Networked Learning Conference 2004

NLC2004 /Proceedings / Individual Research Papers

Determining Research Questions in e-learning


 Patrick McAndrew, Andrew Brasher and Pascale Hardy

The Open University

{P.McAndrew, A.J.Brasher, P.Hardy}



In this paper we consider the future research direction for e-learning in relation to the needs of higher education. E-learning is recognised as a relevant method for the provision of learning across all areas of society and is considered of strategic importance at government level (HEFCE, 2003; DfES, 2003). In setting such strategies there should be sound knowledge of what approaches are effective and significant for the take up of e-learning. However, this would ignore the indications that there are weaknesses in the fundamental research that can inform such knowledge. Methods from other forms of research, such as comparative studies, often overlook the complexity of the learning process and the inevitable lack of impact from changes to methods of delivery. A critical review of the problems in supporting e-learning based on collation of expert views was carried out within a project, funded by the EU (eLearnTN, 2003). The results from that work led to proposed key research areas and research questions that need to be addressed as e-learning is introduced into existing or novel contexts. In particular a division can be made into research needs for work on measures of effectiveness, representation of effective learning designs, and methods to support knowledge sharing.



e-Learning, Roadmap, Delphi survey, research issues, policy.



In late 2002 a short project was established to bring together researchers involved in the development of distance learning using new technology to consider a collective view on what were the research needs for the area of e-learning in Higher Education. The intention was not to solve the research issues but to identifying them. A key objective of the project was to deliver a “roadmap” setting out the direction of future research for this area.

The scope for the study was e-learning in Higher Education but many of the results are motivated by the aim of exploiting the potential of Networked Learning. This can be illustrated by one example that emerged during the study where a low content approach to learning was developed for a Masters level course. The course followed sound design principles of networked learning to ensure support for learner involvement and exchange of ideas, however the resulting course proved unsustainable owing to the cost in time and demands upon tutors and staff. This example and others provided a motivation for the research agenda; what needs to be known in order to help an organisation plan and run such courses successfully? And, how can this knowledge be shared?



Technology roadmapping originally developed as a research and development management and foresight methodology in industry since the mid-1980s. Robert Galvin (Galvin, 1998), former Motorola chairman and prominent advocate of Science and Technology roadmaps, offered this definition:

"A 'roadmap' is an extended look at the future of a chosen field of inquiry composed from the collective knowledge and imagination of the brightest drivers of change in that field. . . . Roadmaps communicate visions, attract resources from business and government, stimulate investigations, and monitor progress.  They become the inventory of possibilities for a particular field."

Therefore, a roadmap provides a consensus view or vision of the future landscape available to decision makers.  The roadmapping process provides a way to identify, evaluate, and select strategic alternatives that can be used to achieve a desired objective. The focus for the roadmap can vary depending on whether the aim is to understand what previous research has enabled a successful innovation, a retrospective roadmap, or to look for either technology-push or requirements-pull in the future, prospective roadmaps. These can also be combined in a retrospective-prospective roadmap  to link past developments with a vision for the future.

Roadmaps link strategy to future actions and explicitly incorporate a plan for needed capabilities and technologies to be in place at the right times. For example:

Da Costa and Punie identified as typical two key interrelated emerging functions (Da Costa, Punie 2002). Firstly, the roadmap methodology produces representations of the state of the art of Science and Technology at a certain point in time and of the nature, rate and direction of potential developments. In this way a roadmap can act as a prospective methodology. Secondly, the representation is put to practical use in negotiating the way forward and in informing decisions about possible future options. As such, a roadmap is also a planning methodology: “a traveller's tool that provides essential understanding, proximity, direction, and some degree of certainty in travel planning.” (Kostoff and Schaller, 2001)

Detailed methodologies are available dependent on the aims, timescale and scope of a roadmap, however characteristic actions include:


Project methodology

The initial aim of the project was to prepare a prospective roadmap. The stages that can be used for a typical technology-driven roadmap are:

The methodology adopted was primarily expert-based through workshops and a survey. This process was somewhat paradoxical in that the appropriate expertise must be employed to develop a roadmap, but the appropriate expertise becomes fully known only after a complete roadmap has been constructed. An iterative roadmap development process is therefore recommended (Kostoff and Schaller, 2001).

For selection of the research lines an iterative process was followed from defining the domain and gathering problem issues, through ranking and analysis of the problems, to redefinition of problems into research questions. At each stage expert input was taken through workshops and direct consultation. The process considered an initial division into the three areas of pedagogy, technology, and business. This division proved unhelpful as overlaps occurred and differences in the treatment. A revision of the project view merged technology and pedagogy and considered the business and organisational aspects as background scenarios. An alternative would have been to apply different methodologies to gain a wider perspective.

In the first two workshops developed by the project areas of measuring pedagogic effectiveness, representation, and knowledge sharing emerged as leading concerns. These areas were then reviewed with reference to existing research activities in order to bring out the research questions that need to be considered while trying to find solutions to the problems. In practice one of the difficulties of applying roadmap methodologies lies in the reporting aspect as the process and stages cannot clearly be separated from the results. In this case the reporting was focused on the outcomes as research areas and research lines. This loses some of the detail in the end report but increases its clarity and simplicity and enabled validation through expert feedback in a Delphi survey and institutional endorsement.


Analysis of problems using visualisation

In order to refine of the implications for the research lines first the key aspects of effectiveness, representation and knowledge sharing were mapped against the broader range of problems to aid understanding using a visualisation tool, Compendium (Selvin et al, 2001) as shown in figure 1.

Figure 1: Screenshot of a visualisation of relationships between identified problems, created using Compendium (Selvin et al 2001).

This mapping draws out the inter-relationships between the different areas (i.e. pedagogy, technology, and business), and allows us to provide a tentative model of dependencies between a variety of research topics. Desk research was then carried out to look at existing work and activity in research groups to propose an overall line of action.


Research areas

As a result of these views the distinction between the working groups was reduced and two new tasks were carried out to identify the scenarios and possible use cases for e-learning, and to review and simplify the problem list from the pedagogy area with integration to the technology area. The scenario work was carried out at two levels of detail, first by considering global issues and previous analysis of future directions from the L-Change project (, and a set of scenarios in use with The Open University, secondly by producing a small number of detailed use cases for reference within the project. The refinement of the pedagogic area was guided by the commentaries and expert ranking of problem areas to allow a key problem area to emerge that should be the focus of research efforts. This research area, was called “Effectiveness” and was further analysed and related to other important issues identified as “Knowledge Sharing” and “Representation”. Each was seen as key for various reasons. 

Effectiveness research area

The ability to both measure pedagogical effectiveness and to understand the causes of variations in pedagogical effectiveness is a prerequisite for sophisticated decision making within the sector of higher education. For example managers of educational institutions need to be able to estimate how pedagogical effectiveness changes with variations in resources (time, materials, infrastructure etc.) to effect profitable organisational changes within their institutions. Similarly, the development of novel pedagogies which exploit advances in ICT requires an understanding of the factors which influence pedagogical effectiveness, to allow the developers of such pedagogies to proceed in an informed, logical and efficient manner.  With respect to e-learning there is a need for fundamental research work to produce models of teaching and learning which can help researchers hypothesise causal relationships and design experiments to test these models and relationships. Such research work must inform and form a basis for the specific effectiveness research questions proposed later in this document.

Representation research area

Representation concerns the issue of representing data and processes so that both people and machines within higher educational systems can exploit the process and/or data.  This research area specifies the research needed to advance implementations of learning materials and environments for learning. It focuses on research into practices for representing learning materials and learning process, and development of tools to support those representations so that benefits to universities and/or students are realised.

Knowledge sharing research area

The area of knowledge sharing addresses interrelated aspects of optimising the transfers of organisational knowledge (within and between organisations) and of individual knowledge (within and between both the staff and student populations). The focus of the research proposed is on development and application of knowledge sharing methods to support the ability of universities and learners to work in new ways. This research is not about new ways to work, but about how the development of mechanisms to share useful knowledge that are optimised for the academic community.

A fourth area emerged with a looser theme taking in other aspects that tend to apply at to the organisational and business concerns of higher education. Research in these areas may broaden out to consider aspects beyond technology, such as change management, curriculum issues and the effects of different social and cultural pressures.


Delphi survey

In the Delphi process (Masini, 1993) the objective is to obtain a reliable consensus of opinion from a group of experts that can be used as a future forecast, while at the same time minimising the undesirable aspects of group interactions. The main instrument used is interrogation of experts through a series of questionnaires giving the survey characteristics of anonymity, iteration with controlled feedback, and statistical group responses. The approach has some acknowledged advantages: statistical response that can include the views of outliers as well as centrists, and encouraging anonymous interactions amongst experts. Disadvantages of the method are that it is time-consuming, can be influenced by the moderator, subjective (and hence dependent on the expert selection) and can lack time for secondary analysis of the results (Linstone and Turoff, 1975).

In this study the Delphi process associated separate research lines with each of the four areas and used an online questionnaire designed by Scienter and managed through the Open University’s PRESTO electronic survey system. The response to the first phase of the survey is represented in figure 2. The approach allows expert views to be gathered and comparisons made for both consensus and to identify new issues. Those new issues can then be revisited in a second phase of the Delphi process. A set of prioritised research lines has been produced and disseminated (eLearnTN 2003); however a secondary analysis of the data allows the broader research areas to be reviewed. In the chart the thick lines show the threshold applied to select the 20 most important issues in two categories of coverage, these were reviewed to then see if the concepts had any coherence. The four clusters shown in the chart (by encircled regions with marked centroids) can be seen to characterise the majority of the important research lines. For effectiveness it can be seen to cluster as more important, but fairly well covered; knowledge sharing and representation produce very similar clusters with practically identical overall importance and coverage area, slightly less important than effectiveness, but less work addressing it. There are also two outlying aspects of representation, skills and resources, which clusters with the effectiveness issues and can arguably be seen to overlap with them, and metadata, which is identified as having more research attention. Finally there are a cluster of important other/organisational issues with low coverage, these can be characterised as the change management requirements of educational organisations.

Figure 2: Plot of research lines from Delphi survey importance (y) against coverage (x). Regions: I) high importance/low coverage, II high importance/high coverage, III) and IV) lower importance.


The Delphi process therefore allows earlier ideas to be supported through the expert feedback. The results can be interpreted in different ways but the cluster analysis encourages a view that effectiveness forms a distinct set of research priorities, while knowledge sharing and representation reflect similar overall priority from the community and may therefore be linked more strongly. The emergence of a set of organisational issues as highly important and not being addressed encourages these aspects to be considered as underlying needs. A third dimension to the Delphi survey was urgency. In the event this data was not used as it was highly correlated with importance, reflection on the questionnaire identified the use of a similar ranking scale for urgency and importance; it is possible that using a time related scale for urgency would have enabled greater use to have been made of that dimension.


Discussion of research areas

As an illustration for each of the key research areas we consider some possible links between current developments and the research questions that may lead on from them. Overall there are many possible questions that could be selected and more of these are presented in the reports from the project (eLearnTN, 2003).



The research necessary to progress in the area we have entitled effectiveness is concerned with the development of mechanisms to understand and quantify pedagogical effectiveness, and how pedagogical effectiveness varies in response to changes in resources.

To develop this analysis and hence generate the relevant research questions in this area we assumed that:

Underlying both these assumptions there is a need to understand the factors which affect pedagogical effectiveness.  Existing work points out that this is an extremely complex topic which encompasses a wide range of research domains, see e.g. (Price and Richardson, 2003; Harris 1998; Schönwetter, 2002). However, the project required the identification and specification of research areas and activities which will generate technology based solutions, and so limited the scope of the research activities it proposed.  Thus with respect to the quantification of pedagogical effectiveness, there are known limitations to many of the currently available techniques, e.g.  media comparison studies (Joy and Garcia 2000), which prompts the need for research into technology based mechanisms for improving these techniques. Some examples of the type of research that could contribute to this are investigations into non-intrusive surveillance techniques (e.g. exploiting technologies such as radio frequency identification (RFID) devices and wireless networks) and their application to quantifying aspects of learning in a variety of learning situations (see for further discussion of this topic).



Much of the work on representation has focussed on descriptive aspects, in particular metadata aspects (as identified in the Delphi response). Newer work on representation is now addressing the pedagogic context and the need to include roles and people within the area. A major contribution to this area is the design of the Educational Modelling Language by the Open University of the Netherlands (Hummel et al 2003) and its emergence as a specification as IMS Learning Design (IMS 2003). Learning Design offers a way to describe the pedagogic structure of course materials and separates out the roles, methods, tools and resources needed to create the final learning experience. The use of a formal approach for each of these areas offers the potential to create stores of designs that embody teaching methods. Learning Design itself is intended to be neutral about the pedagogy that it represents; however it is particularly attractive as a specification because it appears rich enough to represent the complexity involved in networked  learning. For example some of the Open University (UK) activities in its course Learning in the Connected Economy have been produced using Learning Design and it has been found possible to formalise activities such as online debates involving tutor roles and distinct learner roles going through several stages with asynchronous and synchronous collaboration. Initial analysis indicates that a relatively small number of such designs would provide valuable templates for use across the organisation. Research questions that will be assisted by such representations include the formative design of e-learning, to see whether an analysis mechanism, for example building on an activity theoretical model of how interactions should take place may allow validation of the structure represented in a learning design, or tracking learner behaviour feedback into the design process.


Knowledge Sharing

Extending the initial work on problem areas has encouraged us to link knowledge sharing with representation. Essentially representation offers a way to codify the learning and lead to a mechanism for exchange of ideas and materials. Knowledge sharing establishes the importance of the community in any implementation. It also brings a strong link with network-based learning by recognising the value of the network as a transfer mechanism not just for materials but broader aspects of knowledge.

The vision described in (Laurillard and McAndrew, 2002) considers such transfer happening through knowledge sharing across the academic community. Tools for sharing are also now becoming more understood and work underway with the will look at the feasibility of incorporating a Learning Design Engine (developed at OUNL) within a sharing environment developed at the OUUK.  Again a key component of such sharing appears to be the effectiveness of each method, earlier work looking at the sharing of educational software (Twining at al, 1998) emphasised the importance of the context of learning and offers a route to indicating the value of any model or design through thorough description of a case study (with negatives as well as positives). An important area of research would be to examine the integration needed to bring together representation of learning design with a sharing of the knowledge of how effective the design may have been in particular contexts. Working in this way it may then be apparent if there is a broader link between effectiveness and design without attempting the use of comparative studies, which have already been demonstrated to be ineffective in many educational contexts.


Conclusions and further work

Development of educational systems and methods are driven by practical, political and technological agenda. These pressures cannot be ignored but need to be balanced by appropriate consideration of knowledge of methods and theories that enable education to meet its needs. A classification into researching effectiveness, representation, and knowledge sharing offers a way to progress many of the problems identified in the fields of organisation, technology and pedagogy in Higher Education. It helps the development of a roadmap by facilitating both the identification of dependencies amongst research tasks, and the communication of concepts within the roadmap by use of this simple vocabulary (effectiveness, representation, and knowledge sharing) to enable simple overviews to be generated. This broad description for research activity in e-learning has been further analysed and aspects of the three research areas have been used to suggest a future research agenda. Examples of specific actions are to research how to consider effectiveness through the points of interaction in models of teaching and learning, e.g. in the models from (Price & Richardson, 2003), or Laurillard’s Conversational model (Laurillard, 2002); to establish an approach to representation, for example building on the work of EML and Learning Design (Hummel et al, 2003); and, to embed that within Knowledge Sharing mechanisms as considered in (Laurillard & McAndrew, 2002).

In networked learning other authors have presented similar views, Czerniewicz looked at the research drivers impacting on lecturers working on Educational Technology in one organisation (e, 2002). She considered a range of drivers for the research from the external environment as needing understanding of benefits, models of use and appreciation of context and interactions. The views of individual lecturers presented in the paper highlighted qualitative effectiveness. A symposium on the policy issues included Roberts who considered how to bring out research issues in the complexity of networked learning (Roberts, 2002), his conclusions were that a cultural-theoretical view might help set a research agenda for networked learning, incorporating representation and propagation of knowledge. Recent work has highlighted the concept of Powerful Learning Environments to bring together aspects from various models in an “eclectic view” (van Merriënboer and Paas, 2003) that encompasses separate theoretical approaches (e.g. acknowledging the role of transmissive elements alongside constructivist learning). Taking the view that learning is a complex system means that separate comparative studies that seek to isolate effects is of limited value. New approaches to research effectiveness in the context of complexity need to be derived, and mechanisms that offer practical assistance to practitioners are needed.

The research reported here has proved to be a catalyst for further action. A presentation of the findings of the project at the EADTU has influenced the agenda for work that will be sponsored by the higher education distance learning community within EADTU and link to the Europe wide Bologna agreement to work towards lowering barriers for learning across Europe. A companion study is also expected to be initiated looking at e-Learning agenda within the UK. This study will use a modified version of the approach applied in this study to inform future research agenda. Our work has shown the risks and value in such an approach; while it can be unwieldy to analyse and difficult to interpret it can also help gain consensus and agreement in complex areas.



The eLearnTN project (IST-37440-2001) was funded by the European Union. The coordinator was Altran SDB and the partners were The Open University (UK), The Open University of the Netherlands, UNED (Spain), UOC (Spain), EADTU, Scienter, and Lambrakis Research Foundation. We thank our colleagues Alan Woodley and Adam Joinson for providing the electronic survey system and Claudio Dondi and Claudio Del Rio of Scienter for managing the Delphi survey and providing us with analysed results. The content of this paper does not necessarily reflect the position of the European Community, nor does it involve any responsibility on the part of the European Community



Czerniewicz (2002) Research in Educational Technology: framing MEG’s Work. Proceedings of Networked Learning 2002 (Sheffield, UK, 26-28 March 2002), 369-377.

Da Costa O. and Punie Y.(2002) Ambient Intelligence in everyday life: a function-oriented Science and Technology Roadmapping project, European Commission, IPTS/ESTO,

DfES (2003) DfES e-learning strategy consultation. 

eLearnTN (2003) eLearning Thematic Network.

Galvin, R (1998) Science Roadmaps, Science, 280, 803.

Harris, A. (1998) Effective Teaching: a review of the literature. School Leadership & Management, 18, 2, 169-183.

HEFCE (2003). Consultation on HEFCE e-learning strategy. 

Hummel, H., Manderveld, J., Tattersall, C. and Koper, R. (2004) Educational modelling language and learning design: new opportunities for instructional reusability and personalised learning. Int. J. Learning Technology, 1, 1, 111-126.

IMS (2003) Learning Design Specification

Jones, C. (2003). Networks and Learning: communities, practices, and the metaphor of networks, Proceedings of ALT-C 2004 Research Proceeding: Communities of Practice, 234-248.

Joy, E. H. and F. E. Garcia (2000). Measuring Learning Effectiveness: A New Look at No-Significant-Difference Findings. Journal of Asynchronous Learning Networks, 4, 1.

Kostoff R. and Schaller R. (2001) Science and Technology roadmaps, IEEE Transactions on Engineering Management,

Laurillard, D. (2001) Rethinking University Teaching: A Conversational Framework for the Effective Use of Learning Technologies, Routledge, UK.

Laurillard, D. and McAndrew P. (2002). Virtual Teaching Tools: Bringing academics closer to the design of e-learning, Proceedings of Networked Learning 2002 (Sheffield, UK, 26-28 March 2002), 11-16.  

Linstone H. and Turrof M. (1975). The Delphi Method, techniques and applications. Addison-Wesley London.

Masini Barbieri E. (1993) Why Future Studies? Grey Seal, London.

Price, L. and J. Richardson (2003). Why is it difficult to improve student learning? 11th Improving Student Learning Symposium (Hinckley, September 2003).

Roberts, G. (2002) Complexity, Uncertainty and Autonomy: the Politics of Networked Learning. Proceedings of Networked Learning 2002 (Sheffield, UK, 26-28 March 2002), 201-209.

Schönwetter, D.J., Clifton, R. A., & Perry, R. P. (2002). Content Familiarity: Differential Impact Of Effective Teaching On Student Achievement Outcomes. Research in Higher Education, 43(6), 625-655.

Selvin, A., Buckingham Shum, S., Sierhuis, M., Conklin, J., Zimmermann, B., Palus, C., Drath, W., Horth, D., Domingue, J., Motta, E. and Li, G. (2001). Compendium: Making Meetings into Knowledge Events. Knowledge Technologies 2001, (Austin, TX, 4-7 March 2001),

Twining, P., Stratfold, M., Kukulska-Hulme, A. and Tosunoglu, C. (1998) SOURCE – Software Use, Re-use & Customisation in Education, Active Learning, 9, 54-546.

van Merriënboer, J. and Paas, F. (2003) Powerful Learning and the Many Faces of Instructional Design: Toward a Framework for the Design of Powerful Learning Environments. In Powerful Learning Environments: Unravelling Basic Components and Dimensions, De Corte, E., Verschaffel, L., Entwistle, N. and van Merriënboer, J. (Eds), Pergamon, Oxford, UK. 3-20.