Addressing Methodological Issues in
e-Learning Research
Grainne Conole1, Martin Oliver2, Kim Isroff2 and Andrew Ravenscroft3
University of Southampton1, University College
London2, London Metropolitan University3
As a
relative young field, learning technology research suffers in a number of
respects (Conole, 2003). Firstly, the area is not yet clearly defined and
scoped. Secondly, there is criticism of much of the current research
activities, as it is considered too anecdotal, case-based, and lacking theoretical
underpinning. Thirdly, as indeed is true in social science research more
generally, there are divided views on the importance of quantitative versus
qualitative research methods.
This paper
focuses on the first two of these criticisms. Firstly, an overview of the area
will be provided, in response to criticisms of definition and scope. Secondly,
a critique of the methodological issues will be given and in particular which
methodologies are favoured and why. Thirdly, the criticisms that the area suffers
from a lack of theoretical underpinning will be discussed. Finally, the paper
concludes by arguing that learning technology research should be articulated
within a broader and more holistic socio-cultural framework. The paper provides
an overview for the symposium and is followed by three case study papers which
take forward these issues by critiquing specific areas and applications of
learning technology research.
Every
discipline has its own epistemological beliefs and associated culture. This
consists of three inter-related features. Firstly, the area is shaped by the
nature of the research questions being explored. Secondly, it is shaped by the
research methodologies adopted and the ways in which these are used to address
the research questions. Thirdly, it is shaped by the underpinning theory and
theoretical perspectives adopted and used to try and making sense of research
findings.
Learning technology research, methodological issues
Learning technology research in general is concerned with understanding how technology can be used to support learning and teaching. It involves an underlying motivation of improving the student learning experience, as well as exploring the impact of ICT from individual through to institutional level. The fundamental question, perhaps is how can technologies be used to enhance learning? Other central questions include:
These are just some of the questions being addressed by
researchers in an area, which covers a spectrum of topics grouped around three main themes – pedagogical,
technical and organisational.
The first
research theme is concerned with the pedagogy of e-learning and, in particular,
the development of effective models for implementation, including the
application of learning theory to instructional design and use of technologies. Work is needed on the
development of guidelines of good practice and support mechanisms to develop
the e-learning skills of tutors and students. Another area of research lies in
understanding the nature of online communities and different forms of
communication and collaboration, as well as exploring different models for
online courses.
The second area focuses on technology, both
technical tools and the development of architecture to support different types
of learning, as well as standards to ensure interoperability between systems.
This includes exploring mechanisms for tracking activity online, exploration of
the nature of different types of virtual presence, mobile and smart technology
and the development of context sensitive and tailored learning environments.
The third area is concerned with organisational issues, such as formulating strategies for integrating online courses within institutional structures and the seamless linking of different information processes and systems.
It is worth
reflecting on the research methodologies which are used to explore the research
questions outlined above and also to critique the role of ICT itself in
research methods. The choice of methodologies and the way in which it is
carried out in terms of the data collection and analysis will have a critical
impact on the value and worth of the research findings (Conole, forthcoming).
Choice of appropriate research methods will depend both on the nature of the
questions being considered and on the researchers and associated stakeholders
in the research findings. These groups may have conflicting agendas and are
likely to place different values on methodological approaches. Broadly speaking
there is often a tension between the needs of policy makers and senior
managers, academics and support staff and the students engaged in the learning
process. The first group are much more likely to be interested in potential
efficiency gains and cost effectiveness associated with learning technologies
and will want to see evidence-based practice with comparison of the benefits of
new technologies over existing teaching and learning methods. The latter two in
contrast are more likely to be interested in focusing on improving the student
learning experience.
Not
surprisingly given the feeder disciplines from which learning technology has
emerged (education, psychology, computer science etc), there is a wider
spectrum of research approaches being used to conduct learning technology
research. However, some methods are more prevalent than other which can be
accounted for in terms of the characteristics of the area (see Conole, 2004 for
a more detailed discussion of the characteristics of learning technology as a
discipline). For example action research and evaluation are used extensively,
which can be explained in terms of the importance of linking findings back into
practice and the importance of ensuring stakeholder engagement in the issues
being addressed. In contrast, in the UK as least, there is a strong lobby
against evidence-based practice, because there is a belief that there are
fundamental epistemological issues with using this approach in learning
technology (Oliver and Conole, 2003).
New and
innovative research methodologies are also been developed. There has been an
increase in using ICT for data collection and analysis and a move towards mixed
modal methodologies which triangulate across different approaches. Beetham and
Conole have previously described the importance of the development of a sense
of shared ownership and co-participation in learning technology research, in
part because of the practical pragmatic dimension to the area but also because
of the highly political nature and dependences on stakeholder perspectives (Beetham 2001; Beetham and Conole 2001). They describe a multi-modal
approach to data collection. As a
result of this co-participative approach users were much more engaged with the
process of data collection and had a shared sense of ownership and responsibility.
This meant that a much richer set of data were collected, which were also more
authentic and hence representative of the community.
There are
also increasingly innovative uses of the technology itself as a tool for
research. For example there is now a wide range of software available to
facilitate research; for example online web surveys are increasingly being used
for data collection and various statistical and qualitative packages have
emerged to support data analyses. These tools enable the researcher to focus
less on routine collection and calculation and more on the analysis of outputs
(Conole and Dyke, 2004).
Furthermore,
interactive e-journals are now changing the nature of academic discourse and
the relationship between authors and referees (Hey, 1997, Ingraham, 2000),
similarly specialised gateways such as SOSIG have emerged which help to
categorise resources but raise interesting questions about quality control and
scope. Free academically owned publishing mechanisms such as e-Prints are now
becoming more respected and referenced within the research community, but how
does this impact on the future of more traditional journals?
However the increase harnessing technologies for research raises a series of questions, such as what are the particular problems posed in terms of validity, representative-ness, inclusion/exclusion and ethics?
Although the
theoretical position of the researcher is fundamental to their interpretation
of data, this relationship remains largely unexplored within the context of
e-learning research and evaluation (Oliver & Harvey, 2002). Where this
relationship is explored, it is often to lament the lack of scientific rigour
(Mitchell, 2000). Importantly, criticisms such as this focus upon method
(techniques) but remain silent on methodology (the ‘science’ of method,
implying a commitment to a theoretical or philosophical position).
Although similar
criticisms can be made of the wider field of educational research (e.g. Usher,
1996, who criticises its lack of reflexivity), what is distinctive about
e-learning research is that awareness of the problem is generally lacking.
There are some notable exceptions, however. Hodgson et al (2001), for example, note the general vagueness over methodology
in much published research and a tendency towards objectivist perspectives on
e-learning. Such work largely relies of naïve measures of effectiveness such as
exam score difference or Likert-scale expressions of satisfaction. They also
question whether the adoption of research methods designed for face-to-face
settings are appropriate to use online, illustrating (for example) how new
approaches have developed for the analysis of asynchronous computer-mediated
communication (CMC). Likewise, McConnell (2001) explores the relationship
between technology and methods. He observes, for example, that research that
uses CMC seems to encourage participant engagement, whilst that which uses
technology to distribute research instruments diminishes it; that transcripts
automatically arising from networked learning can be used as a stimulus for
discussion in interviews; and that data arising from CMC can be analysed using
traditional methods such as grounded theory or ethnography. However, Jones
(2001) questions whether the assumptions of ethnography require modification
when it becomes ‘virtual ethnography’. He notes the opportunities for
misunderstanding, but
concludes that the ethnographic emphasis on participation is important if the
researcher is to appreciate what this experience is like for their research
participants.
The impact
of such discussions are significant, if largely unobserved. For example, in America, particular theoretical
positions have now been enshrined in law. The “No Child Left Behind” act and
the associated National Research Council report place explicit value on
experimental studies within education. Taking an activity theoretic
perspective, this situation introduces conflicts into current systems of
research. Specifically, in Activity Theoretic terms (Issroff and
Scanlon, 2002), a contradiction arises
between the rules of the community and those of individual subjects. This is
deliberate: by sanctioning (and funding) certain kinds of research at the
expense of others, the government is creating problems for researchers whose
theoretical positions differ from their own. Specifically, researchers working
within a relativist (typically, qualitative) paradigm are disadvantaged by this
development, whilst those with an objectivist perspective (typically,
quantitative) are advantaged. This legislation provides them with new tools (permissions and
funding) allowing more opportunities for research activity than was previously
the case. Such governmental sanctioning of theory via method has been challenged
as being philosophically, socially and morally inappropriate (Oliver &
Conole, 2003).
An explicit
example of this can be seen in research on computer-supported cooperative
learning (CSCL). In Issroff (1993)
guidelines for research are set out. Research within CSCL started in
developmental psychology and initially used rules, tools, divisions of labour
etc developed by that community. Over the last ten years there has been a shift
in research in the area towards naturalistic learning settings with an
emphasis on practical applications and use. This shift has led to a change in
the research methodology used. Issroff’s guidelines reflect this shift. One of
the contradictions which led to this change was that research on the use of
technology in artificial settings led to expectations and findings which then
were difficult to translate into information that was useful in real learning
settings. The realisation that context and other features of the setting make a
difference led to a change in the rules of practice for this community. One
example of this is the way in which students are put into pairs/groups.
Theories of cognitive conflict led researchers to create collaborating pairs of
students who were matched by their conceptual knowledge (either the same or
different). This theoretically-driven
methodology was very fruitful in terms of understanding the role of conflict.
However, researchers realised that other factors also impacted upon
collaboration – for example, friendship. This led to a contradiction between
the tool (the research methodology) and the object (to carry out research that
is relevant to practice).
The
implication of these examples is clear. If, as Hodgson et al (2001) argue,
theoretical commitments remain vague in e-learning research, problems will
continue to occur. Inappropriate inferences will be drawn from published papers
and interdisciplinary teams will encounter problems attempting to undertake
research. Two developments are possible in response to this. Either a single
position will be advocated (as in the US), causing problems for any ‘deviant’
who believes differently, or else all e-learning researchers will have to be
encouraged make their position clear, so that differences can be understood and
respected (for example, by judging each piece of research on its own merits).
Although this latter option is likely to be harder to achieve, it is doubtless
the more desirable, since such community-led revision of acceptable rules of
practice through critique and discussion represents an important opportunity
for social learning within the field.
To conclude, in order to address the criticism that learning
technology research lack scientific rigour, we argue that learning technology
research needs to be articulated within a broader and more holistic
socio-cultural framework for cognitive change (Ravenscroft, 2002, 2003a). There
needs to be a clear conjoining of research into socio-cognitive and
socio-cultural aspects of the educative process. This ambitious approach will
require explicitly integrate theoretical, empirical and computational (or
design-oriented) approaches and consideration
of the methodologies of cognate disciplines within an ‘educational
socio-cognitive science paradigm’.
In light of
this we argue that in learning technology research we need to ‘think outside
the box’, whilst being systematic and ‘scientific’ about how we do this. As part of this reflection we also need to be
mindful of the use of research findings and in particular its relevance to
policy and practice. We must not forget that learning technology research is a practical and applied
discipline which is contextualised in nature.
The paper has provided an overview of learning technology research and associated methodological and theoretical issues. As part of the symposium the paper is linked to three case studies which explore particular aspects of the general arguments discussed here in terms of their relevance for networked learning. Alsop and Tompsett consider the use of two approaches which have gained considerable popularity in the area in recent years; namely the use of grounded theory and activity theory. Jones considers how quantitative and qualitative approaches can be brought together in a meaningful way, given an example of the use of phenomenography. Finally, Steeples reports on the use of an action/participatory research approach. We would like to build on this initial set of papers and invite others to develop case studies and then report on this as a follow up at the ALT-C 2004 conference in September 2004.
Alexander, S. (2003). Closing Keynote - Communities of Practice: a reflection. ALT 10th International Conference: Communities of Practice, Sheffield.
Beetham, H. (2001). Developing learning technology networks through share representations of practice. Proceedings of the 2001 9th International Symposium Improving Student Learning: Improving Student Learning Using Learning Technologies, Oxford: OCSLD, Oxford.
Beetham, H. and G. Conole (2001). Modelling aspects of institutional developments: culture, infrastructure, expertise. Improving Student Learning Using Learning Technologies Conference, Heriott Watt, University, Edinburgh.
Cicourel, A (1964) Method and Measurement in the Social Sciences. New York: The Free Press.
Conole, G., Ingraham, B., and Cook, J., (2003), Learning technology as a community of practice?, research paper, proceedings of the ALT-C 2003 Communities of Practice Conference, 8-10th September, Sheffield, 102-114.
Conole, G. and Dyke, M. (2003), ‘Educational research using online methods’, in Research, Scholarship and Teaching – changing relationships’, Annual Conference of the Society for Research into Higher Education, Royal Holloway College, London, December 2003.
Conole, G. (2004), ‘The role of learning technology practitioners and researchers in understanding networked learning’, Symposium paper, Networked Learning, Lancaster, April 2004.
Conole, G. (Forthcoming), ‘Research questions and methodological issues’ in From Individual Enthusiasm to Institutional Implementation: A Review of Learning Technology in Post Compulsory Education' , Seale, J. (ed), Swets and Zeitlinger, NL.
Erickson, F. & Gutierrez, K. (2002) Culture, Rigor, and Science in Educational Research. Educational Researcher, 31 (8), 21-24.
Glaser, BG & Strauss, AL (1968) The discovery of grounded theory. London: Weidenfield and Nicholson
Goodyear, P., Asensio, M., Jones, C., Hodgson, V., and Steeples, C. (2003) Relationships between conceptions of learning, approaches to study and students’ judgements about the value of their experiences of networked learning. ALT-J 11 (1) 17 – 27.
Hey, J. (1997), ‘E-journals for Research: the User Perspective’, Serials 10(1):65-68.
Hodgson, V., Watland, P. & Asensio, M. (2001) Researching Networked Learning. In Rust, C. (Ed), Proceedings of the 2001 9th International Symposium, Improving Student Learning, 336-347.
Ingraham, B. (2000), ‘Scholarly rhetoric in digital media’, Journal of Interactive Multimedia Education, http://www-jime.open.ac.uk
Issroff, K. (1993) Methodology for research in computer-supported cooperative learning. in PEG 93: AI Tools and the Classroom: Theory into Practice, Moray House Institute of Education, Scotland
Issroff, K. & Scanlon, E. (2002) Using technology in Higher Education: an Activity Theory perspective. Journal of Computer Assisted Learning, 18 (1), 77-83
Jones, C. (2001) What ethnography sees: networked learning as a context for action. In Rust, C. (Ed), Proceedings of the 2001 9th International Symposium, Improving Student Learning, 348-358.
Jones, C. and Boxham, S. (2001) Networked Legal Learning: An Evaluation of Student Learning Experience. International Review of Law Computers & Technology. 15 (3) 317 – 329
Laurillard, D. (2003). Realising the Vision for E-Learning. ALT 10th International Conference: Communities of Practice, Sheffield.
Lynch, M. ( 1991) Method: measurement – ordinary and scientific measurement as ethnomethodological phenomena. In G. Button. Ethnomenthodology and the Human Sciences. Cambridge: Cambridge University Press.
Levy, P (2003) A methodological framework for practice-based research in networked learning. Instructional Science 31 (1-2), 87-109
Marshall, J (1999) Living life as inquiry. Systemic Practice and Action Research 12 (2), 155-171
McAlister, S, Ravenscroft, A & Scanlon, E. (2003). Combining interaction and context design to support collaborative argumentation in education, CALRG Technical Report, Institute of Educational Technology, UK.
McConnell, D (2002) Action research and distributed problem-based learning in continuing professional development. Distance Education 23 (1), 59-83
McConnell, D. (2001) Researching networked learning: issues arising from the use of a variety of different research methods. In Rust, C. (Ed), Proceedings of the 2001 9th International Symposium, Improving Student Learning, 348-370.
McNiff, J (2002) Action research: principles and practice. 2nd edition. London: Routledge Falmer
Mitchel, P. (2000) The impact of educational technology: a radical reappraisal of research methods. In Squires, D., Conole, G. & Jacobs, G. (Eds), The Changing Face of Learning Technology.
Available online: http://www.uwp.co.uk/book_desc/mitchell.pdf.
Oliver, M. & Conole, G. (2003) Evidence-based practice and e-learning in Higher Education: can we and should we? Research papers in education, 18 (4), 385-397.
Oliver, M. & Harvey, J. (2002) What does 'impact' mean in the evaluation of learning technology? Educational Technology & Society, 5 (3), 18-26. http://ifets.ieee.org/periodical/vol_3_2002/v_3_2002.html
Ravenscroft, A. (2002). Communities, communication and cognitive change: social processes and designing engaging e-learning discourse, in Driscoll, M. & Reeves, T. C (eds.) Proceedings of E-Learn 2002, Association for the Advancement of Computing in Education (AACE), Norfolk, USA, pp. 792-797.
Ravenscroft, A. (2003a). From conditioning to highly communicative learning communities: Implications of 50 years of research and development in e-learning interaction design. Association for Learning Technology Journal (ALT-J), accepted, to appear in Vol. 11, No. 3.
Ravenscroft, A. (2003b). Towards highly communicative e-learning communities: Developing a socio-cultural framework for cognitive change. To appear in Land, R & Bayne, S. (Eds.) Ideas in Cyberspace Education. Routledge. (in press).
Salmon, G (2002) Approaches to researching teaching and learning online. In C Steeples and CR Jones (Eds) Networked learning: perspectives and issues. London: Springer-Verlag
Sohng, SSL (1995) Participatory research and community organizing. Working paper presented at the New Social Movement and Community Organizing Conference, University Of Washington, Seattle, WA. November 1-3., 1995 http://www.interweb-tech.com/nsmnet/docs/sohng.htm (Accessed 11th August 2003)
Steeples, C (2003) Networked Learning Environments: Continuing Professional Development and the Creation and Use of Multimedia Artifacts. Unpublished doctoral thesis. Lancaster: University of Lancaster
Usher, R. (1996) Textuality and reflexivity in educational research. In Scott, D. & Usher, R. (Eds), Understanding Educational Research, 33-51.
Winch,P (1958) The Idea of a Social Science. London:
Routledge &Kegan Paul