Networked Learning Conference 2004

NLC2004 /Proceedings / Individual Research Papers

The Pedagogies of e-Learning

 

Wassila Naamani Mehanna

University of Cambridge,

wm212@cam.ac.uk

 

ABSTRACT

There is a great need for empirical examinations of e-learning environments and pedagogical assessments of online interactions and discussions. This paper reports part of the findings from a multi-case study that has analysed online conferences of four different postgraduate programmes in four different UK universities. A model of 29 pedagogical behaviours emerged from the one-year grounded analyses and then applied in analysing additional 5 million words of online interactions. The findings suggest four clusters of pedagogies correlated with students’ grades; however, the effect size calculation reveals an educational significance for all of them. This indicates if they were to be employed in online classrooms they are likely to enhance students’ learning and outcomes.

 

Keywords

Pedagogies, e-learning, online conferences, learning outcomes, content analysis, coding.

“Without appropriate pedagogy, use of high capacity communication services cannot provide significant improvements in learning outcomes.  In general it is the pedagogy that provides for learning, not the technology or the software.” (Carr, 1999, as quoted in Stephenson 2002, p. 53)

 

Technology and pedagogy: how much do we know?

The literature review revealed a struggle to link the potential of online technology with what ideally online education should deliver in terms of students’ learning and achievement.

In their review of 100 published research reports completed in the period 1991-2001, Coomey and Stephenson (2001) found little if any definitive evidence of the overall effectiveness of ‘e-learning’ compared with more conventional methods. This is not to say that this medium is ineffective but rather to say that there is little systematic and empirical work to show evidence of its evaluation. 

The vast and growing bulk of information now available about e-learning, both in print and online has focused on the potential of technology or the enthusiasm of its users and lacked the pedagogical guidance stemming from research that would inform the processes of online course development, review and moderation.  In accordance with this Goodyear (2001) noted: “the literature on learning in higher education is surprisingly quiet with respect to what both lay people and practitioners might expect to be a key construct” (page 62).  Finally, Steeples and Jones (2002) reported: “the big lesson about technology and learning from the 20th century is that less is known about how people learn than many educational researchers are inclined to admit” (page xiv).

 

Technology and pedagogy: how to go?

Boettcher (1997) argued: “Now that the World Wide Web is providing a whole new context for teaching and learning, we have the need to return to the core principles of teaching and learning, and create a new model of teaching and learning.” Therefore, if technology is applied in conjunction with pedagogical concepts, it can create an effective student-centered environment and enhance learning outcomes.

The key issue revolve around the nature and the components of effective teaching and learning and is pointing towards establishing the pedagogies of e-learning or e-pedagogy and presenting recommendations for the e-pedagogues. It became more evident that there is a great need for a pedagogical assessment of online conferences and discussions.

This dearth of content analysis is due to the time required to perform such analyses (Hara, Bonk and Angeli, 2000) and researchers still lack a reliable instrument or an analytical framework to analyse the online discussions. Goodyear (2001) noted: “Analyzing the content of networked learning discussions is a troublesome research area and several commentators have remarked on the difficulty of connecting online texts to discourse to learning” (Page 62).

 

Theoretical framework for the content analysis of online classrooms discussions

The need for specific and comprehensive categories with which to organise the research on online pedagogy materialized as a must and the analysis or the coding of online discussions needs to be developed to show evidence of effective pedagogies.

While the new technologies eventually may lead us to develop new understanding of effective pedagogy that are specific to these learning context, perhaps a good place to start is to look for well-established pedagogies that extant research tells us are effective in other teaching and learning contexts and are well tied with students’ outcomes. However, a researcher trying to collect facts about what research has found about pedagogy is faced with a daunting task. For example, as cited in Marzano (1998), Hattie, Biggs and Purdie (1996) identified over 21,000 studies that one would have to consult for a comprehensive literature review on the factors that affect students’ outcomes and achievement in education. Therefore, looking for effective pedagogies in the meta-analyses research seems a sensible starting point.

This research has therefore built on Marzano’s (1998) meta-analysis which empirically tested nine pedagogical strategies with effect sizes (ES) “showing evidence of their effectiveness in enhancing students’ achievement for all students in all subject areas at all levels” (Op Cit, page 4)

His theory-based meta-analysis posited the interaction of four aspects or systems of human thought operating in most, if not all situations. Those systems are:

·  The self-system processing of presenting tasks. This system contains a network of interrelated beliefs that enable one to make sense of the world (Markus and Ruvulo, 1990) and processes that evaluate the importance of the presenting task relative to a system of goals and assesses the probability of success relative to the individual’s beliefs.(Harter, 1980; Garcia and Pintrich, 1991, 1993, 1995). If the presenting task is judged as important and the rate of success is high, positive affect is generated and the individual is motivated to engage in the presenting task.

·  The use of task-related knowledge. This system is comprised of the information, mental processes, and  psychomotor processes that are specific to a subject matter (Ajzen, 1985; Ajzen and Fishbein, 1977, 1980; Ajzen and Madden, 1986).

·   The cognitive processing of tasks is responsible for the effective processing of the information essential to the presenting task (Anderson, 1995). This system acts on an individual knowledge base (Lindsay and Norman, 1977) and can be organised in four categories: storage and retrieval, information processing, input/output, and knowledge utilisation (Marzano, 1998)

·  The meta-cognitive processing of tasks controls any and all aspects of the knowledge and the cognitive system (Sternberg, 1977, 1984a, 1984b, 1986a, 1986b; Schank and Abeleson, 1977). To this extent, this system has been described as responsible for the “executive control of all processes” (Flavell, 1979, 1987; Brown, 1978, 1980). In other words, it is the “engine for learning” (Marzano,1998)

From those four systems, Marzano (1998) teased out nine specific pedagogies that were further researched in a separate meta-analysis (Marzano, 2000). Based on their effect sizes Marzano recommended their use by all teachers in all subject areas. They are:

For example, a student watches a demonstration on how air flows over the wing of a plane. Later, he applies this in hypothesing that the changing shape of wings in a specific way will have a specific effect on the flow of air, designs a wing with desire shape and test his conjecture. ( Marzano, 1998)

 

Although, the effect sizes of Marzano’s pedagogies strongly correlated with students’ learning and improvement, it was found that they were initially too broad to capture the subtleties of pedagogic activities.

Using a grounded approach in the analysis of one million words or an academic year of online discussions,  twenty-nine more specific pedagogic behaviours have emerged to encompass all the online interactions and were categorised into three distinct groups to form a coding schedule or a Model for Analysing the Pedagogical Content of Online Discussions (MAPCOD).[1] This model includes:

For added rigour, this model was used in the analysis of 11 classes of three different postgraduate programmes, represented by 11 tutors, 160 students and a totality of 4,924,197 words of online discussions. Evidence of all existing categories was there and no new categories have emerged. This has satisfied the saturation required by the grounded theory.

 

Data preparation and coding

The whole process of coding and analysis was iterative and it started by sorting all online interactions according to individuals’ postings. Each student and each tutor had a file.  This data was then converted from HTML into text files in order to load it in NViVo. The researcher went over the transcript sentence by sentence and coded according to the existing 29 categories. Sometimes it was noticeable that one sentence could have double coding and overlapped between two categories.

Once the coding was done. Each student and tutor had a summary sheet that included the occurrences or frequencies for each one the pedagogies. Inter-raters reliability checks showed 97% agreements.

The frequencies of those pedagogies along with students’ grades were entered in SPSS for statistical compilations. To unify the measure, all frequencies were rescaled into percentages which facilitated the comparisons between students within the same group and across groups.

For triangulation, online questionnaires and interviews were administered to solicit students’ and tutors’ accounts with regard to the use of those 29 pedagogic behaviours and their frequency within their online interactions.

 

Statistical Findings

Correlations were calculated for each student’s grade and the total frequencies or Σ of the 29 pedagogic behaviours employed in his/her entire work as well as those contributed by the tutor.  Strong and significant correlation (r =.328** and p <.0001) was observed.

Later, the 29 categories were re-grouped into 9 clusters to fit in Marzano’s original pedagogies and their effect size has been calculated[2]. This regrouping was done to test Marzano’s theory that they work for all classrooms and all age groups. Although, four clusters correlated with students’ grades, the effect size (ES) for each cluster of pedagogies showed evidence of educational significance. According to Cohen’s (1998) classification of ES,

a)      Similarities and differences (for example, all utterances in which Tutor and/or Students analyse two or more elements in terms of their similarities and differences) ES = 0.24

b)      Recognising efforts (Utterances where tutor gives feedback on students’ progress or presented work, praising students’ work and when students give  feedback to colleague’s work and when they also praise their colleagues’ work) ES = 0.2

c)      Cooperative Learning ( Utterances where tutors assign group work, students share note and ideas, ask questions or request help, help each other, add to one another arguments, also when tutor interjects and adds to the argument or to the content of the topic under study, and evidence or recalling prior knowledge) ES = 0.3

d)      Setting objectives and feedback ( Utterances where tutor sets objectives, gives directions ,instructions and or updates to reach that objectives, help students with the content, adds to their arguments and gives feedback and/or  praise to work) ES = 0.34

e)      Generating and testing hypotheses. (Refers to all utterances where students show evidence of setting and generating hypotheses, presenting ideas and/or arguments and where students show evidence of defending an argument set earlier and being justified) ES = 0.2

a)      Summarising and note taking (Utterances in which Students collect, review and summarise notes or re-present them after an analysis to decide what is most important to be included) ES = 0.62

b)      Homework and practice ( Utterances providing evidence of Homework and practice. This could be identified as students’ presenting assignments and applying knowledge in their work) ES = 0.73

 

a)      Nonlinguistic presentations (Utterances were students show evidence or presenting their ideas in a non-linguistic form. Graphs, charts, tables, maps, illustrations, etc ) ES = 1.08

b)      Questions and cues (Utterances where tutors use questioning to help students activate their prior knowledge and by also  providing students with help or feedback in the form of questions hints and cues) ES = 1.01

Determining the significance of a particular effect size requires interpreting what the size of the effect really means. While recent recommendations (Wainer and Robinson, 2003) were made that the Null Hypothesis (NH) could support the calculation of the effect size in determining its practical importance, it was found that this is not applicable in all cases and several criticisms were apparent due to the misuse and misinterpretation of the NS (Cohen, 1990, 1994; Rosenthal, 1991; Thompson, 1994).   An alternative for interpreting the value of ES or its educational significance is “to think meta-analytically” (Thompson, 2002b).

Thompson prefers a model where effect sizes from individual studies are interpreted in the context of previous studies, let alone a meta-analysis in our case. This was an additional reason behind the regrouping of the 29 pedagogic behaviours into Marzano’s original 9.  Comparisons of ES can be easily made in appendix 2.

Further correlations were sought for each cluster of pedagogies and individual students’ grades.

The following was observed:

 

Conclusion and Implications

The research has purposely focused on the pedagogic aspect of e-learning in higher education and a “Model for Analysing the Pedagogical Content of Online Discussions (MAPCOD)” has emerged as a reliable tool that could be implemented in the content analysis of online discussions and could provide guidance to e-pedagogues and course designers.

Findings suggest a number of pedagogies that are worthy of including in the online teaching and learning processes due to their theoretical and empirical backing and link with established learning systems.

Empirical evidence of their correlations with students’ learning and outcomes perhaps would encourage online tutors to adopt the proposed pedagogies and use the devised model as a blue print for analysing the content of their course part of an action research in order to enhance the quality of their courses.

 

Appendix 1: Model for Analysing the Pedagogical Content of Online Discussions (MAPCOD)

No.

Code

Pedagogic behaviour

Definition for Each Category

1

SumNote

Summarising notes and note taking and/or summarising ideas

Refers to all utterances in which Tutor and/or Students collect, review and summarise notes or re-present them after an analysis to decide what is most important to be included.

2

CompTrst

Identifying similarities and differences

(comparing and contrasting)

Refers to all utterances in which Tutor and/or Students analyse two or more elements in terms of their similarities and differences

3

Metafors

Identifying similarities and differences

(using analogies & metaphors)

Refers to all utterances in which Tutor and/or Students analyse two or more elements in terms of their similarities and differences however by creating metaphors and analogies to present them.

4

HWPract

Practice & application of knowledge in Homework.

Refers to all utterances providing evidence of Homework and practice. This could be identified as students’ presenting assignments and applying knowledge in their work

5

Nonling

Non-linguistic Presentations

Refers to all utterances were students show evidence or presenting their ideas in a non-linguistic form. Graphs, charts, tables, maps, illustrations, etc are looked for.

 

 

Students

 

6

Stobject

Students setting objectives

Refers to all utterances where there is evidence of students setting their own learning and course objectives.

7

Mtobject

Students meeting objectives

Refers to all utterances where there is evidence of students meeting set objectives. Those objectives could be their own objectives as well as course objectives.

8

St-share

Students share notes

Refers to all utterances where there is evidence of students exchanging notes and sharing information.

9

Stfeedbk

Students share feedback

Refers to all utterances where there is evidence of students responding to colleagues’ messages/posting either by agreeing with what has been said, praise, or any relevant comments about their colleagues’ work and posting.

10

Sthelpst

Students on this course help each other (directions)

Refers to all utterances where there is evidence of students helping each other. This help could be in the form of answering questions, guiding or facilitating the execution of a task.

11

Hyposet

Students setting hypotheses/presenting arguments

Refers to all utterances where students show evidence of setting and generating hypotheses, presenting ideas and/or arguments.

12

Hypotst

Students testing hypotheses/justifying arguments

Refers to all utterances where students show evidence of defending an argument set earlier and is being justified.

 

13

Stargue

Students building/adding one each other argument

Refers to all utterances where there is evidence displayed that students are building on each other arguments.

14

Stquest

Students asking questions

Refers to all utterances where there is evidence of students are asking questions. Those questions could be for clarification purposes, or for asking ‘how to do’ or ‘what to do’, etc. and they could be addressed to tutor and peers.

15

Strecall

Students recall of prior knowledge

Refers to all utterances where there is evidence of students recall on prior knowledge. It could be in the form of summarising points of previous posted messages or referring each other to a concept or an idea taught and/or discussed earlier.

16

Stpraise

Students providing recognition

Refers to all utterances where there is evidence that the students’ feedback to each other is showing statements of praise and encouragement such as ‘well done’, ‘very good’, ‘good work’, etc.

 

 

TUTORS

 

17

T-quest

Tutors asking questions

Refers to all utterances where there is evidence the tutor is asking questions.

 

18

T-direct

Tutor giving directions

Refers to all utterances where there is evidence the tutor is telling students ‘how to do’ things that facilitate their tasks, assignments and learning.

 

19

T-instr

Tutor giving instructions

Refers to all utterances where there is evidence the tutor is telling students ‘what to do’ ie, assignments, deadlines, rules of posting, group work.

20

T-hints

Tutor gives hints and cues

Refers to all utterances where there is evidence the tutor feedback to students about is  in the form of hints and cues which facilitate students’ completion of tasks and/or understanding concepts.

 

21

T-recall

Tutor calling upon prior knowledge

Refers to all utterances where there is evidence the tutor is calling upon students’ prior knowledge. They could be ideas, concepts, theories, events and any curricular material and information with a view to stimulating recall of students.

22

T-feedbk

Tutor feedback on progress  reinforcing effort.

Refers to all utterances where there is evidence that the tutor’s feedback is showing statements acknowledging students’ efforts and capabilities in achieving tasks.

23

T-praise

Tutor providing recognition

Refers to all utterances where there is evidence that the tutor’s feedback is showing statements of encouragement and praise such as ‘well done’, ‘very good’, ‘good work’, etc.

24

T-contnt

Tutor’s addition to content

Refers to all utterances where there is evidence that the tutor’s feedback is in the form of further addition and or clarification to the course content.

25

T-expln

Tutor explains purpose of module, work requirements. Or clarifying relevant students’ questions

Refers to all utterances where there is evidence that the tutor has clear explanations of the course material, requirements and expectations and also clarifying students’ questions.

26

T-set-obj

Tutor sets objectives for students

Refers to all utterances where there is evidence that the tutor’s feedback is showing statements setting further learning objectives for students. These objectives could be  individual (addressed to one student) as well as collective (addressed to members of one group).

27

T-updates

Tutor announces updates

Refers to all utterances where there is evidence that the tutor keep students posted with any relevant updates. i.e, changes in schedule, changes in dead lines, as well as her/his availability.

 

28

Tasgnwrk

Tutor assigns groups & work

Refers to all utterances where there is evidence that the tutor assigns students’ groups (collaborative work) where students are expected to meet common objectives. ie, a project, an assignment, group report, etc.

29

T-addarg

Tutor builds/adds to students argument

Refers to all utterances where there is evidence that the tutor’s feedback is showing statements that builds on students’ arguments.

 

Appendix 2: Theoretical Links of the Pedagogies and their Effect Size


 


REFERENCES

Boettcher, J. (1997) (Florida State University) Pedagogy and Learning

Strategies. Downloaded from the Web (http://www.csus.edu/pedtech/learning.html)

Cohen, J. (1988). Statistical Power Analysis for the Behavioral Sciences, 2nd ed. Hillsdale, NJ: Erlbaum.

Cohen, J. (1990). Things I have Learned (so far). American Psychologist, Vol. 49, pages 997-1003.

Cohen, J. (1994). The earth is round (p < 0.5) American Psychologist, Vol. 45, pages 1304-1312.

Coomey M and Stephenson, J (2001) Online Learning: it is all about dialogue, involvement, support and control—according to the research, in Teaching and Learning Online: Pedagogies for New Technologies, ed. J. Stephenson, Kogan Page, London, UK

Goodyear, P (1995) Asynchronous Peer Interaction in Distance Education: The  Evolution of Goals, Practices, and Technology. Training Research Journal.  Vol. 1, pp. 71-102

Goodyear, P (2001) Psychological Foundations of Networked Learning in Jones, C and Steeples, C (Eds). Networked Learning: Perspectives and Issues Springer, Godalming, Surrey, UK

Hara, N,  Bonk, C J and Angeli, C (2000) Content Analysis of Online Discussion in an Applied Educational Psychology Course. Instructional Science.  Vol. 28, pp. 115-152

Laurillard, D. (1993) Rethinking University Teaching. Routledge: London

Laurillard, D. and McAndrew (2003). Reusable educational generic software: a basis for generic learning activities. In  Littlejohn, A. (2003) Reusing Online Resources: A sustainable Approach to e-Learning. Kogan Page Limited, London

Marzano, R. (1998). A Theory-Based Meta-Analysis of Research on Instructions. Office of Educational Research and Improvement (OERI). U.S. Department of Education.

Marzano, R. (2000). What Works In Classroom Instructions. USA: Mid-Continent Research For Education and Learning.

Networked Learning Conference 2002.

http://www.learn.uhi.ac.uk/conferences/networked_learning_conference_report.pdf.

Rosenthal, R. (1991). Meta-Analysis Procedures for Social Research. Thousands Oaks, CA: Sage

Steeples, C. and Jones, C. (2002). Networked Learning: Perspectives and Issues. London, Springer-Verlag London Limited

Stephenson, J ( 2002). Teaching & Learning Online: Pedagogies for New Technologies. Kogan Page Limited, London

Thompson, B. (1994).  The Concept of Statistical Significance. (Report No. EDO-TM-94-1). Washington, DC: Office of Educational Research and Improvement.

Thompson, B. (2002b). What Future Social Science Research Should Look Like: Confidence Interval for Effect Sizes. Educational Researcher. Vol. 31, No. 3, pages 24-31

Wainer, H. and Robinson, D. (2003). Shaping Up the Practice of the Null Hypothesis Significance Test. American educational Research association. Vol. 32, No. 7, pages 22-29


[1] Appendix 1

[2] Appendix 2 shows the theoretical link of those 29 categories, the re-grouping into 9 clusters and their effect sizes.