Networked Learning Conference 2004

NLC2004 /Proceedings / Individual Research Papers

Carrying out Research into Learning Through on line Discussion: Opportunities and Difficulties

 

 Michael Hammond and Mongkolchai Wiriyapinit

University of Warwick

m.hammond@warwick.ac.uk, m.wiriyapinit@warwick.ac.uk

 

ABSTRACT

This paper reports on approaches to researching on line discussion. The context is an extended study of an         E business module at the University of Warwick in which the use of communication technology is pivotal. The module requires students to tackle case based activities within on line groups and gives opportunities to explore on line course materials. A study of student learning within this module was undertaken through questionnaire surveys, text analysis and interview. This paper reports on the value of these different approaches to research and highlights the value of a mixed or triangulated approach to the research of on line discussion.

 

Keywords

e-learning, e-tutoring; relationships between research, theory, policy and practice; methodologies for researching NL

 

INTRODUCTION

This paper reports on researching on line discussion. The context is an E business module in which students tackle case based activities within on line groups and access on line course materials. The module lasts for 18 weeks with a two-week introductory period followed by group work and assignment topics. In the first year of our study (2001) there were 46 students taking part, from 17 countries. Communication took place via asynchronous discussion and occasional chat within the conferencing software WebBoard. (For more background see Hammond and Wiriyapinit, 2002.)

We undertook a study of student learning within this module through questionnaire surveys, text analysis and interview. This highlighted the value of a mixed or triangulated approach to the research of on line discussion. The  paper needs to be seen in the context of a widespread concern for message analysis in the reporting of online conferencing but relatively little discussion of the strengths and weaknesses associated with this approach.

 

Our STUDY

Our study set out to ask two central questions about on line discussion within the course: did students feel learning had taken place and, if so, how did it take place? We were keen to reflect on our research methodology as much as to report on findings. In addition to one of the author’s participation in the conference we relied on three research strategies: on line questionnaire surveys; text analysis and interviews.

 

On line questionnaire surveys

Students were asked about their general expectations before the module began, their experiences of collaborative learning mid module and an evaluation of the course at the end of the module. There were also questionnaires specifically for tutors. Response rates ranged from 42 - 75 per cent. Questions mostly required selections from multiple choice responses but there were some open ended questions.

Regarding student expectations we found that a few students had taken a web-based module before but most had some experience of online discussion.  Nearly all students (96%) were taking the module to learn more about e-business.  Some students also wanted to get experience with new learning technologies (41%), to explore online learning (37%), to get to know other distance learning students (22%), and to experience group work (22%). A significant number of students (48%) chose the online module to avoid having to do a formal written examination.  Students expected to have a more enjoyable and more collaborative learning environment.

Later surveys showed a high level of satisfaction with the module and the role of group discussion within it. Nearly all students enjoyed the flexibility of participating in group discussion at times convenient to them, ie asynchronous discussion.  Another positive feature of the group work was being able to go back to read messages, as contributions were permanently stored in WebBoard.   The majority (61%) appreciated the flexibility as to where they could take part. A number of students felt that their communication skills (45%), consensus building skills (52%), and cross cultural understanding (61%) had been improved.  Most students (76%) felt that the online discussion was accessible and coherent. Most students (79%) did not feel anxious when posting messages, and were comfortable with having to leave their contribution permanently recorded.  Students felt that the group activities were interesting and integrated into the module and 94% saw group work as a necessary part of the module. However, half of respondents encountered difficulties in finding the time to take part in online discussion. Table 1 shows the percentage of students who agreed with the following statements asked about online discussion.

 

I like online discussion because:

%

I do not like online discussion because:

%

I can mail when I like.

82

I do not have enough time.

52

I can go back to read messages.

79

I cannot see the audience.

21

I can mail where I like.

61

I do not like the style of messages

9

I can get to know people from different cultures

61

I can’t interrupt other people.

0

I am not interrupted when I contribute.

24

I do not feel confident using computer.

0

I can get to know other people in my group well.

18

 

 

 

Table 1: Students’ evaluation of their experience of online discussion

 

Text analysis

We analysed the online conferences both quantitatively and qualitatively. We looked at messages by week, by topic, by group, by group member, by length and we compared asynchronous messages with synchronous messages (chat) and attachments. It was clear that there was active participation in the conferences. For example, the number of messages sent within the working groups (in which there were 5 or 6 people) ranged from 250-488 messages over 18 weeks. The average length of these messages in the three groups studied in closer depth ranged from 64 – 115 words. Many of these messages carried attachments of much longer, and more heavily edited, pieces of course work. The average length of these attachments ranged from 918 – 1260 words. We found considerable variations in participation rates between individuals, between groups and between weeks. For example figure 1 (overleaf) shows variations in postings between groups, the shadings on each column represent particular phases during the discussion.

We took a sample of discussions within the module and analysed them in order to characterise the style and focus of the discussions. We coded the function of the messages, or more often parts of messages, for example was the writer making a suggestion, agreeing, disagreeing, stating or clarifying?  The focus of the messages was then analysed, for example, was the focus on course content, group processes or was it social in nature? This part of the research was time consuming and difficult. Codings were frequently modified through discussion until an acceptable level of agreement was arrived at. 

A key finding was that messages were more strongly associated with production of course work, for example scheduling of activities and assigning roles within the group, than on open ended exploration of content or reflection on the nature and purpose of group work. In each group the organisation of a group work task was taken on by an individual within the group – a role rotated at different phases of the module. This was identifiable in the message analysis both through more frequent postings and the focus within these messages on scheduling and checking progress and other organisational issues.  There was a marked lack of interactive discussion. Figure 2 shows the ten most common functions of messages. The majority (introducing, stating, reporting, analysis, clarifying, reporting, agreeing) were strongly associated with representing personal positions. Three carried more interactive functions (requesting, suggesting and clarifying). Only two per cent of messages were coded as expressing disagreement.

 

Figure 1: Variations in frequencies of messages within groups and between topics


 


Figure 2: The ten most common functions of messages in a sample of discussions

Interviews

We carried out face to face interviews with a sample of course participants (n= 12). We identified key categories and considered the dimensions associated with each category. For example we found that group work was viewed positively and was strongly associated with flexibility and sense of community. Negative associations with the course were its time consuming nature and the extent of the workload when compared with more conventional distance learning modules. Working groups were described as task-focused, friendly, supportive, collaborative but also as seeking consensus and relatively impersonal. Table 2 shows how interviewees’ comments about other members of the group were recorded following analysis of interview transcripts.

 

Interviewee

A

B

C

D

E

F

G

H

I

J

K

L

Σ

Friendly

1

1

1

1

1

1

1

 

 

1

 

 

8

Non-confrontational

1

1

1

1

 

1

 

1

 

1

 

 

7

Task-focused

1

1

1

 

1

 

 

1

 

1

 

 

6

Collaborative

1

1

 

 

1

1

 

 

 

1

 

1

6

Impersonal

 

1

 

 

1

1

 

1

 

 

 

 

4

Formal

 

 

 

1

 

1

 

 

1

1

 

 

4

Informal

1

1

1

 

 

 

1

 

 

 

 

 

4

Supportive

 

 

1

1

1

1

 

 

 

 

 

 

4

Knowledgeable

 

 

1

 

 

1

 

 

 

1

 

 

3

Democratic

1

1

 

 

 

 

 

 

 

 

 

 

2

Easy-going

 

1

 

 

 

1

 

 

 

 

 

 

2

Assertive

 

 

1

 

 

 

 

 

 

 

 

 

1

Honest

 

 

 

1

 

 

 

 

 

 

 

 

1

Punctual

 

 

 

 

 

1

 

 

 

 

 

 

1

 

Table 2: Tracking associations: how interviewees described the characteristics of other members of their group.

 

The interviews confirmed the pattern to group activity which had been suggested within the message analysis. Interviewees explained that communication was task focused and geared towards the production of reports on case problems. These were structured around interpreting an activity, planning, drafting, and collating a group response (table 3) below.

 

Phase

Typical student activity

Interpreting the task

Individual response to case problem

Planning a response

Volunteer takes on role of coordinator, someone who schedules the production of a group report

Drafting a response

Individuals create a response to aspects of the problem

Sharing responses

Individuals post responses and comments on other colleagues responses

Collating a draft report

Co-ordinator collates and edits responses, posts draft report

Final report submitted

Co-ordinator amends final report on basis of any further comments

 

Table 3: Structure of student activity

There was scope within this framework for wide ranging discussion but in practice interviewees explained that there was not time to do this.  The focus was on fulfilling their individual commitment to the group by meeting schedules for discussion and posting of viewpoints. The collaboration was seen as successful in that tasks were completed and relationships between the group were supportive and ‘democratic’. Here one student comments favourably on the work undertaken by the group:

“ I think it was quite impressive in general. There was collective leadership, we all played a part,  there was no-one that really dominated and there were no major disagreements…”

However this lack of disagreement was picked up more critically by others. One interviewee made the point that:

 “ in the group work everything stays very polite.  That might have been cultural or the fact that we are students.  We were quite polite.  Confrontational discussions were avoided…you could see the difference between contributions, people disagreed but the disagreement was not made explicitly in messages.  When it came to writing a summary of the discussion it very much rested with the group task coordinator, everybody would be okay with whatever direction he wanted to take it to. I think in a business environment there would have been more debate and discussion but here the objective is to get the task done. Getting your opinion in, or your views in the group report, is not that important.”

 

 

Comments on our findings

We have only given a snapshot of our research findings but a picture emerges of a course which was well received, appropriately structured for the subject content and having additional value for participants in terms of motivation, interaction, gaining knowledge of other cultures and providing experience of online working. Set against these positive findings it is clear that collaborative work was constrained. Groups were active, members communicated with each other and commented on each other’s work. However members of groups were reticent about pushing disagreement and the procedures for groupwork were not critically explored.

Several writers have drawn a distinction between cooperative and collaborative group work (eg Curtis and Lawson 2001). The terms are used inconsistently but they do imply variations in the degree of collaboration: cooperative implying specialisation within the group so that each member works on a discrete aspect of the task while collaborative learning involves team members working together to develop a joint solution or response to a problem. At a more sophisticated level, collaboration has been seen as underpinning the idea of a community of practice, borrowed from Lave and Wenger, to imply both reflexive as well as reflective activity. Putz (2001) for example sees five dimensions to an online community of practice: participation and reification; identity and anonymity; local and extended community; designed and emergent structures; grading and alignment. Our research is best summarised as an example of cooperative learning with only occasional evidence of higher order collaboration and reflexive discussion on the nature of the community and the role of members within it. There was a community of learners rather than a learning community. This can be further illustrated through reference to a five stage model on the use of CMC to support learning proposed by Salmon (2000).  Our study shows levels of cooperation characteristic of stages one to three, strong elements of stage four, in which course-related group discussions occur and the interaction becomes more collaborative, but only occasional activity at the final stage in which participants look for more benefits from the system to help them achieve personal goals; explore how to integrate CMC into other forms of learning; and reflect on the learning processes.

The limits on collaboration within the group lay in a combination of personal factors; the pressure on part time students in trying to balance work, family and study commitments; the nature of online discussion which often appears as more time consuming than face to face teaching; a curriculum geared to the production of reports; and an unwillingness on the part of many students to take risks and explore disagreement.  In some cases students’ cautious attitude to collaboration had roots in their uncertainty in using English as a second language and their reticence in making a public permanent contribution to a discussion. This was not ‘on line learning’ as such, students did a lot of their thinking away from Web Board, for example by reflecting on an article or carrying out some individual research, and joined the on line discussion again to report back and to monitor the production of the group assignments. There are undoubtedly changes which the course team could introduce to give students more opportunities for collaboration, for example, reduce the intensity of tasks, encourage tutors to take on a more discursive role, put more emphasis on assessment of individual contribution or introduce peer assessment. However these would not reduce the demands made on students in other areas of their lives. These students pushed the discussion as far as they felt able and as far as they felt it was useful, there was an overwhelming sense that the existing arrangements were well received. To sum up students felt they had gained knowledge and understanding of E business, drew positive comparison between this module and more traditional distance learning modules and developed cooperative working practices. However, collaboration was limited and there was a clear element of instrumentalism (Marton et al 1984) in some students’ approach to learning.  The module is a best fit solution to constraints operating on learners and may provide a viable model for introducing on line courses in related contexts, particularly those in which the subject matter is technology related. 

 

Comments on research strategies

A clear strength in our research methodology was a commitment to a triangulated approach. Lack of triangulation we argue elsewhere (Hammond 2004) is a feature and weakness in many case studies on online learning. We felt more confident in reporting our findings as there was consistency between different sources of data, albeit each source had provided detail which was not easily obtainable by other methods. For example, the surveys provided a positive picture of the course with group work highlighted as a distinctive and valuable part of the course. This was confirmed in the interviews and levels of activity were shown through quantitative message analysis. Similarly pressure of time emerged as an issue in the survey and was confirmed in the interviews in which students were able to go into detail about their study patterns and how they scheduled online activities. The message analysis offered additional evidence of constraints on students by showing a relatively low focus on interactive exchanges.

 

There were, however, weaknesses in our approach. Firstly, with more time we could have carried out more comprehensive data collection, for example, by carrying out telephone interviews with students. (We had limited our research to a representative sample of students who we met at a whole course residential meeting.) Secondly, we would like to have involved students far more in the evaluation methodology, for example by asking them to write logs or diaries of their participation and discussing their experiences in a whole course meeting. The constraint here was again lack of time. Thirdly, an action research orientation to the course may have given us more scope for implementing innovations and reduced the distance between us and students (a concern reinforced for us by a recent commentary by Knobel, 2003.).

 The area of the research in which we felt least secure was our analysis of messages, or content analysis characterised by Anderson et al (2001) as a set of procedures to make valid inferences from text. These procedures are summarised as identifying and defining a target variable, collecting samples of representative text, and devising reliable and valid rules for categorizing segments of the text. The strength of our approach was a commitment to a grounded approach and a refusal to use a priori categories, such as those of Henri (1991) or Gunawardena et al (1997), which is a feature of much reported content analysis. However, our categories need developing and contrasting with other research. For example, the distinction between interactive and non interactive functions  within messages could be compared to studies by Aviv (2000), Hawkey (2002), Thomas (2003) and others who develop independent and interactive categories of message. However it was very clear that message analysis was very time consuming for as yet limited returns. This may have been because of the modesty of our objectives but it is open to question how far any message analysis can throw light on learners’ construction of knowledge.  Rourke at al (2001) discuss procedural problems in categorising ‘latent’ content within messages, but the problem cuts deeper. Content analysis can give valuable insight on the nature of a conference but it is constructed through the perspective of a diligent and interested reader, it is not a reliable guide to the experience of the learner. In retrospect, perhaps we would have discovered more by involving students more directly in message analysis from the start or by focusing much more on critical or illustrative incidents within conferences. Previous proceedings from the Networked Lifelong Learning Conference (Banks et al 2002) demonstrate the strong influence of content analysis. Our research made us more aware of the difficulties in categorisation of messages, of taking account of learning that goes on away from the computers and the need to validate categories of analysis. We finished more committed to an explicitly multi method approach.

 

References

Anderson, T., Rourke, L., Garrison, R., and Archer, W. (2001) Assessing Teaching Presence in a Computer Conferencing Context, Journal of Asynchronous Learning Networks, 5, 2, 1 –17.

Aviv, R. (2000) Educational performance of ALN via content analysis", Journal of Asynchronous Learning Networks, 4, 2, 53 – 72.

Banks, S., Goodyear, P., Hodgson, V. and McConnell, D. (2002) Networked Learning 2002 A research based conference on E learning in higher education and life long learning, University of Sheffield, Sheffield.

Curtis, D. and Lawson, M. (2001) Exploring collaborative online learning, Journal of Asynchronous Learning Networks, 5, 1, 21- 34.

Gunawardena, C., Lowe, C. and Anderson, T. (1997) Analysis of a global online debate and the development of an interaction analysis model for examining social construction of knowledge in computer conferencing, Journal of Educational Computing Research, 17, 4, 395 – 429.

Hammond, M. (2004) Is research into asynchronous discussion any good? An analysis of recent papers on teaching and learning through on line discussion in higher education, paper presented to ITTE Research Conference,  University of Cambridge, 9 - 10 January, 2004.

Hammond, M. and Wiriyapinit, M. (2002) ‘E-MBA Initiative at Warwick Business School: A Study in Collaborative Online Learning in World Congress Networked Learning in a Global Environment Challenges and Solutions for Virtual Education, May 1 - 4, 2002 , NAISO Academic Press, BB Sliedrecht, The Netherlands.

Hawkey, K. (2003)Asynchronous Text-Based Discussion: A Case Study with Trainee Teachers, Education and Information Technologies, 8, 2, 165-177.

Henri, F. (1991) Computer conferencing and context analysis in (ed A. Kaye) Collaborative Learning through Computer Conferencing, Springer-Verlag, Berlin.

Knobel, M. (2003) Rants, ravings and representation: ethical issues in researching online social practices, Education, Communication and Information, 3, 2, 187 – 210.

Marton, F., Hounsell, D. and Entwistle, N. (1984) The Experience of Learning, Scottish Academic Press, Edinburgh.

Putz, P. and Arnold, P. (2001) Communities of Practice: guidelines for the design of online seminars in higher education, Education, Communication and Information, 2001, 1 ,2, 181-195

Rourke, L., Anderson, T., Garrison, D. and Archer, W. (2001) Methodological issues in the content analysis of computer conference transcripts, International Journal of Artificial Intelligence in Education, 12, 8-22.

Salmon, G. (2000) E-moderating: the Key to Teaching and Learning Online, Routledge, London

Thomas, M. (2002) Implementing a CMC tutor group for an existing distance education course, Journal of Computer Assisted Learning, 18,  3,  351-366.