ABSTRACT

Educational dialogue can be used to support learners in the development of reasoning, critical thinking, and argumentation. In this paper we report on the evaluation of an educational design for online peer discussion that guides student dialogue towards more academic interactions and facilitates extended argument. This design includes a mediating interface for synchronous discussion, implemented for this research, which incorporates a dialogue game approach to discussion, using sentence openers to structure interactions. The study employed a broader set of online educational activities – a designed local context, which aims to motivate higher education students to argue in a collaborative working setting. The evaluation compares the impact of using structured interactions with the use of a simple interface without structured interaction. The findings suggest an improved, deeper argumentation process in the structured environment.

Keywords:

Peer Collaboration Online, Argumentation, Synchronous dialogue, CMC

Structuring dialogue

Educational dialogue can be used to support learners in developing skills of reasoning, critical thinking, and argumentation of the sort used in academic practice. This is supported by theoretical studies such as Doise & Mugny (1984), Kuhn (1991), and Lipman (2002), and empirical studies (e.g. see Pilkington, 2001, for a review). In many networked learning situations learners may lack the opportunity to converse, argue and debate face-to-face in order to develop such skills. This paper reports on an educational design for synchronous online peer argumentation incorporating a mediating interface or tool, AcademicTalk. The aim is to use such a tool to explore its effectiveness in guiding and scaffolding students towards more academic dialogue and to facilitate extended and more in-depth discussion. A set of online educational activities have been developed to motivate students to argue through creating suitable conditions to support the structuring of such interactions.

There is a need to identify, characterise and design lower level features of collaborative working - such as roles, rules, strategies and moves to support particular types of dialogue in achieving particular educational goals (e.g. see Ravenscroft & Pilkington, 2000, for a review). This work suggests that structuring and guiding learners dialogue can lead to clear and significant educational benefits. Collaborative educational argumentation is often essential to support the type of dialogue that in turn leads to conceptual development and improved reasoning in learners. Vygotskian (1978) approaches to learning also emphasise the development of higher level mental processes – such as critical reasoning and reflection, through internalising linguistic processes – such as argumentation, that occur ‘in the social’ .

The interface designed for this research – AcademicTalk, manages conversations through the use of topic threads and argument strands which keep reply messages next to their antecedent messages. This innovation prevents the incoherence in sequencing of messages that is typically experienced in simpler interfaces (Chat) and removes the cognitive burden of matching messages for the student (Herring, 1999; Garcia and Jacobs, 1999). In order to scaffold the students' interactions towards academic argumentation at the locution level, sentence openers, such as ‘I think…’, 'Why do you think that?...', 'Is there evidence?...' etc. form the first part of the locutions performed by the students. Forty one opening phrases are collected into six menus by intention, and aim to promote reasoning, use of evidence, and direct engagement with the ideas of others. The set of openers has been developed for higher education students, based on earlier work on collaborative working and effective peer dialogue (Soller & Lesgold, 1999; McManus & Aiken, 1995; and Johnson & Johnson, 1991), but modified to reflect the emphasis on argumentation. Further structure and guidance is provided to the students through offering an optional 'preferred reply' set of openers derived from dialogue game rules and that correspond to notions of well-formed dialogue.

Cultivating the local context

Besides structuring interactions at the interpersonal level the educational design is concerned with cultivating a suitable local context for supporting argumentation and discussion. This context for student discussion is important in providing the motivation to argue since much previous research (e.g. Ravenscroft & Matheson, 2002; Ravenscroft, 2004), has pointed out the necessity to consider the context in which successful argumentation takes place, as students don’t ‘argue with anyone about anything at anytime’. So the discussion activity was designed to integrate with the interface design and prepare students for argumentation in order to support phases of knowledge building (Quignard & Baker 1999, Bereiter 1994, Scardamalia et al., 1994).

Figure 1 – The Collaborative Learning Activity Model

Figure 1 shows the phases which include preparatory readings, two separate online discussion sessions, and a discussion summary provided by a student facilitator. The readings seed argumentative discussion in the online sessions by supplying alternatives perspectives on the topic (Veerman, 2000; Baker, de Vries & Lund, 1999). Two online discussion sessions were used, the first (comparing) to explore the topic breadth and to provide some familiarity and confidence to the students who may only have tentative knowledge at that stage. The second session (debating) involves a formal taking of sides to debate a subtopic the students have expressed an interest in. A student facilitator keeps the group on-task and checks progress while participating in the discussions. Their tasks include helping the group decide the subtopic for debate in the second session and then writing a summary of the arguments used in the whole discussion. This summary is posted to the group members for further reflection. More detail on the phased activity (and the interface) can be found in McAlister et al. (2004).

An Evaluation Study

In order to evaluate the AcademicTalk system, a study was conducted comparing it with Chat over five phased synchronous discussions, with groups of four or five students. Half the groups used only Chat (n=19) for their discussions, and half used only AcademicTalk (Talk, n=15). The purpose of the study was to see if the structured interactions led to a deeper and more extended argumentation process than unstructured interactions. The topics for discussion were chosen to be controversial issues to stimulate argument, and to coincide with the students’ course reading[1]. The 34 students in the sample were from two different cohorts taking a first level development of technology course offered by the UK Open University. Twenty-nine of these were new to higher education, and of these 16 had low formal qualifications[2]. The students were recruited as volunteers from regional student online fora and the study (like the course) was run online without the researchers or the students meeting face-to-face. Two-thirds of the study sample were female, in contrast with the student population which was only one third female.

These were quasi-experimental conditions – replicating the course environment as much as possible, and as the students were volunteers, there was both dropout and latecomers to the study. A few students moved within condition between the two groups, in order to continue to attend, as the groups met on different nights of the week. A supporting asynchronous forum provided a place for the students to meet online and bond as group, and was well used by them. In spite of the time burden for each discussion – the reading preparation, hour-long group sessions on two separate evenings and the occasional facilitation – all the sample students who responded (65%), were unanimous that the time spent was worthwhile, that they enjoyed the discussions and they would do it again. They also agreed it gave them something they could not get elsewhere on the course and that their understanding of the course was helped.

Analysis of the discussion logs

Each of the eighteen discussion logs comprised data from the two online sessions (comparing and debating), which the researcher arranged by episode. These episodes typically contained between 5-15 messages of conversation on the same subtopic, and two illustrative episodes are presented later. Argumentative exchanges were common in each online session, but not all dialogue in the discussions was argumentation, there were off-topic episodes and other discursive episodes which did not contain argument. Initial reading of the transcripts found most off-topic messages in the Chat data, varying by group, but consistently little or none in the Talk data. A count of the off-topic messages totalled 24% of all messages in Chat, whereas in the Talk data there was only 1%, a highly significant difference[3].

In order to analyse only argumentative on-topic dialogue, a selection from the available episodes was made by course tutors who did not know the students or the researchers. Each tutor was provided with several discussions from each condition. They were asked to select four episodes of dialogue from each discussion that they believed showed good interaction between students and had educational value to the students involved. This data subset of 72 independently selected episodes (40 from Talk and 32 from Chat, about 18% of the total data), which contained on-topic argumentation worthy of tutor-selection was analysed further and is presented here (for all results see McAlister (2004, in preparation).

Counts of dialogue moves

Dialogue moves indicate the main communicative function of the message and are derived from a dialogue game approach to analysing conversation, based upon ‘speech acts’ (Searle, 1969). The messages in this dataset were categorised using a general markup scheme for spoken and interactive dialogue (DISCOUNT), which defines the rhetorical predicates and ideational content of the dialogue moves (Pilkington, 1999). The purpose of this was to see if there were different frequencies of dialogue moves in each condition, which would indicate different dialogue games, or styles of interaction.

The main differences between the two conditions are shown in Table 1. There was a highly significant difference[4] between the structured dialogues and the unstructured dialogues in terms of the move frequencies. Each of these differences illustrates the more argumentative dialogue found in the Talk data compared to the Chat. Explore, Withdraw and Nocommitment were more frequently used in Talk, to invite views and to show dissension. The latter two dissenting moves accounted for 17% of Talk moves but only 3% of Chat moves. In Chat, Inform, Inquire and Reply were more frequently used to trade information and opinion.

Table 1 – Counts of dialogue moves from tutor selected episodes, extract

Move	Talk	Chat	Talk%	Chat%	Comment
Explore	47	19	10	5	Invite views on issue
Withdraw	54	6	11	2	State disagreement with argument
Nocomm.	30	4	6	1	Signal that argument is unconvincing
Inform	66	70	14	20	Retrieve facts and opinions
Inquire	10	25	2	7	Request information
Reply	6	26	1	7	Offer information as answer
(All moves)	470	356
Notes: Talk% is percentage of all Talk moves, Chat% is percentage of all Chat moves.

Requesting and providing evidence

Use of evidence is a key skill in academic discussion and AcademicTalk has several openers relating to evidence and example. Counts of the messages using or requesting evidence were made in the tutor-selected dataset, and the difference between the two conditions was highly significant[5]. About 14% of the Talk on-topic messages either requested or referred to evidence compared to 9% of the Chat on-topic messages.

Claims and rebuttals

In order to measure both the complexity of the process of argument and the amount of extended argument, claims and rebuttals were examined using the Toulmin Argument Pattern schema (TAP), proposed by Osborne, Simon & Erduran (2002). In this schema, simple argument begins with claims and counter-claims unsupported by reasons (level 1), while extended argumentation involved claims, reasons and rebuttals (levels 4 & 5). Weak rebuttals (level 3) are attempts to address and rebut an argument, but the rebuttal is incomplete in that the argument still retains at least some validity. Rebuttals are regarded as highly important in the sense they have the potential to change the ideas and thinking of students. Therefore, the main features of the argument framework "include: the extent to which students have made use of the data, claims, warrants, backings and qualifiers; and the extent to which they have engaged in claiming, elaborating, reinforcing or opposing the arguments of each other" (Osborne, Simon and Eduran, 2002, p7).

About one third of the selected Chat data was non-oppositional, in that claims were not challenged, qualified or countered. The Talk data rated more highly on every level of argumentation process (see Fig. 2), a result that was highly significant[6]. Outside the field of ‘hard’ science where TAP originates, and particularly with the normative questions discussed in this study, most rebuttals will fall into the weak category and the TAP schema fails to discriminate in the top two categories which are sparse. So an additional category – extended rebuttals – was created that counts the proportion of episodes that contain multiple rebuttals (weak or strong). While about a third of the Talk episodes demonstrated extended argumentation, none of the Chat episodes did.

Figure 2 - TAP rating of tutor-selected episodes, by % in each category

[e1] Argumentation with Structured Interaction

Table 2 shows an episode from a Debating session with AcademicTalk, in which the students use constructive conflict. This episode shows detailed argumentation about the relevance of ideas to the Internet project. It can be seen that the participants engage with, and challenge arguments proposed using the opener ‘I disagree because…’. These direct challenges, which are infrequent in equivalent Chat dialogues, may result from a sense of ‘permission’ given by using openers in a ‘dialogue game’ which overcomes politeness rules. Each message replies to the previous message and starts with an opener (except the first which begins a thread).

Table 2 – AcademicTalk: Argumentation based on challenge

Topic 'Who do you think made the most significant contribution to the Internet project pre-1990?
91	Andrew	Influential and Inspirational Internet Visionary - Vannevar Bush or JCR Lickleider?
94	Eric	I think... It has to be Lick. He saw the benefit of networking computers. Bush only saw the benefit of organizing information.
96	Dawn	I agree because... Bush's initial idea did not take the idea of the information 'memex' into the realm of networking
100	Andrew	I disagree because... It was Bush who defined a whole new way of thinking about technology and how it should be developed and funded
106	Dawn	Good point.
107	Dawn	I agree because... Bush also had the idea of linking this information (hypertext linking?), just maybe not the networking idea
110	Eric	I disagree because... Although Bush's ideas have been used in the internet. His ideas had nothing to do with computers and networking them. He only visualised a single machine working by itself.
119	Andrew	I disagree because... he has been described as a prophet of cyberspace!
121	Andrew	Because... you have to remember at what time he was talking about this machine and its ability to leave associative trails that could be passed on to future generations
126	Eric	That is valid if... if we consider the internet as only interlinked documents. But what about other applications such as e-mail or file sharing?
133	Andrew	I disagree because... still require human interface with the computer, which is what he predicted would be the most exciting area of computing
145	Eric	A counter-argument is... is that the most exciting area is commicating with other computers and users of computers. Not just your own with all the information locked inside it, like the Memex machine.
146	Andrew	Good point.
Notes: This episode is an extract from the debating session of the fourth discussion in the Talk series; student names changed for anonymity; lines are non-sequential because AcademicTalk is multi-threaded.

In this episode Eric is championing JCR Licklider as the Internet visionary against Andrew’s claims for V Bush some twenty years earlier. Dawn is able to see the merits of both claims. Eric proposes and maintains the essential element of networking in the Internet vision (94, 110, 126, 145), against Andrew’s (and Dawn’s) arguments. By a series of rebuttals, using openers ‘I disagree because…’ and ‘That is valid if…’, he manages to get Andrew’s agreement for this point of view.

Table 3 – Chat: Argumentation based on agreement

Topic 'Who do you think made the most significant contribution to the Internet project pre-1990?
112	Daisy	He [Licklider] saw how to make computers do the mundane stuff
113	Daisy	to allow humans to [do] the important stuff, I believe?
114	Sheila	So are we agreed that being knowledgeable about both disciplines influenced his thinking and ideas?
115	Tracy	an intimate relationship!
116	Tracy	yes
117	Ray	ime sure it must have
118	Daisy	His 'groundbreaking work Man Computer Symbiosis
119	Sheila	Could he have done that without his expertise in both areas?
120	Daisy	No, His paper discussed real time interactive computing to cooperate in making decisions
121	Tracy	i think it gave him a unique insight
122	Ray	i think that because his expertice overlapped this gave him advantages that he utilised
123	Daisy	Yes. An unusual mix methinks
125	Ray	he not only saw the human and computer he managed to see the combination of the two was greater than the patrs?
126	Tracy	absolutely
127	Daisy	Yes, Ray, I think you’re right
Notes: This episode is an extract from the debating session of the third discussion in the Chat series; student names changed for anonymity.

In the Chat extract (Table 3), the majority of dialogue moves are informing moves proposing ideas and opinions, which are generally agreed. The ideas are not challenged or tested and there are no rebuttals. The conversation reviews the contribution of JCR Licklider without comparing the contribution of other contenders and assessing their relative merit, or even offering any counter-arguments. As a result the conversation does not strongly establish the argument for ‘Lick’. Much of the unstructured dialogue in the study followed this style of interaction as the earlier results for dialogue moves and rebuttals showed.

Discussion and Further Work

The extracts presented demonstrate the differing styles of unstructured and structured dialogue, and the dialogue move analysis showed that different dialogue games are being played out. The greater use of explore, withdraw and nocommitment moves demonstrate the use of constructive conflict in the Talk dialogues. It seems that the students in the Talk condition felt able to challenge and oppose ideas, as the dialogue extract demonstrated, and this led to ideas being critiqued and defended with a consequent drawing out of the argument. In contrast, the students in the Chat condition traded ideas, but these ideas were not really tested or challenged, as the Chat extract showed. This uncritical acceptance of ideas and reluctance to critique ideas in unstructured collaborative discussion represents a lost opportunity to delve more deeply into the real arguments and by so doing broaden and deepen understanding.

AcademicTalk appears to successfully scaffold students’ argumentation skills, with this condition showing more on-topic dialogue, more justification for their positions, better use of evidence, the exploration of alternative points of view and less simple ‘trading’ of opinions compared with Chat. Secondly, AcademicTalk supported more rebuttal of positions, extended argument and multiple rebuttals. These are promising results, when we consider that the interface design (i.e. AcademicTalk or Chat) was the key variable, with all contextual factors held constant including the requirement for preparatory activities before the sessions. The interface provided a number of affordances in order to establish a argumentative dialectic between students. Firstly, it required students to reply to, and address the content of, a specific message through the use of sentence openers. Secondly, the managed dialogue clearly preserved individual strands of argument, with replies following their antecedent messages, providing a coherent view of the argument. Thirdly, all earlier messages were readily available, allowing for the argument to develop again later in the discussion after some reflection. Finally, the highlighting of ‘preferred’ openers to afford guidance during opener selection by the student[7].

The results clearly demonstrate the potential of the adopted approach to designing structuring. The design used features of dialogue games (sentence openers and flexible structuring of preferred responses) to guide and scaffold learners’ dialogue, and the results suggest that the design decisions have had a positive impact on the dialogue. The findings suggest that this approach leads to more coherent and improved argumentation than is possible with less structured approaches, such as the use of Chat. This is a valuable insight in most open and distance learning (and probably many other) learning situations. There is an increasing emphasis on the need for improved synchronous online argumentation, but there are virtually no tools – with the exception of AcademicTalk – that explicitly support it. We hope to continue further with the exploration of this tool in other learning settings.

References

Baker, M., de Vries, E. & Lund, K. (1999) Designing computer-mediated epistemic interactions. In Lajoie S.P.& Vivet M. (ed.), Proceedings of the 9th Int. conference on AI in Education. Amsterdam:IOS Press.

Bereiter, C. (1994) Constructivism, socioculturalism, and Popper's World 3. Educational Researcher 23, 7, 21-23.

Doise, W. & Mugny, G. (1984) The Social Development of the Intellect. Oxford:Pergamon.

Garcia, A. & Jacobs, J. (1999) The eyes of the beholder: Understanding the turn-taking system in quasi-synchronous CMC. Research on Language and Social Interaction 32, 4, 337-367.

Herring, S. (1999) Interactional coherence in CMC. J. of CMC 4, 4, (online). Available: http://www.ascusc.org/jcmc/vol4/issue4/herring.html

Johnson, D. & Johnson, R. (1991) Learning Together and Alone. Englewood Cliffs, NJ:Prentice Hall.

Kuhn, D. (1991) The Skills of Argument. Cambridge, MA: Cambridge University Press.

Lipman, M. (2002) Thinking in Education. Cambridge University Press 2nd Ed

McAlister, S., Ravenscroft, A. & Scanlon, E. (2004) Combining interaction and context design to support collaborative argumentation using a tool for synchronous CMC. J. of Computer Assisted Learning (in press).

McAlister, S. (2004) Dialectic and Design for Online Peer Discussion. PhD. Thesis in preparation, The Open University, UK.

McManus, M. & Aiken, R. (1995) Monitoring computer-based collaborative problem solving. J. of AI in Education 6, 4, 307-336.

Osborne, J., Simon, S. & Erduran, S. (2002) Enhancing the Quality of Argumentation in School Science. London:King's College End of Project Report.

Pilkington, R. (1999) Analysing educational discourse: the DISCOUNT scheme. Leeds University CBLU Technical report No. 99/2 Jan 1999.

Pilkington, R. (2001) Introduction to the IJAIED Special Issue on Analysing Educational Dialogue, Part II. Int. J. of AI in Education 12, 1-7

Quignard, M. & Baker, M. (1999) Favouring modellable computer-mediated argumentative dialogue in collaborative problem-solving situations. In Lajoie, S.P. & Vivet, M. (eds.), Proceedings of the 9th Int. conference on AI in Education. Amsterdam:IOS Press.

Ravenscroft, A. (2004) Towards highly communicative eLearning communities: Developing a socio-cultural framework for cognitive change. In Land, R. & Bayne, S. (eds.), Education in Cyberspace. Routledge (in press).

Ravenscroft, A. & Matheson, M.P. (2002) Developing and evaluating dialogue games for collaborative e-learning interaction. J. of Computer Assisted Learning Special Issue: Context, collaboration, computers and learning 18, 1, 93-102.

Ravenscroft, A. & Pilkington, R.M. (2000) Investigation by Design: Developing dialogue models to support reasoning and conceptual change. Int. J. of AI in Education Special Issue: Analysing Educational Dialogue Interaction: From Analysis to Models that Support Learning 11, 1, 273-298.

Scardamalia, M., Bereiter, C. & Lamon, M. (1994) The CSILE project: Trying to bring the classroom into World 3. In McGilly, K. (ed.) Classroom Lessons: Integrating cognitive theory and practice. Cambridge, MA:MIT Press.

Searle, J. (1969) Speech Acts. Cambridge University Press.

Soller, A., Lesgold, A. (1999). Analyzing Peer Dialogue from an Active Learning Perspective. Proceedings of the AI-ED 99 Workshop: Analysing Educational Dialogue Interaction, Le Mans, France, 63-71.

Veerman, A. ( 2000) Computer-Supported Collaborative Learning Through Argumentation. NL:University of Utrecht, published PhD. Thesis

Vygotsky, L. (1978) Mind and society: The development of higher psychological processes. Cambridge, MA:Harvard University Press.

[1] An example topic was “The Windows PC owes its success to features, like the graphical user interface, that were first developed for the Apple Mac”

[3] Test for difference between off-topic counts, Chi-square(1) = 458, p < 0.001

[4] Test for difference in dialogue move frequency, Chi-square(16) = 91.5, p < 0.001, (cell combination).

[5] Test for difference in use of evidence, Chi-square(1) = 65, p < 0.001.

[6] Test for difference in episode frequency in the TAP scheme, Chi-square(2) = 13.0, p < 0.002 (cell combination).

[7] Each preferred set of openers is determined from the opener of the message replied to, and is dynamically adjusted to take account of recent openers.