The Open
University
This brief paper examines the potential of platform based student tracking tools to assist online tutors in monitoring the learning activity of their students. It makes particular reference to a group of students working within one Blackboard mediated classroom by examining how actual behaviour correlated with that monitored automatically by the delivery platform. It concludes that whilst student tracking can be a useful tool in certain circumstances, reliance on it as the sole means of monitoring student participation and learning is unsound.
e-learning, e-tutoring, student tracking, student appraisal
Universities are increasingly moving their programmes online; either as discrete wholly-online entities or as an integral part of what have become known as blended learning programmes where online discussion boards supplement face-to-face teaching. As Bartolic-Zlomislic & Bates (1997) point out, this has benefits for institution and student alike. Institutions gain new markets and economies from resource sharing while students gain access to new partnerships, greater class diversity and access to peer interaction - the latter being especially significant for those previously following paper-based distance programmes. The online horizon is not entirely clear however. Bartolic-Zlomislic & Bates also identify a number of issues that may cloud the sky: not least, the challenge of the time involved in text-based online interaction. Beaudoin (2002) notes an assumption by many educationalists that, ‘a high level of interaction is desirable and increases the effectiveness of both classroom and distance education courses’, although he also notes that, in face-to-face teaching at least, many students who fail to participate actively still do achieve intended learning outcomes. Whilst it is arguable that online students may determine their level of participation and thus their individual time commitment to any class without necessarily jeopardising their future academic achievement, time is much more problematic for tutors who must attend to the needs of the whole class. Furthermore, ‘online learning environments are more challenging than face-to-face settings because the visual and aural clues of student performance are missing’ (Pappas et al., 2001).
Under such circumstances then, how can the online tutor monitor learning appropriately? Pappas et al., (2001) assert that online tutors need to ‘rethink the monitoring of student performance’. A survey of virtual learning environments (VLEs) using the EduTools evaluation database[1], reveals nearly 40 VLE platforms where in-built student tracking tools purport to address this issue. Such tools use login records, automatically gathered in the course of normal VLE operation, to generate analyses of student learning behaviour in the VLE. Wang & Newlin (2002) advise, ‘this type of tracking information is automated and readily accessible by instructors, we strongly recommend its use as an early warning indicator of cyber student performance’. Their own studies suggest that, for example, there is a direct correlation between the number of logins in the first week of a course and the grades students' will achieve at the end. To date, few studies have examined the effectiveness of tracking tools - despite a developing awareness amongst practitioners of the differing needs of an increasingly diverse learner population. In order to examine one such system in action, the present study used student tracking to monitor the apparent learning activity of a volunteer group of ten online learners from an honours/masters class in education using a Blackboard VLE. The study was guided by the question: in what ways can student tracking tools assist the online tutor?
As a first step, an activity profile was sought for the class as a whole, to establish average activity levels for the class. The profile was generated automatically by the Blackboard platform through use of the Control Panel, in particular the Course Statistics facility. The profile consisted of statistics, pie and bar charts to indicate log-ins to, and percentage of total platform logins for, different areas of the VLE. The platform areas were classified by the profile as being: content areas; communications areas; student areas and group areas. Later, profiles were generated for each individual in the volunteer group. Participants were subsequently interviewed individually. During interview, participants were asked to describe areas of the platform they had used, to accord priority to those areas (in terms of time spent there as well as rating the relative importance of each area in terms of their overall learning on the course), and to describe how they had used the online resources available to them. They were also asked to clarify the balance of their study time, online and off, for each area.
According to the activity profiles all students surveyed had used content and communications areas. Some had used student areas and a few had used group areas. The absence of activity in group areas is explained by the fact that these were used on an optional basis by some students and any activity there was not a required part of their study programme. Most students showed an overall preference for use of the communications area.
According to the interview data most students felt they had made more use of the content areas than the communications areas. Subsequent questioning revealed that only one student had used content materials whilst actually online. Most others had printed off all the content materials or had download each content module onto a stand alone PC at home or at work and had then used the materials electronically but not whilst actually online.
Two students in the group provide data which amply illustrates both the potential and the problems involved when using student tracking to monitor online student learning: the platform generated activity profile for student 'Pamela' shows a 100% preference for use of content areas. However, Pamela dropped out of the course at an early stage, her total log-ins were 3. In interview it transpired that she had downloaded all the course subject content, begun studying it off line and then abandoned the class. She had done all her previous learning by print-based distance learning, and she felt unable to deal with the discussion fora and the interaction with other students online which she felt was required of her on this course. The platform generated activity profile for student 'Richard' shows a mere 4% preference for content areas compared to figures of 49% for communications areas and 38% for student areas. In interview it transpired that Richard, being a rancher in the Outback at the time of his studies, had downloaded content modules onto a PC and then printed them off so that he could take them with him to study while he, literally, "waited for the cows to come home". He also reported using the modules offline on his PC and was quite clear that content areas were the ones he used most.
Comparison of the tracking and interview data thus highlighted a number of potential difficulties with the use of student tracking data for monitoring student learning activity:
Also, unless the platform is programmed carefully, it is so zealous in its search for activity that it may record apparently suitable activity even when that activity is not, in fact, suitable (e.g. when asked for a report of overall class activity it carefully included logins for technical and administrative staff in its analysis).
Referring back to Pamela above, who had a profile showing a 100% preference for content areas based on 3 log-ins, interview and reference to the course discussion boards shows that despite the platform registering no logins to the communications areas for her she had in fact been there and posted a message to the class welcome forum. In her interview it became clear that it was this one visit, ‘unseen’ by the student tracking system, which had led her to conclude that did not want to proceed with the course because of its requirement for discussion board messaging interaction.
So, what can tracking offer the time pressed online tutor? Wang & Newlin (2002) suggest that there is no one indicator for identifying an online learner at risk, this study suggests that likewise there is no one way to adequately monitor online activity. Student tracking tools, when their limitations are understood, can help but will provide only one shade of the technicolor picture required to maximise effective online teaching. Above all, tutors need to learn an instinct which makes them question the accuracy of any information the platform may give them. Furthermore, they need to be aware of what the figures don't say as much as what they do say. For example: when presented with a usage figure of 80 for a student's use of content areas they need to realise that this is only indicative of 80 percent of all the logins the student made, which may or may not be a 'healthy' number for learning to result. The tutor will need to probe further if they wish to reassure themselves that an individual learner is on track for success. Monitoring student progress cannot be left entirely to the delivery platform, although the platform may be able to offer the tutor clear indicators as to where they should look in order to identify learners' at risk.
Bartolic-Zlomislic,
S. and A. W. Bates (1999). Investing in Online Learning: Potential Benefits and
Limitations. Vancouver,
University of British Colombia available from URL: http://www.educause.edu/asp/doclib/abstract.asp?ID=CSD2615
Beaudoin, M.
F. (2002). Learning or Lurking? Tracking the "Invisible" Student
Online. Internet and Higher Education,
5: 147-155.
Pappas, G., E.
Lederman, et al. (2001). Monitoring Student Performance in Online Courses: New
Game-New Rules. Journal of
Distance Education, 16(2).
Wang, A. Y.
and M. H. Newlin (2002). Predictors of Performance in the Virtual Classroom.
T.H.E. Journal Online (May 2002).