Agnes Kukulska-Hulme and Lesley Shield
The Open University
A.M.Kukulska-Hulme@open.ac.uk, L.E.Shield@open.ac.uk
Technical knowledge is insufficient when designing and testing websites for e-learning. At the Open University, usability has been at the heart of a project aimed at enabling academic course developers to participate on an equal footing with other professionals in addressing website usability. Our investigations have included interviews with key staff, a review of usability literature, case studies, usability tools, workshops, an investigation of student and tutor experiences, and an analysis of student ratings for websites. We outline our concept of ‘pedagogical usability’, a set of key cosiderations for improving the experience of web-based learning.
Web-based learning, guidelines, pedagogical design, usability, acceptability, website testing
In post-compulsory education,
issues surrounding e-learning website usability, or the educational
effectiveness, practical efficiency and general ‘enjoyability’ of interacting
with web-based e-learning materials, are not yet well
known or understood. There are
numerous recommendations for the design of pages, text, graphics and
navigation, but in spite of this, it is recognized that “severe usability
problems are common” (Brinck, Gergle, & Wood, 2002:4). Why is this the
case? It appears that too often, website developers assume they know users’
wants and behaviours, forgetting their own high level of technical expertise
(Preece, 2000). Indeed, website usability is
often perceived to be the province of the technical expert rather than the
content expert, but technical knowledge is
insufficient when it comes to designing and testing websites intended for
e-learning. Problems also arise when graphic design is prioritized over
usability, so that a page looks attractive but is difficult to read (Brinck et al., 2002).
Increased awareness of student difficulties and frustrations means that website usability is an area that is beginning to touch those who would not previously have thought an understanding of the issues around this to be relevant to their work. It is becoming apparent that for e-learning websites to be usable, what is needed is a complementary appreciation of what students expect from websites, how they learn, what motivates them, and what helps them achieve their learning goals (e.g. Johnson & Hegarty, 2003). These are some of the realisations that led us to initiate a project centred on the issues of e-learning website usability, and to continue developing our ideas over the past three years. According to one recent view, which we would endorse, usability means reaching “a threshold of acceptability beyond which users can begin to interact productively and voluntarily instead of simply acting and reacting” (Hémard, 2003:23). Laurillard expressed a similar sentiment when she wrote: “The aim is to design an interface that never intrudes on the task in hand” (Laurillard, 2002:194).
In this paper we present our methods and findings, and our overall conclusions from the project to date. We then discuss the concept of ‘pedagogical usability’, which is a set of key cosiderations that we believe should be taken on board by all those concerned with improving the experience of web-based learning.
At the Open University (OU), usability issues have been the subject of increasing interest for some time. Context experts involved in developing e-learning websites frequently request guidance about whether their educational website(s) will “work”. While they may not be able to articulate clearly which aspects of a website they wish or need to test, what is apparent is that it is increasingly necessary to provide the means by which to empower them to answer questions like, “Will it work?” for themselves, rather than relying totally on advice from technical experts. For this reason, e-learning website usability was at the heart of a project that was run jointly by staff from the Institute of Educational Technology and the Department of Languages in the Faculty of Education and Language Studies from 2001-2003. A key aim of this project was to enable academic staff and others across the OU to participate on an equal footing with other professionals (for example, graphic designers, web developers, broadcast media professionals) in discussing, checking and evaluating website usability. In this paper, we present the outcomes of our investigations, and discuss our ongoing research in this area.
The first phase of the project, which comprised four major stages, included different activities such as interviews with 14 key staff across the institution, a review of usability literature and guidelines, and two case studies in two different faculties. The activities and outcomes of this phase are summarized in Table 1, below.
|
Stage 1: A review of existing usability guidelines These were gathered from internal OU sources and external sources such as Nielsen’s website at http://www.useit.com. Since such issues bear on the efficiency of interaction and learner satisfaction, the purpose of this literature review was to identify sources that focussed on online learning and/or more general web usability issues such as navigation and on-screen text. There were two major outcomes from this stage: · An intranet website including a usability resource list that is interactive insofar as it invites visitors to the site to make additions to it, thus encouraging contributions from usability experts as well as the sharing of experience by content experts · a provisional set of guidelines for the evaluation of usability of e-learning websites |
|
Stage 2: A series of semi-structured interviews with usability experts within the OU. The purpose of these interviews was to identify which usability guidelines were in use across the institution, whether any of these were specifically tailored for the context of supported open and distance learning and the level of usability awareness among content experts. It was found that no universal or specifically-tailored guidelines were in use and that the usability awareness levels of content experts were varied. |
|
Stage 3: Two case studies The project team worked with content developers from two faculties (Education, and Health and Social Work), to identify which sorts of usability guidance they would find most or least helpful to address their needs. Content experts were most interested in the practical aspects of usability guidelines such as sharing their experiences of usability with other content experts, learning how to carry out low-cost, low-tech usability tests such as paper prototyping and finding out what resources and support were available to them when they were confronted with website usability issues. Abstract and theoretical discussions of usability guidelines were, they believed, of little help to them. |
|
Stage 4: A synthesis of the outcomes of Stages 1 – 3. The aim of this stage of the project was to identify what sort of usability guidance would be most suitable and to develop this in an appropriate format. Several outputs resulted from this work, including: · a list of 10 key usability issues, distilled from the guidelines, interviews and case studies · face-to-face workshops in faculties where project team members led content experts through usability issues they were likely to encounter while involved in e-learning website development · a web-based, self-study staff development module about website usability · a project-related website dealing with wider aspects of website usabilty both generically and for e-learning websites. |
Table 1. Stages in the first phase of the Web Usability project (2001-2002).
Key findings of this phase were that while usability guidelines existed, these had not yet been centralised or rationalised within the university, and that content experts recognised the need for usability testing but were unsure about where to obtain guidance, although they did have views on what would and would not constitute useful guidance. As a result of our investigations, we developed a working definition of usability as ‘the effectiveness, efficiency and satisfaction with which specified users can achieve identified goals in particular environments’. For OU course-related websites, such usability includes: the effectiveness with which students learn according to the stated learning outcomes of the course, the efficiency with which students interact with the website and the enjoyment or level of satisfaction gained from such interaction.
The work and outputs from the first phase of the project were taken forward during 2002-3 in several areas, including the development of usability tools that would be relevant and useful to non-technical staff. We also analysed the relationship between student ratings for websites and ways in which sites are conceived, developed and tested and investigated user experiences through interviews with students and tutors.
We developed the “Challenges” paper-based tool for supporting content experts in developing their understanding of typical pedagogical usability problems and possible solutions. The tool comprises two major sections. The first introduces users to key usability terms and concepts such as the lifecycle of the course website, from design to maintenance via implementation. It also offers practical ‘hints and tips’ about planning and design issues like the importance of adequate planning before handing the website over for programming, and effective methods of testing ideas – for example, paper prototyping – before committing the course developers to a specific approach. The second section of “ Challenges” takes 10 key usability issues or Challenges that were identified during Phase 1 of the project (Table 2) and expands on them, offering comprehensive information and advice about each.
“Challenges” recognizes that how an e-learning website is used varies widely and describes issues relevant to all course-related websites, although the specific advice given within each Challenge may be more or less applicable. In a fast-moving world, it is not possible to develop a definitive, rigid set of guidelines; “Challenges” does not, therefore, contain an exhaustive list that is ‘set in stone’, although it spans the range from technical to pedagogical considerations. Each Challenge is presented in terms of the Risks associated with it, Examples of practice, Suggested Solutions to the problem, Tests to assess progress in addressing the Challenge, with an indication of the optimal time in the development process at which to carry out such tests and, finally, a section on Related Research. Table 3 below explains and illustrates the structure and typical content of a Challenge with examples from a Challenge concerning website maintenance..
|
1. Make the role of the website clear 2. Be pedagogically rather than technologically driven 3. Integrate the learning resources into a seamless whole 4. Organize the site to meet learner needs and expectations 5. Write clearly for the medium 6. Make navigation simple 7. Editorial quality and consistency should be of equivalent quality to those of print media 8. Ensure a printer-friendly version of the website and a help section are available 9. Update the website regularly 10. Offer technical help via the website, providing clearly visible contact information for technical support |
Table 2. 10 key usability issues
|
Structure |
Example |
|
Challenge x. The challenge is stated in the heading. |
Challenge 9. Site
Maintenance: updating the website on a regular basis |
|
The Challenge This section describes a challenge. |
The Challenge To assign roles and resources to ensure that the website
is regularly maintained so that it is up-to-date, with working links. Using
the “updateability” of the website for educational benefit makes the course
more interesting and enjoyable. |
|
The Risks The risks of not responding to the challenge are outlined
in this section. |
The Risks · Sites that are outdated or clearly unmaintained lose credibility. · Not updating a site can undermine one of the key raisons d’être of online resources, i.e. being up to date |
|
Examples To underline the importance of addressing the issues
described by a Challenge users are
presented with real examples drawn
from empirical evidence such as students’ comments about their course
website. This section also draws the
user’s attention to key aspects of the example through a Comment. |
Examples When I go to the site, I just look to see
if there is anything new added to the site.
If not, I don’t stay around. Comment : This student had a strategy of checking to
see if anything new had appeared. If
not, he/she would leave the site. The developers could have capitalised on
this function as an incentive for the student to engage with the site more
regularly. Note also that the student
was making decisions based on what was fun or interesting for him/her, i.e.
his/her personal goal was to have fun/be interested by the course. |
|
Suggested solutions Since course developers’ time is usually constrained, this section offers practical solutions that are easily implemented as well as suggesting when to consider these solutions. Each solution is assigned a weighting according to its importance to and the cost of addressing a Challenge. |
Suggested Solutions Budget resources (personnel, time and money) to ensure that someone is responsible for website maintenance. (This may require both a content and technical support to be budgeted for from the outset). Weight:
Very important. The maintenance
phase of the website MUST be planned for. Cost: Minimal
if planned in advance. Otherwise, very high
in terms of stress. |
|
Tests This section outlines simple, cost-effective tests that
can be used to check that the Challenge
is being addressed at appropriate stages in the development lifecycle. “Challenges”
explains which test to do at which stage and how to do it. |
Tests ·
In the release stage: Is the site maintainer trained and
prepared to begin site maintenance once it goes live? ·
In the maintenance stage: Is the site content being kept fresh? Are
any errors being logged, reported and corrected (if appropriate)? |
|
Related Research The final information presented for a Challenge comprises
– where appropriate - a ‘Related Research’ section for users who wish to
investigate the Challenge in more
depth; this section is under constant development so that it, like course
websites, is up-to-date and relevant to users’ requirements |
Related Research Research at the National University of Singapore has shown that students value websites where content is updated frequently; see Student’s Choice awards at http://edtech.nus.edu.sg/edtech/services/scaward.asp [Accessed 28 January 2004] |
Table 3: The “Challenges” tool: an illustration
Each year, the Annual Courses Survey (Survey) is carried out by the Institute for Educational Technology (IET)'s Student Statistics Team in collaboration with the Programme for Learner Use of Media (PLUM). Students from across the OU are asked to complete a questionnaire designed to elicit their views about their learning experiences. The Survey includes a generic section which is sent to all students and asks questions concerning their experience of all learning media used in their courses In the second phase of the usability project, we examined views about course-related websites as expressed by those students who returned the Survey in 2002. Although students are not asked to give reasons for their answers, which makes it difficult to draw any firm conclusions about the views they expressed, even an examination of the relatively sparse data available revealed a pattern in students’ reported experiences of their course-related websites. Students from Languages courses, on the whole, claimed the least satisfaction with course-related websites while students whose experience with course-related websites appeared to have been the most successful were found in Social Science (most helpful resources), the Open University Business School (most frequently accessed sites) and the Science Faculty (highly satisfied with quality of information, advice and support).
The findings reported in the preceding section led us to look further into the reasons for the relatively low ratings by Languages students and further to investigate this group of students’ experience by carrying out interviews with them. Since the OU offers supported open and distance learning courses, it can be difficult to interview students in person as OU students do not attend classes on campus. They study at home or in the workplace and meet each other and their tutors only occasionally at designated sites within the geographical area in which they live, sometimes in the evening, but also at the weekend at Day Schools. For this phase of the project, students attending a Day School were approached and asked to participate in mini-interviews which aimed to pilot a method for eliciting information about student experiences of their course-related websites. When asked why they did not use their course-related websites (something that became clear from analysing responses concerning website use in the Annual Courses Survey data), three major reasons emerged: lack of time, adequacy and abundance of course materials in other media and aspects of the web technology that students believed to be inadequate, for example, one student noted “A website can’t answer questions like, ‘How do you pronounce ….?’”
The first phase of this project was concerned with the knowledge and experience of technical and content experts in designing usable educational websites, while the second phase took into account the student experience of interacting with websites produced by members of groups interviewed in the first phase of the project. What emerged was that content experts were aware of the need for usability testing but did not know how to implement it. Interviewing students revealed the importance of soliciting their views at an early stage in website development; had students been involved in testing and evaluating languages websites earlier in the development lifecycle, for example, it would have become apparent that these were not perceived as being integral to the course and were thus likely to be rarely used. Had content experts been aware of this, they may well have designed the content differently so as to integrate their websites’ content more obviously into the overall course materials. This relationship between pedagogical content and educational website usability, which is not widely explored in usability literature, led us to consider the concept of pedagogical usability in more detail.
The concept of usability comes from the discipline of HCI (Human-Computer Interaction) and was originally applied to the interaction of one user with one computer. A decade ago, Preece et al. (1994) described usability as a key concept in HCI, “concerned with making systems easy to learn and easy to use” (Preece et al., 1994:14), and distinguishable from the notion of “user experience”, which encompasses a wider set of concerns such as creating systems that are satisfying, enjoyable, fun, entertaining, helpful, aesthetically pleasing, supportive of creativity, rewarding or emotionally fulfilling. Already at that time, HCI researchers recognized that to produce systems with good usability, it was necessary to understand the psychological, ergonomic, organizational and social factors that determine how people operate, and to consider group working, integration and interaction of media, as well as the wider impacts of computer technologies. There was much emphasis on how people learn about computers, but much less was known regarding how they learn with computers. HCI being a specialised area of knowledge, it seems that there was also a gap between what was known and the actual practice of interface design, a problem we would still recognize today.
By 2000, Preece had moved on to considering usability in the context of online communities, concluding that “...software with good usability supports rapid learning, high skill retention, low error rates and high productivity. It is consistent, controllable, and predictable, making it pleasant and effective to use.” (Preece, 2000: 27). To support online communities, Preece suggests that developers have to identify software with suitable usability (the software often being web-based, or embedded in a website), “… then tailor it to more closely to meet the community’s needs” (Preece, ibid: 27), thereby implying that there is a further dimension to consider. In a subsequent book on Interaction Design, Preece et al. (2002) explain that ‘interaction design’ is defined as “designing interactive products to support people in their everyday and working lives” (also described as “finding ways of supporting people”), and it is concerned with a broader range of issues, topics and paradigms than has traditionally been the scope of HCI. It entails “creating user experiences that enhance and extend the way people work, communicate, and interact” (Preece et al., 2002: v). Usability is again distinguished from user experience, the former encompassing effectiveness, efficiency, safety, utility, learnability and memorability, whilst the latter has a focus on aspects such as satisfaction and enjoyability (as mentioned above). A difference between generic design principles and generic usability principles is also identified: “…whereas design principles tend to be used mainly for informing a design, usability principles are used mostly as the basis for evaluating prototypes and existing systems.” (ibid 26). In other words, design principles are general reminders about what to provide or to avoid when designing websites, whilst usability principles are considered to be specific and used to assess the acceptability of interfaces.
Among researchers in educational applications of computing, Laurillard (2002) addresses issues of usability from a pedagogical perspective, focusing on three aspects: user interface, design of learning activities, and checking whether learning objectives have been met. Hale & French (1999) considered the assessment of Web design based on what they described as ‘learning principles’: reducing conflict and frustration; repetition of concepts using variations in technique; positive reinforcement; active student participation; organization of knowledge; learning with understanding; cognitive feedback; individual differences; and motivation. Hall (2001) concludes that in training websites, several themes are important: site organisation, taking advantage of the hyperspace environment by building in flexibility and learner control, and use of case examples from the vast resources available on the Web. Hall also believes that collaborative learning activities should be part of such a Web environment. Nielsen (2001) has remarked that although general usability standards apply equally to e-learning, there are additional considerations, for example the need to keep content fresh in learners’ minds so that they do not forget things whilst trying to accommodate new concepts. Collings and Pearce (2002) have successfully developed a collaborative online workspace to educate university students in usability evaluation from the dual perspectives of a website developer and a user. Their focus was on “teaching and learning the process and value of usability evaluation” (Collings & Pearce, 2002: 269) and the project yielded a number of usability heuristics proposed by the students, such as ‘students should be kept informed where they are and where they should go’, ‘there should be appropriate use of language and media’, and ‘recognition rather than recall’.
In our usability project at the Open University, we have developed the notion that there are several layers of usability, namely context-specific, academic, general and technical (Muir et al., 2003):
· Context-specific usability relates to the requirements of particular disciplines and courses. Each course has its own needs and intended outcomes which make it different from other courses.
· Academic usability deals with educational issues, such as the pedagogical strategy, and the place of websites in relation to other course materials. Expected study behaviour also comes into play. The specifics of e-learning are considered at this level.
· General usability issues are common to most websites and include aspects such as clear navigation and accessibility for users with special needs. They may reflect general HCI concerns or aspects that are specific to the web.
· Technical usability addresses issues such as broken links, server reliability, download times, appropriateness of plug-ins, and accurate HTML. This is also known as the ‘functional’ usability level.
There is no doubt that the levels are mutually dependent on one another, with technical usability underpinning the other levels. For example, excessively slow download times (at the technical level) can mean that learners abandon a website or hardly ever use it. No matter how pedagogically effective the content may be, it is of little use if learners are unable to locate it in a poorly organized website, and a well-designed website may nevertheless fail if it cannot be accessed reliably from the server. Technical usability is therefore the basis for the other three levels, whilst not being sufficient by itself. It is clear that there is much overlap and interplay between the levels. In the process of gathering together our key usability principles (Challenges) for our project, we have made constant reference to the framework of the four usability levels. Nearly all of the principles can be said to belong to the ‘academic’ level of usability, as they concern issues such as making the role of the website clear, integration of learning resources, meeting learner needs, and so forth. Two of the principles - making navigation simple and ensuring that a printer-friendly version is available – most obviously belong to both the levels of general and academic usability.
Although the ‘academic usability’ level can be described alternatively as ‘pedagogical usability’, we have moved on in our thinking towards an interpretation that sees ‘pedagogical usability’ as the focal point that draws in the other aspects. Whilst a separation of levels aids basic understanding of usability, an integrative approach has the advantage of foregrounding the contribution of academic staff who understand what is important in teaching their subject, as well as the contributions of other staff with pedagogical expertise involved in e-learning (e.g. specialist library staff). A pedagogically-oriented approach does not diminish the importance of taking into consideration the perspective of learners – on the contrary, in our interpretation it includes learners’ experiences and views. We consider that knowledge of learners’ needs and behaviours is central to successful e-learning. In recent months we have been focusing on the contribution of ‘context’, taking language learning websites as an example. We have begun to explore the special considerations and requirements that come into play in that discipline, such as issues of linking through to external websites that belong to unfamiliar cultures and require a degree of intercultural competence.
E-learning brings usability into a shared arena, highlighting the need for technical or design experts and academic experts to work together more closely than ever before to produce usable websites. Having considered the difference between design principles and usability principles, we are drawn towards concluding that the distinction is theoretically useful but may lead to a separation of responsibilities that may act against the best interests of e-learning website users. Descriptions of usability which comprise effectiveness, efficiency and ‘enjoyability’, are still a good way of keeping in mind the main facets of this concept. However, to get to the heart of pedagogical usability, we have to understand more about the impact of requirements in relation to specific communities, contexts and disciplines.
We would like to thank Alexander Muir, our Research Fellow on this project, and all the staff and students who participated in interviews.
Brinck, T.,
Gergle, D., and Wood, S.D. (2002) Usability for the Web: Designing Web Sites
that Work. Morgan Kaufmann, San Francisco.
Collings, P.,
and Pearce, J. (2002) Sharing designer and user perspectives of web site
evaluation: a cross-campus collaborative learning experience. British Journal
of Educational Technology, 33, 3, 267-278.
Hale, C., and
French, D. (1999) Web-related Assessment and Evaluation. In: French, D.,
Internet Based Learning: An Introduction and A Framework for Higher Education
and Industry. Stylus Publications, USA.
Hall, R.H.
(2001) Web-Based Training Site Design Principles: A Literature Review and
Synthesis. In: Badrul Khan (ed.) Web-Based Training. Educational Technology
Publications, New Jersey.
Hémard, D.
(2003) Language Learning Online: Designing Towards User Acceptability. In:
Felix, U. (ed.) Language Learning Online: Towards Best Practice. 21-42. Swets
& Zeitlinger, Lisse.
Johnson, R.,
and Hegarty, J.R. (2003) Websites as Educational Motivators for Adults with
Learning Disability. British Journal of Educational Technology, 34, 4, 479-486.
Laurillard, D.
(2002) Rethinking University Teaching: A Conversational Framework for the
Effective Use of Learning Technologies (2nd ed.). Routledge, London and New
York.
Muir, A.,
Shield, L., and Kukulska-Hulme, A. (2003) The Pyramid of Usability: A Framework
for Quality Course Websites. Proceedings of EDEN 12th Annual Conference of the
European Distance Education Network, The Quality Dialogue: Integrating Quality
Cultures in Flexible, Distance and eLearning, Rhodes, Greece, 15-18 June 2003,
188-194.
Nielsen, J.
(2001). Just how important is usability in e-learning?. Elearning Post Website.
Retrieved January 28, 2004, from
http://www.elearningpost.com/features/archives/001015.asp
Preece, J.
(2000) Online Communities: Designing Usability, Supporting Sociability. Wiley,
Chichester.
Preece, J.,
Rogers, Y., and Sharp, H. (2002) Interaction Design: Beyond
Human-Computer-Interaction. Wiley, New York.