ABSTRACT

The formative evaluation of the JISC’s Learning and Teaching Programme within the context of its wider Information Environment development strategy was a three year project (EDNER) funded by JISC and was completed in July 2003. This paper provides a reflection upon the strategies which were deployed in EDNER in conducting the formative evaluation, and upon issues in evaluating large-scale digital initiatives with reference to constructs and methods utilised in UK national initiatives which are concerned with networked learning. Building blocks towards an evaluation framework for large scale national digital initiatives which are related to networked learning are identified.

Keywords

Evaluation; Information Environment; Networked learning.

INTRODUCTION

The formative evaluation of the JISC’s Learning and Teaching Programme within the context of its wider Information Environment development strategy was a three year project (EDNER) funded by JISC and was completed in July 2003. The idea of running an independent evaluation alongside a major development programme so as to encourage it to learn lessons from implementation was relatively new in 2000, as indeed was the proposal by JISC to develop an Information Environment for higher education. The EDNER evaluation was particularly concerned to explore the ways in which the development activities funded by the JISC were producing outcomes which impacted upon learning and teaching in UK higher education.

This paper provides a reflection upon the strategies which were deployed in EDNER in conducting the formative evaluation, and upon issues in evaluating large-scale digital initiatives with reference to models and approaches to evaluation utilised in related UK national initiatives such as the National Learning Network, National Grid for Learning, Learning and Teaching Support Network, LearnDirect and Culture Online. These contexts provide the opportunity to examine the lessons learned from EDNER’s strategy and to posit components of an evaluation framework for national digital initiatives.

It was apparent from the outset of the EDNER evaluation that the task of building and evaluating national-level digital services was complicated by very different perspectives among key stakeholders, and by the lack of any single, clear model on which to base development and evaluative judgements. We recognised early in our work that part of the challenge was to develop a methodological approach which was robust enough to be applied to the elucidation of processes and performance across any major national-level digital initiative and which would support such programmes in maximising the outcomes and impacts of their work. We would not claim that we have been able to give a comprehensive answer to this challenge yet, but consider we have made progress towards it.

A convergence of disciplines

The arguments for developing an evaluation framework are rooted in the premise that digital initiatives emanating from two previously distinct disciplines – Education and Information Science – are now converging. The Common Information Environment Working Group (CIEWG) has been established By JISC to help cement a collaborative, cross-sectoral partnership to build a common on-line information environment. Current membership comprises representatives from the British Educational Communications and Technology Agency (Becta), The British Library , Culture Online, The UK e-Science (GRID) Core Programme , The Joint Information Systems Committee (JISC) , The National Archives, The National Electronic Library for Health (NeLH), Resource: the Council for Museums, Archives and Libraries, and UKOLN. In describing the activities of the Group, Pothen (2003) comments that ‘in an age of specific and far-reaching commitments to lifelong learning, to e-government, to the central place of ICT in our educational and cultural lives, it makes less and less sense to see these audiences as distinct and separate, and our individual activities in isolation’. Lyon’s (2003) perception of the Common Information Environment cuts across the fields of Health, Cultural Heritage, Science and technology and Education, with Education comprising all aspects of lifelong learning from schools through colleges and universities to adult and community programmes. Borgman⁸, in noting that digital libraries support specific activities in specific contexts - classroom instruction, distance learning, digital asset management, scholarship, virtual museums - and in pointing out that digital libraries need to be evaluated as systems and as services to determine how useful, usable, and economical they are and whether they achieve reasonable cost-benefit ratios has called for the development of a digital library evaluation infrastructure.

Indeed, a simple analysis of terminology and keywords used in defined objectives or description of purpose of a range of initiatives which have Information or Education at their root demonstrates convergence in terms of Learning and Teaching in an electronic environment. This is demonstrated in Figure 1.

Figure 1. Keywords in defined objectives and description of purpose of digital initiatives

Keywords located in objectives or description	Learning	Teaching	Access	Support	Education	Delivery	Training	Quality	Advice
Archives Hub
British Educational Communications and Technology Agency
Culture Online
Electronic Libraries Programme
UK eUniversities
Ferl
JISC Information Environment
Learndirect
Learning and Teaching Support Network
National Electronic Library for Health