Towards an Evaluation Framework for Large Scale Networked
Learning Initiatives
Shelagh Fisher
Manchester
Metropolitan University
The formative evaluation of the JISC’s Learning and Teaching Programme
within the context of its wider Information Environment development strategy
was a three year project (EDNER) funded by JISC and was completed in July
2003. This paper provides a reflection
upon the strategies which were deployed in EDNER in conducting the formative
evaluation, and upon issues in evaluating large-scale digital initiatives with
reference to constructs and methods utilised in UK national initiatives which
are concerned with networked learning.
Building blocks towards an evaluation framework for large scale national
digital initiatives which are related to networked learning are identified.
Evaluation; Information Environment; Networked learning.
The formative evaluation of the JISC’s Learning and Teaching Programme
within the context of its wider Information Environment development strategy
was a three year project (EDNER) funded by JISC and was completed in July 2003. The idea of running an independent
evaluation alongside a major development programme so as to encourage it to
learn lessons from implementation was relatively new in 2000, as indeed was the
proposal by JISC to develop an Information Environment for higher education. The
EDNER evaluation was particularly concerned to explore the ways in which
the development activities funded by the JISC were producing outcomes which
impacted upon learning and teaching in UK higher education.
This paper provides a reflection upon the strategies which were deployed
in EDNER in conducting the formative evaluation, and upon issues in evaluating
large-scale digital initiatives with reference to models and approaches to
evaluation utilised in related UK national initiatives such as the National
Learning Network, National Grid for Learning, Learning and Teaching Support
Network, LearnDirect and Culture
Online. These contexts provide the opportunity to examine the lessons learned
from EDNER’s strategy and to posit components of an evaluation framework for
national digital initiatives.
It was apparent from the outset of the EDNER evaluation that the task of
building and evaluating national-level digital services was complicated by very
different perspectives among key stakeholders, and by the lack of any single,
clear model on which to base development and evaluative judgements. We
recognised early in our work that part of the challenge was to develop a
methodological approach which was robust enough to be applied to the
elucidation of processes and performance across any major national-level
digital initiative and which would support such programmes in maximising the
outcomes and impacts of their work. We would not claim that we have been able
to give a comprehensive answer to this challenge yet, but consider we have made
progress towards it.
The arguments for developing an evaluation framework are rooted in the
premise that digital initiatives emanating from two previously distinct
disciplines – Education and Information Science – are now converging. The Common Information Environment Working
Group (CIEWG) has been established By JISC to help cement a collaborative,
cross-sectoral partnership to build a common on-line information environment. Current membership comprises representatives
from the British Educational Communications and Technology Agency (Becta), The
British Library , Culture Online, The UK e-Science (GRID) Core Programme ,
The Joint Information Systems Committee (JISC) , The National Archives,
The National Electronic Library for Health (NeLH), Resource: the
Council for Museums, Archives and Libraries, and UKOLN. In describing the activities of the Group,
Pothen (2003) comments that ‘in an age of specific and far-reaching commitments
to lifelong learning, to e-government, to the central place of ICT in our
educational and cultural lives, it makes less and less sense to see these
audiences as distinct and separate, and our individual activities in
isolation’. Lyon’s (2003) perception of the Common Information Environment cuts
across the fields of Health, Cultural Heritage, Science and technology and
Education, with Education comprising all aspects of lifelong learning from schools through colleges and
universities to adult and community programmes. Borgman8, in noting that digital libraries support
specific activities in specific contexts - classroom instruction, distance
learning, digital asset management, scholarship, virtual museums - and in
pointing out that digital libraries need to be evaluated as systems and as
services to determine how useful, usable, and economical they are and whether
they achieve reasonable cost-benefit ratios has called for the development of a
digital library evaluation infrastructure.
Indeed, a simple analysis of terminology and keywords used in defined
objectives or description of purpose of a range of initiatives which have Information or Education at their
root demonstrates convergence in terms of Learning and Teaching in an
electronic environment. This is
demonstrated in Figure 1.
Figure 1. Keywords
in defined objectives and description of purpose of digital initiatives
|
Keywords located in objectives or
description |
Learning |
Teaching |
Access |
Support |
Education |
Delivery |
Training |
Quality |
Advice |
|
Archives Hub |
|
|
|
|
|
|
|
|
|
|
British Educational Communications and Technology
Agency |
|
|
|
|
|
|
|
|
|
|
Culture Online |
|
|
|
|
|
|
|
|
|
|
Electronic Libraries Programme |
|
|
|
|
|
|
|
|
|
|
UK eUniversities |
|
|
|
|
|
|
|
|
|
|
Ferl |
|
|
|
|
|
|
|
|
|
|
JISC Information Environment |
|
|
|
|
|
|
|
|
|
|
Learndirect |
|
|
|
|
|
|
|
|
|
|
Learning and Teaching Support Network |
|
|
|
|
|
|
|
|
|
|
National Electronic Library for Health |
|
|
|
|
|
|
|
|
|
|
National Grid for Learning |
|
|
|
|
|
|
|
|
|
|
National Learning Network |
|
|
|
|
|
|
|
|
|
|
NHS Direct |
|
|
|
|
|
|
|
|
|
|
NOF-digi |
|
|
|
|
|
|
|
|
|
|
People's Network |
|
|
|
|
|
|
|
|
|
|
Research Support Libraries Programme |
|
|
|
|
|
|
|
|
|
|
UK-Online |
|
|
|
|
|
|
|
|
|
A problem which taxed the evaluators from the outset of EDNER was in
deciding on which areas to focus. The problem arose not because of the lack of
a detailed evaluation plan, but because the ‘thing’ to be evaluated was dynamic
and evolving in volume, scope and purpose.
This problem is intrinsic to the evaluation of large-scale digital
initiatives, whether formative or summative.
In outlining challenges to digital library evaluation, Saracevic6 asserts that key issues inherent in
large-scale evaluation of digital libraries are ‘what to evaluate?’ (Construct)
and ‘how to evaluate?’ (Methods).
Research and development of large scale digital library initiatives over
the last decade has generated significant ongoing debate on the development of
tools and frameworks for evaluation of these large scale initiatives. In proposing an evaluation framework for
digital community information systems, Unruh, Pettigrew and Durrance7
conclude that the critical question of exactly what to attend to in such an
evaluation remains.
Within EDNER, the evaluation ‘problem’ (research question) as defined in
the call for proposals (JISC, 1999) to undertake the evaluation could be
described as ill-defined – ie. to undertake “formative evaluation of the DNER, working closely
with DNER developers in testing user acceptance of the DNER and identifying
improvement strategies”. Also seemingly ill-defined at the outset, were the objectives of
the primary construct to be evaluated – the (then) DNER (ie.
““to provide the globe's high quality digital content to staff and
students in (UK) higher and further education, at any time, and from anywhere.”). It could be argued that such visionary
objectives are impossible to measure, but, as noted earlier, the idea of
running an independent evaluation alongside a major development programme so as
to encourage it to learn lessons from implementation was relatively new in
2000, so both developers and evaluators needed to experiment with ideas and
applications in forging new territory. The EDNER team could not look to
precedents in guiding the evaluation design.
There was no precedent in the developing entity (ie. DNER/Information
Environment) or in approaches to evaluation of such an entity.
Strategies adopted within the EDNER formative evaluation included an
analysis of constituent roles and services of the nascent Information
Environment using document and web content analysis; student evaluation of the
quality attributes of a sample of contributing services using test questions
; analysis of the extent and manner in
which services were becoming embedded within UK HE institutions, using
search/link techniques; close monitoring and expert commentary on the
developing presentation layer, including portals; case studies of two, and a
survey of a larger sample of , HE institutions to determine the impact of the
L&T programme and the developing Information Environment, using interviews
and questionnaires; a critical review of information needs and information use
in HE, using documentary analysis and interviews; and a pedagogical assessment
of L & T projects, using project logic maps, ‘history of the future’
exercises and document analysis.
EDNER also undertook to identify approaches to the evaluation of large
scale national digital initiatives which directly contributed to, or supported,
networked learning to provide intelligence on related evaluation activity to
JISC. It is from this aspect of the
work within EDNER that building blocks towards an evaluation framework for
large scale national digital initiatives which supply platforms for e-learning
or are related to networked learning have been identified and are outlined in
the following section.
This section provides a summary
analysis of evaluation approaches to national digital initiatives and to other
major relevant evaluation activity in a digital information context. This
overview is presented according to the Construct on which evaluations
have focused. Published reports of
evaluation activity for the range of digital initiatives outlined in Figure 1
were scrutinized, in the first instance to identify constructs (what is
being evaluated?) and methods (how is the evaluation being conducted?). It is suggested here that the constructs
provide the primary foundation towards developing a framework for evaluating
large scale networked learning initiatives.
In the main, the constructs were
found to centre around users,
but the evaluations have focussed on a wide range of characteristics and
criteria relating to users. Figure 2
outlines the key constructs used in evaluating initiatives related to networked
learning. (The source evaluation report
is referenced in parentheses)
Figure 2. Evaluation
constructs in UK digital initiatives
|
CONSTRUCT |
DIGITAL INITIATIVE (Evaluation Report) |
|
Content / Landscape |
JISC Information Environment (Brophy, Fisher and Jones, 2004) |
|
Educational potential |
Culture online (Taylor Nelson, 2001) |
|
Evaluation design |
Electronic Libraries Programme
(Kelleher, 1995) |
|
|
People’s Network |
|
Information architecture |
JISC Information Environment (Brophy, Fisher and Jones, 2004) |
|
Institutional impact |
Learning and Teaching Support Network
(Saunders et al, 2002) |
|
|
JISC Information
Environment (Brophy, Fisher and
Jones, 2004) |
|
Mapping activities against objectives |
Electronic Libraries Programme
(Esys, 2000, 2001) |
|
Pedagogical viewpoints |
JISC Information Environment (Brophy, Fisher and Jones, 2004) |
|
Programme processes and outputs |
Electronic Libraries Programme
(Esys, 2000) |
|
Project logic |
JISC Information
Environment (Brophy, Fisher and
Jones, 2004) |
|
Relation to similar
projects/initiatives |
JISC Information
Environment (Brophy, Fisher and
Jones, 2004) |
|
Strategic context |
National electronic Library for Health
(McTaggart, 2001) |
|
Take-up |
NHS Direct Online (Nicholas et al, 2001) |
|
Usability |
NHS Direct online (Nicholas et al, 2001) |
|
|
National electronic Library for Health
(Urquhart, Yeoman and Sharp, 2002) |
|
|
Archives Hub (Ramsden, Kelleher and
Russell, 2002) |
|
|
JISC Information Environment (Brophy, Fisher and Jones, 2004) |
|
Value |
National electronic Library for Health
(McTaggart, 2001) |
|
|
NHS Direct online (Nicholas et al, 2001) |
|
Users |
|
|
Awareness (of
initiative) |
Archives Hub (Ramsden, Kelleher
and Russell, 2002) |
|
|
Learning and Teaching Support
Network (Saunders et al, 2002) |
|
|
JISC Information
Environment (Brophy, Fisher and
Jones, 2004) |
|
Behaviour of |
Culture online (McKinsey and Company, 2001) |
|
|
JISC Information
Environment (Brophy, Fisher and
Jones, 2004) |
|
Demographics of |
Culture online (McKinsey and
Company, 2001) |
|
Expectations
of |
Culture online (DCMS, 2001) |
|
Experience of |
NHS Direct online (Commission for Health
Improvement (2003) |
|
Impact on |
National electronic Library for Health
(McTaggart, 2001) |
|
|
Learning and Teaching Support Network (Saunders et al, 2002)
|
|
|
Ferl Practitioners’ Programme (Ferl,
2003) |
|
|
Learn Direct (Tamkin et al, 2003) |
|
|
National Grid for Learning (Harrison et
al, 2003) |
|
|
National Learning Network (LSDA, 2002) |
|
|
Research Support Libraries Programme
(Tavistock Institute, 1999) |
|
|
UK Online (Wyatt et al, 2003) |
|
Needs of |
Culture online ( McKinsey
& Company, 2001 ) |
|
|
JISC Information
Environment (Brophy, Fisher and
Jones, 2004) |
|
|
RSLG
(HEFCE, 2003) |
|
Skills acquired |
UK online (Wyatt, J et al., 2003) |
|
Value (of initiative
to) |
Archives Hub
(Ramsden, 2003) |
|
|
National electronic
Library for Health (Urquhart et al., 2001) |
In summary, the following
methods were used in the evaluations listed above.
Figure 3. Range of methods utilized in evaluating UK
digital initiatives
|
Analysis of project
reports |
Analysis of search terms |
|
Analytical exercises |
Cartoon tests |
|
Case
studies |
Citation analysis |
|
Computer logs |
Document analysis |
|
Expert commentary |
Group discussions |
|
Focus groups |
Interviews |
|
History of the
Future exercise |
Manager surveys |
|
Literature review |
Project logic maps |
|
Pilot-testing |
Regional
workshops for validation of findings |
|
Self-reporting |
Sentence and story
completion |
|
Surveys |
Usability assessments |
|
User survey |
User testing |
|
Vignettes |
Web site appraisal |
|
Web survey |
|
|
Workbook exercises |
|
In conclusion it can be seen that a wide variety of characteristics,
criteria and methods have been identified both for use within EDNER and being
used in other related evaluations. This illustrates the problem outlined at the
start, namely that developing a robust evaluation methodology for large scale
digital initiatives is extremely complex, especially if the issues of outcomes
and impact are to be taken seriously.
However, components for an evaluation framework for national digital
initiatives can be identified. Building
blocks towards an evaluation framework for large scale national digital
initiatives which supply platforms for e-learning or are related to networked
learning are in existence. Constructs
and Methods identified within evaluation activity for such initiatives as the
JISC Information Environment, National Learning Network, National Grid for
Learning, Learning and Teaching Support Network, People’s Network and Culture
Online etc provide a sound foundation for collaborative activity. These preliminary analyses suggest the need
for an evaluation framework to be developed under the auspices of a
cross-sectoral body such as the Common Information Environment Working Group.
Borgman, C.L. (2002) Fourth DELOS Workshop. Evaluation of Digital Libraries: Testbeds, Measurements, and
Metrics: final report to National Science Foundation, Computer and Information Science Directorate, Information and
Intelligent Systems Division, Digital Libraries Program
http://www.dli2.nsf.gov/internationalprojects/working_group_reports/evaluation.html
Brophy, P., Fisher, S. and Jones, C.R. (2004) EDNER: Final Report. Manchester: CERLIM (Centre for Research in Library
& Information Management). (in press)
Commission for Health Improvement (2003) NHS Direct online. Hampshire Ambulance Service NHS Trust,
NHS Direct Hampshire and Isle of Wight and NHS Direct Online. Clinical Governance Review. Commission for
Health Improvement (CHI).
http://www.chi.nhs.uk/eng/organisations/south_east/hamps_amb_nhs_direct/2003/hampshire_amb.pdf
ESYS Consulting (2000) Summative evaluation of phases 1 and
2 of the eLib initiative: final
report ESYS http://www.ukoln.ac.uk/services/elib/info-projects/phase-1-and-2-evaluation/elib-fr-v1-2.pdf
ESYS Consulting (2001) Summative evaluation of phase 3 of
the eLib initiative: final report
ESYS
http://www.ukoln.ac.uk/services/elib/papers/other/summative-phase-3/elib-eval-main.pdf
Ferl (2003) Ferl Practitioners’ Programme. Pilot Evaluation
Report.
http://ferl.becta.org.uk/content_files/pages/fpp_02/fpp_03/consultation_and_eval/fpp_eval_rpt_final1.doc
Harrison, C. et al (2003)
ImpaCT2: The Impact of
Information and Communication Technologies on Pupil Learning and
Attainment, Becta
http://www.becta.org.uk/page_documents/research/report01.pdf
HEFCE (2003)
Research Support Libraries Group. Final Report http://www.rslg.ac.uk/final/final.pdf
JISC (1999) Developing the DNER for Learning and Teaching
(Circular 5/99) http://www.jisc.ac.uk/index.cfm?name=circular_5_99
LSDA (2002) The National Learning Network Final report to
the Evaluation Steering Group and the NLN Programme Board by the LSDA and
Sheffield Hallam University
http://www.nln.ac.uk/uploadedfiles/NLN_FinalReport.pdf
Lyon, L. The Common Information Environment in Context.
Paper presented at the CIE Awayday, April 2003
http://www.ukoln.ac.uk/ukoln/staff/e.j.lyon/apr03-awayday.ppt
McKinsey & Company (2001) Cultural Web Site Visitor
Survey Results from Cultural Institutions ,
Department for Culture, Media and Sport
http://www.cultureonline.gov.uk/html/Drummond_Madell_col_files/frame.htm
McTaggart, B. (2001). NeLH Pilot Evaluation
Project. Final Report for the NHS Information Authority NCC Group http://www.nhsia.nhs.uk/nelh/pages/background.asp
Nicholas D. et al (2001) NHS Direct Online Interactive
Enquiry Service Evaluation of the Pilot stage. Report submitted to NHS Direct
Online Operational team. London: City University
Pothen, P. (2003) Building a Common Information Environment
Library and Information Update Dec
http://www.cilip.org.uk/update/issues/dec03/article5dec.html
Ramsden, C., Barkat,
S., and Kelleher, J. (2003) Summative Evaluation of the Archives Hub. Tavistock
Institute http://www.archiveshub.ac.uk/tavistock2003.pdf
Ramsden, C., Kelleher, J., and Russell, S. (2002)
Formative Evaluation of the Archives Hub. Tavistock Institute
http://www.archiveshub.ac.uk/tavistock.pdf
Saracevic, T. (2000). Digital Library Evaluation: Toward
Evolution of Concepts. Library Trends, 49, (2) 350-369.
Saunders, M et al. (2002) Evaluating the Learning and
Teaching Support Network: from awareness to adaptation. Deliverable D10. Second
Annual Report to the LTSN Executive. http://www.ltsn.ac.uk/for-us/index.asp?id=163
SRU Ltd (2001) Culture Online: Second Stage Qualitative
Study Report, Department for Culture, Media and Sport
http://www.cultureonline.gov.uk/html/intro/sru_report.pdf
Stern, E and Kelleher, J. (1995) Evaluation of the
Electronic Libraries Programme. report of a brief consultancy for FIGIT/JISC.
The Tavistock Institute
http://www.ukoln.ac.uk/services/elib/papers/tavistock/figit-consultancy/
Tamkin, P. et al (2003) New Learners, New Learning: A
Strategic Evaluation of Ufi .Institute
for Employment Studies
http://www.dfes.gov.uk/research/data/uploadfiles/RR440.pdf
Tavistock Institute (1999) Evaluation of the Research
Support Libraries Programme Preliminary Evaluation Tavistock Institute, Evaluation Development & Review
Unit
http://www.rslp.ac.uk/AboutUs/eval.doc
Taylor Nelson (2001) Results of
omnibus survey of people in England.. Interest
in arts and culture and Culture Online Taylor Nelson Sofres Phonebus.
http://www.cultureonline.gov.uk/html/intro/omnibus_results.pdf
Unruh, K. T., Pettigrew, K. E., &
Durrance, J. C. (2002). Towards effective evaluation of digital community information systems.
In E. G. Toms (Ed.), Proceedings of the 65th Annual Meeting of the American
Society for Information Science & Technology (pp. 250-259). Philadelphia,
PA. Nov. 18-21, 2002.
Urquhart, C. et al
(2001) NeLH Pilot Evaluation Project.
Final Report. NHS Information
Authority http://www.nhsia.nhs.uk/nelh/pages/documents/aber.doc
Wyatt, J et al (2003) Evaluation of CMF funded UK Online
centres – Final report
http://www.helpisathand.gov.uk/info/print/publications/other/cmf-evaluation/evaluation-of-cmf-funded-uk-online-centres-final-report-pdf-280kb.pdf
http://www.archiveshub.ac.uk/
http://www.becta.org.uk/
http://www.cultureonline.gov.uk/
http://www.cerlim.ac.uk/edner/
http://www.ukoln.ac.uk/services/elib/
http://www.ukeu.com/
http://ferl.becta.org.uk/
http://www.jisc.ac.uk/
http://www.jisc.ac.uk/index.cfm?name=wg_cie_home
http://www.learndirect.co.uk/
http://www.ltsn.ac.uk/
http://www.nelh.nhs.uk/
http://www.ngfl.gov.uk/
http://www.nln.ac.uk/
http://www.nhsdirect.nhs.uk/
http://www.nof-digitise.org/
http://www.peoplesnetwork.gov.uk/
http://www.rslp.ac.uk/
http://www.ukonline.gov.uk/