An informal survey was undertaken by Cetis in May and June 2013. Subscribers to a number of email circulation lists – with members coming largely from institutional IT, administration and educational technology responsibilities – were invited to respond.
The purpose of the survey was to:

• Assess the current state of analytics in UK FE/HE.
• Identify the challenges and barriers to using analytics.

Preview
[pdf http://publications.cetis.org.uk/wp-content/uploads/2013/10/analytics-survey.pdf]

This briefing introduces a range of approaches and specifications for recording and exchanging data generated by the interactions of users with resources.

Such data is a form of Activity Data, which can be defined as “the record of any user action that can be logged on a computer”. Meaning can be derived from Activity Data by querying it to reveal patterns and context, this is often referred to as Analytics. Activity Data can be shared as an Activity Stream, a list of recent activities performed by an individual. Initiatives such as OpenSocial, ActivityStreams and TinCan API have produced specifications and APIs to share Activity Data across platforms and applications.

While Activity Streams record the actions of individual users and their interactions with multiple resources and services, other specifications have been developed to record the actions of multiple users on individual resources. This data about how and in what context resources are used is often referred to as Paradata. A specification for recording and exchanging paradata has been developed by the Learning Registry, an open source content-distribution network for storing and sharing information about learning resources.

[pdf http://publications.cetis.org.uk/wp-content/uploads/2013/05/paradataBriefing.pdf]

Over the past five years, as part of its overall developments in teaching and learning, The University of Huddersfield has been active in developing new approaches to assessment and feedback methodologies. This has included the implementation of related technologies such as e-submission and marking tools.

In this case study Dr Cath Ellis shares with us how her interest in learning analytics began and how she and colleagues are making practical use of assessment data both for student feedback and overall course design processes.

Preview
[pdf http://publications.cetis.org.uk/wp-content/uploads/2013/04/Acting-on-Assessment-Analytics-Vol2-No2.pdf]

Jean Mutton, Student Experience Project Manager, University of Derby, shares with us some approaches she has been spearheading in terms of using data and analytics to help improve the student experience. Through their participation in Jisc development programmes, Jean and her team (including paid student interns) have taken a service design approach that focuses on the needs of end user first.

This case study explores the wider issues around using data to inform decision making, and the strategies the University of Derby are developing to improve their student enhancement processes by addressing key questions such as:

• What is actually happening to students, how can we find out?
• What are the touch points with between students and the institution?
• What are the institutional “digital footprints” of our students?
• What really matters to our students?

Preview
[pdf http://publications.cetis.org.uk/wp-content/uploads/2013/03/CETIS-Analytics-Series-Vol-2-No-1.-Case-Study-Engaging-with-Analytics.pdf]

Analytics is notable in that it is a headline grabbing trend in many domains, but has also been around for a long time under various other labels. One consequence of that longevity is that there is a bewildering array of tools available that can support an analytics process in some way.

An exhaustive overview of all such tools is near impossible, and probably out of date the moment it’s finished. What is possible, however, is to provide a map of the major categories of tools, and highlight some landmark tools that are available now.

Because of the diverse history and practice of analytics, many different categorisations are possible, but we choose to group them by tradition, or established approach. One reason is that such an approach makes tools more easily comparable, because they have been developed to meet the needs and expectations of their communities over time. The other reason is that it tallies closely with other papers in the CETIS Analytics Series of which this briefing is a part.

[pdf http://publications.cetis.org.uk/wp-content/uploads/2013/01/Analytics-Tools-and-Infrastructure-Vol-1-No11.pdf]

Many strong claims have been made for Learning Analytics and the potential which it has to transform the education system, which deserve to be treated with caution, particularly as they regard teaching practice.

The introduction of these techniques cannot be understood in isolation from the methods of educational management as they have grown up over the past two centuries. These methods are conditioned by the fact that educational managers are limited in their capability to monitor and act upon the range of states which are taken up by teachers and learners in their learning activities. Strategies for simplification have been developed which classify the range of knowledge as a number of subjects, reduce the subjects to courses, and assign students to cohorts which carry out the same activities. Teachers, meanwhile, deal as best they can with the full variety of learners’ needs in their practice. Over the years, an accommodation has developed between regulatory authorities, management and teaching professionals: educational managers indicate the goals which teachers and learners should work towards, provide a framework for them to act within, and ensure that the results of their activity meet some minimum standards. The rest is left up to the professional skills of teachers and the ethical integrity of both teachers and learners.

This accommodation has been eroded by the efforts of successive governments to increase their control over the education received by both school and higher education students. Learning Analytics radically reduces the effort involved in gathering information on the way in which lecturers deliver the curriculum, and also to automate the work of analysing this information. An alliance of these two trends has the potential to constrain teaching practice, and therefore it is necessary to take a systemic view when assessing the impact of analytics on teaching practice.

Three types of analytics intervention are discussed, in terms of their impact on practice.

• efficiency in the wider functioning of the institution, which has few implications for teaching practice,
• enhanced regulation of the teaching and learning environment, which has potentially negative impact on teaching practice,
• methods and tools intended to help lecturers carry out their tasks more effectively, which have the potential to be a useful tool in teaching practice.

It is concluded that Learning Analytics should not be seen as a short cut to providing teaching professionals with universal advice on ‘what works’, and that its use to increase the accountability of teachers to management may have unintended negative consequences. Rather, the most promising area for enhancing teaching practice is the creation of applications which help teachers identify which of the many interventions open to them are most worthy of their attention, as part of an on-going collaborative inquiry into effective practice.

[pdf http://publications.cetis.org.uk/wp-content/uploads/2013/02/Analytics-for-Teaching-Practice-Vol-1-No-10.pdf]

The potential of analytics according to this definition is to help us to evaluate past actions and to estimate the potential of future actions, so to make better decisions and adopt more effective strategies as organisations or individuals. Analytics allows us to increase the degree to which our choices are based on evidence rather than myth, prejudice or anecdote.

Several factors are coming together at the moment to stimulate interest in making more use of analytics. One of these is the increased availability, detail, volume and variety of data from the near-ubiquitous use of ICT (Information and Communication Technology) throughout almost all facets of our lives. This aspect tends to be the focus of the news media but data alone is not enough to realise benefits from analytics. A less popularised factor driving effective exploitation of analytics is the rich array and maturity of techniques for data analysis; a skilled analyst now has many disciplines to draw inspiration from and many tools in their toolbox. Finally, the increased pressure on business and educational organisations to be more efficient and better at what they do adds the third leg to the stool: data, techniques, need.

This paper, one of the CETIS Analytics Series, is aimed at readers who wish to be introduced to the range of techniques that are being pieced together and labelled as Analytics. It does this by outlining some of the most important communities – each with their own origins, techniques, areas of limitation and typical question types – and suggests how they are contributing to the future, with special reference to the context of post-compulsory education.

The diversity and flexibility of some of the techniques lined up under the analytics flag is evidenced by the numerous different applications of analytics: financial markets, sports analytics, econometrics, product pricing and yield maximisation, fraud, crime detection, spam email filters, marketing, customer segmentation, organisational efficiency and even tracking the spread of infectious disease from web searches i. Behind these applications we can find the roots of analytics in the birth of statistics in the eighteenth century but since then different applications of statistics and IT have led to different communities of practice that now seem to be merging together. We see that Web Analytics pioneers are now expl oiting data from the “social web” by using Social Network Analysis and that the techniques of Information Visualisation are supporting interactive and exploratory forms of analysis rather than just the graphs in management reports. Subjects that some see a s old-hat such as Operational Research and others that are often perceived as futuristic such as Artificial Intelligence are each making contributions in surprising ways. Meanwhile, the education community has made its own contributions; Social Network Analysis and Artificial Intelligence have both emerged from academic research and we are now beginning to see sector – specific variants of analytics being put to work in the form of Educational Data Mining, Learning Analytics and bibliometrics.

[pdf http://publications.cetis.org.uk/wp-content/uploads/2012/12/Analytics-Brief-History-Vol-1-No9.pdf]

This briefing paper is written for managers and early adopters in further and higher education who are thinking about how they can build capability in their institution to make better use of data that is held on their IT systems about the organisation and provision of the student experience. It will be of interest to institutions developing plans, those charged with the provision of analytical data, and administrators or academics who wish to use data to inform their decision making. The document identifies the capabilities that individuals and institutions need to initiate, execute, and act upon analytical intelligence.

For the purpose of this paper, the term Learning Analytics (LA) is used to cover these activities using the definition of:

Analytics is the process of developing actionable insights through problem definition and the application of statistical models and analysis against existing and/or simulated future data. (CETIS, 2012)

The proposition behind learning analytics is not new. In the school sector particularly, good teaching practice has long involved record keeping with pen and paper and the analysis and reflection on this data to inform courses of action, and more recently using technology. Similarly, in different ways, all higher education (HE) and further education (FE) institutions use data to inform their decision making in assessment boards and course committees. However, as institutions increasingly use technology to mediate, monitor, and describe teaching, learning and assessment through Virtual Learning Evironments (VLEs) and other systems, it becomes possible to develop ‘second generation’ learning analytics. The large data sets being acquired are increasingly amenable to new techniques and tools that lower the technical and cost barrier of undertaking analytics. This allows institutions to experiment with data to gain insight, to improve the student learning experience and student outcomes, and identify improvements in efficiencies and effectiveness of provision.

[pdf http://publications.cetis.org.uk/wp-content/uploads/2012/12/Institutional-Readiness-for-Analytics-Vol1-No8.pdf]

This paper, the seventh in the CETIS Analytics Series, considers one way to explore similarities, differences, strengths, weaknesses, opportunities, etc of actual or proposed applications of analytics. It is a framework for asking questions about the high level decisions embedded within a given application of analytics and assessing the match to real world concerns. The Framework of Characteristics is not a technical framework.

This is not an introduction to analytics; rather it is aimed at strategists and innovators in post-compulsory education sector who have appreciated the potential for analytics in their organisation and who are considering commissioning or procuring an analytics service or system that is fit for their own context.

The framework is conceived for two kinds of use:

1. Exploring the underlying features and generally-implicit assumptions in existing applications of analytics. In this case, the aim might be to better comprehend the state of the art in analytics and the relevance of analytics methods from other industries, or to inspect candidates for procurement with greater rigour.
2. Considering how to make the transition from a desire to target an issue in a more analytical way to a high level description of a pilot to reach the target. In this case, the framework provides a starting-point template for the production of a design rationale in an analytics project, whether in-house or commissioned. Alternatively it might lead to a conclusion that significant problems might arise in targeting the issue with analytics.

In both of these cases, the framework is an aid to clarify or expose assumptions and so to help its user challenge or confirm them.

[pdf http://publications.cetis.org.uk/wp-content/uploads/2012/12/A-Framework-of-Characteristics-for-Analytics-Vol1-No7.pdf]

Analytics seeks to expose meaningful patterns in data. In this paper, we are concerned with analytics as applied to the process and outputs of research. The general aim is to help optimise research processes and deliver improved research results.

Analytics is the use of mathematical and algorithmic methods to describe part of the real world, reducing real-world complexity to a more easily understandable form. The users of analytics seek to use the outputs of analytics to better understand that part of the world; often to inform planning and decision-making processes. Applied to research, the aim of analytics is to aid in understanding research in order to better undertake processes of planning, development, support, enactment, assessment and management of research.

Analytics has had a relatively a long history in relation to research: the landmark development of citation-based analytics was approximately fifty years ago. Since then the field has developed considerably, both as a result of the development of new forms of analytics, and, recently, in response to new opportunities for analytics offered by the Web.

Exciting new forms of analytics are in development. These include methods to visualise research for comparison and planning purposes, new methods – altmetrics – that exploit information about the dissemination of research that may be extracted from the Web, and social network and semantic analysis. These methods offer to markedly broaden the application areas of analytics.

The view here is that the use of analytics to understand research is a given part of contemporaneous research, at researcher, research group, institution, national and international levels. Given the fundamental importance of assessment of research and the role that analytics may play, it is of paramount importance for the future of research to construct institutional and national assessment frameworks that use analytics appropriately.

Evidence-based impact agendas are increasingly permeating research, and adding extra impetus to the development and adoption of analytics. Analytics that are used for the assessment of impact are of concern to individual researchers, research groups, universities (and other institutions), cross-institutional groups, funding bodies and governments. UK universities are likely to increase their adoption of Current Research Information Systems (CRIS) that track and summarise data describing research within a university. At the same time, there is also discussion of increased ‘professionalisation’ of research management at an institutional level, which in part refers to increasing standardisation of the profession and its practices across institutions.

The impetus to assess research is, for these and other social, economic and organisational reasons, inevitable. In such a situation, reduction of research to ‘easily understandable’ numbers is attractive, and there is a consequent danger of over-reliance on analytic results without seeing the larger picture.

With an increased impetus to assess research, it seems likely that individual researchers, research groups, departments and universities will start to adopt practices of research reputation management. However, the use of analytics to understand research is an area fraught with difficulties that include questions about the adequacy of proxies, validity of statistical methods, understanding of indicators and metrics obtained by analytics, and the practical use of those indicators and metrics in helping to develop, support, assess and manage research.

To use analytics effectively, one must at least understand some of these aspects of analytics, and certainly understand the limitations of different analytic approaches. Researchers, research managers and senior staff might benefit from analytics awareness and training events.

Various opportunities and attendant risks are discussed in section 5. The busy reader might care to read that section before (or instead of) any others.

[pdf http://publications.cetis.org.uk/wp-content/uploads/2012/12/Analytics-for-Understanding-Research-Vol1-No4.pdf]

There is currently a growing interest in better exploiting data from various sources to help organisations to be more effective and a growing number of strategies for doing this are being developed across different industries. The term “analytics” is frequently being applied to these efforts but often without clarity as to what the word is intended to mean. This makes it difficult to make sense of what is happening or to decide what to appropriate from other industries or research and hinders creative leaps forward in exploring how to adopt analytics.

This paper, part of the CETIS Analytics Series, is aimed at strategists and innovators in post-compulsory education sector who are grappling with these questions in the context of a single institution or the sector at large. It does so by:

• considering a definition of “analytics”; and
• outlining analytics in relation to research management, teaching and learning or whole-institution strategy and operational concerns.

[pdf http://publications.cetis.org.uk/wp-content/uploads/2012/11/What-is-Analytics-Vol1-No-5.pdf]

The collection, processing and retention of data for analytical purposes has become commonplace in modern business, and consequently the associated legal considerations and ethical implications have also grown in importance. Who really owns this information? Who is ultimately responsible for maintaining it? What are the privacy issues and obligations? What practices pose ethical challenges?

This paper in the CETIS Analytics series covers legal, ethical and related management issues surrounding analytics in the context of teaching, learning and research and their underlying business processes. It is based on current UK law, set in the context of publicly funded Further and Higher Education and their mission. With a primary focus on personal data, it considers the rights and expectations of the data subjects (students, researchers, employees) and the responsibilities of institutions, above campus services, suppliers and funders.

[pdf http://publications.cetis.org.uk/wp-content/uploads/2012/11/Legal-Risk-and-Ethical-Aspects-of-Analytics-in-Higher-Education-Vol1-No6.pdf]

A broad view is taken of analytics for Learning and Teaching applications in Higher Education. In this we discriminate between learning analytics and academic analytics: uses for learning analytics are concerned with the optimisation of learning and teaching per se, while uses of educational analytics are concerned with optimisation of activities around learning and teaching, for example, student recruitment.

Some exemplars of the use of analytics for Learning and Teaching are to:

• Identify students at risk so as to provide positive interventions designed to improve retention.
• Provide recommendations to students in relation to reading material and learning activities.
• Detect the need for, and measure the results of, pedagogic improvements.
  Tailor course offerings.
• Identify teachers who are performing well, and teachers who need assistance with teaching methods.
• Assist in the student recruitment process.

Conclusions drawn in this paper include:

• Learning and academic analytics are underutilised in UK Higher Education.
• There are risks to adoption of analytics, including that measures that are not useful are revealed through analytics, and that staff and students lack the knowledge to use analytics. However, there are visualisation techniques that help significantly in non-specialist adoption.
• There is also a risk for institutions that delayed introduction of analytics may lead to missed opportunities and lack of competitiveness in the UK Higher Education market. Our judgment is that this is the most compelling risk to amortise.
• Institutions vary in analytics readiness and maturity, and may to a greater or lesser extent be ready for the introduction of analytics or increases in the use of analytics.
• We stress that there are different scales of analytics projects, from small projects that can be undertaken under a limited budget and time frame, to large projects that may involve, say, creating a data warehouse and employing experienced analytics staff to build complex models.
• A good way of starting is to undertake small limited-scope analytics projects. These enable institutions to develop staff skills and/or raise the profile of analytics in the institution.
• Large-scale analytics may involve the activities of staff who may variously be characterized as analysts, modellers or data scientists. These staff are often in short supply, but reference to local departments of statistics may provide expert help.
• There are commercial solutions that work in conjunction with commonly adopted Virtual Learning Environments and student information systems. Some of these are ‘plug-and-play’ and do not require analyst input. However, before purchase, they should be evaluated for suitability, particularly with respect to intuitional pedagogic approaches and concerns.

[pdf http://publications.cetis.org.uk/wp-content/uploads/2012/11/Analytics-for-Learning-and-Teaching-Vol1-No3.pdf]

The benefits afforded by the longitudinal collection and analysis of key institutional data are not new to enterprise IT managers nor to senior management more generally. Data warehousing and Business Intelligence (BI) dashboards are integral to the modern management mindset and part of the ‘enterprise IT’ architecture of many Higher Education Institutions and Further Education Colleges.

However some things are changing that pose questions about how business intelligence and the science of analytics should be put to use in customer facing enterprises:

• The demonstration by online services ranging from commodity sales to social networks of what can be done in near-real time with well-connected data.
• The emphasis brought by the web to the detection, collection and analysis of user activity data as part of the BI mix, ranging from clicks to transactions.
• The consequent changes in expectations among web users of how services should work, what businesses could do for them, accompanied by shifts in legal and ethical assumptions.
• The availability of new types of tools for managing, retrieving and visualizing very large data that are cheap, powerful and (not insignificantly) accessible to grass roots IT users.

Set against that backdrop, this paper aims to;

• Characterise the educational data ecosystem, taking account of both institutional and individual needs.
• Recognise the range of stakeholders and actors – institutions, services (including shared above-campus and contracted out), agencies and vendors.
• Balance strategic policy approaches with tactical advances.
• Highlight data that may or may not be collected.
• Identify opportunities, issues and concerns arising.

Our focus is therefore not on technology but rather on high value gains in terms of business objectives, the potential for analytics and opportunities for new thinking across the organisation.

[pdf http://publications.cetis.org.uk/wp-content/uploads/2012/11/Analytics-for-the-Whole-Institution-Vol1-No2.pdf]

This paper provides a high level overview to the CETIS Analytics Series. The series explores a number of key issues around the potential strategic advantages and insights which the increased attention on, and use of, analytics is bringing to the education sector. It is aimed primarily at managers and early adopters in Further and Higher Education who have a strategic role in developing the use of analytics in the following areas:

• Whole Institutional Issues,
• Ethical and Legal Issues,
• Learning and Teaching,
• Research Management,
• Technology and Infrastructure.

Preview:

[pdf http://publications.cetis.org.uk/wp-content/uploads/2012/11/Analytics-Vol1-No1-Briefing-Paper-online.pdf]