Developing inclusive measures of learning: The UEL Omnibus Learning Gain Study so far

By Dr Fuad Ali

Context

The political economy of Higher Education (HE) is undergoing rapid transformation, with the proliferation of analytics, continuing marketization, not to mention the ongoing impacts and institutional responses to widened sectoral growth, historic fee rises, and the recent emergence of an alternative demarketised HE horizon.

With the establishment of the Teaching Excellence Framework (TEF) prompting HE administrations to take systematic quantitative stock of teaching and learning, the need and capacity for inclusive and robust ‘performance’ metrics has never been more urgent. To this end, the Omnibus Learning Gain Study is one of 13 HEFCE funded projects exploring what learning gain means in the English HE context. Here learning gain has been defined as the distance travelled, or the improvement in knowledge, skills, work-readiness and personal development demonstrated by students at two points in time.

The Omnibus Study is led by UEL, in partnership with Brunel University London and the University of Roehampton. Our approach has been to develop measures and evidence of institutional value, analyse their variability with time across a wide range of learners, institutions and disciplines and explore interrelations with inequality. Our key output will be an easily administered survey instrument which works effectively across multiple motivated students, HE organisations, disciplinary diversity, and which is supportive of the teaching demands of widening participation and employability.

Methods

Beginning in late 2015 and scheduled to complete at the end of the 2017/18 academic year, a longitudinal, mixed method approach was designed, integrating student focus groups, self-reported surveys administered to students during lectures with the support of teaching and learning support staff, as well as institutional data.

Survey scales included, the 18-item Need for Cognition (NfC) scale (Cacioppo, Petty and Kao, 1984), the Academic Behaviour Confidence (ABC) scale (Sander and Sanders, 2009), portions of the UK Engagement Study (UKES) administered by the Higher Education Academy as well as a 6-item sub scale designed to prompt students self-assessments of their academic standing with respect to their peers over the duration of their studies.

In the academic year 2016/17 over 2000, mainly first year students, participated in the study across the three partner institutions; participation in the first wave at the start of the academic year was far higher than the second wave of surveying towards the end of the teaching year.

Analysis

Although subsequent blog posts will discuss our emerging results further, we would like to share a few of the headlines with readers. At institutional level, the difference in ABC was significant at wave 1, however this significance was not found at the later survey point. The narrative of NfC was the opposite, with no significant difference at wave 1, and a significant difference emerging at wave 2.

Both Academic Behaviour Confidence (ABC) and Need for Cognition (NfC) scores were observed to have dropped over the duration of the academic year in question. This drop off, whether due to students recalibrating themselves with a new learning environment, or fatigue, can be seen in the difference between dashed and solid lines in plot of ABC score distributions for respective institutions below in figure 1.

Figure 1 - UEL Omnibus

The partial United Kingdom Engagement Survey (UKES), administered only during wave 2, allowed considerable contextual insight into the differential challenges unfolding within the lives of students. The panel plot in figure 2 below breaks down participating UEL students by age and gender, then time spent studying, in taught sessions, volunteering, working for pay, caring and commuting.

 

Figure 2 - UEL Omnibus

While only a partial sample of students participating in a survey towards the end of the academic year, the diversity of lifeworld represented here is food for thought for those in the business of designing socially just metrics, and learning provision for HE. While wanting to avoid the mishaps of contextual value add in the schools sector, we propose a Student Viscosity Model to begin a joined up treatment of structural, institutional and behavioural factors in HE.

What next?

Over the coming academic year the team will be trialling our new streamline survey instrument across UEL, Brunel and Roehampton. This streamlined 10 minute instrument was developed following reflection upon experiences administering last year’s survey, analysis of last year’s data, consultation around our organisational ecologies and several student focus groups across campuses. Retaining the Academic Behaviour Confidence scale in its entirety for its Teaching & Learning value, this year we introduce a new 8-item Motivation scale, a new 9-item Employability scale and a truncated 5-item Need for Cognition scale. Additionally, we shift the sampling frame from disciplinary variety, to multi-year cohorts of single academic programmes. Shifting the gaze longways affords us the ability to probe longer programme-scale maturation processes, and connect with and affect Employability-orientated and Programme-level stakeholders in an evidence enabled and solution-driven way.

Following workshop sessions at UEL’s Learning & Teaching Symposium with the Centre for Student Success, we will be continuing our programme of internal and external engagement. We will be speaking at Inside Government and Learning & Skills Research Network events over the coming months.

 

References

Cacioppo, J. T., Petty, R. E., & Kao, C. F. (1984). The efficient assessment of need for cognition. Journal of Personality Assessment, 48 (3): 306-307.

Sander, P. & Sanders, L. (2009) Measuring academic behavioural confidence: the ABC scale revisited. Studies in Higher Education, 34 (1): 19-35.

About Tony Hudson

Currently undertaking doctoral research at the University of East London focusing on the learning career and academic identity of Access HE teachers.
This entry was posted in Uncategorized. Bookmark the permalink.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s