Disproportionate burden assessment: Captioning of teaching resources

This is Lancaster’s assessment of the WCAG regulation requirement for captioning of teaching resources, which supports a claim for disproportionate burden to caption all recordings at 100% accuracy except those cases where accurate captioning is required.

The original assessment, shown below, was in August 2022, followed by an update to current position as of August 2024.

Current responsibilities

The Public Service Bodies Accessibility Regulations 2018 requirement for captioning recorded material is based on WCAG2.2 guidelines, specificallySuccess Criterion 1.2.2: Captions (Pre-recorded).

The current legislation states that captions must include all the dialogue (either verbatim or in essence) as well as all important sounds. To achieve this, captioning accuracy should be 100% or close enough to this such that the caption text can be comprehended.

Original Assessment: August 2022

Lancaster Context / Provision

During the original assessment for the academic year 2020-21, there were at least 30,576 recordings uploaded to the Panopto and planet eStream video platforms used at Lancaster University. In addition, there are many more recordings stored in Microsoft Teams channels (online teaching, recorded divisional meetings and forums, working groups, etc) but due to the storage mechanism used by Microsoft we do not have the ability to identify this information. We do not know the length of recordings, but if we focus on recordings used in a teaching context, the number of timetabled teaching events (excluding physical workshops, fieldwork and lab time) for this academic year totalled 72,636 hours. We know that not all teaching sessions are recorded, but we estimate that 50% of course sessions would be a workable figure based on the data we have; this would total 36,318 hours of recordings.

In 2022, the University’s video hosting services provide automated captioning which was tested for accuracy. The current systems will provide captions with an average 83.9% - 86.4% accuracy rating.

To improve the captioning accuracy to 100% (verbatim), manual correction needs to be done. Lecturers and academic staff do not have the time available in their schedules to do this themselves and it is not a viable use of their time.

At the time of the original assessment, captioning advice was:

Content creators should check captions and correct any major inaccuracies generated by auto-captioning technology to ensure sentence coherence. In particular, ensure all technical terms or acronyms are presented accurately and remove anything offensive or inappropriate.
Content creators should include subject specific terminology on lecture slides or by providing a glossary alongside the course materials.
Where a specific requirement for more accurate captions has been identified, such as in a student’s Inclusive Learning Support Plan (ILSP) or where the content creator has an identified difficulty in correcting the captions (e.g. due to a disability) support was provided by Postgraduate caption editors.

Actions taken by LU to meet legal responsibilities

Guidance for making teaching resources accessible and inclusive has been built into the University’s ‘Embrace Digital’ online portal advice for lecturers about teaching online.

Between March and July 2021, postgraduate students were employed to manually correct captions on a selection of recordings across the faculties, and the time taken to correct these captions was monitored. Each academic department was allocated a small amount of funding and given the responsibility for recruiting and managing their own postgraduate caption editors. Each department identified recordings in need of accurate captions based on a number of factors including the requirements of disabled students

It was found that Caption Editors familiar with the subject took between 2x and 3x the length of the recording to correct the captions (with subjects containing technical language and foreign language subjects taking longest).

To manually correct all captions on recordings used for teaching purposes using the model of employing postgraduate students, the estimated cost would be:

Estimated Recorded hours: 36,318
Min Estimated cost based on 2x recording duration to fix captions: £862,916
Max Estimated cost based on 3x recording duration to fix captions: £1,294,373

A further budget was provided to continue the Postgraduate Caption Editor project into the 2021-22 academic year.

Difficulties identified with the Postgraduate Caption Editor project are:

Caption editors are employed on short term contracts and departments recruit them as needed, so there is a constant need for technical training and support of new staff, plus an overhead in terms of recruitment activity
There is no central monitoring of materials to make sure that all necessary recordings are captioned to the appropriate level, although departments are asked to sign off the completed work.
Compared to the total number of recorded teaching sessions, only a small number of recordings have been passed to the postgraduate editors to have their captions checked for accuracy.

Whilst a useful holding response, it is evident that the current method of employing postgraduate students managed by the department to correct captions is not scalable to address all the recordings made by the university. Neither is the approach manageable or sustainable with respect to the cost of doing this (as identified above), the administration, logistics and technical support requirements.

Options for cost reduction including alternative products

The Digital Accessibility Team have investigated the accuracy and cost of various automated captioning tools. The conclusion of this investigation was that the accuracy of automated captioning using Microsoft Stream and Panopto was similar to other automated captioning solutions that are currently available, 83.9% - 86.4% accuracy rating.

The current automated captioning provision provided by Lancaster's Planet eStream video service was found to be substandard and needed improving. A business case was submitted to improve the accuracy of this service to a more appropriate level by implementing Microsoft Azure auto captioning software into Planet eStream. This will reduce the cost of manually correcting captions using this platform but will only bring them up to the level up the other automated captioning software available.

The only automated captioning solution that provides captions to the legally required level is an external captioning service such as Verbit. Such services produce automated captions that are then corrected by human caption editors. This is similar to our current model of using postgraduate students to correct errors, but the administration costs (recruitment, training, support) would be reduced as the caption editors would be outsourced, and the turnaround times are much quicker (two to three days). However, using Verbit would increase our overall costs:

Estimated Recorded hours: 36,318
Estimated cost for 72hr turnaround using Verbit: £1,786,845
Estimated cost for 24hr turnaround using Verbit: £1,917,590

Timescales of making captions 100% accurate

Uncorrected automatic captions can be produced within a few hours of lecture recordings being made (the time is dependent on the load on the recording system).

The current system of employing postgraduate caption editors to correct the automatic captions on lecture recordings cannot be relied on. The regulations say that captions should be provided within 14 days of the recording being made but delaying the availability of lecture materials by this length of time will disadvantage students with hearing impairments. It can be difficult to meet these timescales due to the ad hoc nature of the employment of editors, the need for the editors to be provided with access to the lecture material and the workloads of both the academic staff and the postgraduate caption editors.

Using a third-party captioning system such as Verbit will provide corrected captions within one to three days of the recording being submitted.

The use of existing resources

Resources that have been captioned to 100% accuracy could be reused in future years as long as the pedagogical content remains the same. These would not be used as a lecture replacement but could be used for reviewing and revision by students who need lecture recordings with accurate captioning.

Equality impact assessment of not making captions 100% accurate

Students relying on teaching materials having accurate captions will be disadvantaged if they need to wait for 14 days for the captions to be produced.
Staff with certain disabilities may have difficulty correcting captions. Support is needed to enable such staff to produce materials with accurate captioning.
Accurate captions are essential for people with hearing impairments. Providing teaching materials and essential university videos with suboptimal captioning will disadvantage students and staff who have a hearing impairment.
People with language processing difficulties also rely on accurate captioning to support comprehension.
People experiencing mental health difficulties may struggle with concentration and can benefit from using captions to support comprehension.
Some students with disabilities benefit from using captions to support note taking. For example, students with visual impairments and specific learning difficulties often use the caption file to produce a transcript of the teaching session for further reference.
Captions are also beneficial for students who have English as a second language and people working in noisy environments.
Externally facing recordings could be viewed by a much wider audience who have not disclosed a disability or impairment to the university. Accurate captions are essential for all externally facing videos.

The number of students requiring accurate captions due to a disability is relatively small but tending to increase each academic year. According to the Lancaster University Student Information (LUSI) database (academic year 2021-22), there are 23 students in the current cohort (~16,000 students) who have disclosed that they have a hearing impairment or other disability and require accurate captions.

Students who have identified as having a disability are provided with an Inclusive Learning Support Plan (ILSP) which can identify their captioning requirements. This information is stored on LUSI and provided to the academic departments.

Disabled students’ allowance (DSA) can provide funding for note taking for students with hearing impairments and visual impairments, but these services are provided by specialist notetakers who are in high demand and are often unable to provide support at the Lancaster campus. Where specialist note taking support cannot be provided through DSA, students rely on the university providing appropriate accessible teaching materials (e.g. recordings with accurate captions) or adjustments to course delivery.

Many students with specific learning difficulties require note taking support and this needs to be provided by the university. Transcripts of lectures can be produced from an accurate caption file to support note taking. However, students using caption files to produce transcripts for note taking purposes (but who are not hearing impaired) may not require 100% accurate captions as they can review the audio from the recording alongside the transcript. Lecture slides that contain essential information (names and technical terminologies that may be misinterpreted by automatic captioning systems) can be provided by the presenter to support note taking or clarify information provided in the captions.

Cost benefit analysis

The cost of producing accurate captioning for all lecture and meeting recordings is prohibitively expensive and far exceeds the current budget being provided by the university. The majority of viewers of these recordings do not need 100% accurate captions (or may not need to use captions at all). For most viewers, captions can be followed and understood even when they aren't 100% accurate.

Possible remediations

Caption everything to a usable level (automated captioning plus a check by the content creator to correct any gross errors and ensure all technical terms or acronyms are presented accurately).
Include subject specific terminology on lecture slides or by providing a glossary alongside the course materials.
ILSP information stored on LUSI will be used to identify the teaching modules and lecture recordings that require accurate captioning.
All externally facing recordings and videos should be captioned to 100% accuracy.
Where accurate captioning is required, these will be provided through a mechanism still to be agreed (e.g. postgraduate caption editors, an external service such as Verbit or by the member of staff)

Current position: August 2024

Since the original disproportionate burden assessment was completed in August 2022, captioning of teaching resources has improved at Lancaster due to improvements in technology, investment in captioning services and increased staff training and awareness.

Advanced Captioning Service for specific students needs

Verbit Advanced Captioning Service is available to those who have specific requirements identified through their inclusive learning support plan (ILSP) or where a content creator is having difficulty creating the captions (e.g. due to disability). This has a video turnaround for 1-3 working days. In academic year 2023-2024, for the 16 students identified as needing complete captioning, 1,090 videos had advanced captioning applied by Verbit.

Improved automatic captioning

In 2022, Microsoft Azure was added into Planet eStream to improve the auto-generated captions quality. In 2024, Microsoft Azure was replaced with Whisper AI which significantly improved the accuracy of automated captioning. Through our in-house testing, using a selection of recordings from different faculties and departments, Whisper AI captions provided 98% accuracy. Panopto automatic caption quality has also improved with system updates throughout the year.

Increased training and awareness

Digital accessibility guidance is embedded within taught and online digital skills training courses and guidance. There is a specific online course around Creating Accessible Resources, which is strongly encouraged, for all computer using staff to complete which refers to captioning. The video production and editing online course has captioning guidance for each system, including Panopto, eStream and Teams.

The Digital Accessibility team have been raising awareness at departmental meetings and via an ongoing communication campaign, as well as and providing bespoke training to staff. The team have run awareness raising events, such as for Global Accessibility Awareness days.

Current guidance

Content creators should check captions and correct any major inaccuracies generated by auto-captioning technology to ensure sentence coherence. In particular, ensure all technical terms or acronyms are presented accurately and remove anything offensive or inappropriate.
Content creators should include subject specific terminology on lecture slides or by providing a glossary alongside the course materials.
Where a specific requirement for more accurate captions has been identified, such as in a student’s Inclusive Learning Support Plan (ILSP) or where the content creator has an identified difficulty in correcting the captions (e.g. due to a disability) support is provided by the Verbit Advanced Captioning Service.

Decision

Lancaster University is treating captioning for teaching resources as a disproportionate burden case based on the evidence in this assessment. Due to the relatively low number of students who require accurate captioning, the number of recordings that will be used by these students (compared to the total number of recordings made each year) and the duration that the recordings are needed for, we consider the cost of captioning all recordings to the required WCAG guidelines to be a disproportionate burden.

Remediation Plan

The university promotes an inclusive environment and should provide accurate captions to those who need it. Unfortunately, doing this in an anticipatory manner is not cost effective and can only be done upon request or by reacting to staff and students who have disclosed an appropriate disability or impairment.

The accuracy of automated captions has improved significantly and is suitable for many viewers. However, new staff join the University throughout the year, so ongoing communication is paramount to raising awareness of expectations with regards to captioning of teaching resources and the use of the advanced captioning service.

University to continue to monitor digital developments around automated captioning and to continue offering an advanced captioning service for students who require it.