Original Assessment: August 2022
Lancaster Context / Provision
During the original assessment for the academic year 2020-21, there were at least 30,576 recordings uploaded to the Panopto and planet eStream video platforms used at Lancaster University. In addition, there are many more recordings stored in Microsoft Teams channels (online teaching, recorded divisional meetings and forums, working groups, etc) but due to the storage mechanism used by Microsoft we do not have the ability to identify this information. We do not know the length of recordings, but if we focus on recordings used in a teaching context, the number of timetabled teaching events (excluding physical workshops, fieldwork and lab time) for this academic year totalled 72,636 hours. We know that not all teaching sessions are recorded, but we estimate that 50% of course sessions would be a workable figure based on the data we have; this would total 36,318 hours of recordings.
In 2022, the University’s video hosting services provide automated captioning which was tested for accuracy. The current systems will provide captions with an average 83.9% - 86.4% accuracy rating.
To improve the captioning accuracy to 100% (verbatim), manual correction needs to be done. Lecturers and academic staff do not have the time available in their schedules to do this themselves and it is not a viable use of their time.
At the time of the original assessment, captioning advice was:
- Content creators should check captions and correct any major inaccuracies generated by auto-captioning technology to ensure sentence coherence. In particular, ensure all technical terms or acronyms are presented accurately and remove anything offensive or inappropriate.
- Content creators should include subject specific terminology on lecture slides or by providing a glossary alongside the course materials.
- Where a specific requirement for more accurate captions has been identified, such as in a student’s Inclusive Learning Support Plan (ILSP) or where the content creator has an identified difficulty in correcting the captions (e.g. due to a disability) support was provided by Postgraduate caption editors.
Actions taken by LU to meet legal responsibilities
Guidance for making teaching resources accessible and inclusive has been built into the University’s ‘Embrace Digital’ online portal advice for lecturers about teaching online.
Between March and July 2021, postgraduate students were employed to manually correct captions on a selection of recordings across the faculties, and the time taken to correct these captions was monitored. Each academic department was allocated a small amount of funding and given the responsibility for recruiting and managing their own postgraduate caption editors. Each department identified recordings in need of accurate captions based on a number of factors including the requirements of disabled students
It was found that Caption Editors familiar with the subject took between 2x and 3x the length of the recording to correct the captions (with subjects containing technical language and foreign language subjects taking longest).
To manually correct all captions on recordings used for teaching purposes using the model of employing postgraduate students, the estimated cost would be:
- Estimated Recorded hours: 36,318
- Min Estimated cost based on 2x recording duration to fix captions: £862,916
- Max Estimated cost based on 3x recording duration to fix captions: £1,294,373
A further budget was provided to continue the Postgraduate Caption Editor project into the 2021-22 academic year.
Difficulties identified with the Postgraduate Caption Editor project are:
- Caption editors are employed on short term contracts and departments recruit them as needed, so there is a constant need for technical training and support of new staff, plus an overhead in terms of recruitment activity
- There is no central monitoring of materials to make sure that all necessary recordings are captioned to the appropriate level, although departments are asked to sign off the completed work.
- Compared to the total number of recorded teaching sessions, only a small number of recordings have been passed to the postgraduate editors to have their captions checked for accuracy.
Whilst a useful holding response, it is evident that the current method of employing postgraduate students managed by the department to correct captions is not scalable to address all the recordings made by the university. Neither is the approach manageable or sustainable with respect to the cost of doing this (as identified above), the administration, logistics and technical support requirements.
Options for cost reduction including alternative products
The Digital Accessibility Team have investigated the accuracy and cost of various automated captioning tools. The conclusion of this investigation was that the accuracy of automated captioning using Microsoft Stream and Panopto was similar to other automated captioning solutions that are currently available, 83.9% - 86.4% accuracy rating.
The current automated captioning provision provided by Lancaster's Planet eStream video service was found to be substandard and needed improving. A business case was submitted to improve the accuracy of this service to a more appropriate level by implementing Microsoft Azure auto captioning software into Planet eStream. This will reduce the cost of manually correcting captions using this platform but will only bring them up to the level up the other automated captioning software available.
The only automated captioning solution that provides captions to the legally required level is an external captioning service such as Verbit. Such services produce automated captions that are then corrected by human caption editors. This is similar to our current model of using postgraduate students to correct errors, but the administration costs (recruitment, training, support) would be reduced as the caption editors would be outsourced, and the turnaround times are much quicker (two to three days). However, using Verbit would increase our overall costs:
- Estimated Recorded hours: 36,318
- Estimated cost for 72hr turnaround using Verbit: £1,786,845
- Estimated cost for 24hr turnaround using Verbit: £1,917,590
Timescales of making captions 100% accurate
Uncorrected automatic captions can be produced within a few hours of lecture recordings being made (the time is dependent on the load on the recording system).
The current system of employing postgraduate caption editors to correct the automatic captions on lecture recordings cannot be relied on. The regulations say that captions should be provided within 14 days of the recording being made but delaying the availability of lecture materials by this length of time will disadvantage students with hearing impairments. It can be difficult to meet these timescales due to the ad hoc nature of the employment of editors, the need for the editors to be provided with access to the lecture material and the workloads of both the academic staff and the postgraduate caption editors.
Using a third-party captioning system such as Verbit will provide corrected captions within one to three days of the recording being submitted.
The use of existing resources
Resources that have been captioned to 100% accuracy could be reused in future years as long as the pedagogical content remains the same. These would not be used as a lecture replacement but could be used for reviewing and revision by students who need lecture recordings with accurate captioning.
Equality impact assessment of not making captions 100% accurate
- Students relying on teaching materials having accurate captions will be disadvantaged if they need to wait for 14 days for the captions to be produced.
- Staff with certain disabilities may have difficulty correcting captions. Support is needed to enable such staff to produce materials with accurate captioning.
- Accurate captions are essential for people with hearing impairments. Providing teaching materials and essential university videos with suboptimal captioning will disadvantage students and staff who have a hearing impairment.
- People with language processing difficulties also rely on accurate captioning to support comprehension.
- People experiencing mental health difficulties may struggle with concentration and can benefit from using captions to support comprehension.
- Some students with disabilities benefit from using captions to support note taking. For example, students with visual impairments and specific learning difficulties often use the caption file to produce a transcript of the teaching session for further reference.
- Captions are also beneficial for students who have English as a second language and people working in noisy environments.
- Externally facing recordings could be viewed by a much wider audience who have not disclosed a disability or impairment to the university. Accurate captions are essential for all externally facing videos.
The number of students requiring accurate captions due to a disability is relatively small but tending to increase each academic year. According to the Lancaster University Student Information (LUSI) database (academic year 2021-22), there are 23 students in the current cohort (~16,000 students) who have disclosed that they have a hearing impairment or other disability and require accurate captions.
Students who have identified as having a disability are provided with an Inclusive Learning Support Plan (ILSP) which can identify their captioning requirements. This information is stored on LUSI and provided to the academic departments.
Disabled students’ allowance (DSA) can provide funding for note taking for students with hearing impairments and visual impairments, but these services are provided by specialist notetakers who are in high demand and are often unable to provide support at the Lancaster campus. Where specialist note taking support cannot be provided through DSA, students rely on the university providing appropriate accessible teaching materials (e.g. recordings with accurate captions) or adjustments to course delivery.
Many students with specific learning difficulties require note taking support and this needs to be provided by the university. Transcripts of lectures can be produced from an accurate caption file to support note taking. However, students using caption files to produce transcripts for note taking purposes (but who are not hearing impaired) may not require 100% accurate captions as they can review the audio from the recording alongside the transcript. Lecture slides that contain essential information (names and technical terminologies that may be misinterpreted by automatic captioning systems) can be provided by the presenter to support note taking or clarify information provided in the captions.
Cost benefit analysis
The cost of producing accurate captioning for all lecture and meeting recordings is prohibitively expensive and far exceeds the current budget being provided by the university. The majority of viewers of these recordings do not need 100% accurate captions (or may not need to use captions at all). For most viewers, captions can be followed and understood even when they aren't 100% accurate.
Possible remediations
- Caption everything to a usable level (automated captioning plus a check by the content creator to correct any gross errors and ensure all technical terms or acronyms are presented accurately).
- Include subject specific terminology on lecture slides or by providing a glossary alongside the course materials.
- ILSP information stored on LUSI will be used to identify the teaching modules and lecture recordings that require accurate captioning.
- All externally facing recordings and videos should be captioned to 100% accuracy.
- Where accurate captioning is required, these will be provided through a mechanism still to be agreed (e.g. postgraduate caption editors, an external service such as Verbit or by the member of staff)
