Written by John Kleeman of Learnosity, Ben Hunter of Caveon and Liberty Munson of Microsoft, with review from members of the ITCC DE&I Workgroup
Three members of the ITCC DE&I Workgroup, John Kleeman of Learnosity, Liberty Munson of Microsoft, and Ben Hunter of Caveon, recently presented at the Conference on Test Security on the connections between validity, inclusivity, and exam security. Because we believe that we can balance security, inclusivity, and validity, we wanted to share a summary of our presentation.
Validity Risks of Failing to be Inclusive
A certification exam seeks to determine if someone is competent or not, (i.e. meets a defined criterion of knowledge and/or skill). In a “perfect” certification exam, everyone who is competent will pass, and everyone who is not competent will fail.
If someone passes the exam and is not competent, that is an error, often called an error of “acceptance.” If someone fails the exam but is competent, then that is also an error, often called an error of “rejection.” The table below shows the possibilities: the lower left and upper right boxes (in green) show a correct decision, and the other boxes (in red) show an error.
Reducing these errors is important to the validity and usefulness of the exam. The reason why exam security is so important is that if someone engages in fraud or cheats on an exam, there is a high risk of an error of acceptance. That person may pass when they shouldn’t, and exam security efforts are directed to reduce the prevalence of such errors.
But if an exam is not inclusive, or if exam security measures penalize people with certain characteristics, then there will be errors of rejection. Some people will fail the exam who shouldn’t.
Both of these reduce validity, and the usefulness and relevance of the certification it is aligned with.
The key message of our presentation was to encourage those putting in place exam security measures to consider their impact on those who may be different from the typical exam candidate and, by doing so, to humanize the exam experience. Test security measures are useful because they improve validity. But if a security measure impacts people in one or more dimensions of difference adversely and inconsistently with what the exam seeks to measure that can reduce validity.
Balancing security and inclusivity
Here are a few of the examples we shared.
Certification programs often require one or more government IDs to identify candidates prior to taking an exam. Many people have a passport or driving license. But not everyone does in every country. Some people don’t drive or travel internationally, especially younger people and those less privileged. This can be a barrier to access an exam.
Another area of challenge is in remote proctoring. Allowing people to take proctored exams at home is extremely useful to people who find it difficult to travel to an exam center. However, exam security rules are often very strict, not allowing for unscheduled breaks, requiring that the candidate always look at the screen, requiring that the exam session be uninterrupted, including prohibiting interruptions by a child, and so on. This can make it difficult for parents or people with certain disabilities or personal circumstances to take the exam.
Both at exam centers and when testing remotely, rules about what a candidate can have at hand during the exam or what they can wear can be a challenge for some. For example, such rules can impact religiously or culturally imposed clothing requirements. They can be difficult for someone who needs to eat or drink frequently or for a neurodiverse person who needs an accessory, such as a fidget device or white noise machine) of some kind to perform to their best capability. Even if it is possible to ask for an accommodation for such things, asking in and of itself is an added burden that “typical” candidates are not required to go through.
To be inclusive and respect diversity,we proposed an evaluative framework when considering security measures that include the following steps:
- Identify the Activity
- Identify Current Problem & Contributing Factors
- Evaluate reasons for current process
- Identify other methods & solutions to resolve
- Implement, Analyze, and improve
The key step is to listen to and understand the views of different kinds of candidates and consider how exam security measures might impact them–to walk a mile in their shoes (or at least try to). If a security measure might impact them negatively, consider and balance the benefit against other more inclusive options that can provide a sufficiently comparative experience.
How Microsoft is Humanizing the Exam Experience
Liberty Munson, Director of Psychometrics, described some of the improvements Microsoft has made within their certification program to humanize the exam experience. In some cases they have adjusted exam security measures to remove barriers.
For example, in the past, all exam accommodations required medical documentation, but they determined that this process was a blocker for some candidates because medical proof can be difficult and expensive to obtain. After carefully reviewing the security risks associated with each type of accommodation, they’ve removed this requirement for any accommodation request that would not lead to an unfair advantage for someone who didn’t need it (and thus, removed the barriers to those who need the accommodation but didn’t have the documentation to request one). This resulted in all but a few accommodation requests being available without documentation. Further, they have ensured that as many accommodations as possible are now part of the experience itself, so no notification is even required in advance of the exam. As past Microsoft CEO, Steve Ballmer, said:
“Accessible design is good design – it benefits people who don’t have disabilities as well as people who do. Accessibility is all about removing barriers and providing the benefits of technology for everyone.”
One important innovation is that they now allow unscheduled breaks in online proctored exams without advance notice. A candidate who needs to visit the bathroom, get a snack or check on a child can select the “Take a Break” icon in the exam UI and take a break.
To ensure the security of the experience, exam time continues while a candidate is on break (as shown in the screenshot right), and after the break, the candidate can’t return to any previously viewed questions even if they were unanswered.
The unscheduled break feature has been very positively received by candidates, with over 10,000 such breaks taken in the first 8 months of 2023. Reviews have suggested that this does not compromise the validity of the scores, it just makes taking an exam more human and reasonable.
Summary
In summary, although exam security is very important to ensure the validity and trustworthiness of exam results, inclusivity is also important. It’s important to review exam security measures in the light of diversity and inclusivity and consider how best to balance the two.
If you’d like to hear a re-run of the presentation, we are running it as an ITCC Education session on 14th December.
