This session was held on Thursday, August 19, 2021
Data Forensics for Test Integrity
Locard’s principle on forensic evidence is summarized by the mantra “every contact leaves a trace.” While generally discussed in terms of physical evidence left at the scene of a crime, we can likewise state that when people interact with tests and test questions in abnormal and fraudulent ways they are likely to leave traces of their acts, occurring as irregularities in their answer choices, score patterns, and response times. Data forensics for test security is the use of statistical trends and analytics to detect these traces of fraudulent behavior. It has successfully been used to detect activity like proxy testing (when the test is completed by someone other than the test taker), item-preknowledge (access to active test questions prior to taking the test), and test site collusion.
Identifying suspected test fraud and taking action as early as possible is a crucial component of test security and the integrity of a certification program. This session will provide practical guidance on the types of analyses proven to detect cheating and procedures for identify associated test questions, test forms, and candidates for further investigative action. Analyses will range from simple statistical checks that can be easily implemented to more complex methods like score pattern and response similarity analysis. Examples will be presented of results consistent with normal testing behavior as well as results from confirmed incidences of test fraud.
Q&A with Nicole Tucker
Why is this topic important to the IT certification industry?
IT certifications have been shown to be extremely desirable which may increase the likelihood of these exams being targeted for fraudulent test taking. The success of a Certification Program is heavily dependent on the validity and fairness of its exams which requires a multi-method approach to test security. Data forensics adds an additional layer to the traditionally used test security protocols.
What key takeaway do you hope attendees learn or implement based on your presentation?
Attendees will be exposed to practical and proven statistical methods for identifying cheating behavior. While some of the methods will be complex and may require the assistance of a statistician, others will include simple steps that can be implemented as a first line of detection.
What’s the biggest change for the IT certification industry that this topic is driving? Or should be aware of? Trends?
The Covid-19 pandemic has resulted in the need for flexible testing options, like remote proctoring also known as online testing. While these options have been extremely helpful in allowing test takers to continue to pursue certifications to advance their careers, it also produces challenges for high stakes exams as tests are taken outside of the traditional test site environment. Introducing data forensics early as part of any overall test security strategy allows time to gather baseline data that will help to establish expected norms thus allowing easier detection if and when any cheating activity starts to occur.
About our Speaker
Director, Statistical Reporting and Analytics
PSI Services, Credentialing
Nicole Tucker is the Director of Statistical Reporting and Analytics at PSI Services LLC, leading a team of Statistical Analysts, Psychometricians, and Research Scientists specializing in psychometric reporting and analytics to support the development of valid and reliable high-stakes assessments. In addition to psychometric reporting, Nicole manages the development and application of data forensic techniques used to detect trends consistent with fraudulent test-taking behavior, like proxy testing and item harvesting. Nicole holds a Master of Science in Statistics, Measurement, and Assessment, and Research Technology (SMART) from the University of Pennsylvania, Graduate School of Education. She also earned a Bachelor of Science in Mechanical Engineering and has over 14 years of experience in test development and quantitative assessments.
