Automatic Item Generation

Share This Post:

Facebook
Twitter
LinkedIn
Email

Automatic Item Generation: Lessons Learned, Next Steps, and Outstanding Questions

This session was held on Thursday, April 15, 2021

The ITCC April Education Session included a presentation by Anjali Weber, Audry Hite, and Nadine McBride from Amazon Web Services (AWS).

Advances in computer-based assessment have dramatically increased the access candidates have to sit for exams. With this increased volume comes the risk of overexposure and item theft, both of which threaten exam validity and value in the field. Traditional item development methods are thorough and produce high-quality items, but they are time consuming, costly. High volume exam programs will need to implement practices that can quickly increase the available pool of items to supplement traditional item writing. Automatic Item Generation (AIG) is an assessment design strategy that has the potential to generate multiple items from a single template. In this session, we will present lessons learned and initial implementation considerations to start your journey.

Q&A with Anjali Weber

Why is this topic important to the IT certification industry?

Employers rely on IT certifications to validate proficiency at the skill levels needed to support emerging markets and navigate an increasingly competitive pool of applicants.

What key takeaway do you hope attendees learn or implement based on your presentation?

  • The importance and nuance of construct validity in item development: AIG, and AE more broadly, focuses on systematically evaluating competency along a proficiency continuum and highlights the importance of intentionally exploring the scope of your construct. 
  • Trust the process. The more theoretical foundation your exam program has (domain analysis, content validation steps, etc.), the smoother your transition to AIG methodologies will be. 
  • Our ability to provide evidence of proficiency along the continuum of a construct depends on our validation of the construct. Under AE, a task model is a carefully constructed specification for a task that requires a particular challenge to the candidate. Changes in difficulty should not be random outcomes as a function of SME interpretation or individual choice that may unintentionally redesign the construct being measured.

What’s the biggest change for the IT certification industry that this topic is driving? Or should be aware of? Trends?

  • Rapid pace of change and obsolescence of content
  • Need for efficiency in scaling efforts to growing candidate volumes
  • Mitigation of impact of content theft and proxy testing

About the Speakers

Anjali Weber is the Manager of Certification Strategic Initiatives at AWS

Audry Hite is the AIG Program Manager at AWS

Nadine McBride is a Psychometrician at AWS

Related Posts

Download Our Free Resource

There’s something about the word “certified” when it precedes a professional title that conveys the consumers and employers a sense of trust, credibility, knowledge and an official “stamp” of approval. This is not a coincidence. IT certification has long been a proven means of differentiation and qualification among professionals in the industry. Employers often include certification as a prerequisite when seeking qualified candidates to fill positions; consumers often trust only those IT professionals who boast credentials proving they have attained a certain level of knowledge.