Master's thesis · August 2026
Redefining Evaluation Using Adaptive AI-Generated Parsons Problems
Mercy Jerop Tum · Master of Science (MS) in Software Engineering, University of Michigan-Dearborn
Advisor: Professor Bruce Maxim · Committee: Professor DeLean Tolbert Smith, Professor Zhiwei Xu
DOI: 10.7302/dspace/30871View on Google Scholar
How it works
- Question generation
- GPT-4.1, checked against a JSON schema and rule-based quality checks
- Question format
- Four-tier diagnostic test: answer, confidence, reasoning, confidence
- Scoring
- Modified Elo rating mapped to a CEFR scale from A1 to C2
- Study
- Quasi-experimental, mixed-method; 8 students and 4 expert reviewers
What the study found
- Jielewe picked up meaningful patterns in what students understood, beyond whether their code was right.
- Its CEFR levels ran higher than the expert reviewers' judgements, so the level thresholds needed calibrating.
- Misconceptions needed more weight in the score, and a larger, more balanced dataset is needed for full validation.