?
Prompt Design for GPT-4 Assessments of EFL Student Reports
This study investigates the impact of different prompt design strategies on the performance of GPT-4 in
assessing undergraduate reports within an English as a Foreign Language (EFL) context. As Large
Language Models (LLMs) increasingly integrate into educational assessment, understanding how
prompt engineering affects grading accuracy and alignment with human judgment is crucial. Three
prompt design methods—TELeR Taxonomy, Six strategies framework, and Zero-shot prompting—
were applied to evaluate 40 student business reports. The assessments were compared against human
teacher scores using quantitative statistical analysis, specifically box-and-whisker plots and inter-rater
agreement between teacher ratings and AI-generated scores using Cohen’s κ and quadratic weighted κ.
Findings indicate that the choice of prompt design significantly influences the alignment between AI
and human grading. The Six strategies method demonstrated strong alignment with teacher assessments,
particularly in evaluating coherence and cohesion. The Zero-shot prompt demonstrated the strongest
alignment with teacher ratings, yielding the most similar score distribution and the highest inter-rater
agreement (quadratic weighted κ = 0.65). The results suggest that although this strategy most effectively
approximates human scoring in AI-assisted assessment, human oversight remains essential for capturing
the subjective and nuanced aspects of student writing.