Evaluating AI-Generated Feedback in EFL Classrooms: Effects of Prompt Writing

Abstract

The introduction of Artificial Intelligence (AI) to English as a Foreign Language (EFL) classrooms has altered writing instruction through AI-mediated text generation and feedback. Given the difficulty of providing efficient personalized feedback to EFL learners’ writing, this research explores the influence of prompts on the quality of AI-generated responses and their successive impact on learners’ writing. The study concerns the effects of three types of prompts viz., corrective, analytical, and developmental given by 134 EFL learners enrolled for various English programs in the Department of English Language and Literature, College of Languages and Humanities, Qassim University, Saudi Arabia.  The output generated by ChatGPT in response to these prompts was utilized by the sample for accomplishing writing tasks in English which were later analyzed to identify the most effective  prompts. Moreover, a mixed survey with a combination of multiple choice, Likert-type, and open-ended questions to gain learners’ perspectives on the efficacy of ChatGPT as an assistive writing tool was utilized to help triangulate findings. Descriptive statistics, ANOVA, paired t-tests, correlation, and Cronbach's Alpha were computed to make inferences. Findings showed positive attitudes to AI feedback (M = 3.88–4.02). ANOVA showed significant differences in the outputs generated by the three types of prompts (F = 9.47, p < 0.001), with the developmental prompt exhibiting the greatest improvement (MD = 1.13) in writing quality. The study concluded that prompt quality affects AI feedback and its efficacy in EFL writing, highlighting the significance of prompt engineering in EFL instruction.