Marking is not feedback: reflections on AI, context, and keeping the human in the loop

18 August 2026

Recently, I presented our EduMark AI work to a South African Community of Practice on AI in Higher Education, alongside colleagues from the University of Johannesburg and a South African EdTech developer. Our pilot with EduMark now features in a sector policy white paper on AI in assessment grading (South African Higher Education Sector, 2026). What struck me was not how different our contexts were, but how firmly we reached the same conclusion from opposite ends of the world. 

The white paper does not overstate its case. It sets out the limitations as squarely as the opportunities. AI is reliable at grading surface features such as fluency, structure and the presence of expected content, but unreliable at the judgements underneath them: whether an argument is sustained, whether evidence is weighed rather than listed, whether a student has grasped the significance of what they describe. 

One study it cites shows the gap clearly (Lundgren, 2024). GPT-4 marked sixty master’s-level essays that academics had already graded. Across the set its mean score sat close to the human mean, so at cohort level the marking looked sound. At the level of the individual script it was not: exact agreement with human markers ran between 25% and 35%, and inter-rater reliability was low (Cohen’s κ = 0.08–0.18). The model clustered its marks in the middle of the range, undervaluing the strongest work and overvaluing the weakest, and changing the prompt made little difference. Recent UK evidence at undergraduate level reports the same central tendency (Talmi et al., 2026). 

This is a validity problem, not just an accuracy problem. A matching cohort average tells us nothing about whether individual students were placed correctly, and it is precisely the boundaries between pass and fail, or merit and distinction, that a qualification exists to certify. If those distinctions are not recognised, the strongest students get no signal that their analytical work was seen, external examining and accreditation lose the judgements they depend on, and students learn to optimise for the fluency the model rewards rather than the thinking the rubric asks for. That is the difference between a tool that assists and one that quietly weakens what is being measured. 

Our own studies at Queen Mary point the same way. When rubrics are explicit and carefully engineered, AI marking aligns closely with educators; we found rubric specificity, not assessor choice, to be the strongest predictor of agreement.  

Many failures blamed on “the model” are really specification failures: gaps in how we articulate our own standards. That is hopeful, because specifications can be fixed. 

But recent work on feedback presses a deeper point worth taking seriously. The Manifesto for feedback in the age of generative AI (Winstone et al. 2025) argues that what these systems produce is information, not feedback: feedback is about learning, dialogue and sense-making, not the production of text. Winstone and colleagues (Winstone et al. 2026) describe feedback as a “care-full craft” where care, judgement and relationality are central. A model can generate accurate, rubric-aligned comments in seconds, which helps students who need timely guidance while the work is still fresh. But a comment becomes feedback only when an educator takes responsibility for it, interpreting it for a particular student and standing ready to discuss it. 

This is why the same principle runs through the white paper and our practice. The point is not simply that a human stays in the loop, but that the educator drives the judgement and keeps the locus of control (Knowles 2024). In EduMark AI, we call this guided grading: the AI handles the repetitive, structured work, while the educator sets the standards the rubric encodes and keeps the decisions it cannot, reading borderline cases, weighing fairness and progression, and holding the dialogue that turns a comment into feedback. 

Two further lessons travel well. The first is that assessment is a deeply contextual practice (Bearman et al. 2016). The judgements educators make, weighing borderline cases and applying policy on fairness and progression, are often tacit and embedded in local practice, and do not transfer cleanly into a generic system. The second is that data governance and local capability are not afterthoughts: the credible path runs through educators designing the assessment logic and institutions holding their own data. 

For Queen Mary, these translate into concrete steps: invest in rubric design and educator-led calibration, begin with formative, low-stakes use, and keep institutional control of our data. The aim was never to automate marking, but to give every student timely, meaningful feedback with a human still in the conversation. 

AI can assist with the marking. Academic judgement must remain at the centre. 

Dr Deepshikha Deepshikha

Teaching Fellow, School of Engineering and Materials Science

https://www.sems.qmul.ac.uk/staff/d.deepshikha 

References 

• Bearman, M., Dawson, P., Boud, D., Bennett, S., Hall, M., & Molloy, E. (2016). Support for assessment practice: developing the Assessment Design Decisions Framework. Teaching in Higher Education, 21(5), 545–556. https://doi.org/10.1080/13562517.2016.1160217 
• Knowles, A. M. (2024). Machine-in-the-loop writing: Optimizing the rhetorical load. Computers and Composition, 71, 102826. https://doi.org/10.1016/j.compcom.2024.102826 
• Lundgren, M. (2024). Large language models in student assessment: Comparing ChatGPT and human graders. arXiv preprint arXiv:2406.16510. https://arxiv.org/abs/2406.16510 
• South African Higher Education Sector. (2026). Artificial Intelligence in the Grading of Higher Education Assessment: An evidence-based policy framework for the responsible adoption of AI-assisted grading at NQF Level 7 and above. Discussion draft, Version 1.0, May 2026. 
• Talmi, D., et al. (2026). AI in university assessment: Evaluating the opportunities and risks of automated marking. OpRaise project report, University of Cambridge. https://www.emotional-cognition.psychol.cam.ac.uk/opraise 
• Winstone, N., Gravett, K., Noble, C., Nicola-Richmond, K., Bearman, M., Jensen, L. X., Jones, A., Corbin, T., de Kleijn, R., Gabelica, C., Kainth, R., Poobalan, A., & Reedy, G. (2025). Manifesto for feedback in the age of generative artificial intelligence. https://doi.org/10.6084/m9.figshare.30195568 
• Winstone, N. E., Gravett, K., Bearman, M., Noble, C., Jensen, L. X., Jones, A., & Nicola-Richmond, K. (2026). The care-full craft of feedback in an age of generative AI. Assessment & Evaluation in Higher Education. Advance online publication. https://doi.org/10.1080/02602938.2026.2643333