LLMs, GenAI, RAG, agents, evaluation · insight-1642

Capture the human's edit of a model draft as feedback, because the diff shows what was wrong

Situation
Agents edit AI-drafted customer replies before sending, and the only feedback collected is a thumbs-up on the draft.
Action
Store the draft, the sent text and the structured diff with the prompt version, then sample edits by size and category (facts, tone, policy, length). Turn recurring edit patterns into prompt rules, eval cases and, where volume allows, reviewed fine-tuning pairs.
Reason
An edit is a paired, in-context correction from the person who owns the outcome. Unlike a rating it localises the error, and a draft sent unchanged is weak evidence of acceptability, and only where review was real.
Boundary
Drafting assistants with a human in the loop, such as replies, summaries and case notes.

Symptom or risk

The most specific signal available, what a skilled person changed, is thrown away, and improvement work relies on vague ratings.

Workflows where edits are rare because users cannot change the text, or where drafts contain personal data that cannot be retained.

Edits mix preference with correctness; label a sample first.

Training on all edits, which teaches one agent's style, or counting an unedited send as quality when review was skipped.