LLMs, GenAI, RAG, agents, evaluation · insight-1502
Measure per-rule compliance as the rule count in one prompt grows, and split dense prompts
- Evidence type
- empirical finding
- Difficulty
- medium
- Roles
- Applied AI, ML engineer
- Industries
- insurance, cross-industry
Summary: situation, action, reason, boundary
Situation
A claims-reply prompt has grown to about 60 rules through patches, and each new rule was checked alone on a handful of examples.
Action
Score each rule separately with the full prompt on a fixed set, repeat with rule groups removed to get compliance against rule count, and split the prompt into stages or move checkable rules into code when compliance drops below target.
Reason
IFScale tested 20 models with up to 500 instructions in one writing task; the best frontier models reached 68% accuracy at the highest density, and models showed a bias toward earlier instructions.
Boundary
Long system prompts that carry many independent formatting and policy rules.
Symptom or risk
Every rule seems to work in isolation while compliance with the full set quietly falls, and the earliest or latest rules are the ones dropped.
Exceptions
Short prompts, or rules that merely restate one behaviour. The benchmark uses synthetic keyword rules, so its numbers are not a threshold for your task.
Trade-offs
Per-rule scoring needs checkable rules or a validated judge, and splitting adds calls and latency.
Pitfalls
Reading the benchmark's 68% as a limit for your own rules instead of measuring your curve.
Lessons and concepts
Claims and their sources
Jaroslawicz et al. evaluated 20 models from seven providers on IFScale (up to 500 keyword-inclusion instructions in a business-report task); the best reached 68% accuracy at 500 instructions, and models showed a bias toward earlier instructions and distinct degradation patterns.