LLMs, GenAI, RAG, agents, evaluation · insight-0511

Use majority voting over sampled answers for verifiable reasoning tasks, and price it

Situation
An LLM computes refund amounts from multi-step fare rules and is wrong on a noticeable share of eval cases with a single chain-of-thought answer.
Action
Sample several reasoning paths at moderate temperature, extract each final answer, take the majority, and send low-agreement cases to a deterministic calculator or human review.
Reason
Self-consistency improved accuracy over greedy chain-of-thought on arithmetic and commonsense benchmarks, because different paths that reach the same answer are more often correct.
Boundary
Tasks with a short, comparable final answer such as a number, label or option.

Symptom or risk

Single-sample reasoning errors reach customers as wrong refund amounts.

Open-ended text has no clean majority; where a deterministic calculator exists, compute the answer instead.

Cost and latency grow with the number of samples.

Treating high agreement as proof; shared misconceptions produce consistent wrong answers.

Wang et al. found that sampling diverse reasoning paths and taking the most consistent answer outperformed greedy chain-of-thought decoding on several reasoning benchmarks.
Self-Consistency Improves Chain of Thought Reasoning in Language Models