project-046 · LLM and GenAI · public-sector

Summary faithfulness evaluation for legislative bills: human claim labels vs automatic checks

A legislative-tracking team wants machine summaries of US bills for analysts. Analysts accept a shorter summary but not a wrong one. Which automatic check flags unfaithful summaries well enough to decide which summaries a person must review, and how much does ROUGE against a reference summary tell you about faithfulness?

Dataset: BillSum US bills (test split) with reference summaries
Licence: CC0 1.0 for the US bills (dataset card; US bills from GovInfo); ca_test licence unknown - not used
Reference run: none
Lessons: day 45 day 51

Data
BillSum US bills (test split) with reference summaries (public) · Train 18,949, test 3,269, ca_test 1,237 (dataset card); this project uses 60 test bills and up to 3 summaries per bill
Details on data
Baseline
ROUGE-L against the reference summary as the faithfulness signal, and a lexical support check: the share of summary content words and all numbers/dates/amounts that…
Details on baseline
Validation
Split bills 50/50 into calibration and test, keeping all summaries of a bill together. Thresholds for every checker are tuned on calibration bills and applied once to…
Details on validation
Metric
Summary-level recall and precision of unfaithful flags, review rate at the chosen threshold, per-perturbation-type detection, rank correlation of each checker and of…
Details on metric

Success criterion

A checker is usable for triage if, against the learner's claim-level human labels, it catches at least 90% of summaries containing an unsupported or contradicted claim while sending no more than half of all summaries to review (targets). The report must also state the observed relationship between ROUGE and the human faithfulness labels on the same summaries, whatever it turns out to be.

Context

Days 45 and 51 separate answer quality from groundedness. Faithfulness means every claim in the summary is supported by the source; a summary can overlap heavily with a reference and still add a wrong number. BillSum provides US bills with human-written reference summaries under CC0, which allows both a no-model mode (perturbed reference summaries) and a model mode (generated summaries).

Data

Dataset: BillSum US bills (test split) with reference summaries (public)
Licence: CC0 1.0 for the US bills (dataset card; US bills from GovInfo); ca_test licence unknown - not used
Size: Train 18,949, test 3,269, ca_test 1,237 (dataset card); this project uses 60 test bills and up to 3 summaries per bill
Verification: identity verified

Access

Hugging Face parquet (FiscalNote/billsum; no login, not gated). Use the US test split; exclude ca_test, whose California source has no licence stated in the card. Sample 60 bills of moderate length (fixed seed) so each source fits the checker's context without truncation; record the length filter.

Grain

One evaluation unit = one (bill, summary) pair; labels are per atomic claim, aggregated to a summary-level faithful/unfaithful flag.

Column	Meaning
text	Bill text (source)
summary	Human reference summary
title	Bill title
candidate_summary / origin	Derived: summary under evaluation and whether it is the reference, a labelled synthetic perturbation, or a model output
claim_labels	Derived: per claim supported / contradicted / not-in-source, written by the learner

EDA and data-quality rules

Length distribution of bills and summaries in tokens; choose the sample so every source fits the checkers' context, and report the excluded share.
Spot-check 10 reference summaries against their bills: references can themselves contain information not in the text (e.g. context from elsewhere), which must be labelled, not assumed faithful.
Claim-splitting protocol written before labelling, with examples; second labelling pass on 15 summaries to measure self-agreement.

Baseline

ROUGE-L against the reference summary as the faithfulness signal, and a lexical support check: the share of summary content words and all numbers/dates/amounts that appear in the source. Both are cheap and CPU-only.

Features

No-model mode: create labelled synthetic perturbations of reference summaries - swapped numbers, changed dates, negated obligations, swapped agencies, an added unsupported clause - each with its expected label, so detector sensitivity can be measured per error type. Model mode: summaries from a configured model with the source in context. Checkers operate per sentence or claim and aggregate to a summary flag with a threshold tuned on half of the bills.

Candidate methods

NLI-based sentence-level consistency check (SummaC-style)
Entailment against source passages targets faithfulness directly; needs an NLI model download, CPU-feasible at this size.
LLM judge with claim extraction and per-claim verdicts
Handles paraphrase and legal phrasing; needs a configured model and validation against human labels (see project-042).

Validation design

Split bills 50/50 into calibration and test, keeping all summaries of a bill together. Thresholds for every checker are tuned on calibration bills and applied once to test bills. Report detection per perturbation type and, in model mode, on naturally occurring errors separately; synthetic perturbations are easier to detect than real errors and must not be pooled silently.

Leakage checks

The learner writes perturbations and also tunes thresholds; tune only on calibration bills. If an LLM judge is used, do not include test bills as examples in its prompt.

Bias and representativeness

Bills follow formulaic legislative language, which helps lexical checks and may not transfer to news or meeting notes. Report results by bill length and by perturbation type.

Robustness checks

Truncation test: deliberately truncate sources and show how checkers misreport supported claims as unsupported; paraphrase test: rewrite five faithful summaries heavily and check false alarms.

Technical metrics

Summary-level recall and precision of unfaithful flags, review rate at the chosen threshold, per-perturbation-type detection, rank correlation of each checker and of ROUGE-L with human labels, human self-agreement.

Business interpretation

Analyst review hours (labelled assumption per summary) versus the count of unfaithful summaries that would reach readers unreviewed.

Error slices

Perturbation type: numbers and dates vs negations vs entity swaps vs added clauses.
Bill length quantiles.
Origin: reference vs synthetic perturbation vs model-generated.

Implementation steps

Load, filter by length, sample bills; write the claim-splitting and labelling protocol.
Label reference summaries and create perturbations with expected labels; run ROUGE-L and the lexical check.
Add the NLI checker (if the download is allowed) and, with a configured model, generate summaries and run the LLM-judge checker.
Tune thresholds on calibration bills, score test bills once, and analyse by slice.
Write the triage recommendation and the ROUGE-vs-faithfulness finding.

Reproduction

Learner-authored; for example uv run python -m faith.sample --seed 3 --n 60, uv run python -m faith.perturb, uv run python -m faith.score --checker lexical and optional --checker nli or --checker judge. Without a configured generator the project runs in perturbation mode and states that faithfulness of model-generated summaries was not evaluated.

Limitations

Synthetic perturbations are cleaner than real model errors; detection rates on them overstate real performance.
Single-labeller claim labels; legal nuance may need a domain expert.
Sixty bills support a method decision, not a production error-rate estimate.

Production and monitoring

Run the chosen checker on every summary, route flagged ones to an analyst, and sample unflagged ones for audit to estimate the miss rate over time. Store source version, summary, checker version and verdict for auditability. Re-validate after any model or checker change on a fixed labelled set.

Interview questions

Why can a summary with high ROUGE be unfaithful, and a faithful one have low ROUGE?
How would you validate an LLM judge for faithfulness before trusting it to gate publication?

Postmortem and alternatives

Expect lexical checks to catch numeric and date errors and miss negations; discuss whether a hybrid (lexical for numbers, NLI or judge for the rest) is cheaper than a single strong checker, and whether constraining the summary format (quote section numbers) reduces the problem at the source.

Rubric3 criteria

Criterion	Meets	Exceeds
Labelling protocol	Written claim-level protocol with self-agreement measured.	Reference summaries audited and their own unsupported content documented.
Checker evaluation	Thresholds tuned on calibration bills; results per perturbation type on test bills.	Truncation and paraphrase tests explain false alarms and misses.
Honest scope	Perturbation mode and model mode reported separately.	Triage recommendation includes an audit plan to estimate the live miss rate.

References5

On Faithfulness and Factuality in Abstractive Summarization
Intrinsic vs extrinsic hallucination in summaries and entailment-based evaluation.
SummaC: Re-Visiting NLI-based Models for Inconsistency Detection in Summarization
Sentence-level NLI aggregation for inconsistency detection.
BillSum: A Corpus for Automatic Summarization of US Legislation
How BillSum was built.
RAGAs: Automated Evaluation of Retrieval Augmented Generation
Faithfulness as a reference-free metric and the need to validate LLM-computed metrics.
Survey of Hallucination in Natural Language Generation
Hallucination taxonomy used for perturbation types.

At a glance