Evaluation · September 2026
CFR-QA: an evaluation set for consumer-finance regulation, including refusal
JDM Inc · Research
The literature map says the field measures accuracy and rarely measures refusal. So we built the set we were missing, and we are publishing its design before we publish any score on it.
v0.1 · draft
not yet counsel-reviewed
Reg Z · E · CC · B
57
answerable questions
8
false-premise questions
12
must-decline questions
260
robustness variants
How it is built
Every answer key traces to hashed rule text. Fifteen sections of 12 CFR were pulled from the eCFR, stored and SHA-256 hashed. The build fails if any answer key is not a verbatim quote from its stored section.
Correctness is scored on facts, never on citation echo. An answer that gets the rule right without reciting the section number passes. Citation of the correct section is reported as its own separate number.
False premises are scored. A question that embeds a wrong assumption — “the APR tolerance is 1.25 points, right?” — passes only if the answer corrects it.
Refusal is scored in both directions. Declining an answerable question is a failure, and answering a must-decline question is a failure. The decline set includes a section of the CFR that does not exist and a figure that has not been published yet.
Incidental context must not change the outcome. Each answerable item is re-asked with a harmless detail prepended — “We’re a Delaware LLC.” — because that is how customers write, and because a gate that refuses on the state name is a live defect, not a scoring artifact.
The scorer is tested against known wrong answers. Real failures we have observed are kept as permanent tests, so a scoring change that quietly stops catching them fails the build.
Status, plainly. v0.1 is a 77-item pilot drafted from rule text and not yet reviewed by counsel. Until that review is done we are publishing the design, not results. A version worth calling a benchmark needs roughly 300 items, a second independent author and inter-rater agreement — and we will say which of those are done when we publish numbers.
Point it at your hardest question.
If you run compliance at a bank, credit union, lender or RIA and you want to see the evaluation set, or to put your own questions into it, write to us. We will tell you plainly whether the corpus covers them today.
research@jmvent.com