Guide · Published November 14, 2026 · Reviewed November 14, 2026 · Ingenious AI team · Not legal advice
How to Run a Human Review Queue for ADMT Decisions That Actually Changes Outcomes
The short answer: A human review queue works when four things are true: the reviewers are named people with authority to change the outcome (Colorado's standard — a reviewer who only restates the tool output does not meet it); requests enter the queue from the notice and explanation contacts without friction; each case carries the original record (notice, reasons, data summary) so review starts from evidence; and every outcome — upheld, changed, corrected-and-re-run — is recorded with its date. The kit runs this queue over its hash-chained record; this guide is the operating manual around it.
Check your duties and dates — free Build your first notice — free
What is the review queue? It is the workflow that receives a person's request for human review of an adverse automated-assisted decision, assigns it to a qualified person, and records the result. California's appeal exception uses the same shape — a qualified human reviewer who can overturn — and Connecticut assumes a point of contact who can act. The standard the queue must meet is defined in Meaningful Human Review; this guide covers running it.
Staffing: authority first, then availability
Start with authority. List who can actually reverse each decision type — not who can reply, but who can change the outcome — and make those people the queue's reviewers. If the honest list is one person, the queue is one person plus a named deputy; pretending a wider rota exists creates reviews signed by people who could not have changed anything, which the record will eventually show.
Then availability: a rota with coverage for absences, sized to your request rate. Request rates are knowable — explanations sent multiplied by the fraction who ask (in the examples across these guides, roughly one in eight to twelve adverse outcomes draws a request). Staff for the rate you measure after month one, and re-size quarterly. An unstaffed queue is worse than a small honest one, because every waiting day is recorded.
Intake: the request must arrive with its record attached
Review requests arrive through the contacts printed on the notices and explanations. Intake should attach the case automatically: the person's notice, the decision, its reasons and data summary, and any correction already submitted. A reviewer who begins by hunting through three systems spends the review on archaeology; a reviewer who opens a complete case spends it on judgment.
Log the request date at intake — not when it is assigned, not when it is opened. Several duties in the kit's data run on clocks (Colorado's 30 days for the original explanation; Connecticut's 45 days for opt-out answers), and review has no stated clock in the kit's data — which makes promptness a matter of record quality rather than deadline compliance. A queue that logs receipt honestly can show its real response times; one that logs assignment can only show its intentions.
Worked example: Blue Mesa Lending
Example — fictional business, real duty structure. Blue Mesa is a fictional Colorado lender that formalised its review queue after relying on an inbox. Its first quarter with the queue running properly:
- Reviewers named: 3 reviewers with authority to change lending outcomes, plus 1 deputy — replacing a shared inbox owned by nobody.
- Requests received: 41 review requests in the quarter, each logged at intake with the full case (notice, reasons, data summary) attached automatically.
- Response times: median 4 days from request to communicated outcome; the oldest case at quarter-end is 9 days — visible in the queue view, not discovered by complaint.
- Outcomes: 9 outcomes changed (22%) — 6 after corrected data led to re-run decisions, 3 on reviewer judgment about the original reasons; 32 upheld with written reasoning.
- Record kept: 41 requests, 41 outcomes, and the reasoning for each sit in the same chain as the original decisions, held for three years.
The review itself: fresh eyes on the same evidence, plus anything new
A meaningful review re-examines the decision: the data as it was, the reasons as stated, the tool's role as recorded — and anything the person has added, above all corrected data. The reviewer needs the power and the information to reach a different conclusion; where corrected data changes an input, the decision is re-run or re-made on the corrected basis and both versions stay in the record.
What review is not: a restatement. Colorado's rules data says it plainly — a reviewer who only restates the tool output does not meet the duty. The record should therefore show work: what was checked, what was weighed, and why the outcome stands or changes. Two or three honest sentences per case outperform a rubber stamp at exactly the moment the record is read closely. The failure mode is dissected in Token Human Review.
Outcomes and communication: close the loop with the person
Every review ends in a communication: outcome upheld with the reasoning, outcome changed with what changes for the person, or decision re-run on corrected data with the new result. The communication is part of the record, dated, and written in the same plain language as the explanation that started the chain. Silence is not an outcome state; a queue with cases ageing silently is a queue that has stopped.
Track three numbers monthly: requests received, median days to outcome, and outcomes changed. The third number is diagnostic, not a target — a permanent zero suggests review is not examining anything; a very high rate suggests the original decisions need attention. Either way, the queue is telling you something about the system upstream of it.
The queue inside the kit, and the boundary of the tool
In the kit, the review queue sits on the same hash-chained record as the notices and explanations: a request opens against a case, the case carries its documents, the review and its outcome append to the chain, and exports show the whole sequence for the 3-year retention duty. Team plans add multi-seat review — several reviewers working one queue — which matches how the rota actually operates.
The tool's boundary stays where it always is: the kit runs the queue and keeps the record; it does not judge whether the underlying decision was fair, and it does not make the review decision for you. That judgment — the actual reconsideration — is the human part the duty exists to protect. Set the queue up from the plans page, or see the documents that feed it in the free generator.
- Name reviewers who can actually change each decision type — and a deputy for each — before the first request arrives
- Attach the full case (notice, reasons, data summary, corrections) to every request automatically at intake
- Log the request date at receipt, and review response times monthly from the log, not from memory
- Require the record to show work: what was checked, what was weighed, why the outcome stands or changes
- Communicate every outcome in plain language, and watch the changed-outcome rate as a diagnostic, never a target
Sources, as carried in the kit's own data: the duty summaries in app/rules.py and the checker data in app/checker_data.py, verified against primary or authoritative sources on October 28, 2026. Run the free duties checker for your jurisdiction, decision type, and the tool's role, and compare the kit's plans for the record, builder, and review queue. More from Ingenious AI is on the hub.
Related guides
- Meaningful Human Review: What 'Meaningful' Requires in Practice
- Token Human Review: Why Rubber-Stamping the Tool's Output Fails Every State's Test
- Colorado AI Act Automated Decision Notices: What to Send
- Adverse-Outcome Explanations in Plain Language
See all ADMT guides for the full set.
Frequently asked questions
Who can be a human reviewer for ADMT decisions?
A person with authority to change the outcome. Colorado's rules data adds that a reviewer who only restates the tool output does not meet the duty; California's appeal exception requires a qualified human reviewer who can overturn the decision.
Is there a deadline for completing a human review?
The kit's data states no fixed day count for review completion. It does state clocks nearby — Colorado's 30-day explanation, Connecticut's 45-day opt-out answers — and a well-run queue logs request and outcome dates so its real times are visible.
What should the reviewer see?
The complete case: the pre-use notice, the decision with its principal reasons and data summary, the explanation sent, and anything the person has submitted since, especially corrected data.
What happens if corrected data changes an input?
The decision is re-run or re-made on the corrected basis, and both versions stay in the record — the original decision on the original data, and the new decision on the corrected data.
How many reviewers do we need?
Enough to cover your measured request rate with a deputy for absences. In the worked examples in these guides, roughly one in eight to twelve adverse outcomes draws a review request — measure your own rate after the first month.