Human in the Loop AI: Design Reviews That Can Keep Up

Human in the loop AI includes people in an AI system’s training, evaluation, or operation. In a business workflow, it might mean an employee corrects an extracted order, approves a proposed refund, or takes over an ambiguous customer case before work continues.

This article focuses on that operational review—not on building a data-labeling workforce. Google Cloud’s HITL overview explains the broader lifecycle.

My recommendation is to design the review as real work: give someone evidence, authority, enough time, and an explicit next step. “An employee will check it” does not establish any of those. Start by checking whether the promised review is even possible at your expected volume.

Can your reviewers keep up?

Suppose an AI-assisted intake process receives 100 cases per hour. You expect 10% to need human review, and each review takes eight active minutes. The person assigned to review has 45 protected minutes per hour; the other 15 minutes are needed for unrelated work.

These are illustrative assumptions, not a measured client operation or a staffing benchmark.

Review workload = incoming cases × review share × active minutes per review

The workload is 100 × 10% × 8 = 80 review minutes per hour. One reviewer supplies only 45. That leaves 35 minutes of review work unhandled every hour, if these rates persist. Even assigning their entire hour would leave a 20-minute shortfall.

Change the assumptions and the staffing decision changes:

The two-reviewer case uses 80 ÷ 90 = 88.9% of protected capacity. That is a feasibility calculation, not a turnaround guarantee. Bursts, difficult cases, absences, breaks, and rework can still create a queue. It also says nothing about whether the decisions are correct.

The four-minute case is a possible redesign target, not an achieved improvement. Showing the original document and relevant policy together might reduce searching, but you must time the resulting work. Do not cut the review itself in half and assume quality survives.

When capacity is insufficient, add protected coverage, improve preparation, reduce intake, or narrow the pilot. Do not silently remove necessary review triggers to make a staffing spreadsheet balance. Cases that cannot proceed need an existing manual or escalation path, not an indefinitely growing inbox.

Choose where a person must intervene

Place review before the consequence you need to prevent. Checking a draft before it is sent is different from discovering an incorrect promise after a customer receives it.

For a small pilot, reviewing every proposed action can be reasonable. You are learning which cases occur and whether the preparation is useful. That recommendation changes as volume rises and you obtain reliable evidence about a well-defined subset of work.

For low-consequence, reversible work, rules may permit routine cases to proceed while people handle exceptions and examine a sample of completed cases. Sampling can reveal quality problems; it cannot prevent every sampled-out error. Retain pre-action review when the potential harm cannot be adequately recovered afterward.

Useful triggers include missing evidence, conflicting records, an action outside delegated authority, or a proposed commitment that needs an accountable decision-maker. Keep triggers explicit enough to test. An unknown document should reach a known intake owner, as shown in AI document classification, rather than being forced into the nearest category.

Do not interpret a model’s “95% confident” statement as proof of 95% correctness. Use a confidence score as a routing input only when you understand its meaning and have evaluated it on relevant cases. Business rules may still require review of a high-consequence action regardless of that score.

Give the reviewer an actual decision

Consider an illustrative refund case, R17. An assistant has prepared a $120 refund to the original payment method. A designated refund approver must authorize the proposed action under this example company’s policy. No payment is made by the example.

The review should show enough to decide—not simply an AI-written paragraph saying “refund recommended.”

If the return receipt is missing, the decision is “request the receipt,” with a named owner for obtaining it. It is not “approve because the assistant sounded confident.” AI can prepare the case; the reviewer still needs access to the source and the authority to stop the action.

Suppose the reviewer approves revision 3. Before execution, someone changes the proposal to $200 in revision 4. The old approval must not authorize the new amount. Return it for approval of the changed proposal, even if the case ID is unchanged.

We checked this narrow rule with a local, deterministic demonstration: the matching revision-3 approval could proceed to the next step; the changed revision, missing evidence, wrong reviewer role, and request-for-information decision could not. No payment provider or AI model was involved. This demonstrates the example rule, not production security or the quality of human judgment.

The practical requirement is simple: the approval must identify what was approved. In a live system, execution also needs fresh authorization and transaction checks. An old approval does not resolve a refund that another process has already issued.

Specify what happens after each decision

Review is unfinished until the workflow knows where the case goes next.

For R17, approval permits the refund execution step; it does not mean the customer has been paid. That requires confirmation from the payment system. If execution fails or its outcome is unknown, the case needs reconciliation before anyone retries. The cash application workflow illustrates why a financial record and its supporting evidence must agree.

For other outcomes, name both the state and its owner:

• Rejected: stop this proposal, preserve the reason, and assign any customer response. Do not let the assistant resubmit the same proposal until it gets a yes.
• Missing information: pause the decision and assign a specific evidence request. Record who will return the case to review.
• No response: keep the action blocked and route it to the agreed backup or manual path when the review deadline expires. Silence is not approval.
• Changed proposal: invalidate the decision for the old version and show what changed before requesting another one.

Choose deadlines from the business consequence and coverage you can actually provide. An urgent case can need a different queue; it does not follow that an unattended case should automatically execute. Prevent two reviewers from independently completing the same live action, and record who currently owns it.

This principle applies beyond refunds. A planner reviewing a demand forecast needs to know whether they are approving an assumption, a forecast version, or a purchase order. The AI demand forecasting example shows why a forecast and a replenishment decision are different objects.

Improve the work, not just the approval rate

An approval rate near 100% can mean preparation is good. It can also mean reviewers lack time, evidence, or a workable reject option. The rate alone cannot distinguish those explanations.

Inspect some approved cases against their source evidence, not only rejected ones. Ask reviewers to explain the decisive fact on ambiguous examples. If qualified people disagree, determine whether the policy is unclear, the evidence is incomplete, or the task really requires judgment. Do not turn disagreement into an automatic assumption that one person is wrong.

Corrections also need an explicit destination. Fixing a refund amount may update that case only. Repeated missing receipts might justify a new intake rule. A corrected extraction could become an evaluation example after appropriate review. None of those actions necessarily retrains the model.

Google’s People + AI Guidebook emphasizes explaining what feedback changes and when; some feedback affects product behavior without changing the underlying model. Tell reviewers what their correction will do instead of promising that “the AI learns” from every click.

Before increasing authority, walk through a correct approval, a rejection, missing evidence, a revised proposal, an absent reviewer, and an uncertain execution result. Then run enough representative work to measure active handling time and inspect decision quality. Replace the capacity example’s assumptions with those observations.

If you cannot provide the evidence or the people required, keep that action in the existing manual process. Human review is useful when someone can meaningfully change the outcome—not when an approval button disguises an unattended queue.

KelenAI’s workflow automation services can help define the handoff between AI preparation, employee judgment, and system execution. Submit your workflow context with the action, expected volume, and current review bottleneck. We will review it and follow up; you do not need to book a call to submit.