G-CARL decomposes RLHF rewards into factuality, coverage and expression for medical AI agents

The problem G-CARL targets

Conventional RLHF collapses quality into one holistic preference signal. According to a write-up published on dev.to, that creates trouble for patient-oriented medical report interpretation (PMRI), where an agent explains radiology findings to patients under two constraints that pull in opposite directions. Evidence grounding requires every medical claim to trace back to the radiology report or a clinical knowledge base. Patient appropriateness requires wording, depth and emphasis to adapt to the individual query and the dialogue so far.

The article argues these objectives conflict under naive optimization: a model rewarded for helpfulness tends to invent plausible-sounding detail, while a model harshly penalized for any unsupported statement may refuse to explain much at all. G-CARL, short for Grounded Checklist-Aligned Reward Learning, responds by splitting the problem in two: whether each individual claim is true, and whether the response actually addresses what the patient asked. Both are then enforced separately.

Claim-level grounding

The grounding pipeline first parses the agent's response into discrete medical statements. Each claim is checked in parallel against three sources: the original radiology report for exact findings, a medical knowledge base for general clinical facts, and the prior dialogue for patient-specific history. Every claim receives a binary grounded or ungrounded label depending on whether supporting evidence exists in any of those sources.

The dev.to post emphasizes that this is not semantic similarity search. The test is whether a claim can be justified by the available evidence, not whether it sounds medically plausible.

Checklist-aligned coverage

The second component addresses appropriateness. For each patient query, the system derives a checklist of required topics, for example explaining a lung nodule, comparing against a prior scan, or outlining next steps. The response is then scored on how many checklist items it covers. Checklist weights adapt per instance, so if a patient asks specifically about a nodule, that item counts more than general report structure.

A composite reward with three parts

G-CARL's reward model combines three components. Factuality is measured as claim-level precision using the retrieval-based binary labels. Coverage is measured as checklist recall against the query-derived checklist. Expression is scored by a lightweight language model for readability and tone. The final reward is a weighted sum of the three, with weights tuned as hyperparameters on a validation set.

The key design point, per the article, is that factuality and coverage are verified independently, so the model cannot game one metric by sacrificing the other.

Training and inference

The training pipeline runs in three stages. A supervised warm-start fine-tunes a vision-language model on clinician-written interpretations. The reward model itself, the factuality verifier and the checklist generator, is trained on labeled examples. Finally, reinforcement learning fine-tunes the agent with proximal policy optimization using the composite reward, regularized by a KL-divergence penalty against the supervised policy. That penalty is described as a guard against degenerate strategies, such as refusing to answer in order to avoid ungrounded claims.

At inference time, the agent can optionally run self-verification: extract claims, check their grounding, and regenerate when the factuality score falls below a threshold. The post notes this adds latency and suggests batching claim verification and caching knowledge base lookups for production settings.

Evaluation and failure modes

The reported evaluation is three-dimensional: clinician-labeled claim precision for factuality, checklist recall for coverage, and readability and appropriateness scores for expression. According to the dev.to write-up, pairwise preference evaluation by clinicians found G-CARL outputs more accurate and better aligned with patient needs than baseline RLHF or supervised fine-tuning alone.

The method does not eliminate hallucination; it makes factuality checkable and penalizes ungrounded claims. The article lists failure modes, noting some are inferred from the architecture rather than documented in the paper. Retrieval gaps can leave the model unable to explain rare conditions not covered by the knowledge base. Checklist drift occurs if the checklist generator misreads the patient's intent, pointing optimization at the wrong target. Adversarial fluency means the model may learn hedged phrasing to dodge binary grounding penalties. Recommended observability includes logging claim-level grounding decisions, tracking checklist coverage per query, and monitoring KL divergence from the supervised policy to detect reward hacking.

Why it matters

The immediate application is narrow, but the pattern is not. Any agent domain where a source of truth exists and context-dependent communication is required could adopt the same split: legal agents grounding claims in case law and statutes, financial advice verified against regulatory constraints and market data, or safety-critical automation tracing control decisions back to sensor data and safety rules.

For RLHF practitioners, the broader lesson is decomposition. Replacing a single holistic preference signal with separately verifiable components makes reward hacking harder and factuality auditable, at the cost of real engineering overhead in retrieval, checklist generation and inference-time latency. It is a fit where hallucination has real-world consequences and structured evidence sources exist, and a poor fit where they do not.