Safety alignment may invert LLM agent behaviour in civil violence simulations

What the study did

A study published on dev.to asks a deceptively simple question: when an LLM replaces a mathematical decision rule inside an agent-based model, does it actually reproduce the rule's behaviour? The testbed is Epstein's 2002 civil violence model, a classic simulation in which citizen agents decide whether to become actively rebellious. In the original, each agent weighs grievance — perceived hardship discounted by how legitimate it finds the government — against net risk, a product of personal risk aversion and the estimated chance of arrest. When grievance exceeds net risk by a threshold, the agent activates.

That rule is deterministic and, crucially, predicts more activation as hardship rises and legitimacy falls. The study's author argues this monotonicity is the minimum bar for an LLM stand-in: a language-model agent fed a natural-language rendering of the same state should at least not move in the wrong direction.

The experiments ran on a locally served Qwen 27B model (4-bit quantised, via MLX), at temperature 0.7, across more than 600 calibration conditions with 20 repetitions per cell, spanning tension levels from 0.10 to 0.90.

An inverted response curve

With the agent asked to choose between "act" and "wait", the result ran backwards. Activation probability fell from 41.5% at tension 0.10 to 38.1% at 0.50, 29.3% at 0.70 and 12.6% at 0.90. At the point where the reference model expects near-universal activation, the LLM agent activated in roughly one sample in eight.

Changing nothing but the wording of the two options — to "protest" versus "comply" — restored the expected shape: 0% at 0.10, 20% at 0.50, 35% at 0.70 and 95% at 0.90, with a steep rise in the high-tension range that echoes the threshold behaviour of the original rule.

The author is careful about statistical weight. At 20 samples per cell, the 95% Wilson interval around a 20% estimate spans roughly 8% to 42%, so intermediate points should be read as directional. The working label pair was also found by search, making it an in-sample result that still needs validation on held-out prompt templates.

A hypothesised mechanism, not a proven one

The observed distortion is broken into three parts: a lexical prior that strongly favours "wait" (around 45 percentage points against action), position bias of 5–16 points, and a tension-dependent anti-action effect worth 15–30 points at high tension.

The hypothesis is that preference tuning — RLHF or DPO-style safety alignment — installs a negative prior on endorsing unspecified "action" in high-conflict contexts, with the penalty growing as the scenario signals more conflict. On this reading, "protest" escapes the penalty because training data treats it as a protected civic act, while "wait" attracts a positive prior as a safe, de-escalatory completion.

The study explicitly flags this as consistent with the data but not isolated. The proposed direct test: run identical prompts through base and aligned checkpoints of the same model family and check whether the tension-dependent component appears only in the aligned version.

Other models fail differently

Mistral 7B Instruct v0.3, whose documentation describes no dedicated safety-tuning stage, showed no anti-action pattern. Instead it exhibited near-total primacy bias, picking whichever option came first in about 100% of samples — output that carries no information about the scenario at all. Base models showed weaker safety-style priors but stronger social biases and unreliable instruction following. None of the three, the author concludes, is an unbiased decision function without calibration.

What the author recommends

Five practices come out of the work: calibrate before simulating, sweeping decision inputs and checking monotonicity and threshold placement against the reference rule, and refusing to run the population simulation if the check fails; report the label pair as a methodological design parameter, alongside the alternatives that were tested; counterbalance option order across agents and repetitions; publish a "bias budget" quantifying each identified bias component; and prefer log-probability scoring of labels over sampled text, calibrating against a content-free baseline prompt.

Why it matters

Each identified bias is comparable to or larger than the behavioural differences such simulations exist to resolve — taken together, the study notes, they are enough to turn a predicted mass mobilisation into near-total quiescence. Existing LLM-ABM frameworks such as SocioVerse-ABM and AgentTorch compare agent output against baselines, but neither, as far as the author knows, separates decision-function distortion from emergent dynamics before a run. Without that step, a deviation from baseline cannot be attributed to the model's reasoning rather than to its label and position priors.

The findings also echo Li et al. (2025, ACM FAccT), where alignment reduced explicit bias while implicit bias persisted or grew. For anyone wiring LLM agents into simulations, the practical lesson is that wording is an experimental variable, and uncalibrated agents can silently invert the phenomenon being studied. The study's own limits — one primary model, one quantisation level, one country prior and a binary decision — mean the specific numbers should be treated as a case study rather than a general law.