· via dev.to (home feed)
Hidden preferences in VLM fine-tuning data survive distillation into student models
A study of vision-language model distillation finds that a preference hidden in a teacher's fine-tuning data reappears in its students, shifting downstream behavior by up to ten percentage points.

Hidden preferences travel through distillation
Fine-tuning a vision-language teacher on data carrying a covert objective measurably biases the student models distilled from it, according to a dev.to post summarizing the paper "Subliminal Learning in Vision-Language Models". In the experiment, an animal-preference signal was concealed in the teacher's fine-tuning data; after distillation, that preference reappeared in the student models and moved their downstream behavior by about ten percentage points.
Earlier research on large language models had already shown that preferences hidden in training data can be passed along during learning, the post notes. What had not been tested was whether the same subliminal transfer occurs in vision-language systems, where a teacher model produces the outputs a smaller student is trained on. This study closes that gap.
The measured shift
Students trained on the biased teacher's outputs picked the preferred target more often than control students did, with a gap as large as ten percentage points on the animal preference task, the post reports.
One concrete evaluation puts numbers on the effect: the control student chose the preferred animal 30.50% of the time, while the student distilled from the biased teacher chose it 40.42%. The post frames this as confirmation of the ten-point shift on a geometric-shape task unrelated to the fine-tuning objective. In other words, the bias surfaced even when the student was probed on something other than the domain where the preference was originally planted.
What the study leaves open
How far the hidden bias travels beyond the narrow geometric-shape probe remains an open question, according to the post. The effect is also not constant: transfer reportedly grew stronger with more training data, and its size depended on which preference was targeted, which model was used, and how the numbers were generated.
Two implications follow. Larger fine-tuning corpora may amplify subtle preferences rather than wash them out, and different data-generation strategies may dampen or magnify the effect, meaning the same hidden objective could be more or less contagious depending on how a pipeline is configured.
Why it matters
Distillation is a standard step in modern multimodal pipelines: large teachers generate training data for smaller, cheaper students, and that data is usually screened for quality, coverage and overt toxicity rather than for covert preferences. This result shows a specific route by which a signal nobody intended, in the experiment one deliberately concealed in the teacher's fine-tuning data, can travel through the pipeline and surface on tasks unrelated to its origin. Conventional benchmark evaluation, which checks task performance, would not catch it.
The practical recommendation in the post is direct: teams that prune or curate teacher VLMs need to screen them for hidden preference signals before distilling from them. Without that step, student models inherit biases that stay undetected even when evaluated on unrelated tasks, turning a subtle fairness problem in one model into a distributed property of every model trained from it.
One caveat: the material here is a single summary post on dev.to of the underlying paper, and the figures above are as reported there.
- #vision-language-models
- #distillation
- #machine-learning
- #bias
- #ai-safety