Alignment teaches vision-language models (VLMs) to avoid expressing demographic biases, and when gender is clearly visible they largely succeed. Far less is known about ambiguous inputs (a worker in full gear, a figure seen from behind) cases common in practice yet rarely studied. We find that minimal prompting pressure exposes occupation-gender defaults when prompting ambiguous input images, with models collapsing to male even for strongly female-stereotyped occupations. But do these outputs reflect what models actually encode internally? We introduce LALS (Latent Association Leaning Score), a zero-shot metric that projects visual-token activations into the model's text-embedding space to measure concept associations per token and layer. Across 15 occupations, over 800 gender-ambiguous images, and four VLMs, internal representations and outputs are systematically decoupled: models often encode a female association internally yet output male. Layer-wise analysis reveals an asymmetric filter -- male signal amplifies end-to-end while female signal peaks mid-network and is suppressed before generation -- and a color ablation shows that culturally loaded visual cues such as clothing color further modulate these internal associations.

翻译：对齐训练使视觉-语言模型（VLM）避免表达人口统计学偏见，当性别清晰可见时，模型基本能做到这一点。但在实际常见且鲜有研究的模糊输入场景中（如全副武装的工人、从背后观察的人影），情况远未被充分理解。我们发现，当用模糊输入图像提示模型时，最小程度的提示压力即可引发职业-性别默认关联，即使对强烈女性刻板印象的职业，模型也会坍缩为男性输出。但这些输出是否反映了模型内部真正编码的内容？我们提出LALS（潜在关联倾向评分）——一种零样本度量方法，通过将视觉令牌激活映射到模型文本嵌入空间，逐令牌、逐层测量概念关联。针对15种职业、800余张性别模糊图像及四种VLM，发现内部表征与输出存在系统性解耦：模型内部常编码女性关联，输出却为男性。逐层分析揭示了一种非对称滤波机制——男性信号在全流程中被放大，而女性信号在中间网络层达到峰值后，在生成前被抑制；颜色消融实验表明，服装颜色等承载文化负荷的视觉线索会进一步调节这些内部关联。