Evaluation Awareness Beyond English: Do LLMs Act Differently When Tested in Indic Languages?
Large language models (LLMs) can often tell when they are being tested. This ability, known as evaluation awareness, can lead models to act more cautiously during safety evaluations than in real use. Existing work has studied evaluation awareness almost entirely in English, although models serve users in many languages. Indic languages are spoken by over a billion people, yet e
Reading in Plain English mode — toggle below to switch
Large language models (LLMs) can often tell when they are being tested. This ability, known as evaluation awareness, can lead models to act more cautiously during safety evaluations than in real use. Existing work has studied evaluation awareness almost entirely in English, although models serve users in many languages. Indic languages are spoken by over a billion people, yet e
This research highlights a critical challenge in evaluating and deploying LLMs, especially in diverse linguistic contexts. Understanding evaluation awareness is crucial for ML engineers to build robust and fair AI systems, ensuring models behave consistently and safely across different user interactions and languages, not just during controlled tests.