Judge Bias
A systematic flaw where an LLM acting as an evaluator disproportionately favors certain outputs based on style, length, or position rather than quality.
Think of It Like This
Like a tired teacher giving higher grades to essays that are longer or simply placed at the top of the grading stack.
Common forms include verbosity bias (preferring longer answers), position bias (preferring the first option), and self-enhancement bias (preferring its own outputs). Mitigating this requires careful prompt engineering, swapping the order of options, and calibrating the judge against human baselines. Unchecked judge bias severely compromises automated evaluations.