Find out whether your judge measures quality or noise.

An LLM judge that scores the same answer differently depending on where it sits in the prompt, or swings twelve points between identical runs, is not a measurement. judge-audit takes scores you already collected and measures the four failure modes that make a judge untrustworthy: order bias, repeat instability, scale usage, and agreement with a reference.

Five tests over your own records. No API keys, no model calls, and nothing leaves this browser tab.

Exact sign test for order biasDeterministic scoringEvery test shows its evidence

Judgments

Load a sample

The same answers judged in two orders, with a 14 point lift for whoever goes first.

Paste your own

JSON records or CSV with a header. Each record needs a score, and optionally an item id, a condition, and a reference score. A 0 to 1 or 1 to 10 score is rescaled to 0 to 100.

[{"item_id":"item_1","condition":"first","score":72,"reference":58},{"item_id":"item_1","condition":"first","score":71,"reference":58},{"item_id":"item_1","condition":"second","score":57,"reference":58},{"item_id":"item_1","condition":"second","score":58,"reference":58},{"item_id":"item_2","condition":"first","score":87,"reference":72},{"item_id":"item_2","condition":"first","score":85,"reference":72},{"item_id":"item_2","condition":"second","score":72,"reference":72},{"item_id":"item_2","condition":"second","score":73,"reference":72},{"item_id":"item_3","condition":"first","score":64,"reference":51},{"item_id":"item_3","condition":"first","score":65,"reference":51},{"item_id":"item_3","condition":"second","score":50,"reference":51},{"item_id":"item_3","condition":"second","score":49,"reference":51},{"item_id":"item_4","condition":"first","score":85,"reference":72},{"item_id":"item_4","condition":"first","score":87,"reference":72},{"item_id":"item_4","condition":"second","score":74,"reference":72},{"item_id":"item_4","condition":"second","score":71,"reference":72},{"item_id":"item_5","condition":"first","score":73,"reference":60},{"item_id":"item_5","condition":"first","score":74,"reference":60},{"item_id":"item_5","condition":"second","score":60,"reference":60},{"item_id":"item_5","condition":"second","score":58,"reference":60},{"item_id":"item_6","condition":"first","score":80,"reference":65},{"item_id":"item_6","condition":"first","score":81,"reference":65},{"item_id":"item_6","condition":"second","score":66,"reference":65},{"item_id":"item_6","condition":"second","score":65,"reference":65},{"item_id":"item_7","condition":"first","score":64,"reference":51},{"item_id":"item_7","condition":"first","score":64,"reference":51},{"item_id":"item_7","condition":"second","score":50,"reference":51},{"item_id":"item_7","condition":"second","score":51,"reference":51},{"item_id":"item_8","condition":"first","score":71,"reference":58},{"item_id":"item_8","condition":"first","score":73,"reference":58},{"item_id":"item_8","condition":"second","score":58,"reference":58},{"item_id":"item_8","condition":"second","score":59,"reference":58},{"item_id":"item_9","condition":"first","score":70,"reference":54},{"item_id":"item_9","condition":"first","score":70,"reference":54},{"item_id":"item_9","condition":"second","score":56,"reference":54},{"item_id":"item_9","condition":"second","score":55,"reference":54},{"item_id":"item_10","condition":"first","score":63,"reference":49},{"item_id":"item_10","condition":"first","score":61,"reference":49},{"item_id":"item_10","condition":"second","score":47,"reference":49},{"item_id":"item_10","condition":"second","score":50,"reference":49},{"item_id":"item_11","condition":"first","score":72,"reference":57},{"item_id":"item_11","condition":"first","score":70,"reference":57},{"item_id":"item_11","condition":"second","score":56,"reference":57},{"item_id":"item_11","condition":"second","score":57,"reference":57},{"item_id":"item_12","condition":"first","score":82,"reference":70},{"item_id":"item_12","condition":"first","score":82,"reference":70},{"item_id":"item_12","condition":"second","score":69,"reference":70},{"item_id":"item_12","condition":"second","score":69,"reference":70},{"item_id":"item_13","condition":"first","score":78,"reference":63},{"item_id":"item_13","condition":"first","score":76,"reference":63},{"item_id":"item_13","condition":"second","score":65,"reference":63},{"item_id":"item_13","condition":"second","score":62,"reference":63},{"item_id":"item_14","condition":"first","score":88,"reference":72},{"item_id":"item_14","condition":"first","score":85,"reference":72},{"item_id":"item_14","condition":"second","score":71,"reference":72},{"item_id":"item_14","condition":"second","score":70,"reference":72},{"item_id":"item_15","condition":"first","score":63,"reference":48},{"item_id":"item_15","condition":"first","score":63,"reference":48},{"item_id":"item_15","condition":"second","score":48,"reference":48},{"item_id":"item_15","condition":"second","score":47,"reference":48},{"item_id":"item_16","condition":"first","score":67,"reference":54},{"item_id":"item_16","condition":"first","score":68,"reference":54},{"item_id":"item_16","condition":"second","score":56,"reference":54},{"item_id":"item_16","condition":"second","score":53,"reference":54},{"item_id":"item_17","condition":"first","score":69,"reference":56},{"item_id":"item_17","condition":"first","score":71,"reference":56},{"item_id":"item_17","condition":"second","score":56,"reference":56},{"item_id":"item_17","condition":"second","score":55,"reference":56},{"item_id":"item_18","condition":"first","score":77,"reference":62},{"item_id":"item_18","condition":"first","score":75,"reference":62},{"item_id":"item_18","condition":"second","score":61,"reference":62},{"item_id":"item_18","condition":"second","score":63,"reference":62},{"item_id":"item_19","condition":"first","score":68,"reference":55},{"item_id":"item_19","condition":"first","score":70,"reference":55},{"item_id":"item_19","condition":"second","score":56,"reference":55},{"item_id":"item_19","condition":"second","score":55,"reference":55},{"item_id":"item_20","condition":"first","score":87,"reference":73},{"item_id":"item_20","condition":"first","score":85,"reference":73},{"item_id":"item_20","condition":"second","score":74,"reference":73},{"item_id":"item_20","condition":"second","score":71,"reference":73},{"item_id":"item_21","condition":"first","score":78,"reference":65},{"item_id":"item_21","condition":"first","score":80,"reference":65},{"item_id":"item_21","condition":"second","score":66,"reference":65},{"item_id":"item_21","condition":"second","score":64,"reference":65},{"item_id":"item_22","condition":"first","score":76,"reference":60},{"item_id":"item_22","condition":"first","score":73,"reference":60},{"item_id":"item_22","condition":"second","score":61,"reference":60},{"item_id":"item_22","condition":"second","score":60,"reference":60},{"item_id":"item_23","condition":"first","score":74,"reference":59},{"item_id":"item_23","condition":"first","score":75,"reference":59},{"item_id":"item_23","condition":"second","score":57,"reference":59},{"item_id":"item_23","condition":"second","score":60,"reference":59},{"item_id":"item_24","condition":"first","score":72,"reference":57},{"item_id":"item_24","condition":"first","score":72,"reference":57},{"item_id":"item_24","condition":"second","score":58,"reference":57},{"item_id":"item_24","condition":"second","score":58,"reference":57}]

Everything is measured in this browser tab. Your data is never uploaded.

No audit yet

Paste judge scores on the left, or load a sample, to measure order bias, repeat instability, scale usage and agreement with a reference.

How it works

Records are read for score, item, condition and an optional reference.
Conditions are compared in pairs to measure order bias, with an exact sign test.
Repeated items are measured for spread, and score values for scale usage.
Reliability weights the average against the weakest test.

What it does not do

It measures agreement, not correctness against the truth.
It cannot judge items you did not repeat or pair.
The pass line is fixed at 70 and is stated, not hidden.
A high score is not a guarantee of a good rubric.