inter annotator agreement

Most teams run human eval as a vibe check with ten examples and a 1-5 slider. The signal is treating it as a designed experiment: pairwise, blinded, powered, and agreement-measured. Here is the protocol that survives scrutiny.

Models can only be as good as their labels, and noisy annotation quietly caps performance. What interviewers want is a measured quality process, not a bigger collection effort. Here is the answer.