Our method
The questions come from published standards and research on how courts should be judged by the people who use them. Here is what we used, why, and what the result cannot tell you.
The rules we followed
- Ask about what people can see (the process), not about legal correctness or the verdict.
- One idea per question.
- Plain language, at about an 8th-grade level.
- Give an honest way out: “I didn’t see this”. Require at least three answers.
- Ask whether the person won or lost, and show it, because losing colors how people rate.
- Show groups of raters separately instead of blending them.
The five judge questions
Measures “voice”. Based on the court-user survey of the National Center for State Courts (“I had a chance to tell my side of the story”), the Arizona juror survey, and research by Lind, Tyler and Huo on what makes people feel a court treated them fairly.
Measures neutrality, the best everyday stand-in for impartiality. Based on NCSC (“I was treated the same as everyone else”) and the Arizona survey.
Measures respect and temperament. Based on NCSC (“courtesy and respect”), the Arizona survey, and the temperament criterion of the ABA guidelines.
Measures understanding. Based on NCSC (“I could follow what was happening”, “I know what to do next”) and the communication criterion of the ABA guidelines.
Measures time and organization. Based on NCSC (“a reasonable amount of time”), the Arizona survey, and the case-management category used by Colorado’s own evaluations. In Rhode Island’s 2023 survey, time was one of the lowest-scoring areas.
The five prosecutor questions
No official questionnaire for prosecutors exists. These are a careful first draft built from the ABA prosecution standards, the same procedural-justice research, and the Stanford “Rate My District Attorney” report.
Respect. Same research as above.
The duty to prepare. Visible in court.
Candor and the standard against inflammatory argument. It asks about conduct a person can see, not whether the facts were true.
Understanding, one of the elements of procedural justice.
Fair conduct toward people in the room. We ask about behavior, not about motives.
Based on research on victims’ and witnesses’ experience, where being informed and listened to matters most. Shown separately and never merged into the main score.
What we deliberately do not ask
- Whether a verdict, sentence or plea was fair, or whether a judge got the law right. People in the room cannot judge that, and answers would mostly reflect who won.
- Whether someone is “good”, “honest” or “corrupt”. We ask about things people did that others can see.
- About charging decisions, plea deals or evidence disclosure. These happen where the public cannot see them.
- About looks, politics or personal life.
Where the questions come from
- ABA Black Letter Guidelines for the Evaluation of Judicial Performance (summary by IAALS)
- Rhode Island Judiciary, 2023 Access and Fairness Survey (uses the NCSC CourTools Measure 1 items)
- Arizona Supreme Court Administrative Order 2010-22 (juror and litigant evaluation questions)
- ABA Criminal Justice Standards for the Prosecution Function
- Prosecutorial Performance Indicators project
- Stanford Law School, “Rate My District Attorney” (2018)
Also: Lind, Tyler and Huo (1997) and Tyler and Huo, “Trust in the Law”, on how people judge courts by fair treatment more than by outcome.
The limits, honestly
- These questions are drafted from standards and research. They have not yet been tested with real users, so we do not claim they are a validated scientific instrument.
- People who choose to rate are not a random sample of everyone who goes to court.
- A score summarizes opinions about conduct in a courtroom. It is not a finding about a judge’s legal skill.
- We could confirm some of our sources only through summaries. We will replace them with the primary documents as we read them.
How we will check and improve them (planned)
- A pilot with real courtroom visitors in Colorado.
- Checking that people understand each question the same way and that the answers hang together.
- Review by independent judicial-performance researchers, survey methodologists and practicing Colorado attorneys.
- Review of the Spanish by native speakers.
- Publishing every change to a question and the reason.