AI

Machine Bias and LLMs: A History Lesson

Troy McKinster

— 4 min read
Share
Machine Bias and LLMs: A History Lesson

In the late 90s, Tim Brennan and Dave Wells founded Northepointe and developed a tool to identify the recidivism risk (that being the chance of a person re-offending following their conviction, how this is measured varies and can include as little as re-arrest without conviction) of defendants and called it the Correctional Offender Management Profiling for Alternative Sanctions, or COMPAS. It's not entirely obvious what the initial purpose of COMPAS was; ostensibly it was to provide a better measure of recidivism risk than the commonly-used Level of Service Inventory (LSI) and help judges when determining eligibility for alternatives to hard time.

Ok, good so far. They wanted a better tool to guess at whether someone was actually rehabilitated (whatever that means) or if they were likely to be incarcerated again. It's a data analysis problem well-suited to an AI solution. Let's set aside discussion about whether or not the US incarceration system is actually designed to "rehabilitate" anyone. Let's also set aside the Minority Report of it all; trying to predict criminal behavior isn't a totally insane approach until people are preemptively punished for future crimes primarily on the word of a predictive system.

Fast forward to 2016, COMPAS has achieved widespread adoption in local judicial systems throughout the US. A small handful of tech journalists with ProPublica publish Machine Bias, which drills down in one Florida County to check if COMPAS does what it's supposed to: produce fairer, more accurate outcomes for people being sentenced to time behind bars. It isn't. COMPAS is producing outcomes that are racially biased. This is clear from the data, but the data hadn't actually been examined at scale until ProPublica got to it. There isn't even a great reason why beyond the fact that the data was messy (ProPublica's team did a ton of work to even be able to analyze it). For anyone curious about the larger context surrounding COMPAS and ProPublica's article on it: Brian Christians' The Alignment Problem breaks down the problems with the data set and the problem with the designers' approach to non-bias.

Anyone familiar with the history of criminal justice in America won't be surprised by the outcome, but why it happened is still interesting:

White Defendants’ Risk Scores

The graph of score distributions for white defendants looked something like this, with risk scores clustered toward lower values

Black Defendants’ Risk Scores

And the graph for black defendants looked more like this, with risk scores more evenly distributed. (Source: ProPublica analysis of data from Broward County, Fla.)

But these graphs should look the same if COMPAS is treating black and white defendants equitably. What went wrong? COMPAS was shown to have similar predictive accuracy for blacks and whites (just below 70%) and its survey data didn't even include defendant's race. What COMPAS did include was a handful of data points which correlated very strongly with race (things like address, level of income, home ownership, etc). This ended up having adverse effects on the model as a whole.

This should surprise no one who knows a little about policing rates.

COMPAS was frequently recommending longer sentences to black defendants and those recommendations were being followed, leading to real time served by black defendants who often had much less criminal history than the white defendants who COMPAS scored similarly.

What are the takeaways here?

  • COMPAS was probably developed with good intentions
  • Northepointe's approach when developing COMPAS was naive with regards to statistics and the tool's potential for use and abuse
  • COMPAS was in use for years across many states before it was independently verified

In common 2026 parlance, COMPAS is only barely an AI tool. It isn't backed by an LLM, it wasn't built to solve all the problems customers might throw at it, and the data set used to build it wasn't particularly large (at least not compared to something like ChatGPT). But how COMPAS ended up being used is remarkably similar to current LLMs in a handful of meaningful ways:

  • customer misuse - COMPAS was meant to be one of several factors used in sentencing, it ended up carrying much greater weight as users leaned on it more and more
  • little or no independent verification of outcomes despite widespread usage - usage spread rapidly and ended up serving as surrogate for credibility when very little research or analysis had been done on outcomes

It isn't hard to connect the dots on customer misuse to contemporary AI systems. As customers continue to consult with LLMs for everything, their dependency only grows. When the users are already unstable or isolated, the consequences can be tragic. ChatGPT wasn't designed to assist in suicide, but customers accustomed to asking for help with everything have continued to find ways to make it tell them what they want to hear.

We need to be careful about when and how we apply AI tools to complex problems. Being "data-driven", AI tends to bring some amount of credibility with it by default. It can be extremely challenging to determine if and how a data set might be biased. These factors combined can result in tools that look as if they perform well, but are actually producing undesired outcomes at the margins. A virtual friend might be a charming novelty to someone with a healthy social circle, but to someone with strange ideas and few friends it might be a lifeline that also validates dangerous or isolating behavior. A program built to identify security vulnerabilities is as valuable to a bank thief as it is to the bank.

UPDATE: For those curious what became of COMPAS, its owner Northpointe was merged with other companies in the justice space under the name "equivant". The software is still widely used and 2024 research indicates its use results in an overall reduction in incarceration while exacerbating racial differences in outcomes.

Read more