Shitty Ask Lemmy

Because "garbage in, garbage out". Statistical models will by design reproduce any bias in the training data.

For instance, in 2018, Amazon scrapped a hiring AI tool because it discriminated against women. It did so because it was trained on previous Amazon hires which, it turns out, had a sexist bias. It's a testament to the power of these statistical methods that the tool managed to discriminate against women in spite of gender having been stripped from the data: the model managed to reconstitute the information based on other elements to a statistically significant degree.

Applying this to crimes, can you think of any biases in police action or the justice system that could be reproduced by the AI? Communities that get unfairly targeted, or others that benefit from blind spots?

Furthermore, there are some other issues.

One is confirmation bias: if the AI predicts there will be crime in an area and a patrol car is sent, it will confirm the prediction.. In that police tends to find crime wherever it goes. If this data is then again used as input, over time this will bias the AI more and more.

Another issue is dilution of responsibility. If a human being gives an order that turns out to be illegal, they are legally responsible and can be prosecuted. If however that same person tells their subordinates to obey the AI, and that AI gives illegal orders, who is responsible? In truth, the same people are still in charge – they can feed and train the AI however they like, and ignore its output if it isn't as they wish. But it is a legally and mediatically gray area that would be very useful for enforcing illegal orders without risking prosecution.