A woman was arrested and jailed because she was mis-identified by facial recognition tools.

Between the mistaken ID and the arrest, no one thought to ask the simple question: "had she even been in the state when the alleged crime took place?" (As her attorney determined, she had not.)

I've said this before about genAI and I'll say it again:

AI's greatest risk isn't an incorrect result; it's AI being put in a situation where it's unfit for the task.

Mix that with insufficient oversight, and every incorrect result leads to disastrous consequences.