A Taco Bell customer crashed its AI-backed ordering system by requesting 18,000 waters.

I get that AI is a novel technology, and that companies are still finding their way. But the problems mentioned in the article are not unique to AI! "Check and filter the inputs" is a 101-level lesson on running unattended, automated systems.

In fact, this strikes me as something a company would catch in advance, as part of a risk assessment / QA / red-teaming exercise.

The fact that it was uncovered in production makes me wonder: did Taco Bell not perform those exercises? Or did it run them, but ignore inconvenient results?

I have no inside information here. But given the widespread push for companies to adopt genAI, it's possible that Taco Bell skipped some steps for the sake of speed.

The takeaway lesson for your company: introducing AI brings new opportunities but also new dangers. You can take proactive steps to reduce your downside risk exposure.

For more details, I recommend my 2023 article "Risk Management for AI Chatbots" – it's available on my website as well as on O'Reilly Radar.