In the past I have said that AI models are akin to "interns" and "factory machines": eager, tireless, but ultimately unaware when they are operating out of their depth.
The Washington Post's Geoffrey Fowler has experienced this first-hand:
"I let ChatGPT’s new ‘agent’ manage my life. It spent $31 on a dozen eggs."
Knowledge and wisdom
It helps to know the difference
Unreliable chatbots
The BBC ran the numbers on chatbot goofs