The Cautionary Tale: AI Use with the Lack of Human Oversight
When an organisation gets caught out for submitting flawed AI-generated work, the same narrative follows: “AI made a mistake.” It’s a convenient excuse, but a misleading one. Artificial intelligence does not make mistakes. It generates outputs based on prompts and data. It has no context, no judgment, and no understanding of consequences. The real problem lies with people who use it without providing the correct context and worse to, proper review. The recent Deloitte incident is a perfect example. A global consultancy, trusted by governments and major corporations, found itself refunding part of a 440,000 Australian dollar contract after errors were found in a report prepared for the Australian government. The issue was not that AI was used, but that human accountability was missing. Until organisations accept that technology is only as reliable as the people managing it, similar incidents will keep happening. The Deloitte Example Deloitte has agreed to repay part of a 440,000 Australian dollar fee after errors were discovered in an Australian government report that it helped produce with generative AI. The Department of Employment and Workplace Relations commissioned the firm to review the Targeted Compliance Framework and related IT systems. After publication, academics and journalists identified fabricated or incorrect references, along with a misdescribed court decision. Deloitte amended the document and disclosed the use of Azure OpenAI GPT-4o tools, while maintaining that the report’s core findings were unchanged. The department confirmed that the recommendations remain intact, yet the credibility damage was already done. A senator criticised the firm for allowing AI to do the heavy lifting with inadequate human checks. The result was a partial refund and a public lesson in accountability. Other outlets reported similar details. Coverage notes that the report included invented or misattributed citations and a fabricated legal quote later removed in revisions. The department accepted a partial refund and said the substance of the review stands, but questions about quality assurance and disclosure persist. What “Hallucination” really means The Deloitte report is a textbook example of AI hallucination in a high-stakes setting. Hallucination is the term for when AI models produce information that is fabricated or incorrect, often with total confidence. Generative models predict likely words based on patterns in data. They do not possess context, institutional memory, or legal expertise. When prompted for citations, they can produce plausible but false references. This is known as hallucination. In the Deloitte case, watchdogs found references that did not exist and a legal citation that did not match the real judgment. After corrections, observers noted that replacing one weak reference with a cluster of alternatives did not fix the underlying problem. The initial claims were not grounded in verified sources. This is not unique to consulting. Courts have sanctioned lawyers for filings that relied on AI text containing invented cases. In a recent Utah matter, the appeals court punished counsel after a brief cited a precedent that could not be found in any database. Local reporting and follow-up coverage make the same point. AI can assist research, but professionals must verify. A Failure of Human Accountability, Not of AI The fault does not lie with AI. It lies with the people who choose to ship AI-assisted work without the checks that any professional deliverable requires. Tools generate drafts. People are responsible for the truth. If a report contains a fabricated reference, that is a process failure. If a legal brief cites a non-existent case, that is a breach of professional duty. When organisations treat AI outputs as finished work instead of raw material, they convert productivity aid into a reputational risk. AI can accelerate research and drafting, but it cannot (and should not) replace human expertise in validating content. When Deloitte’s team delivered a report riddled with fake references and errors, that was a breakdown in their quality control and professional duty. No AI policy or guideline can absolve professionals of accountability for what they present to clients. Just as the court in Utah made clear that lawyers must verify their filings despite using AI, consultants and analysts must ensure their AI-augmented reports are accurate and credible. Blaming the AI alone is irresponsible – the onus is on the humans using the tool to use it correctly. Why This Will Keep Happening The market rewards speed. AI accelerates drafting, so teams move faster. Without AI training and clear process review steps, errors slip through, and clients pay for them. Public scrutiny then focuses on the tool rather than the system that enables misuse. Unless leaders establish clear expectations for verification, disclosure, and ownership of outcomes, similar incidents will continue to surface across various domains, including consulting, legal, financial, policy, and technical. The pattern is already visible. When errors like these come to light, they undermine trust, not just in AI tools, but in the organisations deploying them. As Senator Deborah O’Neill (Senator for New South Wales, representing the Australian Labor Party) joked, why shouldn’t a client “sign up for a ChatGPT subscription” instead of paying a hefty fee to a firm that handed in AI-written material? This sharp critique highlights a serious reputational risk: if consulting firms (or any experts) misuse AI, they may be seen as overcharging for establish clear expectations for verification, disclosure, and ownership of outcomes, similar incidents will continue to surface across various domains, including consulting, legal, financial, policy, and technical work product that is no more reliable than something a layperson could prompt from a chatbot. Moreover, incidents like this can cause backlash against the broader use of AI in business. Government clients, for example, might become wary of allowing consultants to use AI at all. That would be a shame, because when used responsibly, AI can be a powerful asset – saving time on data analysis, generating useful first drafts, and uncovering insights. The key differentiator is responsible use. The fallout from the Deloitte report is a reminder that we will keep seeing AI-related blunders until organisations institute proper oversight and take accountability. In an era of