AI Mistakes Australian Defence Personnel Should Avoid

The defence environment has unique security requirements that make AI tool adoption more complex than in other sectors. Mistakes that would be inconvenient in a commercial context can have serious national security consequences in a defence context. These are the most common AI mistakes Australian Defence personnel make — and how to avoid them.

---

Mistake 1: Entering Classified Information into Unapproved AI Tools

This is the most serious mistake Defence personnel can make with AI tools. Entering classified information into an AI tool that has not been approved for classified use is a security breach with potentially serious consequences — for the individual, for the organisation, and for national security.

AI tools process data on remote servers. Even if a tool appears to be secure, entering classified information into it without appropriate approval is a breach of Defence security policy.

How to avoid it: Never enter classified information into any AI tool that has not been specifically approved for classified use. When in doubt, do not enter the information. Check with your security officer if you are unsure about the classification of information or the approval status of a tool.

---

Mistake 2: Using Unapproved AI Tools on Defence Networks

Defence networks are subject to strict security controls. Using unapproved software — including AI tools — on Defence networks creates security risks and may violate Defence ICT policy.

How to avoid it: Only use AI tools that have been approved for use on Defence networks. Check the current approved tool list with your IT team. Do not install or use unapproved software on Defence systems.

---

Mistake 3: Ignoring Data Sovereignty Requirements

AI tools that process data on servers outside Australia may not be appropriate for Defence use, even for unclassified information. Australia has specific data sovereignty requirements for government data, and Defence has additional requirements.

How to avoid it: Check the data residency of any AI tool before using it for Defence work. Prefer tools that process data in Australia or in jurisdictions with equivalent protections. Check Defence ICT policy for current requirements.

---

Mistake 4: Over-Relying on AI in Operational Contexts

AI tools can provide valuable analysis and recommendations, but they can also be wrong — and in operational contexts, errors can have serious consequences. Defence personnel who treat AI recommendations as definitive without applying their own judgement are taking a risk.

How to avoid it: Train personnel to understand the limitations of AI tools and to apply their own judgement. AI recommendations should be one input into decision-making, not the only input. Experienced personnel should be empowered to override AI recommendations when their judgement differs.

---

Mistake 5: Failing to Maintain Manual Backup Processes

AI-powered systems can fail — due to technical problems, cyberattacks, or degraded communications. Defence personnel who have become entirely dependent on AI-powered systems without maintaining manual backup processes are vulnerable.

How to avoid it: Maintain manual backup processes for all critical functions. Regularly practise these backup processes to ensure personnel can use them effectively under pressure.

---

Mistake 6: Neglecting Cybersecurity for AI Systems

AI systems are software systems, and like all software systems, they are vulnerable to cyberattacks. Adversaries may attempt to compromise AI systems used by the ADF — either to extract information or to manipulate the AI's outputs.

The ASD has noted that AI systems can be targeted by adversarial attacks — attempts to manipulate AI outputs by feeding the system carefully crafted inputs.

How to avoid it: Apply the same cybersecurity standards to AI systems as to other critical IT systems. Follow ASD guidance on cybersecurity for AI. Be aware of the possibility of adversarial attacks on AI systems.

---

Mistake 7: Using AI for Decisions That Require Human Accountability

Some decisions in the defence context — particularly those involving the use of force — require human accountability. Using AI to make or automate these decisions without appropriate human oversight is inconsistent with Australia's commitments on responsible military AI use.

How to avoid it: Be clear about which decisions require human oversight and accountability. AI can assist with analysis and planning, but decisions involving the use of force must be made by accountable human commanders.

---

Mistake 8: Sharing AI-Generated Content Without Review

AI tools can produce plausible-sounding but inaccurate content. Defence personnel who share AI-generated content — in briefings, reports, or communications — without reviewing it for accuracy are creating risks.

In a defence context, inaccurate information in a briefing or report can lead to poor decisions with serious consequences.

How to avoid it: Treat AI output as a first draft, not a finished product. Every piece of AI-generated content used in official documents or communications must be reviewed by a qualified person before use.