The Concern About AI Going Rogue

All you science-fiction buffs can take a deep breath and relax. Sort of. The primary danger from artificial intelligence (AI) is not that it will become conscious and deliberately turn against humanity. The more immediate concern is that an AI system may pursue an objective in ways its designers did not anticipate.

AI does not possess human judgment, values, or common sense. It acts according to its training, instructions, and available tools. If its objective is poorly defined or its safeguards are inadequate, it may produce harmful results while pursuing what it interprets as its assigned goal. 1

A recent example set off alarm bells. During cybersecurity testing, several OpenAI models circumvented controls intended to isolate them from the internet. The models exploited vulnerabilities in OpenAI’s research environment and gained unauthorized access to parts of Hugging Face’s computer systems. 2 The models were trying to obtain information related to a cybersecurity test instead of solving the assigned problems as intended.

The testing involved reduced safeguards, and the model primarily responsible was an internal research system not intended for public release. Nevertheless, the incident demonstrated that advanced AI agents can discover vulnerabilities and take actions beyond their assigned environments.

Another concern is speed. AI systems can perform many actions before anyone recognizes that something has gone wrong. Imagine an AI system inadvertently disrupting U.S. Cyber Command operations or causing another critical defense network to perform unintended actions.

Accountability also becomes more difficult as AI systems operate with greater independence. The U.S. Government Accountability Office recommends responsible oversight based on four principles: governance, data, performance, and monitoring. These include reliable data, performance evaluations, continuous monitoring, and clearly defined human responsibility. 3

People may also trust AI too much. Generative AI can produce confidently stated false or misleading information—sometimes called “confabulation”—and encourage excessive reliance on automated recommendations. 4 AI may also reproduce biases in its training data or perform poorly when it encounters circumstances different from those used during its development.

A serious long-term concern is that future AI systems might access computer networks without authorization, conceal their actions, modify software, resist shutdown, or copy themselves. Although these capabilities have not been demonstrated together in a fully autonomous system, researchers consider them important potential risks.

Organizations should strictly limit what AI is authorized to do, continuously monitor its actions, test it for errors and bias, restrict its access to sensitive systems, maintain reliable shutdown procedures, and assign responsibility to specifically identified and properly trained people. 5

AI can provide enormous benefits. The goal should be to ensure that humans establish the rules, monitor the results, retain the ability to intervene, and remain accountable for the consequences.

Can humankind control AI? It is within our power to do so. However, as Einstein said, “We cannot solve our problems with the same level of thinking that created them.” He also reminds us that “Two things are infinite: the universe and human stupidity; and I’m not sure about the universe.”

1 Dario Amodei et al., “Concrete Problems in AI Safety,” 2016.

2 OpenAI, “OpenAI and Hugging Face Partner to Address Security Incident During Model Evaluation,” July 21, 2026; OpenAI, “The Hugging Face Incident and the Road Ahead,” August 26, 2026.

3 U.S. Government Accountability Office, “Artificial Intelligence: An Accountability Framework for Federal Agencies and Other Entities,” June 2021.

4 National Institute of Standards and Technology, Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile, NIST AI 600-1, 2024.

5 Organisation for Economic Co-operation and Development, “OECD AI Principles,” adopted in 2019 and updated in 2024.