The Biggest Risk of Artificial Intelligence May Still Be Human

Artificial intelligence is becoming more powerful every year. It can analyse enormous amounts of information, assist scientific research, generate computer code, and help people make decisions in seconds. But as these systems become more capable, an important question deserves our attention: What happens when humans begin trusting machines more than their own judgement?

Much of the debate surrounding AI focuses on the possibility that machines could eventually become too intelligent or difficult to control. These concerns deserve serious consideration. Yet history offers another perspective. Some of the most dangerous technological failures have not occurred because machines became intelligent enough to overpower humanity, but because people misunderstood their limitations, ignored warning signs, or placed too much confidence in automated systems.

KanxeY Tales

From a false nuclear alarm during the Cold War to weaknesses in modern water infrastructure, real-world incidents demonstrate how technology can become dangerous when human decisions and technical safeguards fail.

The challenge, therefore, is not simply to build smarter machines. It is to ensure that the people who design, operate, and depend on them remain capable of questioning their results.

When a Machine Mistook a False Alarm for a Nuclear Attack

In September 1983, Soviet military officer Stanislav Petrov faced a decision that could have changed the course of history.

While working at a Soviet early-warning facility, Petrov received an alert indicating that the United States had launched a nuclear missile towards the Soviet Union. Moments later, the system reported additional incoming missiles.

The warning appeared serious. Petrov had only minutes to assess the situation, and his responsibilities required him to report a potential attack to senior military officials.

However, something about the warning did not make sense to him.

He questioned whether the United States would realistically begin a nuclear attack with such a small number of missiles and without other evidence confirming the threat. Rather than treating the computer’s warning as an established fact, he judged that it could be a false alarm.

His decision proved correct. No nuclear attack had occurred. The warning system had misinterpreted information from a satellite surveillance system.

Petrov’s response is widely remembered as an example of human judgement preventing a technological error from escalating into a potentially catastrophic crisis.

The lesson is not that people are always more reliable than computers. Humans make serious mistakes, too. Rather, the incident demonstrates that a machine can detect a signal without necessarily understanding what that signal means.

That distinction becomes especially important when automated systems influence decisions involving human lives, national security, or other consequences that cannot easily be reversed.

As AI enters increasingly sensitive areas, we must ask whether people will retain enough time, information, and authority to question a machine when its conclusions appear convincing but may be wrong.

Why Scientific Knowledge Does Not Automatically Become Practical Power

Another important lesson comes from the activities of Aum Shinrikyo, a Japanese religious cult that attempted to develop biological weapons during the early 1990s.

The organisation had access to trained scientists, laboratory facilities, and specialised equipment. Despite these resources, its attempts to develop effective biological weapons failed.

In July 1993, the group dispersed material containing anthrax from its headquarters in Tokyo. The incident did not result in people becoming ill. Subsequent analysis found that the material contained a strain used in animal vaccination that was not expected to cause disease in healthy people under the circumstances involved.

The equipment also suffered from practical problems that prevented the material from being dispersed effectively.

The episode illustrates a fundamental difference between theoretical knowledge and successful real-world execution.

Understanding a scientific process does not automatically provide the practical skills, experimental experience, equipment, or quality controls required to carry it out successfully.

This distinction matters in the discussion about artificial intelligence.

Modern AI tools can explain complicated scientific concepts, summarise research, and help users explore information that once required years of specialised study. Such capabilities can be beneficial, but they can also raise legitimate concerns about misuse.

However, access to information is not the same as the ability to perform a complex experiment or achieve a particular outcome.

The practical barriers that exist in science do not disappear simply because a person can ask a chatbot for an explanation.

That does not mean AI poses no biological security risks. It means those risks must be evaluated carefully, taking account of what the technology actually enables rather than assuming that access to information automatically translates into practical capability.

Can Artificial Intelligence Make Dangerous Activities Easier?

One of the central questions in AI safety is whether advanced systems can substantially lower the barriers to activities that require specialised knowledge.

A chatbot might help someone understand a scientific paper, organise research findings, or explain an unfamiliar technical concept. More advanced systems can also assist with programming, data analysis, and certain research tasks.

These abilities can save time and make knowledge more accessible.

At the same time, researchers need to understand whether such assistance could make harmful activities easier to carry out.

A study published by RAND in January 2024 examined whether large language models could help participants plan biological attacks. The researchers did not find a statistically significant difference in the assessed viability of plans between participants who received chatbot assistance and those who did not.

Some of the information generated by the models also overlapped with material already available publicly.

The findings offer useful context, but their meaning should not be exaggerated. They do not prove that AI can never increase biological risks. They indicate that the experiment did not demonstrate a significant improvement in the viability of the plans being evaluated.

AI capabilities also continue to evolve, meaning that the results of one experiment cannot settle the question for every future system.

The responsible approach is to evaluate specific capabilities, test them under realistic conditions, and update safety assessments as the technology develops.

AI Is Also Changing Scientific Research

Artificial intelligence is increasingly being used to explore scientific possibilities that would be difficult for humans to examine manually.

In biology, for example, researchers can use computational models to investigate molecular structures, generate candidate designs, and identify promising directions for further study.

Such tools may accelerate parts of the research process. Yet producing a computer-generated design is only one stage of a much larger undertaking.

Researchers must still determine whether a design is scientifically sound, whether it can be produced, and whether experimental results support the predictions.

The difference between generating a theoretical possibility and demonstrating a working result remains important.

As AI becomes more involved in scientific research, safety measures must develop alongside its capabilities. The goal should be to preserve the benefits of faster discovery while preventing powerful tools from creating unnecessary risks.

The Water Treatment Incident That Raised Difficult Questions

In February 2021, an operator at a water treatment facility in Oldsmar, Florida, noticed that the computer cursor appeared to be moving across the screen. The concentration setting for sodium hydroxide, a chemical used in water treatment, was being increased.

At sufficiently high concentrations, sodium hydroxide can pose serious health risks.

The operator quickly reversed the change and notified a supervisor. Local authorities initially described the incident as a suspected cyberattack.

However, the explanation later became less certain.

In 2023, a municipal official stated that the FBI had not found evidence of an external intrusion and suggested that the incident might have originated with the employee. The FBI said it could not confirm whether the event had involved a targeted intrusion.

Consequently, it would be inaccurate to present the Oldsmar incident as a proven external cyberattack.

Nevertheless, the event raises an important question about the security of industrial systems: what happens when a potentially dangerous change can be made through a computer interface, and how quickly can people recognise and stop it?

Water treatment facilities, electricity networks, transport systems, and other essential services depend on interconnected technologies. A failure in these environments can affect far more than a single computer or organisation.

AI may eventually help operators identify unusual behaviour and detect emerging problems. But introducing more automation does not automatically make an insecure system safe.

Strong access controls, independent monitoring, emergency procedures, and properly trained staff remain essential.

Basic Security Failures Can Create Serious Risks

In some cases, the greatest weaknesses in critical infrastructure are surprisingly ordinary.

Poor passwords, outdated software, excessive user permissions, and shared accounts can leave systems vulnerable to unauthorised access or make suspicious activity harder to investigate.

The US Environmental Protection Agency has highlighted cybersecurity deficiencies among water systems it inspected, including weaknesses involving passwords and account management.

These problems deserve attention regardless of how advanced AI becomes.

It makes little sense to focus exclusively on hypothetical future threats from autonomous machines while neglecting basic security failures in systems that already manage essential services.

Artificial intelligence could introduce new risks, but it will also operate within the technological environments humans have built. The security of those environments will influence how safely AI can be deployed.

Human Oversight Is Essential, but It Is Not Enough

A common response to concerns about AI is to insist that a human should always remain in control.

This principle is important, particularly in high-risk applications. However, the presence of a human being somewhere in the process does not automatically guarantee safety.

Imagine an operator responsible for monitoring hundreds of automated alerts. If the system produces warnings continuously, the operator may become exhausted or begin treating them as routine.

Alternatively, a person might receive an AI-generated recommendation that appears authoritative but is based on incomplete information.

If the person lacks the time, expertise, or authority to challenge that recommendation, human oversight may exist in theory while offering little protection in practice.

This is sometimes described as a problem of automation bias: people can place excessive confidence in automated recommendations, even when there are reasons to question them.

The solution is not to exclude humans from technological systems or assume that human intuition will always prevail.

Instead, people and machines should be able to compensate for one another’s weaknesses.

Effective safeguards include:

• Strict access controls: Only authorised personnel should be able to change critical settings or initiate sensitive operations.
• Independent verification: Important decisions should be checked using reliable evidence that does not depend entirely on the same system producing the recommendation.
• Emergency controls: Operators must have practical ways to pause or stop automated processes when something appears unsafe.
• Proper training: Staff should understand the limitations of AI and know how to recognise questionable outputs.
• Clear accountability: Organisations must define who has the authority to approve high-impact decisions and who is responsible for reviewing them.

These protections become particularly important when AI systems move beyond answering questions and begin interacting directly with software, equipment, and external services.

The objective is not merely to keep a person involved. It is to ensure that the person can make a meaningful contribution to the decision.

Why Nuclear Decisions Must Remain Under Human Control

The risks of excessive automation become especially clear when nuclear weapons are involved.

In November 2024, US President Joe Biden and Chinese President Xi Jinping agreed that the decision to employ nuclear weapons should remain under human control. The commitment was announced following their meeting in Lima, Peru.

The agreement reflected a shared recognition that decisions with consequences of this magnitude should not be left to automated systems.

Nuclear decision-making often takes place under intense pressure, with limited time to assess incoming information. An inaccurate warning or misleading recommendation could have consequences far beyond those associated with an ordinary technical error.

Artificial intelligence could potentially help analysts process information, identify patterns, and evaluate different scenarios. Such assistance might be useful, but it also creates questions about reliability, interpretation, and accountability.

What happens if an AI system incorrectly identifies a threat? What if a decision-maker accepts its assessment without checking the underlying evidence? And what if there is too little time to correct the mistake?

These questions concern more than whether a machine can independently control a weapon.

They also concern the influence that automated recommendations can exert on human decisions.

Keeping people responsible for nuclear decisions is therefore an important safeguard. But that safeguard must be supported by reliable information, rigorous procedures, and sufficient time to assess uncertain evidence.

When an AI Error Could Influence a Military Operation

The potential consequences of AI-generated misinformation become even clearer when its output enters a military decision-making process.

The original report describes an incident in which an American military analyst reportedly used a chatbot while preparing an intelligence assessment concerning a Chinese vessel.

According to the account, the chatbot misinterpreted information about the vessel’s cargo and produced an assessment suggesting a possible military action. Aircraft were reportedly already airborne and troops were preparing to board the vessel before the mistake was identified.

This account illustrates the importance of independently verifying AI-generated intelligence before it influences operational decisions.

However, the specific incident should be treated as a reported account rather than independent proof of a general pattern.

The broader concern is straightforward: an inaccurate answer in an ordinary conversation may cause confusion, but the same error can have much more serious consequences when incorporated into an intelligence report or operational plan.

AI-generated information can appear polished, coherent, and convincing even when its underlying assumptions are incorrect.

In high-stakes settings, presentation must never be confused with accuracy.

Human reviewers need access to the original evidence, enough technical knowledge to evaluate the system’s conclusions, and the authority to reject recommendations that cannot be substantiated.

Otherwise, a human approval step risks becoming little more than a formality.

The Real Test Will Come When There Is No Time to Check

The historical examples discussed here do not prove that every feared AI scenario is impossible. Nor do they demonstrate that human supervision alone can prevent every disaster.

They reveal something more immediate: complex technological systems can fail, people can misinterpret information, and mistakes can become dangerous when decisions must be made quickly.

Artificial intelligence may help organisations process information faster and detect problems earlier. Yet it can also accelerate the spread of an error when its output is accepted without sufficient verification.

The danger is particularly serious when several weaknesses occur together: unreliable information, inadequate security, excessive trust in automation, and insufficient time for human review.

Stanislav Petrov’s experience in 1983 offers a powerful historical reminder. A computer system presented him with a warning that appeared credible, but he did not treat that warning as unquestionable truth.

Modern AI systems create a different technological situation, but the underlying challenge remains familiar.

The question is not simply whether machines can make mistakes. It is whether the people depending on them can recognise those mistakes before they lead to irreversible consequences.

Conclusion: The Future of AI Depends on Human Responsibility

Artificial intelligence is neither an automatic path to disaster nor a guarantee of progress. Its effects will depend on the systems we build, the decisions we delegate, and the safeguards we put in place.

AI can improve scientific research, strengthen cybersecurity, support decision-making, and help people solve problems that once required enormous amounts of time and effort.

But these benefits do not remove the need for critical thinking, independent verification, and responsible human judgement.

History shows that dangerous situations can emerge when people place too much confidence in technology, fail to address basic vulnerabilities, or allow automated recommendations to influence decisions without adequate scrutiny.

As AI becomes more deeply integrated into essential services, scientific research, and national security, the responsibility to manage these risks will become increasingly important.

The answer is not to reject technological progress. It is to ensure that progress is accompanied by sound judgement, reliable safeguards, and clear accountability.

Ultimately, the most important question may not be how intelligent artificial intelligence becomes, but how wisely humanity chooses to use it.

The machines we create may become increasingly capable. Whether they make our world safer or more dangerous will depend, in large part, on the choices we make around them.