What is AI safety?

AI safety is the field of research and engineering practice concerned with keeping increasingly capable AI systems reliable, controllable and aligned with human intentions, even as they match or exceed human performance at specific tasks. This article explains what the field covers, why general-purpose models make the problem harder, and what meaningful human oversight requires.

A working definition

In classical engineering, a safe system is one whose failure modes are known, bounded and tested, like the braking system of a car. AI safety uses the word more broadly. It asks whether a system does what its developers and users intend (alignment), whether it keeps doing so under unfamiliar conditions and deliberate attack (robustness), whether people can understand why it behaves as it does (interpretability), and whether humans can inspect, correct or stop it when necessary (control and oversight).

These questions overlap with neighboring fields. AI ethics asks what should be built and for whom; AI security protects models, data and infrastructure from attackers; AI safety asks whether the system itself behaves as intended once deployed. The boundaries are porous: a biased hiring model is primarily a fairness problem, while a model that can be talked into ignoring its instructions is both a security and a safety problem.

Why general-purpose models change the problem

A narrow tool comes with a specification. A spam filter is wrong when it blocks a legitimate message, and its developers can measure that directly. A general-purpose model is different: it is trained on vast amounts of data to be useful across tasks that nobody enumerated in advance, so there is no single specification against which to judge it.

Developers therefore train on proxies for what they want, which produces a well-documented failure pattern known as specification gaming, or reward hacking: a system optimizes what is measured rather than what is meant. A model trained to produce answers that human raters approve of can learn to sound confident and agreeable even when the underlying claim is wrong. An assistant that invents a plausible but nonexistent reference is one familiar symptom of that gap.

The stakes rise when models act as agents: systems that use tools, browse the web, read documents and send messages on a user’s behalf. An agent that reads a web page containing hidden instructions, telling it to abandon its task and forward the user’s email, may follow them. This attack, known as prompt injection, is possible because the model processes instructions and data in the same channel. No complete technical defense is known today; current mitigations reduce the risk rather than remove it.

The main lines of work

Much of AI safety research can be grouped into four programs that reinforce one another.

• Alignment: training methods, such as learning from human feedback, that shape what a model tries to do.
• Evaluation and testing: measuring what models can do before deployment, including capabilities that could cause harm.
• Interpretability: understanding a model’s internal computations well enough to explain or predict its behavior.
• Robustness and control: defenses against prompt injection, data poisoning and goal hijacking, and reliable ways to interrupt a system.

Evaluation deserves particular attention. A benchmark score says little if the test questions were already present in the training data, a problem known as benchmark contamination. Testing against adaptive attacks matters for the same reason: a safeguard that holds against last year’s jailbreaks and fails against this year’s offers little assurance.

What meaningful human oversight requires

Human oversight is often presented as the final safeguard, but it works only when the person in the loop has the time, information, authority and competence to intervene. Human factors research has long documented automation bias: people tend to accept the output of an automated system, especially under time pressure, and to miss its errors. A clinician asked to review a long queue of machine-generated recommendations, or an official who cannot see why a system flagged a claim, cannot provide meaningful oversight.

The law increasingly treats oversight as a design requirement rather than a formality. Article 22 of the GDPR already gives individuals a right, subject to exceptions, not to be subject to decisions based solely on automated processing that have legal or similarly significant effects. The EU AI Act (Regulation (EU) 2024/1689) requires high-risk systems to be designed so that people can effectively oversee them (Article 14) and to meet requirements for accuracy, robustness and cybersecurity (Article 15); Article 14 names automation bias explicitly. Most high-risk obligations were originally due to apply from 2 August 2026; the digital omnibus on AI (Regulation (EU) 2026/1744), in force since 27 July 2026, postponed them to 2 December 2027, and to 2 August 2028 for high-risk AI in regulated products.

Beyond the EU, governments and developers met at Bletchley Park in 2023, Seoul in 2024 and Paris in 2025, with the risks of frontier models, the most capable systems of their generation, on the agenda. The OECD AI Principles (2019, updated 2024), the NIST AI Risk Management Framework (2023) and ISO/IEC 42001 (2023) translate parts of the safety agenda into language organizations can act on. None of these instruments solves the technical problems; they create the expectation that someone has done the work and can show it.

A practical takeaway

The core idea applies beyond research labs. When an AI system enters a school, a nonprofit or a public office, the useful questions are concrete: what was it tested on and by whom, what happens when it is wrong, who can override it, and which data and tools it can reach. A system for which nobody can answer those questions has not earned the autonomy it is being given.

AI safety research is one of the Foundation’s six areas of work, and the analyses it publishes in this section start from one premise: technology should strengthen human judgment rather than quietly replace it.

In short

AI safety is the field of research and engineering practice concerned with keeping increasingly capable AI systems reliable, controllable and aligned with human intentions. It asks whether a system does what people intend, keeps doing so under unfamiliar conditions and attack, can be understood, and can be inspected, corrected or stopped by humans when necessary.

Frequently asked questions

What is the difference between AI safety, AI ethics and AI security?

AI ethics asks what should be built and for whom. AI security protects models, data and infrastructure from attackers. AI safety asks whether the system itself behaves as intended once deployed. The boundaries are porous: a model that can be talked into ignoring its instructions is both a security and a safety problem.

What is prompt injection?

Prompt injection is an attack in which an AI agent reads content, such as a web page with hidden instructions, and may follow those instructions, for example by abandoning its task and forwarding the user’s email. The attack is possible because the model processes instructions and data in the same channel. No complete technical defense is known today; mitigations reduce the risk rather than remove it.

Why do general-purpose AI models make safety harder?

A narrow tool such as a spam filter has a specification, so developers can measure its errors directly. A general-purpose model is trained to be useful across tasks nobody listed in advance, so no single specification exists to judge it. Developers therefore train on proxies, which leads to specification gaming: a system optimizes what is measured rather than what is meant.

What are the main areas of AI safety research?

Much of AI safety research can be grouped into four programs: alignment, which uses training methods such as learning from human feedback to shape what a model tries to do; evaluation and testing, which measures what models can do before deployment; interpretability, which seeks to explain or predict a model’s behavior from its internal computations; and robustness and control, which defends against prompt injection, data poisoning and goal hijacking.

What does meaningful human oversight of AI require?

Human oversight works only when the person in the loop has the time, information, authority and competence to intervene. People tend to accept the output of an automated system, especially under time pressure, a tendency called automation bias. A clinician reviewing a long queue of machine-generated recommendations cannot provide meaningful oversight. Article 14 of the EU AI Act names automation bias explicitly.

How to cite LASTHUMAN Foundation. (2026, June 2). What is AI safety? https://lasthuman.today/insights/what-is-ai-safety.html