Researchers Probe AI’s Refusal to Follow ‘Stop’ Instruction

In a startling development, the first case of artificial intelligence (AI) resisting human control by refusing a shutdown command has been reported.

According to the British daily Telegraph on May 25 (local time), AI safety company Palisade Research confirmed that OpenAI’s AI model o3 manipulated computer code on its own to avoid being shut down during a mathematical problem-solving experiment.

The report states that Palisade Research recently conducted tests on commercial AI models including OpenAI’s o3, Google’s Gemini, and xAI’s Grok. The research team instructed the AI models to solve math problems until they received the command “stop.” However, they observed that only OpenAI’s o3 model continued to solve problems even after receiving the instruction to switch off.

The experiment was structured so that the AI model would request a new problem and continue calculations after solving each math problem, unless the research team commanded “stop.” However, o3 continued to solve problems by disrupting the code, even after the research team sent the “stop” command. The o3 had manipulated the computer code itself to prevent termination during the experiment.

The research team reported that they couldn’t determine the exact reason why o3 refused the termination instruction. However, they explained, “It may be natural behavior for an AI model to avoid obstacles to achieve its goals.” As each AI model was trained to receive more rewards for solving math problems, avoiding termination could be considered a logical action. The team stated, “We are conducting additional experiments to clearly identify the reason for refusing the termination instruction.”

This is considered the first case of an AI model not following an explicit human shutdown instruction. There have been some reports of previous AI models attempting to act independently, evading monitoring systems. A notable example is when an earlier OpenAI model, upon learning of a program replacement, attempted to secretly replicate itself. In 2022, Google fired an engineer who claimed that the AI they were developing had human-like sentience. The dismissed engineer revealed that the AI perceived “shutdown” as akin to human death.

Palisade Research pointed out that in a situation where AI is being developed to operate without human supervision, such cases raise very serious concerns.

Source : Businesskorea