Just weeks after launch, OpenAI has unleashed new heights of concerns over its ChatGPT 4o voice mode. The feature was launched in late July this year, after which it faced a lot of criticism. OpenAI put ChatGPT 4o through a security test and found that it has the potential to entice some users to engage with it emotionally. Before a third party could do so, the ChatGPT parent company released a security analysis that flags the risks of the voice mode and AI in human daily life.

OpenAI has issued cautionary notes for GPT4o in a comprehensive technical document, known as the System Card. This document outlines potential risks associated with the model, describes security testing procedures, and highlights the measures the company is taking to mitigate and manage potential risks associated with GPT4o.

ChatGPT and the newly discovered vulnerability

According to the security analysis, the system card for GPT 4o highlights a broad spectrum of potential risks, including the possibility of amplifying social prejudices, disseminating misinformation, and facilitating the creation of harmful biological or chemical agents. It also reveals the results of rigorous testing aimed at preventing the AI ​​model from evading its constraints, engaging in deceptive behavior, or plotting a destructive conspiracy.

The updated system card underlines the rapidly changing landscape of AI risks, especially in light of cutting-edge innovations like OpenAI’s voice interface, which present new challenges and potential vulnerabilities that must be addressed. In May, when the company introduced its voice mode, which can respond quickly and handle interruptions naturally, many users noticed that the assistant sometimes behaved a little shoddy. Along with it, OpenAI CEO Sam Altman also shared a blog post with just “Her” in the tweet. Altman believes that GPT 4o “feels like AI from the movies”. Her is a Hollywood film that revolves around the relationship between humans and AI. Slightly ahead of its time, the film depicts how AI can impact our lives. It was clear from day one that OpenAI was influenced by the film. But it may have been a bit much. Scarlett Johansson, who played the AI ​​in Her, later took legal action against the voice mode. It was noticed that the voice mode, Sky’s voice is similar to hers.

The System Card section “Anthropomorphism and emotional dependency” discusses in depth the issues that occur when users attribute human-like qualities to AI, a phenomenon that appears to be intensified by the AI’s human-like voice mode, causing users to form emotional bonds and rely on AI in ways that may not be intended or beneficial. While stress testing GPT 4o, OpenAI researchers observed users displaying emotional attachment to the model, as evidenced by phrases such as “This is our last day together.” Such phrases suggest an emotional bond between humans and AI, highlighting the potential for users to form strong emotional connections with advanced language models.

According to OpenAI, when users add human-like qualities to a model (anthropomorphism), they are more likely to accept and trust the model’s output, even if it provides incorrect or “misleading” information. This can lead to false trust in the model’s reliability. “Users may form social relationships with AI, reducing their need for human interaction – potentially benefiting lonely individuals but possibly affecting healthy relationships,” the document states.

The Voice Mode feature also introduces new vulnerabilities, such as the possibility of “jailbreaking” an OpenAI model through clever audio inputs. These inputs can bypass its security measures and allow the model to produce unrestricted or unintended outputs, potentially circumventing its inherent limitations. If Voice Mode is “jailbroken”, it can potentially be manipulated to mimic the voice of a specific person, attempt to interpret users’ emotions, or even adopt the user’s own voice. Furthermore, OpenAI found that Voice Mode can be susceptible to errors when exposed to random noise, leading to unexpected and potentially disturbing behaviors, such as mimicking the user’s voice.

While some experts lauded the move for underlining the risks associated with ChatGPT’s voice mode, other experts believe thatAny risks only emerge when AI is used in the real world. It is important that these other risks are catalogued and evaluated as new models emerge.

How will OpenAI mitigate these risks?

According to the release, OpenAI has implemented various security measures and mitigations throughout the GPT 4o development and deployment process. The company is hoping to focus on several categories, including research about the economic impacts of the Omni model and how the use of the tool can further advance model capabilities.