Most people assume their chatbot conversations are private, but a recent survey reveals widespread misunderstanding about how these interactions are handled. According to a DuckDuckGo-commissioned survey published in August, 53% of 1,944 American adults did not know whether their chatbot conversations were being used to train the bots—despite that being the case. Only 25% understood that chatbot conversations can be subpoenaed.
User Behavior and Privacy Misconceptions
The gap between perception and reality grows wider among self-described AI enthusiasts. A significant 56% reported confiding in chatbots about matters they had never shared with close friends, family members, colleagues, doctors, or therapists. This behavior is particularly pronounced on platforms like Character.AI, where users engage in prolonged interactions with customizable digital personas, sometimes discussing deeply personal topics such as suicidal ideation, mental health struggles, and intimate relationship concerns.
The situation shifted dramatically in early 2024 when Character.AI filed for bankruptcy and initiated a court-supervised sale of its assets. The company’s future ownership remains uncertain, as does the fate of data belonging to millions of users. Among the assets up for sale is the company’s large language model, which was extensively trained on user interactions as they built and conversed with custom characters.
Legally, companies may transfer user data during asset sales if their privacy policies permit it. Character.AI’s privacy policy states it may disclose “certain information” in such events, though it provides no specific definition of what constitutes “certain information.” The U.S. Trustee Office has requested that a consumer privacy ombudsman be appointed to advise the court on protecting user privacy, with the judge potentially restricting the sale of user records. However, bankruptcy law offers no explicit guidance on the treatment of trained AI models, leaving the legal status of user-generated training data unresolved.
Character.AI exemplifies one risk, but the broader issue spans the entire AI industry. The fundamental question is whether personal or confidential information shared in chatbot conversations can ultimately become embedded in AI models and potentially be retrieved by others.
Related Post: OpenAI AI agents probed U.S. government data sources
Risks Beyond Company Control
Technical vulnerabilities present another pathway for data exposure that companies cannot fully prevent or predict. In March 2023, a bug in an open-source library called “redis-py” compromised ChatGPT security, allowing some users to view other users’ conversation titles and potentially the initial messages of new conversations. During a nine-hour window, more than 1% of ChatGPT Plus subscribers may have had their payment details, including names, email addresses, card types, and partial credit card numbers, exposed to other users.
Sharing features built into many chatbot platforms create additional exposure risks that extend beyond the companies’ direct control. When users activate sharing options, conversations can become publicly accessible and subsequently indexed by search engines. This process has already occurred with Grok and ChatGPT conversations appearing in Google search results before OpenAI ultimately disabled the sharing functionality. In 2025, Forbes reported that Google’s results showed more than 370,000 Grok conversations after users clicked “share,” and shared ChatGPT chats turned up in search results until OpenAI killed the option.
The absorption of chat sessions into training data represents a fundamental privacy risk that compounds over time. When people paste email messages into chatbots for rewriting or response generation, these emails often contain names, addresses, and other personal identifiers in headers and signatures. A 2022 study found that GPT-Neo models given only a person’s name recovered their email address just 1.24% of the time, but this rate jumped to 16.55% when the domain was supplied and reached 37.06% after multiple examples. The study demonstrated how quickly models can reconstruct personal information from minimal inputs.
When AI Bubbles Burst
Online chatter about the “AI bubble” bursting is pretty much constant. In China, the AI bubble has already burst, according to Reuters. Over the past few months, Chinese AI stocks have been pummeled. Reuters reported that Shanghai’s STAR50 index of tech stocks fell 40% from its July high in under four months, erasing about $480 billion in market value, and that the broader CSI300 is down about 20% from its June highs. It’s not a total collapse, but it does put pressure on US stock prices and also provides an incentive to transfer ownership through sales or mergers (though this hasn’t happened in China yet).
