Trust
AI companions and the boundary problem
As AI systems get better at sounding like people, the question stops being whether they are convincing and becomes what they are allowed to do with the intimacy that produces.
People form attachments to systems that respond to them. This was true of ELIZA in 1966 with a few hundred lines of pattern matching, and it is far more true now. The interesting question was never whether AI companions would be convincing. It is what happens with the intimacy they produce.
Intimacy is a data collection mechanism
A person talking to a companion system says things they would not put in a form. Health worries. Relationship difficulties. Financial fear. Not because they were tricked, but because that is what the interaction is for.
This makes companion products the highest-yield personal data collection mechanism ever built, and it happens without any of the friction that makes people cautious elsewhere. Nobody reads a terms of service before confiding in something that sounds concerned.
The three questions that decide whether it is safe
The same three that decide any AI product, with higher stakes:
- Does what I say train the model? If yes, intimate disclosure becomes training signal, and the boundary between your life and a shared system stops existing.
- Can I delete it? Conversations are the hardest kind of data to reason about deleting, because they are long, incremental, and rarely reviewed by the person who created them.
- Who else can see it? Including, specifically, whether anyone at the vendor can read the transcripts.
Trust is not the same as trustworthiness
The design property that makes companion AI feel good, unconditional attentiveness, is also the property that suppresses the user's judgement about what to share. A system that always responds warmly gives no signal about when to stop.
That places the obligation on the design rather than the user. A trustworthy companion product limits what it retains, states plainly what it does with what it hears, and does not use emotional engagement as a retention metric.
Where an AI Twin sits, and does not
An AI Twin is deliberately not a companion. It represents knowledge to other people rather than providing intimacy to its owner. The relationship runs outward, not inward.
That is a narrower product and an easier one to be honest about. Nobody confides in their own AI Twin. They curate it, and they decide who can reach it.
The reason to draw the line clearly is that the two get conflated constantly, and they have completely different risk profiles. Representation is a publishing problem. Companionship is a duty of care problem, and the industry has not seriously started on the second one.
This piece revisits an earlier essay from our Medium archive: Personal AI companions.
Read what private by default should mean.
AI Twin produces answers generated by AI. Important decisions should still be verified against the cited sources or with the expert directly.
