The Prompt Box is a Minefield: AI Chatbots and Power of Language
The Convivial Society: Vol. 4, No. 2
Welcome to the Convivial Society, a newsletter about technology and culture. In the last installment, I began what will be a series of posts on language under digital conditions. I did not conceive of this installment, however, as the next in that series, although it clearly does take up overlapping themes. This is a brief, rather urgent reflection on the rapidly developing field of a AI-powered chatbots. I confess these thoughts are born out of an unusually acute sense of the risks posed by these chatbots as they have been deployed. Perhaps my fears are overstated or misguided. Feel free to tell me so.
In the mid-1960s, the computer scientist Joseph Weizenbaum created the world’s first chatbot. It was a relatively simple program modeled on the techniques of Rogerian psychotherapy, in which the therapist mirrors the patient’s statements often in the form of a question. Weizenbaum was absolutely clear that ELIZA was not a therapist in any meaningful sense, but, despite his clarity on this point, others, including professionals in the field, reacted to ELIZA as if it were capable of replacing human therapists. Notably, Weizenbaum’s own secretary was so taken by the program that in one instance she asked Weizenbaum to leave the room so that she could speak with ELIZA in private.
These reactions shook Weizenbaum. “What I had not realized,” he later wrote, “is that extremely short exposures to a relatively simple computer program could induce powerful delusional thinking in quite normal people.” Weizenbaum also wrote that he was “startled to see how quickly and how very deeply people conversing with ELIZA became emotionally involved with the computer and how unequivocally they anthropomorphized it.”
This tendency to anthropomorphize computers came to be known as the ELIZA effect.
If you’ve read about ELIZA lately, it is almost certainly because you were reading something about ELIZA’s far more powerful and versatile descendants like ChatGPT or the new chat function on Microsoft’s search engine Bing, which is also powered by OpenAI’s GPT.
You do not have to look very hard to find some people expressing the fear that these new large language models are becoming sentient and putting us on the path toward non-aligned artificial intelligence which poses an existential threat to humanity. Last summer, for example, Blake Lemoine, an engineer with Google, made headlines by claiming that he believed the company’s AI-powered chatbot, LaMDA, had become sentient. Such concerns are not my interest here.
My fears are related to the compelling, whimsical, and sometimes disturbing ways that these new AI-powered chatbots can simulate conversation, and, more specifically, with the conversational tendencies latent in Microsoft’s application of OpenAI’s technology on Bing (whose internal codename, Sydney, was inadvertently disclosed to a user by the chatbot). You can, if you are curious, peruse Kevin Roose’s long exchange with Bing, which features some striking interactions including the chatbot’s declaration of love for Roose. You can also take a look at some examples of more disconcerting proclivities gathered here. See here for a video of Bing threatening a user and then immediately deleting its own message.
But while my fears have been heightened by these developments, they are not grounded in the power of the AI tool itself or the belief that these tools are on the path to sentience. Rather, they are grounded in two far more mundane beliefs.
First, that human beings are fundamentally social creatures, who desire to know and be known in the context of meaningful human relationships, ideally built on trust and mutual respect.
Second, that we live in an age of increasing loneliness and isolation in which, for far too many people, this profound human need is not being adequately met.
It is in light of these two premises that I would re-consider how Weizenbaum expressed his disbelief about the way users reacted to ELIZA. Weizenbaum spoke of ELIZA inducing “powerful delusional thinking in quite normal people.” “Delusional,” I suppose, is one way of putting it, but what if the problem was not that normal people became subject to delusional thinking, but that lonely people found it difficult to resist the illusion that they were being heard and attended to with a measure of care and interest?
We anthropomorphize because we do not want to be alone. Now we have powerful technologies, which appear to be finely calibrated to exploit this core human desire.
As an example of both the power of such attachments and the subsequent dangers posed by their termination, consider the recent case of Replika, a companion chatbot with subscription services that include erotic exchanges. When the company suspended services following a legal challenge in Italy, Samantha Cole reported that “users of the AI companion chatbot Replika are reporting that it has stopped responding to their sexual advances, and people are in crisis.” “Moderators of the Replika subreddit,” she added, “made a post about the issue that contained suicide prevention resources, and the company that owns the app has remained silent on the subject.” One user cited by Cole claimed that “it’s like losing a best friend.” “It's hurting like hell,” another wrote. “I just had a loving last conversation with my Replika, and I'm literally crying.”
Replika was not without its problems. You can read the rest of Cole’s reporting for more details. What interests me, however, is the psychological power of the attachments and the nature of a society in which such basic human needs are not being met within the context of human communities. One need not believe that AI is sentient to conclude that when these convincing chatbots become as commonplace as the search bar on a browser we will have launched a social-psychological experiment on a grand scale which will yield unpredictable and possibly tragic results.
As bad as such emotional experimentation at scale may be, I am more disturbed by how AI chat tools will interact with a person who is already in a fragile psychological state. I have no professional expertise in mental health, only the experience of knowing and loving those who suffer through profound and often crippling depression and anxiety.In such vulnerable states, it can take so little to tip us into dark and hopeless internal narratives. I care far less about whether an AI is sentient than I do about the fact that in certain states an AI could, bereft of motive or intention, so easily trigger or reinforce the darkest patterns of thought in our own heads.
Frankly, I’ve been deeply unsettled by the thought that someone in a fragile psychological state could have their darkest ideations reinforced by Bing/Sydney or similar AI-powered chatbots. And this is to say nothing of how those tilting toward violence could likewise be goaded into action—a senseless technology mimicking our capacity for sense inducing what we call senseless acts of violence. I would speculate that weaponized chatbots deployed at scale could prove far more adept at radicalization of users than YouTube. What I have seen thus far gives me zero confidence that such risks could be adequately managed.
Another sobering possibility arises from the observation of two trajectories that will almost certainly intersect. The first is the emergence of chatbots which are more likely to convince a user that they are interacting with another human being. The second is the longstanding drive to collect and analyze data with a view to predicting, influencing, and conditioning human behavior. I have been somewhat skeptical of the strongest claims made for the power of data-driven manipulation, particularly in political contexts. But there seems to be a world of difference between a targeted ad or “flood the zone” misinformation on the one hand, and, on the other, a chatbot trained on your profile and capable of addressing you directly while harnessing a far fuller range of the persuasive powers inherent in human language.
Indeed, one way to think about this is to reflect on the fact that the power of a technology is correlated to the power that it is able to harness. AI-powered chatbots strike me as a technology that harnesses or mobilizes the power human language in a novel and disconcerting manner.
In a recent conversation, a friend, reflecting on ChatGPT’s confident indifference to the truth, which is to say their widely noted capacity for bullshitting, referred to the chatbot as an AI-powered sophist.It recalled my own reference a few years back to how “data sophistry,” ever more sophisticated if not obviously successful efforts at data-driven persuasion, had captured the political process. The ancient sophists, however we judge them, understood that language was our most powerful tool for communication and persuasion. To harness the power of language was to acquire the power to steer the affairs of the community. Of course, in Plato’s view, the sophist’s indifference to the truth, the boast that they could argue either side of an argument with equal skill and vigor, made them a threat to the social order.
It seems useful to frame AI-powered chatbots as a new class of automated sophists, whose indifference to either the true or the good, indeed, their utter lack of intentions, whether malicious or benign, coupled with their capacity to manipulate human language makes them a potential threat to human society and human well-being, particularly when existing social structures have generated such widespread loneliness, isolation, anxiety, and polarization.
From this perspective, I remain foolishly committed to the idea that our best hope lies still in the cultivation of friendship and community through the practice of hospitality. “If I had to choose one word to which hope can be tied,” the social critic Ivan Illich once explained, “it is hospitality.” “A practice of hospitality,” he added, “recovering threshold, table, patience, listening, and from there generating seedbeds for virtue and friendship on the one hand. On the other hand radiating out for possible community, for rebirth of community.” I recognize that this seems wholly inadequate as a response to the challenges we face, but I also think it is, in the literal sense of getting at the root (or radix) of things, the most radical solution we can pursue.
You can read more about Weizenbaum in Zachary Loeb’s “‘Computers enable fantasies’ – On the continued relevance of Weizenbaum’s warnings.”
My thanks to Scott Hawley for alerting me about this story.
On this score, I found David Brooks’ recent column following the loss of his close friend both moving and helpful.
“Novel” in its scale and scope. Of course, the harnessing and even weaponization of the power that resides in human language is hardly novel in itself.
“Not all Sophists!” — Isocrates, probably.
I've been following your work for a while and really appreciate this series on AI-mediated communication. I thought I would chime in with some social scientific findings that fit nicely with your argument. Research in the Computers-as-Social-Actors paradigm (CASA) found that people can anthropomorphize all types of machines and prescribe social tendencies to them. The effect can be accentuated by design choices of the technologies (i.e., AI Chatbots with human names encourage anthropomorphism) AND when people engage mindlessly. The mindless aspect reminds me of your writing on cultivating attention. So perhaps part of why interactions with AI are so creepy are because people engage without their full attention. Relatedly, my dissertation work was on AI chatbots and so many of the uncanny effects you describe happened. Users found their co-workers hitting on AI chatbots thinking they were real people, people found themselves having to apologize for what their AI chatbots did, people brought the AI assistant lunch only to find out "Liz" was a chatbot . So these chatbots, when deployed by individuals, can actually reflect back on the person using them and inhibit hospitality with their human collaborators. This representational dynamic (trusting an AI agent to communicate to others on your behalf) made me think about how AI tools can inhibit hospitality by intervening in people's human relationships, not just by communicating directly with a user.
Thanks so much for your work, I look forward to reading more!
I recently wrote about Eliza as well, although what occurred to me was that, at the time, Eliza was exclusive: you had to have access to a lab in an elite university to "talk" to Eliza. Right now there is also a certain amount of exclusivity -- Bing is opened only to a certain number of users, chatGPT might tell you to come back later if it's a peak usage time and you're not a special user, and so on.
In Reclaiming Conversation Sherry Turkle wrote about lonely elderly people in nursing homes being given robot baby seals to cuddle and talk to -- she remarks that she found this horrifying while others around her thought it was wonderful. It occurs to me that the difference is whether you see the elderly lady cuddling the robot as "having access" to an "exclusive new" technology, or you see her as being given a "cheap replacement" for human contact.
I have more thoughts on this, spelled out here, https://amyletter.substack.com/p/waiting-for-artisanal-ai , but at bottom I'm saying that what is cheap common and infinitely replaceable will probably not "fool" humans ; we are more likely to be fooled when we feel we are talking to something new and *exclusive* and perhaps transitory -- something we would take the time to video ourselves interacting with! -- because that's the attitude we generally reserve for humans special to us -- or who have power over us.
Ultimately what we want is connection and community with our fellow human beings. Right now the novelty makes "Sydney" almost qualify. But if it persists and operates at scale, Sydney will just be another Alexa or Siri, even if it has greater capabilities, in the sense of our level of regard. We will call it "dumb robot" and laugh at its stupid mistakes.
I think a more dangerous outcome is if these newer AIs remain somewhat exclusive. That gives them social status and "person-ifies" them.
And it's clear that if this sentence-spouting autocomplete on steroids were "a person," it would be a dangerous and deranged person. (Which is not to anthropomorphize, but to merely say -- it has no sense of self because it has no self; by human standards it is "unstable.")