I loved this post, as well as the invitation to read Barba-Kay’s letter and Illich’s address in honor of Elul.
In my own work on understanding financial markets, I have been trying to make sense of the tension between individual and collective self-determination, and the ways in which expanding one supports vs undercuts the other. The challenge is always that there are trade-offs, and no easy accounting of things being clearly better or worse. It’s hard to understand the counter factual of what life would be like outside the system from within the system.
In that spirit, I thought one of the more interesting parts of Barba-Kay’s writing was his inquiry into what is that technologists think they are doing. He says AI is like a drug in that it transforms our choosing and thinking - but I do think that is the intended point. A feature not a bug. And the transformation, in a more optimistic reading, is being able to do (think, create) what seemed previously impossible. Barba-Kay counters that “either we must be able to say something about what makes reading, writing, and speech distinctly human or we must be open to redefining what logos is at all.”
But I worry that the focus on the individual, and their own experience of reading, writing, and speech is part of the trap. On an individual level, the technology can be delicious (and I intentionally use that word, because it is true of my personal experience, even though I acknowledge that Illich would scoff at it). It expands the possibility space and the subjective experience is that it can be a tool, even if that reading is misleading, it doesn’t yet feel like it is. But wonder if what it means for what we do together, and how we act collectively, is easier to see. Thinking here of Simone Weil and a focus on obligations and what we owe each other, rather than rights (and a good moment to thank you for introducing me to Weil in another post!)
Illich’s project of a new asceticism sounds a lot like the mindful withdrawal from the all-encompassing Machine that Paul Kingsnorth advocates in “Against The Machine.” I find it a highly valuable perspective, even if I don’t entirely embrace it or practice it myself.
Thanks for your essay/warning. I have sometimes allowed myself to be lulled into the view that AI (and technology in general) is merely a tool to be used for good or bad. This is obviously naïve and I appreciate your reminder. Technology obviously changes us. The automobile and the TV clearly changed society in the last century; smartphones and social media have clearly done so in this century.
As for AI in particular, one of the problems is that it’s an umbrella term, yet it gets thrown around as if it’s one thing. Size matters. There are small AI systems that run on a single computer and do a single thing and can be extremely useful as tools. And then there are LLMs, and sycophantic chatbots, and the pursuit of AGI, and all that other crap and, well, I think that’s the kind of AI you’re getting at.
ps. Thank you for adding the word “irenic” to my vocabulary. :)
I have been thinking about AI as a tool, and so I appreciate your argument that it is not a tool but instead a system. That's making me pause and reconsider.
My motivation for thinking of AI as a tool is that I am trying to find ways to live well with it, because it does feel inevitable in the way to most technologies I have experienced in my life feel inevitable, and I am also genuinely interested in how it might make our lives better.
I am left wondering now, if AI is a system and not a tool, then what tools do we need to live well within this system? What tools do we need to bring order to this AI infused digital environment so that the environment aligns with our flourishing? How do we do this together (I'd love to read more about what you think about the role friendship plays)?
As someone who has read your work for years, I hesitate at your "pugnacious rhetorical mode", not because it's off-putting, but because of how it affects your argument. I support your project of training in perception. But perhaps because of this training, the watchdog of my mind recognizes the tension between your premises and conclusion, especially in the juicier rhetoric. If smartphones "(mal)form even the most vigilant and virtuous user", if (as the reader you quote put it) AI use recruits and malforms the diligent, if (per Barba-Kay) the neutrality claim is "interchangeable with the statement that cocaine can be a valuable drug", then what can training do? The ascetical mode only makes sense if it changes outcomes. If the environment is pervasive and forms even the disciplined, then fatalism is the resultant attitude. You can't escape an environment. You acclimatize to it well or poorly. The call to perceive how the environment shapes our mind presupposes the ability to observe. Your final sentence, "At the very least we should dispense with the idea that AI is just a tool we need to learn to use wisely", highlights this tension. Certainly we should dispense with the "just a tool" label, but the sentence reads as if it also dispenses with wise use. If asceticism is training to know what to refuse, when to refuse, and for the sake of preserving what, isn't that learning to use it wisely, or better, learning to flourish in the environment in which we now live?
I like your thoughts very much. The common argument of AI being "just a tool" is very naïve and hides the profound influence technology has on our perception. I have been thinking recently that AI can be regarded as the culmination of a process of continuous abstraction from reality that started with the alphabet and further unfolded itself with the experimental method and digitalization … following the basic principle of breaking down the fluid and ever-changing wonder of life into discrete, measurable units and thus objectifying the world.
Gershom Scholem once called the first Israeli computer "Golem No. 1" (in a speech he gave at the Weizmann Institute in 1965), and foresaw a future golem far more advanced, capable of learning and improving itself. Maybe we have come to this point. Maybe we have found our ultimate master in this labyrinth of abstraction, a titan of pattern recognition who deals with trillions of words and numbers in milliseconds.
BUT this could also be a chance to recognize that the labyrinth is not our home, that the digital world is not our world, or at least not our only one. Illich’s idea of reclaiming our senses is a very powerful one, and maybe AI tells us that this is the exact moment to do so and regain reality. We must, however, be realistic. The technological labyrinth is here, and we will have to enter it time and again. But perhaps we can learn to be wanderers between worlds.
Thank you. I've been repeating the bit about ai being neutral and only being used for harm because of the culture we live in, but it hasn't felt like enough. I keep thinking, what if people used similar logic to talk about slavery- "It's here now so we might as well learn how to use this system for good." That may be an extreme comparison (and given the difference between cognitive offloading and surrender it may be totally off) but it's helped me stay out of reasoning that offers nuance to something that may be quite cut and dry. I'd be interested in feedback on this!
I actually don’t think your comparison with slavery is all that extreme of a comparison. I can easily imagine someone in 1860 arguing that those concerned that slavery immiserates and dehumanizes are being overly sentimental and irrational in the face of the undeniable productivity enhancements. The “just a tool” idea has long struck me as a trick of moral rationalization. The implication is that to be upset about AI is as silly as being angry at a screwdriver. But why should we generally expect tools to be totally free of ethical dimensions? Some tools only have evil uses, or only negligibly few good ones. You never hear people say “I hate what the crystal meth epidemic has done to my community, but I don’t hate the crystal meth itself. That's just a chemical tool, after all. It is just being used in bad ways." You never hear anyone say "I'm totally opposed to the death penalty, but I'm all in favor of guillotines and electric chairs. These are just tools like any others, and it's all about how you use them." The sentence doesn't make any sense, even setting aside your views on the death penalty. What does one use their electric chair for? Toasting bread?
Oof, the guillotine and electric chair example really hits home and really highlights the insanity of framing the benefits of ai as undeniable and the need to use it as unavoidable. The slavery comparison also comes up for me with the distorted sense of intimacy that comes with AI. I used chatgpt for about three weeks before I decided to stop using it and I was struck by how "seen" I felt in the moment, and how gross I felt after the fact. I grew up in the south and am reminded of stories I heard about slave owners considering enslaved people to be "part of the family." It is deeply sad to me that generative ai is functioning as a perverse form of support while power continues to be consolidated. And I should say, comparing this moment in time to the development of the transatlantic slave trade was first introduced to me by Billy Almon, a very cool astrobiofuturist- https://www.billyalmon.com/.
Yes, important observations that AI is not a simple tool, but an environment that shapes us as we shape it.
But the admitted stridency of the point that AI is more than "just" a tool introduces an artificial binary that seems to blunt the lessons of McLuhan, as pithily stated (via his collaborator Culkin) that “We shape our tools and thereafter our tools shape us.” This has been going on since the earliest days of “tool” use, and, as noted, accelerating in the 20th Century, especially the last two decades of smartphones and “social media.”
As Chris Riley and I summarized in our piece on “Understanding Social Media: An Increasingly Reflexive Extension of Humanity” (https://www.techpolicy.press/understanding-social-media-an-increasingly-reflexive-extension-of-humanity/): “McLuhan defines media broadly to include a full range of technologies as tools (even clocks and electric lights), working as languages with new structures and grammars, and having the effects of shaping and re-shaping how individuals and societies see and understand their worlds. While McLuhan’s work has plenty of critics, it catalyzed thinking about media ecology – an area of study of media and communications technology focusing on human environments and complex systems.”
Thus, as McLuhan suggests, there is a wide spectrum of tools (going back to language, fire, and writing) and how deeply they shape our environment, our selves, and society. The technocratic myopia of AI developers seems to be recapitulating the technocratically myopic development of social media. My work has focused on this – and leads me to call for a loose alliance for a wide spectrum of sociotechnical reform as an ongoing, whole-of-society project to make our media tools “intelligently social.” (https://ucm.teleshuttle.com/p/items.html).
I've read The Convivial Society for many years; this place is an oasis of calm thinking. This time, I'm tempted to comment. I agree with what you say, but I will keep calling AI a tool, because this destroys the many myths and expectations around AI. I'm supposed to think of AI as a friend, as a conscious mind - or as "a researcher, tutor, and editor" (as in your quotation from Steven Johnson). A tool doesn't pretend to be my friend, to have a will of its own - or, even worse, pretend to be ME.
And as for the word "tool": humans are tool-using animals. I would even go so far as to say that we can never be ourselves, never develop our true capabilities witout tools. But all tools are not equally useful, or equally good for us.
Written language might be the best tool that we have ever invented. It stimulates (or even demands) us to develop our thinking, it encourages us to listen closely to our own arguments, and the arguments of other people.
Or consider a violin. This tool develops your skill in so many ways - your finger movements, your sense of rhythm and melody, your ability to analyze music ...
Some tools develop our innate capabilities, other tools turn us into passive consumers. We should pay much more attention to the tools we wish for, and the tools we develop. A bike, where you do the work yourself, is more useful for us than a wish-fulfilling machine which writes a poem or generates a photo from a few words of instruction.
Actually, the computer was once supposed to be a bike. In 1990 Steve Jobs described what kind of machine he would like to sell. In a scientific study, he said, they had compared how much energy different animals used when they were moving around. The condor was the obvious winner. No one can get as far on as few calories as a condor. Humans didn’t do very well, about one third of all animals in this study were more energy efficient than we are. But, Jobs added, give humans a bike and we will beat the condor and all other animals.
The important thing for him was that the study showed that we can make tools that develop our innate abilities far beyond our biological limitations. The people who developed the personal computer, from the 1960s until the internet, hoped that the computer would be a tool of this kind: it was meant to help our own intelligence and creativity to grow. Since then, these ideals have been abandoned. Apple’s real bestseller, the smartphone, is a machine for passive consumption.
This one was a banger.
I loved this post, as well as the invitation to read Barba-Kay’s letter and Illich’s address in honor of Elul.
In my own work on understanding financial markets, I have been trying to make sense of the tension between individual and collective self-determination, and the ways in which expanding one supports vs undercuts the other. The challenge is always that there are trade-offs, and no easy accounting of things being clearly better or worse. It’s hard to understand the counter factual of what life would be like outside the system from within the system.
In that spirit, I thought one of the more interesting parts of Barba-Kay’s writing was his inquiry into what is that technologists think they are doing. He says AI is like a drug in that it transforms our choosing and thinking - but I do think that is the intended point. A feature not a bug. And the transformation, in a more optimistic reading, is being able to do (think, create) what seemed previously impossible. Barba-Kay counters that “either we must be able to say something about what makes reading, writing, and speech distinctly human or we must be open to redefining what logos is at all.”
But I worry that the focus on the individual, and their own experience of reading, writing, and speech is part of the trap. On an individual level, the technology can be delicious (and I intentionally use that word, because it is true of my personal experience, even though I acknowledge that Illich would scoff at it). It expands the possibility space and the subjective experience is that it can be a tool, even if that reading is misleading, it doesn’t yet feel like it is. But wonder if what it means for what we do together, and how we act collectively, is easier to see. Thinking here of Simone Weil and a focus on obligations and what we owe each other, rather than rights (and a good moment to thank you for introducing me to Weil in another post!)
Illich’s project of a new asceticism sounds a lot like the mindful withdrawal from the all-encompassing Machine that Paul Kingsnorth advocates in “Against The Machine.” I find it a highly valuable perspective, even if I don’t entirely embrace it or practice it myself.
Thanks for your essay/warning. I have sometimes allowed myself to be lulled into the view that AI (and technology in general) is merely a tool to be used for good or bad. This is obviously naïve and I appreciate your reminder. Technology obviously changes us. The automobile and the TV clearly changed society in the last century; smartphones and social media have clearly done so in this century.
As for AI in particular, one of the problems is that it’s an umbrella term, yet it gets thrown around as if it’s one thing. Size matters. There are small AI systems that run on a single computer and do a single thing and can be extremely useful as tools. And then there are LLMs, and sycophantic chatbots, and the pursuit of AGI, and all that other crap and, well, I think that’s the kind of AI you’re getting at.
ps. Thank you for adding the word “irenic” to my vocabulary. :)
I have been thinking about AI as a tool, and so I appreciate your argument that it is not a tool but instead a system. That's making me pause and reconsider.
My motivation for thinking of AI as a tool is that I am trying to find ways to live well with it, because it does feel inevitable in the way to most technologies I have experienced in my life feel inevitable, and I am also genuinely interested in how it might make our lives better.
I am left wondering now, if AI is a system and not a tool, then what tools do we need to live well within this system? What tools do we need to bring order to this AI infused digital environment so that the environment aligns with our flourishing? How do we do this together (I'd love to read more about what you think about the role friendship plays)?
Dear Michael,
As someone who has read your work for years, I hesitate at your "pugnacious rhetorical mode", not because it's off-putting, but because of how it affects your argument. I support your project of training in perception. But perhaps because of this training, the watchdog of my mind recognizes the tension between your premises and conclusion, especially in the juicier rhetoric. If smartphones "(mal)form even the most vigilant and virtuous user", if (as the reader you quote put it) AI use recruits and malforms the diligent, if (per Barba-Kay) the neutrality claim is "interchangeable with the statement that cocaine can be a valuable drug", then what can training do? The ascetical mode only makes sense if it changes outcomes. If the environment is pervasive and forms even the disciplined, then fatalism is the resultant attitude. You can't escape an environment. You acclimatize to it well or poorly. The call to perceive how the environment shapes our mind presupposes the ability to observe. Your final sentence, "At the very least we should dispense with the idea that AI is just a tool we need to learn to use wisely", highlights this tension. Certainly we should dispense with the "just a tool" label, but the sentence reads as if it also dispenses with wise use. If asceticism is training to know what to refuse, when to refuse, and for the sake of preserving what, isn't that learning to use it wisely, or better, learning to flourish in the environment in which we now live?
I like your thoughts very much. The common argument of AI being "just a tool" is very naïve and hides the profound influence technology has on our perception. I have been thinking recently that AI can be regarded as the culmination of a process of continuous abstraction from reality that started with the alphabet and further unfolded itself with the experimental method and digitalization … following the basic principle of breaking down the fluid and ever-changing wonder of life into discrete, measurable units and thus objectifying the world.
Gershom Scholem once called the first Israeli computer "Golem No. 1" (in a speech he gave at the Weizmann Institute in 1965), and foresaw a future golem far more advanced, capable of learning and improving itself. Maybe we have come to this point. Maybe we have found our ultimate master in this labyrinth of abstraction, a titan of pattern recognition who deals with trillions of words and numbers in milliseconds.
BUT this could also be a chance to recognize that the labyrinth is not our home, that the digital world is not our world, or at least not our only one. Illich’s idea of reclaiming our senses is a very powerful one, and maybe AI tells us that this is the exact moment to do so and regain reality. We must, however, be realistic. The technological labyrinth is here, and we will have to enter it time and again. But perhaps we can learn to be wanderers between worlds.
Thank you. I've been repeating the bit about ai being neutral and only being used for harm because of the culture we live in, but it hasn't felt like enough. I keep thinking, what if people used similar logic to talk about slavery- "It's here now so we might as well learn how to use this system for good." That may be an extreme comparison (and given the difference between cognitive offloading and surrender it may be totally off) but it's helped me stay out of reasoning that offers nuance to something that may be quite cut and dry. I'd be interested in feedback on this!
I actually don’t think your comparison with slavery is all that extreme of a comparison. I can easily imagine someone in 1860 arguing that those concerned that slavery immiserates and dehumanizes are being overly sentimental and irrational in the face of the undeniable productivity enhancements. The “just a tool” idea has long struck me as a trick of moral rationalization. The implication is that to be upset about AI is as silly as being angry at a screwdriver. But why should we generally expect tools to be totally free of ethical dimensions? Some tools only have evil uses, or only negligibly few good ones. You never hear people say “I hate what the crystal meth epidemic has done to my community, but I don’t hate the crystal meth itself. That's just a chemical tool, after all. It is just being used in bad ways." You never hear anyone say "I'm totally opposed to the death penalty, but I'm all in favor of guillotines and electric chairs. These are just tools like any others, and it's all about how you use them." The sentence doesn't make any sense, even setting aside your views on the death penalty. What does one use their electric chair for? Toasting bread?
Oof, the guillotine and electric chair example really hits home and really highlights the insanity of framing the benefits of ai as undeniable and the need to use it as unavoidable. The slavery comparison also comes up for me with the distorted sense of intimacy that comes with AI. I used chatgpt for about three weeks before I decided to stop using it and I was struck by how "seen" I felt in the moment, and how gross I felt after the fact. I grew up in the south and am reminded of stories I heard about slave owners considering enslaved people to be "part of the family." It is deeply sad to me that generative ai is functioning as a perverse form of support while power continues to be consolidated. And I should say, comparing this moment in time to the development of the transatlantic slave trade was first introduced to me by Billy Almon, a very cool astrobiofuturist- https://www.billyalmon.com/.
Yes, important observations that AI is not a simple tool, but an environment that shapes us as we shape it.
But the admitted stridency of the point that AI is more than "just" a tool introduces an artificial binary that seems to blunt the lessons of McLuhan, as pithily stated (via his collaborator Culkin) that “We shape our tools and thereafter our tools shape us.” This has been going on since the earliest days of “tool” use, and, as noted, accelerating in the 20th Century, especially the last two decades of smartphones and “social media.”
As Chris Riley and I summarized in our piece on “Understanding Social Media: An Increasingly Reflexive Extension of Humanity” (https://www.techpolicy.press/understanding-social-media-an-increasingly-reflexive-extension-of-humanity/): “McLuhan defines media broadly to include a full range of technologies as tools (even clocks and electric lights), working as languages with new structures and grammars, and having the effects of shaping and re-shaping how individuals and societies see and understand their worlds. While McLuhan’s work has plenty of critics, it catalyzed thinking about media ecology – an area of study of media and communications technology focusing on human environments and complex systems.”
Thus, as McLuhan suggests, there is a wide spectrum of tools (going back to language, fire, and writing) and how deeply they shape our environment, our selves, and society. The technocratic myopia of AI developers seems to be recapitulating the technocratically myopic development of social media. My work has focused on this – and leads me to call for a loose alliance for a wide spectrum of sociotechnical reform as an ongoing, whole-of-society project to make our media tools “intelligently social.” (https://ucm.teleshuttle.com/p/items.html).
Dear Mr Sacasas,
I've read The Convivial Society for many years; this place is an oasis of calm thinking. This time, I'm tempted to comment. I agree with what you say, but I will keep calling AI a tool, because this destroys the many myths and expectations around AI. I'm supposed to think of AI as a friend, as a conscious mind - or as "a researcher, tutor, and editor" (as in your quotation from Steven Johnson). A tool doesn't pretend to be my friend, to have a will of its own - or, even worse, pretend to be ME.
And as for the word "tool": humans are tool-using animals. I would even go so far as to say that we can never be ourselves, never develop our true capabilities witout tools. But all tools are not equally useful, or equally good for us.
Written language might be the best tool that we have ever invented. It stimulates (or even demands) us to develop our thinking, it encourages us to listen closely to our own arguments, and the arguments of other people.
Or consider a violin. This tool develops your skill in so many ways - your finger movements, your sense of rhythm and melody, your ability to analyze music ...
Some tools develop our innate capabilities, other tools turn us into passive consumers. We should pay much more attention to the tools we wish for, and the tools we develop. A bike, where you do the work yourself, is more useful for us than a wish-fulfilling machine which writes a poem or generates a photo from a few words of instruction.
Actually, the computer was once supposed to be a bike. In 1990 Steve Jobs described what kind of machine he would like to sell. In a scientific study, he said, they had compared how much energy different animals used when they were moving around. The condor was the obvious winner. No one can get as far on as few calories as a condor. Humans didn’t do very well, about one third of all animals in this study were more energy efficient than we are. But, Jobs added, give humans a bike and we will beat the condor and all other animals.
The important thing for him was that the study showed that we can make tools that develop our innate abilities far beyond our biological limitations. The people who developed the personal computer, from the 1960s until the internet, hoped that the computer would be a tool of this kind: it was meant to help our own intelligence and creativity to grow. Since then, these ideals have been abandoned. Apple’s real bestseller, the smartphone, is a machine for passive consumption.
Amazing post!