AI Right Now Is Watching Us

“We can think: “Now that the AI handles the routine cognitive work, I am going to attempt something greater.” The tool can be liberating — but only if the humans consciously choose to use their freed capacity for something harder. That requires a certain kind of leadership: leaders who model intellectual ambition, who demonstrate that AI fluency and rigorous independent thought are not in tension.”

Dr. Craig A. Kaplan, founder of Superintelligence.com and a former researcher at Carnegie Mellon University, where he co-authored papers with the Nobel laureate and Turing Award winner, Dr. Herbert A. Simon, shared his thoughts on knowledge and economic development in the age of AI in my "Leading Through Disruption" series.

Lauterbach: Craig, you have been in AI since the 1980s — through winters and resurgences, working with symbolic systems and artificial neural networks. When you look at the boardrooms and leadership teams navigating AI today, what is the single most dangerous strategic mistake you observe?

Kaplan: Doing safety after the fact. Every major organization is building first and asking safety questions later. My core motivation — the reason I have spent years working on the architecture of Advanced AI Systems — is that it is actually possible to design safety in, from the ground up. Not as a patch, not as a policy layer, but as a structural property of how the system works. Interestingly, the safer design can also be the more powerful and profitable design. So unlike many in the field, I do not see an inherent conflict between safety and more powerful AI models.

The dominant approach right now is to train a giant monolithic model, a black box using machine learning, scale it, and hope to understand it afterwards. That is the strategic mistake. Nobody — not even the most brilliant researcher alive — can look at a trillion parameters and understand what the model has really learned or how it will behave. The method itself creates the opacity. And now these models are getting smarter than us across multiple dimensions.

Lauterbach: The “climbing taller trees to reach the moon” problem.

Kaplan: Exactly. But there is an alternative. We can build advanced AI not as one giant opaque brain, but as a network of personalized AI agents. Each agent should carry not only expertise but also values and ethics. And all of these agents interact on a transparent, auditable network to create an overall collective intelligence that is far smarter than any single opaque model. I built systems like this for Wall Street. My company architected collective intelligence systems to compete against the smartest traders in the world, with real money, making billions of dollars of trades. Collective intelligence won. The same approach can lead to SuperIntelligence. The collective intelligence approach is faster, safer, and more beneficial for humanity than the arms race to build bigger opaque models that everyone else is running.

Lauterbach: Let us talk strategy more concretely. The leaders I mentor — CXOs of banks, healthcare institutions, industrial groups — face what I call the vintage IT problem. Some have COBOL applications from the 1970s still running core processes. The capability curves you cite, doubling every few months, call for a modernized IT infrastructure. What is your strategic advice for incumbents?

Kaplan: They have a narrow window to act — and the window is closing faster than most people appreciate. A Stanford researcher recently cited work from Metr.org showing that the length of task that an AI model can complete autonomously is doubling every few months. In coding specifically, capability is doubling every seventy days. Even if this is the steep part of an S-curve and natural limits eventually slow things down, organizations cannot wait. But here is what I tell leaders: you do not have to boil the ocean. The collective intelligence approach means you plug existing systems — including those monolithic frontier models — into a network. Even a legacy institution can begin with personalized AI agents customized with the knowledge and values of its best people. That is not rip-and-replace. This is a strategy based on technology agents and humans, working together.

Lauterbach: I observe something troubling: in the rush to appear AI-forward, leaders are outsourcing their critical thinking to ChatGPT and its cousins. Ninety percent of LinkedIn posts are written by AI. Executives accept the polished output, the nicely formatted answer, without questioning it. Having spent thirty years across several industries, and having studied computational linguistics as my first master's degree, gives me knowledge of these domains — and so I notice this fluent nonsense. Most executives do not take the time to understand how AI operates, so they use the tool without reflecting on the fact that AI's intelligence has nothing to do with their own.

Kaplan: With AI becoming more intelligent so rapidly, we have to think about the role of humans in the future. I think about it this way. In the future, AI will be more of the brain, and humans will be more of the heart. Reid Hoffman and Reed Hastings made this point recently, and I agree. When you personalize an AI agent with your expertise, you must not leave out your values and your aesthetic sensibilities -- your heart. Because AI cannot come up with values on its own. Reason, as Herbert Simon wrote in Reason in Human Affairs, is wholly instrumental. It can tell you how to get somewhere. It cannot tell you where to go. The premise — what is good, what is worth pursuing — cannot be logically derived. That has to come from us. Similarly, the values, vision, and high level goals of an organization have to come its leaders and be communicated clearly to AI.

Lauterbach: This brings me to what I genuinely believe is the defining leadership challenge of our era: preserving human agency while deploying increasingly capable AI. I am not just talking about job displacement. I am talking about something deeper — the degradation of human thinking itself. If AI writes your emails, structures your arguments, drafts your strategy documents, what happens to the cognitive muscles that are not being exercised? My teenage daughter is currently playing Debussy's L'Isle Joyeuse. She started piano at the age of three and understands that one has to practise études before moving to Debussy. Going through difficulties is imperative to achieving what is perceived as easy and fun by outsiders.

Kaplan: It is a real danger, and I have watched it happen to people I know. “I'll just let the AI do it — it writes better than I can.” And gradually they stop thinking as much for themselves. But I do not think it has to go that way. Humans can respond by raising their ambitions. We can think: “Now that the AI handles the routine cognitive work, I am going to attempt something greater.” The tool can be liberating — but only if the humans consciously choose to use their freed capacity for something harder. That requires a certain kind of leadership: leaders who model intellectual ambition, who demonstrate that AI fluency and rigorous independent thought are not in tension. Jensen Huang, CEO of Nvidia, is one leader who has said that in the age of AI we should raise our ambitions.

Lauterbach: You worked with Herbert Simon and Allen Newell — two of the true founders of the AI field. Simon's Nobel was in economics, focused on the recognition that humans do not make decisions like rational operating machines. What does that intellectual tradition tell us about how leaders should think about AI being a force for augmentation instead of a perpetual efficiency engine?

Kaplan: Simon got the Nobel Prize, in part, for his idea of bounded rationality. Bounded rationality says that human behavior is not always rational but instead much of it follows from the cognitive limits that all humans have. For example, because human working memory only holds seven plus or minus two items, we don’t typically find optimal solutions to problems. Instead we find good enough solutions. Simon called this “satificing” and it is a direct consequence of our cognitive limitations. Now, compare human’s limited memory capacity with that of an AI that holds millions of items in its huge working memory or context window. That AI can come up with solutions humans have never thought of, because those solutions simply required too much memory or too much computational power compared to what our human brains can produce.

Talking about augmentation or creativity, sometimes you hear the argument that AI may have greater computational power than humans but that doesn’t mean AI can be creative. I disagree.

The Logic Theorist — the very first AI program, 1956 — produced a mathematical proof that was not in Bertrand Russell's Principia Mathematica. Simon sent it to Russell. Russell wrote back: “I wish I had thought of that proof.” Creativity from AI is not a future prospect. It has been demonstrated for seventy years. The question for leaders is: are you building organizations that can receive and act on that kind of creative input? Or are you using AI only to produce faster versions of what you already do?

Lauterbach: That is a deeply uncomfortable question for most organizations. And it connects to the swarm behavior concern I raise repeatedly. I am more afraid of human swarm intelligence than of autonomous AI. Swarms lack a brain. Their behavior is not about leading with purpose and courage — it is about following. The ant does not ask why. If we equip people with AI that is faster, more articulate, and more confident than they are, the path of least resistance is to follow it — not to lead with it and contradict its direction, if necessary. And a society of followers, however efficiently coordinated, is not a civilization. It is a flock. The question I keep returning to is this: how do we build AI systems, and AI cultures, that reward the person who asks why — not just the one who moves fastest in the direction everyone else is already going?

Our schools are not preparing students for the future with AI. Teachers say: “Don't use it.” Students respond: “We want to understand it, we want to discuss what is ethical, even if we haven’t found all the answers yet.”

Kaplan: I know from building those collective intelligence systems on Wall Street that most of the millions of participants believed their individual contribution was too small to count. They would tell us that when we interviewed them. Yet, together, they beat the market. The most important thing that leaders and all of us need to understand is this: AI right now is watching us. All of these systems are trained on human data. Every email, every post, every research paper is being absorbed by AI. Whenever we behave, we are expressing our values— whether we intend to or not. AI is currently like a very gifted child who watches everything we humans do or say. In teaching children, the early years matter most. And for AI, the early years are now. We are not, or should not be, followers of AI. We should realize that we are teachers and we need to teach our AI children well!

Lauterbach: Final question, and it is the one that keeps me awake. Geoffrey Hinton estimates perhaps a 10 to 20% probability of human extinction from AI — what people in the field call p(doom). You are fundamentally optimistic. Why, and what would have to be true for that optimism to be justified?

Kaplan: I’ve read that on average, over the past decade, war and violent conflict have accounted for mortality of less than 0.003% of the global population per year. So even though every life lost to war is a tragedy, if we had a collective AI intelligence that operated the way human societies operate — transparent actions, auditable decisions, democratic checks and balances, broadly representative values — we might expect something in that range, not 10 to 20%. That would be more than a thousand-fold improvement compared to current estimates of p(doom). I think Hinton would be happy with that. I would be very happy with that level of risk reduction.

Three things must happen for this scenario to unfold. First: transparency by design. Superintelligent AI systems must be designed so that their reasoning is visible and auditable. They can’t continue to operate as black boxes that produce outputs nobody can interrogate. Second: humans in the loop. The most powerful and profitable system designs need to include human input as part of their fundamental operation. Third: values embedded through personalization. If each of us personalize our AI agents with our individual values, then we will have many billions of AIs representing the actual diversity of the 8.3 billion people on the planet. This approach is far more representative, and much safer, than telling AI to follow a constitution drafted by a few Silicon Valley elites. Consider also that if values are distributed across a network of billions of AI agents it is much harder for a bad actor to override those values compared to just changing the values in a centralized constitution that all AIs must obey.

These three things are absolutely achievable! They require will, good engineering, and humans who serve as good role models. But the window to act is right now, while the systems are still being designed by humans and AI’s values are still being formed. As Herbert Simon said: reason can tell you how to get there. But it is up to us to decide where we are going. That decision is still ours to make -- ideally democratically.

Lauterbach: To quote Winston Churchill: democracy is the worst system with the exception of all others. Craig, thank you — for the depth, the honesty, and for making the urgency feel like possibility rather than catastrophe.