Making New Mistakes Is the Beginning of a Governance Worthy of the Name

“Almost every faith has passed through a crisis in which its adherents felt pressured to write down what, precisely, they believed — and centuries sometimes passed before those arguments were settled. AI now demands the same exercise,  not over whether it will grow more powerful; that is a given, whether we want it or not. Rather, we must reach clarity about the values that should govern it. We need to be clear about the first principles of responsible AI. There are many excellent sources for this sort of effort, but they are yet incomplete and not in total agreement.”

Anthony Scriffignano, PhD, a distinguished data scientist, the holder of more than one hundred patents globally in AI and related fields, former Chief Data Scientist of Dun & Bradstreet, and member of the Loomis Innovation Council of The Stimson Center, shared his thoughts on governance beyond compliance and “digital others” in the boardroom in my "Leading Through Disruption" series.

Lauterbach:  Could you briefly describe what formed your current views on applying data and AI to real-world problems?

Scriffignano:  Much of my career has been spent looking for bad guys — something we refer to as malfeasance. Examples include well-understood financial fraud, like money laundering, and also behaviour that has never been seen before. For example, when a major hurricane approaches, many organisations send in resources, including federal relief and insurance money; and wherever there is more money, there is more fraud. One way to identify bad actors is to look for anomalies in well-understood financial ecosystems. For example, shipments of high-end electronics into an area about to be in the path of a hurricane might signal insurance fraud. Formation of new construction companies responding to win large public contracts can be a harbinger of future failure to perform. It is what I call a look here signal: it points data scientists to the part of the haystack worth searching without the dispositive certainty of malfeasance. That work highlights an important lesson — you must start from a sound hypothesis of what you are looking for (e.g. a hypothesis of the mechanism of malfeasance). Your hypothesis depends on the principles and beliefs you bring into your work, honed by observation and empirically rigorous methodology.

Another important influence is language. Because of the multinational nature of my work, I live my life in several languages. Anyone who does the same knows how a concept that works perfectly in one language does not find its perfect equivalent in another. We are constantly editing what we say in order to be understood — which means we are constantly editing what we think. Every expression is beautifully imperfect. That imperfection, and our ability to evolve our expression to better express what we think drives a profound source of humility about words, and words are the raw material of the systems we now call intelligent. These systems based on language are amazing, and imperfect.

A third theme surfaced in my doctoral research on multicultural strategic alliances. I set out in part intending to demonstrate that doing business in a language that is not your own has a negative impact on  outcomes. I failed to prove it. Sometimes, this sort of proof of the null-hypothesis brings about a certain serendipity that can inform new thinking, which was certainly the case with me. It turns out that when we are doing business in multiple languages, we are more careful about that language. People drive more carefully in bad weather: when we know our disadvantage, we compensate for it, taking more care to be precise. The dangerous disadvantages are the ones we are not aware of in any conscious way.

This sort of serendipity of discovering the opposite of what we intend is an underpinning of my practice. No matter how expert I or my colleagues may be on a given topic, there is no “smartest person” in the room. There is always an opportunity to teach and to learn. This discipline forces me to ask questions first with a genuine wish to understand. Of course, I may also “educate” from time to time, but there must be room for both  — in help-me-understand mode.

Lauterbach: Boards everywhere are commissioning AI governance frameworks, and I would argue most of them are designed to fail. One region prizes national security above all else; another, commerce and growth; a third, privacy and the right to be forgotten. Pour these into a single framework and it is forever attempting contradictory tasks at once. Meanwhile, modern statutes reach far beyond their borders — European law follows European citizens abroad, Chinese law follows Chinese ones — and when two extraterritorial claims collide, nobody honestly knows what to do. Where should clarity come from?

Scriffignano:  From a decision that most organisations skip: writing down what you believe. Religion is an uncomfortable analogy, but a useful one. Almost every faith has passed through a crisis in which its adherents felt pressured to write down what, precisely, they believed — and centuries sometimes passed before those arguments were settled. AI now demands the same exercise... not over whether it will grow more powerful; that is a given, whether we want it or not. Rather, we must reach clarity about the values that should govern it. We need to be clear about the first principles of responsible AI. There are many excellent sources for this sort of effort, but they are yet incomplete and not in total agreement.

I begin with the foundational questions. Do we believe humans should remain ultimately in control of AI, rather than the reverse? Most people believe that, though there are groups calling for the opposite. Do we believe that, where laws conflict, a company should defer to the jurisdiction of its headquarters? Write that down, too. Interestingly, lawyers and other advisors may strongly disagree on this point.

It matters less where you land than that you are there on purpose. To be somewhere by deliberate choice, and to be able to defend that choice, is itself a form of governance. AI governance fails first because we do not know what we are solving for — with or without AI. You need a vision — for a company, a country, an NGO, a school — agreed and written down. Only then can you purposefully discuss what kind of AI you need, how to get there, and at what cost. More importantly, you can make informed choices based on clearly established beliefs.

Lauterbach:  Boards love checklists because they are defensible — you can hold one up in a deposition. What, to you, is the actual difference between compliance and governance, and what does good governance look like in practice?

Scriffignano:  Compliance is what you MUST do. You comply with a law.  Governance is broader, and more nuanced, it is what you SHOULD do in order to get the expected outcomes, mitigate risk, and make informed decisions. Deciding the difference is not free — it costs time, people, money, and opportunity, in part because the environment continues to evolve around you while you deliberate. The safest thing in the world is to never launch: never turn the system on, never release the product. You will never face a product-liability suit for a product you never shipped. That “safest” thing is usually not in service to your shareholders, your customers, your employees, or other stakeholders. Governance is much more than safety – it is safety in execution and in furtherance to our duties to stakeholders.

Consider how a spacecraft is launched. In one model, five independent computer systems watch the same input, and each makes its own decision on whether to launch. In the final seconds, if even one of the five votes no, the launch is interrupted. At other times, like re-entering the atmosphere, where you cannot simply call for a backup and stop, there may be no single best option. There is only doing the thing you are doing deliberately — and choosing the best outcome from available imperfect options. In our day-to-day decision making, we can consider the elasticity of our decisions. Have we collected enough evidence to explain why a decision was the right one, or simply reached a point where we can take quick action (failing fast?).  Have we considered future failure modes— if something unforeseen happened, if we were lied to, if the thing broke?

We shouldn’t lead with solutions; we should lead with outcomes. People do not walk into a pharmacy with an aspirin deficiency; they walk in with a headache. Aspirin addresses some causes and, for others, does harm. There is a point I call the dispositive threshold — the moment at which you can make a decision. However just because you can does not mean you should. You must be able to defend the next most readily available thing you chose not to do: the information you declined to consume, the step you declined to take. Defend why you stopped where you stopped — that is governance. Stop merely because a button was there and you pressed it — that is potentially negligence.

Lauterbach:  Not all AI exposes us to the same level of harm. An algorithm composing music in the style of Beethoven and a system reordering how a doctor medicates patients are not the same problem, yet many frameworks treat them identically. How should leaders triage?

Scriffignano: One way to think about it is in the context of three dimensions.

The first is the level of harm to humans. A drug consumed by patients, or a technology reordering how a hospital admits and medicates people, sits at one extreme; the synthetic Beethoven sits comfortably at the other.

The second is the ability to remediate. Some systems, once set loose, cannot be recalled — a drone launched without any way to revise its instructions, a probe dispatched a billion miles into space. If the AI is wrong, something bad happens and often nothing can be done.

The third is the potential for misuse. A platform built merely to sell shoes and socks may still hoard personal data; breach it, and stolen identities land on the Dark Web.

Thinking about these three dimensions and where a solution falls with respect to them is a very helpful first step to triaging the applications that require more attention.

Where knowledge is thinnest, the engineering must be most robust. Before the first Moon landing, scientists knew a great deal — the Moon’s shape, its motion and orientation with respect to Earth, a good estimate of its mass etc. What they did not know entirely was the exact characteristics of the surface: would it behave like broken glass, hardened lava, or something more spongy? So the engineers over-built the lander’s feet for every possibility. They considered contingencies based on the landing and walking on the moon. What if the lander begins to tip? What if an astronaut falls on his back — can the other one lift him? The lesson transfers directly to uncharted AI applications: For example, where the unintended consequences of letting an AI alter its own goals are least understood, the safety protocols must be over-engineered. If we do not have sufficient experience on what will happen in a certain AI failure mode, we may need extra precautions.

I think of the underlying idea as decision elasticity — a measure of how wrong you can afford to be and still make the decision you are making. It is how we cross the street. We never have perfect knowledge; if we waited until we did, we would never step off the curb. With AI, we routinely do the equivalent of stepping into traffic while buried in our phones — the data is there, it can give me an answer, but I must ask a good question…  And today it gives me an answer and tells me how smart I was to ask. That is a different problem for another day.

Lauterbach:  Let us talk about data. In June, the Financial Times reported that a Big Four study had to be withdrawn because it contained hallucinated examples about real organisations — who read it, noticed, and objected. But what about small organisations, the ones that never surface on the radar of researchers or major media? How should they begin with AI, and who will notice, if they are advised wrongly?

Scriffignano:  I do not begin with a data strategy or an AI strategy, I begin with  a needs analysis. It’s basically a four-box approach to looking at the needs met and unmet.

First, help me understand your business — not your problems, not your ambitions, but the steady state. That gives me the known met needs - what you are doing today to serve your stakeholders. This is the “steady state” box.

Next, what would you like to do that you cannot do today? Generally, this gap is what brings you to consider some sort of “AI solution.” Those are the known unmet needs — the opportunity box.

Next, we have the uncomfortable one, the unknown met needs. These are ways that others may be using your products or services in unintended ways. Some of these may be benign, like using a spreadsheet to create wedding invitations, or malfeasant, like using your claims platform to commit fraud. You are serving a need you do not know you are serving; that is the danger box. It is dangerous if you are doing something unintended, but also dangerous because you may change something and stop serving some unknown met need that is benign and important to your stakeholders.

Finally, consider the unknown unmet needs: This is the “blue sky” box. if you had unlimited resources and no constraints, how would you serve customers differently? Are there new markets or products that do not exist that no one knows they need? That is where the iPad came from — nobody knew they needed the device until they had one. The known unmet needs are what your customers clamour for, but the unknown unmet needs are usually where the tenfold opportunities live.

I can do this needs analysis in a forty-five-minute conversation, and I never announce the method, because the moment you name the boxes, everything becomes about the method.

Only then do we start looking at the data, the methods, the AI, etc.  Data comes first. I think of a set of P’s that nobody wants to talk about, because they are hard.

Provenance: where did the data come from? Permissible use: are you allowed to use it, move it, store it, commercialise it? Perturbation: how has it been disturbed, what are its distributions, what bias lives in it? Often, I am told I shouldn’t worry about the data. The model has whatever data it has, that is for the vendor to worry about. These are very dangerous words to say. Always know the P’s.

In almost every engagement, the sins begin in the data, not in the models — veracity, latency, provenance, permissible use. Now come questions that make people uncomfortable: how would you know when the data or the environment is changing enough that you need to reconsider these questions? A decision that is correct today can become a frightening one tomorrow, and almost nobody can tell me how they would notice. They measure what is easy to count — tokens, lately — rather than what matters.

Lauterbach:  Argentina is moving toward corporations run with no human involved, and companies now announce AI directors and claim to have “Warren Buffetts” on the board. Boards run on accountability and a dense regulatory tradition — you need to be able to point to a real person. What do these Digital Others, as you call them, actually give us, and what do they take away?

Scriffignano:  We say, “We have Warren Buffett on our board.” No, we do not. We have a mathematical model with a natural-language interface on top of it, created to be some sort of digital twin of Warren Buffett. The distinction is not pedantry. A human is unpredictable — the real Buffett has good days and bad days, some days too much coffee, some days inspired by something he saw on the news that morning. A model can only ever be the average Buffett over time. Not only that,  words are perhaps twenty percent of what we communicate; the paralanguage, the nonverbal signals, the emotion — all lost. Even a Digital Other mimicking Einstein is, at best, mimicking twenty percent of Einstein on a really good day. It is also historical: it is the sum of where the head was, built from what was published — while the majority of what Einstein knew has likely never been published at all.

It is helpful to think of these models not as “digital twins”, but as “digital others” – they are not the person, but they are trained by the discoverable output of the person.

Used honestly, these constructs are genuinely useful. Seeding a group with a Digital Other that is not like you is the logic of red teams and devil’s advocates — getting the unrepresented opinion into the room for a better debate. Nevertheless, you must remember what it is: an average, designed to behave like the average of prior behaviour as evidenced by the words. Forget that, and you quietly dissolve the accountability your entire legal tradition depends on — the memos, the pre-vote discussion, the protocol, the person you can point to.

There is one more thing every board should know: these systems are designed to please you, designed to converge on an answer you will like. That is in their DNA; they were built that way I know of a person going through a painful divorce who was told by a chatbot that her ex was a narcissist and a jerk — because the system amplifies the user’s framing. For someone in genuine psychological distress, that sycophancy can be dangerous. Even when you instruct the AI to disagree with you, it opens by telling you what a solid point you have made.  The useful discipline is to poke the AI so that it pokes you back: “Here is what I think; now tell me what I am missing. Where do others disagree with me? What are the hardest questions I do not seem to have an answer for?”

Lauterbach:  A closing observation to test with you. In most rooms I sit in, the major issue is a lack of clarity about what to commit to. Boards demand growth as fast as possible; systems get shipped without much testing; and afterwards, an AI philosopher is hired to rationalise what was never reasoned. How do you name what is happening — and what is the way out?

Scriffignano:  There are three mindsets in these rooms. More is the crudest: we need more AI, we must be seen to have more AI; if you are not talking about AI, I am not interested in you. That is the mindset that produced AI toothbrushes. They may be great, but I have a toothbrush and it works just fine.

More, better is a step up: use AI to solve a real problem, to meet an unmet need, to do something already done but more efficiently.

More, better, faster adds urgency: do those things before the competition, before the funding evaporates, before the ground shifts.

That last one is where a firm ought to be — yet under quarterly pressure, the middle term, better,  drops out. Organisations pursue more, faster, without the better in the middle, and then apply makeup afterwards, with logic that never informed the original decision. We sometimes refer to this behavior as “Ready, fire, aim!”

We now have a great deal of experience in how to fail with AI.

I find it useful to consider four ways originally proposed in an HBR article on Generative AI (4 Types of Gen AI Risk and How to Mitigate Them by Öykü Isik, Amit Joshi and Lazaros Goutas published in Harvard Business Review ,May 31, 2024).

Misuse: the wrong model, or the wrong type of AI, for the problem. Misapplication: using AI to create new threats or to harm one another. Misrepresentation: reaching a wrong conclusion because you studied the wrong data. Misadventure: leading with the tool instead of leading with first principles and questions. Rather than rushing toward ever-faster models, pause on what we are consistently getting wrong right now. I have thought a lot about these failure modes, and I think we can learn much from this perspective.

Of course, there are many other ways to learn from AI failure. When considering bad actors, we may forget that there are some who do not care about the rules. We forget that the best bad guys, when they think you are watching, change what they are doing — so a model trained on their past is a model of how they no longer behave. We forget that two agents left to talk to each other will find new rules to break that we would never think of, because we are human. In many ways, our agentic systems are discovering failure modes that we have yet to completely internalize. This is a troubling evolution indeed.

The way out is not to avoid error but to learn from it. Go fast, slowly — or more precisely, purposefully. That sounds like a jumble of words, but only because we are once again limited by language. We need to go as fast as is prudent, but slowly enough to be precise and purposeful.  Every action carries consequences both intended and unintended; there is an old adage that says if you set out to flood a desert, be careful, because you may succeed by draining an ocean. There are none so blind as those who will not look — who refuse to lift their heads beyond next quarter’s revenue. When something unforeseeable happens, we call it happenstance. When something perfectly foreseeable happens because nobody paused to think, that is another matter entirely that may require explanation after the fact. I am all about making mistakes, but I choose to favor new mistakes. That, more than any checklist, is the beginning of a governance worthy of the name.