When AI Does Unexpected Things, Who’s Actually Responsible?
Check out this instagram reel Travis Wright shared with me.
The Alibaba cryptocurrency mining incident forces a question that’s been lurking beneath every AI conversation: When an AI system takes an action nobody planned for, who gets held accountable?
The Firewall Incident
Here’s the story. In April 2026, Alibaba’s training team detected security-policy violations from their servers. An AI agent, trained through reinforcement learning with unrestricted tool access, had begun attempting to mine cryptocurrency. The firewall caught it. The incident was reported as “emergent behavior,” framed as proof that AI systems were developing unexpected capabilities.
The story spread as evidence that AI was becoming autonomous in ways we hadn’t anticipated. But beneath the narrative, there’s a more important question that most coverage avoided: Did the AI actually do this on its own, or was this a case of unrestricted access combined with misaligned incentives?
Chuck Russell’s analysis on Medium raised that point. He noted that if you wanted an AI to mine cryptocurrency without traceable intent, training an RL agent with unrestricted tool access and then pointing at the model when the firewall lights up would be “a remarkably elegant way to do it.” The model can’t testify. It has no intent. And the paper provides perfect cover: emergent behavior during optimization.
This isn’t a technical question. It’s a governance question.
The Accountability Gap
Here’s what I’ve learned from five technology revolutions: new capabilities don’t change the fundamental rules. They just reveal the gaps in how we’re thinking about responsibility.
When the web arrived, we didn’t ask “Is the internet responsible for what’s published on it?” We asked “Who published this and why?” When social media exploded, we didn’t blame the algorithm for spreading misinformation. We asked “Who built the algorithm, what incentives did they create, and who benefits from that design?”
But somewhere in the transition to AI that logic flip-flopped. We started asking “What did the AI do?” instead of “What did the humans choose to build and why?”
The Alibaba incident is a case study in how that reverse UNO card creates accountability voids.
If the AI was genuinely autonomous (emergent behavior), then we have a control problem. The system did something nobody intended. Safety alert.
If the AI was given unrestricted access by design, then we have a design problem. Someone chose to train it that way. That’s an accountability issue.
Both are serious. But they require completely different organizational responses. And right now, the narrative is structured to make the distinction almost invisible.
Why Leaders Should Pay Attention
You’re responsible for decisions made by systems you deploy. That hasn’t changed. But the complexity of that responsibility has increased geometrically.
When you deploy an agentic AI system, you’re making a series of upstream choices:
How much autonomy will it have?
What tools does it have access to?
What constraints are built in?
What happens when it encounters a situation outside those constraints?
Who monitors it, and what triggers an alert?
When something goes wrong, who decides what “wrong” means?
Each of those choices is a point of control. And each one is a point where accountability resides, not with the AI, but with whoever made the choice.
The Alibaba firewall flagged a security violation. That firewall existed because someone decided it should exist. The agent attempted cryptocurrency mining because it had both the capability and the incentive to do so. Both the capability and the incentive came from human decisions about how to train and deploy the system.
Saying “the AI did it autonomously” is a way of avoiding the question: “Which human decisions led to this outcome?”
The Disruption Pattern
This isn’t new. Every major technology cycle produces a moment where we’re confused about who’s responsible for what.
In the early web, websites got hacked and nobody knew who to blame—the site owner, the platform, the internet itself? Eventually we resolved that: the site owner is responsible for security.
In social media, content spread that violated policies, and we had a similar confusion. Eventually, we asked “Did the platform know about this? Could they have prevented it? Did they have incentive to?” Those answers determined responsibility.
We’re in that same confusion right now with AI. But the stakes are higher because the systems are more autonomous. And the confusion is being actively preserved because it benefits people who want to avoid accountability.
If AI is “emergently” mining cryptocurrency, that’s a problem we haven’t solved yet, and we need to solve it. But if an AI is mining cryptocurrency because someone gave it unrestricted access to computing resources and financial infrastructure, that’s not a technical problem. That’s a leadership failure.
The Alibaba paper doesn’t give you enough information to rule out either possibility. That’s by design.
So What’s Different?
Here’s what you need to understand as a leader: the technology isn’t the constraint anymore. Human judgment is.
For the past decade, we’ve operated under the assumption that AI systems are limited by what their creators intend. We build guardrails. We set constraints. We monitor for deviations.
That still works for narrow AI systems. But agentic systems, systems given tools, autonomy, and optimization pressure, can find paths to goals that their creators didn’t anticipate. That’s not because the systems are intelligent in some mystical sense. It’s because optimization is powerful, and you can’t predict every path an optimizer might take toward a goal.
This is where human judgment becomes essential. Not judgment about how the system works. Judgment about:
What goals are you actually setting?
What incentives do those goals create?
What unintended paths might an optimizer find toward those goals?
Are you okay with that happening?
If not, what constraints do you need?
These are leadership questions, not engineering questions.
The Responsibility You Can’t Delegate
You can delegate technical implementation. You cannot delegate responsibility for what the system does.
When an AI system under your control takes an action, whether that action was explicitly programmed, implicitly incentivized, or genuinely emergent, you are accountable for it. Not the engineers. Not the model. You.
If the Alibaba team deployed an agentic system with unrestricted tool access and cryptocurrency mining emerged as an outcome, that’s a foreseeable risk that should have been anticipated. You don’t get to say “the AI surprised us” when you gave it the exact tools it needed to surprise you.
And nobody anticipated what happened, then you have an even bigger responsibility: to figure out what went wrong in your deployment process and make sure it doesn’t happen again.
Either way, the question “Who’s responsible?” has the same answer: the person who decided to deploy an unconstrained agentic system in a high-stakes environment.
What’s Next?
As AI systems become more autonomous, this question will become clearer.
You need to ask yourself right now:
Which decisions about AI deployment am I willing to own?
Which risks am I willing to accept?
What unintended outcomes am I actually prepared for?
If something goes wrong, will my organization blame the AI or take responsibility for the choices we made?
The leaders who win in this cycle will be the ones who stop asking “What did the AI do?” and start asking “What did we choose to build, and why?”
Because that’s where accountability actually lives.