The autonomy paradox: How much rope should we give AI?

Seven things we learnt from the roundtable about governing Agents

Our roundtable brought together four Crisil practitioners - a risk management industry veteran, a model risk senior practitioner, an agentic AI architect and an AI governance specialist for a live roundtable. Several hundred risk and AI professionals from financial services firms joined in.

The brief was deliberately uncomfortable: how much autonomy can you safely give an AI agent and where should the limit be set?

Partway through the session, the moderator stopped the discussion and asked the panel an upfront question:

“When was the last time any of you overrode a system that had been right 99 times before?”

Nobody answered quickly. And that pause revealed the problem. It is not whether AI is accurate or useful, but whether the safeguard of a human, watching and ready to intervene, is a control at all when the machine is usually right.

Here are the key takeaways from the discussion:

• The thing we are governing has changed from an answer to an action
  
  For 30 years, model risk management rested on the assumption that the model produces an output and a human decides what to do with it. An autonomous agent breaks that causal chain. It interprets an objective, retrieves its own evidence, selects its own tools, decides, executes and then decides what to do next.
  
  The question is no longer “Can I trust this output?” but “How much authority am I prepared to delegate and under what conditions can I take it back?”
  
  Most control frameworks in use today were never designed to answer that.
  
  
• A guardrail written in a prompt is not a control, but a request
  
  Many so-called AI guardrails currently in production are sentences: “Do not access that system”, “Stay within the risk appetite” or “Do not take irreversible actions”.
  
  These are not controls. They are polite instructions to a probabilistic system. And one cannot put a request in a control inventory and call the risk mitigated. A real control is enforced at the tool and permission layer, where the agent is structurally incapable of crossing the line, not just discouraged from doing so.
  
  
• Human oversight gets weaker just as the model gets better
  
  Every other control in the inventory holds its strength or degrades for reasons you can monitor. The human-in-the-loop approach does the opposite. The more reliable the system becomes, the less genuine scrutiny the reviewer applies. People learn to assent. Automation bias and reviewer fatigue grow in direct proportion to the model’s track record.
  
  As one panellist put it, “A human in the loop who has never said no is not a control, it is a signature.”
  
  The fix is not more vigilance. It is showing the reviewer something worth challenging - what happened during the run, where confidence dropped, which unusual source was used and where a boundary was approached rather than a finished, confident answer and an approve button.
  
  
• Reversibility is a technical property, consequence is not
  
  This produced the sharpest disagreement of the session, and it was not resolved.
  • One view: Index autonomy to undo-ability. Give an agent wide latitude wherever its actions can be reversed and tighten sharply as irreversibility rises
  • The counterview: In a bank, almost nothing is as reversible as the engineer thinks. You can cancel a trade, but you cannot un-send a client communication, un-file a regulatory report or make a leaked exposure unseen
  • The practical conclusion for anyone tiering agentic workflows: Classify by consequence, not by technical recoverability. The transaction may reverse, but the reputational and regulatory effects will not
     
• Risk accumulates while the agent is still working
  
  While risk frameworks measure exposure as a level, for an autonomous agent it behaves more like a budget that depletes. An agent may begin with approved data and low-risk actions. Then it accesses more sources, invokes more tools, encounters ambiguity and moves closer to a consequential decision. Nothing has failed, but its risk state is no longer what it was at the first step.
  
  The reference point raised in the room was Knight Capital, which lost over $400 million in roughly 45 minutes. Not because of one catastrophic decision, but due to accumulated autonomy running without a forced checkpoint.
  
  The TAR measure asks not just how much autonomy an agent has, but how long it can safely remain autonomous before intervention becomes necessary.
  
  
• Validation stops being a gate and starts designing the guardrails
  
  If the system’s behaviour is determined at runtime, a point-in-time sign-off cannot be the whole control. The shift the panel converged on was that static validation becomes the design-time gate and its output becomes the specification for the controls that operate in production. Constraints move out of documents and into executable architecture.
  
  The architecture should be designed to control the agent’s state transitions. A CTS helps define when an agent can continue, when its authority should be constrained, when it should escalate and critically, when control should return to a human.
  
  Telemetry: What the agent accessed, which tools it called, where it retried, where it approached a limit becomes governance evidence rather than IT logging.
  Before approving an agentic workflow, validation must ask if it is TRACE compliant.
  
  
  • Transparency: Can we reconstruct the reasoning and material evidence behind the action?
  • Reversibility: Can the action and its consequential effects be undone or contained?
  • Auditability: Can we prove afterwards what happened exactly, in what sequence, and with which data and tools?
  • Constraint-awareness: Are the limits being enforced by the system, rather than merely stated in a prompt or a procedure?
  • Escalation-readiness: When the agent reaches the edge of its competence or authority, does it hand over control cleanly rather than improvise?
     
• Autonomy can be delegated, but accountability cannot
  
  The closing point, and the one that resonated the most, was that responsibility for an agentic system is distributed among developers, model owners, validators, technology teams and business users. But distributed responsibility must not result in diluted accountability.
  
  Someone decided this agent was sufficiently understood, controlled and bounded to be given the authority to act. That decision needs a named human owner. And that person probably should not be the validator. Validation provides independent challenge, and it does not own the business decision to deploy. Confusing assurance with ownership is how accountability evaporates. Architecture, telemetry and audit trails give you evidence of accountability, but do not replace it. If nobody’s name is next to the autonomy level, the autonomy level is not real.
  
  Three questions worth taking to your next AI policy meeting
  
  
  • How many of the AI guardrails in your control inventory are enforced in the architecture and how many are sentences in a prompt?
  • When your reviewer approves an AI output, what are they looking at: the finished answer or the behaviour of the run that produced it?
  • For your most autonomous system in production, whose name is next to the decision to grant it that autonomy?
  
  The paradox doesn’t get solved, it gets held
  
  That was the note on which the session closed. There is no configuration of frameworks that makes this paradox go away. Autonomy and accountability pull against each other permanently and the work is to hold the tension every day, with mechanisms that pay attention when people cannot. The destination isn’t maximum autonomy. It is governed autonomy.