Synergised Consulting
Method

Your AI Agent Passed the Pilot. Production Asks Different Questions.

Synergised Consulting Ltd

7 min read
Title card on the Synergised ground background: the post title in charcoal sans type with a coral rule above, over the label First 100 Days Operations

Last Updated: 11 November 2026

Keeping an AI agent under control in production is not a capability problem; it is an evidence problem. For any action the agent took, you need to be able to show what it saw, which tool it called, who approved it, and what it cost. This month Cisco and Snowflake both shipped tooling built around exactly that proof, and the gap between their announced features and their actual availability tells you what is real today.

What shipped this week

Cisco announced at Cisco Live APJC on 10 November that Splunk Agent Observability gains a cost governance feature called Tokenomics and an Evaluation Engine built on new Eval-G1 evaluation models, while Cisco AI Defense adds red teaming that uses Anthropic's Claude to attack agent systems under named-operator controls [1]. Snowflake's Expedition 2026 announcements on 3 November expanded Cortex AI Gateway with agent tracing, abuse detection, tool-call governance and hard spending limits [3]. Both releases treat the agent as something that must be governed and audited, not just demonstrated. The market is no longer asking whether agents can do the work; it is building the machinery to prove what they did.

That shift is worth a moment of attention from any business owner running or considering agents, because the tools being built for enterprises name the controls that matter everywhere. You do not need a data warehouse to borrow the control list.

What production asks that a pilot never does

A pilot runs on sample data with a person watching each step. Production runs unattended, on live systems, spending real money, and the proof it demands is answers for any given action: what did the agent see, which tool did it call, who approved it, and what did it cost. Runtime observability means recording every step while the agent works, the reasoning step, the tool chosen, the arguments passed, the result returned, the time taken and the tokens spent. A transcript of prompts and answers is not that record.

Cisco frames the same need as a continuous loop of testing, cost and visibility, quoting the requirement for unified, real-time visibility into agent cost, performance and quality [1]. The independent coverage of Snowflake's release makes a point worth holding on to: its observability records land in the customer's own account, not in a vendor's [4]. Where the evidence lives decides who can inspect it later, and that question, vendor platform or your own systems, is worth answering deliberately rather than by default.

What the new tooling actually does

What is actually usable today

The honest summary is that some of this is usable now and some of it is a preview label. At Snowflake, gateway inference, observability and cost management are generally available as of 3 November, while tool-call governance is in public preview, and Agent Trace Intelligence, Agent Abuse Detection and Cortex Sandboxes remain in private preview [3]. Business Ontology, the approved-definitions layer, has no announced public preview date [3]. If your platform shortlist includes Snowflake agents, the generally-available parts are the ones to plan around.

The numbers, and their limits

Cisco's release carries real numbers, and they are company-reported rather than independently measured. Splunk reports that its Eval-G1 evaluation models return an evaluation in about 100 milliseconds, roughly 2.5 times faster than using a large language model as the judge, at up to 98% lower evaluation cost than a commercial LLM, which is what makes checking every production interaction affordable rather than sampling it [1][2]. Availability is staggered too: the evaluation models and the tuner are generally available this month, while Tokenomics enters private preview in December and reaches general availability on AWS Marketplace in early 2027 [1][2].

There is also a caution that marketing pages will not print you. The red teaming controls to targets the customer has registered and is authorised to test, ties every engagement to a named, authenticated operator, and keeps sessions time-bound and stoppable [1][5]. That is the right design, and it exists because an adversarial model pointed at the wrong target is itself an incident. The general lesson transfers even if the product never reaches you: anything that tests or acts aggressively on your behalf needs a registered scope, a named human and an off switch.

The control that buys the most: the approval gate

The cheapest governance control to set up, and the one to start with, is the approval gate: anything irreversible stops and waits for a named human before it runs. The gate does not need to be elaborate software. A queue, an approve action and a recorded identity turn an unattended agent into a supervised one without giving up the speed the agent was bought for. ### The approval gate, worked through

Consider an illustrative walkthrough. An agent drafts a batch of customer invoices each morning. In the ungated version, it sends them. In the gated version, it queues the batch with a summary, the owner approves with a recorded identity, and only then does sending run. The agent keeps its speed; the irreversible step acquires a name and a timestamp. If something is wrong, you know who approved it and when, which is precisely the evidence a pilot never produced.

The caution on spending limits

Cost limits belong in the same conversation, but their behaviour is worth knowing before you trust one. Snowflake's own cost documentation, reported in independent coverage, states that shared budget evaluation can take up to 6.5 hours, or one hour on the low-latency option, while per-user quota blocking applies within minutes of usage being counted; and a gateway block rejects new requests while work already running finishes and is billed [4]. In other words, a spending limit is a fence with a delay, not a light switch. An agent that starts looping burns budget before any limit notices. The practical consequence for a small business is to keep approvals on actions that carry real cost, not only on actions that carry real consequence, and to check spend directly rather than assuming a cap is enforced in real time.

How to set this up before buying anything

None of the enterprise releases are a prerequisite for the controls themselves, and in discovery workshops we find the useful part of this work happens in conversation rather than in a console: walk a normal week task by task, and the steps an agent touches separate quickly into those that are reversible and those that are not. The irreversible ones become the gate list. We also recommend running that session with key team members in the room rather than interviewing one person, because the reflection and the pain points surface in the discussion itself, and the tool permissions you need to scope become visible when people describe how work actually moves between them.

The setup that follows is deliberately unglamorous. Scope each agent's access to the specific systems and mailboxes its task needs, and rotate those credentials on a schedule. Log every step of every run somewhere you control, in the format that answers the production questions: what it saw, what it called, who said yes, what it cost. Put an approval gate in front of anything that moves money, deletes records, contacts a customer or changes a live configuration. Then attempt to break your own agent: write down the outcomes you refuse to accept and try to produce them, an instruction hidden in a document, a tool result that asks the agent to send something it should not. What fails becomes the rule the agent must obey. That record of scoped access, logged steps and named approvals is not paperwork; it is the thing that lets anyone, a customer, an auditor or a future buyer of your business, establish what your agent did without taking anyone's word for it.

Sources

  1. [1] Cisco Newsroom, "Cisco Unveils Frontier AI Red Teaming and Agent Observability to Secure the Agentic Enterprise", 9-10 November 2026:
  2. [2] Splunk Blog, "Splunk Observability at Cisco Live: Agent Evaluation, Tokenomics on AWS, and Simplified Full-Stack Packaging", 10 November 2026:
  3. [3] Data Phoenix, "Snowflake adds Business Ontology and agent controls", 4 November 2026:
  4. [4] AI Ping, "Snowflake makes AI Gateway tracing and cost controls generally available", 5 November 2026:
  5. [5] Network World, "AI agents will break today's enterprise network, and Cisco wants to rebuild it", 11 November 2026: