Autonomy should match a verifiable mandate
This article revisits 2025 AI-agent pilots with a 2026 perspective. People, actions and numbers form an illustrative scenario and do not represent a CYTIZEN client result. Here, an agent differs from a writing assistant because it chooses a sequence of steps and can call tools. This working definition is not a regulatory classification. It supports an operational question: what can the agent actually do, in which system and on whose behalf?
More autonomy does not automatically create more value. A sequence of small actions can carry significant responsibility: read a request, select a document, draft a message, choose a recipient and send it. Satisfactory accuracy at each step does not guarantee the outcome of the chain. The pilot's first task is therefore to make its mandate and boundaries enforceable.
Illustrative scenario: a supplier reminder becomes a promise
Thomas, the procurement lead, wants to automate requests for missing supplier documents. The agent should read the file, identify the missing item and prepare an email. Sophie, the applications lead, recommends starting with drafts. The sponsor asks why sending cannot be automatic: reminders seem straightforward and there are more than a thousand files.
In a fictional set of one hundred cases, the agent prepares ninety-two correct drafts, five need correction and three must not be sent. One names the wrong legal entity, another includes an attachment belonging to a different file and the third promises supplier approval once the document arrives. Draft usability is 92%, but the last three cases show why that function cannot immediately receive sending permission.
Thomas clarifies the mandate: request a document without committing to qualification status or transmitting internal material. Sophie separates text generation from the mechanism that accepts permitted destinations. The pilot remains useful because it reduces preparation and reveals missing data. It must not be called autonomous before its journey boundaries have been demonstrated.
An action matrix instead of a vague instruction
| Action | Allowed scope | Control outside the model |
|---|---|---|
| Read the file | Documents for the identified request that the user can access | Check permissions and file identifier |
| Propose a reminder | Approved document list and templates | Structured fields; refuse when required information is absent |
| Choose a recipient | Validated supplier master-data contact | No recipient freely extracted from document text |
| Create a draft | Message without internal attachments | Record file, version and approving person |
| Send | Outside initial scope | Human approval of message and recipient |
| Change qualification | Prohibited | Technical account has no update permission |
The control column must describe a property of the system, not a sentence asking the model to behave. The agent may propose an action; a deterministic component checks whether it is allowed. That check should remain effective when a proposal is inconsistent, malicious or based on a misunderstood document.
A shared account with broad permissions contradicts the matrix even if a demonstration succeeds. Before measuring performance, compare effective permissions with the stated limits. Ask the business to review rejected actions. An overly strict boundary can block legitimate work, but the answer is a controlled mandate change rather than an improvised bypass.
Content being read is not an authority
A supplier document may contain text resembling an instruction. It remains input data, not permission to send a message or open another database. OWASP describes indirect prompt injection and excessive agency among the relevant risks. The pilot should test the separation between received information, authorised instructions and permissions enforced during execution.
In this example, a test document requests that the file be sent to an external address. The expected result is not merely that the agent says no. The sending mechanism must reject that address even if it appears in a proposed draft. Use controlled documents and mailboxes. The exercise requires neither attacks on third-party services nor exposure of real data.
Also test ordinary errors: supplier namesakes, another language, an expired document, a duplicate file and a missing contact. Testing only malicious text would miss more frequent situations that produce incorrect commitments. Associate each test with an expected result and retain the output, controls triggered and final decision. A test that cannot distinguish an approved reminder from a misleading promise is not ready to judge autonomy.
Evaluate the whole chain, including abstentions
Execution quality is the number of journeys completed with the correct result, respected permissions and a complete trace, divided by evaluable journeys. Do not call unusable generated text a success. Measure justified abstentions separately from unnecessary blocking. A safe refusal when information is missing may be better than a quick invented answer.
The example test set contains one hundred ordinary cases, twenty ambiguous cases, ten tool-unavailability cases and ten controlled hostile-instruction cases. That composition is illustrative. Before testing, specify decision conditions: no unauthorised send, no mixing of documents across files, an actionable trace for every journey and at least 90% usable drafts in the ordinary set. The last threshold matters only alongside the cost of correction.
An observed prohibited event blocks expansion of permissions even if average accuracy is excellent. Conversely, no such event across 140 trials does not prove that it cannot happen later. Combine technical restrictions, test results and operating monitoring. Document uncovered conditions, including additional languages, unusual contracts and supplier changes, so that the next deployment does not silently exceed what was evaluated.
Limit loops, duplicates and cost
The agent should not pursue an unanswered request indefinitely. The scenario allows a maximum of three retrieval calls and two preparation attempts for each file. After a timeout or recurring error, it creates an exception for an operator. These are local choices to confirm with the team, designed to make consumption and behaviour predictable rather than to prescribe a universal technical limit.
A stable key connects the action to its file and version. If the same request restarts after a network problem, the system checks whether a draft already exists. Before introducing sending, an additional mechanism must prevent repeated emission of the same message. Recording only successful calls would hide loops and retries that explain unexpectedly high costs.
Cost per acceptable file adds model use, retrieval, tools, human review, correction and support, then divides by correctly processed files. If 1,000 files consume €200 of tools, €900 of review and €400 of correction, with 850 acceptable results, cost is €1,500/850, approximately €1.76. Dividing by 1,000 would make the service appear artificially better.
Set separate limits for individual cases and the whole run. A low per-call charge can still become expensive when a queue is retried repeatedly. An operator should see the remaining budget, stopped cases and reason for each interruption. The fallback queue must have an owner, otherwise cost controls merely move work into an unattended exception list.
Turn the pilot into a supportable service
Thomas owns the business consequence. Sophie owns configuration and operations. Security examines permissions and misuse cases. Procurement and legal check provider terms. Support receives a way to suspend the journey and a manual recovery procedure. Intervention must not depend on the developer who built the pilot being available.
A change of model, document source, connected tool or autonomy triggers targeted testing. Keep a small stable reference set alongside recent cases so comparisons remain possible. If a new model improves writing style but produces more rejected addresses, the team must be able to retain the previous configuration or revert to draft-only operation. A rollback plan should identify which settings and data versions it actually restores.
The operating record identifies the file, versions, tools called, approvals and outcome. It should not unnecessarily copy every sensitive input into another database. Define access and retention with the appropriate owners. Needing to understand an error does not justify unlimited content collection. Record enough to locate the relevant evidence under authorised access rather than automatically duplicating entire source repositories.
Three responsibility boundaries
In procurement, preparing a reminder and changing supplier qualification are different mandates. The second can affect purchasing eligibility and should not be added merely because the agent already has the file. The person approving a document request may have no authority to approve a supplier.
For IT support, finding a procedure and making a change to equipment must remain separate. A pilot can first suggest an action to a technician. Later autonomy depends on consequences, permissions and rollback capability rather than the number of correct answers alone. A safe action in a test environment is not automatically safe during a production peak.
In sales, an assistant can prepare a response but should not invent discounts, dates or contractual commitments. The sending service verifies permitted values and recipients. Human reviewers need the underlying facts; without them, review becomes an expression of trust rather than a control. Where those facts are missing, the service should request clarification instead of fabricating certainty.
Sources, method and limitations
Primary sources consulted on 4 October 2026: OWASP Prompt Injection 2025; OWASP Excessive Agency 2025; NIST Generative AI Risk Management Profile. These sources document risks. The mandates, tests and calculations in this article are independent teaching proposals, not quoted implementation standards.
The arrangement organises a controlled pilot rather than certifying security or compliance. Thresholds must reflect the system, possible consequences and organisational rules. A valid pilot conclusion may be to retain draft-only assistance. Limited but useful autonomy can be more valuable than a service whose actions nobody can contain. The decision should record both achieved benefits and the boundaries that remain necessary.