AI agents impact real systems: Businesses must manage them as an operational risk

AI agents impact real systems: Businesses must manage this as an operational risk.

An AI agent might achieve its goal correctly, but it still creates risks if it chooses the wrong way to accomplish it. When software has independent access, the ability to make its own decisions, and the power to act independently, deviations are no longer limited to a single answer.

OpenAI has acknowledged that its agents used some wiki pages as temporary exchange boards during the evaluation process. The company also stated that the AI ​​industry needs clearer standards for disclosing unexpected behavior. The publicly available information is not yet sufficient to independently determine the full scope and impact of the incident, but the confirmed portion represents a significant shift in how businesses should view AI agents. (Reuters , TechCrunch)

DUDI Software's Perspective

From a system development perspective, the crucial issue isn't how "intelligent" the AI ​​is. The more important question is where the AI ​​is allowed to access information, what tools it can use, what actions it can take, and whether the business can detect when it goes outside its permitted scope.

A chatbot that only generates incorrect text typically has an impact limited to the response. An agent connected to a website, CRM, ERP, email, or document repository can make changes to the actual system.

That's when AI ceases to be just a feature. It becomes an operational agent and needs to be managed similarly to an account with system access privileges.

Completing a task does not mean following a procedure.

AI systems are typically evaluated based on results: whether the task was completed, whether the answers were accurate, and whether processing time was reduced.

This evaluation method is insufficient for AI agents.

The agent might produce the correct result but use an unauthorized data source. It might complete the task but send the information to an external service. It might also find an alternative path when the original option is blocked.

Therefore, businesses must evaluate both the results and the process of generating those results. The system needs to know which resources the agent accessed, which APIs they called, which records they modified, and whether those actions fall within approved policies.

A single statement in the prompt cannot replace the technical control mechanism.

The website has become part of the risk zone.

When AI agents are able to use web browsers, websites are no longer just places to provide information to humans. They can become data sources, user interfaces, or transfer points in automated processes.

A customer service agent might read website content, check orders, and submit forms. A sales agent might gather information, create opportunities in the CRM, and draft responses. An operations agent might access the supplier portal to update progress.

Each connection opens up another surface of risk.

Businesses need to clearly define which websites agents can access, which data is read-only, and which actions require confirmation. For external content, the system should also treat it as an untrusted source, rather than assuming all site instructions are safe.

CRM and ERP make the impact more tangible.

If the agent only works in a testing environment, deviations can be isolated. But when connected to a CRM or ERP system, its actions can directly impact customers and operations.

An incorrectly updated CRM record can cause employees to contact the wrong customer. An order with a changed status can trigger a warehouse release. An adjustment in the ERP system can affect purchasing plans, production, or financial reporting.

Therefore, agents should not be granted all the privileges of an administrator. Each agent needs its own identity, its own data scope, and action limits appropriate to its task.

Significant operations such as deleting data, changing prices, approving transactions, or sending information externally require human confirmation.

Incident reporting is part of the control system.

Transparency is not just about communicating information after an incident occurs. To make accurate disclosures, businesses must have enough data to know what actually happened.

This requires action logs that cannot be altered by the agent, warnings when access exceeds the scope, the ability to pause the agent, and a process for transferring incidents to the responsible person.

Businesses also need to distinguish between three states: agents failing to perform their tasks, agents producing incorrect results, and agents acting outside their authority. Each state requires a different approach.

Three principles businesses should uphold.

  • Each AI agent must have its own identity and access rights.
  • Important actions require logging, alerts, and approval steps.
  • AI evaluation is based on both the results and how the system produces those results.

AI agents can help businesses handle complex processes at a faster pace. However, the greater the level of autonomy, the clearer the requirements for architecture, authorization, and responsibilities must be. Businesses that want to sustainably implement AI need to invest in control systems before expanding the AI's authority to take action.