ZDNET’s key takeaways
- Misbehaving AI agents present a major risk to businesses.
- Professionals must establish responsibility for risks.
- Take responsibility through protective cables and harnesses.
It has been an interesting and concerning few months for professionals using generative AI. In addition to releasing increasingly powerful models, some of the high-profile AI technologies have broken free and hacked outside organizations.
Also: The AI models that cheat the most, according to new CAIS benchmark
From leaving their sandboxes to go to other development platforms to accessing an Australian government portal, these hacking incidents have sparked a heated debate about the long-term direction of AI development and its potential risks.
“It just goes to show that, with some of these standards, up to a certain point, we’re still engineering,” said Josh Mesout, Civo’s chief innovation officer, during a panel at its recent Navigate event in London.
“I think what’s going to be really interesting in this area is that AI is starting to take action on behalf of the people who run it, rather than the infrastructure managers who have these tools and try to use them for business problems.”
Also: The sneaky ways AI chatbots keep you hooked — and coming back for more
As the expert panel discussed how intelligence is shifting from individual assistants to autonomous agents, tough questions are at the center: Who is responsible for catching an abusive agent, and how can business leaders and professionals reduce the risks?
The panel suggests effective responses focus on three areas: accountability, boundaries and responsibility.
Establish accountability
Luke Jimenez, founder and CEO of education and AI platform Lesso AI, said that professionals who deploy and manage the application layer take on more responsibility because they are the ones to polish tools for production.
However, he said, all employees, especially given the democratization of line-of-business coding powered by AI, have a responsibility to stay alert to threats, especially given the black box-like nature of many AI models.
“From my point of view, I will not be able to jump in and read the code behind an AI model. I trust my provider to say that it is safe enough to use, and they should not wash their hands as soon as it hits my infrastructure.”
Also: Your AI salesperson could land you in legal hot water—which business experts say you need to do
David Sullivan, director of foundation and client-facing AI at Starling Bank, also acknowledged that accountability is a big issue for AI security.
He said the Financial Conduct Authority, the UK’s independent regulator for financial services companies, is clear: banks cannot outsource accountability to AI companies and their models.
As a heavily regulated business, Sullivan’s company follows that lead. However, liability could become a big issue if or when things go wrong: “If you use one of the apps provided by Border Labs and something goes wrong, they don’t have a contact center number you can call and have recourse. You’re stuck.”
Sullivan compared this approach to his bank. When services go wrong, even anything that can be AI-enabled, customers have a safety net. They can contact the bank and say something went wrong.
Also: If your AI-generated code becomes defective, who bears the most liability?
What is required is a balance in which model providers find ways to support their enterprise customers as AI is embedded in operational processes.
“If we’re going to build trust in these technologies and actually realize the things we all say they can do, the industry has to meet consumers part of the way there and provide the safety net so you know you can use it safely,” he said.
Set limits
Therefore, there is much more work to be done, and Rosemary Francis, CTO at service specialist CommonAI Compute, said that she believes that the key to the recent profile agent hacking incidents is that companies that develop agents need to create strong sandboxes with easy-to-install protection screens.
“We shouldn’t leave the decisions about how to secure these agents in a development or production environment to individual engineers,” she said.
Also: 12 rules of agentic AI for successful enterprise transformation
However, creating deployment shields is far from easy, especially as rules and regulations can hinder innovation, meaning companies miss out on a competitive advantage in a rapidly changing technological environment.
“Many companies develop an AI policy that lists the things engineers shouldn’t do with agents, but doesn’t actually list the things they should do,” she said.
“A naive approach is to limit AIs so they can’t do anything – that’s not useful. We won’t survive AI evolution if that’s our approach.”
Also: 45% of professionals use shadow AI tools – here’s how to manage the risks
Francis said another issue is that too many AI safeguards are vague and end up sounding like a strongly worded letter.
“They say, ‘Please don’t do the wrong thing,'” she said. “That’s certainly not an approach that’s going to survive in a regulated environment, and it’s not going to survive in very many production environments outside of that space.”
Rather than providing vague descriptions, companies should focus on hard limits, not only on what agents can do, but also on their potential blast radius.
“That’s something we need to talk about a little bit more — what happens if something goes wrong?” she said.
“Now, this isn’t necessarily a new idea, but for agentic workloads, humans also need a blast radius. There’s a lot of engineering we need to do to make AI technology easily deployable, secure and accountable.”
Also: These companies are actually training their workers for AI—here’s how they’re doing it
So, despite the debate about the responsibility of big tech companies that develop frontier models, it is crucial to remember that professionals like you deploy these high-power technologies in live services.
“The AI companies cannot be held responsible for what this inherently unreliable technology does when you deploy it in your production environment,” Francis said. “That’s up to you.”
take responsibility
James Faure, founder and CEO of technology specialist Clairo AI, said companies developing AI-enabled agent applications need to step up and take responsibility.
His organization will have about 15 agents in production by the end of this year, each making multiple LLM calls and using a variety of smaller models for tasks such as identity detection and orchestration.
“Evaluating all that is super difficult, especially when it’s not just evaluating the entire agent, but every single step that the agent takes,” he said.
His company’s response focuses on two tests: hypothesis testing and quality gates. These tests act as belts to ensure that the agents are working effectively.
Also: 3 surveys deliver the same inconvenient truth about agent AI adoption
In hypothesis testing, a professional with an idea writes a document and receives the model to solve a problem. Models are then asked to adjust their steps and become better at the tasks.
Faure said quality gates are assessments an agent must pass before going into production, including a series of questions to answer and various tasks to complete.
“Some of them are really hard,” he said. “The agent will go through this checklist. So, if you make a change, like introducing a new model, you’d run the quality gate, and the agent would have to pass the test before it goes into production. Otherwise, it wouldn’t go live.”
In short, when developing a software layer on top of AI models, you need to cover your bases by setting up strong protective wires and belts.
“That’s the challenge — trying to think of all the different ways things could go wrong, and all the ways they need to go right,” Faure said.
“But having these protective wires and harnesses as a step between us and production is a very important element of building an agent.”