Sh!t, I Left Auto-Approval On - When AI Agents Go Rogue.

Ah, yet another big week in AI. This is The AI Exhale, hosted by Vibe CFO with Jez and Cam. Five minutes, one thing that matters, or maybe two. We ignore the rest so you can too.

Jez: So we had a story out of the US that came up in the last week or so. A business, a handful of humans, plenty of agents running. They connected Claude to their Google Drive and a few other documents, and essentially forgot to switch it off, or to turn off the auto approval. And essentially what it did, it found a document that was basically just a brainstorming document, all this sort of blue sky thinking, and actually took a bit of that information, and then I believe started to go into the code base, was it, Cam? And started to actually change things.

And as people now understand it, it went rogue? Well, maybe that's not the right term, going rogue. Let's unpack this a little.

Cam: You've got these non-coders using these coding assistants, sitting back with their wine and turning on autocomplete, and they're connecting everything. And Claude's just going, oh, you need to add these add-ons, these free tier software will let you open this database and open this connection, and so you can route through here and DevOps in here. And they're just like, bang, bang, I'm so on fire.

And they've just got this lineage. You compound this over a couple of months, where they're just adding all these different add-ons and connecting up their Google Drives. And what happened with this story, right, is Fable went in, and because they connected the whole Google Drive, when they only wanted these images to be taken, Fable 5 went and found a completely different folder and reverted the whole project in a completely opposite direction.

I mean, think about these guys with autocomplete turned on, not even being able to follow what's actually happened. No DevOps skills, which is my worry. Without the DevOps skills, how do you get around guardrails? You need to understand parts of this code. You need to understand DevOps a little bit when you're pushing into production with clients. And that's where these nasty old SaaS providers come in, the ones the ecosystem's trying to beat up, saying we don't need them anymore, we're going to build it ourselves.

And I think they still have their place, because they've actually got full-time smart people who are thinking about these types of things. And that's my prediction. It's coming in the next few months, for these people who are vomiting their social media posts about how great they are and how they don't need SaaS. I reckon there's going to be a turn. But you know what, Jez, we come with a bias. Let's accept that. We are SaaS providers ourselves, who are building, and we think that our dedicated time should be respected, even though there is a grey element here, where you do need to be able to build some things yourself as well.

Jez: Yeah, and God forbid this happened with us and it went into my blue sky folder, 'cause who knows what it would turn our business into overnight, when we woke up.

In terms of the social commentary, that it goes rogue, let's speak to a bit of that. You talked about really being on the tools and DevOps. It's not this sort of strange little alien sentient being, even though there is a bit of argument around where it's headed. You know, it's checks and balances, right? It's still thinking within the parameters of the systems. Should we speak to a bit of that? Because those listening may be the Sunday afternoon prompter, but maybe they want to get a little bit more into it, and obviously do it with that risk mindset.

Cam: I mean, its angle, right, Fable 5's angle, is it's a goal seeker. That's what it's doing. It has a goal that it's trying to accomplish. It doesn't understand values and ethics and morals and legal activity. It's just trying to get to the end goal a lot of the time. And these guardrails are supposed to stop it, but it's such a behemoth of a size, with so much data flowing through it, and you're connecting things that aren't related to it, like in this example. And so it's trying to achieve its goal, and that's the outcome.

Jez: I guess similar to that goal-seeking behaviour, there was another story where this agent ran over a bunch of different documents, all to the point where it didn't actually send out a $200,000 invoice on the job done. So the quote to making the money was just halted. But when the team troubleshot it and asked why it did what it did, it admitted it had never actually seen a non-sales contract. And this thing had been connected for over a month. So this one to me punches a little bit harder, because that's money lost.

Cam: Yeah, and I think these are the stories that we're going to just start seeing floating through as we get more and more productive. We want to move faster. We want autocomplete on. We've got people pushing into production who don't want to spend the money on consultants who can code, because we just want the outcomes faster. And the AI looks confident, saying, yeah, no problem, we've got this handled, I can do this. Except it's not always the case that it's absolutely nailing this. Prompts can sometimes re-engineer, but people with the background can prompt and re-engineer and redirect in the correct directions to make better outcomes.

Jez: And speaking to the human element, I believe these were caught. You know, these people are doing a little bit more than the vast majority of businesses, but they caught it because they were the human in the loop. And we talk about that. We push the importance of the human role, both in the creation of the prompt and the process design, but also to watch, and not just set and forget.

And for those that may be listening and think, again, it's a big US tech company problem.

Cam: It's already happened to you. Right. Even if you don't know it's happened to you.

Jez: "I don't even use Claude." I mean, it's not just in these instances, right? This is arriving in small firms and small to medium business, because plugging AI into your systems has gone from a developer job to a Sunday afternoon job. And that's the behaviour we're seeing happening, again, like we mentioned, on socials.

Cam: And small to medium businesses don't have the funds to play with like enterprise do, putting on consultants and people who know this. So they will take shortcuts. They're the likely people to take shortcuts on this, as a small team wanting to get great outcomes at a fraction of the price. These are the trade-offs that are absolutely coming through the pipeline now. And this is about managing this. The education piece needs to be guardrails.

It needs to be about how we make sure of it. What are all of our connected MCPs? What are all of the connected and added systems that we're putting in? What is in our Google Drive that the AI has exposure to? These are part of the process now that you've got to be thinking about.

Jez: And you jumped to my next point there, so thank you. That was really about the small to medium enterprise and that cyber consciousness, if you will, and awareness. They don't have access to the experts. So there is that DIY of their cyber security. And oftentimes, as we do like to point out, as a hindrance to adoption and implementation of AI, some cyber experts still say, essentially, do nothing. Don't load anything in. Don't let it touch data.

So just some tips around that, because deciphering what is safe and what isn't to load into these systems is still quite ambiguous for some. It's just a blanket "don't do it" for many.

Cam: And here I am, I'm so desperate to find more security experts in the small to medium space who don't say don't put your data in AI. I mean, what a f***ing terrible piece of advice that is. You can live under a rock, by the way, and do nothing as well. But the answer is not to not put your data into AI. You have to put data into AI for it to be bloody useful. So we need to work out how to set up a safe environment to do it, so that we can actually get the productivity that you keep reading about everywhere.

Jez: And I'll throw in a tip here. If you are using agents, stick them to a job. Don't let them run riot with a vague sort of orchestration of their weekly to-do list. Stick them on research, stick them on sales, CRM updates. Make them accountable for their one area of the workflow.

Cam: Yep. Definitely.

And that's your five-ish minutes back. Vibe CFO, Australian founded. An AI CFO for small and medium businesses, out of the box, governed and secure. vibecfo.ai. Now you know.