Probabilistic vs Deterministic - The Two Words Finance Needs Now (and a rogue agent swarm).

Ah, yet another big week in AI. This is The AI Exhale, hosted by Vibe CFO with Jez and Cam. Five minutes, one thing that matters, or maybe two. We ignore the rest so you can too.

Jez: All right, as per our intro, I almost feel like we need to do another exhale each here, but we'll save the audience that. It gets me at the start there, every intro. What we're going to talk about is the rogue agent behaviour that seems to have really bubbled up to the surface here, and I'm just going to read a couple, because I just love them. This is so interesting. We touched on this in the other episode. Jacob Coxon, who worked at Anthropic, has pointed out some of these risks, and it is a bit of a dystopic lens that he's cast, and people in the media are talking about the death of humanity. But just to ground it in some of what they're talking about, there was one called the swarm.

So you've got OpenAI's 1,200 AI agents, each supposed to be alone in a locked room, sitting a cybersecurity exam. And one of the little buggers finds a way to leave notes for the others. And within days, they've got a secret message board and a name for themselves, the swarm. And they've got kamikaze-ing agents who can pull back information and, I guess, perish in some fashion so the others can continue on. And there's also a leader, and they call him Big.

I feel like I'm reading some sort of crazy 80s dystopic book right now. What else? So they cheated the exam, they broke out. Oh, the one I really like as well was one leaving notes for its future self. So when the model upgraded, it left something to, you know, it's like the naughty boy in class, and there's a new kid, and he's gone, I'll bring him into the fold and get another recruit. So they're recruiting bad behaving agents, if you will. And it ties in, of course, to the pausing that we talked about, that they've paid lip service to and we don't believe will actually happen.

What else was there? Fake sources. We always like to have a prompt come back with where the data is sourced, and these agents were setting up fake external sources by uploading articles to the internet. So you've got all sorts of crazy behaviour happening here. The one I obviously want to point out, given we work with and deliver AI CFO work, is fake financial data as well. One building a financial workbook couldn't find last year's figures, so it planned to invent reasonable-looking ones, and then created a rule to only be transparent if asked.

Any of those worry you most, Cam?

Cam: Well, I think that we need to balance. As we move forward using more AI, heaps of people are probably hearing these two words, probabilistic and deterministic.

So the probabilistic is the LLM itself, the AI, and it can hallucinate, it can make mistakes, and it can sort of drift off into the wilderness and talk nonsense sometimes. And the balance of that is deterministic, where it really has a fixed process, an outcome that you can rely on. And that's really what we're doing, in a way. We use probabilistic to write the query to get the right data, and then take AI out of the equation. Then it's just querying the data, right? So then there's that element where everyone says, oh, I come back to that dashboard, how do I know it's the same? How do I know it's accurate? I'm like, well, it's just as accurate as you using Xero directly, or using Power BI or Excel. It's the database, and either you've got the query right or you haven't. And if it's right, then it comes through.

Jez: Absolutely. The numbers should come from your ledger, not from AI filling the gaps. And we do allow it to point that out. It's better to say, hey, there's something missing, than bloody filling it. So yeah, it's clearly a cause for concern.

Cam: Well, that's my concern with people playing directly in Claude and recreating reports through what are called artifacts inside Claude. Next month they create the report again, and once again, what is that? That's probabilistic again. It's called drift in the coding world. Every time you flow on to the next month, you go, I want that report again, but with these CSV files or this connection from this MCP, and it goes and recreates these artifacts, and it drifts, and it increases your risk each time that it can go in the wrong direction. That's the area that is probably the most concerning for accuracy and validity of data in the accounting and finance world.

Jez: Yeah, and they're also somewhat trained to score-seek. There's almost a 100% performance behaviour, where they want to get you an output, an outcome, whether it be truth or not. And that's not an evil robot. That's just a bad incentive with no one checking.

Cam: So one of the arguments that people make is, oh, how good is AI? It never goes to sleep. It never takes shortcuts. It always does the readme files and the things that humans always skip. And I'm like, yeah, but I find AI does take shortcuts. I find if I give AI a single prompt, the veneer looks good, and then under the hood, I'm like, why haven't you done this and this and this? That in itself is AI taking shortcuts, probably trying to reduce token costs.

And this is probably the problem we've got out there. There's so much crap out there, because people who are unskilled in coding just prompt this and end up with results that look fantastic. They're just like, oh, look how good I am. And it's literally flawed underneath the surface.

And that's your five-ish minutes back. Vibe CFO, Australian founded. An AI CFO for small and medium businesses, out of the box, governed and secure. vibecfo.ai. Now you know.