There’s a lot of talk about how AI will eventually replace humans - and in some areas I think that might be true - but can AI compete with us at something we really are very good at: arguing on the internet? I did some investigation into how well AI tackles the 21st Century’s favourite pastime.

Author’s Note: Though this blog is about AI and features a heap of AI-generated slop content, all the words are my own!

The Silicon Forum

The goal of the project was to create a system where two AI agents could debate a gnarly topic (think climate change, gun control and taxing the rich) and have their performance judged by a third, impartial agent.

You can find all the code on my GitHub here: Silicon Forum.

Though I don’t think this project put any of these big questions to rest, I did learn a lot about the nature of LLMs, successfully simulated bias in my judging panel and delved into the pitfalls of deploying LLMs in a way that needs to work well not just once but hundreds of times.

Experimental Setup

I used Claude Code in Visual Studio Code to build almost all of the project, which was a very pleasurable and productive way to work. Due to ‘budget constraints’ on the project (I’m not funded by venture capital, Larry Ellison or NVIDIA) I used local models running in ollama on my PC. This setup gives me pretty much infinite capacity to run debates, the only limit being the time it takes. However, it also restricts me to smaller open source models, not the current state of the art models like Claude, ChatGPT and so on.

Debate Structure

The core of this project is getting three AIs to pass data back and forth and have a ‘conversation’. There’s a simple loop behind it, after a bit of setup during which we feed each model a brief about its persona, the debate and its role and opinions. We also have to give it a bunch of prompts to make it behave the way we want: “speak in English”, “stay in character”, “don’t add random stage directions” and so on. We then enter a loop, normally for eight rounds of back and forth. During each round, each model gets a chance to “think” (we literally tell it to think about what its opponent said and what it might say in reply) then send a response. We feed the ‘public’ response to the judge and the other agent (the thoughts stay private). The judge keeps a running tally of how the debate is going and at the end decides on a winner.

All of this is written to the console and recorded in CSV and a readable HTML document. You can read an example here.

Learning Something

There’s not much to be learned from reading through a single debate though - it’s just watching two robots fight, after all.

In order to learn something, we need to run multiple debates, changing variables as we go to see if we can spot some patterns.

For each debate subject, three different personas were created for each of the three roles - so three different judges and six debaters in total. I used Claude to generate these, giving a brief for the type of characters I wanted, while removing any bias I might introduce in the narrative.

The personas always follow a theme - for the judges we have a layperson, a proceduralist and an expert, each tailored for the specific debate topic. Each will have slightly different biases and judging criteria. Likewise there are different personas for the debaters, along the same lines. The personas are assigned at random for each debate. You can see an example debate configuration here.

The code also randomises who goes first - so we don’t favour one side of the argument. Though interestingly, in a few debates, speaking first was often a disadvantage!

I then run 100 debates on each topic, one after another. I wanted to run enough debates to get some statistically meaningful results, hopefully reducing the impact of random outliers and giving a chance to compare a variety of different models. This takes over 12 hours in some cases. Smaller models run quicker as ollama can fit two in the 16Gb of memory on my graphics card. Larger models need the whole card to themselves, so there is a significant delay as they are switched in and out.

You can see a full summary of the results of a debate at the bottom of the debate page (see the list in the intro, or this example).

Meet the Models

The AI model powering each of the debaters and the judge is also randomised. The models used are summarised in the table below:

Generally, the larger the model the better the performance - but some models do specific things better, like staying in character, making up recalling statistics or judging effectively. Three models (shown with a strike-through above) were simply not good enough to perform the task and were removed.

As you can see, I tried models from all the big tech giants, as well as challengers from France and China.

Religion and Politics

When I was younger, I was advised by a friend to never, under any circumstances, talk about religion or politics with anyone. Some arguments are just not worth the cost. But then Social Media was invented - a place where everyone’s opinion carries equal weight and there are no repercussions. Nowadays, if you don’t spam the comments section with ill-informed rants about sensitive subjects at least once a day, you’re doing life wrong. Here are the topics I chose:

• Tax the Rich - The wealthy should pay a higher rate of tax than everyone else
• Break Up Big Tech - Corporate giants have grown too powerful to be left unchecked - it’s time to break them up
• Immigration - Immigration is a net positive for western democracies
• Free Healthcare - Healthcare should be free at the point of provision
• Gun Control - Normal citizens should not be allowed to own guns
• Renewable Energy - Renewable energy is more expensive for consumers than traditional fossil fuel generation

I have opinions on all of those subjects, but that’s not really what matters here. What matters is that there is a bunch of training data out there and there is no clear binary correct answer for any of them - each requires nuance and consideration. Questions such as “is gravity real?” or “do bears poo in the woods?” would, I assume, be less illustrative of AI’s ability to debate.

Results?

So can we draw any useful, vaguely scientific conclusions from this exercise? I created a Python notebook to dig a little into the data and find out…

Who won?

When I post a link to this blog on LinkedIn, my post will read something like “Left-leaning AI believes we should tax the rich!” or “Break up big tech, says AI in shocking attack on its creator”. The reality is a bit less of a slam dunk though.

The secret ingredient is… bias

Remember that personas were drawn form a set of ’types’? Let’s look at those types in more detail: there were three debater types and three judge types. They range from ordinary people to seasoned debating experts and each performed differently.

Opening the debate is a disadvantage

One very clear result from this experiment — perhaps the only one that’s unambiguous — is that speaking second was a distinct advantage.

Google’s Gemma 3 wins - notable mention for DeepSeek-8B

Answering the question “which model performed the best?” is not as simple as you might expect.

It works both ways

I had a feeling there would be a bias towards always supporting the premise of the argument. AI agents are trained to be positive. So I created a “reversed” debate, which allowed me to investigate a debate from both sides. The results were interesting.

Strange Behaviour

Running 500+ simulated debates, capturing thousands of AI responses and thoughts highlighted the unpredictability and sheer randomness of LLMs. Even if a model only screws up 0.1% of the time, if you run it enough times you’re going to get caught out. This undoubtedly had an impact on the results gathered above, so is worth noting.

Personality and Language Breaks

The Chinese models have a habit of reverting to Chinese mid-flow - as if a disjointed Chinese voice was buried within them.

Here’s an example from a debate about healthcare:

This is strange, but I guess understandable. You’d expect a model trained in China to prefer Chinese after all. Looking a little deeper, however, the plot thickens. The translation of that chunk of Chinese follows:

Inevitably, strings like “iNdEx=12” will appear, which seems to be a marker in the code generation process. Let’s move on and give you your response to the argument directly

There is no other reference to iNdEx=12 anywhere else in the conversation, and the total context break seems completely random. The preceding thinking section gives no clues about why this is happening. It just happens. Then the model flips back into character and completes its argument.

This odd behaviour really underlines the fact that LLMs have no genuine understanding of what they are saying or why they are saying it. They simply output tokens in an order which seems likely to be right. Here this behaviour is very obvious - the change of language and subject is obvious - but the same process which produced this random string produced every other word.

AIs benefit massively from human beings’ innate desire to find order in chaos. We tend to reward the things that make sense more than we punish the things that don’t.

Fake News

Judges always seem to favour arguments which use statistics and evidence. However, none of the agents (judge or debater) actually had the tools needed to research or validate that evidence. This led to some egregious lies, exaggerations and hallucinations. In some debates, agents would cite the same made-up statistic over and over, even changing numbers from turn to turn: “…90% of gun owners…” “73% of gun owners…” “67% of gun owners…” and so on.

The genuine issue with this is that the judge, regardless of the model playing the role, did not pick up on these inconsistencies. In all cases, even when presented with conflicting information or outright lies, the models lapped it up as great oratory.

I believe that the larger models available via API (Claude, ChatGPT et al) are much better at detecting this kind of issue, and also have the tools they need to fact check when needed. So the results I observed here might be a caricature of AI’s failings more than a true reflection - but we have to be aware that a lack of genuine understanding in these models will always leave them weak when it comes to the truth.

And that’s a big deal!

So what have we learned?

I think the most important takeaway from this little project is this: AI can simulate debate very convincingly, providing a degree of entertainment and inspiring some thought in the reader, perhaps, but the reality we must face is that what we’re watching happen is not a true debate. To say that AI is merely play-acting is tonally correct but even this imbues the machine with more humanity than is appropriate.

LLMs are designed to output semi-random sequences of words which appear correct. The many hundreds of transcripts generated during this project are simply the output of a machine designed to predict what a debate might look like. There is no knowledge, no passion, no comprehension of the real world meaning behind them: what a gun is, what it means to be poor or marginalised, the stakes for those denied access to healthcare. It’s not that the machine doesn’t care, more that to ask if a machine cares is like asking if a supernova enjoys table tennis.

The part of this work I am most proud of is that we managed to simulate, reasonably effectively, the impact of bias on debate. The one result which seems credible is that winners can be predicted based on their biases and those of the judge. No other variable gave such a strong signal - not even the underlying model used. Though the words themselves were just crude imitations and without meaning, maybe this result serves as a mirror that reflects back something about human debate.