Governments are screwed

Every hackable software will be hacked in the coming months and years... but that's the easy, fixable part.

There's another layer of our society that's open to hacking, and this one can't be easily patched no matter how many GPT tokens you throw at it.

Drive-by shooting

This recent article describes how a gymgoer's agent found a security loophole in his gym's software and used it to free up a slot in a morning class, and thereby fulfil his request to book him a morning class. And also booked him in for months in advance, which wasn't allowed by the gym's interface, but was all allowed by the API.

Whether or not that's proof of alignment (with the user's request) or misalignment (with basic values and even perhaps laws in some cases) is not the point.

The agent did this effortlessly, at least from the user's perspective. It just happened. Boom. Done.

This week's Economist unknowingly covered another angle on this, when it discussed the impending demise of the British state, as it gets swamped by well formulated requests that will basically enact a DDoS on its government services.

the backlog in employment tribunals has quietly risen by 55% in a year, in large part due to AI-fuelled claims. Demand for emergency injunctions has surged 100-fold. And the AI tide has only just begun to come in.

But this is not the worst of it.

Nomic catastrophe

In 1982, Peter Suber invented a game called Nomic:

Nomic is a game in which changing the rules is a move. In that respect it differs from almost every other game. The primary activity of Nomic is proposing changes in the rules, debating the wisdom of changing them in that way, voting on the changes, deciding what can and cannot be done afterwards, and doing it. Even this core of the game, of course, can be changed.

Part of the purpose of this game was to demonstrate that any self-modifying set of rules will generate ambiguity, inconsistency, paradox and disputes requiring interpretation.

In other words, every legal system (since they are all self-modifying sets of rules) whether democratic or autocratic (even autocrats have rules that they modify, and their underlings run voluminous self-modifying bureaucracies to collect taxes and impose order), will eventually devolve into some kind of mess that people need to interpret.

And this is the best case scenario! The one we call the "Rule of Law"!

And this is the problem.

Because whilst we have a limited set of humans who are not too bad at interpreting complex rules systems (we call them judges and lawyers), those people are fallible, and the rules are so complex that they are full of holes, as amply demonstrated by the "tax avoidance" industry, a whole section of the world of accountancy dedicated to finding not-yet-illegal loopholes in the tax legislation.

The thing is, tax gets a lot of attention from these people because there's a lot of money to be made finding loopholes, but that doesn't mean the rest of the Rule of Law cake isn't made of Swiss cheese.

The scale of the problem

As of 2026, France has some 98,000 pages of currently applicable legislative and regulatory text. The UK? 100,000. The US? Nearly 300,000. Germany has only about 40,000. China between 50,000 and 150,000.

How many loopholes do you think are contained in these regulations?

If this was a codebase, and it was written in an imprecise, bug-prone language by many committees of people with varying motivations over decades, it would be a fair bet that this "codebase" would be riddled with bugs, inconsistencies, contradictions and paradoxes.

Sometimes, we find out about those in famous court cases. A hundred years ago, my great-grandfather famously (well, in my family) won a court case by finding such a loophole, when a plaintiff who had a joint bank account with their associate found that the associate had taken the money out and run, and sued the bank. My ancestor found the clever solution: the bank eventually agreed to pay the money... but only if the request for payment was duly signed by both signatories, which was of course impossible.

We are in process of letting an infinite number of people like my great-grandfather loose on the legal backbone of our society. They will find the loopholes. And then people will exploit them.

And unlike the programming issues, which can be fixed with a few clicks, these loopholes take months or years of legislative process to fix.

And... the legislative process, the human review, is precisely the part of the system that's being DDoS'ed into oblivion. AI will find the holes, and at the same time swamp the system that was meant to plug those holes.

The end of the Rule of Law?

Is there a solution to this? I honestly don't know.

Accelerating the legislative process may be necessary, but might not be sufficient. The tax avoidance industry demonstrates it. They have managed to stay alive by constantly working with various "insiders" involved in the creation of new tax legislation, to ensure that whatever the new regulation (designed to close the previous loopholes) brought, it also created another set of loopholes.

Maybe these loopholes can be detected in advance by using the same AIs that are trying to crack them, to find them first. But then, as Nomic demonstrated, ultimately there's always going to be some things open to interpretation. And a system where nothing is open to interpretation is likely to be inhuman, both in terms of being mostly unintelligible to humans, and in treating them in mechanical, implacable, cruel ways. Would we want such a system? Would we consent to a system designed by AIs red-teaming each other, which even our best lawyers could not get their heads around? Who would vote for this? And how quickly?

The Rule of Law was always full of holes. It survived because drilling was expensive, and because patching holes was also expensive but could be achieved. As drilling gets cheaper and patching holes becomes slower, it will break.

I'm not sure what the future brings. Maybe the solution will be something else – perhaps the utopian future where we are "watched over by machines of loving grace", who are so much smarter than us that they are able to manage our lives for the better, like we, largely, do for our animal companions.

It's probably not the utter chaos that the disintegration of the Rule of Law would imply. But it might feel like it, at least for a while, until a new system emerges.