> No one actually wants a society being ruled by computers, except maybe the people running those computers.
Humans in charge are known for injustice, kickbacks, favor trading, selective enforcement and other forms of corruption and abuse. Properly engineered and regularly reviewed open source systems with balance checks might just get us closer to a rules-based system that provides a level playing field for everyone. Given all the known biases of current AI systems, we are certainly far from ready for it, but the prospect of transforming large parts of government into an open source "social operating system" that automatically and fairly offers basic services according to clearly coded and broadly enforced rules looks like a desirable goal in the (very) long term.
Many laws can be expressed as computer code. Where they cannot is often due to deliberate vagueness built in to leave scope for future interpretation as new cases arise. This suggests that we could express laws in computer code that raises a HumanInputRequiredException in the cases currently handled with deliberate vagueness. The resulting reduction in vagueness would remove a huge amount of discretion that currently facilitates corruption and abuse of power while ensuring ultimate human control and human-directed evolution of the law.
I want to add a historical remark. Very early forms of human government, such as ancient kingdoms in various parts of the world, had one or a few prominent members of society hold full discretion and decision-making power. Later, we codified rules and decided that even monarchs are not above the law. Endowing the written laws with additional power by making them executable seems like a natural next step.
You can't be serious. EU leadership has always been a bit weak, with a lot of compromises, and not a hard stance on foreign policy, but to prefer an unknown oracle? The current state of AI would even have it overfitted on a small relevant corpus padded with arbitrary other material. How can you expect that to produce better government?
This is at least a step in the right direction.