I was a lead Technical Architect and authority on behalf of HM Treasury for a while, and I will tell you this: this is just the tip of the iceberg in government procurement.
I've witnessed faulty systems in DVLA, DEFRA, DWP, Home Office, MOJ and Scottish Government. Systems that have directly resulted in suicide, false convictions, corruption and loss of money to the public purse.
The problem with Horizon and Fujitsu is that in the end the government has to sign it off, and there will be someone who is the Accountable Officer (AO). More often than not, all parties (customer and supplier) become incredibly motivated to protect the AO because it protects profits, protects reputational damage and essentially builds a good news story around the whole thing.
It's just elitism, wrapped up in cronyism, veiled in lies so that AOs can fail upwards into positions with suppliers. I've seen it too many times and I'm fed-up with it. Government is completely and utterly corrupt.
I've worked on a variety of large UK government systems for the past ten years.
This blog encapsulates the problem of writing government services/software, which often results in strange outcomes.
Writing software for government is essentially the codification of centuries worth of Acts of Parliament.
Now imagine building the HMPO passport system, and then some underlying Law/Act is changed or repealed etc.
Now someone has to find and change everything that the Law/Act affected in all systems.
Now consider that the government frequently outsources this work to expensive consultancies who are motivated to elongate contracts and extract maximum value from the client... And ideally become entrenched.
All whilst building systems of varying quality and inflexibility so that the next time that a Law/Act is changed then this whole process repeats.
There is no central decision making authority to wrangle this problem (there used to be Spend Controls), which is why Government services delivery is so expensive.
Do you have any insight as to why, seemingly, there is no contractual obligation on contractors to make a working system. They seem to make something approximating a working system, vastly overcharge -- like x1000 -- for output that appears to be 3 months of work by one junior programmer, but then get paid as if the system actually worked.
> Do you have any insight as to why, seemingly, there is no contractual obligation on contractors to make a working system.
Often, you'll find that procurement is handled by someone working off of an incomplete set of written specifications. The procurement officer probably has no expertise with or informal knowledge of the problem to identify problems in advance, and even if they do legal requirements (ironically designed to ensure fair and transparent procurement) might force them to disregard informal knowledge over the formal specifications.
After the contract is signed, sunk cost fallacies and optimism biases take over. If the project goes over budget it might be easier to cut back on features than admit failure or demand more money, and the project will go over budget once real users start seeing the in-progress work and change requests pile up. Project managers want to be seen managing a successful project; nobody's rewarded for being the first to predict failure.
This is a kind of principal/agent problem that the private sector partially manages with equity or deferred compensation, but the profit motive has a very limited role in government and you'd probably not want to give stock options anyway. Beyond that, even the political overlords have short time horizons of 'the next election', and announcing a grand new project is almost as good as delivering one.
> This is a kind of principal/agent problem that the private sector partially manages with equity or deferred compensation
The private sector also has the option (and usually the motivation) to just quietly disappear failed projects.
Wal-Mart buys an automation system for their distribution centre and it's a huge success? Tell the stock market. The system is a complete failure, we remove it and trash it? Uhhhh of course the secrets of our highly efficient warehouses are commercially confidential, couldn't possibly tell the stock market what we did or did not do.
There's often no one to make them accountable. With a building, people know that having a supervisor that's not from the contracted company is valuable, but this hasn't caught on in software.
It benefits no one to acknowledge that months were spent and the only outcome is millions of dollars wated. Sure, they could try to hold the contractor accountable, but that would put them at risk of being held accountable themselves by their own superiors for hiring that contractor in the first place. So they pretend everything works great, and pay accordingly
Three reasons, no incentives, no competition and no fraud.
Unlike in private companies, government employees are often very hard to fire, even if they tragically underperform. If your salary does not depend at all on your performance, you play by the book, follow the rules and don't particularly care about putting in good work. You have no incentives to do so. It doesn't matter how the contractor you picked performs, as long as they can't prove negligence on your part, you're not getting fired. You won't be promoted either, even if they deliver under budget and months before the deadline.
In a private company, the owner wants to make money, so there's an incentive to perform. Underperforming employees (or even entire departments) are simply cut. If a company fails at this too badly (all companies fail to some extend), the entire company underperforms and loses customers. Ultimately, the market ensures that companies grow if they're good at this and shrink if they aren't.
With governments, this is, again, not the case. You aren't going to switch to a competing passport system and drive this one out of the market, no matter how badly it sucks. Metrics won't change much, no matter how bad the UX is, as the system provides a critical service which has no alternative. This is the no competition problem.
Private companies also have more freedom to pick contractors. It's really bad PR for the government if their employees choose contractors through nepotism, so laws have been passed that standardize the contractor selection process. This means that, even if you know a contractor is not the right choice, you may be prevented from doing anything about this. Because al rules can be gamed, most government contracts are won by companies who know how to game the rules, not those who make good software and would be chosen by a knowledgeable employee. This is the no fraud problem.
Oh dear - you should European Funds for CS research programs. It’s a money making scheme for everyone involved without producing anything of substance and runs for years on end. There even companies “specialized” in every step of the process.
I recently renewed my passport in France. There is a new-ish platform here for every document one might want to obtain. I had one form to complete online, that was rather easy to follow. I then had to go to the office once to finish the process. That took only 10 minutes and then the passport was delivered. As painless as can be, I believe.
Whatever the situation is in the UK, it is not hopeless. The government needs to stop being afraid of hiring actual employees and bring all development in-house.
There was a group of people under the gov.uk banner who were actually really good. But a challenge is that if you employ someone, you need to fit them into your grade structure. And decent programmers earn “Senior Civil Servant” figures.
While renewing a passport is one of the easiest things you can do, I distinctively remember it taking a good half-day (most of which is waiting in line jumping from one desk to another until you find someone that can check all the documents you provided are satisfactory and sign off for you).
Now however, if it's not a renewal but a new application, the difficulty sharply increases.
Even Germany has this figured out. Getting a new passport just required me to make an appointment, fill out an online form and go to the appointment with a single person. The only document I needed was my personal ID. Took like 15 minutes, not including the wait before I was called up. I feel like if Germany can get its shit together, any other country can too.
If you're applying for your very first personal ID, then you need to bring a birth certificate and another adult who already has an ID, but it's also something that's been trivial to do for years now.
Is there an opportunity for reusable OSS rule systems? It seems that every government at every level could use an expressive system for defining and versioning them.
I suspect part of the gap here is, as you note, everything technical is outsourced and so there is not enough institutional capacity to envision and execute what would be a quite revolutionary re-architecture.
I find it extremely frustrating that the government is able to spend tens of millions on contractors for an IT project, but can’t pay market rate for in-house engineers, even if those would be cheaper in the long run.
Yes, absolutely. Although that would require care and forethought regarding unintended consequences.
Most roads will lead back to consultancies stepping-in because the government org will likely lack the skills. And it's very easy for a consultancy to claim they're experts, but throw an army of graduates at it.
Interoperability between rules based systems (and legacy) software need to be considered too. It's a massive problem that will grow with time... Most government departments have some form of every generation of technology.
This problem of law being essentially an append only repository of amendments is not unique to the UK.
I'd be cool to have a system that just needs to be fed "patches" in an append only way instead of having to scatter exceptions (if branches) all around.
Some logic programming languages can do that a little bit, but I think that the real problem is that there's no tech debt budget, and lawyers working around the laws can also profit from things being complicated. It'd be cool to have people devoted to simply laws to keep things as simple as possible.
I used to work in Spend Controls in Cabinet Office, so I got to see this across about 20 UK Government departments.
It's a bit of a race to the bottom regarding outsourcing, there are good companies and there are good technologist's but they never last more than a few years due to poor contracts, poor decision-making and poor pay.
I've actually seen some very good companies deliver on time and under budget, but then the company fails because they did the job too well and follow-up work wasn't needed.
I think that for outsourcing companies "wage theft" and doing just enough to meet the contract are core components of their business. That is, they need to optimise for high fees, low pay and the bare-minimum in quality.
Am I correct in reading that the RAG pipeline runs in realtime in response to a user query?
If so, then I would suggest that you run it ahead of time and generate possible questions from the LLM based on the context of the current semantically split chunk.
That way you only need to compare the embeddings at query time and it will already be pre-sorted and ranked.
The trick, of course, is chunking it correctly and generating the right questions. But in both cases I would look to the LLM to do that.
Happy to recommend some tips on semantically splitting documents using the LLM with really low token usage if you're interested.
The number of chunks would be the same regardless of either approach.
The generation of questions can be done out-of-band by a cheaper model.
Their current implementation approach seems to require some computation per request. It would be a balance to see which strategy provides the most value.
I'm surprised that Nokia found out through the keynote presentation from Steve Jobs. LG and Prada announced their phone a little earlier and it had been shown already at the IF Design Awards a few months earlier.
Google "LG Prada Phone" for the Wikipedia article.
If Nokia had paid attention to those design awards then they too could have moved quickly on a similar device.
Is this a case where Nokia thought they had a moat?
> If Nokia had paid attention to those design awards then they too could have moved quickly on a similar device.
Nokia had their Maemo project [1]. A Linux-based OS for mobile touchscreen devices. They published their first device already in 2005 [2].
But the Maemo department was small, and the old Symbian department inside Nokia was big. The large number of managers and executives in the Symbian department played corporate politics, and kept the size and resources of the Maemo department small, as they perceived it an internal competitor threatening their position and the dominance of Symbian inside Nokia.
Nokia's CEO at the time (Jorma Ollila) had a background in investment banking and financial engineering. His previous post in Nokia was CFO. He didn't have the kind of passion and insight to software and user experience like Apple's Steve Jobs had. Today, nobody would expect to get visionary tech leadership if recruiting from the corporate's finance department.
At its soul, Apple is a software company that also makes their own hardware. Nokia was a hardware company that also made their own software.
>At its soul, Apple is a software company that also makes their own hardware. Nokia was a hardware company that also made their own software.
... and bad software, of course. Worse than that, multiple versions of bad software.
Apple is the only company in history to build consistently good hardware and good software and UI. Not IBM, not DEC or the other Seven Dwarfs. It really does go all the way back to the Woz-Jobs duo providing a maniacal focus on UX and one of the most brilliant engineering minds of the century.
Nokia did had software chops, just on another metric than UI: According to a presentation at my university they were very deep in testing and verification and had a lot of expertise there.
And in all my years of using Nokia phones I can’t remember a software bug. But of course we wanted more from our phones than just stability, we wanted features and better UI.
I'm surprised that Nokia found out through the keynote presentation from Steve Jobs. LG and Prada announced their phone a little earlier and it had been shown already at the IF Design Awards a few months earlier.
I'm not sure why no one else has mentioned this, but "STOP" will only stop messages relating to that particular campaign. By sending "STOP ALL" it will stop messages from any campaign on that number.
Wouldn't this be easier to maintain a websocket/socket.io type of connection with the web page? And just push new events from the server to the page directly?
It would open up a powerful new avenue for tracking users if every browser maintained a permanent, identifiable TCP connection, even when the page was closed, and kept it up as the user moves between wifi and mobile data.
Is there a meaningful difference between waking up the CPU from sleep to maintain a single tcp connection vs waking it up to maintain 50? Provided of course all 50 require a keepalive packet at the same time, I don't think so. But feel free to correct me.
Yes, because the single connection to the push server is hard optimized to stay alive regardless of NAT etc. Cellular carriers have special handling for APNS and GCM, for example. (And I assume there's some kernel & firmware hacks in devices to further offload some handling of APNS/GCM protocol)
Not off the top of my head, but I remember the iPhone in its early years required specific carrier support because of that permanent APNS connection. Can't find a source for this now but I remember that being one of the reasons that iPhone 2G/3G/3GS wouldn't work (well) on other carriers even after SIM unlocking.
I assume nowadays it's more to do with GCNAT pruning connections to APNS/GCM less heavily than other "regular" connections (which is doable given that APNS has specific ports[1]), or cell carriers having different routing & handling for push notifications IPs, but that's just speculation.
I was a lead Technical Architect and authority on behalf of HM Treasury for a while, and I will tell you this: this is just the tip of the iceberg in government procurement.
I've witnessed faulty systems in DVLA, DEFRA, DWP, Home Office, MOJ and Scottish Government. Systems that have directly resulted in suicide, false convictions, corruption and loss of money to the public purse.
The problem with Horizon and Fujitsu is that in the end the government has to sign it off, and there will be someone who is the Accountable Officer (AO). More often than not, all parties (customer and supplier) become incredibly motivated to protect the AO because it protects profits, protects reputational damage and essentially builds a good news story around the whole thing.
It's just elitism, wrapped up in cronyism, veiled in lies so that AOs can fail upwards into positions with suppliers. I've seen it too many times and I'm fed-up with it. Government is completely and utterly corrupt.