I think this kind of feedback is a good example of something an LLM is very good at suggesting. I regularly feed my important raw texts to an AI, and ask it not to rewrite it (!), but to give me line by line grammatical, style and tone advice, point out uncommon language, idioms or semantics etc. Also, they are good at fact checking, they can quickly verify each statement against web sources etc.
On the other hand, LLMs are very bad writing partners, they are sycophants and very rarely give substantial criticism, the kind of feedback an editor would give and is mentioned in the article.
This is the substantial service an editor will provide going forward in the AI slop era, where everyone and their grandma will self publish some personal masterpiece: a contact with the real world and setting the bar high, to the point you need to struggle to achieve the required quality. Writing a book, especially finishing a great book, is not supposed to be enjoyable, it's hard, grueling work.
In truth, I have never seen a company doing such an insane 180 against it's traditional suppliers as Google is doing against the websites populating its search pages which now Google milks as mere training data. Google Search is dead, from the perspective of the websites, and there is less and less incentive to spend effort and money to appear on the first pages.
Google better have a good strategy because this is bound to have 2nd order effects down the road, like, why would I pay money for advertising on a dead internet search engine? If AI is killing organic results, then AI better make money itself, because dead websites don't buy ads.
You could say the aggregate is air and the cement still performs its role as a binder, it binds all the air bubbles into a stable matrix. Hence "aerated concrete".
The UK crisis involved steel reinforced AAC beams that were used (of all places) to support roofs of schools. UK turned out to be a rainy place, the rain infused into the cellular structure and corroded the steel, with disastrous consequences.
It's a very particular use case of a very particular product, not relevant to the wide majority of AAC uses around the world, which is largely non-structural and not reinforced, or subjected to moderate compressive loads, such as lateral walls for 1-2 stories buildings in non-seismic areas.
The risks were understood (by engineers) and this usage was given a "shelf life". Unfortunately, those risks were put into the "Oh we'll forget about it" or "We'll wait until it looks a bit shifty" categories.
However as any fule (engineer) kno, reinforced and especially pre-stressed conc members will fail in quite a dramatic fashion. Unless you notice rust dribbling out then you can end up with anything from the roof failing to the roof exploding. I don't think anyone was daft enough to pre-stress these things.
I don't know how much money was saved but it was a really stupid application and basically ended up punting far greater costs due to remediation down the road.
While it might technically be true, that surely does not absolve the engineers who did this crap.
There is a general social expectation that new buildings should be structurally sound for a duration on the order of a century. So, if you deliver something that has a mean time before catastrophic failure around 30 years, you also need to account and set up the institutions that will handle the failure, the same way nuclear companies are required to set aside money for their decommissioning. You need to have periodic inspections for signs of early failure etc. and this whole circus needs to be disclosed and priced into your tender.
In reality, this entire fiasco was a dirty and cheapest way to satisfy the contract, ye old "good enough for government work" as evidenced by the fact no substantial number of private buildings of the same period are having this problem.
The maintenance provision was snuck into - or bribed into - some mountain of legalese, but the fuckers knew exactly that they were putting children in harm's way.
The industrial version is produced in an autoclave, this allows precise control of curing, density and final mechanical resistance/insulation values. Hence, the name the material is best known by - AAC.
On the other hand, the video linked attributes too much credit and complexity to the foam manufacturing method, it can certainly be done with very primitive technology. Here are some dudes doing it in a developing country, it's very very basic, the foam generator is basically a steel wool sponge where pressurized air combine with water containing the foaming agent. They give out the complete recipe and details of their tools:
I think it was mostly a branding exercise, Salesforce wanted to signal to its customers that they are on top of this whole AI thing and there is no need to go to some unknown AI startup to "AIfy" their business. So they wanted to capitalize on FOMO / fear of being disrupted while using a bad labor market to improve profitability. They succeeded in this and made news around the world, but maybe not so many new customers.
Makes no sense - why would Salesforce's customers care if the company is using AI or not, other than when it impacts them (the customer) such as worse customer service.
This just seems a poor decision made by C-suite folk who were neither AI-savvy enough to understand the limits of the tech, nor smart enough to run a meaningful trial to evaluate it. A failure of wishful thinking over rational evaluation.
If you consider the extent to which our economy has become financialized, then you see these decisions have little to do with providing a product for customers but rather a stock for investors.
You know you need to be careful when an Amazon engineer will argue for a database architecture that fully leverages (and makes you dependent of) the strengths of their employer's product. In particular:
> Commit-to-disk on a single system is both unnecessary (because we can replicate across storage on multiple systems) and inadequate (because we don’t want to lose writes even if a single system fails).
This is surely true for certain use cases, say financial applications which must guarantee 100% uptime, but I'd argue the vast, vast majority of applications are perfectly ok with local commit and rapid recovery from remote logs and replicas. The point is, the cloud won't give you that distributed consistency for free, you will pay for it both in money and complexity that in practice will lock you in to a specific cloud vendor.
I.e, make cloud and hosting services impossible to commoditize by the database vendors, which is exactly the point.
Skipping flushing the local disk seems rather silly to me:
- A modern high end SSD commits faster than the one way time to anywhere much farther than a few miles away. (Do the math. A few tens of microseconds specified write latency is pretty common. NVDIMMs (a sadly dying technology) can do even better. The speed of light is only so fast.
- Unfortunate local correlated failures happen. IMO it’s quite nice to be able to boot up your machine / rack / datacenters and have your data there.
- Not everyone runs something on the scale of S3 or EBS. Those systems are awesome, but they are (a) exceedingly complex and (b) really very slow compared to SSDs. If I’m going to run an active/standby or active/active system with, say, two locations, I will flush to disk in both locations.
> Skipping flushing the local disk seems rather silly to me
It is. Coordinated failures shouldn't be a surprise these days. It's kind of sad to here that from an AWS engineer. Same data pattern fills the buffers and crashes multiple servers, while they were all "hoping" that others fsynced the data, but it turns out they all filled up and crashed. That's just one case there are others.
Durability always has an asterisk i.e. guaranteed up to N number of devices failing. Once that N is set, your durability is out the moment those N devices all fail together. Whether that N counts local disks or remote servers.
Interestingly, on bare metal or old-school VMs, durability of local storage was pretty good. If the rack power failed, your data was probably still there. Sure, maybe it was only 99% or 99.9%, but that’s not bad if power failures are rare.
AWS etc, in contrast, have really quite abysmal durability for local disk storage. If you want the performance and cost benefits of using local storage (as opposed to S3, EBS, etc), there are plenty of server failure scenarios where the probability that your data is still there hovers around 0%.
This is about not even trying durability before returning a result ("Commit-to-disk on a single system is [...] unnecessary") it's hoping that servers won't crash and restart together: some might fail but others will eventually commit. However that assumes a subset of random (uncoordinated) hardware failures, maybe a cosmic ray blasts the ssd controller. That's fine, but it fails to account for coordinated failure where, a particular workload leads to the same overflow scenario on all servers the same. They all acknowledge the writes to the client but then all crash and restart.
To some extent the only way around that is to use non-uniform hardware though.
Suppose you have each server commit the data "to disk" but it's really a RAID controller with a battery-backed write cache or enterprise SSD with a DRAM cache and an internal capacitor to flush the cache on power failure. If they're all the same model and you find a usage pattern that will crash the firmware before it does the write, you lose the data. It's little different than having the storage node do it. If the code has a bug and they all run the same code then they all run the same bug.
Yeah good point, at least if you wait till you get an acknowledgement for the fsync on N nodes it's already in an a lot better position. Maybe overkill but you can also read the back the data and reverify the checksum. But yeah in general you make a good point, I think that's why some folks deliberately use different drive models and/or raid controllers to avoid cases like that.
This is an aside, but has anyone tried NVDIMMs as the disk, behind in-package HBM for ram? No idea if it would be any good, just kind of a funny thought. It’s like everything shifted one slot closer to the cores, haha, nonvolatile memory where the RAM use to live, memory pretty close to the core.
I think this entire design approach is on its way out. It turns out that the DIMM protocol was very much designed for volatile RAM, and shoehorning anything else in involves changes through a bunch of the stack (CPU, memory controller, DIMMs), which are largely proprietary and were never intended to support complex devices from other vendors according to published standards. Sure, every CPU and motherboard attempts to work with every appropriately specced DIMM, but that doesn’t mean that the same “physical” bits land in the same DRAM cells if you change your motherboard. Beyond interoperability issues, the entire cache system on most CPUs was always built on the assumption that, if the power fails, the contents of memory do not need to retain any well-defined values. Intel had several false starts trying to build a reliable mechanism to flush writes all the way to the DIMM.
Instead the industry seems to be moving toward CXL for fancy memory-like-but-not-quite-memory. CXL is based on PCIe, and it doesn’t have these weird interoperability and caching issues. Flushing writes all the way to PCIe has never been much of a problem, since basically every PCI or PCIe device ever requires a mechanism by which host software can communicate all the way to the device without the IO getting stalled in some buffer on the way.
I think it is fair to argue that there is a strong correlation between criticality of data and network scale. Most small buisnesses don't need anything S3 scale, but they also don't need 24 hour uptime, and losing the most recent day of data is annoying rather than catastrophic, so they can probably get away without flushing but with daily asynchronous backups to a different machine and a 1 minute UPS to allow for safe storage in the event of a power outage.
Committing to NVMe drive properly is really costly. I'm talking using O_DIRECT | OSYNC or fsync here.
Can be in the order of whole milliseconds, easily. And it is much worse if you are using cloud systems.
It is actually very cheap if done right. Enterprise SSDs have write-through caches, so an O_DIRECT|O_DSYNC write is sufficient, if you set things up so the filesystem doesn't have to also commit its own logs.
I just tested the mediocre enterprise nvme I have sitting on my desk (micron 7400 pro), it does over 30000 fsyncs per second (over a thunderbolt adapter to my laptop, even)
Another complexity here besides syncs per second is the size of the requests and duration of this test, since so many products will have faster cache/buffer layers which can be exhausted. The effect is similar whether this is a "non-volatile RAM" area on a traditional RAID controller, intermediate write zones in a complex SSD controller, or some logging/journaling layer on another volume storage abstraction like ZFS.
It is great as long as your actual workload fits, but misleading if a microbenchmark doesn't inform you of the knee in the curve where you exhaust the buffer and start observing the storage controller as it retires things from this buffer zone to the other long-term storage areas. There can also be far more variance in this state as it includes not just slower storage layers, but more bookkeeping or even garbage-collection functions.
You must still commit the WAL to disk, this is why the WAL exists it writes ahead to the log on durable storage. Its doesn't have to commit the main storage to disk only the WAL which is better since its just an append to end rather than placing correctly in the table storage which is slower.
You must have a single flushed write to disk to be durable, but it doesn't need the second write.
The disater plan is to have a few dozens stratum 1 servers spread around the world, each connected to a distinct primary atomic clock, so that a catastrophic disaster needs to take down the global internet itself for all servers to become unreachable.
The failure of a single such server is far from a disaster.
And the disaster plan for the disaster plan is to realize that it isn't that important at the human-level to have a clock meticulously set to correspond to other meticulously-set clocks, and that every attempt to force rigid timekeeping on humans is to try to make humans work more like machines rather than to make machines work more like humans.
I really, really can't get behind this sentiment. Having a reliable, accurate time keeping mechanism doesn't seem like an outlandish issue to want to maintain. Timekeeping has been an important mechanism for humans for as long as recorded history. I don't understand the wisdom of shooting down establishing systems to make that better, even if the direct applicability to a single human's life is remote. We are all part of a huge, interconnected system whether we like it or not, and accurate, synchronized timekeeping across the world does not sound nefarious to me.
> Timekeeping has been an important mechanism for humans for as long as recorded history.
And for 99% of that history, Noon was when the sun was half-way through its daily arc at whatever point on Earth one happened to inhabit. The ownership class are the ones who invented things like time zones to stop their trains from running in to each other, and NTP is just the latest and most-pervasive-and-invasive evolution of that same inhuman mindset.
From a privacy point of view, constant NTP requests are right up there alongside weather apps and software telemetry for “things which announce everyone's computers to the global spy apparatus”, feeding the Palantirs of the world to be able to directly locate you as an individual if need be.
> The ownership class are the ones who invented things like time zones to stop their trains from running in to each other
In a world where this didn't happen, your comment would most likely read:
> The ownership class are the ones who had such indifference toward the lives of the lower class passengers that they didn't bother stopping their trains from running into each other.
Far more things rely on reliable and accurate time-keeping than just being on time to work. Timekeeping is vitally important (even if it's not readily visible) to lots of critical infrastructure worldwide.
Actually, it's really important to me to have a network of atomic clocks available to verify the times I clock in and out, I want to make sure I get paid for an accurate duration of time down to the nanosecond
oh....no, not really, no, the world needs GPS, so, yeah. this is not like scrooge mcduck telling you to be at work on time. scrooge still has a windup watch
> In Iran, some 90 percent of the water abstracted from rivers and underground aquifers is taken for agriculture.
It's quite something they are envisioning a 100 billion dollar project to move the capital instead of limiting water waste in desert agriculture and closing Tehran's water loop by reclaiming sewage, greatly reducing the net demand: https://www.youtube.com/watch?v=WhCNpX3s-D8
Closed loop municipal water systems seem like such a good idea that just needs a little kickstart to become relatively self-sustaining from harvesting what is otherwise treated as waste.
On the other hand, LLMs are very bad writing partners, they are sycophants and very rarely give substantial criticism, the kind of feedback an editor would give and is mentioned in the article.
This is the substantial service an editor will provide going forward in the AI slop era, where everyone and their grandma will self publish some personal masterpiece: a contact with the real world and setting the bar high, to the point you need to struggle to achieve the required quality. Writing a book, especially finishing a great book, is not supposed to be enjoyable, it's hard, grueling work.