I've always seen the breathless Singularitarian worrying about AI Alignment as a smokescreen to distract people from thinking clearly about the more pedestrian hazards of AI that isn't self-improving or superhuman, from algorithmic bias, to policy-washing, to energy costs and acceleration of wealth concentration. It also leads to so-called longtermism - discounting the benefits of solving current real problems and focusing entirely on solving a hypothetical one that you think will someday make them all irrelevant.
My feeling has been that it’s a lot of people that work on B2B SaaS that are sad they hadn’t gotten the chance to work on the Manhattan Project. Be around the smartest people in your field. Contribute something significant (but dangerous! And we need to talk about it!) to humanity. But yeah computer science in the 21st century has not turned out to be as interesting as that. Maybe just as important! But Jeff Bezos important, not Richard Feynman important.
The Singularitarians were breathlessly worrying 20+ years ago, when AI was absolute dogshit - Eliezer once stated that Doug Lenat was incautious in launching Eurisko because it could've gone through a hard takeoff. I don't think it's just an act to launder their evil plans, none of which at the time worked.
Fair. OpenAI totally use those arguments to launder their plans, but that saga has been more Silicon Valley exploiting longstanding rationalist beliefs for PR purposes than rationalists getting rich...
Eliezer did once state his intentions to build "friendly AI", but seems to have been thwarted by his first order reasoning about how AI decision theory should work being more important to him than building something that actually did work, even when others figured out the latter bit.
From my experience people who worry about existential AI risk tend to worry more about the more mundane problems than the wider population. In particular they have been very vocal about scamming, cube security and also wealth (& power!) concentration.
>s a smokescreen to distract people from thinking clearly about the more pedestrian hazards of AI that isn't self-improving or superhuman,
Anything that can't be self-improving or superhuman almost certainly isn't worthy of the moniker "AI". A true AI will be born into a world that has already unlocked the principles of intelligence. Humans in that world would be capable themselves of improving AI (slowly), but the AI itself will (presumably) run on silicon and be a quick thinker. It will be able to self-improve, rapidly at first, and then more rapidly as its increased intelligence allows for even quicker rates of improvement. And if not superhuman initially, it would soon become so.
We don't even have anything resembling real AI at the moment. Generative models are probably some blind alley.
> We don't even have anything resembling real AI at the moment. Generative models are probably some blind alley.
I think that the OP's point was that it doesn't matter whether it's "real AI" or not. Even if it's just a glorified auto-correct system, it's one that has the clear potential to overturn our information/communication systems and our assumptions about individuals' economic value.
I think when the GP says "our assumptions about individuals' economic value." they mean half the workforce becoming unemployed because the auto corrector can do it cheaper.
That's going to be a swift kick to your economy, no matter how strong.
If the "autocorrector" can do anything that used to be a paid wage/salary "cheaper", this isn't the case of another job being automated away. Either something is grossly wrong in that some are willing to have a stupid machine do that poorly (writing Sports Illustrated articles), or something was always grossly wrong in that we were paying someone to do something that never did matter (paying someone to write SEO spam). This isn't a swift kick in the economy, this is a "your economy was so weak a soft breeze knocked it over".
> Either something is grossly wrong in that some are willing to have a stupid machine do that poorly (writing Sports Illustrated articles), or something was always grossly wrong in that we were paying someone to do something that never did matter
I recently had an LLM write a function for me that, for a given RGB color value and another integer n > 1, returned to me a list of n RGB colors equidistantly and sequentially spaced around the color wheel starting at the specified RGB value.
For a given system I'm creating, I might have lots of such tasks. That collection of tasks is something that did matter and took some education and skill to complete well.
In the pre-LLM world - assuming I was too busy to handle all the tasks myself - I would have delegated them to a junior software engineer.
In a post-LLM world, I just ask the LLM to implement tasks like that, and I review the code for correctness.
That seems like a pretty transformational change to me, and not just some kind of "rot" being removed from the process.
I actually think the people developing AI might well not get rich off it.
Instead, unless there's a single winner, we will probably see the knowledge on how to train big LLMs and make them perform well diffuse throughout a large pool of AI researchers, with the hardware to train models reasonably close to the SotA becoming more quite accessible.
I think the people who will benefit will be the owners of ordinary but hard-to-dislodge software firms, maybe those that have a hardware component. Maybe firms like Apple, maybe car manufacturers. Pure software firms might end up having AI assisted programmers as competitors instead, pushing margins down.
This is of course pretty speculative, and it's not reality yet, since firms like Cursor etc. have high valuations, but I think this is what you'd get from the probably pressure if it keeps getting better.
It smacks of a goldrush. The winners will be the people selling shovels (nVidia) and housing (AWS). It may also be the guides showing people the mountains (Cursor, OpenAI, etc).
I suspect you'll see a few people "win" or strike it rich with AI, the vast majority will simply be left with a big bill.
When railroads were first being built across the continental USA, those companies also had high valuations (for the time). Most of them ultimately went bankrupt or were purchased for a fraction of their peak valuation. But the tracks remained, and many of those routes are still in use today.
And today the railroad system in the USA sucks compared to other developed countries and even China.
It turns out that boom-and-bust capitalism isn’t great for building something that needs to evolve over centuries.
Perhaps American AI efforts will one day be viewed similarly. “Yeah, they had an early rush, lots of innovation, high valuations, and robber barons competing. Today it’s just stale old infra despite the high-energy start.”
I think it's unlikely that AI efforts will go as railroads have. I think being an AI foundation model company is more like being an airplane builder than like a railway company, since you develop your technology.
Plenty of those that similarly went bankrupt over the years, and now the USA mostly has Boeing that's reached a set of continual crises and being propped up by the government.
Yes, of course, but they usually live for a long time.
Boeing tries to compete with Airbus, where the engineers make maybe 70% of what they do in the US, and has upon that made some really bad decisions. Many aircraft companies are profitable.
America's passenger rail sucks, it couldn't compete with airplanes and every train company got out of the business, abandoning it to the government. But America does have a great deal of freight rail which sees a lot of use (much more than in Europe, I don't know how it compares to China though.)
One reason the passenger service sucks is that the freight rail companies own the tracks, and are happy to let a passenger train sit behind a freight train for a couple hours waiting for space in the freight yard so it can get out of the way.
Humans are also freight, of course. It is not like the rail companies really care about what kind of fright is on the trains, so long as it is what the customer considers most important (read: most profitable). Humans are deprioritized exactly because they aren't considered important by the customer, which is to say that the customer, who is also the freight in this case, doesn't really want to be on a train in the first place. The customer would absolutely ensure priority (read: pay more, making it known that they are priority) if they wanted to be there.
I understand the train geeks on the internet find it hard to believe that not everyone loves trains like they do, but the harsh reality is that the average American Joe prefers other means of transportation. Should that change in the future, the rail network will quickly accommodate. It has before!
The root cause is Americans, generally, prefer any mode of transit other than rail, so passenger rail isn't profitable, so train companies naturally prioritize freight.
For what it's worth, I like traveling by train and do so whenever I can, but I'm an outlier. Most Americans look at the travel times and laugh at the premise of choosing a train over a plane. And when I say they look at the travel times, I don't mean they actually bother to look up train routes. They just know that airplanes are several times faster. Delays suffered by trains never get factored into the decision because trains aren't taken seriously in the first place.
China hasn't shown that their railroad buildout will work. My understanding is they currently aren't making enough return to payoff debt, yet alone plan for future maintenance. Historically the command economy type stuff looks great in the early years, it's later on we see if that is reality.
You are comparing USA today to the robber baron phase, whose to say China isn't in the same phase? Lots of money being thrown at new railroads and you have Chinese leaders and best and management leaders chasing that money. When happens when it goes low budget/maintenance mode?
The USA today is in a robber baron phase. We only briefly left it for about 2 generations due to the rise of labor power in the late 1800s/early 1900s. F.D.R. was the compromise president put into place to placate labor and prevent a socialist revolution.
> And today the railroad system in the USA sucks compared to other developed countries and even China.
Nonsense. The US has the largest freight rail system in the world, and is considered to have the most efficient rail system in the world to go along with it.
There isn't much in the way of passenger service, granted, but that's because people in the US aren't, well, poor. They can afford better transportation options.
> It turns out that boom-and-bust capitalism isn’t great for building something that needs to evolve over centuries.
It initially built out the passenger rail just fine, but then evolution saw better options come along. Passenger rail disappeared because it no longer served a purpose. It is not like, say, Japan where the median household income is approaching half that of Mississippi and they hold on to rail because that's what is affordable.
> There isn't much in the way of passenger service, granted, but that's because people in the US aren't, well, poor. They can afford better transportation options.
This is so misguided view... Trains (when done right) aren't "for the poor", they are great transportation option, that beats both airplanes and cars. In Poland, which isn't even close to the best, you can travel between big cities with speeds above 200km/h, and you can use regional rail for your daily commute, both those options being very comfortable and convenient, much more convenient than traveling by car.
Poland is approximately the same geographical size as Nevada. In the US, "between cities" is more like New York to Las Vegas, not Las Vegas to... uh, I couldn't think of another city in Nevada off the top of my head. What under-serviced route were you thinking of there?
What gives you the idea that rail would be preferable to flying for the NYC to LAS route if only it existed? Even as the crow flies it is approximately 4,000 km, meaning that at 200 km/h you are still looking at around 20 hours of travel in an ideal case. Instead of just 5 hours by plane. If you're poor an additional 15 hours wasted might not mean much, but when time is valuable?
> In the US, "between cities" is more like New York to Las Vegas, not Las Vegas to... uh, I couldn't think of another city in Nevada off the top of my head. What under-serviced route were you thinking of there?
Why would you constrain the route to within a specific state? In fact, right now a high-speed rail line is being planned between Las Vegas and LA.
But outside of Nevada, there are many equivalent distance routes in the US between major population centers, including:
> In fact, right now a high-speed rail line is being planned between Las Vegas and LA.
Right now and since 1979!
I'll grant you that people love to plan, but it turns out that they don't love putting on their boots and picking up a shovel nearly as much.
> But outside of Nevada, there are many equivalent distance routes in the US between major population centers, including
And there is nothing stopping those lines from being built other than the lack of will to do it. As before, the will doesn't exist because better options exist.
There are a lot more obstacles than lack of will. There are also property rights, environmental reviews, availability of skilled workers, and lack of capital. HN users sometimes have this weird fantasy that with enough political will it's possible to make enormous changes but that's simply not how things operate in a republic with a dual sovereignty system.
> There are also property rights, environmental reviews, availability of skilled workers, and lack of capital.
There is no magic in this world like you seem to want to pretended. All of those things simply boil down to people. Property rights only exist because people say they do, environmental reviews only exist because people say they do, skilled workers are, well, literally people, and the necessary capital is already created. If the capital is being directed to other purposes, it is only because people decided those purposes are more important. All of this can change if the people want it to.
> HN users sometimes have this weird fantasy that with enough political will it's possible to make enormous changes but that's simply not how things operate in a republic with a dual sovereignty system.
Hell, the republic and dual sovereignty system itself only exists because that's what people have decided upon. Believe it or not, it wasn't enacted by some mythical genie in the sky. The people can change it all on a whim if the will is there.
The will isn't there of course, as there is no reason for the will to be there given that there are better options anyway, but if the will was there it'd be done already (like it already is in a few corners of the country where the will was present).
There has been continuous regularly scheduled passenger service between Chicago and Detroit since before the Civil War. The current Amtrak Wolverine runs 110 MPH (180 KPH) for 90% of the route, using essentially the same trainset that Brightline plans to use.
Fair point. Last time I took that train (mid 1990s) it didn't run to Pontiac or Troy, and I recall there being very infrequent service. A far as I know, it's not the major mode of passenger transit between Detroit and Chicago. Cars are. That might be because of the serious lack of last-mile transit connectivity in the Detroit area.
Cars are definitely the major mode. Lots of quick flights, too.
They’ve made a lot of investments since the 1990s. It’s much improved, though perhaps not as nice as during the golden years when it was a big part of the New York Central system (from the 1890s to the 1960s they had daily trains that went Boston/NYC/Buffalo/Detroit/Chicago through Canada from Niagara Falls to Windsor).
During the first Trump administration, Amtrak announced a route that would go Chicago/Detroit/Toronto/Montreal/Quebec City using that same rail tunnel underneath the Detroit River. It was supposed to start by 2030. We’ll see if it happens.
Also, if you go all in and build something equivalent to Chinese bullet trains (that go with speeds up to 350km/h) you could do for example NY to Chicago in 3.5 hours, or even NY to Miami in 6 hours :-D (I know, not very realistic)
Not sure how we got from Scott A being a rationalist to trains, but since we're here, I want to say:
I've taken a Chinese train from Zhengzhou, in central China, to Shenzhen, and it was fantastic. Cheap, smooth, fast, lots of legroom, easy to get on and off or walk around to the dining car. And, there's a thing where boiling hot water is available, so everyone brings instant noodle packs of every variety to eat on the train.
Can't even imagine what the US would be like if we had that kind of thing.
Similar experience here. I'd always prefer it to flying.
Getting to the airport in most major cities takes an hour, and then there's the whole pre-flight security theatre, and the flights themselves are rarely pleasant. To add insult to injury, in the US it's usually a $50 cab ride to the airport and there are $28 ham-and-cheese sandwiches in the terminal if you get hungry.
In China and Japan the trains are centrally located, getting on takes ten minutes, and the rides are extremely comfortable. If such a thing existed in the US I think it would be extremely popular. Even if it was just SF-LA-Vegas.
> Getting to the airport in most major cities takes an hour
Do you mean beginning in the same city? If so, that's downright hilarious. I live 50 miles clear of the city, out in the middle of nowhere, and can be to the airport in the city in less than an hour.
> If such a thing existed in the US I think it would be extremely popular.
I don't know, if people willingly spend an hour getting from one point to another in the same city, the aren't apt to be concerned about how they get from one city to another. I expect they don't put much thought into anything.
If you can't think of another city in Nevada off the top of your head, are you even American? (Reno.)
Anyway, New York to Las Vegas spans most of the US. There are plenty of routes in the US where rail would make sense. Between Boston, New Haven, New York City, Philadelphia, Baltimore, and Washington, D.C. Which has the Amtrak Acela. Or perhaps Miami to Orlando. Which has a privately funded high speed rail connection called Brightline that runs at 200 km/h who's ridership was triple what had been expected at launch.
> Which has a privately funded high speed rail connection called Brightline that runs at 200 km/h
Which proves that when the will is there, it will be done. The only impediment in other places is simply the people not wanting it. If they wanted it, it would already be there.
The US has been here before. It built out a pretty good, even great, passenger rail network a couple of centuries ago when the people wanted it. It eventually died out simply because the people didn't want it anymore.
If they want it again in the future, it will return. But as for the moment...
This analysis seems wrong for at least a couple of reasons.
1. Freight is easier to manage and has better economics on a dedicated network. The US freight network is extremely efficient as others have pointed out. Other networks, e.g., Germany, instead prioritized passenger service. In Germany rail moves a small proportion of freight (19%) compared to trucks. [0] It's really noticeable on the Autobahn and unlike the US where a lot of truck traffic is intermodal loads.
2. The US could have better rail service by investing in passenger networks. Instead we have boondoggles like the California high-speed rail project which has already burned through 10s of billions of dollars with no end in sight. [1] Or the New Jersey Transit system which I had the pleasure to ride on earlier today to Newark Airport. It has pretty good coverage but needs investment.
Yeah the best in the world freight industry that is being sued by every company in America for their decades long price fixing scandal. Cant find the list of plaintiffs right now but it had ford and frito lay and hundreds of others. An update from yesterday:
It's not that it would be terrible, but in the real world people are generally lazy and will only do what they actually want to see happen. Surprisingly, we don't yet have magical AI robots that autonomously go around turning all imagined ideas into reality without the need for human grit.
Since nobody really wants passenger rail in the US, they don't put in the effort to see that it exists (outside of some particular routes where they do want it). In many other countries, people do want board access to passenger rail (because that's all they can afford), so they put in the effort to have it.
~200 years ago the US did want passenger rail, they put in the work to realize it, and it did have a pretty good passenger rail network at the time given the period. But, again, better technology came along, so people stopped maintaining/improving what was there. They could do it again if they wanted to... But they don't.
They won't be allowed to use it unless they serve the capitalists who own it.
It's not social media. It's a model the capitalists train and own. Best the rest of us will have access to are open source ones. It's like the difference between trying to go into court backed by google searches as opposed to Lexis/Nexis. You're gonna have a bad day with the judge.
Here's hoping the open source stuff gets trained on quality data rather than reddit and 4chan. Given how the courts are leaning on copyright, and lack of vetted data outside copyright holder remit, I'm not sanguine about the chances of parity long term.
To be fair, Musk is probably the least subtle megabillionaire of them all, and that reflects in the odd behavior of his silicon child. I don't doubt in the competency of the likes of Thiel to build their techo-monarchy.
The problem for them is that it might be a rather fundamental limitation. We already know that RLHF makes models dumber. It is entirely possible that, in order to make the model buy fully into what those people are peddling, the amount of forceful training required would crater the model's overall performance.
