Interesting, but can't help but question if there's some way to leverage nerf to estimate the keypoints directly, rather than supervise with nerf. It seems a bit likely to be redundant.
Of course they won't. The investment in the Heavy Press Program was the initial build, and just citing one example, the Alcoa 50,000 ton forging press was built in 1955, operated until 2008, and needed ~$100M to get it operational again in 2012.
The investment was made to build the press, which created significant jobs and capital investment. The press, and others like it, were subsequently operated by and then sold to a private operator, which in turn enabled the massive expansion of both military manufacturing, and commercial aviation and other manufacturing.
The Heavy Press Program was a strategic investment that paid dividends by both advancing the state of the art in manufacturing at the time it was built, and improving manufacturing capacity.
A GPU cluster might not be the correct investment, but a strategic investment in increasing, for example, the availability of training data, or interoperability of tools, or ease of use for building, training, and distributing models would probably pay big dividends.
There may however, be a shortage of capital for open source AI, which is the subject under consideration.
As for the why... because there's no shortage of capital for AI. It sounds like the government would like to encourage redirecting that capital to something that's good for the economy at large, rather than good for the investors of a handful of Silicon Valley firms interested only in their own short term gains.
Look at it from the perspective of an elected official:
If it succeeds, you were ahead of the curve. If it fails, you were prudent enough to fund an investigation early. Either way, bleeding edge tech gives you a W.
Yeah. There is alot of over hyped and over funded nonsense that comes out of NASA. Some of it is hype from the marketing and press teams, other hype comes from misinterpretation of releases.
None of that changes that there have been major technical breakthroughs, and entire classes of products and services that didn't exist before those investments in NASA (see https://en.wikipedia.org/wiki/NASA_spin-off_technologies for a short list). There are 15 departments and dozens of Agencies that comprise the US Federal government, many of whom make investments in science and technology as part of their mandates, and most of that is delivered through some structure of public-private partnerships.
What you see as over-hyped and over-funded nonsense could be the next ground breaking technology, and that is why we need both elected leaders who (at least in theory) represent the will of the people, and appointed, skilled bureaucrats who provide the elected leaders with the skills, domain expertise, and experience that the winners of the popularity contest probably don't have.
Yep, there will be waste, but at least with public funds there is the appearance of accountability that just doesn't exist with private sector funds.
If it succeeds the idea gets sold to private corporations or the technology is made public and everyone thinks the corporation with the most popular version created it.
If it fails certain groups ensure everyone knows the government "wasted" taxpayer money.
> A GPU cluster might not be the correct investment, but a strategic investment in increasing, for example, the availability of training data, or interoperability of tools, or ease of use for building, training, and distributing models would probably pay big dividends
Would you mind expanding on these options? Universal training data sounds intriguing.
Sure, just on the training front, building and maintaining a broad corpus of properly managed training data with metadata that provides attribution (for example, content that is known to be human generated instead of model generated, what the source of data is for datasets such as weather data, census data, etc), and that also captures any licensing encumbrance so that consumers of the training data can be confident in their ability to use it without risk of legal challenge.
Much of this is already available to private sector entities, but having a publicly funded organization responsible for curating and publishing this would enable new entrants to quickly and easily get a foundation without having to scrape the internet again, especially given how rapidly model generated content is being published.
I think the EPC (energy performance certificate) dataset in the UK is a nice example of this. Anyone can download a full dataset of EPC data from https://epc.opendatacommunities.org/
Admittedly it hasn't been cleaned all that much - you still need to put a bit of effort into that (newer certificates tend to be better quality), but it's very low friction overall. I'd love to see them do this with more datasets
If the public is going to go to all the trouble of doing something, why would that public not make it clear that there is no legal threat to using any data available?
The public is incredibly lazy, though. Don't expect them to do anything until their hand is forced, which doesn't bode well for the action to meet a desirable outcome.
> Doubtful that GPUs purchased today would be in use for a similar time scale
Totally agree. That doesn't mean it can't generate massive ROI.
> Govt investment would also drive the cost of GPUs up a great deal
Difficult to say this ex ante. On its own, yes. But it would displace some demand. And it could help boost chip production in the long run.
> Not sure why a publicly accessible GPU cluster would be a better solution than the current system of research grants
Those receiving the grants have to pay a private owner of the GPUs. That gatekeeping might be both problematic, if there is a conflict of interests, and inefficient. (Consider why the government runs its own supercomputers versus contracting everything to Oracle and IBM.)
No. But I do want to limit the amount we reward NVIDIA for calling the shots correctly to maximize the benefit to society. For instance by reducing the duration of the government granted monopolies on chip technology that is obsolete well before the default duration of 20 years is over.
That said, it strikes me that the actual limiting factor is fab capacity not nvidia's designs and we probably need to lift the monopolies preventing competition there if we want to reduce prices.
> Why do you think these private entities are willing to invest the massive capital it takes to keep the frontier advancing at that rate?
Because whether they make 100x or 200x they make a shitload of money.
> Why wouldn't NVIDIA be a solid steward of that capital given their track record?
The problem isn't who is the steward of the capital. The problem is that economically efficient thing to do for a single company is (given sufficient fab capacity, and a monopoly) to raise prices to extract a greater share of the pie at the expense of shrinking the size of the pie. I'm not worried about who takes the profit, I'm worried about the size of the pie.
> Because whether they make 100x or 200x they make a shitload of money.
It's not a certainty that they 'make a shitload of money'. Reducing the right tail payoffs absolutely reduces the capital allocated to solve problems - many of which are risky bets.
Your solution absolutely decreases capital investment at the margin, this is indisputable and basic economics. Even worse when the taking is not due to some pre-existing law, so companies have to deal with the additional uncertainty of whether & when future people will decide in retrospect that they got too large a payoff and arbitrarily decide to take it from them.
You can't just look at the costs to an action, you also have to look at the benefits.
Of course I agree I'm going to stop marginal investments from occurring into research into patent-able technologies by reducing the expect profit. But I'm going to do so very slightly because I'm not shifting the expected value by very much. Meanwhile I'm going to greatly increase the investment into the existing technology we already have, and allow many more people to try to improve upon it, and I'm going to argue the benefits greatly outweigh the costs.
Whether I'm right or wrong about the net benefit, the basic economics here is that there are both costs and benefits to my proposed action.
And yes I'm going to marginally reduce future investments because the same might happen in the future and that reduces expected value. In fact if I was in charge the same would happen in the future. And the trade-off I get for this is that society gets the benefit of the same actually happening in the future and us not being hamstrung by unbreachable monopolies.
> But I'm going to do so very slightly because I'm not shifting the expected value by very much
I think you're shifting it by a lot. If the government can post-hoc decide to invalidate patents because the holder is getting too successful, you are introducing a substantial impact on expectations and uncertainty. Your action is not taken in a vacuum.
> Meanwhile I'm going to greatly increase the investment into the existing technology we already have, and allow many more people to try to improve upon it, and I'm going to argue the benefits greatly outweigh the costs.
I think this is a much more speculative impact. Why will people even fund the improvements if the government might just decide they've gotten too large a slice of the pie later on down the road?
> the trade-off I get for this is that society gets the benefit of the same actually happening in the future and us not being hamstrung by unbreachable monopolies.
No the trade-off is that materially less is produced. These incentive effects are not small. Take for instance, drug price controls - a similar post-facto taking because we feel that the profits from R&D are too high. Introducing proposed price controls leads to hundreds of fewer drugs over the next decade [0] - and likely millions of premature deaths downstream of these incentive effects. And that's with a policy with a clear path towards short-term upside (cheaper drug prices). Discounted GPUs by invalidating nvidia's patents has a much more tenuous upside and clear downside.
You have proposed state ownership of all successful IP. That is a massive change and yet you have demonstrated zero understanding of the possible costs.
Your claim that removing a profit motivation will increase investment is flat out wrong. Everything else crumbles from there.
No, I've proposed removing or reducing IP protections, not transferring them to the state. Allowing competitors to enter the market will obviously increase investment in competitors...
This is already happening - its called China.
There's a reason they don't innovate in anything, and they are always playing catch-up, except in the art of copying (stealing) from others.
I do think there are some serious IP issues, as IP rules can be hijacked in the US, but that means you fix those problems, not blow up IP that was rightfully earned
> That said, it strikes me that the actual limiting factor is fab capacity not nvidia's designs and we probably need to lift the monopolies preventing competition there if we want to reduce prices.
Lol it's not "monopolies" limiting fab capacity. Existing fab companies can barely manage to stand-up a new fab in different cities. Fabs are impossibly complex and beyond risky to fund.
It's the kind of thing you'd put government money to making but it's so risky government really don't want to spend billions and fail so they give existing companies billions so if they fail it's not the governments fault.
So, if a private company is successful, you will nationalize its IP under some guise of maximizing the benefit to society? That form of government was tried once. It failed miserably.
Under your idea, we’ll try a badly broken economic philosophy again. And while we’re at it, we will completely stifle investment in innovation.
there is no such thing as a lump-sum transfer, this will shift expectations and incentives going forward and make future large capital projects an increasingly uphill battle
There was a post[0] on here recently about how the US went from producing woefully insufficient numbers of aircraft to producing 300k by the end of world war 2.
One of the things that the post mentioned was the meager profit margin that the companies made during this time.
But the thing is that this set the America auto and aviation industry up to rule the world for decades.
A government going to a company and saying 'we need you to produce this product for us at a lower margin thab you'd like to' isn't the end of the world.
I don't know if this is one of those scenarios but they exist.
They are an intellectual property company holding the rights on plans to make graphic cards, not even a company actually making graphic cards.
The government could launch an initiative "OpenGPU" or "OpenAI Accelerator", where the government orders GPUs from TSMC directly, without the middleman.
It may require some tweaking in the law to allow exception to intellectual property for "public interest".
y'all really don't understand how these actions would seriously harm capital markets and make it difficult for private capital formation to produce innovations going forward.
If we have public capital formation, we don’t necessarily need private capital. Private innovation in weather modelling isn’t outpacing government work by leaps and bounds, for instance.
because it is extremely challenging to capture the additional value that is being produced by better weather forecasts and generally the forecasts we have right now are pretty good.
private capital is absolutely the driving force for the vast majority of innovations since the beginning of the 20th century. public capital may be involved, but it is dwarfed by private capital markets.
It’s challenging to capture the additional value and the forecasts are pretty good because of continual large-scale government investment into weather forecasting. NOAA is launching satellites! it’s a big deal!
Private nuclear research is heavily dependent on governmental contracts to function. Solar was subsidized to heck and back for years. Public investment does work, and does make a didference.
I would even say governmental involvement is sometimes even the deciding factor, to determine if research is worth pursuing. Some major capital investors have decided AI models cannot possibly gain enough money to pay for their training costs. So what do we do when we believe something is a net good for society, but isn’t going to be profitable?
They said remove legally-enforced monopolies on what they produce. Many of these big firms made their tech with millions to billions of taxpayer dollars at various points in time. If we’ve given them millions, shouldn’t we at least get to make independent implementations of the tech we already paid for?
> Those receiving the grants have to pay a private owner of the GPUs.
Along similar lines, I'm trying to build a developer credits program where I get whomever (AMD/Dell) to purchase credits on my super computers, that we then give away to developers to build solutions, which drives more demand for our hardware, and we commit to re-invest those credits back into more hardware. The idea is to create a win-win-win (us, them, you) developer flywheel ecosystem. It isn't a new idea at all, Nvidia and hyperscalers have been doing this for ages.
A much better investment would be to (somehow) revolutionize production of chips for AI so that it's all cheaper, more reliable, and faster to stand up new generations of software and hardware codesign. This is probably much closer to the program mentioned in the top level comment: It wasn't to produce one type of thing, but to allow better production of any large thing from lighter alloys.
> Not sure why a publicly accessible GPU cluster would be a better solution than the current system of research grants.
You mean a better solution than different teams paying AWS over and over, potentially spending 10x on rent rather than using all that cash as a down payment on actually owning hardware? I can't really speak for the total costs of depreciation/hardware maintenance but renting forever isn't usually a great alternative to buying.
In Canada, all three major AI research centers use clusters created with public money. These clusters receive regular additional hardware as new generations of GPUs become available. Considering how these institutions work, I'm pretty confident they've considered the alternatives (renting, AWS, etc). So that's one data point.
sure, I’ll hand it over after you spend your own time first to show that everything everywhere that’s owned instead of leased is a poor financial decision.
I cite how science permits plausibility upon evidence. This is in contrast to OP's wishful dogma, where anything not presently known must be fantasy/sci-based and implausible - which is not how science works.
I didn't say Martians can't be real. I said the evidence that I've encountered (obviously limited by the fact that I'm not a subject matter expert) is insufficient to overcome the threshold of doubt. I then theorized (uncharitably, I'll admit) that many people are predisposed to believe in martians because they've read too much science fiction stories which themselves are actually just metaphors for conditions of life here on Earth. I then made fun of these people, lightly I think.
They likely would not be in this position if they weren’t abusing it for an egregious fee. If they were noble, they wouldn’t have the fee. I think it’s more about holding profits captive than anything else.
Bottom 25th percentile has been zero or negative growth at least back to 1998. This would include minimum wage and support the original argument. This doesn't seem to correct for inflation, so zero or negative wage growth is punished completely by inflation.
Sure but not everyone stays in the bottom 25%. College students would be one category that you would expect this to be true of. I was certainly in the poverty level in college, but not anymore.
What we really care about are the people who are persistently in poverty and whether or not that number grows or shrinks. If it shrinks the economy is helping real people, if it's not, it's helping the well off.
So you are arguing that there is increasing mobility?
Or do you argue that people are bouncing between jobs with lower wage growth and higher wage growth, so the bottom 25% of wages are getting worse and worse, yet they are only temporary?
It seems pretty clear that 25% of earners experience no growth or negative growth YoY while 25% of earners see wage growth that outpaces inflation. Unless they are swapping places constantly, it's not really healthy for this to occur over a long period of time (as it has).
Are you sure that this chart shows bottom 25% of wages and not the bottom 25% of wage change?
They provide this definition: Wage Growth Tracker is a measure of the nominal wage growth of individuals. It is constructed using microdata from the Current Population Survey (CPS), and is the median percent change in the hourly wage of individuals observed 12 months apart.
Chart 1 plots the time series of the median, along with the mean, and the 75th and 25th percentiles of the individual wage growth distribution
I believe this chart will almost always show little or negative numbers for the bottom 25% of wage growth. By definition.
Are you arguing there's not increasing mobility, with this new fangled internet and the series of tubes and what have you? Upward mobility has never been more available across all income brackets.
Again... you want to look at the number of people in poverty. That's the true metric of an economy, not the unemployment number.
True metric of an economy is not just poverty rate.
I can have 3x the income of monthly food budget (poverty threshold). What happens when housing per month is also 3x the cost of food per month?
Poverty threshold is a poor indicator of the economic health. Poverty can lower while the ability for the average citizen to support a family continues to drop.
> Are you arguing there's not increasing mobility, with this new fangled internet and the series of tubes and what have you?
That's not an argument. Technological improvements can result in consolidation of capital and rising inequality. Something like that would be indicated by... top 25% of wage earners outpacing inflation while bottom 25% continue to stagnate or fall behind.
EDIT: The last poverty rate publication showed the supplemental poverty rate rising (taking into account more factors that traditional poverty rate). Previous to the pandemic, they were largely correlated. We will have to wait until the 2024 results to see if the inversion holds true.
One way to think of an economy is GDP/stock market/etc. Another way to think of it is: "What is the maximum level of poverty before there's a revolution?" Moving people from the poverty class to the middle class should be a basic goal for any economy -- because it's likely to affect the most citizens.
> That's not an argument.
So if you're in small town Nebraska in 1910/20/30/40/50/60/70/80/90's, it was super hard to learn calculus (one example) unless you enrolled in college and traveled to the university/college, etc. Why? No one else knew it, nor could they teach it.
Back before 2000's era, people were held back, mostly based upon where they were born. In the internet era, you can be a millionaire/billionaire without having to travel to the shit hole that is silicon valley. Amazing!
Edit:
I saw your edit, and yes, you would expect poverty to go up in a down turn economy. But you would also expect it to go down if economic gains are truly affecting everyone, and not just the rich or upper middle class.
I don’t think college students would be counted in this metric because it only counts people who want a full time job. A full time college student likely does not.
This is basically what HOOBS markets. They've really fallen to the movement of open source though.
I think there's still a world in which you can buy an rPi with software preinstalled and updaters packaged without much effort though.
If that could take off, you could easily have this type of service. However, I think it falls apart because you need to govern the app store and that is expensive and difficult.
