> Yes the team which literally created transformer and almost all the important open research including Bert, T5, imagen, RLHF, ViT don’t have the ability to execute on AI /s.
This, but non-sarcastically. Google has spectacularly, so far, failed to execute on products (even of the “selling shovels” kind, much less end-user products) for generative AI, despite both having lots of consumer products to which it is naturally adaptable and a lot of the fundamental research work in generative AI.
The best explanation is that they actually are, institutionally and structurally, bad at execution in this domain, because they have all the pieces and incentives that rule out most of the other potential explanations for that.
> OpenAI bought into the field. They are good at execution but i havent seen anything novel coming out of them.
Right, OpenAI is good at execution (at least, when it comes to selling-shovels tools, I don’t see a lot of evidence beyond that yet), whereas Google is, to all current evidence, not good at execution in this space.
They're getting Innovator's Dilemma'd, the same way that Bell Labs, DEC, and Xerox did. When you have an exceptionally profitable monopoly, it biases every executive's decision-making toward caution. Things are good; you don't want to upset the golden goose by making any radical moves; and so when your researchers come out with something revolutionary and different you bury it, maybe let them publish a few papers, but certainly don't let it go to market.
Then somebody else reads the papers, decides to execute on it, and hires all the researchers who are frustrated at discovering all this cool stuff but never seeing it launch.
The typical solution to this (assuming there is one internally) is setting up a sub-company and keeping the team isolated from the parent company aka "intrapenuership" but also keeping them well resourced by the parent.
It seems like that's what they were doing with DeepMind for the last decade. But it's also possible DeepMind as an institution lacked the pressure/product sense/leadership to produce consumable products/services. Maybe their instincts were more centered around R&D and being isolated left them somewhat directionless?
So now that AI suddenly really matters as a business, not just some indefinite future potential, Google wants to bring them inside.
They could have created a 3rd entity, their own version of OpenAI, combining DeepMind with some Google management/teams and other acquisitions and spinning it off semi-independently. But this play basically has to be from Google itself for their own reputation's sake - maybe not for practicality's sake but politically/image-wise.
Yeah. It doesn't really work all that well. Xerox tried it with Xerox PARC, Digital with Western Digital, AT&T with Bell Labs, Yahoo with Yahoo Brickhouse, IBM with their PC division, Google with Google X & Alphabet & DeepMind, etc.
Being hungry and scrappy seems to be a necessary precondition for bringing innovative products to market. If you don't naturally come from hungry & scrappy conditions (eg. Gates, Zuckerburg, Bezos, PG), being in an environment where you're surrounded by hungry & scrappy people seems to be necessary.
For that matter, a number of extremely well-resourced startups (eg Color, Juicero, WebVan, Secret, Pets.com, Theranos, WeWork) have failed in spectacular ways. Being well-resourced seems to be an anti-success criteria even for independent companies.
That may have been true in the 70's and 80's. However, I worked for a 2000 person (startup) software company in the 90's that was acquired at 1.8B, another 4000 person (startup) software company in the 90's that was acquired at 3.4B, and then a few years ago, the acquirer of both was itself acquired for 18B.
I survived ALL the layoffs somehow. Boots on the ground agrees with "doesn't really work all that well" but the people collecting rents keep collecting. Given the size all of these received significant DOJ reviews though the only detail I remember is basketball sized court rooms filled with printed paper for the depositions. I'm sure they burned down the Amazon to print all that legalese, speaking of scaling problems.
edit: i take it all back! my memory is not as good as i thought it was re: software companies. i will leave up my sorry list as penance for my crappy recent tech history skills.
Thanks for the comment. Chortle. That's hilarious.
Indeed, you are right on: Legent, Platinum, CA, and Broadcom in order from little fish to big. CA was the second largest software company in the world behind Microsoft then.
The weird part you couldn't see from this telling is that I worked in the Legent office in Pittsburgh, moved to Boston post-CA acquisition and worked in the CA office in Andover. Resigned and went to Platinum in Burlington. Moved to Seattle. Second CA acquisition in 5 years. I should have quit while I was ahead. Moved back to Pittsburgh. Worked in the exact same office I'd worked in 5 years earlier with the same crew. Weird feeling is a mild understatement. I still know people who work for Broadcom now. I should reach out.
i used to read BYTE mag over in the UK in the early 90s before i moved to USA; CA was such a heavy hitter in the early 90s!! i guess it never really was the same in the post-Wang era(s).
The problem with the intrapreneurship idea is that it's really hard to beat desperation as a motivator. I have seen people behave very differently in the context of a startup vs a corporate research lab thanks to this dynamic. Some people thrive in the corporate R&D environment, but the innovator's dilemma eventually gets to their managers.
Cisco has done a great job balancing this, actually - they keep contact with engineers who leave to do startups, and then acquire their companies if they become successful enough to prove the product.
After a bunch of ex-Cisco people ate Cisco’s core router lunch at Juniper, Cisco vowed it would never happen again. Until a bunch of ex-Cisco people ate WebEx’s lunch at Zoom.
Getting a big seed round once makes you want that next round to keep going (and take even more money off the table).
Getting a X-million-per-year budget from a parent company gives you a very different sort of situation. IME this results in less urge to get something out the door and more urge to get "the best thing" built. Shipping early risks your budget in a way that "look at all this cool theoretical progress" doesn't, because the public and press can critique you more directly.
Lack of major owner equity basically means few intrapreneur efforts will succeed unless the 'founder' really couldn't succeed without the daddy company
> But it's also possible DeepMind as an institution lacked the pressure/product sense/leadership to produce consumable products/services. Maybe their instincts were more centered around R&D and being isolated left them somewhat directionless?
It seems like this is more a Google problem than a DeepMind problem though, no? Google created one of the most successful R&D labs for ML/AI research the world has ever known, then failed to have their other business units capitalize on that success. OpenAI observed this gap and swooped in to profit off all of their research outputs (with backing from Microsoft).
IMO what they’re doing here is doubling down on their mistakes: instead of disciplining their other business units for failing to take advantage of this research, they’re forcing their most productive research team to assume responsibility and correct for those failures. I expect this will go about as well as any other instance of subjecting a bunch of research scientists to internal political struggles and market discipline, i.e. very poorly.
They're also paying for their product managers' cancellation culture. (Sorry.) I'm seeing a lot of AI pitch decks; none suggest trusting Google. That saps not only network effects, but what ill term earned research: work done by others on your product. Google pays for all its research and promotion. OpenAI does not.
Are researchers actually frustrated to never see it launch, or are they mostly focused on publishing papers?
I thought OpenAI’s unique advantage over many big tech companies is that they’ve somehow figured out how to fast track research into product, or have researchers much more willing to worry about “production”.
I’m puzzled that stuff like alpha Fold count for nothing in this discussion (having just browsed through most of it).
I saw quotes from independent scientists referring to it as the greatest breakthrough of their lifetime, and I saw similarly strong language used in regard to the potential for good of alpha fold as a product.
So they gave it away, but it is still a product they followed through on and continue to.
Was it wrong of them that they gave it away, and right, that Microsoft’s primary intent with their open AI technology, seems to be to provoke an arms race with google?
Alpha Fold is a game changer, but nowhere near the game changer ChatGPT(4) is, even if ChatGPT was only available for the subset of scientists that benefit from Alpha Fold. We are literally arguing semantics if this is AGI, and you're comparing it to a bespoke ML model that solves a highly specific domain problem (as unsolvable and impressive as it was).
> We are literally arguing semantics if this is AGI,
And if it isn't? Literally every single argument I've seen towards this being AGI is "We don't know at all how intelligence works, so let's say that this is it!!!!!"
> nowhere near the game changer ChatGPT(4) is, even if ChatGPT was only available for the subset of scientists that benefit from Alpha Fold
This is utter nonsense. For anyone who actually knows a field, ChatGPT generates unhelpful, plausible-looking nonsense. Conferences are putting up ChatGPT answers about their fields to laugh at because of how misleadingly wrong they are.
This is absolutely okay, because it can be a useful tool without being the singularity. I'd sure that in a couple of years time, most of what ChatGPT achieves will be in line with most of the tech industry advances in the past decade - pushing the bottom out of the labor market and actively making the lives of the poorest worse in order to line their own pockets.
I really wish people would stop projecting hopes and wishes on top of breathless marketing.
I asked GPT-4 to give me a POSIX compliant C port of dirbuster. It spit one out with instructions for compiling it.
I asked it to make it more aggressive at scanning and it updated it to be multi-threaded.
I asked it for a word list, and it gave me the git command to clone one from GitHub and the command to compile the program and run the output with the word list.
I then told it that the HTTP service I was scanning always returned 200 status=ok instead of a 404 and asked it for a patch file. It generated that and gave me the instructions for applying it to the program.
There was a bug I had to fix: word lists aren’t prefixed with /. Other than that one character fix, GPT-4 wrote a C program that used an open source word list to scan the HTTP service running on the television in my living room for routes, and found the /pong route.
This week it’s written 100% of the API code that takes a CRUD based REST API and maps it to and from SQL queries for me on a cloudflare worker. I give it the method signature and the problem statement, it gives me the code, and I copy and paste.
If you’re laughing this thing off as generating unhelpful nonsense you’re going to get blind sided in the next few years as GPT gets wired into the workflows at every layer of your stack.
> pushing the bottom out of the labor market and actively making the lives of the poorest worse in order to line their own pockets.
I’m in a BNI group and a majority of these blue collar workers have very little to worry about with GPT right now. Until Boston Dynamics gets its stuff together and the robots can do drywalling and plumbing, I’m not sure I agree with your take. This isn’t coming for the “poorest” among us. This is coming for the middle class. From brand consultants and accountants to software engineers and advertisers.
Software engineers with GPT are about to replace software engineers without GPT. Accountants with GPT are about to replace accountants without GPT.
> Literally every single argument I've seen towards this being AGI is
Here is one: it can simultaneously pass the bar exam, port dirbuster to POSIX compliant C, give me a list of competing brands for conducting a market analysis, get into deep philosophical debates, and help me file my taxes.
It can do all of this simultaneously. I can't find a human capable of the simultaneous breadth and depth of intelligence that ChatGPT exhibits. You can find someone in the upper 90th percentile of any profession and show that they can out compete GPT4. But you can't take that same person and ask them to out compete someone in the bottom 50th percentile of 4 other fields with much success.
Artificial = machine, check.
Intelligence = exhibits Nth percentile intelligence in a single field, check
General = exhibits Nth percentile intelligence in more than one field, check
Maybe it's heavily biased towards programming and computing questions? I've tested GPT-4 on numerous physics stuff and it fails spectacularly at almost all of them. It starts to hallucinate egregious stuff that's completely false, misrepresents articles it tries to quote as references etc. It's impressive as a glorified search engine in those cases but can't at all be trusted to explain most things unless they're the most canonical curriculum questions.
This extreme difficulty in discerning what it hallucinates and what is "true" is what it's most obvious problem is. I guess it can be fixed somehow but right now it has to be heavily fact-checked manually.
It does this for computing questions as well, but there is some selection bias so people tend to post the success-stories and not the fails. However it's less dangerous if it's in computing as you'll notice it immediately so maybe require less manual labour to keep it in check.
Hahaha, if you want nit-picking, all the language tasks chatGPT is good at are strictly human tasks. Not general tasks. Human tasks are all related to keeping humans alive and making more of us, they don't span the whole spectrum of possible tasks where intelligence could exist.
Of course inside language tasks it is as general as can be, yet still needs to be placed inside a more complex system with tools to improve accuracy, LLM alone is like brain alone - not that great at everything.
On the other hand if you browse around the web you will find various implementations of dirbuster, probably in C for sure in C++ which are multi-threaded , it’s not to take away from your experience but I mean, without knowing what’s in the training set it may have already been exposed to what you asked for, even several times over.
I have a feeling they had access to a lot of code on GH, who knows how much code they actually accessed. Copilot for a long time said it would use your code as training data, including context, if you didn’t opt out explicitly, so that’s already millions maybe hundreds of millions of lines of code scraped.
The conspiracy theorist in me wonders if MS just didn’t provide access to public and private code to train on, they wouldn’t have even told Open AI, just said, “here’s some nice data”, it’s all secret and we can’t see the models inputs so I’ll leave it at that. I mean they’ve obviously prepared the data for copilot, so it was there waiting to be trained on.
So yeah I feel your enthusiasm but if you think about it a little more, or maybe not so hard to imagine what you saw being actually rather simple ? Every time I write code I feel kind of depressed because I know almost certainly someone has already written the same thing and that it’s sitting in GitHub or somewhere else and I’m wasting my time.
ChatGPT just takes away the knowing where to find something (it’s already seen almost everything the average person can think of) you want and gives it to you directly. Have you never thought of this already ? Like you knew all the code you wanted already was there somewhere, but you just didn’t have an interface to get to it? I’ve thought about this for quite a while and I knew there would big data people doing experiments who could see that probably 80-90% of code on GitHub is pretty much identical.
> If you’re laughing this thing off as generating unhelpful nonsense you’re going to get blind sided in the next few years as GPT gets wired into the workflows at every layer of your stack.
Okay, now try being a scientist in a scientific field that isn't basic coding.
It's not people laughing at pretences, it's people who know even basic facts about their field literally looking at the output today and finding it deeply, fundamentally incorrect.
I do not believe that is a reasonable threshold for AGI. If it were, I believe a significant % of humans would individually fail to meet the threshold of AGI.
I wonder what your personal success rate would be if we did a Turing test with the “people” who “know basic facts about their field.” If they sat at a computer and asked you all these questions, would you get them right? Or would you end up in slide decks being held up as a reason why misnome doesn’t qualify as AGI?
I find comfort in knowing that it can’t “do science.” There is a massive amount of stuff it can do. I’m hopeful there will be stuff left for humans.
Maybe we’ll all be scientists in 10 years and I won’t have to waste my life on all this “basic coding” stuff.
Absolutely not! I created a powershell script for converting one ASM label format to another for retro game development and i used ChatGPT to write it. Now, it fumbled some of the basic program logic, however, it absolutely nailed all of the specific regex and obtuse powershell commands that i needed and that i merely described to it in plain English.
It essentially aced the "hard parts" of the script and i was able to take what it generated and make it fit my needs perfectly with some minor tweaking. The end result was far cleaner and far beyond what i would have been able to write myself, all in a fraction of the time. This ain't no breathless marketing dude: this thing is the real deal.
ChatGPT is an extremely powerful tool and an absolute game changer for development. Just because it is imperfect and needs a bit of hand holding (which it may not soon), do not underestimate it, and do not discount the idea that it may become an absolute industry disrupter in the painfully near future. I'm excited ...and scared
It does, quite often. Not only that, as you describe. But it does.
For example, I asked it what my most cited paper is, and it made up a plausible-sounding but non-existent paper, along with fabricated Google Scholar citation counts. Totally unhelpful.
Right, i think it's a question of how to use this tool in its current state, including prompting practice and learning its strengths. It can certainly be wrong sometimes, but man, it is already a game changer for writing, coding, and i'm sure other disciplines.
If you're a robotresearcher, maybe try getting it to whip up some ...verilog circuits or something? I don't know much about your field or what you do specifically, but tasks like regular expressions or specific code syntax it is absolutely brilliant at, whatever the equivalent to that is in hardware. ...I've only ever replaced capacitors and wired some guitar pickups.
> it made up a plausible-sounding but non-existent paper, along with fabricated Google Scholar citation counts
I ran into a similar issue: I asked it for codebases of similar romhacks to a project i'm doing, and it provided made up Github repos with completely unrelated authors for romhacks that do actually exist: non-existent hyperlinks and everything.
Now, studying the difference in GPT generations, it seems like more horsepower and more data solves alot of GPT problems and produces emergent capabilities with the same or similar architecture and code. The current data points to this trend continuing. I find it both super exciting and super ...concerning.
This seems like the perfect test, because it's something that does have information on the internet - but not infinite information, and you know precisely what is wrong about the answer.
> I'd sure that in a couple of years time, most of what ChatGPT achieves will be in line with most of the tech industry advances in the past decade - pushing the bottom out of the labor market and actively making the lives of the poorest worse in order to line their own pockets
This is not what any of the US economic stats have looked like in the last decade.
Especially since 2019, the poorest Americans are the only people whose incomes have gone up!
I use ChatGPT daily to generate code in multiple languages. Not only does it generate complex code, but it can explain it and improve it when prompted to do so. It's mind blowing.
FWIW, as a non-pathologist with a pathologist for a father, I can almost pass the pathology boards when taken as a test in isolation. Most of these tests are very easy for professionals in their fields, and are just a Jacksonian barrier to entry. Being allowed to sit for the test is the hard part, not the test itself.
As far as I know, the exception to this is the bar exam, which GPT-4 can also pass, but that exam plays into GPT-4's strengths much more than other professional exams.
What is a Jacksonian barrier to entry? I can't find the phrase "Jacksonian barrier" anywhere else on the internet except in one journal article that talks about barriers against women's participation in the public sphere in Columbia County NY during Andrew Jackson's presidency.
I may have gotten the president wrong (I was 95% sure it's named after Jackson until I Googled it), but the word "Jacksonian" was meant to refer to the addition of bureaucracy to a process to make it cost more to do it, and thus discourage people. I guess I should have said "red tape" instead...
Either it's a really obscure usage of the word or I got the president wrong.
"It's difficult to attribute the addition of bureaucracy or increased costs to a specific U.S. president, as many presidents have overseen the growth of the federal government and its bureaucracy throughout American history. However, it is worth mentioning that Lyndon B. Johnson's administration, during the 1960s, saw a significant expansion of the federal government and the creation of many new agencies and programs as part of his "Great Society" initiative. This expansion led to increased bureaucracy, which some argue made certain processes more expensive and inefficient. But it's important to note that the intentions of these initiatives were to address issues such as poverty, education, and civil rights, rather than to intentionally make processes more costly or discourage people.
Exams are designed to be challenging to humans because most of us don’t have photographic memories or RAM based memory, so passing the test is a good predictor of knowing your stuff, i.e. deep comprehension.
Making GPT sit it is like getting someone with no knowledge but a computer full of past questions and answers and a search button to sit the exam. It has metaphorical written it’s answers on it’s arm.
This is essentially true. I explained it to my friends like this:
It knows a lot of stuff, but it can't do much thinking, so the minute your problem and its solution are far enough off the well-trodden path, its logic falls apart. Likewise, it's not especially good at math. It's great at understanding your question and replying with a good plain-english answer, but it's not actually thinking
That's a disservice to your friends, unless you spend a bunch of time defining thinking first, and even then, it's not clear that it, with what it knows and the computing power it has access to, doesn't "think". It totally does a bunch of problem solving; fails on some, succeeds on others (just like a human that thinks); GPT-4's better than GPT-3. It's quite successful at simple reasoning (eg https://sharegpt.com/c/SCeRkT7 and moderately successful at difficult reasoning (eg getting a solution to the puzzle question about the man, the fox, the chicken, and the grain trying to cross the river. GPT-3 fails if you substitute in different animals, but GPT-4 seems to be able to handle that. GPT-4's passed the bar exam, which has a whole section on logic puzzles (sample test questions from '07: https://www.trainertestprep.com/lsat/blog/sample-lsat-logic-... ).
It's able to define new concepts and new words. It's masters have gone to great lengths to prevent it from writing out particular types of judgements (eg https://sharegpt.com/c/uPztFv1). Hell, it's got a great imagination if you look at all the hallucinations it produces.
All of that sum up to many thinking-adjacent things, if not actual thinking! It all really hinges on your definition of thinking.
exactly. it's almost like say dictionaries are better at spelling bee hence smarter than humans, or that computers can easily beat humans in Tetris and smarter because of that.
That's not a response from someone who wrote the answers on the inside of their elbow before coming to class. That's genuine inductive reasoning at a level you wouldn't get from quite a few real, live human students. GPT4 is using its general knowledge to speculate on the answer to a specific question that has possibly never been asked before, certainly not in those particular words.
It is hard to tell what is really happening. At some level though, it is deep reasoning by humans, turned into intelligent text, and run through a language model. If you fed the model garbage it would spit out garbage. Unlike a human child who tends to know when you are lying to them.
If you fed the model garbage it would spit out garbage.
(Shrug) Exactly the same as with a human child.
Unlike a human child who tends to know when you are lying to them.
LOL. If that were true, it might have saved Fox News $800 million. Nobody would bother lying, either to children or to adults, if it didn't work as well as it does.
>We are literally arguing semantics if this is AGI
It isn't and nobody with any experience in the field believes this. This is the Alexa / IBM Watson syndrome all over again, people are obsessed with natural language because it's relatable and it grabs the attention of laypeople.
Protein folding is a major scientific breakthrough with big implications in biology. People pay attention to ChatGPT because it recites the constitution in pirate English.
This is like all other rocket companies undermining what spacex is doing as not a big deal. You can keep arguing semantics while they keep putting actual satellites and people into orbit every month.
I use chatGPT every day to solve real problems as if it’s my assistant, and most people with actual intelligence I know do as well. People with “experience in the field”, in my opinion can often get a case of sour grapes that they internalize and project with their seeming expertise and go blind to persist some sense of calm to avoid reality.
ChatGPT cannot reason from or apply its knowledge - it is nowhere near AGI.
For example, it can describe concepts like risk neutral pricing and replication of derivatives but it cannot apply that logic to show how to replicate something non-trivial (i.e., not repeating well published things).
The domain is the domain of protein structure, something which potentially has gigantic applications to life. Predicting proteins may yet prove more useful than predicting text.
“Predicting proteins”? I’m a biologist and I can assure you knowing the rough structure of a protein from sequence is nowhere near as important to biology as everyone makes it out to be. It is Nobel prize worthy to be sure but Nobel prizes are awarded once a year not once a century.
Except its not, because they gave it away without any kind of commercialization. Its possible to give something away for free in some context and still have it be a product (Stable Diffusion is doing quite a bit of that, though its very unclear if they’ll be able to do it sustainably), but AlphaFold doesn’t seem to be an example. It seems to be an example of something cool they did that they had no desire to make into a product. Which is great! But isn’t the same as executing on product in a space.
This is hacker news, AlphaFold doesn’t have an app, some obscure GitHub repo, a hyped up website or a bunch of VC backing, so it’s basically a waste of time.
Numerous individuals have since transitioned away from Google, with reports suggesting their growing dissatisfaction as the company appeared indecisive about utilizing their technological innovations effectively.
Moreover, it has been quite some time since Google successfully developed and sustained a high-quality product without ultimately discontinuing it. The organizational structure at Google seems to inadvertently hinder the creation of exceptional products, exemplifying Conway's Law in practice.
Generative AI at its current state is still a very new area of research with many issues including hallucination, bias and legal baggage. So for the first few version we are looking at many new startups like open ai, stability, anthropic etc. It is yet to be seen if any of the new breed of startups actually starts to make sizeable revenue. But again there is nothing defensible here unless all the major labs stop publishing paper.
Uh, you snipped in the middle of a clause so you could argue against something it didn’t say.
Here’s the whole thing (leaving out a parenthetical that isn’t important here):
“Google has spectacularly, so far, failed to execute on products […] for generative AI”
You listed a bunch of products in other domains, some of which are the reasons why it has institutional incentives not to push generative AI forward, even if it also stands to lose more if someone else wins in it.
When did anyone realize that there generative AI was actually a product with wide consumer appeal? Or how many use cases there were for it as an API service? I'd say it wasn't really obvious until around Q4 last year, maybe Q3 at the earliest.
That's a pretty short time ago. So it seems that so far it hasn't really been a failure to execute, but more about problems with product vision or with reading the market right leading to not even attempting to have actual products in this space. That's definitely a problem, but not one that's particularly predictive of how well they'll be able to execute now that they're actually working on products.
The hardware costs alone of running something like GPT 3.5 for real time results is 6-7 figures a year. By the time you scale for user numbers and add redundancy... The infra needs to be doing useful work 24/7 to pay for itself.
It's more than possible Google knows exactly what it can do, but was waiting for it to be financially viable before acting on that. Meanwhile Microsoft has decided to throw money at it like no tomorrow - if they corner the market and it becomes financially viable before they lose that it could pay off. That is a major gamble...
> The hardware costs alone of running something like GPT 3.5 for real time results is 6-7 figures a year.
Can you unpack your thinking there? Even at 5% interest for ownership costs to be six figures a year you're talking about millions of dollars in hardware. Inference is just not that expensive, not even with gigantic models.
To the extent that there is operating cost (e.g. energy)-- that isn't generated when the system is offline.
I don't know how big GPT 3.5 is, but I can _train_ LLaMA 65B on hardware at home and it is nowhere near that expensive.
That's 8 $200k GPUs + all the other hardware + power consumption for one instance. You could run it on cheaper hardware, but then you'll get to nowhere near realtime output which is required for the majority of the use cases not already handled well by much smaller models.
Even if Google/Microsoft are getting the hardware at a 50% reduction (bearing in mind these are already not consumer prices) it gets to $1mn in hardware alone - again for a single instance that can handle one user interacting with it at a time.
It makes a lot of the bespoke usecases people are getting excited about (i.e. anything with data privacy concerns) far from financially viable.
If you want a dedicated instance of full capability ChatGPT for example (32K content) OpenAI are charging $468k for a 3 month commitment / $1,584k for a year.
You can purchase 80GB A100s right now for about $12,5k on the open market. I think the list price is $16k. I don't know what discount the big purchasers see, but 30% should be table stakes (probably explains that $12.5k prices), 50% for the big boys wouldn't be at all surprising to me based on my experience with other computing hardware.
So under the assumption that 8 80GB gpus are required, we're talking about a somewhat more than $100k one time cost (for 8x 80gb A100 plus the host) plus power, not 6-7 figures annually. Huge difference!
Evaluating it in a latency limited regime but without enough workload to enable meaningful batching is truly a worst case. I admit that there are applications where you're stuck with that, but there are plenty that aren't.
Anyone in that regime should try to figure out how to get out of it. E.g. concurrently generating multiple completions can sometimes help you hide latency, at least to the extent that you're regenerating outputs because you were unhappy with the first sample.
> that can handle one user interacting with it at a time.
That bit I don't follow. The argument given there is without batching. You can do N samples concurrently at far less than N times the cost.
This, but non-sarcastically. Google has spectacularly, so far, failed to execute on products (even of the “selling shovels” kind, much less end-user products) for generative AI, despite both having lots of consumer products to which it is naturally adaptable and a lot of the fundamental research work in generative AI.
The best explanation is that they actually are, institutionally and structurally, bad at execution in this domain, because they have all the pieces and incentives that rule out most of the other potential explanations for that.
> OpenAI bought into the field. They are good at execution but i havent seen anything novel coming out of them.
Right, OpenAI is good at execution (at least, when it comes to selling-shovels tools, I don’t see a lot of evidence beyond that yet), whereas Google is, to all current evidence, not good at execution in this space.