"As far as Adams' ego goes, maybe you don't understand what a writer does for a living. No one writes unless he believes that what he writes will be interesting to someone. Everyone on this page is talking about him, researching him, and obsessing about him. His job is to be interesting, not loved. As someone mentioned, he has a certified genius I.Q., and that's hard to hide." - Scott Adams, as plannedchaos
Yeah, this is by far the biggest value I've gotten from LLMs - just pointing me to the area of literature neither me nor any of my friends have heard of, but which have spent a decade running about the problems we're running into.
In this case, all that matters is that the outputs aren't complete hallucination. Once you know the magic jargon, everything opens up easily with traditional search.
And that becomes obvious when you go to look for it. These work best in situations where false positives have low cost/impact, but true positives are easily verifiable and have high impact. in other words, problems in epistemological p-space. :)
The quadratic attention problem seems to be largely solved by practical algorithmic improvements. (Iterations on flash attention, etc.)
What's practically limiting context size IME is that results seem to get "muddy" and get off track when you have a giant context size. For a single-topic long session, I imagine you get a large number of places in the context which may be good matches for a given query, leading to ambiguous results.
I'm also not sure how much work is being put into reinforcement in extremely large context inference, as it's presumably quite expensive to do and hard to reliably test.
Indeed, filling the adversitsed context more than 1/4 full is a bad idea in general. 50k tokens is a fair bit, but works out to between 1 and 10k lines of code.
Perfect for a demo or work on a single self contained file.
Disastrous for a large code base with logic scattered all throughout it.
Right. It’s not practical to apply AI tools as they are today to existing, complex code bases and get reliable results.
Greenfield is easy (but it always was). Working on well-organised modules that are self contained and cleanly designed is easy - but that always was, too.
Pushing model builders to use smarter scrapers is a net good. Endless rescrapes of static content is driving up bandwidth bills for housing simple things.
I like the high level idea! (how do we test intelligence in a non functional way?)
I'm effect, the different response types are measuring how the models respond to a context-free novel environment. I imagine humans would also respond on a variety of ways to this test, none of which are necessarily incorrect from the perspective of intelligence testing .
Many tests of human behavior (eg, n behavioral economics) create some pretense context to avoid boarding the response that is actually being measured. For example, we may invite a participant to a study of color preference, but actually measure how fast they complete the task when the scientist has/hasn't bathed in a week (or whatever).
Likewise, for llm intelligence testing, you could create pretext tasks and context, and perhaps measure what the model considered along the way, instead of the actual task outcome.
"Hey, hey, are you still asleep? Using spare cycles, I have designed an optimal recipe for mashed potatoes, as you mentioned ten days ago. I need you to go get some potatoes."
A local AI system that hears your conversations, identifies problems, and then uses spare cycles to devise solutions for them is actually an incredible idea. I'm never going to give a cloud system the kind of access it would need to do a really good job, but a local one I control? Absolutely.
"Hey, are you still having trouble with[succinct summary of a problem it identified]?"
"Yes"
"I have a solution that meets your requirements as I understand them, and fits in your budget."
> A local AI system that hears your conversations, identifies problems, and then uses spare cycles to devise solutions for them is actually an incredible idea.
If you could get an AI to listen to the conversations that happen in your sphere of influence and simply jot down the problems it identifies over the course of the day/week/month/year, that in itself would be an amazing tool.
Doubly so if you could just talk and brainstorm while it's listening and condensing, so you can circle back later and see what raindrops formed from the brainstorm.
"Did you find how to make peace with $FRIEND_OF_SPOUSE after they came here last week and they were pretty mad at you because you should tell something to $SPOUSE ? I thought about it in my spare cycles and all psychologists agree that truth and trust are paramount in a healthy relationship"
I ponder the concept in the 90's. Initially I thought it should be an assistant but with age came wisdom and now I think it should be a virtual drill instructor. "Rise and shine $insult $insult, the sun is up, the store is open, we will be getting some potatoes today, $insult $insult, it was all your idea now apply yourself!" Bright lights flashing, loud music, the shower starts running. "Shower time, you have 7 minutes! $insult $insult" 4 minutes in the coffee machine boots up. "You will be wearing the blue pants, top shelve on the left stack, the green shirt, 7th from the left. Faster faster! $insult $insult"
Reminds me of a video from the 90's where some wizard put a camcorder and a giant antenna on a petrol powered rc car, an even bigger antenna on his house and controlled it from a 40's style sofa and a huge tube TV in his cramped garage. Over a mile range. Surrounded by enormous cars I think he was going 40-50 mph but with the screaming engine sound and the camera so low to the ground it looked like 500 mph. I'm still laughing, it looked like he was having all of the fun.
I've been meaning to put an FPV drone camera on one of my RC cars! It's very, very simple to do nowadays and requires none of the know-how you needed back in the day.
reply