It's a shitty analogy, and TFA is pointing it out.
> Perhaps to frame it better: do humans trade with GOD?
This is also a terrible analogy because:
1. There is no general agreement on whether or not any deities exist, and if so, which ones.
2. Historically, many religions do involve trade with one or more deities (e.g. protect me from this battle and I will sacrifice 2 goats when I get home). While this is more commonly associated with pagan religions, the Abrahamic religions are not entirely devoid of this tradition either.
"Not entirely devoid"? Abrahamic religions are built on a bedrock of contracts (albeit rather one-sidedly negotiated) with God. The 10 commandments, basically the entirety of Leviticus, etc.
Heck, every mainstream religion with dieties has some sort of teaching with the idea that "human does x, God does y".
I say "not entirely devoid" because a large fraction of modern Protestantism is rather far from the contractual roots; they were so obsessed with stamping out pharisaical legalism that they went from "If ye keep my commandments, ye shall abide in my love; even as I have kept my Father's commandments, and abide in his love." to "Salvation is by faith alone"
God in the gnostic Logos sense: the all encompassing universe, the forces that act within it, so on.
In more current vernacular: do humans trade with the big bang? do humans trade with the expansion of the universe, and it's inevitable collapse back into the initial singularity?
It's not a shitty analogy. Most people completely understand what it's getting at. Inability to understand is a reflection on you, not the analogy. And being pedantic is not a positive trait or intellectually impressive.
The distinction is important because an AGI developed by humans is probably going to have (at least vestigial) ability to communicate with humans in some way.
It's entirely possible that it will have no use for humans, but if that's what the analogy is getting at, it's doing so poorly.
If my understanding of mythology is current, it's only the devil who considers bargaining, and it's never to our benefit.
Following that idea, if AI ever needed to "trade" with humans I suspect we'd be taking the worse end of the trade every time (the trade would be for its benefit, and unlikely to benefit us).
Kind of like when the U.S. "trades" with a third-world country. The U.S will just make an offer they can't refuse. But taking the trade often perpetuates their dependence on the U.S. or has long-term consequences for them.
Which is why countries the US have embargoed have advanced leaps and bounds over their neighbors with trade relationships. Oh wait, it is the exact fucking opposite. Seriously it seems some people can't pry off their exploitation and imperialism goggles. The U.S. could commit imperialism by sitting quietly in another room in their minds.
No it's not satire. It's one of those clickbaity contrarian articles like "actually using goto is great" or "sometimes wearing pants on head is the smart thing to do." The maxim that they are reacting to is the idea that because we don't trade with ants, smart AI won't trade with us if the cognitive difference is similar. I think their take is that once you can overcome some minimum threshold of communication ability then it does often make sense to trade with others even if they are much much stupider than you are, and that the reason we don't trade with ants is because of that communication barrier. Presumably they are suggesting that even if we will be much much stupider than AI, we will still be able to communicate with the AI in some manner that exceeds that threshold, and therefore they might want to trade with us in some way. But my main point is that it's a contrarian clickbait take that isn't meant to be taken too seriously or to be taken as satire it's to be taken as something to click on.
The idea that this framing is better than the original really goes to show that superintelligent AIs aren't being discussed because they're a realistic near-future possibility, but because they let techie atheists ponder religious concepts like omnipotence and god without admitting to themselves that that's what they're doing.
>they let techie atheists ponder religious concepts like omnipotence and god without admitting to themselves that that's what they're doing
There's a reason the singularity has been called "the rapture for nerds" since forever. Beliefs about runaway AGI (in particular the assumption that it would have nigh godlike powers,) along with a lot of UFO culture and belief in simulation theory are literally just religion with the serial numbers filed off.
This has been my experience as well. I cannot understand how people are saying it has given them order of magnitude speed improvements in shipping.
Most code on github is by definition average, which is usually a few levels below the quality of an expert programmer. So it's not surprising the output of code LLMs is poor to average quality. For any larger block of code it outputs, a non-trivial amount of editing is required if a higher bar of quality is wanted.
Around half of developers write below average code, by definition. So it isn't surprising that raising the quality to average _for them_, and doing it faster, would be a productivity boon.
I'm a crappy marketing copy writer. A professional writer could do much better. But with chatGPT I too can write hack-quality marketing-copy that roughly conveys the a message and give it others to put in various marketing outlets.
> Around half of developers write below average code, by definition.
By definition, half the code out there is of below median quality. Whether or not half the code out there is of quality below the arithmetic mean depends on assumptions about the distribution of code quality. I would suspect that much more than half the code out there is "below average", so to speak.
Yep, there are assumptions all around. On the developer side, I also suspect there is a huge group casual "low-code" type programmers out there that could be assisted by copilot type tools. My main point was that the argument that the code produced by LLM type systems is low-quality is not really a barrier to adoption, and _might_ actually _raise_ the average quality of systems built in the wild. I'm less concerned with the actually distributions and more noting the point that the different relative value-point exists.
Does anyone get an unpleasant feeling from these typical "founder" types? Something always just feels deeply off about them. They all seem like carbon copies of each other too.
Just here to post this is another reason to get off Gumroad when you get a chance. Do NOT use Gumroad. Using Stripe directly will save you so much money. Still bitter at those exploitive jerks jacking the payments price to a total of 12.9% (and something like 17% when through PayPal).
Gumroads USP isn’t payment processing. Gumroads USP is that they handle international sales tax (also know as Merchant of Record), which, I’ve you’ve ever tried selling something on the Internet without committing tax fraud¹, will know that 12.9% is _cheap_.
¹) I’d estimate some 95% of all small online business are doing sales tax wrong. Hell, Basecamp just now cleaned up their act.
If you're a software dev, you shouldn't be using gumroad. You're not their target market. Gumroad explicitly states that they're for creators, and that software folks are better off just integrating Stripe.
You get a lot more on gumroad for that don’t you? I tried to find MoR services like Gumroad and there are a bunch, so you don’t need to use them. But it’s not that trivial to replace all they offer for that price; that depends of course on how much you sell. It becomes more interesting to move the more you make, but I would skip Stripe too and go to a ‘local’ acquirer which will tons cheaper than Stripe.
Gumroad should’ve gone about it another way; just do fixed price extra’s for instance. Ah well.
Ignoring the ~2.9% of Credit Card Processing Fees ( on average it would still be higher than 2.9% ), I dont see how they charge 10% for their services is exploitive jerks.
Changing the fee from 3%-5% total to 12.9% (17% on paypal!) total when you already have a large customer base there, meaning you'd have to painfully migrate them to another provider, is not a nice thing to do in my eyes.
And also why I'm working on completely migrating off.
I was always surprised at how AMD hasn't already thrown a bunch of money at this problem. Maybe they have and are just incompetent in this area.
My prediction is AMD is already working on this internally, except more oriented around PyTorch not Hotz's Tinygrad, which I doubt will get much traction.
He mentioned ROCm, and apparently had lack luster experience with it.
>The software is called ROCm, it’s open source, and supposedly it works with PyTorch. Though I’ve tried 3 times in the last couple years to build it, and every time it didn’t build out of the box, I struggled to fix it, got it built, and it either segfaulted or returned the wrong answer. In comparison, I have probably built CUDA PyTorch 10 times and never had a single issue.
Not surprising lol. This was also the experience I had while experimenting with MLIR approximately 3 years ago. You'd need to git checkout a very specific commit and then even change some flags in code to have a successful build. I'm sure things are better now but I haven't messed with it since then.
AMD is not going down the path of ROCm; perhaps they claim to do so, but as evidenced by the lack of both effort and results, they clearly are not.
The parent post is surprised that they still aren't making the appropriate investments to make it work. They kind of started to do that a few years ago, but then it fell on the wayside without reaching even table stakes, which in my opinion would require providing a ROCm distribution that works out of the box for most of their recent consumer cards (i.e. those cards which the enthusiasts/students/advocates/researchers might use while choosing which software stack to learn, and afterward base corporate compute cluster purchasing decisions on whether they support the software they wrote for e.g. CUDA+Pytorch), and they seem to be failing at that.
AMD is limited by numerous patent and other legal issues. For this reason small company that releases everything as open source have some chances to beat AMD on their own hardware.
As mentioned, much of our discussion is rote parroting. I can usually go into any hackernews thread and roughly know what the top discussions are going to be. It's not surprising that an AI trained on a large portion of the internet would thus look human like.
If you really poke at GPT, you begin to realize it's fairly shallow. Human intelligence is like a deep well or pond, where as GPT is a vast but shallow ocean.
Making that ocean deeper is not a trivial problem that we can just throw more compute or data at. We've pretty much tapped out that depth with GPT4 and are going to need better designs.
This could only take half a decade or it could be half a century. Plenty of enterprises stagnate for decades.
Sam Altman said it himself. He seems like a reasonable source.
If you're familiar with other fields of AI, adding more and more layers to ResNet was the hotness for awhile, but the trick stopped working after awhile.
Altman didn't really say that. Reading what he actually said rather than a headline, He was alluding to economical walls. He didn't say anything about diminishing returns on scaling. And if anything, the chief scientist, Ilya thinks there's a lot left to squeeze.
Sure Sam Altman, the lying CEO of a tech company (they all do) should be listened to on this matter but not on the part where he thinks AGI within reach using his approach. Selective hearing.
> You can't possibly know that, given that we don't actually understand how LLMs work on a high level.
It's a fair assumption to make however - basically 80/20 rule.
AI research isn't a new thing and I bet you could go back 40/50 years where they thought they were about to have a massive breakthrough to human level intelligence.
> GPT-4 is three months old and you're confident that its working principle cannot be extended further? Where do you get that confidence from?
I'm guessing from actually using it.
GPT4 is super impressive and helpful in a practical way, but having used it myself for a while now I get this feeling also. It feels a bit like "it's been fed everything we have, with all the techniques we have, now what?"
There are dozens and maybe hundreds of different approaches that could theoretically get around the limitations of GPT4 that merely haven't been trained at scale yet. There is absolutely no lack of ideas in this space, including potentially revolutionary ones, but they take time and money to prove out.
Training a model doesn't mean you understand what the neurons actually do to influence output. Nobody knows that. That's where the black box analogies come in. We know what goes in the box and what comes out. We don't know what the box is doing to the data
> Making that ocean deeper is not a trivial problem that we can just throw more compute or data at.
I'd say this is immediately counterindicated by the available evidence. Gpt2 was hopeless for anything other than some fun languagw games like a bot replica of a subreddit or trump. 3.5 is much much bigger, and has semi competent but limited reasoning abilities.
Gpt 4 is a vast improvement over 3.5 in various reasoning tasks. Yes, a priori I would have agreed with you that this has to stop somewhere, but not anymore. I would need to see some data of post gpt4 models to believe you.
The apparent shallowness IMO is due to the lack of long term memory and limited context. In terms of depth, the depth of the human mind is fairly limited. Put any average human to task on any creative endeavor and it's surely a regurgitation of things they've seen. The high standard of true creativity that people hold LLM's to is only a capability of a small minority of humans.
I'd wager it's far more likely 5 years than 50 LLMs get to the full depths all humans are capable of. Simply compare the state of LLMs today vs 2018.
>Using the current approach, we can create AIs with the ability to do any task at the level of the best humans — and some tasks much better. Achieving this requires training on large amounts of high-quality data.
Wrong. Full self driving is still not here despite access to a huge amount of high quality data.
I guess it depends on what you mean by full self driving. I shared an intersection in San Francisco just the other day with a Cruise (GM) self driving car that didn't have anybody in the car. Are there limits? Yes, but there are limits to my ability to drive in certain situations as well. Will it take over the road soon? Probably not.
>I get the cold sweats when I see methods longer than a few lines of code
This guys code might not be as readable as he thinks. I do not want to read through your hundreds of 10 line functions because a few people said "long functions bad".
I liked the portion on naming. My God the amount of AbstractFactorySingletonViewModel garbage I see is astounding.
Simple anecdote: A colleague of mine had a really hard time getting some topologic sort to work. I kept finding issues with his implementations and he was getting frustrated so we aggreed to have a pair programming session. He wanted to put all the code in a single method (not terribly long, maybe 30, 40 lines), but the conditions kept nesting and no matter how much he tried to figure them out, he couldn't get them to work. I convinced him to extract two methods which would only be used once and only inside this sorting method. He was vehemently against it. After he finally agreed he realized the code was so easy to read, it was basically self-explanatory and obviously correct. No ammount of fiddling with the if statements could've produced that.
I don't impose fixed line counts for my methods. But when it makes sense, the readability improves a lot.
I have essentially the same story, so I won't repeat it. The big win was when my colleague seized on the idea of self documenting code. That was all he needed to write better structured code.
Small functions are not about readable code. It's about single responsibility.
Single responsibility is an engineering principle that is not limited to software development. It is the reason that we don't combine the breaking functionality in our car with the am/fm radio. We keep moving parts isolated because the fewer "things" that something does, the less likely it is to break. The less complicated it is.
But there are other side effects to creating small, single-purpose functions (or anything for that matter). It is not always obvious when you will have an opportunity to reuse something. And duplication is not always apparent. When you take single-responsibility as far as you can go, you not only isolate all of your moving parts but you maximize the opportunities for reuse.
And it goes even further than that. Your large functions likely have a few dependencies, at least. Those dependencies will make your functions more difficult to write tests for. And your test cases will be more complicated if you have larger functions because they are doing more than one thing that needs to be captured.
Readability is more about expressing the intent of code. You can do that in "long functions." You can express the intent of code while making it extremely compilcated. Hell, just add lots of verbose comments and your code will be more "readable."
Since you seem to be one of those people who has a stick up you about design patterns (your last comment about AstractFactorySingletonViewModel ... who hurt you?) I will offer you this piece of food for thought: the purpose of "best practices" and design patterns is to SIMPLIFY code. If you ever see a misapplication of them in the wild* then what you are witnessing is not "over" engineering ... it is POOR engineering. Consider that before throwing the baby out with the bathwater.
* I rarely do, so I often wonder to myself if this is a made up problem by lazy devs who don't want to actually study theory. But I do hear that this occurs from time to so I'll take you at your word that there are people out there that don't know how and when to actually apply design patterns properly. The problem is the misapplication, not the patterns themselves - which are just common solutions to recurring problems. Do you not think DRY is a good idea?
Ha! Thanks for sharing “enterprise FizzBuzz”; I’d not seen that before. I was digging through the code before I had a PTSD-related flashback from debugging some OO at my last company…
My friend and I were working together, when one of our seniors came over and asked us for help figuring out how to modify an algorithm that someone new on the project had rewritten. The algorithm was 20-30 well-commented lines of code, until the rewriter decided that it lacked flexibility. It had been rewritten to use a Singleton Factory to instantiate an implementation of an interface that itself delegated every snippet that could possibly be conceived of as a function to one of several different classes (Policy, Actor, Dispatcher, etc.). Every function was three lines or less, and all were entirely illegible taken in part or in whole.
I’m glad that team member got to show us how many times he read about design patterns and how much he knew about OOP, but it took four engineers (we later roped in another) to venture 15 layers down the call stack of virtual functions to figure out where a simple constant came from. Quality code indeed.
> Any rule, principle or pattern taken to its extreme is bad.
That's a non-sequitur. The word "extreme" describes matter of degree. If something is "good" then it does not follow, logically, that "extremely good" becomes "bad" just because it was "taken to the extreme." As you yourself said, context matters.
Then to say "it's always a balance", a balance between what? Good and bad? Simple and complex? What are you talking about?
We are speaking in the abstract, necessarily, when we speak of "small, single purpose functions" because we are taking the context out of it. Can we agree that the simplest solution should always win? If so, why?
After 25 years of developing software professionally, I have seldom seen someone apply design patterns, so-called "best practices" and engineering principles to a point where they shot themselves in the foot. It happens, we are all human after all, but in my experience it is not a huge problem.
What is a huge problem, is that time after time we inherit projects written by very decent hardworking developers who can get something that works out the door quickly but don't know how to write it in such a way that it is easy to change over time. We see massive files, massive functions, lots of cyclomatic complexity, terse variable names, duplication, tight coupling, near zero modularity and very poor separation of concerns. This is the norm, not the exception.
And yet lately, I have seen a trend that is a knee-jerk reaction against decades of work done to address this problem and it should be concerning. If 75% of the projects I worked on were tech-debt free, and we could identify actual examples of software where people wrote it so abstractly that it was incomprehensible then I would be on the same side. But what I see in the wild is not examples of people going "patterns happy" or taking things to a point where it starts to become complex rather than simple. What I see are developers who have never even heard of design patterns, or they learned one and then misapplied it everywhere because suddenly every tool looks like a nail to them.
Therefore my position is that we should be preaching single-responsibility and small functions more, not less.
This is a very level-headed defense of design patterns. I'm kinda like the parent commenter. I have a stick up my ass about these kinds of things. I think they're misused quite often. But it's good to see a reminder that they do have their place.
Even DRY, I've seen DRY cause test suites become hard to work with. Rules of thumb sometimes get pumped really hard and I think it causes people to over-use them. The rule becomes the goal rather than something more material like UX or DX. This code is DRY, DRY is good, therefore this code is good - I hear arguments like that quite often. But your take feels very grounded.
> We keep moving parts isolated because the fewer "things" that something does, the less likely it is to break. The less complicated it is.
It doesn't really work if I'm breaking up a function body into a bunch of smaller functions that will never be called from anywhere except the refactored function. I've just moved the complexity from a large function body into a bunch of smaller functions and the calls to them - but it's still the exact same logic, with the same complexity (but perhaps harder to read, due to all the jumping back and forth).
It’s a different kind of overhead. But I will take the tradeoff between ten 10 line methods and one 100 line method every time. I stand a much better chance of successfully modifying and testing the former.
Small functions are easy enough for me to deal with when they're basically library functions, but when they're split up/distributed in an OOP-y way I find navigating them and keeping track of them way more troublesome.
I think this is one of the more important distinctions, but you never seen it brought up. High level, business oriented functions almost never end up being more readable split up into a bunch of sub functions.
My second job had amazing code quality, and one that that always struck me was how the code was separated cleanly into layers that made the business logic layer super obvious.
It was 3 layers
- API interface (unpacking objects, etc)
- Business Layer
- DB/System Interface.
The business logic changes could often be totally contained, without updating the API or the database. It would be so obvious what layer needed to be edited, and the problems could be fixed so much faster. Rarely did a commit change two layers at once.
You can easily handle AuthN in the API layer, AuthZ in the business layer, and then access to a db in the system layer. Assume API is un-authZ’ed, assume the system layer is fully AuthZ’ed, etc.
That's fair if you only have 100 lines of code, but I don't think it automatically follows that the best solution for 10,000 lines of code is 1,000 ten line functions.
There's a tradeoff between understanding the function itself and understanding how/where the function is used (and therefore understanding what will break when that function is changed).
The best heuristic I've got at the moment is roughly the square root of the size of the module. So, for the 10,000 line module, my vague instinct is that I'll have one hundred functions approximately one hundred lines each. That's not a hard and fast rule, just an observation about tradeoffs and I would definitely expect (quite possibly even most) functions to differ significantly without losing sleep over it.
Agree - if a function is otherwise cohesive and doesnt mix independent concerns, I prefer one 100 line fuction to 25 4-line functions.
Splitting functions is tempting because it makes each individual function seem simper, but the higher cost in complexity and maintainability is not as directly visible.
Shorter functions are fine when appropriate, but length by itself is not an indicator of quality.
Perhaps to frame it better: do humans trade with GOD?