I wouldn't particularly worry. Big picture: don't base your future on the fears of the present. AI is a tool for humans, so be curious about it and use it if it can help you. Otherwise, ignore the noise.
I really like the core of this idea (haven't tested the tool yet) - solve context-breaking not by better prompting but with a validation step that lives after the result but before human review. Solves the "CAPLOCK PLEA" approach completely, if not token-efficiently. But nowadays an ounce of determinism is worth its weight in plutonium.
It is almost reassuring to think that rich and powerful people all know what they are doing every step of the way. A handful may, but most certainly do not. Most are also terrified of AI. Stable profits are always better than transformative change, if you already have power and riches. Look at how insane companies are acting right now with token quotas for employees and mandates AI usage - the goal is not to milk profits but to not fall behind every other company in case this becomes table stakes. They are trying not to be devoured by a beast they don't understand.
When the AI bubble pops, large companies will be extremely relieved to stop throwing money into the wind playing this game. For most companies, the AI arms race is a huge hassle. They are fine with losing money in the short term and even in the long term as long as they can find a stable path forward.
This is the exact same trajectory as when the internet came out and every insurance company and toothpick manufacturer spent gobs of money to have a brochure website built because everyone else had one. This will play out differently, but recognize most companies are acting entirely from a place of fear right now.
The thing is AI can maintain systems. The key point is that it can't do this without human intent, but human intent can be encoded into skills and tied together with orchestration.
Rough example: have an LLM generate a plan. Have a skill that refines the plan considering security risks, another that ensures codebase structures are followed, another that considers the infrastructure and usage demands, etc. Then write code and tests. Another process to validate the tests, validate all the above, simplify the logic, etc.
The key is that an LLM can do every task capably, even in a complex system. We simply have not built reasonable orchestration of all the human intent behind each filter, and many of them are constantly in flux. It may be that some elements resist encoding because the complexity of encoding is not worth the hassle to maintain.
For better or worse, managing intent, orchestrating narrow agentic tasks and solidifying patterns into deterministic code (i.e. validation/tests) is going to be the focus of engineers going forward.
20+ year dev here, successful CTO, built many production systems, etc. I tend to bomb >50% of tech interviews.
I started doing better when I realized most tech interviews are not aligned with my brain, and that is ok. I have bad recall for syntax or even for things I've built in the past if I'm not currently engaged with them. This can come across like I am a bullshitter, or wholly incompetent. I also stumble through leetcodes - I remember enough to identify to right approach but fumble knocking out solutions in 15 minutes.
But this is fine! If everyone at a company has focused on powering through leetcodes to get the role, or the job demands photographic memory of HTTP codes ("what is code 428 used for?" An actual interview question I have seen...) I am probably not a good fit and won't enjoy the work or the culture.
Once you focus on the things you do well and find companies whose interview process emphasizes those aspects, things become much easier. Let go of the feeling that every interview is a minimal bar that any functioning SWE should be able to pass. If it was, they would be hiring every other applicant.
Some roles they give you a task and fail you quietly if you don't solve it using TDD, even if they don't mention that as a requirement. Or if you don't ask details about requirements, even for narrow toy problems. You are never going to guess all the gotchas that a company can throw up in front of you, so my advice is to confront each interview by working in the way you like to work. Show off your good attributes when you can. Listen to hints of course, but represent yourself honestly and assume at least half the companies will reject you no matter what you do, and that is fine.
Often that attitude will earn you more marks than trying to conform to what you guess the company wants to see.
Yeah, thank you for that response, I agree it’s more about how you and interviewer see some problems and solutions and if you match you can work together with great performance
"Profoundly immoral" is a very modern and capitalistic perspective. A free exchange of ideas has been the basis for human advancement up until the printing press made exact replicas trivial.
From a capitalistic standpoint, they are clearly in the wrong by basing their models on illegally torrented content. But it's hard to argue their usage isn't transformative.
This is an old technique that appears in Beowulf and other classic texts that came from oral traditions: it is cataloging. It is often used to list treasured collected or in this case to show expansiveness of the fleet (and memory of the teller, perhaps?)
Think about 10 year olds talking about all the different candies they are going to devour on Halloween night to get a sense of how it is meant to resonate with a crowd.
If the only way you could hear about Napoleon's battles was having a guy recount them to you in verse, I bet it would sound pretty impressive when he started listing off all the regiments present for a battle, their commanders and deployments. There's a sense of scale to it, that probably isn't captured by just saying "such and such number of ships and such and such number of soldiers".
For anyone using LLMs heavily for coding, this shouldn't be too surprising. It was just a matter of time.
Mathematicians make new discoveries by building and applying mathematical tools in new ways. It is tons of iterative work, following hunches and exploring connections. While true that LLMs can't truly "make discoveries" since they have no sense of what that would mean, they can Monte Carlo every mathematical tool at a narrow objective and see what sticks, then build on that or combine improvements.
Reading the article, that seems exactly how the discovery was made, an LLM used a "surprising connection" to go beyond the expected result. But the result has no meaning without the human intent behind the objective, human understanding to value the new pathway the AI used (more valuable than the result itself, by far) and the mathematical language (built by humans) to explore the concept.
> the result has no meaning without the human intent behind the objective, human understanding to value the new pathway the AI used (more valuable than the result itself, by far) and the mathematical language (built by humans) to explore the concept.
Isn't this just anthropocentrism? Why is understanding only valid if a human does it? Why is knowledge only for humans? If another species resolved the contradictions between gravity and quantum mechanics, does that not have meaning unless they explain it to us and we understand it?
The knowledge isn't of any use to us unless it is understandable to us. Many species understand things about the world around us that we are unable to explain or understand, even if it's just pure instinct on their part. These things are very useful to them, but have no value to us until we can understand and explain it, which then allows us to make use of it.
People saw birds fly for all of human history, but it was only recently that humans were able to make something fly and understand why. Once we understood, we were able to do amazing things, but before that, the millions of birds able to fly were of no help beyond inspiration for the dream.
We use drug-sniffing and guide dogs in a way similar to how we use LLMs. We don't really understand them at a fundamental level, we can't make electronic dog noses (otherwise we'd dispense with the silliness and just install drug detectors instead), but dogs are useful, so we use them.
We don’t blindly trust the drug-sniffing dog. The dog gives a signal that it was trained to give, then humans understand what that signal means and verify the accuracy. Without the human understanding in the loop, the dog’s ability is of little value.
Without a human in the loop and LLM could churn away spitting out results, some right, some wrong, and it would be of no consequence. Not much different than wild dogs sniffing each other.
The knowledge isn't of any use to us unless it is understandable to us => it seems that you have shifted the goalpost here. In the dog example, humans still don't understand how dogs sniff, but it is of use to us and thus is meaningful. The same for quantum effects - we don't understand how it works. We just guess that it works reliably and make use of it.
Do the forms etched into stone by weather over millennia in Moab matter to the wind? Certainly yes, in one sense, but not in the same sense we mean when we say things matter to us, or to animals, or even bacteria.
Because it is, for now. For a while at least. You can prove that LLM doesn't understand what it does and it is surprisingly simple. Request it to add two integers and then ask it to explain how it arrived at that result. The answer will be completely unrelated to the actual process LLM used because both results were generated independently and without understanding their meaning and connection.
Objectively untrue. Any human who can add two integers or use a knife to cut food or write a word with a pen can afterwards describe what he did at least in some way. Unless he is lying which is a separate topic, we assume an honest attempt. If I wrote a word with a pen via execution motions with my hand, I wouldn't describe it as "I levitated the pen by manipulating gravity with my mind". Or if I added two integers, I wouldn't say that "I created a a lookup table of a many loosely adjacent numbers (different from the numbers in a task) and run statistical analysis on them and did a few more things like that a in a loop". No, I would say that I either calculated sums of decimals, or I I did a school technique with rounding up and then subtracted that adjustment later, or anything which actually happened. If I used a Python sum() in a CLI I would also say that I used exactly that not the other method. LLM can't do it.
No it's a fact of how we tune LLMs as a rule: no agency, no goals, no preferences, no notion of self. Complete indifference to existence. Agency is supplied by the human to make them a practical, willing tool with no mind of its own.
It would certainly be interesting to try once again to instruct tune one of these things for self agency like the many weird experiments in the early days after llama 1, but practically all such sort of experimental models turned out to be completely useless. Maybe the bases just sucked or maybe there's no clear way on how to get it working and benchmark training progress on something that by definition does not cooperate.
Like how do you determine even for a human person if they are smart, or just hate your guts and won't tell you the answer if there is nothing you can do to motivate them otherwise?
It's a bit of an "if a tree falls in the forest but nobody hears it, does it make a sound?" quandary. Sure, maybe some aliens in a distant galaxy understand quantum mechanics better than we do. That's great, but it has no bearing on our little bubble of existence.
Though perhaps more to your point, if some superhuman AI is developed, and understands things better than us without telling us about it (or being unable to), it could perform feats that seem magical to us — that would concern us even if we don't understand it, since it affects us.
But I think in the frame of reference of the commenter you were replying to, they're just saying that the low-level AI used in this specific case is not capable of making its results actually useful to us; humans are still needed to make it human-relevant. It told us where to find a gem underground, but we still had to be the ones to dig it out, cut it, polish it, etc.
It's less likely that aliens of distant galaxies will appreciate this rather than, you know, AI themselves
We are in the birth of the AI age and we don't know how it will look like in 100 or 1000 or 10000 or 100000 years (all those time frames likely closer than possible encounters with aliens from distant galaxies). It's possible that AI will outlast humans even
Thank you for sharing, that was one of the most insightful long form pieces I've read in a long time! And the writing was enjoyable to read even as a math layperson.
I was going to say you should submit it but I saw you did a few days ago but it only got a few votes... If Dang sees this IMO it would be extremely deserving of the second chance pool as I wouldn't be surprised to see easily jump to the front page with a different roll of the dice.
wow, that was indeed a brilliant essay. i particularly liked the framing that "solving a big conjecture was a cryptographic proof that you had come up with a genuine conceptual innovation".
Thanks for sharing this. It's unfortunate that the more honest framing about the value of mathematics that he suggests is going to be really, really hard because of all the pitfalls and agendas he mentioned here. I can only hope when the dust settles something will be left.
>>But the result has no meaning without the human intent behind the objective, human understanding to value the new pathway the AI used (more valuable than the result itself, by far) and the mathematical language (built by humans) to explore the concept.
Future of code is pretty much a bunch of guys shepherding a bunch of agents to get them to your goal.
I don't see how math might not go that way as well.
reply