I've been watching a bunch of bycloud on YouTube recently, and although he's done a great job reviewing papers from the big AI labs, I feel like I'm missing something - how have all the labs seemingly made a model that's cheaper, faster AND has better performance? Historically `flash` variants (like codex spark as well) have been faster but perform worse
Devin was the first ai developer harness I ever heard of — and in fact the first one I ever used as I had gotten an invite while it was in private beta. Now it seems like the product has faded (into irrelevance?) between other harnesses like cursor especially and all the major harnesses having a cloud mode. Is there some other area Devin has become really popular / has stayed competitive?
I was just thinking recently, isn't this fascination with harnesses kind of backwards?
I mean ChatGPT is the harness most of the time that's calling the tools, MCPs, etc. Putting a loop around it doesn't really seem like it would be something that gets you a lot of extra milage.
It gets an incredible amount of mileage. And I believe it is the interface that allows the model to learn and eventually bake that intelligence into the model.
Look at how Astra scored 100% on Arc-AGI-3. It was largely because of the harness.
The harness increases the chances (often to 100% chance) of a non-deterministic LLM to perform deterministic actions.
Not only that, the harness provides the feedback that becomes training data for the model. So, over time the model bakes those lessons in, and the harness becomes less necessary and the agent becomes more efficient at some tasks.
A harness will likely always be necessary, we may hit some level of complexity or some level of compute that never allows us to bake the lessons into the model, and external tools provide the model with the ability to find leverage and make up for those short comings.
I did not imply harnesses were useless, just that they're merely a functional part of running an LLM. But it's ticking a checkbox. If the harness can support LLM capabilities (tool calling, reading the screen, visual interaction), it's reached the limit of its potential. Like a JSON parser - can it do the job or not?
Astra didn't solve Arc-AGI-3 because of its phenomenal harness, just a harness was needed to support its capabilities.
Giving Astra's harness to a different LLM would not have improved said LLMs ability to solve the task in a meaningful way.
I feel like this is important to point out, seeing how I'm commenting under an article about how a harness is valued at $50B.
Imagine valuing a JSON parser at $50B on the basis that most communication can't be read without one.
A harness is fundamentally required - although we don’t really see it using tools like Claude or codex ai models still only can take tokens in and produce tokens out. Tool calls are literally just the ai model printing “I want to call search_web with parameters abc” and the harness sees that in the text stream and runs it
I think as a hacker/engineer type you’re more aware of what’s possible and because of that you expect more from things in your life. The flip side of that is that it makes features most people would never think of feel like table stakes and combined with the ability to build your own solution - IMO leads to over engineering.
"yes" but that's a philosophical question. I think they're getting better at "early fusion" IE, training the model that "apple" and these visual tokens are the the same concept, but LLMs are fundamentally a pattern matching machine so even with perfect fusion I personally wouldn't call it understanding.
> Cross out stuff you disagree with, draw a big circles around the stuff that resonates.
> Not only does it make consumption a lot more engaging, it makes a revisit of that book extremely rewarding
This is actually the biggest reason I own a Remarkable 2/Kindle Scribe, let's me "destroy" the book while also not having to worry about losing my notes.
Honest question: How confident are you that you'll have access to those notes on your Remarkable/Kindle 20 years from now?
I love my Kobo. I highlight stuff with it. But for anything really important, I then go to my computer, go through each highlight, and make notes in my org file.
(I probably should automate it - I know tools exist to extract all that metadata from the Kobo).
So, this is kind of not directly related, but sort of is.
I teach an adult Bible study class, and I use Logos for all of my notes and highlights. Your question touches on one of my specific concerns, because they use a proprietary format for their datasets. The best solution I have is that I export the notes and format them in Markdown, and I distribute them to the class via one of my blogs. Obviously, I can't include verbatim copies or quotes from the sources I use, but it's better than leaving the notes inside a platform that could (potentially) disappear in 20 years.
I cite all sources used in the footnotes of every session, so at the very least, it should be possible (theoretically) to obtain the source material in printed form and find the exact page number (Logos stores page number metadata in the digital copies). It's probably not ideal, but it's the best thing I've come up with so far that doesn't require a lot of duplication of effort.
I'm personally extremely confident. Those Remarkable tablets are very hackable. I pay for their sub, but I've also whipped up my own app for syncing content between my RM2 and my home server so that I can send news articles to it, mark them up as I see fit and archive them.
That's a decent enough solution if they let you automate somehow (e.g. sync to PC once a week without me thinking about it). I could then build further automations to parse those PDFs.
The Remarkable can spin up a web server if you connect it to your computer via USB. You can then access the files through a web browser by entering a dedicated IP address, which the Remarkable will give you. This features is disabled by default for obvious security reasons, but it's trivial to enable in the Remarkable's settings.
Point being: as this feature works perfectly fine in Airplane Mode, you don't need an internet connection to use your Remarkable.
> This is actually the biggest reason I own a Remarkable 2/Kindle Scribe, let's me "destroy" the book while also not having to worry about losing my notes.
Well... I'm mostly back to paper after losing most of the notes I had taken on my own Remarkable 2.
If phones didn't have secure boot nowadays inevitably little Johnny would (unknowingly) install a rootkit on his mom's phone for the promise of free vbucks or some see through walls app.
It's not open weight, but the point is to be an on device (and thus local, privacy preserving) option. The article mentions that as the caveat
> What this means if you just want good transcription
> If you are on a current iPhone or Mac, the best on-device transcription engine for English is already in the operating system, and the private option is no longer the compromise option
Presumably if you don't trust apple you wouldn't purchase their products and even if you were for example forced to use it via work or something you wouldn't use this feature ... so it doesn't really change the calculus as presented by this article - IF you ALREADY HAVE a MODERN Mac (and trust apple) this is your best option
You are being naive. An Apple device makes dozens of network requests every minute or so to Apple. It is neigh on impossible to verify what is being requested or sent. Also unplugging the Internet and verifying that something still works does not mean the app won't phone home behind your back when it can. These things are designed to fail silently.
There are many alternatives for trying to find out what’s going on. If you don’t want to bother, and most people don’t, well, what else is there to say?
It is generally a good idea to know what software is phoning home, if you can pinpoint it.
If you have any software recommendations, I’d be happy to know.
It's more complex than that. LittleSnitch or other application firewalls won't help when programs like package managers legitimately need to access the Internet. And figuring what what is being sent would require decrypting the traffic. There is no easy way to figure out if Apple or some rogue app is siphoning off your home directory.
reply