I've had a similar experience (but slightly different). The biggest innovation that ChatGPT has brought to me in my programming/workflow tasks is the comfort and ease in which it creates shell scripts.
That's an area that's always been just past my comfort zone ("Create a shell script to solve this problem? Now I have two problems...").
But all of a sudden, I feel like a master at shell scripting! Things that have always been manual-and-slightly-annoying are now scripted away into automation world, and it feels like I have a newfound superpower :) This past week has been a workflow productivity superweek.
This is currently one of the biggest impacts it's had on me: I no longer fear programming languages and DSLs that I'm not completely fluent in.
I've use ChatGPT to confidently produce jq programs, and Bash, and zsh, and ffmpeg scripts, and AppleScript - all tech I've mostly avoided in the past because I hadn't climbed the learning curve far enough to be confident with them.
To me it's an extension of that great feeling when you need to do something in an area you're not as fluent in and you find an SO thread that perfectly addresses your use-case, or at least gets you 90% of the way there. But with copilot you can do it from right within your IDE, or use ChatGPT to iterate on the results.
Using it with “old” systems, like bash and objective-c, is why I decided to shell out $20/month. It easily saves me $1.25 of pain, per work day.
But, it also really highlights that it’s trained on data, and doesn’t “know”. I’ve never seen it produce a bash script/line that properly handled spaces, on its own, because most shell scripts don’t. Even ChatGpt 4.0 backed fails to “correct” the mistakes, a large portion of the time.
Shell scripting and JavaScript are strengths, any kind of server-side language like Scala or C# it really falls apart quite quickly. I think that integrating into existing APIs is the Achilles' heel, GPT-4 doesn't seem to do any better here either. It makes me wonder if someday we might not get better results completely eschewing libraries altogether and letting the transformers generate literally everything--perhaps recreating similar client libraries over and over and perhaps based on specific existing human-authored libraries, but each one unique to the service, e.g. no external dependencies at all.
I think it will be quite the opposite. Generating a lot of code means generating a lot of bugs. LMMs should get the (battle tested) libs they have to use as part of the prompt, and they should just use those APIs.
I’ve also been wondering if no framework coding will be more productive for LLM driven software.
How many times have frameworks like React changed since it was created? A changing framework means the LLM may mix training from multiple versions and create errors. Whereas libraries that don’t change can produce consistent results. So use stable libraries that choose backwards compatibility at any cost, over the ones that change with the times.
They aren’t trained on assembly. Even if they were, the tools for debugging assembly wouldn’t be as helpful if the LLM wanted to iterate when given compiler errors. The compiler errors they’re trained on are errors humans run into.
Now they also have training on frameworks but those keep changing. So I think the best abstraction is plain javascript - it has the most training data and javascript written twenty years ago still works today.
I can agree to that. I've been writing a lot small cli utilities than before, with argument parsers, help message ecc and in relatively small span of time. It is really useful and very fast.
A large time saving I’ve recently discovered is documenting functions. Paste the function in, tell it to write a docstring/whatever, and it’s 95% there, most of the time.
That's an area that's always been just past my comfort zone ("Create a shell script to solve this problem? Now I have two problems...").
But all of a sudden, I feel like a master at shell scripting! Things that have always been manual-and-slightly-annoying are now scripted away into automation world, and it feels like I have a newfound superpower :) This past week has been a workflow productivity superweek.