Not what I do. I'll reformulate the ticket description so that the purpose and as many details as possible about the solution are made clear from the start. Then I tell Opus to go and research the relevant parts of the codebase and what needs to be done, and write its findings to a research.md file. Then I'll review that file, bring answers to any open questions and hash out more details if any parts seem fuzzy. When the research is sound I'll ask Opus to produce a plan.md document that lists all the changes that need to be made as actionable steps (possibly broken into phases). Then I'll let Sonnet execute the steps one by one and quickly review the changes as we go along.
You are making it too hard on yourself. Most people would just paste the ticket URL and type "fix this", then spend the next 3 hours on social media.
OTOH, I try hard to provide all possibly relevant context, manually copy/paste logs to reduce context overhead, always ask to produce an implementation plan and review it before making any code changes. Yet I often feel like a dinosaur here, all coworkers who tout "LLM productivity" just type a few words in and let the agent spin for hours without any guidance.
I'd call that irresponsible use. One of the principles I try to stick by is to never offload any major decisionmaking to the LLM without oversight. Because some percentage of the decisions it makes are going to be wrongful (and more often just against my taste).
I don't know about the GP, but my workflow is similar to theirs, but I aim to ship low thousands of lines per week. The fewer the better. I even tell the agent to only write high SNR tests, otherwise it just adds useless "make sure this function returns this thing we hardcoded".
I usually succeed, BTW. I spend a lot of time planning, but usually each PR is a few hundred lines, and fairly easily reviewable.
I mostly work with Python backends, though these days it might be any language (Ruby, Go, TS).
> What type of features did you implement on your 100k LOC week?
I work on 3rd party API integrations, of which, we have hundreds, each in its own repo. We need to build thousands more at a fraction of the cost. Any given integration historically takes a human a few days up to a few months to build and is subject to ongoing maintenance. We frequently do not have access to the API and we mostly never have a representative data set if we do. Complex APIs tend to expose multiple, entwined data models. Documentation may be wrong or in a foreign language.
I've been building a new framework to do it better. Ideally, we can get an agent to spit them out in a few minutes to hours with a much reduced ops burden for managing the fleet, all with very high confidence. The later requires pushing as much into the type system as possible and leveraging static analysis. Much of the work has been embarrassingly parallelizable. Consider categorizing access patterns across the entire set or ensuring byte for byte parity (over the input space of third party API responses).
This is absolutely not a problem that a human or 2 could tackle prior to AI.
>I've been building a new framework to do it better.
So you're using your software factory to build a software factory. Not building thousands of integrations at a fraction of the cost.
There are many ways to use an LLM to generate a piece of software. I base most of my projects these days around sets of Markdown files where I use AI first to research, then plan and finally track the progress of implementation (which I do step-wise with the plan, always reviewing as I go along). If I was asked to provide documentation for my workflow those files would be it. My code is 99% generated, but I take care to ensure the LLM generates it in a way that I am happy with. I'd argue the result is often better than what I'd have managed on my own.
Yep pretty much same, although if I’m lax at any point of the reviewing (in-progress or final), I’d say the quality quickly drops to below my average manual effort, and then I don’t even have the benefit of thinking it all through as directly. I think getting really quality results out of LLM code generation for non-trivial projects still needs quite a bit of discipline and work.
Anyone test it out for generating 2D art for games? Getting nano banana to generate consistent sprite sheets was seemingly impossible last time i tried a few months ago.
As a AI-aware software engineer currently creating systems that integrate with LLM provider APIs for my company- who also has no idea what an eval is or how a data scientist thinks about RAG. I honestly don't see what value a data scientist would bring to the table for my team. Maybe someone would care to enlighten me?
Your view of what is happening in the neural net of an LLM is too simplistic. They likely aren't subject to any constraints that humans aren't also in the regard you are describing. What I do know to be true is that they have internalised mechanisms for non-verbalised reasoning. I see proof of this every day when I use the frontier models at work.
There must be a mechanism to rate the person submitting the PR. Anyone that wants to submit code to a well-known repo would first need to build a demonstrable history of making high-quality contributions to lesser known projects. I'm not very familiar with the open source scene but I'd find it very surprising if such a mechanism was not already in place. Seems like an obvious solution to the problem of vibe coders submitting slop.
> build a demonstrable history of making high-quality contributions to lesser known projects.
> Seems like an obvious solution
I'm not sure how you would rank quality of submissions for grading contributors like this. Just because a project accepted your PR doesnt make it high quality, the best we can hope for is that it was better than no accepting it?
