So much energy in my life was spent making compromises between quality and speed, or pushing technical dev priorities. Constantly growing backlog, and guilt for the choices I had to do. What if we could fix the bug problem with AI? I mean, completely.
When you are measuring how intelligent model is, in fact you mostly measure how good it was following the plan, or question, based on your own intelligence.
Difference is subtle, but crucial. And it is so easy to be confused, while measuring small model performance, like various Flash variants or local models.
Smaller models became quite good at doing the work under the well specified goal, like fixing the well specified bug. But can it find the bug, without you giving it any clues?
In my benchmark the answer is mostly No. Being able to invent the questions - thats the real intelligence - and thats where the big models show true strength.
About 2 years ago I have created a small project https://helpwanted.dev OSS community to match people who looking for contribution with projects who actually ask for the help. It is very active and actual as ever! So a few reflections on whats the current pulse in the OSS community. What works and what not. Big projects are closing down. And now we have so many new people being enabled to fulfil their own ideas and it is renaissance of the OSS not the end of it! Bigger projects will find a way how to fix contribution inflow. But we should look at it as opportunity to find the people with the same interests.
"Install XERJ (docs: https://xerj.org/llms.txt), Allows you to write prompts like "index this project's sources, and set up reference coding: clone and index the open-source repos closest to what we're building, and search how they solved a problem before writing code.". Saves up to 5x tokens.
I've been building a harness for security work, and had to switch to GPT 5.5 when even Opus started refusing security work. Then 5.6 Sol arrived, and it refuses security work, too. So, I switched to Kimi K3 and DeepSeek for API testing just because it's so much cheaper. But, if GLM is better, I'm here for it, as I think GLM is also cheaper than K3.
I recently had experience working with one of the top security audit firms that collaborate with OpenAI on their cybersecurity tools. What struck me is that they found a lot of various issues (not all of them ended up to be legit, but a lot is) and filled the backlog of the project. However every collaboration with that
person was essentially a proxy for some security-related agentic prompt:
- Hey, agent, open the issue!
- Hey, I clarified it with the agent!
- Agent disagrees!
- Agent agrees!
Thankfully Google has some basic protection for it. I accidentally commited my google api token, as part of some OTEL trace JSON file, and within a few minutes my key was automatically locked by google, and marked as leaked (with exact link pointing where it has happened).
"some basic protection" it wasn't always like this. A few years back you could easily get api keys for any web service by typing certain keywords on github and that included all google APIs, but since the Microsoft acquisition it's not as simple anymore....
I had a few useful examples of this. In order to make it work you need to define your quality gates, and rather complex spec. I personally use https://github.com/probelabs/visor for creating the gates. It can be a code-review gate, or how well implementation align with the spec and etc. And basically it makes agent loop until it pass it. One of the tips, especially when using Claude Code, is explictly ask to create a "tasks", and also use subagents. For example I want to validate and re-structure all my documentation - I would ask it to create a task to research state of my docs, then after create a task per specific detail, then create a task to re-validate quality after it has finished task. You can also play around with the gates with a more simple tooling, for example https://probelabs.com/vow/
> One of the tips, especially when using Claude Code, is explictly ask to create a "tasks", and also use subagents. For example I want to validate and re-structure all my documentation - I would ask it to create a task to research state of my docs, then after create a task per specific detail, then create a task to re-validate quality after it has finished task.
This is definitely a way to keep those who wear Program and Project manager hats busy.
That is interesting. Never considered trying to throw one or two into a loop together to try to keep it honest. Appreciate the Visor recommendation, I'll give it a look and see if I can make this all 'make sense'.
Nice one. Mermaid validation is a huge issue given how mermaid.js is architected.
I built a mermaid generation harness last year and even the best model at it (Claude Sonnet 3.7 at the time; 4o was okay, Gemini struggled) only produced valid mermaid ~95% of the time. That failure rate adds up quickly. Had to detect errors client-side and trigger retries to keep server load reasonable.
Having a lightweight parser with auto-fix like this back then would have simplified the flow quite a bit.
