> Skills have some instructions but are primarily informed repo specific instructions and keep their context away from the rest of the repo to keep things sanitised for me.
Skills and agents in the Claude world can also be extended and evolved over time, as they are committed "code".
For example, we have an agent which can take a statement or a support ticket and identifies the services, tenants and infrastructure components likely meant in the ticket or request. Similar to a skill, Claude can invoke this on demand in a conversation.
This started very simple, but various people spent time tuning it over the last 4-6 months. They have "taught" it to pick up on jargon from different departments, writing style of different departments, how they think about their systems.
With all of that tuning over time it has become quite "clever" in identifying the mentioned systems and - if requested - the train of thought leading to this conclusion.
Similar things are happening with skills for various task, be it Ansible integration tests, upgrade chores and so on. The first version can be fairly underwhelming, but continuously improving it after each usage can make them very powerful.
This becomes fun if you start to prioritize security work on CVEs rated higher, not affecting you, over security work on CVEs rated lower and affecting you.
Or prioritizing chasing CVEs currently not affecting you over architectural work improving security.
The CVE should be fixed, I fully agree, but where does it land as a priority? Less competent security teams push /all/ CVEs as Sev0 over /all/ other topics.
I don't have a good answer to this problem. The reality is security is becoming very important - for good reasons. If you (as an organization) are spending all your time chasing CVEs instead of the other useful things (such as what you named) you have a problem, but not fixing CVEs right away is a terrible answer. There is some hope that LLMs have found most of the existing issues and things will settled down - but only time will tell if this becomes true or not.
I agree this is a scope or focus thing. Different tools for different things.
If you are already maintaining a tool chain, a supply chain and skills in an SPA framework for multiple customer-facing, client-driven projects, I don't think you have much of a use for HTMX. Hence why at work, we have our full-time frontend engineers doing this, because it gives you the most flashy quality.
I am an infrastructure engineer. I have privately tried out a couple of SPA frameworks, The complexity to set this up for the first time is immense. And if you have rarely touched projects, the churn of the tooling and dependency ecosystem is taking up a significant amount of time for something that may be nice to have at work, or a pet project at home.
It's not cool if you have half a day to touch an old project and everything has rotted away around it so nothing interesting gets done. Something like Pydantic, Flask or Django + Jinja generally doesn't do that. And HTMX can make it nicer fairly cheaply.
I'm right with you. My dad was a farmer and taught me: First deal with the irregular, annoying parts on the outside. Create rectangles with a long side and turning space on each shorter side. Then deal with each rectangle by going back and forth along the long edge. The turnarounds are also a great place for some wheelbarrows to collect the clippings closest to the Komposthaufen.
Hence why I went for the irregular strange part on the lower left first, and then the more regular part on the right. That's the weird rain collection, herb garden, thing we have at the back.
> Then deal with each rectangle by going back and forth along the long edge.
I prefer to work in lands after I have cut the headlands. Reduces turning stress on the grass and acknowledges that the machine (the one I own, but the design is common) is optimized for use in one direction.
> People are, in heavy LLM systems, realising that the most valuable commodity is engineers knowing WTF is going on, and that loss of understanding of your codebase is the #1 blocker to actually getting things done. Any senior engineer knows this all too well
I've been pushing against this "maximum LLM driven speed" in our infrastructure as well and been working on a middle ground:
We can generate a change to Ansible or terraform code in half or a third of the time, sure. But we don't use this to make three times the changes in the same time frame. We rather use the freed up time to discuss the change, the context and the affected systems with 2-3 engineers maintaining them. And yes, at times this means us three are sitting around half a day discussing and drawing diagrams about our systems.
I'm just happy to work in a company understanding the value of speeding up less than we could in the feature direction, but investing this time in the direction of control and understanding of the infrastructure.
Yup, this is how I see teams who care about quality, design and architecture seems to use LLMs, not to "produce more and faster" but to retain same speed but with a lot more confidence and reliability. Hoping this will spread eventually, some companies seem to take a more... hazardous approach to the whole thing.
This is one of the unintuitive parts when you get into operations: If you go to lower layers in the stack, production expands towards dev:
To the product developers and operators, customer-facing systems are production.
To us in infra-operations, dev and testing are actually production as well. Maybe with a lower SLA and easier maintenance scheduling, but if we fry dev or testing, a hundred developers can't work and start screaming.
Within the infra-ops team, our config management tests and the deployment pipelines are production. If those don't work, infra operators cannot test or roll out changes to the infrastructure.
This was recently discovered by a dev team providing a cross-cutting service: Their testing environment can halt work for a lot of other teams, so they have to be really careful with their testing environment.
I think the question there is how do you make test actually test? Allowing them to break their test env so they can actually fix something they broke, imo. It's enabling and it's educational. Too often I see it so abstracted away that devs don't even know what their environment is made out of, so how can you expect them to make performant decisions that align with the infra?
Maybe the cost you pay is in the complexity of security. And then where does that land? Is it ops or security when some ephemeral rotating key or token that's controlled by some test service account doesn't work?
IMO having a test environment is a bit of a failure: you should be able to spin up and stop a full test environment easily, ideally (and feasibly for most systems) on the developer's local machine.
Depends on the type of testing. If you're doing integration testing between systems, then it's not just a matter of spinning it up on a developer's local machine.
You need a testing environment, especially if there are other physical elements that are being integrated (embedded and other equipment).
It's bizarre to me that we live in a world where whole machines and their networking stack can be spun up and virtualized and yet integration testing STILL actually has to go out and touch a real resource running on a different machine.
Yeah, obviously this isn't gonna work for embedded development, but not that many people have that specific problem. Is it really so hard to spin up ephemeral testing clusters on a single machine? Everywhere that I've seen it done, it's a horrible kludge of scripts and hacks that's almost as difficult to understand as the code under test.
When you are integrating to other people's stacks, then you need a test facility where people can bring systems together, sometimes there are also physical elements that are required.
I work in Automatic Fare Collection (think railway passenger gates, ticket vending, contactless readers, bus driver consoles and readers etc as well as the backends that deal with the fare calculation and payments).
There are usually multiple vendors involved, and even with the best API and interface documentation and individual system testing against the specifications, you still need to bring the entire environment, including the hardware, together.
These ITFs (Integration Test Facilities) tend to be in existence for the life of the systems, including after hardware refreshes etc.
Yes you can test performance with jmeter etc, but sometimes you need 20 people walking in a circle between an entrance gate and an exit gate.
Fair enough, but IMO even a system like this one should have good mock implementations of the system hardware, down to and including a simple simulator for the physical layout of people near the machine.
In past years I wouldn't say that, but these days it's so easy to get an LLM to write something like this plus a quick and dirty visualization.
I'm not saying this can be a replacement for a physical test setup, but it's so much easier to make changes when the tests can just give a clear cut "yes, this acts like you expect" or "no, this is broken."
Docker has made this much much easier than it used to be, and the ease of publishing images has led to many SaaS companies providing "dockerized" versions of their services.
I write a lot of Terraform to manage internal infrastructure with dependencies on other internal and external infrastructure that's outside of my control. I define a module parameter for each dependency (usually with a default value that points to the real service URL) and then create a second module that uses Docker to spin up and configure local instances of each dependency and overrides the corresponding parameter.
This makes local testing very straightforward: `terraform -chdir=tests apply`
Sure, for integration with some specific hardware, there may be some limits (though I would ask if that could be simulated for most testing, or if multiple could be made available). But it's a goal that should be aimed for and is achievable for most software, if some effort is spent on it. I see a lot of places seem to cargo-cult a dev/staging/prod setup (each of which inevitably winds up slightly different and hard to reproduce for incidental reasons) and it seems like an approach with a lot of headaches that you probably don't want to use unless you're forced to.
When you have a lot of interdepended systems it is not always possible. In our case our test environment cannot be replicated locally on developer machines because it integrates to various 3rd parties to which we only have one set of test credentials.
In our case, "testing" as an environment has the goal to be a fully integrated setup of the entire service landscape. Each team is responsible for the uptime of their service, just like in production.
The teams are running automated end-to-end tests against this environment. On top, pre-sales uses it to build demo and sample environments, sales shows demos on it, solution engineers use it to test/try out the important parts of projects.
That generates a good production-like experience for the team: You have customers using the system, and possibly screaming if you break it. And it has been pretty successful at catching problems missed or integration failures.
This is correct, but there's additional nuance here, which is the security of your development pipeline plays a part in the production system too.
Way back when, you'd see a lot of people stuff their jenkins in their dev account. There's no way that the system that builds, publishes and deploys your application should be treated as anything but as sensitive as the production system itself.
( To be fair there are mitigations such as reproducible builds, but you're now doing a bunch of engineering to tie in your deploy system. )
Agreed. I've started calling our non-production infrastructure environments 'engineering' rather than dev, and nobody's invited. We'll pull in platform teams that are exposed to this shame environmental offset issue, but no customers.
In my experience, business software has narrow and predictable use cases. Note taking: You need to add, delete and update notes and the real meat of the product is usually in the note organization. Even SAP has use cases simple in nature: Account for milk cartons sold. The use cases just grow complex once you work in 30 countries or more. "Pay the right taxes on milk cartons sold" becomes tricky.
However, this is parallelizable: You can work on all 30 countries at once and ship when ready. You can work on taking and organizing notes in parallel. You can work on scaling in parallel. You can streamline UX in parallel while introducing tagging, or trees, or groups, or anything.
A game, and especially a narrative game, has an intrinsic linear component, the plot. You can work on a certain amount in parallel, sure: Once the rough shape of the plot is laid out, you can start working on assets, necessary engine features, sounds, dialog in parallel.
But you still have a strong need that a player or play tester has a good progression from act one to act two to act three, and a good progression from puzzle one to puzzle two to puzzle three. This takes a lot of very slow iteration.
This involves human testing by nature. You get a playtest every few months, if you have the money. You can easily throw more CI hardware to run your integration tests more often for more countries. Playtesters, especially those who don't know the game, are very, very limited.
Then you also want characters and settings to be consistent across the plot. If done well, this takes a lot of adjustments and refinement both back and forth. Sometimes the setting of Act three requires a character to have a certain trait, and that trait should be developed, so you now go back to Acts one and two to see if you can build up the character -- without knocking down the entire house of cards already in place.
And once you get to polish and atmosphere, it becomes even harder and more time consuming, because then you need new playtesters even more.
From a past job and working on some features, I spent a day or two presenting like 6-8 different options, and then spent half a day in different testing groups to figure out which procedural generation option "feels the best" for what the game designer wants. With a better spec, that could've been an hour. This made players more excited. Probably.
Back when we were working in the office, this was honestly a preferred pair programming setup for us. Two people, and each just had their keyboard and mouse.
Interestingly enough, it seemed that the inputs from multiple mice onto the same cursor is summed up by default. So you could fight cursor movement you disagreed with by moving the mouse in the other direction.
But that seems like quibbling about semantics. Thompson says that if a compromised binary generates a binary, you cannot trust the generated binary.
This does not even have to be gcc, it could be vim. Or strip. Or cat. Or dd.
Or an iop to write data to persistent storage if the firmware of your SSD is compromised.
reply