This is probably a controversial opinion, but I think that working in an environment where time-to-iterate is high is actually very beneficial for improving your skills. It's one of those things that may feel overly burdensome in the short term, but is better in the long term. I say this as someone who taught programming with beginners, and observed what many of them will do when given an IDE, which at the scale of the code they're writing, makes the time-to-iterate very short (press a key and you instantly see your changes). They end up getting into a "dopamine feedback loop" that causes them to continually make tiny changes and rebuild/run, and the code written as a result of this looks exactly like what you'd expect: it barely works, and is full of redundancies, "dead ends", and other evidence that its author was probably not thinking of anything more than the next line or two when writing it.
In other words, reducing the iteration time reduces the motivation to get it right the first time, and the associated deep reasoning/"mental execution" skills which are required to do so. In order to develop those skills, one should strive to write as much code as possible before running it, and a high iteration time assists with that.
Also, I'm tempted to continue the title with "...unless it's APL".
Isn't that just what it's like to be a beginner? I think it's difficult to grasp when you've been coding for a few decades that new developers don't immediately see what effects a piece of code has, they can't just look at a loop for 2 seconds and say "oh, there's an off-by-one there". Sure, I can worry about the big picture while I'm typing out the code because I don't need to pay much attention when I'm typing out the code. The syntax of code is a no-brainer activity. I've been adding semicolons at the ends of lines for longer than some developers have been alive, it's not going to be something I miss. A beginner needs to focus on the details of the lines, which means their code becomes myopic and line-focused. I don't think you can meaningfully widen the scope before you're able to do the simple steps automatically.
Experienced developers typically don't sit like Hari Seldon predicting branches 50k instructions into the future with some extremely elaborate grand master plan, they're so fluent in the code and architecture that they don't need to. This is tacit knowledge that is acquired along the way, I don't think it can be meaningfully practiced.
> Experienced developers typically don't sit like Hari Seldon predicting branches 50k instructions into the future with some extremely elaborate grand master plan
I think that final sentence is unduly pessimistic - after all, if practice did not play a big part in expert programmers becoming that way, what did? What hope is there for the novice who is frequently making off-by-one errors and the like?
Maybe you are saying that there is no deliberate method of practice, but I think there is: whenever you correct one of your mistakes, try to figure out how you might have written the correct code the first time. Often, it might be a boneheaded mistake, but a significant number of cases have something to learn from. Can you make an argument for the current version being correct, and the previous one not? If an API did not work the way you expected, did you overlook something in the documentation? (Maybe just that the documentation was ambiguous, and you might have realized that earlier.)
I think OP's point is that slow turnaround encourages that sort of thinking, and it does, though it is quite a cost to pay.
> I think that final sentence is unduly pessimistic - after all, if practice did not play a big part in expert programmers becoming that way, what did? What hope is there for the novice who is frequently making off-by-one errors and the like?
It's not that the knowledge can't be acquired, but this is a type of knowledge often can't be explicitly practiced through some training routine.
A beginner may of course learn to check off common errors they've been told about, and may arduously step through the code in their head line by line, but an expert sees the code is wrong unconsciously and understands what the code does without stepping through it. The only way to learn that is to engage with code for decades. There are no shortcuts.
I agree that competent, experienced programmers look at code differently, and much less linearly, than do novices, and there have been studies that show this. On the other hand, a person could (and people do) spend years programming without learning to look at code in this way. The people who do improve must be doing something different.
There are no shortcuts, AFAIK (though I think most people can achieve most of their potential in less than a decade), and I am sure you cannot get there through rote learning. Are you suggesting, however, that the sort of approach I am advocating is not very helpful, or just that it cannot be explicitly practiced?
> They end up getting into a "dopamine feedback loop" that causes them to continually make tiny changes and rebuild/run,
Yeah that's what I'm doing all the time. Just constantly testing the thing after every little change. If the test is fast, I like it. Probably there is some dopamine thing. I have thought of it in the past as like a slot machine.
> and the code written as a result of this looks exactly like what you'd expect:
Why would you expect any particular thing?
> it barely works, and is full of redundancies, "dead ends", and other evidence that its author was probably not thinking of anything more than the next line or two when writing it.
My code isn't like that though. Maybe that code was like that because it's beginner code.
> In other words, reducing the iteration time reduces the motivation to get it right the first time
I'm always hoping it works on the first time but you know how that goes. Even when it's close there's some little thing. The hope of getting untested code right on the first try isn't really something I entertain too seriously anymore. Short iteration is lovely.
Getting a particular piece if code running from the first time is so unexpected that I'm like "Huh... that's pretty cool" everytime it happens.
no need to spend 3 more minutes looking at the code when a 2 seconds build will tell me there's a Segmentation fault right there, or when Valgrind will tell me I missed a free ().
Well, yesterday I made a rather significant change to the code. Essentially, a reimplementation of an existing function in a completely different way, to avoid a memory consumption issue that cannot be solved in the old approach. 68 insertions, 15 deletions. It, unexpectedly, worked correctly and passed all tests from the first attempt, even though there are threads involved, and they are my weak point. But I didn't have a "Huh... that's pretty cool" moment. I had an "I don't like this code, it's too complex, it must be wrong somehow" moment.
Only a small fraction of segmentation-fault-causing errors will be revealed in 2 seconds of testing. Thinking about all the ways your code could fail, and being able to keep a mental track of the constraints you need to observe, is a useful skill to acquire.
The problem is many times, you can't really reason through the issues your encountering. For example, if you are trying to talk to a server with an undocumented API flow, or you're trying to coax an in-house library to life, you have no option but take baby steps and test every single change to see if you made progress.
In environments (like C++) where the compile times are high, and it's pretty easy to introduce almost impossible-to-trace bugs, I'm often tempted to do a bunch of changes, and then test them all at once, which will invariably result in something breaking and me having to spend an inordinate amount of time to figure out what went wrong.
Yeah, this breaks down in this case. I've wasted so much time making small changes trying to figure out how to adapt a model in the right way for it to work in both real and test code, getting a weird injection to work, etc. on systems with agonizing build times.
Your example is valid for beginners but once you're getting experience you don't code like that anymore.
After 13+ years of C#, on simple projects I can code for a couple hours without compiling and get the pleasant surprise that my code work the first time I run it. And I'm certainly not a genius, I'm a 1X programmer.
I still do. Of course dabbling into REPL-based solutions in my youth might have influenced that.
For me it is crucial to always have have the code in some compile-able and testable state. I iteratively work towards the simplest, most naive solution that gets the job done. After that it is refactoring time and I form it into a proper solution.
The reason I do so is because only after having coded the naive solution, I have an concrete and proven correct understanding of the task and can pick the appropriate abstractions. I always see colleagues with an less iterative style use premature-abstractions and over engineering on the basis of "we might need it" instead of knowing.
I do invest some time into thinking about the right data structure though. (Not much about algorithms, as they mostly follow from the data structures anyway.) Also if I code a well understood problem, I might use a more top down approach.
(Not a critique of your workflow, just wanted to share mine.)
Eh, ten years a professional, I frequently fall into it still, where you forget to think about the best solution and just.. mash. Sometimes it works to get a solution that can then cleaned up, other times I waste an hour before I snap out of it. I saw someone post "Stop thinking and look!" as universal advice once, but I need the inverse just as often.
I have been coding for over 30 years and I can barely write one function without at least making some typos if not logical errors. I would never code for a couple of hours without compiling/testing.
> In other words, reducing the iteration time reduces the motivation to get it right the first time, and the associated deep reasoning/"mental execution" skills which are required to do so. In order to develop those skills, one should strive to write as much code as possible before running it, and a high iteration time assists with that.
A high iteration time [a.k.a long feedback loop] leads to distraction more frequently than to “deep reasoning”. Human brains seem to be wired to find distraction after about eight seconds have elapsed. [1]
Furthermore, let’s not forget that an IDE doing a lot of error checking real-time, often including partial compilation and execution, so nobody running an IDE is coding for a long period without compiling, they’re more likely coding for a long period without integrating.
If we reframe deep reasoning as “stay engaged with the problem while waiting for some feedback”, we can see that there are a lot of tricks for this: Unit tests, for example, or using a different medium like pencil and paper or a whiteboard. (I’m sure this group can come up with a great list.)
There has to be some nuance here. Feedback that comes when I'm not at a stop/review point is just a distraction. Is why aggressive spell checking is annoying.
This was a thing learning to type, for me. Our class would transcribe from paper to document. Those that watched the screen were slower, once folks learned to touch type.
That is, learning to get into deep thought is a skill that eventually doesn't want the distractions. You want to focus and produce, then review.
If you're ramping up, you cannot write a significant amount of code without errors. If you try, it's much harder to figure out which change caused the failure of a previously working system, because you're less aware - by definition, ramping up - of the mapping between edits and failure modes.
Once you're ramped up, yes, it's more efficient to produce more code and fix the odd issue that crops up. I rely more heavily on high level, slower running functional tests on code where I've been the primary author or have worked on the system for a long time. But just be aware of what you're optimizing for: tenured employees.
I wouldn't kid yourself that it's "deep reasoning", beyond people on a dopamine kick, or beginning programmers working with simple data structures. You can't reason much about a system you don't know, because everything is details - millions of lines of code, you can't mentally evaluate it. It's not until you've built a mental model of the moving parts, and are able to mentally abstract away details not relevant to your task, that you can think things through in the large.
That is a really interesting perspective, and something I have noted about myself when working with Lambda in AWS, which has a god awful iteration time.
My two comments however would be that when you don't know your context, lots of probing is required, and that can take a lot of iterating. In industry work you're very often working on a small part of a larger software orchestration, and it's often entirely new to you when you're asked to work on it. So you don't get access or time to understand the full scope of the software to reason about. So you need to "discover" your context as it actually is at run time. This is especially true when bugsquashing.
My second comment, is that we should all have a good debugger. Having a debugger is like having a superpower. I can come into a codebase I've never encountered before, step through as it runs, gain enormously useful context, and solve the problem I wanted to solve, without ever running the code to completion or having to read every bit of logic.
If I'm working on someone else's codebase, usually I spend time reading to figure out the general architecture, then dive right in with a debugger and start stepping through. I find how to trigger the code paths related to my task, see what's happening as it happens, and get a really good understanding of how it's all running.
In what might be another controversial opinion, a working programmer without a debugger is like a working carpenter who's chosen to do everything with a knife. They may produce good work with their own way, but they could be faster with the proper tools.
I even see the mindset spreading to other activities, like writing an abstract, paper, proposal or preparing slides:
I regularly (more often than not) see students using their supervisors as "CI-chain", submitting iterations with lots of stupid errors, half-baked sentences, unclean structure... because they learned to hit CTRL-b and see what comes back.
Of course there are reasons to talk about half-baked ideas, but if your task is writing a proposal or a section, you should at least be happy with what you wrote before you send it to the Prof, you're wasting his time.
But I also want to say that CI, unit tests and quick iterations are still helpful, even if you know how to approach a problem (i.e. think, research, draft, test, revise, etc.)
Your stance looks self-contradictory. An introduced change either brings enhancement, or not. In the latter case you wouldn't expect anybody to be excited. It means one only gets dopamine boost when getting closer to a goal set, i.e. the feedback works absolutely right. If after a bunch of such changes one arrives at "barely working" it indicates poorly defined goal, and/or lack of underlying knowledge. And has nothing to do with length of compilation. Actually, that's what we can expect from beginners, right? With quicker feedback they, most likely, will have a chance to absorb knowledge quicker.
Many early programers, Knuth etc, have described how writing programs on punch cards, submitting the job, picking up the results much later had a big influence on how they thought about programing. The lack of immediate feedback, counterintuitively, made them better programers it seems.
Or the people drawn to the tedium of early programming were just really good and mentally twisted in the right way to enjoy it.
My first programming experiences were BASIC on TRS-80, TI-99/4a, and similar machines. It was instant feedback, for the most part, and I enjoyed being able to make small changes and seeing the results. I was around 12 years old and that worked well with my attention span at that time.
> Or the people drawn to the tedium of early programming were just really good and mentally twisted in the right way to enjoy it.
Agree.
I think what was happening was the the lack of feedback fit the mental models of the people who were attracted to computers early.
For another example of this,Ken Thompson famously said that he never quite understood why anyone would want to see the whole file they were editing instead of the single line.
But instant feedback attracted more people to programing.
I would not conflate slow feedback loops with intentional breaks
For me slow feedback loops push me into dopamine seeking activities, intentional breaks push me into relaxation and deep thought
Recently I got a treadmill desk and the physical exertion reminds me to take breaks where I can get off and relax and potentially think about a problem
Slow tests running causes me to not write tests and look at YouTube
There is a cut off period though. In my current role there is one project that takes 15 minutes to clean build, and another 15 minutes to run the test suite. Incremental builds are fast, but sometimes/often the compiler decides it needs to rebuild everything.
Additionally for some reason on my machine, maybe 25% of the time it gets stuck while building and I need to kill it and start again. Usually I go and get coffee while it's building, but there's only so much coffee or Reddit you can drink. Many times I've taken a 30 minute break while it builds and tests, then come back to find it got stuck and need to start again. Anytime I have to work on this project I try to put it off as much as possible, as I know it's going to be painful...
When I was a beginner I'd write code and run it very fast, after each small change, even though an experienced developer could probably compile it in their head and see whether it worked or not.
I think just as you get more experience you tend to write more code before you run it. I've written an entire (relatively small) application and written the tests for it before I even thought about running it. Run it, few tests fail, iterate a tiny bit, done.
It's just natural as you get more experienced that you can reason about code in your head without resorting to running it to confirm it.
Let me tell you, time to iterate on a huge 50 year old code base is through the roof and reaching out into interstellar space. The resulting code is pure and utter trash.
I would kill to have to have a quick feedback loop and at least be able to address quality issues through training, culture, or technical controls.
The alternative is fighting against a giant, heavy machine with a lot of inertia in the wrong direction.
I'd say the opposite. The quicker the feedback loop, the faster you can learn. Just because some beginners choose to get the feedback faster doesn't mean they are doing something wrong.
This is the same argument in favor of manual photography with film, instead of a digital SLR, or a camera phone.
While you might learn more if forced by the constraint, the opportunity cost is quite high, and these days you'll never replicate the experience of going off with your glass plates and camera packed on a mule to photograph the Sierras.
Not only all of that... but Ansel Adams spent a TON of time in the darkroom, and pretty much invented most of the techniques that Photoshop emulated in their early versions. Even faced with limited chances at exposing a negative, he iterated the heck out of the development process when making prints.
I share your view that there is value in the focus, but I don't think that anyone can afford the opportunity costs.
Those short cycles are great when learning a new language.
Experiment with it, try new things, understand how the language works.
But at some point you are better thinking about what to write, how to write it, which structure you will use.
And this skill has to be acquired as well.
> Also, I'm tempted to continue the title with "...unless it's APL".
Or lisp. Or smalltalk. Or forth. There's an entire subfield of computer science, headed by some prestigious people, who believe that working in a REPL is better precisely for these reasons.
At this point I deploy the left-handed user argument: different developers probably find these tools and workflows have drastically different effects on their productivity.
Perhaps also a "code kata" thing. It may be worthwhile to make beginners do each for a few months just so they have the experience (and find out their own "handedness")
I genuinely don't know what to make of the handedness argument. On the one hand it makes a sort of intuitive sense (we're all individuals! Of course we all do things differently!) On the other we're all running on the same hardware, give or take. It strikes me as like learning styles: yes, people might have expressed preferences, but if you run the numbers we do all learn in fundamentally the same way, independently of what we say we prefer.
I think "high iteration time" taken to the extreme is programming without a computer, like when taking a walk. You construct a very detailed mental model of how the program operates and interrogate it in your head.
This only works once you're a bit more advanced, but the one sure sign of a senior dev is the accuracy of their mental model. Which is also what allows you to write your piece without a continual back and forth with the compiler.
This all said, I greatly enjoy ping pong with a repl or instant hot reloading when doing visuals or sound.
In the same vein: I'm very used to "printf debugging". When I use a language where it's harder to do that, I feel like I improve as a programmer because I spend more time reasoning and thinking about my code instead of directly trying to fix the problem I have right now.
As a more general thing: you can learn a lot by removing tools that you usually use from your toolbox. For example, going from a managed language to manual memory management. Going from an IDE to shell commands. Removing tools like language servers. Each time I did one of those things, I learned a lot. However, there are lots of things to learn and I don't know if this was the "most effective use of my time".
Another example of hard to balance learning: I'm currently working for a company. Should I focus my learning on their business and our codebase, or on general technology? General technology is a transferrable skill, and I'm scared of being "trapped" in a company if I focus on their specific parts too much. But on the other hand, companies want to recruit efficient people, and I think focusing on the business and our codebase would make me more efficient.
I'm a junior engineer, so maybe that's due to a lack of experience. If anyone has insights to share about that, I'd love to hear them.
My strategy is you spend 80% of the time on the company problem, and 20% for learning. By that you have both the time to deeper your professional skills and get new inspiration from new technology.
I worked on projects with long compile times and nowadays I will often write code all day without executing anything and then spend the last hour running and fixing it. Maybe that's a skill I developed because of long compile times, but whenever I work on a project with short ones I feel much more motivated and productive. Your experience is probably biased from watching beginners learning to code.
In my experience developers develop their own "unique style" to deal with with this problem. Most often it is horrible pile of mess. Often without unit tests since they do not give much feedback. And acceptance criteria is simple "it compiles and prints ok".
With short iterative changes I have unit tests and fine grained git history, to untangle it.
This holds true for ML models too when the data is large. I've seen people(myself included) when the data is small, the iteration is faster when selecting the features but when the data is large then you have to think about the features that you want to select a bit more deeply, test it out on a sample and then submit it to the larger data set.
I've observed "trial and error coding" many times in my life. This isn't caused by a short feedback loop at all. The people who work like this will work like this with a long feedback loop too, they will just take a lot longer to do anything. These people are simply incompetent programmers.
> Also, I'm tempted to continue the title with "...unless it's APL".
Came here to say that. But jokes aside, as an APLer I wouldn't be able to bear waiting for things to compile. I regularly edit functions while they're on the stack, then have them continue…
> But automatic lets you pay more attention to the road ahead, whether you are a new or experienced driver.
In my experience, drivers who primarily drive an automatic are easily distracted from the task of driving. They turn around to their children, fiddle with the radio or climate control, I've even seen many adjust their seating position and steering wheel position while in motion. Not to mention the internet-connected smartphone calling them incessantly, which is propped up close to eye level and blocking part of the windscreen.
Contrast to manual drivers, who will happily drive along without music. Far less prone to distractions and far less likely to touch the phone.
One could argue that the focused people who enjoy driving do prefer a manual transmission, to which I'll agree, but I'll counter that the manual transmission itself fosters that focus and enjoyment in the driving experience. The boring automatic is what invites distraction and does not foster enjoyment in the driving experience.
Interesting. I was thinking that you pay more attention with manual. The point of manual shifting is not in how the car works. The point is in knowing what will happen before it will happen.
> whether looked at by human factors leading to accidents, by driver age or by accident type and age, the results all support the conclusion that, among all accidents, the accident rate for AT vehicles is twice as high as for MT vehicles except for head-on collisions
That is a very good argument for controlled large feedback times in a learning environment, also, just some of the time, when thinking about your code is the focus of the exercise.
I'm one if the people that thinks that coding in paper is a great learning task. I tend to not ask people to do that in practice because it's a very effective way to lose control of a classroom, but I do use the second best task that is asking people to explain their code.
Anyway, if thinking about the code is not your focus, the increased feedback cycle is all cost and no gain.
Wow, I never quite formalised this in my head, but I think it's true!
At least for small delays, say 30s-3m, I'm not tempted to go for that (other) dopamine shot of checking messages or whatever at all. I keep thinking about the problem I'm solving, what to try if the thing I'm currently trying doesn't work, and so on. I don't remember ever feeling limited with such cycles - literally only when fiddling around with CSS, which I quite often just do in the browser's dev tools for the fastest possible feedback cycles.
The drawback with long time to iterate is you have to be in flow to do things. That's non-negotiable in environments with a lot of meetings.
There's also a "sketching" element with faster iteration. On slower iterations, you're engineering. The engineering skill is often underdeveloped but sketching is more valuable without a dedicated designer.
Also some things are poorly documented, often things like UI and browser code or legacy support for say, older Android OS. So even if you have your calculations right it ends up wrong.
I would say that practicing in a long-compile-time environment is part of improving your skills, but that once you have a certain level of skill with a language and tech stack, then it is beneficial to go back to having compile times of a few seconds or less. Generally speaking, it is far better to have a rapid iteration time to quickly test that the ground you are working on is solid as you progress.
But definitely having a phase where you are dealing with long build times helps you enhance one aspect of software development!
My longest compile times were when I was doing iOS development (native), and even then I couldn't kick the habit of relying on quickly looking at my changes and using logging for debugging. Not the best.
On the other hand, at least it's a motivator to look at the build pipeline every once in a while and see if there's anything that can be improved.
That said, working in a typed language with good IDE support definitely helps build up the confidence that things will work as intended before you hit compile.
That's what you'd like to think. But the quantity vs quality pottery grading story is a tested and retold one. Also this might just go to show that you've settled for a slower pace of progress. And this term 'correctness' that indicates perfection and the enemy of perfection - bad for getting things off the ground.
Agree with this so much. One of the main things that I try to do is ensure that the turn around time between testing and writing code is as short as possible.
As an example I inherited a codebase about 2 years ago that would take 15 minutes to run the test suite. It was painful. It was just long enough that it would waste an entire day just trying to implement a simple feature. The project was behind schedule and no one wanted to work on it because it just took too long to do anything.
So I spent a few weeks running benchmarks on the tests and figuring out where the slow down was. I did a bunch of different things, reusing SQL tables, only truncating tables ones that were used, refactored how the tests worked, removed some code so that docker was not needed to do the testing, etc.
I got those tests running in 3 seconds down from 15 minutes. And that changed the whole velocity of the project.
Absolutely. What I find astonishing is how undervalued test feedback time is.
I don't know what drives that, but in most jobs I've had to fight to do the right thing.
Just recently I've made some optimizations to get a functional test suite from 65 minutes down to 3.6. Parallelism is my favorite lever, as it scales so well, and core count is still going up!
I think the core point isn't about coherence, but tradeoffs. Some tests are not worth holding onto: ones that are slow, flaky, and test code that is of marginal importance. In those cases, the ideal tradeoff can be to delete the test. Perhaps to rewrite a better one, or perhaps to just 'eat' the risk incurred by removing it in favor of improved iteration times and the other long term benefits of not having to maintain it.
As always, mentioning that Your Mileage May Vary (YMMV) puts the emphasis/decision on the people facing this or that situation rather than blindly applying advice found on the Internet.
The OP mentioned using tests instead of the entire game. You could also consider splitting the tests. The code base I'm on now has 9 test suites. It would be much slower if all 9 were merged into 1. So, if you have a monolithic test suite consider breaking it apart.
Our team also has a CI/CQ so I never run all the tests myself. I run the tests I think my change affects. When it passes I upload to the CQ and let it run all the tests. Then I go work on something else and check later if it all passd.
In games there is often no need to run on the target hardware for 95% of tasks so choose the hardware that's fastest. In other words, if you're making a game that can run on PC, PS5, XBox then develop on PC and only switch to PS5,XBox when you have to (platform specific features, checking perf, etc.). For VR for example I'd test via PC on a link cable or on a Vive/Index/Rift and only test on standalone Quest when I absolutely had to. So much time saved.
The same is true in other places. I work a very large project that run on many platforms (Windows, Mac, Linux, Android). Building and testing on Linux is 10x faster than Windows, 40x faster than Mac. Once I found that out I switched to Linux for my day to day work and only pull out the other devices if the stuff I'm working on is platform specific.
Docker helped me to significantly increase unit test speed on one project. Each test was recreating database, run dozens of DDL scripts over and over (to ensure clean environment). I reimplemented in in a way that DDL scripts were run once, then container with database was commited to an image and that image was re-used for every test. Also multiple containers were run at once, so tests could be run simultaneously. Docker is a miracle technology sometimes.
That's not normal, because the test is now not testing the code in its eventual context (which is : the transaction commits). In my experience unfortunately this is a very common pattern (tests are far removed from "reality").
Only if your database lacks the tools to do it effectively. "The transaction commits" isn't exclusive with "and you can roll back to before it started".
Would it be possible to cp /var/lib/your-database /var/lib/your-database-clean after it's been set up, then after each test just overwrite /var/lib/your-database again? I'm sure that the database will need to be properly shut down and restarted, but that is far less expensive than rebuilding a large database each time.
Disk is cheap. CPU and memory are the expensive assets.
> 3. Does not allow running multiple test simultaneously.
It does with almost every form of transaction isolation mode. Read uncommitted in Postgres still behaves as read committed, so in Postgres you're never missing the mark here.
> 1. That wouldn't work if code under test uses multiple transactions.
A lot of abstractions interpret nested transactions as save points for this reason.
It's not the only valid pattern, but it is a consistently behaving one that's straightforward to implement and gets you most of the UX you want from integration test design.
I still think that tests would be flaky without complete isolation in a different database instances. Transaction isolation is nice, but for serial isolation you would get errors when you can't commit that transaction and you'll need to repeat the test (which might cause further issues) and for lesser isolation levels you would get some phantom reads which might manifest as a rare test failures which is a terrible thing, because you can't reproduce it.
Maintenance type work tends to be chasing issues you don't personally care about in increasingly arcane edge cases or trying to intuit why someone else, who has moved on to something that sounds way more exciting than this, came up with a specific solution.
Even more so in a lot of orgs maintenance work gets hit by the "cost center" mentality and management only rewards not hearing about it.
Not full time, but documenting interfaces, speeding up tests, generating stubs and harnesses, profiling pain points - all these I find to be meditative tasks, and I do them when I'm not 100%.
I can do about 4-5 hours a day of "difficult" programming, but then I can easily do another 4-5 hours of this maintenance type work, and the cumulative effects of repeated maintenance time keeps everything running super smoothly.
I'd add that the cumulative effects of always knowing that the maintenance backlog is under control and that you always have time for the maintenance makes it much easier to consistently deliver the 4-5 hours of "difficult" programming.
I actually love chasing down other people's bugs. I often get to fix other developers new shiny features shortly after they've hit production. And I love the challenge.
It's also nice that bug chasing has less of an expectation of giving good times estimates, which the software industry is famously poor at anyway.
Going from "manual testing" to automated testing is the biggest jump. Unfortunately, lots of folks (both developers and managers) fall into the trap of thinking they will "use up" their development time on automated tests, and not really thinking how much time they're already wasting just to do (incomprehensive and hand-wavy) manual tests each time.
And while this strategy might work for the folks who actually wrote the code (and thus have tacit knowledge about it), the moment they leave and/or someone new joins the team, all that "speed advantage" is lost and it turns from minutes to hours to days instantly.
Automated tests really are the only way to capture the business logic of any code for it to not to become "legacy code" before its time.
This is where I stop understanding modern ui frameworks. Instead of making an ui inner machinery (controllers) as a set of modules which could in theory run even in a non-browser environment, they pile up everything together into dom/etc. UI could be just a thin layer over usually testable ui-controllers, whose methods could be called by either controls or offline test suites (as in most desktop frameworks). But nope, they do mountains of “smart” views and test them instead as a whole.
…I just googled for “React TDD” and the top no-money result suggests finding divs in containers and spying on dom changes. It’s like testing your cli app by hooking into tty driver’s section in /dev/mem.
Interesting, in my experience React allows me to write more testable code in the front-end than I ever could. Given how React will handle updates to the DOM given changed state, I can extract complex state manipulation code (i.e. code that's most important to test) into independent functions that take state and an event and produce new state, rather than having that all intertwined with DOM manipulation code.
And when it comes to testing what your DOM looks like, even though that does often involve spying for changes (after all, you're testing whether they occur), that's not on the actual DOM but on the virtual DOM, which makes it easy and efficient to run in a non-browser environment.
You have a “data” as a model sitting between the DOM and the controller (mainly methods and watchers). There are some gray lines, like computed properties, but otherwise the controller can run without the DOM and vice versa, you can easily force the model into a state to see how the view will look like.
A lot more testable than the old spaghetti of onClick and getElementById().addEventListener.
For what it's worth, a popular Python testing framework uses something like `expect` under the hood to simply check that the output matches a predefined output.
I agree. The issue is when you don’t have the test harness or the code just isn’t testable. Then it really will “use up” your dev time for not much gain. Or at least it’s a harder sell. My “dark pattern” to sell this is to offer to refactor a small portion at a time, then read through as much code as I can on nights/weekends looking for unhandled edge cases. When I find one, that’s the part I make tests for. Wo and behold, it found bugs! Bonus points if the edge case actually causes issues in prod or loss of revenue. The manager is now convinced we need this thing, or at least thinking about giving us time to work on it.
Very good points. In his book 'working effectively with legacy code' Michael Feathers actually defines 'legacy code' as code without tests. Largely because of reasons like you state.
What do you think about code being too smart for current team? I encountered that use-case three times in my career. It was C++ program which was written by very bright person, used template magic, boost and stuff. Rest of the team were ordinary C developers. After that person left, they had to rewrite his program, because it took too much time to understand how it works and to fix or improve it. Second case was when someone wrote some helper program with Haskell which happened to be important. Same story, nobody knew Haskell, nobody wanted to invest into learning, so rewrote in node.js. Third story: someone used reactive API with Java (Spring Flux) and again same story: team don't understand it and slowly rewrote it with ordinary blocking API.
> Unfortunately, lots of folks (both developers and managers) fall into the trap of thinking they will "use up" their development time on automated tests, and not really thinking how much time they're already wasting just to do (incomprehensive and hand-wavy) manual tests each time.
Actually, currently am about to be asked when the feature will be done, whereas i've decided to actually write tests for it (reusable library code across multiple projects). As far as the reality is concerned, the people who say that writing tests "uses up" time are right: being slow now will be more visible than no one understanding why development takes a whole lot of time down the line, which is when many might just shrug their shoulders and go "legacy code". What's the benefit of making others able to develop code faster, if that makes you be slower - both facts probably being visible to management, though oftentimes without the underlying reasons behind those being relevant to them?
I'd argue that that's why even governmental projects in my country (that are often contracted out to random companies) oftentimes have poor test coverage and just generally not a lot of thought is put into the quality and sustainability of those codebases - they got paid, they could iterate reasonably quickly while the contract was ongoing, why should they worry about anything? The people who take over the project then can also deliver features more slowly, not only because of the poor domain understanding, lack of ADRs, lack of documentation, but also lack of tests - and still often get paid on a time material basis. Thus, no one actually wants to improve things, apart from me wanting to tear my hair out whenever i'm expected to work on garbage like that and fix their problems.
Thus, you see some people in the industry adopt an egoistical approach to it all - join a company, focus on the speed of their own iteration without thinking about the project in the long term, spend a few years doing this and then leave for another company where they'd do the same thing. It might be a cultural thing, but working on other people's projects instead of starting my own feels like losing at this point - no READMEs, no automated CI, no Ansible, no Dockerfiles, no IDE run profiles, no common linting rules, no code static analysis, no local DBs with migrations, no local setup scripts etc. Why should i be expected to toil away without results that are visible to the business because someone else was allowed to neglect the codebase?
At this point, i use unit tests and code coverage rules defensively: to prevent someone from jumping into the project and ruining how things should work with breaking changes, which will be caught by the tests, or by introducing untested and undocumented code, which the coverage rules will catch and make CI fail. Of course, depending on the culture, you might find that other devs talk amongst themselves and before long your tests have @Ignore added to them or have been removed entirely, which is the point where it might just be easier to look for a less dysfunctional environment, despite living in a country that others outsource to and therefore quality isn't a priority.
In short: i agree with your point, but i think that there are social problems at play. You need to actually care about the code that you're developing to prevent these issues in the long term. And even then, if you care, that doesn't mean that other team members will.
From my limited experience, governmental projects tend to have the worst code quality ever. At least that's what I see here. It can easily be explained by pervasive corruption — projects go to those who agree to return the largest kickback and are written mostly to steal money from the taxpayers. With such incentives, code quality does not receive much attention.
> working on other people's projects instead of starting my own feels like losing at this point - no READMEs, no automated CI, no Ansible, no Dockerfiles, no IDE run profiles, no common linting rules, no code static analysis, no local DBs with migrations, no local setup scripts etc.
You hit the nail on its head here, my friend.
When working with others' projects, I try to add tests where I've added/modified features, so that at least I can get a sense of it all and make sure I don't break anything. And for a completely untested codebase, it might make sense to add more integration-like tests (while still using mocks for external dependencies), such that simultaneously a bigger proportion of the codebase is covered, even if at slightly longer test times.
I see this is about game development, but in the web/server world idk how anyone ever thinks its acceptable to make development impossible on your local laptop/computer. Everyone time I do a code review I need to ensure they didn't break local development. It blows my mind that I need to do it, like they think its ok to build and deploy for every little test. When I first joined I spent so much time just getting it to work on my laptop. I really just don't understand the mindset. This is has been the case in multiple places. Who was this first person to break local development and decide it was ok. Please fire that person, black list them from getting jobs ever again.
We have so many tools to make this easy now like docker and yet it still gets messed up.
I think this is why I love green field projects so much. More than being able to work on my own code and design from scratch, I appreciate the fact that no one has messed up how testable and runnable the code is from a developer machine.
In some environments, I imposed a rule: “the tests must continue to pass, and I must be able to use the application on my local machine in an airplane on crappy wifi”
Sure they should. I often spend all day on negative numbers of lines (I always say that the best code I write, is the code I don't write).
The complaint seems to center on CI/D services that provide inefficient build times. These increase the pain of "round-trip" implement-test-refactor.
That's quite valid, as this is how we tend to work, these days (at least, that's how I work). I often take it a step further, and iterate the design as I progress[0].
But I am one that got their start in the waning days of "big iron," when compute time was the costliest and most time-consuming part of the whole process. You'd need to schedule for computer runs, which would often happen overnight. This meant that it was very important to have your code complete, and debugged, before submitting it for compilation.
Argh. I miss it like I miss a case of food poisoning. It did teach me to do my homework, though.
> Sure they should. I often spend all day on negative numbers of lines (I always say that the best code I write, is the code I don't write).
You're missing the point of the article. It's not about valuing quantity of code. It's about valuing minimizing the time it takes to determine what you want the code to be.
If it takes you a 30 minute compile loop to add a line of code, it's also going to take a 30 minute compile loop to refactor and eliminate a line of code.
The latter is actually worse. When your iteration time is high, developers will deal with it and push through as necessary to get features implemented because they have to make the software do a certain thing. But they will absolutely not struggle through a shitting iteration cycle if the only result is cleaner refactored code.
If you want nice codebases, you need a nice iteration cycle.
I experienced this while working for a multinational.
- A long build process and linker step that takes ages. 1-2 minutes for a small change.
- Trying the change on a real device took perhaps 5 minutes.
- A review process that can take days or even weeks sometimes.
- There was a process that merged my change set which often failed and needs to be rerun several times - often 1-2 days were spent on trying to commit something where everyone was doing the same.
- There are no unit test because some manger decided that they provide no business value.
- There were no simulation tools for software be because some manger decided that they provide no business value.
- Testing would be done overnight.
With the right tools (unittest and pc simulation) I would be effective and could test my changes prior to testing on a real device. This was for me the bottleneck and not how long it took to build the project. I was testing code that was just C on a complex embedded device that I could have tested/debugged easily on a PC in 1/10 of the time.
This was really a management problem where the management didn't understand what to do to make their employees efficient with the right tools.
At least you are lucky enough to be able try changes yourself. I am in the "fun" situation where team of 50+ SW engineers don't have single piece of test equipment needed for operating the hardware, and everything has to be manually tested by some QA engineer, taking 1-2 hours minimum...
If I had to order the things that make me productive, starting with the most important:
1. It’s easy to have a fair amount of confidence in changes without even needing to run them. Strong static typing, good/clean abstractions, local reasoning - little to no mutability, code is mostly pure functions with side effects pushed to the edges. Being able to just write a bunch of code without needing to run it at all (and having it “just work” the first time, most of the time) is HUGE for productivity
2. Excellent automated test suite, with a nice pyramid (small number of E2E tests, solid number of integration tests, exhaustive unit tests). Also excellent monitoring and alerts, with automated canary releases and automated rollback. Basically make it so that it’s hard for mistakes to fully reach production. Being able to be a bit of a cowboy, safely, is huge for productivity too. I’ve worked on plenty of systems without this, and then I’m MUCH more careful, but when you really do have this, it makes everyone so much faster, especially once you learn to stop worrying and trust the guardrails (for simpler changes)
3. Test suite runs quickly locally, and is not flakey
4. Running the app locally is quick. Also ideally I can run just the one component I’m working on (like a front end, or service, or whatever), but have it fit into a full system running elsewhere
5. Full build/test/deploy process on CI is quick and reliable
This article emphasizes 3 and 4, and they’re certainly very important, but I think 1 and 2 are even more important. With them in place, often I don’t even need to do 3 and 4 for more trivial changes - just “I’m pretty certain this works,” and then the tests all pass in CI, and I’m very certain.
Compared to systems without 1 and 2, where I have to basically change a line, then run tests and/or the app because I’m not sure it’ll work, then do extensive manual testing to be sure because the automated test suite sucks. Muuuuuuch slower.
3 and 4 are great, and necessary for more complex changes, but 1 and 2 let you ship simple changes super quick, without even needing to run anything locally. That’s fastest of all, and can still be quite safe in the right environment.
> Being able to just write a bunch of code without needing to run it at all
I work with a frightening number of programmers who assume that they can do this... so much so that they never bother testing to see if they were correct in the first place.
FWIW, I always write tests for my changes, as long as they have any impact on behaviour (i.e. not writing tests for adding a little logging or metrics, but am for any new or modified business logic). Every single dev having a strong commitment to automated testing is a requirement for an environment where you can sometimes get away without manual testing.
For any changes where it's not obvious that it'll work in practice, despite the tests, I also run the full app locally, and test manually. Or for any UI changes, obviously. For backend changes, for some codebases, running the full app locally is required for ANY change, period - hard to reason about code, little-to-no type safety, weak overall test coverage, etc. But if you have a simple backend change in an easy to reason about code base, that has great guardrails (great automated test coverage, great monitoring/alerts, canary deploys with auto-rollback), there really is not much reason to actually run the full app for a simple change, which is a nice productivity boost. Just make the tweak, write/modify tests if necessary, and if everything passes on CI, you're good to go.
For the 8 years before my current job, I was working in environments that weren't safe enough to do this ever, and was horrified when devs would merge without actually manually testing their changes. But my current company/environment has a codebase with strong typing and nice abstractions/local reasoning, great automated test coverage, and great monitoring/alerts and canary deploys with auto-rollback. Took me a bit, but I've embraced the "no need to run the app for simple changes" approach, and it works great in this environment - highly productive and still safe enough.
Hah fair, I have too :) But generally have found it not hard to convert them to embracing automated testing, as long as there's a strong automated testing culture at the company.
I agree with your list but really need 4 myself to be a happy camper. Often it's neglected once 2 is in place as most don't seem to find any value by it, or don't care to maintain it. I find that having the possibility to run the real thing (in some capacity) in a cozy development environment with debugger is very useful to quickly learn how parts in the system interact with each other. I have a hard time doing that with unit tests, perhaps because they are so boring.
Oh yeah, I definitely use 4 frequently too - for basically any bug ticket, or for any more complex change/new feature, I want to run the full app. Having this be fast/easy/reliable is huge. If it’s a service oriented app, and I’m modifying a single service, I especially love tools like https://www.telepresence.io/ - run my service locally, with an “auto recompile/reload on save” setup (and sometimes a debugger), while all the other services are running in the cloud, but communicate with my local service.
However, a lot of changes are just small, trivial tweaks, and if 1 and 2 are strong, I don’t even have to bother with 3 or 4.
In a future post, I will go over how web developers needs to start taking iteration time more seriously as the influx of new tools and frameworks starts to bloat up build times.
The newer tools in web dev have crushed iteration times to a fraction of what they used to be. A combination of things like esbuild, fast refresh, yarn's offline cache, and a few other bits can get your iteration times on a site you're manually testing down to milliseconds to both build and update (literally). Decent devs have been writing tests for years; the test runners could be a bit faster but people are working on that as well.
The React app I'm working on at the moment takes well under a second to build, under 10 seconds to run the test suite (could be a sign I need more coverage..), and milliseconds to update the code in the browser in dev mode. It's nice to work with. It's not even clever or special. A default Next.js app will do that. Create-React-app 5 launched a few days ago with more of the tooling too. In Vue things like Vite are based on the new fast tooling and it works well. Vue dev work is fast. SvelteKit even goes further by using Snowpack to remove the bundling step entirely. Snowpack claims a default refresh time of 50ms.
I really hope the author's article about web dev is simply "Yeah, update your tools and you'll be fine."
Heh. I remember a time back when I worked at Google, and we're using a proprietary internal language to construct some monitoring stuff. It took three of us a day to implement some simple thing that was a line or two of code, so opaque and intractable the language was. When we got it to do what we expected, we still weren't quite sure it was really correct. It was really kind of an existential moment where we collectively wondered, "wtf are we doing?"
The last time some one told me something like "changing three lines of code shouldn't take all day"[0], I responded with "Fuck you" because the way they managed to work 'faster' was to make and test their changes in production before committing them to version control.
I also once spent a week pursuing a small-but-necessary authentication flow change across two dependencies (one of them owned by a different team) and three tests, only to be fired for talking to the other team members directly (I was a mere contractor) rather than playing 'telephone' and relaying everything through my management and their management, which would have made it take a month instead.
Neves Law says that "the harder a bug is to find, the smaller the fix will be". The worst Neves ratio I ever saw was about a week to two bits, where a plus (0x2b) had to be changed to a minus (0x2d).
How I wish a single line code change only took a 15 second build.
The java app I work on regularly takes 3 minutes for a single line change. Then you just need to pray that jrebel will work today and actually hot reload your change. Otherwise you have another 8 or so minutes of redploying weblogic.
Of course the ideas suggested in this can help, but when you need to run an integration test which also takes minutes to run you realize that 3 lines of code isn't bad for a days work.
Your problem is WebLogic, not Java. Try a lightweight app framework (Dropwizard, SpringBoot...) that embeds the "app server" right into your app. Of course if you're using WebLogic, you probably don't have a choice.
Recently decided on using TDD for some library code and aiming for >90% test coverage at my dayjob. Frankly, things do take longer to develop this way and i needed to put in more thought into how the interfaces should be structured, vs just doing what people sometimes do in other projects - just relying on concrete implementations or using frameworks that use reflection to do what they want.
However, the experience itself feels like a positive one - even if the code takes longer to craft, it's much easier to work with, since now i don't need to worry about sloppily written if/else chains that depend on enums or runtime type checks, but can utilize different implementations as needed. Also, writing the actual tests allowed me to be sure about how everything would actually work, all the way up to discovering that Paths.get (Java) on Linux accepts almost anything as a valid path string, whereas Windows has actual validation rules, leading to many headaches in regards to the code coverage quality gates that i set up, since the coverage differs based on which platform you run on.
But i guess that my point still stands: in certain circumstances slowing down might actually be a good thing, when you really want to know how your code will work under most circumstances and make it maintainable.
Of course, on the other hand, compilation speeds and other factors in regards to the speed of iteration definitely shouldn't be overlooked either, and having everything else apart from writing tests and actually thinking about how everything will fit together be faster is a good thing! I'm not sure what i'd do if my test suite took 10 minutes to run instead of 10 seconds as it currently does. Probably drink lots of coffee.
I guess it depends on slowing down for good reasons, vs just wasting time because of tooling or other sub optimal circumstances (e.g. what was described in the article, personally i've also seen a local API service be pretty chatty with a remote DB which was slow over a VPN, yet no one had a local DB with all of the migrations in place, and a bunch of other things like that).
Offtopic: Anyone remember how fast Pascal compiled? Now that was a really nice stack to do some stuff in, it's a shame that it never got as popular as Java, or didn't have tooling like JetBrains has, it felt like a more ancient Go (which is also pretty good as far as ergonomics go).
I have somewhat the same experience with my first real TDD-based project. Progress was slow, but the amount of debugging afterwards as almost zero. It feels slower, but more predictable.
> Recently decided on using TDD for some library code and aiming for >90% test coverage at my dayjob.
I thought the point of TDD was that you don't write code unless a test requires it? In my mind (and the little experience I have) that translates to 100% coverage, with exceptions of really annoying side-effects that are near-impossible to test reliably.
> In my mind (and the little experience I have) that translates to 100% coverage, with exceptions of really annoying side-effects that are near-impossible to test reliably.
Yep, like Java supposedly being cross platform, but having different file system implementations, because using one implementation across multiple file systems just isn't entirely feasible, for example, due to how paths are validated.
Thus, no matter what i did, i couldn't get the coverage to 100% due to semantics of how i'm supposed to validate that the low level call works and because i have branches in the test, one that checks whether the correct exception is thrown in Windows and a different exception for Linux. Even if technically the code paths are covered, telling that to a subpar code coverage plugin is hard.
So, i just settled on writing code for everything myself, but having 90% coverage be enforced on a class line level to account for the weirdness.
AWS CloudFormation had one of the worst user stories related to iteration time a few years ago.
The lack of feedback between the writing of the yaml file and validation of the structure/type/format was frustratingly slow. I would pay good money for better tooling in the Infrastructure as Code(IaC) space.
Waiting for the resources to update, fail with cryptic error messages, then slowly rollback only to then fail the rollback. Now in an invalid state, manual resource creation was required before the rollback would succeed.
The AWS cdk has improved this significantly. As a result the sun shines just a little bit brighter.
This is why you hire old programmers, so that you have someone to yell "WTF" when they see you have no testbench. We had that stuff figured out in the PS1 days.
In 1989, I was working at my first (very short lived) US job, for Bell Labs (not the Bell Labs, but related). I was working on their "office-scale" phone switches, the kind that allowed you to forward a call from one extension to another. There was a hard-coded limit of (I think) 16 forwards, and for some reason it was decided to increase this to (I think) 64. This involved changing a constant in a header file, a documentation string and one conditional in the code that for some reason didn't use the constant.
Under budget, but at the end of that week, having successfully pushed through this DRAMATIC CHANGE, I drove from Phila. to the Outer Banks (NC), crashed, rolled the car 3 times, totalled the car, broke my arm, could no longer get to work, decided to marry my first wife, and left to work for a company I could commute to with a broken arm. Small change, big ramifications :)
Another thing here is that often times people are so busy "developing" product that really trivial efficiency gains get overlooked. Things like simply removing unnecessary parts from the build at least when you're testing / in dev mode. Also compartmentalizing the build to the portion of the app you're working on, ensuring it's not building the entire app when you're only testing one feature (you'll be surprised how common this is even especially at large companies). It's a huge value add to the rest of your team to bite the bullet and take an hour or so to shave off some minutes from the time you have to wait for a build. That one hour saves you from many multiples more (hours / days / weeks) of fiddling your thumbs in the breakroom waiting for a build to finish.
(better yet to have someone regularly scanning for these efficiency related things)
Was working in a similar product and it was by far the worst dev experience I've ever had. It was actually worse than what the post describes. It was like having to go through a lot of stages to get to the area I'm working and half of them not working at all - so instead I was doing bug reports on other people's areas which was only slightly familiar with. (Yeah, our CI QA gateway was a joke. Our CI in general was a joke).
It was hell. That much so that I jumped to the first half-decent role that came my way.
Something is wrong with the way we build systems that integrate too many other systems. Perhaps if we designed them with a TDD approach all the way up things would be better...or worse. Dunno. Sometimes what we're trying to do is just way into the entropy zone.
I love this. I made an ask HN post about testing practices for games[0]. The answers I got are interesting and an opportunity to reflect. There are clearly some differences in programming cultural practices.
I’m of the opinion that applying great testing practices to unity programming is very important. It’s been fun synthesizing ideas I’ve encountered into my hobby game dev career.
I’m glad there are others out there doing the good work of generating content to popularize the benefits of good testing practice :)
Whilst I very much agree with the premise I do think this exposes a fallacy many programmers fall for. That passing unit tests means your code actually does what you think it does. Using unit tests this way is a crutch to avoid improving startup times, implementing hot reloading and other schemes to skip gameplay. You lose an awful lot by not testing your code for the job it’s actually expected to do in situ. In games this results in increasing the total iteration time as it’s usually now a different person who actually runs the code, find issues and submits them back.
1) Use unit tests to move quickly through to implementing The Feature
2) When The Feature is complete, run in a local environment to confirm it works
3) When that works, move The Feature to a development environment that more closely mimics Prod
4) Deploy to Prod
Each one of those steps takes an order of magnitude longer than the previous one, so should be done an order of magnitude less often. However if you are finding inconsistencies between steps, then alter local/dev to ensure more consistent testing. (i.e. if it works on Dev, it should just work on Prod, however there are always little issues).
Right but 2 often ends up being perfunctory whereas if you can test routinely against the local environment you’ll see a lot more and for example course correct earlier. It depends on what you’d consider an iteration as well. I don’t think the loop is closed until you’ve seen your code running in situ. Unit tests might let you have more confidence that things will work at that point.
Running in situ could also be integration testing FWIW. Depends on what it is you’re making.
No unit test should be mostly to make sure the old assumptions for code you don't know still hold and warn you if you break something.
Ofc if you do the code and the unit test, all you validate is that the code does what you assumed was right. But in 2 years, when someone touches something unrelated, he'll kiss you on the mouth that your test noticed an edge case that started failing.
If it's not your unit test catching unintended changes, it's your client. You may not want that :)
I'm not arguing against unit tests. Just that using them to speed up iteration is not a good way to approach that problem. They can of course be useful for other things. And obviously the things they don't test will still be found by the client in particular all the things you miss by not actually increasing iteration speed.
You need both. If you don't have unit tests your feed back loop becomes way too slow, but then you of course have to test the "real" running of the program properly before shipping. And you need at least one person who didn't write the code to test it.
Unit tests are useful for some things. I can’t unit test my way to a game feeling good. For that I need a fast feedback loop to the real program. There’s more of that in most applications than most people suspect. And routes to get there. As programmers we make a lot of little decisions along the way and those don’t get proper consideration unless you can see that as the feature evolves.
As someone writing Lisp code, this feels so archaic. Then I remind myself that Lisp is older and than those games and that the future just hasn't caught up yet.
Testing changes here could mean progressing through several seasons of career mode in order to test out a change
That's like the Groundhogs Day of programming. Living the same season over and over again, changing things until you get it right and can move on. Sure, ai guess that's kind of programming in general, but on this level it's most of your day. No developed cheat codes?
The "sub-second response loop flow state" he mentions here is "the doherty threshold"[1] just applied to programing as a particular user experience.
After working at a CDN that delivered a couple hundred kb objects in milliseconds around the world I thought "code is data, so why cant code be updated in milliseconds?" I tried starting a startup that could do this, treat code as data, and therefore achieve sub-second round-trip trial-and-error loops.
Did not pan out, but to be honest we never really got to testing that thesis. I still think it could be amazing, just not sure how much of the lang/ide/build/test/deploy/validate cycle you could integrate and how much you'd have to build.
IMHO this is the main thing that made PHP successful. The "edit a file -> alt-tab -> click refresh" test loop being faster than you could click.
By comparison I've worked with someone who pushed a '2 line fix' live which had fallout which lasted >2 weeks because their code was pushed without checking to production and had hard-coded their uid into the fix because it was a quick 'hack'...
The main reason when cornered was 'running pip install is too difficult to run the test-suite' (tests themselves took <30sec to run all ~300).
We eventually (1 week later) got permission to pull the release from our shared (pseudo write-only) storage area but as he was project lead he tagged a new minor version for his '2 line fix' without consulting anyone and our users were reluctant to move to a new bugfix release unless prodded with a red-hot poker.
Can't say I miss working with _some_ programmers turned managers.
Regarding the "testbeds". I recently built this for an (iOS) application and it helps SO MUCH. Each module in the app has its own local target (the testbed) with a menu which lets you open the module for a given scenario. A scenario is a combination of local JSON for endpoints, device fakes (think fingerprint enabled/disabled) and module-specific configurations. The ability to near instantly get to a specific functionality with the same network requests makes everything super simple. I run the UI tests on these targets as well, and they are near perfectly non-flaky.
The best part is when I receive a new bug report from QA, since they include the network logs I usually just need to create a new scenario, register the JSON and fix the reproduced issue.
> This is the sweet spot amount of time where you become tempted to “do something else” while you wait.
I think this is an important observation. The time where you "wait", you can allow your mind to drift off right after being immersed in the problem. In my experience, this time has the potential to give you great insights.
I think fast feedback is great, but only if you know what you're trying to accomplish. Or trying to learn. Don't use it as a tool to throw stuff against the wall and see what sticks. Once you start to do that, spend some time away from the computer. Draw a picture, formulate some hypothesis based on your mental model.
Make a map before you enter the jungle of trial-and-error.
If what is slowing your iteration speed is testing, I'd say you are in a somewhat happy place. At least you can write test, speed up the build, or somehow improve the situation.
What drains my soul is when you need to use legacy or poorly designed tools that slow you down and you can't do much about. Here are real examples of times I wasted entire mornings: tools that, if you make a mistake, leave the environment in an inconsistent state, and then you need to manually fix it (it isn't documented and changes on a case by case basis). If you miss anything or make another mistake, start from the beginning again.
This is something I've worked to get right my entire career. Whenever I start a new project, I make sure the feedback loop is small. If it's a new technology for me, this can take a while to get to a process I'm happy with, but it's worth it.
Way back when I was at uni I remember watching over people's shoulders at the painfully slow iteration process. I couldn't believe not a single one thought of doing anything about it. They looked more like factory workers, endlessly turning the same handle and waiting. If you spend your time doing this, you will have no time left for creativity.
Entirely relying on unit tests for development can never be the only solution, because tests are quite simply not the product, and they shouldn't be a substitute/workaround for fast iteration right on the product. Easier said than done of course, especially in such a complex environment as game development.
The future for fast compiled-code iteration in game development will most likely be hot code reloading, so that code changes are compiled and implanted right into the running game instead of requiring a full linker run and then getting back to the right place in the game to be tested.
I don't think anybody is ever suggesting this - but I do see people suggest the opposite: that all testing should be end-to-end "black box" testing, and unit tests are a waste of time. If you actually want to ship something that works reliably, you have to do both unit testing and end-to-end (integration) testing. I've never seen anybody sacrifice integration testing. I have seen them sacrifice unit testing.
Sounds like EA has (had?) actually pretty good developers. The game itself has so many things gone wrong that from the players perspective it's very hard to find any good words for. Shame that the company hates their customers so then customers started to hate anyone involved in EA. In the forums of the game, devs of this game are considered to be the laziest and the least skilled across the industry.
PS. Anyone did figure out whether DDA is officially in the code or is it just a matter of network quality difference between players and the server (the game stopped to be p2p in any mode)
My first full time job as a junior dev was to maintain a legacy GWT project which makes tons of $ every year.Setting up the development environment took 1 week. For some odd reason we couldn't build the project module by module. Build was taking at least 2 mins with the bare minimum module count. I quickly become depressed. Waiting the builds and half way into build an error pops-up and you start again. This was 4 months ago, i literally became a deppressed junior dev who hates his job in 1 month. I started applying for job after a month and quit that job after 2 months.
Bill Atkinson, the author of Quickdraw and the main user interface designer, who was by far the most important Lisa implementor, thought that lines of code was a silly measure of software productivity. He thought his goal was to write as small and fast a program as possible, and that the lines of code metric only encouraged writing sloppy, bloated, broken code.
I love building test harnesses. I think the trick is making sure that your system is built in such a way that 1. You can actually pull the core of it out to run in a different situation and 2. You keep enough of the real system such that you’re not getting lots of green passes which are going to completely fail in the real world.
The other day I was working on an embedded system and running out of space for the program (512KB). It took me two days to rewrite a few sections to use compressed data instead of the "easier to read but bulkier" original way. The end result to the user was the same, but used 10KB less space.
I worked in lots of projects like this. The most amazing part is most devs didn´t care. Never understood why.
In one of those projects the code got so big and bloated that it took around 1 minute (sometimes more) to move from one line to the next while debugging. For me it was a torture, it was ok for most.
Makes me even sadder that something like GOAL didn't take off. In GOAL, you could make changes at the REPL, compile instantly, push the compiled changes out to the console and see them immediately in the running engine.
There's a reason why old-school Naughty Dog games were so refined.
Haha, so true. People are used to old ways. I know its not c++, but If you run docker on mac or Windows for web dev or any other project that has thousanda of deps you know how painful docker sync is and how slow/crash prone. But hey "docker is easy" :)
IME reducing iteration time is one of the best ways to increase my productivity.
I like Android Studio's approach to running unit tests: It will run them on the host PC instead of deploying them to the target, so you can avoid the often long deployment process.
While it sounds painful because of the inefficient development & testing process, I think that if that day is spent on research instead (looking for a better solution or reading existing code) it is OK to spend 1 day on 3 lines of code.
That really depends on how critical the code is. The amount of attention that has gone into optimizing ObjC message send is astounding with many man hours of top programmer time in each and every single line.
Aside from looking at changes in your dev methods, this is a good way to demonstrate to your management the value of having fast, up to date hardware on your desk.
This guy is young and probably not educated in modern dev. I refuse C++ jobs when I ask how they do unit test and they reply it's too hard in C++, and take Java jobs because there the first question THEY ask is HOW I do unit test, what I think of coverage metrics (hints: they don't matter much) and how I propose to enforce ALWAYS unit testing important code.
In this blog post the guy took 3 job change to DISCOVER unit tests, it's insane :( It's not like they know but don't have time, it's that he's a "champion" just for proposing limited scope testing.
I think the main issue about automatted unit/integration testing is not the language, but the area you are writing code in. Yes, C++ not having good unit test support hurts. But it hugely depends on the problem domain and whether the outputs are machine verifiable, and the inputs are easy to simulate.
It's hard to do unit tests in a large part of the area of gamedev. How do you ensure that a picture is rendered correctly? You can take a screenshot and compare it to a stored one. But then what happens if you do artistic changes that you want to do? You have to update the stored screenshots of the testsuite. Who will review that the changes all made sense? And more importantly, there might be slight differences in the output of GPUs, depending on driver versions, model, etc.
So let's say you have a bug in your game where if you walk through a level in a specific direction, the game crashes. The error is easy to check for: just make sure that there is no crash. But how do you create a reproduction of the bug? You could record controller inputs and play them back. Then a different department changes something how quickly players move and increase their walking speed by 10%. Suddenly your player walks into a wall and the test is basically broken.
Compare this to a CRUD app where you have well defined operations and their impact is well described.
That being said, even in gamedev there are areas that are well testable. You can do a unit test of the low level networking layer by trying to make a server and a client, dropping some packets, then looking if the packets still arrived because the networking layer sent them again.
I once spent the better part of a day figuring out a off by one error, when I pushed the fix it changed two characters. Almost the whole day, imagine that.
From a systems perspective, there is a balance to be achieved.
You want faster iteration times in order to provide feedback, but if they're too fast then you neglect other things or end up swinging wildly. Consider a thermostat that turned on the heat or cooling at just .1 degree below or above its target, or turned it off as soon as it hit the reverse. Or if because the feedback comes in as a torrent of data you attend to that feedback instead of other elements of the system or your capabilities.
As a system gets more complicated you have more factors to consider. In the case of programming, if you rely too much on that near-instant feedback, how much are you internalizing about your system, language, and environment? How often are you making the same errors but recovering quickly (so it's not slowing you down too much, or doesn't appear to be slowing you down too much) because of the feedback? How much faster could you ultimately be if your work were smoother and not so rough and jagged?
Introducing a delay, here, gives you time to contemplate. Even if it's just taking an hour or two a day to sit back from the keyboard and ponder what you've done that day and what you will do the next. Create a plan instead of jump into action, even if the plan doesn't get executed perfectly or ends up being the wrong plan that bit of contemplation is when you learn.
But too long a delay (especially a forced delay) causes other problems. The actually needed feedback (not just compiler errors and such, but your V&V issues discovered from testing and evaluation) getting delayed by a day or more can be too much (especially when working with a team, where other parts are potentially changing around your own changes). Too long a delay also promotes batching many unrelated changes together because you don't want to sit through the whole process again. Consider a system that takes a week or a month to get feedback from an external test team, you'd be tempted to throw many changes at them because of that week or month long delay (or more!) and not just one change.
So strike a balance, find a point where your iteration cycle permits you time to think and not just act so you can really learn (both programming and the particular system you're trying to develop). Smooth out your development so that you're slowed down not by having to take corrective steps but having to ponder logical steps. "Is this the right data structure? Well, if I isolate it in my domain model then I can swap it out later and no one will be impacted." or "I'll take a walk around the building and think about what I actually need here."
In other words, reducing the iteration time reduces the motivation to get it right the first time, and the associated deep reasoning/"mental execution" skills which are required to do so. In order to develop those skills, one should strive to write as much code as possible before running it, and a high iteration time assists with that.
Also, I'm tempted to continue the title with "...unless it's APL".