My understanding is that it has been manually tested. I.e. it has produced correct results to previously intractable problems. I'm not sure how much automated testing would add at that point.
Unit testing usually isn't easily replaced by manual testing. If you have, for example, 3 units that can be in 2 different modes each, that's 2^3 different combinations, but only 2*3 unit modes. Testing the end result is more work than testing the units.
Discovery science is different from web software engineering.
Most discovery scientists use manual testing, not unit testing. Very few actually do integration tests or system tests (this is something I'm trying to change).
And, given the external results of the application, it's unclear to me how much additional value would come from a rigorous testing system.
Sure, I'm trying to take the idea of merging continuous integration with workflow/pipelines. It's all stuff that I learned at Google and is non-proprietary. The idea is have presubmit checks that invoke a full instance of a complex pipeline, but on canned (synthetic or pseudoanonymized or somehow not directly connected to the prod system) data, as an integration test. This catches many errors that would be hard to debug later in a prod workfflow.
In a sense, I see software testing/big web data and modern large scale data processing in science as a continuum and I want to bring the practices from the big web data and testing fields to bear on science pipelines.
> In spite of this, I am in the top 7% of accounts on the site, which is a nice reminder on the power of compound interest!
I'm also in the top 7%. That's down from 12% or so from five years ago. However, I haven't gained many points during the same interval. That leads me to believe it's not about compound interest but new users joining.
I'm not convinced this is a meaningful distinction in this context. Assuming a developer consciously creates a race which turns out to be a bug, how does it matter whether it was a data race or some other kind of race condition?
In general data races can break program invariants, violate memory safety and do other undefined behavior.
In C++, data races can cause jumps to invalid addresses and the compiler will assume data races away and delete checks that are redundant in the single-thread case. The equivalent problem happens in most languages where data races are possible (two Python threads writing at once can put a value into an impossible/illegal state)
An ordinary race condition is just an observable effect of the nondeterministic progress of threads, so it's possible to have a truly benign race condition, so long as the correctness of the whole program doesn't depend on the outcome of the race. I/O causes nondeterminism anyway, so it's not really practical to avoid all ordinary races.
It's possible to have truly benign data races too, e.g. lazily computing a hash of immutable data.
One of the reasons that Rust's ownership model is necessary is most data races cannot otherwise be statically detected. The compiler will be unable to apply any of its theoretical UB optimizations. It will emit its native code and you get the defined behavior of the underlying hardware. There will be no undefined behavior. If that ever changes, the bigger news, I imagine, will be that we solved the halting problem.
Data races are undefined behavior, and race conditions are a logic error. Yes, both are bugs, and both are important to fix. But they have (at least to me) different severities.
(Worth noting that data races have a clear, unambiguous definition, and therefore are possible to eliminate through tooling. Race conditions rely on program intent or logic, and are therefore... not really.)
Also, I think it matters, generally, that we are truthful about what Rust prevents and does not prevent.
> Worth noting that data races have a clear, unambiguous definition, and therefore are possible to eliminate through tooling. Race conditions rely on program intent or logic, and are therefore... not really.
This is a great point. Usually people only talk about the two categories in terms of vague severity, but this is a hard distinction which has lots of implications for how each can be dealt with.
Any kind of race can lead to data corruption, or being very drastic, possible death if the outcome is related to any computer system with direct influence over human lives like factory automation systems.
This is why I think Rust discussions about this subject don't focus enough that the language only validates a very tiny subset of races.
> Data races are undefined behavior, and race conditions are a logic error. Yes, both are bugs, and both are important to fix. But they have (at least to me) different severities.
Some data races really are benign while some other race conditions can lead to data corruption or security vulnerabilities. Why then should data races be strictly more severe than data races? What am I missing?
Actually "this context" was my previous reply. Much like I can nest a lexical scope inside a program and have it reach up and past the containing scope, I can create a new context in a discussion. Please at least try to understand my position before you reply.
You can also create races in safe Rust by using the file system, no concurrency primitives necessary (just create a new file, close it, and re-open it assuming it's still there..).
I'm not sure I buy the premise. The point of the original article was that you shouldn't dig a hole for yourself to get out of later. Compiler writers usually go out of their way, digging extra deep holes, to accept a broad range of invalid input. Not because they like complexity---though, as a group they.. I digress---but because it lets them give more valuable user feedback.
Yeah I am currently writing a compiler, turns out for useful error messages my parser should accept a superset of my language, then validate it afterwards.
Yeah, the main coherent point I take out of this is that it makes sense to provide capabilities rather than to check them, i.e. if you have access to a certain API then you're by construction allowed to use it. It seems this is something the author has been working on.
This is then somehow shoehorned into "Parse / Don't typecheck" but it doesn't make much sense honestly.
No[1], TypeScript is unsound in many ways. Most of them are intentional trade-offs to catch as many bugs as possible while not being too restrictive/allowing most JavaScript code to be annotated.
It's unbelievable how hard to grasp distributed systems are. I recently implemented Paxos in Rust and at certain points I literally thought I was losing my mind.
When you read Paxos Made Simple it really all seems so, well, simple. But then you get inconsistent commits and look at the traces of what happened and just go "How?!"
One of the things that surprised me about this analysis was just how many bugs we found that had to do with the actual Raft implementation. Usually when I test Raft-based systems the bugs are at the edges--like the coupling of the system to the Raft library, treating it like an externally-queryable log rather than the driver of a state machine, and so on. We found integration bugs here too, but also a fair number of issues in the Raft library itself--and this is despite Redis-Raft having existing integration tests!
Has anyone approached Jepsen about running an analysis on the Erlang Ra implementation? I believe they've been running Jepsen tests internally, just curious if they're thinking about getting an official analysis at some point. Thanks for all that you folks do!!
* https://github.com/rabbitmq/ra
Pre-existing! It's a fork of willamt's https://github.com/willemt/raft/, which has been around since 2013, and has property-based fuzz testing! It really does look like it's got its own extensive tests; I'm surprised we found issues.
I'm curious if there's research into "better" primitives in Programming Languages in order to simplify writing distributed systems, analogous to how concurrency primitives beyond Mutexes, Semaphores, and Condition Variables (like Futures, Monitors, etc. or approaches such as Actors) can greatly simplify logic and enhance one's ability to reason about code. Or things like the Rust borrow checker.
There's a lot of research into this, actually! Folks have been working on ways to extract executable code from Alloy, TLA+, Isabelle/HoL, and Coq specifications. That doesn't help with implementations which don't use codegen though--and it doesn't help you with the parts of the program that aren't formalized.
There are two general issues here: there's the nuts and bolts, and then there's the emergent properties of the protocol.
In my experience, async (which includes futures/promises, and actor-like mechanisms) makes the nut-and-bolts problems of avoiding variable race conditions, avoiding deadlock, managing multiple things going on, way easier.
You still need fuzzing and model checking to make sure you got the strategic stuff right.
That said, the team I work on is about to release our first Raft-based product, so I might have a different opinion in a few months.
That's because the core Paxos protocol is relatively simple. However, no production service could ever survive with just that. Once you start considering all the features and optimizations you will really need, things get very hairy.
Well paxos way more complex than raft. I'm not saying building on top of raft is easy, I'm saying making a MVP raft implementation is easier than paxos.
One thing I wish raft had - a learner role which act like a follower that can't start an election until it has catch up with the rest of cluster. etcd has it, but I wish it was part of the raft instead, as well as bulk log transfer.
Article pointing out a very common issue that anyone who tried implementing raft runs into:
Letting a follower forward request to leader on client's behalf is not easy to implement correctly, that's why most popular raft based software (hashicorp stack) doesn't do that. Not worth it.
> Letting a follower forward request to leader on client's behalf is not easy to implement correctly, that's why most popular raft based software (hashicorp stack) doesn't do that. Not worth it.
I'm honestly surprised by this comment! I've written multiple Raft implementations, and request proxying was one of the easiest things to get right--it doesn't have to touch the Raft subsystem at all. Could you talk a little more about this?
Proxying isn't hard to implement but in the erlang Raft implementation, Ra we decided only to do it when the client explicitly declares that it does not care about ordering. When proxying it is always possible that the client will find out the current leader whilst the proxied request is in flight. This may be more of a problem inside erlang where it is easy to address any processes within the erlang cluster.
I don't see anything in this blog that even touches on "distributed systems are hard". Every issue in here should be filed under "Redis has no tests". If you follow basic software engineering principles, you'll find distributed systems easier to approach.
My reading of the article's introduction is that Redis is adding this feature and are (among other things I'm sure) paying jepsen to test it. So this is them having tests.
> If you follow basic software engineering principles, you'll find distributed systems easier to approach.
When I implemented Paxos I had tests and when they failed they spit out an exact trace of what happened in what order and on what node. Sometimes it was still excruciating to figure out what happened. Here's[1] a comment which you can think of as a bug tombstone. It took me half a day to figure out after I had a trace to analyze the issue.
Sure, but now imagine you have no confidence that any part of your paxos implementation works at all, nevermind the paxos part. That's my impression of issue #13 from the article: not only did the software not pass the test, it's clear that nobody ever even tried to use it, at all!
Full-scale blackbox testing of a database system is similar to dogfooding. You only use it when you have high confidence that you have exhausted the possibilities of unit and integration tests. It's clear this project did not start with exhaustive unit tests.
It reminds me a bit of FoundationDB, which is also a terrible program nobody should entrust with data they ever want to see again. The first time I tried to use it it ran out of memory and crashed in about ten seconds. I found the problem, which was that their huge-page-aware allocator, which has no tests, had never actually been used by anybody on a machine with huge pages. It was a core library of a released database which had never been executed by anyone. This Redis thing is the same: nobody had ever said "RAFT SET foo bar", if they had done they would have seen the problem right away.
I'm hesitant to draw too strong a conclusion here, and I can't speak for the Redis Labs team, but I do suspect that this is somewhere where... having an outside tester, like Jepsen (or a suitably adversarial QA team) can help detect missing-stairs sorts of problems. Coming from the perspective of a prospective operator (and having some experience with testing distributed systems), I immediately said "of course I want proxy mode by default", when this wasn't how the Redis-Raft designers necessarily intended things to be used--they intended smart clients to make it so that users wouldn't actually need proxy mode, so they hadn't focused on testing it that way.
Fair enough. I think I misinterpreted the "easier to approach" part of your original answer. Sorry if my answer came across as defensive. My wounds are still fresh. ;)
Not really, given the left bias of technology companies there is zero chance that his act will be ignored by companies. If anything, standing behind free speech is brave nowadays
If nothing works he can probably open a gofundme campaign.
You do realize that Anders Hejlsberg is on the TypeScript team? I don't want to imply that he's superhuman or anything, but.. just what is your idea of "serious compiler engineers"?
Well maybe he doesn't get enough leeway. Maybe he knows that he should be building an engine from scratch but management won't let him. This is very common in companies of that size.
