i've been following kyle's jepsen project for a few months (check out his other posts) - and it seems (thankfully) to be nudging the focus of discussion away from only "performance metrics" (transactions/sec, etc.) of distributed systems and toward data consistency and behavior in the face of partitions and other faults. ...what's the use of being able to make 20k updates per second if half your data is lost during a common failure?
this is perhaps the most honest and accurate description of what you will likely find in raising money - couple with fantastic advice. in fact, it accurately reflects my first experience and rookie mistakes raising money in a secondary market: wasting time by being led on by investors who don't lead; eventually closing a first investor, which started a rush; etc. if you're raising money for the first time, please read this multiple times...for your sanity's sake.
one other thing to keep in mind is that the underlying algorithms (zab and raft) provide different guarantees.
for example, zookeeper/zab allows reading directly from followers with a guarantee to get at least a past value that won't be rolled back. this was one reason zookeer didn't use paxos:
in my understanding, raft doesn't allow reading directly from followers, because the follower logs may get repaired/rolled-back when a new leader is elected. (though i'm sure an implementation can tweak the protocol to provide this support.)
that said, raft has a lot of interesting applications, and, in my opinion, is definitely more understandable than the many versions of paxos. (implementing zab yourself, at this point, would be a futile exercise.)
i found the videos from the raft user study to be very well done (and easier to understand than even their paper):
...however, i think they did paxos a disadvantage by not just focusing on multi-paxos (which is probably the most common implementation). but, it's certainly fair to say that info about paxos is spread out far and wide...with perhaps too many knobs to turn and implementation-related details to fill in yourself.
as a side note: i've just started implementing raft in a set of libraries (multiple languages) that will be open source - along with other protocols.
I'm the original author of go-raft (the implementation used in etcd) so I'll try to address some of the points in this thread.
Raft only updates the state of the system once log entries are committed to a quorum the local state will never be rolled back. Log entries can be thrown out but they haven't been committed to the local state so it doesn't matter.
You can read from the leader if you need to ensure linearizability but that will kill your read scaling. Another approach is to read locally and check that the local raft node isn't in a "candidate" state (which would mean that it hasn't received a heartbeat from the master within the last 150ms). That approach works for a lot of cases.
As far as implementing multi-paxos, the authors behind Google Chubby have talked about how there is a large divide between theoretical multi-paxos and actually implementing multi-paxos. Also, there aren't any standalone multi-paxos Go libraries available. I wrote go-raft at the time because there wasn't an alternative distributed consensus library in Go at the time.
Let me know if you need an extra pair of eyes on your Raft implementation or if you have any questions (ben@skylandlabs.com).
Having followers redirect to the leader is how the paper describes the algorithm.
But I don't think there is anything stopping you from having followers service committed log entry reads, provided you're willing to live with being out-of-date.
one of the most important things - something that is difficult to suss out in an interview - is how well you (or your team) would work with a person. (i think this is perhaps the most important quality - definitely more important than being able to solve brain teasers.)
i've wondered lately if it'd be better, during an interview, to choose a problem that both you and the candidate don't know...and try to solve it together...
interesting FAQ...i like the idea of bringing this info together.
i've found there are lots of more-common things that cause partitions in practice than equipment-in-the-middle failures. human errors are probably the biggest: network configuration changes, fresh bugs in your own software - or in your dependencies, etc.
also, while a network might be asynchronous, there's usually a limit to how long a message can be delayed in practice. ...the limit might be how much memory you have to queue up messages...or perhaps how long your client-side software (or your end-user) is willing to wait for a message when a dialog is more complex than request/response.
when designing distributed software, i've found that it's helpful to ask: when (not if) X process/server/cluster/data-center fails or becomes unreachable - temporarily or forever - how should the rest of my system respond?
so, perhaps the most important take-away from the FAQ for designers is #13: that C and A are "spectrums" that you tune to meet your own requirements when the various failure scenarios happen.
> The theory in taxing capital gains at a lower rate than ordinary income is that the wealth that was invested that produced the capital gains has already been taxed once when it was earned.
Whenever someone argues this point (not fw, in this case) and alludes to being double-taxed, I like to remind them that only the gain (minus any loss) is taxed...so there is no double-tax. ...and why someone would invest in stocks or startups vs bonds is that the rate of return is potentially much greater. I don't think there needs to be an extra incentive (in the form of lower tax rates) for investors. ... That argument seems to be a smokescreen.
On another note: I'd love to see a tax rate (or fair analysis) based on discretionary income.
A progressive tax system addresses this a bit...and, in my opinion, this is why moving to a flat tax would benefit the rich and hurt the poor.
look at all those happy/helpful people, ready to volunteer their time and emotional shares to help you get your project off the ground. (and this is only one great hacker spot in the city.)
in my experience (~10 years in manhattan+brooklyn)...these are the kinds of people/groups that are here...looking to build stuff and have fun.
...but, of course, in a place as large as this, you'll find all types. (as you do in the valley too, really.)
kudos!