After reading the article and all the comments here, and from my own experience, I just don't think it's possible to not have a DB. At best, you write your own basic DB, because you don't need anything fancy.
For example, you write S-expressions to files like Hacker News does. This is clever, because the file system has some of the features of a database system, and files and S-expressions are abstractions that already exist. You do have to manage what data is in memory and what data is on disk at any given time, but the complexity and amount of code are low.
The idea that "event sourcing" somehow keeps you from needing a DB is ridiculous. By the time you've defined the event format, and written the software to replay the logs, etc., which if you're smart will be fairly general and modular, congrats, you've just written a database. At best, you keep complexity low, and it's another example of a small custom DB for a case where you don't need a fancy off-the-shelf DB. Maybe it's the perfect solution for your app, but it's still a database.
"Memory images," as a completely separate matter, are an abstraction that saves you some of the work of making a DB. Just as S-expressions can save you from defining a data model format, and files can save you from a custom key-value store, memory images as in Smalltalk could save you from having to deal with persistence. And if your language has transactions built in, maybe that saves you from writing your own transaction system. In general, though, it's very hard to get the DB to disappear, as there is a constellation of features important to data integrity that you need one way or another. It's usually pretty clear that you're using a DB, writing a DB, or using a system that already has a DB built in. If you think there's no DB, there's a high chance you're writing one. Again, that could be fine if you don't need all the features and properties of a robust general-purpose DB.
Funnily enough, in EtherPad's case, we had full versioned history of documents, and did pretty much everything in RAM and application logic -- a pretty good example of what the article is talking about -- and yet we used MySQL as a "dumb" back-end datastore for persistence. Believe me, we tried not to; we spent weeks trying alternatives, and trying to write alternatives. Perhaps if every last aspect of the data model had been event-based, we could have just logged the events to a text file and avoided SQL. More likely, I think, we would use something like Redis now.
Yes, agreed, which is why I find Fowler's (and others) documents a little hard to take seriously. They ARE writing to disk, and doing a lot of other work to try and keep the data intact in case of failure...which is what a DB does. I'm all for the idea of keeping some data in memory for speed, but moving it all to resident memory is just moving the same components around.
Was there something specific it the data model that made the versioning hard to write? Or was it that, for this to work across the board, the entire model had to be versioned?
It sounds that SQL itself wasn't the problem. Were you looking for versioning alternatives in SQL that weren't up to par?
SQL was a solution more than a problem; we were just hoping for a simpler or more elegant solution.
For example, it's possible that logging all data model changes to a text file would have given us the persistence we were looking for without bridging all our data to SQL. Cutting out SQL from our production set-up would have been an inherent win -- one less process to manage, one less black-box source of complexity, etc.
For example, you write S-expressions to files like Hacker News does. This is clever, because the file system has some of the features of a database system, and files and S-expressions are abstractions that already exist. You do have to manage what data is in memory and what data is on disk at any given time, but the complexity and amount of code are low.
The idea that "event sourcing" somehow keeps you from needing a DB is ridiculous. By the time you've defined the event format, and written the software to replay the logs, etc., which if you're smart will be fairly general and modular, congrats, you've just written a database. At best, you keep complexity low, and it's another example of a small custom DB for a case where you don't need a fancy off-the-shelf DB. Maybe it's the perfect solution for your app, but it's still a database.
"Memory images," as a completely separate matter, are an abstraction that saves you some of the work of making a DB. Just as S-expressions can save you from defining a data model format, and files can save you from a custom key-value store, memory images as in Smalltalk could save you from having to deal with persistence. And if your language has transactions built in, maybe that saves you from writing your own transaction system. In general, though, it's very hard to get the DB to disappear, as there is a constellation of features important to data integrity that you need one way or another. It's usually pretty clear that you're using a DB, writing a DB, or using a system that already has a DB built in. If you think there's no DB, there's a high chance you're writing one. Again, that could be fine if you don't need all the features and properties of a robust general-purpose DB.
Funnily enough, in EtherPad's case, we had full versioned history of documents, and did pretty much everything in RAM and application logic -- a pretty good example of what the article is talking about -- and yet we used MySQL as a "dumb" back-end datastore for persistence. Believe me, we tried not to; we spent weeks trying alternatives, and trying to write alternatives. Perhaps if every last aspect of the data model had been event-based, we could have just logged the events to a text file and avoided SQL. More likely, I think, we would use something like Redis now.