Tandem Computers actually did implement files on top of a database.
Their OS didn't have files. Their system was a replicated database running directly on the hard disk, with no file system.
There's something to be said for this. Your files get ACID properties. If we were serious about file integrity, we'd have file systems that worked like this:
- Unit files. The unit of data is the entire file. Files are written once, and when closed successfully, the file transaction commits and others can read the file. Any update replaces the entire file as an atomic operation. (Many applications need this, and try to do it with various move and rename operations, usually leaving files behind if things fail at the wrong moment.)
- Log files. You can only add at the end. Writes are atomic. In the event of a crash, the file is valid up to some recently completed write. (On many systems, log files can tail off into junk or contain truncated records.)
- Temporary files. When the process or process group exits, they're gone. Random access is OK. (You shouldn't have to clean up junk temporary files.)
- Managed files. These support a database or something with complex structure. There are extra I/O functions for locking, flushing and being sure a write has been committed to disk.
That covers most of the use cases for files. There have been file systems which did some of this, but not in recent years.
OS/400 systems (by IBM, also called AS/400, System i, iSeries, iOS, etc)are a database operating system, there are no files, only "libraries". The entire thing sits on top of a customized DB2 database.
From the README: Copyright 2006, Palmsource, Inc., an ACCESS company.
I wonder if this has anything to do with when PalmOS got a 'filesystem' even though the OS originally only gave programs a database to interface with, way back when...
Good catch. It might have something to do with the Palm Lifedrive[1] and its 4gb limit or the NVFS[2] of the Palm 650 and other models. Who knows? It seems like it could be used by a software services company to make a thin client with an encrypted operating system that could detect tampering or cloning. I'm guessing that what the Guardian Project[3] does and why it gets mentioned along with SQLCipher.
Presentation by Hans Reiser on Reiser4. Reiser just explained the new Linux VFS layer introduced by Reiser4 and the ability to see each file a small directory or a set of records.
Audience: Then what is the difference between a DB and a filesystem?
I too would like to remove unnecessary tersm and concepts. But it seems that splitting into FS / DB / Apps gives people a way to optimize their own way. At the cost of communication/reintegration.
ps: I always found the IBM/COBOL record oriented FS a good idea. You remove a lot of ad-hoc parsing code from loading and writing data.
Storage overhead. The loopback device is still a block device, if you skip this step, you don't have to waste the block overhead when storing a file that's smaller than the block size. If you're going to store a large number of small-sized files, this fuse <-> sql bridge could be a win.
I do not know if the library takes advantage of it, but using an ACID-compliant database as the backend, one could make use of transactions at the filesystem level.
Something that's lacking in the Linux world is an efficient way to determine "what file has changed since that time". Using SQLite db could be really helpful in this situation.
A good use case for this might be a filesystem with a large number of files where one could quickly find recently changed files without having to sequential scan the whole system. Presumably the access and modified time would be stored as separate indexed attributes.
I was going to reply with something along the lines of "modern filesystems should already have meta-data indexes".... because I thought I'd read about that in connection to ext4, btrfs and/or zfs. Now I'm no longer certain ... all I came up with was the following, which might be of interest:
Metadata indexing isn't really a filesystem-level problem though, depending what you're trying to do.
An index can be anywhere, and the data you want to keep is somewhat arbitrary - so why not just use a file on the filesystem anyway?
There's also an important difference: metadata indices can be considered disposable in a lot of cases. File data isn't which means the constraints are different: with metadata you want to pack it all into the tiniest, most local part of the disk you can.
With file-data (and the actual filesystem) you want to distribute and replicate that data as widely as possible to minimize the chances that a cluster of failures wipes out important structures.
The typical example (lifted from one of the links, I forget which) -- is looking at a huge filesystem, quickly seeing the files changed since a given date, which have the "archive"-bit set, query after files by name (unknown path -- ie, run "find") etc.
Indexing on other meta-data, like tags for images and music files -- can be considered a filesystem level problem -- if one considers approaches like Mac (or Amiga) resource forks/info-files.
Perhaps I should have stated "file metadata" as opposed to "just" metadata... One could of course claim that the only thing the filesystem does is take an exact path, and return the data at that path. In such a case, you could replace the path+name with a guiid, and store the filename and path-name info in a file... and update file whenever you accessed a file... and then you end up building some of that into the filesystem interface. So the question really is where the filesystem ends and the "system" starts...
Though not exactly the same thing, Tcl can use its VFS extension to provide a "file system" for an application backed by a database or ZIP archive.
The VFS allows bundling up an app as a self-contained executable (a "starpak") or runnable with the Tcl/Tk interpreter (aka "tclkit"). FS access is transparent to the app--read/write operations are the same for all FS types.
Some work has been done to use sqlite as a VFS data storage medium. I haven't yet tried it myself, but in principle, it's not too hard to accomplish. I'm putting that project on my list...
Out of curiosity, when/why would someone want to use something like this?
SQLite is a file database, in that the database is literally a file, which means it will reside on another already existing filesystem - so you would have:
DMG (a disk image that can be mounted) is the main way to distribute OS X applications, and something this could have fulfilled the same use case if Linux wasn't as fond of centrally distributed packages.
It's not like Linux lacks disk image formats, though, like cloop (used by Knoppix). It's just that when you don't have to distribute weird metadata like resource forks, what's the point of using a whole filesystem versus a tar file?
Even when you do have weird metadata, it's easy enough to retrofit archive formats to handle it, which Mac OS X did years ago. Disk image distribution is a complete anachronism now, and it's odd how persistent it is.
DMGs still have the advantage that you can customise the Finder windows that opens when you double click one [1].
So, from an UX standpoint, they still make a lot of sense to distribute application bundles since it compels users to "install" the application by dragging the bundle icon to the appropriate directory, which is conveniently symlinked in that Finder window. It's also easier to produce than a self-installing package.
With a .zip or .tar.gz users would be left with an app bundle and no idea of where to put it. Generally they would just throw it somewhere random across the filesystem. At least, that's what non-technical Mac users I know happen to do.
I think the proper solution to that, if there's actually a problem with "somewhere random", is to have apps that move themselves on first launch. It's pretty easy to do, and IMO easier than creating a consistent build process for a good-looking dmg.
The solution is to use a .tar file, but give it a custom extension (like, say, .deb) and so when the user double clicks on it, the system knows it's a software package and installs it in the appropriate directory.
One can still install a program to "somewhere random across the filesystem" on most OS's, even Windows. It just so happens that by convention, most programs get installed to the "Program Files" directory, but that is not a requirement.
The only reason I can think of is DRM. Sometimes you are allowed to distribute physcial media, where keys are scrambled outside the regular filesystem, and perhaps there that allows for image distribution as well. But I'm guessing here.
Many people say that DBs are not good for large blobs, hence the advice to put large files such as images in the files system, not in the database. What's different here that makes it acceptable to put files in the database?
There's a universe of archive formats that do this without the need for a structured query language or FUSE. But i'm sure somebody has a need for this...
Funny, I have just spent most of the morning trying to find a user mode file system for windows. The best I can come up with is mounting WebDav or using a .net SMB implementation. I wonder if it would be better to just build a software iSCSI target?
In the "why use this?" department, I think the answer to the question is that you want to distribute massive amounts of content in a single-file (making content updates easy) while also maintaining a POSIX-level control over the content in the container.. so if you've got a massive SQL database full of content - say a dictionary or wikipedia, or whatever - doing a content transform that spits out a single .db to your paid/registered/updating customers is quite useful..
There's something to be said for this. Your files get ACID properties. If we were serious about file integrity, we'd have file systems that worked like this:
- Unit files. The unit of data is the entire file. Files are written once, and when closed successfully, the file transaction commits and others can read the file. Any update replaces the entire file as an atomic operation. (Many applications need this, and try to do it with various move and rename operations, usually leaving files behind if things fail at the wrong moment.)
- Log files. You can only add at the end. Writes are atomic. In the event of a crash, the file is valid up to some recently completed write. (On many systems, log files can tail off into junk or contain truncated records.)
- Temporary files. When the process or process group exits, they're gone. Random access is OK. (You shouldn't have to clean up junk temporary files.)
- Managed files. These support a database or something with complex structure. There are extra I/O functions for locking, flushing and being sure a write has been committed to disk.
That covers most of the use cases for files. There have been file systems which did some of this, but not in recent years.