Were you using Device Mapper purposefully, or just because you couldn't get a more typical storage driver to work? The general setup instructions recommend either 1. installing linux-image-extra and using AUFS (for kernel versions < 4) or 2. modifying the dockerd invocation to specify the OverlayFS storage driver with `-s overlay` (for kernel versions >= 4).
Admittedly, the latter strategy is a bit frustrating and counterintuitive on 16.04. When you install Docker it will start automatically and hang because it can't find the AUFS driver and Device Mapper isn't set up. The solution is to either 1. modify policy-rc.d to prevent services from automatically starting or 2. set up Device Mapper (install dmsetup and run `dmsetup mknodes`) before installing Docker and changing the storage driver. Unfortunately, these workarounds are not particularly well-documented.
I would be interested to see how Docker 1.12 or 1.13 with a modern kernel and OverlayFS would be able to handle your described workload.
So you are saying that all default settings are unusable in production, to the point docker might not start at all, and the only cure is series of 5 obscure advanced system setup/configuration that are almost impossible to figure out by oneself and not documented, yet they should be totally obvious to anyone using Docker, right?
FYI: It's because of this sort of bullshit that there are articles called "Docker in Production: An history of failure".
No, that's not what I'm saying. I'm saying that Ubuntu Server 16.04 LTS comes out of the box without AUFS and Device Mapper, and that you either need to enable one or the other before installing Docker or prevent Docker from starting automatically at install time. It sucks, yes, but your description of the problem and its solution is beyond hyperbolic. Googling "docker ubuntu install hang" takes you right to the GitHub issue with the solution at the bottom. And I think it's fixed in Docker 1.13 (it will prefer OverlayFS if AUFS and Device Mapper are not available), although I have not tested it.
I mean to say, device mapper was the only solution which worked in a production setting for me.
I no longer work with that company and so I can't tell how those improvements would change the reliability. Presently I am using docker with btrfs on a low-throughput workload and it seems to work just fine.
Tangentially related - I have been using OverlayFS without Docker for about a year now. It's pretty great. My read-layers can be on an NFS drive and the penalty for writes is relatively small. It basically gives me 90% of what I wanted from Docker in the first place.
Admittedly, the latter strategy is a bit frustrating and counterintuitive on 16.04. When you install Docker it will start automatically and hang because it can't find the AUFS driver and Device Mapper isn't set up. The solution is to either 1. modify policy-rc.d to prevent services from automatically starting or 2. set up Device Mapper (install dmsetup and run `dmsetup mknodes`) before installing Docker and changing the storage driver. Unfortunately, these workarounds are not particularly well-documented.
I would be interested to see how Docker 1.12 or 1.13 with a modern kernel and OverlayFS would be able to handle your described workload.