Does anyone have experience making tests against real databases fast?
I resonate with the sentiment of this article, but have struggled to find an alternative that’s fast enough as the test suite grows, isn’t flakey in CI, and is able to share the production schema definition for relevant relations.
I’d love to hear more from anyone that’s solved for some of these constraints!
I think one helpful thing is what rails calls transactional tests (https://guides.rubyonrails.org/testing.html#transactions). It basically does the database setup (migrations and seeds) and then executes the tests in a transaction that is rolled back at the end (and thus never committed). This helps with speed and also ensuring that tests don't have as much accidental codependence as they might otherwise.
If you use read replicas in production code this can become tricky though since the transactions don't commit they never become visible to the reader or even different connections to the same database
A real database should not be slow. Even with our tests running against a hosted SQL Server on a separate server, the database is never the slow part. For other tests, we run with the same database in a local Docker container with Docker Compose and it is fast and isolated/resettable.
Most tests should be unit tests, which are super fast. Integration and UI tests that might use the database should be fewer and if the database is slow, it might be related to your specific application or unoptimized database queries, our database calls are usually < 10ms
What I've done is make a clone of the real database, with a sample of data that has enough variety/size to test whatever it is you need to test, but no bigger. It definitely takes some thinking, planning, and writing of code, but it's worth doing.
Unfortunately I maintain an app where the database (read-only) is Snowflake, and being more of a "warehouse" than "database" there's always a lot of overhead in running any query at all. Even just `select 1` can take a few seconds. So there's only so much you can do with that, but setting up your data so that tests can be parallelized helps as well.
However your tests against a proper OLTP database should be plenty fast, unless your app itself is slow or your test fixtures require some really complicated setup.
On my machine -- which is quite middle range at this point, not even high end -- I get by just fine up to 3000 tests, with Elixir at least. When I was at that contract the ~3200 tests ran in something like 40 seconds.
What kinds of troubles do you have with using a real DB for testing?
If you happen to use Postgres, the approach is ultimately portable: it uses Pg database templates (also, regarding perf, the author recommends using a ramdisk and turning off fsync on your test DBs; you'll see this in the project readme). But you’ll have to write the code yourself.
This is so great Peter-- first I've heard of pgtestdb and it's immediately useful for me. How can people donate money to the pgtestdb project? Or hire you for consulting for pgtestdb? I'm joel@joelparkerhenderson.com and would love to help fund your work.
I sincerely appreciate the sentiment and the offer — but pgtestdb is MIT license, actual, for real, not kidding, open source. No payment necessary; please enjoy.
(I'm always open to discuss potential contracts or consulting opportunities. If you have one that you think might be a good fit, my email is in my profile here and on github and on my homepage.)
Really glad to hear it's been working for you with zero flakiness! If you ever do run into any trouble, or have any suggestions for improvements, come on over to the github issues page :)
My app: Kotlin, Ktor, Exposed
Databases:
- Production/Dev: Postgresql
- Test suite/CI: SQLite
Performance:
- ~1000 tests
- ~5 seonds
For testing anything below the ktor layer, I create and roll back transactions for everything, including db table creation (though I should probably fix that, just bigger fish to fry in this project)
For the SQLite / PostgreSQL differences that Exposed doesn't naturally handle, namely jsonb, during the CREATE TABLE phase, I have my test harness create a regular json version of the table rather than a jsonb one. Exposed's ORM seemlessly handles that swap out after table creation. There's a bit of manual upkeep in making sure the *Json test version of the tables are kept up-to-date to the production non-Json version; but that's the sort of thing that's caught on the very first test and fixed in minutes, most of the time (sometimes cascading table creation bites me).
I will eventually probably add a flag or something so the test suite can run against a separate testing partition in my PostgreSQL docker container, but I haven't done that yet.
I've been bitten by using an in memory database instead of the one production uses. If you're using GitHub Actions it's as easy as using "hoverkraft-tech/compose-action" and providing a docker compose yaml with a postgres image a step.
Oh certainly! The problem is that it's yet another thing to add to the pile of things I need to configure, etc, at this point and it's of lower priority than many, many, many other things - and will easily take as long as so many other of the steps, which are all taking a long time due to learning curve.
Tests are usually embarrassingly parallel. Instead of creating one test db (app_test) create many (app_test_0, app_test_1, ...). Run tests in many threads/processes, db per thread.
This works in a lot of cases. In some cases this might not address your bottleneck.
Also someone should write a real, performant, in-memory postgres storage driver. Then we can all be happy (with pg at least).
I've taken a stab at making a solution for it via https://github.com/data-catering/data-caterer. It focuses on making integration tests easier by generating data across batch and real-time data sources, whilst maintaining any relationships across the datasets. You can automatically set it to pick up the schema definition from the metadata in your database to generate data for it. Once your app/job/data consumer(s) use the data, you can run data validations to ensure it runs as expected. Then you can clean up the data at the end (including data pushed to downstream data sources) if run in a shared test environment or locally. All of this runs within 60 seconds.
It also gives you the option of running other types of tests such as load/performance/stress testing via generating larger amounts of data.
We use Testcontainers (https://testcontainers.com/) in our node.js / vitest / kysely stack. Really fast to spin up a temporary postgres instance, and we use kysely migrations to init and seed the db.
Not sure if this is still a valid approach, but on a large Django site with a lot of unit tests, the cumulative setup/teardown/reset cycles was killing us. We found that setting each test to be wrapped with a transaction that was aborted on teardown, caused the per test cleanup to drop radically. We also kept a canned database for testing so that running the test suite didn't have a large startup penalty to populate the database with test fixtures. Keeping that ready db between runs also sped things up a lot for devs.
Doesn’t directly answer your question but at least in Postgres I am curious about UNLOGGED mode and see if it results in faster specs. Trade off being, crash recovery doesn’t work but that’s fine in CI.
There is also something to be said about keeping database transactions atomic (no 3rd party network calls, etc) to keep flakey specs to none. I have some ad hoc thoughts on this, will try to frame it proper in a post.
Depends on how fast you need them to be. I've had success using docker postgres , set up the same as production, where each test creates a connection to it's own test specific database, or is written to run inside a transaction that can clean up after itself using a shared database. In my experience, slower tests that use a real db almost always save net positive dev time.
Tests are embarrassingly parallel. If you can split up the load for CI then at the limit the slowest response is your slowest running test. I wish more tooling would expose this out of the box, but with some effort it is possible.
This is better than nothing but if you have to run migrations before your database is usable, you want an approach like the one I use in pgtestdb — use one server for all your tests, but give them each different databases by cloning from one that's already set up with all the schemas correctly migrated.
I did something similar with pg_tmp as well. I also had a lot of tests around migrations, so there was a lot of tests that would 1) ask for a DB of version x 2) insert things into the DB, etc 3) migrate to version y 4) ensure the DB is appropriate (note: this was not a SaaS).
You could “yield” the record instead of constructing the list. This makes “read_portfolio” into an iterator instead of returning a list. Use a list comprehension or list constructor to convert the iterator to a list if needed.
would old be something like: implementation is no longer relevant (the software itself is obsoleted), but the idea is seminal -- a basic conceptual building block upon which current systems are implemented? Example: Lambda Papers (https://en.wikisource.org/wiki/Lambda_Papers)
This is really interesting. What kind of metrics is the culture focused on? How were they established. Can you think of examples when critical/deliberate decisions were made to pursue the integrity of the culture (rather than shifting or broadening focus)? Has cultural orientation or measurements changed over time? How and why?
Are there other examples of cultures focused on particular metrics that have flourished? Or faded away?
So there are a number of examples, but Bjarne used to stress 'compatibility' for example. It was a really important idea for him, that despite the really clean and smaller language that he could have developed, it was very important for him to have tighter compatibility with C.
In many ways, he discussed how he hasn't built a 'perfect language', and that as languages operate as tools, it's very important to be focused and decide what matters over what doesn't.
Sure you could have a prettier language, but as programmers there is a cost to everything and sometimes we are so eager to see the benefits in any new tool, it's easy to forget to measure the costs (he also really, really enjoyed having us benchmark things to see the cost in using x method over y method).
Making it explicitly clear what you intend to focus on, and setting a standard for not just yourself, but your community encourages the language to grow in a way that it remains useful and sustainable. He mentioned that if he built out this tighter/cleaner language, it's also possible that the language would have failed to generate any traction at all.
I'm not entirely sure of cultures that have flourished or faded particular ideas, but if you look at how despite newer languages are constantly taking old ideas from Lisp, how come Lisp didn't take off (and I say this as a true fan of Lisp)? Again, I'm totally hypothesizing, but it may have to do with compatibility.
You can say that Lisp's syntax (or lack thereof) is really important to not just how macros are developed, but for Lisp programmers to have a language that evolves. Yet at the same time, Lisp's greatest strength is often its biggest weakness. You can't really standardize something that evolves all the time, and as such, can't really have compatability. How do I build a library for a language that despite becoming more powerful for the particular purpose I started using it for, looks nothing like anything else out there?
I think his main point tended to be, pick things that matter to you: metrics that matter to you, ideas that matter to you, a mission that matters to you and just really work on that. It's easy to get distracted and just add features "just because", but make sure they conform to the metrics you set forth for yourself.
He was quite right, this is what attracted me to C++ after Turbo Pascal, in spite of disliking C.
I could have the type safety of Turbo Pascal, its scope of language features and spend less time writing my own FFI wrappers to the OS APIs, that were adopting C as their new systems language.
Of course to achieve that safety like level, I still had to write my own wrapper classes, but it was less effort than full blown FFI code in languages that have completely difference concepts.
> You can't really standardize something that evolves all the time
You standardize the meta-level and then use best practices. There are many possibilities to extend a language using macros, but actual macro mechanisms may favor a few (for example because some macros are straight forward, where others are more difficult to implement). Thus one now has to learn: how the macro mechanism works, what is the best way to write macros, how to debug macros, ... Working on and with this meta-level is more difficult.
> How do I build a library for a language that despite becoming more powerful for the particular purpose I started using it for, looks nothing like anything else out there?
Usually one would build macros in similar ways. There are some domain-independent principles.
Absolutely! Oh, don’t get me wrong (again, Common Lisp/Scheme were instrumental in getting me into programming so you’re preaching to the choir), but I guess the point I was trying to make is that there is a real difference between the Lisp approach to standardization and of course C++. Apologies for not being more clear.
There are definite ways to standardize a Lisp, I mean look at all the awesome work Rich Hickey as done on building out Clojure. I just feel that the Lisp (huge generalization incoming) may not care for standardization in the same way Bjarne does when developing C++.
The point I was trying to make (let’s take an example Bjarne cared about that isn’t compatability) is that people in different language communities care about different things. I think it’s fair to say that Ruby developers don’t think about speed as much as people working on C++. It’s not that they don’t care, but ‘programmer happiness’ in the way Matz thinks about Ruby (having an ‘unless’ keyword for example) is more important to Matz than speed. Again, it would be crazy to say Matz doesn’t care about speed, but he, like Bjarne, obviously prioritizes some things over others.
For a college student taking the class, it may seem obvious (“well of course you should build a language focused on certain metrics”), but it’s easier said than done. I remember working on a compiler my junior year where I all I wanted to do was continually add features to my language. Often the problem with the language was with the goals I’d set forth and addressing the problem I was trying to solve, not simply adding more features.
Focus over features is the idea I think Bjarne got across super well.
Lisp has attempted to standardize (in the sense of writing standards) multiple times: Portable Standard Lisp, Common Lisp, ISLisp, Scheme (RXNS, IEEE Scheme).
There are/were different target communities. Common Lisp addressed the needs of (complex) application developers. Scheme had the both industrial and educational users. ISLisp had slightly different goals.
But what the standard limits is not so much the scope it wants to address, but the application areas with their demands and their capability to pay for it. Stroustrop (and many others) got paid over more than a decade to do standards work.
But back to the original point. I don't think there is a problem to 'standardize' macros or extension languages which use a lot of macros. For the end user a macro is not different from other syntactic constructs, it's just that these syntactic constructs are coming with language updates or with new (embedded) languages. A Java developer learns the Java language and then she needs to learn various XML-based configuration language plus some another extension language. In Lisp this tends to be within one language with syntactical extensions, which share a common language base.
Guido van Rossum also has a similar wisdom and attitude, though very different goals and priorities than Bjarne Stroustrup of course, when he judges Python extension proposals by how "Pythonic" they are.
Even if somebody can come up with a clever syntactically valid way of changing python syntax or semantics (like in all heated discussions about lambda), it still might lack the proper "Pythonicity" that makes it fit in with the rest of the language, and would spoil the Zen of Python's [1] "There should be one -- and preferably only one -- obvious way to do it."-ness of the language.
Bjorn's April Fools joke paper "Generalizing Overloading for C++2000" paper [2] was so hilarious and self-aware because it elegantly conformed to and perfectly parodied the C++ way of doing things, yet showed what can happen if taken to extremes: you get 💩++ [3]
"Summary: An incident on python-dev today made me appreciate (again) that there's more to language design than puzzle-solving. A ramble on the nature of Pythonicity, culminating in a comparison of language design to user interface design."
... I'm not saying that one language or design philosophy is superior to the other, but that they're both quite aware of and well focused on their own different goals. (Virtues lacking from dazzled magpie design of PHP, for example.)
So it can be delicious when you combine them together [5] [6] like chocolate and peanut butter [7]:
I work for a global / English-speaking company in Japan. We don't have futsal and basketball instead of vball, but it's pretty easy to get started. PM me if you're interested.
A related market for some of this tech: remote interviews. I've seen lots of solutions for this: collaborative code editors, hangouts, Skype etc. There are frequently problems with bandwidth and friction with coordination (handoff to other interviewers).
Solution to "frictionless" remote work / collaboration among teams (not just engineers) seems similar to interview problem. The target market may also be distinct (teams don't remote work, but remote interviews are common).
Containers are easier for folks with less operational experience to understand. Or maybe easier to get started with is a better way to put it. It's easy to underestimate easy to use tooling when you aren't the target demographic.
When TCO is "lexically scoped" like loop/recur, the compiler can handle it.
When HoF are involved, you may have a case where a procedure calls a procedure-parameter, which calls another, and another... Something about the runtime has to recognize this or else the compiler has to accept more constrains.
See, for example, how Gambit-C implements tail calls by compiling modules to single C functions and how there is a trade off for procedure calls across module boundaries versus Chicken Scheme's approach to essentially garbage-collecting the call stack.
Okay, but at that point you're talking about things that are way beyond the capabilities of an iterative loop. I think my point still stands - that implementing a tail recursion in place of a loop is not something you will have to pay for. Both structures will map to the same instructions.
The difficulty with tail recursion optimization is related to calling conventions. Some calling conventions expect the caller to do some cleanup after the callee returns, which effectively means that no function calls actually occur in tail position. For example, in the C calling convention the caller caller is responsible for allocating and freeing the stack memory dedicated for the callee's function parameters. This system makes vararg functions like printf easy to implement but makes it hard to do TCO. Another example is Rust, where having destructors automatically run when a variable got out of scope prevented them from implementing TCO in a clean manner. I'm not familiar with the JVM internals but I think the limitations are going to be similar to the ones I mentioned here.
It's not just that, it's that last time there was serious talk of it, LLVM's musttail wasn't properly supported across important platforms for us. So it got put by the wayside, and there's always so much to do, nobody has worked through the semantics now that support is better.
We did reserve a "become" keyword for this purpose though. Instead of "return", you say "become", and TCO is guaranteed. That's the basic idea anyway, we'll see what happens.
Yep, you're absolutely right. I guess that maybe if you care about inventing new kinds of loops (e.g. a DSL that wants to loop as a specialized kind of `fold` construct), there's more flexibility with TCO.
But Clojure shows that there's tons of mileage and composability that can happen by using runtime protocols and tasteful language-design decisions even without TCO.
Capturing continuations is another interesting theoretical outcome of TCO, although as various scheme implementations show: the approach to supporting TCO also imposes some performance constraints on call/cc.
An interesting thing about general tail recursion is that it's composable.
Closure's loop/recur is a special case implementation of a common pattern. That's Ok, but it can't extend.
Any higher order function in scheme can be parameterized with methods that should be tail recursive. No special language support needed. That's why for-each is a procedure instead of a special form in scheme.
This allows libraries (like SFRI-1) or DSLs to be created that expect callers can conform by contract instead of by using special forms. Callers, in turn, can create abstractions over the libraries and DSLs that are equally general, instead of being strongly influenced by what they're built on.
Clojure does a really nice job of creating reusable abstractions. From what I understand, tail-call optimization isn't possible in the JVM, so we have iterators and lazy evaluation first-class instead; which has other nice advantages for creating abstractions.
Yep, and the same can be implemented in any language that supports functions-as-objects (for example, javascript).
It's a practical solution/workaround, but isn't quite composable/generalizable as first-class support for TCO (unless the community is willing to follow some convention like "TCO-like behavior is a 'good thing' and so libraries should try to support and interface compatible with `trampoline`".
This is a little bit like `function` / yield in newer versions of Javascript. It's not TCO, but it is a strong community convention (with the assist of special syntax to enforce the semantic).
Clojure is a great example of how practical/tasteful decisions by language author can create re-usable abstractions that are really powerful (even while at the same time working around limitations of the underlying technology).
Some might also argue that TCO is "bad" because it's less debuggable (e.g. when an exception is thrown, the stack trace is gone). We can see the same phenomena with non-blocking IO, for example. So maybe it's a Good Thing to have "less general" semantics in a language (e.g. function, trampoline, and loop/recur may be good-enough while at the same time preserving debuggability).
There are always tradeoffs. What's nice about general TCO is that the composibility of unrelated libraries/DSLs is more probable whereas with conventions like trampoline, DSL/library authors need to understand the advantages and design the DSL/library in a compatible way in advance. As Clojure shows with lazy sequences, this can absolutely work given enough community awareness of the convention.
Another thing to mention: having guarantee of TCO supports higher-level control-flow abstractions (capturing continuations), which is also an interesting outcome...
I think by 'production', they mean 'ready for general use on developer laptops'. No one in their right mind is deploying actual production software on Docker, on OS X/Windows.
I've been using it on my laptop daily for a month or two now, and it's been great. Certainly much better than the old Virtualbox setup.
>No one in their right mind is deploying actual production software on Docker, on OS X/Windows.
Since the whole point of Docker would be to deploy these in production and not just for development, I don't see how the term 'ready for production' can be used. Isn't this just a beta?
Sorry, CoreOS is Linux as well, but in my mind it's enough of a hyper-specialised immutable auto-updatable container-specific version of Linux that it warrants a separate category when talking about Docker.
I don't think so. That's what Jeffery Snover is working on in Server 2016 with Windows nano server.
Unless something has changed since the last time I checked, The WindowsServerCore docker image was not generally available yet and requires server2016 (I think it was TP6 the last time I checked)
Docker, to my knowledge, is still exclusively Linux flavors. (Though I'm happy to be corrected if someone knows more than me)
Docker images still aren't generally available, but you can now run Windows Container Images based on the NanoServer docker image (and WindowsServerCore image if you replace nanoserver with windowsservercore in their image URL in the docs below) on Windows 10 (insiders build)[0].
you would use kubernetes, dc/os, swarm mode for aws, etc for that. Containers are portable.. nobody is launching a windows vm and doing a "docker run" for their production env
The fact that I can have Bash up and running in any distro I feel like within minutes blows my friggin mind. Docker is the stuff of the future. We were considering moving our development environment to Docker for some Fun, but we're still holding off until it is more stable and speedy.
Leaving containers vs VMs aside, docker for Mac leverages a custom hypervisor rather than VirtualBox. My overall experience with it is that it is more performant (generally), plays better with the system clock and power management, and is otherwise less cumbersome than VirtualBox. They are just getting started, but getting rid of VirtualBox is the big winner for me.
When I used VirtualBox for Docker (using Docker machine/toolbox), I would run out of VM space, have to start and stop the VM, and it was just clunky all around.
Docker.app has a very nice tray menu, I don't know or care anything about the VM it's running on, and generally is just better integrated to OS X. For instance, when I run a container, the port mapping will be on localhost rather than on some internal IP that I would always forget.
On the other hand I find my old setup with VMware much more reliable and performant. And I can continue to use the great tools to manage the VM instead of being limited to what docker provides. Some advanced network configuration is simply impossible in docker's VM.
This isn't a product that's "ready for production"; it's a product company declaring that it is.
This means what it's always meant: that the company believes the sum they'll make by convincing people it's "production ready" is greater than the sum they'll lose from people realizing it isn't.
Keep in mind the optimal state of affairs for Docker Inc. is one where everyone is using Docker and everyone requires an enterprise contract to have it work.
I agree that it is confusing. Production ready in the sense that it is stable for the targeted use-case: local development environments. Not for "production". Damn now i'm confused...
Exactly. "Ready for production" and "industrial" are constantly abused. All these tools are awesome and we use them, but PROPERLY deploying and supporting them in production is far from painless (or easy).
I think many view "ready for production" as a sign of what they do have in place is stable enough and support options are available so that it ticks all the CTO/CEO boxes in business plans.
Which basicly gets down to when your CTO/CEO or some manager comes in preaching docker - we should be doing that, why arn't we has one less argument to dismiss it now than before.
Yes many aspects need improving but case of what is there is deemed to of gained enough run-time in environments to be deemed stable enough to say, we can support this in production for off the shelf usage without you needing lots of grey-bearded wizards to glue it all in place and keep it that way.
I'm not completely disagreeing with you but Debian in recent years has taken massive steps backwards as far as production stability. Jessie for example did not ship with SELinux enabled which was a key deliverable for Jessie to be classed as stable / ready for production, what's worse is it doesn't ship with the require SELinux policies - again another requirement before it was to be marked as stable, it's filled with out of date packages (you know they're old when they're behind RHEL/CentOS!) and they settled on probably the worst 3.x kernel they could have.
You've given one example; SELinux. Did wheezy ship with SELinux enabled? No. So how is that a step backwards? It would have been a step backwards if they shipped with it enabled and it was half-assed. SELinux is notoriously hard to get right across the board. See how many Fedora solutions start with "turn off SELinux." Shipping jessie without SELinux enabled was the right thing to do, if the alternative was: not shipping jessie; or shipping borked jessie with borked SELinux support on by default. Those who know what they are doing can turn it on with all that entails.
You gripe about kernel 3.16 LTS but provide no support for your statement. With a cursory search I can't find any. If it was such a big deal I have to assume I would. For my part I use Jessie on the desktop and server and have not encountered these mysterious kernel problems of which you complain. Again, you may have wished for some reason that they shipped with 3.18 or 4.x, but they shipped. They have 10 official ports and 20K+ packages to deal with, I'm sorry they didn't release with your pet kernel version. Again, those who know what they are doing can upgrade jessie's kernel themselves if they are wedded to the new features.
Oh believe me, we did try to contribute to Debian, in recent years the community has aged poorly and become toxic and hostile, where the Redhat / CentOS community has grown, is more helpful and we have found them to be more accepting of people offering their time than ever.
The sales pitch I usually give people is that any ops person can read a Dockerfile, but most devs can't figure out or help with vagrant or chef scripts.
But it's a hell of a lot easier to get and keep repeatable builds and integration tests working if the devs and the build system are using docker images.
You are doing it wrong then. People run containers in production using orchestration platforms, like ECS, kubernetes, mesos etc. The docker for mac/windows are not designed to serve containers in production environments.
They help you build and run containers locally, but when it comes time to deploy you send the container image to those other platforms.
Using docker like that is like running a production rails app with "rails s"
Which security problems are you referring to? Our containers run web applications, we aren't giving users shell access and asking them to try and break out.
Over large layers: Don't run bloated images with all your build tools. Run lightweight base images like alpine with only your deployment artifact. You also shouldn't be writing to the filesystem, they are designed to be stateless.
Credentials capture in layers. Environment variable oversharing between peer containers (depending on tool).
And the fact that nobody involved in Docker is old enough to remember that half of the exploits against CGI involved exposing environment variables, not modifying them.
Yes, you can capture runtime secrets in your layers, but it's pretty obvious to everyone when you're doing that and usually people clue in pretty quickly that this isn't going to work.
Build time secrets are a whole other kettle of fish and a big unsolved problem that the Docker team doesn't seem to want to own. If you have a proxy or a module repository (eg, Artifactory) with authentication you're basically screwed.
If you only had to deal with production issues there are a few obvious ways to fix this, like changing the order of your image builds to do more work prior to building your image (eg, in your project's build scripts), but then you have a situation where your build-compile-deploy-test cycle is terrible.
Which would also be pretty easy to fix if Docker weren't so opinionated about symbolic links and volumes. So at the end of the day you have security-minded folks closing tickets to fix these problems one way, and you have a different set that won't provide security concessions in the name of repeatability (which might be understandable if one of their own hadn't so famously asserted the opposite http://nathanleclaire.com/blog/2014/09/29/the-dockerfile-is-... )
I like Docker, but I now understand why the CoreOS guys split off and started building their own tools, like rkt. It's too bad their stuff is such an ergonomics disaster. Feature bingo isn't why Docker is popular. It's because it's stupid simple to start using it.
Regarding secrets in builds, I think a long term goal would be to grow the number of ways of building Docker images (beyond just Docker build), and to make image builds more composable and more flexible.
One example is the work we've experimented with in OpenShift to implement Dockerfile build outside of the Docker daemon with https://github.com/openshift/imagebuilder. That uses a single container and Docker API invocations to execute an entire Dockerfile in a container, and also implements a secret-mount function. Eventually, we'd like to support runC execution directly, or other systems like rkt or chroot.
I think many solutions like this are percolating out there, but it has taken time for people to have a direct enough need to invest.
>> Debian is a good example of the opposite end of Docker.
It is not fair to compare Docker with Debian. Docker Inc (who backed Docker) is a for-profit corporation and is backed by investors. It is understandable why they need to push their products into production the soonest time possible.
I use Docker a lot. I also use things like Docker volume plugins and have had to modify code due to API changes/breakages.
"Production ready" in the "container space" for me are Solaris Zones, FreeBSD Jails, and to an extent lxc (it's stable, but I've used it less). I like what Docker/Mesos/etc. bring to the table, but when working with the ecosystem, it takes work to stay on top of what is going on.
It is even harder to consult with a customer or company interested in containers and give the most accurate near/long term option. It becomes a discussion in understanding their application, what approach works now, and guidance for what they should consider down the road.
Networking and Storage are two areas with a lot of churn currently.
What does it matter how fair it is? It's neither fair to compare a monkey to a fish in terms of being able to climb trees, but that doesn't change that one of the two is most likely already sitting on a branch. And ultimately, if you need something that can climb trees, a fish simply won't do, no matter how fair you try to treat it.
I can't get it to work on OSX without the CPU staying at 100%. Still not fixed:
> There are several other threads on this topic already. Setups that docker build an image and rely on in-Docker storage work well; setups that rely heavily on bind-mounting host directories do not. A complex npm install in a bind-mounted directory breaks Docker entirely, according to at least one thread here.
This is another issue that's been preventing my adoption of Docker for Mac: https://forums.docker.com/t/docker-pull-not-using-correct-dn.... The fact that DNS resolution over a VPN still doesn't work correctly makes me wonder how production-worthy this release is. It's a pretty common thing people want to do in my experience.
If you have the time, could you make a report on the issue tracker https://github.com/docker/for-mac/issues and include the contents of /etc/resolv.conf and "scutil --dns" when you connect and disconnect to your VPN? Ideally also include an example resolution of a name by the host with something like "dig @server internalname". I suspect the problem is caused by a DNS server in the "scutil" list being missing from /etc/resolv.conf. We're planning on watching the "scutil --dns" list for changes, but it's not implemented completely yet.
Okay, will do. Resolution of internal hostnames by their FQDN works fine if I set my VPN client (Tunnelblick) to rewrite /etc/resolv.conf. That said, the search domain is not carried into the VM, so name resolution by hostname does not work. Also, Tunnelblick has a name resolution mode that does split DNS (i.e. preserves DHCP-set DNS servers and only forwards DNS requests for the internal domain to the VPN DNS servers). This mode doesn't work at all. Would it be possible to allow forwarding of DNS requests to the host machine like with Virtualbox (VBoxManage modifyvm "VM name" --natdnshostresolver1 on)? I feel like that would simplify things greatly.
I always thought of production ready to be stable, of all things. Feature complete is not a part of it.
Basically, if you can live with the shortcomings a release has (bugs, performance, lack of features) you can use it in production as long as it's stable (and secure).
This bug's been driving us mad because we can't reliably repro it on our machines at Docker, and it only happens to a small subset of users, but is very annoying when it goes trigger. It seems to be related to the OSX version involved, but there's not enough bug reports to reliably hone in on it.
The other aspect that it may be is a long-running Docker.app -- since as developers we are frequently killing and restarting the application, it could happen after a period of time. I've now got two laptops that I work on, and one of them has no Homebrew or developer tools installed outside of containers, and runs the stable version of Docker.app that's just been released. If this can trigger the bug, we will hunt it down and fix it :-) In the meanwhile, if anyone can trigger it and get a backtrace of the com.docker process, that would be most helpful. Bug reports can go on https://github.com/docker/for-mac/issues
The filesystem is still not as fast as I would like, but it's incredibly improved over the last couple months.
One thing I found, was to be a little more cautious about what host volumes you mount into a container: for a Symfony project, mounting `src` instead of the whole folder sped up the project considerably, as Symfony's caching thrashes the file-system by default.
That works for some use cases, but for others (Elasticsearch, Zookeeper, Kafka, etc) the service inside the container needs to bind to an interface associated with an IP that's also addressable by the host. Even in host networking mode, eth0 inside a DFM-powered container will bound something like 192.168.x.y but that 192.168.x.0 subnet is completely inaccessible from the host.
I've been heavily using it since what must have been early closed beta, and cannot recall ever having this issue. Might be something that isn't quite so widespread.
It only happens with a few users, and not at all to the majority. It seems to happen more on older OSX versions, but beyond that there has not been anything identifiable in common about the systems it happens on unfortunately.
Not to mention the lack of host:container socket sharing and the fact that the Moby VM time drifts due to system sleep. I love Docker for Mac, I use it every day, and it's definitely still beta quality.
How much does your time drift? We changed the mechanism so that it should sync from the OSX ntp server now, which seems to be giving good results. If you are having problems can you create an issue and we can look into it.
Host container socket sharing will come, but it is complex as sockets only exist with in a single operating system, so we have to bridge them across two. We are using this for the docker socket, and debugging the issues across Mac and Windows, so it is in the roadmap.
Actually GA may have fixed this. I was able to reproduce it, but may have checked too quickly. I opened https://github.com/docker/for-mac/issues/17 against it, and may end up closing it.
For #3 that is an issue for remote debugging with things like XDebug for PHP. I have been using this command:
sudo ifconfig lo0 alias 10.254.254.254
And setting the remote host to 10.254.254.254 instead of localhost inside the container to work around that issue. It's been working pretty well.
I've been using the Beta version of Docker for Mac for many months and haven't had many issues with it at all. The biggest issue I've seen was the QCow file not releasing space and growing to 60+GB, but deleting it and restarting Docker did the trick (although I had to rebuild or repull any containers).
I had a similar experience trying to switch to docker-machine as it sounds like you've had with the new apps, and ended up giving up.
It's super simple through Vagrant though, just vagrant up and set DOCKER_HOST to the static IP. Plus there are vagrant plugins that let you sync a directory to the vm in a way that gives you inotify events so live build/update tools can run in your containers (which btw is huge, I can't believe the official apps haven't even attempted to address that, as far as I've seen).
The company claimed back in March [0] that Docker for Mac addresses the filesystem events. I observed that it works.
While Docker for Mac has improved somewhat over the beta, unfortunately it's still quite rough. For example, it was only last week that they pushed a fix for the DNS timeout issue [1] (I think maybe it was fixed? I can't check because Docker for Mac is not open source).
The DNS resolving code in Docker for Mac is in the VPNkit project which is open-source: https://github.com/docker/vpnkit. A DNS timeout is a fairly general symptom and it's hard to diagnose fully without a packet capture, but there's one issue that I'm aware of: if the primary server is down then the UDP DNS queries and responses will use the host's second server. However if a response is large and requires TCP then unfortunately we will still forward it to the primary server, which obviously won't work :( I've filed this issue about it: https://github.com/docker/vpnkit/issues/96. We hope to improve DNS, VPN and general proxy support for 1.13 -- please do file issues for any other bugs you find!
> Plus there are vagrant plugins that let you sync a directory to the vm in a way that gives you inotify events so live build/update tools can run in your containers (which btw is huge, I can't believe the official apps haven't even attempted to address that, as far as I've seen).
If you don't mind, what are these plugins? This is one thing that's sorely missed when I do development with Vagrant. I did a small amount of searching and trial and error, but couldn't find a solution that worked for me.
There used to be a separate vagrant plugin for rsync but it's now built-in. There is also built-in support for NFS and virtualbox/vmware synced folders. These all work reasonably well until you start having fairly large numbers of files/directories.
Also if you use a native Linux host with LXC or Docker there is no overhead for sharing directories with the container, it's just a bind mount.
I don't believe NFS supports inotify events? At least, that's what I'm using, and I'm forced to use polling for any file change detection. And rsync is one-way IIRC. But yes, LXC on Linux works great when it's feasible; I've just been looking for something that supports file change detection on other platforms.
The last one, in my experience, is basically a deal breaker. Simple commands (eg rake routes, npm install, etc) take 100x longer.
I'm don't have a firm opinion on what is or isn't 'production ready', but if there are major bugs, then there should be some way of disseminating that information instead of everyone rediscovering the same issues.
Number (3) is specially painful. The fact that their documentation makes it very explicit that the host is bridged by default and containers are "pingable" aggravates it a little bit further as it seems as a very basic pre-requisite for the tool to be usable.
for 4) you can use http://docker-sync.io - its compatible with docker for mac and others, supports rsync/unison/unison+unox and will have NFS support in the near future.
With unison+unox you have full transparent sync while having native performance (no performance loss at all). This is far better the osxfs or nfs.
I resonate with the sentiment of this article, but have struggled to find an alternative that’s fast enough as the test suite grows, isn’t flakey in CI, and is able to share the production schema definition for relevant relations.
I’d love to hear more from anyone that’s solved for some of these constraints!