Even more important, npm doesn't reliably reproduce builds. Different runs of the same shrinkwrapped build could either fail or succeed -- which is odd given that builds should be deterministic. Additionally, shrinkwrap is also broken in the same way.
This is the first time I've heard of ied, a little competition could go a long way!
Could you expand on that? What causes it to be nondeterministic? I haven't had that experience but it's the second time I've read someone reference this behavior
> You can reliably get the same dependency tree by removing your node_modules directory and running npm install whenever you make a change to your package.json.
I have a project that's shrinkwrapped. I check out a new copy of the project and run `npm install`. My co-worker does the exact same thing, his fails, yet mine successfully builds. An install of a shrink-wrapped should be deterinistic, i.e. either succeed or fail, but not both.
From that doc page it sounds like that should be deterministic? Even without shrinkwrapping, a fresh install from package.json with empty node_modules should be deterministic.
> The npm install command, when used exclusively to install packages from a package.json, will always produce the same tree. This is because install order from a package.json is always alphabetical. Same install order means that you will get the same tree.
> You can reliably get the same dependency tree by removing your node_modules directory and running npm install whenever you make a change to your package.json.
Maybe you're experiencing a bug, rather than some in-grained non-determinism in npm?
My understanding is that even when you fix your dependencies to exact versions, your dependencies probably haven't so without shrink-wrap you'll never know _exactly_ what gets installed.
Indeed; a lockfile is just a workaround for the problem of "just give me whatever" dependency declarations. Solve the problem at the root rather than piling hacks on it.
I'm not 100% sure from your comment, but are you suggesting that by default you should vet every version exactly and freeze it at the point you defined it?
As someone writing a lot of ruby, "give me whatever" is analogous to "give me the latest unless I specify otherwise", which I consider to be a very good default. It keeps me up to date with security issues, and incompatibilities between libraries that the respective maintainers resolve amongst themselves with a minimum of manual work.
> are you suggesting that by default you should vet every version exactly and freeze it at the point you defined it?
Yes, this is the way that Guix and Leiningen and rebar3 and a bunch of other things work, and it is wonderful.
Pulling in new code without you asking is a fine idea for something like apt-get where you have a huge team doing QA on the entire system working together before it even hits your repositories, but for most package managers, the dev team is the one doing the testing, and upgrades should be done only with great care.
It does mean you have to watch for security updates, but this is true of all package managers.
> Yes, this is the way that Guix and Leiningen and rebar3 and a bunch of other things work, and it is wonderful.
Even in Nix/Guix, it's still ideal for upstream projects to express their dependencies in terms of ranges (semver-wise), otherwise we run into the problem have either really large run-time dependency closures, or problems around e.g. wanting to use multiple (overly specified) versions of C libs within the same process.
As the current maintainer of Nixpkgs' Bundler-based build infrastructure, I've found the lockfile approach that Bundler uses to be quite frustrating - in part because Bundler's design is antithetical to packaging, but also due to the build times and sizes of the resulting packages, compared to C libraries. (People give C a hard time wrt productivity and security and such, but when it comes to packaging, C libs are usually so much easier to work with than most other higher level languages.)
I would love to see more adoption of semver, and possibly Haskell's PVP (https://wiki.haskell.org/Package_versioning_policy). Granted, dynamic programming languages don't have the benefit of making API breakage obvious at build time, so perhaps the best we can do in such cases -- if we want any certainty that packaged applications will actually function correctly -- is lock down every dependency version precisely per application...
If you want to do this in ruby you can just specify a version manually. If you don't, you still get versions frozen by default with Gemfile.lock. It doesn't pull in anything automatically—by default you get no updates, you can choose to update a single dependency, or the whole thing if you want to verify it works on the latest (useful for libraries for instance). I'm not sure I see the downside.
FWIW I have been doing ruby for over a decade now, and I hold up Bundler as one of the great success stories of open source, and it is one of the reasons I hold Yehuda Katz in high regard, in that he was able to solve a really big problem in the community and hammer it into shape aggressively over a period of two years with a lot of doubters and naysayers (even Rubygems core was against Bundler for a long time), until it finally got so solid for so many use cases (libraries vs apps, private vs public, development vs deployment, etc, etc) where it solved nearly everyone's problems in such a solid way that everyone adopted it.
> I hold up Bundler as one of the great success stories of open source
It's a great success in many ways, but the problem it solves is completely self-inflicted by rubygems. I've also been doing Ruby for over a decade, but I've also learned a lot from other library ecosystems, and I feel pretty confident saying that disallowing version ranges makes all those headaches completely evaporate.
I'm with you that version ranges cause problems, but my belief is that lockfiles are a better solution.
Generally I'd use semver ranges in libraries, and then fixed versions + lockfiles for transitive deps in applications.
I suppose this is roughly equivalent to doing `:pedantic :abort` in leiningen, except you wont't have as many warning to squash - either way you have to rely on the test suite to tell you if the versions you've pegged work.
We use Ruby extensively in testing and infrastructure and have lost months of cumulative developer time to this attitude. It works for a single user constantly making changes to a small piece of code and prepared to work out the solution to dependency change problems as soon as they come up. However, it doesn't scale to a team of people working on very large code bases.
One example of this is having a repeatable developer setup guide. If the dependencies might have changed by the time a new joiner starts your setup guide could very easily end up useless or misleading and that's without anything at all in your own codebase having changed.
Shrinkwrap fixes this, but always gets added after things went wrong several times already. It should be the default.
> Mix, Erlang's package manager (used by Phoenix / Elixir)
Slight nitpick: Mix is part of Elixir, I've never seen a (non-elixir) erlang project that use it. Don't think erlang has an equivalent officially blessed tool, but the most popular one is rebar / rebar3.
(While I'm nitpicking: mix and rebar are more build & dependency managers; mix uses hex for package management. I believe rebar3 can also use hex. In ruby terms, mix ~ bundler (among other things); hex ~ rubygems).
This is a good start, but it has issues. You probably don't want all of your locally installed packages in a requirements.txt file. Instead, they should be curated. I've been using pip-compile [0] for a while now (see the author's blog post [1] for a detailed explanation) and have become a big fan of it. With this model, you should enumerate only what your application uses directly and let pip-compile convert this list to a full version-locked requirements.txt. (Shameless plug: I also wrote a bit about why this is the best currently available option for specifying Python dependencies [2].)
This should be used with virtualenv, not as an alternative to it. I have a bunch of libraries and tools in almost all of my virtual environments that are not application dependencies and have no business being installed in my production environment (e.g. bpython, tox, flake8, and neovim). This approach also handles things like a dependency being dropped gracefully (if some library no longer needs a dependency, it magically gets removed from your locked requirements.txt next time you compile requirements.in). Python's package management tools are making great strides (see pip's recent peep-style hash checking support), and pip-compile is a big win in that category in my opinion.
And it's so awesome when you discover devs have used lockfiles to leave you stuck on known-insecure gems instead of updating their fucking code to support the newer patched version.
Which is still better than having your server crash due to a bug in a dependency of a dependency that got updated without your knowledge when you deployed something trivial. It has happened to be more than once.
same issue here.
this non-deterministic behavior is really fxxked up. there's one time that our building process suddenly begin to fail, spend a few hours on the issue, and it turned out to be one of the babel-core patch release is broken.
some of the very fundamental designs of npm is seriously wrong.
> Canola oil is one of the worst fats one could put in their body because it contains erucic acid that damages the heart.
Terrible piece, but this comment, without references, puts the cherry on top. This guy is about as bad as Food Babe.
I'm not making any judgements on Soylent, since I am not a nutritionist. However, anyone who's actually interested in any dietary supplement, including Soylent, should read about the industry's massive lobbying to prevent regulation by the FDA. Indeed, instead of the companies having to prove that their supplements are not harmful (via clinical studies and trials), it instead falls on the 20 something FDA staff to prove that they are harmful -- against the thousands of supplements available for sale in the United States [1].
> You don't debug a unikernel, mostly; you just kill and replace it
Curious as to how you would drive to root cause the bugs that caused the crash in the first place? If you don't root cause, won't subsequent versions still retain the same bugs?
There are bugs that can only manifest themselves in production. Any system where we don't have the ability to debug and reproduce these classes of problems in prod is essentially a non-starter for folks looking to operate reliable software.
Personally, I wouldn't use unikernels for my custom app-tier code† (unless my app-tier either was Erlang, or had a framework providing all the same runtime infrastructure that Erlang's OTP does.)
Instead, unikernels seem to me to instead be a good way to harden high-quality, battle-tested server software. Of two main kinds:
• Long-running persistent-state servers that have their own management capabilities. For example, hardening Postgres (or Redis, or memcached) into a unikernel would be a great idea. A database server already cleans and maintains itself internally, and already offers "administration" facilities that completely moot OS-level interaction with the underlying hardware. (In fact, Postgres often requires you to avoid doing OS-level maintenance, because Postgres needs to manage changes on the box itself to be sure its own ACID guarantees are maintained. If the software has its own dedicated ops role—like a DBA—it's likely a perfect fit for unikernel-ification.)
• Entirely stateless network servers. Nginx would make a great unikernel, as would Tor, as would Bind (in non-authoritative mode), as would OpenVPN. This kind of software can be harshly killed+restarted whenever it gets wedged; errors are almost never correlated/clustered; and error rates are low enough (below the statistical noise-floor where the problem may as well have been a cosmic ray flipping bits of memory) that you can just dust your hands of the responsibility for the few bugs that do occur (unless you're operating at Facebook/Google scale.) This is the same sort of software that currently works great containerized.
---
† The best solution for your custom app-tier, if you really want to go the unikernels-for-everything route, might be a hybrid approach. If you run your app in both unikernel-image, and OS-process-on-a-Linux-VM-image setups, you'll get automatic instrumentation "for free" from the Linux-process instances, and production-level performance+security from the unikernel instances. You could effectively think of the Linux-process images as being your "debug build."
Two ways of deploying such hybrid releases come to mind:
• Create cluster-pools consisting of nodes from both builds and load-balance between them. Any problems that happen should eventually happen on an instrumented (OS-process) node. This still requires you to maintain Linux VMs in production, though.
• Create a beta program for your service. Run the Linux-process images on the "beta production" environment, and the unikernel-images on the "stable production" environment. Beta users (a self-selected group) will be interacting with your new code and hopefully making it fall down in a place where it's still instrumented; paying customers will be working with a black-box system that—hopefully—most of the faults will have been worked out of. Weird things that happen in stable can be investigated (for a stateful app) by mirroring the state to the beta environment, or (for a stateless app) by accessing the data that causes the weird behavior through the beta frontend.
> In fact, Postgres often requires you to avoid doing OS-level maintenance, because Postgres needs to manage changes on the box itself to be sure its own ACID guarantees are maintained.
With unikernels you get a lot more consistency. E.g. I once saw a bug that came down to one server using reiserfs and another using ext2. But there's no way to have that problem with a unikernels.
But sure, you need a debugger. So you use one. I'm not sure why the author seems to think that's so hard.
> But sure, you need a debugger. So you use one. I'm not sure why the author seems to think that's so hard.
The author wrote and continues to contribute to DTrace, which is an incredibly advanced facility for debugging and root causing problems. GDB (for example) doesn't help you solve performance problems or root-cause them, because now your performance problem has become ptrace (or whatever tracing facility GDB uses on that system).
The point he was making is that there are problems with porting DTrace to a unikernel (it violates the whole "let's remove everything" principle, and you couldn't practically modify what DTrace probes you're using at runtime becuase the only process is your misbehaving app -- good luck getting it to enable the probes you'd like to enable).
You can't modify them from within your app, sure. You modify them from a (privileged) outside context. Allowing the app to instrument itself that thoroughly violates the principle of least privilege the author was so fond of.
There's a lot more that can happen differently there. Docker doesn't hide all the details of the filesystem, kernel version, or the like. With EC2 instances you're still running a kernel that has a lot of moving parts of its own.
It's a lot harder to get consistency out of a non-unikernel system running on EC2 - e.g. IME the linux boot/hardware probing process can behave nondeterministically before it even starts running your user program.
Right, I agree with this. There's nothing magic about any of this -- ultimately it's all just software, hardware and physics.
To describe something as magic in a serious context is toxic. We're not sleight of hand performers, we're engineers. It's more helpful to state something like, "this library is complicated and the internals are beyond the scope of this post". A pointer to documentation and source code on libuv would be even better. In face, this series of articles on libuv are a great start for those who are interested: https://nikhilm.github.io/uvbook/introduction.html
If all occurrences of "this is [magic]" are treated as "this is [sufficiently complicated internally and beyond the scope of this post]," then what's the difference? The former makes for a point that's expressed more succinctly and eloquently at little real loss.
Definition 2a seems related: "Producing [...] remarkable results, like those attributed to magic [...]; effecting or permitting change, success, etc., as if by magic."
Also the 'magic box'.
The Urban Dictionary (the place where I look for colloquialisms) has this definition: "with or through complexity either spurios to the contemproary context or to long to be reasonable or desired of explanation". [1]
This is what we like to call a "figure of speech". It means that we do not believe to be literally magic, but are using hyperbole to describe a situation where something happens and it's not immediately apparent why.
I don't think it's fair to say "every single piece of software", as the claim that it's impossible to write secure software is just a myth. It's not very hard to write a secure "hello world".
Then there's also Coq and such.
Of course, usually the amount of vulnerabilities exponentially correlates to the size of the codebase.
Nope. HTTPie is much, much more focused than cURL (the CLI tool). It is exclusively an HTTP client designed primarily for interactive use. cURL, on the other hand, is meant to be used mostly non-interactively and is more of a general client for "URLs" (it also supports DICT, FILE, FTP, FTPS, GOPHER, IMAP, IMAPS, LDAP, LDAPS, POP3, POP3S, RTMP, RTSP, SCP, SFTP, SMTP, SMTPS, TELNET and TFTP).
HTTPie's goal is user-friendliness, so it comes with features like built-in JSON support, output formatting, syntax highlighting, extensions, etc. Comparison of the two in one screenshot can be seen in the README:
I use and love httpie but I start to regret that it's written in Python as the Python platform is a very very late adopter of modern TLS standards. Even today, in 2015, it's still a hurdle to use httpie with a SNI endpoint, and forget about it making it talk to a ECDSA-based endpoint like those exposed by the free tier of Cloudflare (so called "modern TLS"). I wonder how long it will take to HTTP2 to land.
I tried a random rewrite of httpie in Go and it worked perfectly with those endpoints, as expected, and Go is getting full HTTP2 with 1.5 in June. Too bad the rewrite was still feature incomplete, when I tried it.
The situation is not ideal but more recent versions of Python (2.7.9 and 3.4) have a much improved TLS support and things like SNI work out of the box.
For older Python versions there are some additional dependencies to be installed manually (will happen automatically in next version of HTTPie):
Serves the same purpose, for sure. However, I think the interface is more intuitive/clean for json rest apis.
I usually end up using HTTPie for manual work / ad-hoc checks. And then when I script it I tend to end up using curl, because curl is available everywhere and I never need to worry if it's installed everywhere I'll want to the script to run.
This is great. I was hoping the data charges would be cheaper. The T-mobile $30 prepaid plan gives you unlimited data with the first 5 gigs at LTE speeds. You also get 100 minutes. This works incredibly well for me as I mostly use my phone for data. If I need to make a call, I'll use google hangouts over LTE.
Sure, but TMobile doesn't have the best coverage outside of cities. I have this plan as well, but once I leave NYC and get out to rural Pennsylvania, the coverage is abysmal. Often minimal cell service, let alone data.
They've offered it for a while I think? It's a pre-paid, look in Other monthly plans. It's pretty hidden, I don't think they want everyone to know about it. Also can be Walmart only, but I think you can do it online just fine. http://prepaid-phones.t-mobile.com/prepaid-plans
I can definitely tell the difference. A friend of mine has the AXE-FX. We tried testing its Mesa Boogie model against my actual Mesa Boogie amp -- and the difference was night and day.
Like one of the comments from above, part of the magic of tube guitar amplification is that it is much more responsive dynamically to inputs. You can turn the volume up a tick, or hit the strings harder or softer, or change your picking style, and the results are drastically different on a tube amp.
This gives the artist more space to express themselves -- which is why tube amps are still favoured amongst guitar players today -- decades after transistor's have taken over the rest of the electronic industry. You can't get the nuanced expressiveness in a crystal lattice that you can in the free space of a vacuum. [1]
[1] I'm paraphrasing from H. Alexander Dumble, one of the gurus of guitar amplification.
Sure. Interesting how confident you are in this assumption - there are plenty of audiophiles who claim they can definitely night and day hear the difference in 2 brands of speaker cable. So presumably you've blind tested yourself and are definitely sure it's not bias, or wishful thinking, or feeling special because you've got the real deal.
The AXE FX II models the differences with tube amplification that you're talking about.
I also don't like that quote any - by the logic of that quote MP3s and CDs are equally rubbish recordings that have no room for expression? Or are we only extending it to what suits your view?
There are a vast number of ways to customise a real tube amp (I've owned a Mesa Boogie Mark IV) and an Axe FX II (which I currently own.) A poorly setup mesa vs. one that's been carefully configured will be night and day as well. Did you factor that in? Was there a tone match? Actually the Axe FX doesn't claim to emulate particular models precisely in many cases (see http://wiki.fractalaudio.com/axefx2/index.php?title=Amp:_all...)
My point is you dismiss it as if you can know for sure - go do some blind testing where these factors are taken into account (you can start with a skeptic's view at https://www.youtube.com/watch?v=-EtxHlJ2FPo) and then see if your certainty is affected.
Finally, Metallica have chosen to do all live shows using only Axe FX's. So obviously you know better than this small band who barely tour.
I know HN is basically turning into the new reddit, but comments like these where you just make a statement and then act as if it's established then go on to quote some guy really make me wonder whether reading the comments section is going to continue to depress me forever at this point.
Are you hoping that I just... made up that point? I find it interesting the degree of denial audio guys have with this stuff. The Axe FX is a really amazing and inspiring piece of machinery that allows for a huge artistic freedoms and is basically the product of one guy's dedication to accuracy (many amps are modelled to the extent of their entire circuitry being simulated as well as precise physical models of the valve sets which you can change, etc. etc.) - when I wonder why the Axe hasn't got the love it deserves (though there's growing love for it) the reaction of 'not the same as analogue' guys reminds me why :)
>You can't get the nuanced expressiveness in a crystal lattice that you can in the free space of a vacuum.
Cute, but at the end of the day it's just different nonlinearities in the transfer function. If you like how ancient technology sounds, that's fine, no need to ascribe mystical properties to it.
There is a difference, like you say (I owned an Axe-Fx). It sounded very good, especially the Plexi, but at the end of the day it sounds like a mic'd amp. You can't make it sound like an unmic'd cabinet, leading to lots of [1] discussion [2] about getting that 'in the room' sound [3]. It sounds pretty much exactly like what you would hear on a CD; it's definitely good at getting that studio mix sound.
From what I understand and what I've read on Axe-Fx II emulation, it emulated the circuit components of the amps, including analogue ones, to emulate the signal processing. If that is true, I can see how some aspects of emulation could be worse, but a drastic difference in dynamics doesn't seem possible. I also own Axe-FX II, and can't tell the difference between it and the real amps, in any regard.
This is the first time I've heard of ied, a little competition could go a long way!