I've found there to be a pretty huge difference in those offices, particularly the European offices. The staff is far more diverse (e.g. I'm on a 40 person team and the largest number of people from one country is 4) and the culture and attitude to work is completely different. Much less koolaid-drinking and much better work/life balance. People actually use their (much more generous) vacation time!

At the 50-person company I worked at 6-8 years ago, my team's product was custom Ubuntu-based OS images that booted appliance-style on customer hardware and ran our product full-screen. We had some very nice tooling based on the live CD creation stuff that would a) build our cross-platform code into Debian packages b) patch and rebuild a couple of Ubuntu packages, or fail the build c) generate a nice OSS compliance bundle and d) output a .iso as well as a directory of .debs for internal interactive testing (so you could upgrade a machine you'd already installed without reinstalling it). We wrote some small custom tooling to snapshot the Ubuntu repo metadata periodically (https://github.com/mokafive/apt-mirror) and build using the snapshot as of the time the SVN branch was created. Keeping this system running, once it was built, took very little time - that project was briefly down to one engineer and I could keep writing code and trust the build system to work fine. If I needed to implement that today using the best commodity tools I could, I'd probably be in some sort of Dockerfile hell, and in fact I'd probably be strongly inclined to design the customer system on top of Docker somehow, even though it would be absolutely the wrong fit, simply because that's what everyone else's OS-image-building problems look like these days.

I guess the flip side is that there are probably few startups building products like that, and many more startups where you're building a CRUD app in Rails^WNode.js and your value to the customer is that you're just doing a really good job of writing a CRUD app. That's fine and valuable, and there's an interpretation where it's my personal failing that I'm not really excited by that, even if it's delivering business value / helping the world / etc.. But it is how I feel (and my guess is the author feels the same way), and I once did leave a company because I felt, for lack of a better term, bored - I was having to invent my own opportunities to grow as an engineer, and it was made clear to me that I would get better performance reviews if I worked on less ambitious projects. And when I've been interviewing people at my current place, I've heard enough stories of people being equally bored at their current jobs at large companies.

So - this isn't a BigCorp vs. startup argument, I think. It's an argument that different companies have different levels of appetite for build-and-encourage-internal-tooling vs. buy-and-conform-to-the-outside-world in their culture, and if you're an engineer who's going to be frustrated by the latter, look for a company (of any size) that believes in the former.

Like, HN comes from YC, it's not surprising that there's a focus on startups and the sorts of BigCorps that grew from startups in living memory. But Java, C# and VB.NET are all in the Tiobe top-10, and I feel like HN has a real blindspot for the people who are writing line-of-business apps. It's like they're all dark-matter developers.

This is why I'm always frowning during job interviews with engineers telling stories about they migrations from monolith to microservices and putting it all into k8s, while their headcount in the firs tens and their DAU in about 10000 users.

Disclaimer: I worked in Yandex, virtually Russian Google. Loads and operations scale there are comparable with second rank international tech giants. Yandex has and have lots of internal tools to handle that load and scale, but I've almost never really needed them in smaller companies. Maybe "nice to have" couple of times.

The only reason the OP can boast about the experience with these proprietary tools is because they're used by FAANG. No other reason — it's just an indirect and inconspicuous boasting about having worked at FAANG.

If it was yet another proprietary non-OSS language used by one of the investment companies that has no direct analogues at any of their competitors — good luck boasting on your resume that you have skills in programming languages that noone cares about.

Proprietary software sucks. Just because it's written and used internally by FAANG doesn't make it suck any less. When you're evaluating whether or not to join a company, presence of such tools should be more of a red flag, if anything. Same goes with software-as-a-service. Tools should be OSS, and easily portable between the workplaces.

Another case in point — it kind of works both ways — many companies start around with Erlang and ejabberd, because it's a really great tool, but then, after having issues finding developers who can understand it properly, they eventually, one way or another, convert their stack back to the more common languages; I recall this has happened with Facebook Messenger and Heroku, and also at a random startup I was interviewing with during the course of my interview (they just hired a manager who knew Java — and, what do you know, he decides to set a direction to eventually replace all Erlang with Java).

Writing any type of UI in Swing is just painful - but this system made it so much nicer. It also had a WYSIWYG editor that you could interop with actually writing the code. Plus you could write either the 'markup language' or the "Java" flavour directly.

Making complex responsive UIs was a breeze. It's a shame it did not get open sourced - but it was tied into a great amount of internal tooling. (The entire codebase was 28+ million lines of code when I left).

It's one of the few non-open source libraries I've worked with that really impressed me.

Different companies have different problems. Big companies need big expensive tools to help with their problems. Little companies usually get by with smaller more portable tools. Let's not use this to demean people at small companies with some false sense of superiority please...

What takes months is feeling like you understand the most important tools and aspects of the environment to be as effective in them as those around you, or to match how you felt in an environment where the scope is smaller. It’s not like you’re sitting in classes for months - you’re doing your job (maybe even more effectively than normal due to the availability of these tools), but every once in a while you’re encountering something where you need to ask someone else or explore or watch a video about how to do something, and that might break your momentum.

Dont under estimate the profitability of a dev. I-make my employer three times as much as he pays to keep me, the industry really misses that.

Let them train the juniors, and solve their issues. Its worth it.

I think the main issue is fad-tooling. It is annoying and often not needed. Its also a complex issue really... what about saving educational costs ?

Say my devs want k8s. Dows my scale allow me to send them to confs, or do we go and shred our monilith? Maybe start a new project? Stuff like that. Its not an easy problem to solve.

That is, I expect the 5+ months is more how long it takes to find a good project and ramp up on it and deliver something of value while figuring out the social processes, not how long it takes to learn how to use the tools. I would not expect that developers just aren't writing code for their first four months.

For instance - I am generally under the impression that Google interns get something done (not necessarily a lot, but they do complete a project) in a 2-4 month internship, and they use the same tools as everyone else.

It is expected that a new employee takes that much to actually be part of company culture.

Isn't the point that they often don't need those tools? I've lost track of the number of "I replaced Hadoop with Excel!" blog posts I've read.

> but which didn't exist on the outside world.

> You're not allowed to include things you or your friends built

If you work for a service provider, you probably have zero to not much internal tooling, because you're usually solving the customer's problems, not your company's. When I did, we open sourced everything cool we built - sometimes tried and true defaults, or sensible Vagrantboxes or something like that. (Today that would be Dockerfiles, I guess) - and I consider myself lucky to have worked at a smaller company that was close to the FOSS scene, and not something like Accenture. Companies like ThoughtWorks might be an outlier here.

You're unlikely to build something so immensely cool in your first 1-2 years in a startup that it will change your live (as a developer) completely. (This is not about your company's product, but internal developer tooling, so if your startup does developer tools as a product, you're the 1% where this would apply).

And finally, if your startup grows.. you might probably still not be the next Google or Facebook where the internal team providing nice things to your fleet of developers is bigger than some companies. Internal tooling will often be seen more of a cost factor than a tool for productivity enhancement. And often it's actually the case. In my current company we just have so many software development teams creating parts of our products.. so many languages (with good reason) - the only internal things I could think about that would even make sense for all of them would be some code review tool, or anything targeting source control or maybe CI. Anything else would have to be really workflow-centric and language-agnostic. Anything regarding the mentioned borg/k8s/BigTable would be completely useless, for example.

That said, at my last company, I think we created some pretty cool things that made a lot of things a lot easier for the team. But we were two small teams, so point to "we built it ourselves, for ourselves" applies. Also it probably wasn't so revolutionary in the greater sense, but everything made sense for that company and I'm often still sad if I have to solve some similar problem these days and don't have my old setup (more than a specific tool).

Those were nice tools which worked quite well most of the time. They worked in any browser on any OS, unlike some of the ActiveX-only stuff they tried to replace it with.

This is not the Rackspace you probably know. This is the teeny little one that nobody had heard of outside of a few advertisements in Linux Journal (and the occasional rant about spammers and malware hosting). This is them back before the whole cloud thing - over fifteen years ago.

I've not had so many jobs at product companies, but I've never seen this even when I was doing consulting-type work and came around a bit, and many companies (at least here in Germany) being decades old are simply not digital-first. Maybe the mechanical engineers building the (non-computer) hardware have enough clout, but software is really often not more than a cost factor. I guess the bigger and older the company the more likely that everyone hates internal tooling, and you were focusing on the exceptions, whereas my life was the opposite. Waiting for an internal server just to deploy your shiny new thing could take months.

There's a reason to hire specialists in this industry, but we conflate one-track ponies with specialists and it's to our detriment.

Perhaps it’s true that most glue is poorly written, but doesn’t that imply that writing good glue is, in fact, difficult, and therefore it’s as hard — or maybe harder - than building “real” things?

Basically: structured log events with dozens or hundreds of columns, no write time aggregation, and arbitrary folding and slicing of data for investigation and debugging.

The reason Facebook and Google built their own is that they're at such a huge scale both in terms of usage and number of engineers, that none of the off-the-shelf things work for them. For most companies that's not the case.

Having said that, I do miss the data that’s available at FAANG companies (at least ones I’ve been at). I could dig up all sorts of system and business data, and really get good insight into both what my application and the users are doing. Tends to lead to a lot better and more clear decisions and thus helps to build the right thing.

That being said, I think most of the things she mentions are not customer facing but still pretty differentiating to the businesses.

Another perspective is that the first crop of truly successful Internet companies were in the position to solve the big backend problems with running big Internet companies in-house. Nowadays getting to the scale while building some fundamental in-house stuff makes less economic sense (you'd lose to someone who doesn't). It's probably better to have some unique technology but more closely related to the specific business.

These days, I very rarely enjoy using tools that aren't open source, or at least generic and very widely available. I simply don't want to spend my limited brain cells learning a piece of software that I won't be able to use in the future.

And in any case, large, internally developed tools are usually awful. Unless I wrote them. ;-)

For every piece of infrastructure or developer tooling you need, you have to make a buy vs. build decision based on cost/benefit analysis. Your cost to "buy" needs to include the upfront cost of developer time to configure the software for your needs, and the ongoing support cost to make sure it continues to work. When you include this, even open source software isn't "free", although it may be cheap. Your cost to "build" includes the upfront engineering time to design & code, ongoing support time to keep it working, and opportunity costs incurred because until the system is actually ready, no one can use it, and they have to either meet their needs in some other way or simply block until the tool is complete.

Very large engineering organizations like Facebook, Netflix, etc. can amortize the cost of tool and infrastructure development over thousands of developers and an enormous infrastructure budget. If a tool would make most of Facebook's developer's 3% more efficient, than it's worth staffing a team to build that tool. If better caching infrastructure could reduce Netflix's network usage by 3%, then it's worth staffing a team to build that infrastructure.

If you're a small- or medium-sized business, then that math doesn't pencil out. A 3% (or even 10%) developer efficiency improvement doesn't even pay for the opportunity cost of one developer. A 10% decrease in a single dimension of your AWS bill probably doesn't pay for the developer time either.

If you're at a mega-corp with a huge engineering staff: Yes, absolutely invest in tools that make your engineers more productive, especially if you've identified truly unique needs that can't be met by off-the-shelf software.

If you're at a smaller business: Focus your developer energy on your user-facing problems, and almost always prefer to "buy" instead of "build" (where "buy" includes using open source software).

As such, my guiding principle has been to consider the layer at which the thing you’re trying to build/buy fits. For most companies it doesn’t make much sense to build data centers and any cloud vendor will do just fine. Infrastructure is therefore somewhat easier... almost always buy. But if your product requires some kind of integration that is critical for customers, I would think hard about it before buying.

The costs of integrating a bought tool are also overlooked in most of these conversations. External tools are marketed well but don’t always deliver on their promises of being easy to use or setup.

Fundamentally I genuinely believe that if your product is tech heavy you absolutely need a CTO or a founding engineer who has a solid experience in building things. There is no replacement or magic pill to replace that just yet. And still, many fall into this trap.

Exactly, for a lot of problem domains, the cost of integrating a third party solution is nearly equal to the cost of writing from scratch, because the hard part isn’t the problem space but the environment-specific configuration.

> If you're at a mega-corp with a huge engineering staff:

Netflix is an odd choice for number two on a list of two. I’d guess it has an order of magnitude fewer engineers than Facebook.

So what’s the rough cutoff here? 1000 software engineers?

What if the only tech that solves the problem is Oracle, and your business model can't cope with the costs that would incur?

(I have a specific example in mind but it's private. Poor evidence for an argument; but I'll stand by the fact that some business problems require too much glue with too many moving parts, and can admit simpler, more vertically integrated solutions whose maintenance is still within reach of your average dev.)

When doing the math: A common mistake is to calculate the cost of "build" as SUM(compensation of developer team on the project) over the time it would take to build the thing. You should actually use SUM(business value created by developers working on the next-highest impact items on your roadmap). In a well-run business the latter should be much higher than the former.

If it's not a particularly complex problem, but not one that other businesses broadcast how they are solving it, it's quite possible you can solve it less expensively than finding out how other businesses are solving it.

(And, no, finding out how solution vendors tell you other businesses are handling it isn't the same as finding out how other businesses are handling it, since such a presentation, while readily available, will always distort the facts to serve the sales interests of the vendor.)

How does one do that? Do other businesses just tell you how they integrated? I remain highly skeptical of this. Bigger corporations especially are quick to tout their successes but hide their failures (publicly) which makes it hard to know how happy/sad these businesses are with the decisions they made.

(I'd end up writing an essay if I was to fill this out to full detail.)

The point is to only build things that are necessary for your product, not to avoid building anything.

Or perhaps that we've mostly worked in crap dev jobs/companies? There are easily enough of them around to encompass a career.

I have a friend who worked in the TensorFlow team and still quit to pursue a PhD. In his words "all engineering jobs are boring" from his perspective. I don't think being in a BigCorp or not makes much difference at all.

Having someone/a team to ensure network uptime is fantastic and that have the scale and experience to anticipate and mitigate potential trouble before it comes up. Likewise all infra in general. Having a legal department to ask "Can I do this?" where often the answer may be "We don't know" but that's still an answer. A security force, property, accounting, providing peace of mind. Someone making sure the projector's setup, tiny things that add up quickly.

There are downsides to this, such 'organisation' often grinds things to a glacial pace, hence why I'm breaking out by myself now, but... not tooling, I can take care of that myself and I'm incredibly thankful for the SaaS ecosystem that would have made this impossible even a decade ago, or at least very inconvenient with 3rd party consultants providing such services for high fees and little more. There are a lot of things an organisation provides that we can take for granted, and having that experience provides a great bonus in expectations of robustness not only in technology but in all processes of a business. Tooling is, in the end, in the mind.

Sure, sometimes paying for a hosted solution for your CI, deployments, container platform, data pipeline, etc. isn't the most ideal route, depending on your business, but I'd much rather focus on building actual product over reinventing the deployment system wheel for the Nth time.

> People from Google probably miss Borg (and lament the whole Kubernetes thing) or BigTable.

I've never worked at Google, and thus have no experience with Borg, but Kubernetes is exactly what I want to see companies open-sourcing, or, if they must, selling. I don't see what there is to lament. Being able to start a new company and not have to reinvent and rebuild your own deployment/container platform is a huge gift.

For the most part I find this blog to be insightful and fun to read, but this post IMO completely misses the mark. Given that I think the author works a lot on the operations side of the house, I guess I'm not surprised. Internal platform tools are there to serve the needs of the product teams; they don't exist just to make someone feel good about their ability to build something instead of buying instead, which she unfairly looks down on so disdainfully, as if it's not real engineering. Part of doing engineering is putting your ego aside when it's more cost-effective and robust to use someone else's solution instead of rolling your own.

https://nshipster.com/bug-reporting/

As for Prometheus, I like the rules engine VERY much (unsurprising, since they basically reimplemented borgmon, there). I'd take Viceroy any day over Grafana, though.

You should build your own tools, that you bring with you, that is what all artisans have done for all time.

No reason to change that just because you can use the newest buss-word every week.

I never been attached to work tools, or technology fads, things are ephemeral, best practices of today are the legacy codebase that no one wants to work on tomorrow.

The world of open source tooling and easily re-usable SAAS offerings means everyone has access to the best tools, whether you’re a small startup or a big company.

Anyone who longs for internal, corporate tooling baffles me when they can use things that actually have polish, user experience and likely better implementations under the hood.

Companies should spend their time/energy building things unique to their problem domain, not weak also-ran corporate tooling.

There are reasons that make them the best for many, if not most, companies: more investment, more mindshare, easier to hire employees with prior experience, and so on.

But there are also costs in having a wobbly stack of glued together stuff, especially if the parts aren't quite right for your goal.

Sometimes the best tool is more focused, more vertically integrated, in a different language or for a different operating system because those choices integrate better with the rest of your stuff.

The constraints of the company are also part of the problem domain. Using the wrong tools can be quite expensive, and the public tools may all be just a bit wrong in a way that compounds.

Frankly, it sucks. Our tools are mostly very good, but it took us a long time to get there, and the internal fights over getting funding to really invest in our internal platform have been exhausting for all involved. I get that it's not zero work or zero time to use something off the shelf, but as someone who has been playing in the grass on the other side of that particular fence, it takes a lot of work to keep that grass green.

The build tool was multi-language. It allowed depending on arbitrary packages which had been imported to the Amazon package repository. It allowed package owners to annotate the packages with guidance - experimental, deprecated, forbidden (in the case of security issues), etc. You could also declare conflicts, which would notify consumers at compiletime and force them to resolve in some way. The tool deferred to a number of standard build tools in whatever language(s) you were using; it was just about getting and packaging the dependencies.

When you committed code, a build was submitted to a distributed build system. It would run your build, and then it would run the builds of every package which declared a dependency on your version. If those builds failed, your build failed (so, bump your version or make your change backwards compatible). On completion, it imported an immutable bundle of your artifact + dependencies to the deployment infrastructure.

This is the part I miss - the build stuff was great but I largely find that open source and paid options aren't so bad here. What I miss is how easy it was to manage the journey of a built package to your machines. They had a tool called Pipelines that had a visualization of this progression. Each stage, with associated machines, was linked here. The tool knew how to add stages of environments, each with their own set of deployment configurations. You could set up approval workflows: integration tests, manual approvals, etc. You could feed one pipeline into another. For each artifact, you could configure autobuilds into your pipeline, so that new versions flowed as long as tests/approvals allowed. There was support for sanity checks: if those failed, the tool would automatically rollback. In fact, if any stage fails, your pipeline would block, and you'd be notified. In some cases, a newer build that was fully functional could unblock your pipeline.

Pipelines was a pleasure to use - it really just got the hell out of my way, and nothing I've used since is as simple to integrate with.

And tying this all together, there was a tool that would allow you to initialize the end to end infrastructural pieces needed. You go to a wizard, tell it your language and purpose (webapp, CLI tool, service) and it initializes a repository, package, and pipeline with environments. Get a coffee, come back, and Do Your Job.

I imagine similar things exist at places like Google, but man everywhere else I've worked, so much developer productivity has been lost not even approximating the level of "let me do the interesting stuff" that Amazon provided.

A previous employer had an all-proprietary stack: language, IDE, version control, deployment system, database, job scheduler. It was, frankly, amazing. The most productive environment I’ve ever worked in. Time-to-market was this company’s competitive advantage and this stack let them leave their competitors in the dust, scrabbling for a distant second place.

everyone has access to the best tools,

It’s 5 years since I left there and the FOSS tooling we use at my present company doesn’t come close, and it never will, because choices are driven from the basic assumption that FOSS is always best, and it just isn’t true. We are at least 5 years behind where the previous company was when I left it and they won’t have stood still in that time.

I'd rather have engineers working on products that customers would miss.

The view that such work just constitutes writing "glue" is super reductive. So much time is also spent empathizing with others: talking to stakeholders, understanding existing processes, determining requirements, and simplifying. So if we can reduce the solution to glue instead of needing to write a complicated system, that's hands down a win.

The worst engineers are not those who just glue stuff together, it's the people who cv-drivenly develop 100x too slow, resource intensive and broken buzzword bingo clusterfucks that could be solved, correctly and cheaply, with a page of shell script.

If you go through the build/buy/partner decision, and decide to buy or partner with development comprising of glue code - then that's probably the optimal solution for your org. And you made the right engineering choice.

Of course less code is good, all things being equal.

But taking that idea literally, basically no tooling code gets ever written and people develop some sort of learned helplessness around dysfunctional workflows that are just about feasible but fantastically wasteful of engineering time. This tends to happen a lot, also because tooling is often perceived by management as an unnecessary luxury.

The tech stack is full of OSS or industry standard tools, not a bunch of internal NIH things.

My current stack is: React, Node, Kubernetes, Linux (for dev and deploy), Git for sc, GitHub, vs code.

I can’t even think of some internal tool that I depend on at the moment.

You can easily put GitHub on your resume, and everyone would know what you're talking about. Of course, by itself, it's not much of an experience, but, surprisingly, it is often listed as a requirement and/or nice-to-have for certain SWE job posts.