Google has objfs, a authenticed network mounted drive that is an object file cache. Usually most of the tree has already been compiles so just your change needs to get recompiled and relinked.
Further, there's then GOMA which is the distcc equivalent. I frequently build Android with 500 cores in the cloud. For that reason, I don't think Google engineers will ever focus on the compiler speed of llvm. That said, LLD (llvm's linker) is the fastest linker in town by far.
Nice info. I've wondered for quite a while about the impact of the G level of compile parallelization :). A couple of followup questions, if I may:
- How long do incremental and non-incremental builds take with the build configuration you describe?
- How does core<->engineer allocation work? Is 500 cores an average number? High-priority work presumably commands more access, but I'm curious if allocation is generally team-specific or engineer-specific (eg, more responsibilities = more resources). (I guess the reason I ask this is that 500 cores sounds kind of impressive and I'm going "surely they'll run out of cores if they're handing them out like candy???"), but then thinking about it that's only like 11 computers...)
- I'm curious if you happen to know how Chromium gets built (IIUC Android builds it 2 (or is it 3?) times). It's tricky to distill (presumably/understandably deliberately so) how many cores the CI/trybot/etc infra is using. Overall my curiosity is what the core/time ratios/graphs are, ie how many cores translates to how short of a build.
- As an aside, I vaguely recall absorbing by osmosis several years ago that Chromium takes approximately 15 minutes to build on a Very Cool™ local workstation (HP Z series). Not sure how out of date this info is. Do engineers still do test builds on local hardware (and if yes, how long does this take? probably longer than 500 cores, heh); or is everything done in throwaway cloud instances nowadays?
Thanks in advance for whatever answers you're happy to provide :)
Google has unlimited resources. Cores and codebase size don't matter to them. Only concern is what's on the critical path, e.g. final link step. It's very rare for builds to not be incremental. AFAIK JavaScript is the only language that can't be compiled incrementally. Probably the only engineers who would need to run a build solely on local hardware are ones that help support open source tools.
Js can and does get compiled incrementally if people do it correctly. It's fully supported by blaze. It's just that a bunch of old codebases globbed everything, so it appeared no incremental.
Is that with the js_binary/library rules or something new? Last time I checked, all the work that goes into compiling the JS sources and minifying them into a single "binary" blob happens in the js_binary definition, and the js_library definitions traditionally served only to construct a total ordering of raw source source file dependencies.
Two partial answers: there's a newer better thing thats now the default and splits the building, but even under old js_library, there were some gains from not globbing everything together...
Interesting. I searched for some more information on that but don't see anything except the GitHub repostoryitself. I wonder how would one go about integrating that into .Net stack.
Check out Bazel's remote execution feature and Google RBE (remote build execution) [1]. Bazel itself is language agnostic (language support comes in the form of rules).
There's also an opensource goma client + server which proxies to RBE (or potentially other bazel remote execution API implementations). Which is great if you can't or don't want to use the bazel client, but can use a compiler wrapper.
Further, there's then GOMA which is the distcc equivalent. I frequently build Android with 500 cores in the cloud. For that reason, I don't think Google engineers will ever focus on the compiler speed of llvm. That said, LLD (llvm's linker) is the fastest linker in town by far.