lwn.net
Back to storiesRanked by HN
Comments
63 total · 63 loaded
Until a few years back most of my systems used Linux From Scratch. It's amazing for learning your system and how all its libraries integrate. I created my own boot scripts, my own pacman package manager repository, etc. One of the main reasons I stopped using LFS was the amount of work required for updates.
Firefox is bad to compile yourself, because it's slow and often requires a library version you don't have, such as a new ICU version. Updating ICU also requires you to build packages that depend on it: harfbuzz, ffmpeg, qt5, poppler and the list goes on.
Chromium takes it to a whole new level. While Linux packages typically have an automake flow (configure, make, make install) Chromium requires ninja, not just ninja but a specific version depending on the Chromium version. Then you need a ton of libraries with specific versions (or custom patches) or use the built-in versions provided by chromium, which may or may not compile at all. Then there is clang, while most software is compiler agnostic, Chromium does not compile with gcc. Of course, it needs a specific clang release, sometimes even a pre-release, or again a load of patches to compile. It seems the chromium build process is mostly built by engineers desperate to reinvent the wheel. While everything can work fine with sane Linux defaults (GCC, automake, system libraries) they convinced themselves their software is so complex and unique it requires a special treatment.
Chromium looks like a build system for a complex integrated project where the engineers have one priority: make the application work as best it can. You will quickly find that when you focus on that then making the build system convenient, especially convenient for distribution packagers with their myriad of own rules and modifications that they want to make on top of your system, is quite a big blocker to the primary goal. You can see similar things with applications like OBS, where they bundle modified libraries to fix problems and then distributions unbundle them and release broken packages, causing no end of grief for users and the developers.
The Chromium build process is just a consequence of the monorepo mindset. For Google, depending on specific versions of tools and dependencies is not a problem because all their engineers will automatically get those versions. For the Linux distribution model it's a pain.
> Linux defaults (GCC, automake, system libraries)
Last time I tried Gentoo desktop I ended up with 8 separate autotools versions, and gcc & glibc was not that far behind in count. And just like in https://xkcd.com/927/ every C developer tries to improve the situation by creating another build system.
> Last time I tried Gentoo desktop I ended up with 8 separate autotools versions, and gcc & glibc was not that far behind in count.
When was this?
AFAIK, you simply can't end up with more than one glibc version as that package hasn't had multiple Slots. [0] GCC is Slotted but I can't remember a time when there were more than three or four Slots defined for it. IME, it's rare for a package to declare an upper bound on GCC version, but it does happen. It's definitely true that there are many Slots for the various autotools tools.
[0] A "Slot" defines a particular subset of package versions that can be installed alongside other "Slots" of the same package. For example, every sys-kernel/gentoo-sources version has its own Slot, every minor version of dev-lang/python [1] has its own Slot, etc, etc, etc.
[1] ...that is, 3.9, 3.10, 3.11...
This is also why I stopped using Gentoo. I used it regularly from 2004-2005 (which I admit was a very long time ago now) on my graphite G4, because support was very good for PowerPC. But then one day emerge suggested I update glibc. I don’t know if I botched the update or if it really was suggested in error, but I was just starting my studies in CS so I did not know what that particular “library” did and said yes. The update must have included an ABI change. Not surprisingly, the machine did not come back up after a reboot.
Gentoo was fun while it lasted (fluxbox user here!). I decided that MacOS X wasn’t so bad after all after that.
gentoo's emerge is awesome. I keep a gentoo vm just to compile firefox, because it is impossible any other way. even though i don't use gentoo overall.
the comments trying to defens chrome (and old OBS) are completely corporate shills or missing the point. yeah obs at first did that because they had to, since it was such a small and niche project the devs had no pull upstream. after distros did all the unbundling job and helped them move patches upstream now every software benefit from those libraries fixes.
Google took chromium et al from open source and actively moved in the wrong direction, while having all the pull and influence like no open source project will ever have! its malice and/or incompetence. if you don't see that you're either shillings or lying to yourself. or have no knowledge of what open source is about.
> Firefox is bad to compile yourself, because it's slow and often requires a library version you don't have, such as a new ICU version. Updating ICU also requires you to build packages that depend on it
> Chromium does not compile with gcc. Of course, it needs a specific clang release, sometimes even a pre-release, or again a load of patches to compile
And that is what is wrong with today software: it is an uncompilable mess. It is "This program requires Windows 98 and Internet Explorer 4" brought to a new level.
Software has become balkanized.
There is not some arbitrary notion of wheel reinvention just for browsers. Calling browsers 'just browsers' is doing their feature set a huge disservice: they are bona fide operating systems or virtual machines, which have to load and run untrusted, unverified data and code within critical system libraries (many of which with kernel-mode access, such as GPU drivers, sound drivers, USB subsystems, and more), and do a huge amount of sandboxing to guard against escalations.
Within Google, a Chromium build uses remote build execution by default, upstream Clang by default (for the latest features and fixes, and Chromium developers regularly contribute to LLVM), and needs to cross-compile easily for at least a dozen different OS-architecture triples (Linux x86-64, Linux ARM, macOS, Windows x86-64, Windows on ARM, iOS, Android x86-64, Android ARM, Android TV, some BSDs, etc).
Everything you've listed—from ninja, to Clang, to libraries with custom patches, to Clang releases, including requiring pre-releases sometimes—is optimised for Google's scale, rather than the lone warrior developer. What you call 'sane Linux defaults' (especially the examples you have listed: GCC, automake, so-called 'system libraries') are highly antagonistic to all of the above.
They may make life easy for the so-called 'default case' of desktop Linux (arguable; Chromium was first released on Windows, and secondly since desktop Linux's most common C library, glibc has an unstable ABI, making back-portability impossible), but make it incredibly hard to get out a project of such immense complexity. All of Chromium's tooling and 'complexity' was developed in response to the 'sane' defaults being not so sane after all.
To get a GCC cross-compile for a given set of architecture triples, one needs a complete graph of GCC toolchains targeting every pair of host and target. On the other hand Clang is natively a cross-compiler; pass in `--sysroot` and the host compiler will redirect for the target instantly. Automake (and the autotools in general) is slow as molasses compared to ninja. It is based on computing and compilation from the 1970s, and their defaults probe for long-since-dead string comparison functions that are expected to work by default nowadays on most hosted systems. Not to mention it has an extraordinarily complicated configure step[1], and an equally appalling macro-based configure language.
Ninja is purely declarative, and is not even meant to be human-readable, as it is a back-end for a variety of build generator tools (CMake, XMake, build2, Bazel (Blaze internally at Google), Meson, probably more). Google have in fact rewritten Ninja to be a remote-build-capable tool from the outset[2], and they call it Siso. Siso has more than halved my Android build times even compared to Ninja because of its incredible use of RBE.
Frankly I think it should be evident why they have opted for 'rewriting the world' including vendoring and modifying system libraries, as well as including efforts like Skia, ANGLE, and more: directly relying on 'system libraries' is painful. I mentioned GPU drivers. Imagine rewriting the same code over and over to have precisely the same behaviour on D3D, Vulkan, OpenGL, Metal, and every single vendor GPU driver on every single system (and there are a lot more on mobile than on desktop: ARM Mali, Qualcomm Adreno, Samsung Xclipse, Imagination PowerVR, Huawei, MediaTek; this is not even close to exhaustive, just look at the Khronos vendor list).
When comments like yours are made I wonder if there is ever a consideration beyond 'it makes the developer's life hard'. As developers, our ultimate target audience are the users of our software. We should be willing to do anything come hell or high water, even making the dev-ex extraordinarily convoluted, if it means the user experience will be the smoothest and fastest possible. Sure, there is a balance, but I believe it should be strongly skewed towards the user story first.
[1]: https://upload.wikimedia.org/wikipedia/commons/8/84/Autoconf...
[2]: https://chromium.googlesource.com/build/+/refs/heads/main/si...
>glibc has an unstable ABI, making back-portability impossible
Of all the libraries to call out for this, glibc is not the one. It's basically the only library on linux that actually has a strong commitment to providing a stable ABI, with its own symbol versioning system to allow it. It is a bit of a PITA to compile for an older version of that ABI, but it's very possible to be compatible with glibc across a wide range of versions and distributions (I don't know why there isn't a flag to do it: the most widely supported option is compiling on a system with the oldest glibc you want to support, and there's some header hacks which claim to be able to avoid that).
It's everything else necessary to interact with the system (especially anything desktop/multimedia) that's wildly unstable.
Fair response.
I singled it out because I've been badly burnt by it in the past, and I dislike how it works compared to Win32. Back-targeting old glibc from newer (a very common case because developers tend to have newer distros than their users), requires either changes in source code (not build code, which is where it ought to be) to explicitly pin versions, or the ridiculousness of firing up an entire Docker container just for old glibc. It has regularly caused issues for things like Steam on Linux.
If you build on random developer's machines rather than a unified build environment you've already failed at so many things that the glibc problem is certainly not your biggest concern.
All I can say is that Gentoo specifically forbids downgrading glibc. I think it's the only package where they did this. [1] The .ebuild is full of patches and workarounds.
[1] See line 873. https://github.com/gentoo/gentoo/blob/master/sys-libs/glibc/...
Yeah, that's forwards-compatibility, which is different. Something built targeting a newer version of glibc is unlikely to work with an older one. (and gentoo will by default build against the latest you have installed, so yeah, downgrading has a good chance of breaking near everything).
Fair enough for poor Google, if only they did not strong-armed Chrome on everyone's computer by leveraging their default search engine/home page position, their ads and pop-under "Try Chrome" that were plastered everywhere 15 years ago or so.
Big company does big company things like cross-category marketing for a new product.
This is not really relevant within the discussion of Chromium's build system.
It's relevant because Chrome market dominance still has a very strong influence on websites working (tested?) only with Chrome. Add to that Google's NIH syndrome and its "move fast" mantra at the cost of interoperability with the wider Linux community and then both things are relevant.
> which have to load and run untrusted, unverified data and code within critical system libraries (many of which with kernel-mode access, such as GPU drivers, sound drivers, USB subsystems, and more),
No, they don't have to. The fact that someone decided that a browser shall run remote code (JS) is sad.
I tried Gentoo some 20 years ago or so for fun at home. I guess my naive thinking was that software specifically compiled for my hardware would run noticeably faster and "better".
But I didn't last for very long. I remember starting the install for a bigger app before going to bed, only to find the thing still building when I woke up. And as for the dream of more tailored builds, I never really noticed anything much different - I probably was too uneducated about build configurations, or - more likely - the differences just didn't matter too much.
Eventually, I switched to Ubuntu and never looked back.
So, I'm curious - is anyone using Gentoo as their daily driver, and what is the main incentive for you?
I'm using it as my main OS. I have it installed on 4 machines: a RK3288 SBC, a RK3588 desktop, a x86-64 desktop and a x86-64 laptop. Main incentive for me is customization.
- It makes it very very easy to patch any package before they're installed - just put the patches in /etc/portage/patches/<pkg name>/, and that includes the kernel too.
- I can exclude package features that I don't like. For example, my Firefox doesn't try/offer to turn on the wifi.
- I can also exclude entire packages that I don't like, because the entire dependency chains can be eliminated by disabling USE flags. For example, none of my systems have systemd or dbus. Since last year I don't even have openrc anymore (the other Gentoo service manager). One of them doesn't even have udev. I tweaked it so much, I'm not even sure I can still call it Gentoo.
Daily Gentoo desktop user since 2007. Compilations/ricing is one thing, the variety and flexibility of the portage and USE flag system is another. For me it just makes sense to stay with this proven distro. I use Ubuntu variants on my work PC and they just have all this crap that I don't have to tolerate on my gentoo installs.
P.S.: I've got to admit, though, that I was always impressed by the really great quality of Gentoo's documentation!
My understanding is that compiler flags yielding speed boosts where an actual thing back in the day, but not anymore really.
Nowadays the speed come from other things. For example a few years ago i realised my firefox on linux was doing video encoding/decoding on cpu without using the intel gpu’s hardware encoder/decoder. I did not have to rebuild firefox from scratch but rather fiddle with configurations and environment variables to make sure Firefox wouldn’t assume the lowest common denominator.
That saved battery on my laptop and yielded better performance.
I used it on an off as my daily machine for around 15-20 years I think. It is fast in terms of execution-on-hardware speed, and genuinely puts you in control. For a certain kind of user this is more important than anything.
These days I'm on OSX mostly because of the frequency of business trips meant I needed something small and portable a few years ago, and now the hardware is so much better than what you can get for the PC world. Plus, I am hyper-conscious of my time and can't really afford to be fiddling with new system libraries, kernel configurations and so forth.
Frankly Gentoo was never polished, at some point the system state would break and you'd have to reinstall, and after enough iterations I'd started to err to Ubuntu at some point. But it was by far the most fun of any Linux I used, save perhaps Slackware which was my first.
I think in a few years' time the next machine I build will be a non-Apple ARM custom portable, hopefully running Gentoo, with a lot of custom machining and hardware features.
> at some point the system state would break and you'd have to reinstall
My advice is to start using btrfs snapshots. Make a backup snapshot, run the updates, and if anything breaks, just "btrfs set-default" to the backup snapshot and everything is working again.
> Frankly Gentoo was never polished, at some point the system state would break and you'd have to reinstall
Not at all. I have even recovered from bricking the package manager (my fault, these days there are protections in place against that). Worst case you manually extract working binaries and then clean up the mess. And I have never run into issues that needed such drastic measures without doing stupid things.
It is however not a distro for those unwilling to understand how it works.
> It is fast in terms of execution-on-hardware speed, and genuinely puts you in control.
After installing Gentoo I felt I should make some kind of change, and I wanted to replace the GNOME foot in the GNOME menu with an icon I found of the Gentoo "g" in a wizard hat.
I found documentation suggesting that the menu's icon was controlled by some setting somewhere, and I changed that setting to point to the wizard-hat icon.
The menu was unaffected. If I viewed the menu bar's properties, the menu was associated with the wizard-hat icon in the properties display, but it continued to show the foot icon in its name.
I had nothing else to go on and gave up. I'm still mad about that.
> Frankly Gentoo was never polished, at some point the system state would break and you'd have to reinstall
Well, that did happen to me once. But it happened after I uninstalled glibc. I can't really blame Gentoo for that. I always saw it as positive that Gentoo allowed me to do something that ridiculous.
I remember triggering my third recompile of kde-base because I wanted a gui calculator. I moved away not long after that.
> Measures need to be taken by somebody qualified that this doesn't keep happening in packages as important as these.
To the single guy maintaining it, and no offer of help. Crazy entitlement.
The irony being that ChromeOS is actually Gentoo under the hood
I always wondered how long it would last tbh.
I've always used source-based packages, but chromium was the one I simply dropped in favour of Firefox.
On a 16c32t / 64G machine, compiling Firefox takes ~1h with LTO and PGO, while Chromium already took 5+h in 2019 without PGO. Then they removed jumbo-build because there were some issues and kept adding code and toolchain deps (jumbo-build is a build step that merges a few hundred source files before compiling to do the header processing once, shrinking the build from 5h+ to ~1-2h). Last time I checked the build had 70k+ build steps, with the final link step easily taking 10m+ with LLD. Answering some other comments: ccache doesn't help, because every version will have some minor header changes somewhere, so basically the entire cache gets invalidated even within the same major (e.g. 151.44.22.11 to 151.44.23.28).
As an ex-Gentoo user, the pain in a was always bundled Vs shared libraries.
Shared libraries breaking ABI could mean recompiling half your operating system
On the other hand, bundled libraries meant insanely long build times and a sense of impurity/uncleanliness
Unfortunately, it looks like projects like ABI tracker are long dead:
Compiling Chromium took absolutely forever 5+ years ago. I think a whole system took like 20% of the time that Chromium would take to compile. I can't even imagine with rust extensions added on top now. Gentoo was so awesome though but I had to switch to debian because I would just play with my configuration all the time and break things. One day I hope to return to using it though or setup another machine to use it.
So basically they'd be new CVE's appearing by the time someone built the previous release at this pace? :P
I was ok with Chromium alone taking ~6h. I moved away when I started experimenting with Rust.
No ccache?
Chromium is a pyramid of components and the higher you go up, the longer the compilation of single file takes, because they depend on so many things. This makes caching difficult. A small change at the bottom can invalidate many files, especially the heavy ones at the top. The build system ninja already does caching, but it only offers limited benefits. Then there is so much activity that if you plan to do a weekly build, you can do it from scratch.
ccache for portage is broken since at least 2010. Because each package source gets extracted and compiled in it's own versioned directory, with each new version the cache is invalidated by the different path.
ccache is not very effective accross different versions of software as it only takes one small change in a header that's included everywhere to make the cache completely useless.
Am I the only one who noticed just how entitled this Žilka person is? If they are so good at maintenance maybe they should take it over and learn the hard way that things like this are not at all remotely simple to pull off and that maintainers are (usually) unpaid volunteers who have their own lives (and in some cases, they maintain many other packages).
Edit: I should note that it definitely looks like entitlement based on how the article presents it, and doesn't at all seem like this person is taking into consideration all the maintenance policies/rules/whatnot.
Very related and interesting discussion that got posted in the comments about the new Crubit thingie: https://groups.google.com/a/chromium.org/g/embedder-dev/c/W_...
This is what we get when we go all in on a monolithic bloated platform, and then let its direction be primarily determined by one company. We'd be better off with a simpler, albeit less featureful, platform, or at least one that's more modular.
why don't they leverage ai for maintaining packages?
Maintaining Gentoo ebuilds for complex packages such as Chromium is as peak Linux as one can get. You're talking about developers who _often_ discover and then submit upstream patches for the most obscure and difficult bugs that upstream projects haven't heard of. Often because Gentoo users are the only ones trying different compilation options, or they're the first to try building with more modern dependencies than the upstream project has yet attempted.
For just one of many examples, see [1] where the Gentoo developer had to submit a patch upstream for a Chromium build toolchain bug that to 99% of experienced Linux users is bordering on unintelligible due to how much specific knowledge is needed to understand the bug.
The main problems seem to be that it takes a long time to build all combinations of USE flags and that users complain that volunteers are not working fast enough, causing annoyance of the maintainers. Can you explain how AI would help here?
ai can work test the options and fix the builds
You still need the hardware resources to do builds. And thanks to Slop Generator providers gobbling up alllllllll the RAM and storage they can get their icky hands on, it simply practically infeasible for everyone not affiliated with a tech BigCo.
Most ai wouldn't know how to write a proper ebuild to save their lives.
That's funny cause I use ai to maintain dozens of ebuilds. They are usually one shots.
I tried some months ago, and it'll make a blank one fine, but if you need a complex configure or install phase, I felt I was better off doing it by hand.
Cool, you can adopt chromium then!
Problem is solved!