All 1,400 Google Chrome CLI flags(peter.sh)
peter.sh
All 1,400 Google Chrome CLI flags
https://peter.sh/experiments/chromium-command-line-switches/
168 comments
This is expected because it's their flag-ship product.
I cannot believe I'm upvoting this...
I hate you
--0 No description
--1 No description
--10000 No description
--100000 No description
--1000000 No description
--2 No description
--3d-display-mode[1] No description
--4 No description
--50000 No description
--500000 No description
--5000000 No description
--7 No description
--?
PR reviews on that team must go smoothly
Many of these flags are not intended to be used on their own. For example, you can use "./chrome --memlog-sampling-rate=10000" but not "./chrome --10000". This information is, understandably, not explicitly stated on this page.
There's only a single flag in `--memlog-sampling-rate=10000` though.
The docs say, uh, the exact opposite though.
--10000 is a single flag
--memlog-sampling-rate=10000 is uh, well, also a single flag
according to these docs both flags should be valid?
--10000 is a single flag
--memlog-sampling-rate=10000 is uh, well, also a single flag
according to these docs both flags should be valid?
AFAIK, only "--memlog-sampling-rate=10000" is valid. Neither standalone "--10000" nor "--memlog-sampling-rate --10000" is valid.
See the comment https://source.chromium.org/chromium/chromium/src/+/main:bas...
Also see how the flag is actually parsed: https://source.chromium.org/chromium/chromium/src/+/main:com...
See the comment https://source.chromium.org/chromium/chromium/src/+/main:bas...
Also see how the flag is actually parsed: https://source.chromium.org/chromium/chromium/src/+/main:com...
Since I was at Google 12 years, this doesn't surprise me at all. It's not that there are 1,400 flags in ONE module; it's that all the many systems it includes all have their own. No one looks over that list and says, "Hmm, which of these 1,400 do I need today?"
There's absolutely no reward for scouring the system and getting rid of the 90% of the flags that serve no purpose anymore. All you would do is annoy people and risk breaking something. So no one does it.
In fact, as I recall, you could either "list the flags" for a system, or "list all the flags AND all the included systems' flags." Hardly anyone wants to do the latter.
There's absolutely no reward for scouring the system and getting rid of the 90% of the flags that serve no purpose anymore. All you would do is annoy people and risk breaking something. So no one does it.
In fact, as I recall, you could either "list the flags" for a system, or "list all the flags AND all the included systems' flags." Hardly anyone wants to do the latter.
There is some incentive to clean up flags since there has been horror stories about dead flags causing problems[1].
I seem to recall there is some automation that proposes changes to remove flags that has not been set to non-default values for a sufficiently long time, but maybe that's not done for binaries that run outside of Google's network.
[1] Knight Capital's $440 million glitch:
https://news.ycombinator.com/item?id=31239033
https://news.ycombinator.com/item?id=4329101
I seem to recall there is some automation that proposes changes to remove flags that has not been set to non-default values for a sufficiently long time, but maybe that's not done for binaries that run outside of Google's network.
[1] Knight Capital's $440 million glitch:
https://news.ycombinator.com/item?id=31239033
https://news.ycombinator.com/item?id=4329101
That doesn’t really apply to software like Chrome modulo backup / restore accidentally confused but the risk of that depends on how that’s implemented (feels unlikely).
Note that a decent amount of these are ChromeOS specific. The whole visual UI is orchestrated by the "chrome" binary.
So Chrome's not just a browser but a window manager too?
You are close.
“It's an operating system (pretending to be a browser).” [1]
1: https://news.ycombinator.com/item?id=14737739 from a past discussion of building Chrome “24-core CPU and I can’t move my mouse”
“It's an operating system (pretending to be a browser).” [1]
1: https://news.ycombinator.com/item?id=14737739 from a past discussion of building Chrome “24-core CPU and I can’t move my mouse”
It depends on what we mean when we say "Chrome". When the chrome binary is built for ChromeOS, it includes a shell called "ash". It's got a window manager, virtual desks, app launcher, login screen and more. The fact that this all lives in a binary called "chrome" and uses a nontrivial amount of the flags that this post is about is in some ways a historical artifact. The source for it is here [1]. None of that stuff is included in a regular desktop (Windows, Mac, Linux) build of Chrome.
[1] https://source.chromium.org/chromium/chromium/src/+/main:ash
[1] https://source.chromium.org/chromium/chromium/src/+/main:ash
Yep. Web browsers are new Xorg(In more way then most people realize. I wonder at this point, we could make web browsers some X Wayland style daemon.
This is sorta how many "embedded" systems with a display work these days. Start a chromium window in kiosk mode at boot and use a webapp for the UI. You'd be surprised how many touchscreens out in the world do exactly this.
Yeah, that was kind of the point of them moving to Aura. More details on it: https://www.chromium.org/developers/design-documents/aura-de...
Browsers are de-facto windows managers in the unfortunate world we've created where everything is a webapp with 5% of the perf that it'd have as a native app.
Everything-as-a-web-app is not good.
However...
I didn't measure it, but switching tabs in an Asus C201 Chromebook sure seemed to have a much lower latency than tab switching (or clicking on a different app in a menubar) for any Linux distro I've ever used on any piece of hardware. That includes this Dell XPS laptop running Ubuntu, which is fast and well-supported with firmware.
I speculate without measuring because switching browser tabs on that Chromebook feels uncanny. It's as if there's some ahead-of-time trick that utilizes the camera to guess my impending trackpad click or keystroke. That makes me think I've gotten used to a fairly high floor of latency on my Linux machines.
Add to that the long battery life while browsing in Chrome. Add to that the fact that I open and close that Chromebook at will with the browser in nearly any state, and it never got hot, crashed, or failed to come back up when I reopened it[1].
Those things being true, it makes more sense from a performance standpoint to use Linux from within Chrome than the other way around. That is-- as long as you spend non-trivial time in a browser, it's more efficient to do Linux-Inside-Chrome than Chrome-Running-Inside-Linux. (And I have never found a Linux Firefox config on any piece of hardware that can reliably play videos on Reddit, which makes me think it's not handling video acceleration properly.)
1: The one exception: when I was running a low-latency audio app on that Chromebook which grabbed exclusive control of ALSA.
However...
I didn't measure it, but switching tabs in an Asus C201 Chromebook sure seemed to have a much lower latency than tab switching (or clicking on a different app in a menubar) for any Linux distro I've ever used on any piece of hardware. That includes this Dell XPS laptop running Ubuntu, which is fast and well-supported with firmware.
I speculate without measuring because switching browser tabs on that Chromebook feels uncanny. It's as if there's some ahead-of-time trick that utilizes the camera to guess my impending trackpad click or keystroke. That makes me think I've gotten used to a fairly high floor of latency on my Linux machines.
Add to that the long battery life while browsing in Chrome. Add to that the fact that I open and close that Chromebook at will with the browser in nearly any state, and it never got hot, crashed, or failed to come back up when I reopened it[1].
Those things being true, it makes more sense from a performance standpoint to use Linux from within Chrome than the other way around. That is-- as long as you spend non-trivial time in a browser, it's more efficient to do Linux-Inside-Chrome than Chrome-Running-Inside-Linux. (And I have never found a Linux Firefox config on any piece of hardware that can reliably play videos on Reddit, which makes me think it's not handling video acceleration properly.)
1: The one exception: when I was running a low-latency audio app on that Chromebook which grabbed exclusive control of ALSA.
The number of flags in a typical internal google binary can easily be 1 or even 2 orders of magnitude higher.
Didn’t Google have to patch their kernel to increase the maximum length of command line arguments, because the number of flags they actually use is so high?
Why would you do this vs. having config files?
/meme why not both!
Googler, opinions are my own.
Sometimes we use flags on my team for controlling experiments and whatnot, flags tend to be used for more static settings, like the name of other jobs we call.
We have a few configuration languages at Google, and many of them will push to production very quickly. We tend to use these for experiment controls.
The problem with flags as experiment controls, if you need to roll back quickly. It's a lot slower because you have to wait for all the jobs to restart.
Googler, opinions are my own.
Sometimes we use flags on my team for controlling experiments and whatnot, flags tend to be used for more static settings, like the name of other jobs we call.
We have a few configuration languages at Google, and many of them will push to production very quickly. We tend to use these for experiment controls.
The problem with flags as experiment controls, if you need to roll back quickly. It's a lot slower because you have to wait for all the jobs to restart.
Maybe it’s used differently at Google, for us experiments are something you roll out to a (sticky) subset of users matching certain criteria. So it cannot be merely data (flag or config), but a library function of user ID and whatever other metadata.
Depending on the type of app, it could also refresh it’s flags or have values pushed to it.
scripting/automation
I don't follow, is it easier to automate a ton of args than to automate a config file? Or better yet, put the config file in the bundle you're distributing, since it probably benefits from the same change management flow?
> I don't follow, is it easier to automate a ton of args than to automate a config file?
Yes? With just flags, the only artifact that you depend on is the program. With a config file, now there is a separate artifact to maintain for every distinct configuration, which could be a lot. I'm not saying its not solvable, but it does strike me as more complicated.
Yes? With just flags, the only artifact that you depend on is the program. With a config file, now there is a separate artifact to maintain for every distinct configuration, which could be a lot. I'm not saying its not solvable, but it does strike me as more complicated.
I guess if you're distributing standalone binaries, that makes sense. The distribution systems I have seen are all tarballs, packages, or container images, where one more file is no problem.
There must be some kind of config system where the flags are being driven from though, unless someone is manually typing them out each time.
There must be some kind of config system where the flags are being driven from though, unless someone is manually typing them out each time.
They could done what bwrap does where there is a special CMD flag that tell program to read the flags from FD. So in flatpak which generally has to give a few kbs of flag to bwrap just pass this via FD. Now whether this works on all supported platform is an another question in it self
On automation, if you are building the config dynamically from external or computed parameters, in the case of a config file you'll need to:
- open some descriptor on the system
- write the file content
- execute the command
- get rid of the file
Of course you'll also need to deal with the constraints/failure of any of these steps (write the file in a location the program can access, take care it still exists or is not overwritten when you run the command, so have pseudo random names and/or lock the file, etc.)
In comparison, passing it all as args is one single step.
- open some descriptor on the system
- write the file content
- execute the command
- get rid of the file
Of course you'll also need to deal with the constraints/failure of any of these steps (write the file in a location the program can access, take care it still exists or is not overwritten when you run the command, so have pseudo random names and/or lock the file, etc.)
In comparison, passing it all as args is one single step.
Fair. There are all those "configuration management" frameworks that make templating out config files from data a fairly solved problem, but if you're launching programs from other programs that you also wrote, I guess it's less code to have an arbitrarily long arguments list vs. also dealing with files.
[deleted]
Several drawbacks about config files:
+ Another artifact to ship and deploy
+ What file format to use? Obviously different teams will use different formats too. This compounds as you add engineers.
+ How do you find the initial file? Whichever mechanism you use besides hardcoding it is another gateway to configuring the program. It will be used in addition to the config file and compose in weird ways.
I think the limitations of the command line actually make it scalable especially in the Google context.
+ Another artifact to ship and deploy
+ What file format to use? Obviously different teams will use different formats too. This compounds as you add engineers.
+ How do you find the initial file? Whichever mechanism you use besides hardcoding it is another gateway to configuring the program. It will be used in addition to the config file and compose in weird ways.
I think the limitations of the command line actually make it scalable especially in the Google context.
Config files and flags should be equivalent and substitutable. There's no data difference between text-at-command-line and text-at-command-time.
At my current job I wrote a simple Python class to abstract settings being read from either command line args, environment variables or a TOML-formatted config file. The class first reads settings from the config file if it is available, then environment variables if found, and finally from the command line args if set by the user. The last context to provide a setting value is treated as the actual setting to propagate to the app. If desired the app can then save its current settings state back to a file, either the original config or to a new config file.
I arrived at this solution after getting tired of both having to track down where the value for a particular setting originated at runtime, as well as having multiple implementations of the same kinds of argparse/os.environ/file-based settings systems across our internal tools and scripts.
I arrived at this solution after getting tired of both having to track down where the value for a particular setting originated at runtime, as well as having multiple implementations of the same kinds of argparse/os.environ/file-based settings systems across our internal tools and scripts.
I just put configuration settings in config.py and import them.
Problem solved.
Plus I have the full programming language available, not some textual subset with Yet Another Dependency to read it.
Problem solved.
Plus I have the full programming language available, not some textual subset with Yet Another Dependency to read it.
Agreed. Tried to reconcile this for some of my projects a while back and spent much more time I anticipated. This is an unexpected rabbit hole, between type casting, scoping of flags to subcommands and flags > configuration file > environment variables > default values precedence. And TOML / YAML / JSON / XML / INI multi-format support.
I made my hacks open-source, so if you have a Click-based Python CLI, try click-extra, contribute back, and help me solves this problem once and for all: https://kdeldycke.github.io/click-extra/config.html
I made my hacks open-source, so if you have a Click-based Python CLI, try click-extra, contribute back, and help me solves this problem once and for all: https://kdeldycke.github.io/click-extra/config.html
What about when an option is present in both the file and the command line? Which one takes precedent? If an option occurs 2x on the command line typically the last one wins. If an option appears 2x in the file does the last one win too?
With restraint and thought you can make the 2 equivalent from a data perspective. That's a ton of effort and if you get it wrong it is pretty hard to change it later on because someone is going to start relying on previous semantics.
With restraint and thought you can make the 2 equivalent from a data perspective. That's a ton of effort and if you get it wrong it is pretty hard to change it later on because someone is going to start relying on previous semantics.
This problem has been solved for a long time now: http://www.catb.org/~esr/writings/taoup/html/ch10s02.html
Multiple decades of unix software configuration and pointless differences between distro configuration would seem to indicate otherwise.
Esr even contradicts himself at the end of that link when he says that site wide configs may be immutably set in the system directory.
Esr even contradicts himself at the end of that link when he says that site wide configs may be immutably set in the system directory.
I believe that the problem has been solved, it has just been poorly implemented in some cases. The general gist that more local configs takes precedence over more global configs is a simple and powerful mechanism which has worked. Yes, distros can be a mess where there are five different config directories but I believe that specific problem is orthogonal to how we solve config precedence.
It is even poorly implemented in such ubiquitous tools as ls. For example if I always want output to be sorted by last modified the _only_ point of configuration is the command line. But I can control colors via either the command line or an environment variable.
I've lost count of the numbers of times I've straced commands to see where they pull configuration from or grepped thru codebases looking for getenv calls. Any unix consensus for configuration is in theory only not practice.
File location is actually pretty germane to precedence. A more "global" configuration cannot be applied if the program is looking for it in the wrong location in the first place. Let's not even discuss the horrible state of where to store local configurations in a users home directory. As long as it starts with a . it's fine :).
I've used Linux exclusively for nearly 20 years - but not because I loved its configuration management story.
I've lost count of the numbers of times I've straced commands to see where they pull configuration from or grepped thru codebases looking for getenv calls. Any unix consensus for configuration is in theory only not practice.
File location is actually pretty germane to precedence. A more "global" configuration cannot be applied if the program is looking for it in the wrong location in the first place. Let's not even discuss the horrible state of where to store local configurations in a users home directory. As long as it starts with a . it's fine :).
I've used Linux exclusively for nearly 20 years - but not because I loved its configuration management story.
ls is a very poor example as it was a program built in v1 Unix over 50 years ago which has remained the same due to backwards compatibility and being standardized in POSIX.
We are arguing over whether the glass is half empty or half full. I can also come up with plenty of examples of poorly designed programs, but I can also bring up examples of programs which are not perfect but much more usable than ls. I’m sure you could have also picked a program which is better designed than ls as well.
We are arguing over whether the glass is half empty or half full. I can also come up with plenty of examples of poorly designed programs, but I can also bring up examples of programs which are not perfect but much more usable than ls. I’m sure you could have also picked a program which is better designed than ls as well.
Do you have any examples that aren’t confusing and super complicated?
Kind of related but not: if you are interested in cli design I recommend this guide as well: https://clig.dev/
I think people are overly focused on how crappy Unix tools from the previous millennia are (ls, sendmail, dd, etc…) but I do truly believe that cli tools can be ergonomic because I have used and built tools which feel ergonomic.
I think people are overly focused on how crappy Unix tools from the previous millennia are (ls, sendmail, dd, etc…) but I do truly believe that cli tools can be ergonomic because I have used and built tools which feel ergonomic.
I think the aws cli is a decent example. It isn’t perfect by any means but it demonstrates how to handle config files, environment variables, and command line arguments in a usable way: https://docs.aws.amazon.com/cli/latest/userguide/cli-configu...
In fact, some programs (mpv comes to mind) make them literally the same, with the config file as a text file containing command line options in the exact same format that you would provide from a shell, with the only difference being that newlines are permitted noop characters.
Command line flags are miserably at representing anything beyond key=value. A config file will easily handle a hierarchical tree 5 levels deep.
These are solvable problems, no?
Are they worth solving though?
Flags at command line time can be equivalent to flags in file from a data perspective.
If you work on it they can behave with similar semantics too.
If you work even harder you can ensure you deploy the config file atomically with the binary. You can also make sure that the file is properly secured, and config values from it are observable and auditable.
Or you can just mandate that programs are configured by the command line flags passed to them and not have to do all that extra work.
Flags at command line time can be equivalent to flags in file from a data perspective.
If you work on it they can behave with similar semantics too.
If you work even harder you can ensure you deploy the config file atomically with the binary. You can also make sure that the file is properly secured, and config values from it are observable and auditable.
Or you can just mandate that programs are configured by the command line flags passed to them and not have to do all that extra work.
IMO, the advantage of flags is in the fact that a human can copy/paste the command line invocation and get all they need to reproduce an execution. That’s great.
But when the command line invocation is so big that you can barely even copy/paste it any more, and it exceeds the max command line length on a stock Linux kernel, you’re way past the point of convenience being the important factor. All the normal places you’d want to see the flags (set +e in a bash script echoing them back to you, etc) are suddenly overloaded with megabytes of text for a command line invocation… who’s gonna look at all that? Who is this serving any more?
“Flags for everything” would seem like a good idea taken way too far to me.
But when the command line invocation is so big that you can barely even copy/paste it any more, and it exceeds the max command line length on a stock Linux kernel, you’re way past the point of convenience being the important factor. All the normal places you’d want to see the flags (set +e in a bash script echoing them back to you, etc) are suddenly overloaded with megabytes of text for a command line invocation… who’s gonna look at all that? Who is this serving any more?
“Flags for everything” would seem like a good idea taken way too far to me.
If you routinely have so many CLI flags that you have to save the CLI invocations to a file, then what you have is an ad-hoc, informally-specified, bug-ridden config file.
The config flags themselves are ad-hoc, informally-specified and bug-ridden in the first place, so what’s the big deal here?
So you can run `top` and see exactly what config the process is running under. If it was started with a config file, that file could’ve changed or disappeared since startup.
Huh, this is a totally legit reason, but the solution is so horrible it shouldn't justify it. Yet I cannot come up with a "proper" way to handle this. Is there any?
At my work, all services have a "debug" HTTP port with endpoints to, among other things, dump the running configuration. But I'll admit that's less general than flags, since it depends on the server processes being built from our standard frameworks.
Also for us, we can't SSH onto hosts, and the scheduler that actually knows about args/pids/etc is fairly deep behind the curtain, but HTTP access is easy. But you could see different companies having different constraints.
Also for us, we can't SSH onto hosts, and the scheduler that actually knows about args/pids/etc is fairly deep behind the curtain, but HTTP access is easy. But you could see different companies having different constraints.
A logging overlay that stores configs in a database and monitors each process. If you're testing at that scale, why would you limit yourself to what you can do with just top? To do it manually with flags and monitoring with top is very pets not cattle.
Put the config in a file and have a launcher program that turns it into flags, and then stop thinking about the flags.
That's a silly reason. At simplest, you could
a) log the entire config at startup.
or
b) create a specific file with config used, if logging is too messy.
or
c) make an API to return the running config.
or
d) use envvars in a container, that you can just dump from 'env' at will.
All of which would be cleaner and saner than browsing top.
a) log the entire config at startup.
or
b) create a specific file with config used, if logging is too messy.
or
c) make an API to return the running config.
or
d) use envvars in a container, that you can just dump from 'env' at will.
All of which would be cleaner and saner than browsing top.
[deleted]
If you want to stop the inevitable new chrome 0-days:
--js-flags="--jitless"
How do you set that to run on auto-run when the OS starts? Or in general, when Chrome is launched in some other way than you typing that on the command-line?
At least on Windows, you should be able to modify the target property of the program’s shortcut to add any desired flags. Then use that shortcut to launch, or drop it in the startup folder.
On Linux idk, maybe an alias?
What shortcut though? Auto-launch is automatically set by Chrome in the registry, and Chrome really doesn't like you to mess with things it manages in general, and would eventually overwrite this in any case. Also I _think_ messing with shortcuts sometimes has weird effects with Chrome, like on taskbar grouping, though it's been a while since I've tried.
In Windows, at least, you can make a batch file to launch Chrome with that option, and then use a shortcut to that batch file to open Chrome.
You can skip the batch file and just include the flag in the shortcut.
How much of a performance penalty is there for turning off JIT? Are there any other consequences?
Crypto libraries that rely on WebAssembly (disabled by jitless mode) break. Some services like protonmail that require to use client device encryption will fail.
Could you explain a little bit more in depth on how to do this on firefox?
You can turn off the “javascript.options.baselinejit” and “javascript.options.ion” (Ion is the optimizing JIT) preferences in about:config, but you are definitely voiding the warranty and running an untested configuration by doing this!
https://searchfox.org/mozilla-central/rev/57527d50ef5d3df412...
https://searchfox.org/mozilla-central/rev/57527d50ef5d3df412...
[deleted]
Sure, you can use the enterprise policy:
https://chromeenterprise.google/policies/#DefaultJavaScriptJ...
And this existed before SDSM... in fact SDSM depends on the code landed in Chrome that backs this policy.
https://chromeenterprise.google/policies/#DefaultJavaScriptJ...
And this existed before SDSM... in fact SDSM depends on the code landed in Chrome that backs this policy.
This may sound like an insane number. But as a Chrome developer, the number of flags was never an issue for me. You don't need to test every permutation of flags because almost all of them are unsupported, best-effort kinds of things with a very small scope. You only need to test the default configuration. Compared to the other sources of complexity in Chrome (primarily portability, sandboxing, backwards compatibility, and the sheer scope of the provided APIs), flags are nothing.
This is right, in many cases flags are supposed to control specific launches ("flag-flip") or being a developer shortcut rather than giving users meaningful choices.
Sounds like they forgot to cleanup after themselves
It's largely an artifact of trunk-driven development. Everything is behind a feature flag. The benefits of this approach vastly outweigh the drawbacks IMO, but one of those drawbacks is that you tend to end up with large numbers of dead codepaths behind forgotten flags, because cleaning up old flags becomes a multi-stage process, with no incentive to follow through but your team's "standard" processes (that probably nobody's checking on) and your own desire to keep things tidy.
Or there might be hundreds of on-going genuine launches/experiments. There are literally thousands of engineers in the Chrome org and each launch usually takes several quarters thanks to the nature of being a customer facing platform.
[deleted]
A friend used to call this kind of setup flag-oriented programming.
I've suffered it when working for engineering companies where the PMs understood the logic but didn't understand why it's a bad idea to just create branches for everything.
Not sure why programmers would inflict this "methodology" on themselves. And no, sorry, I'm not going to read the article. Too soon.
I've suffered it when working for engineering companies where the PMs understood the logic but didn't understand why it's a bad idea to just create branches for everything.
Not sure why programmers would inflict this "methodology" on themselves. And no, sorry, I'm not going to read the article. Too soon.
Actual research has demonstrated that trunk based development works well. Flags are a downside, but there are enormous upsides.
It's great to test/disable features. Most people never need them but it saves a recompile when you do
Oh, for testing we used other tricks, config files or the more modern instrumentation. The flags that I dislike are the ones used to add functionality as an afterthought.
bazel is also pretty good with the number of flags. I didn't count them but there are quite a few of them: https://bazel.build/reference/command-line-reference
is this part of the Google culture?
is this part of the Google culture?
Yea, a lot of the core libraries at Google have flags in them, so when you link them into your cli, you get those flags as well.
They once submitted a patch to raise the Linux kernel’s limit on process command lines, because 128 KB was no longer enough for all their flags.
is there a source for this patch?
I am not a global variable sir, I am a command line flag.
"gflags:" https://gflags.github.io/gflags/
*Not to be confused with the windows sdk utility of the same name
Ha yes, yes it is. There are so many flags in a typical google3 program that there are a bunch of metaflags to change the way the usage message is printed. Because --helpfull is going to show you several megabytes of junk.
Yes. This is part of the no long lived branches[0] culture. Fwiw, I have found it superior to having to do huge merges.
[0] https://dl.acm.org/doi/10.1145/2854146
[0] https://dl.acm.org/doi/10.1145/2854146
I've worked in code bases like this, although not in Google. I think it's a good way to go but it does increase code size and sometimes you end up doing some convoluted dances with trying to change behavior based on a flag.
Also, good hygiene is important and implementing some kind of tracking ("no more than N flags older than X") is IMO a good practice. I wonder how Google does this?
Also, good hygiene is important and implementing some kind of tracking ("no more than N flags older than X") is IMO a good practice. I wonder how Google does this?
I recall that folks did analyze this, every server running in production has its flags logged, and every command line tool run on user workstations is also logged. Then it's just a matter of having a registry of every flag (aka the source code) and diffing the set of flags used vs declared. As you would imagine most flags were never used.
Now removing them is another thing entirely -- but some teams do have code cleanup fixits.
Yes it does make source code smaller and simplier to read. But if you're referring to binary code size - well it pales in comparison to what protobuf contributes to generated code size. I'd say that proto code makes up >60% of Google binary size.
Now removing them is another thing entirely -- but some teams do have code cleanup fixits.
Yes it does make source code smaller and simplier to read. But if you're referring to binary code size - well it pales in comparison to what protobuf contributes to generated code size. I'd say that proto code makes up >60% of Google binary size.
The Bazel documentation does tend to repeat them in multiple sections, though.
It also doesn't include many "undocumented"/experimental flags, which are hidden in release notes or only mentioned in the source code.
It also doesn't include many "undocumented"/experimental flags, which are hidden in release notes or only mentioned in the source code.
because there are no branches in google3 (the monorepo which stores virtually all internal code) flags are used to guard code paths rather than branches.
E.g I know in just one client-facing app there is well over a thousand flags, all individually owned and monitored by various individuals and teams wishing to either experiment or launch features.
E.g I know in just one client-facing app there is well over a thousand flags, all individually owned and monitored by various individuals and teams wishing to either experiment or launch features.
Some of these, like "--memlog", that have no description seem pretty clear about what they do.
Some, like "--zygote", have no description and also are totally unclear.
Love it.
Some, like "--zygote", have no description and also are totally unclear.
Love it.
The word "Zygote" will actually be really clear to anyone who's been exposed to Chrome's multiprocess architecture.
https://chromium.googlesource.com/chromium/src/+/HEAD/docs/l...
> A zygote process is one that listens for spawn requests from a main process and forks itself in response. Generally they are used because forking a process after some expensive setup has been performed can save time and share extra memory pages.
With that background, --zygote isn't actually a flag at all. If you click on it it takes you to a string constant, but that string is actually a value for the --type flag. e.g. anzygote process would have `--type=zygote`.
`--type` is itself an internal flag, explaining it's lax description of "Flags spied upon from other layers.". However it is briefly described in the documentation I linked above.
https://chromium.googlesource.com/chromium/src/+/HEAD/docs/l...
> A zygote process is one that listens for spawn requests from a main process and forks itself in response. Generally they are used because forking a process after some expensive setup has been performed can save time and share extra memory pages.
With that background, --zygote isn't actually a flag at all. If you click on it it takes you to a string constant, but that string is actually a value for the --type flag. e.g. anzygote process would have `--type=zygote`.
`--type` is itself an internal flag, explaining it's lax description of "Flags spied upon from other layers.". However it is briefly described in the documentation I linked above.
Fascinating -- thank you!
[deleted]
Here is some critique why this is awful in context of bazillion cli options in compilers. [0]
[0] Programming Language Q&A, December 2022 - Jonathan Blow https://youtu.be/OAIqCpqszVw?t=3974
[0] Programming Language Q&A, December 2022 - Jonathan Blow https://youtu.be/OAIqCpqszVw?t=3974
That is a crazy amount of flags. I wonder how many conflict with each other too
Internally all Google apps have mind bending numbers of command line flags. I believe this drove kernel patches to increase the size of the command line.
Yes flag hygiene is a bit of a problem too :)
Yes flag hygiene is a bit of a problem too :)
That was something else. Most command line flags were untouched from the defaults. What was a problem was GFE's command line. For a number of reasons, it had to list every public service and its config. There were many hundreds of those, probably thousands by now. And there was a kernel limit of 128KB, which was kept at bay for a while with tricks (hacks?), until eventually a proper solution was implemented.
Hacks such as: the ability to unpack the command line from a single compressed and base64-encoded arg.
Fun fact about Linux's now-unlimited, demand-paged argv: the JVM was a bit behind the ball on that one. As soon as Linux systems with enormous args limits appeared, `xargs` happily adapted. But the JVM at that time had quadratic command line parsing. So anyone using `xargs java` suddenly had cron jobs (or whatever) that never finished.
Fun fact about Linux's now-unlimited, demand-paged argv: the JVM was a bit behind the ball on that one. As soon as Linux systems with enormous args limits appeared, `xargs` happily adapted. But the JVM at that time had quadratic command line parsing. So anyone using `xargs java` suddenly had cron jobs (or whatever) that never finished.
What prevents a more general solution like using @flagfiles instead of literally specifying them as arguments?
As far as I recall, until the plumbing was heavily redone a decade later or so, you could only push (propagate) an updated command line to replicas, not a separate file as well. On top of that, I vaguely remember something about bootstrapping and ensuring that the remaining flags and the flag file always be updated atomically, in lockstep.
For regular Unix folks, imagine issuing a remote ssh command. You can't easily do that AND bundle an auxiliary data file at the same time.
For regular Unix folks, imagine issuing a remote ssh command. You can't easily do that AND bundle an auxiliary data file at the same time.
I don't know. Judging from source blame, flagfile has been in gflags since 2007 at the latest, and 2007 is also when the command line length limit was lifted: https://github.com/torvalds/linux/commit/b6a2fea39318e43fee8...
I remember 128KB being an issue past mid 2007, which is when that patch was merged. I think that the compression trick actually happened a bit after that. Looking at the patch, it appears that yes, it's one of the temporary hacks I was referring to. It did add some breathing room, because it no longer constrains the aggregate of argv plus the environment, but it keeps 128KB as the max size for each flag, which was still problematic for good old GFE and its service list. I can't remember if they managed to implement external flag files first or if they had to resort to --flag, --flag2, as web folks do nowadays to work around e.g. cookie limits (I'm looking at you, oauth2-proxy, embedding massive claims).
My only guess is that it plays badly with borg run somehow but having the borg cli autogenerate a flagfile seems uncomplicated.
Edit: Although there's arguably security implications to letting me push a file from my workstation, but that wouldn't have been an issue in 2006.
Edit: Although there's arguably security implications to letting me push a file from my workstation, but that wouldn't have been an issue in 2006.
At that point, why not just use them as a library and call them as an API from bespoke command line tools?
I remember someone telling me about a requested feature in Chrome that got rejected because it would create "yet another option" that would need to be tested etc.
1400 seems high until you see all the internal experiment flags. The configuration is literally several megabytes of configuration code that generates tens of megabytes of configuration protobuf message.
Most important flag: Which one gives you ability to run chrome without browser profile being connected to Google account in the browser session?
I tried messing with these flags a month or so ago to see how close one could get to creating a standalone web app without needing to use Electron or similar. I assume the binary executable that a PWA installs is essentially just setting these flags on startup, so I decided to try to figure out which ones.
As someone else stated, most of the app oriented flags are ChromeOS specific, so I didn't have much success. I tried a lot of different combos and scrolled through Chromium's .h headers to see what I could find. It was a good way to waste an afternoon and learn a bit more about the browser. Kiosk modes, dummy profiles, full screen, single window, security flags, etc. Lots of interesting features you don't run across regularly.
As someone else stated, most of the app oriented flags are ChromeOS specific, so I didn't have much success. I tried a lot of different combos and scrolled through Chromium's .h headers to see what I could find. It was a good way to waste an afternoon and learn a bit more about the browser. Kiosk modes, dummy profiles, full screen, single window, security flags, etc. Lots of interesting features you don't run across regularly.
If you create a PWA on Linux all it does is drop a new ".desktop" file in `$XDG_DATA_HOME/applications`, which you can then inspect to see what flags it sets. I imagine it's mostly the same flags as used on other platforms.
Turns out it just calls Chromium and passes an app id:
[Desktop Action Edit]
Name=Edit
Exec=/usr/lib/chromium-browser/chrome --profile-directory=Default --app-id=idjenohckefppahppaeclfgpbfkfppaf --app-launch-url-for-shortcuts-menu-item=https://www.hypertext.plus/editor/Oh, wow. Thanks! I will totally look at that!! I was even trying to look for strings within the Mac binary, and browsing around the repo for the PWA creation code (which I didn't find). I'm super curious what's in the .desktop file. I'm definitely going to check that out this evening.
And that list is not complete, I use at lease two additional switches in development instance of chrome: --ignore-certificate-errors --ignore-ssl-errors
I've used this page many times. Pretty useful. It's a hierarchy and pretty well organized.
I think this more reflects the complexity of emerging web technologies than Chrome-specific bloat. I think many people would be surprised by the sheer number of new web tech that's been added or is on the horizon. Check out some of the stuff here for an example: https://chromestatus.com/features Web browsers today are more like operating systems than basic document viewers. It really is crazy what a web browser can do now. Eventually all native APIs will be web-based and all software will run in / on a web page.
My guess is the people who thought this was interesting, rather than just a source of reference, have never configured their own Linux installation. Sigh.
From man cdrecord:
Bugs
Cdrecord has even more options than ls.
networksetup enters the chat
This makes about:config look elegant.
Can't wait for hacker news to find out about --enable-features and --disable-features.
Funny to see that some of them use `_`s instead of `-`s as the separator character.
The underscore is the preferred method for naming flags inside Google.
it also simplifies the search for the flag in the source code. Rather than having to switch between `-` and `_`, you have only one textual representation for the flag.
I personally like to use `-` in filenames and flag names because it's one less shift key to press. But it leads to the requirement to translate the `-` into `_` in the code. Eg, `resource-filename.json` becomes something like `resource_filename` in the code.
I personally like to use `-` in filenames and flag names because it's one less shift key to press. But it leads to the requirement to translate the `-` into `_` in the code. Eg, `resource-filename.json` becomes something like `resource_filename` in the code.
Now, the real question is, what happens if you run chrome with all of them!
Speaking of Chrome:
There was a recent Chrome on macOS beachballing problem related to dragging a tab. Multiple users noticed it at MAANG megacorp.
Resolution:
Running from the CLI changed some state that eliminated it, so logging in verbose mode was impossible. Instead, a SWE sent a spindump over to Chromium attached to an opened issue.
There was a recent Chrome on macOS beachballing problem related to dragging a tab. Multiple users noticed it at MAANG megacorp.
Resolution:
Running from the CLI changed some state that eliminated it, so logging in verbose mode was impossible. Instead, a SWE sent a spindump over to Chromium attached to an opened issue.
Is Firefox this complicated?
Firefox has a boatload of things available in about:config.
So does Chrome though. Can't speak to the overlap with flags. If a large part of it's just exposing all that then I suppose that's good... But wouldn't you just have it be the same key:value pair as on that page, behind a single flag?
Chrome doesn't have an about:config. There is about:flags (redirects to chrome://flags/) but that intentionally doesn't list all of these.
I doubt it. Having used Firefox for over a decade, I never had to use flags other than a profile selection option and a safe mode flag.
I have come across more flags during my occasional use of Chrome.
Firefox's configuration flags are unified in `about:config` page. It has a whole bunch of configurations too, but they are neatly named (e.g `browser.ui.something.enabled`) and I prefer configuration options than flags.
I have come across more flags during my occasional use of Chrome.
Firefox's configuration flags are unified in `about:config` page. It has a whole bunch of configurations too, but they are neatly named (e.g `browser.ui.something.enabled`) and I prefer configuration options than flags.
What we need is, organized flags into meaningful -standardized- taxonomy.
just one more flag, bro. That will be the last one
Think about the number of ways in which these can be combined and the kind of headache that would cause during testing.
Maybe they would benefit from a project aimed at reducing this to a more manageable list?
Maybe they would benefit from a project aimed at reducing this to a more manageable list?
These Googler comments crack me up. "It may seem like this is out of control, etc., but for Google/my team..." That's just straight cruft, whether you are at Google or Billy Joe Bob's software shop.
I don’t know, engineers default to griping about their experience with mundane things much less extraordinary ones. I find so little griping about something seemingly extraordinary pretty persuasive. That’s bolstered by the fact that unlike, say, curl or ffmpeg, even as an engineer myself using Chrome constantly I’ve almost never had to care about a single one of its flags.
Ok there's some cruft there, but if you remove the cruft, you'll still be left with, say, 700 flags, which a lot. So the point is that it's basically ok to have a lot of flags.
and none to disable their telemetry, spying or tracking
flag creep
at some point you gotta start engineering a solution instead of just adding to the list of flags.. isn't that what engineers are supposed to do?
Seems like they got themselves into a tech debt situation now
Seems like they got themselves into a tech debt situation now
Well it's either flags, or a config file (firefox's user.js). Just two different ways of achieving the same thing.
Browsers are complex these days, and it's good that we have options to change behaviour that is problematic for us. I run chrome/firefox as part of a batch job on a virtual linux display, so I really need to turn off all the crap like first-start wizards, bookmarks and the like.
Browsers are complex these days, and it's good that we have options to change behaviour that is problematic for us. I run chrome/firefox as part of a batch job on a virtual linux display, so I really need to turn off all the crap like first-start wizards, bookmarks and the like.
> Well it's either flags, or a config file
Actually: No. Well, at least not in the ridiculous way that Chrome or Firefox do it. Command line options or config files are both fine - when used in moderation.
The problem here (at least in part) is that it is extremely easy for devs to just add a configuration option. But it is much harder to think about whether a user would actually ever want to change the setting, if it had a good default value.
In fact, I think maybe about 20 command line options would cover 99.9% of all usecases for chrome. - And it would be so much more understandable. - Both for the actual developers and the users.
I absolutely despise it when a software product pushes effort downstream this way.
---
Note: I fully understand that in Chrome these flags at least partially leak out of their "trunk-driven development". - But this is no excuse: In no way should the development environment leak out into a released product this way! - And it would be exceptionally easy to stop this too: Just disable all development flags in the release build.
Actually: No. Well, at least not in the ridiculous way that Chrome or Firefox do it. Command line options or config files are both fine - when used in moderation.
The problem here (at least in part) is that it is extremely easy for devs to just add a configuration option. But it is much harder to think about whether a user would actually ever want to change the setting, if it had a good default value.
In fact, I think maybe about 20 command line options would cover 99.9% of all usecases for chrome. - And it would be so much more understandable. - Both for the actual developers and the users.
I absolutely despise it when a software product pushes effort downstream this way.
---
Note: I fully understand that in Chrome these flags at least partially leak out of their "trunk-driven development". - But this is no excuse: In no way should the development environment leak out into a released product this way! - And it would be exceptionally easy to stop this too: Just disable all development flags in the release build.
What is the alternative? Make them compile-time options? Expose them to the UI?
For all sorts of testing reasons, there has to be a way to disable the JIT. There has to be a way to disable sandboxing. There has to be a way to bypass the GPU acceleration blacklist, and there has to be a way to launch without GPU acceleration on non-blacklisted hardware. Thank god the options exist!
I mostly wish the documentation was better. It would be great if I could filter by platform. And, um, the definitive source probably shouldn't be some guy's personal website.
For all sorts of testing reasons, there has to be a way to disable the JIT. There has to be a way to disable sandboxing. There has to be a way to bypass the GPU acceleration blacklist, and there has to be a way to launch without GPU acceleration on non-blacklisted hardware. Thank god the options exist!
I mostly wish the documentation was better. It would be great if I could filter by platform. And, um, the definitive source probably shouldn't be some guy's personal website.
The alternative is to not have (most of them) at all.
The reality is that 99.5% of users have perhaps 10 or 20 actual usecases/requrements. - You named a few of the most prominent ones. These need to be supported (e.g. with flags) and documented properly. Everything else is excess can be removed.
The problem with this is: You need to find out what users want to do. But many software projects are just too lazy and push all the effort downstream.
The reality is that 99.5% of users have perhaps 10 or 20 actual usecases/requrements. - You named a few of the most prominent ones. These need to be supported (e.g. with flags) and documented properly. Everything else is excess can be removed.
The problem with this is: You need to find out what users want to do. But many software projects are just too lazy and push all the effort downstream.
I was just thinking that in my own code I have a tendency to add more command line arguments as a quick fix and then (hopefully) go back and limit them to what is essential and add config files if necessary, etc. It does seem like a sign that too many people are able to come in and do some debugging or experiment via flags, without cleaning up afterwards.
Yeah - my first thought, on seeing the list, "the project is completely out of control and it's amazing that it works as well as it does!"
I guess it's a tribute to what you can get if you throw enough money and effort at something! They must be running flat-out just keeping the project going and the ever-changing product running.
Just imagine what they could have done with that money & effort, if it was spent on clear, focused projects.
I guess it's a tribute to what you can get if you throw enough money and effort at something! They must be running flat-out just keeping the project going and the ever-changing product running.
Just imagine what they could have done with that money & effort, if it was spent on clear, focused projects.
If you see evidence like this that conflicts with your prejudices, it should make you rethink your prejudices, not denounce the evidence. Facts we are aware of: Chrome is an extremely successful project with many users and developers, and it uses flag-oriented feature control. Your prejudice: flag-oriented feature switches are "out of control". Now, how are you going to reconcile this? Doesn't it call into question the utility of your prejudice? This is demonstrably an extremely successful project that ignores your rule.
I've been around enough projects to pick up on warning signs :) That google has been able to manage such a project is truly impressive.
Chrome is clearly the most popular browser out there but, for me, popularity counts for little. Chrome is also extremely heavyweight by most measures (installed size, memory usage, cpu usage, build time!)
Re popularity: Imagine you're at a retail store buying a coffee maker. They tell you "this model is very popular". For something that people purchase/install occasionally, popularity isn't a good indicator of quality, though it might be a good indicator of price, advertising, feature-list, or sales commission!
Personally, I want my coffee maker to make great coffee, and to keep doing that for a long time. And I want my browser to be fast, lightweight, and minimally intrusive.
Chrome is clearly the most popular browser out there but, for me, popularity counts for little. Chrome is also extremely heavyweight by most measures (installed size, memory usage, cpu usage, build time!)
Re popularity: Imagine you're at a retail store buying a coffee maker. They tell you "this model is very popular". For something that people purchase/install occasionally, popularity isn't a good indicator of quality, though it might be a good indicator of price, advertising, feature-list, or sales commission!
Personally, I want my coffee maker to make great coffee, and to keep doing that for a long time. And I want my browser to be fast, lightweight, and minimally intrusive.
The "it uses lots of memory" argument is sort of silly. That's the behavior you want. Unused ram is wasted, as long as chrome is willing to play nice and free it, using otherwise unused memory to prefetch/cache is good user experience.
Counterpoint: well-designed feature flags encourage good design and modularity. Rather than sneaking my hacks all over the codebase, if I’m going to sneak a non-backwards compatible change somewhere, I’ll need to explicitly put it behind my feature flag, and it’s immediately apparent not only to reviewers but also future maintainers that my feature is the one that changed behavior. It’s also documentation: this is why this change is necessary. And removing a flag, even when the feature is stable, removes those guardrails.
Counter counterpoint: Chrome also has an entire system for gating features behind flags without needing to write a new command line flag for each one.
Counter counter counterpoint: I like command line flags and you can take them from my cold dead hands.
Counter counter counterpoint: I like command line flags and you can take them from my cold dead hands.
If it doesn't hail from the command line flag region of France, can it really be called a feature, or is it just sparkling technical debt?
flagrant!