I wonder what impact this will have on Mozilla's OpenSSH configuration guide[0], which currently specifies `[email protected]` as its primary cipher. Should that be dropped to rely solely on AES ciphers?
I'm the CTO of a popular Secrets Management platform. It's fair to say that I personally have a lot of experience with secrets and requirements around them, based on conversations with customers.
The primary missing feature here seems to be multiline support. That's super common for keys, certificates, and other JSON configuration. Based on the Node PR, they appear open to adding that later (big +1 on shipping incrementally).
The other missing feature that folks tend to heavily rely on is variable expansion. For that to truly work well, I recommend using a holistic platform like Doppler. That allows for expansion/referencing across environments and projects, like when you have multiple independent services that need access to the same set of secrets (e.g. database creds, error reporting tool, stripe key, etc). You can then update the secret once and have the change propagate to all the places it's used.
Lastly, I'd be remiss if I didn't mention the Doppler CLI and our own fairly unique support for .env files. We've traditionally taken a dim view of .env files because they represent a static, long-lived collection of sensitive information that lives offline. Often, these get checked into a git repo. This is probably fine for personal projects, but a major issue for companies and the security aware. However, .env files are a pseudo standard and folks want a way of continuing to consume their secrets via them. Our CLI's approach is to mount a named pipe that we can write secrets to when a reader attaches. That allows us to limit the amount of times the "file" can be read (e.g. once), it guarantees that the file's contents are unavailable once the application process dies, and it uses the same open/read interface as a standard file.
In all, this is an exciting development for Node. I'm glad to see more standard features make it into core and hope that multiline support is a fast follow.
When this deal was originally announced, Adobe's stock took a ~10% hit. Now that the deal is getting blocked, their stock is again taking a hit? I certainly can't claim to understand the public markets.
You're exactly right about those two problems, and they actually go together quite well. If a user who previously had access to a secret suddenly has that access revoked, you should rotate that secret. Technically that's only really necessary if the user ever actually saw that secret value, but most secret managers don't expose that information.
At Doppler, we're currently in the process of building out our rotation engine[0]. It allows you to automatically rotate secrets with third party providers (including your database) on a schedule you define. We also provide access logs to see exactly who saw what and when. The ultimate goal is to automatically rotate your secrets as users' access changes. And if you get notification of a breach, you can rotate everything with a single click.
Our process never includes ghosting anybody, even when we pass on a candidate, so something must have gone very wrong here. Could you email me at thomas@[company domain]? I'll look into this further.
At Doppler we have a strong culture of engineering excellence, thorough PR review, security, automation/CI, and helping each other succeed. We give engineers 30% time to work on things they think are important and are remote first (but also have an office in SF). Our engineering team hangs out in Discord all day to make pair programming/debugging easy. It helps that our CEO used to be an engineer.
Co-founder/CTO of Doppler here. I'd agree that Doppler currently focuses very heavily on UX. We aim to fit into developer workflows, regardless of where devs run their code. This includes local development (macOS/Windows/Linux), CI/CD, Kubernetes, Vercel, and just about anywhere else. We've found that security tools that are difficult to use end up being worked around by developers, which ultimately decreases an org's security posture (see HashiCorp Vault). This does come with some tradeoffs- specifically that you must trust Doppler with your secrets, given our tokenization model[0].
We do recognize that the current security tradeoffs of Doppler aren't going to satisfy everyone. For Enterprises, we offer Enterprise Key Management (EKM), which allows orgs to encrypt their secrets using a cloud KMS. Of course, this still doesn't satisfy every concern. And so, for customers requiring additional security guarantees, stay tuned!
Doppler solves this problem by storing your secrets in the cloud *hand wave*.
In actuality, the Doppler CLI (a Go binary) fetches your secrets from Doppler's API and injects them as environment variables into your specified process. That looks something like `doppler run -- printenv`. This prevents your secrets from being written to the filesystem in plain text, and prevents the environment variables from being available more broadly. In the case of docker, you would bake the Doppler CLI into your image, thereby sidestepping the documented `docker inspect` pitfall.
Of course, the CLI still needs a way of authenticating to Doppler's API. You authenticate and authorize the CLI by running `doppler login`. This initiates a login flow that you complete in your browser. Once completed, your newly generated auth token is sent back to the CLI. The CLI then saves the auth token to your system keyring for later use. The identifier needed to access that keyring entry is then stored in plain text in the CLI's config file (~/.doppler/.doppler.yaml), which is only readable by the user.
We're exploring other means of injecting your secrets into your application, as some users are wary of using environment variables. This is a challenging problem though as there are few means of injecting secrets that don't require substantially changing your application's logic.
This solves a very real problem that some services like GitHub [0] have started to address. Auth tokens are being committed to public repos at an alarming rate. Detecting this and ideally preventing it as early as possible is key to avoiding account compromise. There are two components to this: identification of a secret and attribution. Identification is non-trivial and requires determining if some text really is a secret and not just a random hash, uuid, or other high entropy string. Most tokens today are generic, alphanumeric patterns; false positives abound. Attribution is tricky too, currently relying on either parsing the variable name (`AWS_SECRET_KEY=XYZ`), commit message, file name, or some other metadata. In the rare case, a service will have designed their auth tokens with this in mind, prepending a unique, static prefix to their tokens.
The URI scheme proposed in the linked RFC will squarely solve the first problem. It will allow for highly accurate CI scanners and pre-commit hooks. The scheme doesn't appear to address attribution, assuming all service providers use the same `secret-token` scheme. However attribution is a nice-to-have, allowing for automated revocation once the secret has gone public. If done right, identification alone could be used to prevent most of the token leakage that occurs today.
Really interesting development in the section titled "Crash Monitoring?":
> An interesting side effect of the new processing pipeline is that imagent is now able to detect when an incoming message caused a crash in BlastDoor (it will receive an XPC error)... As can be seen, imagent is apparently informing the iMessage servers that the message with the UUID 0x3a4912626c9645f98cb26c7c2d439520 (fU key) has caused a crash in BlastDoor. It is unclear what the purpose of this is without access to the server’s code. While these notifications may simply be used for statistical purposes, they would also give Apple a fairly clear signal about attacks against iMessage involving brute-force and a somewhat weaker signal about any failed exploits against the BlastDoor service.
> In my experiments, after observing one of these crash notifications, the server would start directly sending delivery receipts to the sender for messages that hadn't actually been processed by the receiver yet. Possibly this is another, independent effort to break the crash oracle technique by confusing the sender, but that is hard to verify without access to the code running on the server.
Apparently Apple has added the ability to detect crashes when parsing iMessages, report that data back to the mothership (with some attribution), and then mislead the sender of the likely malicious message with fake delivery receipts. This is cool on so many different levels.
I haven't gone as far as emailing support, but my single tweet asking if there would be a postmortem after a previous incident did not receive a reply. My other tweets to them, such as inquiring about a recent increase in spam, received replies.
I've admittedly put in very little effort to obtain postmortems from Fastmail. But I also posit that I shouldn't have to. Postmortems should be made readily accessible, just as the outage was. I reserve my detective skills for figuring out my own service's outages.
I'm a happy Fastmail customer, but the number of outages recently has been a little concerning. There has been at least one outage per month for the past several months. And what's worse is that I've never seen a postmortem published. It's hard to maintain confidence in a service that isn't being transparent about why they keep having outages, or what they're doing to prevent future outages.
This is fantastic! I often workout specific muscle groups and cycle through different exercises to target each, so this tool very nicely aligns with that. The videos are a great touch to help me make sure I have proper form. And overall seems to entirely avoid "broscience."
One feature request: list what complementary muscles each exercise targets (e.g. list "triceps" when displaying dips as a chest exercise). This would help me plan my workout when targeting multiple muscle groups, like a chest/tri or back/bi day.
Lots of really great questions in here! Starting at the top-
The process demoed in the video is for user-based auth. Service tokens can be issued programmatically and provide read-only access to one config. You can pass these into your environment via puppet and other configuration tools, or even via LDAP, in much the same way you bootstrap other OS config like SSH keys.
I really like your thoughts around more niche access roles, and it's definitely where we're headed. Admittedly our RBAC is currently more limited with a basic Member, Admin, Owner model. We do have existing audit capabilities for monitoring secrets changes (without revealing the actual secrets) and support shipping those logs to Slack or a webhook.
Audit history is based on your specific plan[0], but we currently offer up to a year. Some customers do have requirements for longer time periods and we can easily configure that for them.
Regarding your last point, this is what we refer to as secret referencing. You can absolutely reference secrets across different configs and projects to avoid repeating common values. Here's our announcement from when we shipped this feature back in August[1].
The local fallback file is updated each time new secrets are fetched. It is possible for your Doppler secrets to change between your last successful fetch and your next unsuccessful fetch, and in that scenario you wouldn't be operating off the latest secrets. We don't see the fallback file as an excuse for availability; our service needs to be up.
The fallback file is encrypted using AES-GCM with a key derived from the auth token and other metadata using PBKDF2. It is theoretically possible for an ex-employee to store the fallback file and retain a copy of the auth token and other metadata, even after the token has been revoked. In this case, they could construct the key using their privileged information. However, this attack would require active malfeasance by an authorized party during the period in which they're still authorized. It would be easier for the bad actor to store the raw secrets (by logging process.env, for example) than storing the encrypted file. To really solve this issue, the secrets would need to be time-bound and dynamic. Dynamic secrets are on our roadmap, but the issue you point out is a really hard problem. (And if you'd like to help, we're hiring![0])
We fully support using Doppler on bare metal servers, VMs, and just about any other cloud service via our CLI [0]. This would be identical to how you'd access your secrets locally when developing. You'd wrap your command with our CLI so that your secrets are injected into the process's environment (e.g. `npm start` becomes `doppler run -- npm start`).
Client certificates are an interesting use case that we'd be open to, but admittedly the request hasn't come up a whole bunch yet from our customers. Same with Kerberos tokens- this would definitely be used in the Enterprise space, which is an offering we're still in the early stages of. Other forms of identity-based auth would fall into this category as well.
Thanks for pointing this out, you're absolutely right. I'm linking to our Security page[0] and our security docs[1] below, but we'll definitely be updating our marketing site to place more of an emphasis on this. I hope the rest of this post and our comments here illustrate that trust and security are things we think a lot about, despite our oversight with the landing page.
[0] https://infosec.mozilla.org/guidelines/openssh