HackerTrans
TopNewTrendsCommentsPastAskShowJobs

jamiesonbecker

no profile record

comments

jamiesonbecker
·3개월 전·discuss
Rotating keys is easy with the right software. (I work @ Userify) Agree with the auditing point

Token-based keys, to tptacek's point, is that they can be a giant pain once you start scripting across fleets.
jamiesonbecker
·3개월 전·discuss
One key per device is exactly what we recommend too. Private keys should always be protected as much as possible within that device and should never leave that device.

Just paste all of your devices' public keys into your authorized_keys file and leave a comment at the end for what device it's for. in Userify, it literally goes right into your nodes' authorized_keys file almost verbatim. (disclaimer: I work at https://Userify.com)

And then, if you leave your token or laptop at the airport or whatever, just remove that key right from your phone and it'll take effect in seconds across all the nodes/instances (if you're using Userify) or you can just write a quick for-inline-sed loop to remove it from your authorized keys everywhere.
jamiesonbecker
·3개월 전·discuss
The next one linked at the bottom, https://jonno.nz/posts/stealing-nanoclaw-patterns-for-webapp... has this bold and frankly unbelievable claim:

"70% of startups fail due to premature scaling"

.. which is a link to another blog post somewhere else that says nothing even slightly related.
jamiesonbecker
·3개월 전·discuss
Then install your own:

    curl i.userify.com | sudo - sE
jamiesonbecker
·3개월 전·discuss
The experience might be better right up until you're running it in prod and someone happens to ask about:

   Cert revocation (or even expiration)

   Sudo roles

   User removal and process termination

   Is the cert server HA and locked down

   How you log in when the cert server is down or under attack (rich target!)

   How to easily add Alice to server group A, Bob to B, and Carlos to both A and B, and then to remove them..
(disclaimer we're celebrating our 15th anniversary at https://Userify.com, but those are actually legit concerns and not only a sales pitch. You certainly can build a solid and secure ssh cert infra, but doing it in production is just not an easy set-it-and-forget-it sort of thing.)
jamiesonbecker
·3개월 전·discuss
Honestly, we used to replace a lot of pam_ldap and similar sorts of awful solutions. With those, if your LDAP went down even for a heartbeat, you couldn't log in at all.

So I totally agree: if I had to do certificates and didn't have something like Userify, a 1 hour (or even shorter if possible) expiration seems quite worth chasing, especially with suitable highly available configuration. (Of course, TFA doesn't even bother mentioning revocation and expiration, which should give you a clue as to how much fun those are lol)

And for more normal, lower-security requirements or non-HA, 6 or 8 hours or so would probably work and give you plenty of time for even serious system outages before the certs expired.

Not to hard shill or anything (apologies in advance, just skip if you're not interested), but there are two significant security and reliability differences between standard SSH (with or without certificates) and Userify:

1. Userify Cloud updates by default every three minutes, and on-premise Userify Express/Enterprise updates every ten seconds, but it doesn't have to update at all; even if your Userify server goes offline forever, you can still log in because the accounts are standard UNIX accounts (literally created with `useradd`)

2. When accounts are removed, Userify also completely nukes the user account, removes its sudo perms, and totally kill -9 's any tmux/screen/etc sessions (all processes owned by the user are terminated across the entire enterprise within seconds), which is also not something that a certificate expiration would ever do.
jamiesonbecker
·3개월 전·discuss
Great question. Not yet ;)
jamiesonbecker
·3개월 전·discuss
Well, TOFU is really just the model for how the chain of trust is established.

In practice there isn’t really trust on first use: there’s verify the key matches what’s expected, or distribute keys out-of-band (including certs).

If that verification step isn’t happening, then it’s not TOFU, it’s just blind trust.

From an automation/autoscaling angle, the same thing shows up again:

1. either keys are pre-baked / distributed

2. or, something signs them at boot

Signing an instance key is just another way of distributing trust. It doesn’t remove the need for a root of trust, it moves it.

Certificates just add extra steps around the same underlying task.
jamiesonbecker
·3개월 전·discuss
That works for authn in the happy path: short-lived cert, grab it, connect, done.

Except for everything around that:

* user lifecycle (create/remove/rename accounts)

* authz (who gets sudo, what groups, per-host differences)

* cleanup (what happens when someone leaves)

* visibility (what state is this box actually in right now?)

SSH certs don’t really touch any of that. They answer can this key log in right now, not what should exist on this machine.

So in practice, something else ends up managing users, groups, sudoers, home dirs, etc. Now there are two systems that both have to be correct.

On the availability point: "reasonably available" is doing a lot of work ;)

Even with 1-hour certs:

* new sessions depend on the signer

* fleet-wide issues hit everything at once

* incident response gets awkward if the signer is part of the blast radius

The failure mode shifts from a few boxes don't work to nobody can get in anywhere

The pull model just leans the other way:

* nodes converge to desired state

* access continues even if control plane hiccups

* authn and authz live together on the box

Both models can work - it’s more about which failure mode is tolerable to you.
jamiesonbecker
·3개월 전·discuss
SSH certs quietly hurt in prod. Short-lived creds + centralized CA just moves complexity upward without solving the core problem: user management.

The system shifts from many small local states to one highly coupled control point. That control point has to be correct and reachable all the time. When it isn’t, failures go wide instead of narrow.

Example: a few boxes get popped and start hammering the CA. Now what? Access is broken everywhere at once.

Common friction points:

     1. your signer that has to be up and correct all the time
     2. trust roots everywhere (and drifting)
     3. TTL tuning nonsense (too short = random lockouts, too long = what was the point)
     4. limited on-box state makes debugging harder than it should be
     5. failures tend to fan out instead of staying contained
Revocation is also kind of a lie. Just waiting for expiry and hoping that’s good enough.

What actually happens is people reintroduce state anyway: sidecars, caches, agents… because you need it.

We went the opposite direction:

     1. nodes pull over outbound HTTPS
     2. local authorized_keys is the source of truth locally
     3. users/roles are visible on the box
     4. drift fixes itself quickly
     5. no inbound ports, no CA signatures (WELL, not strictly true*!)
You still get central control, but operation and failure modes are local instead of "everyone is locked out right now."

That’s basically what we do at Userify (https://userify.com). Less elegant than certs, more survivable at 2am. Also actually handles authz, not just part of authn.

And the part that usually gets hand-waved with SSH CAs:

     1. creating the user account
     2. managing sudo roles
     3. deciding what happens to home directories on removal
     4. cleanup vs retention for compliance/forensics
Those don’t go away - they're just not part of the certificate solution.

* (TLS still exists here, just at the transport layer using the system trust store. That channel delivers users, keys, and roles. The rest is handled explicitly instead of implied.)
jamiesonbecker
·3개월 전·discuss
But then you can't log in if your box goes offline for any reason.
jamiesonbecker
·3개월 전·discuss
We're in the process of updating the experience to this century! ;)

We've always taken the stance that crusty is better than vulnerable, but it turns out that not having a modern experience after 15 years is starting to feel like maybe we need to step up the features and shininess :)
jamiesonbecker
·3개월 전·discuss
Exactly. We'd had discussions about building https://Userify.com (plug!) around SSH certificates, but elected to go with keys instead, because Userify delivers most of the good things around certificates without the jank and insecurity.

It's not that certificates themselves are insecure themselves, it's that the workflows (as the parent points out) are awful. We might still add some automation around that (and I think I saw some competitor tooling out there if you're committed to that path) but I personally feel like it's an answer to the wrong question.
jamiesonbecker
·4개월 전·discuss
Classic OpenSSH safety check: if /home/$user (or ~/.ssh) is too open, or ownership/modes are off, sshd will refuse pubkey auth. Annoying, but correct.

If you still have some access (console, password login, another sudo user), this usually fixes it:

    username=bob
    sudo chown "$username:$username" /home/$username
    sudo chmod 700 /home/$username

    sudo install -d -m 700 -o "$username" -g "$username" /home/$username/.ssh
    echo "ssh-ed25519 AAAA....insertyourpubkeyhere" | sudo tee /home/$username/.ssh/authorized_keys >/dev/null
    sudo chown "$username:$username" /home/$username/.ssh/authorized_keys
    sudo chmod 600 /home/$username/.ssh/authorized_keys
(optional, if the user needs sudo)

    echo "$username ALL=(ALL) NOPASSWD: ALL" | sudo tee /etc/sudoers.d/$username >/dev/null
    sudo chmod 440 /etc/sudoers.d/$username
Not to shill too hard, but this exact "keys/perms/sudo drift" failure mode is why Userify exists (est. 2011): local accounts on every box + a tiny outbound-only agent that polls and overwrites desired state (keys, perms, sudo role). If scp/rsync/deploy steps clobber stuff, the next poll re-converges it (cloud default ~90s; self-host default ~10s; configurable). Removing a user also kills their sessions. No inbound ports to nodes, no PAM/NSS hooks, auditable.

Shim (old but readable): https://github.com/userify/shim/blob/master/shim.py#L308 (obligatory): https://userify.com
jamiesonbecker
·7개월 전·discuss
at least it had a minimum of Clause. Clause. Punchline.
jamiesonbecker
·7개월 전·discuss
I like it too, even though it has that distinctive odor of being totally written by chatgpt though. (a bit distracting tbh)
jamiesonbecker
·7개월 전·discuss
One of the big issues was latency.
jamiesonbecker
·7개월 전·discuss
Thank you for the comprehensive answers!

(By the way: NVIDIA AIstore is NOT a proxying/caching engine, although it can, which is somewhat unique among these types of stores. AIstore is actually a full S3 engine in its own right, and it's actually extremely capable, although live cluster resizing and ETL requires k8s :( )
jamiesonbecker
·7개월 전·discuss
These questions are meant to be constructively critical, but not hyper-critical: I'm genuinely interested and a big fan of open-source projects in this space:

* In terms of a high-performance AI-focused S3 competitor, how does this compare to NVIDIA's AIstore? https://aistore.nvidia.com/

* What's the clustering story? Is it complex like ceph, requires K8s like AIstore for full functionality, or is it more flexible like Garage, Minio, etc?

* You spend a lot of time talking about performance; do you have any benchmarks?

* Obviously most of the page was written by ChatGPT: what percentage of the code was written by AI, and has it been reviewed by a human?

* How does the object storage itself work? How is it architected? Do you DHT, for example? What tradeoffs are there (CAP, for example) vs the 1.4 gazillion alternatives?

* Are there any front-end or admin tools (and screenshots)?

* Can a cluster scale horizontally or only vertically (ie Minio)

* Why not instead just fork a previous version of Minio and then put a high-speed metadata layer on top?

* Is there any telemetry?

* Although it doesn't matter as much for my use case as for others, what is the specific jurisdiction of origin?

* Is there a CLA and does that CLA involve assigning rights like copyright (helps prevent the 'rug-pull' closing-source scenario)?

* Is there a non-profit Foundation, goal for CNCF sponsorship or other trusted third-party to ensure that the software remains open source (although forks of prior versions mostly mitigates that concern)?

Thanks!
jamiesonbecker
·7개월 전·discuss
> 8. Google kills Gemini Cloud Services (killedbygoogle.com)

So, Google renamed itself to Gemini?

Instead of Google having a product named Gemini, Gemini has a product named Google.

pattern recognition much?