If you're telling me I worked there at literally the worst possible time frame, I'd believe it. I may have my experience skewed through the perspective of Nick's influence, but tbh many of my issues were unrelated to him or his sphere of influence.
The C level thing may not have been a "big" mystery, but it was to me, and as somebody who was running the dev of a flagship software product (UniFi) it set off alarm bells that nobody I talked to could explain who was handling the roles of those execs. I'm not exaggerating when I say I effectively got "I dunno" as a response when I inquired, and I dug.
It is good to know, though, that what I experienced wasn't chronic for the entire company's existence.
To clarify on the China thing, I wasn't trying to imply that anything nefarious was actually happening. Just that it warranted some scrutiny when a security focused product was being developed on the Chinese mainland and by a team of Chinese citizens that are subject to CCP laws. Given some of the things that have happened around that country's involvement in tech in recent years, I don't think such scrutiny is unwarranted, especially when the team has a track record of security "goofs".
Oh god don't remind me about Trace. I had to deal with the Controller side of that and it was a damn nightmare.
He basically dictated that you couldn't use any kind or repo+deployment pipeline except for what his team was building. Which wasn't actually functional for like 8 months. So we never even got a dev or staging tier to test against for months.
And then when I ended up with access to push things along, the actual apps for the trace system we're... not well implemented.
Ugh... I could bitch about this stuff for literal days but I gotta drop my kids off.
> Nick was hired out of his job at Amazon because he was supposed to be the AWS expert.
This always cracked me up. From what I can tell, he was a mid level dev on the Alexa web api team. He knew AWS sure, but he did not have the cred at all to justify the position and responsibility he was given at Ubiquiti.
I've since heard that the repo has been taken down and all the keys rotated, but just kinda makes you wonder how many APs and switches and cloud keys, etc are still out there using compromised keys.
Also, even though they may have had read access, not many knew it existed. But it wasn't super hard to find (I stumbled across it basically).
Oh and then there the whole metrics collection debacle, where the controller basically phoned home about the topology of every network that it managed. Even if you opted out. Opting out just meant they fuzzed your ID so any given record couldn't be linked back to PII. Which may or may not be legal, IANAL.
But either way it definitely wasn't clear that opting out meant data was still collected. Super sketchy.
Hoo boy, this is gonna be a fun one. For reference, I spent a year (mid-2018 to mid-2019) running the UniFi Network team and worked with Nick during that time.
> * Why was it so easy for a lead engineer to get access to a root AWS user without anyone else being notified? I.e. AWS GuardDuty provides FREE alerting for when an AWS root IAM account is logged in or used, this account should be under lock and key and when used, confirmed and audited by relevant persons or teams.
The "Cloud Lead" that Nick took over from gave zero fucks. He ran all the AWS stuff for Ubiquiti under his personal AWS account. Nick came in and started putting "proper" AWS structure and security in place, primarily by scaring Robert (the CEO) into giving him the keys to the castle (my own personal opinion of Robert is... not the greatest).
One thing to understand about Ubiquiti (at least during those times) is that the company had zero C-level execs. There was Robert.... and then nobody knows. I asked repeatedly why we didn't have a CTO, or a COO, or a CFO, or CMO or ANYTHING and I got nothing but shrugs and "idunno" as a response for the whole year I was there.
So when Nick came in, a very... let's just say "forceful" personality, he immediately won over Robert and ended up with carte blanche over pretty much all of Ubiquiti's cloud accounts. Which were basically... everything. All the UniFi Network services, UniFi Protect services, you name it. If it was connected to the cloud in any way, Nick had access to it.
So why wasn't anybody else notified? Simple. Because he was basically "god". If anybody was gonna be notified, it would've been Nick. He was the top of the totem pole company-wide when it came to AWS.
Also, for some perspective, at that time Ubiquiti kept all the hardware signing keys in a private GitHub repo that every employee had read access to. And they were in plain-text. So... yeah.
> * Furthermore on the root account being easily accessed, the root account in the companies I've worked at had MFA enabled, and the QR code is locked in a safe only accessible by two people agreeing it needs to be accessed in a break glass situation, where warranted.
See above for the quality of security processes and practices this company had in place.
> * Why was he also able to delete critical CloudTrail logs and reduce their retention to 1 day? I.e. These logs should be in a S3 bucket or other environment where such changes cannot be made. Alternatively, they should be shipped to a redundant service that manages this risk to prevent data deletion
See above. (re: "god") Nick answered only to Robert. And he'd already successfully hoodwinked him. He could do whatever he wanted. Eventually he fell from Robert's good graces, but seeing as Ubiquiti as a company didn't really have a ton of checks and balances, he kept his god-level access far longer than he should've.
> * Why did Ubiquti not announce they were compromised sooner? The hack started in early December, Ubiquiti noticed the compromise on Dec. 28. Ubiquiti told the market on January 11th. Is that a satisfactory turn around? Giving them some credit for the XMas break I'll say this partially understandable.
Simple. Fear of share price falling. I was constantly given this as a reason we couldn't be transparent. Not by Robert, nor where he could hear. But it was pretty much well known that the company kept shit quiet for fear of the share price dipping.
> All the AWS configuration I'm speaking of above, I would describe as Security 101.
To keep with the metaphor, Ubiquiti couldn't even get Pre-school level security in place, much less 101. I have no idea how something even more massive hasn't happened yet. Must be dumb luck.
Speaking of, by the time I left the company, the team that was handling the door entry-way systems (UniFi "Access" I guess) had been caught with numerous security issues, not the least of which was logging user credentials in plain text (not just storing, but logging, in response to authentication events). They were also based in China and subject to Chinese laws around government access, so take that how you will.
And that doesn't really even cover most of it. That year took a toll on my physical, mental, and emotional health, not to mention put a crazy strain on my marriage. I'd rather honestly forget it, but the schadenfreude of what's going on is too delicious to ignore.
Dude, let's not be generous. Could he write code? Yes. But this is a guy who wrote everything in Node, but absolutely _refused_ to use any existing libraries except for ones he personally wrote. He didn't "trust" them.
He wasn't even hired on as a dev, he was hired to be the "Cloud guy", essentially a sysadmin for AWS, and basically spooked the CEO into giving him the keys to the castle.
Greed. 100% greed. While I was there, the CEO loved to just fly between offices (randomly) on his private jet. You never knew where he'd pop up, and that put everybody on edge, because when he was unhappy he tended to fire people in large chunks (and shut down entire offices). Every decision was motivated by how it affected the stock price.
IMO, the CEO had a bit of a Steve Jobs hero-worship complex, but only all the bad parts. I can absolutely see him putting two teams on the same project, and "may the best product win".
The team that "lost" would get canned, obviously (I saw it happen to two separate offices while I was there).
If you're telling me I worked there at literally the worst possible time frame, I'd believe it. I may have my experience skewed through the perspective of Nick's influence, but tbh many of my issues were unrelated to him or his sphere of influence.
The C level thing may not have been a "big" mystery, but it was to me, and as somebody who was running the dev of a flagship software product (UniFi) it set off alarm bells that nobody I talked to could explain who was handling the roles of those execs. I'm not exaggerating when I say I effectively got "I dunno" as a response when I inquired, and I dug.
It is good to know, though, that what I experienced wasn't chronic for the entire company's existence.
To clarify on the China thing, I wasn't trying to imply that anything nefarious was actually happening. Just that it warranted some scrutiny when a security focused product was being developed on the Chinese mainland and by a team of Chinese citizens that are subject to CCP laws. Given some of the things that have happened around that country's involvement in tech in recent years, I don't think such scrutiny is unwarranted, especially when the team has a track record of security "goofs".