Heroku: We’ve Heard Your Feedback(blog.heroku.com)
blog.heroku.com
Heroku: We’ve Heard Your Feedback
https://blog.heroku.com/we-heard-your-feedback
150 comments
> 2. Customers sign up at "heroku.com", the platform is called "Heroku", the CLI is "heroku", everything's heroku, so don't send emails from a parent company (Salesforce), send them from "Heroku".
They're working on something called "Project Periwinkle" that is intended to remove all Heroku branding and make everything Salesforce branded. Periwinkle being a colour between blue (salesforce) and purple (heroku). No more Heroku signups, you'll need a Salesforce account to use it. No more free tier either.
Heroku has been in the process of being sunset for years now. New features have been banned for years. Only "keep the lights on" projects are allowed. Not that they could do anything with the skeleton crew they have running the platform.
Bob Wise's LinkedIn doesn't even mention Heroku, only Salesforce. Lenora mentions the project here: https://www.lenoraporter.com/portfolio/salesforce
Source: I'm a former employee that left in protest because of this project.
They're working on something called "Project Periwinkle" that is intended to remove all Heroku branding and make everything Salesforce branded. Periwinkle being a colour between blue (salesforce) and purple (heroku). No more Heroku signups, you'll need a Salesforce account to use it. No more free tier either.
Heroku has been in the process of being sunset for years now. New features have been banned for years. Only "keep the lights on" projects are allowed. Not that they could do anything with the skeleton crew they have running the platform.
Bob Wise's LinkedIn doesn't even mention Heroku, only Salesforce. Lenora mentions the project here: https://www.lenoraporter.com/portfolio/salesforce
Source: I'm a former employee that left in protest because of this project.
I work for a non-Heroku tech vendor, and I have always spoken highly of Heroku to my customers from my past experience of going all-in on Heroku, and receiving a top quality of support from Heroku back in 2017-2019. I'm going to point folks to the recent Hacker News threads now.
It is difficult for me to understand why Salesforce is not aware of the strength of the Heroku brand among experienced technology workers, and how much they have destroyed that brand in the last 2-3 years.
It is difficult for me to understand why Salesforce is not aware of the strength of the Heroku brand among experienced technology workers, and how much they have destroyed that brand in the last 2-3 years.
The amount of Salesforce fluff in the post is quite palpable. There is a clear lack of control with the leaders in Heroku judging off this post because of the salesforce transition.
This project Periwinkle sounds awful. Basically thats the end of using Heroku for us. If it remains like this its something to judge from.
This project Periwinkle sounds awful. Basically thats the end of using Heroku for us. If it remains like this its something to judge from.
There really aren't any Heroku leaders anymore. I think anyone director level or higher is Salesforce—not Heroku.
It's worth mentioning that this isn't a result of Salesforce acquiring Heroku. That happened 12 years ago when Heroku was next to nothing. Salesforce gets credit for investing in and making Heroku. Why they ultimately have decided to give up on it I have no idea. I hear it's because salespeople had a hard time understanding how to sell it which seems like a strange reason to give up entirely.
It's worth mentioning that this isn't a result of Salesforce acquiring Heroku. That happened 12 years ago when Heroku was next to nothing. Salesforce gets credit for investing in and making Heroku. Why they ultimately have decided to give up on it I have no idea. I hear it's because salespeople had a hard time understanding how to sell it which seems like a strange reason to give up entirely.
Salesforce is where good software goes to die (or at best become stagnant and lame). Same thing happened to quip and will happen to slack.
Oh man, I somehow missed Slack had been acquired by Salesforce. So many things make sense now!
I didn't know Slack had been acquired by Salesforce.
That's sad, I guess.
That's sad, I guess.
This happens after every single Salesforce acqusition. If you think you're different, compare how many new internal promotions there are, how many senior management have left and how many Salesforce transplants there are.
Everyone thought they were the darling child, at least for the first 18 months.
Everyone thought they were the darling child, at least for the first 18 months.
> because salespeople had a hard time understanding how to sell it which seems like a strange reason to give up entirely.
Yet unfortunately from my experience with large orgs and sales, the sales people get all the control and freely shit on the people actually making the software they sell.
Yet unfortunately from my experience with large orgs and sales, the sales people get all the control and freely shit on the people actually making the software they sell.
>I have no idea. I hear it's because
So you had an idea after all?
So you had an idea after all?
[deleted]
Periwinkle never materialized. Neither did Shinrai, or a number of other projects that people brought up in hn threads about this incident. It’s clear that Heroku has lost a ton of great talent and momentum due to questionable business decisions but some of the most controversial fail to ever make it past codename status.
It's on ice. As you say it's also not the first time they've attempted something like this.
Eventually they'll get there, but customers should know that's the direction. They'll be users of "Compute Cloud" writing APEX instead of Procfiles someday.
Eventually they'll get there, but customers should know that's the direction. They'll be users of "Compute Cloud" writing APEX instead of Procfiles someday.
[deleted]
> [..] No more Heroku signups, you'll need a Salesforce account to use it. No more free tier either.
Guess I'm packing it up as I'm no longer seen as target audience (its my hobby project platform)
> New features have been banned for years. Only "keep the lights on" projects are allowed.
What???
Guess I'm packing it up as I'm no longer seen as target audience (its my hobby project platform)
> New features have been banned for years. Only "keep the lights on" projects are allowed.
What???
> New features have been banned for years. Only "keep the lights on" projects are allowed.
That's what it's looked like from the outside -- that no features were for whatever reason(s) no longer going to happen. But still dismaying to hear it from the inside in those terms.
That's what it's looked like from the outside -- that no features were for whatever reason(s) no longer going to happen. But still dismaying to hear it from the inside in those terms.
You've gotta go way back on the Heroku Changelog to find anything that isn't a language version upgrade or feature removal: https://devcenter.heroku.com/changelog
I think the feature freeze happened in 2018
I think the feature freeze happened in 2018
The interesting thing is that heroku still is so at the top of what it does, as far as developer UX and ease and reliability. It's hard to describe exactly what i mean, but I know plenty of other people agree. Other things do other things better -- but it's only in the past year or two that some things have started to come close or equal.
I don't know what's going on exactly. Those who set up heroku for the first ~5 years somehow did such a good job that they could coast for another 5-10 and still stay on top of what they were on top of.
I don't know what's going on exactly. Those who set up heroku for the first ~5 years somehow did such a good job that they could coast for another 5-10 and still stay on top of what they were on top of.
Any suggestions on a rival PaaS that has a similar engineering philosophy to the old Heroku?
I was a long time Heroku user/lover (~10 years) for both personal and company projects. I've recently moved to Render[1], and so far it's been great. I haven't been using them long enough to put it through it's paces, but I'd certainly say it's worth a look. Other alternatives I've heard good things about are Fly.io[2] and Porter[3].
[1] - https://render.com/
[2] - https://fly.io/
[3] - https://porter.run/
[1] - https://render.com/
[2] - https://fly.io/
[3] - https://porter.run/
Checked out render.com in the context of side projects that I want on the air but don't expect anyone to ever use (I have so many on heroku, who got rewarded for their generosity with many thousands of dollars of business from me), which means I'm willing to pay a few dollars but not to pay per-app. Looks great except that they delete your free databases after 90 days, making it completely useless for this use case. Their announcement blog post says they plan to remove this limitation "early 2022", but it hadn't happened yet.
I next looked at fly.io, which seem ok for the first two apps, assuming they use the same database. If I ever want more than two apps, it seems I will need to start hosting different apps together, which is the opposite of the headache-free experience I'm looking for.
Porter runs on my own cloud account, so I can't trust it to not cost too much.
Maybe I can get a Kubernetes cluster somewhere (DigitalOcean?) and deploy all my small apps to it, but it sounds like a headache.
I'm staying with heroku.
I next looked at fly.io, which seem ok for the first two apps, assuming they use the same database. If I ever want more than two apps, it seems I will need to start hosting different apps together, which is the opposite of the headache-free experience I'm looking for.
Porter runs on my own cloud account, so I can't trust it to not cost too much.
Maybe I can get a Kubernetes cluster somewhere (DigitalOcean?) and deploy all my small apps to it, but it sounds like a headache.
I'm staying with heroku.
Dokku on top of a Digital Ocean droplet is a pretty cheap and easy option if you're OK with a single-server solution (i.e. small side project) and you're looking for something Heroku-like.
What about DO App Platform? I was thinking of trying that next. I’ve read a lot about slow build times and random build failures, but at the same time they at least seem to be actively developing the service.
Elsewhere in this thread is a link to a blogpost comparing it to doing things manually with GCP and quickly dismissing it as extremely limited, so I didn't delve further.
Heroku only gives you one free app too? Fly.io pricing is a lot cheaper. You can run a small service for $2/month whereas Heroku is $7 minimum.
If you want a more minimal do-it-yourself option you could self-host something like Dokku[1].
[1] https://dokku.com/
[1] https://dokku.com/
I feel confident nobody using heroku is doing it because they want a more minimal do-it-yourself option.
fly.io is an excellent alternative. I am slowly migrating all of my clients over.
Cloud Foundry maybe? Some companies are offering it as a service (e.g: IBM Cloud Foundry)
There's Clever Cloud which I use personally
Youch.
I have paid apps and free apps (staticman comment processing, and very simple apps I create while following learning tutorials). This project sounds bad! I’ve been slowly exploring alternatives, and was about to abandon my search due to demands on my time. Guess I need to keep exploring.
I have paid apps and free apps (staticman comment processing, and very simple apps I create while following learning tutorials). This project sounds bad! I’ve been slowly exploring alternatives, and was about to abandon my search due to demands on my time. Guess I need to keep exploring.
Well, that’s sad. I guess it’s Fly or Digital Ocean for all future projects then.
There is some comparison of DO's App platform and Google's Cloud offerings here if you are interested (not my blog, just found it helpful: https://blog.verygoodsoftwarenotvirus.ru/posts/greener-cloud...)
Don’t forget render.com
When I got an email from Salesforce talking about Heroku I assumed phishing. Huge +1 on that 2nd point :)
re: 2
interestingly, when i log in or reauthenticate, or log in through the heroku cli, I get sent to https://verify.salesforce.com/v1/verify/
interestingly, when i log in or reauthenticate, or log in through the heroku cli, I get sent to https://verify.salesforce.com/v1/verify/
When I saw the Salesforce email I thought I was being spammed or phished or getting a job offer.
Those are all the same thing
This all somehow seems unsurprising. If anyone has tried to use Heroku in easily the last year, the number of times you get failed builds over really trivial things is noticeable.
I haven’t experienced a failed build in a few years.
I did try a heroku competitor recently and my builds failed. And there was no detailed log to show why. So I couldn’t troubleshoot it and I immediately gave up!
As an aside, I was really hoping this post was going to be “we read that one competitor’s blog post last week clearly detailing all the areas where we can improve, and we HEAR you.”
I did try a heroku competitor recently and my builds failed. And there was no detailed log to show why. So I couldn’t troubleshoot it and I immediately gave up!
As an aside, I was really hoping this post was going to be “we read that one competitor’s blog post last week clearly detailing all the areas where we can improve, and we HEAR you.”
Considering how badly Salesforce handled communication and bug fixing of log4shell in Tableau that's not really surprising.
“I have a lifelong enthusiasm for developers and the experience they have building software together”
And then drops a link to contact them, via LinkedIn…
LinkedIn is the polar opposite of GitHub. It’s the worst example of social media, from its news feed, to spam invites. And it’s broken every rule in the “be a good netizen” play book, from constant spam, to slurping your email contacts and surveillance to the extreme.
I struggle to imagine a developer saying “I’d like to contact xxx, and I’d love to do it via LinkedIn”
Why not drop your email? Or a GitHub profile with a public email, and readme containing other contact methods, would have been more dev centric.
And then drops a link to contact them, via LinkedIn…
LinkedIn is the polar opposite of GitHub. It’s the worst example of social media, from its news feed, to spam invites. And it’s broken every rule in the “be a good netizen” play book, from constant spam, to slurping your email contacts and surveillance to the extreme.
I struggle to imagine a developer saying “I’d like to contact xxx, and I’d love to do it via LinkedIn”
Why not drop your email? Or a GitHub profile with a public email, and readme containing other contact methods, would have been more dev centric.
It hadn't really clicked until your comment that both Github and LinkedIn are owned by the same company.
True, but both of them have stuck to their original objectives even now. GitHub at doing things Devs like and LinkedIn at doing things HR likes.
WaPo and AWS are both owned by the same person. That doesn't mean both (have to) charter in the same territory.
WaPo and AWS are both owned by the same person. That doesn't mean both (have to) charter in the same territory.
I agree with you about LinkedIn. I also would have agreed with you about how silly that is, until I started working with giant banks and other huge finance industry people. Many of the devs there legit do use LinkedIn. A number of people actually blog and post updates and stuff on LinkedIn. It was really surprising. Startup culture and big corporate tech, despite using many of the same technologies, are miles apart culturally.
Devs use LinkedIn for blog posts? Are they using it as way to get other devs to read it, or as a way to get potential employers to notice them?
Tbh, I don't a single dev who uses LinkedIn unless they looking for a new role I.e they're certainly not using LinkedIn as a dev-oriented news feed
Tbh, I don't a single dev who uses LinkedIn unless they looking for a new role I.e they're certainly not using LinkedIn as a dev-oriented news feed
Yes, surprisingly many of the devs used LinkedIn for blog posts. The type of blog posts that they did were somewhat different as well. It was normal to see posts that had to do with engineering culture and how to structure an engineering department, things like that. They would also often talk about technology from a high level. For example "why I think vert.x is the future of Java high performance programming", or something like that.
I assume it’s so that they can mine your LinkenIn connection data and make it available to their sales teams to upsell you and people you know. I’d imagine they would end up storing all that data in some kind of web-based CRM, possibly sold as a SAAS product.
> LinkedIn is the polar opposite of GitHub. It’s the worst example of social media, from its news feed, to spam invites.
I agree with the general sentiment of your comment, but LinkedIn is the worst? Really?
I can think of a couple social media platforms that add less value to society.
LinkedIn has helped democratise the recruitment landscape significantly across many parts of Africa; we rely on it extensively.
Not wanting to pick a fight here, but that’s one helluva hyperbole I just couldn’t let slide ^_^
I agree with the general sentiment of your comment, but LinkedIn is the worst? Really?
I can think of a couple social media platforms that add less value to society.
LinkedIn has helped democratise the recruitment landscape significantly across many parts of Africa; we rely on it extensively.
Not wanting to pick a fight here, but that’s one helluva hyperbole I just couldn’t let slide ^_^
Email address is good but also an issue tracker or better yet IRC or something public would be good.
Agreed, for the specific issue. But it seems like he’s asking ppl to contact him more generally.
It does pose the question, what is the most developer friendly contact method??
GET /contacts
POST /message
Interested to hear ideas.
It does pose the question, what is the most developer friendly contact method??
GET /contacts
POST /message
Interested to hear ideas.
I guess it depends on what "developer friendly" means. If it means 'easiest method for a developer to contact me' that's email. It's ubiquitous. Everyone has at least one. It's easy.
If it means 'easiest method for only developers to contact me' that would probably be some documented REST API.
Either way, I agree the answer isn't Linkedin.
If it means 'easiest method for only developers to contact me' that would probably be some documented REST API.
Either way, I agree the answer isn't Linkedin.
Email address and / or GitHub. No one's saying he needs to show off what a great programmer he is through his contact methods, just don't use a site that's infamously bad for dark patterns and recruiter spam.
As a former herokai, communication was always the #1 point of discussion internally. This idea that they "can" do better is a half truth. They DID do better. I leave it as an exercise to the reader to determine what the limiting resource was here.
By the way, how many senior devs and cofounders are left at Heroku Bob? Why doesn't it show up at dreamforce anymore?
By the way, how many senior devs and cofounders are left at Heroku Bob? Why doesn't it show up at dreamforce anymore?
(Former) Salesforce employee here, long enough to have been there a couple of years before the Heroku acquisition. I can't let this stand without a comment.
Heroku never seemed to want to integrate, but instead be the "cool kids", those who just do not have to worry about the enterprisy stuff such as automated backups, DR, high availability, enabling Java, etc. Everything they did was great, everything "Salesforce" sucked, yet they never made any money before the acquisition. The acquisition was a waste of money in first place (IMHO).
The founders did not even try and so they left.
Heroku never seemed to want to integrate, but instead be the "cool kids", those who just do not have to worry about the enterprisy stuff such as automated backups, DR, high availability, enabling Java, etc. Everything they did was great, everything "Salesforce" sucked, yet they never made any money before the acquisition. The acquisition was a waste of money in first place (IMHO).
The founders did not even try and so they left.
Trying to integrate with SFDC was a mistake. The product never fit inside of the company. Of course their investment made Heroku successful over the first few years (and eventually highly profitable as I understand) but they should've just spun it off and let it succeed without trying to figure out how to "enterprise" it.
What we knew how to do at Heroku was build a great platform for developers to launch apps. We never claimed to know how to make that model work for enterprise. In fact we were awful at trying.
What we knew how to do at Heroku was build a great platform for developers to launch apps. We never claimed to know how to make that model work for enterprise. In fact we were awful at trying.
Former Heroku employee here: everything you list happened within the first year of acquisition.
You’re correct that the lack of real integration with SFDC was a large part of the problem though. Though as far as SFDC acquisitions go Heroku was cheap. More expensive ones had far worse outcomes even with integration.
You’re correct that the lack of real integration with SFDC was a large part of the problem though. Though as far as SFDC acquisitions go Heroku was cheap. More expensive ones had far worse outcomes even with integration.
> Heroku never seemed to want to integrate, but instead be the "cool kids", those who just do not have to worry about the enterprisy stuff such as automated backups, DR, high availability, enabling Java, etc.
Seems like Salesforce got exactly what they bought. Not sure why they’d be surprised about it.
Seems like Salesforce got exactly what they bought. Not sure why they’d be surprised about it.
I knew the writing was on the wall for Heroku a few years back. I had asked a group of Salesforce sales reps about Heroku Connect (two-way sync between postgres and salesforce - amazing!), and they had no absolutely no idea what it was. They checked back with their team and weren't able to get any information. We basically couldn't buy it.
That's crazy! I work with Salesforce reps a lot for some pre-sales bids and they've been pushing connect (both salesforce connect and heroku connect) for the last few years. Cost is in the range of 5k/m or so for salesforce connect (from what I recall).
Is Matz still there?
Matz might have been on the payroll, but he never really worked for Heroku.
Sure, that was always clear enough. I'm asking whether or not the arrangement continues.
Heroku is a one of kind platform, many tried to replicate, none have come close, the ability to setup a whole app with a few clicks on dynos and add-ons is great. The price was always salty $50/month for 1gb RAM shared CPU, and in the few past years no newly released features comes to mind. As a customer I have no idea what this security issue impact, the communication has been poor as this post does not clarify much, you have to click a status post link to see a feed of what's going on.
Sorry but Render at their lunch entirely. Try it. https://render.com/docs/deploy-rails
Want simple? Use a build.sh
Want a bit complexity? Use a Dockerfile.
Their docs are really copy paste for 99.9% applications. Whoever in that company made the decision to invest in docs was right on the money. It made switching to them a much easier choice. https://render.com
Want simple? Use a build.sh
Want a bit complexity? Use a Dockerfile.
Their docs are really copy paste for 99.9% applications. Whoever in that company made the decision to invest in docs was right on the money. It made switching to them a much easier choice. https://render.com
I'm moving a small amount of stuff ($250/mo spend) off Heroku this weekend. It's either Render or Fly, and I guess I'm going to start with Render.
Here's where I'm starting: https://render.com/docs/migrate-from-heroku
Here's where I'm starting: https://render.com/docs/migrate-from-heroku
Update: Migrating the simplest possible Rails app from free-tier to free-tier didn't work on Render (deploys fail, no logs to be found), so I'm bailing out to try Fly.
To any render people reading this, it appears to have been an issue with the migration plugin and the Heroku-18 stack. Manually deploying the app worked fine.
To any render people reading this, it appears to have been an issue with the migration plugin and the Heroku-18 stack. Manually deploying the app worked fine.
Thanks for the heads up and good to hear you were able to deploy the app manually. We might deprecate the migrator because it's hard to keep up with all the Heroku stack and buildpack configs and because manual migrations are relatively straightforward.
Completely understandable. I’ve been configuring stuff with Render all morning and I’m really starting to like this service.
Is there a way to assign an existing Env Group when manually creating a new service? So far I've needed to create and then go back in and assign.
Is there a way to assign an existing Env Group when manually creating a new service? So far I've needed to create and then go back in and assign.
Thanks! Re: Env Groups, not yet, but we are overhauling the new service flow and will definitely incorporate adding env groups.
Update for fairness: Fly.io failed even quicker.
Creating postgres cluster redacted-db in organization redacted Launching...⣻ Error failed creating the Postgres cluster redacted-db: Timeout on CreatePostgresCluster.app
Every attempt fails. There's no way I can run this for $250/mo on AWS. Now I have to pretend I'm a sysadmin and use Hetzner.
Creating postgres cluster redacted-db in organization redacted Launching...⣻ Error failed creating the Postgres cluster redacted-db: Timeout on CreatePostgresCluster.app
Every attempt fails. There's no way I can run this for $250/mo on AWS. Now I have to pretend I'm a sysadmin and use Hetzner.
> Whoever in that company made the decision to invest in docs was right on the money.
Also the reason Django has always been so approachable (particularly in the early days, as it is orders of magnitude more complex now) and therefore popular
Older still, IMHO also why mIRC scripting was so much fun despite its shortcomings. The help file had literally everything you could want to learn about the language and it almost begged to be read. It's how I got into coding
Also the reason Django has always been so approachable (particularly in the early days, as it is orders of magnitude more complex now) and therefore popular
Older still, IMHO also why mIRC scripting was so much fun despite its shortcomings. The help file had literally everything you could want to learn about the language and it almost begged to be read. It's how I got into coding
+1, Django docs were gold standard when I last used it. Also DRF. I still miss them, 5 years later.
I hadn't heard of Render prior to this incident, but now I see it recommended in almost every related thread. Of course, Heroku's free tier had always worked well enough for my hobby apps, so I had no reason to search for alternatives. Looking forward to giving Render a try.
Posting this at 5pm on a Friday sounds about right, yeah?
That isn’t a very charitable take. It’s been 24x7 for them, the acknowledgement is most welcome, in my opinion. I’m not sure waiting until Monday 9AM Pacific just to avoid these type of comments would be the right choice. The internet is 24x7 after all.
There doesn't seem to be anything different now than 6 hours ago. So posting it after 8pm EDT/5pm PDT Friday seems very intentional.
With what intention? Honest question, I don't see a strategic advantage in posting this on a Friday. More of a disadvantage, actually, with less visibility.
The point is less visibility/criticism. Since this is confirming pretty bad news (the attackers had access to the environment variables.) The concept of a Friday night news dump is a fairly well-worn, but basically true, trope.
> With what intention? Honest question, I don't see a strategic advantage in posting this on a Friday. More of a disadvantage, actually, with less visibility.
The intention presumably is to be able to say they communicated this while (intentionally) limiting the amount of awareness that's spread about this.
The intention presumably is to be able to say they communicated this while (intentionally) limiting the amount of awareness that's spread about this.
[deleted]
With any hope it will slide in the news cycle and be buried by any developments over the weekend.
[deleted]
[deleted]
[deleted]
> I started as Heroku GM a few weeks ago with intense enthusiasm to be a part of such a storied team.
Wow talk about terrible timing.
Wow talk about terrible timing.
Well, it's a great timing.
Fail it's because I am new; Succeed then it's my superior capability; plus, the learning rate during crisis is at least 10 times higher than the peaceful time, let alone the nerves and mental fortitude where peaceful time would never train one for.
Fail it's because I am new; Succeed then it's my superior capability; plus, the learning rate during crisis is at least 10 times higher than the peaceful time, let alone the nerves and mental fortitude where peaceful time would never train one for.
Maybe he's the guy who posted a couple of days ago about pulling a disk out of a RAID array on his first day in a leadership position...
Let’s give Bob a chance. I worked with him at AWS and prior. He is very committed to the customer, hence his post. We all try to communicate the best we can, so please try to support him in his new role.
I launched several startups on Heroku over the last decade +, and feel they have gravity on trust. Some of the best devs I have worked with.
I launched several startups on Heroku over the last decade +, and feel they have gravity on trust. Some of the best devs I have worked with.
> Additionally, we have no evidence that the attacker has accessed any customer accounts or decrypted customers’ environment variables.
Now the attackers had access to encrypted environment variables?
Now the attackers had access to encrypted environment variables?
More context, from the latest email notification:
> We also wanted to address a question regarding impact to environment variables. While we confirmed that the threat actor had access to encrypted Heroku customer secrets stored in config var, the secrets are encrypted at rest and the threat actor did not access the encryption key necessary to decrypt config var secrets.
> We also wanted to address a question regarding impact to environment variables. While we confirmed that the threat actor had access to encrypted Heroku customer secrets stored in config var, the secrets are encrypted at rest and the threat actor did not access the encryption key necessary to decrypt config var secrets.
Where do I find those notifications? I only got a single email, stating I should "reset my user password as mentioned in our previous notification". Nothing prior, nothing since, and their password reset form shows nothing but "Internal Server Error". I've been getting some details from HN but I think I'm missing a lot of the picture since they obviously don't send everyone every email.
> and the threat actor did not access the encryption key necessary to decrypt config var secrets.
If the threat actor had access to any of the systems that use the key, they may not have needed to. Even this statement isn't clear that they couldn't have done it, but suggests that they don't think it's true...
This is really bad incident response messaging.
If the threat actor had access to any of the systems that use the key, they may not have needed to. Even this statement isn't clear that they couldn't have done it, but suggests that they don't think it's true...
This is really bad incident response messaging.
Sorry, but this is a joke of a response.
When they started sending out password reset emails, they should have explained why.
Not only when people started complaining, and the media picked up on the lack of transparency.
When they started sending out password reset emails, they should have explained why.
Not only when people started complaining, and the media picked up on the lack of transparency.
It's ok: Bob says "Trust as our #1 value".
can somebody fill me in what happened? is this related to the oAuth vulnerability through github a while back?
They got hacked about a month ago and have been extremely cagey about what exactly was hacked ever since.
Turns out the master database with encrypted username/passwords got leaked and encrypted environment variables were also leaked but it was like pulling teeth to get them to answer whether or not these happened or even admit that it might have been possible. Presumably more than this was also leaked but so far they haven't said anything on that. Env vars were the biggest concern on everyone's mind.
They gave the absolute least amount of information over the longest period they could muster.
The problem wasn't the hack really, it was the lack of transparency in the response.
See: https://twitter.com/jacobian/status/1522782890957819906
Turns out the master database with encrypted username/passwords got leaked and encrypted environment variables were also leaked but it was like pulling teeth to get them to answer whether or not these happened or even admit that it might have been possible. Presumably more than this was also leaked but so far they haven't said anything on that. Env vars were the biggest concern on everyone's mind.
They gave the absolute least amount of information over the longest period they could muster.
The problem wasn't the hack really, it was the lack of transparency in the response.
See: https://twitter.com/jacobian/status/1522782890957819906
> Turns out the master database with encrypted username/passwords got leaked and encrypted environment variables were also leaked
WTF!!!!
That alone is disastrous enough, they should be reprimanded for this. Are there I'm sure, class action lawsuits happening?
How much of an impact will this have on Salesforce? I mean imagine the data from that alone would be immensely valuable.
WTF!!!!
That alone is disastrous enough, they should be reprimanded for this. Are there I'm sure, class action lawsuits happening?
How much of an impact will this have on Salesforce? I mean imagine the data from that alone would be immensely valuable.
I should've said "hashed" not encrypted passwords. But the env vars are the real problem. They haven't categorically dismissed the hacker somehow getting access to the actual environment variables either. Only said there isn't evidence of that happening.
If it comes out that the hacker did get to unencrypted env vars I think it's game over for Heroku. Nobody should trust them with sensitive data.
If it comes out that the hacker did get to unencrypted env vars I think it's game over for Heroku. Nobody should trust them with sensitive data.
> we have no evidence
It means they don't know. They must say "we are confident" otherwise.
It means they don't know. They must say "we are confident" otherwise.
Heroku has always been the opposite of transparent.
I still don't forget how they handled the Intelligent Routing fiasco from back in the day.
I still don't forget how they handled the Intelligent Routing fiasco from back in the day.
>The Heroku team and their colleagues have worked around the clock, including nights and weekends
Can someone more familiar with an event like this tell me what they are working so hard on? I imagine securing the vulnerable service and resetting various credentials doesn't take that much work.
Can someone more familiar with an event like this tell me what they are working so hard on? I imagine securing the vulnerable service and resetting various credentials doesn't take that much work.
You have to make sure the attackers aren’t still in your network, you have to get them out if they are, you have to fully scope out what they messed with, and restore anything that’s plausibly connected. Even in the best case with all the right monitoring systems in place there is a lot of manual work involved from owners of the various different effected services and just managing the overall response adequately. I would say the amount of effort involved is comparable to managing a novel, ongoing SEV1.
Additionally there are legal concerns, for both regulatory compliance and for prepping for inevitable litigation. Those concerns aren’t necessarily a blocker for service restoration, but it really depends on the systems involved. If service could be restored by rebooting a system, for example, but that system also has data related to customers’ (and their customers’) PII and it might have been accessed by the attacker, then you need to make sure it’s all properly preserved forensically first, so that you can comply with regulations regarding breach notifications. The forensic analysis could then happen, but it’s definitely a “measure twice, cut once” situation with lots of lawyers involved (they won’t understand the systems, but they’ll make you explain everything so they can make decisions about risk; and they are in charge).
Also, generally, it’s a “fog of war” scenario, where you can have so many unknowns to work through in a compressed time period, and sometimes there’s an active attacker and they get a vote, too.
Also, generally, it’s a “fog of war” scenario, where you can have so many unknowns to work through in a compressed time period, and sometimes there’s an active attacker and they get a vote, too.
One quick example: combing through logs which can go back months or years can be expensive in time, especially if those are stored in something like Glacier and need to be pulled back into a system which permits analysis.
Proper incident response involves a lot more than just "securing the vulnerable service and resetting various credentials" and it will in this case probably involve a lot of different teams.
For starters, you have to document everything. At the very minimum the legal team for the company should insist on this, if no other measures, just as a CYA move.
If an attacker had broad access, it's entirely possible an all hands on deck approach is required to help identify (and document) what systems were compromised. Yes, you definitely want a team working on patching the hole ASAP. You also need a team hunting for any possible persistence. Another team probably involving standing up brand new "safe" systems and failing over client systems over to those running the patched software. While that happens yet more people may start doing audits of what was compromised on the original systems.
I've seen incidents where 50-80 people were pulled in to work on an incident at a company of about 150 employees. Depending on how well-funded your SoC is they can cut that number down substantially.
For starters, you have to document everything. At the very minimum the legal team for the company should insist on this, if no other measures, just as a CYA move.
If an attacker had broad access, it's entirely possible an all hands on deck approach is required to help identify (and document) what systems were compromised. Yes, you definitely want a team working on patching the hole ASAP. You also need a team hunting for any possible persistence. Another team probably involving standing up brand new "safe" systems and failing over client systems over to those running the patched software. While that happens yet more people may start doing audits of what was compromised on the original systems.
I've seen incidents where 50-80 people were pulled in to work on an incident at a company of about 150 employees. Depending on how well-funded your SoC is they can cut that number down substantially.
I imagine they are examining their whole attack surface area. Probably bringing in consultants to discover what could possibly be attacked.
Besides that, lots of incident response work to figure out the scope and impact of the breach, I'd imagine.
[deleted]
They're working on their optics, amigo
I moved over to Render and cancelled Heroku. I didn't mind it before because it was reliable (if not expensive) but this Github fiasco made me move. Happy to get rid.
If you're on Heroku how do you deploy now? It says it's still weeks away before you can deploy - did I read this right?
Still weeks away from enabling the GitHub integration again, you can still manually deploy via the heroku CLI.
Our team realised heroku was going to take weeks and we’ve replicated our “review app” development workflow with GitHub actions that clone apps on PR, push code and rebuild them on push and destroy them on PR close. It’s not as seamless as the heroku GitHub integration but it’s good enough for now.
Our team realised heroku was going to take weeks and we’ve replicated our “review app” development workflow with GitHub actions that clone apps on PR, push code and rebuild them on push and destroy them on PR close. It’s not as seamless as the heroku GitHub integration but it’s good enough for now.
You don't even need the "heroku cli" to deploy -- just git. That was heroku's original famous innovation, you may recall!
Woah… this would be an amazing thing to open source
Same way we’ve done for years, via our CD (circle CI), which uses the Heroku CLI
We’ve been lucky that this entire event has, theoretically, not touched us as we never connected GitHub. That may change as more information comes to light.
We’re still strongly considering moving to AWS, and are in the process of getting quotes from vendors.
We’ve been lucky that this entire event has, theoretically, not touched us as we never connected GitHub. That may change as more information comes to light.
We’re still strongly considering moving to AWS, and are in the process of getting quotes from vendors.
Are you thinking AWS, or a provider like Render, etc?
We're looking at a simple Lift & Shift to AWS. We're building our next product on AWS, so makes sense to move everything there, and the agency that we have doing the 24 hours monitoring runs a bunch of rails apps on their AWS stack so they've got everything in place to do it pretty simply.
You can deploy, you just can't trigger the builds automatically by pushing to Github. (This also means the Review Apps system is non-functional.)
Deploy using heroku CLI: https://devcenter.heroku.com/articles/git#deploy-your-code
Unfortunately, git lfs isn't supported for CLI pushes, haven't found a way that works yet.
Unfortunately, git lfs isn't supported for CLI pushes, haven't found a way that works yet.
git push heroku master
Glad I didn’t even realize there was github integration!
Glad I didn’t even realize there was github integration!
Curious why they would need to keep the GitHub integration down for a while longer. Are they afraid someone could still grab credentials?
GitHub is not allowing them to turn it back on yet (source: multiple people on both sides)
GitHub should insist on it being converted to a GitHub App before they allow it to be re-enabled.
That way org admins can see the requested permissions and control exactly which repos are permitted. GitHub OAuth apps are an absolute nightmare to audit or control.
That way org admins can see the requested permissions and control exactly which repos are permitted. GitHub OAuth apps are an absolute nightmare to audit or control.
They were using a legacy GitHub integration. Maybe it's inherently unsafe and they need to rewrite things using the new API.
Audit?
The author seems like they really care and are competent.
They should go work for render.com
They should go work for render.com
Too little, too late
Just cut the fucking marketing corpospeak, please, we're begging you. This post is just saturated with it.
So I looked at their product offerings and the Wikipedia page about Heroku and I'm still left scratching my head as to what it actually even is. They basically rent out other cloud computer providers' time at a higher price and make you use some kind of shitty middleware? Is that all this actually is? Please help me to understand what in the hell this even is.
You broadly have it right, but it's not "shitty" middleware, despite their problems the actual process of launching an app on Heroku is stellar and was especially groundbreaking 10 years ago. It would provision a server, install dependencies, configure, then launch your app just by pushing a git repository to them.
They were the gold standard for rails deployments back in the day when rails was popular—which was important since rails was actually pretty hard to host on a server compared to say PHP.
They were the gold standard for rails deployments back in the day when rails was popular—which was important since rails was actually pretty hard to host on a server compared to say PHP.
Well, those, and:
1. Speed. It took days for heroku customers to be told about this.
2. Customers sign up at "heroku.com", the platform is called "Heroku", the CLI is "heroku", everything's heroku, so don't send emails from a parent company (Salesforce), send them from "Heroku".
3. Unambiguous info on what customers need to do. I had to guess based on HN comments whether config vars were accessed. Config vars are 100x more sensitive than code. Comms should be unambiguous and complete, and if incomplete for any reason, explain that (e.g. we don't know yet).
4. I still don't know whether having 1 Github Deploy on my Heroku account allowed unauthorized access to all heroku applications on my heroku account (i.e. those using other deploy methods, like `git push heroku main`). Were all my apps' repositories able to be accessed, or just the one(s) deployed via Github Deploys?
5. I still don't know whether unauthorized access was gained to all other GitHub respositories on my GitHub account, i.e. the repos that aren't heroku apps.
These said, I still really appreciate that security incidents happen and aren't easy to deal with, and there's no obligation for anyone at a profitable company to actually care about semi-captive customers, so thanks to Heroku for the efforts; it's genuinely appreciated.