"I'll just do this small release on a Friday before heading out" - thought a software engineer before AI-slopification period. Today it was probably some AI agent which fixed all of the performance issues by deleting everything.
I've also used Netdata for years on many servers now. One of the problems with it is its way to fail silently when some healtcheck or alarm is configured badly instead of erroring out on the process start. I understand the reasoning behind it - it tries to monitor by default as much as possible and if some collector fails for whatever reason (let's say you don't have MySQL installed) then it will be just disabled and only way to know that is by looking at the dashboard and not finding it there. I'm only using local agent and not the cloud which means that if server dies totally then there's also no monitoring insights - something to keep in mind.
Creating complex software is not at all complicated, and sometimes it is done right at the beginning of a project, even before solving a single business problem.
Double-checked - no e-mails prior 18th of August from "Google Play Console". Also, double-checked "Google Play Console" (even under "Archived messages") - first message about deprecation I see is from 17th of August (a day before e-mail). Even if there were something a year ago, then 3 month, 1 month followup would make sense instead of 2-3 week reminder.
Maybe it's possible that different roles in "Google Play Console" will receive different type of messages at different time? I'm not an admin for that app and it's possible that someone else has gotten prior warnings, but might have ignored these, because usually they are not with technical background.
But this is also kind of a problem created by Google - why force from 30 to 33 and not force from 30 to 31, then 31 to 32 etc? Gradual API level upgrades would make more sense in my world and probably would be less risky for every party.
What do you mean by that? Are you effectively trying to say that allowing upgrades does not have any risk of attack surface? I'm pretty sure that updating things have also a pretty high risk on introducing new previously non-existing security issues into your code-base/product.
I'm the OP and thank you for thinking along with me here. As stated in numerous replies already then I totally agree that I could have done better in terms of testing things out - of course, there's always room for improvement in that regard. There was a deadline set by Google (again, first time I heard about it was at 18th of August, not before), change seemed trivial at time and since app worked on an old Android version as it was before then I didn't expect it to fail so miserably. Again, I'm not a seasoned Android dev, but have 15+ years of experience in software development in general so I have some expectations how things will work or not and what to expect and be afraid of. I really didn't know that "best practice" is to do a staged roll-out of "99.99999999%" to have a way of partial "yank" possibility of the latest release. To find out that there's no way to cancel/delete a latest release to fall back to previous working version was just something I did not expect in my wildest dreams (I guess this is something you only learn during situations like these). Yes, everyone can blame me for not testing every functionality with every Android version and I do the same, but please open your eyes and understand that the way releases are currently handled by Play Store is not a sane person would do outside of Play Store. Everyone will have a problem like this at one point and I do hope that this article will and thread in here will lower the number of developers experiencing situation similar to this.
I'm the OP and wanted to clarify in case you missed some points - it is a legacy application which does not have any active dev teams on it and needs only developers attention when Google says so and as mentioned by multiple other commenters here the first time I got that e-mail from Google, was at 18th of August. I would not agree that I have been lazy, but instead trying to solve this problem in the time-constraints set by Google and failing to do so because of the inability to put a fix to production and/or pull back current release version. Of course I admit that there's always ways to improve quality assurance.
There has been zero communication towards me from Google until two weeks until deadline. Yes, maybe if I would have logged into Play Console then there might have been some notifications, but there have been no reason to do so until that e-mail (I'm usually not involved with Android projects, otherwise I might have noticed similar warnings via other projects early on).
Actually nowadays it's not that bad anymore. Android browser itself offers installable PWA and there is an event called beforeinstallprompt event (https://developer.mozilla.org/en-US/docs/Web/API/Window/befo...), which can be used to perform PWA installation on user interaction. Of course it's not supported in every browser.
iOS is more difficult since user needs to understand that "saving to home screen" is same as installing "app" and there's no way to trigger it programmatically or help user in any other way than with visual illustrations.
Typical "legacy application situation where no team is assigned to" here. At least I've never seen in my 15+ year career a legacy application, which has a very good maintenance/testing model in place. It will just not work because maintenance/testing needs also resources and updates, but if priorities are in different places then it's not possible. Of course as stated already multiple times - things could have handled in better ways, but this doesn't mean that Google should not allow to stop release at least. Anyway, lessons learned.
It's true that in the future I would do some more thorough testing, but never-ever for this legacy application can I build a bullet-proof automated testing solution - there will not be a budget for that for sure. However, even with a fancy solution mistakes will happen and you still can't stop release propagating. It's just a matter of time when it happens.
Totally understand that a roll-back is more complex, but it doesn't explain why pulling back/pausing/deleting/yanking a release is not possible either - customers who got the newest faulty release can uninstall & install to get the previous version back (no need to worry about incompatibilities here) and the ones who did not get the updated version yet can stay using the old one until a new fixed version will be released.
A-ha - good tip about roll-out, uninstall & install. Was not aware of that possibility. But yeah, it's still a hack and there should be a better way - just let me delete/cancel latest release even if roll-out is 100% for whatever reasons.
About better testing - there's always room to improve testing, but no way it's going to happen with a legacy application where no active team is assigned. Only these irregular updates mainly forced by Google are done. Unfortunately.
I agree that things could have been done better in many ways. However, as explained, it is a legacy application, which does not see any active development nowadays and it would just not make sense to build such a robust QA. This particular app have been written long time ago by another company and there's not even simple unit tests. That's the hard truth. But still, even as complex setup as you have, there's still going to happen mistakes and the real problem is that there is no way to pull-back/cancel/rollback release.
How would staged roll-out help in this situation for all customers? When end-user gets the faulty version of the app, does he/she have a way of getting the non-faulty version somehow?
Agreed about conflicting message. And even better - e-mail was pretty vague and had to Google a separate link (https://support.google.com/googleplay/android-developer/answ...) for more info. If I could go back in time then I would press that button, which asks for time until November, because as happened to us - it was not possible to go back to previous version in any way.