Microsoft Ubuntu repositories are broken because of space issues(github.com)
github.com
Microsoft Ubuntu repositories are broken because of space issues
https://github.com/dotnet/core/issues/6381
70 comments
/etc/apt/trusted.gpg.d is being deprecated also in favour of signed-by=/usr/share/keyrings/keyfile.gpg
https://www.linuxuprising.com/2021/01/apt-key-is-deprecated-...
https://www.linuxuprising.com/2021/01/apt-key-is-deprecated-...
Thanks for reminding me. I really wish apt-secure(8) and apt-key(8) would mention this!
I have zero trust on how they handle this, judging by your post it is justified.
This is why I decided to bite the bullet and go flatpak for the likes of teams, slack and other proprietary software.
This is why I decided to bite the bullet and go flatpak for the likes of teams, slack and other proprietary software.
> This is why I decided to bite the bullet and go flatpak for the likes of teams, slack and other proprietary software.
I always recommend this now if it is an option. Even if Microsoft does nothing wrong, it is way too easy for me to make an error and somehow enable Microsoft antivirus stuff on my Fedora machine which I still don't know how I managed to do.
I always recommend this now if it is an option. Even if Microsoft does nothing wrong, it is way too easy for me to make an error and somehow enable Microsoft antivirus stuff on my Fedora machine which I still don't know how I managed to do.
Flatpak is best iff the vendor is producing the packages. I have found all too often than I'm stuck on an outdated version of a orphaned, unofficial pacakge...
Still I wonder if you'd author all non-proprietary packages how many of these violations you'd find. Especially in smaller projects. Don't have any examples ready since I also try to avoid anything but established software, but I'm certain in the past I've had issues like mentioned above, even persisting after removal of the package.
>explicitly changes /usr/share/teams/chrome-sandbox to be setuid
These kinds of hacks are pretty consistant with my few experiences with Microsoft (and other vendor's) closed software on Linux. It's all best avoided for community maintained solutions.
These kinds of hacks are pretty consistant with my few experiences with Microsoft (and other vendor's) closed software on Linux. It's all best avoided for community maintained solutions.
What SLA are Ubuntu mirrors held to? This doesn't seem great, but won't a quick run of mirrorselect remove the offending repos and let you get on with life?
For your home setup: sure, it should be resolved soon, not too big of an issue.
What's much more annoying is that this breaks CI/Docker builds. Right now you can't deploy new container images that depend on a MS repository. I hope it's resolved soon.
What's much more annoying is that this breaks CI/Docker builds. Right now you can't deploy new container images that depend on a MS repository. I hope it's resolved soon.
We had the same problems already in the 90s and resolved them by caching, it's unthinkable that the whole build is broken by a problem with an external dependence - if you assume everything will be 100% available you basically make sure a problem will happen.
I wrote "hashcache" [0] [1] for this for my Linux distribution (which is the build from source variety) to avoid issues with upstream going away, or worse changing the contents of a file.
Many different caches can be used, in case one is down as well.
Simple example download script at [2].
[0] http://hashcache.rkeene.org/ [1] https://chiselapp.com/user/rkeene/repository/hashcache/ [2] https://kitcreator.rkeene.org/fossil/artifact/d546c4b48dfce8...
Many different caches can be used, in case one is down as well.
Simple example download script at [2].
[0] http://hashcache.rkeene.org/ [1] https://chiselapp.com/user/rkeene/repository/hashcache/ [2] https://kitcreator.rkeene.org/fossil/artifact/d546c4b48dfce8...
https://wiki.debian.org/AptCacherNg still exists and still works fine.
Aptly [0] is also really nice for maintaining mirrors and one's own Apt repositories though it is unmaintained.
[0]: https://github.com/aptly-dev/aptly
[0]: https://github.com/aptly-dev/aptly
Not caching... mirroring or vendoring.
As the other comment says: cache it yourself. Not only will you save time downloading it, avoid sudden version changes, your build environment should really not depend on pulling things out of your control every single time
The modern argument would be that you're more likely to have problems with your own cache than someone else, and you should just use caching as a service.
It's a lot easier to remove yourself from the critical path if your cache has problems than it is to add the caching after the fact when a third party is having problems, though.
Exactly this!
You plan for failure. What happens if your cache fails? You fall back to Microsoft. But what happens if Microsoft fails and you don't have a cache? Well you build a cache in advance and default to that.
Caches can also improve build time, allow for security tools to scan, and other such benefits too. So there are other benefits outside of creating a redundancy.
You plan for failure. What happens if your cache fails? You fall back to Microsoft. But what happens if Microsoft fails and you don't have a cache? Well you build a cache in advance and default to that.
Caches can also improve build time, allow for security tools to scan, and other such benefits too. So there are other benefits outside of creating a redundancy.
We do local caching, and it’s caused more outages, downtime, headaches and lost productivity than I can count. Although the hardware requirement is minimal, all that other stuff makes it expensive. So run a risk calculation for your own use case.
The modern argument is most likely made by people who sell caching as a service and have an interest in you blowing $500/month on a glorified FTP or a script to setup Artifactory.
But this is Microsoft's cache, so it is already essentially on the same network, and is free.
well we cached the final docker images ;-) unfortunatly if you need to add another dependency for our build image were done. (well we could pull the old one and add the dependency and repush this image)
also there are cache misses :-(
also there are cache misses :-(
The “mirrors” in this case are Microsoft run servers specifically for Microsoft made packages for Ubuntu. Rather than the actual Ubuntu mirrors for all Ubuntu packages.
Those actually work pretty well from my experience and are usually faster than the default archive in docker containers.
had this last night :( thanks for posting
Not only Ubuntu. And not for the first time [0], [1], [3] (Stack Overflow). Would be surprised, however, if the servers are in space instead of under water [2].
[0] https://social.msdn.microsoft.com/Forums/en-US/5d090bb8-6c22...
[1] https://github.com/MicrosoftDocs/windowsserverdocs/issues/50...
[2] https://news.microsoft.com/innovation-stories/project-natick...
[3] https://stackoverflow.com/questions/59624773/failed-to-fetch...
[0] https://social.msdn.microsoft.com/Forums/en-US/5d090bb8-6c22...
[1] https://github.com/MicrosoftDocs/windowsserverdocs/issues/50...
[2] https://news.microsoft.com/innovation-stories/project-natick...
[3] https://stackoverflow.com/questions/59624773/failed-to-fetch...
The packages.microsoft.com repositories are regularly broken, which results in perennial build failures if you use GitHub Actions which includes it by default.
GitHub's response to "your default configuration is broken" was basically "welp, NOMFP":
https://github.com/actions/virtual-environments/issues/3410
GitHub's response to "your default configuration is broken" was basically "welp, NOMFP":
https://github.com/actions/virtual-environments/issues/3410
Uh somewhat off topic but is NOMFP a typo of NMFP (="not my fine problem")?
Hadn't seen it before and big G suggests NMFP is way more common, with no hit that I saw explaining the 'O'.
Hadn't seen it before and big G suggests NMFP is way more common, with no hit that I saw explaining the 'O'.
https://www.google.com/search?client=firefox-b-d&q=NOMFP
(It's a quote from the UK political satire series "The Thick Of It", in which the character Malcolm Tucker remarks, "NOMFP. N-O-M-F-P. Not My Fucking Problem. I quite like that. Did you like that? I'll use that quite a lot today.")
(It's a quote from the UK political satire series "The Thick Of It", in which the character Malcolm Tucker remarks, "NOMFP. N-O-M-F-P. Not My Fucking Problem. I quite like that. Did you like that? I'll use that quite a lot today.")
Is this another typo? NFMP = Not Fine My Problem?
That said, I’d also like to know.
That said, I’d also like to know.
Typo yes, that's what I get from HN:ing on my phone. Thanks, and fixed.
I’d have guessed „none of my fucking problems“
Maybe “Not on my fucking plate”?
Is there a way to easily change the default and if so what is a good mirror?
I have no need for MS packages, and so I simply
for broken_apt_repository in `grep -lr microsoft /etc/apt/sources.list.d/`; do sudo rm $broken_apt_repository; done
Oh, had this several times and removed MS repos in some projects. Maybe it worth to create the extension.
Wanted to update my Teams on Ubuntu 20 in the morning (after coming to office after months working from home) and had the same issue. Removed the package and installed it from snap. Works like charm so far.
it's not just ubuntu, it's basically all of them. basically my debian docker build failed because of that.
Same. More or less all of my company's pipelines are currently broken
Of all the social issues in the world, apt-get and wget 404ing for my version of linux are the only things that trigger me
[deleted]
I've also found the ':' character in filenames also breaks Linux under Windows :(
None commented yet on the suggested issue? They ran out of space? Really? Azure Monitoring, Azure Sentinel - surely they have the simplest metric: used vs remaining space.
Or did they hit a limit in the blob/file storage. Still: no monitoring?
You may have noticed I have so many questions on Microsoft Ops.
Or did they hit a limit in the blob/file storage. Still: no monitoring?
You may have noticed I have so many questions on Microsoft Ops.
rm -rf --no-preserve-root is a space issue, too.
Reminds me of an F1 story. A car was able to finish the race but stopped before reaching the pit lane. The team communicated it as being a fuel pick up problem. The underlying cause was that there was no fuel left to be picked up.
> because of space issues
yeah, spaces in filenames. A well-known yet insidious problem.
Is there any mount option to change raw spaces in the exposed filesystem with an unicode non-breaking space, or something?
yeah, spaces in filenames. A well-known yet insidious problem.
Is there any mount option to change raw spaces in the exposed filesystem with an unicode non-breaking space, or something?
> yeah, spaces in filenames. A well-known yet insidious problem.
It'll be fixed as soon as they finish their Backslash Conversion Project!
It'll be fixed as soon as they finish their Backslash Conversion Project!
its probably a silly question, but why are people using microsoft to fetch debian packages?
Microsoft's packages of its own software for Debian.
I'd assume they host some of their own packages plus it's probably the default for the Azure space. (just an assumption)
Microsoft repos were silently added to Raspberry Pi OS earlier this year, ostensibly "in case anyone wants to use VSCode" and definitely not just as a super shitty way of adding telemetry to every Raspberry Pi OS install without asking the user if they were cool with this.
For example, my team's services run on Debian, but are .NET Core-based, so our dev boxes & CI machines rely on Microsoft deb repos.
[deleted]
[deleted]
Just to clarify the title, Space in this context is the 'Biggest Key on your Keyboard' kind not the 'On Disk' kind.
It looks like a parsing issue where something in MS's automated process is replacing various parts of filenames with spaces. Looks like / and _ at a glance, but there could be more.
Just trying to prevent this type of comment, which I can only assume came from HN as nowhere in the Github issue does anyone mention disk space. -- https://github.com/dotnet/core/issues/6381#issuecomment-8631...
It looks like a parsing issue where something in MS's automated process is replacing various parts of filenames with spaces. Looks like / and _ at a glance, but there could be more.
Just trying to prevent this type of comment, which I can only assume came from HN as nowhere in the Github issue does anyone mention disk space. -- https://github.com/dotnet/core/issues/6381#issuecomment-8631...
"nowhere in the Github issue does anyone mention disk space"
What is mentioned is ambiguous. So while disk space isn't specifically mentioned, neither is spaces in the context of filenames.
Here's what people are reading in that issue:
"Update: our infra team is still working to resolve this issue. They ran into some space issues but this issue should be resolved quickly. Unfortunately, I do not have an ETA yet."
There isn't enough context to know what it means.
What is mentioned is ambiguous. So while disk space isn't specifically mentioned, neither is spaces in the context of filenames.
Here's what people are reading in that issue:
"Update: our infra team is still working to resolve this issue. They ran into some space issues but this issue should be resolved quickly. Unfortunately, I do not have an ETA yet."
There isn't enough context to know what it means.
"escaping space characters" would be clear.
I thought there was some diskspace issue
I thought there was some diskspace issue
[deleted]
[deleted]
e.g., azuredatastudio contains code that manages a /usr/local/bin/code symlink...
... and it contains code that converts a PGP key in armored form to binary form and dumps it in /etc/apt/trusted.gpg.d, even if the sysadmin already took the time to verify the PGP key and put it into their own .asc file in the same directory (ok, I guess at least they aren't force-appending their key to the old/deprecated/monolithic /etc/apt/trusted.gpg file like many others)...
... oh and in doing so they dump out microsoft.gpg to the current directory, whatever that may be... they should at least be using mktemp!
... and they are doing other things that they should be relying on debhelper to do for them, e.g. installing shared-mime-info-spec files manually rather than with dh_installmime; installing desktop-entry-spec files manually rather than relying on the triggers that already handle installation of such files...
As for teams, it has its own oddball way of monkeying with /etc/apt, and one other weirdness: it explicitly changes /usr/share/teams/chrome-sandbox to be setuid. If that file is supposed to be setuid then ship it as such... shipping it non-setuid and then modifing it in a maintainer script sets off alarm bells and breaks dpkg-statoverride...