APFS filesystem format(blog.cugu.eu)
blog.cugu.eu
APFS filesystem format
https://blog.cugu.eu/post/apfs/
147 comments
Anyone know how Apple pulled off an in place filesystem upgrade ? Is the data copied to your upgrade laptop and back ?
No, nothing like that necessary. Data remains in the same spot, and during the conversion new metadata and file tables/pointers are created in APFS format. Then the volume is remounted using the new format, and the old hfs stuff goes bye-bye.
Does anyone know or can speculate on the purpose of the 4 "alignment" bytes in the node block header? Are they for alignment (in which case it's strange to have them at the beginning instead of the end)? Or are they describing alignment (in which case it's strange to need 32 bits)?
I just called it alignment. It's just a bit that decides wether the following entries are fixed length or variable length.
Is there a filesystem that can fragment a large, existing file to perform arbitrary inserts?
fallocate(FL_INSERT_RANGE) can do that, but only on block boundaries
Common in the mainframe world, like VSAM. That would be variable length record oriented files, versus stream files. Someone did do this for GEOS: http://justsolve.archiveteam.org/wiki/GEOS_VLIR , but I've never seen that for anything remotely unix like.
Unlikely. It's difficult to implement, and can be worked around by applications that need it.
Look, we get it, you bought an Android and you want to use iMessage and FaceTime.
That would be stupid, Apple doesn't make money from users data, Apple doesn't listen to conversations like Google does to sell ads.
So why would they support servers (and there are a lot more of android users than iOS users) for people that didn't bought anything from them? For people that bought Androids to like them? They don't need that.
It has nothing to do with open source, the open source community doesn't need this and Apple couldn't care less about the GNU types after GPLv3 incompatibilizes with their business (while Android OEMs break the license every single day, but the community turns a blind eye, because it's Google)
Also, they would lose a few of their product sales and lots of 30% shares on sticker packs and other Apps for iMessage they sell.
That would be stupid, Apple doesn't make money from users data, Apple doesn't listen to conversations like Google does to sell ads.
So why would they support servers (and there are a lot more of android users than iOS users) for people that didn't bought anything from them? For people that bought Androids to like them? They don't need that.
It has nothing to do with open source, the open source community doesn't need this and Apple couldn't care less about the GNU types after GPLv3 incompatibilizes with their business (while Android OEMs break the license every single day, but the community turns a blind eye, because it's Google)
Also, they would lose a few of their product sales and lots of 30% shares on sticker packs and other Apps for iMessage they sell.
We detached this subthread from https://news.ycombinator.com/item?id=14175206 and marked it off-topic.
That's not at all why I posted this or where I was going. I do happen to have an Android, but don't use apps other than a web browser and whatever the SMS app is. I have no desire to video chat, or any need for SMS beyond what a Nokia can do.
I am just interested in knowing more about why Apple has the reputation it does around open source. The thread has delivered on that. Also delivered on the idea that apple fans are oddly defensive. Honestly not trolling, but the reaction seems as if I did succeed in that somehow.
I am just interested in knowing more about why Apple has the reputation it does around open source. The thread has delivered on that. Also delivered on the idea that apple fans are oddly defensive. Honestly not trolling, but the reaction seems as if I did succeed in that somehow.
Ends a little abruptly (as in no closing) but other than that interesting read.
It's a description of the format, not a review of it. What conclusion were you after?
I wasn't even after a conclusion or anything. But since there isn't a footer I just thought maybe my internet connection died and it only loaded a portion of the page.
Fair point! I looked at it more of a blog post, but I suppose I should've looked at it more as a resource. My mistake.
Kaitai which was used as a parser for this looks pretty interesting. I need to reverse engineer some of the Allen Bradley Ethernet Industrial Protocol and its client communication. This seems pretty useful for documenting that.
http://kaitai.io/
Also check out https://ohmlang.github.io/ and view the maths demo in your local copy.
Also check out https://ohmlang.github.io/ and view the maths demo in your local copy.
Holy shit this is so cool. I will never have a need for this but it's really amazing to poke around with. It kind of feels like Wireshark for files instead of packets.
They also have a web version[0] with a bunch of example files if you want to check it out without installing it locally.
[0] https://ide.kaitai.io/
They also have a web version[0] with a bunch of example files if you want to check it out without installing it locally.
[0] https://ide.kaitai.io/
Typo thread!
"this hight result in" => "might"
"and seams to be related" => "seems"
"sometimes spaceman" => "sometimes called spaceman" ?
Very interesting read, thanks!
"this hight result in" => "might"
"and seams to be related" => "seems"
"sometimes spaceman" => "sometimes called spaceman" ?
Very interesting read, thanks!
Very, very cool exercise. For anyone who hasn't been following APFS much though, worth knowing that Apple has stated in their AFPS FAQ [1] that they do intend to directly "document and publish the APFS volume format specification" themselves later this year. Presumably (as they do with other stuff they develop) they want to be able to more easily make breaking changes during the shakedown period of initial public deployment (first on iOS right now, later on Macs, though maybe not a default option until 10.13 this Summer/Fall). That'd be the most helpful in getting read/write into FUSE and other platforms should anyone choose to do so.
---
1. https://developer.apple.com/library/content/documentation/Fi...
---
1. https://developer.apple.com/library/content/documentation/Fi...
They said the same about Facetime and iMessages but never opened or documented those, so let's see when and if it actually happens. Apple has a history of false promises when it comes to opening and documenting their protocols & data formats.
Edit: I misrecalled that iMessage was announced as something they were opening.
Edit: I misrecalled that iMessage was announced as something they were opening.
I guess you missed the whole lawsuit thing [0] that forced them to make (initially P2P) Facetime completely centralized at which point it was pointless to open it up.
PS: If you're gonna hate on Apple, at least find some valid points about them to hate instead of just spreading bullshit.
[0] https://arstechnica.com/tech-policy/2013/08/report-after-pat...
PS: If you're gonna hate on Apple, at least find some valid points about them to hate instead of just spreading bullshit.
[0] https://arstechnica.com/tech-policy/2013/08/report-after-pat...
Apple did say they were opening it up; then they didn't. How's that bullshit? Even if they did have a legal excuse, that doesn't make it any less true. It could happen again.
I think it's a bit unfair to paint a genuine and honest intention, that is even pursued as far as a court of law, but which is unexpectedly prevented by a third party a 'false promise'.
After having just read that article, and knowing nothing else about it: ... How is that patentable?
They never promised to open up iMessage. The story behind FaceTime is a little bit more complicated. Steve Jobs said he would open it up without asking the engineers and lawyers who were just as surprised as anyone else about his announcement.
As far as Apple not open sourcing anything.
https://opensource.apple.com
They also published the full git commit history of Swift from the initial commit.
As far as Apple not open sourcing anything.
https://opensource.apple.com
They also published the full git commit history of Swift from the initial commit.
Apple's relationship with open source is pretty average. In the end, they use open source strategically. If there's no benefit to open-sourcing something, they won't.
Clang, WebKit and Swift are "proper" open source projects, done in the open. But there aren't many of those. They have no open source apps, and most frameworks such as Cocoa, AppKit etc. are completely proprietary.
Some of their open source efforts have been half-hearted; Darwin was enthusiastically embraced by the community, then essentially shut down by Apple. The XNU sources are APL, but there's nothing open about its development. I/O Kit is closed, I believe.
Clang, WebKit and Swift are "proper" open source projects, done in the open. But there aren't many of those. They have no open source apps, and most frameworks such as Cocoa, AppKit etc. are completely proprietary.
Some of their open source efforts have been half-hearted; Darwin was enthusiastically embraced by the community, then essentially shut down by Apple. The XNU sources are APL, but there's nothing open about its development. I/O Kit is closed, I believe.
> In the end, they use open source strategically. If there's no benefit to open-sourcing something, they won't.
Wouldn't that apply for most companies? Let's not pretend that Facebook, Microsoft et al do open source as a charity
Wouldn't that apply for most companies? Let's not pretend that Facebook, Microsoft et al do open source as a charity
They don't but they both perform much better than Apple on the subject.
React is one of the leaders in JS dev and several other Facebook projects are quite useful.
Microsoft is the company with the biggest number of open source contributors and several of their projects like TypeScript are extensively used and they are the biggest contributors to LINUX https://octoverse.github.com/
Apple made Swift an open source language that runs only on their proprietary devices and that's it...
React is one of the leaders in JS dev and several other Facebook projects are quite useful.
Microsoft is the company with the biggest number of open source contributors and several of their projects like TypeScript are extensively used and they are the biggest contributors to LINUX https://octoverse.github.com/
Apple made Swift an open source language that runs only on their proprietary devices and that's it...
> made Swift an open source language that runs only on their proprietary devices
Incorrect. Besides Linux, like the other poster mentioned, it even runs on Raspberry Pi:
http://dev.iachieved.it/iachievedit/swift-3-0-on-raspberry-p...
Incorrect. Besides Linux, like the other poster mentioned, it even runs on Raspberry Pi:
http://dev.iachieved.it/iachievedit/swift-3-0-on-raspberry-p...
Swift does not run "only on their proprietary devices". It runs on at least Linux as well.
I'm not saying that Apple is better, or denying that Facebook does a 'better job', but instead just reinforcing the concept that all these technology companies don't open source out of the goodness of their heart, but instead for the same strategic decisions that Apple does it for.
I'm not saying that Apple is better, or denying that Facebook does a 'better job', but instead just reinforcing the concept that all these technology companies don't open source out of the goodness of their heart, but instead for the same strategic decisions that Apple does it for.
>> Apple's relationship with open source is pretty average.
That's literally what he said.
That's literally what he said.
I think you literally mean figuratively.
A lot of companies routinely open-source lots of little things that could easily have stayed internal, but are genuinely useful to everyone else. (Admittedly, I think about 30% of them are Go logging libraries, but the point still stands.) I.e., open source by default as opposed to being the exception.
That, of course, can be a strategy. A company can look cooler and attract talent by developing everything in the open.
That, of course, can be a strategy. A company can look cooler and attract talent by developing everything in the open.
It's called "commoditizing your complements"
Google open sourced Android to get advertising revenue.
IBM supports open source and made money from server sells and still makes money from services.
Apple open sourced WebKit (more than they had to) to gain industry support and later to kill off Flash.
https://www.joelonsoftware.com/2002/06/12/strategy-letter-v/
Google open sourced Android to get advertising revenue.
IBM supports open source and made money from server sells and still makes money from services.
Apple open sourced WebKit (more than they had to) to gain industry support and later to kill off Flash.
https://www.joelonsoftware.com/2002/06/12/strategy-letter-v/
Webkit was open-source long before anticipating killing Flash. I think, like Darwin, it was originally open-source because it had to be, due to its origin in an existing open-source project that required derivatives also be open-source.
Most of WebKit today is actually permissive (BSD license), with some that is still LGPL.
Darwin was originally based on BSD code - there is no requirement to keep BSD code open.
Why does Apple continue to publish Darwin code then? I have always been curious.
Why not?
It doesn't harm Apple's competitiveness. Apple doesn't publish source to the hire level OS X frameworks.
It doesn't harm Apple's competitiveness. Apple doesn't publish source to the hire level OS X frameworks.
I think to help third party driver developers?
Apple didn't own webkit they couldn't keep it closed
The original web rendering engine code was originally licensed under BSD. Apple could have forked it and closed it.
JavascriptCore was also based on BSD code, but was then rewritten by the WebKit team and made GPL.
JavascriptCore was also based on BSD code, but was then rewritten by the WebKit team and made GPL.
Fairly sure KHTML was LGPL, Apple could not have closed it even if they wanted to. Same goes for what became JSCore which also remains LGPL.
Pretty sure you are in fact wrong.
Fairly sure KHTML was LGPL, Apple could not have closed it even if they wanted to. Same goes for what became JSCore which also remains LGPL.
You're right. WebKit itself is BSD but Webcore and the original original JSCore are LGPL. A really weird combination.
You're right. WebKit itself is BSD but Webcore and the original original JSCore are LGPL. A really weird combination.
They do have at least one open source application: https://www.cups.org/
it's fully open-source, and wholly owned by Apple, however they bought it, they didn't invent it.
They also lead development of LLVM and clang, which is a _huge_ project, and very useful (although sometimes without full realization of that fact) to the Open Source ecosystem.
Interesting, I did not know that Apple is the maintainer of CUPS since 2007.
They hired the guy who originally made it, Michael Sweet [1].
[1] https://en.wikipedia.org/wiki/Michael_Sweet_(programmer)
[1] https://en.wikipedia.org/wiki/Michael_Sweet_(programmer)
[deleted]
Perhaps the exception that proves the rule, but TextEdit is open source!
https://developer.apple.com/library/content/samplecode/TextE...
https://developer.apple.com/library/content/samplecode/TextE...
Specifically, there were major lawsuits over patent infringement immediately after Facetime launched.
And those lawsuits forced Apple to re-engineer how Facetime worked, from peer-to-peer direct connections to a more Apple-intermediated model[1]. At that point, creating an open standard is less interesting, since you can only communicate with folks using the same "relay server" network.
[1] https://arstechnica.com/tech-policy/2013/08/report-after-pat...
[1] https://arstechnica.com/tech-policy/2013/08/report-after-pat...
Regarding Apple’s Open Source strategy, the WebKit origin story is a major disaster.
They forked an existing project, only did sourcedumps every few months, often stripped of all comments, refused to work with the original devs that they had forked from (the KDE KHTML engine), etc.
They forked an existing project, only did sourcedumps every few months, often stripped of all comments, refused to work with the original devs that they had forked from (the KDE KHTML engine), etc.
Since 2010, KDE has been moving away from KHTML to the official WebKit fork.
Good.
GPLv3 software can't be on Apple products, yet every Chromebook ships with it, because Saint Google.
GPLv3 software can't be on Apple products, yet every Chromebook ships with it, because Saint Google.
Signal is GPLv3 and readily available on Apples App Store. So no, you are mistaken.
It's violating GPLv3.
Read your parent again:
> GPLv3 software can't be on Apple products
it is not saying:
> App Store
> GPLv3 software can't be on Apple products
it is not saying:
> App Store
Looks like I'm not getting my point across,
Here is the Tanegashima's post:
> ...GPLv3 software can't be on Apple products...
And here is jwildeboer's reply:
> ... available on Apples App Store. So no, you are mistaken
And my point is:
No he is not mistaken because you are talking about completely different topics.
Here is the Tanegashima's post:
> ...GPLv3 software can't be on Apple products...
And here is jwildeboer's reply:
> ... available on Apples App Store. So no, you are mistaken
And my point is:
No he is not mistaken because you are talking about completely different topics.
[deleted]
[deleted]
Open sourcing Swift, though, probably benefits Apple more than it does the community. Sans the Apple UI, it doesn't create some opportunity to lose a sale or have Apple like consumer facing apps show up on some other platform. Apple apps already mostly rely on non-apple backend servers.
Opening up iMessage or FaceTime would have been more of a statement. Either would allow interoperability on platforms they don't control.
Is there something significant they've open sourced, despite some perceived risk?
Opening up iMessage or FaceTime would have been more of a statement. Either would allow interoperability on platforms they don't control.
Is there something significant they've open sourced, despite some perceived risk?
> Open sourcing Swift, though, probably benefits Apple more than it does the community.
I am no fan of Apple myself, but it's worth giving them credit where credit is due, they did everything right when open-sourcing Swift and are very responsive to the community, there's really no downside for anybody here, so why the need to criticise? It only needlessly provides the wrong incentives.
I am no fan of Apple myself, but it's worth giving them credit where credit is due, they did everything right when open-sourcing Swift and are very responsive to the community, there's really no downside for anybody here, so why the need to criticise? It only needlessly provides the wrong incentives.
It is not criticism. It's an observation, followed by a question. I'm honestly interested in why Apple has the reputation it does around open source. The replies in the sub thread have been enlightening.
WebKit? Darwin? ResearchKit?
Apple gets this bad rap for OSS, because of Google's good PR a few years back on how "Android was open." But it's not like Google has a better track record of supporting OSS when it comes to products that influence their bottom line. Adwords, Google Search, AdSense, Analytics, etc are not open source, and no one ever complains about that.
Apple gets this bad rap for OSS, because of Google's good PR a few years back on how "Android was open." But it's not like Google has a better track record of supporting OSS when it comes to products that influence their bottom line. Adwords, Google Search, AdSense, Analytics, etc are not open source, and no one ever complains about that.
The comment you're replying to, and all the parents up to the root, like most comments here made no mention of Google. It was a discussion of Apple on its merits.
Why do we seem to tie our personal identities to corporate affiliations, to the point of bringing up corporate rivalries out of context and taking sides in them?
Why do we seem to tie our personal identities to corporate affiliations, to the point of bringing up corporate rivalries out of context and taking sides in them?
Those are good examples. It was an honest question though, not a proclamation that Apple hasn't open sourced anything.
Edit: Maybe Google is getting more credit because of uptake? Chromium, golang, protobufs, material design, etc. So perhaps not how much was open source, but how much of it ended up in non Google projects?
Edit: Maybe Google is getting more credit because of uptake? Chromium, golang, protobufs, material design, etc. So perhaps not how much was open source, but how much of it ended up in non Google projects?
Chromium was a fairly recent fork of WebKit. Webkit was already being adopted by every browser vendor except MS and Mozilla.
As far as Golang, I doubt that it has the uptake ofClang (even used by MS), CUPS, or Bonjour.
Yes I realized that CUPS was open source before being acquired by Apple but the copyright was owned by Sweets and they could have closed all future enhancement over the last decade.
As far as Golang, I doubt that it has the uptake ofClang (even used by MS), CUPS, or Bonjour.
Yes I realized that CUPS was open source before being acquired by Apple but the copyright was owned by Sweets and they could have closed all future enhancement over the last decade.
You're talking about Blink. Chromium is not a rendering engine, but a full browser. Chrome is just chromium with branded assets and a few proprietary features that link it to Google Services.
It was first released in 2008, only 3 years after WebKit was open sourced, and about the same it was starting to gain traction outside Apple. Most of the browsers which didn't have their own engine back then were actually using Gecko or embedding an IE webview. Google embracing WebKit was a huge factor in pushing it to become the mainstream rendering engine.
It was first released in 2008, only 3 years after WebKit was open sourced, and about the same it was starting to gain traction outside Apple. Most of the browsers which didn't have their own engine back then were actually using Gecko or embedding an IE webview. Google embracing WebKit was a huge factor in pushing it to become the mainstream rendering engine.
They didn't open WebKit. It's a fork of the LGPL'd KHTML code base. Apple is legally obliged to keep it open, so whatever anyone thinks about Apple's open source record using WebKit as an example makes little sense.
Eh, not quite. Apple could satisfy its LGPL obligations by throwing source dumps of WebCore over the wall - without releasing the rest of WebKit, exposing SVN history, or having a public bug tracker. In fact, for the first two years of Safari's life, that's exactly what they did. Then in 2005 they opened it up properly, voluntarily:
https://dot.kde.org/2005/06/07/apple-opens-webkit-cvs-and-bu...
https://dot.kde.org/2005/06/07/apple-opens-webkit-cvs-and-bu...
"Voluntarily" after they were rapped over and over by KDE developers and community...
Ah yes, the KDE community, holders of the ear of Steve Jobs.
Frankly, I'm not sure Apple ever cared for their thoughts on the matter.
Frankly, I'm not sure Apple ever cared for their thoughts on the matter.
Of course, they only cared about PR: by pure coincidence, they "voluntarily" changed development model on webkit right after several news outlets had picked up the umpteen blog post by a grumbling KDE developer.
Yes, Apple historically doesn't give a shit about the OSS community unless they can exploit it or it becomes a dangerous nuisance. And this is why it gets a bad rep in OSS circles.
Yes, Apple historically doesn't give a shit about the OSS community unless they can exploit it or it becomes a dangerous nuisance. And this is why it gets a bad rep in OSS circles.
Yes because Apple in general and Steve Jobs in particular is a company that goes out of it way to be accommodating to the masses....
I'm pretty sure you're wrong about this one:
> Steve Jobs in particular is a company
> Steve Jobs in particular is a company
The big one is maps...though that's not so much open source as open data.
Google has nothing to do with this. Apple get their bad rep from their behavior on Darwin, webkit and the GNU bait-and-switch (first adopted with great fanfare, then left to rot and eventually purged mercilessly). The fact that they strenuously refused to let some of their desktop goodness leak back into the OSS world, from the very first day OSX was announced, is also an old pain point.
> Open sourcing Swift, though, probably benefits Apple more than it does the community.
As opposed to whom? Choosing an open source approach is a strategic decision, presumably to the benefit of the chooser.
This was even true of Cygnus, the original Open Source business.
As opposed to whom? Choosing an open source approach is a strategic decision, presumably to the benefit of the chooser.
This was even true of Cygnus, the original Open Source business.
The context bubbles up to the post that said "Apple has a history of false promises when it comes to opening and documenting their protocols & data formats". That comment was down voted, but had me curious.
I was asking for examples of something they open sourced that would have exposed something like that, something more risky to the walled garden concept. I perhaps could have worded it better, but HN does seem to react to any questioning of Apple with a defensive posture.
I was asking for examples of something they open sourced that would have exposed something like that, something more risky to the walled garden concept. I perhaps could have worded it better, but HN does seem to react to any questioning of Apple with a defensive posture.
something more risky to the walled garden concept. I perhaps could have worded it better, but HN does seem to react to any questioning of Apple with a defensive posture.
What has any company open sourced that would cause them to loose their competitive advantage?
What has any company open sourced that would cause them to loose their competitive advantage?
Certainly, Android is more open than Apple's mobile OS. Google does hold some important bits back. But, still, it is "more risky". Hence the question. I may be missing other areas where Apple was open sourcing things that were more core to them.
Android is "more open" except for the parts that Google cares about - Google Play Services, and all of the other open source components that Google is abandoning.
https://arstechnica.com/gadgets/2013/10/googles-iron-grip-on...
And there are few if any open source drivers for the cellular chips or graphics chips.
Android is "more open" except for the parts that Google cares about - Google Play Services, and all of the other open source components that Google is abandoning.
https://arstechnica.com/gadgets/2013/10/googles-iron-grip-on...
And there are few if any open source drivers for the cellular chips or graphics chips.
I feel like you missed the "Google does hold some important bits back" part that you quoted.
I get the point that you don't like this line of inquiry, but "something more risky" isn't "definitely lose your competitive advantage". And, again, it's a question, not an accusation. I'm curious...trying to understand the Apple reputation in the open source area, how much of it was deserved or not.
Certainly, Android is more open than Apple's mobile OS. Google does hold some important bits back. But, still, it is "more risky". Hence the question. I may be missing other areas where Apple was open sourcing things that were more core to them.
Certainly, Android is more open than Apple's mobile OS. Google does hold some important bits back. But, still, it is "more risky". Hence the question. I may be missing other areas where Apple was open sourcing things that were more core to them.
It's not Apple's fault that they didn't open up FaceTime. They absolutely were going to, and then they were sued by a patent troll (VirtnetX), and was forced to change how FaceTime works so it relays through Apple's servers instead of going peer-to-peer. Since it was now going through Apple's servers, there was no point in opening up the protocol.
IRC also needs someone to run a server. Just because a protocol isn't p2p there's plenty of reason to open it. You can independently develop third party clients, use your own server instead of Apple's etc.
They simply lied. They never said "We'll open Facetime, but only if it's implemented as a p2p protocol & client"
They simply lied. They never said "We'll open Facetime, but only if it's implemented as a p2p protocol & client"
> They simply lied
Changing one's mind based on changing circumstances isn't lying- it's common sense. The 'thing' that Steve Jobs wanted to publish as an open protocol stopped existing due to a lawsuit, the 'thing' that replaces it shared only its name.
Changing one's mind based on changing circumstances isn't lying- it's common sense. The 'thing' that Steve Jobs wanted to publish as an open protocol stopped existing due to a lawsuit, the 'thing' that replaces it shared only its name.
I think that's an absurdly low standard to apply to corporate speech.
The CEO of a multi-billion dollar company gets on stage during their main annual product conference and says a newly announced product is going to become an open industry standard.
This turns out not to possible in exactly the way the company had planned due to a lawsuit, so the company drops the plan entirely.
That's a lie by definition. You asserted that you'd do something and then you didn't do it. Does breaking a promise imply that there can't be any mitigating circumstances? No, I'm sure there were some reasons for why FaceTime wasn't opened.
People & companies make product buying decisions based on things like these, e.g. some company might have bought a bunch of iPads for video conferencing expecting that in a couple of years they'd be inter-operable with any device they could buy, including non-Apple devices.
This excuse that they couldn't open the p2p thing they initially planned to open is absurd. Apple could absolutely have just opened & standardized what FaceTime ended up being, i.e. an open video chat client/server allowing anyone to run a server. I.e. some sort of IRC or Jabber for video messaging.
Even if they couldn't open or release some specific component due to patent concerns they could still have opened & standardized almost everything else, see e.g. how open source video players ship pluggable codec support without direct support for some proprietary & patented codecs.
Bringing this back to my original comment on APFS, do I think they won't open that? No, they probably will. I just thought I'd point out that Apple has a documented history of abandoning their open source / standardization plans after making some very public announcements about them, so people should be wary of counting their chickens before they're hatched when it comes to Apple.
The CEO of a multi-billion dollar company gets on stage during their main annual product conference and says a newly announced product is going to become an open industry standard.
This turns out not to possible in exactly the way the company had planned due to a lawsuit, so the company drops the plan entirely.
That's a lie by definition. You asserted that you'd do something and then you didn't do it. Does breaking a promise imply that there can't be any mitigating circumstances? No, I'm sure there were some reasons for why FaceTime wasn't opened.
People & companies make product buying decisions based on things like these, e.g. some company might have bought a bunch of iPads for video conferencing expecting that in a couple of years they'd be inter-operable with any device they could buy, including non-Apple devices.
This excuse that they couldn't open the p2p thing they initially planned to open is absurd. Apple could absolutely have just opened & standardized what FaceTime ended up being, i.e. an open video chat client/server allowing anyone to run a server. I.e. some sort of IRC or Jabber for video messaging.
Even if they couldn't open or release some specific component due to patent concerns they could still have opened & standardized almost everything else, see e.g. how open source video players ship pluggable codec support without direct support for some proprietary & patented codecs.
Bringing this back to my original comment on APFS, do I think they won't open that? No, they probably will. I just thought I'd point out that Apple has a documented history of abandoning their open source / standardization plans after making some very public announcements about them, so people should be wary of counting their chickens before they're hatched when it comes to Apple.
> Apple could absolutely have just opened & standardized what FaceTime ended up being, i.e. an open video chat client/server allowing anyone to run a server. I.e. some sort of IRC or Jabber for video messaging.
But why? There's literally no point to doing that. The whole point of opening up FaceTime was so people using non-Apple devices could talk to people using Apple devices. After the lawsuit that became no longer possible.
But why? There's literally no point to doing that. The whole point of opening up FaceTime was so people using non-Apple devices could talk to people using Apple devices. After the lawsuit that became no longer possible.
>That's a lie by definition
Telling a lie is to say something you know isn't true. If they honestly intended to open source FaceTime and then changed their mind (for whatever reason) that is not a lie by any definition.
But I agree with you that it casts doubt on their determination to follow through on their open source promises. And that sort of doubt is extremely detrimental to any uptake of open sourced software.
Telling a lie is to say something you know isn't true. If they honestly intended to open source FaceTime and then changed their mind (for whatever reason) that is not a lie by any definition.
But I agree with you that it casts doubt on their determination to follow through on their open source promises. And that sort of doubt is extremely detrimental to any uptake of open sourced software.
The whole point of opening it up is so other people's clients can talk to FaceTime users and vice versa. But Apple's not going to provide their own servers for third parties. And without that, there's no interoperability, which means there's no point in opening up the protocol.
Yet they said the same about Swift, ResearchKit, etc. Those are successful open source projects. FaceTime is the only recent project they said they would open up and didn't...and that arguably wasn't their fault.
Timestamps are 64bit nanoseconds
Why would this go into a new standard? Is there a practical use for such a high level of granularity?
edit: off by a factor of 10 in my calculation... It's actually
2554-07-21 23:34:33.709551615 UTC
By my math, we're fine for almost 600 years:
http://wolframalpha.com/input/?i=2%5E64+nanoseconds+from+now...
http://wolframalpha.com/input/?i=2%5E64+nanoseconds+from+now...
But that should be 'from 1.1.1970', i.e. from the unix epoch. Still far out, though.
2^64 nanoseconds is > 580 years. Even cutting it in half for signed values is still likely 10X the expected lifetime of this particular iteration of APFS.
The granularity is probably useful for establishing an ordering (even somewhat arbitrarily) for highly concurrent file system operations, and thus potentially skipping a bunch of expensive synchronization.
The granularity is probably useful for establishing an ordering (even somewhat arbitrarily) for highly concurrent file system operations, and thus potentially skipping a bunch of expensive synchronization.
2^64 nanoseconds is ~584 years, though.
Found this: https://github.com/jbenet/nanotime. No idea if related in any way or used for APFS but it was interesting enough for a quick read. This one is MIT licensed. A little more searching I found a variant but it's done with big-endian so it's much different.
64-bit nanosecond values are actually pretty common on Darwin already. See e.g. clock_gettime_nsec_np(3).
Is there any reason to use APFS on desktop?
There are a couple features missing from HFS+. Some of these are implemented in APFS:
* Case sensitivity
* Clones extents / ranges (reflinking)
* Holes in the middle of files
* ACLs
* Collapsing file ranges
* Per file compression and per file encryption
These are useful for apps like SQLite or other stores which implement a multi user, log structured store.
These are useful for apps like SQLite or other stores which implement a multi user, log structured store.
FYI - HFS is capability of case sensitivity and is an option during formatting. I believe it even used to be the default.
A quick note to any readers... I'd suggest avoiding a case-sensitive HFS+ boot volume with macOS. I used one for over a year, and it just brings trouble.
The biggest problems I encountered were (1) occasional bugs with Homebrew formulas, and (2) Adobe's applications cannot be installed on a case-sensitive file system.
The biggest problems I encountered were (1) occasional bugs with Homebrew formulas, and (2) Adobe's applications cannot be installed on a case-sensitive file system.
HFS and HFS+ were never case-sensitive by default on Mac OS. Case-sensitive HFS+ was made available as an option in Mac OS 10.0, but was always glitchy (especially for boot volumes), because many applications depended on case insensitivity.
I think possibly the point of confusion is that very early versions of MacOS X (the original MacOS X server, and some of the MacOS X DPs, anyway) used UFS by default, not HFS+. UFS is case sensitive.
I have a case sensitive HFS+ and everything works fine.
That was a problem with older versions of Adobe suite, but since lots of people on Macs used Adobe Apps...
That was a problem with older versions of Adobe suite, but since lots of people on Macs used Adobe Apps...
Isn't the problem still present? A few months back I tried installing the latest version of Photoshop and Illustrator, and it didn't let me.
> was always glitchy (especially for boot volumes), because many applications depended on case insensitivity.
So CS HFS+ is not glitchy, and CS APFS will not magically fix applications with CI dependency.
So CS HFS+ is not glitchy, and CS APFS will not magically fix applications with CI dependency.
What about snapshots? Does APFS offer efficient user-created snapshots? I run Ubuntu with btrfs, and for me this is a killer feature.
> Does APFS offer efficient user-created snapshots?
Yes: https://arstechnica.com/apple/2017/02/testing-out-snapshots-...
I'm hoping it will make Time Machine a lot better.
Yes: https://arstechnica.com/apple/2017/02/testing-out-snapshots-...
I'm hoping it will make Time Machine a lot better.
Isn't there a flag to make hfs case sensitive? Programs react poorly because it's rare, but it's available.
FWIW, HFS+ fully supported NFSv4 ACLs since... no idea, but they were already there in 2008.
I believe one reason APFS is touted is because the HFS+ data structures have one big global lock, whereas APFS allows for multithreading which ought to improve performance on multi-core machines.
I heard there will be copy-on-write for files. For web devs this would save a lot of space and time when copying files from yarn cache to node_modules.
I wonder what it means for appending data to a large log file. Does the entire log file need to get copied on every write (or just a block of the file?)
My hope is that it will improve Time Machine performance / reliability.
COW should do wonders for performance, but COW with no data checksums would make me more nervous about reliability (only one copy of files to corrupt)
I agree to a point, one factor that can help mitigate this however is using a quality SSD where it's onboard memory has ECC, I suspect Apple sees the near future of end user computing as gaining ECC for all memory by standard which is something I've wanting for years but games have been played by CPU manufacturers to prevent the use of high end desktop CPUs in server environments by I believe purposely kneecapping them from using ECC memory (citation probably needed here).
I haven't done any serious analysis of Time Machine (or of APFS for that matter), but I think in general Time Machine also only keeps one copy of files. So APFS isn't worse than the existing situation. Arbitrarily keeping multiple copies blows up the disk requirements. You can intentionally keep old copies around in APFS via snapshots, which is what APFS will probably do when implementing Time Machine.
What Apple added for the existing Time Machine on HFS+ was an extension of the Unix concept of hard links to files (where there is only one copy). They added hard links to directories. Generic Unix avoids doing that for a number of reasons.
What Apple added for the existing Time Machine on HFS+ was an extension of the Unix concept of hard links to files (where there is only one copy). They added hard links to directories. Generic Unix avoids doing that for a number of reasons.
Isn't is already true that there is one copy to corrupt? The whole idea of using hard-linked trees (a la rsnapshot [1]) presumes that when a file doesn't change, there is only one copy.
[1] http://rsnapshot.org/
[1] http://rsnapshot.org/
If someone wants to know like me why Apple came up with a whole new FS - http://www.cultofmac.com/435718/apfs-new-apple-file-system/
Great link! I'm very interested in the mention of per file keys, one for data and one for metadata. I wonder what data structure they store these in. It sounds a lot like cryptree [0].
[0] https://raw.githubusercontent.com/ianopolous/Peergos/master/...
[0] https://raw.githubusercontent.com/ianopolous/Peergos/master/...
So it is copy-on-write. Interesting.
Apple should publish something like TN1150 for APFS. several years ago, I had to implement HFS+ for an embedded system, and TN1150 proved very helpful, saved me a lot of hassle.
https://developer.apple.com/legacy/library/technotes/tn/tn11...
https://developer.apple.com/legacy/library/technotes/tn/tn11...
For those who, like me, don't follow developments in this area closely:
APFS is Apple's new filesystem, intended to be used across all of their devices (Macs, iOS, watchOS, etc). For Macs, it will be replacing HFS+, a direct descendant of the Hierarchical Filing System which has been used since 1985. (The original Macintosh Filing System, from 1984, was suitable only for floppy disks; among other things, it was a flat filing system and didn't really support folders.) The main improvements HFS+ made over HFS were support for larger files and Unicode; Apple has since added some more features but the core filesystem remains a 30-year-old format which was designed for early hard disks.
APFS is Apple's new filesystem, intended to be used across all of their devices (Macs, iOS, watchOS, etc). For Macs, it will be replacing HFS+, a direct descendant of the Hierarchical Filing System which has been used since 1985. (The original Macintosh Filing System, from 1984, was suitable only for floppy disks; among other things, it was a flat filing system and didn't really support folders.) The main improvements HFS+ made over HFS were support for larger files and Unicode; Apple has since added some more features but the core filesystem remains a 30-year-old format which was designed for early hard disks.
So what features does the new filesystem bring?
IIRC there's COW blocks (crash protection), metadata checksumming, shared storage pools (a container can span multiple physical storage devices), snapshots, built-in encryptions, sparse files, I/O QoS and access patterns more suitable for solid-state, 64b across the board (64b inodes, 64b nanosecond-precision timestamps, …), better multithreading/multiprocessing behaviour.
If you ask what innovations APFS brings to the overall FS landscape, the answer is probably "none".
If you ask what innovations APFS brings to the overall FS landscape, the answer is probably "none".
It replaced (as of iOS 10.3) HFS+ on the mobile devices as well, didn't it?
Almost correct. 32-bit devices that received iOS 10.3 did not receive APFS. It was exclusively delivered to 64-bit devices.
It's also worth noting that the first 64-bit iPhone/iPad were released in 2013, so they've been around for a while.
The iPhone 5c will have the shortest support window from its release, at 4 years.
The iPhone 5c will have the shortest support window from its release, at 4 years.
Yes.
Some things that came to mind: I like that some of the block types are "acronyms" of their meanings: 5 = S(pace man), B = B(-tree), C = C(heckpoint). Also, I wonder why the volume superblock magic is "APSB", while the root (container) magic is "NXSB"; maybe this refers to NextStep? Seems a bit anachronistic, but I don't know.