The NOVA filesystem(lwn.net)
lwn.net
The NOVA filesystem
https://lwn.net/SubscriberLink/729812/1557fcd18e4dd96f/
29 comments
It's not so grim.
"The origin of the CPU-count dependence is that NOVA divides PMEM into per-CPU allocation regions. We use the current CPU ID as a hint about which region to use and avoid contention on the locks that protect it. So moving from a smaller number of CPUs to a larger number of CPUs just means more contention for the locks. Moving from a larger number to a smaller number is no problem at all. So, our current plan is to set the CPU count very high (like 256) when the file system is created." - comment under the post by one of the designers of the fs
"The origin of the CPU-count dependence is that NOVA divides PMEM into per-CPU allocation regions. We use the current CPU ID as a hint about which region to use and avoid contention on the locks that protect it. So moving from a smaller number of CPUs to a larger number of CPUs just means more contention for the locks. Moving from a larger number to a smaller number is no problem at all. So, our current plan is to set the CPU count very high (like 256) when the file system is created." - comment under the post by one of the designers of the fs
Just wait 10 years until we all have 1000 core CPUS...
I don't see it as such a big problem. Most filesystem images aren't intended to be ported directly between machines in general. As long as they have a tool to read the filesystem on a different configuration, a filesystem with this restriction could still be very useful for generic bulk storage on a single node.
CPUs with hyperthreading typically allow you to disable it in the BIOS. If you're running a new, beta-ish CPU microarchitecture (such as Ryzen/Threadripper) you might wish to disable HT as part of troubleshooting a CPU freeze/hang.
Changing a single CPU-related flag in the BIOS should under no circumstances render a filesystem unmountable.
Changing a single CPU-related flag in the BIOS should under no circumstances render a filesystem unmountable.
Changing the number of cores given to a VM is not so uncommon.
What if you upgrade the machine with a new CPU that has more cores?
A comment from one of the devs says there should be no impact as long as the number of CPU's stays below the number of locks the filesystem was created with, which will be very high by default. If it goes above, there is a possibility of more lock contention but it will still work.
[deleted]
Archive and restore or transfer to new media.
Agreed. They might fix it at some point, but if there's a performance advantage to making the filesystem non-portable between machines with different numbers of CPUs, many people might happily take that performance advantage, at the cost of having to do file-level backups/transfers rather than image-level.
> Most filesystem images aren't intended to be ported directly between machines in general.
Wouldn't a Server hardware cluster potentially prove problematic?
Wouldn't a Server hardware cluster potentially prove problematic?
> Most filesystem images aren't intended to be ported directly between machines in general
You don't have external hard drives or flash drives...?
You don't have external hard drives or flash drives...?
This file system only works on nvdimms, which are typically in RAM sockets. It won't work on normal block devices.
Er, the comment was, "Most filesystem images aren't intended to be ported directly between machines in general."
The entire complaint itself is the fact that this file system isn't able to do what people expect from most file system images (read: pretty much every one, except this one). Citing the fact that this particular one fails that pattern isn't a justification, it's circular logic.
The entire complaint itself is the fact that this file system isn't able to do what people expect from most file system images (read: pretty much every one, except this one). Citing the fact that this particular one fails that pattern isn't a justification, it's circular logic.
This feels like a file system designed for mobile devices, where moving the storage volume would not be a relevant concern in most cases. Based on the design, it should be more responsive, use less CPU and thus less battery.
Simple answer may be treating it the way hash partioning treats shards, i.e. overallocate to start and map the over allocation to the physical layer.
From the comments, it appears that that's what they're doing:
> The origin of the CPU-count dependence is that NOVA divides PMEM into per-CPU allocation regions. We use the current CPU ID as a hint about which region to use and avoid contention on the locks that protect it.
> So moving from a smaller number of CPUs to a larger number of CPUs just means more contention for the locks. Moving from a larger number to a smaller number is no problem at all. So, our current plan is to set the CPU count very high (like 256) when the file system is created.
> The origin of the CPU-count dependence is that NOVA divides PMEM into per-CPU allocation regions. We use the current CPU ID as a hint about which region to use and avoid contention on the locks that protect it.
> So moving from a smaller number of CPUs to a larger number of CPUs just means more contention for the locks. Moving from a larger number to a smaller number is no problem at all. So, our current plan is to set the CPU count very high (like 256) when the file system is created.
It seems like the developers could relatively easily add a slow compatibility mode that allows accessing filesystems from systems with different numbers of CPUs. It would just impose additional locking accessing all of the previous per-CPU datastructures.
Yeah this would be a deal-breaker. I can imagine the devs will be working on a way to "convert" one to the other, but I can also imagine that comes way down the priority list, so I am not disturbed by this fact, at this time.
This is required, I assume (after 5 min scan so can be wrong), that in order to speed things up, information about the CPUs are embedded in the filesystem.
So is this not a classic "hardcode for speed" vs "generic but slow" trade off?
This is required, I assume (after 5 min scan so can be wrong), that in order to speed things up, information about the CPUs are embedded in the filesystem.
So is this not a classic "hardcode for speed" vs "generic but slow" trade off?
Agreed, something they should have a plan for but not too worrying to see it with a few caveats at this stage.
I actually wrote a comment in response to loeg elsewhere in the thread and lost it by refreshing (and then lost interest, hah!) on the subject of conversion. I think there are some interesting problems around introducing a new FS and having to build "migrations" into it from the start. Migration feels like an expensive (in time/effort) word and I'd be concerned about issues coming up when you need to rewrite chunks of your NOVA FS because the per-CPU region has to become larger and everything else needs to shift along. It wouldn't need to be a full rewrite since only some data would need to be moved but it still seems clunky.
As mentioned in another comment, it seems like the solution being put forward is to oversubscribed the per-CPU data region for 256 CPUs but I wonder how long until that turns out to be a bad assumption. Who knows, maybe it doesn't matter. Maybe it's just out of scope for NOVA. One FS doesn't need to rule the world after all!
I actually wrote a comment in response to loeg elsewhere in the thread and lost it by refreshing (and then lost interest, hah!) on the subject of conversion. I think there are some interesting problems around introducing a new FS and having to build "migrations" into it from the start. Migration feels like an expensive (in time/effort) word and I'd be concerned about issues coming up when you need to rewrite chunks of your NOVA FS because the per-CPU region has to become larger and everything else needs to shift along. It wouldn't need to be a full rewrite since only some data would need to be moved but it still seems clunky.
As mentioned in another comment, it seems like the solution being put forward is to oversubscribed the per-CPU data region for 256 CPUs but I wonder how long until that turns out to be a bad assumption. Who knows, maybe it doesn't matter. Maybe it's just out of scope for NOVA. One FS doesn't need to rule the world after all!
It's interesting that the assumptions in the comments here all assume one thinks of this as a disk, as opposed to a non-volatile RAM disk.
No one thinks twice about having to "do something" with their RAM disk to upgrade the CPU.
It's time we realized the new class of nonvolatile memory hardware is a new entry in the storage hierarchy. One shouldn't think of it as RAM, but it's much, much faster than traditional disk (or even SSDs) and should be thought of something different.
It's a mistake to put the same restrictions on the potential performance of this just to make it fit our preconceived notions of storage.
No one thinks twice about having to "do something" with their RAM disk to upgrade the CPU.
It's time we realized the new class of nonvolatile memory hardware is a new entry in the storage hierarchy. One shouldn't think of it as RAM, but it's much, much faster than traditional disk (or even SSDs) and should be thought of something different.
It's a mistake to put the same restrictions on the potential performance of this just to make it fit our preconceived notions of storage.
Agreed. It is surprisingly hard to lose some of the assumptions spinning rust has left us with. Treating this as a log-structured 'soup', I think, opens up interesting possibilities.
Interop is vital, though. I'm less concerned with being able to pull and seat a drive between machines[1] than sensible replication/backup, and a 'send/receive' type-thing shouldn't be too big of a deal.
[1] Thumb drives are an obvious exception.
Interop is vital, though. I'm less concerned with being able to pull and seat a drive between machines[1] than sensible replication/backup, and a 'send/receive' type-thing shouldn't be too big of a deal.
[1] Thumb drives are an obvious exception.
I would definitely expect a separate piece of hardware to be swappable without impacting on the rest of my machine. I can change/upgrade the CPU without breaking the north bridge or losing my RAID mapping so why should the data storage mechanism on my new NVMe device (which is where my mind leaps for NOVA) be any different?
What type of usage am I missing where losing/having to do an offline rebuild of your data would be acceptable during a hardware upgrade? NOVA seems to be aiming to address the recognised issue by overprovisioning (which I disagree with but such is life). I don't really understand your argument.
What type of usage am I missing where losing/having to do an offline rebuild of your data would be acceptable during a hardware upgrade? NOVA seems to be aiming to address the recognised issue by overprovisioning (which I disagree with but such is life). I don't really understand your argument.
You can't change your CPU without losing the data in RAM (unless you are on zOS), and I think that's a precedent which people are ignoring.
Imagine a hardware architecture which considers NV-RAM as a very large and safe cache, and software which understands that.
This would be great for things like a (huge) RDMS which traditionally requires tables and especially indexes to be in RAM. Suddenly you can get adequate performance on multi-terrabyte indexes, and providing the software understands the storage hierarchy there is no explicit rebuild needed - the data is also stored on spinning rust and on a WAL maybe on a SSD, and will be reloaded into NV-RAM on read.
Imagine a hardware architecture which considers NV-RAM as a very large and safe cache, and software which understands that.
This would be great for things like a (huge) RDMS which traditionally requires tables and especially indexes to be in RAM. Suddenly you can get adequate performance on multi-terrabyte indexes, and providing the software understands the storage hierarchy there is no explicit rebuild needed - the data is also stored on spinning rust and on a WAL maybe on a SSD, and will be reloaded into NV-RAM on read.
seems like a dealbreaker. What use is a FS if it isn't portable? At least it sounds like they're very aware of this issue.
I'd be very interested to see what they end up doing to make this behave better prior to upstream consideration. I wonder if a linked list journal of inode table changes (with space drawn from per-CPU freelists) would be safe/fast enough. That could be periodically coalesced and remapped to the NOVA device's non-CPU aware inode table.