Perfect NAS Solution(vermaden.wordpress.com)
vermaden.wordpress.com
Perfect NAS Solution
https://vermaden.wordpress.com/2024/08/04/perfect-nas-solution/
25 comments
It's lovely that these devices get a chance at being productive again.
Perfect NAS involves drives in ZFS mirror and ECC memory.
It is very cool that nvme drives are now cheap. It looks like pricing is linear up to 4TB now. That probably means the time for 10gb copper at home is finally here. (edit 10gb RJ45 switches also appear to be sufficiently available now)
It is very cool that nvme drives are now cheap. It looks like pricing is linear up to 4TB now. That probably means the time for 10gb copper at home is finally here. (edit 10gb RJ45 switches also appear to be sufficiently available now)
ZFS is good when someone else pays for the drives. It's a pain in the butt when you have a mismatch of random drives found during sales and want to create a single unified storage pool of them.
Synology's SHR and Unraid do it easily.
Synology's SHR and Unraid do it easily.
Lets assume You got 4TB 6TB 5TB and 8TB drives ... just do a 4TB 1st primary partition on each of them and create a ZFS pool from that - I do not understand where the 'pain' is ...
You can then add 1TB 2nd partition on 6TB and 5TB and 8TB drives and add another ZFS VDEV to the same pool - or create another pool.
... and You can also add 3rd partition.
Also its really easy to get used drives in the same or similar size ... unless you live in some really 'hard to live' part of the world.
You can then add 1TB 2nd partition on 6TB and 5TB and 8TB drives and add another ZFS VDEV to the same pool - or create another pool.
... and You can also add 3rd partition.
Also its really easy to get used drives in the same or similar size ... unless you live in some really 'hard to live' part of the world.
"just" :)
As far as I know you can't have mismatching drives as a single contiguous space in ZFS. Maybe it has changed.
And when I want to add a new drive of a random size to extend the size of the same directory it's again a pain of adding and extending stuff.
As far as I know you can't have mismatching drives as a single contiguous space in ZFS. Maybe it has changed.
And when I want to add a new drive of a random size to extend the size of the same directory it's again a pain of adding and extending stuff.
> As far as I know you can't have mismatching drives as a single contiguous space in ZFS. Maybe it has changed.
Yes you can, example below:
But with ZFS you have plenty of options to use disks of different sizes.
Example below, RAIDZ from 4 disks of 2TB and added a mirror of two 5TB disks to the same ZFS pool.
Regards,
vermaden
Yes you can, example below:
vermaden ~ % truncate -s 1G FILE.0
vermaden ~ % truncate -s 2G FILE.1
vermaden ~ % mdconfig.sh -c FILE.0
IN: created vnode at /dev/md0
vermaden ~ % mdconfig.sh -c FILE.1
IN: created vnode at /dev/md1
vermaden ~ % lsblk md0
DEVICE MAJ:MIN SIZE TYPE LABEL MOUNT
md0 2:17 1.0G zfs - -
vermaden ~ % lsblk md1
DEVICE MAJ:MIN SIZE TYPE LABEL MOUNT
md1 2:18 2.0G zfs - -
vermaden ~ % doas zpool create test md0 md1
vermaden ~ % zpool status test
pool: test
state: ONLINE
config:
NAME STATE READ WRITE CKSUM
test ONLINE 0 0 0
md0 ONLINE 0 0 0
md1 ONLINE 0 0 0
errors: No known data errors
But with ZFS you have plenty of options to use disks of different sizes.
Example below, RAIDZ from 4 disks of 2TB and added a mirror of two 5TB disks to the same ZFS pool.
vermaden ~ % doas zpool create test raidz md{0,1,2,3}
vermaden ~ % zpool status test
pool: test
state: ONLINE
config:
NAME STATE READ WRITE CKSUM
test ONLINE 0 0 0
raidz1-0 ONLINE 0 0 0
md0 ONLINE 0 0 0
md1 ONLINE 0 0 0
md2 ONLINE 0 0 0
md3 ONLINE 0 0 0
errors: No known data errors
vermaden ~ % doas zpool add test mirror md4 md5
vermaden ~ % zpool status test
pool: test
state: ONLINE
config:
NAME STATE READ WRITE CKSUM
test ONLINE 0 0 0
raidz1-0 ONLINE 0 0 0
md0 ONLINE 0 0 0
md1 ONLINE 0 0 0
md2 ONLINE 0 0 0
md3 ONLINE 0 0 0
mirror-1 ONLINE 0 0 0
md4 ONLINE 0 0 0
md5 ONLINE 0 0 0
errors: No known data errors
Regards,
vermaden
Yup, my personal setup involves 2 ZFS pools of 3 drives each in RAIDZ1, 2 NVMe drives partitioned up using LVM. Each pool gets a partition on each of the NVMe drives for a read cache. The primary/larger pool gets a partition that is mirrored across the 2 NVMe drives as a log/write cache. The system has 64GB of ECC DDR4 and a 10Gbe NIC. As far as I could tell, there was exactly 1 motherboard that actually allowed for this setup (2 full speed NVMe, ECC support, AMD4 socket), so it was not easy, but you can do it!
This is my 4th NAS iteration, so I've worked out exactly how I want it over the course of about 10 years. Partial parts list here: https://pcpartpicker.com/user/chiefnoah/saved/CxjNyc
This is my 4th NAS iteration, so I've worked out exactly how I want it over the course of about 10 years. Partial parts list here: https://pcpartpicker.com/user/chiefnoah/saved/CxjNyc
That was my initial thought too. I clicked hoping for some new chassis/hardware that would make things easier.
Like maybe a rack mounted low power quad core with lots of memory slots, 6 or so 3.5” spinny slots, and some hot swap nvme bays for an affordable price.
I dont need exotic hw, I just want the ability to have some big slow storage mixed with some fast storage.
My current set is a 1U atom setup with an esas card hooked up to a 1U drive shelf. With dual bonded 10gbps uplink.
Like maybe a rack mounted low power quad core with lots of memory slots, 6 or so 3.5” spinny slots, and some hot swap nvme bays for an affordable price.
I dont need exotic hw, I just want the ability to have some big slow storage mixed with some fast storage.
My current set is a 1U atom setup with an esas card hooked up to a 1U drive shelf. With dual bonded 10gbps uplink.
I agree with You.
I believe there would be a 'follow up' to this article when 8 TB (or at least 6 TB) NVMe SSD modules will be available on 2280 and 2242 sizes :)
Then I would make ZFS mirror of them ... and hopefully with ECC RAM.
I believe there would be a 'follow up' to this article when 8 TB (or at least 6 TB) NVMe SSD modules will be available on 2280 and 2242 sizes :)
Then I would make ZFS mirror of them ... and hopefully with ECC RAM.
I’ve had unrecoverable data errors in mirrored ZFS pools before. I had remote backup of course, but now instead of mirror I use ZFS pools with many disks and set up with groups of raidz-3.
It seems that almost every DIY-NAS article these days is an excercise on minimizing power consumption. People seem to forget the purpose of the NAS: storage and (secondarily) expandability.
SSDs are getting cheaper, but their capacity is still lagging. E.g. a 4TB Samsung NVMe drive is >2.5 times more expensive than a same capacity NAS-grade HDD, and ~4.5 times more expensive per GB than a NAS-grade 18TB HDD at the same price.
What is the point of bean-counting on KWhs if the $ you pay for storage will either burn a hole in your pocket, or leave you with a NAS that will be full in a year and virtually no expandability options?
SSDs are getting cheaper, but their capacity is still lagging. E.g. a 4TB Samsung NVMe drive is >2.5 times more expensive than a same capacity NAS-grade HDD, and ~4.5 times more expensive per GB than a NAS-grade 18TB HDD at the same price.
What is the point of bean-counting on KWhs if the $ you pay for storage will either burn a hole in your pocket, or leave you with a NAS that will be full in a year and virtually no expandability options?
I recently set up one of these: https://www.lincplustech.com/products/lincstation-n1-network...
and it seems to strike a pretty excellent balance. 4x NVME + 2x 2.5" can get you a lot of storage, but it's still using just a 15W CPU. Build quality is quite good as well.
> E.g. a 4TB Samsung NVMe drive is >2.5 times more expensive than a same capacity NAS-grade HDD
Samsung is far from the only player in the NVMe space. DRAM-less NVMe drives are quite a bit cheaper (~$50/TB), which if you only want 1-2TB is a great option - ie, if you're just looking for a personal dropbox equivalent. Which was exactly my use case for the lincstation - I just wanted to own my own documents, I didn't need to backup blurays or whatever.
and it seems to strike a pretty excellent balance. 4x NVME + 2x 2.5" can get you a lot of storage, but it's still using just a 15W CPU. Build quality is quite good as well.
> E.g. a 4TB Samsung NVMe drive is >2.5 times more expensive than a same capacity NAS-grade HDD
Samsung is far from the only player in the NVMe space. DRAM-less NVMe drives are quite a bit cheaper (~$50/TB), which if you only want 1-2TB is a great option - ie, if you're just looking for a personal dropbox equivalent. Which was exactly my use case for the lincstation - I just wanted to own my own documents, I didn't need to backup blurays or whatever.
In my case I also use the "NAS" for other things like home assistant and plex. So it's not just capacity and power consumption but also CPU. Especially if you are living in a household with photovoltaics a few kW extra don't matter.
Die you consider Fujitsu Celsius Workstations (e.g. Celsius W570 / W580)?
These machines are pretty big, but cheap and i managed to get a
These machines are pretty big, but cheap and i managed to get a
Fujitsu D3417-A board
Bojiadafast Adapter 24-16pin aliex.
160W streacom nano
Xeon 1225v5
64GB ECC RAM
WD SN850x 2tb nvme
7 WattsTo be honest - no - I already have something 'big' - https://vermaden.wordpress.com/2023/07/30/amd-based-freebsd-... - described here in details - and needed something tiny :)
... but thank You for another point of view - these 'Celsius' machines does look interesting - need to spend some time to check them.
... but thank You for another point of view - these 'Celsius' machines does look interesting - need to spend some time to check them.
There is also the FriendlyElec NAS Kit[1][2] and the Odroid H4 (Ultra) [3], only if someone else might consider this.
I'd personally prefer the Gigabyte MC12-LE0 Board for 50 Bucks with a Ryzen 5600 / 5650, which supports IPMI and Bifurcation. However, the power consumption exceeds 15W in most cases and the NVMe is stepped down to something around 1GB/s... and no 2.5Gbe without an extra card.
1: https://www.friendlyelec.com/index.php?route=product/product...
2: https://www.youtube.com/watch?v=QsM6b5yix0U&embeds_referring...
3: https://www.hardkernel.com/shop/odroid-h4/
I'd personally prefer the Gigabyte MC12-LE0 Board for 50 Bucks with a Ryzen 5600 / 5650, which supports IPMI and Bifurcation. However, the power consumption exceeds 15W in most cases and the NVMe is stepped down to something around 1GB/s... and no 2.5Gbe without an extra card.
1: https://www.friendlyelec.com/index.php?route=product/product...
2: https://www.youtube.com/watch?v=QsM6b5yix0U&embeds_referring...
3: https://www.hardkernel.com/shop/odroid-h4/
I'll never use a NAS without ECC RAM for anything else than disposable content (i.e. downloaded stuff).
It will take 100 hours to consume 1 KWh at 10 W. So at, say, 10 cents per KWh, this amounts to like ten dollars per year of 24/365 operation or to $800 for a full human life.
This is something that needs to be taken into account if one is planning to buy a new machine to reduce power draw by like 5W.
This is not taking into account the footprint of producing that extra machine earlier than it would need to be produced.
This is something that needs to be taken into account if one is planning to buy a new machine to reduce power draw by like 5W.
This is not taking into account the footprint of producing that extra machine earlier than it would need to be produced.
When spending money to limit electricity consumption by such trivial amounts it makes sense to have a look at spending the same amount of money to generate electricity. If you already have solar panels on a roof it is probably much more cost effective to just add a panel or two to offset the power consumption of whatever gadgetry you're running than to spend that money on more power-efficient widgets. If you don't have solar panels you can get a balcony solar installation for around €200 which easily offsets the power used by your existing NAS server assuming you're not using an ex-lease rack-mount multi-purpose server like I do. If you do use one of those you'll have to spend a bit more to get a bigger setup but even in that case it probably ends up being more cost-effective to produce your own power - a single 400W panel covers the power use of a rack-mount server after all.
Many places have much higher electricity costs than that. More power also means more heat and cooling. (Yes, 5W isn’t much either way.)
Here in EU SSD prices are much higher now but anyway for me the point is not much going "mini" but being accessible and upgradable, in a home setup I do not care much "the NAS" but the homeserver: I do not have a datacenter-as-a-computer where some machines host a gazillion of nvme serving them via nvme or iscsi and so on to some blades allowing easier physical storage administration and greater flexibility, I typically have some desktops used as desktops, a homeserver or two (the second as a spare), so the main point is having a machine to do anything not just backup. A classic desktop rackmounted in a 4-5U, a switch, a patch panel, a UPS, a "router" it's the machine room. Having many small boxes is not much comfy to manage in case of hw issues, having a rack, well accessible, with any component easy to replace is valuable to me.
Since the home already consume much more than few W for ventilation and so on the consumption of a modern PC does not change much, it's not an old multi-opteron server.
Since the home already consume much more than few W for ventilation and so on the consumption of a modern PC does not change much, it's not an old multi-opteron server.
I've written up my experience (going into the 10th year now!) with a DIY home NAS [1],[2]. I recommend a dual setup:
- a software RAID 1 for important data (eg. contracts, receipts, emails, family pictures) paired with par2archive to counter bit rot - a JBOD with snapraid for large media collections
The posts go into details, eg. properly configuring ext4.
[1] https://blog.georgovassilis.com/2020/04/01/building-the-perf...
[2] https://blog.georgovassilis.com/2022/10/28/my-home-nas-setup...
- a software RAID 1 for important data (eg. contracts, receipts, emails, family pictures) paired with par2archive to counter bit rot - a JBOD with snapraid for large media collections
The posts go into details, eg. properly configuring ext4.
[1] https://blog.georgovassilis.com/2020/04/01/building-the-perf...
[2] https://blog.georgovassilis.com/2022/10/28/my-home-nas-setup...
OS for home NAS I highly recommend Unraid. I have enough trouble at work dealing with OS upgrades, commands, configurations, etc. I am happy that Unraid just works and is worth every penny, huge community of plugins, etc.
[deleted]
I mix and match components from various computers and drop debian on them then run docker containers for various purposes. I've given my parents, cousins and friends Frankenstein servers like this.
You'd be amazed at what people throw away, not uncommon to find working laptops with no passwords that log straight into Windows filled with all kinds of family photos.
Sometimes unlocked iPhones from 5 years ago. It's a sick world we live in.
We deserve everything thats coming for us.