Mayastor: Lightning Fast Storage for Kubernetes(percona.com)
percona.com
Mayastor: Lightning Fast Storage for Kubernetes
https://www.percona.com/community-blog/2020/10/23/mayastor-lightning-fast-storage-for-kubernetes/
30 comments
Trying to filter all the marketing speak since there is almost zero info on the actual technology of Mayastor engine. Is this thing basically NVMf array orchestrator for k8s? So if your NVMf target crashes your PV goes bye bye?
According to https://blog.mayadata.io/openebs-mayastor-0.3.0-lands-soon-k...
> But the nexus is also capable of performing transformations on the I/O passing through it. For example, for reasons of availability and durability, we might wish to maintain more than one copy of the data contained by a PV. The nexus supports this by dispatching multiple copies of any writes which are received for the volume, to replicas hosted on other Mayastor Storage Nodes within the cluster (the actual replica count is defined by the Volume’s Storage Class). Only when all replicas have acknowledged their writes will the nexus signal completion of the transaction back to the consumer. That is to say, policy-based workload protection in Mayastor is based on synchronous replication.
Presumably life can go on so long as a replica survives.
> But the nexus is also capable of performing transformations on the I/O passing through it. For example, for reasons of availability and durability, we might wish to maintain more than one copy of the data contained by a PV. The nexus supports this by dispatching multiple copies of any writes which are received for the volume, to replicas hosted on other Mayastor Storage Nodes within the cluster (the actual replica count is defined by the Volume’s Storage Class). Only when all replicas have acknowledged their writes will the nexus signal completion of the transaction back to the consumer. That is to say, policy-based workload protection in Mayastor is based on synchronous replication.
Presumably life can go on so long as a replica survives.
Ah i see so it’s kind of like network raid1... I’m guessing this still implies that you can’t write while backup target is offline?
Later in the same blog post:
> MOAC’s response to the available replica count of a Volume falling below the desired count defined within its Storage Class is to instruct a Mayastor Storage Node to create a new, “empty” replica, and to share it with the degraded nexus. The introduction of a new replica to the nexus will cause it to start a rebuild process, bringing it into synchronization with the other replicas. This process can successfully complete without having to suspend workload I/O to the affected PV.
> MOAC’s response to the available replica count of a Volume falling below the desired count defined within its Storage Class is to instruct a Mayastor Storage Node to create a new, “empty” replica, and to share it with the degraded nexus. The introduction of a new replica to the nexus will cause it to start a rebuild process, bringing it into synchronization with the other replicas. This process can successfully complete without having to suspend workload I/O to the affected PV.
Your PV would likely still exist but the `mount` command on the pod's cgroup will fail and you'll get into a crashloop with a NotReady (I think).
Yes by “bye bye” i meant your workload writes or reads are going to fail/stall and all sorts of bad things happen
From the above: "This process can successfully complete without having to suspend workload I/O to the affected PV"
Just want to say for anyone running a clusters of machinese (with kubernetes for example) -- OpenEBS provides one of the most approachable and easily to use solutions for storage out there. It scales from hobby grade to enterprise grade in my opinion, and although I can't point you to a 1000+ node cluster I run or anything (maybe OpenEBS can), they offer enterprise editions and support which is usually enough to qualify as the minimum viable definition of "enterprise grade".
OpenEBS has many different drivers[0] (which is a good sign IMO) I only use one of them but am planning a post (that may never actually get written) to expand on all of them if/when I can:
- Jiva[1] (based on Longhorn[1], iSCSI-based, bring your own disk driver)
- cStor[2] (baked by OpenEBS, zfs-based used to be the golden child before MayaStor)
- MayaStor (written in rust, iSCSI/NVMe-oF based, bring your own lower layer)
- zfs-localpv[3]
Funnily enough, I've been really interested in zfs and btrfs lately so have been learning a lot about the field, but cStor didn't work when I first ran it.
I know "written in rust" has become somewhat of a buzzword, but don't forget that it has actual meaning -- it means that they will be able to get the raw performance possible without a language runtime or garbage collection to hold them back. This is critical for "systems" programming, and is the domain rust was built for. When I see rust, I basically think -- this could be as fast as average C (and safer) if you get even just pretty good developers.
Also a note on this -- one of the things that's nice about OpenEBS's offerings is that writes are synchronous. This is obviously a problem[4] for performance of course, but it's a massive relief for building resiliency into systems. I'm not a classical sysadmin, but it puts me at ease that I can give up some throughput to know that if one of my servers goes completely kaput, the other can pick up at the last write on the other node, with zero guesswork on my end. The switchover is of course a little hairy but this is what OpenEBS smoothes out for you.
For this not to be completely an ad for OpenEBS, I do want to note that the other prominent option that scales from hobbyist to enterprise (and arguably more so) is Rook[5] which manages Rook clusters. Rook the organization seems to be branching out to position themselves as Cassandra storage and a bunch of other stuff, but the key thing they brought to the table was a mostly hands-off operator for operating Ceph[6] which is certified enterprise grade(tm) FREE software. Ceph is basically the linux of the storage appliance world.
Ceph (via Rook) is another really good choice for getting storage up and running on a kubernetes (or whatever else) because it offers similar benefits to OpenEBS's solutions but does it slightly different. Instead of container-attached-storage, it solves the features come at the filesystem layer-- you get a tunable "ceph" installation (you should definitely choose BlueStore btw) which gives you:
- replication (like the iSCSI approaches of OpenEBS)
- checksumming (like ZFS and zfs-backed OpenEBS)[7]
- striping (like ZFS/btrfs and zfs-backed OpenEBS depending on setup)
- compression (like ZFS)[7]
Anyway, I don't run any servers that use NVMe but nice to see this article, hope the people at OpenEBS are still doing well, love their entry into the space.
[EDIT] - I totally forgot and should mention, you can obviously always bring your own lower layer, ex creating a xfs-formatted ZVOL on top of ZFS on the host and then exposing that drive to whatever. Excellent example is a recent talk from Japan which I've mentioned before[8][9] which uses this to run Ceph on ZFS.
[0]: https://docs.openebs.io/docs/next/casengines.html
[1]: https://docs.openebs.io/docs/next/jiva.html
[2]: https://docs.openebs.io/docs/next/alphafeatures.html#mayasto...
[3]: https://github.com/openebs/zfs-localpv/
[4]: https://github.com/longhorn/longhorn/issues/1242#issuecommen...
[5]: https://rook.io/docs/rook/
[6]: https://docs.ceph.com/en/latest/architecture/
[7]: https://ceph.io/community/new-luminous-bluestore/
[8]: https://speakerdeck.com/takutakahashi/ceph-on-zfs
[9]: https://news.ycombinator.com/item?id=24784596
OpenEBS has many different drivers[0] (which is a good sign IMO) I only use one of them but am planning a post (that may never actually get written) to expand on all of them if/when I can:
- Jiva[1] (based on Longhorn[1], iSCSI-based, bring your own disk driver)
- cStor[2] (baked by OpenEBS, zfs-based used to be the golden child before MayaStor)
- MayaStor (written in rust, iSCSI/NVMe-oF based, bring your own lower layer)
- zfs-localpv[3]
Funnily enough, I've been really interested in zfs and btrfs lately so have been learning a lot about the field, but cStor didn't work when I first ran it.
I know "written in rust" has become somewhat of a buzzword, but don't forget that it has actual meaning -- it means that they will be able to get the raw performance possible without a language runtime or garbage collection to hold them back. This is critical for "systems" programming, and is the domain rust was built for. When I see rust, I basically think -- this could be as fast as average C (and safer) if you get even just pretty good developers.
Also a note on this -- one of the things that's nice about OpenEBS's offerings is that writes are synchronous. This is obviously a problem[4] for performance of course, but it's a massive relief for building resiliency into systems. I'm not a classical sysadmin, but it puts me at ease that I can give up some throughput to know that if one of my servers goes completely kaput, the other can pick up at the last write on the other node, with zero guesswork on my end. The switchover is of course a little hairy but this is what OpenEBS smoothes out for you.
For this not to be completely an ad for OpenEBS, I do want to note that the other prominent option that scales from hobbyist to enterprise (and arguably more so) is Rook[5] which manages Rook clusters. Rook the organization seems to be branching out to position themselves as Cassandra storage and a bunch of other stuff, but the key thing they brought to the table was a mostly hands-off operator for operating Ceph[6] which is certified enterprise grade(tm) FREE software. Ceph is basically the linux of the storage appliance world.
Ceph (via Rook) is another really good choice for getting storage up and running on a kubernetes (or whatever else) because it offers similar benefits to OpenEBS's solutions but does it slightly different. Instead of container-attached-storage, it solves the features come at the filesystem layer-- you get a tunable "ceph" installation (you should definitely choose BlueStore btw) which gives you:
- replication (like the iSCSI approaches of OpenEBS)
- checksumming (like ZFS and zfs-backed OpenEBS)[7]
- striping (like ZFS/btrfs and zfs-backed OpenEBS depending on setup)
- compression (like ZFS)[7]
Anyway, I don't run any servers that use NVMe but nice to see this article, hope the people at OpenEBS are still doing well, love their entry into the space.
[EDIT] - I totally forgot and should mention, you can obviously always bring your own lower layer, ex creating a xfs-formatted ZVOL on top of ZFS on the host and then exposing that drive to whatever. Excellent example is a recent talk from Japan which I've mentioned before[8][9] which uses this to run Ceph on ZFS.
[0]: https://docs.openebs.io/docs/next/casengines.html
[1]: https://docs.openebs.io/docs/next/jiva.html
[2]: https://docs.openebs.io/docs/next/alphafeatures.html#mayasto...
[3]: https://github.com/openebs/zfs-localpv/
[4]: https://github.com/longhorn/longhorn/issues/1242#issuecommen...
[5]: https://rook.io/docs/rook/
[6]: https://docs.ceph.com/en/latest/architecture/
[7]: https://ceph.io/community/new-luminous-bluestore/
[8]: https://speakerdeck.com/takutakahashi/ceph-on-zfs
[9]: https://news.ycombinator.com/item?id=24784596
One important aspect that should be mentioned is that OpenEBS is essentially a block storage system (think virtual hard drives, which need to be formatted with dfs/ext4) and by its nature can't expose the same files to multiple pods.
Ceph(FS) and other distributed file systems allow you to access your data from any number of machines. Also they provide faster and more robust failover as they don't have to unplug the virtual hard drive and repair the local file system.
Ceph(FS) and other distributed file systems allow you to access your data from any number of machines. Also they provide faster and more robust failover as they don't have to unplug the virtual hard drive and repair the local file system.
> by its nature can't expose the same files to multiple pods.
I don't know OpenEBS, but this isn't true of block storage systems in general. Both AWS and Google allow mounting virtual disks to multiple pods or nodes if all but one are mounted read-only.
Even multi-node writing is possible - see https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ebs-volu... :
> Amazon EBS Multi-Attach enables you to attach a single Provisioned IOPS SSD (io1) volume to up to 16 Nitro-based instances that are in the same Availability Zone. You can attach multiple Multi-Attach enabled volumes to an instance or set of instances. Each instance to which the volume is attached has full read and write permission to the shared volume.
Google offers this too: https://cloud.google.com/sdk/gcloud/reference/beta/compute/d...
Of course this does have constraints compare to a true distributed filesystem, which the AWS page describes.
I don't know OpenEBS, but this isn't true of block storage systems in general. Both AWS and Google allow mounting virtual disks to multiple pods or nodes if all but one are mounted read-only.
Even multi-node writing is possible - see https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ebs-volu... :
> Amazon EBS Multi-Attach enables you to attach a single Provisioned IOPS SSD (io1) volume to up to 16 Nitro-based instances that are in the same Availability Zone. You can attach multiple Multi-Attach enabled volumes to an instance or set of instances. Each instance to which the volume is attached has full read and write permission to the shared volume.
Google offers this too: https://cloud.google.com/sdk/gcloud/reference/beta/compute/d...
Of course this does have constraints compare to a true distributed filesystem, which the AWS page describes.
You are able to attach the block devices, but local file systems do not support being mounted on more than one host.
For that you need to use a cluster file system, but that's a different beast.
For that you need to use a cluster file system, but that's a different beast.
The multi-reader scenario definitely supports multiple readers of the same file system - we use it both in and out of Kubernetes on GCP.
Kubernetes has specific support for this with its ReadOnlyMany access mode: https://kubernetes.io/docs/concepts/storage/persistent-volum...
However, I do see on that page that AWS EBS doesn't support this mode, so perhaps that's what you're thinking of.
Kubernetes has specific support for this with its ReadOnlyMany access mode: https://kubernetes.io/docs/concepts/storage/persistent-volum...
However, I do see on that page that AWS EBS doesn't support this mode, so perhaps that's what you're thinking of.
Would you mind going into detail on your setup? What do you use to do this outside and inside GCP?
Both scenarios I was referring to are on GCP, but one is on k8s (GKE) and one uses ordinary GCP VMs. In both cases we're using the multi-reader mounting feature of GCP persistent disks, all that changes is how that's specified.
Outside GCP, we do have an on-premise product that can deploy on any k8s cluster. In that case, the default is for the customer to provide an NFS mount for the data in question. We use the same k8s ReadOnlyMany access type for that.
Outside GCP, we do have an on-premise product that can deploy on any k8s cluster. In that case, the default is for the customer to provide an NFS mount for the data in question. We use the same k8s ReadOnlyMany access type for that.
As layman, which has been not following all the file store tech for a while. Is GFS still a thing and useful ?
By GFS do you mean GlusterFS[0]? I never ended up using it because it was seemingly bad for databases -- it is probably nuseful but in comparison it wasn't that great.
[0]: https://www.gluster.org/
[0]: https://www.gluster.org/
Ahh sorry yes, I should have been more clear. Yes Gluster.
Thanks for feedback, I'm currently looking for a dist-file-store. Doesn't need to be super-fast just reliant and distributed.
So I have no idea what would work for you and am not in a position to make a solid recommendation.... buuuut this is the internet so I'll list some possibilities in order of perceived production worthiness:
- Good ol' trusty NFS[0][1] (of course someone has to take an arrow to the knee and become an NFS admin essentially)
- CephFS[2] via Rook Shared filesystem[3]
- Swift[4] though it looks like a huge PITA to deploy outside of openstack, almost like it's hard on purpose
- Dragonfly[5]? I remember think of it for containers but it's technically for random files too
- SeaweedFS[6]? It's got a comparison section which is a good sign??
- Could you just... offload to some S3-like thing?
[0]: https://kubernetesguide.com/kubernetes/nfs-volumes-in-kubern...
[1]: https://www.padok.fr/en/blog/readwritemany-nfs-kubernetes
[2]: https://docs.ceph.com/en/latest/cephfs/
[3]: https://rook.io/docs/rook/v1.4/ceph-filesystem-crd.html
[4]: https://docs.openstack.org/swift
[5]: https://github.com/dragonflyoss/Dragonfly
[6]: https://github.com/chrislusf/seaweedfs#compared-to-other-fil...
- Good ol' trusty NFS[0][1] (of course someone has to take an arrow to the knee and become an NFS admin essentially)
- CephFS[2] via Rook Shared filesystem[3]
- Swift[4] though it looks like a huge PITA to deploy outside of openstack, almost like it's hard on purpose
- Dragonfly[5]? I remember think of it for containers but it's technically for random files too
- SeaweedFS[6]? It's got a comparison section which is a good sign??
- Could you just... offload to some S3-like thing?
[0]: https://kubernetesguide.com/kubernetes/nfs-volumes-in-kubern...
[1]: https://www.padok.fr/en/blog/readwritemany-nfs-kubernetes
[2]: https://docs.ceph.com/en/latest/cephfs/
[3]: https://rook.io/docs/rook/v1.4/ceph-filesystem-crd.html
[4]: https://docs.openstack.org/swift
[5]: https://github.com/dragonflyoss/Dragonfly
[6]: https://github.com/chrislusf/seaweedfs#compared-to-other-fil...
Oh wow thanks for taking the time and writing this up !
Lol agreed with NFS. Trying to avoid that yes.
I think I will need to setup some tests and go from there.
Thanks again !
Thanks again !
"For the input you can choose which type of protocol you want through the storage classes. At the output, let's say, you can do iSCSI, NVMe, and local. But the iSCSI work was there mostly for us to get us going because by default we simply use NVMe Over Fabrics or local..."
https://youtu.be/_5MfGMf8PG4?t=2866
EDIT: Thanks for sharing your experience. I'm looking at Mayastor, but remembered this quote from a video I watched a while back suggesting that while iSCSI works, it is probably not the way to the most from this software.
https://youtu.be/_5MfGMf8PG4?t=2866
EDIT: Thanks for sharing your experience. I'm looking at Mayastor, but remembered this quote from a video I watched a while back suggesting that while iSCSI works, it is probably not the way to the most from this software.
Yup this is mayastor -- I noted that (iscsi & NVMe-oF) -- NVMe and NVMe over Fabric.
iSCSI would be 90% of the usecases I'd think though -- unless you know every server you're getting NVMe attached, most providers are SSD or HDD right?
iSCSI would be 90% of the usecases I'd think though -- unless you know every server you're getting NVMe attached, most providers are SSD or HDD right?
NVMe-oF is a transport protocol between an initiator (e.g. Linux nvmf module) and a target (e.g. spdk, Linux nvmf module, etc.) much as iSCSI is a transport protocol. The storage behind the target does not need to be NVMe for NVMe-oF nor does it need to be SCSI for iSCSI.
TIL -- I had no idea NVMe-oF was a transport protocol just like iSCSI was, so rather I should characterize it as NVMe-oF/iSCSI transport protocols. Thanks for the clarification
”This site uses cookies and other tracking technologies to assist with navigation, analyze your use of our products and services, assist with promotional and marketing efforts, allow you to give feedback, and provide content from third parties. If you do not want to accept cookies, adjust your browser settings to deny cookies or exit this site. Cookie Policy”
Great cookie policy there with no easy opt-out.
Great cookie policy there with no easy opt-out.
I think github link would be more useful for most people here: https://github.com/openebs/Mayastor
Here is a competing solution that I am sort of fond of. https://storidge.com/ Note sure how it compares to Mayastor though.
The blast radius of this sort of per workload storage is much smaller than that of shared everything storage such as CEPH.