jaltman · HackerTrans

8 miesięcy temudiscuss

Most likely with other storage systems which are already in use. https://its.umich.edu/projects/afs-retirement

8 miesięcy temudiscuss

Heidelberg shutdown their AFS cell in 2023. https://www.urz.uni-heidelberg.de/en/newsroom/afs-network-fi...

8 miesięcy temudiscuss

AuriStorFS is the next generation AFS with an extended ACL model which might be useful to you. https://www.auristor.com/documentation/man/linux/7/auristorf...

jaltman

9 miesięcy temudiscuss

Correct, and the point-to-point bandwidth is limited by the maximum RX window size because of the bandwidth delay product. As round-trip latency increases, at some point the window size becomes insufficient to keep the pipe full, at which point data transfers stall.

One site which recently lifted and shifted their AFS cell to a cloud made the following observations:

We observed the following performance while copying a 1g file from local disk into AFS.

  AuriStor Client (2021.05-65) -> OpenAFS server (1.6.24): 3m.11s

  AuriStor Client (2021.05-65) -> AuriStor Server (2021.05-65): 1m

  AuriStor Client (2025.00.11) -> AuriStor Server (2025.00.11): 30s

All of the above tests were performed from clients located on campus to fileservers located in in the cloud.

There are many RX implementation differences between the three versions. It is important to note that the window size grows from 32 -> 128 -> 512.

jaltman

9 miesięcy temudiscuss

Not everyone ignored it but unlike nfs it didn't come in the box with the operating system, and you had to pay for it. In addition, AFS provided strong cryptographic authentication and wire privacy which meant that it couldn't be licensed in many countries because the U.S. government did not grant appropriate export licenses.

I often wonder how the world would be different if AFS 3.0 could have been freely distributed world wide in 1989 precluding the need for HTTP to be developed at CERN.

jaltman

9 miesięcy temudiscuss

https://workshop.openafs.org/afsbpw08/wed_keynote.html

https://www.usenix.org/legacy/publications/library/proceedin...

https://workshop.openafs.org/afsbpw08/talks/wed_1/OpenAFS_an...

jaltman

9 miesięcy temudiscuss

DCE DFS (developed at Transarc) was originally supposed to be AFS 4.0 before it was contributed to DCE. After the contribution it became backward incompatible with AFS 3.x. The RPC layer, the authentication protocol, the protection service (user/group management) were all replaced to leverage technology contributions from other DCE participants.

IMO IBM/Transarc died for two reasons. First, there was significant brand confusion after the release of Windows Active Directory and Windows DFS since no trademarks were obtained for DCE service names. Second, the file system couldn't be deployed without the rest of the DCE infrastructure.

There was an unofficial effort within IBM to create the Advanced Distributed File System (ADFS) which would have decoupled DFS from the DCE Cell Directory Service and Security Service as well as replaced DCE/RPC. However, the project never saw the light of day.

https://en.wikipedia.org/wiki/DCE_Distributed_File_System

jaltman

9 miesięcy temudiscuss

The cephfs model of a file system logically constructed from an object store closely mirrors the AFS architecture. The AFS fileserver is horribly misnamed. Whereas AFS 1.0 fileserver exported the contents of local filesystems much as NFS and CIFS do, AFS 2.x/3.x/OpenAFS/AuriStorFS fileservers export objects (aka vnodes) which are stored in an object store. Each AFS vice partition stored zero or more object stores each consisting of the objects belonging to a single volume group. A volume group consists of one or more of the RWVOL, ROVOL and/or BACKVOL instances.

The AFS consistency model is fairly strong. Each client (aka cache manager) is only permitted to access the data/metadata of a vnode if it has been issued a callback promise from the AFS fileserver. File lock transitions, metdata modifications, and data modifications as well as volume transactions cause the fileserver to break the promise. At which point the client is required to fetch updated status information before it can decide it is safe to reuse the locally cached data.

Unlike optimistic locking models, the AFS model permits cached data to be validated after an extended period of time by requesting up to date metadata and a new callback promise.

An AFS fileserver will not permit a client to perform a state changing operation as long as there exist broken callback promises which have yet to be successfully delivered to the client.

jaltman

9 miesięcy temudiscuss

The Amdahl's Law limitations are specific to the implementation and not at all tied to the protocols. The 1990 AFS 3.0 server design was built upon a cooperative threading system ("Light Weight Processes") designed by James Gosling as part of the Andrew Project. Cooperative processing influences the design of the locking model since there isn't any simultaneous between tasks. When the AFS fileserver was converted to pthreads for AFS 3.5, the global state of each library was protected by wrapping it with a global mutex. Each mutex was acquired when entering the library and dropped when exiting it. To complete any fileserver RPC required acquisition of at least six or seven global mutexes depending upon the type of vnode being be accessed. In practice, the global mutexes restricted the fileserver process to 1.7 cores regardless of how many cores were present in the system.

AuriStor's RX and UBIK protocol and implementation improvements would be worthless if the application services couldn't scale. To accomplish this required converting each subsystem so it could operate with minimal lock contention.

This 2023 presentation by Simon Wilkinson describes the improvements that were made to AuriStor's RX implementation up to that point.

https://www.auristor.com/downloads/auristor-rx-hare-and-the-...

The RX tortoise is catching up with the TCP hare.

  Connecting to [10.0.2.15]:2345
  RECV: threads   1, times        1, bytes        2000000000:          881 msec   [18.15 Gbit/s]

comments