What is https://httpyac.github.io/ missing in comparison to postman, insomnia or hopscotch? I really like being able to run .http files as unit tests in ci as well as during debugging.
I would be happy to hear what you think of ownCloud Infinite Scale, a go microservice implementation of the ownCloud WebDAV and sharing APIs. We are limiting ourselves to file sync and share, but integrations with collabora and OnlyOffice are there. Just don't expect server side encryption. It was only added to the oc10 codebase to prevent storage admins to read the data. It does not protect against evil sysadmins in the least. Imo an encrypted filesystem or overlay filesystem can be used to solve that use case.
If you need E2EE cryptomator works just fine.
That being said, oCIS does add other things to the table: for an out of box experience we bundle a small OpenID Connect Identity provider that should suffice for a small home lab.
And we now support parts of the libre graph API, which is an effort to build a federated graph API closely following Microsoft Graph. Obviously, our team focuses on file sync and share, but e.g. the Kopano guys are working on the PIM parts.
The team is busy preparing the next release and customer projects based on the new stack, but we are happy with what we have achieved so far.
Back in 2011 we built yatta solutions as a startup around template based roundtrip engineering. UML lab would use templates to not only generate the source code from UML diagrams but also to 'parse' the code.
I don't know the current state of https://www.uml-lab.com/en/uml-lab/ because I left for other adventures, but if you are looking for a tool that can actually keep code and UML in sync give it a try.
Hm, let me give a short history of owncloud reva and ocis.
Many years ago CERN chose ownCloud over other solutions because of the state based sync. They could use that to let researchers sync petabytes of data residin on EOS, their custom built storage solution. That ownCloud supports custom storage implementations made this a lot easier.
The did suffer some database bottlenecks and decided to extend EOS with features that the sync clients needs: tree modification time propagation (so that the etag of the root changes when anything in the tree changes) and size tree accounting (so you can see how many bytes are hidden in a folder including all children).
The basically maintained a fork that was half PHP, half c++. Together, we added APIs and interfaces to the codebase to make the file cache implementation exchangeable.
They went ahead and implemented a golang service that could serve the api requests, while the web ui was rendered in PHP. It should be possible to dig into the details by looking at the public CernBox repo. Code archeology ;)
Anyway, all that was before the nextloud fork.
To be honest, CERN has tried to convince ownCloud to switch to a different architecture for years. With some long held opinions leaving the company we were free to reassess our options.
ownCloud has long had problems with long running operations being killed by eg. php-fpm timeouts. So I was evaluating and comparing different PHP frameworks like reactphp, swoole and amphp. I wanted to be able to offload workflows that are triggered after an upload has finished to a proper background job. And I really wanted to stay in PHP land because of all the already written code and existing apps.
But I noticed that they all had one thing in common. The all reimplemented a redis and a mysql library which made me wonder why?The existing drivers would block network IO, killing every concurrency gains you could gain by using a reactive framework or the go like coroutines of swoole.
It finally dawned on me that PHP may not be the best language to implement a service that has to deal with file IO. A systems language is more fitting.
Go or rust? Two years ago that was way easier to answer. Furthermore, CERN had a working server side API of the ownCloud webdav and ocs endpoints written in golang.
We sat down and discussed how o file sync and share solution would have to look like at the protocol level. What services are necessary and how could we make the existing code more modular to support other storage backend a than EOS? How can we get rid of the centralised database?
The result were changes to the CS3 api, making reva a reference implementation and using ocis to tie it together with user management openid connect and thinking about migration strategies. There is still the possibility to wrap PHP in a sandbox like service using roadrunner.
I would not have dared to start from scratch. But with an existing codebase that was used in production the decision became easier. The story is not over, yet.
Yes, we are leaving things behind. But we can embrace new things as well. And I am happy to be able to work with an awesome team to see this through. Every helping hand is welcome.
No, but thx for the pointer. AFAICT the difference in the sync protocol is a block (syncthing) vs file based (owncloud) protocol. There are several reasons why we prefer the file based protocol.
Currently, all our clients sync using webdav. Changing the protocol would require rewriting that part of every client.
Another reason is that ownCloud as well as ocis are used to access files that resides in other existing storages, eg. S3 or CEPH. Translating those protocols to a file based sync is a lot easier than adapting syncthing (just my gut feeling).
Another aspect is that the file based sync is also state based. We can use the etag to detect changes and immediately start syncing in a breadth first approach. We are not doing that, yet. But the windows cloud vfs we implemented for the desktop clients makes this one of the next steps.
The sync in ownCloud feels slow because the clients are polling. There is currently no persistent connection that the server could use to push changes. But to be honest we have been bitten by firewalls between server and client so often that I doubt we can do real push notifications. Which is why I personally am aiming for long polling. For mobile devices we have to rely on the existing notification infrastructure from Google and apple anyway. Anyway, there are ways to speed up sync. Which switching to go made a lot more implementable. But one step after another.
We already implemented TUS for uploading files in ocis and all clients. It is a well designed extendable upload protocol that covers a lot of corner cases we have experienced first hand with all our clients. We contributed our experiences and are planning a batch extension that we would use to group lots of small files in a single upload. Delta sync should be a TUS extension as well, IMO. But that is mostly in my head.
First, we want to get the basic file sync and share features fully implemented.
So ... Yeah ... Changing the language and the architecture kind of made a knot disappear and we are moving forward.
In that regard, I personally have an eye on the Microsoft graph api which we could use for file management. Input welcome.
Lukas! Yep, for now, we focus on file sync and share.
Personally, I am looking into kopano or etesync to replace calendar and contacts. We already embed kopano konnectd as the OpenID provider so, there are synergies.
We do maintain backwards compatibility for the WebDAV, OCS and OCM apis and the clients are all working. The inner hook system of oc 10 obviously is gone, just as well as the internal PHP APIs. Instead we now rely on the CS3 APIs that describe a set of services in a microservice architecture to extend or customize functionality. Something that has always been fragile, because a bug in an oc/nc app could affect all other services.
Furthermore, we did not have to start from scratch but contributed to reva, which has been powering CernBox for a few years. It is the reference implementation for the CS3 api.
So, we are very much Protocol driven, which makes changing the implementation language possible if needed. And everyone is free to implement his own service replacement if he needs to. The extensive acceptance testsuite will tell you what is not working compared to oc 10 or ocis. For now, we are pretty happy with go and the initial benchmarks.
One more thing regarding core developers having moved from oc to nc. The server is just one piece of the puzzle. None, literally, none of the desktop, iOS or Android developers paid by ownCloud moved to the fork. It feels as if they were not even considered important. The sync protocol and the end user experience on his daily driver really is what matters. I am very happy that we as ownCloud finally grabbed the chance to really tackle the file sync and share part with the right tools for the job. Without being pulled in a gazillion of directions. I cannot express my gratitude for the rest of the oc sales and support team that keeps oc 10 well running and bought us the time to actually make this step. Truly exciting times
Keep in mind this is still tech preview. There is still a lot of work to do. Helping hands welcome. And yes we need to work on our communication skills...
Full disclosure: oc employee no 7 and ocis tech lead here. Originally hired to work on full text search in ownCloud 10. Good old times. Cheers to all the oc and nc devs: it should be possible to implement the internal PHP API of oc/nc using roadrunner. That might allow existing PHP apps to be wrapped in a dedicated microservice. But that is not something we are investing time in. Feel free to ping me on https://talk.owncloud.com