I'm working on an evolutionary, accumulate-only database system called SirixDB written in Java (and a module in Kotlin) in my spare time and in my day to day job I'm also working as a software engineer (embedded; besides I'm dancing a lot -- swing dances, mainly Lindy Hop and travel to festivals in Europe often times in my holidays) :-)
Furthermore, I'm working on Brackit (http://brackit.io), a query engine for JSON and XML using a JSONiq like query language. It can be used as ad-hoc in-memory query processor as the query processor of a data store. We envision, that it encapsulates common proven optimizations whereas the individual data stores can add further stages to rewrite the queries for index matching and physical optimizations.
SirixDB (https://sirix.io or https://github.com/sirixdb/sirix) is all about efficient versioning of your data. That is on the one hand it reduces the storage cost of storing a new revision during each transaction-commit while balancing read- and write-performance through a novel sliding snapshot algorithm and dynamic page compression. On the other hand Sirix supports easy query capabilities for instance to open a specific revision by a timestamp or several revisions by a given timespan, to navigate to future or past versions of nodes in the tree-structure and so on. It basically never overwrites data and is heavily inspired by ZFS and Git and borrows some ideas and puts these to test on the sub-file level.
In stark contrast to other approaches SirixDB combines copy-on-write semantics with node-level versioning and does not require a write-ahead-log for consistency.
It all started around 2006 as a university / Ph.D. project of Marc Kramis and I worked on the project since 2007 and already did my Bachelor's Thesis, Master's Thesis as well as several HiWi-Jobs on topics regarding the project and I'm still more eager than ever to put forth the idea of a versioned, analytics plattform to perform analytical tasks based on current as well as the history of the data.
I'm working on a side-project in my spare time since the end of 2012 (before at the university) with some gaps, which is an append-only DBS with time travel capabilities using a custom storage engine based on COW tries and a (still I think) novel page versioning strategy called Sliding Snapshot. Recently I began work on pooling pages for reuse and a custom allocator for variable page sizes. Before, I had created new instances whenever a page was read from disk instead of reusing instances, so the allocation rate was really high for parallel transactions.
I once implemented the backend of a calendar and resource control for a low code platform.
The control is highly customizable, with a lot of views to chose from, daily, monthly, yearly... but also resource views (you can book resources with custom groupings, by plugin, by the resource-ID, whatever...), define "plugins" on the data sources, what's the from- and to- columns, the title column, what's the resource (may be from a foreign key / 1:1 relationship or 1:N if it's from a "child" data source or from the same data source/table).
Furthermore I've implemented different appointment series, to chose from (monthly, weekly (which weekdays), daily...), which column values should be copied. Also appointment conflicts (or only conflicts if they book the same resource). You could also configure buffers before and after appointments where no other appointment can be.
That was a lot of fun and also challenge sometimes regarding time zones and summer/winter time in Europe and so on :-)
So, I think in my local bubble noone is for instance as excited about DB systems as I am, so in essence I thought I could even spend some money to get some expert opinions or rather insights I'm struggling with (currently for instance with bad throughput of my immutable OSS DBS). That said I think noone so far wanted money and some even offered help, but so far I think they didn't have time, thus didn't answer any "pings". So, as I can't spend too much time (and of course not too much money) either on profiling and debugging right now it's kind of a dilemma, as it would IMHO be very interesting to know what's slowing down N read-only trxs in my system :-) that said a couple of years ago I also asked about help with a frontend without much luck. I guess it has to have some value of course, so maybe at least spending some money (even if it's a non profit spare time project since 11 or even more years) should be OK :-)
Throughput. The code can be "suspended" on a blocking call (I/O, where the platform thread usually is wasted, as the CPU has nothing to do during this time). So, the platform thread can do other work in the meantime.
We're using a similar trie structure as the main document (node) index in SirixDB[1]. Lately, I got some inspiration for different page-sizes based on the ART and HAMT basically for the rightmost inner pages (as the node-IDs are generated by a simple sequence generator and thus also all inner pages (we call them IndirectPage) except for the rightmost are fully occupied (the tree height is adapted dynamically depending on the size of the stored data. Currently, always 1024 references are stored to indirect child pages, but I'll experiment with smaller sized, as the inner nodes are simply copied for each new revision, whereas the leaf pages storing the actual data are versioned themselfes with a novel sliding snapshot algorithm.
You can simply compute from a unique nodeId each data is assigned (64bit) the page and reference to traverse on each level in the trie through some bit shifting.
I think it depends, but I wonder if anything can be done about the problem with checked exceptions in lambdas / for instance the streams. I think the enhanced switch with handling failure is only part of the solution, but I'm also a proponent of having only unchecked exceptions.
What about storing the data and thus, the indexes in Kafka. Would it make sense? Let's say currently, I'm storing SirixDB resources in files. However, instead of offsets into a file the index pages could be stored in Kafka optionally (or Pulsar...). Is Kafka too slow for this or only for specific row-tuples? We could make a combined storage caching the pages locally or also storing in the file system and asynchronous storing in Kafk, S3 or whatever.
It's fundamentally how SirixDB approaches this (basically also storing checksums) as also written in another reply :-)
Every commit directly syncs the binary data to the durable storage (currently a file) and incrementally adds data. Furthermore, it stores optionally the changes (type of change/ctx node/updatePosition... in JSON files). For instance, lately I've implemented a simple copy mechanism based on this. Copy a given revision and optionally apply all changes with intermediate commits to also copy the full history up to the most recent revision). However, the main idea is to use the change tracking also for diff visualizations... maybe even stream these via web sockets.
I'm working on an evolutionary, accumulate-only database system called SirixDB written in Java (and a module in Kotlin) in my spare time and in my day to day job I'm also working as a software engineer (embedded; besides I'm dancing a lot -- swing dances, mainly Lindy Hop and travel to festivals in Europe often times in my holidays) :-)
Furthermore, I'm working on Brackit (http://brackit.io), a query engine for JSON and XML using a JSONiq like query language. It can be used as ad-hoc in-memory query processor as the query processor of a data store. We envision, that it encapsulates common proven optimizations whereas the individual data stores can add further stages to rewrite the queries for index matching and physical optimizations.
SirixDB (https://sirix.io or https://github.com/sirixdb/sirix) is all about efficient versioning of your data. That is on the one hand it reduces the storage cost of storing a new revision during each transaction-commit while balancing read- and write-performance through a novel sliding snapshot algorithm and dynamic page compression. On the other hand Sirix supports easy query capabilities for instance to open a specific revision by a timestamp or several revisions by a given timespan, to navigate to future or past versions of nodes in the tree-structure and so on. It basically never overwrites data and is heavily inspired by ZFS and Git and borrows some ideas and puts these to test on the sub-file level.
In stark contrast to other approaches SirixDB combines copy-on-write semantics with node-level versioning and does not require a write-ahead-log for consistency.
It all started around 2006 as a university / Ph.D. project of Marc Kramis and I worked on the project since 2007 and already did my Bachelor's Thesis, Master's Thesis as well as several HiWi-Jobs on topics regarding the project and I'm still more eager than ever to put forth the idea of a versioned, analytics plattform to perform analytical tasks based on current as well as the history of the data.
e-mail address: [email protected]