interesting.
> And that's stored in Dynamodb, which is very much a KV store and it was somewhat of a nightmare to keep track of these things.
what problems did you face to keep track of these things in a KV store?
good thing about badgerdb in our case is badgerDB data files live on the same filesystem as our file based manifests and so in essence its just a fancy file based manifest that supports efficient features like append only operations, partial reads, crash recovery etc. In theory we can implement all such features on our existing manifest files.
With badgerDB we just get them out of the box and snapshots still work as expected. So we are trying to move all the information in manifest files to separate badgerDB instances for each shard.
But till we have source of truth in manifest files as well as badgerDB (while we are in transition), we have to do the dance of synchronizing data between the two.