LiteDB: A .NET embedded NoSQL database(litedb.org)
litedb.org
LiteDB: A .NET embedded NoSQL database
https://www.litedb.org/
47 comments
If you just need a fast key-value map that scales well with threads/cores, might I interest you in non-blocking drop-in replacement for ConcurrentDictionary?
https://www.nuget.org/packages/NonBlocking
https://www.nuget.org/packages/NonBlocking
LiteDB is a database engine, so it needs persistence.
For the link you provided, I wonder if Microsoft would accept a pull request to merge it into the standard library to replace the built-in concurrent dictionary…?
For the link you provided, I wonder if Microsoft would accept a pull request to merge it into the standard library to replace the built-in concurrent dictionary…?
Spot on, this was the original intent of the linked implementation but the PR eventually did not progress and got rejected[0].
However, it might not be off the table yet, because scaling per core is dramatically better especially in moderate to write-heavy scenarios which remains the main use case for ConcurrentDictionary. I have been benchmarking it on arm64 lately and it shows different numbers much more favoring non-blocking implementation without any regressions discussed in the PR.
[0] https://github.com/dotnet/runtime/pull/50337
However, it might not be off the table yet, because scaling per core is dramatically better especially in moderate to write-heavy scenarios which remains the main use case for ConcurrentDictionary. I have been benchmarking it on arm64 lately and it shows different numbers much more favoring non-blocking implementation without any regressions discussed in the PR.
[0] https://github.com/dotnet/runtime/pull/50337
SQLite integrates nicely with .Net and EF Core. Would there be any reason to use this over SQLite with a JSON field?
Saying a document database is the same as a JSON field is a bit reductive.
I think you chose this instead of MongoDB like you would chose SQLite instead of MSSQL. The library is very similar to Mongo's .Net library, so its almost a 1:1 code replacement.
I think you chose this instead of MongoDB like you would chose SQLite instead of MSSQL. The library is very similar to Mongo's .Net library, so its almost a 1:1 code replacement.
I agree. The way I see it, a document DB (I have used Firebase) has features that make it nicer to work with documents. For example Firebase let's you query and index based on the data within the "JSON" (in quotes because it is not really represented in Firebase as JSON, but is close in structure).
Another thing is you might want the lack of features. I.e. you want it so people don't do joins etc. Because you want the team to use the same paradigm with local data as with server hosted. I can think of reasons why.
Another thing is you might want the lack of features. I.e. you want it so people don't do joins etc. Because you want the team to use the same paradigm with local data as with server hosted. I can think of reasons why.
You can create an index based on expressions so you can index fields inside JSON data with sqlite.
SQLite lets you do those JSON things as well.
> I think you chose this instead of MongoDB like you would chose SQLite instead of MSSQL. The library is very similar to Mongo's .Net library, so its almost a 1:1 code replacement.
I guess it depends on what's important.
Some people primarily care about the API. Other people focus on the operational story.
I guess it depends on what's important.
Some people primarily care about the API. Other people focus on the operational story.
Interop overhead makes SQLite in .NET a lot slower than an embedded database, in my experience - though this may be a problem with the stock SQLite wrappers and not an unfixable problem.
In my experience the overhead is only noticeable if you use Entity Framework Core to interface with Sqlite.
The native Sqlite wrapper libraries such as Microsoft's Microsoft.Data.Sqlite have faster performance than EF, but because you're forced to pass around SQL strings, most .NET dev's don't prefer to use them.
The native Sqlite wrapper libraries such as Microsoft's Microsoft.Data.Sqlite have faster performance than EF, but because you're forced to pass around SQL strings, most .NET dev's don't prefer to use them.
I would suggest using system.data.sqlite instead, it's the official version from sqlite and is actively maintained. Additionally, it's self contained whereas the Microsoft version requires separate dlls. Personally I've used dapper as a wrapper over sqlite and had decent performance but you do have to be aware of transactions and increase the cache size (and maybe set wal mode). There's also things like sqlite-net which looks promising but I haven't tried it so I can't comment on perf.
Perf with litedb is pretty awful in my experience if you're using it as a schemaful db. But it's probably the wrong tool for the job if you're comparing to sqlite. I've yet to come across any embedded db that can do schema + document with decent performance.
Perf with litedb is pretty awful in my experience if you're using it as a schemaful db. But it's probably the wrong tool for the job if you're comparing to sqlite. I've yet to come across any embedded db that can do schema + document with decent performance.
Microsoft.Data.Sqlite makes it easier to use encrypted databases compared to System.Data.Sqlite out of the box.
I am yet to find a modern up-to-date tutorial and implementation of System.Data.Sqlite that offers encrypted database support without making my brain melt.
I am yet to find a modern up-to-date tutorial and implementation of System.Data.Sqlite that offers encrypted database support without making my brain melt.
A few years ago, I needed faster SQLite interop than Microsoft.Data.Sqlite, and ended up writing my own: https://github.com/zmj/sqlite-fast
The ergonomics could be better, but this enabled query execution to avoid heap allocations and reflection.
The ergonomics could be better, but this enabled query execution to avoid heap allocations and reflection.
I wonder if anyone's working on compiling SQLite's C code into C# so it can run without interop.
One reason:
SQLite is slower than LiteDB in this benchmark project created by the LiteDB inventor
https://github.com/mbdavid/LiteDB-Benchmark
https://github.com/mbdavid/LiteDB/issues/291
Another (lesser) reason is the similarity to MongoDB methods, if that's what you are used to it will feel familiar, but no MongoDB server needed.
SQLite is slower than LiteDB in this benchmark project created by the LiteDB inventor
https://github.com/mbdavid/LiteDB-Benchmark
https://github.com/mbdavid/LiteDB/issues/291
Another (lesser) reason is the similarity to MongoDB methods, if that's what you are used to it will feel familiar, but no MongoDB server needed.
Before checking this out, people might want to take a look through the issues and pull requests of which there are 500+ and 50+ respectively [1]. I was really optimistic about this project and it was headed in a great direction, but it's not in a production ready state, and it seems that the main guy behind it has decided to move onto other things. It's been about a year since there was any significant activity.
I just mention this because a lot of these little issues might only become more apparent after integrating the db into your project and so it can be a bit annoying.
[1] - https://github.com/mbdavid/LiteDB
I just mention this because a lot of these little issues might only become more apparent after integrating the db into your project and so it can be a bit annoying.
[1] - https://github.com/mbdavid/LiteDB
My favorite has ( and will) always be MartenDB on PostgresSQL
Have a look: https://martendb.io/documents/
How to query: https://martendb.io/documents/indexing/duplicated-fields.htm...
Have a look: https://martendb.io/documents/
How to query: https://martendb.io/documents/indexing/duplicated-fields.htm...
I like Marten, but they broke stuff around timestamps pretty badly with the latest release. A good chunk of that is Npgsql's fault, but we lost the ability to duplicate / index on any timestamp fields, which threw performance of the system into the bin. I've been migrating the system over to EFCore slowly. Honestly, I don't think the system was a good fit for a document DB in the first place.
Isn't this why a qa environment is for?
I used this library in a thing I was working on years ago.
Ended up ripping it out, it caused no end of crashes and bugs. Hopefully its more stable now.
Ended up ripping it out, it caused no end of crashes and bugs. Hopefully its more stable now.
the left side reads "Embedded NoSQL database for .NET"
the right sides shows
the right sides shows
SELECT title, year, genres FROM movies...NoSQL stands for Not only SQL... It does not mean No SQL.
That definitely has shifted in meaning over the years then. I remember when the selling point of these DBs was the exclusion of SQL.
It's a really overloaded term these days. In some cases it's also used as a synonym for document DB's or K/V stores.
Best way to read it nowadays is probably "not like the popular DB's of the 90's".
Best way to read it nowadays is probably "not like the popular DB's of the 90's".
Good product, but the lack of editors on OSX and Linux really is a show stopper.
Are there few or none?
Nice project. A couple questions: is it EF Core compatible? Can I save the DB file/files to azure blob storage?
I use LiteDB for a game state store and am able to store the database as a gzipped blob. LiteDB allows using memory streams for usage and doesn’t require a file stream so you should be able to use any data store to store your database blobs.
Do you have more information about your game and how you are using LiteDB?
I can’t talk about much about the game as it’s in development and unannounced, but it’s a 3d adventure game.
We use LiteDB to hold all game state: objects on the ground, button/switch states, ai variables, inventory, etc. LiteDB allows using C# classes directly with the BSON mapper, allowing easy interoperability between the database and using the data in game. LiteDB allows using MemoryStreams which are convenient for loading and saving from a binary storage like a cloud service or a file. It’s fast at querying, but we’ve had issue with upsert performance so we perform all upserts at once in a separate thread to not impact the main game thread. Over all has worked well for the project and has had minimal issues over a year plus of development.
We use LiteDB to hold all game state: objects on the ground, button/switch states, ai variables, inventory, etc. LiteDB allows using C# classes directly with the BSON mapper, allowing easy interoperability between the database and using the data in game. LiteDB allows using MemoryStreams which are convenient for loading and saving from a binary storage like a cloud service or a file. It’s fast at querying, but we’ve had issue with upsert performance so we perform all upserts at once in a separate thread to not impact the main game thread. Over all has worked well for the project and has had minimal issues over a year plus of development.
I just found it today, so I might not be the best to answer this, but I doubt it is EF compatible at all as it is a document database (like MongoDB). As for azure blob storage, a quick google search shows a handful of libraries that accomplish that.
EF Core briefly had a (beta-level) MongoDB-compatible provider. It wasn't very useful in practice, but EF Core blogs every now and then talk about returning to document database support as a possibility.
I don't think I would use EF Core with a document database, but I can understand the appeal in sharing models with EF Core, and maybe even utilizing a shared "Change Tracker". Unless you are doing your own things, most EF Core models should be POCOs easily enough serializable to JSON to directly use in your document DB, and EF Core contexts at this point have an easy enough Change Tracker API to query that you could possibly reuse it easily enough for a document DB (though that is probably overkill because you don't need per-property tracking at that point usually, just per-document tracking).
I don't think I would use EF Core with a document database, but I can understand the appeal in sharing models with EF Core, and maybe even utilizing a shared "Change Tracker". Unless you are doing your own things, most EF Core models should be POCOs easily enough serializable to JSON to directly use in your document DB, and EF Core contexts at this point have an easy enough Change Tracker API to query that you could possibly reuse it easily enough for a document DB (though that is probably overkill because you don't need per-property tracking at that point usually, just per-document tracking).
Everybody can write a linq provider, it's just not that easy.
Ravendb has a great linq provider and is a nosql db.
Ravendb has a great linq provider and is a nosql db.
When using an App Service, you can have it mount a Blob Container or File Share. Dunno how well that works for a database though.
It was quite some time (embarrassingly so) before I realised that this has nothing to do with SQLite.
Used it for a while in production along with Serilog, but ended up ditching it. LiteDB Studio is not as useful as it could be, and there were issues when accessing a shared database that I couldn't solve.
Could you please elaborate on the shared access issues? Is this about multiple processes using the same database?
(I recently found LiteDB while researching log destinations.)
(I recently found LiteDB while researching log destinations.)
> Is this about multiple processes using the same database
For me, yes. I experience a pretty large performance hit on shared vs direct mode. That being said, shared mode does seem to work with multiple processes; However, all processes must be on the same machine, as he uses a mutex around critical code. Or at least, he did last time I looked at the codebase.
If your DB is on a network location and the processes are on different machines, bad things will happen.
My experience is that LiteDB in direct mode is faster than SQLite, but in shared mode is slower.
For me, yes. I experience a pretty large performance hit on shared vs direct mode. That being said, shared mode does seem to work with multiple processes; However, all processes must be on the same machine, as he uses a mutex around critical code. Or at least, he did last time I looked at the codebase.
If your DB is on a network location and the processes are on different machines, bad things will happen.
My experience is that LiteDB in direct mode is faster than SQLite, but in shared mode is slower.
Yes. Basically some processes failed to open the shared database.
What was your problem with Serilog? Currently I'm using the `Microsoft.Extensions.Logging` in a project, and I feel that Serilog was far handier.
I have only used Serilog for pet projects so far, but liked it really much, and have uses MS / Nlog and other "battle tested" solution is larger deployments so far. had some letdowns with those as well.
I have only used Serilog for pet projects so far, but liked it really much, and have uses MS / Nlog and other "battle tested" solution is larger deployments so far. had some letdowns with those as well.
Serilog wasn't the issue, I'm still using it. The problem was LiteDB not being able to open a database sometimes, and also reading those logs through LiteDB Studio was a PITA.
Why would you use this over something like Realm, which is a much more solid embedded database and already syncs directly with MongoDB?
No async.
Wow! You're not wrong: https://github.com/mbdavid/LiteDB/issues/1783
Surprising..
Surprising..
Building on top of Microsoft's FASTER[1] Log and FASTER KV would be a good option for a storage engine, as that is also C# and notably provides higher performance than the in-memory ConcurrentDictionary class in the standard library!
LiteDB uses locks for writers, which means it has virtually no chance of being anywhere near as performant as a lock-free log.
[1] https://github.com/microsoft/FASTER