wtatum · HackerTrans

2 lata temudiscuss

Not to go too off topic but I'm extremely interested in the throwaway comment about holding enough equity in your service provider of choice to justify calling investor relations when you have an issue. Is this a real thing that people do? If it works it sounds like an amazing life hack but I have my doubts how much influence they would have over the "real" support.

wtatum

2 lata temudiscuss

That's this https://apacorp.com/ Apache

wtatum

2 lata temudiscuss

Having been the target of a Crosslake report a few times it felt like the process was actually almost entirely focused on points like this. Key person risk, talent acquisition pipeline, process maturity, "definition of done", etc. It felt like "hard" tech debt items around code quality, measured SLA performance, etc were present for the sake of completeness but were of almost no consequence to the Crosslake team or those who would read the report.

Does that mirror your experience working in DD?

wtatum

3 lata temudiscuss

I see a lot of projects started in this space and all of them appear to have multi-writer as a goal. I've been interested for a long time (and have started and stopped) in a solution for single-write multi-read with eventual consistency. I chatted with @benbjohnson on a LiteStream ticket about the possibility of adding a mobile client to receive the replicas to mobile devices but I think that option isn't really consistent with the new direction of that work for LiteFS at Fly.

To me the multi-writer "collaborative" use case is super powerful but also has a lot of challenges. I personally would see a lot of value in a solution for eventually consistent read-replicas that are available for end-client (WASM or mobile native) replication but still funnel updates through traditional APIs with store-and-forward for the offline case.

Is anybody aware of an open-source project pursuing that goal that maybe I haven't come across?

wtatum

3 lata temudiscuss

Do you know of a straightforward way to identify that this is happening: where one node is using DERP or one link between your nodes is falling back to DERP?

wtatum

3 lata temudiscuss

Thanks that observation is extremely helpful. If I have it down then the intended flow looks something like?

- Web client is directed to some token vending service. This service implement authn in a manner of its choosing (i.e. OAuth) then sets a NATS client JWT in the cookie per https://docs.nats.io/running-a-nats-service/configuration/se... - Nats.ws client connection provides cookie during connection to perform client auth - If further authz/fine-grained control is needed the auth callout mechanism can be used. This would have access to the provided cookie/token so any claims needed for access control could be stapled on during step one and used at this point?

For GPs original question -- I'm running a fairly old Keycloak version (v8) but it does appear to set a JWT in KEYCLOAK_IDENTITY and KEYCLOAK_IDENTITY_LEGACY.

Am I right in understanding that IFF the token is signed with Ed25519 and both sub and iss are an NKEY value this is sufficient for NATS to accept that cookie as a credential?

wtatum

3 lata temudiscuss

Same. I tried out some toy use-cases using nats.js over websocket a few months ago. The prospect of possibly being able to "directly" consume messaging or key/value store from the browser with only a thin gateway between was really interesting to me but I couldn't square up the NATS-internal JWT cookie thing with how you would handle auth in a traditional web-app (OAuth client on a gateway plus a session cookie).

I found some threads saying auth callout in 2.10 would solve this and decided to table the project until 2.10 but it's really really unclear how to work through the details of converting a "traditional" OAuth access token into the NATS-specified access token required by the auth callout contract.

wtatum

3 lata temudiscuss

Not (necessarily) relevant to this solution, but I've recently come across a use case where it would be valuable to allow third-party (semi-trusted) individuals to sandbox/build/test some data processing/transformation pipelines in Jupyter and then "operationalize" that into ongoing ETL in my main webapp/API once they're finished. Not the same use case as Mercury but still in the bucket of hoisting a notebook into a more repeatable/operationalizable runtime. Does anyone have experience with something like that?

wtatum

3 lata temudiscuss

Also using Tauri in SQLite for a project. I only briefly looked at the provided SQLite plugin and quickly decided it didn't support everything I needed (custom scalar functions for example but there were others) but as far as I know all "official" Tauri plugins use the same event/command RPC mechanism available to you in userland for calling into Rust so I don't actually think the Tauri SQLite plugin "does the access in the UI" in the truest sense -- otherwise it wouldn't be a Tauri plugin it would just be a vanilla JS lib.

If you've looked more closely and know that not to be the case would like to hear what you've seen but my understanding is that anything that leaves the WebView sandbox is using RPC to make calls to Rust.

wtatum

3 lata temudiscuss

Thanks for the info that's really helpful. I was immediately curious whether SQLite-WASM would run in Deno and so I'm glad to hear you say that's something you've tried. It felt immediately like it "should" since Deno mirrors web APIs so well and supports a WASM runtime but I wasn't sure.

I have a current project that is starting out by building a client-side SQLite DB (currently in Tauri but WASM SQLite would be an option as well) that would benefit from an option to offload building really large DBs so a shared server process. Knowing same or very similar code could run in Deno opens up some really interesting possibilities.

wtatum

3 lata temudiscuss

Looks to be using the kvvfs which uses local storage key value pairs as the DB VFS. The file load-db.js (https://craft-of-emacs.kebab-ca.se/load-db.js) pre-loads the values into localStorage. I assume the values were determined by loading local storage through another method and then extracting it during development.

wtatum

3 lata temudiscuss

Could you comment on which "flavor" of SQLite for WASM this is using and how you built or pulled the glue code? I think sql.js and absurd-sql are the best known solutions for this but it's not clear from the blog if you're using those or rolled your own. Information on how it was built or what prebuilt you're using would be fantastic.

I also see that load-db.js is loading the known search index values into localStorage ... do you have some tooling for creating this file from a known SQLite base file or was it just handrolled from a localStorage already holding a valid DB?

EDIT: Looking in the code I found this const db = new sqlite3.oo1.DB({ filename: 'local', // The 't' flag enables tracing. flags: 'r', vfs: 'kvvfs' });

Googling lead me to the "official" SQLite WASM pages which otherwise don't appear too prominently in search results for whatever reason. This (https://sqlite.org/wasm/doc/trunk/about.md) seems like a good starting point and notes that both sql.js and absurd-sql are in fact inspirations

wtatum

4 lata temudiscuss

This is an interesting idea but I do wonder if it can be applied to generic collaborative editing "as is". The approach for synchronized ticks (I think most RTS games do something similar to the Blizzard approach) does require each participant to "ack" the tick. You can see this when one participant is lagging in that the simulation freezes for all players until they acknowledge. The online gaming approach is to let other players vote to kick someone who is lagging the game but that probably wouldn't be considered suitable in something like enterprise content management.

Can some variation of the approach be taken without the strict requirement that: - There is a known set of participants at any time - The list of participants has a reasonable well-bounded size - Peers can communicate with each other directly (WebRTC is available but there are sharp corners) - It's acceptable to freeze all players based on one lagging player - Need an unambiguous way to handle simultaneous player actions against shared state

Most likely the "fix" for the friction is that you just slow the ticks down a lot compared to what you would do in a multi-player game.

comments