This is a fair point and also called out in the discussion section. To some degree, this could be mitigated by hotloading shader code (and compiling shaders in debug mode). However, this remains as a fundamental downside of the approach.
Personally, I think that this is a price worth paying!
Would you be willing to share details about the fine-tuning procedure, such as the initialization, learning rate schedule, batch size, etc.? I'd love to learn more.
Background: I've been playing around with generating image sequences from sliding windows of audio. The idea roughly works, but the model training gets stuck due to the difficulty of the task.
As far as HNSW implementations go, this one appears to be almost entirely unfinished. Node insertion logic is missing (https://github.com/swapneel/hnsw-rust/blob/b8ef946bd76112250...) and so is the base layer beam search.