PrivateGPT is a great starting point for using a local model and RAG.
Text-generation-ui, oogabooga, using superbooga V2 is very nice and more customizable.
I’ve used both for sensitive internal SOPs, and both work quite well. Private gpt excels at ingesting many separate documents, the other excels at customization. Both are totally offline, and can use mostly whatever models you want.
Not only is the demo funny, but this worked, surprisingly, as advertised. Had to restart the environment a few times for some reason. Not sure I understand the authors security concerns, but this is a fantastic early implementation.
I’ve used both for sensitive internal SOPs, and both work quite well. Private gpt excels at ingesting many separate documents, the other excels at customization. Both are totally offline, and can use mostly whatever models you want.