HackerLangs
TopNewTrendsCommentsPastAskShowJobs

acutesoftware

no profile record

comments

acutesoftware
·19 giorni fa·discuss
> and maintain

This right here is the key difference! Yes anyone can vibe code a replacement for many apps - but will it still run 2 years later (assuming they get it 'running' in an prod environment at all
acutesoftware
·4 mesi fa·discuss
This highlights that all RAG systems should be using metadata embedded into each of the vectorstores. Any result from the LLM needs to have a link to a document / chunk - which is turn links to a 'source file' which (should) have the file system owners id or another method of linking to a person.

If the 'source information' cannot be linked to a person in the organisation, then it doesnt really belong in the RAG document store as authorative information.
acutesoftware
·6 mesi fa·discuss
Nice to see old tech revitalised - I had fun with the Australian version of a Z80 single board computer - https://en.wikipedia.org/wiki/TEC-1
acutesoftware
·6 mesi fa·discuss
I am using LangChain with a SQLite database - it works pretty well on a 16G GPU, but I started running it on a crappy NUC, which also worked with lesser results.

The real lightbulb moment is when you realise the ONLY thing a RAG passes to the LLM is a short string of search results with small chunks of text. This changes it from 'magic' to 'ahh, ok - I need better search results'. With small models you cannot pass a lot of search results ( TOP_K=5 is probably the limit ), otherwise the small models 'forget context'.

It is fun trying to get decent results - and it is a rabbithole, next step I am going into is pre-summarising files and folders.

I open sourced the code I was using - https://github.com/acutesoftware/lifepim-ai-core
acutesoftware
·6 mesi fa·discuss
I am working on a local RAG LLM designed for lower end PC's - ability for people to try out searching their own documents, seeing it was such a learning curve to get to this stage - hoping others can learn from my mistakes.

https://github.com/acutesoftware/lifepim-ai-core

Only been public a few days, so please let me know if there are glaring issues.
acutesoftware
·9 mesi fa·discuss
Switching models when running locally is fairly easy - as long as you have them downloaded you can switch them in and out with a just a config setting - cant quite remember, but you may need to rebuild the vectorstore when switching though.

LangChain has the embeddings for major providers:

  def build_vectorstore(docs):
    """
    Create vectorstore from documents using configured embedding model.
    """
    # Choose embedding model
    if cfg.EMBED_MODEL.lower() == "openai":
        embeddings = OpenAIEmbeddings(model="text-embedding-3-small")
    elif cfg.EMBED_MODEL.lower() == "huggingface":
        from langchain_community.embeddings import HuggingFaceEmbeddings
        embeddings = HuggingFaceEmbeddings(model_name="sentence-transformers/all-MiniLM-L6-v2")
    elif cfg.EMBED_MODEL.lower() == "nomic-embed-text":
        from langchain_ollama import OllamaEmbeddings
        embeddings = OllamaEmbeddings(model=cfg.EMBED_MODEL)
acutesoftware
·8 anni fa·discuss
RIP - he was a great inspiration to so many people