HackerTrans
TopNewTrendsCommentsPastAskShowJobs

acutesoftware

no profile record

comments

acutesoftware
·há 20 dias·discuss
> and maintain

This right here is the key difference! Yes anyone can vibe code a replacement for many apps - but will it still run 2 years later (assuming they get it 'running' in an prod environment at all
acutesoftware
·há 4 meses·discuss
This highlights that all RAG systems should be using metadata embedded into each of the vectorstores. Any result from the LLM needs to have a link to a document / chunk - which is turn links to a 'source file' which (should) have the file system owners id or another method of linking to a person.

If the 'source information' cannot be linked to a person in the organisation, then it doesnt really belong in the RAG document store as authorative information.
acutesoftware
·há 6 meses·discuss
Nice to see old tech revitalised - I had fun with the Australian version of a Z80 single board computer - https://en.wikipedia.org/wiki/TEC-1
acutesoftware
·há 6 meses·discuss
I am using LangChain with a SQLite database - it works pretty well on a 16G GPU, but I started running it on a crappy NUC, which also worked with lesser results.

The real lightbulb moment is when you realise the ONLY thing a RAG passes to the LLM is a short string of search results with small chunks of text. This changes it from 'magic' to 'ahh, ok - I need better search results'. With small models you cannot pass a lot of search results ( TOP_K=5 is probably the limit ), otherwise the small models 'forget context'.

It is fun trying to get decent results - and it is a rabbithole, next step I am going into is pre-summarising files and folders.

I open sourced the code I was using - https://github.com/acutesoftware/lifepim-ai-core
acutesoftware
·há 6 meses·discuss
I am working on a local RAG LLM designed for lower end PC's - ability for people to try out searching their own documents, seeing it was such a learning curve to get to this stage - hoping others can learn from my mistakes.

https://github.com/acutesoftware/lifepim-ai-core

Only been public a few days, so please let me know if there are glaring issues.
acutesoftware
·há 9 meses·discuss
Switching models when running locally is fairly easy - as long as you have them downloaded you can switch them in and out with a just a config setting - cant quite remember, but you may need to rebuild the vectorstore when switching though.

LangChain has the embeddings for major providers:

  def build_vectorstore(docs):
    """
    Create vectorstore from documents using configured embedding model.
    """
    # Choose embedding model
    if cfg.EMBED_MODEL.lower() == "openai":
        embeddings = OpenAIEmbeddings(model="text-embedding-3-small")
    elif cfg.EMBED_MODEL.lower() == "huggingface":
        from langchain_community.embeddings import HuggingFaceEmbeddings
        embeddings = HuggingFaceEmbeddings(model_name="sentence-transformers/all-MiniLM-L6-v2")
    elif cfg.EMBED_MODEL.lower() == "nomic-embed-text":
        from langchain_ollama import OllamaEmbeddings
        embeddings = OllamaEmbeddings(model=cfg.EMBED_MODEL)
acutesoftware
·há 8 anos·discuss
RIP - he was a great inspiration to so many people