3d27

2 years agodiscuss

Checkout this instead: https://github.com/confident-ai/deepeval

Also has native ragas implementation but supports all models.

3d27

2 years agodiscuss

This is great. I'm also building an LLM evaluation framework with all these benchmarks integrated in one place so anyone can go benchmark these new models on their local setup in under 10 lines of code. Hope someone finds this useful: https://github.com/confident-ai/deepeval

3d27

2 years agodiscuss

(found this interesting post on medium, this is not my original work)

3d27

2 years agodiscuss

There's a lot more in the evaluation space, including this one: https://github.com/confident-ai/deepeval

3d27

2 years agodiscuss

I'm just imagining Jensen Huang laughing in his sleep right now...

3d27

3 years agodiscuss

How did you calculate accuracy and bias?

3d27

3 years agodiscuss

The package I built is like a provider for 10+ different evaluation metrics that run both locally on your machine using models from hugging-face but also on the cloud IF you want more functionality.

If you want to evaluate a fine-tuned model, we have integrations with LM Harness and Stanford HELM coming out. If you want to evaluate a RAG application, we have 7+ metrics available for that.

You can also create your custom metrics using our interface!

Submissions

How to evaluate multi-turn LLM chatbots

We wrote a comprehensive guide on LLM security

How to generate synthetic data using SOTA data evolution methods

How to build your own LLM evaluation framework

Overview of All Major LLM Benchmarks

I wrote an article about everything I know about LLM metrics

Best practices I learnt from helping health tech enterprise test LLMs

Am I too needy? From a data science perspective

Best Practices for Unit Testing RAG Systems in Prod

Tried Apple's Vision Pros, would not recommend it

Everything I know about LLM evaluation metrics

Google 2024 Layoffs on a rolling-basis

Meta Going All in on GenAI

I used QAG to implement an LLM text summarization evals

I found a way to code like Shakespear

I implemented 12+ LLM evaluation metrics so you don't have to

AI Makes Commercial Masterpiece [video]

Show HN: I implemented evals metrics for LLMs that runs locally on your machine

Overcoming the biggest barrier to practical quantum computers

Google's new model is good but the demo's not reproducible in Bard

comments