Do you want to reduce the error-rate of responses from OpenAI’s o1 LLM by over 20% and also catch incorrect responses in real-time?
These 3 benchmarks demonstrate this can be achieved with the Trustworthy Language Model (TLM) framework.
TLM wraps any base LLM to automatically: score the trustworthiness of its responses and produce more accurate responses.
As of today: o1-preview is supported as a new base model within TLM. The linked benchmarks reveal that TLM outperforms o1-preview consistently across 3 datasets.
TLM helps you build more trustworthy AI applications than existing LLMs, even the latest Frontier models.
This article demonstrates an agentic system to ensure reliable answers in Retrieval-Augmented Generation, while also ensuring that latency and compute costs do not exceed the processing needed to accurately respond to complex queries. Our system relies on trustworthiness scores for LLM outputs, in order to dynamically adjust retrieval strategies until sufficient context has been retrieved to generate a trustworthy RAG answer.
Large language models are famous for their ability to make things up—in fact, it’s what they’re best at. But their inability to tell fact from fiction has left many businesses wondering if using them is worth the risk.
A new tool created by Cleanlab, an AI startup spun out of a quantum computing lab at MIT, is designed to give high-stakes users a clearer sense of how trustworthy these models really are. Called the Trustworthy Language Model, it gives any output generated by a large language model a score between 0 and 1, according to its reliability. This lets people choose which responses to trust and which to throw out. In other words: a BS-o-meter for chatbots.
We open sourced cleanlab as a Python library to quickly identify dataset problems in any Machine Learning project. While manual issue detection is often done during data prep prior to model training, your trained ML model captures a lot of information about its dataset that can reveal critical issues if the right algorithms are applied. The cleanlab package offers a data-centric AI platform to run many such algorithms and detect common problems in ML datasets like: mislabeling, outliers, (near) duplicates, drift, etc.
Would you trust medical AI that’s been trained on pathology/radiology images where tumors/injuries were overlooked by data annotators or otherwise mislabeled? Most image segmentation datasets today contain tons of errors because it is painstaking to annotate every pixel.
We have added semantic segmentation to automatically catch annotation errors in image segmentation datasets, before they harm your models! Quickly use cleanlab open source to detect bad data and fix it before training/evaluating your segmentation models. This is the easiest way to increase the reliability of your data & AI!
We've feely open-sourced our new method for improving segmentation data and published a paper on the research behind it. https://arxiv.org/abs/2307.05080
Multi-label classification utilizes data where each example can belong to multiple (or none) of the K classes. One example of this could be an image of a face that is labeled with wearing_glasses and wearing_necklace as opposed to standard multi-class classification where each example has only one label.
Ensuring high quality labels in multi-label classification datasets is really hard, as they often contain tons of tagging errors because annotating such data requires many decisions per example.
This article explores the challenges of multi-label data quality and demonstrate how to automatically identify and rectify problems with an enterprise no-code AI data correction tool.
When generating synthetic data with LLMs (GPT4, Claude, …) or diffusion models (DALLE 3, Stable Diffusion, Midjourney, …), how do you evaluate how good it is?
Introducing: Quality scores to systematically evaluate a synthetic dataset with just one line of code! Use Cleanlab’s synthetic dataset scores to rigorously guide your prompt engineering (much better signal than just manually inspecting samples). These scores also help you tune settings of any synthetic data generator (eg. GAN or probabilistic model hyperparameters) and compare different synthetic data providers.
Cleanlab scores comprehensively evaluate a synthetic dataset for different shortcomings including: unrealistic examples, low diversity, overfitting/memorization of real data, and underrepresentation of certain real scenarios. These scores are universally applicable to image, text, and structured/tabular data!
Would you deploy a self-driving car model that was trained on images for which data annotators accidentally forgot to highlight some pedestrians?
Annotators of real-world object detection datasets often make such errors and many other mistakes. To avoid training models on erroneous data and save QA teams significant time, you can now use automated algorithms invented by our scientists.
Our newest paper introduces Cleanlab Object Detection: a novel algorithm to assess label quality in any object detection dataset and catch errors (named ObjectLab for short). Extensive benchmarks show Cleanlab Object Detection identifies mislabeled images with better precision/recall than other approaches. When applied to the famous COCO dataset, Cleanlab Object Detection automatically discovers hundreds of mislabeled images, including errors where annotators mistakenly: overlooked an object that should’ve had a bounding box, sloppily drew a box in a poor location, or chose the wrong class label for an annotated object.
We’ve open-sourced one line of code to find errors in any object detection dataset via Cleanlab Object Detection, which can utilize any existing object detection model you’ve trained.
Years ago, we showed the world it was possible to automatically detect label errors in classification datasets via machine learning. Since that moment, folks have asked whether the same is possible for regression datasets?
Figuring out this question required extensive research since properly accounting for uncertainty (critical to decide when to trust machine learning predictions over the data itself) poses unique challenges in the regression setting.
Today we have published a new paper introducing an effective method for “Detecting Errors in Numerical Data via any Regression Model”. Our method can find likely incorrect values in any numerical column of a dataset by utilizing a regression model trained to predict this column based on the other data features.
We’ve added our new algorithm to our open-source cleanlab library for you to algorithmically audit your own datasets for errors. Use this code for applications like detecting: data entry errors, sensor noise, incorrect invoices/prices in your company’s / client’s records, mis-estimated counts (eg. of cells in biological experiments).
Extensive benchmarks reveal cleanlab’s algorithm detects erroneous values in real numeric datasets better than alternative methods like RANSAC and conformal inference.
I'm excited to share our newest release of cleanlab which helps you clean data and labels by automatically detecting issues in a ML dataset. To facilitate machine learning with messy, real-world data, this data-centric AI package uses your existing models to estimate dataset problems that can be fixed to train even better models.
With this release, it now supports:
- Regression (NEW)
- Object detection (NEW)
- Image segmentation (NEW)
- Outlier detection
- Binary, multi-class, and multi-label classification
- Token classification
- Classification with data labeled by multiple annotators
- Active learning with multiple annotators
Few-shot prompting is a pretty common technique used for LLMs. By providing a few examples of your data in the prompt, the model learns "on the fly" and produces better results -- but what happens if the examples you provide are error-prone?
I spent some time playing around with Open AI's davinci LLM and I discovered that real-world data is messy and full of issues, which led to poor quality few-shot prompts and unreliable LLM predictions.
This article explains my methodology and how I used data-centric AI to automatically clean the messy few-shot examples pool and increased model performance by 30%.
Successful litigation hinges on identifying the relevant data for legal actions through processes like e-discovery and relevance determination. However, the effectiveness of these processes is inevitably compromised by mistakes like incorrect labeling and misinterpretation when just relying on human annotators like busy paralegals and lawyers. Maintaining impartial and accurate categorization of evidence and data sources is often hard and can be extremely expensive, in some cases millions of dollars per day. This post introduces Cleanlab Studio, an AI solution for automatic document review that autonomously detects mis-categorized legal documents enhancing the accuracy of relevance determination.
“In my experience, the phrase ‘you are what you eat’ is exponentially more applicable to AI than to humans.”
This tweet by @WirelessPuppet reflects how folks are finally realizing that AI is becoming data-centric. But what does the future hold?
What data curation and modeling work will be done manually vs automated?
I believe significant automation is necessary to ensure the health of ever-increasing amounts of data and models. Our newest article outlines a vision of how automation-aided AI workflows should look (Hint: AI itself can facilitate many of steps needed to turn raw data into reliable model deployments)
The article outlines: how we plan to get there, why we are building open-source and AI platform software, and key differences between these offerings. Read it to learn why data itself should now be improved using AI.
Many folks using LLMs to generate data nowadays, but how do you know which synthetic data is good?
Introducing synthetic data quality assessment! Without writing ANY code, you can quickly identify which synthetic data is unrealistic (ie. low-quality) and which real data is underrepresented in the synthetic samples. This tool works seamlessly across synthetic text, image, and tabular datasets.
In this blogpost we demonstrate how to automatically detect issues in synthetic customer reviews data generated from the [Gretel.ai](http://Gretel.ai) LLM synthetic data generator.
Having accurate product listings is vital for E-commerce business for various reasons, such as obtaining accurate analytics, enhancing customer experience, improving product discoverability, and maximizing the effectiveness of SEO and advertising campaigns. In this blog we discuss how you can automatically rectify errors in your E-commerce product catalog, enabling your business to succeed. The best part is that you can achieve this without the need for coding.
Check out the full blog post to learn how you can revolutionize your E-commerce game.
Most LLMs allow you to specify the temperature parameter that governs the randomness and thus the creativity of the responses. For this experiment I used a very low temperature to ensure consistency for a given prompt.
It's pretty common now for data scientists and ML engineers to validate the quality of their training data being fed into these LLMs, but what about their test data used to evaluate them?
I spent some time playing around with the FLAN-T5 open-source LLM from Google Research and I discovered that noisy test/evaluation data can actually cause you to choose sub-optimal prompts.
Given two prompts A and B, I found multiple cases where prompt A performed better on the observed (noisy) test data, yet worse on the high-quality test data. In reality, this means that you would choose A as the "best prompt" when prompt B is actually the better one. I also proved the accuracy difference to be significant via McNemar’s Test.
This article explains my methodology and how I used data-centric AI to automatically clean the noisy test data in order to ensure optimal prompt selection.
Before modeling a dataset, do you remember to check if it seems IID?
Distribution drift and interactions between datapoints (autocorrelation) are common violations of the Independent and Identically Distributed (IID) assumption which make data-driven inference untrustworthy.
Presented is an automated check for such IID violations that you can quickly run on any {numeric, image, text, audio, etc.} dataset! This method helps you understand: does the order in which my data were collected matter? When the answer is yes, you must take special precautions in modeling to ensure proper generalization from data violating the IID property. Almost all of standard Machine Learning and Statistics relies on this fundamental property!
Don’t let such issues mess up your data analysis, use automated software to detect them before you dive into modeling!
Have you ever wanted to use handy scikit-learn functionalities with your neural networks, but couldn’t because TensorFlow models are not compatible with the scikit-learn API? Now you can with one-line wrappers for TensorFlow/Keras models that enable you to use TensorFlow models within scikit-learn workflows with features like Pipeline, GridSearch and more.
All you have to do is swap out: keras.Model → KerasWrapperModel, or keras.Sequential → KerasSequentialWrapper. The wrapper objects have all the same methods of their keras counterparts, plus you can use them with tons of awesome scikit-learn methods!
Labeled data is key to train models, but data annotators often make mistakes. One can collect multiple annotations per datapoint to get a more reliable consensus label, but this is expensive! To train the best ML model with the least data labeling, a key question is: which new data should I label or which of my current labels should be checked again?
ActiveLab automatically answers this question for you, allowing you to train the most accurate ML model via a smaller number of total annotations than required to reach similar accuracy with popular active learning methods. ActiveLab is highly practical — it runs quickly and works with: any type of ML model, batch settings where many examples are (re)labeled before model retraining, and settings where multiple annotators can label an example (or just one annotator).
These 3 benchmarks demonstrate this can be achieved with the Trustworthy Language Model (TLM) framework.
TLM wraps any base LLM to automatically: score the trustworthiness of its responses and produce more accurate responses. As of today: o1-preview is supported as a new base model within TLM. The linked benchmarks reveal that TLM outperforms o1-preview consistently across 3 datasets.
TLM helps you build more trustworthy AI applications than existing LLMs, even the latest Frontier models.