While working with LLMs for structured web data extraction, we saw issues with invalid JSON and broken links in the output. This led me to build a library focused on robust extraction and enrichment:
- Clean HTML conversion: transforms HTML into LLM-friendly markdown with an option to extract just the main content
- LLM structured output: Uses Gemini 2.5 flash or GPT-4o mini to balance accuracy and cost. Can also also use custom prompt
- JSON sanitization: If the LLM structured output fails or doesn't fully match your schema, a sanitization process attempts to recover and fix the data, especially useful for deeply nested objects and arrays
- URL validation: all extracted URLs are validated - handling relative URLs, removing invalid ones, and repairing markdown-escaped links
I'd love to hear if anyone else has experimented with LLMs for data extraction or if you have any questions about this approach!
- Clean HTML conversion: transforms HTML into LLM-friendly markdown with an option to extract just the main content - LLM structured output: Uses Gemini 2.5 flash or GPT-4o mini to balance accuracy and cost. Can also also use custom prompt - JSON sanitization: If the LLM structured output fails or doesn't fully match your schema, a sanitization process attempts to recover and fix the data, especially useful for deeply nested objects and arrays - URL validation: all extracted URLs are validated - handling relative URLs, removing invalid ones, and repairing markdown-escaped links
I'd love to hear if anyone else has experimented with LLMs for data extraction or if you have any questions about this approach!