Cross-Lingual News Dedup at $100/Month – Embeddings, Pgvector, and UnionFind(yingjiezhao.com)
yingjiezhao.com
Cross-Lingual News Dedup at $100/Month – Embeddings, Pgvector, and UnionFind
https://yingjiezhao.com/en/articles/Cross-Lingual-News-Dedup-at-100-Dollar-a-Month/
3 comments
its harder than it first appears. Even with good embeddings, semantic similarity across languages often breaks when articles include local context or idioms. Curious whether you found a threshold strategy that works reliably across languages, or if it still needs manual tuning.
Good question. The short answer: a single global threshold (cosine similarity ≥ 0.7) works surprisingly well for news, but it's not because embeddings handle idioms perfectly — it's because news articles are structurally constrained.
News articles about the same event tend to share named entities (people, places, organizations), numbers, and factual structure even across languages. "EU approves AI regulation" is a factual statement that embeds similarly regardless of language. This is very different from, say, opinion pieces or cultural commentary where idioms and local framing would diverge more.
That said, similarity alone isn't enough. The real reliability comes from non-semantic constraints layered on top:
- Time gap ≤ 18 hours between article and story — prevents "same topic, different month" false merges
- Story age ≤ 36 hours — old stories stop absorbing new articles
- Two-pass design — matching against refined story embeddings (average of recent articles) is more stable than raw article-to-article comparison
Where it does break: regional stories with heavy local context. A Japanese domestic politics article and an English wire service summary of the same event sometimes land just below threshold because the framing is so different. I accept some missed merges there rather than lowering the threshold and getting false positives.
No per-language thresholds so far — the embedding model (Qwen3) seems to normalize well across the languages I cover. But I wouldn't be surprised if that changes when adding languages with less training data representation.
News articles about the same event tend to share named entities (people, places, organizations), numbers, and factual structure even across languages. "EU approves AI regulation" is a factual statement that embeds similarly regardless of language. This is very different from, say, opinion pieces or cultural commentary where idioms and local framing would diverge more.
That said, similarity alone isn't enough. The real reliability comes from non-semantic constraints layered on top:
- Time gap ≤ 18 hours between article and story — prevents "same topic, different month" false merges
- Story age ≤ 36 hours — old stories stop absorbing new articles
- Two-pass design — matching against refined story embeddings (average of recent articles) is more stable than raw article-to-article comparison
Where it does break: regional stories with heavy local context. A Japanese domestic politics article and an English wire service summary of the same event sometimes land just below threshold because the framing is so different. I accept some missed merges there rather than lowering the threshold and getting false positives.
No per-language thresholds so far — the embedding model (Qwen3) seems to normalize well across the languages I cover. But I wouldn't be surprised if that changes when adding languages with less training data representation.
Happy to answer questions about the pgvector setup, Cloudflare Workers constraints, or the clustering algorithm tuning.