As a former data scientist and deep learning practitioner, I believe the main reason for the field decline isn't its commoditaztion but the fact that it just didn't deliver its promises for "AI".
One of the most famous word embedding methods, Word2Vec, actually approximates word co-occurrence PMI matrix through matrix decomposition. Check out Omer Levy and Yoav Goldberg's work.
In that case given enough dimensions it could approximate any matrix well - i.e. linearity shouldn't be a limiting factor.
Correlation (vs causation) and linearity are orthogonal concepts.
tl;dr: Pre-computed ElasticSearch term results are put on CDN and are fetched individually on demand - without requiring an online search engine or loading all text into client memory. There's a demo for searching Wikibooks' cookbook (4K documents) in the linked page