If there was only one alert criteria, that'd be simple. Our alerts can be configured for any data filters (eg. only matching logs with column `level='error'`); we would have to create a unique MV for each alerts' filter condition.
We insert from our alerts worker because we want the aggregation to happen per alert (with the aggregated data filtered by the particular alert definition). As each alert is evaluated, we run the following [1] INSERT INTO ... SELECT ... statement based on the alert definition. We can't aggregate with an MV since we'd need to create an MV per unique alert that a customer may set up.
Forget the exact details, but we ran into issues with tuning the two setting you're describing on the Cloud instance (these settings were user level settings that seemed to get reset by ClickHouse Cloud because they were updated based on the cluster size). Perhaps this could change with Cloud and isn't an issue with the OSS version.
Once we hit >100k inserts per second, async inserts didn't work well for us because we had limited control over the background async insert batching happening on the cluster. The background inserts would be too small, resulting in many merges running, causing high CPU, causing high back-pressure (latency) on async inserts which would just result in an ingestion delay.
Plus, async inserts are only available on ClickHouse Cloud.
Exploring bloom filter index merges would be an interesting addition. I do wish it were easier to profile merge performance to break down where most of the CPU time is being spent.
Our browser client would help with tracing what network requests are being made by said apps since it would capture all network requests. However, Shopify may restrict what our browser client can do in your frontend store (I'm guessing that the 3rd party apps are added to a shop as iframes or are otherwise sandboxed).
Hey HN, I'm Vadim, the CTO at highlight.io. We offer session replay to engineering teams and have had a lot of questions around client-side performance.
Our errors ingest is quite affordable (https://www.highlight.io/pricing) and let's you filter out data you are not interested in ingesting. Our logs ingest is even cheaper because of the underlying ClickHouse storage we use.
Regarding model adaptation, we haven't yet explored a fine-tuned model, but it makes a lot of sense for a given class of errors. For a given code-base that is using highlight, the errors will typically be of a given language / infrastructure, so fine-tuning the model to those errors should be beneficial.
As for predicting potential errors, this will particularly make sense as an anomaly detection mechanism across metrics and logs. A class of 'important' errors based on the LLM's understand of the error, as well as historical comparison to normal error rates, is something we're exploring with an 'interesting errors' concept - stay tuned for more there!
> Have you looked into finetuning the embedding model to your use case?
Not yet, though this is definitely one of the next steps for us. The `gte-large` model we use is trained on a variety of text, but a hypothesis is one trained or fine-tuned on technical / code-related content may work better.
> Have you considered faster foundation models?
Any in particular that you would suggest? We're still pretty new to this so would love to learn about other recommendations. Would a foundation model perform as well at this task?
> How far could you go with this idea? Could this be the basis of a new monitoring platform?
Certainly; there are traditional ML approaches that could be applied to monitoring as well and we're heavily exploring this (ie. for metric anomaly detection). Another area we're exploring embeddings grouping is for filtering ingest to help folks only ingest / retain data that they actually want, but without the overhead of strict filter rules. Tons more to explore in this space, and you will certainly be hearing more from us here.
Agreed, but the argument for causing damage still requires a human actor to execute GPT-4's advice. Should we be censoring LLMs just because we are offended by the output or find it misleading? There is plenty of misinformation on the internet already; in my opinion, the power to deal damage is still in the hands of humans.
We ran into some challenges with async inserts at highlight.io [2]. Namely, ClickHouse Cloud has an async flush size configured (that can't be changed AFAIK) that isn't large enough for our scale. Once you async insert more than can be flushed, you get back pressure on your application waiting to write while Clickhouse flushes the queue. We found that implementing our own batched flushing via kafka [3] is far more performant, allowing us to insert 500k+ RPS on the smallest cloud instance type.
How do you think about the query syntax? Are you defining your own or are you following an existing specification? I particularly love the trace view you have, connecting a frontend HTTP request to server side function-level tracing.
The hardest errors to group / tag are ones that are further removed from the actual error cause (think frontend bug that causes a bad request that then causes a backend error, or an OOM because of a gradual memory leak). For these, we're looking at other ways to give more context to the LLM about an error (for example, associating frontend errors with backend ones and providing both as part of the request) but we also have limits on the LLM input size (512 tokens for the mdoel we're using, gte-large).
Both - tagging errors into broad buckets can help with filtering and searching for categories of errors, while finding similar errors surfaces ones that might be more familiar to an unknown one. We're learning as we go though and also want to surface solutions to errors long term.
As for hosting the model, Hugging Face has a number of guides (ie. [1]) for this but overall the process was quite simple. All you need to do is find a model compatible with the kind of inference you want to do (ie. sentence embeddings vs. text generation). Once it's deployed, Hugging Face gives you an API endpoint similar to OpenAI's embeddings API that returns the embedding vector for a given input. For picking the model, there are a bunch of good resources with benchmarks comparing them depending on the use case (ie. [2]).