Show HN: Estimating what users ask ChatGPT about your company(github.com)
github.com
Show HN: Estimating what users ask ChatGPT about your company
https://github.com/syntropicsignal-ai/ai-visibility-audit
4 comments
The 'reverse engineering prompts' approach is interesting, but the variance in LLM responses based on temperature and system prompt updates makes consistency a major hurdle for this type of monitoring. One of the biggest technical challenges is distinguishing between when a model retrieves your site via RAG (live search) versus when it relies on stale training data. In the latter case, you can't really optimize for visibility without a new training cutoff, whereas RAG visibility can be influenced by site structure and indexing. Have you found a way to reliably trigger the search-tool use in your pipeline to ensure you're getting live results? Disclosure: I'm building Sivon HQ, where we track similar AI search visibility metrics.
Thanks for feedback! A couple of things:
Temperature settings only matter for api usage. Nevertheless, stochastic nature of LLM responses does produce a distribution of responses for a single prompt query. It could potentially be a good idea to run the same query couple of times in each iteration of the monitoring tool to get a better look at the distribution of responses.
As for live search, we focus pretty much only on queries that refer to brands or products. Such queries do use web search tool almost 100% of the time so you will not encounter the stale training data issue in our tool.
Happy to discuss it in more details if you want
Temperature settings only matter for api usage. Nevertheless, stochastic nature of LLM responses does produce a distribution of responses for a single prompt query. It could potentially be a good idea to run the same query couple of times in each iteration of the monitoring tool to get a better look at the distribution of responses.
As for live search, we focus pretty much only on queries that refer to brands or products. Such queries do use web search tool almost 100% of the time so you will not encounter the stale training data issue in our tool.
Happy to discuss it in more details if you want
Current limitation: we don't extract shopping carousels and product suggestions though that's WIP
Turns out the only way you find out currently if you're cited at scale is if you reverse engineer the prompts users might be asking about your brand and track ChatGPT/Gemini/etc. responses for them.
I've built a pipeline, that:
1. crawls your website and your competition
2. analyzes top search keywords from search engines
3. retrieves similar conversations from WildChat (~1M ChatGPT conversations)
4. generates most likely user prompts for your brand.
The result is a set of prompts that users are likely to ask about your company or your products and tracking them shows where LLMs suggest you and where do they reference your competition.
Repo:
https://github.com/syntropicsignal-ai/ai-visibility-audit
I'd love feedback on:
- whether this methodology makes sense
- alternative datasets to WildChat
- better ways to estimate prompt distributions