> Maybe you mentioned it in your demo and I missed it, but how does this differ pasting the log messages to ChatGPT / Claude / another LLM? Is it mainly that yours can iterate over a large logfile without blowing up the context window?
We do quite a bit of aggregation over the log file, and generate summary stats and choose what bits to stuff in the LLM. Plan to support more platforms than just spark.
> Does it suffer from the same issue as other LLMs, where it will always identify potential optimizations or improvements even if none are truly needed?
Funnily enough, instructing sonnet-3.7 to not suggest unnecessary optimisations seems to have done the trick!
thanks for the feedback! the first version had a lot more detailed code but decided to go with linking to our GitHub than copying all the code. Wanted to illustrate the core touch points involved in extending DF.
your guide literally helped me hack through a networking class i took in college a decade ago. dont work anywhere close to that stack but this brought back memories.