Hello HN, this a pet project I've been working on:
It allows to visualize logs from raw files in a faceted manner in a sql database.
It creates templates of recurring logs, extracting the 'moving' parts into columns values it stores in a column oriented DB
For now it is possible to:
- load a raw log file
- provide a dissect pattern to extract meta fields such as the timestamp ('ts') and message ('msg')
- Conduct analysis on the extracted data in grafana:
- Aggregate (sum, avg, percentile) on metrics, group by facets (extracted strings)
- Compute elapsed times between any given pair of events with sql (example to come)
It allows to visualize logs from raw files in a faceted manner in a sql database. It creates templates of recurring logs, extracting the 'moving' parts into columns values it stores in a column oriented DB For now it is possible to:
- load a raw log file
- provide a dissect pattern to extract meta fields such as the timestamp ('ts') and message ('msg')
- Conduct analysis on the extracted data in grafana:
I came to work on this mostly after taking interest in templating algorithms and particularly drain: http://jiemingzhu.github.io/pub/pjhe_icws2017.pdf
Let me know if you have remarks or if you think something like this would be useful to you!