LogLead – Fast and Integrated Log Loader, Enhancer, and Anomaly Detector
CoRR(2023)
摘要
This paper introduces LogLead, a tool designed for efficient log analysis
benchmarking. LogLead combines three essential steps in log processing:
loading, enhancing, and anomaly detection. The tool leverages Polars, a
high-speed DataFrame library. We currently have Loaders for eight systems that
are publicly available (HDFS, Hadoop, BGL, Thunderbird, Spirit, Liberty,
TrainTicket, and GC Webshop). We have multiple enhancers with three parsers
(Drain, Spell, LenMa), Bert embedding creation and other log representation
techniques like bag-of-words. LogLead integrates to five supervised and four
unsupervised machine learning algorithms for anomaly detection from SKLearn. By
integrating diverse datasets, log representation methods and anomaly detectors,
LogLead facilitates comprehensive benchmarking in log analysis research. We
show that log loading from raw file to dataframe is over 10x faster with
LogLead compared to past solutions. We demonstrate roughly 2x improvement in
Drain parsing speed by off-loading log message normalization to LogLead. Our
brief benchmarking on HDFS indicates that log representations extending beyond
the bag-of-words approach offer limited additional benefits. Tool URL:
https://github.com/EvoTestOps/LogLead
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要