Effective Anomaly Detection for Microservice Systems with Real-Time Feature Selection.

Microservice architecture is getting increasingly popular in recent years for building web-based systems. Finding runtime anomalies in such systems is crucial for improving their reliability. For this purpose, existing AIOps research has proposed various machine learning-based algorithms. However, a common limitation of existing algorithms is that they are sensitive to the settings of thresholds for anomaly identification when dealing with the high-dimensional multivariate time series data collected by monitoring the running instances of microservices. As a result, the performance of anomaly detection can be easily influenced by threshold changes. To tackle this problem, we propose a new anomaly detection framework called COAD (Combinatorial Optimization enhanced Anomaly Detection), which can work with various anomaly detection algorithms and enhance their detection process by performing real-time feature selection via metaheuristic algorithms. We have evaluated our method on three different testbeds based on a representative microservice system open-sourced by Google. The results show that real-time feature selection can significantly reduce the underlying algorithms' sensitivity to threshold settings (142% reduction on average). At the same time, the best anomaly detection performance (evaluated by f1-score) is improved by 5.67% on average. These results demonstrate the effectiveness and potential usefulness of the approach.