AI-driven log reduction and storage optimization for security operations

Abstract

In this study, we present an AI-driven framework that integrates semantic log reduction with compliance-aware storage optimization, specifically designed for security operations center (SOC) and managed security service provider (MSSP) environments. Traditional approaches such as uniform compression, keyword filtering, and static tiering often either miss critical anomalies or preserve redundant noise, leading to excessive storage use, slower search performance, and analyst fatigue. The proposed framework addresses these challenges by combining three components: semantic reduction of repetitive entries, anomaly-focused retention supported by self-supervised models, and adaptive tiering aligned with regulatory requirements. Evaluations on HDFS, BGL, CICIDS2017, and Suricata datasets achieved 70%–80% log reduction, 55%–65% storage savings, recall rates above 95%, and a one-third reduction in query latency. These results demonstrate that pre-index reduction, together with anomaly- and compliance-aware retention, offers a scalable and regulator-ready solution for operational security environments.