Kafka-machine learning based storage benchmark kit for estimation of large file storage performance

Abstract

Efficient storage and maintenance of big data is important with respect to assuring accessibility and cost-friendliness to improve risk management and achieve an effective comprehension of the user requirements. Managing the extensive data volumes and optimizing storage performance poses a significant challenge. To address this challenge, this research proposes the Kafka-machine learning (ML) based storage benchmark kit (SBK) designed to evaluate the performance of the file storage system. The proposed method employs Kafka-ML and a drill-down feature to optimize storage performance and enhance throughput. Kafka-ML-based SBK has the capability to optimize storage efficiency and system performance through space requirements and enhance data handling. The drill-down search feature precisely contributes through reducing disk space usage, enabling faster data retrieval and more efficient real-time processing within the Kafka-ML framework. The SBK aims to provide transparency and ease of utilization for benchmarking purposes. The proposed method attains maximum throughput and minimum latency of 20 MBs and 70 ms, respectively on the number of data bytes is 10, as opposed to the existing method SBK Kafka.