| Abstract: |
Conventional Intrusion Detection and Prevention Systems (IDPS) frequently fail to handle the dynamic nature of modern cyber threats, struggling to detect zero-day attacks or generalize across diverse network environments. The operational deployment of machine learning (ML) for intrusion detection also remains a significant challenge. This research addresses these limitations by designing, developing, and evaluating an adaptable modular IDPS framework that leverages machine learning to balance detection efficacy, model generalization, and operational efficiency. The proposed cloud-native architecture employs a two-stage process. Stage 1 utilizes a semi-supervised model to learn the behavioral baseline from unlabeled production traffic, distinguishing novel zero-day threats. Stage 2 uses a supervised model for fine-grained, multiclass classification of detected anomalies. To overcome the lack of labeled data, this research introduces a methodology for creating a composite evaluation dataset by unifying attack classes across public benchmarks (CIC-IDS-2017, CIC-IoT-2023) and adapting them to the real network of a modern organization -- a major European scientific research facility -- using statistical domain transfer. The system's operational sensitivity is managed by a novel, recall-focused thresholding strategy, while a weighted scoring system ranks models based on a holistic set of criteria including recall, precision, and inference time. Analysis of ML experiments shows larger models excel in-domain, but regularization is critical for cross-domain generalization, revealing trade-offs between accuracy and operational efficiency. Crucially, by deploying computationally efficient, offline-trained models via container orchestration, the framework achieves zero-downtime operational adaptability while mitigating the severe risks of adversarial attacks associated with online learning. The primary contribution is not a single optimal model, but a flexible, production-ready framework that can be continuously optimized to meet specific organizational security requirements. |