Behavior-Driven Encrypted Malware Detection with Robust Traffic Representation

Nowadays, network traffic encryption techniques are widely adopted to protect data confidentiality and prevent privacy leakage during data transmission. However, malware often leverages traffic encryption techniques to conceal their malicious activities or camouflage their traffic as benign traffic. To cope with this problem, most existing encrypted malware traffic detection methods employ machine learning or deep learning models to learn distinct features between malware and benign traffic. Nevertheless, existing methods still encounter one challenge, i.e., robustness to an imbalanced dataset. In this paper, we propose BDMF, a behavior-driven malware fingerprinting method based on deep learning, to achieve encrypted malware traffic detection. We first design a novel traffic representation named Traffic Behavior Matrix (TBM), which can abstract traffic behavior patterns initiated by malware compared with benign traffic. Subsequently, we design an effective classifier based on Convolutional Neural Networks (CNNs), which extract distinctive, robust features to achieve effective malware traffic detection. The robust behavior-driven traffic representation enables the CNN-based model to achieve robustness to an imbalanced dataset. We conduct extensive experiments with a real-world dataset to evaluate the detection performance of BDMF. The experimental results demonstrate BDMF outperforms all baseline methods in different evaluation metrics. Specifically, when the proportion of benign and malware traffic samples reaches 25:1, BDMF achieves an F1 score of 88.28%, which is 19.01% higher than the SOTA method. Moreover, BDMF maintains at least 0.89 precision and 0.85 recall with a relatively low time overhead.