RL-MAGE: Strengthening Malware Detectors Against Smart Adversaries

Abstract

Today, android dominates the smartphone operating systems market. As per Google, there are over 3 billion active android users. With such a large population depending on the platform for their daily activities, a strong need exists to protect android from adversaries. Historically, techniques like signature and behavior were used in malware detectors. However, machine learning and deep learning models have now started becoming a core part of next-generation android malware detectors. In this paper, we step into malware developers/adversary shoes and ask: Are machine learning based android detectors resilient to reinforcement learning based adversarial attacks? Therefore, we propose the RL-MAGE framework to investigate the adversarial robustness of android malware detectors. The RL-MAGE framework assumes the grey-box scenario and aims to improve the adversarial robustness of malware detectors. We designed three reinforcement learning based evasion attacks A2C-MEA, TRPO-MEA, and PPO-MEA, against malware detectors. We investigated the robustness of 30 malware detection models based on 2 features (android permission and intent) and 15 distinct classifiers from 4 different families (machine learning classifiers, bagging based classifiers, boosting based classifiers, and deep learning classifiers). The designed evasion attacks generate adversarial applications by adding perturbations into the malware so that they force misclassifications and can evade malware detectors. The attack agent ensures that the adversarial applications’ structural, syntactical, and behavioral integrity is preserved, and the attack’s cost is minimized by adding minimum perturbations. The proposed TRPO-MEA evasion attack achieved a mean evasion rate of $93.27\%$ while reducing the mean accuracy of 30 malware detectors from $85.81\%$ to $50.29\%$. We also propose the ARShield defense strategy to improve the malware detectors’ classification performance and robustness. The TRPO-MEA ARShield models achieved $4.10\%$ higher mean accuracy and reduced the mean evasion rate of re-attack from $93.27\%$ to $1.05\%$. Finally, we conclude that the RL-MAGE framework improved the detection performance and adversarial robustness of malware detectors.