Clinacanthus nutans Lindau (C. nutans) is a well-known medicinal plant in South-East Asia that recently has attracted attention due to its therapeutic characteristics and cosmeceutical applications. However, delivering the beneficial attributes of the bioactive ingredients into a formulation is challenging due to poor solubility and low bioavailability of bioactive substances, which may hinder their effective transdermal delivery. Therefore, a nanoemulsion has been chosen to be a carrier in the encapsulation of bioactive ingredients of C. nutans extract for pharmaceutical and cosmeceutical formulations. In this work, a D-optimal mixture design was used to determine the optimal composition of a nanoemulsion-based system loaded with C. nutans leaves extract. The ultimate goal of the present work was to determine the optimum level of five independent variables (surfactant, oil, xanthan gum, bioactive extract, and water) in the optimal C. nutans nanoemulsion composition with minimum average droplet size. Under the optimal conditions, the predicted average droplet size was 99.99 nm. The actual response showed that the model was in good agreement with the predicted value with residual standard error (RSE) of 2.61%. The optimal nanoemulsion composition was observed to be stable under an accelerated stability study during storage at 25 and 45 °C for 90 days, centrifugal force and freeze–thaw cycles. Physicochemical characterization of the optimal nanoemulsion showed its suitability for transdermal application due to its stability against phase separation.