Carbon-based aerogels have been widely accepted to be a type of promising electromagnetic microwave (EMW) absorber due to their merits of low density, high specific surface area, and a three-dimensional porous structure, but the contradiction between their strong dielectric loss capacity and poor impedance matching often results in a weak EMW absorption capacity. In this work, a cellulose derived carbon/BN composite aerogel (CCA/BN) was prepared by a simple freeze-drying and high-temperature pyrolysis process. Here, both the BN loading and pyrolysis temperature could directly affect the dielectric loss capacity of the prepared CCA/BN. Specifically, the abundant interfaces between BN and the carbon matrix and the pyrolysis-generated carbon defects contributed to the polarization loss, and the BN loading and pyrolysis temperature dependent electrical conductivity affected the conduction loss. More importantly, good impedance matching was achieved by adjusting the BN loading and pyrolysis temperature, enabling as much EMW as possible to enter the absorber to be dissipated. What's more, the porous structure also contributed to enhanced EMW absorption efficiency through multiple reflection/scattering inside the material. As a result, CCA/BN with 10 wt% BN and a pyrolysis temperature of 680 °C (CCA/BN10-680) exhibited a superior EMW absorption performance with a minimum RL value (RL_min) of −49.2 dB at 11.28 GHz and 3.75 mm thickness and a maximum effective absorption bandwidth (EAB_max) of 8.16 GHz (9.68–17.84 GHz) at 3.45 mm thickness. This study provides a reliable reference for preparing a highly efficient biomass carbon based EMW absorber.