Composites of ferromagnetic metal and carbon with hierarchical structures are favoured for strengthening microwave absorption capability due to the synergistic effects of hybrid structures and enhanced interfacial polarization. However, it remains a great challenge to fabricate composites with desirable structures due to the collapse of the design structure following post-treatment. Herein, a facile strategy was developed to synthesize micrometre-sized hollow rod-like hierarchical structures assembled by CoFe/C nanosheets via using MgO micrometre rods as a sacrificial template. The morphology of the primary unit and the loading amount of carbon can be tailored by controlling the reaction conditions, both of which play important roles in improving the microwave absorption performance. The optimized CoFe/C hierarchical composite shows excellent microwave absorption performance. With a matching thickness of 2.15 mm, the strongest reflection loss (RL) reaches up to −40.5 dB at 9.1 GHz, while the effective absorption (RL <−10 dB) bandwidth is 5.8 GHz (7.6–13.4 GHz), which can completely cover the X-band. Remarkably, the widest effective bandwidth can reach 8.4 GHz (9.6–18.0 GHz) with a thickness of only 1.65 mm. Such an excellent microwave absorption performance shows potential for practical application of the as-prepared CoFe/C composites. The present strategy provides an effective way to design high-performance microwave absorbing materials with small thickness, broad bandwidth response and strong absorbing capacity.