Real-time 3D reconstruction has always been a hot problem in mobile robotics. However, feature extraction algorithms used in traditional 3D reconstruction systems cannot work stably in challenging environments such as low-textured areas. The feature extraction method based on deep learning has higher accuracy and stability than traditional methods, but the complicated network structure leads to the lack of real-time performance. To overcome the limitations, a real-time 3D reconstruction system using multi-task feature extraction network and surfel is proposed. To enhance the stability and accuracy, we design a simplified convolutional neural network to extract feature. Moreover, surfel model is employed to implement the fusion and optimization of 3D point cloud. According to the experiments on the public dataset and real environments, the proposed system can run on Robot Operating System in real time, maintain high pose estimation accuracy in challenging scenes, and complete precise 3D reconstruction. The overall performance of the proposed system is better than that of the traditional 3D reconstruction system.