Photonic integrated circuits (PICs) for next-generation optical communication interconnects and all-optical signal processing require efficient (∼A/W) and fast (≥25 Gbs-1) light detection at low (<pJbit-1) power consumption, in devices compatible with Si processing, so that the monolithic integration of electro-optical materials and electronics can be achieved consistently at the wafer scale. Graphene-based photodetectors can meet these criteria, thanks to their broadband absorption, ultra-high mobility, ultra-fast electron interactions, and strong photothermoelectric effect. High responsivities (∼ 1 A/W), however, have only been demonstrated in biased configurations, which introduce dark current, noise, and power consumption, while unbiased schemes, with low noise and zero consumption, have remained in the ∼ 0.1 A/W regime. Here, we consider the unbiased asymmetric configuration and show that optimized plasmonic enhanced devices can reach for both transverse-electric and transverse-magnetic modes (at λ = 1550 nm), ∼A/W responsivity, and ∼ 100 GHz operation speed at zero power consumption. We validate the model and material parameters by simulating experimental devices and derive analytical expressions for the responsivity. Our comprehensive modeling paves the way for efficient, fast, and versatile optical detection in PICs with zero power consumption.