Several substances present in the environment, now classified as endocrine disruptors (EDs), strongly interfere with both androgen and oestrogen actions in reproductive tissues. However, nowadays it is well recognized that these sex steroid hormones are more than regulators of gonadal functions. In fact, they, in synergy with genes, are responsible of sex-related differences in anatomical, physiological, and behavioral traits which characterize males and females of many vertebrate species, including humans. Thus, even if EDs are present in minute amount (part for trillion) in environment, their effects in male and female physiology could be greater than before expected also prejudicing the sex-steroid hormone-induced integrated physiological responses in women and men. In addition, differences in male and female susceptibility to EDs could be present even if scarce information on this aspect is still available. Here we have reviewed the state of the art on the sex-related susceptibility to EDs underlying the mechanism at the root of these effects.