The existing literature has shown that increases in housing wealth, driven by unexpected house price shocks, have a positive effect on birth rates of homeowners. According the canonical model, a decrease in housing wealth has a symmetric negative impact on fertility behavior of households. That is, housing gains and losses of the same size should have identical quantitative effects (in an absolute sense) on fertility. In comparison, the theory of reference-dependent preferences suggests that people care more about housing losses than about equivalent gains, leading to an asymmetric housing wealth effect on a fertility decision. In our model, a utility from having a baby is weighted by a utility from house price where reference levels based on the house price at the time of purchase. The theoretical model suggests that the probability of giving birth is kinked at a reference housing wealth level and the wealth effects are discontinuously larger below the kink than above the kink. This theoretical prediction is tested using the recent survey data of Japanese households (Keio Household Panel Survey, KHPS). The KHPS is a nationally-representative, large-scale panel data started in 2004 with initial sample of approximately 4,000 households. Our empirical results suggest that, consistent with the theoretical prediction, homeowners' fertility responses are substantially larger when their housing wealth is below its reference level than when housing wealth is above reference level. Specifically, while estimated marginal effect is significantly positive when housing wealth is below its reference level, it is still positive but insignificant when housing wealth is above reference level. Furthermore, we also find that asymmetric wealth effects are robust to a number of alternative specifications, including controlling for possible measurement errors and unobserved heterogeneity of households.