Efforts to understand the controls on landslides rely heavily on manually mapped landslide inventories, but these are costly and time-consuming to collect, and their reproducibility is not typically well constrained. To test the performance of manual mapping we compare two or more manually mapped inventories of landslides triggered by five recent earthquakes: Kashmir in 2005, Aysén in 2007, Wenchuan in 2008, Haiti in 2010, and Gorkha in 2015. We find surprisingly poor agreement between these maps (at worst 8 % overlap and at best 30 %). This has implications both for how future models and/or classifiers are tested and for the interpretations that are based on these inventories. We then test the ability of a new automated landslide detection index (ALDI) to recover landslide locations. In more than 50% of cases, ALDI more skilfully reproduces landslide locations from one inventory (treated as the ground truth) than a second inventory for the same site based on ROC curve analysis. Finally we examine the spatial pattern of landslides identified in the different maps and the patterns of their agreement and disagreement to show that: 1) much of the disagreement appears to be due to georeferencing rather than landslide identification; and 2) the ALDI map, which is quick and easy to produce, can be used to identify georeferencing errors and thus post-process and improve manual maps.