Elements of the tropical vertex group are formal families of symplectomorphisms of the $2$-dimensional algebraic torus. We prove that ordered product factorizations in the tropical vertex group are equivalent to calculations of certain genus zero relative Gromov-Witten invariants of toric surfaces. The relative invariants which arise have full tangency to a toric divisor at a single unspecified point. The method uses scattering diagrams, tropical curve counts, degeneration formulas, and exact multiple cover calculations in orbifold Gromov-Witten theory