We report on the inelastic response of AV₂Al₂₀ (with A = Sc, La and Ce) probed by high-resolution inelastic neutron scattering experiments. Intense signals associated with the dynamics of Sc, La and Ce are identified in the low-energy range at 6–14 meV in ScV₂Al₂₀ and at 8–16 meV in LaV₂Al₂₀ and CeV₂Al₂₀. Their response to temperature changes between 2 and 300 K reveals a very weak softening of the modes upon heating in LaV₂Al₂₀ and CeV₂Al₂₀ and a distinguished blue shift by about 2 meV in ScV₂Al₂₀. By means of density functional theory (DFT) and lattice dynamics calculations (LDC) we show that the unusual anharmonicity of the Sc-dominated modes is due to the local potential of Sc featured by a strong quartic term. The vibrational dynamics of ScV₂Al₂₀ as well as of LaV₂Al₂₀ and CeV₂Al₂₀ is reproduced by a set of eigenmodes. To screen the validity of the DFT and LDC results they are confronted with data from X-ray diffraction measurements. The effect of the strong phonon renormalization in ScV₂Al₂₀ on thermodynamic observables is computed on grounds of the LDC derived inelastic response. To set the data in a general context of AV₂Al₂₀ compounds and their physical properties we report in addition computer and experimental results of the binary V₂Al₂₀ compound.