Our previous study (Zhang, Xuan, Fu, & Pylyshyn, in press) showed that object-location working memory (WM) was impaired by a secondary multiple object tracking (MOT) task but non-spatial visual WM could survive the MOT task. Thus there might be no competition between perceptual objects selected and objects maintained in visual WM. Considering that spatio-temporal WM and short-term object-location tasks might use different memory mechanisms (Zimmer, Speiser, & Seidler, 2003), competition of perception and attention system was examined by looking at the dual-task interference between a spatio-temporal WM task (Corsi Block Task, CBT) and a secondary MOT task in the present study. In Experiment 1, CBT performance was shown to be impaired by the secondary MOT task, while passively viewing the MOT scene but tracking none of the objects did not result in any damage to the CBT performance. In Experiment 2, we found that tracking per se could be harmful to the spatio-temporal WM. In Experiment 3, results showed that tracking more targets or tracking in higher speed caused more impairment to the spatio-temporal WM. In experiment 4, low spatio-temporal WM load (2 or 3 locations) was also showed to be impaired by the MOT task. In sum, spatio-temporal WM seemed to be vulnerable to a secondary MOT task. In a CBT task, to remember a temporal location sequence needs to shift spatial attention between these locations in rehearsal and impairment will be present when interrupted by tracking. Furthermore, in all experiments, MOT performance was shown to be not impaired by the spatio-temporal WM load, indicating that perceptual system has a priority in accessing limited resource over memory system.