The crack initiation mechanism of cast aluminum alloys is addressed through the image-based finite element analysis. By using the synchrotron radiation CT, the initiation and extension of fatigue crack has been observed. The fatigue crack is initiated by the break of silicon particles near the pores, while its mechanism is not clear. To understand the stress and strain fields around silicon particles under cyclic loading, the finite element elasto-plastic finite-strain analyses were carried out. From the fatigue test and its in situ CT observation, the broken and unbroken silicon particles were identified. Their finite element models were semi-automatically generated from the initial CT image, where a few tens of silicon particles were included. By comparing the history of stress and strain between broken and unbroken silicon particles, we examined the trigger of breakage of silicon particle, that is, the cause of crack initiation.