Aging typically results in reduced visual acuity, both from changes within the eye and from acquired neural deficits. It is not known, however, to what extent aging affects visual field map organization in human cortex. In addition, patients with Alzheimer's disease (AD) often present with visual deficits as one of their earliest complaints. It is possible that measurements of changes in visual cortex in these patients could aid early detection, accurate diagnosis and timely treatment of dementia. Here we investigate the differences and similarities of visual fields map organization and population receptive fields (pRFs) between patients with mild-to-moderate AD and healthy age-matched controls. We measured visual field map organization and pRFs across visual cortex using fMRI in healthy young volunteers ages 20–40, normally-aging subjects ages 55–85, and age-matched patients with mild-to-moderate AD. Retinotopic stimuli consisted of black and white, drifting checkerboards 11° in radius comprising wedges, rings, and/or bars. Normally-aging subjects do not show major visual field map organizational deficits, but do have increased pRF sizes in the central foveal representations of occipital and parietal visual field maps. AD patients do show visual field map organizational deficits and increases in pRFs increase in size and variability relative to age-matched controls.