Abstract Background Minimal validity evidence exists for scales assessing the built environment for physical activity. The purpose of this study was to assess the test-retest reliability and invariance of a three-factor model (Neighborhood Characteristics, Safety/Crime, and Access to Physical Activity Facilities) across gender, race, geographic location, and level of physical activity. Methods To assess measurement invariance, a random sample of 1,534 adults living in North Carolina or Mississippi completed a computer assisted telephone interview that included items examining perceptions of the neighborhood for physical activity. Construct level test-retest reliability data were collected from a purposeful sample of 106 participants who were administered the questionnaire twice, approximately two weeks apart. Fit indices, Cronbach's alpha, Mokken H and Spearman correlation coefficients (SCC) were used to evaluate configural and co/variance invarianc,e and intraclass correlation coefficients (ICC) were used to assess reliability. Results Construct test-retest reliability was strong (ICC 0.90 to 0.93). SCC for Neighborhood Characteristics and Crime/Safety were weak with Access (0.21 and 0.25), but strong between Crime/Safety and Neighborhood Characteristics (0.62). Acceptable fit and evidence of measurement invariance was found for gender, race (African American and White), geographic location, and level of physical activity. Fit indices consistently approached or were greater than 0.90 for goodness of fit index, normed fit index, and comparative fit index which is evidence of configural invariance. There was weak support of variance and covariance invariance for all groups that was indicative of factorial validity. Conclusions Support of the validity and reliability of the three-factor model across groups expands the possibilities for analysis to include latent variable modeling, and suggests these built environment constructs may be used in other settings and populations.