Abstract Background Following from previous work suggesting that neurobehavioral features distinguish fragile X and idiopathic variants of autism, we investigated the relationships between four forms of repetitive behavior (stereotypy, self-injury, compulsivity, ritual behavior) and caudate nuclei volume in two groups: boys with fragile X syndrome, a subset of whom met criteria for autism, and a comparison group of boys with idiopathic autism. Methods Bilateral caudate nuclei volumes were measured in boys aged 3 to 6 years with fragile X syndrome (n = 41), the subset of boys with fragile X syndrome and autism (n = 16), and boys with idiopathic autism (n = 30). Repetitive behaviors were measured using the Repetitive Behavior Scales-Revised. Results For boys with idiopathic autism, left caudate volume was modestly associated with self-injury, while both compulsive and ritual behaviors showed significant positive correlations with bilateral caudate nuclei volumes, replicating previous results. For boys with fragile X syndrome, there was no such association between caudate volume and compulsive behaviors. However, we did identify significant positive correlations between self-injury total scores and number of self-injury topographies with bilateral caudate nuclei volumes. Conclusions These findings suggest a specific role for the caudate nucleus in the early pathogenesis of self-injurious behavior associated with both idiopathic autism and fragile X syndrome. Results further indicate that the caudate may be differentially associated with compulsive behavior, highlighting the utility of isolating discrete brain-behavior associations within and between subtypes of autism spectrum disorder.