Abstract Background Literature on difficult doctor-patient relationships has focused on the "difficult patient." Our objective was to determine physician and practice characteristics associated with greater physician-reported frustration with patients. Methods We conducted a secondary analysis of the Physicians Worklife Survey, which surveyed a random national sample of physicians. Participants were 1391 family medicine, general internal medicine, and medicine subspecialty physicians. The survey assessed physician and practice characteristics, including stress, depression and anxiety symptoms, practice setting, work hours, case-mix, and control over administrative and clinical practice. Physicians estimated the percentage of their patients who were "generally frustrating to deal with." We categorized physicians by quartile of reported frustrating patients and compared characteristics of physicians in the top quartile to those in the other three quartiles. We used logistic regression to model physician characteristics associated with greater frustration. Results In unadjusted analyses, physicians who reported high frustration with patients were younger (p < 0.001); worked more hours per week (p = 0.041); and had more symptoms of depression, stress, and anxiety (p < 0.004 for all). In the final model, factors independently associated with high frustration included age < 40 years, work hours > 55 per week, higher stress, practice in a medicine subspeciality, and greater number of patients with psychosocial problems or substance abuse. Conclusion Personal and practice characteristics of physicians who report high frustration with patients differ from those of other physicians. Understanding factors contributing to physician frustration with patients may allow us to improve the quality of patient-physician relationships.