This study focuses on how non-expert assessors judge relevance guided by mental models of relevance developed and applied during the assessment process. Components of relevance models are identified as well as challenges and changes associated with their construction and use. Study participants evaluated the relevance of news articles with respect to an assigned search topic. They commented on their reasoning in assessing each article, challenges they experienced in determining relevance and changes in their ability to assess relevance over the course of the evaluation session. Content analysis of these comments revealed that relevance models are derived from participants' understandings of the search topic, the documents they viewed and the relationships between them. Relevance manifestations (topical, situational, cognitive) and criteria (information scope, specificity and detail) guide the development and application of the relevance models, which may also be influenced by situational, cognitive and motivational factors.