Immune infiltration in solid tumors has emerged as an important aspect of cancer biology. In particular, the presence of tumor-infiltrating lymphocytes (TILs) within the tumor microenvironment is associated with improved outcomes and response to treatment in a wide variety of tumor types. These beneficial TIL infiltrates, comprising both B-cells and T-cells, may represent the successful targeting of tumor-associated antigens by the immune system leading to direct tumor cell killing. The success of immune checkpoint inhibitors has shown that this immune response may be manipulated to induce durable responses in several tumor types, and has strengthened the notion that an antigen-directed TIL response may occur in some cancers. However, the precise patient and tumor characteristics associated with these beneficial TIL responses are unclear. In this work, a genomic, mRNA-sequencing approach was taken to address TIL infiltration in diverse human tumor types, with a focus on breast cancer. Associations between signatures of TIL infiltration, B-cell TIL clonal diversity, genomic subtype, and other clinical correlates were identified, and show that significant B-cell TIL infiltrates typically represent less clonally diverse populations. Furthermore, in subtype-specific mouse models of breast cancer, T-cell and B-cell TIL clonal diversity are directly assessed through immune repertoire sequencing and a mouse model of claudin-low breast cancer is shown to elicit a recurrent, clonally-restricted T- and B-cell TIL response. This work clarifies the paradigm of conflicting immunosuppressive (pro-tumor) and anti-tumor immune responses within a large set of solid human tumors. B-cells within these tumors, especially, are shown to be an important part of the highly diverse tumor immune infiltrate, and are clonally restricted specifically in tumor types and subtypes associated with prognostic TILs. Immune repertoire sequencing of T- and B-cell TILs in mouse models of breast cancer establishes a mouse model of the claudin-low breast cancer subtype as a consistent model of a clonally-restricted, potentially antigen-directed TIL response within the tumor microenvironment. These findings complement the results of human clinical trials of immune checkpoint inhibitors and reiterate the importance of the immune system in solid tumor biology and patient outcomes.