A comprehensive system for evaluation of remote sequence similarity detection Public Deposited

Downloadable Content

Download PDF
Creator
  • Sadreyev, Ruslan I
    • Other Affiliation: Howard Hughes Medical Institute, University of Texas Southwestern Medical Center, 5323, Harry Hines Blvd, Dallas, TX 75390-9050, USA
  • Qi, Yuan
    • Affiliation: N.C. Cancer Hospital, UNC Lineberger Comprehensive Cancer Center, School of Medicine
    • Other Affiliation: Department of Biochemistry, University of Texas Southwestern Medical Center, 5323, Harry Hines Blvd, Dallas, TX 75390-9050, USA
  • Kim, Bong-Hyun
    • Other Affiliation: Department of Biochemistry, University of Texas Southwestern Medical Center, 5323, Harry Hines Blvd, Dallas, TX 75390-9050, USA
  • Grishin, Nick V
    • Other Affiliation: Howard Hughes Medical Institute, Department of Biochemistry, University of Texas Southwestern Medical Center, 5323, Harry Hines Blvd, Dallas, TX 75390-9050, USA
  • Wang, Yong
    • Other Affiliation: Department of Biochemistry, University of Texas Southwestern Medical Center, 5323, Harry Hines Blvd, Dallas, TX 75390-9050, USA
Abstract
  • Abstract Background Accurate and sensitive performance evaluation is crucial for both effective development of better structure prediction methods based on sequence similarity, and for the comparative analysis of existing methods. Up to date, there has been no satisfactory comprehensive evaluation method that (i) is based on a large and statistically unbiased set of proteins with clearly defined relationships; and (ii) covers all performance aspects of sequence-based structure predictors, such as sensitivity and specificity, alignment accuracy and coverage, and structure template quality. Results With the aim of designing such a method, we (i) select a statistically balanced set of divergent protein domains from SCOP, and define similarity relationships for the majority of these domains by complementing the best of information available in SCOP with a rigorous SVM-based algorithm; and (ii) develop protocols for the assessment of similarity detection and alignment quality from several complementary perspectives. The evaluation of similarity detection is based on ROC-like curves and includes several complementary approaches to the definition of true/false positives. Reference-dependent approaches use the 'gold standard' of pre-defined domain relationships and structure-based alignments. Reference-independent approaches assess the quality of structural match predicted by the sequence alignment, with respect to the whole domain length (global mode) or to the aligned region only (local mode). Similarly, the evaluation of alignment quality includes several reference-dependent and -independent measures, in global and local modes. As an illustration, we use our benchmark to compare the performance of several methods for the detection of remote sequence similarities, and show that different aspects of evaluation reveal different properties of the evaluated methods, highlighting their advantages, weaknesses, and potential for further development. Conclusion The presented benchmark provides a new tool for a statistically unbiased assessment of methods for remote sequence similarity detection, from various complementary perspectives. This tool should be useful both for users choosing the best method for a given purpose, and for developers designing new, more powerful methods. The benchmark set, reference alignments, and evaluation codes can be downloaded from ftp://iole.swmed.edu/pub/evaluation/.
Date of publication
Identifier
  • doi:10.1186/1471-2105-8-314
  • 17725841
Resource type
  • Article
Rights statement
  • In Copyright
Rights holder
  • Yuan Qi et al.; licensee BioMed Central Ltd.
License
Journal title
  • BMC Bioinformatics
Journal volume
  • 8
Journal issue
  • 1
Page start
  • 314
Language
  • English
Is the article or chapter peer-reviewed?
  • Yes
ISSN
  • 1471-2105
Bibliographic citation
  • BMC Bioinformatics. 2007 Aug 28;8(1):314
Access
  • Open Access
Publisher
  • BioMed Central Ltd
Parents:

This work has no parents.

Items