The probability of a gene tree topology within a phylogenetic network with applications to hybridization detection

Type of content
Journal Article
Thesis discipline
Degree name
Publisher
University of Canterbury. Mathematics and Statistics
Journal Title
Journal ISSN
Volume Title
Language
Date
2012
Authors
Yu, Y.
Degnan, J.H.
Nakhleh, L.
Abstract

Gene tree topologies have proven a powerful data source for various tasks, including species tree inference and species delimitation. Consequently, methods for computing probabilities of gene trees within species trees have been developed and widely used in probabilistic inference frameworks. All these methods assume an underlying multispecies coalescent model. However, when reticulate evolutionary events such as hybridization occur, these methods are inadequate, as they do not account for such events. Methods that account for both hybridization and deep coalescence in computing the probability of a gene tree topology currently exist for very limited cases. However, no such methods exist for general cases, owing primarily to the fact that it is currently unknown how to compute the probability of a gene tree topology within the branches of a phylogenetic network. Here we present a novel method for computing the probability of gene tree topologies on phylogenetic networks and demonstrate its application to the inference of hybridization in the presence of incomplete lineage sorting. We reanalyze a Saccharomyces species data set for which multiple analyses had converged on a species tree candidate. Using our method, though, we show that an evolutionary hypothesis involving hybridization in this group has better support than one of strict divergence. A similar reanalysis on a group of three Drosophila species shows that the data is consistent with hybridization. Further, using extensive simulation studies, we demonstrate the power of gene tree topologies at obtaining accurate estimates of branch lengths and hybridization probabilities of a given phylogenetic network. Finally, we discuss identifiability issues with detecting hybridization, particularly in cases that involve extinction or incomplete sampling of taxa.

Description
Citation
Yu, Y., Degnan, J.H., Nakhleh, L. (2012) The probability of a gene tree topology within a phylogenetic network with applications to hybridization detection. PLoS Genetics, 8(4), pp. e1002660.
Keywords
Ngā upoko tukutuku/Māori subject headings
ANZSRC fields of research
Fields of Research::31 - Biological sciences::3104 - Evolutionary biology::310410 - Phylogeny and comparative analysis
Field of Research::06 - Biological Sciences::0604 - Genetics::060411 - Population, Ecological and Evolutionary Genetics
Fields of Research::49 - Mathematical sciences::4901 - Applied mathematics::490102 - Biological mathematics
Rights