Simulating and summarizing sources of gene tree incongruence

We introduce a gene tree simulator that is designed for use in conjunction with approximate Bayesian computation approaches. We show that it can be used to determine the relative importance of hybrid speciation and introgression compared with incomplete lineage sorting (ILS) in producing patterns of incongruence across gene trees. Important features of the new simulator are (1) a choice of models to capture the decreasing probability of successful hybrid species formation or introgression as a function of genetic distance between potential parent species; (2) the ability for hybrid speciation to result in asymmetrical contributions of genetic material from each parent species; (3) the ability to vary the rates of hybrid speciation, introgression, and divergence speciation in different epochs; and (4) incorporation of the coalescent, so that patterns of incongruence due to ILS can be compared with those due to hybrid evolution. Given a set of gene trees generated by the simulator, we calculate a set of statistics, each measuring in a different way the discordance between the gene trees. We show that these statistics can be used to differentiate whether the gene tree discordance was largely due to hybridization, or only due to lineage sorting.