Which conditions is the basis for a species to be reproductively isolated from other members?

Uniform Selection (Mutation-Order Speciation)

Reproductive isolation might also evolve during a process of mutation-order speciation, defined as the evolution of reproductive isolation by the fixation of different advantageous mutations in separate populations experiencing similar selection pressures, that is, uniform selection. In essence, different populations find different genetic solutions to the same selective problem. In turn, the different genetic solutions (i.e., mutations) are incompatible with one another, causing reproductive isolation. During ecological speciation, different alleles are favored between two populations. By contrast, during mutation-order speciation, the same alleles are favored in both populations, but divergence occurs anyway because, by chance, the populations do not acquire the same mutations or fix them in the same order. Divergence is therefore stochastic, but the process involves selection, and, thus, is distinct from random genetic drift. Selection can be ecologically based under mutation-order speciation, but ecology does not favor divergence as such, and an association between ecological divergence and reproductive isolation is not expected. How might mutation-order speciation arise? Sexual selection may cause mutation-order speciation if reproductive isolation evolves by the fixation of alternative advantageous mutations in different populations living in similar ecological environments.

Genetic Analysis of Reproductive Isolation: Principles

Reproductive isolation is unique in evolution because it is not a trait possessed by members of a single species, but a composite character that is the joint property of a pair of species. A single species can be reproductively isolated only with respect to another. Moreover, by its very nature, reproductive isolation is a trait that almost always involves epistatic interaction between alleles – but alleles occurring in different species. Hybrid inviability, for example, results from genes that produce normal viability in members of their own species but are lethal when interacting with alien genes in hybrids. Similarly, sexual isolation is caused when females evolved to prefer traits of conspecific males encounter different traits in other species. This composite and epistatic nature of reproductive isolation guarantees that speciation will not only show emergent genetic and phenotypic properties not seen in studies of a single species (e.g., Haldanes rule; see below), but also that mathematical theories of speciation will be different – and perhaps more complicated – than models of evolution in single lineages. While genetic analysis of reproductive isolation has occurred since the mid-1930s, mathematical theories of speciation are only now beginning to appear.

There are several reasons for studying the genetic basis of speciation. First, just as with a trait that evolves within a lineage, one wants to know whether a reproductive isolating mechanism has a ‘simple’ genetic basis (i.e., involves only a few genes of large effect) or is based on the accumulation of many genes. The number of genes involved may, in turn, allow inferences about the evolutionary process producing reproductive isolation. For example, if the difference in plumage color between males of two sexually dimorphic bird species is due to many genes of small effect, one may posit that these differences arose by sexual selection during which the male trait evolved step-by-step in concert with the female preference for that trait.

Similarly, the pattern of genetic differences causing reproductive isolation may give clues to the underlying evolutionary processes. It has been found, for example, that there are often many more genes causing hybrid male than female sterility between closely related species of Drosophila. This has led to the idea that hybrid sterility may result from sexual selection acting in isolated populations. Such selection, based on female choice, may cause more evolutionary change in males than in females, leading to the preferential sterility of male hybrids as an accidental outcome. Finally, genetic analysis can help localize small sections of chromosomes containing genes causing reproductive isolation, a necessary prelude to cloning and sequencing these genes. Such molecular work is essential for understanding the developmental basis of reproductive isolation, including the question of how a gene that works normally within a species causes deleterious effects in hybrids. At this writing we understand the developmental basis of only one case of reproductive isolation: the formation of lethal melanomas in hybrids between the swordfish and platyfish. This hybrid lethality is based on an oncogene in one species that is normally suppressed by another gene in the same species; the absence of suppression in hybrids causes the appearance of tumors.

Ideally, a study of the genetics of speciation should involve only reproductive isolating mechanisms that evolved up to the point at which gene exchange between populations was first reduced to zero, for it is at that point that speciation is complete. Because of divergent evolution, however, reproductive isolation continues to accumulate even after species cannot exchange genes, but such isolation is incidental to speciation. A proper study of speciation thus requires identifying the isolating mechanisms leading up to complete isolation (there may, of course, be more than one). This is not easy, as it requires that one must find either incipient species that have not yet evolved complete reproductive isolation, or species in which gene flow is prevented by only a single form of reproductive isolation. This has been possible in some cases, as with polyploidy in plants (see below), but in no group of animals or plants have there been systematic attempts to determine which forms of reproductive isolation are the first to evolve. Instead, there are only tentative conclusions based on general impressions. It has been suggested, for example, that sexual isolation is the most important factor causing speciation in birds, as closely related species do not hybridize in the wild but will produce fertile hybrids when forcibly crossed in the laboratory. Such suppositions are intriguing, but neglect possible ecological isolation, and must be buttressed by systematic analysis of populations at different stages of evolutionary divergence.

Prezygotic Isolation

Reproductive isolation represents a breakdown in the ability to reproduce successfully with sexual partners of another type of organism, and speciation requires a build up of reproductive isolation between diverging types of organism until gene flow is sufficiently rare or ineffective that the entities are considered ‘good species.’ Traditionally this was thought to require complete or near complete cessation of gene flow, though increasingly absolute reproductive isolation is thought to be too stringent a criterion (Mallet, 1995; Wu, 2001). Factors which influence prezygotic isolation are those that come into play before gametes of the different types meet and form zygotes. After this point postzygotic isolation occurs, and this simple classification of categories of reproductive isolation based on pre- and post-gametic fusion has been widely adopted since Dobzhansky originally categorized major factors influencing the origin of species into various ‘reproductive isolating mechanisms’ (Dobzhansky, 1937). However, it is important to appreciate that all factors influencing reproductive isolation act in combination. If cross-matings between males and females of different types are half as likely as within types we say their isolation index (I) is 0.5. If the viability of their offspring is also around 50% these are equally effective barriers to gene flow, and acting together will produce a combined I of 0.75, though prezygotic isolation will have made a greater contribution to the overall isolation only because it occurs first (Coyne and Orr, 2004).

Temperament and temporal reproductive isolation

Reproductive isolation can occur when individuals or populations exhibit differences in the timing of various activities, including foraging or mating (Pascarella, 2007; Nosil, 2012). Well known examples include differences in hatching schedule in Enchenopa treehoppers found on different host plants (Wood and Guttman, 1982) and differences in spawning seasonality in salmon (Fillatre et al., 2003). Individuals with different personalities are predicted to become reproductively isolated when personality influences temporal activity, and when the timing of these activities in turn results in a reduction in mating encounters. For example, individuals that have bold behavioral types might be more likely to feed, or to mate, during times of high-predation risk than individuals with shy temperament types that are less likely to be active during risky periods. Unfortunately, little work has explored the relationship between personality and temporal activity levels, and no work to my knowledge has explored the connection between personality-dependent differences in the timing of activity and reproductive isolation. This offers a promising area for future research.

5.2 Reproductive isolation in sympatry

For a long time, the idea of the sympatric speciation as a mechanism of evolution of new species was dismissed because of the apparent incompatibility with the biological concept of species and the idea of the absence of the gene flow as an indispensable condition of speciation.

Reproductive isolation is a condition of speciation and the primary criterion of the biological species concept. In the well-known Ernst Mayr’s definition, “species are groups of actually or potentially interbreeding natural populations, which are reproductively isolated from other such groups” (Mayr, 1942). Now, eight decades after the formulation, the definition seems too restrictive by implying the absence of the gene flow and the reproductive isolation as conditions of speciation: as shown earlier, gene flow is demonstrated to occur between sister species and sensu stricto sister species are not reproductively isolated but can hybridize and produce viable offspring.

Now we know, as will be shown, that many species, while being potentially interbreeding, are nevertheless reproductively isolated by neurocognitive mechanisms. They have evolved interspecific neurocognitive mechanisms to establish and maintain reproductive isolation of populations or species in the same geographic area while remaining potentially capable of hybridizing and producing viable offspring.

Reproductive isolation is the first stage in the process of sympatric speciation. It takes place in random-mating populations of an area as a result of changes in mating preferences in groups of individuals whose mating preference shifts toward conspecifics of the opposite sex displaying specific phenotypic characters. When changes are heritable, they lead to the formation of a separate mating group within the original population. Formation of separate mating groups spatially may result from

emergence in a group of individuals of sensory-determined preferences for individuals of the opposite sex displaying specific phenotypic traits, or

migration of a particular group to a new niche because of a sensory preference.

In the first case, reproductive isolation of populations in sympatry requires a shift in mating preferences in a population, whereas in the second case, it is an automatic result of the population that carves a new niche.

Now reproductive isolation in sympatry is a scientifically demonstrated fact, accepted by most biologists. It is a behavioral trait based on neurobiological mechanisms; it seems the most likely way of separating and isolating reproductively populations in sympatry.

What are Speciation Genes?

Studies of reproductive isolation have attempted to isolate the genes responsible for reproductive isolation. However, the very definition of what a speciation gene is makes it difficult to agree how many speciation genes have been found, and whether they reveal patterns of speciation (Rieseberg and Blackman, 2010; Presgraves, 2010a,b). Under the view that speciation is the evolution of reproductive isolation, it is clear that genes whose alleles contribute to some form of reproductive isolation can be considered a speciation gene (Nosil and Schluter, 2011). On one hand, there are genes contributing to DBM genetic incompatibilities, and therefore to the evolution of intrinsic reproductive isolation (hybrid sterility and inviability). On the other hand there are genes responsible for local adaptation and the evolution of extrinsic reproductive isolation (immigrant inviability and extrinsic postzygotic isolation), and those responsible for gametic recognition, for male–female interactions, and mate choice. In this sense, the list of speciation genes that have been discovered is perhaps larger than anticipated; yet they still have not clearly revealed whether there are special categories of speciation genes.

Questions about the contribution of a particular gene to speciation are similar to those applied to measurements of reproductive isolation. It would be important to know when the speciation gene arose, and whether or not speciation genes create full or weak reproductive isolation (Nosil and Schluter, 2011; Coyne and Orr, 2004). In relative terms, a speciation gene causing strong reproductive isolation might see its effects reduced to a very small magnitude if it arose very late in the speciation process. In a similar fashion, a gene whose expression is late during the life cycle of the organism (e.g., hybrid sterility), might also contribute relatively little to total reproductive isolation if genes expressed earlier during development (e.g., genes responsible for seed germination in plants) already produced high levels of isolation between populations.

What are species? The biological species concept

Disagreement as to what precisely constitutes a species is to be expected, given that the concept serves so many functions (Vane-Wright, 1992). We may be interested in classification as such, or in the evolutionary implications of species; in the theory of species, or in simply how to recognize them; or in their reproductive, physiological, or husbandry status.

Most non-specialists probably have some vague idea that species are defined by not interbreeding with each other; usually, that hybrids between different species are sterile, or that they are incapable of hybridizing at all. Such an impression ultimately derives from the definition by Mayr (1940), whereby species are “groups of actually or potentially interbreeding natural populations which are reproductively isolated from other such groups” (the Biological Species Concept). Mayr never actually said that species can't breed with each other, indeed he denied that that this was in any way a necessary part of reproductive isolation; he merely said that, under natural conditions, they don't.

Reproductive isolation, in some form, stands at the basis of what a species is. Having said this, it must be admitted that it is no longer possible to follow Mayr's concept as definitive. In a recent book (Groves, 2001, see especially Chapter 3) I sketched the main reasons why this is so:

It offers no guidance for the allocation of allopatric populations.

Many distinct species actually do breed with each other under natural conditions, but manage to remain distinct.

The interrelationships of organisms under natural conditions are often (usually?) unknown.

Many species do not reproduce sexually anyway.

4.8.2 Breaking the interspecific isolation barriers

Reproductive isolation between species works as a mechanism for maintenance of separation and individuality of different gene pools that are incompatible and/or adapted to different environments. Among the related species such isolation is usually not complete, which enables some level of gene exchange and taking benefit of those genes, which are or may easily become compatible to the other species’ genetic background.

Ploidy level and differences in genomic composition are principal factors influencing effectiveness of crossing and fertility of the F1 wide hybrids. Practically, no difficulties are to be expected at hybridization of hexaploid cultivated oats A. sativa and A. byzantina with their close relatives A. sterilis, A. fatua, A. occidentalis, and A. ludoviciana, which show the same ploidy level and common genomic formula (AACCDD). Occasional occurrence of univalents in meiosis of hybrids within this group (McMullen et al., 1982) does not restrict the interspecific gene flow. Hybrids of A. sativa with A. sterilis and A. fatua occur even spontaneously (Andersson and Carmen de Vincente, 2010) and the alien variation is easily assimilated into cultivated forms. Some high-yielding cultivars released in the second half of the last century contained even 50% of germplasm from A. sterilis, for example, “Ozark” (Bacon, 1991) or from A. fatua, for example, “Mesa” (Thompson, 1967). According to Leggett and Thomas (1995) these species are classified as the primary gene pool for A. sativa. For the tetraploid cultivated oat A. abyssinica the similar most closely related group is formed by A. barbata, A. vaviloviana (spontaneous hybrids reported by Baum (1977)), and A. agadiriana (all with common genome formula AABB). A corresponding primary gene pool for diploid cultivated oat A. strigosa would involve A. atlantica, A. hirtula, and A. wiestii (carriers of the As genome).

For A. sativa, species of oat with 4x ploidy level but the same homologous genomes have been classified by Leggett and Thomas as the secondary gene pool. Crossability of common oat with these species is relatively high but sterility occurs in pentaploid F1 hybrids (Table 4.7) due to incomplete chromosome pairing in meiosis. In spite of this, fertility is usually sufficient to perform successful backcrossing to a parental species (usually A. sativa). The secondary gene pool includes tetraploids A. maroccana and A. murphyi sharing the A and C genomes with A. sativa. The recently discovered tetraploid A. insularis is another candidate to the group as its genomic formula is close to CCDD and hybrids with A. sativa are also partially fertile (Ladizinsky, 1999).

Very low levels of crossability and F1 fertility are reported for crosses of common oat with diploid species carrying variants of the C genome (A. pilosa, A. ventricosa, Avena bruhnsiana) or the A genome (A. longiglumis, A. canariensis, and A. hirtula) (Table 4.7). All diploids, together with the AABB tetraploids (A. abyssinica, A. barbata, A. agadiriana, A. vaviloviana) and tetraploid A. macrostachya, have been classified by Leggett and Thomas to the tertiary gene pool. Larger differences in ploidy level, lack of homology between different genomes’ chromosomes, and lack of coadaptation between genes from A. sativa and species of this group cause strong reproductive isolation, which cannot be overcome without special procedures such as embryo rescue and doubling of chromosome number (usually with colchicine). More detailed data on crossability and sterility between all Avena species are available in Table 4.7 and in the review of Loskutov (2001).

For the highest chance of seed set in the interploidy crosses of oats, Rajhathy and Thomas (1974) recommend use of the lower ploidy species as a pistillate parent. However, the opposite cross direction may work better in some crosses (e.g., A. sativa × A. macrostachya (Łapiński et al., 2013)). Reciprocal crossing is worthy of trial because of the possible cytoplasmic effects, which are to be expected even in less distant crosses of oat, as enhanced yield of the lines with A. sterilis cytoplasm reported by Beavis and Frey (1987) or different level of disease resistance (Simons et al., 1985).

Application of growth regulators after pollination, embryo rescue, colchicine treatment, and intense vegetative propagation of highly sterile hybrids are powerful procedures that greatly increase chances for alien gene transfer. However, for the most recalcitrant interploidy crosses the best solution may be chromosome doubling in the lower ploidy parent or production of an artificial bridging alloploid (Sadanaga and Simons, 1960). Increasing of ploidy level (and genetic diversity) through a combination of a preliminary wide cross and chromosome doubling proved to be much useful in oat. At higher ploidy levels, the presence of common genomes in F1 hybrids exerts a buffering effect on disturbances caused by the lack of meiotic pairing or incompatible gene action. Therefore, it is recommended, based on a variety of diploid and polyploid species with recognized genomic composition, to produce first a proper artificial alloploid (optimally hexaploid or octoploid) with not more than one critical genome different than in the acceptor species. Next, the alloploid should be used as a cross partner with the target species. This way proved to be successful in the transfer of numerous genes from the tertiary gene pool, for example, for powdery mildew resistance from A. pilosa (Sebesta et al., 1986). On the other hand, the use of a bridge species third component may complicate restitution of fully fertile and agronomically acceptable lines (Rines et al., 2007).

Difficulties at alien transfer are often not restricted to crossability and F1 sterility. More serious problems may appear later, as a restricted recombination between native and alien chromosomes, which disturbs separation of target genes from unadapted or undesired neighbor genes from the same chromosome. In this context, the line Cw57 of A. longiglumis is a particularly valuable cross partner for preliminary crosses. It causes increase of homeologous pairing and intrachromosomal recombination frequency (Thomas et al., 1980a), acting similarly to the deficiency of Ph2 locus used in wheat chromosome engineering.

Transfer of mildew resistance genes from A. barbata (tertiary gene pool) to A. sativa is an example of the classical procedure. It started from colchicine-facilitated production of sativa + barbata alloploid (10x), followed by backcrossing to A. sativa and selfing, aimed at the production of a disomic addition line. Next, irradiation of this line caused fragmentation of the alien chromosome, which led to the desired translocation (Aung et al., 1977). In another methodical version the same addition line was crossed to an alloploid (8x) carrying the A. longiglumis “CW57” genome (added to A. sativa cv. “Pendek”) in order to promote intergenomic crossing over (Thomas et al., 1980a). Another irradiation stimulating the transfer of stem rust resistance from A. barbata was described by Brown (1985). The “CW57” system was used in the Pc94 crown rust resistance gene transfer from A. strigosa (Aung et al., 1996).

Intense vegetative propagation (up to thousands of plants) of colchicine-treated highly sterile F1 hybrids, followed by extensive spontaneous pollination with mixture of A. sativa lines on a specially prepared field, was a successful strategy to overcome the strong sterility barrier in the Polish A. sativa × A. macrostachya crosses (on average, one germinable seed was formed per 210 panicles of F1 generation). In one of the F1 hybrids, the procedure facilitated occurrence of very rare reduced functional gametes from a semirandom assortment in the disturbed meiotic divisions. Among 57 germinable seeds obtained nearly half gave rise to semisterile plants with chromosome numbers between 40 and 49 (the rest were of alloploid type with ∼70 or 56 chromosomes). The variation among lines derived from this material was sufficient to select hexaploid forms with large grain and winter hardiness superior to the lines used as standards (Łapiński et al., 2013).

The ability to participate in interspecific hybridization is not equally distributed in the populations and frequently is restricted to a small proportion of cross-compatible genotypes. Therefore, much may depend on the number and diversity of parental forms selected for a difficult wide cross. Independently on different ploidy and different genomic composition the individual allelic differences remain an essential factor for crossability, F1 sterility, and genetic compatibility with the other parent. There is no evidence on the possible links between an individual crossability or sterility and usefulness of the resulting hybrids. Anyway, in case of a highly difficult cross, the better breeding strategy is hybridization between possibly high number of genetically differentiated parents instead of crossing a pair of genotypes.

Experimentation with gynogenetic production of doubled haploid lines of oat contributed to the extension of the tertiary gene pool beyond the Avena genus. Matzk (1996) succeeded in crossing oats with maize (Zea mays L.) and pearl millet (Pennisetum americanum L.). More recently, Kynast et al. (2004) reported on the production of a whole set of oat lines carrying disomic additions of each of 10 maize chromosomes. The advantages of this material have not been recognized yet. Surely, more intergeneric cross combinations will appear in the future.

Progress in the recombinant DNA technology is expected to omit any reproductive isolation barriers and its role will be growing. However, especially for the quantitative traits, the theoretically unlimited area of choice among genetic resources is remarkably restricted by cognitive and technological possibilities of science. Incomprehensive variation in wild gene pools and their evolving character create dynamic systems with thousands of variables and astronomic number of solutions. Only a minute part of these solutions have been realized. Therefore, no decline is expected for the usefulness of classical search for new variation through wide crossing.

Speciation and Behavior

Finally, the formation of species (called speciation) provides a powerful tool for asking questions about the history of traits, including behavioral traits. So far, this review of evolution has addressed changes in gene frequency and the manner in which those changes happen. How can such genetic shifts within a population of animals result in the biological diversity we see today? Something else has to happen: the flow of genes within that population must be interrupted. This can occur when a population is subdivided and parts of that population are isolated from each other. Eventually, if environments of the subdivided groups differ, natural selection will favor different traits in the two new populations. As time passes, differences accumulate, and the two populations will no longer be able to interbreed were they to have that opportunity. For instance, foxes, coyotes, wolves, and domestic dogs all evolved from the same ancestor, with foxes diverging earliest in evolutionary time. Differences have accumulated so that now foxes cannot successfully interbreed with any of the other species. However, coyotes, wolves, and domestic dogs split more recently and remain so similar that hybrids are common. By the way, complete isolation is not necessary for species formation; a small amount of gene flow does not counteract the accumulation of differences.

Such reproductive isolation can result from a variety of causes:

Geographic barriers. If a population is subdivided by the emergence of a mountain range, river, or other inhospitable habitat, animals on one side of the barrier will be unable to breed with animals on the other side. The same effect occurs if part of the population moves away.

Resource shifts. For animals that live and reproduce on a resource, the ability to colonize new resources decreases the likelihood that they will encounter or mate with individuals in the parent population.

Mate choice. If females diverge in their preferences for male characteristics, for instance, and if that divergence has a genetic basis, then eventually there will be two distinct gene pools, each sporting one or the other preferred male trait.

Genetic change. Mutations that prevent proper meiosis can produce individuals that cannot mate with other members of the population. This is thought to be the origin of about 4% of plant species.

V.B. Patterns of Genetic Divergence for Other Species Traits

Postzygotic reproductive isolation is only one trait that has generally diverged between species. In order to formulate a broader picture of the genetic patterns of divergence, it is necessary to compare the genetic architecture of this trait with what is seen for other traits that diverge between species (reviewed in Hollocher, 1998). The traits that have been looked at in some genetic detail include interspecific mate discrimination and interspecific differences in secondary sexual characteristics. Overall, the general pattern that emerges is that these traits, too, are governed by many genes of small effect. However, in contrast to what was found for the genetic basis of postzygotic reproductive isolation, epistasis does not play a dominant role in governing the evolution of these traits.

Comparison of patterns of genetic variation within species versus patterns of genetic variation between species for the same traits can be useful for gaining insights into the evolutionary mechanisms that may have played a role during species divergence (reviewed in Hollocher, 1998). If these within-species versus between-species comparisons reveal strong similarities, then generally it can be concluded that speciation proceeded through the same general action of evolutionary forces (in terms of type, direction, and strength) normally operating on these traits within species. In contrast, strikingly different patterns of within-species versus between-species comparisons could reveal the operation of a different set of evolutionary forces operating during divergence of species than what normally occurs within species.

Interestingly, within- and between-species patterns of genetic variation for mate discrimination and secondary sexual characteristics are very similar, indicating that divergence of these traits probably reflects the direct extension of the same evolutionary forces (most likely directional sexual selection in this case) that operate on these traits within species. In contrast, within-and between-species genetic patterns of sterility and inviability do not show similar patterns at all. Not only is the role of epistasis drastically different between the two comparisons, the relative frequency of genes that affect sterility versus inviability is completely reversed depending on whether within- or between-species patterns are considered. This disjunction between the genetic patterns observed within species versus those observed between species suggests that the evolution of postzygotic reproductive incompatibilities may result from the accumulated action of relatively rare evolutionary events happening over long periods of time. Such rare events may include periodic episodes of random genetic drift happening alone or in combination with natural and sexual selection working on these traits with varying intensity or changes in direction over the course of evolution.