Augustus [accuracy]

AUGUSTUS is used in many genome annotation projects. Below are some accuracy values in comparison to other programs. As accuracy measure we use sensitivity (Sn) and specificity (Sp). For a feature (coding base, exon, transcript, gene) the sensitivity is defined as the number of correctly predicted features divided by the number of annotated features. The specificity is the number of correctly predicted features divided by the number of predicted features. A predicted exon is considered correct if both splice sites are at the annotated position of an exon. A predicted transcript is considered correct if all exons are correctly predicted and no additional exons not in the annotation. A predicted gene is considered correct if any of its transcripts are correct, i.e. if at least one isoform of the gene is exactly as annotated in the reference annotation.

Accuracy results from the rGASP Assessment (round 2) using RNA-Seq

The complete accuracy statistics is available on a page from the Conputational Genomics Lab, Barcelona. Below pictures are loaded only after confirming the site authenticity.
human coding exon level human coding transcript level
human CDS level human CDS level
fly coding exon level fly coding gene level
fly CDS level fly CDS level
worm coding exon level
worm CDS level
Above accuracy plots are from Josep Abril, Computational Genomics Lab. Our AUGUSTUS predictions are labelled Mar.*. The worst performing prediction of AUGUSTUS (there are 3 sets in human, and worm each, and 2 sets in fly) are ab initio predictions and do not use any RNA-Seq at all. Other participant codes are here.

Accuracy results from recent nGASP assessment on C. elegans: transcript-based
programbaseexontranscriptgene
SnSpSnSpSnSpSnSp
AUGUSTUS99.090.592.580.268.347.180.151.8
Fgenesh++97.689.790.480.965.553.478.354.2
MGENE98.791.991.080.657.748.070.651.1
EUGENE98.585.192.170.360.831.568.836.1
ExonHunter93.792.081.276.937.239.745.640.5
Gramene98.295.488.571.841.719.648.737.2
MAKER92.988.580.766.341.319.650.747.6
Above accuracy values are taken from Coghlan et al. (2008): nGASP: the nematode genome annotation assessment project.

Accuracy results from recent nGASP assessment on C. elegans: ab initio
programbaseexontranscriptgene
SnSpSnSpSnSpSnSp
AUGUSTUS97.089.086.172.650.128.761.138.4
Fgenesh98.287.186.473.647.134.657.835.4
GeneMark.hmm98.383.183.265.637.724.046.324.5
MGENE97.291.584.678.644.640.954.842.3
GeneID93.988.277.068.636.222.844.425.1
Agene93.883.468.961.19.813.112.014.1
CRAIG95.690.980.278.235.736.343.837.8
EUGENE94.089.580.373.049.128.860.230.2
ExonHunter95.486.072.662.515.518.619.119.2
GlimmerHMM97.687.684.471.447.329.358.030.6
SNAP94.084.574.661.332.618.640.019.1
Above accuracy values are taken from Coghlan et al. (2008): nGASP: the nematode genome annotation assessment project.

Accuracy results on human ENCODE regions (ab initio)
long human sequences Program
AUGUSTUS GENSCAN GENEID GENEMARK GENEZILLA
base level sensitivity 78.65% 84.17% 76.77% 76.09% 87.56%
base level specificity 75.29% 60.60% 76.48% 62.94% 50.93%
exon level sensitivity 52.39% 58.65% 53.84% 48.15% 62.08%
exon level specificity 62.93% 46.37% 61.08% 47.25% 50.25%
gene level sensitivity 24.32% 15.54% 10.47% 16.89% 19.59%
gene level specificity 17.22% 10.13% 8.78% 7.91% 8.84%
Above accuracy values are taken from Guigó et al. (2006): EGASP: the human ENCODE Genome Annotation Assessment Project.

Accuracy results on fruit fly data set adh222
long drosophila sequence Program
AUGUSTUS GENEID GENIE
base level sensitivity (std1) 98% 96% 96%
base level specificity (std3) 93% 92% 92%
exon level sensitivity (std1) 86% 71% 70%
exon level specificity (std3) 66% 62% 57%
gene level sensitivity (std1) 71% 47% 40%
gene level specificity (std3) 39% 33% 29%

Accuracy results on Arabidopsis data set araset
multi-gene sequences Program
AUGUSTUS
base level sensitivity 97%
base level specificity 72%
exon level sensitivity 89%
exon level specificity 70%
gene level sensitivity 62%
gene level specificity 39%

adh222 is a single sequence of drosophila melanogaster and 2.9Mb long.
There are two sets of annotations. The first, smaller set, called std1, was chosen so that the genes in it are likely to be correctly annotated and the second larger set, called std3, was chosen to be as complete as possible.
This dataset and the annotation was taken from here.

In the corrected version std1 contains 38 genes with a total of 111 exons and std3 contains 222 genes with a total of 909 exons.
The genes lie on both strands.

araset is a set of 74 multi-gene sequences with 168 genes of Arabidopsis thaliana. The specificity is likely to be underestimated because there are sometimes genes at the boundaries of a sequence that are not annotated.

The datasets can be downloaded.