optimize_augustus.pl results in lower prediction accuracy??

Discussions about training AUGUSTUS from various sources of evidence. Not discussed here: BRAKER1 and WebAUGUSTUS!

Moderator: bioinf

Post Reply
jonpalmer
Posts: 3
Joined: Mon Apr 25, 2016 4:20 pm

optimize_augustus.pl results in lower prediction accuracy??

Post by jonpalmer »

Hello,

I'm getting some strange results from optimize_augustus.pl, essentially it is not resulting in improving the training parameters. I'm using ~ 1200 high quality models that I split into a test set (200) and training set, run etraining, run an initial prediction with augustus which yields gene level sensitivity at 0.745 and specificity 0.73. I then run the optimize_augustus.pl script, after it is complete, the gene level prediction sensitivity is now 0.733 and specificity is 0.707. So it seems like after "optimization" the prediction ability is actually worse. I've seen a very similar pattern with different genomes and gene models. I'm running augustus v3.2.1 compiled using gcc-5.

Thanks,
Jon
User avatar
katharina
Site Admin
Posts: 531
Joined: Wed Nov 18, 2015 6:14 pm
Location: Greifswald
Contact:

Re: optimize_augustus.pl results in lower prediction accuracy??

Post by katharina »

Hi Jon,

thank you for pointing out that this can happen. I have also seen this in more than one genome annotation project.

Always make a copy of original parameters before optimize_augustus.pl

Always visualize prediction results in a browser with before/after optimize_augustus.pl

Based on visualization and on the numbers, you can make a decision whether you would like to use the "new" parameters, or the "old" ones.

Best,

Katharina
Post Reply