Running happy with rtg vcfeval engine does not result with FP.AL 0

[kubranarci]

Hello, I am experimenting happy running with rtg vcfeval engine, and a potential bug (or an error from my side) took my attention. Running happy with rtg vcfeval engine didnt result with FP.AL 0. Can someone please take a look over my run and give an explanation over this?

hap.py \
    HG002_GRCh38_CMRG_smallvar_v1.00.reheader.vcf.gz \
    test2.chr21.reheader.vcf.gz \
    --engine=vcfeval --leftshift \
    --reference genome.fa \
    --threads 4 \
    -f HG002_GRCh38_CMRG_smallvar_v1.00.bed \
     \
     \
     \
    -o test2.bcftools

Type,Filter,TRUTH.TOTAL,TRUTH.TP,TRUTH.FN,QUERY.TOTAL,QUERY.FP,QUERY.UNK,FP.gt,FP.al,METRIC.Recall,METRIC.Precision,METRIC.Frac_NA,METRIC.F1_Score,TRUTH.TOTAL.TiTv_ratio,QUERY.TOTAL.TiTv_ratio,TRUTH.TOTAL.het_hom_ratio,QUERY.TOTAL.het_hom_ratio
INDEL,ALL,3629,104,3525,13196,18,13069,14,4,0.028658,0.858268,0.990376,0.055464,,,1.4202586206896552,1.9555156144686587
INDEL,PASS,3629,104,3525,13196,18,13069,14,4,0.028658,0.858268,0.990376,0.055464,,,1.4202586206896552,1.9555156144686587
SNP,ALL,17603,535,17068,73238,35,72670,17,13,0.030393,0.9383799999999999,0.992244,0.058878,2.353456484479147,1.8436966309579257,1.4225565985643291,2.0340296775263202
SNP,PASS,17603,535,17068,73238,35,72670,17,13,0.030393,0.9383799999999999,0.992244,0.058878,2.353456484479147,1.8436966309579257,1.4225565985643291,2.0340296775263202

I also made a comparison run running without the engine, with default settings:

hap.py \
    HG002_GRCh38_CMRG_smallvar_v1.00.reheader.vcf.gz \
    test2.chr21.reheader.vcf.gz \
     \
    --reference genome.fa \
    --threads 4 \
    -f HG002_GRCh38_CMRG_smallvar_v1.00.bed \
     \
     \
     \
    -o test2.bcftools

Type,Filter,TRUTH.TOTAL,TRUTH.TP,TRUTH.FN,QUERY.TOTAL,QUERY.FP,QUERY.UNK,FP.gt,FP.al,METRIC.Recall,METRIC.Precision,METRIC.Frac_NA,METRIC.F1_Score,TRUTH.TOTAL.TiTv_ratio,QUERY.TOTAL.TiTv_ratio,TRUTH.TOTAL.het_hom_ratio,QUERY.TOTAL.het_hom_ratio
INDEL,ALL,3605,101,3504,13196,23,13069,10,7,0.028017,0.8188979999999999,0.990376,0.05417999999999999,,,1.4202586206896552,1.9555156144686587
INDEL,PASS,3605,101,3504,13196,23,13069,10,7,0.028017,0.8188979999999999,0.990376,0.05417999999999999,,,1.4202586206896552,1.9555156144686587
SNP,ALL,17559,532,17027,73238,36,72670,13,12,0.030298000000000002,0.9366200000000001,0.992244,0.058697000000000006,2.3617224880382777,1.8436966309579257,1.4225565985643291,2.0340296775263202
SNP,PASS,17559,532,17027,73238,36,72670,13,12,0.030298000000000002,0.9366200000000001,0.992244,0.058697000000000006,2.3617224880382777,1.8436966309579257,1.4225565985643291,2.0340296775263202

[Lenbok]

rtg vcfeval does in fact support the loose positional matching annotation that hap.py uses to compute this metric, so a non-zero value in this column is possible.

[kubranarci]

Dear Lenbok, I would suggest to update readme https://github.com/Illumina/hap.py/blob/master/doc/happy.md#comparison-engines accordingly then. I guess that behaviour changed with the latest versions.