samtools error

[hth4]

Hi,

I got an error with samtools 1.18-7-gae19296 using htslib 1.18-17-g5acbc15, both on mac and centos7
I used the exact commands on Centos
On MAC I got a samtools config error (ld: unknown option: -R...) and continued without the LDFLAG option

$HOME/opt/samtools/bin/samtools view -H /tmp/secret.bam

[W::bam_hdr_read] bgzf_check_EOF: Bad file descriptor
@hd VN:1.6 SO:coordinate
@sq SN:MN908947.3 LN:29903 UR:https://www.ncbi.nlm.nih.gov/nuccore/MN908947.3?report=fasta AS:MN908947.3 M5:105c82802b67521950854a851fc6eefd SP:SARS-CoV-2 isolate Wuhan-Hu-1
@pg PN:bwa ID:bwa VN:0.7.17-r1188 CL:bwa mem -t 4 MN908947.3.fa 36316_2#282_1_val_1.fq.gz 36316_2#282_2_val_2.fq.gz
@pg PN:dehumanizer ID:dehumanizer.20210215 VN:0.8.1 CL:/lustre/scratch121/esa-analysis-20200609/tmp/kl2/miniconda3/bin/dehumanise /lustre/scratch121/esa-analysis-20200609/tmp/kl2/ftc/manifest.txt /dev/stdin --preset sr --bam -o /dev/stdout --trash-minalen 25 --log dhlog.log PP:bwa
@pg ID:samtools PN:samtools PP:dehumanizer.20210215 VN:1.11 CL:/software/pkgg/samtools/1.11.0/bin/samtools view -h -
@pg ID:samtools.1 PN:samtools PP:samtools VN:1.11 CL:/software/pkgg/samtools/1.11.0/bin/samtools view -b -
@pg ID:samtools.2 PN:samtools PP:samtools.1 VN:1.11 CL:/software/pkgg/samtools/1.11.0/bin/samtools view -C -
@pg ID:samtools.3 PN:samtools PP:samtools.2 VN:1.18-7-gae19296 CL:/Users/hthiele/opt/samtools/bin/samtools view -b -o crypt4gh:/tmp/secret.bam /tmp/MILK-11786A3.cram
@pg ID:samtools.4 PN:samtools PP:samtools.3 VN:1.18-7-gae19296 CL:/Users/hthiele/opt/samtools/bin/samtools view -H /tmp/secret.bam
samtools(11308,0x10cfd6600) malloc: *** error for object 0x834bb8db41c0: pointer being freed was not allocated
samtools(11308,0x10cfd6600) malloc: *** set a breakpoint in malloc_error_break to debug
zsh: abort $HOME/opt/samtools/bin/samtools view -H /tmp/secret.bam

Thanks

[grbot]

@hth4 initially I thought there might have been something with your downloaded CRAM file or somewhere where it has been manipulated. I however tested with the same versions you mentioned and get the same memory issue.

$HOME/opt/samtools/bin/samtools view -H /tmp/secret.bam
[W::bam_hdr_read] bgzf_check_EOF: Bad file descriptor
@HD	VN:1.6	SO:coordinate
@SQ	SN:MN908947.3	LN:29903	UR:https://www.ncbi.nlm.nih.gov/nuccore/MN908947.3?report=fasta	AS:MN908947.3	M5:105c82802b67521950854a851fc6eefd	SP:SARS-CoV-2 isolate Wuhan-Hu-1
@PG	PN:bwa	ID:bwa	VN:0.7.17-r1188	CL:bwa mem -t 4 MN908947.3.fa 36316_2#282_1_val_1.fq.gz 36316_2#282_2_val_2.fq.gz
@PG	PN:dehumanizer	ID:dehumanizer.20210215	VN:0.8.1	CL:/lustre/scratch121/esa-analysis-20200609/tmp/kl2/miniconda3/bin/dehumanise /lustre/scratch121/esa-analysis-20200609/tmp/kl2/ftc/manifest.txt /dev/stdin --preset sr --bam -o /dev/stdout --trash-minalen 25 --log dhlog.log	PP:bwa
@PG	ID:samtools	PN:samtools	PP:dehumanizer.20210215	VN:1.11	CL:/software/pkgg/samtools/1.11.0/bin/samtools view -h -
@PG	ID:samtools.1	PN:samtools	PP:samtools	VN:1.11	CL:/software/pkgg/samtools/1.11.0/bin/samtools view -b -
@PG	ID:samtools.2	PN:samtools	PP:samtools.1	VN:1.11	CL:/software/pkgg/samtools/1.11.0/bin/samtools view -C -
@PG	ID:samtools.3	PN:samtools	PP:samtools.2	VN:1.18-7-gae19296	CL:/home/gerrit/opt/samtools/bin/samtools view -b -o crypt4gh:/tmp/secret.bam /tmp/MILK-11786A3.cram
@PG	ID:samtools.4	PN:samtools	PP:samtools.3	VN:1.18-7-gae19296	CL:/home/gerrit/opt/samtools/bin/samtools view -H /tmp/secret.bam
Segmentation fault (core dumped)

@daviesrob maybe you have an idea of what is going on here?

[hth4]

There is this error when I try to test the samtools compilation (this time on rocky8):

The command failed [32512]: cat /tmp/Vl6bjmqJZ9/bgzip.dat | bgzip -ci -I/tmp/Vl6bjmqJZ9/bgzip.dat.gz.gzi > /tmp/Vl6bjmqJZ9/bgzip.dat.gz
out:
err:bash: line 1: bgzip: command not found

at test/test.pl line 99.
main::error("The command failed [32512]: cat /tmp/Vl6bjmqJZ9/bgzip.dat | b"..., "out:\x{a}", "err:bash: line 1: bgzip: command not found\x{a}\x{a}") called at test/test.pl line 212
main::cmd("cat /tmp/Vl6bjmqJZ9/bgzip.dat | bgzip -ci -I/tmp/Vl6bjmqJZ9/b"...) called at test/test.pl line 513
main::test_bgzip(HASH(0x55b48cdc44d8)) called at test/test.pl line 39
make: *** [Makefile:241: test] Error 127

[daviesrob]

I've managed to reproduce the crash. It looks like something's changed in how HTSlib closes files which is causing it to delete the crypt4gh handle's data before it's been closed correctly. I'll see if I can fix it.

[daviesrob]

Hmm, looks like the problem is due to samtools/htslib#1577, which fixed crypt4gh redirection with htsget, but seems to have broken it for other file types. It does work if you tell samtools that you're reading a crypt4gh file though:

$HOME/opt/samtools/bin/samtools view -c crypt4gh:/tmp/secret.bam
772698

Getting the redirection to work again shouldn't be too difficult, although there are some issues with how to clean up properly in hts_hopen() if it detects an error after having discovered that a file is crypt4gh...

[grbot]

Thank you @daviesrob

@hth4 for now, please do as Rob suggests. When using samtools specify that you are using a crypt4gh file.

[hth4]

@grbot,@daviesrob

I will follow your suggestions.
Thank you!

[hth4]

@daviesrob

May I ask you another question regarding crypt4ghfs? How can I unmount the mount point? Killing the process seems not to be a good idea ...

Thanks

update: of course just using umount ...