Reduces memory usage and improves speed

[courcelm]

Current slicing/join was concatenating the whole chromosome sequence when only a slice was required. That required 2-3 GB of memory in some case and it was slow in my use case:

Before

I suggest reverting back to original pyGeno code that was changed by @ericloud .

After

@ericloud
Is this a problem with your bug reported here:
2fd3fbe#diff-9a0352f44c9b0e9b00e4e2df44eae54b

[ericloud]

If my memories are correct, I change this part of code to return the correct sequence when insertion, deletion or indel are present.

example:
_ _ _ _ D _ _ _ _ _ S _ _ _ _ _ _ _ _ _ _
_: 1 nucleotide without change
D: a deletion on 1 nuc
S: a SNP on 1 nuc
And we want to return X nuc on the left and on the right of the SNP. This case is very common when you try to load dbSNP.

If we apply the slice on data (old version), the function will return only (X-1) nuc on the left, because one of them is deleted.

But as you mentioned, apply the slice on sequence (new version 3bb0518 and 2fd3fbe) involve to load all the chromosome and increase memory usage and time.

There are certainly a more efficient way to do it, but go back to the old version, just change the problem to somewhere else.

[ericloud]

Also have you try to scan your mutations by Chromosome?
In that way the sequence should stay in cache and be loaded only one time.

Something like:

for chr in genome.iterGet(Chromosome):
    for snp in chr.get(dbSNPSNP):
        ...

[courcelm]

I'm already processing by chromosome and it reloads the sequence for every transcript.

[ericloud]

Even with something like :

for chr in genome.iterGet(Chromosome):
    chr_seq = chrom.getSequenceData()
    for snp in chr.get(dbSNPSNP):
        bin_seq = NucBinarySequence(chr_seq[slice(snp.start,snp.end,1))])
        ...

It's my last suggestion to help you.

I work on other project for the moment, sorry but I don't have the time to improve this specific case.

[courcelm]

I can't apply this since I don't want to load the whole chromosome sequence to memory. Anyway I have solve my problem. Thanks for your help.

[tariqdaouda]

What is your solution?

[courcelm]

I don't have a solution that fixes both the high memory usage and @ericloud bug. I found a way in my script to skip the call to _getSequence

[tariqdaouda]

I also have noticed this drop in performances. This is something that would need to be improved in the future.