-
Notifications
You must be signed in to change notification settings - Fork 20
v2.0.0: Refactor for sample-wise parameterisation #171
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
nschan
wants to merge
110
commits into
nf-core:dev
Choose a base branch
from
nschan:refactor-assemblers
base: dev
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
* add prefix to singularity container for report * add files exist check for references, closes nf-core#165
* Template update for nf-core/tools version 3.2.1 * Template update for nf-core/tools version 3.3.1 * merge template 3.3.1 - fix linting * update pre-commit * merge template 3.3.1 - fix linting * pre-commit config? * pre-commit config? * reinstall links * try larger runner * smaller run, disable bloom filter for hifiasm test * updated test snapshot * updated test snapshot * update nftignore * update nftignore * update nftignore * update nftignore * update nftignore * update nftignore * update nftignore * update nftignore * update nftignore * Update .github/actions/nf-test/action.yml Co-authored-by: Matthias Hörtenhuber <mashehu@users.noreply.github.com> * Update docs/output.md Co-authored-by: Matthias Hörtenhuber <mashehu@users.noreply.github.com> * remove .nf-test.log --------- Co-authored-by: Niklas Schandry <niklas@bio.lmu.de> Co-authored-by: Matthias Hörtenhuber <mashehu@users.noreply.github.com>
Member
|
Warning Newer version of the nf-core template is available. Your pipeline is using an old version of the nf-core template: 3.3.2. For more documentation on how to update your pipeline, please see the nf-core documentation and Synchronisation documentation. |
* update awk regex * update snapshot
* add prefix to singularity container for report * include gawk in gfa2fa env * include gawk in gfa2fa env
* add prefix to singularity container for report * include gawk in gfa2fa env * include gawk in gfa2fa env * mawk version * mawk version in stub * update CHANGELOG * Update CHANGELOG.md Co-authored-by: Simon Pearce <24893913+SPPearce@users.noreply.github.com> * [automated] Fix code linting --------- Co-authored-by: Simon Pearce <24893913+SPPearce@users.noreply.github.com> Co-authored-by: nf-core-bot <core@nf-co.re>
…ls before joining instead of doing two joins.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
As suggested here this is full refactor of
genomeassemblerto support sample-level parameterisation of everything.Currently, this PR contains the full pipeline, tested with stub runs of heterogenous samples in a single sample sheet.
Why?
Often when doing genome assembly, we do not know what works best. With this change, this pipeline can be used to compare different settings for the same set of reads, to compare the assembly outcome. Samples that share the same value in
groupwill be combined during reporting to facilitate comparisons of strategies on the same input(s). The report process / script will be updated to fit this new design (ongoing).Details
This was a bit more tricky than I had initially hoped. Essentially, all
paramsare stuffed into a main channel, which contains a map. I think a map is the only way to handle this channel safely, since sometimes entries are replaced and I am afraid that positional indexing would be too confusing (for me).This works fine, but channels containing maps cannot be joined. For this reason, a pattern that looks like:
map_channel_1 // Convert to list for join .map { it -> it.collect { entry -> [ entry.value, entry ] } } .join( map_channel_2 // Convert to list for join .map { it -> it.collect { entry -> [ entry.value, entry ] } } ) // After joining re-create the maps from the stored map .map { it -> it.collect { _entry, map -> [ (map.key): map.value ] }.collectEntries() }is used throughout to join map channels and recover the map after joining.
Generally, to facilitate asynchronous movement
mix()is used andjoin()is (hopefully) used considerately to avoid blocking resulting from waiting for processes.The overall sample-wise parameterisation is offloaded to
subworkflows/local/utils_nfcore_genomeassembler_pipeline/main.nf; currently this does not produce errors (even though it should). This should be a minor fix.This also does some validation, and consolidates conflicts that may arise from
paramsthat are incompatible with certain samples, e.g.medakacannot be used if there are no ONT reads.Currently, there are no tests included since I would like to get some feedback on whether this is at all reasonable, or if there would be better ways to do things.
I have tested this with a samplesheet that looks like:
In combination with different params (e.g.
--polish_pilon,--scaffold_longstitch, etc) in stub runs, so I think the overall logic is fine.