From 8f99c4e788222adf456d12bcd2c23bdd4759d032 Mon Sep 17 00:00:00 2001 From: Tyler Britten Date: Fri, 7 Apr 2023 11:02:54 -0400 Subject: [PATCH 1/2] update readme --- pipeline-templates/README.MD | 25 ++++++++++++++++++++++--- 1 file changed, 22 insertions(+), 3 deletions(-) diff --git a/pipeline-templates/README.MD b/pipeline-templates/README.MD index 10ffbfa..307f9f2 100644 --- a/pipeline-templates/README.MD +++ b/pipeline-templates/README.MD @@ -1,6 +1,25 @@ # Pipeline Templates -A collection of jsonnet templates that provide specific capabilities. +A collection of jsonnet pipeline templates that provide specific capabilities. -Templates: -* `split-combine` can be used to split text files into multiple files every x lines or combine multiple files into a single file. +## Template Subdirectories + +--- +## `split-combine` +### Description: +can be used to split text files into multiple files every x lines or combine multiple files into a single file. +### Usage +Splitting: +This will take any files in the input repo `myrepo` and split it into seperate files every `1000` lines +```bash +pachctl create pipeline --jsonnet https://raw.githubusercontent.com/pachyderm/examples/master/pipeline-templates/split-combine/splitcombine.jsonnet \ +--arg name="mypipeline" --arg mode="split" \ +--arg lines=1000 --arg source="myrepo" +``` +Combining: +This will combine all the files in the `mypipeline_split` repo and combine them into a single file +```bash +pachctl create pipeline --jsonnet https://raw.githubusercontent.com/pachyderm/examples/master/pipeline-templates/split-combine/splitcombine.jsonnet \ +--arg name="mypipeline_2" --arg mode="combine" --arg source="mypipeline_split" +``` +--- From b8e393f26da15bbb7ccfceac8274049ed1e70775 Mon Sep 17 00:00:00 2001 From: Tyler Britten Date: Fri, 7 Apr 2023 11:05:06 -0400 Subject: [PATCH 2/2] update combine syntax --- pipeline-templates/README.MD | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/pipeline-templates/README.MD b/pipeline-templates/README.MD index 307f9f2..6a765f1 100644 --- a/pipeline-templates/README.MD +++ b/pipeline-templates/README.MD @@ -17,9 +17,9 @@ pachctl create pipeline --jsonnet https://raw.githubusercontent.com/pachyderm/ex --arg lines=1000 --arg source="myrepo" ``` Combining: -This will combine all the files in the `mypipeline_split` repo and combine them into a single file +This will combine all the files in the `mypipeline_split` repo and combine them into a single file called `/pfs/out/combined.csv` ```bash pachctl create pipeline --jsonnet https://raw.githubusercontent.com/pachyderm/examples/master/pipeline-templates/split-combine/splitcombine.jsonnet \ ---arg name="mypipeline_2" --arg mode="combine" --arg source="mypipeline_split" +--arg name="mypipeline_2" --arg mode="combine" --arg source="mypipeline_split" --arg output="/pfs/out/combined.csv" ``` ---