Added Preprocessing Bundle to ML_Core by vzeufack · Pull Request #25 · hpcc-systems/ML_Core

vzeufack · 2020-09-10T19:45:56Z

The current version of the Preprocessing Bundle includes:

LabelEncoder
OneHotEncoder
StandardScaler
MinMaxScaler
Normaliz
Split
StratifiedSplit

richardkchapman · 2020-09-17T09:23:13Z

@RogerDev Please review

lilyclemson

@vzeufack Good code and testing! There are a few merge conflicts and minor typos. Please resolve.

Preprocessing/LabelEncoder.ecl

Preprocessing/OneHotEncoder.ecl

lilyclemson · 2020-09-11T15:58:32Z

Preprocessing/StandardScaler.ecl

+  SHARED numberLayout := Preprocessing.Types.numberLayout;
+
+  /**
+    * Computes averages and stdevs for each feature in baseData.


Please try to void shorthand: stdevs = standard deviation (stdevs)

lilyclemson · 2020-09-23T13:53:29Z

Preprocessing/LabelEncoder.ecl

+    RETURN Result;
+  ENDMACRO;
+
+<<<<<<< HEAD


Remove line 59. It looks like a leftover from a merge conflict

lilyclemson · 2020-09-23T14:02:51Z

Preprocessing/Test/Functional/RunFunctionalTests.ecl

+$.TestOneHotEncoder.RunOneHotEncoderTests;
+$.TestStandardScaler.RunStandardScalerTests;
+$.TestMinMaxScaler.RunMinMaxScalerTests;
+<<<<<<< HEAD


Please resolve the merge conflict

lilyclemson · 2020-09-23T14:05:10Z

Preprocessing/Test/Functional/TestUtils/TestDatasetComparator/RunDatasetComparatorTests.ecl

+$.TestAreEqualRows.TestDifferentRows();
+
+$.TestCompare.TestEqualData();
+<<<<<<< HEAD


Please resolve the merge conflict

lilyclemson · 2020-09-23T14:05:21Z

Preprocessing/Test/Functional/TestUtils/TestDatasetComparator/TestCompare.ecl

+  END;
+
+  /**
+<<<<<<< HEAD


Please resolve the merge conflict

Preprocessing/Types.ecl

lilyclemson · 2020-09-23T14:09:31Z

Preprocessing/Utils/DatasetComparator/Compare.ecl

+                     PROJECT(ROWS(LEFT), XF(LEFT)));
+
+  #UNIQUENAME(comparisonResult)
+<<<<<<< HEAD


Please resolve the merge conflict

lilyclemson

Some minor typos need correction. A few descriptions need more details.
Great job! @vzeufack

lilyclemson · 2020-09-24T20:20:35Z

Preprocessing/LabelEncoder.ecl

+  *   </pre>
+  */
+  EXPORT GetMapping(key) := FUNCTIONMACRO
+    IMPORT Preprocessing.Utils.LabelEncoder;


It may cause broken code. please use relative path.

lilyclemson · 2020-09-24T20:36:11Z

Preprocessing/LabelEncoder.ecl

+    *   <p> Data with categorical values replaced by numbers.
+    */
+  EXPORT Encode(dataToEncode, key) := FUNCTIONMACRO
+    IMPORT Preprocessing.Utils;


It may cause broken code. please use relative path.

lilyclemson · 2020-09-24T20:38:10Z

Preprocessing/LabelEncoder.ecl

+    *   <p> Data with categorical values replaced by their original labels.
+    */
+  EXPORT Decode(dataToDecode, encoderKey) := FUNCTIONMACRO
+    IMPORT Preprocessing.Utils;


It may cause broken code. please use relative path.

lilyclemson · 2020-09-24T20:45:20Z

Preprocessing/Test/Performance/TestLabelEncoder/1_Data_Import.ecl

+IMPORT STD;
+IMPORT $.Files;
+
+spray := STD.File.SprayDelimited('192.168.56.101',


Please abstract the IP address

lilyclemson · 2020-09-24T20:45:30Z

Preprocessing/Test/Tutorial/01_DataImport.ecl

+IMPORT STD;
+IMPORT $.Files;
+
+STD.File.SprayDelimited('192.168.56.101',


Please abstract the IP address

lilyclemson · 2020-09-25T13:58:01Z

Preprocessing/LabelEncoder.ecl

+   *
+   * @param partialKey: same record structure as the key (see below).                
+   *   <p> Mapping between feature names and categories. 
+   *   Some names are mapped to empty categories such that 


Please add a description for the case where the names are mapped to non-empty categories.

lilyclemson · 2020-09-25T15:56:20Z

Preprocessing/MinMaxScaler.ecl

+t_FieldReal := MLC.types.t_FieldReal;
+
+/**
+ * shifts the values in a range [min, max].


Minor typo: Shift

lilyclemson · 2020-09-25T15:58:17Z

Preprocessing/MinMaxScaler.ecl

+
+
+  /**
+    * scales the data using the following formula:


Typo: Scale

lilyclemson · 2020-09-25T16:06:38Z

Preprocessing/OneHotEncoder.ecl

+  END;
+
+  /**
+    * Produces a mapping between numbers when encoded to numbers when decoded.


Please rephrase this line for better description of the function.

lilyclemson · 2020-09-25T16:09:09Z

Preprocessing/StratifiedSplit.ecl

+END;
+
+/**
+ * Determines y stats from full data.


Please explain y stats

gravesee · 2021-04-08T18:28:50Z

Is this branch going to be merged? Analytics would like to use the preprocessing module to prepare data for deep learning training. I have cloned @vzeufack 's repository yet run into this error when trying to import the module:

Not sure if merging the branch would resolve this issue or not. Please advise!

lilyclemson · 2021-04-08T19:00:01Z

@Zelazny7 If ML_Core bundle is already installed previously, it may conflict with the downloaded ML_Core bundle. Rename the downloaded bundle should solve the issue. Please let me know if anything is unclear.

vzeufack force-pushed the master branch from e119eb1 to 59628f5 Compare September 19, 2020 05:29

lilyclemson suggested changes Sep 23, 2020

View reviewed changes

vzeufack force-pushed the master branch 5 times, most recently from e6dff78 to 92ac0e2 Compare September 23, 2020 22:02

lilyclemson reviewed Sep 25, 2020

View reviewed changes

Added Preprocessing Bundle to ML_CORE

0e9d735

vzeufack force-pushed the master branch from 92ac0e2 to 0e9d735 Compare September 29, 2020 15:04

vzeufack added 2 commits January 27, 2021 23:09

Added Documentation

b0ced51

Update README.md

f7fb58b

Conversation

vzeufack commented Sep 10, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

richardkchapman commented Sep 17, 2020

Uh oh!

lilyclemson left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

lilyclemson left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

gravesee commented Apr 8, 2021

Uh oh!

lilyclemson commented Apr 8, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

vzeufack commented Sep 10, 2020 •

edited

Loading