Skip to content
This repository was archived by the owner on Apr 8, 2021. It is now read-only.

Comments

Initial Spark Support (Port of Scalding)#38

Open
tixxit wants to merge 6 commits intostripe-archive:masterfrom
tixxit:topic/initial-spark-support
Open

Initial Spark Support (Port of Scalding)#38
tixxit wants to merge 6 commits intostripe-archive:masterfrom
tixxit:topic/initial-spark-support

Conversation

@tixxit
Copy link
Contributor

@tixxit tixxit commented Mar 5, 2015

This isn't merge ready yet, but since I know a few other people are interested (eg @non), I figured I'd make the PR now so I can get some more eyes on it.

A few thing worth mentioning:

  • Spark doesn't really have support for something like Execution, so we just use its strict Unit-happy API everywhere.
  • We only write to files when explicitly asked and use .cache() for memoizing the trees in-between steps.
  • Basically just implements expandTimes and expandInMemory, but not prune or validate (not hard, just haven't done it).

@tixxit tixxit changed the title Initial Spark Support Initial Spark Support (Port of Scalding) Mar 5, 2015
@tixxit
Copy link
Contributor Author

tixxit commented Mar 5, 2015

You can run the example with something like: SPARK_HOME=/path/to/spark ./iris-spark

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We have to do this because Algebrid's MutableBackedMap isn't Serializable!

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Soon we can get rid of this: twitter/algebird#424

@CLAassistant
Copy link

CLA assistant check
Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
You have signed the CLA already but the status is still pending? Let us recheck it.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants