GoStream - A library for writing scalable data streaming workflows

Overview

GoStream is a library for writing data streaming workflows using the Flow-based programming paradigm. Although other libraries exist, GoStream provides the following advantages

• A set of standard and abstract operators for writing scalable workflows [see our CVPR'14 Paper].
• A simple programming interface for constructing workflow graphs [see our ICDSC'16 Paper].
• An automatic end-to-end throughput versus latency optimization of workflow performance.
• Enabling feedback loops for performance and parameter tuning [see our UCCV'14 Paper].
• Runtime switching between different data processing algorithms for the same operator [see our ICDSC'16 Paper].

We are also working on adding the following features:

• Adding support to distribute workflows using Docker Swarm.
• Integration with the NATS High-Performance server.
• Elastic scaling of workflows while enabling efficient utilization of cloud resources using our end-to-end throughput versus latency optimization module.

GoStream Module for Computer Vision (Currently available on request)

We provide a module for programming online computer vision algorithms processing image and video streams. The module supports the following features,

• GUI windows for displaying images or controlling workflows using Go-GTK
• GNU Plots for displaying dynamic charts using Go-Gnuplot
• OpenCV support using Go-OpenCV

GoStream Module for Machine Learning (Currently available on request)

Learning from data streams is called data stream mining and it aims to extract information models and structures from continuous and possibly infinite streams. We are currently supporting the following algorithms:

• Online Hierarchical Clustering Kanen et al., ICDM'09
• Bayesian Optimization [See Snoek et al. paper]

GoStream Module for Automatic Algorithm Configuration (Currently available on request)

As workflow streaming operators implements different algorithms, it becomes hard to manually identify the best parameter settings for the best performance of every algorithm. The algorithm configuration module implements automatic methods for adaptive parameters selection and tuning. In literature, the problem is refered to by Self-tuning and Autonomic computing. The module provides the following features,

• An implementation of the the time-bounded sequential parameter optimization algorithm [see Hutter et al., LION'10].
• Examples showing the integration of parameter optimization with feedback control.

Install

###Linux & Mac OS X

go get github.com/helala/gostream
cd $GOPATH/src/github.com/helala/gostream/samples
go run word-count.go