Arrow Flight Example

A simple Arrow Flight example in C++.

Note that the below instructions assume Ubuntu 20.04.

Prerequisites

Install Arrow Libraries

wget https://apache.jfrog.io/artifactory/arrow/$(lsb_release --id --short | tr 'A-Z' 'a-z')/apache-arrow-apt-source-latest-$(lsb_release --codename --short).deb
sudo apt install -y ./apache-arrow-apt-source-latest-$(lsb_release --codename --short).deb
sudo apt update
rm apache-arrow-apt-source-latest-$(lsb_release --codename --short).deb

# See the list of available Arrow packages with "apt search libarrow"
sudo apt install -y libarrow-dev libarrow-dataset-dev libarrow-flight-dev

Install gRPCurl

wget https://github.com/fullstorydev/grpcurl/releases/download/v1.8.6/grpcurl_1.8.6_linux_x86_64.tar.gz
tar -zxvf grpcurl_1.8.6_linux_x86_64.tar.gz
sudo mv grpcurl /usr/local/bin
rm grpcurl_1.8.6_linux_x86_64.tar.gz

grpcurl --version

Install Jupyter Notebook

In order to use the example Arrow Flight client (Python), install Jupyter Notebook. For example, deploy the jupyter/all-spark-notebook in Docker.

Usage

Build Arrow Flight Server

cmake .
make

Run Arrow Flight Server

# Use "./server -?" to see available options
./server

Execute Arrow Flight Method (gRPCurl)

# Must provide protocol file to each call unless the gRPC reflection API is implemented
wget https://raw.githubusercontent.com/apache/arrow/master/format/Flight.proto

# List the available flights
grpcurl -import-path . -proto Flight.proto -plaintext localhost:1234 arrow.flight.protocol.FlightService/ListFlights

# Get information on a particular flight
grpcurl -import-path . -proto Flight.proto -plaintext -d @ localhost:1234 arrow.flight.protocol.FlightService/GetFlightInfo <<EOF
{
    "type": "PATH",
    "path": [
        "userdata1.parquet"
    ]
}
EOF

# Retrieve the data with a ticket
grpcurl -import-path . -proto Flight.proto -plaintext -d @ localhost:1234 arrow.flight.protocol.FlightService/DoGet <<EOF
{
    "ticket": "dXNlcmRhdGExLnBhcnF1ZXQ="
}
EOF

# List the available actions
grpcurl -import-path . -proto Flight.proto -plaintext localhost:1234 arrow.flight.protocol.FlightService/ListActions

# Delete a dataset using the dopr_dataset action
grpcurl -import-path . -proto Flight.proto -plaintext -d @ localhost:1234 arrow.flight.protocol.FlightService/DoAction <<EOF
{
    "type": "drop_dataset",
    "body": "$(printf userdata1.parquet | base64)"
}
EOF

Execute Arrow Flight Method (Python)

Open the Jupyter Notebook (arrow-flight-client.ipynb) and execute it to see an example of calling the various Arrow Flight methods.