TinySystems Module SDK

A Kubebuilder-based Kubernetes operator SDK for building flow-based workflow engines. This SDK provides the complete infrastructure for developing modular operators that can be composed into visual workflows.

Overview

TinySystems Module SDK enables developers to create module operators (like common-module, http-module, grpc-module) that bring specific functionality into a Kubernetes-native flow engine. Each module provides reusable components that can be connected through a port-based architecture to create complex workflows.

Key Features

Port-Based Component Architecture: Visual programming model with input/output ports
JSON Schema-Driven Configuration: Automatic schema generation with UI hints
Message Routing & Retry Logic: Intelligent routing with exponential backoff
Multi-Module Communication: gRPC-based inter-module communication
Expression-Based Data Transformation: Mustache-style {{expression}} syntax with JSONPath for flexible data mapping
Kubernetes-Native: Everything is a CRD with standard controller patterns
OpenTelemetry Integration: Built-in observability with tracing and metrics
CLI Tools: Complete tooling for running and building modules

Architecture

Core Concepts

The SDK is built around several key abstractions:

Custom Resource Definitions (CRDs)

TinyNode: Core execution unit representing a component instance in a flow
- Contains module name, version, component type, port configurations, and edges
- Status includes module/component info, port schemas, and error state
TinyModule: Registry of available modules for service discovery
- References container image
- Status contains module address (gRPC), version, and list of components
TinyFlow: Logical grouping of nodes representing a workflow
- Uses labels to associate nodes together
TinyProject: Top-level organizational unit grouping multiple flows
TinySignal: External trigger mechanism for node execution
- Specifies target node, port, and data payload
- One-off: deleted immediately after successful delivery
- Works like webhooks or manual triggers
TinyTracker: Execution monitoring for detailed tracing
TinyWidgetPage: Custom UI dashboard pages for visualization

Component System

Components are the building blocks of workflows. Each component:

Implements the Component interface (module/component.go)
Defines ports for input/output with positions (Top/Right/Bottom/Left)
Handles messages through the Handle() method
Provides JSON Schema configuration for each port

Message Flow

Standard message flow:

External Trigger (TinySignal)
  → TinySignal Controller
    → Scheduler.Handle()
      → Runner.MsgHandler()
        → Component.Handle()
          → output() callback
            → Next node via edges
  → TinySignal deleted (one-off)

Port System

Ports enable component communication:

Source Ports: Output ports that send data
Target Ports: Input ports that receive data
System Ports:
- _reconcile: Triggers node reconciliation
- _client: Receives Kubernetes client for resource operations
- _settings: Configuration port
- _control: Dashboard control port

Each port can have:

Configuration: JSON schema defining expected input structure
Response Configuration: Schema for output data structure

Edge Configuration

Edges connect ports between nodes and support data transformation using mustache-style expressions:

Expression Syntax: Use {{expression}} to evaluate JSONPath expressions against incoming data.

{
  "body": "{{$.request.body}}",
  "statusCode": 200,
  "greeting": "Hello {{$.user.name}}!",
  "isAdmin": "{{$.user.role == 'admin'}}"
}

Expression Types:

Pure expression ("{{$.field}}") - Returns the actual type (string, number, boolean, object)
Literal value ("hello", 200, true) - Static values passed through as-is
String interpolation ("Hello {{$.name}}!") - Embeds expression results in strings
JSONPath with operators ("{{$.count + 1}}", "{{$.method == 'GET'}}") - Supports arithmetic and comparison

Key Features:

Type preservation: "{{$.count}}" returns a number, not a string
Graceful error handling: If source data is unavailable, expressions return nil
Full JSONPath support via ajson library

Built-in Functions:

The expression engine includes many built-in functions:

Category	Functions
Array	`length()`, `first()`, `last()`, `avg()`, `sum()`, `size()`
String	`upper()`, `lower()`, `trim()`, `reverse()`, `b64encode()`, `b64decode()`
String (multi-arg)	`split(str, sep)`, `join(arr, sep)`, `contains(str, sub)`, `hasprefix(str, prefix)`, `hassuffix(str, suffix)`, `replace(str, old, new)`, `substr(str, start[, len])`, `index(str, sub)`
Math	`abs()`, `ceil()`, `floor()`, `round()`, `sqrt()`, `pow10()`
Logic	`not()`, ternary `? :`

String Function Examples:

{
  "kind": "{{first(split($.target, '/'))}}",
  "name": "{{last(split($.target, '/'))}}",
  "hasNamespace": "{{contains($.args, '-n ')}}",
  "upperName": "{{upper($.name)}}",
  "label": "{{replace($.label, '=', ': ')}}",
  "domain": "{{first(split(last(split($.url, '//')), '/'))}}",
  "message": "{{hasprefix($.error, 'NotFound') ? 'Resource not found' : $.error}}"
}

SDK Components

The SDK provides several packages for module developers:

`/module/` - Core Interfaces

Component: Main interface for component implementation
Port: Port definitions and configuration
Handler: Callback function type for message routing

`/pkg/resource/` - Resource Manager

Unified Kubernetes client providing:

Node/Module/Flow/Project CRUD operations
Signal creation for triggering nodes
Ingress/Service management for exposing ports
Helm release management

`/pkg/schema/` - JSON Schema Generator

Automatic schema generation from Go structs
Custom struct tags for UI hints: configurable, shared, propertyOrder, tab, align
Definition sharing and override mechanism

`/pkg/evaluator/` - Expression Evaluator

Mustache-style {{expression}} syntax processing
JSONPath evaluation via ajson library
Type-preserving evaluation (numbers stay numbers, booleans stay booleans)
Graceful error handling when source data is unavailable

`/pkg/metrics/` - Observability

OpenTelemetry integration with spans
Metrics (gauges, counters)
Tracker system for flow execution monitoring

`/internal/scheduler/` - Message Routing

Manages component instances
Routes messages between nodes
Handles cross-module communication via gRPC

`/internal/client/` - Client Pool

Manages gRPC connections to other modules
Connection pooling and lifecycle management

`/cli/` - Command-Line Tools

run: Complete operator runtime
build: Module building and publishing

Communication Patterns

Same-Module Communication

When nodes belong to the same module, messages are routed directly through the scheduler for optimal performance.

Cross-Module Communication

When nodes belong to different modules:

Scheduler identifies the target module via TinyModule CRD
Client pool establishes/reuses gRPC connection
Message is sent to target module's gRPC server
Target module's scheduler routes to the appropriate component

Retry Mechanism

Transient Errors: Exponential backoff (1s → 30s max)
Permanent Errors: Marked via PermanentError wrapper to stop retries
Context cancellation stops all retries

Design Patterns

Eventual Consistency with Reconciliation: Periodic reconciliation (every 5 minutes) plus signal-based immediate updates
Leader Election: Leader handles control operations and blocking handlers for multi-pod coordination
Schema-Driven Configuration: Go structs automatically generate JSON schemas for UI integration
Expression-Based Transformation: Mustache-style {{expression}} syntax with JSONPath enables flexible data mapping without code changes
Definition Sharing: Components mark fields as shared:true or configurable:true for cross-node type safety
One-off Signals: TinySignals are deleted after delivery for clean trigger semantics

Scalability

The SDK supports horizontal scaling of module operators with leader election for coordination.

Leader Election

Uses Kubernetes native leader election via k8s.io/client-go/tools/leaderelection with Lease resources
Configuration: 15s lease duration, 10s renew deadline, 2s retry period
Each pod identifies itself using the HOSTNAME environment variable
Leadership state tracked via isLeader atomic boolean
Purpose: Only leader runs blocking handlers (e.g., HTTP servers, long-running processes)

One-off TinySignals

TinySignals are deleted immediately after successful delivery
Clean trigger semantics without signal accumulation

Project Structure

.
├── api/v1alpha1/          # CRD definitions (TinyNode, TinyModule, TinyFlow, etc.)
├── module/                # Core SDK interfaces for component developers
│   ├── component.go       # Component interface
│   ├── node.go           # Port definitions
│   └── handler.go        # Handler function type
├── pkg/                   # Reusable SDK packages
│   ├── resource/         # Kubernetes resource manager
│   ├── schema/           # JSON schema generator
│   ├── evaluator/        # JSONPath expression evaluator
│   ├── errors/           # Error handling utilities
│   └── metrics/          # OpenTelemetry integration
├── internal/              # Internal operator implementation
│   ├── controller/       # Kubernetes controllers
│   ├── scheduler/        # Message routing and execution
│   ├── server/           # gRPC server
│   └── client/           # gRPC client pool
├── cli/                   # Command-line tools (run, build)
├── registry/              # Component registration system
├── config/                # Kubernetes manifests and CRD definitions
│   ├── crd/              # CRD YAML files
│   ├── samples/          # Example resources
│   └── rbac/             # RBAC configurations
└── charts/                # Helm charts for deployment

Getting Started

You’ll need a Kubernetes cluster to run against. You can use KIND to get a local cluster for testing, or run against a remote cluster. Note: Your controller will automatically use the current context in your kubeconfig file (i.e. whatever cluster kubectl cluster-info shows).

Helm charts

helm repo add tinysystems https://tiny-systems.github.io/module/
helm repo update # if you already added repo before
helm install my-corp-data-processing-tools --set controllerManager.manager.image.repository=registry.mycorp/tools/data-processing  tinysystems/tinysystems-operator

Running on the cluster

Install Instances of Custom Resources:

kubectl apply -f config/samples/

Build and push your image to the location specified by IMG:

make docker-build docker-push IMG=<some-registry>/operator:tag

Deploy the controller to the cluster with the image specified by IMG:

make deploy IMG=<some-registry>/operator:tag

Undeploy controller

UnDeploy the controller from the cluster:

make undeploy

Contributing

// TODO(user): Add detailed information on how you would like others to contribute to this project

How it works

This project aims to follow the Kubernetes Operator pattern.

It uses Controllers, which provide a reconcile function responsible for synchronizing resources until the desired state is reached on the cluster.

Test It Out

Install the CRDs into the cluster:

make install

Run your controller (this will run in the foreground, so switch to a new terminal if you want to leave it running):

make run

NOTE: You can also run this in one step by running: make install run

Modifying the API definitions

If you are editing the API definitions, generate the manifests such as CRs or CRDs using:

make manifests

Create new api

kubebuilder create api --group operator --version v1alpha1 --kind TinySignal

NOTE: Run make --help for more information on all potential make targets

More information can be found via the Kubebuilder Documentation

Module Development

This SDK provides everything you need to build custom module operators. Here's a complete guide to developing your own module.

Quick Start

Create a New Go Project

mkdir my-module
cd my-module
go mod init github.com/myorg/my-module

Add SDK Dependency

go get github.com/tiny-systems/module

Implement a Component

Create components/hello.go:

package components

import (
    "context"
    "github.com/tiny-systems/module/module"
)

type Hello struct{}

// Configuration for the input port
type HelloInput struct {
    Name string `json:"name" configurable:"true"`
}

// Configuration for the output
type HelloOutput struct {
    Greeting string `json:"greeting" shared:"true"`
}

func (h *Hello) Instance() module.Component {
    return &Hello{}
}

func (h *Hello) GetInfo() module.ComponentInfo {
    return module.ComponentInfo{
        Name:        "hello",
        Description: "Greets a person by name",
        Info:        "Simple greeting component example",
        Tags:        []string{"example", "greeting"},
    }
}

func (h *Hello) Ports() []module.Port {
    return []module.Port{
        {
            Name:          "input",
            Label:         "Input",
            Source:        false, // This is an input port
            Position:      module.Left,
            Configuration: &HelloInput{},
        },
        {
            Name:     "output",
            Label:    "Output",
            Source:   true, // This is an output port
            Position: module.Right,
            Configuration: &HelloOutput{},
        },
        {
            Name:     "error",
            Label:    "Error",
            Source:   true,
            Position: module.Bottom,
        },
    }
}

func (h *Hello) Handle(ctx context.Context, output module.Handler, port string, message any) any {
    if port == "input" {
        // Parse input configuration
        input := message.(*HelloInput)

        // Create greeting
        greeting := "Hello, " + input.Name + "!"

        // Send to output port
        output(ctx, "output", &HelloOutput{
            Greeting: greeting,
        })
    }
    return nil
}

Register Component and Create Main

Create main.go:

package main

import (
    "github.com/myorg/my-module/components"
    "github.com/tiny-systems/module/cli"
    "github.com/tiny-systems/module/registry"
)

func main() {
    // Register all components
    registry.Register(&components.Hello{})

    // Run the operator
    cli.Run()
}

Run Your Module

# Build
go build -o my-module

# Run locally (connects to your current kubectl context)
./my-module run --name=my-module --version=1.0.0 --namespace=tinysystems

Deploy to Kubernetes

# Build and push Docker image
docker build -t myregistry/my-module:1.0.0 .
docker push myregistry/my-module:1.0.0

# Install using Helm
helm repo add tinysystems https://tiny-systems.github.io/module/
helm install my-module \
  --set controllerManager.manager.image.repository=myregistry/my-module \
  --set controllerManager.manager.image.tag=1.0.0 \
  tinysystems/tinysystems-operator

Component Interface Deep Dive

GetInfo() - Component Metadata

func (c *MyComponent) GetInfo() module.ComponentInfo {
    return module.ComponentInfo{
        Name:        "my-component",      // Unique identifier
        Description: "Does something",     // Short description
        Info:        "Detailed info...",   // Long description
        Tags:        []string{"tag1"},     // Searchable tags
    }
}

Ports() - Define Component Ports

Ports define how components connect to each other:

func (c *MyComponent) Ports() []module.Port {
    return []module.Port{
        {
            Name:          "input",           // Unique port name
            Label:         "Input Data",      // Display label
            Source:        false,             // Input port
            Position:      module.Left,       // Visual position
            Configuration: &InputConfig{},    // Expected data structure
        },
        {
            Name:              "output",
            Label:             "Output Data",
            Source:            true,          // Output port
            Position:          module.Right,
            Configuration:     &OutputConfig{}, // Output data structure
        },
    }
}

Port Positions: module.Top, module.Right, module.Bottom, module.Left

Handle() - Process Messages

The Handle method is called when a message arrives on a port:

func (c *MyComponent) Handle(
    ctx context.Context,
    output module.Handler,
    port string,
    message any,
) any {
    switch port {
    case "input":
        // Type assert the message
        input := message.(*InputConfig)

        // Do work
        result := processData(input)

        // Send to output port
        output(ctx, "output", &OutputConfig{
            Result: result,
        })

    case "_reconcile":
        // Handle reconciliation (called periodically)
        // Use this for cleanup, state sync, etc.
    }

    return nil
}

Key Points:

ctx: Context with tracing span and cancellation
output: Callback function to send data to other ports
port: Name of the port that received the message
message: The actual data (type assert to your config struct)
Return value is currently unused

Configuration Schemas

The SDK automatically generates JSON Schemas from your Go structs. Use struct tags to control the UI:

type Config struct {
    // Basic field
    Name string `json:"name"`

    // Configurable in UI (can reference other node outputs)
    UserID string `json:"userId" configurable:"true"`

    // Shared definition (other nodes can reference this)
    Result string `json:"result" shared:"true"`

    // Control UI layout
    APIKey string `json:"apiKey" propertyOrder:"1" tab:"auth"`

    // Nested object
    Settings struct {
        Timeout int `json:"timeout" configurable:"true"`
    } `json:"settings"`

    // Array
    Items []string `json:"items" configurable:"true"`
}

Struct Tags:

configurable:"true": Field can accept values from other nodes via {{expression}} syntax
shared:"true": Field definition is available to other nodes for type-safe mapping
propertyOrder:"N": Controls field order in UI
tab:"name": Groups field under a tab in UI
align:"horizontal": Layout hint for UI

System Ports

Special ports available to all components:

`_reconcile` Port

Called periodically (every 5 minutes) and on node changes:

case "_reconcile":
    // Clean up resources
    // Sync state
    // Check for drift

`_client` Port

Provides Kubernetes client for resource operations:

case "_client":
    client := message.(resource.Manager)

    // Create a signal
    client.CreateSignal(ctx, resource.CreateSignalRequest{
        Node: "target-node",
        Port: "input",
        Data: map[string]any{"key": "value"},
    })
    
    // Get node information
    node, err := client.GetNode(ctx, "node-name")

`_settings` Port

Receives initial configuration (no "from" connection required):

case "_settings":
    settings := message.(*MyConfig)
    // Store settings for later use

Error Handling

Transient Errors

Return regular errors for automatic retry with exponential backoff:

func (c *MyComponent) Handle(ctx context.Context, output module.Handler, port string, msg any) any {
    data, err := fetchFromAPI()
    if err != nil {
        // Will retry automatically
        return err
    }
    // ...
}

Permanent Errors

Use PermanentError to stop retries:

import "github.com/tiny-systems/module/pkg/errors"

func (c *MyComponent) Handle(ctx context.Context, output module.Handler, port string, msg any) any {
    if !isValid(msg) {
        // Won't retry - send to error port instead
        output(ctx, "error", errors.PermanentError{
            Err: fmt.Errorf("invalid input"),
        })
        return nil
    }
    // ...
}

Using the Resource Manager

Access Kubernetes resources from your component:

import "github.com/tiny-systems/module/pkg/resource"

func (c *MyComponent) Handle(ctx context.Context, output module.Handler, port string, msg any) any {
    if port == "_client" {
        c.client = msg.(resource.Manager)
        return nil
    }

    if port == "create-flow" {
        // Create a new flow
        flow, err := c.client.CreateFlow(ctx, resource.CreateFlowRequest{
            Name:    "dynamic-flow",
            Project: "my-project",
        })

        // Create nodes in the flow
        node, err := c.client.CreateNode(ctx, resource.CreateNodeRequest{
            Name:      "node-1",
            Flow:      flow.Name,
            Module:    "http-module",
            Component: "request",
            Settings: map[string]any{
                "url": "https://api.example.com",
            },
        })

        // Trigger the node
        c.client.CreateSignal(ctx, resource.CreateSignalRequest{
            Node: node.Name,
            Port: "trigger",
            Data: map[string]any{},
        })
    }

    return nil
}

Observability

OpenTelemetry is built-in. The context includes a span:

import "go.opentelemetry.io/otel"

func (c *MyComponent) Handle(ctx context.Context, output module.Handler, port string, msg any) any {
    // Get tracer
    tracer := otel.Tracer("my-module")

    // Create child span
    ctx, span := tracer.Start(ctx, "processing")
    defer span.End()

    // Add attributes
    span.SetAttributes(
        attribute.String("input.size", "large"),
    )

    // Do work...
    result := doWork(ctx)

    output(ctx, "output", result)
    return nil
}

Testing Components

package components_test

import (
    "context"
    "testing"
    "github.com/myorg/my-module/components"
)

func TestHello(t *testing.T) {
    component := &components.Hello{}

    var outputData *components.HelloOutput
    outputHandler := func(ctx context.Context, port string, data any) any {
        if port == "output" {
            outputData = data.(*components.HelloOutput)
        }
        return nil
    }

    // Send message to input port
    component.Handle(context.Background(), outputHandler, "input", &components.HelloInput{
        Name: "World",
    })

    // Verify output
    if outputData.Greeting != "Hello, World!" {
        t.Errorf("Expected 'Hello, World!', got '%s'", outputData.Greeting)
    }
}

Best Practices

Keep Components Focused: Each component should do one thing well
Use System Ports: Implement _reconcile for cleanup
Handle Context Cancellation: Respect ctx.Done() for graceful shutdown
Leverage Schemas: Use struct tags to create great UI experiences
Share Definitions: Mark output fields as shared:true for type-safe flows
Use Permanent Errors: Don't retry validation errors or user mistakes
Add Observability: Create spans for long operations
Document with Tags: Use meaningful tags in GetInfo() for discoverability
Always Return Handler Results: See critical section below

CRITICAL: Handler Response Propagation

ALWAYS return the result of handler() calls. Never ignore the return value.

The TinySystems SDK uses blocking I/O for request-response patterns. When a component like HTTP Server sends a request, it blocks waiting for a response to flow back through the same handler chain. If any component in the chain ignores the handler return value, the response is lost and the original caller times out.

BAD - Breaks blocking I/O:

func (c *Component) handleError(ctx context.Context, handler module.Handler, req Request, errMsg string) any {
    if c.settings.EnableErrorPort {
        _ = handler(ctx, "error", Error{...})  // WRONG: ignores return value!
        return nil  // Response is lost, HTTP Server times out
    }
    return errors.New(errMsg)
}

GOOD - Propagates response correctly:

func (c *Component) handleError(ctx context.Context, handler module.Handler, req Request, errMsg string) any {
    if c.settings.EnableErrorPort {
        return handler(ctx, "error", Error{...})  // CORRECT: returns handler result
    }
    return errors.New(errMsg)
}

Why this matters:

When HTTP Server → Slack Command → Router → HTTP Server:Response:

HTTP Server blocks on handler(ctx, "request", req)
Message flows to Slack Command via gRPC
Slack Command processes and calls handler(ctx, "error", error)
Edge transforms Error → Response and sends to HTTP Server's response port
HTTP Server's handleResponse() returns the Response
Response must flow back through the entire chain to unblock HTTP Server

If Slack Command does _ = handler(...) and returns nil, the Response is lost.

Rules:

Always return handler(ctx, port, data) for output ports
Exception: _reconcile port calls can ignore returns (internal system port)
Exception: Fire-and-forget async operations where no response is expected

Component Naming Convention

All component names must follow a consistent naming pattern using snake_case:

Pattern: [technology_][resource]_[action] or [resource]_[action]

Rules:

Always use snake_case (lowercase with underscores)
Resource/noun comes before action/verb
Include technology prefix when multiple implementations exist
Keep names concise (2-3 words max)

Examples by category:

Category	Name	Description
HTTP	`http_server`	HTTP server
	`http_request`	Make HTTP request
	`http_auth_parse`	Parse auth header
Encoding	`json_encode`	Encode to JSON
	`json_decode`	Decode from JSON
	`go_template`	Render Go template
Email	`smtp_send`	Send email via SMTP
	`sendgrid_send`	Send email via SendGrid API
Messaging	`slack_send`	Send Slack message
	`slack_command`	Receive Slack command
Kubernetes	`pod_status_get`	Get pod status
	`pod_logs_get`	Get pod logs
	`deployment_restart`	Restart deployment
	`resource_watch`	Watch K8s resources
Utilities	`transform`	Transform/passthrough data
	`delay`	Delay execution
	`router`	Route messages

When to include technology prefix:

Multiple implementations of same concept (e.g., smtp_send vs sendgrid_send)
Technology-specific behavior (e.g., go_template vs handlebars_template)
Clarity about protocol/API used (e.g., grpc_call vs http_request)

Code Style

Follow idiomatic Go patterns:

Early returns - Avoid nested ifs, return early on errors
Flat structure - Extract logic into small, focused functions
Error handling - Use if err != nil { return } pattern
No deep nesting - Maximum 2-3 levels of indentation

// Good - early return
func (c *Component) Handle(ctx context.Context, handler module.Handler, port string, msg any) error {
    if port != "request" {
        return fmt.Errorf("unknown port: %s", port)
    }

    req, ok := msg.(Request)
    if !ok {
        return errors.New("invalid request")
    }

    result, err := c.process(ctx, req)
    if err != nil {
        return c.handleError(ctx, handler, req, err)
    }

    return handler(ctx, "output", result)
}

// Bad - nested ifs
func (c *Component) Handle(ctx context.Context, handler module.Handler, port string, msg any) error {
    if port == "request" {
        if req, ok := msg.(Request); ok {
            if result, err := c.process(ctx, req); err == nil {
                return handler(ctx, "output", result)
            } else {
                return c.handleError(ctx, handler, req, err)
            }
        }
    }
    return fmt.Errorf("unknown port: %s", port)
}

Component Design Principles

1. Minimize Settings

Settings should ONLY contain:

Port flags - EnableErrorPort, EnableStatusPort (affect port visibility)
Precompiled resources - Go templates, JS code (compiled once on change)
Defaults - Rarely-changing default values

Settings should NOT contain:

Credentials (API keys, tokens, signing secrets)
Runtime URLs/endpoints
Per-request parameters
Anything that varies between executions

// Good - minimal settings
type Settings struct {
    EnableErrorPort bool  `json:"enableErrorPort" title:"Enable Error Port"`
    DefaultLines    int64 `json:"defaultLines" title:"Default Lines"`
}

// Bad - credentials and runtime config in settings
type Settings struct {
    APIKey      string `json:"apiKey"`       // Should be in request
    Endpoint    string `json:"endpoint"`     // Should be in request
    Namespace   string `json:"namespace"`    // Should be in request
}

2. Credentials via Input Ports

All credentials and runtime configuration come through input ports:

type Request struct {
    Context   any    `json:"context,omitempty" configurable:"true"`

    // Credentials - from upstream (e.g., secret manager)
    APIKey    string `json:"apiKey" required:"true" configurable:"true"`

    // Runtime config - varies per execution
    Endpoint  string `json:"endpoint" required:"true" configurable:"true"`
    Namespace string `json:"namespace" required:"true" configurable:"true"`
}

Why: Settings are spread across flows, not programmatically configurable, and storing credentials in settings is a security anti-pattern.

3. Context Passthrough

Every component MUST pass context through for correlation:

type Request struct {
    Context any `json:"context,omitempty" configurable:"true" title:"Context" description:"Arbitrary data passed through to output"`
    // ... other fields
}

type Response struct {
    Context any `json:"context,omitempty" title:"Context"`
    // ... other fields
}

func (c *Component) Handle(ctx context.Context, handler module.Handler, port string, msg any) error {
    req := msg.(Request)

    result := c.process(req)

    // Always pass context through
    handler(ctx, "output", Response{
        Context: req.Context,  // Pass through!
        Data:    result,
    })
    return nil
}

4. Flow-Driven Configuration

Everything should be configurable via edges/signals, not the settings panel:

Credentials mapped from upstream components (secret managers, vaults)
Runtime parameters from user input
Makes flows self-contained, portable, and version-controllable

Example Modules

Check out these example modules for reference:

common-module: Basic utilities (delay, switch, merge, signal)
http-module: HTTP client/server components
grpc-module: gRPC service components

CLI Reference

The SDK includes a CLI for running and building modules:

# Run module locally
./my-module run --name=my-module --version=1.0.0 --namespace=tinysystems

# Build (if custom build logic is needed)
./my-module build

# Get help
./my-module --help

License

This project is licensed under the Business Source License 1.1.

Key terms:

Free for production and non-production use
Not permitted: offering this software as a managed service, cloud service, or SaaS
Converts to Apache License 2.0 on January 11, 2031

For commercial licensing inquiries, contact Tiny Systems Limited.

Name		Name	Last commit message	Last commit date
Latest commit History 439 Commits
.github/workflows		.github/workflows
api/v1alpha1		api/v1alpha1
charts		charts
cli		cli
cmd		cmd
config		config
dist		dist
hack		hack
internal		internal
module		module
pkg		pkg
registry		registry
tools/build		tools/build
.dockerignore		.dockerignore
.gitignore		.gitignore
CLAUDE.md		CLAUDE.md
LICENSE		LICENSE
Makefile		Makefile
PROJECT		PROJECT
README.md		README.md
api-docs.md		api-docs.md
crd-api-doc-config.yaml		crd-api-doc-config.yaml
go.mod		go.mod
go.sum		go.sum
release.sh		release.sh

License

tiny-systems/module

Folders and files

Latest commit

History

Repository files navigation

TinySystems Module SDK

Overview

Key Features

Table of Contents

Architecture

Core Concepts

Custom Resource Definitions (CRDs)

Component System

Message Flow

Port System

Edge Configuration

SDK Components

/module/ - Core Interfaces

/pkg/resource/ - Resource Manager

/pkg/schema/ - JSON Schema Generator

/pkg/evaluator/ - Expression Evaluator

/pkg/metrics/ - Observability

/internal/scheduler/ - Message Routing

/internal/client/ - Client Pool

/cli/ - Command-Line Tools

Communication Patterns

Same-Module Communication

Cross-Module Communication

Retry Mechanism

Design Patterns

Scalability

Leader Election

One-off TinySignals

Project Structure

Getting Started

Helm charts

Running on the cluster

Undeploy controller

Contributing

How it works

Test It Out

Modifying the API definitions

Create new api

Module Development

Quick Start

Component Interface Deep Dive

GetInfo() - Component Metadata

Ports() - Define Component Ports

Handle() - Process Messages

Configuration Schemas

System Ports

_reconcile Port

_client Port

_settings Port

Error Handling

Transient Errors

Permanent Errors

Using the Resource Manager

Observability

Testing Components

Best Practices

CRITICAL: Handler Response Propagation

Component Naming Convention

Code Style

Component Design Principles

Example Modules

CLI Reference

License

About

Resources

License

Uh oh!

Stars

Watchers

Forks

Releases 28

Packages 0

Uh oh!

Contributors 2

`/module/` - Core Interfaces

`/pkg/resource/` - Resource Manager

`/pkg/schema/` - JSON Schema Generator

`/pkg/evaluator/` - Expression Evaluator

`/pkg/metrics/` - Observability

`/internal/scheduler/` - Message Routing

`/internal/client/` - Client Pool

`/cli/` - Command-Line Tools

`_reconcile` Port

`_client` Port

`_settings` Port

Packages