Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion apps/gameservers-service/go.mod
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,6 @@ go 1.25
require (
connectrpc.com/connect v1.19.1
github.com/acarl005/stripansi v0.0.0-20180116102854-5a71ef0e047d
github.com/docker/go-connections v0.6.0
github.com/joho/godotenv v1.5.1
github.com/moby/moby/api v1.52.0
github.com/moby/moby/client v0.2.1
Expand All @@ -23,6 +22,7 @@ require (
github.com/containerd/errdefs/pkg v0.3.0 // indirect
github.com/dgryski/go-rendezvous v0.0.0-20200823014737-9f7001d12a5f // indirect
github.com/distribution/reference v0.6.0 // indirect
github.com/docker/go-connections v0.6.0 // indirect
github.com/docker/go-units v0.5.0 // indirect
github.com/felixge/httpsnoop v1.0.4 // indirect
github.com/go-logr/logr v1.4.3 // indirect
Expand Down
48 changes: 48 additions & 0 deletions apps/sftp-service/Dockerfile
Original file line number Diff line number Diff line change
@@ -0,0 +1,48 @@
# Build stage for SFTP Service
FROM golang:1.25-alpine AS go-builder

# Accept build arguments for multi-arch support
ARG TARGETARCH=amd64
ARG TARGETOS=linux

# Install build dependencies
# Use --no-scripts to disable triggers and avoid QEMU emulation issues
RUN apk add --no-cache --no-scripts git make

# Set working directory
WORKDIR /build

# Enable Go build cache and module cache
ENV GOCACHE=/root/.cache/go-build
ENV GOMODCACHE=/root/go/pkg/mod

RUN --mount=type=bind,source=.,target=/src,rw \
--mount=type=cache,target=/root/go/pkg/mod \
--mount=type=cache,target=/root/.cache/go-build \
sh -c "export GOWORK=off && \
cd /src/apps/shared && go mod download && \
cd /src/apps/sftp-service && go mod download && \
CGO_ENABLED=0 GOOS=${TARGETOS} GOARCH=${TARGETARCH} \
GOWORK=off go build -trimpath -ldflags=\"-w -s -extldflags '-static'\" \
-o /build/sftp-service /src/apps/sftp-service"

# Final stage
FROM alpine:latest

# Install CA certificates, curl, and netcat for health checks
# Use --no-scripts to disable triggers and avoid QEMU emulation issues
RUN apk update && apk --no-cache --no-scripts add ca-certificates curl netcat-openbsd

WORKDIR /app

# Copy binary from builder
COPY --from=go-builder /build/sftp-service .

# Create directory for SFTP files and host key
RUN mkdir -p /var/lib/sftp

# Expose SFTP port (2222) and HTTP health port (3020)
EXPOSE 2222 3020

# Run the service
CMD ["./sftp-service"]
196 changes: 196 additions & 0 deletions apps/sftp-service/IMPLEMENTATION.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,196 @@
# SFTP Service Implementation Summary

## Overview

Successfully implemented a complete SFTP data plane microservice for the Obiente Cloud platform. The service provides secure SFTP file transfer capabilities with API key authentication, scoped permissions, comprehensive audit logging, and organization-based isolation.

## Key Features

### 1. Security
- **API Key Authentication**: No password-based auth; all authentication via API keys
- **SHA-256 Hashing**: API keys are hashed before storage and comparison
- **Scoped Permissions**: Granular control with `sftp:read`, `sftp:write`, `sftp:*`, and `sftp` scopes
- **Organization Isolation**: Files organized by org/user, preventing cross-org access
- **Path Traversal Protection**: All paths validated to stay within user's directory
- **No Symlinks**: Symlink operations disabled for security

### 2. Architecture
- **Microservice Pattern**: Follows existing patterns (audit-service, auth-service, etc.)
- **Graceful Shutdown**: Proper cleanup of SFTP and HTTP servers
- **Health Checks**: HTTP endpoint on port 3020 for monitoring
- **Database Integration**: PostgreSQL for API key storage
- **Audit Logging**: TimescaleDB for operation logs

### 3. Operations
- **Read Operations**: Download, list, stat (requires `sftp:read`)
- **Write Operations**: Upload, delete, mkdir, rename (requires `sftp:write`)
- **Wildcard Scopes**: `sftp:*` and `sftp` grant both read and write
- **Audit Trail**: All operations logged with user, org, path, and result

## Files Created

### Core Package (`apps/shared/pkg/sftp/`)
- `server.go`: SFTP server implementation with SSH integration
- `handler.go`: File operation handlers with permission checking

### Microservice (`apps/sftp-service/`)
- `main.go`: Service entry point with HTTP and SFTP servers
- `internal/service/auth.go`: API key validator and audit logger
- `Dockerfile`: Multi-stage build following existing patterns
- `README.md`: Comprehensive documentation
- `go.mod`: Dependency management

### Database (`apps/shared/pkg/database/`)
- `api_keys.go`: APIKey model with GORM annotations

### Infrastructure
- `migrations/001_create_api_keys_table.sql`: Database schema
- `scripts/create-api-key.sh`: Utility for creating API keys
- Updated `docker-compose.yml`: Local development config
- Updated `docker-compose.swarm.yml`: Production swarm config
- Updated `go.work`: Workspace configuration

## Configuration

### Environment Variables
- `SFTP_PORT`: SFTP server port (default: 2222)
- `SFTP_BASE_PATH`: Base directory for files (default: /var/lib/sftp)
- `SFTP_HOST_KEY_PATH`: SSH host key location (default: /var/lib/sftp/host_key)
- `PORT`: HTTP health check port (default: 3020)
- Standard database and auth variables from existing services

### Docker
- **Port 2222**: SFTP access
- **Port 3020**: HTTP health checks
- **Volume**: `sftp-data` for persistent storage
- **Networks**: `obiente-network` overlay

## API Key Scopes

| Scope | Read | Write | Description |
|-------|------|-------|-------------|
| `sftp:read` | ✅ | ❌ | Download and list files |
| `sftp:write` | ❌ | ✅ | Upload, delete, modify files |
| `sftp:*` | ✅ | ✅ | Full access |
| `sftp` | ✅ | ✅ | Full access |

## Directory Structure

Files are organized as:
```
/var/lib/sftp/
├── org-123/
│ ├── user-456/
│ │ ├── file1.txt
│ │ └── subdir/
│ └── user-789/
│ └── file2.txt
└── org-abc/
└── user-def/
└── file3.txt
```

## Usage Examples

### Creating an API Key
```bash
./apps/sftp-service/scripts/create-api-key.sh \
"My SFTP Key" \
user-123 \
org-456 \
"sftp:read,sftp:write"
```

### Connecting via SFTP
```bash
# Using command-line client
sftp -P 2222 user@hostname
# When prompted for password, enter your API key

# Using FileZilla
Host: sftp://hostname
Port: 2222
User: any_username
Password: your-api-key
```

### Testing Health
```bash
curl http://localhost:3020/health
```

## Audit Logging

All operations are logged to TimescaleDB with:
- User ID and Organization ID
- Operation type (upload, download, delete, etc.)
- File path
- Success/failure status
- Bytes transferred
- Timestamp

## Deployment

### Docker Compose (Development)
```bash
docker compose up -d sftp-service
```

### Docker Swarm (Production)
```bash
docker stack deploy -c docker-compose.swarm.yml obiente
```

## Security Considerations

1. **API Keys**:
- Stored as SHA-256 hashes
- Never logged in plain text
- Include expiration and revocation support

2. **File Isolation**:
- Each user restricted to their directory
- Path traversal attempts blocked
- No symlink support

3. **Audit Trail**:
- All operations logged
- Failed attempts recorded
- User and organization tracked

4. **Network Security**:
- SSH protocol for transport encryption
- Host key verification
- No password fallback

## Code Review Fixes

1. ✅ Replaced custom string splitting with `strings.Split`
2. ✅ Implemented SHA-256 hashing for API keys
3. ✅ Fixed wildcard scope permissions (`sftp:*` and `sftp` now grant both read/write)

## Testing Checklist

- [x] Service builds successfully
- [x] Code review completed and issues fixed
- [x] Security scan passed (no CodeQL issues)
- [ ] Manual testing with SFTP client
- [ ] API key authentication verification
- [ ] Permission scope enforcement testing
- [ ] Audit log verification
- [ ] Cross-org isolation testing

## Future Enhancements

1. **Rate Limiting**: Implement per-user/org rate limits
2. **Quota Management**: Enforce storage quotas per user/org
3. **Web UI**: Admin interface for API key management
4. **Metrics**: Prometheus metrics for SFTP operations
5. **WebDAV**: Add WebDAV support for web-based file access

## References

- SFTP Protocol: RFC 4251-4254
- Go SSH Package: golang.org/x/crypto/ssh
- Go SFTP Package: github.com/pkg/sftp
- Existing Microservices: audit-service, auth-service, etc.
137 changes: 137 additions & 0 deletions apps/sftp-service/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,137 @@
# SFTP Service

A secure SFTP data plane microservice with API key authentication, permission scoping, and comprehensive audit logging.

## Features

- **API Key Authentication**: Authenticate using API keys instead of passwords
- **Permission Scoping**: Separate read and write permissions (sftp:read, sftp:write)
- **User Isolation**: Each user has their own isolated directory (organized by org/user)
- **Audit Logging**: All operations are logged to the audit service
- **Secure by Design**: No symlinks, path traversal protection, permission enforcement

## Configuration

Environment variables:

- `SFTP_PORT`: SFTP server port (default: 2222)
- `SFTP_BASE_PATH`: Base directory for SFTP files (default: /var/lib/sftp)
- `SFTP_HOST_KEY_PATH`: Path to SSH host key (default: /var/lib/sftp/host_key)
- `PORT`: HTTP health check port (default: 3020)
- `DATABASE_URL`: PostgreSQL database URL
- `METRICS_DATABASE_URL`: TimescaleDB URL for audit logs
- `LOG_LEVEL`: Logging level (debug, info, warn, error)

## API Key Scopes

API keys must have one or more of these scopes:

- `sftp:read` - Allow reading/downloading files and listing directories
- `sftp:write` - Allow uploading, deleting, and modifying files
- `sftp:*` or `sftp` - Grant both read and write permissions

## Usage

### Creating an API Key

API keys must be created through the auth service or admin interface with the appropriate SFTP scopes.

Example:
```
Scopes: sftp:read,sftp:write
```

### Connecting via SFTP

```bash
# Using command-line SFTP client
sftp -P 2222 -o User=any_username -o IdentityFile=/path/to/key user@hostname

# When prompted for password, enter your API key

# Using FileZilla or other GUI clients
Host: sftp://hostname
Port: 2222
User: any_username (username doesn't matter, authentication is via API key)
Password: your-api-key
```

### Directory Structure

Files are organized by organization and user:

```
/var/lib/sftp/
├── org-123/
│ ├── user-456/
│ │ ├── file1.txt
│ │ └── subdir/
│ └── user-789/
│ └── file2.txt
└── org-abc/
└── user-def/
└── file3.txt
```

Each user can only access their own directory within their organization.

## Operations

All operations are audited and logged:

- **upload**: Upload a file
- **download**: Download a file
- **delete**: Delete a file or directory
- **mkdir**: Create a directory
- **rename**: Rename/move a file
- **list**: List directory contents
- **stat**: Get file information

## Security

- **Path Traversal Protection**: Users cannot escape their directory
- **No Symlinks**: Symlink creation and reading is disabled
- **Permission Enforcement**: Operations are checked against API key scopes
- **Audit Trail**: All operations are logged with user, org, and result
- **API Key Tracking**: Last used timestamp is updated on each connection

## Health Check

HTTP endpoint available at `http://localhost:3020/health`

Returns:
```json
{
"status": "healthy",
"service": "sftp-service",
"timestamp": "2024-01-17T20:00:00Z",
"details": {
"sftp_address": "0.0.0.0:2222",
"base_path": "/var/lib/sftp"
}
}
```

## Development

```bash
# Build
go build -o sftp-service

# Run
./sftp-service

# Test connection
sftp -P 2222 test@localhost
# Enter your API key when prompted for password
```

## Architecture

The service consists of three main components:

1. **SFTP Server** (`pkg/sftp/server.go`): Handles SSH/SFTP protocol
2. **Auth Validator** (`internal/service/auth.go`): Validates API keys against database
3. **Audit Logger** (`internal/service/auth.go`): Logs operations to TimescaleDB

The server uses the standard Go SSH and SFTP libraries with custom handlers for permission checking and audit logging.
Loading
Loading