Overview Databricks-Container-Volumes (DCV) is a local, content-addressed container artifact store built on top of Databricks Unity Catalog Volumes (UC Volumes). It provides a lightweight, OCI-compliant storage layer for container images, including MCP server images, without requiring a full remote registry. By leveraging UC Volumes as the persistent backend, DCV enables reproducible, auditable, and concurrency-safe storage and retrieval of container images across Databricks clusters as well as other remote clusters.
Key Features
- OCI-compliant storage: All images, layers, and manifests adhere to the Open Container Initiative specifications.
- MCP Server Image Storage: Entire MCP servers (as Docker images) can be persisted in UC Volumes, allowing local deployment or distribution across clusters.
- Content-addressed immutability: Images and layers are stored by digest (SHA256), guaranteeing once written, content cannot be modified.
- Local-first design: Uses UC Volumes for storage, providing persistent, shared access to images without relying on a networked registry.
- Thread-safe ingestion: Supports concurrent writing of layers, guaranteeing deterministic correctness even under parallel uploads.
- Manifest-driven images: Each image (including MCP servers) is represented by a manifest that references immutable layers, enabling deterministic builds and reproducible deployments.
- Auditability: The structure of manifests and digests allows verification of content integrity and change tracking.
Architecture & Workflow
- Image Sources: Local clients and remote MCP servers can be packaged as Docker images and pushed into DCV.
- Threaded Blob Ingest: Multiple worker threads upload and hash image layers concurrently into UC Volumes.
- Content-Addressed Storage: Layers are stored in
blobs/sha256/by digest; immutable once written. - Manifests & Index: Image manifests and
index.jsontrack image structure, tags, and versions, including MCP server images. - UC Volumes: Serves as the persistent, shared storage layer that holds all container images, including entire MCP servers, for local or cross-cluster deployment.
Use Cases
- Persisting prebuilt MCP server Docker images for local Databricks clusters.
- Sharing container images, including MCP servers, between multiple jobs, clusters, or DCV-enabled environments.
- Accelerating reproducible ML/ETL workflows using deterministic container environments.
- Enabling governance and auditing of MCP server deployments as container images.
- Serving as a foundation for local or private image distribution pipelines without a public registry.
Technical Notes
- Implements the OCI Image Layout on UC Volumes.
- Uses atomic file operations to ensure safety in concurrent environments.
- Supports extensible metadata for tagging and versioning images, including MCP server images.
- Fully compatible with OCI-aware tools for image verification and inspection.