VEuPathDB Dataset Installer Service (VDI)

Table of Contents

• Links
  • Compose Stack & Configuration
  • Documentation
  • Schema
• Development
  • Repository Structure
  • Building VDI Components
    • Docker Images
    • Source Compilation
  • Common Tasks
    • Release a New Version
    • Update Dependencies
    • Update Cache DB Schema
• License

Links

Compose Stack & Configuration

• https://github.com/VEuPathDB/vdi-compose-stack

Documentation

• HTTP API

• Full Stack Config

• Plugin Config

Schema

JSON schema files for use with editor validation & autocomplete when editing configuration files.

• Service Stack Configuration Schema

• Plugin-Specific Configuration Schema
  (a subset of the full config schema)

Development

Repository Structure

Component Source

• Gradle Build Conventions

• Shared Components

  • Config File JSON Schema

• Core VDI Service Components

  • REST Service

  • Dataset Processing Lanes

  • Background Processes

  • App DB Queries

  • Reconciler Core

• VDI Plugin Handler Server

• Internal Cache DB DDL

Building VDI Components

Docker Images

VDI Core Server Image

make build-image

Internal Cache DB Image

make build-db

Source Compilation

Regenerate RAML-Based Java Code

make raml-gen

Compile VDI Core Server

make build-core-server

Compile Plugin Server

make build-plugin-server

Common Tasks

Release a New Version

The core service and plugin server are versioned together, however they are presently built differently.

Plugin Server

When pushing up a new git tag, a GitHub workflow will automatically be triggered to build and release the plugin server. That workflow will then open pull requests on each of the individual plugin repositories to update their copies of the server to the newest version.

Core Server

The core server docker image build and push to DockerHub is handled in Jenkins.

Update Dependencies

Dependency version updates are automatically suggested once per week by GitHub’s Dependabot, however if manual updates are needed, the JVM dependencies for the full stack are declared in the dependency catalog.
(ref Gradle Version Catalogs)

Update Cache DB Schema

Relevant Locations

• Cache DB Schema Definitions

• Version Migration SQL

• Database JVM Code (Queries & Wrappers)

Making a Schema Change

1. Make the changes to the stack-db schema definition.

2. Add a new migration directory containing migration scripts to update existing database instances in the stack-db migrations.

3. Update the impacted queries, database managers, and POJO types in the core.lib.db.internal subproject.

Migrations: Additional Explanation

Physically, a migration is a directory full of queries used to apply the changes necessary for a target database schema version. The directories are named after the version that they will migrate the database schema to, e.g. a directory named 012 would be expected to migrate the database schema to version 12. The directories will be checked in ascending numeric order, only being used if the database schema version is less than the version indicated by the directory name.

❗	VDI does not update the recorded db schema version itself. The migration is expected to perform that update, preferably as its final step/statement.

The migration scripts are baked into the service jar as resources.

On startup, VDI will test the stack-internal cache-db to fetch the current db schema version. VDI will then iterate through the migration directories, executing the SQL scripts contained in any directory whose numeric name is greater than the current schema version.

🔥	VDI will not notice or apply migration scripts that have been altered in schema versions less than or equal to the current db schema version.

License

Copyright 2020 VEuPathDB

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0.

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.