Skip to content

Improve DB Import/Export Scripts and Add Tests #166

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
pirog merged 7 commits into from
May 9, 2024
Merged

Improve DB Import/Export Scripts and Add Tests #166

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
3bd9243
Add test cases for sql import/export scripts
AaronFeledy May 9, 2024
12a5ece
healthchecks because scripts are faster than me
AaronFeledy May 9, 2024
ed53297
Use DROP/CREATE strategy for MySQL DB reset
xurizaemon Jul 21, 2023
6ddc1d1
Should work whether or not the database exists
AaronFeledy May 9, 2024
9e8a7f3
Turn down that racket!
AaronFeledy May 9, 2024
71f096a
loggin
AaronFeledy May 9, 2024
bfe781d
merge main into sql-import-drop
AaronFeledy May 9, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .github/workflows/pr-core-tests.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,7 @@ jobs:
- examples/orchestrator
- examples/plugin-commands
- examples/services
- examples/sql-helpers
- examples/tooling
node-version:
- "18"
Expand Down
3 changes: 3 additions & 0 deletions CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,8 @@
## {{ UNRELEASED_VERSION }} - [{{ UNRELEASED_DATE }}]({{ UNRELEASED_LINK }})

* Added `leia` tests for `sql-import.sh` and `sql-export.sh`
* Improved `sql-import.sh` to drop and recreate `mysql` and `mariadb` tables before importing

## v3.21.0-beta.19 - [May 9, 2024](https://github.com/lando/core/releases/tag/v3.21.0-beta.19)

### New Features
Expand Down
1 change: 1 addition & 0 deletions examples/sql-helpers/.gitignore
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
sqlhelpers
58 changes: 58 additions & 0 deletions examples/sql-helpers/.lando.sqlhelpers.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,58 @@
name: lando-sqlhelpers
services:
mariadb:
api: 3
type: lando
healthcheck: mysqladmin ping -h mariadb -u test -ptest
services:
image: bitnami/mariadb:10.4
command: /opt/bitnami/scripts/mariadb/entrypoint.sh /opt/bitnami/scripts/mariadb/run.sh
environment:
ALLOW_EMPTY_PASSWORD: yes
MARIADB_DATABASE: lando_test
MYSQL_DATABASE: lando_test
MARIADB_USER: test
MARIADB_PASSWORD: test

mysql57:
api: 3
type: lando
healthcheck: mysqladmin ping -h mysql57 -u test -ptest
services:
image: bitnami/mysql:5.7
command: /opt/bitnami/scripts/mysql/entrypoint.sh /opt/bitnami/scripts/mysql/run.sh
environment:
ALLOW_EMPTY_PASSWORD: yes
MYSQL_AUTHENTICATION_PLUGIN: mysql_native_password
MYSQL_DATABASE: lando_test
MYSQL_PASSWORD: test
MYSQL_USER: test

mysql80:
api: 3
type: lando
healthcheck: mysqladmin ping -h mysql80 -u test -ptest
services:
image: bitnami/mysql:8.0
command: /opt/bitnami/scripts/mysql/entrypoint.sh /opt/bitnami/scripts/mysql/run.sh
environment:
ALLOW_EMPTY_PASSWORD: yes
MYSQL_AUTHENTICATION_PLUGIN: caching_sha2_password
MYSQL_DATABASE: lando_test
MYSQL_PASSWORD: test
MYSQL_USER: test

postgres16:
api: 3
type: lando
healthcheck: pg_isready -h postgres16 -U postgres
services:
image: bitnami/postgresql:16
command: /opt/bitnami/scripts/postgresql/entrypoint.sh /opt/bitnami/scripts/postgresql/run.sh
environment:
ALLOW_EMPTY_PASSWORD: yes
POSTGRESQL_DATABASE: lando_test
POSTGRES_DB: lando_test

plugins:
"@lando/core": "../../.."
101 changes: 101 additions & 0 deletions examples/sql-helpers/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,101 @@
SQL Import/Export Example
============

This example exists primarily to test the `sql-import.sh` and `sql-export.sh` helper scripts.

Start up tests
--------------

```bash
# Should init and start a lando app
rm -rf sqlhelpers && mkdir -p sqlhelpers
cp -rf .lando.sqlhelpers.yml sqlhelpers/.lando.yml
cp -rf testdata1.sql sqlhelpers/testdata1.sql
cp -rf testdata2.sql sqlhelpers/testdata2.sql
cd sqlhelpers && lando start
```

Verification commands
---------------------

Run the following commands to verify things work as expected

```bash
# Should import test data into mariadb
cd sqlhelpers
lando ssh -s mariadb -c "/helpers/sql-import.sh testdata1.sql"
lando ssh -s mariadb -c "mysql -utest -ptest lando_test -e 'select * from lando_test'" | grep "lando_original"

# Should import test data into mysql57
cd sqlhelpers
lando ssh -s mysql57 -c "/helpers/sql-import.sh testdata1.sql"
lando ssh -s mysql57 -c "mysql -utest -ptest lando_test -e 'select * from lando_test'" | grep "lando_original"

# Should import test data into mysql80
cd sqlhelpers
lando ssh -s mysql80 -c "/helpers/sql-import.sh testdata1.sql"
lando ssh -s mysql80 -c "mysql -utest -ptest lando_test -e 'select * from lando_test'" | grep "lando_original"

# Should import test data into postgres16
cd sqlhelpers
lando ssh -s postgres16 -c "/helpers/sql-import.sh testdata1.sql"
lando ssh -s postgres16 -c "psql -U postgres -d lando_test -c 'select * from lando_test'" | grep "lando_original"

# Should export gzipped files from mariadb
cd sqlhelpers
lando ssh -s mariadb -c "/helpers/sql-export.sh mariadb_dump.sql" -u root
gzip -d mariadb_dump.sql.gz

# Should export gzipped files from mysql57
cd sqlhelpers
lando ssh -s mysql57 -c "/helpers/sql-export.sh mysql57_dump.sql" -u root
gzip -d mysql57_dump.sql.gz

# Should export gzipped files from mysql80
cd sqlhelpers
lando ssh -s mysql80 -c "/helpers/sql-export.sh mysql80_dump.sql" -u root
gzip -d mysql80_dump.sql.gz

# Should export gzipped files from postgres16
cd sqlhelpers
lando ssh -s postgres16 -c "/helpers/sql-export.sh postgres16_dump.sql" -u root
gzip -d postgres16_dump.sql.gz

# Should have the correct data in all exported files
cd sqlhelpers
grep "lando_original" mariadb_dump.sql
grep "lando_original" mysql57_dump.sql
grep "lando_original" mysql80_dump.sql
grep "lando_original" postgres16_dump.sql

# Should be able to replace data with testdata2 in mariadb
cd sqlhelpers
lando ssh -s mariadb -c "/helpers/sql-import.sh testdata2.sql"
lando ssh -s mariadb -c "mysql -utest -ptest lando_test -e 'select * from lando_test'" | grep -v "lando_original" | grep "lando_updated"

# Should be able to replace data with testdata2 in mysql57
cd sqlhelpers
lando ssh -s mysql57 -c "/helpers/sql-import.sh testdata2.sql"
lando ssh -s mysql57 -c "mysql -utest -ptest lando_test -e 'select * from lando_test'" | grep -v "lando_original" | grep "lando_updated"

# Should be able to replace data with testdata2 in mysql80
cd sqlhelpers
lando ssh -s mysql80 -c "/helpers/sql-import.sh testdata2.sql"
lando ssh -s mysql80 -c "mysql -utest -ptest lando_test -e 'select * from lando_test'" | grep -v "lando_original" | grep "lando_updated"

# Should be able to replace data with testdata2 in postgres16
cd sqlhelpers
lando ssh -s postgres16 -c "/helpers/sql-import.sh testdata2.sql"
lando ssh -s postgres16 -c "psql -U postgres -d lando_test -c 'select * from lando_test'" | grep -v "lando_original" | grep "lando_updated"
```

Destroy tests
-------------

```bash
# Should destroy sqlhelpers successfully
cd sqlhelpers && lando destroy -y

# Should poweroff
lando poweroff
```
6 changes: 6 additions & 0 deletions examples/sql-helpers/testdata1.sql
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
CREATE TABLE lando_test (
id SERIAL PRIMARY KEY,
column1 VARCHAR(255) DEFAULT NULL,
column2 VARCHAR(255) DEFAULT NULL
);
INSERT INTO lando_test VALUES (1, 'lando_original', 'lando_original');
6 changes: 6 additions & 0 deletions examples/sql-helpers/testdata2.sql
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
CREATE TABLE lando_test (
id SERIAL PRIMARY KEY,
column1 VARCHAR(255) DEFAULT NULL,
column2 VARCHAR(255) DEFAULT NULL
);
INSERT INTO lando_test VALUES (1, 'lando_updated', 'lando_updated');
33 changes: 10 additions & 23 deletions scripts/sql-import.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -95,34 +95,21 @@ echo "Preparing to import $FILE into database '$DATABASE' on service '$SERVICE'

# Wipe the database if set
if [ "$WIPE" == "true" ]; then
echo ""
echo "Emptying $DATABASE... "
lando_pink "\nEmptying $DATABASE... "
lando_yellow "NOTE: See the --no-wipe flag to avoid this step!"

# DO db specific wiping
if [[ ${POSTGRES_DB} != '' ]]; then
# Drop and recreate database
lando_yellow "\t\tDropping database ...\n\n"
psql postgresql://$USER@$HOST:$PORT/postgres -c "drop database $DATABASE"

lando_green "\t\tCreating database ...\n\n"
psql postgresql://$USER@$HOST:$PORT/postgres -c "create database $DATABASE"
psql postgresql://$USER@$HOST:$PORT/postgres -c "DROP DATABASE IF EXISTS $DATABASE" --quiet --echo-errors
psql postgresql://$USER@$HOST:$PORT/postgres -c "CREATE DATABASE $DATABASE" --quiet --echo-errors
else
# Build the SQL prefix
SQLSTART="mysql -h $HOST -P $PORT -u $USER ${LANDO_EXTRA_DB_IMPORT_ARGS} $DATABASE"

# Gather and destroy tables
TABLES=$($SQLSTART -e 'SHOW TABLES' | awk '{ print $1}' | grep -v '^Tables' || true)

# PURGE IT ALL! Drop views and tables as needed
for t in $TABLES; do
echo "Dropping $t from $DATABASE database..."
$SQLSTART <<-EOF
SET FOREIGN_KEY_CHECKS=0;
DROP VIEW IF EXISTS \`$t\`;
DROP TABLE IF EXISTS \`$t\`;
EOF
done
# Connection string
SQLSTART="mysql -h $HOST -P $PORT -u $USER ${LANDO_EXTRA_DB_IMPORT_ARGS}"

# Drop and recreate database
$SQLSTART -e "DROP DATABASE IF EXISTS ${DATABASE}"
$SQLSTART -e "CREATE DATABASE ${DATABASE}"
fi
fi

Expand Down Expand Up @@ -151,7 +138,7 @@ fi

# Build DB specific import command
if [[ ${POSTGRES_DB} != '' ]]; then
CMD="$CMD | psql postgresql://$USER@$HOST:$PORT/$DATABASE"
CMD="$CMD | psql --quiet --echo-errors postgresql://$USER@$HOST:$PORT/$DATABASE"
else
CMD="$CMD | mysql -h $HOST -P $PORT -u $USER ${LANDO_EXTRA_DB_IMPORT_ARGS} $DATABASE"
fi
Expand Down