TERRAFORM_TERRAGRUNT_POSTGRES

❯ Manages Lifecycle for records in Master Table

Table of Contents

• Overview
• Features
• Project Structure
  • Project Index
• Getting Started
  • Prerequisites
  • Installation
  • Usage
  • Testing
• Project Roadmap
• Contributing
• License
• Acknowledgments

---

Overview

❯ This project aims to automate master table record lifecycle via terraform/terragrunt.

---

Features

• For DDLs, make SQL Alchemy models are defined in models directory.
• Generate an alembic revision and upgrade to head/required revision
• Add records in the terragrunt.hcl file as per the required environment
• Add required terraform configuration as well as record management script in scripts. (1 example is already shown)
• The table records are saved in the terraform state files. If you change the records in the terragrunt.hcl, same changes will be reflected in the table as well.

---

Project Structure

└── terraform_terragrunt_postgres/
    ├── Dockerfile
    ├── LICENSE
    ├── README.md
    ├── alembic
    │   ├── README
    │   ├── env.py
    │   ├── script.py.mako
    │   └── versions
    ├── alembic.ini
    ├── docker-compose.yaml
    ├── entrypoint.sh
    ├── environments
    │   └── dev
    ├── models
    │   ├── base.py
    │   └── core
    ├── requirements.txt
    ├── root.hcl
    ├── set_env.sh
    └── terraform_source
        ├── main.tf
        ├── product.tf
        ├── providers.tf
        ├── scripts
        └── variables.tf

Project Index

TERRAFORM_TERRAGRUNT_POSTGRES/

__root__

set_env.sh	❯ Contains DB information. You can source the script via Jenkins or build tool of your choice. This is here just as an example.
alembic.ini	❯ Contains alembic configuration
root.hcl	❯ root.hcl file for terragrunt. Contains terraform source as well as state management configurations (state management configs are commented here but you can add/replace them a per your requirement
entrypoint.sh	❯ Contains steps to create the initial environment to further extend the code. It runs alembic migration provided with the project as well as terragrunt commands.
Dockerfile	❯ Sets up the containers to run the project
docker-compose.yaml	❯ Contains configuration for containers
requirements.txt	❯ packages required to run the project

models

base.py

❯ Declared Base class

core

product.py

❯ SQL Alchemy model file for Product table.

terraform_source

product.tf	❯ Contains terraform code for Product table
main.tf	❯ Main logic for terraform code.
providers.tf	❯ providers configuration for terraform.
variables.tf	❯ Declaration of required variables.

scripts

manage_product.py

❯ Python script for insert/update operation on the corresponding table.

alembic

script.py.mako	❯ Mako template file for alembic
env.py	❯ env.py file generated via alembic init

versions

92ec2f8e1da0_initial_migration.py

❯ Migration file generated via alembic

environments

dev

terragrunt.hcl

❯ terragrunt.hcl file that provides table records information to terraform modules. I have only included DEV environment here but you can further extend to different environments. Make sure to make changes to set_env.sh file according to your environment variables.

---

Getting Started

Prerequisites

Before getting started with terraform_terragrunt_postgres, ensure your runtime environment meets the following requirements:

• Programming Language: Python
• Package Manager: Pip
• Container Runtime: Docker

Installation

Install terraform_terragrunt_postgres using one of the following methods:

Build from source:

Clone the terraform_terragrunt_postgres repository:

❯ git clone https://github.com/rdrishabh38/terraform_terragrunt_postgres

Navigate to the project directory:

❯ cd terraform_terragrunt_postgres

Using docker  

❯ docker compose up -d

This will create two containers for you

❯ docker ps -a
CONTAINER ID   IMAGE                                            COMMAND                  CREATED             STATUS             PORTS                                         NAMES
2a95565b988c   postgres:15                                      "docker-entrypoint.s…"   About an hour ago   Up About an hour   0.0.0.0:5433->5432/tcp, [::]:5433->5432/tcp   postgres_master
b14d20ed0cde   terraform_terragrunt_postgres-terraform_runner   "/terraform_terragru…"   About an hour ago   Up About an hour                                                 terraform_runner

You will have 1 table in the postgres_master container with following records

❯ master_db=# select * from core.product;
 id |           product_key            |       create_timestamp        |       update_timestamp        |  record_effective_timestamp   |  record_expiration_timestamp  | current_record_indicator | product_code | product_name |        product_description
----+----------------------------------+-------------------------------+-------------------------------+-------------------------------+-------------------------------+--------------------------+--------------+--------------+------------------------------------
  1 | 8630830e8ca7dad26bfb051a6c64f275 | 2025-02-23 20:55:58.754735+00 | 2025-02-23 20:58:06.611174+00 | 2025-02-23 20:55:58.754735+00 | 2025-02-23 20:58:06.611174+00 | Y                        | PRODUCT_1    | product_1    | first product
  2 | 58477df3e645dd60ce96cd020ef3290f | 2025-02-23 20:55:58.755098+00 | 2025-02-23 20:55:58.755098+00 | 2025-02-23 20:55:58.755098+00 | 2099-12-31 23:59:59+00        | Y                        | PRODUCT_2    | product_2    | second product

If you want to make any changes to product table, login into the container and modify environments/dev/terragrunt.hcl file

❯ $ docker exec -it b14d20ed0cde /bin/bash

root@b14d20ed0cde:/terraform_terragrunt_postgres# cat environments/dev/terragrunt.hcl
# /terraform_terragrunt_postgres/envs/dev/root.hcl
include {
  path = find_in_parent_folders("root.hcl")
}

inputs = {
  product_data = {
    PRODUCT_1 = {
      product_name = "product_1",
      product_description = "first product modified description" -- here I have changed the project description 
    },
    PRODUCT_2 = {
      product_name = "product_2",
      product_description = "second product"
    }
  }
  working_dir = "/terraform_terragrunt_postgres/terraform_source"
}

Run the terragrunt plan and terragrunt apply commands from environments/dev directory:

❯ cd /terraform_terragrunt_postgres/environments/dev
terragrunt plan
terragrunt apply --auto-approve

This will expire the old record and make a new entry with updated description in to the product table:

❯ select * from core.product;
 id |           product_key            |       create_timestamp        |       update_timestamp        |  record_effective_timestamp   |  record_expiration_timestamp  | current_record_indicator | product_code | product_name |        product_description
----+----------------------------------+-------------------------------+-------------------------------+-------------------------------+-------------------------------+--------------------------+--------------+--------------+------------------------------------
  1 | 8630830e8ca7dad26bfb051a6c64f275 | 2025-02-23 20:55:58.754735+00 | 2025-02-23 20:58:06.611174+00 | 2025-02-23 20:55:58.754735+00 | 2025-02-23 20:58:06.611174+00 | N                        | PRODUCT_1    | product_1    | first product
  3 | 8630830e8ca7dad26bfb051a6c64f275 | 2025-02-23 20:58:06.748467+00 | 2025-02-23 20:58:06.748467+00 | 2025-02-23 20:58:06.748467+00 | 2099-12-31 23:59:59+00        | Y                        | PRODUCT_1    | product_1    | first product modified description
  2 | 58477df3e645dd60ce96cd020ef3290f | 2025-02-23 20:55:58.755098+00 | 2025-02-23 20:55:58.755098+00 | 2025-02-23 20:55:58.755098+00 | 2099-12-31 23:59:59+00        | Y                        | PRODUCT_2    | product_2    | second product

---

Project Roadmap

• Task 1: Implement core functionality of managing table records.
• Task 2: Implement an event_log table which will keep a track of when a change was done in the master tables.
• Task 3: Add more environment configurations - test and prod environments.

---

Contributing

• 💬 Join the Discussions: Share your insights, provide feedback, or ask questions.
• 🐛 Report Issues: Submit bugs found or log feature requests for the terraform_terragrunt_postgres project.
• 💡 Submit Pull Requests: Review open PRs, and submit your own PRs.

Contributing Guidelines

1. Fork the Repository: Start by forking the project repository to your github account.
2. Clone Locally: Clone the forked repository to your local machine using a git client.

   git clone https://github.com/rdrishabh38/terraform_terragrunt_postgres

3. Create a New Branch: Always work on a new branch, giving it a descriptive name.

   git checkout -b new-feature-x

4. Make Your Changes: Develop and test your changes locally.
5. Commit Your Changes: Commit with a clear message describing your updates.

   git commit -m 'Implemented new feature x.'

6. Push to github: Push the changes to your forked repository.

   git push origin new-feature-x

7. Submit a Pull Request: Create a PR against the original project repository. Clearly describe the changes and their motivations.
8. Review: Once your PR is reviewed and approved, it will be merged into the main branch. Congratulations on your contribution!

Contributor Graph

---

License

This project is protected under the GPL-3.0 license License. For more details, refer to the LICENSE file.

---

Acknowledgments

• inspired by real-world requirement of a use case in my organisation.
• Thanks to https://github.com/eli64s/README-AI for assisting in README.md file generation.

---