CI/CD Setup¶

DbtBuildKit includes reusable GitHub Actions workflows that automatically configure all CI/CD infrastructure. This feature eliminates the need to manually configure S3 buckets, IAM roles, and OIDC providers.

Overview¶

The CI/CD setup consists of three reusable workflows that can be called from any GitHub repository:

setup-cicd.yml: Automatically sets up CI/CD infrastructure (S3, IAM, OIDC)
ci.yml: Runs Terraform plan for review
cd.yml: Runs Terraform apply to provision infrastructure

How It Works¶

Initial Setup: The setup-cicd.yml workflow checks if infrastructure exists and creates it if needed
Reusable Workflows: The ci.yml and cd.yml workflows can be used in your CI/CD pipelines
Secure Authentication: Uses OIDC for GitHub → AWS authentication (no permanent credentials needed)

Quick Start¶

Step 1: Configure GitHub Secrets¶

Go to Settings > Secrets and variables > Actions in your repository and add:

Required Secrets: - AWS_ACCOUNT_ID: AWS Account ID (REQUIRED for all workflows) - AWS_ACCESS_KEY_ID: AWS Access Key ID (REQUIRED for setup-cicd.yml) - AWS_SECRET_ACCESS_KEY: AWS Secret Access Key (REQUIRED for setup-cicd.yml)

Optional Secrets (for setup-cicd.yml): - AWS_POLICY_ARN: ARN of custom IAM policy to attach to the created role (optional, uses default policy if not provided) - AWS_SECRET_TOKEN: AWS Session Token (optional, required for SSO/temporary credentials)

Note: - If AWS_POLICY_ARN is not provided, a default policy optimized for dbt projects CI/CD will be automatically created - If you’re using SSO/temporary credentials, you must also provide AWS_SECRET_TOKEN along with AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY

Step 2: Create Your CI/CD Workflow¶

Create a .github/workflows/cicd.yml file in your repository:

Important: You must include the required permissions at the workflow level. The reusable workflows require these permissions to function properly:

id-token: write - Required for OIDC authentication with AWS
contents: read - Required to checkout the repository
pull-requests: write - Required to comment on Pull Requests in CI workflow

name: CI/CD Pipeline

on:
  push:
    branches: [main]
  pull_request:
    branches: [main]

permissions:
  id-token: write
  contents: read
  pull-requests: write

jobs:
  setup-cicd:
    uses: dbtbuildkit/dbtbuildkit-infra/.github/workflows/setup-cicd.yml@main
    with:
      environment: 'dev'
      aws_region: 'us-east-1'
    secrets:
      AWS_ACCOUNT_ID: ${{ secrets.AWS_ACCOUNT_ID }}
      AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
      AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
      AWS_SECRET_TOKEN: ${{ secrets.AWS_SECRET_TOKEN }}
      AWS_POLICY_ARN: ${{ secrets.AWS_POLICY_ARN }}

  ci:
    needs: setup-cicd
    uses: dbtbuildkit/dbtbuildkit-infra/.github/workflows/ci.yml@main
    with:
      environment: 'dev'
      aws_region: 'us-east-1'
    secrets:
      AWS_ACCOUNT_ID: ${{ secrets.AWS_ACCOUNT_ID }}

  cd:
    needs: [setup-cicd, ci]
    if: github.ref == 'refs/heads/main'
    uses: dbtbuildkit/dbtbuildkit-infra/.github/workflows/cd.yml@main
    with:
      environment: 'dev'
      aws_region: 'us-east-1'
      auto_approve: true
    secrets:
      AWS_ACCOUNT_ID: ${{ secrets.AWS_ACCOUNT_ID }}

Note: Replace dbtbuildkit/dbtbuildkit-infra with your DbtBuildKit repository path.

Available Workflows¶

setup-cicd.yml¶

Automatically checks if CI/CD infrastructure exists and creates it if needed.

Inputs:

environment (optional): Environment (dev, stg, prd) - default: ‘dev’
aws_region (optional): AWS region - default: ‘us-east-1’
resource_name_prefix (optional): Prefix for resource names (bucket, role, policy) to avoid conflicts - default: ‘dbt-kit’

Secrets:

AWS_ACCOUNT_ID (required): AWS Account ID
AWS_ACCESS_KEY_ID (required): AWS Access Key ID
AWS_SECRET_ACCESS_KEY (required): AWS Secret Access Key
AWS_SECRET_TOKEN (optional): AWS Session Token (required for SSO/temporary credentials)
AWS_POLICY_ARN (optional): ARN of custom IAM policy to attach to the created role (uses default policy if not provided)

Outputs:

role_arn: ARN of the created IAM role

What it creates:

S3 bucket for Terraform state (with versioning enabled) following the pattern: {prefix}-{environment}-{region}-{aws_account_id}--tfstates (max 63 chars)
IAM Role for GitHub Actions following the pattern: {prefix}-github-actions-role-{aws_account_id}-{region} (max 64 chars)
IAM Policy (default policy optimized for dbt projects CI/CD, or custom policy if AWS_POLICY_ARN is provided) following the pattern: {prefix}-github-actions-policy-{aws_account_id}-{region} (max 128 chars)
OIDC Provider for GitHub authentication

Note: The default prefix is dbt-kit. You can customize it using the resource_name_prefix input to avoid conflicts with existing resources.

Important: GitHub organization names are case-sensitive in OIDC tokens. The workflow automatically preserves the original case of your organization name when creating the trust policy for the IAM role. This ensures that the OIDC authentication works correctly.

ci.yml¶

Runs terraform plan for review. Can comment on Pull Requests with the plan output.

Inputs:

environment (required): Environment (dev, stg, prd)
aws_region (optional): AWS region - default: ‘us-east-1’
terraform_directory (optional): Directory containing Terraform files - default: ‘.’
terraform_version (optional): Terraform version - default: ‘1.10.0’
resource_name_prefix (optional): Prefix for resource names (bucket, role, policy) to avoid conflicts - default: ‘dbt-kit’

Secrets:

AWS_ACCOUNT_ID (required): AWS Account ID (used to construct the IAM role ARN following the pattern: {prefix}-github-actions-role-{aws_account_id}-{region})

Features:

Generates Terraform plan
Uploads plan as artifact
Comments on Pull Requests (if triggered by PR)

cd.yml¶

Runs terraform apply to provision or update infrastructure.

Inputs:

environment (required): Environment (dev, stg, prd)
aws_region (optional): AWS region - default: ‘us-east-1’
terraform_directory (optional): Directory containing Terraform files - default: ‘.’
terraform_version (optional): Terraform version - default: ‘1.10.0’
auto_approve (optional): Auto approve apply - default: false
resource_name_prefix (optional): Prefix for resource names (bucket, role, policy) to avoid conflicts - default: ‘dbt-kit’

Secrets:

AWS_ACCOUNT_ID (required): AWS Account ID (used to construct the IAM role ARN following the pattern: {prefix}-github-actions-role-{aws_account_id}-{region})

Features:

Downloads plan artifact (if available)
Applies Terraform changes
Supports auto-approval for automated deployments

State Management¶

The workflows automatically configure Terraform state management:

S3 Backend: State is stored in S3 bucket
State Locking: Uses S3 lockfile (use_lockfile = true)
Encryption: State files are encrypted at rest
Versioning: S3 bucket versioning is enabled for state recovery

The bucket name follows the pattern: {prefix}-{environment}-{region}-{aws_account_id}--tfstates (max 63 characters)

State key follows the pattern: org={repo-owner}/repo={repo-name}/terraform.tfstate

The IAM role ARN is constructed using: arn:aws:iam::{aws_account_id}:role/{prefix}-github-actions-role-{aws_account_id}-{region} (max 64 characters for role name)

Note: Resource names are automatically truncated to respect AWS limits. The default prefix is dbt-kit, but you can customize it using the resource_name_prefix input in all workflows.

Backend Configuration:

You can use an empty backend configuration in your backend.tf file:

terraform {
  backend "s3" {}
}

The workflows automatically configure the backend during terraform init, so you don’t need to manually specify bucket, key, or region. This makes it easier to manage multiple environments and repositories without worrying about backend configuration.

Multi-Environment Support¶

You can use different environments (dev, stg, prd) by passing the environment input:

jobs:
  setup-dev:
    uses: dbtbuildkit/dbtbuildkit-infra/.github/workflows/setup-cicd.yml@main
    with:
      environment: 'dev'
      aws_region: 'us-east-1'

  setup-stg:
    uses: dbtbuildkit/dbtbuildkit-infra/.github/workflows/setup-cicd.yml@main
    with:
      environment: 'stg'
      aws_region: 'us-east-1'

Each environment will have its own: - S3 bucket for Terraform state (following the pattern: {prefix}-{environment}-{region}-{aws_account_id}--tfstates, max 63 chars) - IAM Role (shared across environments in the same region, following the pattern: {prefix}-github-actions-role-{aws_account_id}-{region}, max 64 chars) - IAM Policy (following the pattern: {prefix}-github-actions-policy-{aws_account_id}-{region}, max 128 chars) - The provided IAM Policy is attached to the role

Note: The default prefix is dbt-kit. All resource names are automatically truncated to respect AWS limits.

AWS Region Support¶

All workflows support choosing the AWS region through the aws_region parameter:

jobs:
  setup:
    uses: dbtbuildkit/dbtbuildkit-infra/.github/workflows/setup-cicd.yml@main
    with:
      environment: 'dev'
      aws_region: 'sa-east-1'  # São Paulo

The chosen region will be used to create all AWS resources (S3, IAM, etc.).

Security¶

OIDC Authentication: Uses OpenID Connect for secure GitHub → AWS authentication
No Permanent Credentials: No need to store AWS access keys in GitHub secrets
Least Privilege: IAM policies are scoped to necessary permissions
State Encryption: Terraform state files are encrypted at rest
Versioning: S3 bucket versioning enabled for state recovery

IAM Permissions¶

The default IAM policy includes permissions for:

Terraform state management (S3)
CodeBuild management
CodeConnections management
ECR management
IAM management (limited)
CloudWatch Logs
Secrets Manager
Data engines (Athena, Glue, Redshift, Lake Formation)
S3 artifacts
SSM Parameter Store

You can optionally provide a custom IAM policy ARN via the AWS_POLICY_ARN secret. If not provided, a default policy optimized for dbt projects CI/CD will be automatically created.

Examples¶

Basic Example¶

Simple CI/CD pipeline with plan and apply:

name: CI/CD

on:
  push:
    branches: [main]
  pull_request:
    branches: [main]

permissions:
  id-token: write
  contents: read
  pull-requests: write

jobs:
  setup-cicd:
    uses: dbtbuildkit/dbtbuildkit-infra/.github/workflows/setup-cicd.yml@main
    with:
      environment: 'dev'
    secrets:
      AWS_ACCOUNT_ID: ${{ secrets.AWS_ACCOUNT_ID }}
      AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
      AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
      AWS_POLICY_ARN: ${{ secrets.AWS_POLICY_ARN }}

  ci:
    needs: setup-cicd
    uses: dbtbuildkit/dbtbuildkit-infra/.github/workflows/ci.yml@main
    with:
      environment: 'dev'
    secrets:
      AWS_ACCOUNT_ID: ${{ secrets.AWS_ACCOUNT_ID }}

  cd:
    needs: [setup-cicd, ci]
    if: github.ref == 'refs/heads/main'
    uses: dbtbuildkit/dbtbuildkit-infra/.github/workflows/cd.yml@main
    with:
      environment: 'dev'
      auto_approve: true
    secrets:
      AWS_ACCOUNT_ID: ${{ secrets.AWS_ACCOUNT_ID }}

Multi-Environment Example¶

Using different environments:

jobs:
  setup-dev:
    uses: dbtbuildkit/dbtbuildkit-infra/.github/workflows/setup-cicd.yml@main
    with:
      environment: 'dev'
      aws_region: 'us-east-1'
    secrets:
      AWS_ACCOUNT_ID: ${{ secrets.AWS_ACCOUNT_ID }}
      AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
      AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
      AWS_POLICY_ARN: ${{ secrets.AWS_POLICY_ARN }}

  ci-dev:
    needs: setup-dev
    uses: dbtbuildkit/dbtbuildkit-infra/.github/workflows/ci.yml@main
    with:
      environment: 'dev'
      aws_region: 'us-east-1'
    secrets:
      AWS_ACCOUNT_ID: ${{ secrets.AWS_ACCOUNT_ID }}

  setup-prod:
    uses: dbtbuildkit/dbtbuildkit-infra/.github/workflows/setup-cicd.yml@main
    with:
      environment: 'prd'
      aws_region: 'us-east-1'
    secrets:
      AWS_ACCOUNT_ID: ${{ secrets.AWS_ACCOUNT_ID }}
      AWS_POLICY_ARN: ${{ secrets.AWS_POLICY_ARN }}

  cd-prod:
    needs: setup-prod
    uses: dbtbuildkit/dbtbuildkit-infra/.github/workflows/cd.yml@main
    with:
      environment: 'prd'
      aws_region: 'us-east-1'
      auto_approve: true
    secrets:
      AWS_ACCOUNT_ID: ${{ secrets.AWS_ACCOUNT_ID }}

Terraform in Subdirectory¶

If your Terraform files are in a subdirectory:

jobs:
  ci:
    uses: dbtbuildkit/dbtbuildkit-infra/.github/workflows/ci.yml@main
    with:
      environment: 'dev'
      terraform_directory: 'terraform'
    secrets:
      AWS_ACCOUNT_ID: ${{ secrets.AWS_ACCOUNT_ID }}

Troubleshooting¶

Workflow Not Found¶

If you see “Unable to find reusable workflow” errors:

Make sure you’ve replaced OWNER/REPO with the actual DbtBuildKit repository path
Verify the branch/tag exists (e.g., @main)
Check that the workflow file exists in the repository

Authentication Errors¶

If you see AWS authentication errors (e.g., “Not authorized to perform sts:AssumeRoleWithWebIdentity”):

Verify AWS_ACCOUNT_ID and AWS_POLICY_ARN secrets are configured correctly
Check that the IAM role (following the pattern {prefix}-github-actions-role-{aws_account_id}-{region}) has the necessary trust relationship with GitHub OIDC
Important: GitHub organization names are case-sensitive in OIDC tokens. The workflow automatically preserves the original case of your organization name when creating the trust policy. If you manually created the role, ensure the trust policy uses the exact case of your GitHub organization name.
Verify the trust policy condition matches: repo:{your-org}/*:* (note the format with two colons)
Ensure the workflow has permissions: id-token: write configured
If using SSO/OIDC, ensure your GitHub Actions environment has proper AWS credentials configured
If using access keys, verify AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY are correct

State Lock Errors¶

If you see state lock errors:

Check if another Terraform operation is running
Verify S3 bucket permissions
Check if lockfile exists in S3 and remove if stale

State Key Format¶

The state key follows the pattern: org={repo-owner}/repo={repo-name}/terraform.tfstate

This format helps organize state files by organization and repository, making it easier to manage multiple projects in the same S3 bucket.

Resource Naming¶

All resources use a prefix (default: dbt-kit) to avoid conflicts:

S3 Bucket: {prefix}-{environment}-{region}-{account}--tfstates (max 63 chars)
IAM Role: {prefix}-github-actions-role-{account}-{region} (max 64 chars)
IAM Policy: {prefix}-github-actions-policy-{account}-{region} (max 128 chars)

Resource names are automatically truncated to respect AWS limits. You can customize the prefix using the resource_name_prefix input in all workflows.

More Information¶

Example Workflow: See .github/workflows/example-user-workflow.yml for a complete example
GitHub Actions Docs: Reusable Workflows

DbtBuildKit Documentation - Home
Quick Start Guide - Quick Start Guide
Usage Examples - Usage Examples
DBT Module - DBT Module
DbtBuildKit Module - DbtBuildKit Module

CI/CD Setup¶

Overview¶

How It Works¶

Quick Start¶

Step 1: Configure GitHub Secrets¶

Step 2: Create Your CI/CD Workflow¶

Available Workflows¶

setup-cicd.yml¶

ci.yml¶

cd.yml¶

State Management¶

Multi-Environment Support¶

AWS Region Support¶

Security¶

IAM Permissions¶

Examples¶

Basic Example¶

Multi-Environment Example¶

Terraform in Subdirectory¶

Troubleshooting¶

Workflow Not Found¶

Authentication Errors¶

State Lock Errors¶

State Key Format¶

Resource Naming¶

More Information¶

Navigation¶