Secrets Management

A secret is any sensitive credential that grants access to a resource: API keys, database passwords, TLS private keys, OAuth client secrets, and encryption keys. Poor secrets management is responsible for a large proportion of cloud breaches - secrets accidentally committed to Git, stored in plaintext environment variables, or embedded in container images are routinely harvested by automated scanners.

---

The Problem with Naive Secrets Handling

Secrets in Code (The Most Common Mistake)

# ❌ WRONG - hardcoded in source code
db_password = "SuperSecret123"
api_key = "sk-live-abcdefghijklmnop"

# These get committed to Git → discoverable by:
#   - Anyone with access to the repo (current or historical)
#   - GitHub secret scanning bots
#   - Automated scanners when repo goes public
#   - git log, even after "deletion" (history preserved)

# Even after removing the secret, it remains in Git history
git log --all --full-history -- config.py   # still shows old commits
git show <commit-hash>:config.py            # still shows the secret

Caution

Git history is forever. If a secret is committed, assume it is compromised. You must rotate the secret immediately - removing it from the current commit is not enough. Tools like git filter-repo can rewrite history, but if the repo is public or cloned anywhere, the secret should be considered exposed regardless.

Environment Variables - Better, But Not Safe Enough

# ✅ Better: use environment variables
export DB_PASSWORD="SuperSecret123"
python app.py

# ⚠️  But environment variables are still risky:
#   - Visible in /proc/PID/environ (any process on same host can read)
#   - Logged in CI/CD output if not careful
#   - Available to all child processes
#   - Often appear in error dumps, stack traces, debug logs
#   - Docker inspect exposes them: docker inspect <container>

# Check environment variables of a running process
cat /proc/$(pgrep nginx)/environ | tr '\0' '\n' | grep -i pass

# Docker exposes env vars in inspect output
docker inspect mycontainer | jq '.[].Config.Env'
# Shows all environment variables set in the container

---

.env Files - Hygiene Rules

.env files store environment variables for local development. They must never be committed.

# ✅ Correct .env hygiene
echo ".env" >> .gitignore          # immediately; before any secrets are added
echo ".env.*" >> .gitignore        # catch .env.local, .env.production, etc.
echo "*.pem" >> .gitignore         # private keys
echo "*.key" >> .gitignore

# .env file format
DB_HOST=localhost
DB_PORT=5432
DB_USER=appuser
DB_PASSWORD=dev_password_only_for_local
API_KEY=dev-key-xxxx

# .env.example - commit this instead (no real values)
DB_HOST=localhost
DB_PORT=5432
DB_USER=
DB_PASSWORD=
API_KEY=

# Check if .env files have been accidentally committed
git ls-files | grep -E "\.env$|\.env\."

# Search the entire Git history for secrets (truffleHog)
pip install truffleHog
trufflehog git file://. --since-commit HEAD~100

# Or with gitleaks (fast, configurable)
gitleaks detect --source . --report-path gitleaks-report.json
gitleaks git --log-opts="--all"   # scan entire git history

---

Secret Scanning - Automated Detection

GitHub Secret Scanning

GitHub automatically scans all public repos (and private repos in paid plans) for known secret formats:

# GitHub Advanced Security - enable in repo Settings
# Detects: AWS keys, GCP service account keys, GitHub tokens, Stripe keys,
#           Slack tokens, Twilio credentials, and 200+ partner patterns

# View detected secrets:
# GitHub repo → Security → Secret scanning alerts

Pre-commit Hooks

Stop secrets before they reach the repo:

# Install pre-commit framework
pip install pre-commit

# .pre-commit-config.yaml in repo root:
repos:
  - repo: https://github.com/gitleaks/gitleaks
    rev: v8.18.0
    hooks:
      - id: gitleaks

  - repo: https://github.com/Yelp/detect-secrets
    rev: v1.4.0
    hooks:
      - id: detect-secrets
        args: ['--baseline', '.secrets.baseline']

# Install hooks
pre-commit install

# Run manually
pre-commit run --all-files

---

HashiCorp Vault - The Standard for Secrets Management

Vault is a purpose-built secrets management tool that centralises storage, auditing, and access control for secrets.

Key capabilities:
  - Dynamic secrets (generate DB creds on-demand, auto-expire)
  - Secret leasing (every secret has a TTL; auto-revoked when expired)
  - Encryption as a service (encrypt data without exposing keys)
  - Full audit log of every secret read/write/lease
  - Fine-grained policies (who can read which secrets?)
  - Multiple auth methods (LDAP, Kubernetes, AWS IAM, GitHub, etc.)

Vault Setup and Basics

# Install Vault
curl -fsSL https://apt.releases.hashicorp.com/gpg | gpg --dearmor > /usr/share/keyrings/hashicorp.gpg
echo "deb [signed-by=/usr/share/keyrings/hashicorp.gpg] https://apt.releases.hashicorp.com $(lsb_release -cs) main" > /etc/apt/sources.list.d/hashicorp.list
apt update && apt install vault

# Start dev server (in-memory; for learning only)
vault server -dev
export VAULT_ADDR='http://127.0.0.1:8200'
export VAULT_TOKEN='root'   # dev mode root token

# Production: initialise and unseal
vault operator init            # generates unseal keys and initial root token
vault operator unseal          # unseal (requires 3 of 5 unseal keys by default)
vault login <root-token>

KV Secrets Engine (Key-Value Store)

# Enable KV secrets engine at path 'secret/'
vault secrets enable -path=secret kv-v2

# Write a secret
vault kv put secret/myapp/database \
  password="db_secret_here" \
  username="appuser"

# Read a secret
vault kv get secret/myapp/database
vault kv get -field=password secret/myapp/database   # just the value

# List secrets at a path
vault kv list secret/myapp/

# Update a specific field (creates new version)
vault kv patch secret/myapp/database password="new_password"

# Get a specific version
vault kv get -version=1 secret/myapp/database

# Delete (soft delete - versions kept)
vault kv delete secret/myapp/database

# Permanently destroy a version
vault kv destroy -versions=1 secret/myapp/database

# Retrieve secret in JSON (useful for scripting)
vault kv get -format=json secret/myapp/database | jq -r '.data.data.password'

Dynamic Secrets - Database

Dynamic secrets are generated on-demand and automatically expire. The application never knows a long-lived credential.

# Enable database secrets engine
vault secrets enable database

# Configure a PostgreSQL connection
vault write database/config/mypostgres \
  plugin_name=postgresql-database-plugin \
  allowed_roles="app-role" \
  connection_url="postgresql://{{username}}:{{password}}@localhost:5432/mydb" \
  username="vault_admin" \
  password="vault_admin_password"

# Create a role (defines what creds are created and their TTL)
vault write database/roles/app-role \
  db_name=mypostgres \
  creation_statements="CREATE ROLE \"{{name}}\" WITH LOGIN PASSWORD '{{password}}' VALID UNTIL '{{expiration}}'; GRANT SELECT ON ALL TABLES IN SCHEMA public TO \"{{name}}\";" \
  default_ttl="1h" \
  max_ttl="24h"

# Generate dynamic credentials (a new DB user is created each time)
vault read database/creds/app-role
# Returns: username=v-app-role-random123, password=A1b2C3d4, lease_duration=1h

# After 1 hour, Vault automatically revokes the credentials
# After 24 hours maximum, even a renewed credential is revoked

Vault Policies - Access Control

# Policies are written in HCL (HashiCorp Configuration Language)
cat > app-policy.hcl << 'EOF'
# Allow read-only access to myapp's secrets
path "secret/data/myapp/*" {
  capabilities = ["read"]
}

# Allow renewing tokens and leases
path "auth/token/renew-self" {
  capabilities = ["update"]
}

# Allow the app to generate dynamic DB creds
path "database/creds/app-role" {
  capabilities = ["read"]
}
EOF

# Write the policy
vault policy write myapp-policy app-policy.hcl

# Create a token with this policy
vault token create -policy=myapp-policy -ttl=8h

# List policies
vault policy list
vault policy read myapp-policy

Vault in Applications

# Python: read Vault secrets at runtime
import hvac
import os

client = hvac.Client(
    url=os.environ['VAULT_ADDR'],
    token=os.environ['VAULT_TOKEN']  # token from environment, not code
)

response = client.secrets.kv.v2.read_secret_version(
    path='myapp/database',
    mount_point='secret'
)
db_password = response['data']['data']['password']

# Shell: fetch secret in CI/CD script
DB_PASS=$(vault kv get -field=password secret/myapp/database)
export DB_PASS

# Or use the Vault agent (recommended for production)
# vault-agent automatically authenticates and refreshes tokens
# writes secrets to a temp file or environment variable

---

AWS Secrets Manager

For AWS-hosted workloads, Secrets Manager is the managed alternative to Vault.

# AWS CLI: store a secret
aws secretsmanager create-secret \
  --name "myapp/database/password" \
  --description "Production DB password" \
  --secret-string '{"username":"appuser","password":"db_password_here"}'

# Retrieve a secret
aws secretsmanager get-secret-value \
  --secret-id "myapp/database/password" \
  --query SecretString \
  --output text

# Retrieve specific field
aws secretsmanager get-secret-value \
  --secret-id "myapp/database/password" \
  --query SecretString \
  --output text | python3 -c "import sys,json; print(json.load(sys.stdin)['password'])"

# Enable automatic rotation (every 30 days)
aws secretsmanager rotate-secret \
  --secret-id "myapp/database/password" \
  --rotation-days 30

# List secrets
aws secretsmanager list-secrets --query 'SecretList[].Name'

# Python boto3: read secret at runtime (no hardcoded credentials)
import boto3
import json

def get_secret(secret_name):
    client = boto3.client('secretsmanager', region_name='us-east-1')
    # IAM role on the EC2/Lambda handles authentication - no keys in code
    response = client.get_secret_value(SecretId=secret_name)
    return json.loads(response['SecretString'])

secret = get_secret("myapp/database/password")
db_password = secret['password']

---

Secrets in CI/CD Pipelines

GitHub Actions

# Store secrets in GitHub: Settings → Secrets and variables → Actions
# Then reference them as:

name: Deploy
on: [push]
jobs:
  deploy:
    runs-on: ubuntu-latest
    steps:
      - name: Deploy to production
        env:
          DB_PASSWORD: ${{ secrets.DB_PASSWORD }}     # injected from GitHub Secrets
          API_KEY: ${{ secrets.PRODUCTION_API_KEY }}
        run: |
          # ⚠️ Never echo secrets - this leaks them to logs
          echo "Deploying..."          # OK
          # echo "$DB_PASSWORD"        # ❌ would leak to logs
          ./deploy.sh

      # Fetch from Vault in CI (using Vault GitHub auth)
      - name: Import secrets from Vault
        uses: hashicorp/vault-action@v2
        with:
          url: https://vault.example.com
          method: github
          githubToken: ${{ secrets.VAULT_GITHUB_TOKEN }}
          secrets: |
            secret/data/myapp/database password | DB_PASSWORD ;
            secret/data/myapp/api key | API_KEY

Common CI/CD Mistakes

# ❌ Secrets visible in runner environment
env | grep -i "secret\|key\|password\|token"  # any process can read these

# ❌ Secrets in artifact archives / build outputs
docker build --build-arg API_KEY=$API_KEY .    # embedded in image layer!
docker history myimage                          # reveals build args

# ✅ Correct: use build secrets (never stored in image)
docker build --secret id=api_key,src=./api_key.txt .
# In Dockerfile: RUN --mount=type=secret,id=api_key cat /run/secrets/api_key

# ✅ Correct: multi-stage build to strip secrets
FROM build AS builder
RUN --mount=type=secret,id=npm_token \
  NPM_TOKEN=$(cat /run/secrets/npm_token) npm install
FROM node:alpine
COPY --from=builder /app/node_modules ./node_modules  # secrets not copied

---

Comparison: Secrets Storage Options

Option	Security	Auditability	Ease	Use for
Hardcoded in code	❌ None	❌ None	✅ Easy	Never
Plaintext .env file	⚠️ Low	❌ None	✅ Easy	Local dev only
OS environment variables	⚠️ Medium	❌ Low	✅ Easy	Non-sensitive configs
CI/CD built-in secrets	✅ Good	✅ Medium	✅ Easy	CI/CD pipeline secrets
Cloud secrets manager (AWS/GCP/Azure)	✅✅ High	✅✅ High	✅ Medium	Cloud-native apps
HashiCorp Vault	✅✅ High	✅✅ High	⚠️ Complex	Multi-cloud, dynamic secrets
Hardware security module (HSM)	✅✅✅ Highest	✅✅ High	❌ Complex	Signing keys, root CA keys, PCI

---

Secret Rotation

Every secret should have a defined rotation schedule. Treat rotation as a normal operation, not an emergency.

# Audit: find long-lived AWS access keys
aws iam generate-credential-report
aws iam get-credential-report --query 'Content' --output text | base64 -d | \
  python3 -c "
import sys, csv
reader = csv.DictReader(sys.stdin)
for row in reader:
    if row.get('access_key_1_active') == 'true':
        print(f\"{row['user']}: key created {row['access_key_1_last_rotated']}\")
"

# Rotation best practices:
# - API keys: rotate every 90 days
# - Service account passwords: rotate every 30 days
# - TLS certs: rotate before expiry (Let's Encrypt: 90 day, auto-renewal)
# - GPG/SSH private keys: rotate when personnel changes or compromise suspected
# - Database passwords: rotate via dynamic secrets (then TTL is the rotation)

# Vault: manually revoke a lease (before TTL expires)
vault lease revoke database/creds/app-role/LEASE_ID
vault lease revoke -prefix database/creds/app-role/   # revoke all for this role