Skip to main content

CI/CD and DevOps Best Practices

Table of Contents​

  1. CI/CD Fundamentals
  2. GitHub Actions
  3. GitLab CI
  4. Jenkins
  5. ArgoCD - GitOps
  6. CI/CD Best Practices
  7. DevOps Best Practices
  8. Pipeline Design and Delivery Fundamentals
  9. GitHub Actions in Depth
  10. Supply Chain Security and Secrets
  11. Release Strategies, Rollback and GitOps
  12. Advanced Real-World CI/CD Scenarios

CI/CD Fundamentals​

CI/CD Pipeline Overview​

┌─────────────────────────────────────────────────────────────┐
│ CI/CD PIPELINE │
├─────────────────────────────────────────────────────────────┤
│ │
│ ┌─────────┐ ┌─────────┐ ┌─────────┐ ┌─────────┐ │
│ │ CODE │───▶│ BUILD │───▶│ TEST │───▶│ DEPLOY │ │
│ │ COMMIT │ │COMPILE │ │ UNIT │ │ STAGING │ │
│ └─────────┘ └─────────┘ └─────────┘ └────┬────┘ │
│ │ │
│ ▼ │
│ ┌─────────┐ ┌─────────┐ ┌─────────┐ ┌─────────┐ │
│ │ MONITOR │◀───│ VERIFY │◀───│ E2E │◀───│ PROD │ │
│ │ │ │SMOKE │ │ TESTS │ │ ROLLING │ │
│ └─────────┘ └─────────┘ └─────────┘ └─────────┘ │
│ │
└─────────────────────────────────────────────────────────────┘

Pipeline Stages:
1. Source → Code commit triggers pipeline
2. Build → Compile code, create artifacts
3. Test → Unit tests, integration tests
4. Deploy → Deploy to environments
5. Monitor → Verify deployment, rollback if needed

Pipeline Types​

TypeTriggerPurpose
Continuous Integration (CI)Every commitBuild, test, validate code
Continuous Delivery (CD)Manual or CI passDeploy to staging automatically
Continuous DeploymentAutomaticDeploy to production automatically

GitHub Actions​

Basic Workflow​

# .github/workflows/ci.yml
name: CI Pipeline

on:
push:
branches: [main, develop]
pull_request:
branches: [main]

env:
NODE_VERSION: "18"
PYTHON_VERSION: "3.12"

jobs:
# ============================================
# LINTING JOB
# ============================================
lint:
name: Lint
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4

- name: Set up Python
uses: actions/setup-python@v5
with:
python-version: ${{ env.PYTHON_VERSION }}

- name: Install dependencies
run: |
python -m pip install --upgrade pip
pip install flake8 black isort

- name: Run linters
run: |
flake8 . --max-line-length=100
black --check .
isort --check .

# ============================================
# UNIT TESTS JOB
# ============================================
test:
name: Unit Tests
runs-on: ubuntu-latest
needs: lint
services:
postgres:
image: postgres:16
env:
POSTGRES_USER: test
POSTGRES_PASSWORD: test
POSTGRES_DB: testdb
options: >-
--health-cmd pg_isready
--health-interval 10s
--health-timeout 5s
--health-retries 5
ports:
- 5432:5432

redis:
image: redis:7-alpine
options: >-
--health-cmd "redis-cli ping"
--health-interval 10s
--health-timeout 5s
--health-retries 5
ports:
- 6379:6379

steps:
- uses: actions/checkout@v4

- name: Set up Python
uses: actions/setup-python@v5
with:
python-version: ${{ env.PYTHON_VERSION }}
cache: "pip"

- name: Cache dependencies
uses: actions/cache@v4
with:
path: ~/.cache/pip
key: ${{ runner.os }}-pip-${{ hashFiles('requirements*.txt') }}

- name: Install dependencies
run: |
pip install -r requirements.txt
pip install pytest pytest-cov pytest-asyncio

- name: Run tests with coverage
run: |
pytest tests/ \
--cov=src \
--cov-report=xml \
--cov-report=html \
--junitxml=test-results.xml

- name: Upload coverage to Codecov
uses: codecov/codecov-action@v4
with:
files: ./coverage.xml
fail_ci_if_error: true

- name: Publish Test Results
uses: EnricoMi/publish-unit-test-result-action@v2
if: always()
with:
files: test-results.xml

# ============================================
# BUILD JOB
# ============================================
build:
name: Build Docker Image
runs-on: ubuntu-latest
needs: test
steps:
- uses: actions/checkout@v4

- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v3

- name: Login to Container Registry
uses: docker/login-action@v3
with:
registry: ghcr.io
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}

- name: Extract metadata
id: meta
uses: docker/metadata-action@v5
with:
images: ghcr.io/${{ github.repository }}
tags: |
type=sha,prefix=
type=ref,event=branch
type=semver,pattern={{version}}

- name: Build and Push
uses: docker/build-push-action@v5
with:
context: .
push: ${{ github.event_name != 'pull_request' }}
tags: ${{ steps.meta.outputs.tags }}
labels: ${{ steps.meta.outputs.labels }}
cache-from: type=gha
cache-to: type=gha,mode=max
build-args: |
VERSION=${{ github.sha }}
BUILD_DATE=${{ github.event.head_commit.timestamp }}

# ============================================
# SECURITY SCANS
# ============================================
security:
name: Security Scan
runs-on: ubuntu-latest
needs: build
permissions:
security-events: write
contents: read
steps:
- uses: actions/checkout@v4

- name: Run Trivy vulnerability scanner
uses: aquasecurity/trivy-action@master
with:
scan-type: "fs"
scan-ref: "."
format: "sarif"
output: "trivy-results.sarif"
severity: "CRITICAL,HIGH"

- name: Upload Trivy scan results
uses: github/codeql-action/upload-sarif@v2
with:
sarif_file: "trivy-results.sarif"

- name: Run Trivy in DB mode
uses: aquasecurity/trivy-action@master
with:
image-ref: "ghcr.io/${{ github.repository }}:${{ github.sha }}"
format: "table"
exit-code: "1"
severity: "CRITICAL"

Deploy to Kubernetes​

# .github/workflows/deploy.yml
name: Deploy to Kubernetes

on:
push:
tags:
- "v*"

env:
REGISTRY: ghcr.io
IMAGE_NAME: ${{ github.repository }}

jobs:
deploy:
name: Deploy to Production
runs-on: ubuntu-latest
environment: production
steps:
- uses: actions/checkout@v4

- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v3

- name: Login to Container Registry
uses: docker/login-action@v3
with:
registry: ${{ env.REGISTRY }}
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}

- name: Extract metadata
id: meta
uses: docker/metadata-action@v5
with:
images: ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}
tags: |
type=semver,pattern={{version}}

- name: Build and Push
uses: docker/build-push-action@v5
with:
push: true
tags: ${{ steps.meta.outputs.tags }}

- name: Configure kubectl
uses: azure/k8s-set-context@v3
with:
kubeconfig: ${{ secrets.KUBE_CONFIG }}

- name: Deploy to Kubernetes
run: |
# Update image tag in deployment
kubectl set image deployment/myapp \
myapp=${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}:${{ github.ref_name }}

# Wait for rollout
kubectl rollout status deployment/myapp --timeout=300s

- name: Verify deployment
run: |
kubectl rollout status deployment/myapp
kubectl get pods -l app=myapp

Matrix Strategy for Multiple Tests​

jobs:
test:
runs-on: ubuntu-latest
strategy:
matrix:
python-version: ["3.10", "3.11", "3.12"]
db: [postgres:14, postgres:15, postgres:16]
fail-fast: false

services:
postgres:
image: ${{ matrix.db }}
env:
POSTGRES_USER: test
POSTGRES_PASSWORD: test
POSTGRES_DB: testdb
ports:
- 5432:5432

steps:
- uses: actions/checkout@v4
- name: Set up Python ${{ matrix.python-version }}
uses: actions/setup-python@v5
with:
python-version: ${{ matrix.python-version }}
- name: Run tests
run: pytest tests/

GitLab CI​

.gitlab-ci.yml​

# ============================================
# GITLAB CI CONFIGURATION
# ============================================

stages:
- lint
- test
- build
- deploy

variables:
DOCKER_DRIVER: overlay2
DOCKER_TLS_CERTS_DIR: "/builds/$CI_PROJECT_PATH_SLUG/tmp"

# ============================================
# DOCKER IN DOCKER SETUP
# ============================================
.services:
- docker:24-dind

# ============================================
# LINT STAGE
# ============================================
lint:
stage: lint
image: python:3.12-slim
before_script:
- pip install flake8 black isort
script:
- flake8 . --max-line-length=100
- black --check .
- isort --check .
only:
- merge_requests
- main

# ============================================
# TEST STAGE
# ============================================
test:unit:
stage: test
image: python:3.12-slim
services:
- postgres:15
- redis:7
variables:
POSTGRES_DB: testdb
POSTGRES_USER: test
POSTGRES_PASSWORD: test
DATABASE_URL: postgresql://test:test@postgres:5432/testdb
REDIS_URL: redis://redis:6379
before_script:
- pip install -r requirements.txt
- pip install pytest pytest-cov pytest-asyncio
script:
- pytest tests/ --cov=src --cov-report=xml --cov-report=html
coverage: '/(?<=TOTAL)\s+\d+\s+\d+\s+(\d+\%?)/'
artifacts:
reports:
coverage_report:
coverage_format: cobertura
path: coverage.xml
paths:
- coverage/
expire_in: 1 week
only:
- merge_requests
- main

test:e2e:
stage: test
image: cypress/base:18
dependencies:
- build
before_script:
- npm ci
script:
- npm run cypress:run
artifacts:
when: always
paths:
- cypress/videos/
- cypress/screenshots/
reports:
junit:
- cypress/results/*.xml
only:
- merge_requests

# ============================================
# BUILD STAGE
# ============================================
build:
stage: build
image: docker:latest
services:
- docker:24-dind
before_script:
- docker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY
script:
- docker build -t $CI_REGISTRY_IMAGE:latest -t $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA .
- docker push $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA
- docker push $CI_REGISTRY_IMAGE:latest
only:
- main

build:beta:
stage: build
image: docker:latest
services:
- docker:24-dind
before_script:
- docker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY
script:
- docker build -t $CI_REGISTRY_IMAGE:beta-$CI_COMMIT_SHORT_SHA .
- docker push $CI_REGISTRY_IMAGE:beta-$CI_COMMIT_SHORT_SHA
only:
- develop
except:
- main

# ============================================
# DEPLOY STAGE
# ============================================
deploy:staging:
stage: deploy
image: bitnami/kubectl:latest
environment:
name: staging
url: https://staging.example.com
script:
- kubectl config use-context staging
- kubectl set image deployment/myapp myapp=$CI_REGISTRY_IMAGE:$CI_COMMIT_SHA
- kubectl rollout status deployment/myapp
only:
- develop

deploy:production:
stage: deploy
image: bitnami/kubectl:latest
environment:
name: production
url: https://example.com
when: manual
script:
- kubectl config use-context production
- kubectl set image deployment/myapp myapp=$CI_REGISTRY_IMAGE:$CI_COMMIT_SHA
- kubectl rollout status deployment/myapp
only:
- main
when: manual

Jenkins​

Jenkinsfile (Declarative Pipeline)​

// ============================================
// JENKINS DECLARATIVE PIPELINE
// ============================================

pipeline {
agent {
docker {
image 'python:3.12-slim'
args '-u root:root'
}
}

environment {
DOCKER_IMAGE = "myregistry/myapp"
DOCKER_REGISTRY_CREDENTIALS = 'docker-hub-credentials'
}

options {
buildDiscarder(logRotator(numToKeepStr: '10'))
disableConcurrentBuilds()
timeout(time: 30, unit: 'MINUTES')
timestamps()
}

stages {
stage('Checkout') {
steps {
checkout scm
}
}

stage('Lint') {
steps {
sh '''
pip install flake8 black isort
flake8 . --max-line-length=100
black --check .
isort --check .
'''
}
}

stage('Test') {
steps {
sh '''
pip install -r requirements.txt
pip install pytest pytest-cov
pytest tests/ --cov=src --cov-report=xml
'''
}
post {
always {
junit 'test-results/*.xml'
cobertura coberturaReportFile: 'coverage.xml'
}
}
}

stage('Build') {
steps {
script {
def imageTag = "${env.BUILD_NUMBER}-${env.GIT_COMMIT[0..7]}"
env.IMAGE_TAG = imageTag

sh """
docker build -t ${DOCKER_IMAGE}:${IMAGE_TAG} .
docker tag ${DOCKER_IMAGE}:${IMAGE_TAG} ${DOCKER_IMAGE}:latest
"""
}
}
}

stage('Security Scan') {
steps {
sh '''
pip install safety bandit
safety check
bandit -r . -f json -o bandit-report.json
'''
}
}

stage('Build Additional Platforms') {
steps {
script {
def platforms = ['linux/amd64', 'linux/arm64']
def images = []

platforms.each { platform ->
def m = platform.split('/')
def os = m[0]
def arch = m[1]

images << docker.image("${DOCKER_IMAGE}:${IMAGE_TAG}")
.withRun("-v /var/run/docker.sock:/var/run/docker.sock") {
sh """
docker buildx create --use
docker buildx build --platform ${platform} \
-t ${DOCKER_IMAGE}:${IMAGE_TAG}-${arch} \
--push .
"""
}
}
}
}
}

stage('Push to Registry') {
steps {
withCredentials([usernamePassword(
credentialsId: DOCKER_REGISTRY_CREDENTIALS,
usernameVariable: 'DOCKER_USERNAME',
passwordVariable: 'DOCKER_PASSWORD'
)]) {
sh '''
docker login -u $DOCKER_USERNAME -p $DOCKER_PASSWORD
docker push ${DOCKER_IMAGE}:${IMAGE_TAG}
docker push ${DOCKER_IMAGE}:latest
docker logout
'''
}
}
}

stage('Deploy to Staging') {
when {
branch 'develop'
}
steps {
sh '''
kubectl config use-context staging
kubectl set image deployment/myapp myapp=${DOCKER_IMAGE}:${IMAGE_TAG}
kubectl rollout status deployment/myapp
'''
}
}

stage('Deploy to Production') {
when {
branch 'main'
}
steps {
input message: 'Deploy to production?', ok: 'Deploy'
sh '''
kubectl config use-context production
kubectl set image deployment/myapp myapp=${DOCKER_IMAGE}:${IMAGE_TAG}
kubectl rollout status deployment/myapp
'''
}
}
}

post {
always {
cleanWs()
}
success {
echo 'Pipeline completed successfully!'
}
failure {
echo 'Pipeline failed!'
}
}
}

Jenkinsfile (Scripted Pipeline)​

// Scripted Pipeline Example
node('docker') {
stage('Checkout') {
checkout scm
}

stage('Build') {
def image = docker.build("${env.DOCKER_IMAGE}:${env.BUILD_NUMBER}")
}

stage('Test') {
image.inside {
sh 'pytest tests/'
}
}

stage('Push') {
docker.withRegistry('https://registry.example.com', 'docker-registry') {
image.push('latest')
image.push("${env.BUILD_NUMBER}")
}
}
}

ArgoCD - GitOps​

ArgoCD Application​

# ============================================
# ARGOCD APPLICATION
# ============================================
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: myapp-production
namespace: argocd
finalizers:
- resources-finalizer.argocd.argoproj.io
spec:
project: production
source:
repoURL: https://github.com/myorg/myapp.git
targetRevision: main
path: deploy/k8s/production
kustomize:
images:
- myregistry/myapp:v2.0.0
destination:
server: https://kubernetes.default.svc
namespace: production
syncPolicy:
automated:
prune: true
selfHeal: true
allowEmpty: false
syncOptions:
- CreateNamespace=true
- PruneLast=true
- RespectIgnoreDifferences=true
retry:
limit: 5
backoff:
duration: 5s
factor: 2
maxDuration: 3m
ignoreDifferences:
- group: apps
kind: Deployment
jsonPointers:
- /spec/replicas
revisionHistoryLimit: 10

ArgoCD AppProject​

# ============================================
# ARGOCD APPPROJECT
# ============================================
apiVersion: argoproj.io/v1alpha1
kind: AppProject
metadata:
name: production
namespace: argocd
spec:
description: Production environment
sourceRepos:
- https://github.com/myorg/myapp.git
- https://github.com/myorg/infra.git
destinations:
- server: https://kubernetes.default.svc
namespace: production
- server: https://kubernetes.default.svc
namespace: production-*
clusterResourceWhitelist:
- group: "*"
kind: "*"
namespaceResourceWhitelist:
- group: "*"
kind: "*"
roles:
- name: production-admin
description: Production admin role
groups:
- production-admins@example.com
policies:
- p, proj:production:production-admin, applications, *, production/*, allow

ArgoCD CLI Commands​

# ============================================
# ARGOCD COMMANDS
# ============================================

# Login
argocd login argocd.example.com --username admin --password $PASSWORD

# Create application
argocd app create myapp \
--repo https://github.com/org/repo.git \
--path deploy \
--dest-server https://kubernetes.default.svc \
--dest-namespace production

# Sync application
argocd app sync myapp
argocd app sync myapp --force

# View application status
argocd app get myapp
argocd app status myapp

# List applications
argocd app list
argocd app list -n argocd

# Watch application
argocd app watch myapp

# Manage resources
argocd app resources myapp
argocd app rollback myapp 1

# Update application
argocd app set myapp --auto-prune
argocd app set myapp --self-heal

# Delete application
argocd app delete myapp

# Sync with revision
argocd app sync myapp --revision v2.0.0

# Get logs
argocd app logs myapp --follow

CI/CD Best Practices​

┌─────────────────────────────────────────────────────────────┐
│ CI/CD BEST PRACTICES │
├─────────────────────────────────────────────────────────────┤
│ │
│ ✅ DO: │
│ ─────────────────────────────────────────────────────────│
│ • Commit small, frequent changes │
│ • Run tests on every commit │
│ • Keep build fast (< 10 minutes) │
│ • Use meaningful commit messages │
│ • Implement branch protection │
│ • Require code review before merge │
│ • Use semantic versioning for releases │
│ • Store secrets in secure vaults │
│ • Implement proper error handling │
│ • Use caching for dependencies │
│ • Run security scans early in pipeline │
│ │
│ ❌ DON'T: │
│ ─────────────────────────────────────────────────────────│
│ • Commit directly to main/production │
│ • Skip tests for "quick fixes" │
│ • Hardcode credentials in pipelines │
│ • Build same code multiple times │
│ • Deploy without testing in staging │
│ • Ignore flaky tests │
│ • Skip security scanning │
│ │
└─────────────────────────────────────────────────────────────┘

Secrets Management​

# GitHub Actions - Using secrets
- name: Deploy
env:
DATABASE_URL: ${{ secrets.DATABASE_URL }}
API_KEY: ${{ secrets.API_KEY }}

# Jenkins - Using credentials
withCredentials([
usernamePassword(
credentialsId: 'aws-credentials',
usernameVariable: 'AWS_ACCESS_KEY_ID',
passwordVariable: 'AWS_SECRET_ACCESS_KEY'
)
]) {
sh 'aws deploy ...'
}

# Using HashiCorp Vault in pipeline
- name: Fetch secrets from Vault
uses: hashicorp/vault-action@v2
with:
url: https://vault.example.com
method: aws
role: my-role
secrets: |
secret/data/myapp/database | DATABASE_URL = url
secret/data/myapp/api | API_KEY = key

DevOps Best Practices​

Infrastructure as Code​

# Terraform backend configuration
terraform {
backend "s3" {
bucket = "my-terraform-state"
key = "prod/terraform.tfstate"
region = "us-east-1"
dynamodb_table = "terraform-locks"
encrypt = true
}

required_providers {
aws = {
source = "hashicorp/aws"
version = "~> 5.0"
}
kubernetes = {
source = "hashicorp/kubernetes"
version = "~> 2.0"
}
}
}

# Terragrunt for DRY configurations
# terragrunt.hcl
locals {
environment = basename(dirname(get_terragratch_dir()))
}

generate "provider" {
path = "provider.tf"
if_exists = "overwrite"
contents = <<EOF
provider "aws" {
region = var.aws_region
default_tags {
tags = {
Environment = local.environment
ManagedBy = "Terraform"
}
}
}
EOF
}

inputs = {
aws_region = "us-east-1"
environment = local.environment
}

Monitoring Integration​

# Prometheus metrics endpoint in CI
- name: Run tests with coverage
run: |
pytest tests/ \
--cov=src \
--cov-report=xml \
--cov-prometheus

# Prometheus operator ServiceMonitor
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
name: myapp
labels:
release: prometheus
spec:
selector:
matchLabels:
app: myapp
endpoints:
- port: metrics
interval: 15s
path: /metrics
namespaceSelector:
matchNames:
- production

Complete GitHub Actions with Monitoring​

name: Complete CI/CD Pipeline

on:
push:
branches: [main, develop]
pull_request:
branches: [main]

env:
REGISTRY: ghcr.io
IMAGE_NAME: ${{ github.repository }}

jobs:
# ... (lint, test, build jobs)

deploy-staging:
name: Deploy to Staging
needs: build
runs-on: ubuntu-latest
if: github.ref == 'refs/heads/develop'
environment: staging
steps:
- uses: actions/checkout@v4

- name: Deploy to staging
run: |
kubectl apply -f k8s/staging/
kubectl set image deployment/myapp myapp=${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}:${{ github.sha }}
kubectl rollout status deployment/myapp -n staging --timeout=300s

- name: Run smoke tests
run: |
sleep 30
curl -f https://staging.example.com/health || exit 1

- name: Notify Slack
if: always()
uses: slackapi/slack-github-action@v1.25
with:
channel-id: "staging-deploys"
payload: |
{
"text": "Staging deployment ${{ job.status }}",
"blocks": [
{
"type": "section",
"text": {
"type": "mrkdwn",
"text": "*Staging Deployment*\nStatus: ${{ job.status }}\nCommit: ${{ github.sha }}"
}
}
]
}
env:
SLACK_BOT_TOKEN: ${{ secrets.SLACK_BOT_TOKEN }}

deploy-production:
name: Deploy to Production
needs: [build, deploy-staging]
runs-on: ubuntu-latest
if: github.ref == 'refs/heads/main'
environment: production
steps:
- uses: actions/checkout@v4

- name: Deploy to production
run: |
kubectl apply -f k8s/production/
kubectl set image deployment/myapp myapp=${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}:${{ github.sha }}
kubectl rollout status deployment/myapp -n production --timeout=300s

- name: Verify deployment
run: |
sleep 60
kubectl get pods -n production
kubectl rollout status deployment/myapp -n production

- name: Create GitHub Release
if: startsWith(github.ref, 'refs/tags/')
uses: actions/create-release@v1
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
with:
tag_name: ${{ github.ref }}
release_name: Release ${{ github.ref }}
draft: false
prerelease: false

- name: Notify Slack on Production
if: always()
uses: slackapi/slack-github-action@v1.25
with:
channel-id: "production-deploys"
payload: |
{
"text": "Production deployment ${{ job.status }}",
"blocks": [
{
"type": "section",
"text": {
"type": "mrkdwn",
"text": "*Production Deployment*\nStatus: ${{ job.status }}\nCommit: ${{ github.sha }}"
}
}
]
}
env:
SLACK_BOT_TOKEN: ${{ secrets.SLACK_BOT_TOKEN }}

Interview Questions​

Q1: What is the difference between CI and CD?​

  • CI (Continuous Integration): Automatically build and test every code change
  • CD (Continuous Delivery): Every change that passes CI is automatically built, tested and deployed to staging, so it is always releasable; the production release is a manual, one-click decision
  • Continuous Deployment: Automatically deploy every change to production

Q2: How do you handle secrets in CI/CD?​

  • Use secret management tools (HashiCorp Vault, AWS Secrets Manager)
  • Never commit secrets to version control
  • Use environment variables injected by CI/CD platform
  • Rotate secrets regularly

Q3: What is GitOps?​

  • Infrastructure/deployments managed through Git
  • ArgoCD or Flux watches Git repo for changes
  • Changes in Git automatically deployed to cluster
  • Provides audit trail and easy rollback

Q4: How do you rollback a deployment?​

  • GitOps: Revert Git commit, ArgoCD syncs automatically
  • kubectl: kubectl rollout undo deployment/myapp
  • Helm: helm rollback my-release
  • GitHub Actions: Re-trigger with previous image tag

Q5: How do you keep build times fast?​

  • Use caching for dependencies
  • Run jobs in parallel where possible
  • Use lightweight base images
  • Run fastest tests first (fail fast)
  • Split tests across parallel jobs

Q6: What is trunk-based development?​

All developers merge small changes to a single shared main branch frequently (daily+), using feature flags to hide incomplete work. It reduces merge conflicts, enables continuous deployment, and avoids long-lived branches. Alternative: git-flow/integration branches for stricter release control.

Q7: What are feature flags and how do they help CD?​

Feature flags gate code paths at runtime without redeployment. You can ship code dark, roll out gradually (canary %), instant rollback by flipping a flag, and run A/B tests. Tools: LaunchDarkly, Flagsmith, custom config. Use short-lived flags; remove them after full rollout to avoid flag debt.

Q8: What is the difference between blue-green and canary deployment?​

Blue-green: run two full environments (blue=old, green=new); switch traffic atomically, rollback = switch back. Canary: send a small % of traffic to the new version, monitor, then increase. Blue-green is simple and fast rollback but doubles capacity; canary is finer-grained risk control.

Q9: What are ephemeral environments?​

Preview environments created per pull request that mirror production (usually lightweight/limited), destroyed after merge. They catch integration issues early without waiting for staging. Key: fast provisioning via IaC, data anonymization, and automated teardown to control cost.

Q10: How do you make CI fast?​

Parallelize jobs (matrix), cache dependencies (package caches), warm Docker layer caches, use lighter images, skip unchanged paths (path filters), and split slow test suites (test sharding). Reduce from "slow full suite" to "fast affected-tests-only" per commit.

Q11: What is shift-left security in CI?​

Move security checks earlier in the pipeline: secret scanning, SAST, dependency scanning, and container scanning run on every PR/commit instead of only pre-release. Fail fast on critical findings; keep findings near the developer who introduced them.

Q12: What is a CHANGELOG/release automation best practice?​

Generate release notes from conventional commits (feat/fix/breaking) automatically with tools like semantic-release. Tag versions with semver: MAJOR for breaking, MINOR for features, PATCH for fixes. This gives auditable, machine-parsed releases.

Q13: What is the difference between GitHub Actions and a self-hosted CI?​

GitHub Actions: hosted runners, YAML workflows, marketplace ecosystem, simple for most repos. Self-hosted (Jenkins, self-managed GitLab, Buildkite agents): full control over runners, secrets, and hardware; needed for special GPUs, on-prem compliance, or custom caching. Cost/complexity trade-off.

Q14: What is artifact promotion?​

Building once per commit, then promoting the same immutable artifact through envs (dev → staging → prod), never rebuilding per environment. Guarantees what you tested is what you ship. Use signed container images and a promotion registry policy.

Q15: How do you handle database migrations in CD?​

Run migrations as a separate, ordered pipeline stage before/around app deploy: expand-and-contract pattern (additive first, backfill, then cutover, then drop old columns), never auto-run destructive DDL in prod, keep migrations idempotent and versioned, and test them against a copy of prod data.

Q16: What is GitOps for CI/CD delivery?​

Git as the single source of truth: a GitOps operator (Argo CD/Flux) reconciles cluster state to match the repo. Deploys are merges (reviewable, audit-logged), drift is auto-corrected, rollback = revert commit. Ideal for Kubernetes; applies to infra with tools like Terraform Cloud.

Q17: What is a build pipeline caching strategy?​

Cache layers that rarely change: pull base images first, install deps before copying source, cache pip cache/npm cache/Gradle/Maven caches keyed by lockfile hash, and bust only when dependencies change. Cache invalidation design is the difference between 1-minute and 10-minute builds.

Q18: What is the difference between continuous delivery and continuous deployment?​

Continuous delivery: every change is releasable; releasing to prod is a manual approval. Continuous deployment: every change that passes the pipeline goes to prod automatically. CDel gives control for regulated industries; CDeploy maximizes speed when test coverage and monitoring are strong.

Q19: How do you test in CI for production parity?​

Run integration tests, contract tests, and E2E tests against production-like services (Testcontainers, ephemeral preview envs), use the same artifact that will deploy, and enforce environment parity for dependency versions and config. The more parity, the fewer "works on CI, breaks in prod" surprises.

Q20: What is a pipeline as code and why adopt it?​

Define pipelines as YAML/DSL in the repo next to app code (GitHub Actions, GitLab CI, Buildkite, Tekton). Benefits: reviewable diffs, versioned with app code, reusable templates, no click-ops config drift. Pipeline-as-code is the minimum bar for GitOps-compatible delivery.


Senior / Principal — Real World CI/CD Scenarios​

Q21: Flaky CI pipeline — passes on retry​

Level: Senior

Scenario: A CI pipeline fails randomly (flaky). Developers re-run until green. Merge queue blocked. Trust in CI erodes.

Impact: Wasted CI minutes. Delayed releases. Merged bugs hidden by retries.

Root Cause: Test ordering dependence, shared state, network flakiness, resource contention, non-deterministic tests, or insufficient retry isolation.

Simple explanation: A machine works when you kick it but breaks again later. You kick it until it works, then ship it broken.

Technical Answer:

  1. Reproduce locally with --repeat and random order.
  2. Isolate tests: function-scoped fixtures, clean DB, unique data, mock external services.
  3. Quarantine flaky tests (--reruns with tracking) and fix before re-enabling.
  4. Use matrix builds with fixed seeds; pin dependency versions (lockfile).
  5. Add retry only for known transient infra issues, never for logic tests.

Code/Syntax Justification:

# GitHub Actions: retry the flaky infra step, never the test step
- name: Install dependencies (network can be flaky)
uses: nick-fields/retry@v3
with:
max_attempts: 3
timeout_minutes: 10
command: pip install -r requirements.txt
- name: Run tests (no retry, a failure is a real signal)
run: pytest

Workflow Diagram:

CI fails randomly
|
v
Re-run until green (BAD)
|
v
Reproduce with --repeat + random order
|
v
Fix root cause: isolation, mocks, seeds
|
v
CI deterministic: pass every run

🔍 Behind the Question: Flaky test detection, isolation, reproducibility, quarantine, retry policy.

❌ The Red Flags: Re-run-until-green; retrying all failures indiscriminately; no flaky test tracking.


Q22: Deployment rollback — bad release in production​

Level: Principal

Scenario: A deploy introduces a breaking change. Error rate spikes to 30%. The team needs to roll back within 5 minutes (SLA). Rolling forward (fix + redeploy) takes 20 minutes.

Impact: Revenue loss. User impact. SLA breach.

Root Cause: No rollback plan. No blue/green or canary. No feature flags. No artifact immutability.

Simple explanation: You install a new engine in a car while driving. It breaks. You need to swap back to the old engine fast.

Technical Answer:

  1. Immutable artifacts: deploy the same Docker image tested in staging.
  2. Blue/green deployment: switch traffic back to old version instantly.
  3. Feature flags: toggle off the breaking feature without redeploy.
  4. Canary: deploy to 5% first, monitor error rate, then promote or rollback.
  5. Rollback runbook: documented, tested, one-command rollback.

Code/Syntax Justification:

# Kubernetes: roll back to the previous revision
kubectl rollout undo deployment/my-app
# Or switch the feature off with a flag (LaunchDarkly Python SDK)
show_new_checkout = ld_client.variation("new-checkout-flow", context, False)

Workflow Diagram:

Bad release deployed
|
v
Error rate spikes
|
v
Rollback (blue/green: switch to old version) OR Fix forward (20 min)
| |
v v
Service restored in < 5 min Service restored in 20 min

🔍 Behind the Question: Rollback strategy, blue/green, canary, feature flags, immutable artifacts, runbook.

❌ The Red Flags: No rollback plan; mutable artifacts; fix-forward-only; no feature flags.


Pipeline Design and Delivery Fundamentals​

Q23: What stages does a production CI/CD pipeline have, and why in that order?​

A good pipeline is ordered so the cheapest, fastest, most likely to fail checks run first and the expensive, slow, environment-touching steps run last:

  1. Static checks (seconds): formatting, linting, type checks, secret scanning.
  2. Unit tests (1-3 min): fast, no network, run in parallel.
  3. Build the artifact once (container image, wheel, JAR) and record its immutable identifier (image digest, checksum).
  4. Security scans on the artifact: dependency/CVE scan, SBOM generation, signing.
  5. Integration/contract tests against the built artifact (Testcontainers, ephemeral environment).
  6. Deploy to staging with the same artifact, then smoke tests.
  7. Production deploy (approval gate for continuous delivery, automatic for continuous deployment), progressive rollout, automated verification against SLOs.

The principle is "fail fast, build once, verify the thing you ship". Anything that touches a shared environment comes after the code is known to be good, and every later stage consumes the artifact produced in stage 3 instead of rebuilding.

lint/secrets -> unit tests -> build (digest) -> scan + sign -> integration -> staging -> prod (canary -> full)
seconds minutes minutes minutes minutes minutes minutes-hours

Q24: Why deploy container images by digest instead of by tag?​

Tags are mutable pointers: myapp:1.4.2 or myapp:latest can be re-pushed to point at different bytes, and a node that cached an older image under the same tag may run something different from what was tested. A digest (myapp@sha256:...) is the content hash of the image manifest and can never change.

Deploying by digest gives you:

  • Reproducibility: the bytes tested in staging are exactly the bytes in production.
  • Reliable rollback: the previous digest still identifies the previous build.
  • Supply-chain verification: signatures and attestations are attached to digests, not tags.
jobs:
build:
runs-on: ubuntu-latest
permissions:
contents: read
packages: write
outputs:
digest: ${{ steps.push.outputs.digest }}
steps:
- uses: actions/checkout@v4
- uses: docker/setup-buildx-action@v3
- uses: docker/login-action@v3
with:
registry: ghcr.io
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}
- id: push
uses: docker/build-push-action@v6
with:
push: true
tags: ghcr.io/${{ github.repository }}:${{ github.sha }}

deploy-staging:
needs: build
runs-on: ubuntu-latest
steps:
- run: echo "Deploying ghcr.io/${{ github.repository }}@${{ needs.build.outputs.digest }}"

Tags remain useful for humans (v1.4.2, the commit SHA), but the deploy manifest should pin the digest.

Q25: What is a merge queue and what problem does it solve?​

On a busy repository, two PRs can each pass CI against an older main and still break main when combined (a "semantic merge conflict"). Requiring "branch must be up to date" fixes this but forces serial rebases and re-runs.

A merge queue (GitHub merge queue, GitLab merge trains, Mergify, Bors-style tools) takes approved PRs, builds a temporary branch of main + PR1 (+ PR2 ...), runs CI on that combination, and only fast-forwards main if it passes. Failing PRs are ejected and the rest are re-tested.

In GitHub Actions, workflows required by the queue must listen to the merge_group event:

on:
pull_request:
merge_group:

Trade-offs: more CI minutes (each queue entry is tested), and a slow pipeline makes the queue a bottleneck, so it pairs naturally with fast, sharded CI.

Q26: How does a Kubernetes rolling update work and what makes it safe?​

A Deployment with strategy.type: RollingUpdate replaces pods gradually. Two knobs control the pace:

  • maxSurge: how many extra pods above replicas may exist during the rollout.
  • maxUnavailable: how many pods below replicas may be unavailable.

A rolling update is only as safe as the readiness probe: Kubernetes considers a new pod available when it is Ready (for at least minReadySeconds), and only then terminates old pods. Without a meaningful readiness probe, broken pods are counted as healthy and the rollout completes anyway.

apiVersion: apps/v1
kind: Deployment
metadata:
name: orders-api
spec:
replicas: 6
minReadySeconds: 10
progressDeadlineSeconds: 300
strategy:
type: RollingUpdate
rollingUpdate:
maxSurge: 25%
maxUnavailable: 0
selector:
matchLabels:
app: orders-api
template:
metadata:
labels:
app: orders-api
spec:
containers:
- name: app
image: ghcr.io/acme/orders-api@sha256:3f1c0e8a9b7d6c5e4f3a2b1c0d9e8f7a6b5c4d3e2f1a0b9c8d7e6f5a4b3c2d1e
ports:
- containerPort: 8080
readinessProbe:
httpGet:
path: /healthz/ready
port: 8080
periodSeconds: 5
failureThreshold: 3

In the pipeline, block on the result so a stuck rollout fails the job:

kubectl rollout status deployment/orders-api --timeout=5m

Limitations: rolling updates mix old and new versions for the duration of the rollout (APIs and schemas must be backward compatible), and they have no automatic metric-based abort. For that, use canary analysis (Argo Rollouts, Flagger) on top.

Q27: What are the DORA metrics and how do you measure them from a pipeline?​

The DORA research program identified four key delivery metrics that correlate with organizational performance:

MetricQuestion it answersTypical data source
Deployment frequencyHow often do we deploy to production?Deploy events from CD system
Lead time for changesHow long from commit to running in production?Commit timestamp vs deploy timestamp
Change failure rateWhat percent of deploys cause a failure needing remediation?Deploys linked to incidents/rollbacks/hotfixes
Time to restore serviceHow long to recover from a failed deploy? (newer DORA reports call this failed deployment recovery time)Incident open/resolve times

The first two measure throughput, the last two stability. The research finding interviewers want to hear: high performers are better at both at once; speed and stability are not a trade-off when changes are small and automated.

Measurement tips:

  • Emit a structured "deployment" event (service, version/digest, commit SHA, timestamp, environment) from the pipeline; GitHub deployments API, Argo CD notifications, or a webhook into your metrics store.
  • Link incidents and rollbacks to the deployment that caused them, or change failure rate is guesswork.
  • Use them to find bottlenecks for a team, never to rank individuals; once they become targets they get gamed (for example, splitting deploys to inflate frequency).

GitHub Actions in Depth​

Q28: In GitHub Actions, how do jobs share data, and what is the difference between outputs and artifacts?​

Each job runs on a fresh runner, so nothing on disk survives between jobs. Jobs run in parallel unless ordered with needs. There are two ways to pass data:

  • Job outputs: small strings (a version, an image digest, a flag). A step writes name=value to the file at $GITHUB_OUTPUT, the job maps it under outputs, and downstream jobs read needs.<job>.outputs.<name>.
  • Artifacts: files (build output, test reports, a Terraform plan) uploaded with actions/upload-artifact and downloaded with actions/download-artifact in a later job of the same run.
jobs:
version:
runs-on: ubuntu-latest
outputs:
version: ${{ steps.v.outputs.version }}
steps:
- uses: actions/checkout@v4
- id: v
run: echo "version=$(cat VERSION)-${GITHUB_SHA::7}" >> "$GITHUB_OUTPUT"

build:
needs: version
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- run: python -m pip install build && python -m build
- uses: actions/upload-artifact@v4
with:
name: dist-${{ needs.version.outputs.version }}
path: dist/
retention-days: 7

publish:
needs: [version, build]
runs-on: ubuntu-latest
steps:
- uses: actions/download-artifact@v4
with:
name: dist-${{ needs.version.outputs.version }}
path: dist/
- run: ls -l dist/

Do not confuse artifacts with actions/cache: the cache is a best-effort speed optimization keyed by content (it may be evicted), whereas artifacts are the guaranteed output of a run.

Q29: How do concurrency groups work and why do deploy workflows need them?​

By default, two pushes to main in quick succession start two workflow runs that can deploy at the same time, and the older commit may finish last and overwrite the newer one. A concurrency group allows at most one running and one pending run per group key.

# Pull request CI: a new push makes the old run pointless, so cancel it
concurrency:
group: ci-${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }}
cancel-in-progress: true
# Production deploy: never cancel a deploy halfway; queue instead
jobs:
deploy:
runs-on: ubuntu-latest
environment: production
concurrency:
group: deploy-production
cancel-in-progress: false
steps:
- run: ./scripts/deploy.sh

Important detail: only one run can be pending per group. If a third run arrives while one is running and one is pending, the older pending run is cancelled. For deploys this is usually desirable (you only care about deploying the newest commit), but it means a concurrency group is not a FIFO queue.

Q30: Reusable workflows vs composite actions: when do you use each?​

Reusable workflowComposite action
Defined withon: workflow_call in .github/workflows/*.ymlruns: using: composite in action.yml
Called atJob level (jobs.<id>.uses)Step level (steps[*].uses)
Can containMultiple jobs, runners, environments, secretsSteps only, run inside the caller's job
Typical useStandard "build-test-deploy" pipeline shared across many reposA reusable sequence of steps, such as "set up toolchain and cache"

Reusable workflow (callee):

# .github/workflows/deploy-service.yml in acme/platform-workflows
on:
workflow_call:
inputs:
service:
type: string
required: true
environment:
type: string
required: true
secrets:
SLACK_WEBHOOK:
required: false

jobs:
deploy:
runs-on: ubuntu-latest
environment: ${{ inputs.environment }}
steps:
- uses: actions/checkout@v4
- run: ./deploy.sh "${{ inputs.service }}" "${{ inputs.environment }}"

Caller:

jobs:
deploy-staging:
uses: acme/platform-workflows/.github/workflows/deploy-service.yml@v3
with:
service: orders-api
environment: staging
secrets: inherit

Composite action:

# .github/actions/setup-python-deps/action.yml
name: Setup Python deps
description: Install Python and project dependencies with pip caching
inputs:
python-version:
description: Python version
default: "3.12"
runs:
using: composite
steps:
- uses: actions/setup-python@v5
with:
python-version: ${{ inputs.python-version }}
cache: pip
- run: pip install -r requirements.txt
shell: bash

Platform teams usually publish versioned reusable workflows so that security controls (scanning, signing, OIDC) are enforced once instead of copy-pasted into hundreds of repositories.

Q31: How do GitHub environments and deployment protection rules work?​

An environment (for example staging, production) is a named deployment target configured in repository settings. A job opts in with environment: production. Environments provide:

  • Required reviewers: the job pauses until an authorized person approves.
  • Wait timer: a delay before the job proceeds (bake time).
  • Deployment branch and tag rules: only main or v* tags can deploy to production.
  • Environment-scoped secrets and variables: production credentials are only exposed to jobs that passed the protection rules.
  • A deployment history per environment, which gives an audit trail and a source for DORA metrics.
jobs:
deploy-prod:
runs-on: ubuntu-latest
environment:
name: production
url: https://api.example.com
steps:
- uses: actions/checkout@v4
- run: ./deploy.sh production
env:
DEPLOY_TOKEN: ${{ secrets.PROD_DEPLOY_TOKEN }} # environment secret

Environments also matter for OIDC: the token's sub claim includes environment:production, so a cloud role can trust only jobs that ran in the protected environment (see the next question).

Q32: How do you authenticate GitHub Actions to AWS without long-lived access keys?​

Use OpenID Connect (OIDC). GitHub issues the job a short-lived signed JWT describing the run (repository, ref, environment). AWS STS validates it against an IAM OIDC identity provider for token.actions.githubusercontent.com and returns temporary credentials for a role whose trust policy matches the claims. No secret is stored in GitHub, and credentials expire after the session duration.

Workflow:

permissions:
id-token: write # allow the job to request an OIDC token
contents: read

jobs:
deploy:
runs-on: ubuntu-latest
environment: production
steps:
- uses: actions/checkout@v4
- uses: aws-actions/configure-aws-credentials@v4
with:
role-to-assume: arn:aws:iam::123456789012:role/github-deploy-prod
aws-region: us-east-1
- run: aws sts get-caller-identity

IAM role trust policy (the important part is the sub condition):

{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"Federated": "arn:aws:iam::123456789012:oidc-provider/token.actions.githubusercontent.com"
},
"Action": "sts:AssumeRoleWithWebIdentity",
"Condition": {
"StringEquals": {
"token.actions.githubusercontent.com:aud": "sts.amazonaws.com",
"token.actions.githubusercontent.com:sub": "repo:acme/orders-api:environment:production"
}
}
}
]
}

Common mistakes: a wildcard sub such as repo:acme/* (any repo or branch in the org, including PR branches, can assume the production role), or no sub condition at all. The same pattern works for GCP (Workload Identity Federation) and Azure (federated credentials).

Q33: How do you apply least privilege to GITHUB_TOKEN, and why is pull_request_target dangerous?​

Every job gets an automatically generated GITHUB_TOKEN. Its default permissions depend on repository/organization settings, so set them explicitly. Start with read-only at workflow level and elevate per job:

permissions:
contents: read

jobs:
test:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- run: make test

release:
needs: test
runs-on: ubuntu-latest
permissions:
contents: write # create a GitHub release
steps:
- uses: actions/checkout@v4
- run: gh release create "v${GITHUB_RUN_NUMBER}" --generate-notes
env:
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}

pull_request vs pull_request_target:

  • pull_request from a fork runs the PR's code with a read-only token and no secrets. Safe by design.
  • pull_request_target runs in the context of the base repository, with secrets and a potentially write-scoped token, so it can label or comment on fork PRs. If such a workflow checks out the PR head (ref: ${{ github.event.pull_request.head.sha }}) and runs its build scripts or tests, an attacker's PR executes with your secrets. This is the "pwn request" pattern.

Rules: never build or run untrusted PR code under pull_request_target; use it only for metadata operations. Also treat values like github.event.pull_request.title as untrusted input: pass them through env: instead of interpolating ${{ }} directly into a run: script, which allows shell injection.

Q34: How do you avoid building and testing everything on every commit in a monorepo?​

Options, from simplest to most powerful:

  1. Workflow path filters: run a service's workflow only when its files change.
on:
pull_request:
paths:
- "services/payments/**"
- "libs/common/**"
- ".github/workflows/payments.yml"
  1. Change detection inside one workflow: compute which packages changed (git diff --name-only against the merge base) and feed the list into a dynamic matrix.
  2. Build-graph-aware tools: Bazel, Pants, Nx, Turborepo and Gradle know the dependency graph, so a change to libs/common rebuilds and tests every dependent package and nothing else, with remote caching of results across machines.

Pitfalls:

  • Path filters do not understand dependencies: if payments imports libs/common and you forget to list it, a breaking library change ships untested.
  • A required status check from a workflow skipped by path filters stays "pending" and blocks the merge. Common fixes are a single always-running aggregator job that is the required check, or a change-detection job instead of trigger-level filters.
  • Run the full suite periodically (nightly or on main) as a safety net for gaps in the dependency graph.

Supply Chain Security and Secrets​

Q35: Why should third-party GitHub Actions be pinned to a full commit SHA?​

uses: some-org/some-action@v2 references a git tag, and tags can be moved. If the action's maintainer account or repository is compromised, the attacker can re-point v2 (and every other tag) to malicious code, and every workflow using it runs that code on its next run, with access to the job's secrets and token. This happened in March 2025 with the widely used tj-actions/changed-files action, whose tags were re-pointed to code that dumped runner memory, including secrets, into workflow logs.

A full 40-character commit SHA is immutable:

steps:
- uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2

To find the commit for a tag yourself:

git ls-remote --tags https://github.com/actions/checkout 'refs/tags/v4.2.2*'
# For an annotated tag, use the line ending in ^{} (the peeled commit), not the tag object

Practices that go with pinning:

  • Keep the version in a trailing comment so Dependabot or Renovate can propose SHA bumps with release notes.
  • Restrict which actions may run (organization setting: GitHub-owned, verified creators, or an explicit allow-list).
  • Prefer a few well-maintained actions; for trivial logic, a run: step is less risk than a dependency.
  • Least-privilege permissions: limits what a compromised action can do with the token.

Q36: What is SLSA, and what is build provenance?​

SLSA (Supply-chain Levels for Software Artifacts, pronounced "salsa") is an OpenSSF framework for describing how trustworthy an artifact's build process is. The SLSA v1.0 Build track defines levels:

  • Build L1: provenance exists, describing how the artifact was built.
  • Build L2: builds run on a hosted build platform that generates and signs the provenance.
  • Build L3: the build platform is hardened so that builds cannot influence each other or forge provenance (runs are isolated from each other, and the provenance signing material is not accessible to user-defined build steps).

Provenance is a signed statement (an in-toto attestation) saying "artifact with digest X was built from repository Y at commit Z by workflow W on builder B". Consumers verify it before deploying, which defends against a tampered artifact being pushed to the registry out-of-band.

GitHub can generate provenance attestations natively:

jobs:
build:
runs-on: ubuntu-latest
permissions:
contents: read
packages: write
id-token: write # sign with Sigstore via OIDC
attestations: write # store the attestation
steps:
- uses: actions/checkout@v4
- uses: docker/login-action@v3
with:
registry: ghcr.io
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}
- id: push
uses: docker/build-push-action@v6
with:
push: true
tags: ghcr.io/${{ github.repository }}:${{ github.sha }}
- uses: actions/attest-build-provenance@v2
with:
subject-name: ghcr.io/${{ github.repository }}
subject-digest: ${{ steps.push.outputs.digest }}
push-to-registry: true

Verification before deploy:

gh attestation verify oci://ghcr.io/acme/orders-api@sha256:3f1c0e8a9b7d6c5e4f3a2b1c0d9e8f7a6b5c4d3e2f1a0b9c8d7e6f5a4b3c2d1e --owner acme

Q37: How do you sign and verify container images with Sigstore cosign?​

Cosign signs an image digest and stores the signature in the registry next to the image. In CI the recommended mode is keyless signing: cosign uses the job's OIDC identity to get a short-lived certificate from Sigstore's Fulcio CA, signs, and records the signature in the Rekor transparency log. There is no private key to manage or leak.

jobs:
sign:
needs: build
runs-on: ubuntu-latest
permissions:
id-token: write
packages: write
steps:
- uses: sigstore/cosign-installer@v3
- uses: docker/login-action@v3
with:
registry: ghcr.io
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}
- run: cosign sign --yes "ghcr.io/${{ github.repository }}@${{ needs.build.outputs.digest }}"

Verification pins who signed, not just "someone signed":

cosign verify \
--certificate-identity-regexp '^https://github.com/acme/orders-api/\.github/workflows/' \
--certificate-oidc-issuer https://token.actions.githubusercontent.com \
ghcr.io/acme/orders-api@sha256:3f1c0e8a9b7d6c5e4f3a2b1c0d9e8f7a6b5c4d3e2f1a0b9c8d7e6f5a4b3c2d1e

Signing only adds security if something enforces verification. In Kubernetes that is an admission controller such as Sigstore policy-controller or Kyverno verifyImages, configured to reject unsigned images or images signed by an unexpected workflow identity.

Q38: What is an SBOM and how do you use it in CI?​

A Software Bill of Materials is a machine-readable inventory of every component in an artifact: packages, versions, licenses, and hashes. The common formats are SPDX and CycloneDX.

Why it matters:

  • When a new CVE drops (the Log4Shell scenario), you query SBOMs to find which deployed images contain the vulnerable version in minutes instead of grepping repositories.
  • License compliance checks.
  • Increasingly a procurement or regulatory requirement for software sold to governments and enterprises.

Generating and scanning in a pipeline with Anchore's open-source tools:

# Generate an SBOM from the built image
syft ghcr.io/acme/orders-api:"${GIT_SHA}" -o spdx-json > sbom.spdx.json

# Scan the SBOM for known vulnerabilities and fail on high or critical
grype sbom:./sbom.spdx.json --fail-on high

Then attach the SBOM to the image as an attestation (for example cosign attest --type spdxjson --predicate sbom.spdx.json ...) and store it, so it can be re-scanned later as new CVEs are published. An SBOM is an inventory, not a fix: it needs a process (continuous re-scanning, ownership, patch SLAs) to reduce risk.

Q39: How do you keep secrets out of CI logs?​

Defense in layers:

  1. Prefer no static secrets at all: OIDC federation to cloud providers, short-lived tokens from Vault, so a leak expires quickly.
  2. Platform masking: GitHub Actions masks values of registered secrets in logs. Values you derive at runtime (a token fetched from Vault, a decoded JSON field) are not known to the masker unless you register them:
TOKEN="$(vault kv get -field=token secret/ci/deploy)"
echo "::add-mask::${TOKEN}"
echo "DEPLOY_TOKEN=${TOKEN}" >> "$GITHUB_ENV"
  1. Know what masking misses: masking is literal string matching on log output. It fails on transformed values (base64, URL-encoded, split across lines), on structured secrets where only the whole JSON blob is registered, and on files uploaded as artifacts.
  2. Avoid echoing: no set -x around commands that take secrets as arguments, no env or printenv debugging steps, no verbose flags (curl -v prints Authorization headers).
  3. Pass secrets via environment variables or stdin, not command-line arguments (which can show up in process listings and error messages).
  4. Scan: run a secret scanner (for example gitleaks or GitHub secret scanning with push protection) on commits, and treat any secret printed to a log as compromised: rotate first, then clean up.

Release Strategies, Rollback and GitOps​

Q40: How do you automatically roll back a canary when it breaches an SLO?​

Progressive delivery controllers (Argo Rollouts, Flagger, AWS CodeDeploy with CloudWatch alarms, Spinnaker Kayenta) shift traffic in steps and run analysis between steps. If a metric query fails its success condition too many times, the controller aborts and shifts traffic back to the stable version without a human.

Argo Rollouts example: 10% then 50% then 100%, with a Prometheus success-rate check running in the background from the first pause onward.

apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
name: checkout
spec:
replicas: 10
selector:
matchLabels:
app: checkout
template:
metadata:
labels:
app: checkout
spec:
containers:
- name: checkout
image: ghcr.io/acme/checkout@sha256:3f1c0e8a9b7d6c5e4f3a2b1c0d9e8f7a6b5c4d3e2f1a0b9c8d7e6f5a4b3c2d1e
ports:
- containerPort: 8080
strategy:
canary:
analysis:
templates:
- templateName: success-rate
startingStep: 1
args:
- name: service
value: checkout
steps:
- setWeight: 10
- pause: {duration: 10m}
- setWeight: 50
- pause: {duration: 10m}
---
apiVersion: argoproj.io/v1alpha1
kind: AnalysisTemplate
metadata:
name: success-rate
spec:
args:
- name: service
metrics:
- name: success-rate
interval: 1m
failureLimit: 2
successCondition: result[0] >= 0.99
provider:
prometheus:
address: http://prometheus.monitoring.svc:9090
query: |
sum(rate(http_requests_total{service="{{args.service}}",code!~"5.."}[5m]))
/
sum(rate(http_requests_total{service="{{args.service}}"}[5m]))

Design points interviewers look for:

  • Gate on user-facing SLIs (error rate, latency percentiles), ideally comparing canary vs stable rather than a fixed threshold, so a global incident does not get blamed on the canary.
  • The canary needs enough traffic for the metric to be statistically meaningful; at 1% of low traffic, a 10-minute window may contain a handful of requests.
  • Without a traffic-routing integration (service mesh or ingress), setWeight is approximated by replica ratios.
  • Automated rollback only reverts the code. Schema changes, emitted events and external side effects must be backward compatible for rollback to be safe.

Q41: How does CI hand off to a GitOps controller, and how do you promote between environments?​

In GitOps, CI does not run kubectl apply against the cluster. The split is:

  • CI (push model, has build credentials): test, build, sign, push image, then write the new image digest into the environment configuration repository (or open a PR there).
  • CD controller (pull model, runs inside the cluster: Argo CD or Flux): watches the config repo and reconciles the cluster to match. The cluster's credentials never leave the cluster.

Typical repo layout with Kustomize overlays:

deploy/
base/
deployment.yaml
kustomization.yaml
overlays/
staging/kustomization.yaml # image digest for staging
production/kustomization.yaml # image digest for production

CI step updating staging:

cd deploy/overlays/staging
kustomize edit set image "ghcr.io/acme/orders-api@${DIGEST}"
git commit -am "orders-api: promote ${DIGEST} to staging"
git push

Promotion to production is a PR that copies the same digest from the staging overlay to the production overlay. The PR is the approval and audit record, and rollback is git revert. Argo CD Image Updater and Flux image automation controllers can automate the write-back step.

Argo CD Application with automated sync:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: orders-api-production
namespace: argocd
spec:
project: default
source:
repoURL: https://github.com/acme/deploy-config.git
targetRevision: main
path: deploy/overlays/production
destination:
server: https://kubernetes.default.svc
namespace: orders
syncPolicy:
automated:
prune: true
selfHeal: true

Q42: How do you ship a hotfix when you practice trunk-based development?​

The default rule is fix forward on trunk: commit the fix to main, let it go through the normal (fast) pipeline, and deploy. If the pipeline is too slow for an emergency, that is the problem to fix, not a reason to bypass it.

When releases are cut from short-lived release branches (mobile apps, on-prem software, regulated releases):

  1. Fix on main first, with a test that reproduces the bug.
  2. Cherry-pick the commit onto the release branch (git cherry-pick -x <sha> records the origin).
  3. Build from the release branch, run the same pipeline, and release a patch version.
git checkout release/2.7
git cherry-pick -x 4e1d2c9
git push origin release/2.7

Fixing on main first prevents regressions: if you fix only the release branch, the bug reappears in the next release cut from main. Things to avoid: hotfixes built on a laptop, skipping tests "because it is urgent", and long-lived release branches that diverge from trunk. For bugs in code behind a feature flag, the fastest mitigation is usually turning the flag off, followed by a normal fix.

Advanced Real-World CI/CD Scenarios​

Q43: Rollback was impossible: mutable tags and a forward-only migration​

Level: Senior

Scenario: A release causes 5xx errors on checkout. The on-call engineer re-deploys orders-api:latest from the previous day, but the error continues. It turns out latest was overwritten by today's build, and the "old" pods also fail because today's release dropped a column the old code still reads. Recovery takes 70 minutes and requires restoring a database backup.

Impact: Over an hour of failed checkouts, data written between the backup and the restore is lost or has to be replayed manually, and the team loses confidence in deploying.

Root Cause:

  • Deployments reference a mutable tag, so "the previous version" is not identifiable or retrievable.
  • The release coupled a destructive schema change (DROP COLUMN) with the code change, so the database was no longer compatible with the previous code.
  • Rollback had never been rehearsed; nobody knew it would not work until it was needed.

Fix:

  1. Deploy by immutable digest and record every deployment (service, digest, commit, time). Rollback means "re-deploy the previous recorded digest", a single automated command.
  2. Make every release N-1 compatible: the database schema after release N must work with code N-1. Use expand/contract: add new columns or tables in one release, migrate reads and writes, and drop old structures only in a later release once N-1 can no longer be rolled back to.
  3. Add a pipeline check that blocks destructive DDL in the same release as the code that stops using it.
  4. Rehearse rollback: run it in staging as part of the pipeline or in game days.
# Rollback workflow: redeploy a known-good digest recorded by earlier deploys
name: rollback
on:
workflow_dispatch:
inputs:
digest:
description: "Known-good image digest (sha256:...)"
required: true
type: string

permissions:
contents: read
id-token: write

concurrency:
group: deploy-production
cancel-in-progress: false

jobs:
rollback:
runs-on: ubuntu-latest
environment: production
steps:
- uses: actions/checkout@v4
- uses: aws-actions/configure-aws-credentials@v4
with:
role-to-assume: arn:aws:iam::123456789012:role/github-deploy-prod
aws-region: us-east-1
- run: aws eks update-kubeconfig --name prod-cluster --region us-east-1
- name: Deploy previous digest
env:
DIGEST: ${{ inputs.digest }}
run: |
kubectl set image deployment/orders-api app="ghcr.io/acme/orders-api@${DIGEST}"
kubectl rollout status deployment/orders-api --timeout=5m

Workflow Diagram:

Release N deployed (code N + DROP COLUMN)
|
v
Errors spike -> redeploy ":latest" -> same broken image (tag was overwritten)
|
v
Redeploy old code -> fails, column is gone -> restore backup (70 min)

Fixed process:
Release N : add new column, write both, read old (N-1 still works)
Release N+1 : read new column (N still works)
Release N+2 : drop old column (N+1 still works)
Rollback : redeploy previous digest -> minutes

Interviewer Notes: Strong answers identify that rollback has two halves, artifact and data, and that the data half is the one that usually fails. They mention immutable digests, N-1 schema compatibility, expand/contract, and rehearsed rollback. Principal-level candidates add policy (a migration linter in CI) and organizational follow-up (blameless postmortem, measuring time to restore).

Red Flags: "Just use latest"; "we restore the database backup to roll back" as the primary plan; treating schema migrations as part of the same atomic step as the code deploy; no deployment record of what was running before.


Q44: A cloud credential was printed in a public CI log​

Level: Senior

Scenario: A developer adds a debug step run: env to investigate a failing deploy job in a public repository. The job uses AWS access keys stored as repository secrets and a database URL assembled at runtime from parts. The access key is masked, but the composed database URL (which contains the password) appears in plain text. The log stays public for two days.

Impact: Production database credentials exposed publicly; potential data breach and regulatory disclosure obligations. Long-lived AWS keys were one typo away from the same fate.

Root Cause:

  • Log masking only redacts exact registered secret values; the composed URL was a new string the masker did not know about.
  • Debug step that dumps the environment; no review of workflow changes by an owner.
  • Long-lived static credentials (AWS keys, a database password) that stay valid after leaking.

Fix:

Immediate response, in this order:

  1. Rotate the database password and deactivate the AWS keys first. Deleting the log does not un-leak a secret, so assume it is compromised.
  2. Delete the workflow run logs, then review database and CloudTrail logs for use of the leaked credentials during the exposure window.
  3. Follow the incident and disclosure process.

Prevention:

  1. Replace AWS keys with OIDC role assumption (no stored cloud secret at all).
  2. Fetch the database credential at runtime and mask every derived value before use.
  3. Protect workflow files with CODEOWNERS and require review by the platform team.
  4. Secret scanning on commits and push protection; alerting on credential use from unexpected IP ranges.
permissions:
contents: read
id-token: write

jobs:
migrate:
runs-on: ubuntu-latest
environment: production
steps:
- uses: actions/checkout@v4
- uses: aws-actions/configure-aws-credentials@v4
with:
role-to-assume: arn:aws:iam::123456789012:role/github-migrate-prod
aws-region: us-east-1
- name: Load DB credentials and mask derived values
run: |
SECRET_JSON="$(aws secretsmanager get-secret-value --secret-id prod/orders/db --query SecretString --output text)"
DB_PASSWORD="$(echo "$SECRET_JSON" | jq -r .password)"
DB_HOST="$(echo "$SECRET_JSON" | jq -r .host)"
DATABASE_URL="postgresql://app:${DB_PASSWORD}@${DB_HOST}:5432/orders"
echo "::add-mask::${DB_PASSWORD}"
echo "::add-mask::${DATABASE_URL}"
echo "DATABASE_URL=${DATABASE_URL}" >> "$GITHUB_ENV"
- run: ./scripts/migrate.sh
# .github/CODEOWNERS
/.github/workflows/ @acme/platform-team

Workflow Diagram:

Secret visible in log
|
v
1. Rotate / revoke credential (minutes, before anything else)
|
v
2. Delete logs, audit usage window (CloudTrail, DB audit logs)
|
v
3. Root-cause and prevent:
OIDC instead of keys -> runtime fetch + add-mask -> CODEOWNERS on workflows
|
v
4. Postmortem + secret scanning / push protection

Interviewer Notes: The first sentence of a good answer is "rotate the credential", not "delete the log". Look for understanding of why masking failed (derived values), the move from long-lived to short-lived credentials, and controls on who can change workflows. Senior candidates also scope the blast radius: what could this credential reach, and is there evidence it was used?

Red Flags: "Delete the log and move on"; believing masking makes printing secrets safe; storing more static keys as the fix; no audit of whether the credential was used.


Q45: The pipeline takes 45 minutes and developers batch their changes​

Level: Senior

Scenario: CI on a Python monolith takes 45 minutes per PR. Developers stop pushing small commits, open large PRs, and re-run failed jobs late in the day. Lead time for changes is several days and merge conflicts are frequent.

Impact: Slow feedback, larger and riskier changes, lost engineering hours waiting, and pressure to skip CI for "urgent" fixes.

Root Cause: Measured with a timing breakdown of the slowest runs:

  • Dependencies installed from scratch on every job (no cache): 6 minutes.
  • Docker image rebuilt without layer cache: 8 minutes.
  • 4,000 tests run serially in one job: 25 minutes.
  • Everything runs sequentially even though lint, tests and image build are independent.

Fix:

  1. Measure first: per-step durations over many runs, not one anecdote.
  2. Cache dependencies keyed on the lockfile, and Docker layers via the BuildKit GitHub Actions cache.
  3. Run independent jobs in parallel; fail fast on lint.
  4. Shard tests across a matrix (here with the pytest-split plugin, which balances by recorded durations).
  5. Cancel superseded runs on new pushes.
  6. Longer term: run only affected tests per PR with the full suite on main, and delete or fix slow tests.
name: ci
on:
pull_request:

concurrency:
group: ci-${{ github.event.pull_request.number }}
cancel-in-progress: true

permissions:
contents: read

jobs:
lint:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-python@v5
with:
python-version: "3.12"
cache: pip
- run: pip install ruff
- run: ruff check .

test:
runs-on: ubuntu-latest
strategy:
fail-fast: false
matrix:
shard: [1, 2, 3, 4, 5, 6]
steps:
- uses: actions/checkout@v4
- uses: actions/setup-python@v5
with:
python-version: "3.12"
cache: pip
- run: pip install -r requirements-dev.txt
- run: pytest --splits 6 --group ${{ matrix.shard }}

image:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: docker/setup-buildx-action@v3
- uses: docker/build-push-action@v6
with:
push: false
tags: orders-api:ci
cache-from: type=gha
cache-to: type=gha,mode=max

Workflow Diagram:

Before (serial, ~45 min):
deps(6) -> lint(2) -> tests(25) -> docker(8) -> misc(4)

After (parallel, ~9 min wall clock):
+-> lint (1 min)
checkout -+-> tests x6 shards (cached deps, ~7 min each)
+-> docker build with layer cache (~2 min)

Interviewer Notes: Good candidates start with data (which steps are slow) rather than jumping to "buy bigger runners". They know the main levers (caching, parallelism, sharding, test selection, cancelling stale runs) and connect pipeline speed to DORA lead time and change size. Principal candidates discuss a target (for example under 10 minutes for PR feedback), ownership of test suite health, and the trade-off of running fewer tests per PR with a safety net on main.

Red Flags: "Run the tests nightly instead"; adding retries to hide slowness; caching without lockfile-based keys (stale dependencies); parallelizing tests that share a database without isolating them.


Q46: A deploy fails intermittently because the migration is not idempotent​

Level: Senior

Scenario: Each pod runs database migrations on startup. During a rollout with 6 new pods, two pods start migrating at the same time. One succeeds; the other fails on CREATE INDEX ("relation already exists"), crash-loops, and the rollout stalls. On another day the migration job times out halfway (a long index build), the pipeline retries it, and the retry fails because the first half already applied.

Impact: Unpredictable deploys, manual database fixes in production, and table locks during peak hours when ALTER TABLE waits behind long-running queries.

Root Cause:

  • Migrations run concurrently from every replica instead of once.
  • Statements are not idempotent and not safe to retry.
  • Large DDL (index builds, table rewrites) is run as a blocking, transactional step during deploy.

Fix:

  1. Run migrations once, as a dedicated pipeline step or Kubernetes Job before the rollout, not in application startup. Use a migration tool that takes a lock (Liquibase uses its DATABASECHANGELOGLOCK table; Flyway also locks during migration) so accidental concurrency is serialized.
  2. Write idempotent, retry-safe DDL, and set a lock_timeout so a migration fails fast instead of blocking production traffic behind it.
  3. Build large indexes with CONCURRENTLY (PostgreSQL) in their own migration, outside a transaction.
  4. Keep the migration N-1 compatible (expand/contract) so it can run before the new code ships.
-- PostgreSQL: retry-safe expand migration
SET lock_timeout = '5s';
SET statement_timeout = '15min';

ALTER TABLE orders ADD COLUMN IF NOT EXISTS fulfillment_status text;

-- Must not run inside a transaction block; keep it in a separate migration
CREATE INDEX CONCURRENTLY IF NOT EXISTS idx_orders_fulfillment_status
ON orders (fulfillment_status);

Caveat: if CREATE INDEX CONCURRENTLY fails midway, it leaves an invalid index that IF NOT EXISTS will then skip. The migration should check pg_index.indisvalid and drop and recreate an invalid index.

# Kubernetes Job run by the pipeline before updating the Deployment
apiVersion: batch/v1
kind: Job
metadata:
name: orders-migrate-4e1d2c9
spec:
backoffLimit: 2
activeDeadlineSeconds: 1800
template:
spec:
restartPolicy: Never
containers:
- name: migrate
image: ghcr.io/acme/orders-api@sha256:3f1c0e8a9b7d6c5e4f3a2b1c0d9e8f7a6b5c4d3e2f1a0b9c8d7e6f5a4b3c2d1e
command: ["alembic", "upgrade", "head"]

Workflow Diagram:

Before:
pod1 --migrate--> OK
pod2 --migrate--> "already exists" -> CrashLoopBackOff -> rollout stuck

After:
pipeline: migrate Job (once, locked, idempotent, lock_timeout)
|
v success
rollout Deployment (pods never run DDL)
|
v
later release: contract (drop old columns)

Interviewer Notes: Look for "migrations run once, separately from app startup", idempotent and retry-safe DDL, awareness of lock contention (lock_timeout, CONCURRENTLY), and expand/contract for rollback safety. Strong candidates mention that retries in CI are only safe if the step is idempotent, and the invalid-index edge case.

Red Flags: Running migrations in every replica's entrypoint; "wrap everything in one transaction" for operations that cannot run in a transaction; ignoring locking on large tables; manual hotfixes to the schema in production.


Q47: A compromised third-party action exfiltrated secrets​

Level: Principal

Scenario: A popular third-party action referenced as @v35 in 120 repositories is compromised: the attacker re-points all version tags to a commit that dumps runner process memory and prints secrets to the logs (the pattern of the March 2025 tj-actions/changed-files incident). Your organization learns about it from a security advisory several hours later.

Impact: Every secret available to any job that ran the action during the window may be exposed (cloud keys, registry tokens, npm publish tokens). Public repositories have world-readable logs. Downstream risk: an attacker publishing malicious packages or images using stolen tokens.

Root Cause:

  • Actions referenced by mutable tags.
  • No inventory of which workflows use which actions.
  • Broad secrets available to all jobs, including long-lived publish tokens and cloud keys.
  • Default GITHUB_TOKEN permissions broader than needed.

Fix:

Incident response:

  1. Find every affected run: search the organization for the action reference, then list workflow runs in the exposure window.
  2. Rotate every secret available to those jobs, prioritizing publish and cloud credentials; audit the registry, package and cloud logs for use of them.
  3. Delete logs of affected runs; replace the action reference with a pinned safe SHA or remove the dependency.

Prevention (platform-level):

  1. Pin all actions by full SHA; let Dependabot or Renovate update the pins with review.
  2. Organization policy: allow-list of permitted actions.
  3. Replace static secrets with OIDC; give publish credentials only to a dedicated, protected-environment job.
  4. Default permissions: contents: read everywhere.
  5. Egress monitoring or restriction on runners to detect unusual outbound connections.
# Inventory: which repositories reference the action?
gh search code "tj-actions/changed-files" --owner acme --extension yml --limit 200

# Recent runs of a given workflow in one repository, to scope the exposure window
gh run list --repo acme/orders-api --workflow ci.yml --created ">=2025-03-14" --limit 100
# .github/dependabot.yml: keep SHA-pinned actions up to date
version: 2
updates:
- package-ecosystem: github-actions
directory: /
schedule:
interval: weekly

Workflow Diagram:

Advisory published
|
v
Inventory: which repos/workflows/runs used the action? (code search + run history)
|
v
Rotate every secret those jobs could read -> audit usage of each credential
|
v
Remove/pin the action, delete exposed logs
|
v
Platform hardening: SHA pins + allow-list + OIDC + least-privilege tokens + egress monitoring

Interviewer Notes: Principal candidates treat this as a supply-chain incident with organization-wide scope: inventory, blast-radius analysis, rotation and audit, then systemic controls. They should explain why SHA pinning helps (immutable reference) and what it does not solve (a compromised commit you deliberately upgrade to; pinned composite actions that internally reference other actions by tag). Mention of least privilege and OIDC shows they reduce what any compromise can steal.

Red Flags: "Just bump to the fixed version"; rotating only the secrets they think were used; no inventory capability; relying on log masking to protect secrets from code running inside the job (code in the job can read secrets directly).


Q48: A fork pull request stole secrets through pull_request_target​

Level: Senior

Scenario: To post test coverage comments on PRs from forks, a team switches their test workflow from pull_request to pull_request_target and checks out the PR head so the tests run on the contributor's code. Weeks later, a PR from a new account modifies conftest.py to send environment variables to an external server. The workflow runs automatically, and the npm publish token and a write-scoped GITHUB_TOKEN are exfiltrated.

Impact: Attacker can publish a malicious package version under your name and push to the repository; every downstream user of the package is at risk.

Root Cause: pull_request_target runs with the base repository's secrets and token permissions. Checking out and executing untrusted PR code in that context hands the attacker those privileges. A build script, test file, package.json lifecycle script, or Makefile in the PR is enough.

Fix:

  1. Revoke and rotate the npm token and any other secrets; audit npm publish history and repository pushes.
  2. Split the workflow into an unprivileged half and a privileged half:
    • pull_request runs the untrusted code with no secrets and a read-only token, and uploads results as an artifact.
    • A workflow_run workflow (runs in the base repository context) downloads the artifact and posts the comment. It never executes PR code, and treats the artifact contents as untrusted data.
  3. Require approval for workflow runs from first-time or outside contributors (repository Actions settings).
  4. Move package publishing to a separate release workflow on tags, in a protected environment, using trusted publishing (OIDC) where the registry supports it.
# .github/workflows/test.yml - untrusted code, no secrets
name: test
on:
pull_request:

permissions:
contents: read

jobs:
test:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-python@v5
with:
python-version: "3.12"
- run: pip install -r requirements-dev.txt
- run: pytest --cov=app --cov-report=term > coverage.txt
- run: echo "${{ github.event.pull_request.number }}" > pr_number.txt
- uses: actions/upload-artifact@v4
with:
name: coverage
path: |
coverage.txt
pr_number.txt
# .github/workflows/coverage-comment.yml - privileged, never runs PR code
name: coverage-comment
on:
workflow_run:
workflows: [test]
types: [completed]

permissions:
pull-requests: write
actions: read

jobs:
comment:
if: github.event.workflow_run.event == 'pull_request' && github.event.workflow_run.conclusion == 'success'
runs-on: ubuntu-latest
steps:
- uses: actions/download-artifact@v4
with:
name: coverage
run-id: ${{ github.event.workflow_run.id }}
github-token: ${{ secrets.GITHUB_TOKEN }}
- name: Post comment
env:
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
run: |
PR_NUMBER="$(head -n 1 pr_number.txt)"
# The artifact was produced next to untrusted code: validate before use
if ! [[ "$PR_NUMBER" =~ ^[0-9]+$ ]]; then
echo "Invalid PR number in artifact" >&2
exit 1
fi
{
echo '```text'
tail -n 20 coverage.txt
echo '```'
} > body.md
gh pr comment "$PR_NUMBER" --repo "$GITHUB_REPOSITORY" --body-file body.md

Note: github.event.workflow_run.pull_requests is empty for PRs opened from forks, which is why the PR number travels inside the artifact. Because the PR's code ran in the job that produced the artifact, the privileged workflow validates the value strictly and only ever uses it as data.

Workflow Diagram:

Fork PR
|
v
[pull_request] untrusted code, read-only token, no secrets
| uploads coverage.txt (data, untrusted)
v
[workflow_run] base repo context, scoped token
| reads data only, never executes PR code
v
Comment posted on PR

Interviewer Notes: The key insight is "privilege plus untrusted code equals compromise". Candidates should know the difference between pull_request and pull_request_target, the split-workflow pattern via workflow_run and artifacts, and that the artifact itself is attacker-controlled input. Bonus: script injection through ${{ github.event.* }} interpolation in run: blocks.

Red Flags: "Add an if that checks the author's name"; believing a code review after the run protects you (the workflow already ran); keeping publish tokens available to test workflows.


Q49: Two merges deployed out of order and production ran stale code​

Level: Senior

Scenario: Commits A and then B are merged to main two minutes apart. Both trigger the deploy workflow. A's run is slow (cache miss) and finishes after B's. Production ends up running A, without B's fix, while the dashboard says B was deployed. The bug fixed by B reappears and nobody understands why.

Impact: Confusing incidents, "fixed" bugs reappearing, and wrong information in the deployment history, which corrupts debugging and DORA data.

Root Cause:

  • Deploy jobs from different runs execute concurrently with no serialization.
  • The deploy step is "last writer wins" with no check that the artifact being deployed is newer than what is running.

Fix:

  1. Serialize production deploys with a concurrency group (one running, newest pending).
  2. Make the deploy step check that the commit being deployed descends from the currently deployed commit, and skip if it is older.
  3. In GitOps setups, the config repository's history provides the ordering: each promotion is a commit and stale writers fail on push (non-fast-forward) and must rebase.
name: deploy
on:
push:
branches: [main]

permissions:
contents: read
id-token: write

jobs:
deploy:
runs-on: ubuntu-latest
environment: production
concurrency:
group: deploy-production
cancel-in-progress: false
steps:
- uses: actions/checkout@v4
with:
fetch-depth: 0
- uses: aws-actions/configure-aws-credentials@v4
with:
role-to-assume: arn:aws:iam::123456789012:role/github-deploy-prod
aws-region: us-east-1
- name: Skip if an equal or newer commit is already deployed
id: guard
run: |
DEPLOYED_SHA="$(aws ssm get-parameter --name /orders-api/prod/deployed-sha --query Parameter.Value --output text)"
if git merge-base --is-ancestor "$GITHUB_SHA" "$DEPLOYED_SHA"; then
echo "Commit $GITHUB_SHA is not newer than deployed $DEPLOYED_SHA; skipping"
echo "skip=true" >> "$GITHUB_OUTPUT"
else
echo "skip=false" >> "$GITHUB_OUTPUT"
fi
- if: steps.guard.outputs.skip == 'false'
run: |
./scripts/deploy.sh "$GITHUB_SHA"
aws ssm put-parameter --name /orders-api/prod/deployed-sha --value "$GITHUB_SHA" --type String --overwrite

Workflow Diagram:

Without serialization:
run A: build......(slow)..........deploy A <- finishes last, wins
run B: build..deploy B
Result: prod = A (stale)

With concurrency group + ancestry guard:
run A: build......deploy A
run B: build..(waits)........deploy B <- serialized
If A arrives after B: "A is an ancestor of deployed B" -> skip
Result: prod = B

Interviewer Notes: Candidates should recognize this as a classic race condition and propose serialization plus an ordering check, not just "deploy less often". Knowing that a GitHub concurrency group keeps only one pending run (it cancels older pending runs, which here is helpful) shows real hands-on experience.

Red Flags: "Only allow one merge at a time"; relying on timestamps from different runners for ordering; no single source of truth for "what is deployed".


Q50: Persistent self-hosted runners leaked state between jobs​

Level: Principal

Scenario: To speed up builds, a company runs long-lived self-hosted runners on EC2 instances with an attached instance profile that has broad AWS permissions. The runners are registered at organization level and also serve a public open-source repository. A malicious PR to that repository runs a build step that installs a background process on the runner. Later jobs from private repositories run on the same machine; the process captures their checkout tokens and source code, and uses the instance profile to read S3 buckets.

Impact: Source code and data exfiltration across many private repositories; cloud access through the runner's instance role; difficult forensics because the runner was never rebuilt.

Root Cause:

  • Non-ephemeral runners: state (files, processes, credentials, Docker cache) persists between jobs from different trust levels.
  • Public repository workflows can run on runners shared with private repositories.
  • The runner host has standing cloud permissions available to any job via the instance metadata service.

Fix:

  1. Isolate and rebuild: take the runners offline, preserve images for forensics, rotate credentials reachable from them, and audit CloudTrail for the instance role.
  2. Use ephemeral runners: one job per runner, then destroyed (--ephemeral registration, or autoscaling solutions such as Actions Runner Controller on Kubernetes).
  3. Never use self-hosted runners for public repositories, or put them in a separate runner group restricted to that repository on isolated infrastructure.
  4. Remove the instance profile; give jobs cloud access through OIDC role assumption scoped per repository and environment.
  5. Segment runner groups by trust level (production deploy runners separate from general CI), and restrict network egress.
# Register a runner that accepts exactly one job and then unregisters itself
./config.sh --url https://github.com/acme --token "${RUNNER_REG_TOKEN}" \
--ephemeral --unattended --labels linux,x64,ci-general --runnergroup general-ci
./run.sh

Workflow Diagram:

Before:
public PR job ---> runner-1 (long-lived, instance role) <--- private repo jobs
| malware persists
v
tokens, code, S3 data stolen

After:
public repo -> GitHub-hosted runners only
private CI -> runner group "general-ci" -> fresh VM/pod per job, no cloud role
prod deploys -> runner group "deploy" -> fresh per job, OIDC role per environment

Interviewer Notes: Principal-level answers talk about trust boundaries: who can schedule code onto a runner, what persists between jobs, and what credentials the host exposes. Expect ephemeral runners, runner groups, OIDC instead of instance roles, and GitHub's guidance that self-hosted runners should not be used with public repositories. Strong candidates also discuss the cost trade-off (ephemeral runners lose warm caches) and solutions such as remote build caches.

Red Flags: "Clean the workspace directory between jobs" as the fix; sharing one runner pool across all trust levels; giving runners admin cloud roles "to make deployments easy".


Q51: A kubectl edit hotfix was silently reverted by Argo CD​

Level: Senior

Scenario: During an incident, an engineer raises memory limits on a crash-looping deployment with kubectl edit. The pods stabilize. Three minutes later Argo CD, with selfHeal: true, restores the Git-declared limits, the pods start crash-looping again, and the incident restarts. In another cluster where self-heal is off, a similar manual change stays forever, and the next deploy unexpectedly reverts it weeks later.

Impact: Extended incident, confusion during response, and hidden configuration drift that causes a surprise outage later.

Root Cause:

  • The team treated the cluster as the source of truth during an incident, while the GitOps controller treats Git as the source of truth.
  • No documented "break glass" procedure for emergency changes under GitOps.
  • Drift not alerted on where self-heal is disabled.

Fix:

  1. The emergency fix is a Git commit: a small, fast-tracked PR (or a direct revert) to the config repository. With a healthy pipeline, Argo CD applies it within its sync interval, or immediately via a webhook or argocd app sync.
  2. If a manual change is truly needed first, follow a documented break-glass path: disable auto-sync for that application, apply the change, then commit the same change to Git and re-enable auto-sync.
  3. Alert on OutOfSync applications so drift is never silent.
  4. Restrict direct write access to production clusters for humans; use read-only access by default and audited elevation for break-glass.
# Break-glass: pause automation for one app, fix, then reconcile through Git
argocd app set orders-api-production --sync-policy none
kubectl -n orders patch deployment orders-api --type=json \
-p='[{"op":"replace","path":"/spec/template/spec/containers/0/resources/limits/memory","value":"1Gi"}]'

# ...commit the same change to deploy-config, merge, then:
argocd app set orders-api-production --sync-policy automated --self-heal
argocd app sync orders-api-production

Workflow Diagram:

Incident
|
+-- normal path: PR to config repo -> merge -> argocd sync -> fixed (durable)
|
+-- break-glass: disable auto-sync -> kubectl change -> commit same change
-> re-enable auto-sync -> app Synced (no drift)

Anti-pattern:
kubectl edit -> fixed -> selfHeal reverts -> broken again

Interviewer Notes: Candidates should understand reconciliation loops and that "Git is the source of truth" applies during incidents too. Good answers include a break-glass process, drift alerting, and making the Git path fast enough to use under pressure. Note that --sync-policy automated alone does not enable self-heal or prune; those are separate flags.

Red Flags: "Turn off self-heal everywhere"; making manual changes without committing them back; giving everyone cluster-admin to "move fast during incidents".


Q52: Moving a team from a two-week release train to continuous delivery​

Level: Principal

Scenario: A product group of 40 engineers ships once every two weeks. Each release bundles 150+ changes, goes through a 2-day manual QA cycle, and about one in three releases needs a hotfix or rollback. Leadership asks you to "deploy faster without breaking things more".

Impact: Lead time of 1-3 weeks, high change failure rate, painful releases that require all-hands support, and slow incident diagnosis because each release contains so many changes.

Root Cause:

  • Large batch size: many changes per release means more risk and harder root-causing.
  • Manual regression testing as the main quality gate.
  • Release and deploy are the same event, so unfinished features block or endanger the release.
  • Weak production observability, so problems are found by customers.

Fix: A staged program, measured with DORA metrics from day one:

  1. Baseline the four DORA metrics and publish them per team.
  2. Automate the regression suite that QA runs by hand, starting with the highest-risk flows; QA shifts to exploratory testing and test design.
  3. Trunk-based development with short-lived branches and feature flags, so incomplete work merges safely and deploy is decoupled from release.
  4. Build once, promote the same artifact, with automated staging verification.
  5. Progressive delivery in production: canary plus automated SLO analysis and rollback.
  6. Increase cadence step by step: weekly, then daily, then on every merge for services that have earned it. Keep a manual approval (continuous delivery) where regulation requires it.
  7. Invest in observability: per-version error and latency dashboards, so a bad deploy is visible in minutes.
# Pipeline shape the program converges on
name: delivery
on:
push:
branches: [main]

permissions:
contents: read
id-token: write
packages: write

jobs:
build:
uses: acme/platform-workflows/.github/workflows/build-sign.yml@v3
secrets: inherit

staging:
needs: build
uses: acme/platform-workflows/.github/workflows/deploy-service.yml@v3
with:
service: orders-api
environment: staging
secrets: inherit

production:
needs: staging
uses: acme/platform-workflows/.github/workflows/deploy-service.yml@v3
with:
service: orders-api
environment: production # environment rules decide: approval or automatic
secrets: inherit

Workflow Diagram:

Before:
150 changes -> 2-week branch -> 2-day manual QA -> big-bang release -> 1 in 3 needs hotfix

After:
small PR -> CI (<10 min) -> build/sign once -> staging auto-verify
-> prod canary 10% -> SLO analysis -> 100% (or auto-rollback)
Feature visibility controlled separately by flags.

Metrics tracked monthly: deploy frequency up, lead time down,
change failure rate down, recovery time down.

Interviewer Notes: This is a socio-technical question. Principal answers balance tooling with process and people: smaller batches, test automation, flags, progressive delivery, observability, and measuring progress with DORA metrics. They sequence the work (you cannot deploy daily before regression testing is automated) and address risk concerns from stakeholders with data instead of slogans.

Red Flags: "Buy a new CI tool"; increasing deploy frequency without automated tests or rollback; using DORA metrics to rank individuals; ignoring regulatory or approval requirements instead of designing for them.