Tuesday, March 18, 2025

Change & Release Management: Procedure.

Understanding the difference between Change Management and Release Management is crucial for ensuring smooth deployments, minimizing risk, and maintaining system reliability. Here’s how they differ:

1. Definition & Focus

  • Change Management focuses on evaluating, approving, and controlling changes in the IT environment to minimize risks and disruptions.
  • Release Management is about planning, scheduling, and deploying software or infrastructure changes in a controlled manner.

Aspect

Change Management

Release Management

Purpose

Ensure changes are reviewed, approved, and documented to minimize risks.

Ensure new software, features, or updates are successfully deployed to production.

Scope

Covers any IT change (code, infrastructure, configuration, security policies, etc.).

Covers software, application, and infrastructure releases as part of the SDLC.

Process

Request → Review → Approval → Implementation → Monitoring

Planning → Build → Testing → Deployment → Validation

Key Activities

Risk assessment, impact analysis, rollback planning, documentation, approvals.

Packaging, testing, deployment, rollback strategies, version control.

Owner

Change Advisory Board (CAB), ITSM teams, SREs, DevOps Engineers.

DevOps, Release Engineers, SREs, QA teams.

Frequency

Happens for each change request (can be daily or weekly).

Follows a fixed release cycle (e.g., weekly, bi-weekly, continuous delivery).

Governance

Often follows ITIL, SOC2, ISO 27001 for compliance and audit requirements.

Aligned with DevOps practices (CI/CD, Agile, GitOps) for automation & efficiency.

2. Key Differences

A,  Change Management = Decision Making & Risk Mitigation

  • Ensures changes (e.g., code updates, infrastructure modifications, security patches) are reviewed, tested, and approved before implementation.
  • Uses frameworks like ITIL (Information Technology Infrastructure Library) for structured governance.
  • Example: A database schema change needs CAB approval before implementation.

B,  Release Management = Deployment & Delivery Process

  • Focuses on safely deploying updates into production using automation (CI/CD pipelines).
  • Ensures smooth rollouts with feature flags, blue-green deployments, and canary releases.
  • Example: A new feature rollout happens via phased deployment using a release pipeline.

3. How They Work Together

Change Management approves changes, while Release Management executes them.
change request (RFC - Request for Change) may trigger a release cycle.
Release Management follows the Change Management approval process to ensure safe deployments.

Example Workflow:

A.     Change Management:

    • DevOps team proposes an infrastructure change (e.g., upgrading Kubernetes from v1.24 to v1.26).
    • Change Request (RFC) is submitted and reviewed for impact and risk.
    • After approval, it's scheduled for release.

B.     Release Management:

    • The update is packaged into a deployment pipeline.
    • canary release is used to test in production with real traffic.
    • If successful, the update is fully rolled out and monitored for issues.

4. When to Use Each

 Use Change Management when:

  • The change has a high risk (e.g., firewall rule updates, database schema changes).
  • Compliance or security approvals are required.
  • A rollback plan needs to be documented.

 Use Release Management when:

  • You are delivering features, patches, or infrastructure changes via an automated pipeline.
  • You need to coordinate deployments across multiple environments (Dev, Staging, Prod).
  • You are following CI/CD or Agile methodologies.

twtech  Thoughts:

Change Management = "Should we make this change?"
Release Management = "How do we safely deploy this change?"

Here are key resources that should always be backed up before a new release to prevent data loss, system failures, or security risks:

1. Database Backups (Most Critical )

  • Always back up databases (SQL & NoSQL) before any deployment that might alter schema, data, or stored procedures.
  • Use point-in-time recovery (PITR) and snapshots for rapid rollback.
  • For cloud-based DBs (AWS RDS, Azure SQL, etc.), enable automated backups and store copies in a separate region.

2. Configuration Files & Environment Variables

  • Backup application config files, environment variables, and feature flags (e.g., .env, Kubernetes ConfigMaps/Secrets).
  • In Kubernetes, backup Helm values, kubectl get all -o yaml, and etcd data before changes.
  • Store backups in version control (Git, S3, or secure vaults like HashiCorp Vault/AWS SSM).

3. Infrastructure-as-Code (IaC) State

  • Backup Terraform state files (terraform.tfstate) if using Terraform.
  • Store state in remote backends (S3, Azure Blob, Google Cloud Storage) with versioning enabled.
  • If using CloudFormation, take stack exports before modifications.

4. Application Artifacts & Binaries

  • Retain a stable version of the application (Docker images, JAR/WAR files, etc.) in an artifact repository (ECR, DockerHub, Nexus, Artifactory).
  • Ensure rollback versions are always accessible in case of a bad deployment.

5. Logs & Monitoring Data

  • Back up system logs (e.g., /var/log/), application logs, and cloud provider logs.
  • Store logs in long-term storage (AWS S3, ELK, Loki, Datadog, etc.) for post-mortem analysis if rollback is needed.

6. Security Certificates & Access Policies

  • Backup SSL/TLS certificates, API keys, IAM policies, security groups, and firewall rules.
  • Ensure access permissions are versioned and auditable in case of accidental misconfiguration.

7. Persistent Storage & File Systems

  • For applications storing user uploads, reports, and critical documents, backup mounted volumes (EBS, EFS, NFS, Ceph, etc.).
  • Use cloud snapshots (AWS EBS snapshots, Azure Managed Disk snapshots) before changes.

8. Kubernetes Cluster State & Resources

  • Take a backup of Kubernetes manifests (kubectl get all -o yaml > backup.yaml).
  • Use velero to backup cluster state and persistent volumes.
  • Ensure Helm releases can be rolled back (helm rollback <release> <revision>).

9. CI/CD Pipeline Configurations

  • Backup Jenkins jobs, GitHub Actions workflows, GitLab CI/CD YAML, ArgoCD configs before modifying pipelines.
  • Keep a version-controlled history of all changes to prevent CI/CD failures.

10. Load Balancer & DNS Records

  • Backup load balancer configurations, ingress rules, and DNS records (Route 53, Cloudflare, etc.).
  • Misconfigurations can cause service outages, so having snapshots ensures quick recovery.

twtech:

Best Practices for Backups Before Releases

Automate Backups using cron jobs, scripts, or backup services (AWS Backup, Velero, etc.).
Store Backups in a Separate Location (Multi-region, Multi-cloud).
Encrypt & Protect Backups with IAM roles, KMS encryption, or HashiCorp Vault.
Test Restore Procedures regularly—a backup is useless if it can’t be restored.

No comments:

Post a Comment

Kubernetes Clusters | Upstream Vs Downstream.

  The terms "upstream" and "downstream" in the context of Kubernetes clusters often refer to the direction of code fl...