Sunday, November 23, 2025

AWS DMS (Database Migration Service) | Deep Dive & Hands-On.

A deep dive into AWS Database Migration Service (DMS)

Scope:

Architecture,
Use cases,
Replication internals,
Endpoints,
Networking,
Performance tuning,
Monitoring,
Security,
Advanced concepts,
Best practices.

Breakdown:

Benefits of AWS DMS,
Core DMS Components,
Types of Migrations,
DMS Replication Internals,
Multi-AZ High Availability,
Networking Architecture,
Performance Tuning,
Monitoring & Troubleshooting,
Security,
Schema Conversion,
DMS Use Cases,
Limitations & Challenges,
Best Practices Summary.

Intro:

AWS DMS is a fully managed service that helps twtech to migrate databases & applications to AWS quickly and securely, with minimal downtime.
AWS DMS supports homogeneous migrations (e.g., Oracle → Oracle).
AWS DMS also support heterogeneous migrations (e.g., Oracle → PostgreSQL, SQL Server → MySQL, MongoDB → DocumentDB).

1. Benefits of AWS DMS

AWS DMS enables:

Continuous data replication
Minimal downtime migrations
Zero-interruption cutover
Supports multiple engines
High availability in multi-AZ
Schema-agnostic migration

Works for:

Lift-and-shift
Replication
Oracle offloading
Change Data Capture (CDC)
Data warehouse loading

2. Core DMS Components

DMS has three main components:

2.1 Replication Instance

Compute engine that:

Reads data from source
Applies changes to target
Manages ongoing replication (CDC)

Key instance attributes:

vCPU
Memory
Network throughput
Available storage (cache buffer)

2.2 Source Endpoint

Defines:

Source DB engine
Connection parameters
Authentication
Encryption
CDC configuration

Supported sources include:

Oracle
SQL Server
PostgreSQL
MySQL/MariaDB
SAP ASE
MongoDB
Amazon S3
Kafka
Db2
DocumentDB
and more.

2.3 Target Endpoint

Defines where DMS will write.

Targets supported:

RDS (all engines)
Aurora (all engines)
DynamoDB
Redshift
S3 (CSV, Parquet, ORC)
Kinesis
OpenSearch
Snowflake (via S3)
Many more via JDBC or Kafka

3. Types of Migrations

DMS supports 3 main migration types.

3.1 Full Load Only

Migrates only existing data.

Used for:

Non-production DBs
Initial bulk load
Simple migrations

3.2 Full Load + CDC

Most common.

Process:

Bulk load existing data
Start Change Data Capture (CDC) capture
Catch up target with changes
During cutover, stop app traffic and switch DB
Disable CDC

NB:

This Provides near-zero downtime.

3.3 CDC Only

Used when:

Full load done outside DMS
Ongoing replication needed
Real-time pipelines

4. DMS Replication Internals

4.1 Full Load

Reads tables in parallel
Uses batching
Loads into target using bulk operations
Can enable/disable indexes to speed up load
Primary keys recommended

4.2 CDC (Change Data Capture)

DMS captures changes by reading:

Database logs (REDO, WAL, binlog, LSN)
Timestamps
Triggers (legacy mode)

CDC supports:

Inserts
Updates
Deletes
DDL changes (depending on engine)

CDC Flow

Source DB logs → Replication Instance Cache → Target Endpoint Apply Engine

5. Multi-AZ High Availability

DMS supports HA via:

Multi-AZ enabled replication instance
Synchronous replication of replication instance storage
Automatic failover within minutes

Not to be confused with:

Database high availability
Cross-region replication

6. Networking Architecture

DMS requires access to both Source and Target.

Typical setups:

VPC to on-prem via VPN
VPC to on-prem via Direct Connect
DMS inside VPC
Public database endpoints
AWS PrivateLink
Inter-region VPC Peering

Firewalls must allow:

Inbound from replication instance subnets
Outbound to source and target DB ports

Best practice:

Place DMS in same VPC and AZ as target for performance.

7. Performance Tuning

Key Factors:

Replication instance size
Source/target DB instance class
Logging format and speed
Table parallelism
Batch sizes

Important parameters:

MaxFullLoadSubTasks (parallelism)
BatchApplyEnabled
BatchSize
CommitRate
LobChunkSize

Performance Bottlenecks:

Underpowered replication instance
Source DB logging not optimized
Network latency
Large LOB columns
No primary keys
Large transactions on source

8. Monitoring & Troubleshooting

Use:

CloudWatch metrics
Enhanced DMS logging (JSON logs)
Task status dashboards
Table statistics

Critical metrics:

CDCLatencySource
CDCLatencyTarget
CDCIncomingChanges
FullLoadThroughput
ReplicationServerFreeStorageSpace

9. Security

DMS supports:

SSL/TLS encryption in transit
KMS encryption at rest
IAM role-based access
Secrets Manager for credentials
VPC security (SGs, NACLs)
No admin access to replication instance OS

Best practices:

Use KMS CMKs
Store passwords in Secrets Manager
Use PrivateLink for RDS
Limit SGs to least privilege

10. Schema Conversion

DMS does not convert schema.

Use:

AWS Schema Conversion Tool (AWS SCT)

SCT converts:

Tables
Procedures
Functions
Views
Triggers
Packages

Also generates:

Migration assessment reports

11. DMS Use Cases

11.1 Homogeneous migration

Oracle → Oracle
MySQL → MySQL
PostgreSQL → PostgreSQL

11.2 Heterogeneous migration

Oracle → Aurora PostgreSQL
SQL Server → MySQL
MongoDB → DocumentDB

11.3 Real-time analytics

DB → S3 (Parquet) → Glue → Athena

11.4 CDC pipelines

DB → Kafka/Kinesis → Data Lake

11.5 Multi-Region replication

Cross-region DB replication via DMS Change Data Capture (CDC)

12. Limitations & Challenges

Does not migrate stored procedures (SCT required)
Limited support for DDL replication
Memory-intensive workloads may require large RI
CDC fails if source logs are purged early
Large LOB data reduces throughput
Rebuild task required for table schema changes

13. Best Practices Summary

Before migration

Validate schemas with Schema Conversion Tool (SCT)
Enable supplemental logging
Ensure source logs retention
Pre-create indexes
Choose adequate RI size

During migration

Use parallel full load
Tune LOB parameters
Monitor Change Data Capture (CDC) latency
Avoid maintenance windows on source

After migration

Validate row counts
Compare checksums
Switch traffic only after Change Data Capture (CDC) = 0
Disable DMS tasks cleanly

Up Next

Project: Hands-On

How twtech Migrates databases within its environment.

Search for AWS service: DMS

Useful Youtube resource link:

What DMS is all about.

https://youtu.be/lOqySgvxUfI

Determining the Migration Target for on-premises Databases in the AWS Cloud (Amazon Web Services)

https://youtu.be/2UmTXVIlDLw

Introduction to the DMS Schema Conversion (Amazon Web Services)

https://youtu.be/ki3r92Cfqwk

Migrating your PostgreSQL database to Amazon RDS for PostgreSQL with AWS Database Migration Service

https://youtu.be/HOJfrR6lcuU

AWS Data Migration Service (Aurora MYSQL) Hands-On

https://www.youtube.com/watch?v=3v3Q7WVbsT4

Focus:

twtech demonstrates how to migrate on-premise MYSQL database to AWS Aurora MYSQL database,
Below are the steps that we will be performing,
List of services needed,
Create source (MYSQL) and target (Aurora MySQL) Databases,
Create a DMS instance,
Create source & target endpoints,
Test endpoints’ Connection,
Create Data Migration Task,
Monitor Data Migration Task.

Step-1:

Search and Create a database in: AWS RDS

Create a database:

Choose a database creation method and Select: Engine Version

Templates, Settings and Encryption:

Cluster Identifier (Name): twtechtargetauroradb
Credential settings (must be remembered): Needed for subsequent login
twtech: twtech123abc
Cluster storage configuration & instance configuration.

Availability & durability:

Connectivity:

VPC security group:

Read replica write forwarding:

Monitoring

Additional monitoring settings:

Addition configuration:

Backup:

Create database:

NB:

It takes a couple of minutes to fully created: about 5 or more munites

From: Creating

To: Available

Step-2

Wait until Source database is fully created, then follow the same steps to create the Source database

Create a database:

Choose a database creation method & Engine Version:

Templates,Settings and Encryption:
Cluster Identifier (Name): twtechtsourauroradb
Credential settings (must be remembered): Needed for subsequent login
twtech: twtech123abc

Cluster storage configuration & instance configuration.

Connectivity:

VPC security group

Additional configuration:

Monitoring:

Additional settings:

Additional configuration:

Backup:

Create database(second)

Step-3:

Create a DMS instance (also called a replication instance)
Search for AWS service: DMS (Database Migration Service)

From DMS navigation tab, select: Migration / Provisioned (Replication) Instances.

Get started to Create replication instance:

Create replication instance

Instance configuration

Connectivity and security

Create replication instance

NB:

It takes also a couple of minutes to be fully created: 5 to 7 munites

From: creating

To:

Step-4

Created endpoints for: both databases

A, source endpoint database

Create: source-database endpoint

Endpoint configuration, Endpoint identifier : twtechsource-endpoint-mysql
Source engine: Amazon Aurora PostgreSQL (use MYSQL for On-Prem)
Access to endpoint database: Manual
From database console, copy the endpoint database name(click open to access details):

· twtechsourceauroradb.cluster-cbgsxxxx0z8i.us-east-2.rds.amazonaws.com

Create endpoint:

B, Target endpoint

From the same steps above, create another endpoint: twtechtarget-endpoint-MySQl
From database console, copy the endpoint database name(click open to access details): twtechsourceauroradb.cluster-cbxxxxx8i.us-east-2.rds.amazonaws.com