Wednesday, July 2, 2025

AWS Snowball | An Offline Device For Secure, Large-Scale Data Migration.

 

AWS Snowball is a physical device designed for secure, large-scale data transfers between twtech on-premises infrastructure and AWS cloud.

It helps move large datasets efficiently without saturating twtech network. Here’s a concise overview of data migrations using AWS Snowball:

 1. Use Cases for Snowball Data Migration

  • Large data transfers (typically over 10 TB)
  • Limited or slow internet connectivity
  • One-time migrations or recurring batch transfers
  • Migrations to Amazon S3, EBS Snapshots, or into an Amazon Glacier vault

 2. Types of Snowball Devices

Device

Capacity (usable)

Compute Capabilities

Notes

Snowball Edge Storage Optimized

~80 TB

Optional EC2, Lambda

For large-scale data transfer and local processing

Snowball Edge Compute Optimized

~42 TB

Enhanced EC2, GPU, Lambda

For edge computing with local analytics/ML

Snowcone

8 TB

Lightweight compute

Small scale, portable, can run offline

 3. Migration Workflow

  1. Create a Snowball job in AWS Snow Family Management Console.
  2. AWS ships the device to twtech location (encrypted and tamper-resistant).
  3. Connect the device to twtech local network via Ethernet.
  4. Use Snowball client tools (Snowball CLI, S3 SDK Adapter) to transfer data.
  5. Ship the device back to AWS using the pre-attached label.
  6. AWS uploads the data to twtech specified S3 bucket.
  7. Data is automatically deleted from the device after successful transfer.

 4. Data Security

  • 256-bit encryption with AWS Key Management Service (KMS)
  • Tamper-evident enclosures
  • Trusted Platform Module (TPM) for hardware-based key protection
  • Chain of custody tracking

 5. Best Practices

  • Use Snowball Edge for compute-intensive tasks (e.g., preprocessing before upload)
  • Verify checksums before and after data transfer
  • Parallelize uploads using multiple workers
  • Organize data to optimize transfer (e.g., avoid too many small files)

 6. Alternatives & Comparisons

Option

Best For

Speed

Limits

Snowball

10s of TBs+

Days (shipping + transfer)

Physical shipping time

Direct Connect

Continuous large data pipelines

Fastest (Gbps-level)

Setup cost & time

AWS DataSync

Online migrations (up to 10s of TBs)

Slower than Snowball

Dependent on network

S3 Transfer Acceleration

Internet-based S3 uploads

Faster than standard S3

Extra cost, depends on latency


No comments:

Post a Comment

Kubernetes Clusters | Upstream Vs Downstream.

  The terms "upstream" and "downstream" in the context of Kubernetes clusters often refer to the direction of code fl...