Migrating 100 VMs from AWS to On-Premises

When our organization moved to a new facility with a purpose-built server room, we made a decision that surprised even our DevOps team: migrate production workloads from AWS back to on-premises.

Cloud-to-on-prem migrations go against conventional wisdom. Here's why we did it and how it went.

Why Leave AWS?

AWS worked fine. But several factors made on-premises attractive:

Cost predictability. Our AWS bill varied month to month. On-premises is capital expenditure upfront, then predictable operational costs.

Latency. Some workloads are latency-sensitive. Eliminating cloud network hops helped.

We had the facility. The new building included a properly equipped server room. Not using it seemed wasteful.

The DevOps team (myself included) questioned this initially. But the business case was solid once we ran the numbers.

What We Built

Specification	Value
Server CPU	240 Cores
Server RAM	2,200 GB
SSD Storage	64 TB

Three hosts running VMware vSphere, clustered with HA and DRS. Enterprise storage with RAID configurations optimized for our I/O patterns.

What We Migrated

Workload	Count
Virtual Machines	100
Kubernetes Cluster	1
NFS Shares	4

The Process

1. Capacity Planning

We analyzed AWS Cost Explorer to understand our actual resource usage. This informed hardware procurement - we sized for current load plus growth headroom.

2. Physical Setup

Hardware installation, rack mounting, cable management, power connections (grid + UPS). Nothing glamorous, but done properly.

3. Network Configuration

Link aggregation for bandwidth and failover. VLAN segmentation for traffic isolation. Redundant paths everywhere - no single points of failure.

4. vSphere Deployment

Installed ESXi on hosts, configured vCenter, enabled HA and DRS. The cluster handles automatic failover if a host dies.

5. Storage Integration

Dell EMC storage integrated with the cluster. RAID configurations balanced performance and redundancy.

6. Migration Execution

VMs migrated in batches during maintenance windows. We kept AWS running in parallel until each workload was validated on-premises.

7. Backup Strategy

Synology Active Backup for daily VM backups. Off-site replication to S3 for disaster recovery. We didn't abandon cloud entirely - it's still useful for backup.

8. Monitoring

Containerized Grafana deployment for metrics and alerting. We see resource utilization, can alert at 80% thresholds, and catch issues before they impact production.

What I Learned

VMware skills matter. Our team was AWS-focused. vSphere administration was a learning curve. Plan for training time.

Test everything twice. We validated connectivity, failover, and performance at each phase. Found issues early that would have been painful to debug in production.

Documentation saves time. Detailed runbooks for troubleshooting. When something breaks at 2 AM, you don't want to figure it out from scratch.

Hybrid makes sense. We still use S3 for backup replication. Pure on-premises or pure cloud is less practical than using each where it fits.

Was It Worth It?

Yes, for our situation. We had the facility, the workloads justified it, and the cost model worked.

Would I recommend this for everyone? No. If you don't have data center space, expertise, or workloads that benefit from local deployment, cloud is probably better.

Key Takeaways

Cloud-to-on-prem migrations can make sense when the business case is solid
Capacity planning from actual usage data prevents over or under-provisioning
VMware clustering (HA, DRS) provides resilience similar to cloud availability zones
Hybrid approach works - we still use cloud for backups and disaster recovery
Factor in training time if your team is cloud-native
Monitor proactively - alert at 80% utilization, not 100%