AWS Storage Gateway AL2023 Migration: Why You Automate or Bleed

On paper this reads like a chore: an operating-system version bumps, you reboot a few appliances, you move on. That framing is what gets fleets in trouble. The real stakes show up when you multiply one routine rebuild by a few hundred gateways sitting under a hard date, and the "chore" becomes a sequencing problem that can quietly lose data.

A colleague pinged me last month with a spreadsheet. Two hundred and forty S3 File Gateways, spread across nine accounts and four Regions, all still on Amazon Linux 2. His question was simple: "We have a deadline, no in-place upgrade, and a change-freeze calendar that says we get maybe six maintenance windows a quarter. How is this not impossible?" He had already priced out doing it by hand and quietly concluded he could not finish in time.

He is not unusual. Amazon Linux 2 reaches end of standard support on June 30, 2026, and after that date every AL2-based Storage Gateway appliance stops receiving software updates, security patches, and bug fixes. There is no in-place upgrade to Amazon Linux 2023.

Each gateway has to be rebuilt: stop the applications, detach the cache disks, launch a fresh AL2023 instance, reattach the disks, fire the migration API, and verify. AWS itself calls doing this manually across a fleet "time-consuming, error-prone, and difficult to scale." To address it, AWS released a migration example inside the Storage Gateway Terraform module in March 2026, paired with an Ansible playbook that orchestrates the disk-swap and the API call.

I work on the S3-compatible side of this world rather than inside AWS, and I have run enough fleet migrations to have a strong opinion about where they actually go wrong. It is almost never the part you fear. The migration API is reliable. The Terraform plan is boring. What kills these projects is sequencing and contact integrity at scale, and that is exactly what the automation does and does not protect you from. So what follows is more an argument about how to spend your maintenance windows than a step-by-step tutorial.

The replacement model is the whole constraint

Read the procedure carefully and one design decision dominates everything: cache preservation. The supported method replaces the instance while keeping the original cache disks and the gateway ID, so cached data does not have to be re-downloaded from Amazon S3. For a gateway fronting a large, warm cache, that is the difference between a 90-minute window and a multi-day re-warm. It is the right default for any latency-sensitive or large-cache deployment.

The cost of that choice is that the migration is fundamentally a sequence of stateful EC2 and EBS operations: stop, detach, attach, migrate, detach again, restart. Sequences are where humans fail under time pressure. AWS quotes 1–2 hours of downtime per gateway for the window and roughly 15–30 minutes for the automated run itself. The gap between those two numbers is your verification and recovery budget. Manual operators tend to spend that budget fixing the previous step instead of checking the next one.

This is why I push back on teams who treat the Terraform module as optional polish. The module exists because the failure mode at scale is operational, not technical: any one engineer can do one gateway by hand. Doing two hundred by hand guarantees that some of them get the steps out of order, and an out-of-order disk-swap is precisely the thing that loses data.

Two phases, and the seam between them is the risk

The solution splits cleanly. Terraform provisions; Ansible executes. Understanding the seam is what keeps you out of trouble.

In phase one, Terraform takes a single required input, the existing `gateway_id`, for example `sgw-12A3456B`, and a helper script (which depends on `jq`) calls the Storage Gateway API to find the underlying EC2 instance. Terraform then reads the VPC, subnet, Availability Zone, security group, SSH key, and root-disk settings straight off that instance and stands up a new AL2023 instance beside it, on the latest Storage Gateway AMI, in the same subnet. It leaves the old instance and its volumes untouched.

That read-only discovery is elegant, but it has a single point of failure worth naming: if the API lookup cannot resolve the gateway to an instance (wrong ID, missing `jq`, or credentials lacking `storagegateway:DescribeGatewayInformation`) the whole run stalls before anything useful happens.

Phase two is the dangerous half. Ansible stops the old instance, detaches every volume, attaches the cache disks (and temporarily the old root) to the new AL2023 instance, triggers the migration over the HTTP API, then detaches the old root and restarts the new instance in its final shape, rejoining Active Directory if the gateway was domain-joined. The reason this belongs in a playbook rather than a runbook is ordering: each step assumes the previous one completed, and the volume classification (root versus cache) has to be exactly right or you attach the wrong disk to the wrong slot.

The prerequisites are not a formality

Most of the genuine pain hides in the checklist you run before `terraform apply`. I treat these as hard gates, because each one maps to a specific stall I have seen.

`CachePercentDirty` is the one that catches people. Treat it as a mandatory gate. If that metric is above zero when volumes detach, the writes that had not yet flushed to S3 are gone, and no amount of automation recovers them. The playbook checks it and warns, but I would not lean on the warning. Confirm it yourself, and confirm that every application writing to the share is actually stopped rather than nominally stopped.

One correction worth making, because the wrong version circulates: the IAM action the migration genuinely needs is `storagegateway:DescribeGatewayInformation`. If your lookup fails with "Can't find EC2 instance ID," check that permission, the gateway ID, and `jq`, in that order. Do not go hunting for exotic `ec2` mutate permissions the procedure never asks for.

Client connectivity is the silent outage

Here is the failure that does not show up in any Terraform output and ruins the morning after. The new AL2023 instance comes up with a new private IP address. If your gateway is referenced by neither a DNS name nor an Elastic IP, every client has to remount its file shares against the new IP after migration, NFS and SMB alike. You will have "successfully migrated" and simultaneously cut off every consumer of the share.

The fix is to decide your addressing before you touch the module. If clients reach the gateway by DNS name, update the record to the new instance's IP after migration and resolvers do the rest, no remount. If you use an Elastic IP, disassociate it from the old instance and associate it with the new one once migration completes. This is a thirty-second decision that, skipped, becomes a fleet-wide remount fire drill. I rank it above the technical steps in importance precisely because the automation will not save you from it: the playbook migrates the gateway, but it does not touch your DNS hygiene.

Rollback has a hard boundary, and you should know which side you are on

The recoverability of the whole operation hinges on one line in the sequence: the migration API call.

Before that call, steps 1 through 9 of the playbook, nothing destructive has happened. The old instance is stopped and its volumes are detached, but they can be reattached to the old instance at their original device paths, and the gateway comes back as it was. The playbook even writes a `migration-volumes-<epoch>.txt` file recording the original mappings so you can reverse the move. Fully reversible.

After that call, step 10 and beyond, you are in automatic-recovery territory only. Rerun the playbook and it detects that volumes already moved to the new instance, skips the detach/attach, and retries the API directly. There is no "put it back the way it was" anymore. This is also why the cleanup guidance is emphatic: do not run `terraform destroy` after a successful migration, because that deletes your new AL2023 gateway. Use destroy only to tear down a *failed* attempt you intend to roll back.

If you are scripting your own wrapper around this, the single most valuable thing you can log is which side of the API call each gateway is on at any moment. When a window runs long and someone has to make a call, "pre-API, reversible" versus "post-API, retry-only" is the entire decision.

A defensible rollout, without the false precision

I will not give you a magic number for how many gateways to migrate in parallel, because anyone who does is inventing it. The real constraints, though, are concrete. Two stalled instances running at once, old not yet cleaned up and new not yet verified, cost you double compute and risk an Availability-Zone mismatch on volume reattachment, which is one of the documented failure modes (cache volumes must be in the same AZ as the new host).

So the discipline is this: parallelize across Regions and accounts where blast radius is isolated, and keep concurrency within a single Region low enough that one human can still intervene if the playbook hangs on a volume attach. Match the concurrency to how many failures your on-call can babysit in one window. Do not match it to a calendar date someone pulled from the air.

The deeper point is worth keeping in view. At fleet scale the bottleneck stops being any single migration and becomes the cumulative debt of your maintenance windows. Two hundred gateways at 15–30 minutes of active automation each is manageable. The hard part is that each of those two hundred gateways needs a scheduled, change-approved, client-coordinated window. Treat this like a delivery pipeline with dependency tracking instead of a checklist you grind through, and the deadline becomes arithmetic instead of a cliff.