WES custom hosting

Removing Dead Files During an S3 Publish

Purpose

Update a static website in S3 without temporarily breaking pages, while safely removing files that are no longer included in the latest WES publish.

This process replaces the current external ZIP-unpacking step with a controlled CodePipeline and CodeBuild deployment.

Current Publish Flow

The recommended AWS flow is:

WES
  ↓
Versioned incoming S3 bucket
  ↓
CodePipeline
  ↓
CodeBuild
  ↓
Customer serving bucket

WES is responsible for producing and uploading the ZIP. AWS handles validation, deployment, and cleanup.

Recommended Cleanup Method: Manifest-Based Deletion

Each publish should produce a manifest containing every live object key.

Example:

index.html
404/index.html
css/site.123abc.css
js/site.456def.js
images/logo.svg
about/index.html

During deployment:

  1. Generate a manifest from the new package.
  2. Upload the new site.
  3. Store the new manifest.
  4. Compare it with the previous manifest.
  5. Identify keys that existed previously but do not exist now.
  6. Wait through the grace period.
  7. Delete only those orphaned keys.

This is safer than blindly running aws s3 sync --delete because the cleanup process has an explicit list of expected live files.

CodePipeline Structure

Stage 1: Source

Configure an S3 source action using the bucket and key where WES uploads the project ZIP.

The source bucket must have versioning enabled.

Example source key:

incoming/boldo/project.zip

Use CodePipeline when that object is created or updated.

Stage 2: Deploy

Use CodeBuild to:

  1. Validate the source artifact.
  2. Generate the new manifest.
  3. Upload non-HTML files.
  4. Upload HTML files.
  5. Save the new manifest.
  6. Queue or perform delayed orphan cleanup.

CodePipeline passes source output as an artifact to downstream actions. CodeBuild makes the extracted artifact contents available in its source directory. Confirm whether the project files appear directly in $CODEBUILD_SRC_DIR or inside a top-level project directory before finalizing the build commands.

CodeBuild IAM Permissions

The CodeBuild service role needs:

Source bucket

s3:GetObject
s3:GetObjectVersion
s3:ListBucket

Serving bucket

s3:GetObject
s3:PutObject
s3:ListBucket
s3:DeleteObject

DeleteObject is only required for the cleanup step.

Limit the resources to the specific incoming and serving bucket prefixes.

Example Deployment Buildspec

This example assumes CodePipeline has extracted the website package directly into $CODEBUILD_SRC_DIR.

env:
  variables:
    SERVE_BUCKET: "customer-site-bucket"
    DEPLOYMENT_PREFIX: ".wes"

phases:
  pre_build:
    commands:
      - set -euo pipefail
      - SITE_DIR="$CODEBUILD_SRC_DIR"

      # Validate required files before changing production.
      - test -f "$SITE_DIR/index.html" || { echo "Missing index.html"; exit 1; }
      - test -f "$SITE_DIR/404/index.html" || { echo "Missing 404/index.html"; exit 1; }

      # Generate a normalized manifest of the new site.
      - |
        cd "$SITE_DIR"
        find . -type f \
          ! -path "./.wes/*" \
          -print \
          | sed 's#^\./##' \
          | LC_ALL=C sort \
          > /tmp/new-manifest.txt

      # Download the previous manifest when one exists.
      - |
        aws s3 cp \
          "s3://$SERVE_BUCKET/$DEPLOYMENT_PREFIX/current-manifest.txt" \
          /tmp/previous-manifest.txt \
          --only-show-errors \
          || touch /tmp/previous-manifest.txt

  build:
    commands:
      # Step 1: Upload assets first. Do not delete anything.
      - |
        aws s3 sync "$SITE_DIR" "s3://$SERVE_BUCKET" \
          --exclude "*" \
          --include "*.css" \
          --include "*.js" \
          --include "*.mjs" \
          --include "*.json" \
          --include "*.xml" \
          --include "*.txt" \
          --include "*.svg" \
          --include "*.png" \
          --include "*.jpg" \
          --include "*.jpeg" \
          --include "*.gif" \
          --include "*.webp" \
          --include "*.avif" \
          --include "*.ico" \
          --include "*.woff" \
          --include "*.woff2" \
          --include "*.ttf" \
          --include "*.otf" \
          --cache-control "public,max-age=31536000,immutable" \
          --only-show-errors

      # Step 2: Upload any remaining non-HTML files.
      - |
        aws s3 sync "$SITE_DIR" "s3://$SERVE_BUCKET" \
          --exclude "*.html" \
          --exclude "$DEPLOYMENT_PREFIX/*" \
          --only-show-errors

      # Step 3: Flip HTML files after their dependencies exist.
      - |
        aws s3 sync "$SITE_DIR" "s3://$SERVE_BUCKET" \
          --exclude "*" \
          --include "*.html" \
          --content-type "text/html; charset=utf-8" \
          --cache-control "public,max-age=60,must-revalidate" \
          --only-show-errors

  post_build:
    commands:
      # Calculate files that existed previously but are absent now.
      - |
        comm -23 \
          /tmp/previous-manifest.txt \
          /tmp/new-manifest.txt \
          > /tmp/orphaned-files.txt

      # Save deployment records before deleting anything.
      - DEPLOYMENT_ID="$(date -u +%Y%m%dT%H%M%SZ)"
      - |
        aws s3 cp \
          /tmp/new-manifest.txt \
          "s3://$SERVE_BUCKET/$DEPLOYMENT_PREFIX/manifests/$DEPLOYMENT_ID.txt" \
          --content-type "text/plain" \
          --cache-control "no-store" \
          --only-show-errors

      - |
        aws s3 cp \
          /tmp/new-manifest.txt \
          "s3://$SERVE_BUCKET/$DEPLOYMENT_PREFIX/current-manifest.txt" \
          --content-type "text/plain" \
          --cache-control "no-store" \
          --only-show-errors

      - |
        aws s3 cp \
          /tmp/orphaned-files.txt \
          "s3://$SERVE_BUCKET/$DEPLOYMENT_PREFIX/orphans/$DEPLOYMENT_ID.txt" \
          --content-type "text/plain" \
          --cache-control "no-store" \
          --only-show-errors

      - echo "Publish complete. Orphan cleanup should run after the grace period."

Delayed Cleanup Job

Run cleanup separately from the main deployment.

Good options include:

  • A second CodeBuild project invoked after a delay
  • A scheduled Lambda function
  • An EventBridge Scheduler job
  • A scheduled cleanup pipeline

The cleanup job should read the orphan file generated by the deployment and delete those keys.

Example shell logic:

set -euo pipefail

SERVE_BUCKET="customer-site-bucket"
ORPHAN_FILE="/tmp/orphaned-files.txt"

while IFS= read -r key; do
  [ -z "$key" ] && continue

  case "$key" in
    .wes/*)
      echo "Skipping protected key: $key"
      continue
      ;;
  esac

  aws s3 rm \
    "s3://$SERVE_BUCKET/$key" \
    --only-show-errors
done < "$ORPHAN_FILE"

Only process orphan lists that are older than the configured grace period.

Simpler Alternative: Delayed sync --delete

A separate cleanup job can unpack the same deployment and run:

aws s3 sync site "s3://$SERVE_BUCKET" \
  --delete \
  --exclude ".wes/*" \
  --only-show-errors

This is simpler but less controlled.

Use it only when:

  • The bucket or prefix contains only generated website files.
  • The cleanup runs after a grace period.
  • Operational files are explicitly excluded.
  • The deployment package is retained and known to be complete.

Do not run this command in the primary publish pass.

Rollback

Keep:

  • Versioning enabled on the incoming source bucket
  • Previous project ZIPs
  • Historical manifests
  • Serving-bucket versioning when practical

To roll back:

  1. Select the previous source ZIP version.
  2. Run the deployment again.
  3. Upload its assets.
  4. Flip its HTML files back into place.
  5. Generate a new cleanup comparison.

Do not immediately restore deleted assets solely by reverting HTML. The previous package should be redeployed so all of its required assets are present before its HTML goes live.

Cache-Control Guidance

Fingerprinted files

Use:

Cache-Control: public,max-age=31536000,immutable

This is appropriate only when changing the file contents also changes the filename.

HTML

Use:

Cache-Control: public,max-age=60,must-revalidate

A short HTML cache lifetime helps browsers discover the new site promptly and limits the required cleanup grace period.

Non-fingerprinted assets

Do not automatically assign a one-year immutable cache policy to files such as:

robots.txt
sitemap.xml
favicon.ico
manifest.json

These files can retain the same key while their contents change. Give them a shorter cache policy or handle them separately.

Reliability Checklist

  • Incoming S3 bucket has versioning enabled.
  • CodePipeline is triggered by the intended ZIP key.
  • Build fails when index.html is absent.
  • No deletion occurs during the primary publish.
  • A new manifest is generated for every publish.
  • Previous and current manifests are compared.
  • Operational prefixes are protected.
  • Cleanup runs only after a grace period.
  • Cleanup has access to the exact orphan list.
  • Previous ZIP versions are retained for rollback.
  • Fingerprinted and non-fingerprinted files receive appropriate cache headers.

Recommended Implementation

Use the manifest-based process rather than relying exclusively on aws s3 sync --delete.

It provides:

  • An auditable list of deleted files
  • Protection for non-site objects
  • Safer rollback
  • A configurable cleanup delay
  • A clear separation between publishing and destructive cleanup
  • Reliable removal of old Webflow pages and fingerprinted assets