Implement zero-downtime deployment with graceful shutdown

[granstel]

Summary

This PR implements a complete zero-downtime deployment system for the service, including graceful shutdown handling, health checks for load balancer integration, and deployment automation scripts.

Key Changes

Deployment Infrastructure

• deploy/rollout.sh: Bash script that orchestrates zero-downtime updates by:

  • Pulling the new image version
  • Starting a new container alongside the old one
  • Waiting for the new instance to report healthy via /health endpoint
  • Gracefully stopping the old instance with traffic drain timeout
  • Cleaning up dangling images

• deploy/docker-compose.yml: Traefik edge proxy configuration that:

  • Terminates TLS and manages Let's Encrypt certificates
  • Routes traffic to service instances based on docker labels
  • Performs health checks on /health endpoint
  • Automatically redirects HTTP to HTTPS

• deploy/env.example and deploy/app-env.example: Configuration templates for deployment parameters and application secrets

• deploy/README.md: Comprehensive deployment documentation covering architecture, setup, and operational procedures

Application Health & Graceful Shutdown

• ReadinessState: Tracks whether the instance is ready to accept traffic

• ReadinessHealthCheck: Health check implementation that reports unhealthy during shutdown

• GracefulShutdownService: Hosted service that:

  • Marks instance as not ready when shutdown signal is received
  • Pauses for DrainDelaySeconds to allow load balancer to remove instance from rotation
  • Allows current requests and background tasks to complete before actual shutdown

• ShutdownConfiguration: Configuration class for drain delay and shutdown timeout values

CI/CD Pipeline

• .github/workflows/docker-publish.yml: Updated to:
  • Build and publish Docker images on every push to master
  • Extract version from .csproj file
  • Tag images with version, commit SHA, and latest
  • Deploy via SSH by executing rollout.sh on the server (when DEPLOY_ENABLED is true)
  • Use concurrency control to prevent parallel deployments

Configuration & Testing

• appsettings.json: Added shutdown configuration section with default drain delay (10s) and timeout (30s)

• AppConfiguration: Added ShutdownConfiguration property

• HealthTests: Integration tests verifying:

  • /health returns 200 OK when running normally
  • /health returns 503 Service Unavailable during shutdown

• Dockerfile: Added curl to base image for health checks

• .gitignore & .gitattributes: Updated to track deployment scripts with LF line endings while excluding secrets and certificates

Implementation Details

The deployment strategy ensures zero downtime by:

1. Running old and new instances simultaneously during updates
2. Using Traefik's load balancing to distribute traffic between healthy instances
3. Gracefully draining traffic from old instance before termination
4. Allowing in-flight requests and background jobs to complete
5. Automatically rolling back if new instance fails health checks

The health check endpoint (/health) is critical to this design—it allows the load balancer to detect when an instance is shutting down and remove it from rotation before the port closes, preventing request loss.

https://claude.ai/code/session_019QThhRUApAXHFb3t4Amaij

[chatgpt-codex-connector]

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.