Graceful Shutdown in Go: Draining, Context, and Signals #Software_Engineering

Thuan Nguyen · August 4, 2026

When a process orchestrator like Kubernetes, systemd, or Docker stops a service container, it sends an OS termination signal (SIGTERM). If an application exits instantly upon receiving SIGTERM, in-flight HTTP requests fail with broken connections or 502 Bad Gateway errors at the ingress proxy, database transactions are aborted mid-statement, and background cleanup jobs never execute.

A graceful shutdown flow allows an application to:

  1. Stop accepting new incoming requests.
  2. Allow active HTTP requests and background tasks to complete within a bounded grace period.
  3. Cleanly close underlying resources like database connection pools, cache clients, and message queues.
  4. Exit with status code zero when finished, or force-terminate if the timeout expires.

Here is a step-by-step guide to building a production-ready graceful shutdown pattern in Go using standard library primitives.

1. Capturing Termination Signals with signal.NotifyContext

Before Go 1.16, signal handling required manually creating a buffered channel of os.Signal and calling signal.Notify. Introduced in Go 1.16, signal.NotifyContext simplifies this by returning a child context.Context that automatically cancels when specified OS signals arrive.

package main

import (
	"context"
	"fmt"
	"os/signal"
	"syscall"
)

func main() {
	// Create a context that cancels on SIGINT (Ctrl+C) or SIGTERM (Kubernetes/systemd)
	ctx, stop := signal.NotifyContext(context.Background(), syscall.SIGINT, syscall.SIGTERM)
	defer stop()

	<-ctx.Done()
	fmt.Println("Shutdown signal received")
}

Why defer stop() Is Essential

Calling stop() unregisters the signal handler and frees associated runtime resources. Crucially, calling stop() restores the operating system default signal behavior.

If a service gets stuck during shutdown, a second SIGINT or SIGTERM sent by a developer or orchestrator will immediately trigger the default OS behavior (terminating the process instantly). Always defer stop() right after creating the signal context.

2. Draining HTTP Servers with http.Server.Shutdown

Go’s standard library net/http package provides http.Server.Shutdown(ctx context.Context) (added in Go 1.8).

When Shutdown is called:

  • It closes all active listeners immediately so new TCP connections are refused.
  • It closes all idle connections.
  • It waits for active in-flight requests to complete.
  • If the provided timeout context expires before all requests complete, Shutdown returns the context error (context.DeadlineExceeded).

Simultaneously, calling Shutdown causes ListenAndServe to return http.ErrServerClosed immediately. A robust server must check for http.ErrServerClosed to distinguish a graceful shutdown from an unexpected server startup failure.

package main

import (
	"context"
	"errors"
	"log"
	"net/http"
	"os/signal"
	"syscall"
	"time"
)

func main() {
	// 1. Set up signal context
	ctx, stop := signal.NotifyContext(context.Background(), syscall.SIGINT, syscall.SIGTERM)
	defer stop()

	mux := http.NewServeMux()
	mux.HandleFunc("/healthz", func(w http.ResponseWriter, r *http.Request) {
		w.WriteHeader(http.StatusOK)
		w.Write([]byte("ok"))
	})

	srv := &http.Server{
		Addr:    ":8080",
		Handler: mux,
	}

	// 2. Start HTTP server in a separate goroutine
	go func() {
		if err := srv.ListenAndServe(); err != nil && !errors.Is(err, http.ErrServerClosed) {
			log.Fatalf("HTTP server failed: %v", err)
		}
	}()

	// 3. Block until OS signal is received
	<-ctx.Done()
	log.Println("Shutting down HTTP server...")

	// 4. Create a hard timeout for connection draining (e.g., 10 seconds)
	shutdownCtx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
	defer cancel()

	if err := srv.Shutdown(shutdownCtx); err != nil {
		log.Printf("HTTP shutdown error: %v", err)
	} else {
		log.Println("HTTP server stopped cleanly")
	}
}

Note: http.Server.Shutdown does not automatically close hijacked connections such as WebSockets or gRPC streams. If your app uses WebSockets, register shutdown callbacks using srv.RegisterOnShutdown(fn) to send close frames to active WebSocket clients.

3. Propagating Cancellation to Background Workers

In addition to HTTP handlers, production services often run worker routines (e.g., polling queues or processing async jobs). Workers should accept a context.Context and monitor ctx.Done().

Use sync.WaitGroup to track active worker goroutines and ensure the main function waits for workers to finish before exiting.

package main

import (
	"context"
	"fmt"
	"sync"
	"time"
)

func runWorker(ctx context.Context, id int, wg *sync.WaitGroup) {
	defer wg.Done()
	ticker := time.NewTicker(500 * time.Millisecond)
	defer ticker.Stop()

	for {
		select {
		case <-ctx.Done():
			fmt.Printf("Worker %d received cancellation, finishing active item...\n", id)
			return
		case t := <-ticker.C:
			fmt.Printf("Worker %d processed job at %s\n", id, t.Format("15:04:05"))
		}
	}
}

Preserving Cleanup Work with context.WithoutCancel

When a parent context is canceled due to shutdown, passing that canceled context to downstream operations like audit logging or releasing distributed locks will fail immediately with context.Canceled.

Go 1.21 introduced context.WithoutCancel(parentCtx). It returns a copy of the parent context that retains all key-value data but detaches from the parent’s cancellation tree and deadline:

func cleanupResource(ctx context.Context, resourceID string) {
	// Create a context that inherits parent values but ignores parent cancellation
	detachedCtx, cancel := context.WithTimeout(context.WithoutCancel(ctx), 3*time.Second)
	defer cancel()

	// Operations using detachedCtx will succeed even if the main application context is canceled
	saveAuditLog(detachedCtx, resourceID, "shutdown_cleaned")
}

4. Resource Shutdown Ordering: The Lifecycle Chain

Order matters during shutdown. Closing a database connection pool before the HTTP server finishes draining in-flight requests will cause active HTTP handlers to fail with sql: database is closed.

The correct shutdown sequence is:

  1. Signal Catching: Intercept SIGINT/SIGTERM via signal.NotifyContext.
  2. Ingress Termination & HTTP Draining: Call http.Server.Shutdown. This stops new connections and waits for active HTTP handlers to return.
  3. Background Worker Draining: Signal worker contexts to stop taking work and call wg.Wait().
  4. Dependency Teardown: Close database pools, Redis clients, gRPC connections, and log flushers.
  5. Exit: Return main or call os.Exit(0).
[OS Signal (SIGTERM)]
        |
        v
1. Cancel Signal Context
        |
        v
2. srv.Shutdown(shutdownCtx)  --> Stops accepting new HTTP requests & drains active HTTP calls
        |
        v
3. Worker Pool Context Cancel --> Signals background workers to finish in-flight jobs (wg.Wait())
        |
        v
4. Close Resources           --> db.Close(), redis.Close(), logger.Sync()
        |
        v
5. Exit (Status 0)

5. Complete Production Worked Example

Here is a complete, executable Go program demonstrating HTTP server draining, worker pool coordination, and ordered resource cleanup.

package main

import (
	"context"
	"errors"
	"log"
	"net/http"
	"os/signal"
	"sync"
	"syscall"
	"time"
)

type Database struct {
	closed bool
}

func (db *Database) Close() error {
	db.closed = true
	return nil
}

func main() {
	// 1. Create root signal context for SIGINT and SIGTERM
	ctx, stop := signal.NotifyContext(context.Background(), syscall.SIGINT, syscall.SIGTERM)
	defer stop()

	db := &Database{}

	// 2. Set up HTTP router and server
	mux := http.NewServeMux()
	mux.HandleFunc("/work", func(w http.ResponseWriter, r *http.Request) {
		if db.closed {
			http.Error(w, "Database unavailable", http.StatusServiceUnavailable)
			return
		}
		// Simulate brief request processing time
		select {
		case <-time.After(2 * time.Second):
			w.WriteHeader(http.StatusOK)
			w.Write([]byte("Job completed successfully\n"))
		case <-r.Context().Done():
			http.Error(w, "Client disconnected", http.StatusGatewayTimeout)
		}
	})

	srv := &http.Server{
		Addr:    ":8080",
		Handler: mux,
	}

	// 3. Start HTTP server
	go func() {
		log.Println("Starting HTTP server on :8080...")
		if err := srv.ListenAndServe(); err != nil && !errors.Is(err, http.ErrServerClosed) {
			log.Fatalf("Server unexpected error: %v", err)
		}
	}()

	// 4. Start background workers
	var wg sync.WaitGroup
	wg.Add(1)
	go func() {
		defer wg.Done()
		for {
			select {
			case <-ctx.Done():
				log.Println("Worker received shutdown signal, finishing...")
				return
			case <-time.After(1 * time.Second):
				log.Println("Worker heartbeat tick")
			}
		}
	}()

	// 5. Block main goroutine until signal is received
	<-ctx.Done()
	log.Println("Initiating graceful shutdown sequence...")

	// 6. Step A: Drain HTTP Server (10s grace period)
	shutdownCtx, cancelShutdown := context.WithTimeout(context.Background(), 10*time.Second)
	defer cancelShutdown()

	if err := srv.Shutdown(shutdownCtx); err != nil {
		log.Printf("HTTP shutdown timeout or error: %v", err)
	} else {
		log.Println("HTTP server drained and closed successfully")
	}

	// 7. Step B: Wait for background workers to complete
	log.Println("Waiting for background workers to exit...")
	wg.Wait()
	log.Println("All background workers exited")

	// 8. Step C: Close shared infrastructure dependencies
	log.Println("Closing database connections...")
	if err := db.Close(); err != nil {
		log.Printf("Error closing database: %v", err)
	} else {
		log.Println("Database connection closed")
	}

	log.Println("Graceful shutdown completed successfully")
}

Summary Checklist

  • Use signal.NotifyContext: Intercept syscall.SIGINT and syscall.SIGTERM.
  • Always defer stop(): Restores normal OS signal behavior so a second signal can force-kill if stuck.
  • Filter http.ErrServerClosed: Prevent false-alarm failure logs when srv.Shutdown is called.
  • Enforce Grace Period: Pass a bounded context.WithTimeout to srv.Shutdown.
  • Order Teardown Sequentially: Drain HTTP requests and workers BEFORE closing database or storage connections.
  • Use context.WithoutCancel: Execute post-cancellation audit logs or cleanup routines without being killed by parent context cancellation.