A Production Cookbook for Managing Apache Flink¶

Executive Summary¶

A “cookbook” for managing Flink in production is essentially a curated set of short, repeatable runbooks (“recipes”) that cover the lifecycle of Flink applications: deploying, operating, diagnosing, and evolving them safely.

• It is organized by operational scenario (e.g., “Roll a Flink job upgrade with savepoints,” “Recover from failed checkpointing,” “Backfill data for a job”) rather than by API or feature.
• Each recipe follows a standard template: Context, Preconditions, Steps, Validation, Rollback, and Gotchas, so SREs and data engineers know exactly what to do during incidents or changes.
• The cookbook spans cluster management, job lifecycle & state, resource management & scaling, observability & SLOs, data quality & compatibility, and common incident handling.
• We may want to start small (10–15 critical recipes) and evolve it over time as new classes of incidents or operational patterns emerge.
• If you’re running Flink on Kubernetes/Flink Operator or on a managed platform (e.g., Confluent Cloud Flink), the core recipes are similar; the concrete commands and UIs differ, but the cookbook structure remains the same.

Audience¶

A Flink production cookbook is written primarily for:

• Platform / SRE teams who own the Flink clusters or managed Flink accounts.
• Data engineers / stream app owners who own the logic of the jobs and must deploy, tune, and troubleshoot them.
• On-call responders who need precise, low-ambiguity steps during incidents.

Scope of the Cookbook¶

The cookbook is not documentation for “how to write Flink code.” It assumes jobs already exist and focuses on Day-2 operations:

• Provisioning / Cluster ops: Spinning up clusters, upgrading Flink versions, adjusting HA settings, etc.
• Job lifecycle & state: Deploying jobs, restarting, upgrading with/without state, managing savepoints and checkpoints.
• Resources & scaling: Tuning parallelism, memory, slots, auto-scaling strategies, backpressure handling.
• Observability: Metrics, logs, traces, alerts, SLOs, debugging performance issues.
• Data & schema: Handling schema evolution (e.g., with Kafka/Confluent Schema Registry), reprocessing, backfills.
• Failure handling: Job failures, checkpoint failures, state corruption, external system outages.

For each of the chapter / section we will try to address Apache Flink, Confluent Platform and Confluent Cloud for Flink.