Confluent Platform for Flink¶

The official product documentation after 12/2024 release is here. The main points are:

• Fully compatible with open-source Flink.
• Deploy on Kubernetes using Helm
• Define environment, which group applications
• Deploy application with user interface and task manager cluster
• Exposes custom kubernetes operator for specific CRD

The figure below presents the Confluent Flink components deployed on Kubernetes:

• CFK supports the management of custom CRD, based as of not on the Flink for Kubernetes Operator
• CMF adds security control, and REST API server for the cli or HTTP client
• FKO is the open source Flink for Kubernetes Operator
• Flink cluster are created from command and CRDs and run Flink applications within an environment

Be sure to have confluent cli.

See dedicated session for Kubernetes deployment

• Install the Flink Kubernetes Operator (FKO), to the default namespace. It can take sometime to pull the image.

  helm upgrade --install cp-flink-kubernetes-operator confluentinc/flink-kubernetes-operator --set watchNamespaces={flink}
  

• Install Confluent Manager for Flink using the cmf helm chart to the cpf namespace:

  helm upgrade --install cmf confluentinc/confluent-manager-for-apache-flink --namespace cpf
  kubectl get pods -n cpf
  

• Install Confluent Platform (Kafka) operator for kubernetes:

  helm upgrade --install confluent-operator confluentinc/confluent-for-kubernetes --namespace confluent
  kubectl apply -f cp-kafka.yaml
  kubectl get pods -n  confluent
  

• Do a port forwarding to the CMF REST api, so the Confluent cli can interact with the operator:

  # 
  kubectl get svc -n cpf 
  kubectl port-forward -n cpf svc/cmf-service 8080:80
  

• Create a Flink environment

  confluent flink environment create development --kubernetes-namespace flink --url http://localhost:8080
  

Important source of information for deployment¶

• Deployment overview and for Apache Flink.
• CP Flink supports K8s HA only.
• Flink fine-grained resource management documentation.

Metadata management service for RBAC¶

• Metadata Service Overview
• Single broker Kafka+MDS Deployment
• Git repo with working CP on K8s deployment using RBAC via SASL/Plain and LDAP
• Configure CP Flink to use MDS for Auth
• Additional Flink RBAC Docs
• How to secure a Flink job with RBAC
• Best Practices for K8s + RBAC

Docker local¶

See the cp-all-in-one for local Confluent platform docker compose files with Flink too.

Same cli commands:

confluent flink  environment create myenv --kubernetes-namespace flink --url http://localhost:8080
confluent flink application list --environment myenv --url http://localhost:8080

Monitoring and Troubleshouting¶