Center of Excellence for Stream Processing Establishment¶
CoE is a centralized team or a virtual organization whose mission is to establish and disseminate best practices, provide expert consultation, and accelerate the adoption and successful, standardized deployment for real-time stream processing applications.
Goals¶
Data Streaming CoE needs to address architecture, high availability, DR, security, multi-tenancy, streaming applications deployment best practices, business transformation with streaming capabilities.
In particular the following items need to be addresses and built:
- Goal and Responsibilities: of the CoE team, or community of practices team.
- Roles and Responsibilities: to outline the roles to run streaming applications and platform.
- Governance model: to define the decisions to take when building, maintaining, and growing of the streaming platform. Should include Multi-tenancy, how shared services are charged back to different lines of business.
- Architectural Patterns: To define reference architectures for typical use cases, addressing different DR patterns base on RPO and RTO.
- Access control: To address how to manage user and group access, to data sources, and Flink applications.
- Development best practices: Recommend practices to move to a streaming platform.
- Operations best practices: Reference runbooks to maintain and manage the specific implementation of a streaming data platform. Includes Application savepoints, checkpoints, blue/green deployment, failover, troubleshooting.
- Sizing: Deliver tools and knowledge to size their platform for growth.
- Project plan references: to highlight what are the development and deployment high level activities for a streaming project.
- Community development: to define the communications, collaboration, and engagement strategies to increase awareness and adoption of stream processing.
User Roles¶
There will be light modification of roles when using a managed services, versus managing the platform.