Skip to content

Lab: Setup Shift_left tool

This Lab focuses on how to get started withthe shift_left tool. This tools is not yet a public pipy module so cloning the repository and do some minimum installation are needed.

To install the CLI, which is based on Python, use a Python virtual environment like venv and a Python module manafer like pip or uv. All the development is based on uv so we recommend to use it.

1 - Preparation

We assume you have a Confluent Cloud account, an environment sets up, a Kafka Cluster available.

  1. Clone this repository (this will not be necessary once the CLI will be available in pypi): 

    git clone  https://github.com/jbcodeforce/shift_left_utils.git
cd shift_left_utils

  2. On Windows - enable WSL2. The shift_left tool was developed on Mac and tested on Linux. Windows WSL should work. Powershell will not work as of now (11/2025).

  3. All Platforms - install git

  4. make is used to encapsulate the confluent cli, to make it easier for Data Engineers to deploy Flink statement during development: It is not used by shift_left tool, but shift_left creates the Makefile with the shift_left table init command (see the recipe section).

  5. All Platforms - install confluent cli. This is not mandatory to run shift_left tool only if you want to use make.

  6. Create virtual environment:

  • Once uv installed, verify the version. It should be fine to be greater or equal to 0.9.13 

    uv self version
> ...uv 0.9.13

  • Create a new virtual environment in any folder: 

    # can be in the `shift_left` folder
uv venv --python 3.12
    This will create a .venv folder.

  • Activate the environment: 
    source .venv/bin/activate
  • Verify the list of wheels available in the src/shift_left/dist/ to take the last one. The github also includes the list of releases to see the last available version number.
  • Install the cli: 
    uv tool list
# in case it was already installed
uv tool uninstall shift_left
# The weel number may change!
ls src/shift_left/dist
 # >.. shift_left-0.1.46-py3-none-any.whl
uv tool install src/shift_left/dist/shift_left-0.1.46-py3-none-any.whl
  • Install python 3.12.xx
  • Create a Python virtual environment: 
    python -m venv .venv
  • Activate the environment: 
    source .venv/bin/activate
  • Be sure to use the pip install in the virtual environment: 
    python -m pip --version
# if pip not present
python -m ensurepip
# install requirements
python -m pip install -r requirements.txt
  • Install the shift_left CLI using the command (this is temporary once the CLI will be loaded to pypi): To get the list of version of the available wheels, look at the content under the src/shift_left/dist folder.
    ls src/shift_left/dist
# >.. shift_left-0.1.45-py3-none-any.whl
# install the last one for example:
pip install src/shift_left/dist/shift_left-0.1.45-py3-none-any.whl

2- A Demonstration Project

  1. Clone a git repository with data as a product project done in Flink and in Spark. This project includes different demonstrations, but we will use the Customer c360 analytics project to support the current labs. 
    cd ..
git clone https://github.com/jbcodeforce/flink_project_demos

3- Get Confluent Cloud Information

The shift_left tool will access Confluent Cloud REST endpoint and will do operations on behalf of your Confluent user. You need to set environment variables and config file.

  1. Get the shift_left config file template to the tutorial folder: 
    # From the `shift_left_utils` folder
cp src/shift_left/src/shift_left/core/templates/min_config_tmpl.yaml tutorial/setup/config.yaml

3.1 Get Environment id from Confluent Cloud

Go to Environments from the left menu, select the environment you want to use

Select the Details tab for the ID, Provider and region

and update the config.yaml 

 confluent_cloud:
    environment_id:  env-<TO_FILL>


confluent login
confluent environment list

     ID     |           Name           | Stream Governance Package |
------------+--------------------------+----------------------------
 env-nxxxx  | j9r-env                  | ADVANCED
and update the config.yaml 
 confluent_cloud:
    environment_id:  env-<TO_FILL>

3.2 Get Organization ID

Go to the contextual menu on the top right side, and then Organization settings, then copy the ID and update the config.yaml 

 confluent_cloud:
    organization_id: <TO_FILL>


confluent login
confluent organization list
Copy the ID and update the config.yaml 
 confluent_cloud:
    organization_id: <TO_FILL>

3.3 Get Kafka Cluster Information

Go to Environments > Clusters > Select on Kafka Cluster > Cluster Settings > General tab and update the config.yaml 

kafka:
   bootstrap.servers: <To FILL with endpoint>
   cluster_id: <TO_FILL WITH Cluster ID>
confluent_cloud:
    region:  <TO_FILL with Region>
    provider:  <TO_FILL with Provider>
flink:
    catalog_name: <TO FILL with environment_name>
    database_name: <TO FILL with cluster_name>

For the cloud provider and region 

confluent kafka cluster list
    ID     |   Name     |  Cloud | Region    |
-----------+------------+--------+-----------+
lkc-3xxxxx | j9r-kafka  | aws    | us-west-2 |
and update the config.yaml 
kafka:
   bootstrap.servers: <To FILL with endpoint>
   cluster_id: <TO_FILL WITH ID>
confluent_cloud:
    region:  <TO_FILL with Region>
    provider:  <TO_FILL with Cloud>
flink:
    catalog_name: <TO FILL with environment_name>
    database_name: <TO FILL with cluster_name>

4- Get Confluent Cloud Keys and Secrets

4.1 Confluent Cloud API Keys and Secrets

Go to the contextual menu on the top right side, and then Add API Key button:

Select User or Service Account

Then resource scope

Set a name and description and download the created API.

List existing keys for your user 

 confluent api-key list | grep username

Create a Cloud api 

confluent api-key create --resource cloud

  • Edit the tutorial/setup/set_env_var file for the following variables: 
    export SL_CONFLUENT_CLOUD_API_KEY=
export SL_CONFLUENT_CLOUD_API_SECRET=

4.2 Kafka Cluster API Keys and Secrets

Go to Environments > Cluster > API Keys > Select the Create

List existing keys for your user 

 confluent api-key list | grep username | grep kafka

Create a Kafka api for a cluster id: 

confluent api-key create  --resource lkc-123456

  • Edit the tutorial/setup/set_env_var file for the following variables: 
    export SL_KAFKA_API_KEY=
export SL_KAFKA_API_SECRET=

Go to Environments > Flink > API Keys tab

List existing keys for your user 

 confluent api-key list | grep username | grep kafka

Create a Kafka api for a cluster id: 

confluent api-key create  --resource lkc-123456

  • Edit the tutorial/setup/set_env_var file for the following variables: 
    export SL_FLINK_API_KEY=
export SL_FLINK_API_SECRET=

5- Finalize environment variables

Shift_left uses the CONFIG_FILE and PIPELINES environment variables all the time. So it is important to set them correctly. The following setting should work for the different labs of this tutorial.

export FLINK_PROJECT=$PWD/../flink_project_demos/customer_360/c360_flink_processing
export PIPELINES=$FLINK_PROJECT/pipelines
export STAGING=$FLINK_PROJECT/staging
export SRC_FOLDER=$FLINK_PROJECT/../c360_spark_processing
export CONFIG_FILE=$PWD/tutorial/setup/config.yaml
  • Set your terminal shell with those environment variables: 
    source tutorial/setup/set_env_var

6- Validate the configuration

The shift_left CLI works with 3 top level commands: project, table and pipeline. With the project the validate_config helps to debug the config.yaml specified by the CONFIG_FILE environment variables.

shift_left version
shift_left project validate_config

A CONFIG_FILE matches a specific environment, kafka cluster and Flink. So when working with different environments or different Kafka Clusters within an environment, it is recommended to use different config.yaml and environment variables.

Security access

The config.yaml and environment variable files should ignored in Git.