Contribute to this repository¶
This chapter addresses how to support the development of this open source project.
Anyone can contribute to this repository and associated projects.
There are multiple ways to contribute: report bugs and suggest improvements, improve the documentation, and contribute code.
Bug reports, documentation changes, and feature requests¶
If you would like to contribute to the project in the form of encountered bug reports, necessary documentation changes, or new feature requests, this can be done through the use of the repository's Issues list.
Before opening a new issue, please check the existing list to make sure a similar or duplicate item does not already exist. When you create your issues, please be as explicit as possible and be sure to include the following:
-
Bug reports
- Specific project version
- Deployment environment
- A minimal, but complete, setup of steps to recreate the problem
-
Documentation changes
- URL to existing incorrect or incomplete documentation (either in the project's GitHub repo or external product documentation)
- Updates required to correct current inconsistency
- If possible, a link to a project fork, sample, or workflow to expose the gap in documentation.
-
Feature requests
- Complete description of project feature request, including but not limited to, components of the existing project that are impacted, as well as additional components that may need to be created.
- A minimal, but complete, setup of steps to recreate environment necessary to identify the new feature's current gap.
The more explicit and thorough you are in opening GitHub Issues, the more efficient your interaction with the maintainers will be. When creating the GitHub issue for your bug report, documentation change, or feature request, be sure to add as many relevant labels as necessary (that are defined for that specific project). These will vary by project, but will be helpful to the maintainers in quickly triaging your new GitHub issues.
Code contributions¶
We really value contributions, and to maximize the impact of code contributions, we request that any contributions follow the guidelines below. If you are new to open source contribution and would like some more pointers or guidance, you may want to check out Your First PR and First Timers Only. These are few projects that help on-board new contributors to the overall process.
Coding and Pull Requests best practices¶
-
Please ensure you follow the coding standard and code formatting used throughout the existing code base.
- This may vary project by project, but any specific diversion from normal language standards will be explicitly noted.
-
One feature / bug fix / documentation update per pull request
- Always pull the latest changes from upstream and rebase before creating any pull request.
- New pull requests should be created against the
integration
branch of the repository, if available. - This ensures new code is included in full-stack integration tests before being merged into the
main
branch
-
All new features must be accompanied by associated tests.
- Make sure all tests pass locally before submitting a pull request.
- Include tests with every feature enhancement, improve tests with every bug fix
Github and git flow¶
The following core principles for the management of this code is using the gitflow
process with separate main
and develop
branches for a structured release process.
- main Branch: This branch always reflects the production-ready, stable code. Only thoroughly tested and finalized code is merged into
main
. Commits are tagged inmain
with version numbers for easy tracking of releases. - develop Branch: This branch serves as the integration point for all new features and ongoing development. Feature branches are created from
develop
and merged back into it after completion. - Creating a Release Branch: When a set of features in develop is deemed ready for release, a new release branch is created from
develop
. This branch allows for final testing, bug fixes, and release-specific tasks without interrupting the ongoing development work in develop. - Finalizing the Release: Only bug fixes and necessary adjustments are made on the
release
branch. New feature development is strictly avoided. -
Merging and Tagging: Once the release branch is stable and ready for deployment, it's merged into two places:
main
: The release branch is merged into main, effectively updating the production-ready code with the new release.develop
: The release branch is also merged back into develop to ensure that any bug fixes or changes made during the release preparation are incorporated into the ongoing development work.
-
Tagging: After merging into main, the merge commit is tagged with a version number (e.g., v1.0.0) to mark the specific release point in the repository's history.
- Cleanup: After the release is finalized and merged, the release branch can be safely deleted
git commands for the different development tasks¶
-
Create a
feature
branch from thedevelop
branch -
Do your work:
- Write your code
- Write your unit tests and potentially your integration tests
- Pass your tests locally
- Commit your intermediate changes as you go and as appropriate to your
feature
branch - Repeat until satisfied
-
Create a pull request against the same targeted upstream
develop
branch.
Forked repository special treatment
If you forked the main repository, once the pull request has been reviewed, accepted and merged into the develop
branch, you should synchronize your remote and local forked github repository main
branch with the upstream main branch. To do so:
-
Pull to your local forked repository the latest changes upstream (that is, the pull request).
-
Push those latest upstream changes pulled locally to your remote forked repository.
Release management:
-
When code is ready, create a new release branch off the
develop
branch. Run all tests and fixes on this release branch until ready for release. -
Merge into main:
- Tag the main branch:
- Merge into the
develop
branch: - In gihub, create a new release referencing the newly created tag
- Delete the release branch:
Environment set up for developers¶
We are using uv as a new Python package manager. See uv installation documentation then follow the next steps to set your environment for development:
- Fork the
https://github.com/jbcodeforce/shift_left_utils
repo in your github account. - Clone your repo to your local computer.
- Add the upstream repository: see guide for step 1-3 here: forking a repo
-
Verify you have set up the pre-requisited with the below differences:
- Create a new virtual environment in any folder, but could be the :
uv venv
- Create a new virtual environment in any folder, but could be the :
-
Activate the environment:
- Get the credentials for the Confluent Cloud Kafka cluster and Flink compute pool, modify the config.yaml file
the structure of the config.yaml
This file is to keep configuration to access Confluent Cloud resources. It is grouped in 5 parts:
- kafka to access Kafka cluster
- confluent_cloud to manage resources at the organization level
- flink: to manage Statement and compute pools
- app: to define CLI parameters
- registry: to access schema registry (not used)
The Flink, and Confluent Cloud api keys and secrets are different.
- Set the CONFIG_FILE environment variable to point to the config.yaml file. The following is used for integration tests.
Development activities¶
See the code explanation and design approach chapter.
Install dependencies and the cli as uv tool¶
-
Install dependencies and tool for iterative development, under the
src/shift_left
folder (the one with theuv.lock
andpyproject.toml
files) -
Verify it is installed locally (version will differ)
-
Uninstall:
(The version number is specified in pyproject.toml
file)
Testing¶
This section addresses the things to know for running tests for the CLI.
Test projects under data
The tests/data
folder includes two test projects:
1/ flink-project and 2/ dbt-project
The dbt-project
include a dbt project to migrate source SQL from. The flink-project is the outcome of the migration with data engineering tuning the SQL and deploy them with Make.
Those projects are used for unit tests and even to explain how to use the CLI by end user.
- Be sure to set the PIPELINES environment variable to point to the
tests/data/flink-project/pipelines
(be sure to be under thesrc/shift_left
folde):
- Running the cli from python code, be sure to be under the
src/shift_left
folder and use:
- It is also possible to test the CLI with python:
To avoid redundant tests, the tests are grouped in three sets:
- The CLI modules tests: will test the CLI functions and the delegate functions from core
- The core modules tests: to test functions not addressed by CLI tests
- The utils modules tests: to test function not addressed by CLI or core tests
The test should supports debugging and test in Terminal, so all file accesses are relative to the test class.
Tests are executed in a virtual environment with python 3 and pytest.
uv run pytest -s tests/it/core/test_table_mgr.py
uv run pytest -s tests/ut/core/test_project_mgr.py
uv run pytest -s tests/ut/core/test_pipeline_mgr.py
- Test the CLIs
Debug core functions¶
Use the following settings for vscode based IDE
{
"name": "Python Debugger: current file",
"type": "debugpy",
"request": "launch",
"program": "${file}",
"console": "integratedTerminal",
"cwd": "${fileDirname}",
"env": {
"CONFIG_FILE": "${fileWorkspaceFolder}/config.yaml"
},
},
Build¶
-
To install the tool locally with uv and to support hot code update
-
Rebuild the CLI as a wheel packaging:
under the src/shift_left project. -
Deploy the CLI from the wheel: 1/ first the last wheel number and then 2/ execute the commands like:
-
To Build the Command.md documentation from the code run:
-
Recompile a requirements.txt for pip users: