Kedro plugins¶
Kedro plugins allow you to create new features for Kedro and inject additional commands into the CLI. Plugins are developed as separate Python packages that exist outside of any Kedro project.
Overview¶
Kedro’s extension mechanism is built on pluggy
, a solid plugin management library that was created for the pytest ecosystem. pluggy
relies on entry points, a Python mechanism for packages to provide components that can be discovered by other packages using importlib.metadata
.
Example of a simple plugin¶
Here is a simple example of a plugin that prints the pipeline as JSON:
kedrojson/plugin.py
import click
from kedro.framework.project import pipelines
@click.group(name="JSON")
def commands():
pass
@commands.command()
@click.pass_obj
def to_json(metadata):
"""Display the pipeline in JSON format"""
pipeline = pipelines["__default__"]
print(pipeline.to_json())
The plugin provides the following entry_points
config in setup.py
:
setup(
entry_points={"kedro.project_commands": ["kedrojson = kedrojson.plugin:commands"]}
)
Once the plugin is installed, you can run it as follows:
kedro to_json
Extend starter aliases¶
It is possible to extend the list of starter aliases built into Kedro. This means that a custom Kedro starter can be used directly through the starter
argument in kedro new
rather than needing to explicitly provide the template
and directory
arguments. A custom starter alias behaves in the same way as an official Kedro starter alias and is also picked up by kedro starter list
.
You need to extend the starters by providing a list of KedroStarterSpec
, in this example it is defined in a file called plugin.py
.
Example for a non-git repository starter:
# plugin.py
starters = [
KedroStarterSpec(
alias="test_plugin_starter",
template_path="your_local_directory/starter_folder",
)
]
Example for a git repository starter:
# plugin.py
starters = [
KedroStarterSpec(
alias="test_plugin_starter",
template_path="https://github.com/kedro-org/kedro-starters/",
directory="pandas-iris",
)
]
The directory
argument is optional and should be used when you have multiple templates in one repository as for the official kedro-starters. If you only have one template, your top-level directory will be treated as the template. For an example, see the pandas-iris starter.
In your setup.py
, you need to register the specifications to kedro.starters
.
setup(
entry_points={"kedro.starters": ["starter = plugin:starters"]},
)
After that you can use this starter with kedro new --starter=test_plugin_starter
.
Note
If your starter lives on a git repository, by default Kedro attempts to use a tag or branch labelled with your version of Kedro, e.g. 0.18.10.
. This means that you can host different versions of your starter template on the same repository, and the correct one will automatically be used. If you do not wish to follow this structure, you should override it with the checkout
flag, e.g. kedro new --starter=test_plugin_starter --checkout=main
.
Working with click
¶
Commands must be provided as click
Groups
The click Group
will be merged into the main CLI Group. In the process, the options on the group are lost, as is any processing that was done as part of its callback function.
Project context¶
When they run, plugins may request information about the current project by creating a session and loading its context:
from pathlib import Path
from kedro.framework.startup import _get_project_metadata
from kedro.framework.session import KedroSession
project_path = Path.cwd()
session = KedroSession.create(project_path=project_path)
context = session.load_context()
Initialisation¶
If the plugin initialisation needs to occur prior to Kedro starting, it can declare the entry_point
key kedro.init
. This entry point must refer to a function that currently has no arguments, but for future proofing you should declare it with **kwargs
.
global
and project
commands¶
Plugins may also add commands to the Kedro CLI, which supports two types of commands:
global - available both inside and outside a Kedro project. Global commands use the
entry_point
keykedro.global_commands
.project - available only when a Kedro project is detected in the current directory. Project commands use the
entry_point
keykedro.project_commands
.
Suggested command convention¶
We use the following command convention: kedro <plugin-name> <command>
, with kedro <plugin-name>
acting as a top-level command group. This is our suggested way of structuring your plugin bit it is not necessary for your plugin to work.
Hooks¶
You can develop hook implementations and have them automatically registered to the project context when the plugin is installed. To enable this for your custom plugin, simply add the following entry in your setup.py
:
setup(entry_points={"kedro.hooks": ["plugin_name = plugin_name.plugin:hooks"]})
where plugin.py
is the module where you declare hook implementations:
import logging
from kedro.framework.hooks import hook_impl
class MyHooks:
@hook_impl
def after_catalog_created(self, catalog): # pylint: disable=unused-argument
logging.info("Reached after_catalog_created hook")
hooks = MyHooks()
Note
hooks
should be an instance of the class defining the Hooks.
CLI Hooks¶
You can also develop hook implementations to extend Kedro’s CLI behaviour in your plugin. To find available CLI hooks, please visit kedro.framework.cli.hooks. To register CLI hooks developed in your plugin with Kedro, add the following entry in your project’s setup.py
:
setup(entry_points={"kedro.cli_hooks": ["plugin_name = plugin_name.plugin:cli_hooks"]})
where plugin.py
is the module where you declare hook implementations:
import logging
from kedro.framework.cli.hooks import cli_hook_impl
class MyCLIHooks:
@cli_hook_impl
def before_command_run(self, project_metadata, command_args):
logging.info(
"Command %s will be run for project %s", command_args, project_metadata
)
cli_hooks = MyCLIHooks()
Contributing process¶
When you are ready to submit your code:
Create a separate repository using our naming convention for
plugin
s (kedro-<plugin-name>
)Choose a command approach:
global
and / orproject
commands:All
global
commands should be provided as a singleclick
groupAll
project
commands should be provided as anotherclick
groupThe
click
groups are declared through the entry points mechanism
Include a
README.md
describing your plugin’s functionality and all dependencies that should be includedUse GitHub tagging to tag your plugin as a
kedro-plugin
so that we can find it
Supported Kedro plugins¶
Kedro-Datasets, a collection of all of Kedro’s data connectors. These data connectors are implementations of the
AbstractDataSet
Kedro-Docker, a tool for packaging and shipping Kedro projects within containers
Kedro-Airflow, a tool for converting your Kedro project into an Airflow project
Kedro-Viz, a tool for visualising your Kedro pipelines
Community-developed plugins¶
See the full list of plugins using the GitHub tag kedro-plugin.
Note
Your plugin needs to have an Apache 2.0 compatible license to be considered for this list.
Kedro-Pandas-Profiling, by Justin Malloy, uses Pandas Profiling to profile datasets in the Kedro catalog
find-kedro, by Waylon Walker, automatically constructs pipelines using
pytest
-style pattern matchingkedro-static-viz, by Waylon Walker, generates a static Kedro-Viz site (HTML, CSS, JS)
steel-toes, by Waylon Walker, prevents stepping on toes by automatically branching data paths
kedro-wings, by Tam-Sanh Nguyen, simplifies and speeds up pipeline creation by auto-generating catalog datasets
kedro-great, by Tam-Sanh Nguyen, integrates Kedro with Great Expectations, enabling catalog-based expectation generation and data validation on pipeline run
Kedro-Accelerator, by Deepyaman Datta, speeds up pipelines by parallelizing I/O in the background
kedro-dataframe-dropin, by Zain Patel, lets you swap out pandas datasets for modin or RAPIDs equivalents for specialised use to speed up your workflows (e.g on GPUs)
kedro-mlflow, by Yolan Honoré-Rougé and Takieddine Kadiri, facilitates MLflow integration within a Kedro project. Its main features are modular configuration, automatic parameters tracking, datasets versioning, Kedro pipelines packaging and serving and automatic synchronization between training and inference pipelines for high reproducibility of machine learning experiments and ease of deployment. A tutorial is provided in the kedro-mlflow-tutorial repo. You can find more information in the kedro-mlflow documentation.
Kedro-Neptune, by Jakub Czakon and Rafał Jankowski, lets you have all the benefits of a nicely organized Kedro pipeline with Neptune: a powerful user interface built for ML metadata management. It lets you browse and filter pipeline executions, compare nodes and pipelines on metrics and parameters, and visualize pipeline metadata like learning curves, node outputs, and charts. For more information, tutorials and videos, go to the Kedro-Neptune documentation.
kedro-dolt, by Max Hoffman and Oscar Batori, allows you to expand the data versioning abilities of data scientists and engineers
kedro-kubeflow, by GetInData, lets you run and schedule pipelines on Kubernetes clusters using Kubeflow Pipelines
kedro-airflow-k8s, by GetInData, enables running a Kedro pipeline with Airflow on a Kubernetes cluster
kedro-vertexai, by GetInData, enables running a Kedro pipeline with Vertex AI Pipelines service
kedro-azureml, by GetInData, enables running a Kedro pipeline with Azure ML Pipelines service
kedro-sagemaker, by GetInData, enables running a Kedro pipeline with Amazon SageMaker service
kedro-partitioned, by Gabriel Daiha Alves and Nickolas da Rocha Machado, extends the functionality on processing partitioned data.
kedro-auto-catalog, by Waylon Walker A configurable replacement for
kedro catalog create
that allows you to create default dataset types other than MemoryDataset.