Kedro plugins¶

Kedro plugins allow you to create new features for Kedro and inject additional commands into the CLI. Plugins are developed as separate Python packages that exist outside of any Kedro project.

Overview¶

Kedro's extension mechanism is built on pluggy, a solid plugin management library that was created for the pytest ecosystem. pluggy relies on entry points, a Python mechanism for packages to provide components that can be discovered by other packages using importlib.metadata.

Example plugin¶

Here is an example plugin that prints the pipeline as JSON:

kedrojson/plugin.py

import click
from kedro.framework.project import pipelines


@click.group(name="JSON")
def commands():
    pass


@commands.command(name="to_json")
@click.pass_obj
def to_json(metadata):
    """Display the pipeline in JSON format"""
    pipeline = pipelines["__default__"]
    print(pipeline.to_json())

From version 0.18.14, Kedro replaced setup.py with pyproject.toml. The plugin needs to provide entry points in either file. If you use setup.py, see the 0.18.13 documentation.

To add the entry point to pyproject.toml, the plugin needs to provide the following entry_points configuration:

[project.entry-points."kedro.project_commands"]
kedrojson = "kedrojson.plugin:commands"

Once the plugin is installed, you can run it as follows:

kedro to_json

Working with `click`¶

Commands must be provided as click Groups

The click Group will be merged into the main CLI Group. In the process, the options on the group are lost, as is any processing that was done as part of its callback function.

Project context¶

When they run, plugins may request information about the current project by creating a session and loading its context:

from pathlib import Path

from kedro.framework.session import KedroSession


project_path = Path.cwd()
session = KedroSession.create(project_path=project_path)
context = session.load_context()

Initialisation¶

If the plugin initialisation needs to occur before Kedro starts, it can declare the entry_point key kedro.init. This entry point must point to a function that has no arguments, but for future proofing you should declare it with **kwargs.

`global` and `project` commands¶

Plugins may also add commands to the Kedro CLI, which supports two types of commands:

global - available both inside and outside a Kedro project. Global commands use the entry_point key kedro.global_commands.
project - available when a Kedro project is detected in the current directory. Project commands use the entry_point key kedro.project_commands.

Suggested command convention¶

We use the following command convention: kedro <plugin-name> <command>, with kedro <plugin-name> acting as a top-level command group. This structure is optional.

Advanced: Lazy loading of plugin commands¶

If your plugin includes a sizable set of CLI commands or heavy dependencies that are slow to import, consider lazy loading. This approach improves the performance of the plugin as well as the Kedro CLI. Follow the instructions in the click documentation on lazy loading of commands. From Kedro 0.19.7, the Kedro commands are declared as lazy loaded command groups that you can use as a reference for the implementation.

Consider the previous example of the kedrojson plugin. Suppose the plugin has two commands, kedro to_json pipelines and kedro to_json nodes. The to_json pipelines command is used more frequently than the to_json nodes command and the to_json nodes command requires a large library to be imported. In this case, you can define your commands with lazy loading and delayed imports as follows:

In kedrojson/plugin.py:

import click
from kedro.framework.project import pipelines
from kedro.framework.cli.utils import LazyGroup

@click.group()
def commands():
    pass

@commands.group(
    name="to_json",
    cls=LazyGroup,
    lazy_subcommands={
        "nodes": "kedrojson.plugin.nodes",
        "pipelines": "kedrojson.plugin.pipelines"
        }
)
def to_json():
    """Convert Kedro nodes and pipelines to JSON"""
    pass

@click.command(name="nodes")
def nodes():
    """Convert Kedro nodes to JSON"""
    import some_large_library
    print("Converting nodes to JSON")
    ...

@click.command("pipelines")
def pipelines():
    """Convert Kedro pipelines to JSON"""
    print("Converting pipelines to JSON")
    ...

The loading of the individual nodes and pipelines commands, and the related imports, will be delayed until the respective commands are called.

Hooks¶

You can develop hook implementations and have them automatically registered to the project context when the plugin is installed.

To enable this for your custom plugin, add the following entry in pyproject.toml

To use pyproject.toml, specify

[project.entry-points."kedro.hooks"]
plugin_name = "plugin_name.plugin:hooks"

where plugin.py is the module where you declare hook implementations:

import logging

from kedro.framework.hooks import hook_impl


class MyHooks:
    @hook_impl
    def after_catalog_created(self, catalog):
        logging.info("Reached after_catalog_created hook")


hooks = MyHooks()

Note

hooks should be an instance of the class defining the Hooks.

CLI Hooks¶

You can also develop Hook implementations to extend Kedro's CLI behaviour in your plugin. To find available CLI Hooks, see our kedro.framework.cli.hooks API documentation. To register CLI Hooks developed in your plugin with Kedro, add the following entry in your project's pyproject.toml:

[project.entry-points."kedro.cli_hooks"]
plugin_name = "plugin_name.plugin:cli_hooks"

(where plugin.py is the module where you declare Hook implementations):

import logging

from kedro.framework.cli.hooks import cli_hook_impl


class MyCLIHooks:
    @cli_hook_impl
    def before_command_run(self, project_metadata, command_args):
        logging.info(
            "Command %s will be run for project %s", command_args, project_metadata
        )


cli_hooks = MyCLIHooks()

Contributing process¶

When you are ready to submit your code:

Create a separate repository using our naming convention for plugins (kedro-<plugin-name>)
Choose a command approach: global and / or project commands:
All global commands should be provided as a single click group
All project commands should be provided as another click group
The click groups are declared through the entry points mechanism
Include a README.md describing your plugin's functionality and all dependencies that should be included
Use GitHub tagging to tag your plugin as a kedro-plugin so that we can find it

Supported Kedro plugins¶

Kedro-Datasets, a collection of Kedro's data connectors. These data connectors are implementations of the AbstractDataset
Kedro-Docker, a tool for packaging and shipping Kedro projects within containers
Kedro-Airflow, a tool for converting your Kedro project into an Airflow project
Kedro-Viz, a tool for visualising your Kedro pipelines

Community-developed plugins¶

Several community-developed plugins are available and a comprehensive list of plugins is published on the awesome-kedro GitHub repository. The list below is a small snapshot of some of those under active maintenance.