Welcome to Kedro’s documentation!

Introduction

• What is Kedro?
  • Learn how to use Kedro
  • Assumptions

First steps

• Set up Kedro
  • Summary
  • Installation prerequisites
  • Create a virtual environment for your Kedro project
  • How to install Kedro using pip
  • How to verify your Kedro installation
  • How to upgrade Kedro
  • How to install a development version of Kedro
• Create a new Kedro project
  • Summary
  • Create a new empty project
  • Create a new project from a configuration file
  • Create a new project containing example code
  • Run the project
  • Visualise a Kedro project
  • Where next?
  • More information about the pandas-iris example project
• Kedro concepts
  • Summary
  • Node
  • Pipeline
  • Data Catalog
  • Kedro project directory structure

Next steps: Tutorial

• Introducing spaceflights
  • Scenario
  • Get help
  • Terminology
• Set up the spaceflights project
  • Create a new project
  • Install project dependencies
  • Optional: logging and configuration
• Set up the data
  • Project datasets
  • Dataset registration
  • Further information
• Create a data processing pipeline
  • Introduction
  • Data preprocessing node functions
  • The data processing pipeline
  • Test the example
  • Preprocessed data registration
  • Create a table for model input
  • Model input table registration
  • Test the example again
  • Visualise the project
  • Checkpoint
• Create a data science pipeline
  • Data science nodes
  • Input parameter configuration
  • Model registration
  • Data science pipeline
  • Test the pipelines
  • Modular pipelines
  • Optional: Kedro runners
• Package an entire Kedro project
  • Add documentation to a Kedro project
  • Package a Kedro project
• Spaceflights tutorial FAQs
  • How do I resolve these common errors?

Visualisation with Kedro-Viz

• Get started with Kedro-Viz
  • Visualise the spaceflights project
  • Automatic visualisation updates
  • Visualise layers
  • Share a pipeline visualisation
• Visualise charts in Kedro-Viz
  • Visualisation with Plotly
  • Visualisation with Matplotlib
• Experiment tracking in Kedro-Viz
  • Experiment tracking demonstration using Kedro-Viz
  • Kedro versions supporting experiment tracking
  • When should I use experiment tracking in Kedro?
  • Set up a project
  • Set up the session store
  • Set up experiment tracking datasets
  • Modify your nodes and pipelines to log metrics
  • Generate the run data
  • Access run data and compare runs
  • View and compare plots
  • View and compare metrics data

Notebooks & IPython users

• Kedro and Jupyter Notebooks
  • A custom Kedro kernel
  • Iris dataset example
  • %reload_kedro line magic
  • %run_viz line magic
  • Convert functions from Jupyter Notebooks into Kedro nodes
  • Useful to know…
  • Find out more
• Kedro as a data registry
  • Usage

Kedro project setup

• Kedro starters
  • How to use Kedro starters
  • Starter aliases
  • List of official starters
  • Starter versioning
  • Use a starter with a configuration file
  • How to create a Kedro starter
• Dependencies
  • Project-specific dependencies
  • Install project-specific dependencies
  • Workflow dependencies
• Configuration
  • Configuration root
  • Local and base configuration environments
  • Additional configuration environments
  • Template configuration
  • Configuration with OmegaConf
  • Parameters
  • Credentials
• Lifecycle management with KedroSession
  • Overview
  • Create a session
• Project settings
  • Application settings
  • Project metadata

Data Catalog

• The Data Catalog
  • Use the Data Catalog within Kedro configuration
  • Specify the location of the dataset
  • Data Catalog *_args parameters
  • Use the Data Catalog with the YAML API
  • Create a Data Catalog YAML configuration file via CLI
  • Adding parameters
  • Feeding in credentials
  • Load multiple datasets with similar configuration
  • Transcode datasets
  • Version datasets and ML models
  • Use the Data Catalog with the Code API
• Kedro IO
  • Error handling
  • AbstractDataSet
  • Versioning
  • Partitioned dataset

Nodes and pipelines

• Nodes
  • How to create a node
  • **kwargs-only node functions
  • How to tag a node
  • How to run a node
  • How to use generator functions in a node
• Pipelines
  • How to build a pipeline
  • Bad pipelines
• Modular pipelines
  • What are modular pipelines?
  • How do I create a modular pipeline?
  • Using the modular pipeline() wrapper to provide overrides
  • Combining disconnected pipelines
  • Using a modular pipeline multiple times
  • How to use a modular pipeline with different parameters
• The pipeline registry
  • Pipeline autodiscovery
• Micro-packaging
  • Package a micro-package
  • Package multiple micro-packages
  • Pull a micro-package
  • Pull multiple micro-packages
• Run a pipeline
  • Runners
  • Custom runners
  • Load and save asynchronously
  • Run a pipeline by name
  • Run pipelines with IO
  • Output to a file
  • Configure kedro run arguments
• Slice a pipeline
  • Slice a pipeline by providing inputs
  • Slice a pipeline by specifying nodes
  • Slice a pipeline by specifying final nodes
  • Slice a pipeline with tagged nodes
  • Slice a pipeline by running specified nodes
  • How to recreate missing outputs

Extend Kedro

• Common use cases
  • Use Case 1: How to add extra behaviour to Kedro’s execution timeline
  • Use Case 2: How to integrate Kedro with additional data sources
  • Use Case 3: How to add or modify CLI commands
  • Use Case 4: How to customise the initial boilerplate of your project
• Custom datasets
  • Scenario
  • Project setup
  • The anatomy of a dataset
  • Implement the _load method with fsspec
  • Implement the _save method with fsspec
  • Implement the _describe method
  • The complete example
  • Integration with PartitionedDataSet
  • Versioning
  • Thread-safety
  • How to handle credentials and different filesystems
  • How to contribute a custom dataset implementation
• Kedro plugins
  • Overview
  • Example of a simple plugin
  • Extend starter aliases
  • Working with click
  • Project context
  • Initialisation
  • global and project commands
  • Suggested command convention
  • Hooks
  • CLI Hooks
  • Contributing process
  • Supported Kedro plugins
  • Community-developed plugins

Hooks

• Hooks
  • Introduction
  • Concepts
  • Under the hood
• Common use cases
  • Use Hooks to extend a node’s behaviour
  • Use Hooks to customise the dataset load and save methods
  • Use Hooks to load external credentials
• Hooks examples
  • Add memory consumption tracking
  • Add data validation
  • Add observability to your pipeline
  • Add metrics tracking to your model
  • Modify node inputs using before_node_run hook

Logging

• Logging
  • Default framework-side logging configuration
  • Project-side logging configuration
  • Perform logging in your project

Development

• Set up Visual Studio Code
  • Advanced: For those using venv / virtualenv
  • Setting up tasks
  • Debugging
  • Configuring the Kedro catalog validation schema
• Set up PyCharm
  • Set up Run configurations
  • Debugging
  • Advanced: Remote SSH interpreter
  • Advanced: Docker interpreter
  • Configure Python Console
  • Configuring the Kedro catalog validation schema
• Kedro’s command line interface
  • Autocompletion (optional)
  • Invoke Kedro CLI from Python (optional)
  • Kedro commands
  • Global Kedro commands
  • Customise or Override Project-specific Kedro commands
• Debugging
  • Introduction
  • Debugging Node
  • Debugging Pipeline
• Automated Testing
  • Introduction
  • Set up automated testing with pytest
  • Add test coverage reports with pytest-cov
• Linting
  • Introduction
  • Set up linting tools
  • Automating linting with pre-commit hooks

Deployment

• Deployment guide
  • Deployment choices
• Single-machine deployment
  • Container-based
  • Package-based
  • CLI-based
• Distributed deployment
  • 1. Containerise the pipeline
  • 2. Convert your Kedro pipeline into targeted platform’s primitives
  • 3. Parameterise the runs
  • 4. (Optional) Create starters
• Deployment with Argo Workflows
  • Why would you use Argo Workflows?
  • Prerequisites
  • How to run your Kedro pipeline using Argo Workflows
• Deployment with Prefect
  • Prerequisites
  • How to run your Kedro pipeline using Prefect
• Deployment with Kubeflow Pipelines
  • Why would you use Kubeflow Pipelines?
  • Prerequisites
  • How to run your Kedro pipeline using Kubeflow Pipelines
• Deployment with AWS Batch
  • Why would you use AWS Batch?
  • Prerequisites
  • How to run a Kedro pipeline using AWS Batch
• Deployment to a Databricks cluster
  • Prerequisites
  • Running Kedro project from a Databricks notebook
  • How to use datasets stored on Databricks DBFS
• How to integrate Amazon SageMaker into your Kedro pipeline
  • Why would you use Amazon SageMaker?
  • Prerequisites
  • Prepare the environment
  • Update the Kedro project
  • Run the project
  • Cleanup
• How to deploy your Kedro pipeline with AWS Step Functions
  • Why would you run a Kedro pipeline with AWS Step Functions
  • Strategy
  • Prerequisites
  • Deployment process
  • Limitations
  • Final thought
• How to deploy your Kedro pipeline on Apache Airflow with Astronomer
  • Strategy
  • Prerequisites
  • Project Setup
  • Deployment process
  • Final thought
• Deployment to a Dask cluster
  • Why would you use Dask?
  • Prerequisites
  • How to distribute your Kedro pipeline using Dask

PySpark integration

• Build a Kedro pipeline with PySpark
  • Centralise Spark configuration in conf/base/spark.yml
  • Initialise a SparkSession using a hook
  • Use Kedro’s built-in Spark datasets to load and save raw data
  • Spark and Delta Lake interaction
  • Use MemoryDataSet for intermediary DataFrame
  • Use MemoryDataSet with copy_mode="assign" for non-DataFrame Spark objects
  • Tips for maximising concurrency using ThreadRunner

Resources

• Frequently asked questions
  • Kedro project development
  • Working with Jupyter
  • Datasets and the Data Catalog
  • Nodes and pipelines
  • Visualisation
• Kedro glossary
  • Data Catalog
  • Data engineering vs Data science
  • Kedro
  • KedroContext
  • KedroSession
  • Kedro-Viz
  • Layers (data engineering convention)
  • Modular pipeline
  • Node
  • Node execution order
  • Pipeline
  • Pipeline slicing
  • Runner
  • Starters
  • Tags
  • Workflow dependencies

Contribute to Kedro

• Introduction
• Guidelines for contributing developers
  • Introduction
  • Before you start: development set up
  • Get started: areas of contribution
  • Create a pull request
  • Need help?
• Backwards compatibility & breaking changes
  • When should I make a breaking change?
  • The Kedro release model
• Contribute to the Kedro documentation
  • How do I rebuild the documentation after I make changes to it?
  • Extend Kedro documentation
  • Kedro documentation style guide
• Join the Technical Steering Committee
  • Responsibilities of a maintainer
  • Requirements to become a maintainer
  • Application process
  • Voting process
• How to deploy a development version of Kedro to Databricks
  • Introduction
  • Prerequisites
  • How to install a build of Kedro onto Databricks

API documentation

kedro

Kedro is a framework that makes it easy to build robust and scalable data pipelines by providing uniform project templates, data abstraction, configuration and pipeline assembly.

Indices and tables

• Index

• Module Index