Welcome to Kedro’s documentation!

Introduction

• What is Kedro?
  • Learn how to use Kedro
  • Assumptions

Get started

• Installation prerequisites
  • Virtual environments
• Install Kedro
  • Verify a successful installation
  • Install a development version
• A “Hello World” example
  • Node
  • Pipeline
  • DataCatalog
  • Runner
  • Hello Kedro!
• Create a new project
  • Create a new project interactively
  • Create a new project from a configuration file
  • Initialise a git repository
• Iris dataset example project
  • Create the example project
  • Run the example project
  • Under the hood: Pipelines and nodes
• Kedro starters
  • How to use Kedro starters
  • List of official starters
  • Starter versioning
  • Use a starter in interactive mode
  • Use a starter with a configuration file

Tutorial

• Kedro spaceflights tutorial
  • Kedro project development workflow
  • Optional: Git workflow
• Set up the spaceflights project
  • Create a new project
  • Install dependencies
  • Configure the project
• Set up the data
  • Add your datasets to data
  • Register the datasets
  • Custom data
• Create a pipeline
  • Data processing pipeline
  • Data science pipeline
  • Kedro runners
  • Slice a pipeline
• Visualise pipelines
  • Install Kedro-Viz
  • Visualise a whole pipeline
  • Exit an open visualisation
  • Visualise layers
  • Share a pipeline
  • Visualise Plotly charts in Kedro-Viz
• Namespace pipelines
  • Adding a namespace to the data_processing pipeline
  • Adding namespaces to the data_science pipeline
  • Nesting modular pipelines
• Set up experiment tracking
  • Set up a project
  • Set up the session store
  • Set up tracking datasets
  • Set up your nodes and pipelines to log metrics
  • Generate the Run data
  • Access run data and compare runs
• Package a project
  • Add documentation to your project
  • Package your project

Kedro project setup

• Dependencies
  • Project-specific dependencies
  • kedro install
  • Workflow dependencies
• Configuration
  • Configuration root
  • Local and base configuration environments
  • Additional configuration environments
  • Template configuration
  • Parameters
  • Credentials
  • Configure kedro run arguments
• Lifecycle management with KedroSession
  • Overview
  • Create a session
• The mini-kedro Kedro starter
  • Introduction
  • Usage
  • Content

Data Catalog

• The Data Catalog
  • Using the Data Catalog within Kedro configuration
  • Specifying the location of the dataset
  • Data Catalog *_args parameters
  • Using the Data Catalog with the YAML API
  • Creating a Data Catalog YAML configuration file via CLI
  • Adding parameters
  • Feeding in credentials
  • Loading multiple datasets that have similar configuration
  • Transcoding datasets
  • Transforming datasets
  • Versioning datasets and ML models
  • Using the Data Catalog with the Code API
• Kedro IO
  • Error handling
  • AbstractDataSet
  • Versioning
  • Partitioned dataset

Nodes and pipelines

• Nodes
  • How to create a node
  • **kwargs-only node functions
  • How to tag a node
  • How to run a node
• Pipelines
  • How to build a pipeline
  • Bad pipelines
• Modular pipelines
  • What are modular pipelines?
  • How do I create a modular pipeline?
  • Using the modular pipeline() wrapper to provide overrides
  • Combining disconnected pipelines
  • Using a modular pipeline multiple times
  • How to use a modular pipeline with different parameters
• Micro-packaging
  • Package a micro-package
  • Package multiple micro-packages
  • Pull a micro-package
  • Pull multiple micro-packages
• Run a pipeline
  • Runners
  • Custom runners
  • Load and save asynchronously
  • Run a pipeline by name
  • Run pipelines with IO
  • Output to a file
• Slice a pipeline
  • Slice a pipeline by providing inputs
  • Slice a pipeline by specifying nodes
  • Slice a pipeline by specifying final nodes
  • Slice a pipeline with tagged nodes
  • Slice a pipeline by running specified nodes
  • How to recreate missing outputs

Extend Kedro

• Common use cases
  • Use Case 1: How to add extra behaviour to Kedro’s execution timeline
  • Use Case 2: How to integrate Kedro with additional data sources
  • Use Case 3: How to add CLI commands that are reusable across projects
  • Use Case 4: How to customise the initial boilerplate of your project
• Hooks
  • Introduction
  • Concepts
  • Common use cases
  • Under the hood
  • Hooks examples
• Custom datasets
  • Scenario
  • Project setup
  • The anatomy of a dataset
  • Implement the _load method with fsspec
  • Implement the _save method with fsspec
  • Implement the _describe method
  • The complete example
  • Integration with PartitionedDataSet
  • Versioning
  • Thread-safety
  • How to handle credentials and different filesystems
  • How to contribute a custom dataset implementation
• Kedro plugins
  • Overview
  • Example of a simple plugin
  • Working with click
  • Project context
  • Initialisation
  • global and project commands
  • Suggested command convention
  • Hooks
  • CLI Hooks
  • Contributing process
  • Supported Kedro plugins
  • Community-developed plugins
• Create a Kedro starter
  • How to create a Kedro starter
  • Configuration variables
• Dataset transformers (deprecated)
  • Develop your own dataset transformer
• Decorators (deprecated)
  • How to apply a decorator to nodes
  • How to apply multiple decorators to nodes
  • How to apply a decorator to a pipeline
  • Kedro decorators

Logging

• Logging
  • Configure logging
  • Use logging
  • Logging for anyconfig
• Experiment tracking
  • Enable experiment tracking
  • Community solutions

Development

• Set up Visual Studio Code
  • Advanced: For those using venv / virtualenv
  • Setting up tasks
  • Debugging
  • Configuring the Kedro catalog validation schema
• Set up PyCharm
  • Set up Run configurations
  • Debugging
  • Advanced: Remote SSH interpreter
  • Advanced: Docker interpreter
  • Configure Python Console
  • Configuring the Kedro catalog validation schema
• Kedro’s command line interface
  • Autocompletion (optional)
  • Invoke Kedro CLI from Python (optional)
  • Kedro commands
  • Global Kedro commands
  • Project-specific Kedro commands
• Linting your Kedro project
• Debugging
  • Introduction
  • Debugging Node
  • Debugging Pipeline

Deployment

• Deployment guide
  • Deployment choices
• Single-machine deployment
  • Container based
  • Package based
  • CLI based
• Distributed deployment
  • 1. Containerise the pipeline
  • 2. Convert your Kedro pipeline into targeted platform’s primitives
  • 3. Parameterise the runs
  • 4. (Optional) Create starters
• Deployment with Argo Workflows
  • Why would you use Argo Workflows?
  • Prerequisites
  • How to run your Kedro pipeline using Argo Workflows
• Deployment with Prefect
  • Prerequisites
  • How to run your Kedro pipeline using Prefect
• Deployment with Kubeflow Pipelines
  • Why would you use Kubeflow Pipelines?
  • Prerequisites
  • How to run your Kedro pipeline using Kubeflow Pipelines
• Deployment with AWS Batch
  • Why would you use AWS Batch?
  • Prerequisites
  • How to run a Kedro pipeline using AWS Batch
• Deployment to a Databricks cluster
  • Prerequisites
  • Run the Kedro project with Databricks Connect
  • Run Kedro project from a Databricks notebook
• How to integrate Amazon SageMaker into your Kedro pipeline
  • Why would you use Amazon SageMaker?
  • Prerequisites
  • Prepare the environment
  • Update the Kedro project
  • Run the project
  • Cleanup
• How to deploy your Kedro pipeline with AWS Step Functions
  • Why would you run a Kedro pipeline with AWS Step Functions
  • Strategy
  • Prerequisites
  • Deployment process
  • Limitations
  • Final thought
• How to deploy your Kedro pipeline on Apache Airflow with Astronomer
  • Strategy
  • Prerequisites
  • Project Setup
  • Deployment process
  • Final thought

Tools integration

• Build a Kedro pipeline with PySpark
  • Centralise Spark configuration in conf/base/spark.yml
  • Initialise a SparkSession in custom project context class
  • Use Kedro’s built-in Spark datasets to load and save raw data
  • Spark and Delta Lake interaction
  • Use MemoryDataSet for intermediary DataFrame
  • Use MemoryDataSet with copy_mode="assign" for non-DataFrame Spark objects
  • Tips for maximising concurrency using ThreadRunner
• Use Kedro with IPython and Jupyter Notebooks/Lab
  • Why use a Notebook?
  • Kedro and IPython
  • Kedro and Jupyter
  • How to use context
  • Global variables
  • Convert functions from Jupyter Notebooks into Kedro nodes
  • IPython extension
  • IPython loader

FAQs

• Frequently asked questions
  • What is Kedro?
  • Who maintains Kedro?
  • What are the primary advantages of Kedro?
  • How does Kedro compare to other projects?
  • What is data engineering convention?
  • How do I upgrade Kedro?
  • How can I use a development version of Kedro?
  • How can I find out more about Kedro?
  • How can I cite Kedro?
  • How can I get my question answered?
• Kedro architecture overview
  • Kedro project
  • Kedro starter
  • Kedro library
  • Kedro framework
  • Kedro extension
• Kedro Principles
  • 1. Modularity at the core ️📦
  • 2. Grow beginners into experts 🌱
  • 3. User empathy without unfounded assumptions 🤝
  • 4. Simplicity means bare necessities 🍞
  • 5. There should be one obvious way of doing things 🎯
  • 6. A sprinkle of magic is better than a spoonful of it ✨
  • 7. Lean process and lean product 👟

Resources

• Images and icons
  • White background
  • Black background
• Kedro glossary
  • Data Catalog
  • Data engineering vs Data science
  • Kedro
  • KedroContext
  • KedroSession
  • Kedro-Viz
  • Layers (data engineering convention)
  • Modular pipeline
  • Node
  • Node execution order
  • Pipeline
  • Pipeline slicing
  • Runner
  • Starters
  • Tags
  • Workflow dependencies

Contribute to Kedro

• Introduction
• Guidelines for contributing developers
  • Introduction
  • Before you start: development set up
  • Get started: areas of contribution
  • Create a pull request
  • Need help?
• Backwards compatibility & breaking changes
  • When should I make a breaking change?
  • The Kedro release model
• Contribute to the Kedro documentation
  • How do I rebuild the documentation after I make changes to it?
  • Extend Kedro documentation
  • Kedro documentation style guide
• Join the Technical Steering Committee
  • Responsibilities of a maintainer
  • Requirements to become a maintainer
  • Application process
  • Voting process

API documentation

kedro

Kedro is a framework that makes it easy to build robust and scalable data pipelines by providing uniform project templates, data abstraction, configuration and pipeline assembly.

Indices and tables

• Index

• Module Index