Welcome to Kedro’s award-winning documentation!

Learn about Kedro

• Introduction to Kedro
• First steps
  • Set up Kedro
  • Create a new Kedro project
  • Kedro concepts
  • Create a Minimal Kedro Project
• Learn Kedro with hands-on video
  • Who is this course for?
  • What you’ll learn
  • Index of videos

Tutorial and basic Kedro usage

• Next steps: Tutorial
  • Set up the spaceflights project
  • Set up the data
  • Create a data processing pipeline
  • Create a data science pipeline
  • Test a Kedro project
  • Package an entire Kedro project
  • Spaceflights tutorial FAQs
  • Watch the video
  • Get help
  • Terminology
• Visualisation with Kedro-Viz
• Kedro for notebook users
• FAQs and resources
  • FAQs
  • Kedro glossary
  • Migration guide

Kedro projects

• Customise a new project
• Configuration
  • Configuration
  • Credentials
  • Parameters
  • Migration guide for config loaders
  • Advanced configuration
• Data Catalog
  • Introduction to the Data Catalog
  • Data Catalog YAML examples
  • Kedro dataset factories
  • Advanced: Access the Data Catalog in code
  • Advanced: Partitioned and incremental datasets
  • Advanced: Tutorial to create a custom dataset
  • KedroDataCatalog (experimental feature)
• Nodes and pipelines
  • Nodes
  • Pipeline objects
  • Modular pipelines
  • Reuse pipelines and group nodes with namespaces
  • The pipeline registry
  • Micro-packaging
  • Run a pipeline
  • Slice a pipeline
• Anonymous Telemetry
  • Collected data fields:
  • How do I withdraw consent?

Integrations

• PySpark integration
  • Centralise Spark configuration in conf/base/spark.yml
  • Initialise a SparkSession using a hook
  • Use Kedro’s built-in Spark datasets to load and save raw data
  • Spark and Delta Lake interaction
  • Use MemoryDataset for intermediary DataFrame
  • Use MemoryDataset with copy_mode="assign" for non-DataFrame Spark objects
  • Tips for maximising concurrency using ThreadRunner
• How to add MLflow to your Kedro workflow
  • Prerequisites
  • Simple use cases
  • Advanced use cases
• Data and pipeline versioning with Kedro and DVC
  • Versioning data with .dvc files
  • Advanced use cases
• Data versioning with Delta Lake
  • Prerequisites
  • Using Delta tables in catalog
  • Inspect the dataset in interactive mode
  • Using Delta tables with Spark
• Data versioning with Iceberg
  • Prerequisites
  • Define a custom dataset to use Iceberg tables
  • Using Iceberg tables in the catalog
  • Inspect the dataset in interactive mode

Development

• Set up Visual Studio Code
  • Kedro VS Code Extension
  • Setting up venv / virtualenv in VS Code
  • Setting up tasks
  • Setting a custom Kedro project path
  • Real time catalog validation with Kedro LSP
  • Visualise the pipeline with Kedro-Viz
  • Debugging
• Set up PyCharm
  • Set up Run configurations
  • Debugging
  • Advanced: Remote SSH interpreter
  • Advanced: Docker interpreter
  • Configure Python Console
  • Configuring the Kedro catalog validation schema
• Debugging
• Automated testing
  • Introduction
  • Set up automated testing with pytest
  • Add test coverage reports with pytest-cov
• Code formatting and linting
  • Introduction
  • Set up Python tools
  • Automated formatting and linting with pre-commit hooks

Advanced usage

• Project setup
  • Dependencies
  • Lifecycle management with KedroSession
  • Project settings
• Extend Kedro
  • Common use cases
  • Kedro plugins
  • Kedro architecture overview
  • Create a Kedro starter
• Hooks
  • Introduction to Hooks
  • Common use cases
  • Hooks examples
• Logging
  • Default logging configuration
  • How to perform logging in your Kedro project
  • How to customise Kedro logging
  • Custom CONF_SOURCE with logging
  • Advanced logging
  • How to customise the rich handler
  • How to enable file-based logging
  • How to use plain console logging
  • How to enable rich logging in a dumb terminal
  • How to enable rich logging in Jupyter
• Deployment

Contribute to Kedro

• Contribute to Kedro

CLI reference

• Kedro’s command line interface
  • Autocompletion (optional)
  • Invoke Kedro CLI from Python (optional)
  • Kedro commands
  • Global Kedro commands
  • Customise or override project-specific Kedro commands

API documentation

kedro

Kedro is a framework that makes it easy to build robust and scalable data pipelines by providing uniform project templates, data abstraction, configuration and pipeline assembly.

Indices and tables

• Index

• Module Index