Welcome to Kedro’s award-winning documentation!

Learn about Kedro

• Introduction to Kedro
• First steps
  • Set up Kedro
  • Create a new Kedro project
  • Kedro concepts
• Learn Kedro with hands-on video
  • Who is this course for?
  • What you’ll learn
  • Index of videos

Tutorial and basic Kedro usage

• Next steps: Tutorial
  • Set up the spaceflights project
  • Set up the data
  • Create a data processing pipeline
  • Create a data science pipeline
  • Test a Kedro project
  • Package an entire Kedro project
  • Spaceflights tutorial FAQs
  • Watch the video
  • Get help
  • Terminology
• Visualisation with Kedro-Viz
• Kedro for notebook users
• FAQs and resources
  • FAQs
  • Kedro glossary
  • Migration guide

Kedro projects

• Customise a new project
• Configuration
  • Configuration
  • Credentials
  • Parameters
  • Migration guide for config loaders
  • Advanced configuration
• The Kedro Data Catalog
  • Introduction to the Data Catalog
  • Data Catalog YAML examples
  • Kedro dataset factories
  • Advanced: Access the Data Catalog in code
  • Advanced: Partitioned and incremental datasets
  • Advanced: Tutorial to create a custom dataset
• Nodes and pipelines
  • Nodes
  • Pipeline objects
  • Modular pipelines
  • Reuse pipelines with namespaces
  • The pipeline registry
  • Micro-packaging
  • Run a pipeline
  • Slice a pipeline
• Anonymous Telemetry
  • Collected data fields:
  • How do I withdraw consent?

Integrations

• PySpark integration
  • Centralise Spark configuration in conf/base/spark.yml
  • Initialise a SparkSession using a hook
  • Use Kedro’s built-in Spark datasets to load and save raw data
  • Spark and Delta Lake interaction
  • Use MemoryDataset for intermediary DataFrame
  • Use MemoryDataset with copy_mode="assign" for non-DataFrame Spark objects
  • Tips for maximising concurrency using ThreadRunner
• How to add MLflow to your Kedro workflow
  • Prerequisites
  • Simple use cases
  • Advanced use cases

Advanced usage

• Project setup
  • Dependencies
  • Lifecycle management with KedroSession
  • Project settings
• Extend Kedro
  • Common use cases
  • Kedro plugins
  • Kedro architecture overview
  • Create a Kedro starter
• Hooks
  • Introduction to Hooks
  • Common use cases
  • Hooks examples
• Logging
• Default logging configuration
  • How to perform logging in your Kedro project
  • How to customise Kedro logging
  • Custom CONF_SOURCE with logging
  • Advanced logging
  • How to customise the rich handler
  • How to enable file-based logging
  • How to use plain console logging
  • How to enable rich logging in a dumb terminal
  • How to enable rich logging in Jupyter
• Development
  • Set up Visual Studio Code
  • Set up PyCharm
  • Kedro’s command line interface
  • Debugging
  • Automated testing
  • Code formatting and linting
• Deployment

Contribute to Kedro

• Contribute to Kedro

API documentation

kedro

Kedro is a framework that makes it easy to build robust and scalable data pipelines by providing uniform project templates, data abstraction, configuration and pipeline assembly.

Indices and tables

• Index

• Module Index