kedro_datasets.ibis.TableDataset¶

class kedro_datasets.ibis.TableDataset(*, table_name, database=None, connection=None, load_args=None, save_args=None, metadata=None)[source]¶

TableDataset loads/saves data from/to Ibis table expressions.

Example usage for the YAML API:

cars:
  type: ibis.TableDataset
  table_name: cars
  connection:
    backend: duckdb
    database: company.db
  save_args:
    materialized: table

motorbikes:
  type: ibis.TableDataset
  table_name: motorbikes
  connection:
    backend: duckdb
    database: company.db

Example usage for the Python API:

 import ibis
 from kedro_datasets.ibis import TableDataset

 data = ibis.memtable({"col1": [1, 2], "col2": [4, 5], "col3": [5, 6]})

 dataset = TableDataset(
...     table_name="test",
...     connection={"backend": "duckdb", "database": tmp_path / "file.db"},
...     save_args={"materialized": "table"},
... )
 dataset.save(data)
 reloaded = dataset.load()
 assert data.execute().equals(reloaded.execute())

Attributes

`DEFAULT_CONNECTION_CONFIG`
`DEFAULT_LOAD_ARGS`
`DEFAULT_SAVE_ARGS`
`connection`	The `Backend` instance for the connection configuration.

Methods

`exists`()	Checks whether a dataset's output already exists by calling the provided _exists() method.
`from_config`(name, config[, load_version, ...])	Create a dataset instance using the configuration provided.
`load`()	Loads data by delegation to the provided load method.
`release`()	Release any cached data.
`save`(data)	Saves data by delegation to the provided save method.
`to_config`()	Converts the dataset instance into a dictionary-based configuration for serialization.

DEFAULT_CONNECTION_CONFIG: ClassVar[dict[str, Any]] = {'backend': 'duckdb', 'database': ':memory:'}¶

DEFAULT_LOAD_ARGS: ClassVar[dict[str, Any]] = {}¶

DEFAULT_SAVE_ARGS: ClassVar[dict[str, Any]] = {'materialized': 'view', 'overwrite': True}¶

__init__(*, table_name, database=None, connection=None, load_args=None, save_args=None, metadata=None)[source]¶

Creates a new TableDataset pointing to a table.

TableDataset connects to the Ibis backend object constructed from the connection configuration. The backend key provided in the config can be any of the supported backends. The remaining dictionary entries will be passed as arguments to the underlying connect() method (e.g. ibis.duckdb.connect()).

The dataset establishes a connection to the relevant table for the execution backend. Therefore, Ibis doesn’t fetch data on load; all compute is deferred until materialization, when the expression is saved. In practice, this happens when another TableDataset instance is saved, after running code defined across one more more nodes.

Parameters:

table_name (str) – The name of the table or view to read or create.
database (Optional[str]) – The name of the database to read the table or view from or create the table or view in. If not passed, then the current database is used. Provide a tuple of strings (e.g. (“catalog”, “database”)) or a dotted string path (e.g. “catalog.database”) to reference a table or view in a multi-level table hierarchy.
connection (Optional[dict[str, Any]]) – Configuration for connecting to an Ibis backend. If not provided, connect to DuckDB in in-memory mode.
load_args (Optional[dict[str, Any]]) – Additional arguments passed to the Ibis backend’s read_{file_format} method.
save_args (Optional[dict[str, Any]]) – Additional arguments passed to the Ibis backend’s create_{materialized} method. By default, ir.Table objects are materialized as views. To save a table using a different materialization strategy, supply a value for materialized in save_args.
metadata (Optional[dict[str, Any]]) – Any arbitrary metadata. This is ignored by Kedro, but may be consumed by users or external plugins.

property connection: BaseBackend¶

The Backend instance for the connection configuration.

Return type:: BaseBackend

exists()[source]¶

Checks whether a dataset’s output already exists by calling the provided _exists() method.

Return type:: bool
Returns:: Flag indicating whether the output already exists.
Raises:: DatasetError – when underlying exists method raises error.

classmethod from_config(name, config, load_version=None, save_version=None)[source]¶

Create a dataset instance using the configuration provided.

Parameters:

name (str) – Data set name.
config (dict[str, Any]) – Data set config dictionary.
load_version (Optional[str]) – Version string to be used for load operation if the dataset is versioned. Has no effect on the dataset if versioning was not enabled.
save_version (Optional[str]) – Version string to be used for save operation if the dataset is versioned. Has no effect on the dataset if versioning was not enabled.

Return type:

AbstractDataset

Returns:

An instance of an AbstractDataset subclass.

Raises:

DatasetError – When the function fails to create the dataset from its config.

load()[source]¶

Loads data by delegation to the provided load method.

Return type:: Table
Returns:: Data returned by the provided load method.
Raises:: DatasetError – When underlying load method raises error.

release()[source]¶

Release any cached data.

Raises:: DatasetError – when underlying release method raises error.
Return type:: None

save(data)[source]¶

Saves data by delegation to the provided save method.

Parameters:

data (Table) – the value to be saved by provided save method.

Raises:

DatasetError – when underlying save method raises error.
FileNotFoundError – when save method got file instead of dir, on Windows.
NotADirectoryError – when save method got file instead of dir, on Unix.

Return type:

None

to_config()[source]¶

Converts the dataset instance into a dictionary-based configuration for serialization. Ensures that any subclass-specific details are handled, with additional logic for versioning and caching implemented for CachedDataset.

Adds a key for the dataset’s type using its module and class name and includes the initialization arguments.

For CachedDataset it extracts the underlying dataset’s configuration, handles the versioned flag and removes unnecessary metadata. It also ensures the embedded dataset’s configuration is appropriately flattened or transformed.

If the dataset has a version key, it sets the versioned flag in the configuration.

Removes the metadata key from the configuration if present.

Return type:: dict[str, Any]
Returns:: A dictionary containing the dataset’s type and initialization arguments.