kedro_datasets.ibis.TableDataset

class kedro_datasets.ibis.TableDataset(*, filepath=None, file_format=None, table_name=None, connection=None, load_args=None, save_args=None)[source]

TableDataset loads/saves data from/to Ibis table expressions.

Example usage for the YAML API:

cars:
  type: ibis.TableDataset
  filepath: data/01_raw/company/cars.csv
  file_format: csv
  table_name: cars
  connection:
    backend: duckdb
    database: company.db
  load_args:
    sep: ","
    nullstr: "#NA"
  save_args:
    materialized: table

motorbikes:
  type: ibis.TableDataset
  table_name: motorbikes
  connection:
    backend: duckdb
    database: company.db

Example usage for the Python API:

 import ibis
 from kedro_datasets.ibis import TableDataset

 data = ibis.memtable({"col1": [1, 2], "col2": [4, 5], "col3": [5, 6]})

 dataset = TableDataset(
...     table_name="test",
...     connection={"backend": "duckdb", "database": tmp_path / "file.db"},
...     save_args={"materialized": "table"},
... )
 dataset.save(data)
 reloaded = dataset.load()
 assert data.execute().equals(reloaded.execute())

Attributes

DEFAULT_LOAD_ARGS

DEFAULT_SAVE_ARGS

connection

rtype:

BaseBackend

Methods

exists()

Checks whether a data set's output already exists by calling the provided _exists() method.

from_config(name, config[, load_version, ...])

Create a data set instance using the configuration provided.

load()

Loads data by delegation to the provided load method.

release()

Release any cached data.

save(data)

Saves data by delegation to the provided save method.
DEFAULT_LOAD_ARGS: ClassVar[dict[str, Any]] = {}
DEFAULT_SAVE_ARGS: ClassVar[dict[str, Any]] = {'materialized': 'view', 'overwrite': True}
__init__(*, filepath=None, file_format=None, table_name=None, connection=None, load_args=None, save_args=None)[source]

Creates a new TableDataset pointing to a table (or file).

TableDataset connects to the Ibis backend object constructed from the connection configuration. The backend key provided in the config can be any of the supported backends. The remaining dictionary entries will be passed as arguments to the underlying connect() method (e.g. ibis.duckdb.connect()).

If filepath and file_format are given, the corresponding read method (e.g. read_csv()) is used to load the file with the backend. Note that only the data is loaded; no link to the underlying file exists past TableDataset.load().

If table_name is given (and filepath isn’t), the dataset establishes a connection to the relevant table for the execution backend. Therefore, Ibis doesn’t fetch data on load; all compute is deferred until materialization, when the expression is saved. In practice, this happens when another TableDataset instance is saved, after running code defined across one more more nodes.

Parameters:

  • filepath (Optional[str]) – Path to a file to register as a table. Most useful for loading data into your data warehouse (for testing).

  • file_format (Optional[str]) – Specifies the input file format for filepath.

  • table_name (Optional[str]) – The name of the table or view to read or create.

  • connection (Optional[dict[str, Any]]) – Configuration for connecting to an Ibis backend.

  • load_args (Optional[dict[str, Any]]) – Additional arguments passed to the Ibis backend’s read_{file_format} method.

  • save_args (Optional[dict[str, Any]]) – Additional arguments passed to the Ibis backend’s create_{materialized} method. By default, ir.Table objects are materialized as views. To save a table using a different materialization strategy, supply a value for materialized in save_args.

property connection: BaseBackend
Return type:

BaseBackend

exists()

Checks whether a data set’s output already exists by calling the provided _exists() method.

Return type:

bool

Returns:

Flag indicating whether the output already exists.

Raises:

DatasetError – when underlying exists method raises error.

classmethod from_config(name, config, load_version=None, save_version=None)

Create a data set instance using the configuration provided.

Parameters:

  • name (str) – Data set name.

  • config (dict[str, Any]) – Data set config dictionary.

  • load_version (Optional[str]) – Version string to be used for load operation if the data set is versioned. Has no effect on the data set if versioning was not enabled.

  • save_version (Optional[str]) – Version string to be used for save operation if the data set is versioned. Has no effect on the data set if versioning was not enabled.

Return type:

AbstractDataset

Returns:

An instance of an AbstractDataset subclass.

Raises:

DatasetError – When the function fails to create the data set from its config.

load()

Loads data by delegation to the provided load method.

Return type:

TypeVar(_DO)

Returns:

Data returned by the provided load method.

Raises:

DatasetError – When underlying load method raises error.

release()

Release any cached data.

Raises:

DatasetError – when underlying release method raises error.

Return type:

None

save(data)

Saves data by delegation to the provided save method.

Parameters:

data (TypeVar(_DI)) – the value to be saved by provided save method.

Raises:

  • DatasetError – when underlying save method raises error.

  • FileNotFoundError – when save method got file instead of dir, on Windows.

  • NotADirectoryError – when save method got file instead of dir, on Unix.

Return type:

None