kedro_datasets.huggingface.HFDataset

class kedro_datasets.huggingface.HFDataset(*, dataset_name, dataset_kwargs=None)

HFDataset loads Hugging Face datasets using the datasets library.

Example usage for the YAML API:

yelp_reviews:
  type: kedro_hf_datasets.HFDataset
  dataset_name: yelp_review_full

Example usage for the Python API:

 from datasets.utils.logging import disable_progress_bar, set_verbosity, ERROR
 disable_progress_bar()  # for doctest to pass
 set_verbosity(ERROR)  # for doctest to pass
 from kedro_datasets.huggingface import HFDataset
 dataset = HFDataset(dataset_name="openai_humaneval")
 ds = dataset.load()  
Downloading and preparing dataset ...
Dataset ...
 assert "test" in ds
 assert len(ds["test"]) == 164

Methods

exists()	Checks whether a data set's output already exists by calling the provided _exists() method.
from_config(name, config[, load_version, ...])	Create a data set instance using the configuration provided.
list_datasets()
load()	Loads data by delegation to the provided load method.
release()	Release any cached data.
resolve_load_version()	Compute the version the dataset should be loaded with.
resolve_save_version()	Compute the version the dataset should be saved with.
save(data)	Saves data by delegation to the provided save method.

exists()

Checks whether a data set’s output already exists by calling the provided _exists() method.

Return type:

bool

Returns:

Flag indicating whether the output already exists.

Raises:

DatasetError – when underlying exists method raises error.

classmethod from_config(name, config, load_version=None, save_version=None)

Create a data set instance using the configuration provided.

Parameters:

• name (str) – Data set name.

• config (dict[str, Any]) – Data set config dictionary.

• load_version (str | None) – Version string to be used for load operation if the data set is versioned. Has no effect on the data set if versioning was not enabled.

• save_version (str | None) – Version string to be used for save operation if the data set is versioned. Has no effect on the data set if versioning was not enabled.

Return type:

AbstractDataset

Returns:

An instance of an AbstractDataset subclass.

Raises:

DatasetError – When the function fails to create the data set from its config.

static list_datasets()

load()

Loads data by delegation to the provided load method.

Return type:

TypeVar(_DO)

Returns:

Data returned by the provided load method.

Raises:

DatasetError – When underlying load method raises error.

release()

Release any cached data.

Raises:

DatasetError – when underlying release method raises error.

Return type:

None

resolve_load_version()

Compute the version the dataset should be loaded with.

Return type:

str | None

resolve_save_version()

Compute the version the dataset should be saved with.

Return type:

str | None

save(data)

Saves data by delegation to the provided save method.

Parameters:

data (TypeVar(_DI)) – the value to be saved by provided save method.

Raises:

• DatasetError – when underlying save method raises error.

• FileNotFoundError – when save method got file instead of dir, on Windows.

• NotADirectoryError – when save method got file instead of dir, on Unix.

Return type:

None