kedro_datasets.snowflake.SnowparkTableDataset¶

class kedro_datasets.snowflake.SnowparkTableDataset(*, table_name, schema=None, database=None, load_args=None, save_args=None, credentials=None, session=None, metadata=None)[source]¶

SnowparkTableDataset loads and saves Snowpark DataFrames.

As of October 2024, the Snowpark connector works with Python 3.9, 3.10, and 3.11. Python 3.12 is not supported yet.

Example usage for the YAML API:

weather:
  type: kedro_datasets.snowflake.SnowparkTableDataset
  table_name: "weather_data"
  database: "meteorology"
  schema: "observations"
  credentials: db_credentials
  save_args:
    mode: overwrite
    column_order: name
    table_type: ''

You can skip everything but “table_name” if the database and schema are provided via credentials. This allows catalog entries to be shorter when all Snowflake tables are in the same database and schema. Values in the dataset definition take priority over those defined in credentials.

Example: The credentials file provides all connection attributes. The catalog entry for “weather” reuses the credentials parameters, while the “polygons” catalog entry reuses all credentials parameters except for specifying a different schema. The second example demonstrates the use of externalbrowser authentication.

catalog.yml:

weather:
  type: kedro_datasets.snowflake.SnowparkTableDataset
  table_name: "weather_data"
  database: "meteorology"
  schema: "observations"
  credentials: snowflake_client
  save_args:
    mode: overwrite
    column_order: name
    table_type: ''

polygons:
  type: kedro_datasets.snowflake.SnowparkTableDataset
  table_name: "geopolygons"
  credentials: snowflake_client
  schema: "geodata"

credentials.yml:

snowflake_client:
  account: 'ab12345.eu-central-1'
  port: 443
  warehouse: "datascience_wh"
  database: "detailed_data"
  schema: "observations"
  user: "service_account_abc"
  password: "supersecret"

credentials.yml (with externalbrowser authentication):

snowflake_client:
  account: 'ab12345.eu-central-1'
  port: 443
  warehouse: "datascience_wh"
  database: "detailed_data"
  schema: "observations"
  user: "john_doe@wdomain.com"
  authenticator: "externalbrowser"

Attributes

`DEFAULT_LOAD_ARGS`
`DEFAULT_SAVE_ARGS`
`session`	Retrieve or create a session.

Methods

`exists`()	Checks whether a dataset's output already exists by calling the provided _exists() method.
`from_config`(name, config[, load_version, ...])	Create a dataset instance using the configuration provided.
`load`()	Load data from a specified database table.
`release`()	Release any cached data.
`save`(data)	Check if the data is a Snowpark DataFrame or a Pandas DataFrame, convert it to a Snowpark DataFrame if needed, and save it to the specified table.
`to_config`()	Converts the dataset instance into a dictionary-based configuration for serialization.

DEFAULT_LOAD_ARGS: dict[str, Any] = {}¶

DEFAULT_SAVE_ARGS: dict[str, Any] = {}¶

__init__(*, table_name, schema=None, database=None, load_args=None, save_args=None, credentials=None, session=None, metadata=None)[source]¶

Creates a new instance of SnowparkTableDataset.

Parameters:

table_name (str) – The table name to load or save data to.
schema (Optional[str]) – Name of the schema where table_name is. Optional as can be provided as part of credentials dictionary. Argument value takes priority over one provided in credentials if any.
database (Optional[str]) – Name of the database where schema is. Optional as can be provided as part of credentials dictionary. Argument value takes priority over one provided in credentials if any.
load_args (Optional[dict[str, Any]]) – Currently not used
save_args (Optional[dict[str, Any]]) – Provided to underlying snowpark save_as_table To find all supported arguments, see here: https://docs.snowflake.com/en/developer-guide/snowpark/reference/python/api/snowflake.snowpark.DataFrameWriter.saveAsTable.html
credentials (Optional[dict[str, Any]]) – A dictionary with a snowpark connection string. To find all supported arguments, see here: https://docs.snowflake.com/en/user-guide/python-connector-api.html#connect
metadata (Optional[dict[str, Any]]) – Any arbitrary metadata. This is ignored by Kedro, but may be consumed by users or external plugins.

exists()[source]¶

Checks whether a dataset’s output already exists by calling the provided _exists() method.

Return type:: bool
Returns:: Flag indicating whether the output already exists.
Raises:: DatasetError – when underlying exists method raises error.

classmethod from_config(name, config, load_version=None, save_version=None)[source]¶

Create a dataset instance using the configuration provided.

Parameters:

name (str) – Data set name.
config (dict[str, Any]) – Data set config dictionary.
load_version (Optional[str]) – Version string to be used for load operation if the dataset is versioned. Has no effect on the dataset if versioning was not enabled.
save_version (Optional[str]) – Version string to be used for save operation if the dataset is versioned. Has no effect on the dataset if versioning was not enabled.

Return type:

AbstractDataset

Returns:

An instance of an AbstractDataset subclass.

Raises:

DatasetError – When the function fails to create the dataset from its config.

load()[source]¶

Load data from a specified database table.

Returns:: The loaded data as a Snowpark DataFrame.
Return type:: DataFrame

release()[source]¶

Release any cached data.

Raises:: DatasetError – when underlying release method raises error.
Return type:: None

save(data)[source]¶

Check if the data is a Snowpark DataFrame or a Pandas DataFrame, convert it to a Snowpark DataFrame if needed, and save it to the specified table.

Parameters:: data (pd.DataFrame | DataFrame) – The data to save.
Return type:: None

property session: Session¶: Retrieve or create a session. :returns: The current session associated with the object. :rtype: Session

to_config()[source]¶

Converts the dataset instance into a dictionary-based configuration for serialization. Ensures that any subclass-specific details are handled, with additional logic for versioning and caching implemented for CachedDataset.

Adds a key for the dataset’s type using its module and class name and includes the initialization arguments.

For CachedDataset it extracts the underlying dataset’s configuration, handles the versioned flag and removes unnecessary metadata. It also ensures the embedded dataset’s configuration is appropriately flattened or transformed.

If the dataset has a version key, it sets the versioned flag in the configuration.

Removes the metadata key from the configuration if present.

Return type:: dict[str, Any]
Returns:: A dictionary containing the dataset’s type and initialization arguments.