Skip to content

SparkHiveDataset

SparkHiveDataset loads and saves data to Hive tables using Apache Spark.

kedro_datasets.spark.SparkHiveDataset

SparkHiveDataset(
    *,
    database,
    table,
    write_mode="errorifexists",
    table_pk=None,
    save_args=None,
    metadata=None
)

Bases: AbstractDataset[DataFrame, DataFrame]

SparkHiveDataset loads and saves Spark dataframes stored on Hive. This dataset also handles some incompatible file types such as using partitioned parquet on hive which will not normally allow upserts to existing data without a complete replacement of the existing file/partition.

This Dataset has some key assumptions:

  • Schemas do not change during the pipeline run (defined PKs must be present for the duration of the pipeline).
  • Tables are not being externally modified during upserts. The upsert method is NOT ATOMIC to external changes to the target table while executing. Upsert methodology works by leveraging Spark DataFrame execution plan checkpointing.

Examples:

Using the YAML API:

hive_dataset:
  type: spark.SparkHiveDataset
  database: hive_database
  table: table_name
  write_mode: overwrite

An external Hive table can be configured by passing Spark's path option through save_args. Spark stores the table metadata in the Hive metastore and the table data at the supplied location:

external_hive_dataset:
  type: spark.SparkHiveDataset
  database: hive_database
  table: external_table
  write_mode: overwrite
  save_args:
    format: parquet
    path: s3a://bucket/path/to/external_table
    partitionBy:
      - name

If save_args.path is omitted, Spark's default managed-table location is used. The path must be a non-empty local or Hadoop-compatible filesystem path when it is provided. The existing append, error, errorifexists, overwrite and upsert modes are passed through unchanged; overwrite and upsert replace data at the configured external location.

Using the Python API:

>>> from kedro_datasets.spark import SparkHiveDataset
>>> from pyspark.sql import SparkSession
>>> from pyspark.sql.types import StructField, StringType, IntegerType, StructType
>>>
>>> schema = StructType(
...     [StructField("name", StringType(), True), StructField("age", IntegerType(), True)]
... )
>>> data = [("Alex", 31), ("Bob", 12), ("Clarke", 65), ("Dave", 29)]
>>> spark_df = SparkSession.builder.getOrCreate().createDataFrame(data, schema)
>>>
>>> dataset = SparkHiveDataset(
...     database="test_database", table="test_table", write_mode="overwrite"
... )
>>> dataset.save(spark_df)
>>> reloaded = dataset.load()
>>> reloaded.take(4)

Parameters:

  • database (str) –

    The name of the hive database.

  • table (str) –

    The name of the table within the database.

  • write_mode (str, default: 'errorifexists' ) –

    insert, upsert or overwrite are supported.

  • table_pk (list[str] | None, default: None ) –

    If performing an upsert, this identifies the primary key columns used to resolve preexisting data. Is required for write_mode="upsert".

  • save_args (dict[str, Any] | None, default: None ) –

    Optional mapping of any options, passed to the DataFrameWriter.saveAsTable as kwargs. Key example of this is partitionBy which allows data partitioning on a list of column names. Set path to a non-empty local or Hadoop-compatible filesystem path to create an external table at that location. If omitted, Spark uses its default managed-table location. Other HiveOptions can be found here: https://spark.apache.org/docs/latest/sql-data-sources-hive-tables.html#specifying-storage-format-for-hive-tables

  • metadata (dict[str, Any] | None, default: None ) –

    Any arbitrary metadata. This is ignored by Kedro, but may be consumed by users or external plugins.

Note

For users leveraging the upsert functionality, a checkpoint directory must be set, e.g. using spark.sparkContext.setCheckpointDir("/path/to/dir") or directly in the Spark conf folder.

Raises:

Source code in kedro_datasets/spark/spark_hive_dataset.py
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
def __init__(  # noqa: PLR0913
    self,
    *,
    database: str,
    table: str,
    write_mode: str = "errorifexists",
    table_pk: list[str] | None = None,
    save_args: dict[str, Any] | None = None,
    metadata: dict[str, Any] | None = None,
) -> None:
    """Creates a new instance of ``SparkHiveDataset``.

    Args:
        database: The name of the hive database.
        table: The name of the table within the database.
        write_mode: ``insert``, ``upsert`` or ``overwrite`` are supported.
        table_pk: If performing an upsert, this identifies the primary key columns used to
            resolve preexisting data. Is required for ``write_mode="upsert"``.
        save_args: Optional mapping of any options,
            passed to the `DataFrameWriter.saveAsTable` as kwargs.
            Key example of this is `partitionBy` which allows data partitioning
            on a list of column names.
            Set `path` to a non-empty local or Hadoop-compatible filesystem path
            to create an external table at that location. If omitted, Spark uses
            its default managed-table location.
            Other `HiveOptions` can be found here:
            https://spark.apache.org/docs/latest/sql-data-sources-hive-tables.html#specifying-storage-format-for-hive-tables
        metadata: Any arbitrary metadata.
            This is ignored by Kedro, but may be consumed by users or external plugins.

    Note:
        For users leveraging the `upsert` functionality,
        a `checkpoint` directory must be set, e.g. using
        `spark.sparkContext.setCheckpointDir("/path/to/dir")`
        or directly in the Spark conf folder.

    Raises:
        DatasetError: Invalid configuration supplied
    """
    _write_modes = ["append", "error", "errorifexists", "upsert", "overwrite"]
    if write_mode not in _write_modes:
        valid_modes = ", ".join(_write_modes)
        raise DatasetError(
            f"Invalid 'write_mode' provided: {write_mode}. "
            f"'write_mode' must be one of: {valid_modes}"
        )
    if write_mode == "upsert" and not table_pk:
        raise DatasetError("'table_pk' must be set to utilise 'upsert' read mode")

    self._write_mode = write_mode
    self._table_pk = table_pk or []
    self._database = database
    self._table = table
    self._full_table_address = f"{database}.{table}"
    self._save_args = deepcopy(self.DEFAULT_SAVE_ARGS)
    if save_args is not None:
        self._save_args.update(save_args)

    if "path" in self._save_args:
        external_location = self._save_args["path"]
        if isinstance(external_location, os.PathLike):
            external_location = os.fspath(external_location)
            self._save_args["path"] = external_location
        if not isinstance(external_location, str) or not external_location.strip():
            raise DatasetError(
                "'save_args.path' must be a non-empty local or Hadoop-compatible "
                "filesystem path when configuring an external Hive table"
            )

    self._external_location = self._save_args.get("path")
    self._format = self._save_args.pop("format", None) or "hive"
    self._eager_checkpoint = self._save_args.pop("eager_checkpoint", None) or True

    self.metadata = metadata

DEFAULT_SAVE_ARGS class-attribute instance-attribute

DEFAULT_SAVE_ARGS = {}

metadata instance-attribute

metadata = metadata

load

load()
Source code in kedro_datasets/spark/spark_hive_dataset.py
186
187
def load(self) -> DataFrame:
    return get_spark().read.table(self._full_table_address)

save

save(data)
Source code in kedro_datasets/spark/spark_hive_dataset.py
189
190
191
192
193
194
195
196
197
198
199
200
def save(self, data: DataFrame) -> None:
    self._validate_save(data)
    if self._write_mode == "upsert":
        # check if _table_pk is a subset of df columns
        if not set(self._table_pk) <= set(self.load.__wrapped__(self).columns):  # type: ignore[attr-defined]
            raise DatasetError(
                f"Columns {str(self._table_pk)} selected as primary key(s) not found in "
                f"table {self._full_table_address}"
            )
        self._upsert_save(data=data)
    else:
        self._create_hive_table(data=data)