kedro.config.TemplatedConfigLoader¶
- class kedro.config.TemplatedConfigLoader(conf_source, env=None, runtime_params=None, config_patterns=None, *, base_env='base', default_run_env='local', globals_pattern=None, globals_dict=None)[source]¶
Extension of the
ConfigLoader
class that allows for template values, wrapped in brackets like: ${…}, to be automatically formatted based on the configs.The easiest way to use this class is by setting the CONFIG_LOADER_CLASS constant in settings.py.
Example:
# in settings.py from kedro.config import TemplatedConfigLoader CONFIG_LOADER_CLASS = TemplatedConfigLoader CONFIG_LOADER_ARGS = { "globals_pattern": "*globals.yml", }
The contents of the dictionary resulting from the globals_pattern get merged with the
globals_dict
. In case of conflicts, the keys inglobals_dict
take precedence. If the formatting key is missing from the dictionary, the default template value is used (the format is “${key|default value}”). If no default is set, aValueError
will be raised.Global parameters can be namespaced as well. An example could work as follows:
globals.yml
bucket: "my_s3_bucket" environment: "dev" datasets: csv: "pandas.CSVDataSet" spark: "spark.SparkDataSet" folders: raw: "01_raw" int: "02_intermediate" pri: "03_primary" fea: "04_feature"
catalog.yml
raw_boat_data: type: "${datasets.spark}" filepath: "s3a://${bucket}/${environment}/${folders.raw}/boats.csv" file_format: parquet raw_car_data: type: "${datasets.csv}" filepath: "s3://${bucket}/data/${environment}/${folders.raw}/cars.csv"
This uses
jmespath
in the background. For more information see: https://github.com/jmespath/jmespath.py and https://jmespath.org/.Attributes
Property method to return deduplicated configuration paths.
Methods
clear
()copy
()fromkeys
(iterable[, value])get
(*patterns)Tries to resolve the template variables in the config dictionary provided by the
ConfigLoader
(super class)get
method using the dictionary of replacement values obtained in the__init__
method.items
()keys
()pop
(k[,d])If key is not found, d is returned if given, otherwise KeyError is raised.
popitem
()as a 2-tuple; but raise KeyError if D is empty.
setdefault
(k[,d])update
([E, ]**F)If E present and has a .keys() method, does: for k in E: D[k] = E[k] If E present and lacks .keys() method, does: for (k, v) in E: D[k] = v In either case, this is followed by: for k, v in F.items(): D[k] = v
values
()- __init__(conf_source, env=None, runtime_params=None, config_patterns=None, *, base_env='base', default_run_env='local', globals_pattern=None, globals_dict=None)[source]¶
Instantiates a
TemplatedConfigLoader
.- Parameters
conf_source – Path to use as root directory for loading configuration.
env – Environment that will take precedence over base.
runtime_params – Extra parameters passed to a Kedro run.
config_patterns – Regex patterns that specify the naming convention for configuration files so they can be loaded. Can be customised by supplying config_patterns as in CONFIG_LOADER_ARGS in settings.py.
base_env –
default_run_env –
globals_pattern – Optional keyword-only argument specifying a glob pattern. Files that match the pattern will be loaded as a formatting dictionary.
globals_dict – Optional keyword-only argument specifying a formatting dictionary. This dictionary will get merged with the globals dictionary obtained from the globals_pattern. In case of duplicate keys, the
globals_dict
keys take precedence.
- clear() None. Remove all items from D. ¶
- property conf_paths¶
Property method to return deduplicated configuration paths.
- copy()¶
- classmethod fromkeys(iterable, value=None)¶
- get(*patterns)[source]¶
Tries to resolve the template variables in the config dictionary provided by the
ConfigLoader
(super class)get
method using the dictionary of replacement values obtained in the__init__
method.- Parameters
*patterns – Glob patterns to match. Files, which names match any of the specified patterns, will be processed.
- Returns
A Python dictionary with the combined configuration from all configuration files. Note: any keys that start with _ will be ignored. String values wrapped in ${…} will be replaced with the result of the corresponding JMESpath expression evaluated against globals.
- Raises
ValueError – malformed config found.
- items() a set-like object providing a view on D's items ¶
- keys() a set-like object providing a view on D's keys ¶
- pop(k[, d]) v, remove specified key and return the corresponding value. ¶
If key is not found, d is returned if given, otherwise KeyError is raised.
- popitem() (k, v), remove and return some (key, value) pair ¶
as a 2-tuple; but raise KeyError if D is empty.
- setdefault(k[, d]) D.get(k,d), also set D[k]=d if k not in D ¶
- update([E, ]**F) None. Update D from mapping/iterable E and F. ¶
If E present and has a .keys() method, does: for k in E: D[k] = E[k] If E present and lacks .keys() method, does: for (k, v) in E: D[k] = v In either case, this is followed by: for k, v in F.items(): D[k] = v
- values() an object providing a view on D's values ¶