kedro.config.TemplatedConfigLoader¶
-
class
kedro.config.
TemplatedConfigLoader
(conf_source, env=None, runtime_params=None, config_patterns=None, *, base_env='base', default_run_env='local', globals_pattern=None, globals_dict=None)[source]¶ Extension of the
ConfigLoader
class that allows for template values, wrapped in brackets like: ${…}, to be automatically formatted based on the configs.The easiest way to use this class is by setting the CONFIG_LOADER_CLASS constant in settings.py.
Example:
# in settings.py from kedro.config import TemplatedConfigLoader CONFIG_LOADER_CLASS = TemplatedConfigLoader CONFIG_LOADER_ARGS = { "globals_pattern": "*globals.yml", }
The contents of the dictionary resulting from the globals_pattern get merged with the
globals_dict
. In case of conflicts, the keys inglobals_dict
take precedence. If the formatting key is missing from the dictionary, the default template value is used (the format is “${key|default value}”). If no default is set, aValueError
will be raised.Global parameters can be namespaced as well. An example could work as follows:
globals.yml
bucket: "my_s3_bucket" environment: "dev" datasets: csv: "pandas.CSVDataSet" spark: "spark.SparkDataSet" folders: raw: "01_raw" int: "02_intermediate" pri: "03_primary" fea: "04_feature"
catalog.yml
raw_boat_data: type: "${datasets.spark}" filepath: "s3a://${bucket}/${environment}/${folders.raw}/boats.csv" file_format: parquet raw_car_data: type: "${datasets.csv}" filepath: "s3://${bucket}/data/${environment}/${folders.raw}/cars.csv"
This uses
jmespath
in the background. For more information see: https://github.com/jmespath/jmespath.py and https://jmespath.org/.Attributes
Property method to return deduplicated configuration paths.
Methods
clear
()copy
()fromkeys
(iterable[, value])get
(*patterns)Tries to resolve the template variables in the config dictionary provided by the
ConfigLoader
(super class)get
method using the dictionary of replacement values obtained in the__init__
method.items
()keys
()pop
(k[,d])If key is not found, d is returned if given, otherwise KeyError is raised.
popitem
()as a 2-tuple; but raise KeyError if D is empty.
setdefault
(k[,d])update
([E, ]**F)If E present and has a .keys() method, does: for k in E: D[k] = E[k] If E present and lacks .keys() method, does: for (k, v) in E: D[k] = v In either case, this is followed by: for k, v in F.items(): D[k] = v
values
()-
__init__
(conf_source, env=None, runtime_params=None, config_patterns=None, *, base_env='base', default_run_env='local', globals_pattern=None, globals_dict=None)[source]¶ Instantiates a
TemplatedConfigLoader
.- Parameters
conf_source (
str
) – Path to use as root directory for loading configuration.env (
Optional
[str
]) – Environment that will take precedence over base.runtime_params (
Optional
[Dict
[str
,Any
]]) – Extra parameters passed to a Kedro run.config_patterns (
Optional
[Dict
[str
,List
[str
]]]) – Regex patterns that specify the naming convention for configuration files so they can be loaded. Can be customised by supplying config_patterns as in CONFIG_LOADER_ARGS in settings.py.base_env (
str
) –default_run_env (
str
) –globals_pattern (
Optional
[str
]) – Optional keyword-only argument specifying a glob pattern. Files that match the pattern will be loaded as a formatting dictionary.globals_dict (
Optional
[Dict
[str
,Any
]]) – Optional keyword-only argument specifying a formatting dictionary. This dictionary will get merged with the globals dictionary obtained from the globals_pattern. In case of duplicate keys, theglobals_dict
keys take precedence.
-
clear
() → None. Remove all items from D.¶
-
property
conf_paths
¶ Property method to return deduplicated configuration paths.
-
copy
()¶
-
classmethod
fromkeys
(iterable, value=None)¶
-
get
(*patterns)[source]¶ Tries to resolve the template variables in the config dictionary provided by the
ConfigLoader
(super class)get
method using the dictionary of replacement values obtained in the__init__
method.- Parameters
*patterns – Glob patterns to match. Files, which names match any of the specified patterns, will be processed.
- Return type
Dict
[str
,Any
]- Returns
A Python dictionary with the combined configuration from all configuration files. Note: any keys that start with _ will be ignored. String values wrapped in ${…} will be replaced with the result of the corresponding JMESpath expression evaluated against globals.
- Raises
ValueError – malformed config found.
-
items
() → a set-like object providing a view on D’s items¶
-
keys
() → a set-like object providing a view on D’s keys¶
-
pop
(k[, d]) → v, remove specified key and return the corresponding value.¶ If key is not found, d is returned if given, otherwise KeyError is raised.
-
popitem
() → (k, v), remove and return some (key, value) pair¶ as a 2-tuple; but raise KeyError if D is empty.
-
setdefault
(k[, d]) → D.get(k,d), also set D[k]=d if k not in D¶
-
update
([E, ]**F) → None. Update D from mapping/iterable E and F.¶ If E present and has a .keys() method, does: for k in E: D[k] = E[k] If E present and lacks .keys() method, does: for (k, v) in E: D[k] = v In either case, this is followed by: for k, v in F.items(): D[k] = v
-
values
() → an object providing a view on D’s values¶
-