kedro.config.OmegaConfigLoader¶
-
class
kedro.config.
OmegaConfigLoader
(conf_source, env=None, runtime_params=None, *, config_patterns=None, base_env='base', default_run_env='local')[source]¶ Recursively scan directories (config paths) contained in
conf_source
for configuration files with ayaml
,yml
orjson
extension, load and merge them throughOmegaConf
(https://omegaconf.readthedocs.io/) and return them in the form of a config dictionary.The first processed config path is the
base
directory insideconf_source
. The optionalenv
argument can be used to specify a subdirectory ofconf_source
to process as a config path afterbase
.When the same top-level key appears in any two config files located in the same (sub)directory, a
ValueError
is raised.When the same key appears in any two config files located in different (sub)directories, the last processed config path takes precedence and overrides this key and any sub-keys.
You can access the different configurations as follows:
import logging.config from kedro.config import OmegaConfigLoader from kedro.framework.project import settings conf_path = str(project_path / settings.CONF_SOURCE) conf_loader = OmegaConfigLoader(conf_source=conf_path, env="local") conf_logging = conf_loader["logging"] logging.config.dictConfig(conf_logging) # set logging conf conf_catalog = conf_loader["catalog"] conf_params = conf_loader["parameters"]
OmegaConf
supports variable interpolation in configuration https://omegaconf.readthedocs.io/en/2.2_branch/usage.html#merging-configurations. It is recommended to use this instead of yaml anchors with theOmegaConfigLoader
.This version of the
OmegaConfigLoader
does not support any of the built-inOmegaConf
resolvers. Support for resolvers might be added in future versions.To use this class, change the setting for the CONFIG_LOADER_CLASS constant in settings.py.
Example:
# in settings.py from kedro.config import OmegaConfigLoader CONFIG_LOADER_CLASS = OmegaConfigLoader
Methods
clear
()copy
()fromkeys
(iterable[, value])get
(key[, default])Retrieve the corresponding layout by the string key.
items
()keys
()load_and_merge_dir_config
(conf_path, patterns)Recursively load and merge all configuration files in a directory using OmegaConf, which satisfy a given list of glob patterns from a specific path.
pop
(k[,d])If key is not found, d is returned if given, otherwise KeyError is raised.
popitem
()as a 2-tuple; but raise KeyError if D is empty.
setdefault
(k[,d])update
([E, ]**F)If E present and has a .keys() method, does: for k in E: D[k] = E[k] If E present and lacks .keys() method, does: for (k, v) in E: D[k] = v In either case, this is followed by: for k, v in F.items(): D[k] = v
values
()-
__init__
(conf_source, env=None, runtime_params=None, *, config_patterns=None, base_env='base', default_run_env='local')[source]¶ Instantiates a
OmegaConfigLoader
.- Parameters
conf_source (
str
) – Path to use as root directory for loading configuration.env (
Optional
[str
]) – Environment that will take precedence over base.runtime_params (
Optional
[Dict
[str
,Any
]]) – Extra parameters passed to a Kedro run.config_patterns (
Optional
[Dict
[str
,List
[str
]]]) – Regex patterns that specify the naming convention for configuration files so they can be loaded. Can be customised by supplying config_patterns as in CONFIG_LOADER_ARGS in settings.py.base_env (
str
) – Name of the base environment. Defaults to “base”. This is used in the conf_paths property method to construct the configuration paths.default_run_env (
str
) – Name of the default run environment. Defaults to “local”. Can be overridden by supplying the env argument.
-
clear
() → None. Remove all items from D.¶
-
copy
()¶
-
classmethod
fromkeys
(iterable, value=None)¶
-
get
(key, default=None)¶ Retrieve the corresponding layout by the string key.
When there isn’t an exact match, all the existing keys in the layout map will be treated as a regex and map against the input key again. The first match will be returned, based on the key insertion order. Return None if there isn’t any match found.
- Parameters
key – the string key as the query for the layout.
- Returns
Corresponding layout based on the query.
-
items
() → a set-like object providing a view on D’s items¶
-
keys
() → a set-like object providing a view on D’s keys¶
-
load_and_merge_dir_config
(conf_path, patterns, read_environment_variables=False)[source]¶ Recursively load and merge all configuration files in a directory using OmegaConf, which satisfy a given list of glob patterns from a specific path.
- Parameters
conf_path (
str
) – Path to configuration directory.patterns (
Iterable
[str
]) – List of glob patterns to match the filenames against.read_environment_variables (
Optional
[bool
]) – Whether to resolve environment variables.
- Raises
MissingConfigException – If configuration path doesn’t exist or isn’t valid.
ValueError – If two or more configuration files contain the same key(s).
ParserError – If config file contains invalid YAML or JSON syntax.
- Return type
Dict
[str
,Any
]- Returns
Resulting configuration dictionary.
-
pop
(k[, d]) → v, remove specified key and return the corresponding value.¶ If key is not found, d is returned if given, otherwise KeyError is raised.
-
popitem
() → (k, v), remove and return some (key, value) pair¶ as a 2-tuple; but raise KeyError if D is empty.
-
setdefault
(k[, d]) → D.get(k,d), also set D[k]=d if k not in D¶
-
update
([E, ]**F) → None. Update D from mapping/iterable E and F.¶ If E present and has a .keys() method, does: for k in E: D[k] = E[k] If E present and lacks .keys() method, does: for (k, v) in E: D[k] = v In either case, this is followed by: for k, v in F.items(): D[k] = v
-
values
() → an object providing a view on D’s values¶
-