Settings¶
Provides global settings and options for accessing OBS services like swift
and configuring methods such as SwiftPath.upload and SwiftPath.download.
Settings are stored internally as nested dictionaries. When using update or
use, this dictionary structure should be followed.
Examples:
An example settings dictionary.
example_settings = { 'swift:delete': { 'object_threads': 10 }, 'swift:download': { 'container_threads': 10, 'object_threads': 10, 'shuffle': True, 'skip_identical': True }, 'swift:upload': { 'changed': False, 'checksum': True, 'leave_segments': True, 'object_threads': 10, 'segment_size': 1073741824, 'segment_threads': 10, 'skip_identical': False, 'use_slo': True } }
Precedence¶
Settings can be configured in the following ways in order of precedence:
When using the CLI, a configuration file specified using the
--configflag.Setting environment variables.
User-specified configuration in a
~/.stor.cfgfile.
Default Settings¶
[stor]
[s3]
# See boto3 docs for more detail on these parameters - all passed directly to boto3.session.Session *if* set
# aws_access_key_id (string) -- AWS access key ID
aws_access_key_id =
# aws_secret_access_key (string) -- AWS secret access key
aws_secret_access_key =
# aws_session_token (string) -- AWS temporary session token
aws_session_token =
# region_name (string) -- Default region when creating new connections
region_name =
# profile_name (string) -- The name of a profile to use. If not given, then the default profile is used.
profile_name =
[s3:upload]
# segment_size (int|str): Upload files in segments no larger than
# <segment_size> (in bytes). Sizes may also be expressed as bytes with
# the B suffix, kilobytes with the K suffix, megabytes with the M suffix
# or gigabytes with the G suffix.
segment_size = 8388608 # 8 MB
# object_threads (int): The number of threads to use when uploading full
# objects to s3.
object_threads = 10
# segment_threads (int): The number of threads to use when uploading object
# segments to s3 in multipart upload.
segment_threads = 10
[s3:download]
# segment_size (int|str): Download files in segments no larger than
# <segment_size> (in bytes). Sizes may also be expressed as bytes with
# the B suffix, kilobytes with the K suffix, megabytes with the M suffix
# or gigabytes with the G suffix.
segment_size = 8388608 # 8 MB
# object_threads (int): The number of threads to use when downloading objects
# from s3.
object_threads = 10
# segment_threads (int): The number of threads to use when downloading object
# segments to s3 in multipart download.
segment_threads = 10
[swift]
# username (str): The swift username. If not set, the ``OS_USERNAME``
# environment variable will be used.
username =
# password (str): The swift password. If not set, the ``OS_PASSWORD``
# environment variable will be used.
password =
# auth_url (str): The swift authentication URL. If not set, the
# ``OS_AUTH_URL`` environment variable will be used.
auth_url =
# temp_url_key (str): The key for generating temporary URLs.
# If not set, the ``OS_TEMP_URL_KEY environment variable will be used.
temp_url_key =
# num_retries (int): The number of times to retry. Uses the
# ``OS_NUM_RETRIES`` environment variable or defaults to 0.
num_retries = 0
[swift:delete]
# object_threads (int): The number of threads to use when deleting objects
object_threads = 10
[swift:upload]
# segment_size (int|str): Upload files in segments no larger than
# <segment_size> (in bytes) and then create a "manifest" file that will
# download all the segments as if it were the original file. Sizes may also
# be expressed as bytes with the B suffix, kilobytes with the K suffix,
# megabytes with the M suffix or gigabytes with the G suffix.
segment_size = 1073741824 # 1 GB
# object_threads (int): The number of threads to use when uploading full
# objects.
object_threads = 10
# segment_threads (int): The number of threads to use when uploading object
# segments.
segment_threads = 10
# use_slo (bool): When used in conjunction with segment_size, it will create
# a Static Large Object instead of the default Dynamic Large Object.
use_slo = True
# leave_segments (bool): Indicates that you want the older segments of
# manifest objects left alone (in the case of overwrites).
leave_segments = True
# changed (bool): Upload only files that have changed since last upload.
changed = False
# skip_identical (bool): Skip uploading files that are identical on both
# sides. Note this incurs reading the contents of all pre-existing local
# files.
skip_identical = False
# checksum (bool): Peform checksum validation of upload.
checksum = True
[swift:download]
# object_threads (int): The amount of threads to use for downloading objects.
object_threads = 10
# container_threads (int): The amount of threads to use for downloading
# containers.
container_threads = 10
# skip_identical (bool): Skip downloading files that are identical on both
# sides. Note this incurs reading the contents of all pre-existing local
# files.
skip_identical = True
# shuffle (bool): When downloading a complete container, download order is
# randomised in order to reduce the load on individual drives when doing
# threaded downloads. Disable this option to submit download jobs to
# the thread pool in the order they are listed in the object store.
shuffle = True
[dx]
# auth_token (str): the login token for dxpy. If not set, ``DX_AUTH_TOKEN``
# env variable will be used
auth_token =
# wait_on_close (int): The number of seconds to wait for DXFile to go to
# closed state. Passed to DXFile.wait_on_close
wait_on_close = 0
# file_proxy_url (str): A custom proxy service for dx paths. It's assumed that
# you can construct URLs as urljoin({file_proxy_url}, f'{project}/{resource}').
# If this is set, temp_url() will generate HTTP paths using the proxy;
# otherwise it'll use make_download_url().
# May also set via DX_FILE_PROXY_URL env var as well.
file_proxy_url =
Settings API¶
-
stor.settings.get()[source]¶ Returns a deep copy of global settings as a dictionary.
This function should always be used rather than accessing
global_settingsdirectly.
-
stor.settings.parse_config_file(filename)[source]¶ Parses a configuration file and returns a settings dictionary.