Swift

Retry Settings

stor.swift.initial_retry_sleep = 1

The time to sleep before the first retry

stor.swift.retry_sleep_function = <function _default_retry_sleep_function>

The function that increases sleep time when retrying.

This function needs to take two integer arguments (time slept last attempt, attempt number) and return a time to sleep in seconds.

SwiftPath

class stor.swift.SwiftPath(pth)

Provides the ability to manipulate and access resources on swift with a similar interface to the path library.

open(mode='r')

Opens a OBSFile that can be read or written to and is uploaded to the remote service.

For examples of reading and writing opened objects, view OBSFile.

Parameters

• mode (str) – The mode of object IO. Currently supports reading (“r” or “rb”) and writing (“w”, “wb”)

• encoding (str) – text encoding to use. Defaults to locale.getpreferredencoding(False)

Returns

The file object for Swift/S3/DX.

Return type

OBSFile

Raises

• SwiftError – A swift client error occurred.

• DNAnexusError – A dxpy client error occured.

• RemoteError – A s3 client error occurred.

glob(pattern, num_objs_cond=None, **retry_args)

Globs all objects in the path with the pattern.

This glob is only compatible with patterns that end in * because of swift’s inability to do searches other than prefix queries.

Note that this method assumes the current resource is a directory path and treats it as such. For example, if the user has a swift path of swift://tenant/container/my_dir (without the trailing slash), this method will perform a swift query with a prefix of mydir/pattern.

This method retries num_retries times if swift is unavailable or if the number of globbed patterns does not match the condition condition. View module-level documentation for more information about configuring retry logic at the module or method level.

Parameters

• pattern (str) – The pattern to match. The pattern can only have up to one ‘*’ at the end.

• condition (function(results) -> bool) – The method will only return when the number of results matches the condition.

Returns

Every matching path.

Return type

List[SwiftPath]

Raises

• SwiftError – A swift client error occurred.

• ConditionNotMetError – Results were returned, but they did not meet the condition.

exists(**retry_args)

Checks existence of the path.

Returns True if the path exists, False otherwise.

See module-level retry documentation for more.

Returns

True if the path exists, False otherwise.

Return type

bool

Raises

SwiftError – A non-404 swift client error occurred.

remove(**retry_args)

Removes a single object.

This method retries num_retries times if swift is unavailable. View module-level documentation for more information about configuring retry logic at the module or method level.

Raises

• ValueError – The path is invalid.

• SwiftError – A swift client error occurred.

rmtree(**retry_args)

Removes a resource and all of its contents. This method retries num_retries times if swift is unavailable. View module-level documentation for more information about configuring retry logic at the module or method level.

Note that when removing a container, the associated segment container will also be removed if it exists. So, if one removes swift://tenant/container, swift://tenant/container_segments will also be deleted.

Note

Calling rmtree on a directory marker will delete everything under the directory marker but not the marker itself.

Raises

• SwiftError – A swift client error occurred.

• ConditionNotMetError – Listing the objects after rmtree returns results, indicating something went wrong with the rmtree operation

copy(dest, swift_retry_args=None)

Copies a source file to a destination file.

Note that this utility can be called from either OBS, posix, or windows paths created with stor.Path.

Parameters

• source (path|str) – The source directory to copy from

• dest (path|str) – The destination file or directory.

• swift_retry_options (dict) – Optional retry arguments to use for swift upload or download. View the swift module-level documentation for more information on retry arguments

Examples

Copying a swift file to a local path behaves as follows:

>>> import stor
>>> swift_p = 'swift://tenant/container/dir/file.txt'
>>> # file.txt will be copied to other_dir/other_file.txt
>>> stor.copy(swift_p, 'other_dir/other_file.txt')

Copying from a local path to swift behaves as follows:

>>> from stor import Path
>>> local_p = Path('my/local/file.txt')
>>> # File will be uploaded to swift://tenant/container/dir/my_file.txt
>>> local_p.copy('swift://tenant/container/dir/')

Because of the ambiguity in whether a remote target is a file or directory, copy() will error on ambiguous paths.

>>> local_p.copy('swift://tenant/container/dir')
Traceback (most recent call last):
...
ValueError: OBS destination must be file with extension or directory with slash

copytree(dest, copy_cmd='cp -r')

Copies a source directory to a destination directory. Assumes that paths are capable of being copied to/from.

Note that this function uses shutil.copytree by default, meaning that a posix or windows destination must not exist beforehand.

For example, assume the following file hierarchy:

a/
- b/
- - 1.txt

Doing a copytree from a to a new posix destination of c is performed with:

Path('a').copytree('c')

The end result for c looks like:

c/
- b/
- - 1.txt

Note that the user can override which copy command is used for posix copies, and it is up to them to ensure that their code abides by the semantics of the provided copy command. This function has been tested in production using the default command of cp -r and using mcp -r.

Using OBS source and destinations work in a similar manner. Assume the destination is a swift path and we upload the same a folder:

Path('a').copytree('swift://tenant/container/folder')

The end swift result will have one object:

Path('swift://tenant/container/folder/b/1.txt')

Similarly one can do:

Path('swift://tenant/container/folder/').copytree('c')

The end result for c looks the same as the above posix example:

c/
- b/
- - 1.txt

Parameters

• source (path|str) – The source directory to copy from

• dest (path|str) – The directory to copy to. Must not exist if its a posix directory

• copy_cmd (str) – If copying to / from posix or windows, this command is used instead of shutil.copytree

• use_manifest (bool, default False) – See SwiftPath.upload and SwiftPath.download.

• condition (function(results) -> bool) – See SwiftPath.upload and SwiftPath.download.

• headers (List[str]) – See SwiftPath.upload.

Raises

• ValueError – if two OBS paths are specified

• OSError – if destination is a posix path and it already exists

upload(to_upload, segment_size=DEFAULT_SEGMENT_SIZE, use_slo=True, segment_container=None, leave_segments=False, changed=False, object_name=None, object_threads=10, segment_threads=10, condition=None, use_manifest=False, **retry_args)

Uploads a list of files and directories to swift.

This method retries num_retries times if swift is unavailable or if the returned upload result does not match the condition condition. View module-level documentation for more information about configuring retry logic at the module or method level.

For example:

with path('/path/to/upload/dir'):
    path('swift://tenant/container').upload(['.'])

Notes

• This method upload on paths relative to the current directory. In order to load files relative to a target directory, use path as a context manager to change the directory.

• When large files are split into segments, they are uploaded to a segment container named .segments_${container_name}

Parameters

• to_upload (List) – A list of file names, directory names, or OBSUploadObject objects to upload.

• condition (function(results) -> bool) – The method will only return when the results of upload matches the condition. In the event of the condition never matching after retries, partially uploaded results will not be deleted. Note that users are not expected to write conditions for upload without an understanding of the structure of the results.

• use_manifest (bool) –

  Generate a data manifest and validate the upload results are in the manifest. In case of a single directory being uploaded, the manifest file will be created inside this directory. For example:

  stor.Path('swift://AUTH_foo/bar').upload(['logs'], use_manifest=True)
  

  The manifest will be located at swift://AUTH_foo/bar/logs/.data_manifest.csv

  Alternatively, when multiple directories are uploaded, manifest file will be created in the current directory. For example:

  stor.Path('swift://AUTH_foo/bar').upload(
      ['logs', 'test.txt'], use_manifest=True)
  

  The manifest will be located at swift://AUTH_foo/bar/.data_manifest.csv

• headers (List[str]) – A list of object headers to apply to every object. Note that these are not applied if passing OBSUploadObjects directly to upload. Headers must be specified as a list of colon-delimited strings, e.g. [‘X-Delete-After:1000’]

Raises

• SwiftError – A swift client error occurred.

• ConditionNotMetError – The condition argument did not pass after num_retries or the use_manifest option was turned on and the upload results could not be verified. Partially uploaded results are not deleted.

Returns

A list of every operation performed in the upload by the

underlying swift client

Return type

List[dict]

download(dest, object_threads=10, container_threads=10, condition=None, use_manifest=False, **retry_args)

Downloads a directory to a destination.

This method retries num_retries times if swift is unavailable or if the returned download result does not match the condition condition. View module-level documentation for more information about configuring retry logic at the module or method level.

Note that the destintation directory will be created automatically if it doesn’t exist.

Parameters

• dest (str) – The destination directory to download results to. The directory will be created if it doesn’t exist.

• condition (function(results) -> bool) – The method will only return when the results of download matches the condition. In the event of the condition never matching after retries, partially downloaded results will not be deleted. Note that users are not expected to write conditions for download without an understanding of the structure of the results.

• use_manifest (bool) – Perform the download and use the data manfest file to validate the download.

Raises

• SwiftError – A swift client error occurred.

• ConditionNotMetError – Results were returned, but they did not meet the condition.

Returns

A list of every operation performed in the download by the

underlying swift client

Return type

List[dict]

download_objects(dest, objects, object_threads=10, container_threads=10, **retry_args)

See baseclass for complete method documentation

Parameters

• dest (str) – The destination folder to download to. The directory will be created if it doesnt exist.

• objects (List[str|PosixPath|SwiftPath]) – The list of objects to download. The objects can paths relative to the download path or absolute swift paths. Any absolute swift path must be children of the download path

Returns

A mapping of all requested objs to their location on

disk

Return type

dict

Raises

ValueError – This method was called on a path that has no container

first(**retry_args)

Returns the first result from the list results of the path

See module-level retry documentation for more.

Raises

SwiftError – A swift client error occurred.

list(starts_with=None, limit=None, condition=None, use_manifest=False, **retry_args)

List contents using the resource of the path as a prefix.

This method retries num_retries times if swift is unavailable or if the number of objects returned does not match the condition condition. View module-level documentation for more information about configuring retry logic at the module or method level.

Parameters

• starts_with (str) – Allows for an additional search path to be appended to the resource of the swift path. Note that the current resource path is treated as a directory

• limit (int) – Limit the amount of results returned

• condition (function(results) -> bool) – The method will only return when the results matches the condition.

• use_manifest (bool) – Perform the list and use the data manfest file to validate the list.

Returns

Every path in the listing.

Return type

List[SwiftPath]

Raises

• SwiftError – A swift client error occurred.

• ConditionNotMetError – Results were returned, but they did not meet the condition.

post(options=None, **retry_args)

Post operations on the path.

This method retries num_retries times if swift is unavailable. View module-level documentation for more information about configuring retry logic at the module or method level.

Parameters

options (dict) –

A dictionary containing options to override the global options specified during the service object creation. These options are applied to all post operations performed by this call, unless overridden on a per object basis. Possible options are given below:

{
    'meta': [],   # Meta values will be prefixed with X-Object-Meta
                  # (or X-Container*  X-Account* depending on path type)
    'header': [], # Header values will not be manipulated like meta values
                  # when being posted.
    'read_acl': None,   # For containers only
    'write_acl': None,  # For containers only
    'sync_to': None,    # For containers only
    'sync_key': None    # For containers only
}

Raises

SwiftError – A swift client error occurred.

stat(**retry_args)

Performs a stat on the path.

Note that the path can be a tenant, container, or object. Using stat on a directory of objects will produce a NotFoundError.

This method retries num_retries times if swift is unavailable. View module-level documentation for more information about configuring retry logic at the module or method level.

The return value is dependent on if the path points to a tenant, container, or object.

For tenants, an example return dictionary is the following:

{
    'headers': {
        'content-length': u'0',
        'x-account-storage-policy-3xreplica-container-count': u'4655',
        'x-account-meta-temp-url-key': 'temp_url_key',
        'x-account-object-count': '428634',
        'x-timestamp': '1450732238.21562',
        'x-account-storage-policy-3xreplica-bytes-used': '175654877752',
        'x-trans-id': u'transaction_id',
        'date': u'Wed, 06 Apr 2016 18:30:28 GMT',
        'x-account-bytes-used': '175654877752',
        'x-account-container-count': '4655',
        'content-type': 'text/plain; charset=utf-8',
        'accept-ranges': 'bytes',
        'x-account-storage-policy-3xreplica-object-count': '428634'
    },
    'Account': 'AUTH_seq_upload_prod',
    # The number of containers in the tenant
    'Containers': 31,
    # The number of objects in the tenant
    'Objects': '19955615',
    # The total bytes used in the tenant
    'Bytes': '24890576770484',
    'Containers-in-policy-"3xreplica"': '31',
    'Objects-in-policy-"3xreplica"': '19955615',
    'Bytes-in-policy-"3xreplica"': '24890576770484',
    # The tenant ACLs. An empty dict is returned if the user
    # does not have admin privileges on the tenant
    'Account-Access': {
        'admin': ['swft_labprod_admin'],
        'read-only': ['seq_upload_rnd','swft_labprod'],
        'read-write': ['svc_svc_seq']
    }
}

For containers, an example return dictionary is the following:

{
    'headers': {
        'content-length': '0',
        'x-container-object-count': '99',
        'accept-ranges': 'bytes',
        'x-storage-policy': '3xReplica',
        'date': 'Wed, 06 Apr 2016 18:32:47 GMT',
        'x-timestamp': '1457631707.95682',
        'x-trans-id': 'transaction_id',
        'x-container-bytes-used': '5389060',
        'content-type': 'text/plain; charset=utf-8'
    },
    'Account': 'AUTH_seq_upload_prod',
    'Container': '2016-01',
    # The number of objects in the container
    'Objects': '43868',
    # The size of all objects in the container
    'Bytes': '55841489571',
    # Read and write ACLs for the container
    'Read-ACL': '',
    'Write-ACL': '',
    'Sync-To': '',
    'Sync-Key': ''
}

For objects, an example return dictionary is the following:

{
    'headers': {
        'content-length': '0',
        'x-delete-at': '1459967915',
        'accept-ranges': 'bytes',
        'last-modified': 'Wed, 06 Apr 2016 18:21:56 GMT',
        'etag': 'd41d8cd98f00b204e9800998ecf8427e',
        'x-timestamp': '1459966915.96956',
        'x-trans-id': 'transaction_id',
        'date': 'Wed, 06 Apr 2016 18:33:48 GMT',
        'content-type': 'text/plain',
        'x-object-meta-mtime': '1459965986.000000'
    },
    'Account': 'AUTH_seq_upload_prod',
    'Container': '2016-01',
    'Object': PosixPath('object.txt'),
    'Content-Type': 'application/octet-stream',
    # The size of the object
    'Content-Length': '112',
    # The last time the object was modified
    'Last-Modified': 'Fri, 15 Jan 2016 05:22:46 GMT',
    # The MD5 checksum of the object. NOTE that if a large
    # object is uploaded in segments that this will be the
    # checksum of the *manifest* file of the object
    'ETag': '87f0b7f04557315e6d1e6db21742d31c',
    'Manifest': None
}

Raises

NotFoundError – When the tenant, container, or object can’t be found.

temp_url(lifetime=300, method='GET', inline=True, filename=None)

Obtains a temporary URL to an object.

Parameters

• lifetime (int) – The time (in seconds) the temporary URL will be valid

• method (str) – The HTTP method that can be used on the temporary URL

• inline (bool, default True) – If False, URL will have a Content-Disposition header that causes browser to download as attachment.

• filename (str, optional) – A urlencoded filename to use for attachment, otherwise defaults to object name

abspath()

No-op for ‘abspath’

property container

Returns the container name from the path or None.

property content_type

Returns content-type. Empty string if not set or if an account or container

download(dest, condition=None, use_manifest=False)

Downloads a directory to a destination.

This method retries num_retries times if swift is unavailable or if the returned download result does not match the condition condition. View module-level documentation for more information about configuring retry logic at the module or method level.

Note that the destintation directory will be created automatically if it doesn’t exist.

Parameters

• dest (str) – The destination directory to download results to. The directory will be created if it doesn’t exist.

• condition (function(results) -> bool) – The method will only return when the results of download matches the condition. In the event of the condition never matching after retries, partially downloaded results will not be deleted. Note that users are not expected to write conditions for download without an understanding of the structure of the results.

• use_manifest (bool) – Perform the download and use the data manfest file to validate the download.

Raises

• SwiftError – A swift client error occurred.

• ConditionNotMetError – Results were returned, but they did not meet the condition.

Returns

A list of every operation performed in the download by the

underlying swift client

Return type

List[dict]

download_object(out_file)

Downloads a single object to an output file.

This method retries num_retries times if swift is unavailable. View module-level documentation for more information about configuring retry logic at the module or method level.

Parameters

out_file (str) – The output file

Raises

ValueError – This method was called on a path that has no container or object

download_objects(dest, objects)

See baseclass for complete method documentation

Parameters

• dest (str) – The destination folder to download to. The directory will be created if it doesnt exist.

• objects (List[str|PosixPath|SwiftPath]) – The list of objects to download. The objects can paths relative to the download path or absolute swift paths. Any absolute swift path must be children of the download path

Returns

A mapping of all requested objs to their location on

disk

Return type

dict

Raises

ValueError – This method was called on a path that has no container

exists()

Checks existence of the path.

Returns True if the path exists, False otherwise.

See module-level retry documentation for more.

Returns

True if the path exists, False otherwise.

Return type

bool

Raises

SwiftError – A non-404 swift client error occurred.

expanduser()

No-op for ‘expanduser’

first()

Returns the first result from the list results of the path

See module-level retry documentation for more.

Raises

SwiftError – A swift client error occurred.

getsize()

Returns content-length of object in Swift.

Note that for containers / tenants, there will be no content-length, in which case this function returns 0 (os.path.getsize has no contract)

glob(pattern, condition=None)

Globs all objects in the path with the pattern.

This glob is only compatible with patterns that end in * because of swift’s inability to do searches other than prefix queries.

Note that this method assumes the current resource is a directory path and treats it as such. For example, if the user has a swift path of swift://tenant/container/my_dir (without the trailing slash), this method will perform a swift query with a prefix of mydir/pattern.

This method retries num_retries times if swift is unavailable or if the number of globbed patterns does not match the condition condition. View module-level documentation for more information about configuring retry logic at the module or method level.

Parameters

• pattern (str) – The pattern to match. The pattern can only have up to one ‘*’ at the end.

• condition (function(results) -> bool) – The method will only return when the number of results matches the condition.

Returns

Every matching path.

Return type

List[SwiftPath]

Raises

• SwiftError – A swift client error occurred.

• ConditionNotMetError – Results were returned, but they did not meet the condition.

is_segment_container()

True if this path is a segment container

isdir()

See: os.path.isdir()

isfile()

Checks the object exists & is not a directory sentinel on Swift.

list(starts_with=None, limit=None, condition=None, use_manifest=False, list_as_dir=False, ignore_segment_containers=True, ignore_dir_markers=False, **kwargs)

List contents using the resource of the path as a prefix.

This method retries num_retries times if swift is unavailable or if the number of objects returned does not match the condition condition. View module-level documentation for more information about configuring retry logic at the module or method level.

Parameters

• starts_with (str) – Allows for an additional search path to be appended to the resource of the swift path. Note that the current resource path is treated as a directory

• limit (int) – Limit the amount of results returned

• condition (function(results) -> bool) – The method will only return when the results matches the condition.

• use_manifest (bool) – Perform the list and use the data manfest file to validate the list.

Returns

Every path in the listing.

Return type

List[SwiftPath]

Raises

• SwiftError – A swift client error occurred.

• ConditionNotMetError – Results were returned, but they did not meet the condition.

listdir(ignore_segment_containers=True, **kwargs)

Lists the path as a dir, returning top-level directories and files

For information about retry logic on this method, see SwiftPath.list

post(options=None)

Post operations on the path.

This method retries num_retries times if swift is unavailable. View module-level documentation for more information about configuring retry logic at the module or method level.

Parameters

options (dict) –

A dictionary containing options to override the global options specified during the service object creation. These options are applied to all post operations performed by this call, unless overridden on a per object basis. Possible options are given below:

{
    'meta': [],   # Meta values will be prefixed with X-Object-Meta
                  # (or X-Container*  X-Account* depending on path type)
    'header': [], # Header values will not be manipulated like meta values
                  # when being posted.
    'read_acl': None,   # For containers only
    'write_acl': None,  # For containers only
    'sync_to': None,    # For containers only
    'sync_key': None    # For containers only
}

Raises

SwiftError – A swift client error occurred.

read_object()

Reads an individual object from OBS.

Returns

the raw bytes from the object on OBS.

Return type

bytes

This method retries num_retries times if swift is unavailable or if the object is not found. View module-level documentation for more information about configuring retry logic at the module or method level.

realpath()

No-op for ‘realpath’

remove()

Removes a single object.

This method retries num_retries times if swift is unavailable. View module-level documentation for more information about configuring retry logic at the module or method level.

Raises

• ValueError – The path is invalid.

• SwiftError – A swift client error occurred.

remove_container()

Remove swift container if it’s not empty.

property resource

Returns the resource as a PosixPath object or None.

A resource can be a single object or a prefix to objects. Note that it’s important to keep the trailing slash in a resource name for prefix queries.

rmtree()

Removes a resource and all of its contents. This method retries num_retries times if swift is unavailable. View module-level documentation for more information about configuring retry logic at the module or method level.

Note that when removing a container, the associated segment container will also be removed if it exists. So, if one removes swift://tenant/container, swift://tenant/container_segments will also be deleted.

Note

Calling rmtree on a directory marker will delete everything under the directory marker but not the marker itself.

Raises

• SwiftError – A swift client error occurred.

• ConditionNotMetError – Listing the objects after rmtree returns results, indicating something went wrong with the rmtree operation

stat()

Performs a stat on the path.

Note that the path can be a tenant, container, or object. Using stat on a directory of objects will produce a NotFoundError.

This method retries num_retries times if swift is unavailable. View module-level documentation for more information about configuring retry logic at the module or method level.

The return value is dependent on if the path points to a tenant, container, or object.

For tenants, an example return dictionary is the following:

{
    'headers': {
        'content-length': u'0',
        'x-account-storage-policy-3xreplica-container-count': u'4655',
        'x-account-meta-temp-url-key': 'temp_url_key',
        'x-account-object-count': '428634',
        'x-timestamp': '1450732238.21562',
        'x-account-storage-policy-3xreplica-bytes-used': '175654877752',
        'x-trans-id': u'transaction_id',
        'date': u'Wed, 06 Apr 2016 18:30:28 GMT',
        'x-account-bytes-used': '175654877752',
        'x-account-container-count': '4655',
        'content-type': 'text/plain; charset=utf-8',
        'accept-ranges': 'bytes',
        'x-account-storage-policy-3xreplica-object-count': '428634'
    },
    'Account': 'AUTH_seq_upload_prod',
    # The number of containers in the tenant
    'Containers': 31,
    # The number of objects in the tenant
    'Objects': '19955615',
    # The total bytes used in the tenant
    'Bytes': '24890576770484',
    'Containers-in-policy-"3xreplica"': '31',
    'Objects-in-policy-"3xreplica"': '19955615',
    'Bytes-in-policy-"3xreplica"': '24890576770484',
    # The tenant ACLs. An empty dict is returned if the user
    # does not have admin privileges on the tenant
    'Account-Access': {
        'admin': ['swft_labprod_admin'],
        'read-only': ['seq_upload_rnd','swft_labprod'],
        'read-write': ['svc_svc_seq']
    }
}

For containers, an example return dictionary is the following:

{
    'headers': {
        'content-length': '0',
        'x-container-object-count': '99',
        'accept-ranges': 'bytes',
        'x-storage-policy': '3xReplica',
        'date': 'Wed, 06 Apr 2016 18:32:47 GMT',
        'x-timestamp': '1457631707.95682',
        'x-trans-id': 'transaction_id',
        'x-container-bytes-used': '5389060',
        'content-type': 'text/plain; charset=utf-8'
    },
    'Account': 'AUTH_seq_upload_prod',
    'Container': '2016-01',
    # The number of objects in the container
    'Objects': '43868',
    # The size of all objects in the container
    'Bytes': '55841489571',
    # Read and write ACLs for the container
    'Read-ACL': '',
    'Write-ACL': '',
    'Sync-To': '',
    'Sync-Key': ''
}

For objects, an example return dictionary is the following:

{
    'headers': {
        'content-length': '0',
        'x-delete-at': '1459967915',
        'accept-ranges': 'bytes',
        'last-modified': 'Wed, 06 Apr 2016 18:21:56 GMT',
        'etag': 'd41d8cd98f00b204e9800998ecf8427e',
        'x-timestamp': '1459966915.96956',
        'x-trans-id': 'transaction_id',
        'date': 'Wed, 06 Apr 2016 18:33:48 GMT',
        'content-type': 'text/plain',
        'x-object-meta-mtime': '1459965986.000000'
    },
    'Account': 'AUTH_seq_upload_prod',
    'Container': '2016-01',
    'Object': PosixPath('object.txt'),
    'Content-Type': 'application/octet-stream',
    # The size of the object
    'Content-Length': '112',
    # The last time the object was modified
    'Last-Modified': 'Fri, 15 Jan 2016 05:22:46 GMT',
    # The MD5 checksum of the object. NOTE that if a large
    # object is uploaded in segments that this will be the
    # checksum of the *manifest* file of the object
    'ETag': '87f0b7f04557315e6d1e6db21742d31c',
    'Manifest': None
}

Raises

NotFoundError – When the tenant, container, or object can’t be found.

temp_url(lifetime=300, method='GET', inline=True, filename=None)

Obtains a temporary URL to an object.

Parameters

• lifetime (int) – The time (in seconds) the temporary URL will be valid

• method (str) – The HTTP method that can be used on the temporary URL

• inline (bool, default True) – If False, URL will have a Content-Disposition header that causes browser to download as attachment.

• filename (str, optional) – A urlencoded filename to use for attachment, otherwise defaults to object name

property tenant

Returns the tenant name from the path or return None

to_url()

Returns URI for object (based on storage URL)

Returns

the HTTP url to the object

Return type

str

Raises

UnauthorizedError – if we cannot authenticate to get a storage URL

upload(to_upload, condition=None, use_manifest=False, headers=None)

Uploads a list of files and directories to swift.

This method retries num_retries times if swift is unavailable or if the returned upload result does not match the condition condition. View module-level documentation for more information about configuring retry logic at the module or method level.

For example:

with path('/path/to/upload/dir'):
    path('swift://tenant/container').upload(['.'])

Notes

• This method upload on paths relative to the current directory. In order to load files relative to a target directory, use path as a context manager to change the directory.

• When large files are split into segments, they are uploaded to a segment container named .segments_${container_name}

Parameters

• to_upload (List) – A list of file names, directory names, or OBSUploadObject objects to upload.

• condition (function(results) -> bool) – The method will only return when the results of upload matches the condition. In the event of the condition never matching after retries, partially uploaded results will not be deleted. Note that users are not expected to write conditions for upload without an understanding of the structure of the results.

• use_manifest (bool) –

  Generate a data manifest and validate the upload results are in the manifest. In case of a single directory being uploaded, the manifest file will be created inside this directory. For example:

  stor.Path('swift://AUTH_foo/bar').upload(['logs'], use_manifest=True)
  

  The manifest will be located at swift://AUTH_foo/bar/logs/.data_manifest.csv

  Alternatively, when multiple directories are uploaded, manifest file will be created in the current directory. For example:

  stor.Path('swift://AUTH_foo/bar').upload(
      ['logs', 'test.txt'], use_manifest=True)
  

  The manifest will be located at swift://AUTH_foo/bar/.data_manifest.csv

• headers (List[str]) – A list of object headers to apply to every object. Note that these are not applied if passing OBSUploadObjects directly to upload. Headers must be specified as a list of colon-delimited strings, e.g. [‘X-Delete-After:1000’]

Raises

• SwiftError – A swift client error occurred.

• ConditionNotMetError – The condition argument did not pass after num_retries or the use_manifest option was turned on and the upload results could not be verified. Partially uploaded results are not deleted.

Returns

A list of every operation performed in the upload by the

underlying swift client

Return type

List[dict]

walkfiles(pattern=None, **kwargs)

Iterates over listed files that match an optional pattern.

Parameters

pattern (str, optional) – Only return files that match this pattern

Returns

All files that match the optional pattern. Swift directory

markers are not returned.

Return type

Iter[SwiftPath]

write_object(content: bytes)

Writes an individual object.

Note that this method writes the provided content to a temporary file before uploading. This allows us to reuse code from swift’s uploader (static large object support, etc.).

For information about the retry logic of this method, view SwiftPath.upload.

Parameters

content (bytes) – raw bytes to write to OBS

Using Swift Conditions with Retry Settings

Swift is a storage system with eventual consistency, meaning (for example) that uploaded objects may not be able to be listed immediately after being uploaded. In order to make applications more resilient to consistency issues, various swift methods can take conditions that must pass before results are returned.

For example, imagine your application is downloading data using the SwiftPath.download method. In order to ensure that your application downloads exactly 10 objects, one can do the following:

SwiftPath('swift://tenant/container/dir').download('.', condition=lambda results: len(results) == 10)

In the above, condition takes the results from SwiftPath.download and verifies there are 10 elements. If the condition fails, SwiftPath.download will retry based on retry settings until finally throwing a ConditionNotMetError if the condition is not met. If condition passes, download returns results.

Note that if you want to combine multiple conditions, you can do this easily as:

condition = lambda results: all(f(results) for f in my_list_of_conditions)

SwiftUploadObject

stor.swift.SwiftUploadObject

alias of stor.obs.OBSUploadObject

Utilities

stor.utils.file_name_to_object_name(p)

Given a file path, construct its object name.

Any relative or absolute directory markers at the beginning of the path will be stripped, for example:

../../my_file -> my_file
./my_dir -> my_dir
.hidden_dir/file -> .hidden_dir/file
/absolute_dir -> absolute_dir

Note that windows paths will have their back slashes changed to forward slashes:

C:\my\windows\file -> my/windows/file

Parameters

p (str) – The input path

Returns

The object name. An empty path will be returned in

the case of the input path only consisting of absolute or relative directory markers (i.e. ‘/’ -> ‘’, ‘./’ -> ‘’)

Return type

PosixPath