Main Interface

stor

Stor is a library that aims to make it easy to write code that works with both local posix filesystems and Swift Object Based Storage. In general, you should be able to replace most uses of os.path and open with:

import stor as path
from stor import open

And your code will work either with posix paths or swift paths (defined as strings in the format swift://<TENANT>/<CONTAINER>/<OBJECT>). Stor also provides an object-oriented API similar to Python 3’s new pathlib, accessible via stor.Path.

stor is heavily inspired by / based on the path.py library, but modified to avoid the need to know whether you have a Path or a string for most functions.

See stor.swift for more information on Swift-specific functionality.

stor.abspath(path, *args, **kwargs)

See os.path.abspath()

stor.normcase(path, *args, **kwargs)

See os.path.normcase()

stor.normpath(path, *args, **kwargs)

See os.path.normpath()

stor.realpath(path, *args, **kwargs)

See os.path.realpath()

stor.expanduser(path, *args, **kwargs)

See os.path.expanduser()

stor.expandvars(path, *args, **kwargs)

See os.path.expandvars()

stor.dirname(path, *args, **kwargs)

See parent, os.path.dirname()

stor.basename(path, *args, **kwargs)

See name, os.path.basename()

stor.expand(path, *args, **kwargs)

Clean up a filename by calling expandvars(), expanduser(), and normpath() on it.

This is commonly everything needed to clean up a filename read from a configuration file, for example.

stor.join(path, *args, **kwargs)

Join first to zero or more Path components, adding a separator character (first.module.sep) if needed. Returns a new instance of first.path_class.

See: os.path.join()

stor.split(path, *args, **kwargs)

p.splitpath() -> Return (p.parent, p.name).

(naming is to avoid colliding with str.split)

See: parent, name, os.path.split()

stor.splitext() → Return ``(p.stripext(), p.ext)``.

Split the filename extension from this path and return the two parts. Either part may be empty.

The extension is everything from '.' to the end of the last path segment. This has the property that if (a, b) == p.splitext(), then a + b == p.

See: os.path.splitext()

stor.glob(pth, pattern)

Glob for pattern relative to pth.

Note that Swift currently only supports a single trailing *

stor.exists(path, *args, **kwargs)

Checks whether path exists on local filesystem or on swift.

For directories on swift, checks whether directory sentinel exists or at least one subdirectory exists

stor.isabs(path, *args, **kwargs)

See: os.path.isabs()

Always True with SwiftPath

stor.isdir(path, *args, **kwargs)

See: os.path.isdir()

stor.isfile(path, *args, **kwargs)

See: os.path.isfile()

stor.islink(path, *args, **kwargs)

See: os.path.islink()

Always False on Swift.

stor.ismount(path, *args, **kwargs)

See: os.path.ismount()

Always True on Swift.

stor.getsize(path, *args, **kwargs)

Returns size, in bytes of path.

For Swift containers and tenants, will return 0. For POSIX directories, returns an undefined value.

Raises

• os.error – if file does not exist or is inaccessible

• NotFoundError/UnauthorizedError – from swift

stor.copy(source, dest, **kwargs)

Copies a source file to a destination file.

Note that this utility can be called from either OBS, posix, or windows paths created with stor.Path.

Parameters

• source (path|str) – The source directory to copy from

• dest (path|str) – The destination file or directory.

• swift_retry_options (dict) – Optional retry arguments to use for swift upload or download. View the swift module-level documentation for more information on retry arguments

Examples

Copying a swift file to a local path behaves as follows:

>>> import stor
>>> swift_p = 'swift://tenant/container/dir/file.txt'
>>> # file.txt will be copied to other_dir/other_file.txt
>>> stor.copy(swift_p, 'other_dir/other_file.txt')

Copying from a local path to swift behaves as follows:

>>> from stor import Path
>>> local_p = Path('my/local/file.txt')
>>> # File will be uploaded to swift://tenant/container/dir/my_file.txt
>>> local_p.copy('swift://tenant/container/dir/')

Because of the ambiguity in whether a remote target is a file or directory, copy() will error on ambiguous paths.

>>> local_p.copy('swift://tenant/container/dir')
Traceback (most recent call last):
...
ValueError: OBS destination must be file with extension or directory with slash

stor.copytree(source, dest, copy_cmd=None, use_manifest=False, headers=None, condition=None, **kwargs)

Copies a source directory to a destination directory. Assumes that paths are capable of being copied to/from.

Note that this function uses shutil.copytree by default, meaning that a posix or windows destination must not exist beforehand.

For example, assume the following file hierarchy:

a/
- b/
- - 1.txt

Doing a copytree from a to a new posix destination of c is performed with:

Path('a').copytree('c')

The end result for c looks like:

c/
- b/
- - 1.txt

Note that the user can override which copy command is used for posix copies, and it is up to them to ensure that their code abides by the semantics of the provided copy command. This function has been tested in production using the default command of cp -r and using mcp -r.

Using OBS source and destinations work in a similar manner. Assume the destination is a swift path and we upload the same a folder:

Path('a').copytree('swift://tenant/container/folder')

The end swift result will have one object:

Path('swift://tenant/container/folder/b/1.txt')

Similarly one can do:

Path('swift://tenant/container/folder/').copytree('c')

The end result for c looks the same as the above posix example:

c/
- b/
- - 1.txt

Parameters

• source (path|str) – The source directory to copy from

• dest (path|str) – The directory to copy to. Must not exist if its a posix directory

• copy_cmd (str) – If copying to / from posix or windows, this command is used instead of shutil.copytree

• use_manifest (bool, default False) – See SwiftPath.upload and SwiftPath.download.

• condition (function(results) -> bool) – See SwiftPath.upload and SwiftPath.download.

• headers (List[str]) – See SwiftPath.upload.

Raises

• ValueError – if two OBS paths are specified

• OSError – if destination is a posix path and it already exists

stor.remove(path, *args, **kwargs)

Delete single path or object

stor.rmtree(path, *args, **kwargs)

Delete entire directory (or all paths starting with prefix).

See shutil.rmtree

stor.walkfiles(path, *args, **kwargs)

Iterate over files recursively.

Parameters

pattern (str, optional) – Limits the results to files with names that match the pattern. For example, mydir.walkfiles('*.tmp') yields only files with the .tmp extension.

Returns

Files recursively under the path

Return type

Iter[Path]

stor.is_filesystem_path(p)

Determines if the path is posix or windows filesystem.

Parameters

p (str) – The path string

Returns

True if p is a Windows path, False otherwise.

Return type

bool

stor.is_swift_path(p)

Determines if the path is a Swift path.

All Swift paths start with swift://

Parameters

p (str) – The path string

Returns

True if p is a Swift path, False otherwise.

Return type

bool

stor.is_obs_path(p)

Determines if the path is an OBS path (an S3 or Swift path).

Parameters

p (str) – The path string

Returns

bool: True if p is an OBS path, False otherwise.

stor.NamedTemporaryDirectory(suffix='', prefix='tmp', dir=None, change_dir=False)

Context manager for creating and deleting temporary directory.

Mimics the behavior of tempfile.NamedTemporaryFile.

Parameters

• suffix (str) – If specified, the dir name will end with it.

• prefix (str) – If specified, the dir name will start with it, otherwise ‘tmp’ is used.

• dir (str) – If specified, the dir will be created in this directory.

• change_dir (bool) – If specified, will change to the temporary directory.

Yields

Path – The temporary directory.

Note

Name is CamelCase to match tempfile.NamedTemporaryFile.

Examples

>>> from stor import NamedTemporaryDirectory
>>> with NamedTemporaryDirectory() as d:
>>>     # Do operations within "d", which will be deleted afterwards

Path Class

The stor.Path class is the parent to all concrete base classes and should be used as the main entry point to the object oriented API.

class stor.base.Path(path)

Wraps path operations with an object-oriented API that makes it easier to combine and also to work with OBS and local paths via a single API. Methods on this class will be implemented by all subclasses of path.

Using the class-level constructor returns a concrete subclass based on prefix and current environment.

Examples:

>>> from stor import Path
>>> Path('/some/path')
PosixPath('/some/path')
>>> Path('swift://AUTH_something/cont/blah')
SwiftPath('swift://AUTH_something/cont/blah')
>>> Path('s3://bucket/prefix/key')
S3Path('s3://bucket/prefix/key')

abspath()

See os.path.abspath()

basename()

See name, os.path.basename()

copy(dest, **kwargs)

Copies a source file to a destination file.

Note that this utility can be called from either OBS, posix, or windows paths created with stor.Path.

Parameters

• source (path|str) – The source directory to copy from

• dest (path|str) – The destination file or directory.

• swift_retry_options (dict) – Optional retry arguments to use for swift upload or download. View the swift module-level documentation for more information on retry arguments

Examples

Copying a swift file to a local path behaves as follows:

>>> import stor
>>> swift_p = 'swift://tenant/container/dir/file.txt'
>>> # file.txt will be copied to other_dir/other_file.txt
>>> stor.copy(swift_p, 'other_dir/other_file.txt')

Copying from a local path to swift behaves as follows:

>>> from stor import Path
>>> local_p = Path('my/local/file.txt')
>>> # File will be uploaded to swift://tenant/container/dir/my_file.txt
>>> local_p.copy('swift://tenant/container/dir/')

Because of the ambiguity in whether a remote target is a file or directory, copy() will error on ambiguous paths.

>>> local_p.copy('swift://tenant/container/dir')
Traceback (most recent call last):
...
ValueError: OBS destination must be file with extension or directory with slash

copytree(dest, copy_cmd=None, use_manifest=False, headers=None, condition=None, **kwargs)

Copies a source directory to a destination directory. Assumes that paths are capable of being copied to/from.

Note that this function uses shutil.copytree by default, meaning that a posix or windows destination must not exist beforehand.

For example, assume the following file hierarchy:

a/
- b/
- - 1.txt

Doing a copytree from a to a new posix destination of c is performed with:

Path('a').copytree('c')

The end result for c looks like:

c/
- b/
- - 1.txt

Note that the user can override which copy command is used for posix copies, and it is up to them to ensure that their code abides by the semantics of the provided copy command. This function has been tested in production using the default command of cp -r and using mcp -r.

Using OBS source and destinations work in a similar manner. Assume the destination is a swift path and we upload the same a folder:

Path('a').copytree('swift://tenant/container/folder')

The end swift result will have one object:

Path('swift://tenant/container/folder/b/1.txt')

Similarly one can do:

Path('swift://tenant/container/folder/').copytree('c')

The end result for c looks the same as the above posix example:

c/
- b/
- - 1.txt

Parameters

• source (path|str) – The source directory to copy from

• dest (path|str) – The directory to copy to. Must not exist if its a posix directory

• copy_cmd (str) – If copying to / from posix or windows, this command is used instead of shutil.copytree

• use_manifest (bool, default False) – See SwiftPath.upload and SwiftPath.download.

• condition (function(results) -> bool) – See SwiftPath.upload and SwiftPath.download.

• headers (List[str]) – See SwiftPath.upload.

Raises

• ValueError – if two OBS paths are specified

• OSError – if destination is a posix path and it already exists

dirname()

See parent, os.path.dirname()

exists()

Checks whether path exists on local filesystem or on swift.

For directories on swift, checks whether directory sentinel exists or at least one subdirectory exists

expand()

Clean up a filename by calling expandvars(), expanduser(), and normpath() on it.

This is commonly everything needed to clean up a filename read from a configuration file, for example.

expanduser()

See os.path.expanduser()

expandvars()

See os.path.expandvars()

property ext

The file extension, for example '.py'.

fnmatch(pattern, normcase=None)

Return True if name matches the given pattern.

See also

fnmatch.fnmatch()

Parameters

• pattern (str) – A filename pattern with wildcards, for example '*.py'. If the pattern contains a normcase attribute, it is applied to the name and path prior to comparison.

• normcase (func, optional) – A function used to normalize the pattern and filename before matching. Defaults to self.module(), which defaults to os.path.normcase().

getsize()

Returns size, in bytes of path.

For Swift containers and tenants, will return 0. For POSIX directories, returns an undefined value.

Raises

• os.error – if file does not exist or is inaccessible

• NotFoundError/UnauthorizedError – from swift

glob(pattern)

Glob for pattern relative to this directory.

Note that Swift currently only supports a single trailing *

isabs()

See: os.path.isabs()

Always True with SwiftPath

isdir()

See: os.path.isdir()

isfile()

See: os.path.isfile()

islink()

See: os.path.islink()

Always False on Swift.

ismount()

See: os.path.ismount()

Always True on Swift.

joinpath(*others)

Join first to zero or more Path components, adding a separator character (first.module.sep) if needed. Returns a new instance of first.path_class.

See: os.path.join()

list(*args, **kwargs)

List all contents using the path as a prefix.

Note: Skips broken symlinks.

makedirs_p(mode=511)

Like os.makedirs(), but does not raise an exception if the directory already exists.

property namebase

The same as name, but with one file extension stripped off.

For example, Path('/home/guido/python.tar.gz').name == 'python.tar.gz', but Path('/home/guido/python.tar.gz').namebase == 'python.tar'.

normcase()

See os.path.normcase()

normpath()

See os.path.normpath()

open(mode: Optional[str] = None, encoding: Optional[str] = None)

Open a file-like object.

The only cross-compatible arguments for this function are listed below.

Parameters

• mode – first positional arg, mode of file descriptor

• encoding – text encoding to use

parts_class

alias of Path

path_class

alias of Path

realpath()

See os.path.realpath()

remove()

Delete single path or object

rmtree()

Delete entire directory (or all paths starting with prefix).

See shutil.rmtree

splitdrive() → Return ``(p.drive, <the rest of p>)``.

Split the drive specifier from this path. If there is no drive specifier, p.drive is empty, so the return value is simply (Path(''), p). This is always the case on Unix.

See: os.path.splitdrive()

splitext() → Return ``(p.stripext(), p.ext)``.

Split the filename extension from this path and return the two parts. Either part may be empty.

The extension is everything from '.' to the end of the last path segment. This has the property that if (a, b) == p.splitext(), then a + b == p.

See: os.path.splitext()

splitpath() → Return ``(p.parent, p.name)``.

(naming is to avoid colliding with str.split)

See: parent, name, os.path.split()

walkfiles(pattern=None)

Iterate over files recursively.

Parameters

pattern (str, optional) – Limits the results to files with names that match the pattern. For example, mydir.walkfiles('*.tmp') yields only files with the .tmp extension.

Returns

Files recursively under the path

Return type

Iter[Path]

Common Path Methods

By using the top-level functions in the stor module (or the main Path interface) users can write code that is portable across Swift storage, Windows, and Posix-based filesystems. In particular, the following methods:

• open: Opens the resource identified by the path and returns a file (or file-like) object.

• glob: Globs the path based on an input pattern. Returns matching path objects.

• exists: Returns True if the resource exists, False otherwise.

• remove: Removes the resource.

• rmtree: Removes a directory and all resources it contains.

• copy: Copies the path to a destination.

• copytree: Copies a directory to a destination.

• walkfiles: Recursively walks files and matches an optional pattern.

Note

Path fully implements the above methods, but SwiftPath may only partially implement the method (e.g. only allowing prefix globbing). Read the SwiftPath documentation below about restrictions on the methods.

Note

Copying from object storage to windows is currently not supported.

SwiftPath

SwiftPath objects returned from the path factory partially support the common path methods listed above along with supporting the basic path manipulation methods (e.g. ‘my’ / ‘path’ == ‘my/path’). For more information on accessing swift, go to the Swift section.

S3Path

S3Path objects returned from the Path factory partially support the common path methods listed above along with supporting the basic path manipulation methods (e.g. ‘my’ / ‘path’ == ‘my/path’). For more information on accessing swift, go to the S3 section.

PosixPath

PosixPath objects return from the path factory fully support the common path methods listed above and also provide additional functionality. For more information, go to the Posix section.

WindowsPath

WindowsPath objects return from the path factory fully support the common path methods listed above and also provide additional functionality. For more information, go to the Windows section.