api_storage_os.rst

Storage OS API

Lithops provides a transparent way to interact with the storage backend.

The module lithops.storage.cloud_proxy mimics the os and the built-in function open to access Cloud Storage as if it were a local file system.

By default, the configuration is loaded from the lithops config file, so there is no need to provide any parameter to use cloud_proxy:

from lithops.storage.cloud_proxy import os, open

with open('dir/file.txt', 'r') as f:
    content = f.read()

Alternatively, you can instantiate a CloudFileProxy with different lithops configuration through a dictionary. In this case, it will load the storage backend set in the storage key of the lithops section:

from lithops.storage.cloud_proxy import CloudStorage, CloudFileProxy

config = {'lithops' : {'storage_config' : 'ibm_cos'},
          'ibm_cos': {'region': 'REGION', 'api_key': 'API_KEY'}}

cloud_storage = CloudStorage(config)
cloud_file_proxy = CloudFileProxy(cloud_storage)

with cloud_file_proxy.open('dir/file.txt', 'r') as f:
    content = f.read()

Cloud Proxy Storage API

open()

Similar to Python's built-in function open().

Manipulate an object stored in Cloud Object Storage.

Parameter	Description
file	File path. Must be an absolute path.
mode	Specify the mode in which the file is opened ('r' for read and 'w' for write text, 'b' for binary).

with open('bar/foo.txt', 'w') as f:
    f.write('Hello world!')

os

Similar to Python's os library, except only file-related functionalities are implemented.

os.listdir

List all objects located in a directory.

Parameter	Description	Default
path	File path. Must be an absolute path.
suffix_dirs	Append a slash to directories listes	False

with open('bar/foo.txt', 'w') as f:
    f.write('Hello world!')

files = os.listdir('/bar') 
print(files)  # Prints ['foo.txt']

os.walk

List recursively all files and directories in a root path.

Parameter	Description	Default
path	Root path. Must be an absolute path.
topdown	List directory tree top-down instead of bottom-up	True

files = ['/bar/foo.txt', '/bar/image.jpg', '/bar/subdir/data.csv']
for file in files:
    with open(file, 'w') as f:
        f.write('Hello world!')

for root, dirs, files in os.walk('/'): 
    print(root, dirs, files)  # Prints '/' ['bar'] [], '/bar' ['subdir'] ['foo.txt', 'image.jpg'], '/bar/subdir' [] ['data.csv']

os.remove

Delete a file. If the directory where the file is located is empty after the file is deleted, this directory it is also removed.

Parameter	Description	Default
path	File path. Must be an absolute path.

with open('bar/foo.txt', 'w') as f:
    f.write('Hello world!')

os.remove('/bar/foo.txt')
files = os.listdir('/')
print(files)  # Prints []

os.path

Similar to Python's os.path, except only file-realted functionalities are implemented.

os.path.isfile

Return True if a path is a file.

Parameter	Description	Default
path	File path. Must be an absolute path.

with open('bar/foo.txt', 'w') as f:
    f.write('Hello world!')

print(os.path.isfile('/bar/foo.txt'))  # Prints ``True``
print(os.path.isfile('/bar'))  # Prints ``False``

os.path.isdir

Return True if a path is a directory.

Parameter	Description	Default
path	Directory path. Must be an absolute path.

with open('bar/foo.txt', 'w') as f:
    f.write('Hello world!')

print(os.path.isdir('/bar/foo.txt'))  # Prints False
print(os.path.isdir('/bar'))  # Prints True

os.path.exists

Retrun True if a path corresponds to an existing file or directory in Cloud Object Storage.

Parameter	Description	Default
path	Target path. Must be an absolute path.

with open('bar/foo.txt', 'w') as f:
    f.write('Hello world!')

print(os.path.exists('/bar/foo.txt'))  # Prints True
print(os.path.exists('/baz/foo.txt'))  # Prints False