Quick Start

Basic Terminology

  • Blobs are objects, keys, or files.
  • Containers (buckets) manage blobs.
  • Storage Driver initiates a connection to the storage backend and manage containers.

Connecting to Storage

Let’s start with creating a Local File System storage driver (replace key argument with a folder path of your choosing):

from cloudstorage.drivers.local import LocalDriver

storage = LocalDriver(key='/home/webapp/storage', secret='<my-secret>')
# <Driver: LOCAL>

Alternatively, the driver can be initialized with its name. This is useful if you have different configurations for testing vs production. For example, a Flask app might use the LOCAL driver for testing and S3 for production.

from cloudstorage import get_driver_by_name

driver_cls = get_driver_by_name('LOCAL')
storage = driver_cls(key='/home/webapp/storage', secret='<my-secret>')
# <Driver: LOCAL>

Creating a Container

Creating a container:

container = storage.create_container('container-name')
# <Container container-name LOCAL>

Accessing a Container

Getting a container:

container = storage.get_container('container-name')
# <Container container-name LOCAL>

Deleting a Container

All of the blob objects in a container must be deleted before the container itself can be deleted:

container = storage.get_container('container-name')

for blob in container:
    blob.delete()

container.delete()

Uploading a Blob

Storing data from a file, stream, or string:

picture_path = '/path/picture.png'
picture_blob = container.upload_blob(picture_path)
# <Blob picture.png container-name LOCAL>
with open('/path/picture.png', 'rb') as picture_file:
    picture_blob = container.upload_blob(picture_file, blob_name='picture.png')
    # <Blob picture.png container-name LOCAL>

Cloud Storage will attempt to guess the uploaded file’s Content-Type using mimetypes and python-magic. The Content-Type can be overridden with the content_type argument:

with open('/path/picture.png', 'rb') as picture_file:
    picture_blob = container.upload_blob(filename=picture_file,
                                         content_type='application/octet-stream')
    # <Blob picture.png container-name LOCAL>
    picture_blob.content_type
    # 'application/octet-stream'

Important

Always use read binary mode rb when uploading a file like object.

Warning

The effect of uploading to an existing blob depends on the “versioning” and “lifecycle” policies defined on the blob’s container. In the absence of those policies, upload will overwrite any existing contents. As of now, Cloud Storage does not supporting versioning/generation.

Accessing a Blob

To get a blob from a container and its attributes:

container = storage.get_container('container-name')
picture_blob = container.get_blob('picture.png')
picture_blob.name
# 'picture.png'
picture_blob.size
# 50301
picture_blob.checksum
# '2f907a59924ad96b7478074ed96b05f0'
picture_blob.etag
# 'bf506fc6ffbc3c4a2756eac85a0b4d2f3f227fee'
picture_blob.content_type
# 'image/png'
picture_blob.created_at
# datetime.datetime(2017, 4, 19, 18, 38, 26, 335373)

Downloading a Blob

Downloading a blob data to a file path:

picture_blob = container.get_blob('picture.png')
picture_blob.download('/path/picture-copy.png')

Or to a file like object:

picture_blob = container.get_blob('picture.png')
with open('/path/picture-copy.png', 'wb') as picture_file:
    picture_blob.download(picture_file)

Important

Always use write binary mode wb when downloading a blob to a file like object.

Deleting a Blob

Deleting a blob:

picture_blob = container.get_blob('picture.png')
picture_blob.delete()

Generate a Download Url

Generates a signed URL to download a blob:

from urllib.parse import urlencode

import requests

storage_url = 'http://localhost/storage'

picture_blob = container.get_blob('picture.png')
signature = picture_blob.generate_download_url(expires=120)

url_params = {
    'signature': signature,
    'filename': 'picture.png',
}
download_url = storage_url + '?' + urlencode(url_params)
# 'http://localhost/storage?signature=<generated-signature>&filename=picture.png'

response = requests.get(download_url)
# <Response [200]>

with open('/path/picture-download.png', 'wb') as picture_file:
    for chunk in response.iter_content(chunk_size=128):
        picture_file.write(chunk)

Generate an Upload FormPost

Generate a signature and policy for uploading objects to a container:

import requests

container = storage.get_container('container-name')
form_post = container.generate_upload_url('avatar.png', expires=120)

url = form_post['url']
fields = form_post['fields']
multipart_form_data = {
    'file': open('/path/picture.png', 'rb'),
}

response = requests.post(url, data=fields, files=multipart_form_data)
# <Response [204]>

Iteration of Containers and Blobs

Storage and containers are both iterable:

for container in storage:
    container.name
    # 'container-a', 'container-b', ...

    for blob in container:
        blob.name
        # 'blob-1', 'blob-2', ...

Check if a container or container name exists in storage:

container = storage.get_container('container-name')
container in storage
# True
'container-name' in storage
# True

Check if a blob or blob name exists in a container:

container = storage.get_container('container-name')
picture_blob = container.get_blob('picture.png')
picture_blob in container
# True
'picture.png' in container
# True

Metadata and Extra Arguments

If supported by the driver, extra arguments can be included with operations on containers and blobs. For example, meta_data can be saved to a blob object or Content-Disposition set to inline or attachment.

options = {
    'acl': 'public-read',
    'content_disposition': 'attachment; filename="user-1-avatar.png"',
    'content_type': 'image/png',
    'meta_data': {
        'owner-id': '1',
        'owner-email': 'user.one@startup.com',
    }
}

picture_path = '/path/picture.png'
picture_blob = container.upload_blob(picture_path, **options)
picture_blob.content_disposition
# 'attachment; filename="user-1-avatar.png"'
picture_blob.meta_data
# {'owner-id': '1', 'owner-email': 'user.one@startup.com'}

Tip

It is recommended to save to meta data keys with dashes, owner-id, instead of with underscores, owner_id. Some drivers will allow underscores but other drivers will automatically convert them to dashes.

Proceed to the Advanced section for individual driver documentation and advanced usages like generating presigned upload and download URLs.