API Reference

This section provides detailed API documentation for all classes and methods in Conflux S3 Utils.

S3Uri

class conflux_s3_utils.S3Uri(bucket, path)

A type-safe representation of an S3 URI.

Parameters:

• bucket (str)

• path (str)

bucket: str

Alias for field number 0

path: str

Alias for field number 1

static from_str(s3_uri_str)

Parse an S3 URI string into an S3Uri object. S3 URI format: s3://bucket_name/path/to/object

Parameters:

s3_uri_str (str)

Return type:

S3Uri

__str__()

Convert the S3Uri object back to an S3 URI string. S3 URI format: s3://bucket_name/path/to/object

Return type:

str

join_path(*paths)

Parameters:

paths (str)

Return type:

S3Uri

__truediv__(path)

Parameters:

path (str)

Return type:

S3Uri

with_path(path)

Parameters:

path (str)

Return type:

S3Uri

property filename: str

S3Client

class conflux_s3_utils.S3Client(client=None)

A S3 client that interacts with S3 using type-safe S3Uri objects and provides convenient methods for common operations.

Parameters:

client (Boto3S3Client | None)

__init__(client=None)

Initialize an S3Client. If client is not provided, a default boto3 S3 client will be created

Parameters:

client (Boto3S3Client | None)

open(s3uri, mode='rb')

Open an S3 object as a file-like object using fsspec. Note that this can be used with a context manager:

s3 = S3Client()
with s3.open(s3uri) as f:
    # Do stuff with `f`.
    # It will automatically be closed when exiting the block.

Parameters:

• s3uri (S3Uri)

• mode (str)

Return type:

OpenFile

open_local(s3uri, mode='rb', multipart_chunksize=8388608)

Interact with an object in S3 by opening a corresponding file locally.

With read mode (mode contains “r”), the object is downloaded from S3 into a temporary directory and then a file handle to the local file is provided. The file and the temporary directory are cleaned up when the context manager exits.

With write mode (mode contains “w”), a file is created in a temporary directory and a file handle to the local file is provided. Upon exiting the context manager, the file is uploaded to S3. The file and temporary directry are cleaned up when the context manager exits.

Parameters:

• s3uri (S3Uri)

• mode (str)

• multipart_chunksize (int)

object_exists(s3uri)

Check if an object exists in S3.

Parameters:

s3uri (S3Uri)

Return type:

bool

delete_object(s3uri)

Delete an object in S3.

Parameters:

s3uri (S3Uri)

list_objects(s3uri, recursive=False)

List objects under the given S3 URI, assuming it is a “directory”. If recursive is False, only list objects directly under the given path.

Parameters:

s3uri (S3Uri)

Return type:

Iterable[S3Uri]

copy_object(src, dest)

Copy an object from src to dest.

Parameters:

• src (S3Uri)

• dest (S3Uri)

Return type:

None

upload_file(filepath, s3uri, multipart_chunksize=8388608)

Upload a file to the specified S3 URI.

Parameters:

• filepath (Path)

• s3uri (S3Uri)

• multipart_chunksize (int)

Return type:

None

download_file(s3uri, filepath, multipart_chunksize=8388608)

Download a file from the specified S3 URI.

Parameters:

• s3uri (S3Uri)

• filepath (Path)

• multipart_chunksize (int)

Return type:

None

upload_directory(dirpath, s3uri, multipart_chunksize=8388608, concurrency=10, exclude={})

Upload a local directory to the specified S3 URI. The directory structure is preserved in S3.

Parameters:

• dirpath (Path)

• s3uri (S3Uri)

• multipart_chunksize (int)

• concurrency (int)

• exclude (set[Path])

Return type:

None