s3
Last updated: 2024-09-27 10:47:24.794800 File source: link on GitLab
s3
Table of Contents
Specification
Description
This sub package is an implementation of StorageProvider
interface for S3 storage.
Structure and Organisation
Here is quick overview of the contents of this pacakge:
README: Current file which is aimed towards developers who wish to use and modify the functionality.
download: This file defines functionality to allow download of files and directories from S3 buckets.
helpers: This file defines helper functions for working with AWS S3 storage.
init: This file initializes an Open Telemetry logger for the package.
s3_test: This file contains unit tests for the package functionality.
s3: This file defines methods to interact with S3 buckets using the AWS SDK.
specs_decoder: This file defines S3 input source configuration with validation and decoding logic.
upload: This file defines S3 storage implementation for uploading files (including directories) from a local volume to an S3 bucket.
Class Diagram
The class diagram for the s3
sub-package is shown below.
Source file
Rendered from source file
Functionality
NewClient
signature:
NewClient(config aws.Config, volController storage.VolumeController) (*S3Storage, error)
input #1:
AWS configuration object
from AWS SDKinput #2:
Volume Controller object
output (sucess): new instance of type
storage.s3.S3Storage
output (error): error
NewClient
is a constructor method. It creates a new instance of storage.s3.S3Storage
struct.
Upload
For function signature refer to the package readme
Upload
uploads all files (recursively) from a local volume to an S3 bucket. It returns an error when
there is error in decoding input specs
there is an error in file system operations (ex - opening files on the local file system etc)
there are errors from the AWS SDK during the upload process
Download
For function signature refer to the package readme
Download
fetches files from a given S3 bucket. It can handle folders but except for x-directory. It depends on the file system provided by storage.VolumeController
. It will return an error when
there is error in decoding input specs
there is error in creating a storage volume
there is an issue in determining the target S3 objects based on the provided key
there are errors in downloading individual S3 objects
there are issues in creating directories or writing files to the local file system
the
storage.VolumeController
fails to lock the newly created storage volume
Size
For function signature refer to the package readme
Size
determines the size of an object stored in an S3 bucket.
It will return an error when
there is error in decoding input specs
there is an issue in AWS API call to retrieve the object size
Data Types
storage.s3.S3Storage
:TBD
storage.s3.s3Object
:TBD
storage.s3.S3InputSource
:TBD
Refer to package readme for other data types.
Testing
The various unit tests for the package functionality are defined in s3_test.go
file.
Proposed Functionality / Requirements
List of issues
All issues that are related to the implementation of storage
package can be found below. These include any proposals for modifications to the package or new data structures needed to cover the requirements of other packages.
References
Last updated