Last updated: 2024-11-07 21:05:07.534461 File source: link on GitLab
This sub package is an implementation of StorageProvider interface for S3 storage.
Here is quick overview of the contents of this pacakge:
README: Current file which is aimed towards developers who wish to use and modify the functionality.
download: This file defines functionality to allow download of files and directories from S3 buckets.
helpers: This file defines helper functions for working with AWS S3 storage.
init: This file initializes an Open Telemetry logger for the package.
s3_test: This file contains unit tests for the package functionality.
s3: This file defines methods to interact with S3 buckets using the AWS SDK.
specs_decoder: This file defines S3 input source configuration with validation and decoding logic.
upload: This file defines S3 storage implementation for uploading files (including directories) from a local volume to an S3 bucket.
The class diagram for the s3 sub-package is shown below.
Source file
Rendered from source file
NewClient
signature: NewClient(config aws.Config, volController storage.VolumeController) (*S3Storage, error)
input #1: AWS configuration object from AWS SDK
input #2: Volume Controller object
output (sucess): new instance of type storage.s3.S3Storage
output (error): error
NewClient is a constructor method. It creates a new instance of storage.s3.S3Storage struct.
Upload
For function signature refer to the package readme
Upload uploads all files (recursively) from a local volume to an S3 bucket. It returns an error when
there is error in decoding input specs
there is an error in file system operations (ex - opening files on the local file system etc)
there are errors from the AWS SDK during the upload process
Download
For function signature refer to the package readme
Download fetches files from a given S3 bucket. It can handle folders but except for x-directory. It depends on the file system provided by storage.VolumeController. It will return an error when
there is error in decoding input specs
there is error in creating a storage volume
there is an issue in determining the target S3 objects based on the provided key
there are errors in downloading individual S3 objects
there are issues in creating directories or writing files to the local file system
the storage.VolumeController fails to lock the newly created storage volume
Size
For function signature refer to the package readme
Size determines the size of an object stored in an S3 bucket.
It will return an error when
there is error in decoding input specs
there is an issue in AWS API call to retrieve the object size
storage.s3.S3Storage: TBD
storage.s3.s3Object: TBD
storage.s3.S3InputSource: TBD
Refer to package readme for other data types.
The various unit tests for the package functionality are defined in s3_test.go file.
List of issues
All issues that are related to the implementation of storage package can be found below. These include any proposals for modifications to the package or new data structures needed to cover the requirements of other packages.
Last updated: 2024-11-07 21:05:06.709764 File source: link on GitLab
The storage package is responsible for disk storage management on each DMS (Device Management Service) for data related to DMS and jobs deployed by DMS. It primarily handles storage access to remote storage providers such as AWS S3, IPFS etc. It also handles the control of storage volumes.
Here is quick overview of the contents of this pacakge:
README: Current file which is aimed towards developers who wish to use and modify the package functionality.
storage: This file defines the interface responsible for handling input/output operations of files with remote storage providers.
volumes: This file contains the interfaces and structs related to storage volumes.
subpackages
basic_controller: This folder contains the basic implementation of VolumeController interface.
s3: This contains implementation of storage functionality for S3 storage bucket.
The class diagram for the storage package is shown below.
Source file
Rendered from source file
The functionality with respect to storage package is offered by two main interfaces:
StorageProvider
VolumeController
These interfaces are described below.
StorageProvider Interface
The StorageProvider interface handles the input and output operations of files with remote storage providers such as AWS S3 and IPFS. Basically it provides methods to upload or download data and also to check the size of a data source.
Its functionality is coupled with local mounted volumes, meaning that implementations will rely on mounted files to upload data and downloading data will result in a local mounted volume.
Notes:
If needed, the availability-checking of a storage provider should be handled druing instantiation of the implementation.
Any necessary authentication data should be provided within the types.SpecConfig parameters
The interface has been designed for file based transfer of data. It is not built with the idea of supporting streaming of data and non-file storage operations (e.g.: some databases). Assessing the feasiblity of such requirement if needed should be done while implementation.
The methods of StorageProvider are as follows:
Upload
signature: Upload(ctx context.Context, vol StorageVolume, target *types.SpecConfig) (*types.SpecConfig, error)
input #1: Context object
input #2: storage volume from which data will be uploaded of type storage.StorageVolume
input #3: configuration parameters of specified storage provider of type types.SpecConfig
output (sucess): parameters related to storage provider like upload details/metadata etc of type types.SpecConfig
output (error): error message
Upload function uploads data from the storage volume provided as input to a given remote storage provider. The configuration of the storage provider is also provided as input to the function.
Download
signature: Download(ctx context.Context, source *types.SpecConfig) (StorageVolume, error)
input #1: Context object
input #2: configuration parameters of specified storage provider of type types.SpecConfig
output (sucess): storage volume which has downloaded data of type storage.StorageVolume
output (error): error message
Download function downloads data from a given source, mounting it to a certain local path. The input configuration received will vary from provider to provider and hence it is left to be detailed during implementation.
It will return an error if the operation fails. Note that this can also happen if the user running DMS does not have access permission to the given path.
Size
signature: Size(ctx context.Context, source *types.SpecConfig) (uint64, error)
input #1: Context object
input #2: configuration parameters of specified storage provider of type types.SpecConfig
output (sucess): size of the storage in Megabytes of type uint64
output (error): error message
Size function returns the size of a given storage provider provided as input. It will return an error if the operation fails.
Note that this method may also be useful to check if a given source is available.
VolumeController Interface
The VolumeController interface manages operations related to storage volumes which are data mounted to files/directories.
The methods of VolumeController are as follows:
CreateVolume
signature: CreateVolume(volSource storage.VolumeSource, opts ...storage.CreateVolOpt) -> (storage.StorageVolume, error)
input #1: predefined values of type string which specify the source of data (ex. IPFS etc)
input #2: optional parameter which can be passsed to set attributes or perform an operation on the storage volume
output (sucess): storage volume of type storage.StorageVolume
output (error): error message
CreateVolume creates a directory where data can be stored, and returns a StorageVolume which contains the path to the directory. Note that CreateVolume does not insert any data within the directory. It's up to the caller to do that.
VolumeSource contains predefined constants to specify common sources like S3 but it's extensible if new sources need to be supported.
CreateVolOpt is a function type that modifies storageVolume object. It allows for arbitrary operations to be performed while creating volume like setting permissions, encryption etc.
CreateVolume will return an error if the operation fails. Note that this can also happen if the user running DMS does not have access permission to create volume at the given path.
LockVolume
signature: LockVolume(pathToVol string, opts ...storage.LockVolOpt) -> error
input #1: path to the volume
input #2: optional parameter which can be passsed to set attributes or perform an operation on the storage volume
output (sucess): None
output (error): error message
LockVolume makes the volume read-only. It should be used after all necessary data has been written to the volume. It also makes clear whether a volume will change state or not. This is very useful when we need to retrieve volume's CID which is immutable given a certain data.
LockVolOpt is a function type that modifies storageVolume object. It allows for arbitrary operations to be performed while locking the volume like setting permissions, encryption etc.
LockVolume will return an error if the operation fails.
DeleteVolume
signature: DeleteVolume(identifier string, idType storage.IDType) -> error
input #1: path to the volume or CID
input #2: integer value associated with the type of identifier
output (error): error message
DeleteVolume function deletes everything within the root directory of a storage volume. It will return an error if the operation fails. Note that this can also happen if the user running DMS does not have the requisite access permissions.
The input can be a path or a Content ID (CID) depending on the identifier type passed.
IDType contains predefined integer values for different types of identifiers.
ListVolumes
signature: ListVolumes() -> ([]storage.StorageVolume, error)
input: None
output (sucess): List of existing storage volumes of type storage.StorageVolume
output (error): error message
ListVolumes function fetches the list of existing storage volumes. It will return an error if the operation fails or if the user running DMS does not have the requisite access permissions.
GetSize
signature: GetSize(identifier string, idType storage.IDType) -> (int64, error)
input #1: path to the volume or CID
input #2: integer value associated with the type of identifier
output (success): size of the volume
output (error): error message
GetSize returns the size of a volume. The input can be a path or a Content ID (CID) depending on the identifier type passed. It will return an error if the operation fails.
IDType contains predefined integer values for different types of identifiers.
storage.StorageVolume: This struct contains parameters related to a storage volume such as path, CID etc.
TBD Note: EncryptionType is not yet defined in types package
types.SpecConfig: This allows arbitrary configuration/parameters as needed during implementation of a specific storage provider. The parameters include authentication related data (if applicable).
storage.VolumeSource: This represents the source of data for a storage volume, for example IPFS, S3 etc.
storage.CreateVolOpt: This allows arbitrary operations on storage.StorageVolume to passed as input during volume creation.
storage.LockVolOpt: This allows arbitrary operations on StorageVolume to passed as input during locking of volume.
storage.IDType: This defines integer values for different types of identifiers of a storage volume.
types.EncryptionType: TBD
Note: The definition below should be moved to types package
TBD
List of issues
All issues that are related to the implementation of storage package can be found below. These include any proposals for modifications to the package or new data structures needed to cover the requirements of other packages.
Last updated: 2024-11-07 21:05:07.018668 File source: link on GitLab
This sub package offers a default implementation of the volume controller.
Here is quick overview of the contents of this pacakge:
README: Current file which is aimed towards developers who wish to use and modify the functionality.
basic_controller: This file implements the methods for VolumeController interface.
basic_controller_test: This file contains the unit tests for the methods of VolumeController interface.
The class diagram for the basic_controller sub-package is shown below.
Source file
basic_controller Class Diagram
Rendered from source file
NewDefaultVolumeController
signature: NewDefaultVolumeController(db *gorm.DB, volBasePath string, fs afero.Fs) -> (storage.basic_controller.BasicVolumeController, error)
input #1: local database instance of type *gorm.DB
input #2: base path of the volumes
input #3: file system instance of type afero.FS
output (sucess): new instance of type storage.basic_controller.BasicVolumeController
output (error): error
NewDefaultVolumeController returns a new instance of storage.basic_controller.BasicVolumeController struct.
BasicVolumeController is the default implementation of the VolumeController interface. It persists storage volumes information in the local database.
CreateVolume
For function signature refer to the package readme
CreateVolume creates a new storage volume given a storage source (S3, IPFS, job, etc). The creation of a storage volume effectively creates an empty directory in the local filesystem and writes a record in the database.
The directory name follows the format: <volSource> + "-" + <name> where name is random.
CreateVolume will return an error if there is a failure in
creation of new directory
creating a database entry
LockVolume
For function signature refer to the package readme
LockVolume makes the volume read-only, not only changing the field value but also changing file permissions. It should be used after all necessary data has been written to the volume. It optionally can also set the CID and mark the volume as private
LockVolume will return an error when
No storage volume is found at the specified
There is error in saving the updated volume in the database
There is error in updating file persmissions
DeleteVolume
For function signature refer to the package readme
DeleteVolume deletes a given storage volume record from the database. The identifier can be a path of a volume or a Content ID (CID). Therefore, records for both will be deleted.
It will return an error when
Input has incorrect identifier
There is failure in deleting the volume
No volume is found
ListVolumes
For function signature refer to the package readme
ListVolumes function returns a list of all storage volumes stored on the database.
It will return an error when no storage volumes exist.
GetSize
For function signature refer to the package readme
GetSize returns the size of a volume. The input can be a path or a Content ID (CID).
It will return an error if the operation fails due to:
error while querying database
volume not found for given identifier
unsupported identifed provided as input
error while caculating size of directory
Custom configuration Parameters
Both CreateVolume and LockVolume allow for custom configuration of storage volumes via optional parameters. Below is the list of available parameters that can be used:
WithPrivate() - Passing this as an input parameter designates a given volume as private. It can be used both when creating or locking a volume.
WithCID(cid string) - This can be used as an input parameter to set the CID of a given volume during the lock volume operation.
storage.basic_controller.BasicVolumeController: This struct manages implementation of VolumeController interface methods.
Refer to package readme for other data types.
The unit tests for the package functionality are defined in *_test.go file.
List of issues
All issues that are related to the implementation of storage package can be found below. These include any proposals for modifications to the package or new data structures needed to cover the requirements of other packages.