types package defines and keeps data structures and interfaces that are used across the whole DMS component by different packages.
Structure and Organisation
Here is quick overview of the contents of this pacakge:
README: Current file which is aimed towards developers who wish to use and modify the package functionality.
capability: This file contains data structures to describe machine capability.
comparison: This file contains constans and types used for capability comparison.
constants: This file contains constants that are used across different packages.
deployment: This file contains data structure related to job deployment.
elk_stats: This file contains data structure to be sent to elasticsearch collector.
encryption: This file contains data structure related to encryption in DMS.
execution: This file contains data structure related to executor functionality.
firecracker: This file contains data structure related to firecracker.
machine: This file contains data structure related to the machine - resources, peer details, services etc.
network: This file contains data structure related to networking functionality of DMS
network_config: This file defines message types, network types (libp2p, NATS) with configurations, and libp2p specific configurations (DHT, keys, peers, scheduling etc).
onboarding: This file contains data structure related to compute provider onboarding.
resource: This file contains data structures of GPU and execution resources.
spec_config: This file defines a SpecConfig struct for configuration data with type, parameters, normalization, validation, and type checking functionalities.
storage: This file contains data structures related to storage.
telemetry: This file defines structs related to telemetry configuration and methods to load configuration from environment variables.
types: This file defines a base model for entities in the application with auto-generated UUIDs, timestamps, and soft delete functionality using GORM hooks.
types package holds interfaces and methods that are used by multiple packages. The functionality of these interfaces/methods are typically implemented in other packages.
NewFailedExecutionResult creates a new ExecutionResult object for a failed execution. It sets the error message from the provided error and sets the exit code to -1.
Config interface TBD
type Config interface {
GetNetworkConfig() *SpecConfig
}
GetNetworkConfig will return the network configuration parameters.
NewSpecConfig
signature: NewSpecConfig(t string) *SpecConfig
input: type for the configuration object
output: types.SpecConfig
NewSpecConfig creates new SpecConfig with the given type and an empty params map.
WithParam
signature: (s *SpecConfig) WithParam(key string, value interface{}) *SpecConfig
input #1 : key
input #2 : value associated with the key
output: types.SpecConfig
WithParam adds a new key-value pair to the spec parameters and returns the updated SpecConfig object.
Normalize
signature: (s *SpecConfig) Normalize()
input: None
output: None
Normalize ensures that the spec config is in a valid state by trimming whitespace from the Type field and initializing the Params map if empty.
Validate
signature: (s *SpecConfig) Validate() error
input: None
output: error
Validate checks if the spec config is valid. It returns an error if the SpecConfig is nil or if the Type field is missing (blank). Otherwise, it returns no error indicating a valid configuration.
IsType
signature: (s *SpecConfig) IsType(t string) bool
input: type
output: bool
IsType checks if the SpecConfig matches the provided type, ignoring case sensitivity. It returns true if there's a match, otherwise false.
IsEmpty
signature: (s *SpecConfig) IsEmpty() bool
input: None
output: bool
IsEmpty checks if the SpecConfig is empty, meaning it's either nil or has an empty Type and no parameters. It returns true if empty, otherwise false.
LoadConfigFromEnv loads the telemetry configuration from environment variables. This includes observabilitty level and collector configuration. It returns the final configuration as types.TelemetryConfig object.
parseObservabilityLevel
signature: parseObservabilityLevel(levelStr string) int
input: observability level
output: int value corresponding to the input observability level
parseObservabilityLevel returns the integer representation of the provided observability level string. When the input string does not match the defined observability levels, default observability level INFO is considered and its integer value is returned.
Data Types
Deployment
types.DeploymentRequest
type DeploymentRequest struct {
RequesterWalletAddress string `json:"address_user"` // service provider wallet address
MaxNtx int `json:"max_ntx"`
Blockchain string `json:"blockchain"`
TxHash string `json:"tx_hash"`
ServiceType string `json:"service_type"`
Timestamp time.Time `json:"timestamp"`
MetadataHash string `json:"metadata_hash"`
WithdrawHash string `json:"withdraw_hash"`
RefundHash string `json:"refund_hash"`
Distribute_50Hash string `json:"distribute_50_hash"`
Distribute_75Hash string `json:"distribute_75_hash"`
Params struct {
ImageID string `json:"image_id"`
ModelURL string `json:"model_url"`
ResumeJob struct {
Resume bool `json:"resume"`
ProgressFile string `json:"progress_file"` // file path
} `json:"resume_job"`
Packages []string `json:"packages"`
RemoteNodeID string `json:"node_id"` // NodeID of compute provider (machine to deploy the job on)
RemotePublicKey string `json:"public_key"` // Public key of compute provider
LocalNodeID string `json:"local_node_id"` // NodeID of service provider (machine triggering the job)
LocalPublicKey string `json:"local_public_key"` // Public key of service provider
MachineType string `json:"machine_type"`
} `json:"params"`
Constraints struct {
Complexity string `json:"complexity"`
CPU int `json:"cpu"`
RAM int `json:"ram"`
Vram int `json:"vram"`
Power int `json:"power"`
Time int `json:"time"`
} `json:"constraints"`
TraceInfo struct {
TraceID string `json:"trace_id"`
SpanID string `json:"span_id"`
TraceFlags string `json:"trace_flags"`
TraceStates string `json:"trace_state"`
} `json:"traceinfo"`
}
type DeploymentRequestFlat struct {
types.BaseDBModel
DeploymentRequest string `json:"deployment_request"`
// represents job status from services table; goal is to keep then in sync (both tables are on different DMSes).
JobStatus string `json:"job_status"`
}
types.BlockchainTxStatus
type BlockchainTxStatus struct {
TransactionType string `json:"transaction_type"` // No need of this param maybe be deprecated in future
TransactionStatus string `json:"transaction_status"`
TxHash string `json:"tx_hash"`
}
ELK
types.NewDeviceOnboarded
// NewDeviceOnboarded defines the schema of the data to be sent to stats db when a new device gets onboarded
type NewDeviceOnboarded struct {
PeerID string
CPU float32
RAM float32
Network float32
DedicatedTime float32
Timestamp float32
}
types.DeviceStatusChange
// DeviceStatusChange defines the schema of the data to be sent to stats db when a device status gets changed
type DeviceStatusChange struct {
PeerID string
Status string
Timestamp float32
}
types.DeviceResourceChange
// DeviceResourceChange defines the schema of the data to be sent to stats db when a device resource gets changed
type DeviceResourceChange struct {
PeerID string
ChangedAttributeAndValue struct {
CPU float32
RAM float32
Network float32
DedicatedTime float32
}
Timestamp float32
}
types.DeviceResourceConfig
// DeviceResourceConfig defines the schema of the data to be sent to stats db when a device resource config gets changed
type DeviceResourceConfig struct {
PeerID string
ChangedAttributeAndValue struct {
CPU float32
RAM float32
Network float32
DedicatedTime float32
}
Timestamp float32
}
types.NewService
// NewService defines the schema of the data to be sent to stats db when a new service gets registered in the platform type NewService struct { ServiceID string ServiceName string ServiceDescription string Timestamp float32 }
types.ServiceCall
// ServiceCall defines the schema of the data to be sent to stats db when a host machine accepts a deployement request
type ServiceCall struct {
CallID float32
PeerIDOfServiceHost string
ServiceID string
CPUUsed float32
MaxRAM float32
MemoryUsed float32
NetworkBwUsed float32
TimeTaken float32
Status string
AmountOfNtx int32
Timestamp float32
}
types.ServiceStatus
// ServiceStatus defines the schema of update the status of service to stats db of the job being executed on host machine
type ServiceStatus struct {
CallID float32
PeerIDOfServiceHost string
ServiceID string
Status string
Timestamp float32
}
types.ServiceRemove
// ServiceRemove defines the schema of the data to be sent to stats db when a new service gets removed from the platform
type ServiceRemove struct {
ServiceID string
Timestamp float32
}
types.NtxPayment
// NtxPayment defines the schema of the data to be sent to stats db when a payment is made to device for the completion of service.
type NtxPayment struct {
CallID float32
ServiceID string
AmountOfNtx int32
PeerID string
SuccessFailStatus string
Timestamp float32
}
Executor
types.Executor: This defines the type of executor that is used to execute the job.
type Executor struct {
ExecutorType ExecutorType `json:"executor_type"`
}
type ExecutorType string
types.ExecutionRequest: This is the input that executor receives to initiate a job execution.
type ExecutionRequest struct {
// ID of the job to execute
JobID string
// ID of the execution
ExecutionID string
// Engine spec for the execution
EngineSpec types.SpecConfig
// Resources for the execution
Resources types.Resources
// Input volumes for the execution
Inputs []storage.StorageVolume
// Output volumes for the results
Outputs []storage.StorageVolume
// Directory to store the results
ResultsDir string
}
types.ExecutionResult: This contains the result of the job execution.
type ExecutionResult struct {
// STDOUT of the execution
STDOUT string `json:"stdout"`
// STDERR of the execution
STDERR string `json:"stderr"`
// Exit code of the execution
ExitCode int `json:"exit_code"`
// Error message if the execution failed
ErrorMsg string `json:"error_msg"`
}
types.SpecConfigTBD: This allows arbitrary configuration/parameters as needed during implementation of specific executor.
// SpecConfig represents a configuration for a spec
// A SpecConfig can be used to define an engine spec, a storage volume, etc.
type SpecConfig struct {
// Type of the spec (e.g. docker, firecracker, storage, etc.)
Type string `json:"type"`
// Params of the spec
// This allows passing arbitrary parameters to the spec implementation
Params map[string]interface{} `json:"params,omitempty"`
}
types.LogStreamRequest: This is the input provided when a request to stream logs of an execution is made.
type LogStreamRequest struct {
JobID string // ID of the job
ExecutionID string // ID of the execution
Tail bool // Tail the logs
Follow bool // Follow the logs
}
Firecracker
types.BootSource: This contains configuration parameters for booting a Firecracker VM.
types.Machine: This contains the configuration parameters of the machine.
type Machine struct {
types.BaseDBModel
NodeId string
PeerInfo int
IpAddr string
AvailableResources int
FreeResources int
TokenomicsAddress string
TokenomicsBlockchain string
}
types.FreeResources: This contains the resources currently available for a job.
// FreeResources are the resources free to be used by new services,
// plugins and any other processes started by DMS. It's basically
// the subtraction between AvailableResources and the amount of resources
// already used by DMS and its processes (mostly services)
type FreeResources struct {
BaseDBModel
TotCpuHz int `json:"tot_cpu_hz"`
PriceCpu float64 `json:"price_cpu"`
Ram int `json:"ram"`
PriceRam float64 `json:"price_ram"`
Vcpu int `json:"vcpu"`
Disk float64 `json:"disk"`
PriceDisk float64 `json:"price_disk"`
NTXPricePerMinute float64 `json:"ntx_price"`
}
types.AvailableResources: This contains the resources onboarded to Nunet by the user.
// AvailableResources are the amount of resources onboarded which
// can be used by NuNet
type AvailableResources struct {
types.BaseDBModel
TotCpuHz int
CpuNo int
CpuHz float64
PriceCpu float64
Ram int
PriceRam float64
Vcpu int
Disk float64
PriceDisk float64
NTXPricePerMinute float64
}
types.ServicesTBD: This contains the details of the services running on the machine.
type Services struct {
types.BaseDBModel
TxHash string
TransactionType string // transaction type can be running, done, withdraw, refund and distribute
JobStatus string // whether job is running or exited; one of these 'running', 'finished without errors', 'finished with errors'
JobDuration int64 // job duration in minutes
EstimatedJobDuration int64 // job duration in minutes
ServiceName string
ContainerID string
ResourceRequirements int
ImageID string
LogURL string
LastLogFetch time.Time
ServiceProviderAddr string
ComputeProviderAddr string
MetadataHash string
WithdrawHash string
RefundHash string // saving hashes for call the `/request-reward` endpoint by SPD
Distribute_50Hash string
Distribute_75Hash string
SignatureDatum string
MessageHashDatum string
Datum string
SignatureAction string // saving signatures for removing redundancy of calling Oracle
MessageHashAction string
Action string
// TODO: Add ContainerType field
}
types.ServiceResourceRequirements: This contains the resource requirements for a service.
type ServiceResourceRequirements struct {
types.BaseDBModel
CPU int
RAM int
VCPU int
HDD int
}
types.ContainerImages: This contains parameters of a container image.
// BlockchainAddressPrivKey holds Ethereum wallet address and private key from which the
// address is derived.
type BlockchainAddressPrivKey struct {
Address string `json:"address,omitempty"`
PrivateKey string `json:"private_key,omitempty"`
Mnemonic string `json:"mnemonic,omitempty"`
}
types.CapacityForNunet
// CapacityForNunet is a struct required in request body for the onboarding
type CapacityForNunet struct {
Memory int64 `json:"memory,omitempty"`
CPU int64 `json:"cpu,omitempty"`
NTXPricePerMinute float64 `json:"ntx_price,omitempty"`
Channel string `json:"channel,omitempty"`
PaymentAddress string `json:"payment_addr,omitempty"`
ServerMode bool `json:"server_mode,omitempty,"`
IsAvailable bool `json:"is_available"`
}
types.Provisioned
// Provisioned struct holds data about how much total resource
// host machine is equipped with
type Provisioned struct {
CPU float64 `json:"cpu,omitempty"`
Memory uint64 `json:"memory,omitempty"`
NumCores uint64 `json:"total_cores,omitempty"`
}
type GPU struct {
// Index is the self-reported index of the device in the system
Index int
// Name is the model name of the GPU e.g. Tesla T4
Name string
// Vendor is the maker of the GPU, e.g. NVidia, AMD, Intel
Vendor types.GPUVendor
// PCIAddress is the PCI address of the device, in the format AAAA:BB:CC.C
// Used to discover the correct device rendering cards
PCIAddress string
// Model of the GPU, e.g. A100
Model string `json:"model" description:"GPU model, ex A100"`
// TotalVRAM is the total amount of VRAM on the device
TotalVRAM uint64
// UsedVRAM is the amount of VRAM currently in use
UsedVRAM uint64
// FreeVRAM is the amount of VRAM currently free
FreeVRAM uint64
// Gorm fields
ResourceID uint `gorm:"foreignKey:ID"`
}
type CPU struct {
// Model represents the CPU model, e.g., "Intel Core i7-9700K", "AMD Ryzen 9 5900X"
Model string
// Vendor represents the CPU manufacturer, e.g., "Intel", "AMD"
Vendor string
// ClockSpeedHz represents the CPU clock speed in Hz
ClockSpeedHz uint64
// Cores represents the number of physical CPU cores
Cores int
// Threads represents the number of logical CPU threads (including hyperthreading)
Threads int
// Architecture represents the CPU architecture, e.g., "x86", "x86_64", "arm64"
Architecture string
// Cache size in bytes
CacheSize uint64
}
types.RAM: RAM details.
type RAM struct {
// Size in bytes
Size uint64
// Clock speed in Hz
ClockSpeedHz uint64
// Type represents the RAM type, e.g., "DDR4", "DDR5", "LPDDR4"
Type string
}
types.Disk: Disk details.
type Disk struct {
// Model represents the disk model, e.g., "Samsung 970 EVO Plus", "Western Digital Blue SN550"
Model string
// Vendor represents the disk manufacturer, e.g., "Samsung", "Western Digital"
Vendor string
// Size in bytes
Size uint64
// Type represents the disk type, e.g., "SSD", "HDD", "NVMe"
Type string
// Interface represents the disk interface, e.g., "SATA", "PCIe", "M.2"
Interface string
// Read speed in bytes per second
ReadSpeed uint64
// Write speed in bytes per second
WriteSpeed uint64
}
types.NetworkInfo: Network details.
type NetworkInfo struct {
// Bandwidth in bits per second (b/s)
Bandwidth uint64
// NetworkType represents the network type, e.g., "Ethernet", "Wi-Fi", "Cellular"
NetworkType string
}
types.Resource: resources resources required to execute a task
Spec_config
types.SpecConfig: This allows arbitrary configuration to be defined as needed.
// SpecConfig represents a configuration for a spec
// A SpecConfig can be used to define an engine spec, a storage volume, etc.
type SpecConfig struct {
// Type of the spec (e.g. docker, firecracker, storage, etc.)
Type string `json:"type"`
// Params of the spec
Params map[string]interface{} `json:"params,omitempty"`
}
Storage
types.StorageVolume: This contains the parameters related to the storage volume that is created by the DMS on the local machine.
// StorageVolume represents a prepared storage volume that can be mounted to an execution
type StorageVolume struct {
// Type of the volume (e.g. bind)
Type string `json:"type"`
// Source path of the volume on the host
Source string `json:"source"`
// Target path of the volume in the execution
Target string `json:"target"`
// ReadOnly flag to mount the volume as read-only
ReadOnly bool `json:"readonly"`
}
Telemetry Config
types.CollectorConfig: This contains the parameters for a collector.
type CollectorConfig struct {
CollectorType string
CollectorEndpoint string
}
types.TelemetryConfig: This defines the telemetry parameters such as obervability level, collector configurations etc.
type TelemetryConfig struct {
ServiceName string
GlobalEndpoint string
ObservabilityLevel int
CollectorConfigs map[string]CollectorConfig
}
ObservabilityLevel is an enum that defines the level of observability. Currently logging is done at these observability levels.
// BaseDBModel is a base model for all entities. It'll be mainly used for database
// records.
type BaseDBModel struct {
ID string `gorm:"type:uuid"`
CreatedAt time.Time
UpdatedAt time.Time
DeletedAt gorm.DeletedAt `gorm:"index"`
}
Testing
Test are defined in other packages where functionality is implemented.
Proposed Functionality / Requirements
List of issues
All issues that are related to the implementation of types package can be found below. These include any proposals for modifications to the package or new data structures needed to cover the requirements of other packages.
These are placeholder interface definitions which will be developed in the future.
Encryptor must support encryption of files/directories
type Encryptor interface {
Encrypt([]byte) ([]byte, error)
}
Decryptor must support decryption of files/directories
type Decryptor interface {
Decrypt([]byte) ([]byte, error)
}
proposed Network types and methods
This section contains the proposed data types and methods related to network functionality.
types.NetworkSpec
// NetworkSpec is a stub. Please expand based on requirements.
type NetworkSpec struct {
}
types.NetConfig
// NetConfig is a stub. Please expand it or completely change it based on requirements.
type NetConfig struct {
NetworkSpec SpecConfig `json:"network_spec"` // Network specification
}
types.NetConfig struct will implement a GetNetworkConfig method which returns network configuration parameters.
// NetworkStats should contain all network info the user is interested in.
// for now there's only peerID and listening address but reachability, local and remote addr etc...
// can be added when necessary.
type NetworkStats struct {
ID string `json:"id"`
ListenAddr string `json:"listen_addr"`
}
types.MessageInfo
// MessageInfo is a stub. Please expand it or completely change it based on requirements.
type MessageInfo struct {
Info string `json:"info"` // Message information
}
proposed Network configuration data type
types.MessageEnvelope
type MessageType string
type MessageEnvelope struct {
Type MessageType
Data []byte
}
types.NetworkConfig
type NetworkType string
type NetworkConfig struct {
Type NetworkType
// libp2p
types.Libp2pConfig
// nats
NATSUrl string
}
types.Libp2pConfig
// Libp2pConfig holds the libp2p configuration type Libp2pConfig struct { DHTPrefix string //crypto is Go-libp2p package that implements various cryptographic utilities PrivateKey crypto.PrivKey BootstrapPeers []multiaddr.Multiaddr Rendezvous string Server bool Scheduler *bt.Scheduler CustomNamespace string ListenAddress []string PeerCountDiscoveryLimit int PrivateNetwork types.PrivateNetworkConfig GracePeriodMs int GossipMaxMessageSize int }
types.PrivateNetworkConfig
type PrivateNetworkConfig struct {
// WithSwarmKey if true, DMS will try to fetch the key from
// `<config_path>/swarm.key`.
WithSwarmKey bool
// ACL defines the access control list for the private network.
ACL []multiaddr.Multiaddr
}
