1 of 3

internal

Last updated: 2024-11-21 22:06:05.928463 File source: link on GitLab

internal

Description
Structure and Organisation
Class Diagram
Functionality
Data Types
Testing
Proposed Functionality/Requirements
References

Specification

Description

This package contains all code that is very specific to the whole of the dms, which will not be imported by any other packages and used only on the running instance of dms (like config and background task).

Structure and Organisation

Here is quick overview of the contents of this pacakge:

README: Current file which is aimed towards developers who wish to use and modify the package functionality.
init: This file handles controlled shutdown and initializes OpenTelemetry-based Zap logger.
websocket: This file contains communication protocols for a websocket server including message handling and command execution.

subpackages

config: This sub-package contains the configuration related data for the whole dms.
background_tasks: This sub-package contains functionality that runs in the background.

Class Diagram

Source

internal class diagram

Rendered from source file

!$rootUrlGitlab = "https://gitlab.com/nunet/device-management-service/-/raw/main"
!$packageRelativePath = "/internal"
!$packageUrlGitlab = $rootUrlGitlab + $packageRelativePath
 
!include $packageUrlGitlab/specs/class_diagram.puml

Functionality

TBD

Data Types

internal.WebSocketConnection

// WebSocketConnection is pointer to gorilla/websocket.Conn
type WebSocketConnection struct {
	*websocket.Conn
}

internal.Command

// Command represents a command to be executed
type Command struct {
	Command string
	NodeID  string // ID of the node where command will be executed
	Result  string
	Conn    *WebSocketConnection
}

Note: The data types are expected to change during refactoring of DMS

Testing

TBD

Proposed Functionality / Requirements

List of issues

All issues that are related to the implementation of internal package can be found below. These include any proposals for modifications to the package or new functionality needed to cover the requirements of other packages.

internal package implementation TBD

References

background_tasks

Last updated: 2024-11-21 22:06:06.410726 File source:

background_tasks

Specification

1. Description

The background_tasks package is an internal package responsible for managing background jobs within DMS. It contains a scheduler that registers tasks and run them according to the schedule defined by the task definition.

proposed Other packages that have their own background tasks register through this package:

Registration
1. The task itself, the arguments it needs
2. priority
3. event (time period or other event to trigger task)
Start , Stop, Resume
Algorithm that accounts for the event and priority of the task (not yet clear)
Monitor resource usage of tasks (not yet clear)

2. Structure and Organisation

Here is quick overview of the contents of this pacakge:

Files with *_test.go naming convention contain unit tests of the functionality in corresponding file.

3. Class Diagram

Source

Rendered from source file

4. Functionality

NewScheduler

signature: NewScheduler(maxRunningTasks int) *Scheduler
input: maximum no of running tasks
output: internal.background_tasks.Scheduler

NewScheduler function creates a new scheduler which takes maxRunningTasks argument to limit the maximum number of tasks to run at a time.

Scheduler methods

Scheduler struct is the orchestrator that manages and runs the tasks. If the Scheduler task queue is full, remaining tasks that are triggered will wait until there is a slot available in the scheduler.

It has the following methods:

AddTask

signature: AddTask(task *Task) *Task
input: internal.background_tasks.Task
output: internal.background_tasks.Task

AddTask registers a task to be run when triggered.

RemoveTask

signature: RemoveTask(taskID int)
input: identifier of the Task
output: None

RemoveTask removes a task from the scheduler. Tasks with only OneTimeTrigger will be removed automatically once run.

Start

signature: Start()
input: None
output: None

Start starts the scheduler to monitor tasks.

Stop

signature: Stop()
input: None
output: None

Stop stops the scheduler.

runTask

signature: runTask(taskID int)
input: identifier of the Task
output: None

runTask executes a task and manages its lifecycle and retry policy.

runTasks

signature: runTasks()
input: None
output: None

runTasks checks and runs tasks based on their triggers and priority.

runningTasksCount

signature: runningTasksCount() int
input: None
output: number of running tasks

runningTasksCount returns the count of running tasks.

Trigger Interface

Its methods are explained below:

IsReady

signature: IsReady() bool
input: None
output: bool

IsReady should return true if the task should be run.

Reset

signature: Reset()
input: None
output: None

Reset resets the trigger until the next event happens.

There are different implementations for the Trigger interface.

PeriodicTrigger: Defines a trigger based on a duration interval or a cron expression.
EventTrigger: Defines a trigger that is set by a trigger channel.
OneTimeTrigger: A trigger that is only triggered once after a set delay.

5. Data Types

internal.background_tasks.Scheduler

internal.background_tasks.RetryPolicy

internal.background_tasks.Execution

internal.background_tasks.Task

Task is a struct that defines a job. It includes the task's ID, Name, the function that is going to be run, the arguments for the function, the triggers that trigger the task to run, retry policy, etc.

internal.background_tasks.PeriodicTrigger

internal.background_tasks.EventTrigger

internal.background_tasks.OneTimeTrigger

6. Testing

Unit tests for each functionality are defined in files with *_test.go naming convention.

7. Proposed Functionality / Requirements

List of issues

8. References

config

Last updated: 2024-11-21 22:06:06.685865 File source: link on GitLab

Specification

Description

This package contains all configuration related code such as reading config file and functions to configure at runtime.

proposed There are two sides to configuration:

default configuration, which has to be loaded for the fresh installation of new dms;
dynamic configuration, which can be changed by a user that has access to the DMS; this dynamic configuration may need to be persistent (or not).

proposed Default configuration

Default configuration should be included into DMS distribution as a config.yaml in the root directory. The following is loosely based on general practice of passing yaml configuration to Go programs (see e.g. A clean way to pass configs in a Go application). DMS would parse this file during onboarding and populate the internal.config.Config variable that will be imported to other packages and used accordingly.

`proposed` Dynamic configuration

Dynamic configuration would use the same internal.config.Config variable, but would allow for adding new values or changing configuration by an authorized DMS user -- via DMS CLI or REST API calls.

The mechanism of dynamic configuration will enable to override or change default values. For enabling this functionality, the internal.config.Config variable will have a synchronized copy in the local DMS database, defined with db package.

Structure and Organisation

Here is quick overview of the contents of this pacakge:

README: Current file which is aimed towards developers who wish to use and modify the package functionality.
config: This file contains data structures for this package.
load: This file establishes a configuration loader using Viper, supports loading JSON files from various locations, applies defaults, and exposes functions to manage and obtain the loaded configuration.

Class Diagram

Source

config class diagram

Rendered from source file

!$rootUrlGitlab = "https://gitlab.com/nunet/device-management-service/-/raw/main"
!$packageRelativePath = "/internal/config"
!$packageUrlGitlab = $rootUrlGitlab + $packageRelativePath
 
!include $packageUrlGitlab/specs/class_diagram.puml

Functionality

The methods of this package are explained below:

getViper

signature: getViper() *viper.Viper
input: None
output: A pointer to a viper.Viper struct

getViper function creates a new viper.Viper instance used for configuration management with search paths.

setDefaultConfig

signature: setDefaultConfig() *viper.Viper
input: None
output: A pointer to a viper.Viper struct

Sets default values for various configuration options in a viper.Viper instance.

LoadConfig

signature: LoadConfig()
input: None
output: None

LoadConfig loads the configuration from the search paths.

SetConfig

signature: SetConfig(key string, value interface{})
input #1 : key for the configuration value
input #2 : value corresponding to the key provided
output: None

SetConfig Sets a specific configuration value using key value pair provided. In case of any error, the deafult configuration values are applied.

GetConfig

signature: GetConfig() *Config
input: None
output: internal.config.Config

GetConfig returns the loaded configuration data.

findConfig

signature: findConfig(paths []string, filename string) ([]byte, error)
input # 1: list of search paths
input # 1: name of the configuration file
output: contents of the configuration file
output(error): error message

findConfig Searches for the configuration file in specified paths and returns content or error.

removeComments

signature: removeComments(configBytes []byte) []byte
input: The byte array containing the configuration file content
output: byte array with comments removed

removeComments Removes comments from the configuration file content using a regular expression.

Data Types

internal.config.Config: holds the overall configuration with nested structs for specific sections

type Config struct {
	General `mapstructure:"general"`
	Rest    `mapstructure:"rest"`
	P2P     `mapstructure:"p2p"`
	Job     `mapstructure:"job"`
}

internal.config.General: Configuration related to general application behavior (data paths, debug mode)

type General struct {
	MetadataPath string `mapstructure:"metadata_path"`
	DataDir      string `mapstructure:"data_dir"`
	Debug        bool   `mapstructure:"debug"`
}

internal.config.Rest: Configuration for the REST API (port number)

type Rest struct {
	Port int `mapstructure:"port"`
}

internal.config.P2P: Configuration for the P2P network

type P2P struct {
	ListenAddress  []string `mapstructure:"listen_address"`
	BootstrapPeers []string `mapstructure:"bootstrap_peers"`
}

internal.config.Job: Configuration for background tasks (log update interval, target peer for deployments, container cleanup interval)

type Job struct {
	LogUpdateInterval int    `mapstructure:"log_update_interval"` // in minutes
	TargetPeer        string `mapstructure:"target_peer"`         // specific peer to send deployment requests to - XXX probably not a good idea. Remove after testing stage.
	CleanupInterval   int    `mapstructure:"cleanup_interval"` // docker container and images clean up interval in days
}

Testing

proposed Unit tests for each functionality are defined in files with *_test.go naming convention.

Proposed Functionality / Requirements

List of issues

internal package implementation TBD

proposed Functionalities

Following Gherkin feature files describe the proposed functionality for config package.

Load default DMS configuration: see scenario definition
Restore default DMS configuration: see scenario definition
Load existing DMS configuration: see scenario definition

References

background_tasks

Last updated: 2024-11-21 22:06:06.410726 File source:

background_tasks

Specification

1. Description

proposed Other packages that have their own background tasks register through this package:

Registration
1. The task itself, the arguments it needs
2. priority
3. event (time period or other event to trigger task)
Start , Stop, Resume
Algorithm that accounts for the event and priority of the task (not yet clear)
Monitor resource usage of tasks (not yet clear)

2. Structure and Organisation

Here is quick overview of the contents of this pacakge:

: Current file which is aimed towards developers who wish to use and modify the package functionality.
: This file initializes OpenTelemetry-based Zap logger.
: This file This file defines a background task scheduler that manages task execution based on triggers, priority, and retry policies.
: This file contains background task structs and their properties.
: This file defines various trigger types (PeriodicTrigger, EventTrigger, OneTimeTrigger) for background tasks, allowing execution based on time intervals, cron expressions, or external events

Files with *_test.go naming convention contain unit tests of the functionality in corresponding file.

3. Class Diagram

Source

Rendered from source file

!$rootUrlGitlab = "https://gitlab.com/nunet/device-management-service/-/raw/main"
!$packageRelativePath = "/internal/background_tasks"
!$packageUrlGitlab = $rootUrlGitlab + $packageRelativePath
 
!include $packageUrlGitlab/specs/class_diagram.puml

4. Functionality

NewScheduler

signature: NewScheduler(maxRunningTasks int) *Scheduler
input: maximum no of running tasks
output: internal.background_tasks.Scheduler

NewScheduler function creates a new scheduler which takes maxRunningTasks argument to limit the maximum number of tasks to run at a time.

Scheduler methods

It has the following methods:

AddTask

signature: AddTask(task *Task) *Task
input: internal.background_tasks.Task
output: internal.background_tasks.Task

AddTask registers a task to be run when triggered.

RemoveTask

signature: RemoveTask(taskID int)
input: identifier of the Task
output: None

RemoveTask removes a task from the scheduler. Tasks with only OneTimeTrigger will be removed automatically once run.

Start

signature: Start()
input: None
output: None

Start starts the scheduler to monitor tasks.

Stop

signature: Stop()
input: None
output: None

Stop stops the scheduler.

runTask

signature: runTask(taskID int)
input: identifier of the Task
output: None

runTask executes a task and manages its lifecycle and retry policy.

runTasks

signature: runTasks()
input: None
output: None

runTasks checks and runs tasks based on their triggers and priority.

runningTasksCount

signature: runningTasksCount() int
input: None
output: number of running tasks

runningTasksCount returns the count of running tasks.

Trigger Interface

type Trigger interface {
	IsReady() bool // Returns true if the trigger condition is met.
	Reset()        // Resets the trigger state.
}

Its methods are explained below:

IsReady

signature: IsReady() bool
input: None
output: bool

IsReady should return true if the task should be run.

Reset

signature: Reset()
input: None
output: None

Reset resets the trigger until the next event happens.

There are different implementations for the Trigger interface.

PeriodicTrigger: Defines a trigger based on a duration interval or a cron expression.
EventTrigger: Defines a trigger that is set by a trigger channel.
OneTimeTrigger: A trigger that is only triggered once after a set delay.

5. Data Types

internal.background_tasks.Scheduler

// Scheduler orchestrates the execution of tasks based on their triggers and priority.
type Scheduler struct {
	tasks           map[int]internal.background_tasks.Task // Map of tasks by their ID.
	runningTasks    map[int]bool  // Map to keep track of running tasks.
	ticker          *time.Ticker  // Ticker for periodic checks of task triggers.
	stopChan        chan struct{} // Channel to signal stopping the scheduler.
	maxRunningTasks int           // Maximum number of tasks that can run concurrently.
	lastTaskID      int           // Counter for assigning unique IDs to tasks.
	mu              sync.Mutex    // Mutex to protect access to task maps.
}

internal.background_tasks.RetryPolicy

// RetryPolicy defines the policy for retrying tasks on failure.
type RetryPolicy struct {
	MaxRetries int           // Maximum number of retries.
	Delay      time.Duration // Delay between retries.
}

internal.background_tasks.Execution

// Execution records the execution details of a task.
type Execution struct {
	StartedAt time.Time   // Start time of the execution.
	EndedAt   time.Time   // End time of the execution.
	Status    string      // Status of the execution (e.g., "SUCCESS", "FAILED").
	Error     string      // Error message if the execution failed.
	Event     interface{} // Event associated with the execution.
	Results   interface{} // Results of the execution.
}

internal.background_tasks.Task

// Task represents a schedulable task.
type Task struct {
	ID            int                          // Unique identifier for the task.
	Name          string                       // Name of the task.
	Description   string                       // Description of the task.
	Triggers      []internal.background_tasks.Trigger                    // List of triggers for the task.
	Function      func(args interface{}) error // Function to execute as the task.
	Args          []interface{}                // Arguments for the task function.
	RetryPolicy   internal.background_tasks.RetryPolicy                  // Retry policy for the task.
	Enabled       bool                         // Flag indicating if the task is enabled.
	Priority      int                          // Priority of the task for scheduling.
	ExecutionHist []internal.background_tasks.Execution                  // History of task executions.
}

internal.background_tasks.PeriodicTrigger

// PeriodicTrigger triggers at regular intervals or based on a cron expression.
type PeriodicTrigger struct {
	Interval      time.Duration // Interval for periodic triggering.
	CronExpr      string        // Cron expression for triggering.
	lastTriggered time.Time     // Last time the trigger was activated.
}

internal.background_tasks.EventTrigger

// EventTrigger triggers based on an external event signaled through a channel
type EventTrigger struct {
	Trigger chan bool // Channel to signal an event.
}

internal.background_tasks.OneTimeTrigger

// OneTimeTrigger triggers once after a specified delay.
type OneTimeTrigger struct {
	Delay        time.Duration // The delay after which to trigger.
	registeredAt time.Time     // Time when the trigger was set.
}

6. Testing

Unit tests for each functionality are defined in files with *_test.go naming convention.

7. Proposed Functionality / Requirements

List of issues

TBD

8. References

config

Last updated: 2024-11-21 22:06:06.685865 File source: link on GitLab

config

Description
Structure and Organisation
Class Diagram
Functionality
Data Types
Testing
Proposed Functionality/Requirements
References

Specification

Description

This package contains all configuration related code such as reading config file and functions to configure at runtime.

proposed There are two sides to configuration:

default configuration, which has to be loaded for the fresh installation of new dms;
dynamic configuration, which can be changed by a user that has access to the DMS; this dynamic configuration may need to be persistent (or not).

proposed Default configuration

`proposed` Dynamic configuration

Dynamic configuration would use the same internal.config.Config variable, but would allow for adding new values or changing configuration by an authorized DMS user -- via DMS CLI or REST API calls.

Structure and Organisation

Here is quick overview of the contents of this pacakge:

README: Current file which is aimed towards developers who wish to use and modify the package functionality.
config: This file contains data structures for this package.
load: This file establishes a configuration loader using Viper, supports loading JSON files from various locations, applies defaults, and exposes functions to manage and obtain the loaded configuration.

Class Diagram

Source

config class diagram

Rendered from source file

!$rootUrlGitlab = "https://gitlab.com/nunet/device-management-service/-/raw/main"
!$packageRelativePath = "/internal/config"
!$packageUrlGitlab = $rootUrlGitlab + $packageRelativePath
 
!include $packageUrlGitlab/specs/class_diagram.puml

Functionality

The methods of this package are explained below:

getViper

signature: getViper() *viper.Viper
input: None
output: A pointer to a viper.Viper struct

getViper function creates a new viper.Viper instance used for configuration management with search paths.

setDefaultConfig

signature: setDefaultConfig() *viper.Viper
input: None
output: A pointer to a viper.Viper struct

Sets default values for various configuration options in a viper.Viper instance.

LoadConfig

signature: LoadConfig()
input: None
output: None

LoadConfig loads the configuration from the search paths.

SetConfig

signature: SetConfig(key string, value interface{})
input #1 : key for the configuration value
input #2 : value corresponding to the key provided
output: None

SetConfig Sets a specific configuration value using key value pair provided. In case of any error, the deafult configuration values are applied.

GetConfig

signature: GetConfig() *Config
input: None
output: internal.config.Config

GetConfig returns the loaded configuration data.

findConfig

signature: findConfig(paths []string, filename string) ([]byte, error)
input # 1: list of search paths
input # 1: name of the configuration file
output: contents of the configuration file
output(error): error message

findConfig Searches for the configuration file in specified paths and returns content or error.

removeComments

signature: removeComments(configBytes []byte) []byte
input: The byte array containing the configuration file content
output: byte array with comments removed

removeComments Removes comments from the configuration file content using a regular expression.

Data Types

internal.config.Config: holds the overall configuration with nested structs for specific sections

type Config struct {
	General `mapstructure:"general"`
	Rest    `mapstructure:"rest"`
	P2P     `mapstructure:"p2p"`
	Job     `mapstructure:"job"`
}

internal.config.General: Configuration related to general application behavior (data paths, debug mode)

type General struct {
	MetadataPath string `mapstructure:"metadata_path"`
	DataDir      string `mapstructure:"data_dir"`
	Debug        bool   `mapstructure:"debug"`
}

internal.config.Rest: Configuration for the REST API (port number)

type Rest struct {
	Port int `mapstructure:"port"`
}

internal.config.P2P: Configuration for the P2P network

type P2P struct {
	ListenAddress  []string `mapstructure:"listen_address"`
	BootstrapPeers []string `mapstructure:"bootstrap_peers"`
}

internal.config.Job: Configuration for background tasks (log update interval, target peer for deployments, container cleanup interval)

type Job struct {
	LogUpdateInterval int    `mapstructure:"log_update_interval"` // in minutes
	TargetPeer        string `mapstructure:"target_peer"`         // specific peer to send deployment requests to - XXX probably not a good idea. Remove after testing stage.
	CleanupInterval   int    `mapstructure:"cleanup_interval"` // docker container and images clean up interval in days
}

Testing

proposed Unit tests for each functionality are defined in files with *_test.go naming convention.

Proposed Functionality / Requirements

List of issues

internal package implementation TBD

proposed Functionalities

Following Gherkin feature files describe the proposed functionality for config package.

Load default DMS configuration: see scenario definition
Restore default DMS configuration: see scenario definition
Load existing DMS configuration: see scenario definition

internal

internal

Table of Contents

Specification

Description

Structure and Organisation

Class Diagram

Functionality

Data Types

Testing

Proposed Functionality / Requirements

References

background_tasks

background_tasks

Table of Contents

Specification

1. Description

2. Structure and Organisation

3. Class Diagram

4. Functionality

5. Data Types

6. Testing

7. Proposed Functionality / Requirements

8. References

config

config

Table of Contents

Specification

Description

proposed Dynamic configuration

Structure and Organisation

Class Diagram

Functionality

Data Types

Testing

Proposed Functionality / Requirements

References

internal

internal

Table of Contents

Specification

Description

Structure and Organisation

Class Diagram

Functionality

Data Types

Testing

Proposed Functionality / Requirements

References

background_tasks

background_tasks

Table of Contents

Specification

1. Description

2. Structure and Organisation

3. Class Diagram

4. Functionality

5. Data Types

6. Testing

7. Proposed Functionality / Requirements

8. References

config

config

Table of Contents

Specification

Description

proposed Dynamic configuration

Structure and Organisation

Class Diagram

Functionality

Data Types

Testing

Proposed Functionality / Requirements

References

`proposed` Dynamic configuration

`proposed` Dynamic configuration