Last updated: 2024-11-07 21:04:52.329610 File source: link on GitLab

background_tasks

• Project README

• Release/Build Status

• Changelog

• License

• Contribution Guidelines

• Code of Conduct

• Secure Coding Guidelines

Table of Contents

1. Description

2. Structure and Organisation

3. Class Diagram

4. Functionality

5. Data Types

6. Testing

7. Proposed Functionality/Requirements

8. References

Specification

1. Description

The background_tasks package is an internal package responsible for managing background jobs within DMS. It contains a scheduler that registers tasks and run them according to the schedule defined by the task definition.

proposed Other packages that have their own background tasks register through this package:

1. Registration

   1. The task itself, the arguments it needs

   2. priority

   3. event (time period or other event to trigger task)

2. Start , Stop, Resume

3. Algorithm that accounts for the event and priority of the task (not yet clear)

4. Monitor resource usage of tasks (not yet clear)

2. Structure and Organisation

Here is quick overview of the contents of this pacakge:

• README: Current file which is aimed towards developers who wish to use and modify the package functionality.

• init: This file initializes OpenTelemetry-based Zap logger.

• scheduler: This file This file defines a background task scheduler that manages task execution based on triggers, priority, and retry policies.

• task: This file contains background task structs and their properties.

• trigger: This file defines various trigger types (PeriodicTrigger, EventTrigger, OneTimeTrigger) for background tasks, allowing execution based on time intervals, cron expressions, or external events

Files with *_test.go naming convention contain unit tests of the functionality in corresponding file.

3. Class Diagram

Source

background_tasks class diagram

Rendered from source file

!$rootUrlGitlab = "https://gitlab.com/nunet/device-management-service/-/raw/main"
!$packageRelativePath = "/internal/background_tasks"
!$packageUrlGitlab = $rootUrlGitlab + $packageRelativePath
 
!include $packageUrlGitlab/specs/class_diagram.puml

4. Functionality

NewScheduler

• signature: NewScheduler(maxRunningTasks int) *Scheduler

• input: maximum no of running tasks

• output: internal.background_tasks.Scheduler

NewScheduler function creates a new scheduler which takes maxRunningTasks argument to limit the maximum number of tasks to run at a time.

Scheduler methods

Scheduler struct is the orchestrator that manages and runs the tasks. If the Scheduler task queue is full, remaining tasks that are triggered will wait until there is a slot available in the scheduler.

It has the following methods:

AddTask

• signature: AddTask(task *Task) *Task

• input: internal.background_tasks.Task

• output: internal.background_tasks.Task

AddTask registers a task to be run when triggered.

RemoveTask

• signature: RemoveTask(taskID int)

• input: identifier of the Task

• output: None

RemoveTask removes a task from the scheduler. Tasks with only OneTimeTrigger will be removed automatically once run.

Start

• signature: Start()

• input: None

• output: None

Start starts the scheduler to monitor tasks.

Stop

• signature: Stop()

• input: None

• output: None

Stop stops the scheduler.

runTask

• signature: runTask(taskID int)

• input: identifier of the Task

• output: None

runTask executes a task and manages its lifecycle and retry policy.

runTasks

• signature: runTasks()

• input: None

• output: None

runTasks checks and runs tasks based on their triggers and priority.

runningTasksCount

• signature: runningTasksCount() int

• input: None

• output: number of running tasks

runningTasksCount returns the count of running tasks.

Trigger Interface

type Trigger interface {
	IsReady() bool // Returns true if the trigger condition is met.
	Reset()        // Resets the trigger state.
}

Its methods are explained below:

IsReady

• signature: IsReady() bool

• input: None

• output: bool

IsReady should return true if the task should be run.

Reset

• signature: Reset()

• input: None

• output: None

Reset resets the trigger until the next event happens.

There are different implementations for the Trigger interface.

• PeriodicTrigger: Defines a trigger based on a duration interval or a cron expression.

• EventTrigger: Defines a trigger that is set by a trigger channel.

• OneTimeTrigger: A trigger that is only triggered once after a set delay.

5. Data Types

• internal.background_tasks.Scheduler

// Scheduler orchestrates the execution of tasks based on their triggers and priority.
type Scheduler struct {
	tasks           map[int]internal.background_tasks.Task // Map of tasks by their ID.
	runningTasks    map[int]bool  // Map to keep track of running tasks.
	ticker          *time.Ticker  // Ticker for periodic checks of task triggers.
	stopChan        chan struct{} // Channel to signal stopping the scheduler.
	maxRunningTasks int           // Maximum number of tasks that can run concurrently.
	lastTaskID      int           // Counter for assigning unique IDs to tasks.
	mu              sync.Mutex    // Mutex to protect access to task maps.
}

• internal.background_tasks.RetryPolicy

// RetryPolicy defines the policy for retrying tasks on failure.
type RetryPolicy struct {
	MaxRetries int           // Maximum number of retries.
	Delay      time.Duration // Delay between retries.
}

• internal.background_tasks.Execution

// Execution records the execution details of a task.
type Execution struct {
	StartedAt time.Time   // Start time of the execution.
	EndedAt   time.Time   // End time of the execution.
	Status    string      // Status of the execution (e.g., "SUCCESS", "FAILED").
	Error     string      // Error message if the execution failed.
	Event     interface{} // Event associated with the execution.
	Results   interface{} // Results of the execution.
}

• internal.background_tasks.Task

Task is a struct that defines a job. It includes the task's ID, Name, the function that is going to be run, the arguments for the function, the triggers that trigger the task to run, retry policy, etc.

// Task represents a schedulable task.
type Task struct {
	ID            int                          // Unique identifier for the task.
	Name          string                       // Name of the task.
	Description   string                       // Description of the task.
	Triggers      []internal.background_tasks.Trigger                    // List of triggers for the task.
	Function      func(args interface{}) error // Function to execute as the task.
	Args          []interface{}                // Arguments for the task function.
	RetryPolicy   internal.background_tasks.RetryPolicy                  // Retry policy for the task.
	Enabled       bool                         // Flag indicating if the task is enabled.
	Priority      int                          // Priority of the task for scheduling.
	ExecutionHist []internal.background_tasks.Execution                  // History of task executions.
}

• internal.background_tasks.PeriodicTrigger

// PeriodicTrigger triggers at regular intervals or based on a cron expression.
type PeriodicTrigger struct {
	Interval      time.Duration // Interval for periodic triggering.
	CronExpr      string        // Cron expression for triggering.
	lastTriggered time.Time     // Last time the trigger was activated.
}

• internal.background_tasks.EventTrigger

// EventTrigger triggers based on an external event signaled through a channel
type EventTrigger struct {
	Trigger chan bool // Channel to signal an event.
}

• internal.background_tasks.OneTimeTrigger

// OneTimeTrigger triggers once after a specified delay.
type OneTimeTrigger struct {
	Delay        time.Duration // The delay after which to trigger.
	registeredAt time.Time     // Time when the trigger was set.
}

6. Testing

Unit tests for each functionality are defined in files with *_test.go naming convention.

7. Proposed Functionality / Requirements

List of issues

All issues that are related to the implementation of internal package can be found below. These include any proposals for modifications to the package or new functionality needed to cover the requirements of other packages.

• internal package implementation TBD

8. References

Last updated: 2024-11-07 21:04:51.724700 File source: link on GitLab

internal

• Project README

• Release/Build Status

• Changelog

• License

• Contribution Guidelines

• Code of Conduct

• Secure Coding Guidelines

Table of Contents

1. Description

2. Structure and Organisation

3. Class Diagram

4. Functionality

5. Data Types

6. Testing

7. Proposed Functionality/Requirements

8. References

Specification

Description

This package contains all code that is very specific to the whole of the dms, which will not be imported by any other packages and used only on the running instance of dms (like config and background task).

Structure and Organisation

Here is quick overview of the contents of this pacakge:

• README: Current file which is aimed towards developers who wish to use and modify the package functionality.

• init: This file handles controlled shutdown and initializes OpenTelemetry-based Zap logger.

• websocket: This file contains communication protocols for a websocket server including message handling and command execution.

subpackages

• config: This sub-package contains the configuration related data for the whole dms.

• background_tasks: This sub-package contains functionality that runs in the background.

Class Diagram

Source

internal class diagram

Rendered from source file

!$rootUrlGitlab = "https://gitlab.com/nunet/device-management-service/-/raw/main"
!$packageRelativePath = "/internal"
!$packageUrlGitlab = $rootUrlGitlab + $packageRelativePath
 
!include $packageUrlGitlab/specs/class_diagram.puml

Functionality

TBD

Data Types

• internal.WebSocketConnection

// WebSocketConnection is pointer to gorilla/websocket.Conn
type WebSocketConnection struct {
	*websocket.Conn
}

• internal.Command

// Command represents a command to be executed
type Command struct {
	Command string
	NodeID  string // ID of the node where command will be executed
	Result  string
	Conn    *WebSocketConnection
}

Note: The data types are expected to change during refactoring of DMS

Testing

TBD

Proposed Functionality / Requirements

List of issues

All issues that are related to the implementation of internal package can be found below. These include any proposals for modifications to the package or new functionality needed to cover the requirements of other packages.

• internal package implementation TBD

References

Last updated: 2024-11-07 21:04:52.838997 File source: link on GitLab

config

• Project README

• Release/Build Status

• Changelog

• License

• Contribution Guidelines

• Code of Conduct

• Secure Coding Guidelines

Table of Contents

1. Description

2. Structure and Organisation

3. Class Diagram

4. Functionality

5. Data Types

6. Testing

7. Proposed Functionality/Requirements

8. References

Specification

Description

This package contains all configuration related code such as reading config file and functions to configure at runtime.

proposed There are two sides to configuration:

1. default configuration, which has to be loaded for the fresh installation of new dms;

2. dynamic configuration, which can be changed by a user that has access to the DMS; this dynamic configuration may need to be persistent (or not).

proposed Default configuration

Default configuration should be included into DMS distribution as a config.yaml in the root directory. The following is loosely based on general practice of passing yaml configuration to Go programs (see e.g. A clean way to pass configs in a Go application). DMS would parse this file during onboarding and populate the internal.config.Config variable that will be imported to other packages and used accordingly.

proposed Dynamic configuration

Dynamic configuration would use the same internal.config.Config variable, but would allow for adding new values or changing configuration by an authorized DMS user -- via DMS CLI or REST API calls.

The mechanism of dynamic configuration will enable to override or change default values. For enabling this functionality, the internal.config.Config variable will have a synchronized copy in the local DMS database, defined with db package.

Structure and Organisation

Here is quick overview of the contents of this pacakge:

• README: Current file which is aimed towards developers who wish to use and modify the package functionality.

• config: This file contains data structures for this package.

• load: This file establishes a configuration loader using Viper, supports loading JSON files from various locations, applies defaults, and exposes functions to manage and obtain the loaded configuration.

Class Diagram

Source

config class diagram

Rendered from source file

!$rootUrlGitlab = "https://gitlab.com/nunet/device-management-service/-/raw/main"
!$packageRelativePath = "/internal/config"
!$packageUrlGitlab = $rootUrlGitlab + $packageRelativePath
 
!include $packageUrlGitlab/specs/class_diagram.puml

Functionality

The methods of this package are explained below:

getViper

• signature: getViper() *viper.Viper

• input: None

• output: A pointer to a viper.Viper struct

getViper function creates a new viper.Viper instance used for configuration management with search paths.

setDefaultConfig

• signature: setDefaultConfig() *viper.Viper

• input: None

• output: A pointer to a viper.Viper struct

Sets default values for various configuration options in a viper.Viper instance.

LoadConfig

• signature: LoadConfig()

• input: None

• output: None

LoadConfig loads the configuration from the search paths.

SetConfig

• signature: SetConfig(key string, value interface{})

• input #1 : key for the configuration value

• input #2 : value corresponding to the key provided

• output: None

SetConfig Sets a specific configuration value using key value pair provided. In case of any error, the deafult configuration values are applied.

GetConfig

• signature: GetConfig() *Config

• input: None

• output: internal.config.Config

GetConfig returns the loaded configuration data.

findConfig

• signature: findConfig(paths []string, filename string) ([]byte, error)

• input # 1: list of search paths

• input # 1: name of the configuration file

• output: contents of the configuration file

• output(error): error message

findConfig Searches for the configuration file in specified paths and returns content or error.

removeComments

• signature: removeComments(configBytes []byte) []byte

• input: The byte array containing the configuration file content

• output: byte array with comments removed

removeComments Removes comments from the configuration file content using a regular expression.

Data Types

• internal.config.Config: holds the overall configuration with nested structs for specific sections

type Config struct {
	General `mapstructure:"general"`
	Rest    `mapstructure:"rest"`
	P2P     `mapstructure:"p2p"`
	Job     `mapstructure:"job"`
}

• internal.config.General: Configuration related to general application behavior (data paths, debug mode)

type General struct {
	MetadataPath string `mapstructure:"metadata_path"`
	DataDir      string `mapstructure:"data_dir"`
	Debug        bool   `mapstructure:"debug"`
}

• internal.config.Rest: Configuration for the REST API (port number)

type Rest struct {
	Port int `mapstructure:"port"`
}

• internal.config.P2P: Configuration for the P2P network

type P2P struct {
	ListenAddress  []string `mapstructure:"listen_address"`
	BootstrapPeers []string `mapstructure:"bootstrap_peers"`
}

• internal.config.Job: Configuration for background tasks (log update interval, target peer for deployments, container cleanup interval)

type Job struct {
	LogUpdateInterval int    `mapstructure:"log_update_interval"` // in minutes
	TargetPeer        string `mapstructure:"target_peer"`         // specific peer to send deployment requests to - XXX probably not a good idea. Remove after testing stage.
	CleanupInterval   int    `mapstructure:"cleanup_interval"` // docker container and images clean up interval in days
}

Testing

proposed Unit tests for each functionality are defined in files with *_test.go naming convention.

Proposed Functionality / Requirements

List of issues

All issues that are related to the implementation of internal package can be found below. These include any proposals for modifications to the package or new functionality needed to cover the requirements of other packages.

• internal package implementation TBD

proposed Functionalities

Following Gherkin feature files describe the proposed functionality for config package.

1. Load default DMS configuration: see scenario definition

2. Restore default DMS configuration: see scenario definition

3. Load existing DMS configuration: see scenario definition

References