1 of 38

release

Last updated: 2024-11-21 22:05:58.821093 File source: link on GitLab

Device Management Service (DMS)

Device Management Service (DMS)

About

Device Management Service or DMS enables a machine to join the decentralized NuNet network both as a compute provider, offering its resources to the network, or to leverage the compute power of other machines in the network for processing tasks. Eventually users with available hardware resources will get compensated whenever their machine is utilized for a computational job by other users in the network. The ultimate goal of the platorm is to create a decentralized compute economy that is able to sustain itself.

Payment

All transactions on the Nunet network are expected to be conducted using the platform's utility token NTX. However, DMS is currently in development, and payment isn't part of v0.5.0-boot release. NTX payments are expected to be implemented in the Public Alpha Mainnet milestone within later release cycles.

Note: If you are a developer, please check out the DMS specifications and Building from Source sections of this document.

Installation

You can install Device Management Service (DMS) via binary releases or building it from source.

Binary releases

You can find all binary releases here and other builds in-between releases in the package registry. We currently support ARM and AMD64 architectures. You may check your architecture with appropriate command (uname -p for linux) and refer to the architecture name mapping e.g. here for figuring correct package to download.

Note: If you intalled the binary from a release and you would like to act as compute provider, you may need to check permissions and features to enable some required and optional features.

Ubuntu/Debian

Download the latest .deb package from the package registry
Install the Debian package with apt or dpkg:

sudo apt update
sudo apt install ./nunet-dms_<latest>.deb -y

Some dependencies such as docker and libsystemd-dev might be missing so it's recommended to fix install by running:

sudo apt -f install

Building from source

We currently support Linux and MacOS (Darwin).

Dependencies

iproute2 (linux only)
build-essential (linux only)
libsystemd-dev (linux only)
go (v1.21.7 or later)

Clone the repository:

git clone https://gitlab.com/nunet/device-management-service.git

Build the CLI:

cd device-management-service
make

This will result in a binary file in builds/ folder named as dms_linux_amd64 or dms_darwin_arm64 depending on the platform.

Note: If you built from source and would like to act as a compute provider, you may need to check permissions and features to enable some required and optional features.

To cross compile to arm, cross compilers need to be installed. In particular arm-linux-gnueabihf and aarch64-linux-gnu. For debian systems, install with:

apt install gcc-arm-linux-gnueabihf gcc-aarch64-linux-gnu

You can add the compiled binary to a directory in your $PATH. See the Usage section for more information.

Permissions and features (for compute providers)

The following applies only for compute providers. If you're running a client/orchestrator, you do not need to set any permissions.

If you built DMS from source or installed the binary from one of our releases, you may need to set some permissions to the binary to enable some features.

Darwin users: unfortunately, the DMS can't work with granular permissions on Mac. So, for now, if running a compute provider, you will have to run the nunet daemon (nunet run) as root.

For Linux users, granular permissions will have to be set to the binary (optionally, but NOT recommended, you can run the binary as root).

Net-admin permission and IP over libp2p

Note: cap_net_admin is a required capability for compute providers.

Setting the permission enables IP over libp2p which is a feature that enhances the capabilities of compute providers, allowing them to participate in a wider range of jobs. One capability enabled with this feature is to do port forwarding which it won't be possible without setting the right unix permissions.

One of the reasons for requiring this permission is because this feature depends on creating and managing tun interfaces.

To enable this feature, the nunet binary requires network-admin capabilities. These capabilities allow the application to perform network configuration tasks without needing to run the entire application as root, which is a more secure approach.

To set the necessary capabilities, run the following command:

sudo setcap cap_net_admin+ep /usr/bin/nunet

The above command depends on: libcap2-bin (Debian/Ubuntu) or libcap (CentOS/RHEL/Arch...)

Firecracker (for compute providers)

Note: Linux only.

To act as a compute provider capable of receiving and running jobs with Firecracker, ensure your user is part of the kvm group. You can do this by running:

sudo usermod -aG kvm $USER

Installation on VMs

Skip doing an unattended installation for the new Ubuntu VM as it might not add the user with administrative privileges.
Enable Guest Additions when installing the VM (VirtualBox only).
Always change the default NAT network setting to Bridged before booting the VM.
Install Extension Pack if using VirtualBox (recommended).
Install VMware Tools if using VMware (recommended).
ML on GPU jobs on VMs are not supported.

Installation on WSL

Install WSL through the Windows Store.
Install the Update KB5020030 (Windows 10 only).
Install Ubuntu 20.04 through WSL.
Enable systemd on Ubuntu WSL.
ML Jobs deployed on Linux cannot be resumed on WSL.

Though it is possible to run ML jobs on Windows machines with WSL, using Ubuntu 20.04 natively is highly recommended to avoid unpredictability and performance losses.

If you are using a dual-boot machine, make sure to use the wsl --shutdown command before shutting down Windows and running Linux for ML jobs. Also, ensure your Windows machine is not in a hibernated state when you reboot into Linux.

System Requirements

CPU-only machines

Minimum System Requirements

We require you to specify CPU (MHz x no. of cores) and RAM, but your system must meet at least the following requirements before you decide to onboard it:

CPU: 2 GHz
RAM: 4 GB
Free Disk Space: 10 GB
Internet Download/Upload Speed: 4 Mbps / 0.5 MBps

If the above CPU has 4 cores, your available CPU would be around 8000 MHz. So if you want to onboard half your CPU and RAM on NuNet, you can specify 4000 MHz CPU and 2000 MB RAM.

Recommended System Requirements

CPU: 3.5 GHz
RAM: 8-16 GB
Free Disk Space: 20 GB
Internet Download/Upload Speed: 10 Mbps / 1.25 MBps

GPU Machines

Minimum System Requirements

CPU: 3 GHz
RAM: 8 GB
GPU: 4 GB VRAM (NVIDIA, AMD, or Intel discrete GPU with manually installed drivers)
Free Disk Space: 50 GB
Internet Download/Upload Speed: 50 Mbps

Note: For AMD64 platforms, we recommend using HiveOS as it comes with all necessary drivers pre-installed. For other setups, proper GPU drivers must be manually installed. See the GPU Driver Installation section for instructions.

Recommended System Requirements

CPU: 4 GHz
RAM: 16-32 GB
GPU: 8-12 GB VRAM (NVIDIA, AMD, or Intel discrete GPU with manually installed drivers)
Free Disk Space: 100 GB
Internet Download/Upload Speed: 100 Mbps

GPU Driver Installation

NuNet DMS requires properly installed GPU drivers to function correctly. We do not automatically install drivers to ensure compatibility and flexibility across different user setups.

For AMD64 Platforms:

We recommend using the Ubuntu-based HiveOS for the easiest setup.

If you prefer to use a different operating system or need to install drivers manually, please follow these steps:

NVIDIA GPUs:

Visit the NVIDIA Official Driver Downloads page.
Select your GPU model and operating system.
Download and install the recommended driver.
Reboot your system after installation.

AMD GPUs:

Visit the AMD Drivers and Support for Processors and Graphics page.
Select your GPU model and operating system.
Download and install the recommended driver.
Reboot your system after installation.

Intel Discrete GPUs:

Visit the Intel® software for general purpose GPU capabilities documentation page.
Select your GPU model and operating system.
Download and install the recommended driver.
Reboot your system after installation.

For detailed instructions specific to your operating system, please refer to the documentation provided by NVIDIA, AMD, or Intel.

Note: Ensure that you have the correct permissions to install drivers on your system. On Linux systems, you may need to use sudo or log in as root to install drivers.

Usage

Quick Start

Before starting, ensure that you have properly installed GPU drivers if you're using a GPU-enabled machine. For AMD64 platforms, we recommend using HiveOS for the easiest setup. For other configurations, refer to the GPU Driver Installation section for instructions.

This quick start guide will walk you through the process of setting up a Device Management Service (DMS) instance for the first time and getting it running. We'll cover creating identities, setting up capabilities, and running the DMS.

The NuNet CLI

The NuNet CLI is the command-line interface for interacting with the Nunet Device Management Service (DMS). It provides commands for managing keys, capabilities, configuration, running the DMS, and more. It's essential for setting up and administering your DMS instance.

Key Concepts

Actor: An independent entity in the Nunet system capable of performing actions and communicating with other actors.
Capability: Defines the permissions and restrictions granted to actors within the system.
Key: A cryptographic key pair used for authentication and authorization within the DMS.

You can find a detailed documentation here.

Creating identities

The first step is to generate identities/keys and capability contexts. It is recommended that two keys are setup: one for the user (default name user) and another for the dms (default name dms)

A capability context is created with the nunet cap new <context> command and it is anchored on a key with the context name.

To set up a new identity/create a new key, run the command:

$ nunet key new <identity>

Then, to initialize its capability context:

$ nunet cap new <identity>

In this example, we are going to set up two identities:

First for the user

$ nunet key new user

then for the dms instance.

$ nunet key new dms

We then setup the capability contexts for each identity:

First the user

$ nunet cap new user

then the dms instance.

$ nunet cap new dms

If you use a ledger wallet for your personal key, you can create the user context as follows:

Create a new key for the user

$ nunet key did ledger

Then create the capability context for the user

$ nunet cap new ledger:user

You can create as many identities as you want, specially if you want to manage multiple DMS instances.

The key new command returns a DID key for the specified identity. Remember to secure your keys and capability contexts, as they control access to your NuNet resources. They are encrypted and stored under $HOME/.nunet by default.

Each time a new identity is generated it will prompt the user for a passphrase. The passphrase is associated with the created identity, thus a different passphrase can be set up for each identity. If you prefer, it's possible to set a DMS_PASSPHRASE environment variable to avoid the command prompt.

Using a Ledger Wallet

It is also possible to use a Ledger Wallet instead of creating a new key; this is recommended for user contexts, but you should not use it for the dms context as it needs the key to sign capability tokens.

To set up a user context with a Ledger Wallet, you need the ledger-cli script from NuNet's ledger wallet tool. The tool uses the Eth application and specifically the first Eth account with signing of personal messages. Everything that needs to be signed (namely capability tokens) will be presented on your Nano's screen in plaintext so that you can inspect it.

You can get your Ledger wallet's DID with:

$ nunet key did ledger

Setting up Capabilities

NuNet's network communication is powered by the NuActor System, a zero-trust system that utilizes fine-grained capabilities, anchored on DIDs, following the UCAN model.

Once both identities are created, you'll need to set up capabilities. Specifically:

Create capability contexts for both the user and each of your DMS instances.
Add the user's DID as a root anchor for the DMS capability context. This ensures that the DMS instance fully trusts the user, granting complete control over the DMS (the root capability).
If you want your DMS to participate in the public NuNet testnet (and eventually the mainnet), you'll need to set up capability anchors for your DMS:
1. Create a capability anchor to allow your DMS to accept public behavior invocations from authorized users and DMSs in the NuNet ecosystem.
2. Add this token to your DMS as a require anchor.
3. Request a capability token from NuNet to invoke public behaviors on the network.
4. Add the token as a provide anchor in your personal capability context.
5. Delegate to your DMS the ability to make public invocations using your token.
6. Add the delegation token as a provide anchor in your DMS.

Add a root anchor for your DMS context

You can do this by invoking the dms cap anchor command:

$ nunet cap anchor --context dms --root <user-did>

Where <user-did> is the user did created above in Creating identities and can be obtained by:

$ nunet key did <user>

or if you are using a Ledger Wallet

$ nunet key did ledger

Setup your DMS for the public testnet

The NuNet DID

did:key:zzCHUybNYmK8QsttZwXqUX8aDLoBGHnMCakDX2RpsGwmXmYHEW

Create a capability anchor for public behaviors

Create the grant

$ nunet cap grant --context user --cap /public --cap /broadcast --topic /nunet --expiry 2024-12-31 <nunet-did>

or if you are using a Ledger Wallet

$ nunet cap grant --context ledger:user --cap /public --cap /broadcast --topic /nunet --expiry 2024-12-31 <nunet-did>

And the granted token as a require anchor

$ nunet cap anchor --context dms --require <the-grant-output>

The first command grants nunet authorized users the capability to invoke public behaviors until December 31, 2024, and outputs a token.

The second command consumes the token and adds the require anchor for your DMS

Ask NuNet for a public network capability token

To request tokens for participating in the testnet, please go to did.nunet.io and submit the did you generated along with your gitlab username and an email address to receive the token. It's highly recommended that you use a Ledger hardware wallet for your keys.

Use the NuNet granted token to authorize public behavior invocations in the public network

3.1 Add the provide anchor to your personal context

$ nunet cap anchor --context user --provide <the-token-you-got-from-nunet>

or if you are using a Ledger Wallet

$ nunet cap anchor --context ledger:user --provide <the-token-you-got-from-nunet>

3.2 Delegate to your DMS

$ nunet cap delegate --context user --cap /public --cap /broadcast --topic /nunet --expiry 2024-12-31 <your-dms-did>

or if you are using a Ledger Wallet

$ nunet cap delegate --context ledger:user --cap /public --cap /broadcast --topic /nunet --expiry 2024-12-31 <your-dms-did>

3.3 Add the delegation token as a provide anchor in your DMS

$ nunet cap anchor --context dms --provide <the-delegate-output>

The first command ingests the NuNet provided token and the last two commands use this token to delegate the public behavior capabilities to your DMS.

Running DMS

If everything was setup properly, you should be able to run:

$ nunet run

Darwin users: If you plan to onboard your computer power to the network, You may need to run with sudo. See the optional features and permissions section for more information.

By default, DMS runs on port 9999.

Provide Compute Power to the Network

If you want to contribute your computer's resources (CPU, RAM, GPU, storage) to the network, you have to onboard your machine.

Follow our Compute Provider Guide to get started.

Deploy Jobs on the Network

Every node on the network can deploy workloads across available compute resources, given the necessary capabilities. Learn how deployments work by following our Deployments Guide.

REST Endpoints

Refer to the api package README for the list of all endpoints. Head over to project's issue section and create an issue with your question.

Configuration

Config file

The DMS searches for a configuration file dms_config.json in the following locations, in order of priority whenever it's started:

The current directory (.)
$HOME/.nunet
/etc/nunet

The configuration file must be in JSON format and it does not support comments. It's recommended that only the parameters that need to be changed are included in the config file so that other parameters can retain their default values.

It's possible to manage configuration using the config subcommand as well. nunet config set allows setting each parameter individually and nunet config edit will open the config file in the default editor from $EDITOR

Run Two DMS Instances Side by Side

As a developer, you might find yourself needing to run two DMS instances, one acting as an SP (Service Provider) and the other as a CP (Compute Provider).

Step 1:

Clone the repository to two different directories. You might want to use descriptive directory names to avoid confusion.

Step 2:

You need to modify some configurations so that both DMS instances do not end up trying to listen on the same port and use the same path for storage. For example, ports on p2p.listen_address, rest.port, general.user_dir etc... neeed to be different for two instances on the same host.

The dms_config.json file can be used to modify these settings. Here is a sample config file that can be modified to your preference:

{
  "p2p": {
    "listen_address": ["/ip4/0.0.0.0/tcp/9100", "/ip4/0.0.0.0/udp/9100/quic-v1"]
  },
  "general": {
    "user_dir": "/home/user/.config/nunet/dms/",
    "debug": true
  },
  "rest": {
    "port": 10000
  }
}

Prefer to use absolute paths and have a look at the config structure for more info.

Tests

Some packages contain tests, and it is always best to run them to ensure there are no broken tests before submitting any changes. Before running the tests, the Firecracker executor requires some test data, such as a kernel file, which can be downloaded with:

make testdata

After the download is complete, all unit tests can be run with the following command. It's necessary to include the unit tag due to the existence of files that contain functional and integration tests.

go test --tags unit ./...

Help in contributing tests is always appreciated :)

Specification

Description

NuNet is a computing platform that provides globally distributed and optimized computing power and storage for decentralized networks, by connecting data owners and computing resources with computational processes in demand of these resources. NuNet provides a layer of intelligent interoperability between computational processes and physical computing infrastructures in an ecosystem which intelligently harnesses latent computing resources of the community into the global network of computations.

Detailed information about the NuNet platform, concepts, architecture, models, stakeholders can be found in these two papers:

DMS (Device Management Service) acts as the foundation of the NuNet platform, orchestrating the complex interactions between various computing resources and users. DMS implementation is structured into packages, creating a more maintainable, scalable, and robust codebase that is easier to understand, test, and collaborate on. Here are the existing packages in DMS and their purposes:

dms: Responsible for starting the whole application and core DMS functionality such as onboarding, job orchestration, job and resource management, etc.
internal: Code that will not be imported by any other packages and is used only on the running instance of DMS. This includes all configuration-related code, background tasks, etc.
db: Database used by the DMS.
storage: Disk storage management on each DMS for data related to DMS and jobs deployed by DMS. It also acts as an adapter to external storage services.
api: All API functionality (including REST API, etc.) to interact with the DMS.
cmd: Command line functionality and tools.
network: All network-related code such as p2p communication, IP over Libp2p, and other networks that might be needed in the future.
executor: Responsible for executing the jobs received by the DMS. Interface to various executors such as Docker, Firecracker, etc.
telemetry: Logs, traces, and everything related to telemetry.
plugins: Defined entry points and specs for third-party plugins, registration, and execution of plugin code.
types: Contains data models imported by various packages.
utils: Utility tools and functionalities.
tokenomics: Interaction with blockchain for the crypto-micropayments layer of the platform.

Design and Architecture

Conceptual Basis

Main concepts of the architecture of DMS, the main component of the NuNet platform, can be found in the Yellow Paper.

Ontology

The Nunet Ontology, which forms the basis of the design, is explained in the articles below:

Architecture

Refer to the following items to understand DMS architecture at a high level.

Research

Relevant research work that has informed the design of DMS can be found below:

Functionality

DMS is currently being refactored and new functionality will be added.

Data Types

Refer to the DMS global class diagram in this section and various packages for data models.

References

In addition to the relevant links added in the sections above, you can also find useful links here: NuNet Links.

Class Diagram

The global class diagram for the DMS is shown below.

Source File

Global Class Diagram

actor

Last updated: 2024-11-21 22:05:59.080133 File source: link on GitLab

NuActor: Secure Actor Programming in Decentralized Systems

NuActor is a framework designed for secure actor oriented programming in decentralized systems. The framework utilizes zero trust interactions, whereby every message is authenticated individually at the point of interaction. The system supports fine-grained capabilities, anchored in decentralized identifiers (see DID) and effected with user controlled authorization networks (see UCAN).

Secure Interactions in Decentralized Systems

Decentralized systems are distributed systems where there are different stake holders and controlling entities who are mutually distrustful. Actors are ideally suited for modeling and programming such systems, as they are able to express concurrency, distribution, and agency on behalf of their controllers.

However, given the open ended computing nature of decentralized systems, there is a fundamental problem in securing interactions. Because the system is open, there is effectively no perimeter; the messages are coming from the Internet, and can potentially originate in malicious or hostile actors.

NuActor takes the following approach:

The only entity an actor can fully trust is itself and its controller.
All messages invoking a behavior carry with them capability tokens that authorize them to perform the invocation.
Invocations are checked at dispatch so that it is always verified whether an invocation is allowed, anchored on the entities the actor trusts for the required capabilities.
There is no central authority; every entity (identified by a DID) can issue their own capability tokens and anchor trust wherever they want.
There are certain entities in the open public networks that may be marginally trusted to vet users (KYC) for invoking public behaviors. The set of such entities is open, and everyone is free to trust whoever they want. The creators of the network at bootstrap are good candidates for such entities.
Trust is ephemeral and can be revoked at all times.
In effect, users are in control of authorization in the network (UCAN!)

Capabilities

Behaviors and the Capability Namespace

Capabilities are defined in a hierarchical namespace, akin to the UNIX file system structure. The root capability, which implicitly has all other capabilities, is /. Every other capability extends this path, separating the namespace with additional /s. A capability is narrower than another if it is a subpath in the UNIX sense. So /A imples /A/B and so on, but /A does not imply /B.

Behaviors have names that directly map to capabilities. So the behavior namespace is also hierarchical, allowing for easy automated matching of behaviors to capabilities.

Capability Tokens

Capabilities are expressed with a token, which is a structured object signed by the private key of the issuer. The issuer is in the token as a DID, which allows any entity inspecting the token to verify by retrieving the public key associated with the DID. Typically these are key DIDs, which embed the public key directly.

The structure of the token is as following:

type Token struct {
	// DMS tokens
	DMS *DMSToken `json:"dms,omitempty"`
}

type DMSToken struct {
	Action     Action       `json:"act"`
	Issuer     did.DID      `json:"iss"`
	Subject    did.DID      `json:"sub"`
	Audience   did.DID      `json:"aud"`
	Topic      []Capability `json:"topic,omitempty"`
	Capability []Capability `json:"cap"`
	Nonce      []byte       `json:"nonce"`
	Expire     uint64       `json:"exp"`
	Depth      uint64       `json:"depth,omitempty"`
	Chain      *Token       `json:"chain,omitempty"`
	Signature  []byte       `json:"sig,omitempty"`
}

The Subject is the DID of the entity to which the Issuer grants (if the chain is empty) or delegates the capabilities listed in the Capability field and the broadcast topics listed in the Topic field. The audience may be empty, but when present it restricts the receiver of invocations to a specified entity.

The Action can be any of Delegate, Invoke or Broadcast, with revocations to be added in the very near future.

If the Action is Delegate then the Issuer confers to the Subject the ability to further create new tokens, chained on this one.

If the Action is Invoke or Broadcast, then the token confers to the Subject the capability to make an invocation or broadcast to a behavior. Such tokens are terminal and cannot be chained further.

The Chain field of the token inlines the chain of tokens (could be a single one) on which the capability transfer is anchored on.

Note that the delegation spread can be restricted by the issuer of a token using the Depth field. If set, it is the maximum chain depth at which a token can appear. If it appears deeper in the chain, the token chain fails verification.

Finally, all capabilities have an expiration time (in UNIX nanoseconds). An expired token cannot be used any more and fails verification.

Anchors of Trust

In order to sign and verify token chains, the receiver needs to install some trust anchors. Specifically, we distinguish 3 types of anchors:

root anchors which are DIDs that are fully trusted for input with implicit root capability. Any valid chain anchored on one of our roots will be admissible.
require anchors which are tokens that act as side chains for marginal input trust. These tokens admit a chain anchored in their subject, as long as the capability and depth constraints are satisfied.
provide anchors which are tokens that anchour the actor's output invocation and broadcast tokens. These are delegations which the actor can use to prove that it has the required capabilities, beside self-signing.

Token Chain Verification

The token chain is verified with strict rules:

The entire chain must not have expired.
Each token in the chain cannot expire before its chain.
Each token must match the Issuer with the Subject of its chain.
Each token in the chain can only narrow (attenuate) the capabilities of its chain.
Each token in the chain can only narrow the audience; an empty audience ("to whom it may concern") can only be narrowed once to an audience DID and all chains build on top must concern the same audience.
The chain of a token can only delegate.
The signature must verify.
The whole chain must recursively verify.

Programming NuActors

The `actor` package

The Go implementation of NuActor lives in the actor package of DMS.

To use it:

import "gitlab.com/nunet/device-management-service/actor"

The network substrate for NuActor is currently implemented with libp2p, with broadcast using gossipsub.

The Security Context

Each actor has a key pair for signing its messages; the actor's id is the public key itself and is embedded in every message it sends. The private key for the actor lives inside the actor's SecurityContext.

In general:

each actor has its own SecurityContext; however, if the actor wants to create multiple subactors and act as an ensemble, it can share it.
the key pair is ephemeral; however, the root actor in the process has a persistent key pair, which matches the libp2p key and Peer ID. This makes the actor reachable by default given its Peer ID or DID.
every actor in the process shares a DID, which is the ID of the root actor.

Each Security Context is anchored in a process wide CapabilityContext, which stores anchors of trust and ephemeral tokes consumed during actor interactions.

The CapabilityContext itself is anchored on a TrustCotext, which contains the private key for the root actor and the process itself.

Actor Handles

type Handle struct {
	ID      ID      `json:"id"`
	DID     DID     `json:"did"`
	Address Address `json:"addr"`
}

type Address struct {
	HostID       string `json:"host,omitempty"`
	InboxAddress string `json:"inbox,omitempty"`
}

Messages

type Envelope struct {
	To         Handle          `json:"to"`
	Behavior   string          `json:"be"`
	From       Handle          `json:"from"`
	Nonce      uint64          `json:"nonce"`
	Options    EnvelopeOptions `json:"opt"`
	Message    []byte          `json:"msg"`
	Capability []byte          `json:"cap,omitempty"`
	Signature  []byte          `json:"sig,omitempty"`

	Discard func() `json:"-"`
}

type EnvelopeOptions struct {
	Expire  uint64 `json:"exp"`
	ReplyTo string `json:"cont,omitempty"`
	Topic   string `json:"topic,omitempty"`
}

The Actor Interfaces

type Actor interface {
	Context() context.Context
	Handle() Handle
	Security() SecurityContext

	AddBehavior(behavior string, continuation Behavior, opt ...BehaviorOption) error
	RemoveBehavior(behavior string)

	Receive(msg Envelope) error
	Send(msg Envelope) error
	Invoke(msg Envelope) (<-chan Envelope, error)

	Publish(msg Envelope) error
	Subscribe(topic string, setup ...BroadcastSetup) error

	Start() error
	Stop() error

	Limiter() RateLimiter
}

Sending Messages

The following code shows how to send a message at the call site:

	msg, _ := actor.Message(
		myActor.Handle(),
		destinationHandle,
		"/some/behavior",
		MyMessage{ /*...*/ },
	)
	_ = myActor.Send(msg)

At the receiver this is how we can react to the message:

	_ = myActor.AddBehavior("/some/behavior", func(msg Envelope) {
		defer msg.Discard()
		// process message
	})

Notice the _ for errors, please don't do this in production.

Interactive Invocations

Interactive invocations are a combinations of a synchronous send and wait for a reply.

At the call site:

	msg, _ := actor.Message(
		myActor.Handle(),
		destinationHandle,
		"/some/behavior",
		MyMessage{ /*...*/ },
	)
	replyChan, _ := myActorInvoke(msg)
	reply := <-replyChan
	defer reply.Discard()
	// process reply ...

At the receiver this is how we can create an interactive behavior:

	_ = myActor.AddBehavior("/some/behavior", func(msg Envelope) {
		defer msg.Discard()
		reply, _ := actor.ReplyTo(
			msg
			MyReply{ /*...*/ },
		)
		_ = mayActor.Send(MyReply)
	})

Again, notice the _ for errors, please don't do this in production.

Broadcast

We can easily broadcast messages to all interested parties in a topic.

At the broadcast site:

	msg, _ := actor.Message(
		myActor.Handle(),
		destinationHandle,
		"/some/broadcast/behavior",
		MyMessage{ /*...*/ },
		actor.WithMessageTopic("/some/topic"),
	)
	_ = actor.Publish(msg)

At the receiver:

	_ = myActor.Subscribe("/some/topic")
	_ = myActor.AddBehavior("/some/broadcast/behavior", func(msg Envelope) {
		defer msg.Discard()
		// process the message
	}, actor.WithBehaviorTopic("/some/topic"),
	)

Discarding Capability Tokens

Notice all these defer msg.Discard() in the examples above; this is necessary to ensure deterministic cleanup of tokens exchanged during the interaction. Please do not forget that.

Behind the Scenes

Sending a Message

Invoking a Behavior

Verifying Capabilities

api

Last updated: 2024-11-21 22:05:59.567392 File source: link on GitLab

api

Specification

Description

The api package contains all API functionality of Device Management Service (DMS). DMS exposes various endpoints through which its different functionalities can be accessed.

Structure and Organisation

Here is quick overview of the contents of this directory:

README: Current file which is aimed towards developers who wish to use and modify the api functionality.
api.go: This file contains router setup using Gin framework. It also applies Cross-Origin Resource Sharing (CORS) middleware and OpenTelemetry middleware for tracing. Further it lists down the endpoint URLs and the associated handler functions.
actor.go: Contains endpoints for actor interaction.

Functionality

Configuration

The REST server by default binds to 127.0.0.1 on port 9999. The configuration file dms_config.json can be used to change to a different address and port.

The parameters rest.port and rest.addr define the port and the address respectively.

You can use the following format to construct the URL for accessing API endpoints

http://localhost:<port>/api/v1/<endpoint>

Currently, all endpoints are under the /actor path

/actor/handle

Retrieve actor handle with ID, DID, and inbox address

Response:

{
    "id": <actor_id>,
    "did" : <actor_did>,
    "addr" : <inbox_address>
}

/actor/send

Send a message to actor

The request should be an enveloped message

{
    "to": {
        "id": <actor_id>,
        "did" : <actor_did>,
        "addr" : <inbox_address>
    },
    "be": <behavior>,
    "from": {
        "id": <actor_id>,
        "did" : <actor_did>,
        "addr" : <inbox_address>
    },
    "nonce": <nonce>,
    "opt": {
        "cont": <topic_to_publish_to>,
        "exp": <time_to_expire>
    },
    "msg": <message>,
    "cap": <capability>,
    "sig": <signature>
}

/actor/invoke

Invoke actor with message

The request should be an enveloped message

{
    "to": {
        "id": <actor_id>,
        "did" : <actor_did>,
        "addr" : <inbox_address>
    },
    "be": <behavior>,
    "from": {
        "id": <actor_id>,
        "did" : <actor_did>,
        "addr" : <inbox_address>
    },
    "nonce": <nonce>,
    "opt": {
        "cont": <topic_to_publish_to>,     // needs to be specified for invoke
        "exp": <time_to_expire>
    },
    "msg": <message>,
    "cap": <capability>,
    "sig": <signature>
}

/actor/broadcast

Broadcast message to actors

The request should be an enveloped message

{
    "to": {},                                 // should be empty for broadcast
    "be": <behavior>,
    "from": {
        "id": <actor_id>,
        "did" : <actor_did>,
        "addr" : <inbox_address>
    },
    "nonce": <nonce>,
    "opt": {
        "topic": <topic_to_publish_to>,
        "exp": <time_to_expire>
    },
    "msg": <message>,
    "cap": <capability>,
    "sig": <signature>
}

For more details on these Actor API endpoints, refer to the cmd/actor package on how the they are used.

cmd

Last updated: 2024-11-21 22:05:59.815188 File source: link on GitLab

Introduction

The Nunet Command-Line Interface (CLI) serves as a powerful tool for interacting with the Nunet ecosystem, enabling you to manage network configurations, control capabilities, and handle cryptographic keys. It provides a comprehensive set of commands to streamline various tasks and operations within the Nunet network.

Top-Level Commands

Actor System

nunet actor

This command provides a suite of operations tailored for interacting with the Nunet Actor System. It enables you to communicate with actors within the network, facilitating actions like sending messages, invoking specific behaviors, and broadcasting information to multiple actors simultaneously.

Detailed documentation can be found here.

Capability Management

nunet cap

This command focuses on capability management within the Nunet ecosystem. It allows you to define, delegate, and control the permissions and authorizations granted to different entities, ensuring secure and controlled interactions within the network.

Detailed documentation can be found here.

Configuration Management

nunet config

The config command allows to interact with and manage your configuration file directly from the command line. This allows you to view existing settings, modify them as needed, and ensure Nunet DMS is tailored to your preferences.

Usage

nunet config COMMAND

Available Commands

edit: Opens the configuration file in your default text editor for manual adjustments.
get: Retrieve and display the current value associated with a specific configuration key.
set: Modify the configuration file by assigning a new value to a specified key.

Flags

-h, --help: Display help information for the config command and its subcommands.

Key Management

nunet key

The Nunet Key Management CLI allows generating new keypairs and retrieve the Decentralized Identifier (DID) associated with a specific key.

Main Command

nunet key
- Description: The primary command to manage keys within the Nunet DMS
- Usage: nunet key COMMAND
- Available Commands:
  - did: Retrieve the DID for a specified key
  - new: Generate a new keypair
- Flags:
  - -h, --help: Display help information for the main key command.

Subcommands

nunet key did
- Description: Retrieves and displays the DID associated with a specified key. This DID uniquely identifies the key within the Nunet network.
- Usage: nunet key did <key-name> [flags]
- Arguments:
  - <key-name>: The name of the key for which the DID is to be retrieved
- Flags:
  - -h, --help: Display help information for the did command
nunet key new
- Description: Generates a new keypair and securely stores the private key in the user's local keystore. The corresponding public key can be used for various cryptographic operations within the Nunet DMS
- Usage: nunet key new <name> [flags]
- Arguments:
  - <name>: A name to identify the newly generated keypair
- Flags:
  - -h, --help: Display help information for the new command.

Important Considerations:

Keep your private keys secure, as they provide access to your identity and associated capabilities within the Nunet DMS
Choose descriptive names for your keypairs to easily identify their purpose or associated devices.

Run Command

nunet run

Starts the Nunet Device Management Service (DMS) process, responsible for handling network operations and device management.

Usage:

nunet run [flags]

Flags:

-c, --context string: Specifies the key and capability context to use (default: "dms").
-h, --help: Displays help information for the run command.

Example:

nunet run

This starts the Nunet DMS with the default "dms" context.

TAP Command

`nunet tap`

Purpose: Creates a TAP (network tap) interface to bridge the host network with a virtual machine (VM). It also configures network settings like IP forwarding and iptables rules.

Key Points:

Root Privileges Required: This command necessitates root or administrator privileges for execution due to its manipulation of network interfaces and system-level settings.

Usage:

nunet tap [main_interface] [vm_interface] [CIDR] [flags]

Arguments:

main_interface: (e.g., eth0) The name of the existing network interface on your host machine that you want to bridge with the TAP interface.
vm_interface: (e.g., tap0) The name you want to assign to the newly created TAP interface.
CIDR: (e.g., 172.16.0.1/24) The Classless Inter-Domain Routing (CIDR) notation specifying the IP address range and subnet mask for the TAP interface. This ensures that the VM or container connected to the TAP has its own IP address within the specified network.

Flags:

-h, --help: Displays help information for the tap command.

Example:


sudo nunet tap eth0 tap0 172.16.0.1/24

This command will create a TAP interface named 'tap0' bridged to your host's 'eth0' interface. The 'tap0' interface will be assigned an IP address of '172.16.0.1' with a subnet mask of '/24'. This configuration allows a VM connected to 'tap0' to access the network through your host's 'eth0' interface.

Important Notes:

Ensure you have the necessary permissions to execute this command.
Be cautious when configuring network settings, as incorrect configurations can disrupt your network connectivity.

GPU Command

`nunet gpu`

Purpose: The nunet gpu command provides gpu related apis.

Usage:

nunet gpu <operation>

Available Operations:

list: List all the available GPUs on the system.
test: Test the GPU deployment on the system using docker.

Flags:

-h, --help: Display help information for the gpu command and its subcommands.

Example:

nunet gpu list

This command will list all the available GPUs on the system.

nunet gpu test

This command will test the GPU deployment on the system using docker.

actor

Last updated: 2024-11-21 22:06:00.127930 File source: link on GitLab

Nunet Actor System CLI

The Nunet Actor System CLI provides a set of commands for interacting with the Nunet actor system, enabling you to send messages to actors, invoke behaviors, and broadcast messages across the network.

Basic Commands

nunet actor msg: Constructs a message for an actor.
nunet actor send: Sends a constructed message to an actor.
nunet actor invoke: Invokes a behavior in an actor and returns the result.
nunet actor broadcast: Broadcasts a message to a topic, potentially reaching multiple actors.
nunet actor cmd: Invokes a predefined public behavior on an actor.

`nunet actor msg`

This command is used to create a message that can be sent to an actor. It encapsulates the behavior to be invoked and the associated payload data.

Usage

nunet actor msg <behavior> <payload> [flags]

Arguments

<behavior>: The specific behavior you want the actor to perform upon receiving the message
<payload>: The data accompanying the message, providing context or input for the behavior

Flags

-b, --broadcast string: Designates the topic for broadcasting the message.
-c, --context string: Specifies the capability context name
-d, --dest string: Identifies the destination handle for the message.
-e, --expiry time: Sets an expiration time for the message.
-h, --help: Displays help information for the msg command.
-i, --invoke: Marks the message as an invocation, requesting a response from the actor.
-t, --timeout duration: Sets a timeout for awaiting a response after invoking a behavior.

`nunet actor send`

This command delivers a previously constructed message to an actor.

Usage

nunet actor send <msg> [flags]

Arguments

<msg>: The message, created using the nunet actor msg command, to be sent.

Flags

-h, --help: Displays help information for the send command.

`nunet actor invoke`

This command directly invokes a specific behavior on an actor and expects a response.

Usage

nunet actor invoke <msg> [flags]

Arguments

<msg>: The message, crafted with nunet actor msg, containing the behavior and payload

Flags

-h, --help: Displays help information for the invoke command

`nunet actor broadcast`

This command disseminates a message to a designated topic, potentially reaching multiple actors who have subscribed to that topic.

Usage

nunet actor broadcast <msg> [flags]

Arguments

<msg>: The message to be broadcasted

Flags

-h, --help: Displays help information for the broadcast command.

Please let me know if you have any other questions.

`nunet actor cmd`

This command invokes a behavior on an actor.

Usage

nunet actor cmd [flags]
nunet actor cmd [command]

Available Commands

/broadcast/hello: Invoke /broadcast/hello behavior on an actor.
/dms/node/onboarding/offboard: Invoke /dms/node/onboarding/offboard behavior on an actor.
/dms/node/onboarding/onboard: Invoke /dms/node/onboarding/onboard behavior on an actor.
/dms/node/onboarding/resource: Invoke /dms/node/onboarding/resource behavior on an actor.
/dms/node/onboarding/status: Invoke /dms/node/onboarding/status behavior on an actor.
/dms/node/peers/connect: Invoke /dms/node/peers/connect behavior on an actor.
/dms/node/peers/dht: Invoke /dms/node/peers/dht behavior on an actor.
/dms/node/peers/list: Invoke /dms/node/peers/list behavior on an actor.
/dms/node/peers/ping: Invoke /dms/node/peers/ping behavior on an actor.
/dms/node/peers/score: Invoke /dms/node/peers/score behavior on an actor.
/dms/node/peers/self: Invoke /dms/node/peers/self behavior on an actor.
/dms/node/vm/list: Invoke /dms/node/vm/list behavior on an actor.
/dms/node/vm/start/custom: Invoke /dms/node/vm/start/custom behavior on an actor.
/dms/node/vm/stop: Invoke /dms/node/vm/stop behavior on an actor.
/public/hello: Invoke /public/hello behavior on an actor
/public/status: Invoke /public/status behavior on an actor

Flags

-c, --context string: Capability context name.
-d, --dest string: Destination DMS DID, peer ID or handle.
-e, --expiry time: Expiration time.
-h, --help: Help for the cmd command.
-t, --timeout duration: Timeout duration.

Broadcast Commands

/broadcast/hello
- Description: Invokes the /broadcast/hello behavior on an actor. This sends a "hello" message to a broadcast topic for polite introduction.
- Usage: nunet actor cmd /broadcast/hello [<param> ...] [flags]
- Flags:
  - -h, --help: Display help information for the /broadcast/hello command

DMS Node Commands

/dms/node/onboarding/offboard
- Description: Invokes the /dms/node/onboarding/offboard behavior on an actor. This is used to offboard a node from the DMS (Device Management Service).
- Usage: nunet actor cmd /dms/node/onboarding/offboard [<param> ...] [flags]
- Flags:
  - -f, --force: Force the offboarding process overriding any safety checks and invalid states.
  - -h, --help: Display help information for the /dms/node/onboarding/offboard command.
/dms/node/onboarding/onboard
- Description: Invokes the /dms/node/onboarding/onboard behavior on an actor. This is used to onboard a node to the DMS, making its resources available for use.
- Usage: nunet actor cmd /dms/node/onboarding/onboard [<param> ...] [flags]
- Flags:
  - -a, --available: Set the node as unavailable for job deployment (default: false).
  - -z, --cpu int: Set number of CPU cores.
  - -h, --help: Display help information for the /dms/node/onboarding/onboard command.
  - -l, --local-enable: Set server mode (enable for local) (default: true).
  - -m, --memory uint: Set the value for memory usage in GB.
  - -x, --ntx-price float: Set the price in NTX per minute for the onboarded compute resource.
  - -w, --wallet string: Set the wallet address.
/dms/node/onboarding/resource
- Description: Invokes the /dms/node/onboarding/resource behavior on an actor. This retrieves or manages resource information related to the onboarding process.
- Usage: nunet actor cmd /dms/node/onboarding/resource [<param> ...] [flags]
- Flags:
  - -h, --help: Display help information for the /dms/node/onboarding/resource command.
/dms/node/onboarding/status
- Description: Invokes the /dms/node/onboarding/status behavior on an actor. This is used to check the onboarding status of a node.
- Usage: nunet actor cmd /dms/node/onboarding/status [<param> ...] [flags]
- Flags:
  - -h, --help: Display help information for the /dms/node/onboarding/status command
/dms/node/peers/connect
- Description: Invokes the /dms/node/peers/connect behavior on an actor. This initiates a connection to a specified peer.
- Usage: nunet actor cmd /dms/node/peers/connect [<param> ...] [flags]
- Flags:
  - -a, --address string: The peer address to connect to
  - -h, --help: Display help information for the /dms/node/peers/connect command.
/dms/node/peers/dht
- Description: Invokes the /dms/node/peers/dht behavior on an actor. This interacts with the Distributed Hash Table (DHT) used for peer discovery and content routing
- Usage: nunet actor cmd /dms/node/peers/dht [<param> ...] [flags]
- Flags:
  - -h, --help: Display help information for the /dms/node/peers/dht command.
/dms/node/peers/list
- Description: Invokes the /dms/node/peers/list behavior on an actor. This retrieves a list of connected peers
- Usage: nunet actor cmd /dms/node/peers/list [<param> ...] [flags]
- Flags:
  - -h, --help: Display help information for the /dms/node/peers/list command.
/dms/node/peers/ping
- Description: Invokes the /dms/node/peers/ping behavior on an actor. This sends a ping message to a specified host to check its reachability
- Usage: nunet actor cmd /dms/node/peers/ping [<param> ...] [flags]
- Flags:
  - -h, --help: Display help information for the /dms/node/peers/ping command
  - -H, --host string: The host address to ping
/dms/node/peers/score
- Description: Invokes the /dms/node/peers/score behavior on an actor. This retrieves a snapshot of the peer's gossipsub broadcast score.
- Usage: nunet actor cmd /dms/node/peers/score [<param> ...] [flags]
- Flags:
  - -h, --help: Display help information for the /dms/node/peers/score command
/dms/node/peers/self
- Description: Invokes the /dms/node/peers/self behavior on an actor. This retrieves information about the node itself, such as its ID or addresses
- Usage: nunet actor cmd /dms/node/peers/self [<param> ...] [flags]
- Flags:
  - -h, --help: Display help information for the /dms/node/peers/self command.
/dms/node/vm/list
- Description: Invokes the /dms/node/vm/list behavior on an actor. This retrieves a list of virtual machines (VMs) running on the node
- Usage: nunet actor cmd /dms/node/vm/list [<param> ...] [flags]
- Flags:
  - -h, --help: Display help information for the /dms/node/vm/list command.
/dms/node/vm/start/custom
- Description: Invokes the /dms/node/vm/start/custom behavior on an actor. This starts a new VM with custom configurations.
- Usage: nunet actor cmd /dms/node/vm/start/custom [<param> ...] [flags]
- Flags:
  - -a, --args string: Arguments to pass to the kernel
  - -z, --cpu float32: CPU cores to allocate (default 1)
  - -h, --help: Display help information for the /dms/node/vm/start/custom command.
  - -i, --initrd string: Path to initial ram disk
  - -k, --kernel string: Path to kernel image file.
  - -m, --memory uint: Memory to allocate (default 1024)
  - -r, --rootfs string: Path to root fs image file
/dms/node/vm/stop
- Description: Invokes the /dms/node/vm/stop behavior on an actor. This stops a running VM
- Usage: nunet actor cmd /dms/node/vm/stop [<param> ...] [flags]
- Flags:
  - -h, --help: Display help information for the /dms/node/vm/stop command
  - -i, --id string: Execution id of the VM

Public Commands

/public/hello
- Description: Invokes the /public/hello behavior on an actor. This broadcasts a "hello" for a polite introduction.
- Usage: nunet actor cmd /public/hello [<param> ...] [flags]
- Flags:
  - -h, --help: Display help information for the /public/hello command
/public/status
- Description: Invokes the /public/status behavior on an actor. This retrieves the status or health information of the actor or system
- Flags:
  - -h, --help: Display help information for the /public/status command

Global Flags

These flags can be used with any of the above commands:

-c, --context string: Specifies the capability context name. This is used for authorization or access control.
-d, --dest string: Specifies the destination for the command. This can be a DMS DID (Decentralized Identifier), a peer ID, or a handle.
-e, --expiry time: Sets an expiration time for the message or command.
-t, --timeout duration: Sets a timeout duration for the command. If the command does not complete within the specified duration, it will time out.
-h, --help: Display help information for the commands

cap

Last updated: 2024-11-21 22:06:00.646189 File source: link on GitLab

Nunet Capability Management CLI Documentation

The Nunet Capability Management CLI provides commands to manage capabilities within the Nunet ecosystem. Capabilities define the permissions and authorizations granted to different entities within the system. These commands allow you to create, modify, delegate, and list capabilities.

Commands

nunet cap anchor: Add or modify capability anchors in a capability context.
nunet cap delegate: Delegate capabilities for a subject.
nunet cap grant: Grant (delegate) capabilities as anchors and side chains from a capability context.
nunet cap list: List all capability anchors in a capability context
nunet cap new: Create a new persistent capability context for DMS or personal usage
nunet cap remove: Remove capability anchors in a capability context.

`nunet cap anchor`

This command is used to add new or modify existing capability anchors within a specific capability context. Anchors serve as the foundation for defining capabilities, establishing the core permissions and restrictions

Usage

nunet cap anchor [flags]

Flags

-c, --context string: Specifies the operation context, defining the key and capability context to use (defaults to "user").
-h, --help: Displays help information for the anchor command
--provide string: Adds tokens as a "provide" anchor in JSON format, defining capabilities the context can offer
--require string: Adds tokens as a "require" anchor in JSON format, defining capabilities the context demands
--root string: Adds a DID as a "root" anchor, establishing the root authority for the context

`nunet cap delegate`

This command delegates specific capabilities to a subject, granting them permissions within the system

Usage

nunet cap delegate <subjectDID> [flags]

Arguments

<subjectDID>: The Decentralized Identifier (DID) of the entity receiving the delegated capabilities

Flags

-a, --audience string: (Optional) Specifies the audience DID, restricting the delegation to a specific recipient
--cap strings: Defines the capabilities to be granted or delegated (can be specified multiple times)
-c, --context string: Specifies the operation context (defaults to "user")
-d, --depth uint: (Optional) Sets the delegation depth, controlling how many times the capabilities can be further delegated (default 0)
--duration duration: Sets the duration for which the delegation is valid
-e, --expiry time: Sets an expiration time for the delegation
-h, --help: Displays help information for the delegate command
--self-sign string: Specifies self-signing options: 'no' (default), 'also', or 'only'
-t, --topic strings: Defines the topics for which capabilities are granted or delegated (can be specified multiple times)

`nunet cap grant`

This command grants (delegates) capabilities as anchors and side chains from a specified capability context

Usage

nunet cap grant <subjectDID> [flags]

Arguments

<subjectDID>: The Decentralized Identifier (DID) of the entity receiving the granted capabilities

Flags

-a, --audience string: (Optional) Specifies the audience DID
--cap strings: Defines the capabilities to be granted or delegated
-c, --context string: Specifies the operation context (defaults to "user")
-d, --depth uint: (Optional) Sets the delegation depth
--duration duration: Sets the duration for which the grant is valid
-e, --expiry time: Sets an expiration time for the grant
-h, --help: Displays help information for the grant command
-t, --topic strings: Defines the topics for which capabilities are granted

`nunet cap list`

This command lists all capability anchors within a specified capability context

Usage

nunet cap list [flags]

Flags

-c, --context string: Specifies the operation context (defaults to "user")
-h, --help: Displays help information for the list command

`nunet cap new`

This command creates a new persistent capability context, which can be used for DMS or personal purposes

Usage

nunet cap new <name> [flags]

Arguments

<name>: The name for the new capability context

Flags

-h, --help: Displays help information for the new command

`nunet cap remove`

This command removes capability anchors from a specified capability context

Usage

nunet cap remove [flags]

Flags

-c, --context string: Specifies the operation context (defaults to "user")
-h, --help: Displays help information for the remove command
--provide string: Removes tokens from the "provide" anchor in JSON format
--require string: Removes tokens from the "require" anchor in JSON format
--root string: Removes a DID from the "root" anchor

db

Last updated: 2024-11-21 22:06:00.916260 File source: link on GitLab

Specification

Description

This package defines the local database functionality for the Device Management Service (DMS). Currently two repository structures have been implemented:

gorm: which is a SQlite database implementation.
clover: which is a NoSQL or document oriented database implementation.

Structure and Organisation

Here is quick overview of the contents of this pacakge:

README: Current file which is aimed towards developers who wish to use and modify the package functionality.
db.go: This file defines the method which opens an SQlite database at a path set by the config parameter work_dir, applies migration and returns the db instance.
repositories: This folder contains the sub-packages of the db package.
specs: This folder contains the class diagram of the package.

Class Diagram

The class diagram for the db package is shown below.

Source file

db Class diagram

Rendered from source file

!$rootUrlGitlab = "https://gitlab.com/nunet/device-management-service/-/raw/main"
!$packageRelativePath = "/db"
!$packageUrlGitlab = $rootUrlGitlab + $packageRelativePath
 
!include $packageUrlGitlab/specs/class_diagram.puml

Package Specification

Refer to the README file defined in the repositories folder for specification of the package.

repositories

Last updated: 2024-11-21 22:06:01.188971 File source: link on GitLab

db

Description
Structure and Organisation
Class Diagram
Functionality
Data Types
Testing
Proposed Functionality/Requirements
References

Specification

Description

The db package contains the configuration and functionality of database used by the DMS

Structure and Organisation

Here is quick overview of the contents of this pacakge:

Files

README: Current file which is aimed towards developers who wish to use and modify the database functionality.
generic_repository: This file defines the interface defining the main methods for db pacakge. It is designed using generic types and can be adapted to specific data type as needed.
generic_entity_repository: This file contains the interface for those databases which will hold only a single record.
deployment: This file specifies a database interface having types.DeploymentRequestFlat data type.
elk_stats: This file specifies a database interface having types.RequestTracker data type.
errors: This file specifies the different types of errors.
firecracker: This file specifies a database interface having types.VirtualMachine data type.
machine: This file defines database interfaces of various data types.
utils: This file contains some utility functions with respect to database operations.
utils_test: This file contains unit tests for functions defined in utils.go file.

Subpackages

gorm: This folder contains SQlite database implementation using gorm.
clover: This folder contains CloverDB database implementation.

Class Diagram

The class diagram for the db package is shown below.

Source file

db Class diagram

Rendered from source file

!$rootUrlGitlab = "https://gitlab.com/nunet/device-management-service/-/raw/main"
!$packageRelativePath = "/db/repositories"
!$packageUrlGitlab = $rootUrlGitlab + $packageRelativePath
 
!include $packageUrlGitlab/specs/class_diagram.puml

Functionality

There are two types of interfaces defined to cover database operations:

GenericRepository
GenericEntityRepository

These interfaces are described below.

GenericRepository Interface

GenericRepository interface defines basic CRUD operations and standard querying methods. It is defined with generic data types. This allows it to be used for any data type.

interface definition: type GenericRepository[T ModelType] interface

The methods of GenericRepository are as follows:

Create

signature: Create(ctx context.Context, data T) -> (T, error)
input #1: Go context
input #2: Data to be added to the database. It should be of type used to initialize the repository
output (success): Data type used to initialize the repository
output (error): error message

Create function adds a new record to the database.

Get

signature: Get(ctx context.Context, id interface{}) -> (T, error)
input #1: Go context
input #2: Identifier of the record. Can be any data type
output (success): Data with the identifier provided. It is of type used to initialize the repository
output (error): error message

Get function retrieves a record from the database by its identifier.

Update

signature: Update(ctx context.Context, id interface{}, data T) -> (T, error)
input #1: Go context
input #2: Identifier of the record. Can be any data type
input #3: New data of type used to initialize the repository
output (success): Updated record of type used to initialize the repository
output (error): error message

Update function modifies an existing record in the database using its identifier.

Delete

signature: Delete(ctx context.Context, id interface{}) -> error
input #1: Go context
input #2: Identifier of the record. Can be any data type
output (success): None
output (error): error message

Delete function deletes an existing record in the database using its identifier.

Find

signature: Find(ctx context.Context, query Query[T]) -> (T, error)
input #1: Go context
input #2: Query of type db.query
output (success): Result of query having the data type used to initialize the repository
output (error): error message

Find function retrieves a single record from the database based on a query.

FindAll

signature: FindAll(ctx context.Context, query Query[T]) -> ([]T, error)
input #1: Go context
input #2: Query of type db.query
output (success): Lists of records based on query result. The data type of each record will be what was used to initialize the repository
output (error): error message

FindAll function retrieves multiple records from the database based on a query.

GetQuery

signature: GetQuery() -> Query[T]
input: None
output: Query of type db.query

GetQuery function returns an empty query instance for the repository's type.

GenericEntityRepository Interface

GenericEntityRepository defines basic CRUD operations for repositories handling a single record. It is defined with generic data types. This allows it to be used for any data type.

interface definition: type GenericEntityRepository[T ModelType] interface

The methods of GenericEntityRepository are as follows:

Save

signature: Save(ctx context.Context, data T) -> (T, error)
input #1: Go context
input #2: Data to be saved of type used to initialize the database
output (success): Updated record of type used to initialize the repository
output (error): error message

Save function adds or updates a single record in the repository

Get

signature: Get(ctx context.Context) -> (T, error)
input: Go context
output (success): Record of type used to initialize the repository
output (error): error message

Get function retrieves the single record from the database.

Clear

signature: Clear(ctx context.Context) -> error
input: Go context
output (success): None
output (error): error

Clear function removes the record and its history from the repository.

History

signature: History(ctx context.Context, qiery Query[T]) -> ([]T, error)
input #1: Go context
input #2: query of type db.query
output (success):List of records of repository's type
output (error): error

History function retrieves previous records from the repository which satisfy the query conditions.

GetQuery

signature: GetQuery() -> Query[T]
input: None
output: New query of type db.query

GetQuery function returns an empty query instance for the repository's type.

Data Types

db.Query: This contains parameters related to a query that is passed to the database.

type Query[T any] struct {
	Instance   T                // Instance is an optional object of type T used to build conditions from its fields.
	Conditions []db.QueryCondition // Conditions represent the conditions applied to the query.
	SortBy     string           // SortBy specifies the field by which the query results should be sorted.
	Limit      int              // Limit specifies the maximum number of results to return.
	Offset     int              // Offset specifies the number of results to skip before starting to return data.
}

db.QueryCondition: This contains parameters defining a query condition.

type QueryCondition struct {
	Field    string      // Field specifies the database or struct field to which the condition applies.
	Operator string      // Operator defines the comparison operator (e.g., "=", ">", "<").
	Value    interface{} // Value is the expected value for the given field.
}

GenericRepository has been initialised for the following data types:

types.DeploymentRequestFlat
types.VirtualMachine
types.PeerInfo
types.Machine
types.Services
types.ServiceResourceRequirements
types.Connection
types.ElasticToken

GenericEntityRepository has been initialised for the following data types:

types.FreeResources
types.AvailableResources
types.Libp2pInfo
types.MachineUUID

Testing

The unit tests for utility functions are defined in utils_test.go. Refer to *_test.go files for unit tests of various implementations covered in subpackages.

Proposed Functionality / Requirements

List of issues

All issues that are related to the implementation of db package can be found below. These include any proposals for modifications to the package or new functionality needed to cover the requirements of other packages.

db package implementation

References

clover

Last updated: 2024-11-21 22:06:01.675190 File source: link on GitLab

clover

Description
Structure and Organisation
Class Diagram
Functionality
Data Types
Testing
Proposed Functionality/Requirements
References

Specification

Description

This sub package contains CloverDB implementation of the database interfaces.

Structure and Organisation

Here is quick overview of the contents of this pacakge:

README: Current file which is aimed towards developers who wish to use and modify the database functionality.
generic_repository: This file implements the methods of GenericRepository interface.
generic_entity_repository: This file implements the methods of GenericEntityRepository interface.
deployment: This file contains implementation of DeploymentRequestFlat interface.
elk_stats: This file contains implementation of RequestTracker interface.
firecracker: This file contains implementation of VirtualMachine interface.
machine: This file contains implementation of interfaces defined in machine.go.
utils: This file contains utility functions with respect to clover implementation.

All files with *_test.go naming convention contain unit tests with respect to the specific implementation.

Class Diagram

The class diagram for the clover package is shown below.

Source file

clover Class diagram

Rendered from source file

!$rootUrlGitlab = "https://gitlab.com/nunet/device-management-service/-/raw/main"
!$packageRelativePath = "/db/repositories/clover"
!$packageUrlGitlab = $rootUrlGitlab + $packageRelativePath
 
!include $packageUrlGitlab/specs/class_diagram.puml

Functionality

GenericRepository

NewGenericRepository

signature: NewGenericRepository[T repositories.ModelType](db *clover.DB) -> repositories.GenericRepository[T]
input: clover Database object
output: Repository of type db.clover.GenericRepositoryclover

NewGenericRepository function creates a new instance of GenericRepositoryclover struct. It initializes and returns a repository with the provided clover database.

Interface Methods

See db package readme for methods of GenericRepository interface

query

signature: query(includeDeleted bool) -> *clover_q.Query
input: boolean value to choose whether to include deleted records
output: CloverDB query object

query function creates and returns a new CloverDB Query object. Input value of False will add a condition to exclude the deleted records.

queryWithID

signature: queryWithID(id interface{}, includeDeleted bool) -> *clover_q.Query
input #1: identifier
input #2: boolean value to choose whether to include deleted records
output: CloverDB query object

queryWithID function creates and returns a new CloverDB Query object. The provided inputs are added to query conditions. The identifier will be compared to primary key field value of the repository.

Providing includeDeleted as False will add a condition to exclude the deleted records.

GenericEntityRepository

NewGenericEntityRepository

signature: NewGenericEntityRepository[T repositories.ModelType](db *clover.DB) repositories.GenericEntityRepository[T]
input: clover Database object
output: Repository of type db.clover.GenericEntityRepositoryclover

NewGenericEntityRepository creates a new instance of GenericEntityRepositoryclover struct. It initializes and returns a repository with the provided clover database instance and name of the collection in the database.

Interface Methods

See db package readme for methods of GenericEntityRepository interface.

query

signature: query() -> *clover_q.Query
input: None
output: CloverDB query object

query function creates and returns a new CloverDB Query object.

Data Types

db.clover.GenericRepositoryClover: This is a generic repository implementation using clover as an ORM

type GenericRepositoryClover[T repositories.ModelType] struct {
	db         *clover.DB // db is the Clover database instance.
	collection string     // collection is the name of the collection in the database.
}

db.clover.GenericEntityRepositoryClover: This is a generic single entity repository implementation using clover as an ORM

type GenericEntityRepositoryClover[T repositories.ModelType] struct {
	db         *clover.DB // db is the Clover database instance.
	collection string     // collection is the name of the collection in the database.
}

For other data types refer to db package readme.

Testing

Refer to *_test.go files for unit tests of different functionalities.

Proposed Functionality / Requirements

List of issues

db package implementation

References

gorm

Last updated: 2024-11-21 22:06:01.931280 File source: link on GitLab

gorm

Description
Structure and Organisation
Class Diagram
Functionality
Data Types
Testing
Proposed Functionality/Requirements
References

Specification

Description

This sub package contains Gorm implementation of the database interfaces.

Structure and Organisation

Here is quick overview of the contents of this pacakge:

README: Current file which is aimed towards developers who wish to use and modify the database functionality.
generic_repository: This file implements the methods of GenericRepository interface.
generic_entity_repository: This file implements the methods of GenericEntityRepository interface.
deployment: This file contains implementation of DeploymentRequestFlat interface.
elk_stats: This file contains implementation of RequestTracker interface.
firecracker: This file contains implementation of VirtualMachine interface.
machine: This file contains implementation of interfaces defined in machine.go.
utils: This file contains utility functions with respect to Gorm implementation.

All files with *_test.go naming convention contain unit tests with respect to the specific implementation.

Class Diagram

The class diagram for the gorm package is shown below.

Source file

gorm Class diagram

Rendered from source file

!$rootUrlGitlab = "https://gitlab.com/nunet/device-management-service/-/raw/main"
!$packageRelativePath = "/db/repositories/gorm"
!$packageUrlGitlab = $rootUrlGitlab + $packageRelativePath
 
!include $packageUrlGitlab/specs/class_diagram.puml

Functionality

GenericRepository

NewGenericRepository

signature: NewGenericRepository[T repositories.ModelType](db *gorm.DB) -> repositories.GenericRepository[T]
input: Gorm Database object
output: Repository of type db.gorm.GenericRepositoryGORM

NewGenericRepository function creates a new instance of GenericRepositoryGORM struct. It initializes and returns a repository with the provided GORM database.

Interface Methods

See db package readme for methods of GenericRepository interface.

GenericEntityRepository

NewGenericEntityRepository

signature: NewGenericEntityRepository[T repositories.ModelType](db *gorm.DB) -> repositories.GenericEntityRepository[T]
input #1: Gorm Database object
output: Repository of type db.gorm.GenericEntityRepositoryGORM

NewGenericEntityRepository creates a new instance of GenericEntityRepositoryGORM struct. It initializes and returns a repository with the provided GORM database.

Interface Methods

See db package readme for methods of GenericEntityRepository interface.

Data Types

db.gorm.GenericRepositoryGORM: This is a generic repository implementation using GORM as an ORM.

type GenericRepositoryGORM[T repositories.ModelType] struct {
	db *gorm.DB
}

db.gorm.GenericEntityRepositoryGORM: This is a generic single entity repository implementation using GORM as an ORM

type GenericEntityRepositoryGORM[T repositories.ModelType] struct {
	db *gorm.DB // db is the GORM database instance.
}

For other data types refer to db package readme.

Testing

Refer to *_test.go files for unit tests of different functionalities.

Proposed Functionality / Requirements

List of issues

db package implementation

References

dms

Last updated: 2024-11-21 22:06:02.188516 File source:

dms

Specification

Description

This package is responsible for starting the whole application. It also contains various core functionality of DMS:

Onboarding compute provider devices
Job orchestration and management
Resource management
Actor implementation for each node

Structure and Organisation

Here is quick overview of the contents of this pacakge:

Subpackages

proposed: All files with *_test.go naming convention contain unit tests with respect to the specific implementation.

Class Diagram

The class diagram for the dms package is shown below.

Source file

Rendered from source file

Functionality

TBD

Data Types

TBD

Testing

proposed Refer to *_test.go files for unit tests of different functionalities.

Proposed Functionality / Requirements

List of issues

All issues that are related to the implementation of dms package can be found below. These include any proposals for modifications to the package or new functionality needed to cover the requirements of other packages.

Interfaces & Methods

proposed Capability_interface

add method will combine capabilities of two nodes. Example usage - When two jobs have to be run on a single machine, the capability requirements of each will need to be combined.

subtract method will subtract two capabilities. Example usage - When resources are locked for a job, the available capability of a machine will need to be reduced.

Data types

proposed dms.Capability

The Capability struct will capture all the relevant data that defines the capability of a node to perform the job. At the same time this will be used to define capability requirements that a job requires from a node.

An initial data model for Capability is defined below.

proposed dms.Connectivity

type Connectivity struct {

}

proposed dms.PriceInformation

proposed dms.TimeInformation

type TimeInformation struct { // Units holds the units of time ex - hours, days, weeks Units string

}

References

hardware

Last updated: 2024-11-21 22:06:02.447676 File source:

dms

Specification

Description

The hardware package is responsible for handling the hardware related functionalities of the DMS.

Structure and Organisation

Here is quick overview of the contents of this package:

Functionality

GetMachineResources()

signature: GetMachineResources() (types.MachineResources, error)
input: None
output: types.MachineResources
output(error): error

GetCPU()

signature: GetCPU() (types.CPU, error)
input: None
output: types.CPU
output(error): error

GetRAM()

signature: GetRAM() (types.RAM, error)
input: None
output: types.RAM
output(error): error

GetDisk()

signature: GetDisk() (types.Disk, error)
input: None
output: types.Disk
output(error): error

Data Types

Testing

The tests can be found in the *_test.go files in the respective packages.

References

jobs

Last updated: 2024-11-21 22:06:02.764635 File source:

jobs

Specification

Description

This pacakge manages local jobs and their allocation, including relation to execution environments, etc. It will manage jobs through whatever executor it's running (Vontainer, VM, Direct_exe, Java etc).

Structure and Organisation

Here is quick overview of the contents of this directory:

TBD

Class Diagram

The class diagram for the jobs package is shown below.

Source file

Rendered from source file

Functionality

TBD

Data Types

TBD

Testing

TBD

Proposed Functionality / Requirements

List of issues

Interfaces & Methods

proposed Job interface

getPods: will fetch list of pods currently running on the machine

proposed Pod interface

combineCapabilities: will combine the capability requirements of different jobs to calculate total capabality needed for a Pod.

proposed JobLink interface

validateRelations: It validates the JobLink properties provided.

proposed Allocation interface

start: starts the allocation execution

sendMessage: sends a message to another actor (Node/Allocation)

register: registers the allocation with the node that it is running on

Data types

proposed dms.jobs.Job: Nunet job which will be sent to the network wrapped as a BidRequest. If needed it will have child jobs to be executed. The relation between parent and child job needs to be specified.

proposed dms.jobs.JobLink: specifies the properties that relate parent and child job.

proposed dms.jobs.Pod: collection of jobs that need to be executed on the same machine.

proposed dms.jobs.Allocation: maps the job to the process on a executor. Each Allocation is an Actor.

proposed dms.jobs.AllocationID: identifier for Allocation objects.

References

node

Last updated: 2024-11-21 22:06:03.022073 File source:

node

Specification

`proposed` Description

This package is responsible for creation of a Node object which is the main actor residing on the machine as long as DMS is running. The Node gets created when the DMS is onboarded.

The Node is responsible for:

Communicating with other actors (nodes and allocations) via messages. This will include sending bid requests, bids, invocations, job status etc
Checking used and free resource before creating allocations
Continuous monitoring of the machine

Structure and Organisation

Here is quick overview of the contents of this pacakge:

Class Diagram

The class diagram for the node package is shown below.

Source file

Rendered from source file

Functionality

TBD

Data Types

TBD

Testing

proposed Refer to *_test.go files for unit tests of different functionalities.

Proposed Functionality / Requirements

List of issues

Interfaces & Methods

proposed Node_interface

getAllocation method retrieves an Allocation on the machine based on the provided AllocationID.

checkAllocationStatus method will retrieve status of an Allocation.

routeToAllocation method will route a message to the Allocation of the job that is running on the machine.

benchmarkCapability method will perform machine benchmarking

setRegisteredCapability method will record the benchmarked Capability of the machine into a persistent data store for retrieval and usage (mostly in job orchestration functionality)

getRegisteredCapability method will retrieve the benchmarked Capability of the machine from the persistent data store.

setAvailableCapability method changes the available capability of the machine when resources are locked

getAvailableCapability method will return currently available capability of the node

lockCapability method will lock certain amount of resources for a job. This can happen during bid submission. But it must happen once job is accepted and before invocation.

getLockedCapabilities method retrieves the locked capabilities of the machine.

setPreferences method sets the preferences of a node as dms.orchestrator.CapabilityComparator

getPreferences method retrieves the node preferences as dms.orchestrator.CapabilityComparator

getRegisteredBids method retrieves list of bids receieved for a job.

startAllocation method will create an allocation based on the invocation received.

Data types

proposed dms.node.Node

An initial data model for Node is defined below.

proposed dms.node.NodeID

References

onboarding

Last updated: 2024-11-21 22:06:03.286178 File source:

onboarding

Specification

Description

This file explains the onboarding functionality of Device Management Service (DMS). This functionality is catered towards compute providers who wish provide their hardware resources to Nunet for running computational tasks as well as developers who are contributing to platform development.

Structure and Organisation

Here is quick overview of the contents of this directory:

Class Diagram

The class diagram for the onboarding package is shown below.

Source file

Rendered from source file

Functionality

Onboard

signature: Onboard(ctx context.Context, config types.OnboardingConfig) error
input #1: Context object
input #2: types.OnboardingConfig
output (error): Error message

Onboard function executes the onboarding process for a compute provider based on the configuration provided.

Offboard

signature: Offboard(ctx context.Context, force bool) error
input #1: Context object
input #2: force parameter
output: None
output (error): Error message

Offboard removes the resources onboarded to Nunet. If the force parameter is True, then offboarding process will continue even in the presence of errors.

IsOnboarded

signature: IsOnboarded(ctx context.Context) (bool, error)
input #1: Context object
output #1: bool
output #2: error

IsOnboarded checks if the compute provider is onboarded.

Info

signature: Info(ctx context.Context) (types.OnboardingConfig, error)
input #1: Context object
output #1: types.OnboardingConfig
output #2: error

Info returns the configuration of the onboarding process.

Data Types

types.OnboardingConfig: Holds the configuration for onboarding a compute provider.

Testing

Proposed Functionality / Requirements

List of issues

References

orchestrator

Last updated: 2024-11-21 22:06:03.564896 File source:

orchestrator

Specification

Description

The orchestrator is responsible for job scheduling and management (manages jobs on other DMSs).

A key distinction to note is the option of two types of orchestration mechanisms: push and pull. Broadly speaking pull orchestration works on the premise that resource providers bid for jobs available in the network, while push orchestration works when a job is pushed directly to a known resource provider -- constituting to a more centralized orchestration. push orchestration develops on the idea that users choose from the available providers and their resources. However, given the decentralized and open nature of the platform, it may be required to engage the providers to get their current (latest) state and preferences. This leads to an overlap with the pull orchestration approach.

The default setting is to use pull based orchestration, which is developed in the present proposed specification.

proposed Job Orchestration

The proposed lifecyle of a job on Nunet platform consists of various operations from job posting to settlement of the contract. Below is a brief explanation of the steps involved in the job orchestration:

Job Posting: The user posts a job request to the DMS. The job request is validated and a Nunet job is created in the DMS.
Search and Match:
a. The Service provider DMS requests for bids from other nodes in the network.
b. DMS on compute provider compares the capability of the available resources against job requirements. If all the requirements are met, it then decides whether to submit a bid.
c. The received bids are assessed and the best bid is selected.
Job Request: In case the shortlisted compute provider has not locked the resources while submitting the bid, the job request workflow is executed. This requires the compute provider DMS to lock the necessary resources required for the job and re-submit the bid. Note that at this stage compute provider can still decline the job request.
Contract Closure: The service provider and the shortlisted compute provider verify that the counterparty is a verified entity and approved by Nunet Solutions to participate in the network. This in an important step to establish trust before any work is performed.
If job does not require any payment (Volunteer Compute), contract is generated by both Service Provider and Compute Provider DMS. This is then verified by Contract-Database. Otherwise, proof of contract needs to be received from the Contract-Database before start of work.
Invocation and Allocation: When the contract closure workflow is completed, both the service provider and compute provider DMS have an agreement and proof of contract with them. Then the service provider DMS will send an invocation to the compute provider DMS which results in job allocation being created. Allocation can be understood as an execution space / environment on actual hardware that enables a Job to be executed.
Job Execution: Once allocation is created, the job execution starts on the compute provider machine.
Contract Settlement: After job is completed, service provider DMS verifies the work done. If the work is correct, the Contract-Database makes the necessary transactions to settle the the contract.

Structure and Organisation

Here is quick overview of the contents of this directory:

Subpackages

Class Diagram

Source

Rendered from source file

Functionality

TBD

Data Types

TBD

Testing

TBD

Proposed Functionality / Requirements

List of issues

Interfaces & Methods

proposed Orchestrator interface

publishBidRequest: sends a request for bid to the network for a particular job. This will depend on the network package for propagation of the request to other nodes in the network.

compareCapability: compares two capabilities and returns a CapabilityComparison object. Expected usage is to compare capability required in a job with the available capability of a node.

acceptJob: looks at the comparison between capabilities and preferences of a node in the form of CapabilityComparator object and decides whether to accept a job or not.

sendBid: sends a bid to the node that propagated the BidRequest.

selectBestBid: looks at all the bids received and selects the best one.

sendJobRequest: sends a job request to the shortlisted node whose bid was selected. The compute provider node needs to accept the job request and lock its resources for the job. In case resources are already locked while submitting the bid, this step may be skipped.

sendInvocation: sends an invocation request (as a message) to the node that accepted the job. This message should have all the necessary information to start an Allocation for the job.

orchestrateJob: this will be called when a job is received via postJob endpoint. It will start the orchestration process. It is also possible that this method could be called via a timer for jobs scheduled in the future.

proposed Actor interface

sendMessage: sends a message to another actor (Node / Allocation).

processMessage: processes the message received and decides on what action to take.

proposed Mailbox interface

receiveMessage: receives a message from another Node and converts it into a telemetry.Message object.

handleMessage: processes the message received.

triggerBehavior: this is where actions taken by the actor based on the message received will be defined.

getKnownTopics: retrieves the gossip sub topics known to the node.

getSubscribedTopics: retrieves the gossip sub topics subscribed by the node.

subscribeToTopic: subscribes to a gossip sub topic.

unsubscribeFromTopic: un-subscribes from a gossip sub topic.

proposed Other methods

Methods for job request functionality a. check whether resources are locked b. lock resources c. accept job request
Methods for contract closure a. validate other node as a registered entity b. generate contract c. kyc validation
Methods for job exeuction a. handle job updates
Methods for contract settlement a. job verification

Note that the above methods not an exhaustive list. These are to be considered as suggestions. The developer implementing the orchestrator functionality is free to make modifications as necessary.

Data types

proposed dms.orchestrator.Actor: Actor has a identifier and a mailbox to send/receive messages.

proposed dms.orchestrator.Bid: Consists of information sent by the compute provider node to the requestor node as a bid for the job broadcasted to the network.

proposed dms.orchestrator.BidRequest: A bid request is a message sent by a node to the network to request for bids.

proposed dms.orchestrator.PriceBid: Contains price related information of the bid.

proposed dms.orchestrator.TimeBid: Contains time related information of the bid.

proposed dms.orchestrator.CapabilityComparator: Preferences of the node which has an influence on the comparison operation.

TBD

proposed dms.orchestrator.CapabilityComparison: Result of the comparison operation.

TBD

proposed dms.orchestrator.Invocation: An invocation is a message sent by the orchestrator to the node that accepted the job. It contains the job details and the contract.

proposed dms.orchestrator.Mailbox: A mailbox is a communication channel between two actors. It uses network package functionality to send and receive messages.

proposed Other data types

Data types related to allocation, contract settlement, job updates etc are currently omitted. These should be added as applicable while implementation.

References

Orchestration steps research blogs

The orchestrator functionality of DMS is being developed based on the research done in the following blogs:

graph

Last updated: 2024-11-21 22:06:03.828829 File source:

graph

This whole package is proposed status and therefore documentation is missing, save for the proposed functionality part.

Specification

Description

TBD

Structure and organisation

TBD

Class Diagram

Source

Rendered from source file

Functionality

TBD

Types

TBD

Testing

TBD

Proposed Functionality / Requirements

List of issues

All issues that are filed in GitLab related to the implementation of dms/orchestrator package can be found below. These include any proposals for modifications to the package or new functionality needed to cover the requirements of other packages.

Proposed functionalities

TBD

Data types

proposed LocalNetworkTopology more complex deployments may need a data structure, which considers local network topology of a node / dms -- i.e. for reasoning about speed of connection (as well as capabilities) between neighbors.

References

Related research blogs

TBD

release

Last updated: 2024-11-21 22:05:58.821093 File source: link on GitLab

Device Management Service (DMS)

Device Management Service (DMS)

About

Payment

Note: If you are a developer, please check out the DMS specifications and Building from Source sections of this document.

Installation

You can install Device Management Service (DMS) via binary releases or building it from source.

Binary releases

Note: If you intalled the binary from a release and you would like to act as compute provider, you may need to check permissions and features to enable some required and optional features.

Ubuntu/Debian

Download the latest .deb package from the package registry
Install the Debian package with apt or dpkg:

sudo apt update
sudo apt install ./nunet-dms_<latest>.deb -y

Some dependencies such as docker and libsystemd-dev might be missing so it's recommended to fix install by running:

sudo apt -f install

Building from source

We currently support Linux and MacOS (Darwin).

Dependencies

iproute2 (linux only)
build-essential (linux only)
libsystemd-dev (linux only)
go (v1.21.7 or later)

Clone the repository:

git clone https://gitlab.com/nunet/device-management-service.git

Build the CLI:

cd device-management-service
make

This will result in a binary file in builds/ folder named as dms_linux_amd64 or dms_darwin_arm64 depending on the platform.

Note: If you built from source and would like to act as a compute provider, you may need to check permissions and features to enable some required and optional features.

To cross compile to arm, cross compilers need to be installed. In particular arm-linux-gnueabihf and aarch64-linux-gnu. For debian systems, install with:

apt install gcc-arm-linux-gnueabihf gcc-aarch64-linux-gnu

You can add the compiled binary to a directory in your $PATH. See the Usage section for more information.

Permissions and features (for compute providers)

The following applies only for compute providers. If you're running a client/orchestrator, you do not need to set any permissions.

If you built DMS from source or installed the binary from one of our releases, you may need to set some permissions to the binary to enable some features.

Darwin users: unfortunately, the DMS can't work with granular permissions on Mac. So, for now, if running a compute provider, you will have to run the nunet daemon (nunet run) as root.

For Linux users, granular permissions will have to be set to the binary (optionally, but NOT recommended, you can run the binary as root).

Net-admin permission and IP over libp2p

Note: cap_net_admin is a required capability for compute providers.

One of the reasons for requiring this permission is because this feature depends on creating and managing tun interfaces.

To set the necessary capabilities, run the following command:

sudo setcap cap_net_admin+ep /usr/bin/nunet

The above command depends on: libcap2-bin (Debian/Ubuntu) or libcap (CentOS/RHEL/Arch...)

Firecracker (for compute providers)

Note: Linux only.

To act as a compute provider capable of receiving and running jobs with Firecracker, ensure your user is part of the kvm group. You can do this by running:

sudo usermod -aG kvm $USER

Installation on VMs

Skip doing an unattended installation for the new Ubuntu VM as it might not add the user with administrative privileges.
Enable Guest Additions when installing the VM (VirtualBox only).
Always change the default NAT network setting to Bridged before booting the VM.
Install Extension Pack if using VirtualBox (recommended).
Install VMware Tools if using VMware (recommended).
ML on GPU jobs on VMs are not supported.

Installation on WSL

Install WSL through the Windows Store.
Install the Update KB5020030 (Windows 10 only).
Install Ubuntu 20.04 through WSL.
Enable systemd on Ubuntu WSL.
ML Jobs deployed on Linux cannot be resumed on WSL.

Though it is possible to run ML jobs on Windows machines with WSL, using Ubuntu 20.04 natively is highly recommended to avoid unpredictability and performance losses.

System Requirements

CPU-only machines

Minimum System Requirements

We require you to specify CPU (MHz x no. of cores) and RAM, but your system must meet at least the following requirements before you decide to onboard it:

CPU: 2 GHz
RAM: 4 GB
Free Disk Space: 10 GB
Internet Download/Upload Speed: 4 Mbps / 0.5 MBps

If the above CPU has 4 cores, your available CPU would be around 8000 MHz. So if you want to onboard half your CPU and RAM on NuNet, you can specify 4000 MHz CPU and 2000 MB RAM.

Recommended System Requirements

CPU: 3.5 GHz
RAM: 8-16 GB
Free Disk Space: 20 GB
Internet Download/Upload Speed: 10 Mbps / 1.25 MBps

GPU Machines

Minimum System Requirements

CPU: 3 GHz
RAM: 8 GB
GPU: 4 GB VRAM (NVIDIA, AMD, or Intel discrete GPU with manually installed drivers)
Free Disk Space: 50 GB
Internet Download/Upload Speed: 50 Mbps

Recommended System Requirements

CPU: 4 GHz
RAM: 16-32 GB
GPU: 8-12 GB VRAM (NVIDIA, AMD, or Intel discrete GPU with manually installed drivers)
Free Disk Space: 100 GB
Internet Download/Upload Speed: 100 Mbps

GPU Driver Installation

NuNet DMS requires properly installed GPU drivers to function correctly. We do not automatically install drivers to ensure compatibility and flexibility across different user setups.

For AMD64 Platforms:

We recommend using the Ubuntu-based HiveOS for the easiest setup.

If you prefer to use a different operating system or need to install drivers manually, please follow these steps:

NVIDIA GPUs:

Visit the NVIDIA Official Driver Downloads page.
Select your GPU model and operating system.
Download and install the recommended driver.
Reboot your system after installation.

AMD GPUs:

Visit the AMD Drivers and Support for Processors and Graphics page.
Select your GPU model and operating system.
Download and install the recommended driver.
Reboot your system after installation.

Intel Discrete GPUs:

Visit the Intel® software for general purpose GPU capabilities documentation page.
Select your GPU model and operating system.
Download and install the recommended driver.
Reboot your system after installation.

For detailed instructions specific to your operating system, please refer to the documentation provided by NVIDIA, AMD, or Intel.

Note: Ensure that you have the correct permissions to install drivers on your system. On Linux systems, you may need to use sudo or log in as root to install drivers.

Usage

Quick Start

The NuNet CLI

Key Concepts

Actor: An independent entity in the Nunet system capable of performing actions and communicating with other actors.
Capability: Defines the permissions and restrictions granted to actors within the system.
Key: A cryptographic key pair used for authentication and authorization within the DMS.

You can find a detailed documentation here.

Creating identities

The first step is to generate identities/keys and capability contexts. It is recommended that two keys are setup: one for the user (default name user) and another for the dms (default name dms)

A capability context is created with the nunet cap new <context> command and it is anchored on a key with the context name.

To set up a new identity/create a new key, run the command:

$ nunet key new <identity>

Then, to initialize its capability context:

$ nunet cap new <identity>

In this example, we are going to set up two identities:

First for the user

$ nunet key new user

then for the dms instance.

$ nunet key new dms

We then setup the capability contexts for each identity:

First the user

$ nunet cap new user

then the dms instance.

$ nunet cap new dms

If you use a ledger wallet for your personal key, you can create the user context as follows:

Create a new key for the user

$ nunet key did ledger

Then create the capability context for the user

$ nunet cap new ledger:user

You can create as many identities as you want, specially if you want to manage multiple DMS instances.

Using a Ledger Wallet

You can get your Ledger wallet's DID with:

$ nunet key did ledger

Setting up Capabilities

NuNet's network communication is powered by the NuActor System, a zero-trust system that utilizes fine-grained capabilities, anchored on DIDs, following the UCAN model.

Once both identities are created, you'll need to set up capabilities. Specifically:

Create capability contexts for both the user and each of your DMS instances.
Add the user's DID as a root anchor for the DMS capability context. This ensures that the DMS instance fully trusts the user, granting complete control over the DMS (the root capability).
If you want your DMS to participate in the public NuNet testnet (and eventually the mainnet), you'll need to set up capability anchors for your DMS:
1. Create a capability anchor to allow your DMS to accept public behavior invocations from authorized users and DMSs in the NuNet ecosystem.
2. Add this token to your DMS as a require anchor.
3. Request a capability token from NuNet to invoke public behaviors on the network.
4. Add the token as a provide anchor in your personal capability context.
5. Delegate to your DMS the ability to make public invocations using your token.
6. Add the delegation token as a provide anchor in your DMS.

Add a root anchor for your DMS context

You can do this by invoking the dms cap anchor command:

$ nunet cap anchor --context dms --root <user-did>

Where <user-did> is the user did created above in Creating identities and can be obtained by:

$ nunet key did <user>

or if you are using a Ledger Wallet

$ nunet key did ledger

Setup your DMS for the public testnet

The NuNet DID

did:key:zzCHUybNYmK8QsttZwXqUX8aDLoBGHnMCakDX2RpsGwmXmYHEW

Create a capability anchor for public behaviors

Create the grant

$ nunet cap grant --context user --cap /public --cap /broadcast --topic /nunet --expiry 2024-12-31 <nunet-did>

or if you are using a Ledger Wallet

$ nunet cap grant --context ledger:user --cap /public --cap /broadcast --topic /nunet --expiry 2024-12-31 <nunet-did>

And the granted token as a require anchor

$ nunet cap anchor --context dms --require <the-grant-output>

The first command grants nunet authorized users the capability to invoke public behaviors until December 31, 2024, and outputs a token.

The second command consumes the token and adds the require anchor for your DMS

Ask NuNet for a public network capability token

Use the NuNet granted token to authorize public behavior invocations in the public network

3.1 Add the provide anchor to your personal context

$ nunet cap anchor --context user --provide <the-token-you-got-from-nunet>

or if you are using a Ledger Wallet

$ nunet cap anchor --context ledger:user --provide <the-token-you-got-from-nunet>

3.2 Delegate to your DMS

$ nunet cap delegate --context user --cap /public --cap /broadcast --topic /nunet --expiry 2024-12-31 <your-dms-did>

or if you are using a Ledger Wallet

$ nunet cap delegate --context ledger:user --cap /public --cap /broadcast --topic /nunet --expiry 2024-12-31 <your-dms-did>

3.3 Add the delegation token as a provide anchor in your DMS

$ nunet cap anchor --context dms --provide <the-delegate-output>

The first command ingests the NuNet provided token and the last two commands use this token to delegate the public behavior capabilities to your DMS.

Running DMS

If everything was setup properly, you should be able to run:

$ nunet run

Darwin users: If you plan to onboard your computer power to the network, You may need to run with sudo. See the optional features and permissions section for more information.

By default, DMS runs on port 9999.

Provide Compute Power to the Network

If you want to contribute your computer's resources (CPU, RAM, GPU, storage) to the network, you have to onboard your machine.

Follow our Compute Provider Guide to get started.

Deploy Jobs on the Network

Every node on the network can deploy workloads across available compute resources, given the necessary capabilities. Learn how deployments work by following our Deployments Guide.

REST Endpoints

Refer to the api package README for the list of all endpoints. Head over to project's issue section and create an issue with your question.

Configuration

Config file

The DMS searches for a configuration file dms_config.json in the following locations, in order of priority whenever it's started:

The current directory (.)
$HOME/.nunet
/etc/nunet

Run Two DMS Instances Side by Side

As a developer, you might find yourself needing to run two DMS instances, one acting as an SP (Service Provider) and the other as a CP (Compute Provider).

Step 1:

Clone the repository to two different directories. You might want to use descriptive directory names to avoid confusion.

Step 2:

The dms_config.json file can be used to modify these settings. Here is a sample config file that can be modified to your preference:

{
  "p2p": {
    "listen_address": ["/ip4/0.0.0.0/tcp/9100", "/ip4/0.0.0.0/udp/9100/quic-v1"]
  },
  "general": {
    "user_dir": "/home/user/.config/nunet/dms/",
    "debug": true
  },
  "rest": {
    "port": 10000
  }
}

Prefer to use absolute paths and have a look at the config structure for more info.

Tests

make testdata

go test --tags unit ./...

Help in contributing tests is always appreciated :)

Specification

Description

Detailed information about the NuNet platform, concepts, architecture, models, stakeholders can be found in these two papers:

dms: Responsible for starting the whole application and core DMS functionality such as onboarding, job orchestration, job and resource management, etc.
internal: Code that will not be imported by any other packages and is used only on the running instance of DMS. This includes all configuration-related code, background tasks, etc.
db: Database used by the DMS.
storage: Disk storage management on each DMS for data related to DMS and jobs deployed by DMS. It also acts as an adapter to external storage services.
api: All API functionality (including REST API, etc.) to interact with the DMS.
cmd: Command line functionality and tools.
network: All network-related code such as p2p communication, IP over Libp2p, and other networks that might be needed in the future.
executor: Responsible for executing the jobs received by the DMS. Interface to various executors such as Docker, Firecracker, etc.
telemetry: Logs, traces, and everything related to telemetry.
plugins: Defined entry points and specs for third-party plugins, registration, and execution of plugin code.
types: Contains data models imported by various packages.
utils: Utility tools and functionalities.
tokenomics: Interaction with blockchain for the crypto-micropayments layer of the platform.

Design and Architecture

Conceptual Basis

Main concepts of the architecture of DMS, the main component of the NuNet platform, can be found in the Yellow Paper.

Ontology

The Nunet Ontology, which forms the basis of the design, is explained in the articles below:

Architecture

Refer to the following items to understand DMS architecture at a high level.

Research

Relevant research work that has informed the design of DMS can be found below:

Functionality

DMS is currently being refactored and new functionality will be added.

Data Types

Refer to the DMS global class diagram in this section and various packages for data models.

References

In addition to the relevant links added in the sections above, you can also find useful links here: NuNet Links.

Class Diagram

The global class diagram for the DMS is shown below.

Source File

Global Class Diagram

types

Last updated: 2024-11-21 22:06:10.873536 File source: link on GitLab

types

Description
Structure and Organisation
Class Diagram
Functionality
Data Types
Testing
Proposed Functionality/Requirements
References

Specification

Description

types package defines and keeps data structures and interfaces that are used across the whole DMS component by different packages.

Structure and Organisation

Here is quick overview of the contents of this pacakge:

README: Current file which is aimed towards developers who wish to use and modify the package functionality.
capability: This file contains data structures to describe machine capability.
comparison: This file contains constans and types used for capability comparison.
constants: This file contains constants that are used across different packages.
deployment: This file contains data structure related to job deployment.
elk_stats: This file contains data structure to be sent to elasticsearch collector.
encryption: This file contains data structure related to encryption in DMS.
execution: This file contains data structure related to executor functionality.
firecracker: This file contains data structure related to firecracker.
machine: This file contains data structure related to the machine - resources, peer details, services etc.
network: This file contains data structure related to networking functionality of DMS
network_config: This file defines message types, network types (libp2p, NATS) with configurations, and libp2p specific configurations (DHT, keys, peers, scheduling etc).
onboarding: This file contains data structure related to compute provider onboarding.
resource: This file contains data structures of GPU and execution resources.
spec_config: This file defines a SpecConfig struct for configuration data with type, parameters, normalization, validation, and type checking functionalities.
storage: This file contains data structures related to storage.
telemetry: This file defines structs related to telemetry configuration and methods to load configuration from environment variables.
types: This file defines a base model for entities in the application with auto-generated UUIDs, timestamps, and soft delete functionality using GORM hooks.

Class Diagram

Source

types class diagram

Rendered from source file

!$rootUrlGitlab = "https://gitlab.com/nunet/device-management-service/-/raw/main"
!$packageRelativePath = "/types"
!$packageUrlGitlab = $rootUrlGitlab + $packageRelativePath
 
!include $packageUrlGitlab/specs/class_diagram.puml

Functionality

types package holds interfaces and methods that are used by multiple packages. The functionality of these interfaces/methods are typically implemented in other packages.

Here are some methods defined in types package:

NewExecutionResult

signature: NewExecutionResult(code int) *ExecutionResult
input: exit code
output: types.ExecutionResult

NewExecutionResult creates a new ExecutionResult object.

NewFailedExecutionResult

signature: NewFailedExecutionResult(err error) *ExecutionResult
input: error
output: types.ExecutionResult

NewFailedExecutionResult creates a new ExecutionResult object for a failed execution. It sets the error message from the provided error and sets the exit code to -1.

Config interface TBD

type Config interface {
	GetNetworkConfig() *SpecConfig
}

GetNetworkConfig will return the network configuration parameters.

NewSpecConfig

signature: NewSpecConfig(t string) *SpecConfig
input: type for the configuration object
output: types.SpecConfig

NewSpecConfig creates new SpecConfig with the given type and an empty params map.

WithParam

signature: (s *SpecConfig) WithParam(key string, value interface{}) *SpecConfig
input #1 : key
input #2 : value associated with the key
output: types.SpecConfig

WithParam adds a new key-value pair to the spec parameters and returns the updated SpecConfig object.

Normalize

signature: (s *SpecConfig) Normalize()
input: None
output: None

Normalize ensures that the spec config is in a valid state by trimming whitespace from the Type field and initializing the Params map if empty.

Validate

signature: (s *SpecConfig) Validate() error
input: None
output: error

Validate checks if the spec config is valid. It returns an error if the SpecConfig is nil or if the Type field is missing (blank). Otherwise, it returns no error indicating a valid configuration.

IsType

signature: (s *SpecConfig) IsType(t string) bool
input: type
output: bool

IsType checks if the SpecConfig matches the provided type, ignoring case sensitivity. It returns true if there's a match, otherwise false.

IsEmpty

signature: (s *SpecConfig) IsEmpty() bool
input: None
output: bool

IsEmpty checks if the SpecConfig is empty, meaning it's either nil or has an empty Type and no parameters. It returns true if empty, otherwise false.

GetID

signature: (m types.BaseDBModel) GetID() string
input: None
output: identifier of the entity

GetID returns the identifier of the entity.

BeforeCreate

signature: (m *Model) BeforeCreate(tx *gorm.DB) error
input: gorm database object to be created
output: bool

BeforeCreate sets the ID and CreatedAt fields before creating a new entity.

BeforeUpdate

signature: (m *Model) BeforeUpdate(tx *gorm.DB) error
input: gorm database object to be updated
output: bool

BeforeUpdate returns true if the spec config is empty.

LoadConfigFromEnv

signature: LoadConfigFromEnv() (*TelemetryConfig, error)
input: none
output: types.TelemetryConfig
output (error): error message

LoadConfigFromEnv loads the telemetry configuration from environment variables. This includes observabilitty level and collector configuration. It returns the final configuration as types.TelemetryConfig object.

parseObservabilityLevel

signature: parseObservabilityLevel(levelStr string) int
input: observability level
output: int value corresponding to the input observability level

parseObservabilityLevel returns the integer representation of the provided observability level string. When the input string does not match the defined observability levels, default observability level INFO is considered and its integer value is returned.

Data Types

Deployment

types.DeploymentRequest

type DeploymentRequest struct {
	RequesterWalletAddress string    `json:"address_user"` // service provider wallet address
	MaxNtx                 int       `json:"max_ntx"`
	Blockchain             string    `json:"blockchain"`
	TxHash                 string    `json:"tx_hash"`
	ServiceType            string    `json:"service_type"`
	Timestamp              time.Time `json:"timestamp"`
	MetadataHash           string    `json:"metadata_hash"`
	WithdrawHash           string    `json:"withdraw_hash"`
	RefundHash             string    `json:"refund_hash"`
	Distribute_50Hash      string    `json:"distribute_50_hash"`
	Distribute_75Hash      string    `json:"distribute_75_hash"`
	Params                 struct {
		ImageID   string `json:"image_id"`
		ModelURL  string `json:"model_url"`
		ResumeJob struct {
			Resume       bool   `json:"resume"`
			ProgressFile string `json:"progress_file"` // file path
		} `json:"resume_job"`
		Packages        []string `json:"packages"`
		RemoteNodeID    string   `json:"node_id"`          // NodeID of compute provider (machine to deploy the job on)
		RemotePublicKey string   `json:"public_key"`       // Public key of compute provider
		LocalNodeID     string   `json:"local_node_id"`    // NodeID of service provider (machine triggering the job)
		LocalPublicKey  string   `json:"local_public_key"` // Public key of service provider
		MachineType     string   `json:"machine_type"`
	} `json:"params"`
	Constraints struct {
		Complexity string `json:"complexity"`
		CPU        int    `json:"cpu"`
		RAM        int    `json:"ram"`
		Vram       int    `json:"vram"`
		Power      int    `json:"power"`
		Time       int    `json:"time"`
	} `json:"constraints"`
	TraceInfo struct {
		TraceID     string `json:"trace_id"`
		SpanID      string `json:"span_id"`
		TraceFlags  string `json:"trace_flags"`
		TraceStates string `json:"trace_state"`
	} `json:"traceinfo"`
}

types.DeploymentResponse

type DeploymentResponse struct {
	Success bool   `json:"success"`
	Content string `json:"content"`
}

types.DeploymentUpdate

type DeploymentUpdate struct {
	MsgType string `json:"msg_type"`
	Msg     string `json:"msg"`
}

types.DeploymentRequestFlat

type DeploymentRequestFlat struct {
	types.BaseDBModel
	DeploymentRequest string `json:"deployment_request"`
	// represents job status from services table; goal is to keep then in sync (both tables are on different DMSes).
	JobStatus string `json:"job_status"`
}

types.BlockchainTxStatus

type BlockchainTxStatus struct {
	TransactionType   string `json:"transaction_type"` // No need of this param maybe be deprecated in future
	TransactionStatus string `json:"transaction_status"`
	TxHash            string `json:"tx_hash"`
}

ELK

types.NewDeviceOnboarded

// NewDeviceOnboarded defines the schema of the data to be sent to stats db when a new device gets onboarded
type NewDeviceOnboarded struct {
	PeerID        string
	CPU           float32
	RAM           float32
	Network       float32
	DedicatedTime float32
	Timestamp     float32
}

types.DeviceStatusChange

// DeviceStatusChange defines the schema of the data to be sent to stats db when a device status gets changed
type DeviceStatusChange struct {
	PeerID    string
	Status    string
	Timestamp float32
}

types.DeviceResourceChange

// DeviceResourceChange defines the schema of the data to be sent to stats db when a device resource gets changed
type DeviceResourceChange struct {
	PeerID                   string
	ChangedAttributeAndValue struct {
		CPU           float32
		RAM           float32
		Network       float32
		DedicatedTime float32
	}
	Timestamp float32
}

types.DeviceResourceConfig

// DeviceResourceConfig defines the schema of the data to be sent to stats db when a device resource config gets changed
type DeviceResourceConfig struct {
	PeerID                   string
	ChangedAttributeAndValue struct {
		CPU           float32
		RAM           float32
		Network       float32
		DedicatedTime float32
	}
	Timestamp float32
}

types.NewService

// NewService defines the schema of the data to be sent to stats db when a new service gets registered in the platform type NewService struct { ServiceID string ServiceName string ServiceDescription string Timestamp float32 }

types.ServiceCall

// ServiceCall defines the schema of the data to be sent to stats db when a host machine accepts a deployement request
type ServiceCall struct {
	CallID              float32
	PeerIDOfServiceHost string
	ServiceID           string
	CPUUsed             float32
	MaxRAM              float32
	MemoryUsed          float32
	NetworkBwUsed       float32
	TimeTaken           float32
	Status              string
	AmountOfNtx         int32
	Timestamp           float32
}

types.ServiceStatus

// ServiceStatus defines the schema of update the status of service to stats db of the job being executed on host machine
type ServiceStatus struct {
	CallID              float32
	PeerIDOfServiceHost string
	ServiceID           string
	Status              string
	Timestamp           float32
}

types.ServiceRemove

// ServiceRemove defines the schema of the data to be sent to stats db when a new service gets removed from the platform
type ServiceRemove struct {
	ServiceID string
	Timestamp float32
}

types.NtxPayment

// NtxPayment defines the schema of the data to be sent to stats db when a payment is made to device for the completion of service.
type NtxPayment struct {
	CallID            float32
	ServiceID         string
	AmountOfNtx       int32
	PeerID            string
	SuccessFailStatus string
	Timestamp         float32
}

Executor

types.Executor: This defines the type of executor that is used to execute the job.

type Executor struct {
	ExecutorType ExecutorType `json:"executor_type"`
}

type ExecutorType string

types.ExecutionRequest: This is the input that executor receives to initiate a job execution.

type ExecutionRequest struct {
	// ID of the job to execute
	JobID string

	// ID of the execution
	ExecutionID string

	// Engine spec for the execution
	EngineSpec types.SpecConfig

	// Resources for the execution
	Resources types.Resources

	// Input volumes for the execution
	Inputs []storage.StorageVolume

	// Output volumes for the results
	Outputs []storage.StorageVolume

	// Directory to store the results
	ResultsDir string
}

types.ExecutionResult: This contains the result of the job execution.

type ExecutionResult struct {
	// STDOUT of the execution
	STDOUT string `json:"stdout"`

	// STDERR of the execution
	STDERR string `json:"stderr"`

	// Exit code of the execution
	ExitCode int `json:"exit_code"`

	// Error message if the execution failed
	ErrorMsg string `json:"error_msg"`
}

types.SpecConfig TBD: This allows arbitrary configuration/parameters as needed during implementation of specific executor.

// SpecConfig represents a configuration for a spec
// A SpecConfig can be used to define an engine spec, a storage volume, etc.
type SpecConfig struct {
	// Type of the spec (e.g. docker, firecracker, storage, etc.)
	Type string `json:"type"`

	// Params of the spec
	// This allows passing arbitrary parameters to the spec implementation
	Params map[string]interface{} `json:"params,omitempty"`
}

types.LogStreamRequest: This is the input provided when a request to stream logs of an execution is made.

type LogStreamRequest struct {
	JobID       string // ID of the job
	ExecutionID string // ID of the execution
	Tail        bool   // Tail the logs
	Follow      bool   // Follow the logs
}

Firecracker

types.BootSource: This contains configuration parameters for booting a Firecracker VM.

type BootSource struct {
	KernelImagePath string `json:"kernel_image_path"`
	BootArgs        string `json:"boot_args"`
}

types.Drives: This contains properties of a virtual drive for Firecracker VM.

type Drives struct {
	DriveID      string `json:"drive_id"`
	PathOnHost   string `json:"path_on_host"`
	IsRootDevice bool   `json:"is_root_device"`
	IsReadOnly   bool   `json:"is_read_only"`
}

types.MachineConfig: This defines the configuration parameters of the machine to be used while creating a new Firecracker VM.

type MachineConfig struct {
	VCPUCount  int `json:"vcpu_count"`
	MemSizeMib int `json:"mem_size_mib"`
}

types.NetworkInterfaces: This defines the network configuration parameters.

type NetworkInterfaces struct {
	IfaceID     string `json:"iface_id"`
	GuestMac    string `json:"guest_mac"`
	HostDevName string `json:"host_dev_name"`
}

types.MMDSConfig: This contains a list of the network configuration parameters defined by NetworkInterfaces struct.

type MMDSConfig struct {
	NetworkInterface []string `json:"network_interfaces"`
}

types.MMDSMsg TBD: This contains the latest metadata of the machine.

type MMDSMsg struct {
	Latest struct {
		Metadata struct {
			types.MMDSMetadata
		} `json:"meta-data"`
	} `json:"latest"`
}

types.MMDSMetadata TBD: This contains the metadata of the machine.

type MMDSMetadata struct {
	NodeId string `json:"node_id"`
	PKey   string `json:"pkey"`
}

types.Actions TBD: This contains the type of action to be performed on the Firecracker VM.

type Actions struct {
	ActionType string `json:"action_type"`
}

types.VirtualMachine: This contains the configuration parameters of Firecracker virtual machine.

type VirtualMachine struct {
	types.BaseDBModel
	SocketFile string `json:"socket_file"`
	BootSource string `json:"boot_source"`
	Filesystem string `json:"filesystem"`
	VCPUCount  int    `json:"vcpu_count"`
	MemSizeMib int    `json:"mem_size_mib"`
	TapDevice  string `json:"tap_device"`
	State      string `json:"state"`
}

Machine

types.IP

type IP []any

types.PeerInfo TBD: This contains parameters of the peer node.

type PeerInfo struct {
	types.BaseDBModel
	NodeID    string `json:"nodeID,omitempty"`
	Key       string `json:"key,omitempty"`
	Mid       string `json:"mid,omitempty"`
	PublicKey string `json:"public_key,omitempty"`
	Address   string `json:"_address,omitempty"`
}

types.Machine: This contains the configuration parameters of the machine.

type Machine struct {
	types.BaseDBModel
	NodeId               string
	PeerInfo             int
	IpAddr               string
	AvailableResources   int
	FreeResources        int
	TokenomicsAddress    string
	TokenomicsBlockchain string
}

types.FreeResources: This contains the resources currently available for a job.

// FreeResources are the resources free to be used by new services,
// plugins and any other processes started by DMS. It's basically
// the subtraction between AvailableResources and the amount of resources
// already used by DMS and its processes (mostly services)
type FreeResources struct {
	BaseDBModel
	TotCpuHz          int     `json:"tot_cpu_hz"`
	PriceCpu          float64 `json:"price_cpu"`
	Ram               int     `json:"ram"`
	PriceRam          float64 `json:"price_ram"`
	Vcpu              int     `json:"vcpu"`
	Disk              float64 `json:"disk"`
	PriceDisk         float64 `json:"price_disk"`
	NTXPricePerMinute float64 `json:"ntx_price"`
}

types.AvailableResources: This contains the resources onboarded to Nunet by the user.

// AvailableResources are the amount of resources onboarded which
// can be used by NuNet
type AvailableResources struct {
	types.BaseDBModel
	TotCpuHz          int
	CpuNo             int
	CpuHz             float64
	PriceCpu          float64
	Ram               int
	PriceRam          float64
	Vcpu              int
	Disk              float64
	PriceDisk         float64
	NTXPricePerMinute float64
}

types.Services TBD: This contains the details of the services running on the machine.

type Services struct {
	types.BaseDBModel
	TxHash               string
	TransactionType      string // transaction type can be running, done, withdraw, refund and distribute
	JobStatus            string // whether job is running or exited; one of these 'running', 'finished without errors', 'finished with errors'
	JobDuration          int64  // job duration in minutes
	EstimatedJobDuration int64  // job duration in minutes
	ServiceName          string
	ContainerID          string
	ResourceRequirements int
	ImageID              string
	LogURL               string
	LastLogFetch         time.Time
	ServiceProviderAddr  string
	ComputeProviderAddr  string
	MetadataHash         string
	WithdrawHash         string
	RefundHash           string // saving hashes for call the `/request-reward` endpoint by SPD
	Distribute_50Hash    string
	Distribute_75Hash    string
	SignatureDatum       string
	MessageHashDatum     string
	Datum                string
	SignatureAction      string // saving signatures for removing redundancy of calling Oracle
	MessageHashAction    string
	Action               string
	// TODO: Add ContainerType field

}

types.ServiceResourceRequirements: This contains the resource requirements for a service.

type ServiceResourceRequirements struct {
	types.BaseDBModel
	CPU  int
	RAM  int
	VCPU int
	HDD  int
}

types.ContainerImages: This contains parameters of a container image.

type ContainerImages struct {
	gorm.Model
	ImageID   string
	ImageName string
	Digest    string
}

types.Libp2pInfo: This contains parameters of Libp2p node.

type Libp2pInfo struct {
	BaseDBModel
	PrivateKey []byte `json:"private_key"`
	PublicKey  []byte `json:"public_key"`
	ServerMode bool   `json:"server_mode"`
	Available  bool   `json:"available"`
}

types.MachineUUID: This defines the unique identifier for the machine.

type MachineUUID struct {
	BaseDBModel
	UUID string `json:"uuid"`
}

types.Gpu: This contains the GPU parameters of the machine.

type Gpu struct {
	Name     string `json:"name"`
	TotVram  uint64 `json:"tot_vram"`
	FreeVram uint64 `json:"free_vram"`
}

types.resources: This defines the resource parameters of the machine.

type resources struct {
	TotCpuHz  float64
	PriceCpu  float64
	Ram       int
	PriceRam  float64
	Vcpu      int
	Disk      float64
	PriceDisk float64
}

types.PeerData: This contains the details of the peer node.

type PeerData struct {
	PeerID               string        `json:"peer_id"`
	IsAvailable          bool          `json:"is_available"`
	HasGpu               bool          `json:"has_gpu"`
	GpuInfo              []Gpu         `json:"gpu_info"`
	TokenomicsAddress    string        `json:"tokenomics_addrs"`
	TokenomicsBlockchain string        `json:"tokenomics_blockchain"`
	AvailableResources   FreeResources `json:"available_resources"`
	Services             []Services    `json:"services"`
	Timestamp            int64         `json:"timestamp,omitempty"`
}

types.Connection: TBD

type Connection struct {
	types.BaseDBModel
	PeerID     string `json:"peer_id"`
	Multiaddrs string `json:"multiaddrs"`
}

types.PingResult: The contains the details of the ping result.

type PingResult struct {
	RTT     time.Duration
	Success bool
	Error   error
}

types.Machines: TBD

type Machines map[string]PeerData

types.KadDHTMachineUpdate: This contains machine info for KAD-DHT.

// machine info for KAD-DHT
type KadDHTMachineUpdate struct {
	Data      []byte `json:"data"`
	Signature []byte `json:"signature"`
}

types.ElasticToken: TBD

type ElasticToken struct {
	types.BaseDBModel
	NodeId      string
	Token       string
	ChannelName string
}

Onboarding

types.BlockchainAddressPrivKey

// BlockchainAddressPrivKey holds Ethereum/Cardano wallet address and private key from which the
// address is derived.
type BlockchainAddressPrivKey struct {
	Address    string `json:"address,omitempty"`
	PrivateKey string `json:"private_key,omitempty"`
	Mnemonic   string `json:"mnemonic,omitempty"`
}

types.CapacityForNunet

// CapacityForNunet is a struct required in request body for the onboarding
type CapacityForNunet struct {
	Memory            int64   `json:"memory,omitempty"`
	CPU               int64   `json:"cpu,omitempty"`
	NTXPricePerMinute float64 `json:"ntx_price,omitempty"`
	Channel           string  `json:"channel,omitempty"`
	PaymentAddress    string  `json:"payment_addr,omitempty"`
	ServerMode        bool    `json:"server_mode,omitempty,"`
	IsAvailable       bool    `json:"is_available"`
}

types.Provisioned

// Provisioned struct holds data about how much total resource
// host machine is equipped with
type Provisioned struct {
	CPU      float64 `json:"cpu,omitempty"`
	Memory   uint64  `json:"memory,omitempty"`
	NumCores uint64  `json:"total_cores,omitempty"`
}

types.Metadata

// Metadata - machine metadata of onboarding parameters
type Metadata struct {
	Name            string `json:"name,omitempty"`
	UpdateTimestamp int64  `json:"update_timestamp,omitempty"`
	Resource        struct {
		MemoryMax int64 `json:"memory_max,omitempty"`
		TotalCore int64 `json:"total_core,omitempty"`
		CPUMax    int64 `json:"cpu_max,omitempty"`
	} `json:"resource,omitempty"`
	Available struct {
		CPU    int64 `json:"cpu,omitempty"`
		Memory int64 `json:"memory,omitempty"`
	} `json:"available,omitempty"`
	Reserved struct {
		CPU    int64 `json:"cpu,omitempty"`
		Memory int64 `json:"memory,omitempty"`
	} `json:"reserved,omitempty"`
	Network           string  `json:"network,omitempty"`
	PublicKey         string  `json:"public_key,omitempty"`
	NodeID            string  `json:"node_id,omitempty"`
	GpuInfo           []Gpu   `json:"gpu_info,omitempty"`
	Dashboard         string  `json:"dashboard,omitempty"`
	NTXPricePerMinute float64 `json:"ntx_price,omitempty"`
}

types.OnboardingStatus

type OnboardingStatus struct {
	Onboarded    bool   `json:"onboarded"`
	Error        error  `json:"error"`
	MachineUUID  string `json:"machine_uuid"`
	MetadataPath string `json:"metadata_path"`
	DatabasePath string `json:"database_path"`
}

types.LogBinAuth: This stores the authorisation token for LogBin.

type LogBinAuth struct {
	types.BaseDBModel
	PeerID      string `json:"peer_id"`
	MachineUUID string `json:"machine_uuid"`
	Token       string `json:"token"`
}

Resource

types.Resources: resources defined for the machine.

type Resources struct {
    CPU      float64
    NumCores uint64
    GPU      []types.GPU `gorm:"foreignKey:ResourceID"`
    RAM      uint64
    Disk     uint64
}

types.AvailableResources: resources onboarded to Nunet.

type AvailableResources struct {
    types.BaseDBModel
    Resources
}

types.FreeResources: resources currently available for new jobs.

type FreeResources struct {
    types.BaseDBModel
    Resources
}

types.RequiredResources: resources required by the jobs running on the machine.

type RequiredResources struct {
    types.BaseDBModel
    Resources
}

types.GPUVendor: GPU vendors available on the machine.

type GPUVendor string

const (
	GPUVendorNvidia  GPUVendor = "NVIDIA"
	GPUVendorAMDATI  GPUVendor = "AMD/ATI"
	GPUVendorIntel   GPUVendor = "Intel"
	GPUVendorUnknown GPUVendor = "Unknown"
	None             GPUVendor = "None"
)

types.GPU: GPU details.

type GPU struct {
	// Index is the self-reported index of the device in the system
	Index int
	// Vendor is the maker of the GPU, e.g. NVidia, AMD, Intel
	Vendor types.GPUVendor
	// PCIAddress is the PCI address of the device, in the format AAAA:BB:CC.C
	// Used to discover the correct device rendering cards
	PCIAddress string
	// Model represents the GPU model name, e.g., "Tesla T4", "A100"
	Model string `json:"model" description:"GPU model, e.g., Tesla T4, A100"`
	// TotalVRAM is the total amount of VRAM on the device
	TotalVRAM uint64
	// UsedVRAM is the amount of VRAM currently in use
	UsedVRAM uint64
	// FreeVRAM is the amount of VRAM currently free
	FreeVRAM uint64

	// Gorm fields
	ResourceID uint `gorm:"foreignKey:ID"`
}

types.GPUList: A slice of GPU.

type GPUList []types.GPU

types.CPUInfo: CPU information of the machine.

type CPUInfo struct {
    NumCores   uint64
    MHzPerCore float64
    Compute    float64
}

types.SpecInfo: detailed specifications of the machine.

type SpecInfo struct {
	CPUs    []types.CPU
	GPUs    []types.GPU
	RAMs    []types.RAM
	Disks   []types.Disk
	Network NetworkInfo
}

types.CPU: CPU details.

type CPU struct {
	// Model represents the CPU model, e.g., "Intel Core i7-9700K", "AMD Ryzen 9 5900X"
	Model string

	// Vendor represents the CPU manufacturer, e.g., "Intel", "AMD"
	Vendor string

	// ClockSpeedHz represents the CPU clock speed in Hz
	ClockSpeedHz uint64

	// Cores represents the number of physical CPU cores
	Cores int

	// Threads represents the number of logical CPU threads (including hyperthreading)
	Threads int

	// Architecture represents the CPU architecture, e.g., "x86", "x86_64", "arm64"
	Architecture string

	// Cache size in bytes
	CacheSize uint64
}

types.RAM: RAM details.

type RAM struct {
	// Size in bytes
	Size uint64

	// Clock speed in Hz
	ClockSpeedHz uint64

	// Type represents the RAM type, e.g., "DDR4", "DDR5", "LPDDR4"
	Type string
}

types.Disk: Disk details.

type Disk struct {
	// Model represents the disk model, e.g., "Samsung 970 EVO Plus", "Western Digital Blue SN550"
	Model string

	// Vendor represents the disk manufacturer, e.g., "Samsung", "Western Digital"
	Vendor string

	// Size in bytes
	Size uint64

	// Type represents the disk type, e.g., "SSD", "HDD", "NVMe"
	Type string

	// Interface represents the disk interface, e.g., "SATA", "PCIe", "M.2"
	Interface string

	// Read speed in bytes per second
	ReadSpeed uint64
	// Write speed in bytes per second
	WriteSpeed uint64
}

types.NetworkInfo: Network details.

type NetworkInfo struct {
	// Bandwidth in bits per second (b/s)
	Bandwidth uint64

	// NetworkType represents the network type, e.g., "Ethernet", "Wi-Fi", "Cellular"
	NetworkType string
}

types.Resource: resources resources required to execute a task

Spec_config

types.SpecConfig: This allows arbitrary configuration to be defined as needed.

// SpecConfig represents a configuration for a spec
// A SpecConfig can be used to define an engine spec, a storage volume, etc.
type SpecConfig struct {
	// Type of the spec (e.g. docker, firecracker, storage, etc.)
	Type string `json:"type"`
	// Params of the spec
	Params map[string]interface{} `json:"params,omitempty"`
}

Storage

types.StorageVolume: This contains the parameters related to the storage volume that is created by the DMS on the local machine.

// StorageVolume represents a prepared storage volume that can be mounted to an execution
type StorageVolume struct {
	// Type of the volume (e.g. bind)
	Type string `json:"type"`
	// Source path of the volume on the host
	Source string `json:"source"`
	// Target path of the volume in the execution
	Target string `json:"target"`
	// ReadOnly flag to mount the volume as read-only
	ReadOnly bool `json:"readonly"`
}

Telemetry Config

types.CollectorConfig: This contains the parameters for a collector.

type CollectorConfig struct {
	CollectorType     string
	CollectorEndpoint string
}

types.TelemetryConfig: This defines the telemetry parameters such as obervability level, collector configurations etc.

type TelemetryConfig struct {
	ServiceName        string
	GlobalEndpoint     string
	ObservabilityLevel int
	CollectorConfigs   map[string]CollectorConfig
}

ObservabilityLevel is an enum that defines the level of observability. Currently logging is done at these observability levels.

    TRACE ObservabilityLevel = 1
	DEBUG ObservabilityLevel = 2
	INFO  ObservabilityLevel = 3
	WARN  ObservabilityLevel = 4
	ERROR ObservabilityLevel = 5
	FATAL ObservabilityLevel = 6

Types

types.BaseDBModel

// BaseDBModel is a base model for all entities. It'll be mainly used for database
// records.
type BaseDBModel struct {
	ID        string `gorm:"type:uuid"`
	CreatedAt time.Time
	UpdatedAt time.Time
	DeletedAt gorm.DeletedAt `gorm:"index"`
}

Testing

Test are defined in other packages where functionality is implemented.

Proposed Functionality / Requirements

List of issues

All issues that are related to the implementation of types package can be found below. These include any proposals for modifications to the package or new data structures needed to cover the requirements of other packages.

types package implementation TBD

proposed Encryption interfaces

These are placeholder interface definitions which will be developed in the future.

Encryptor must support encryption of files/directories

type Encryptor interface {
	Encrypt([]byte) ([]byte, error)
}

Decryptor must support decryption of files/directories

type Decryptor interface {
	Decrypt([]byte) ([]byte, error)
}

proposed Network types and methods

This section contains the proposed data types and methods related to network functionality.

types.NetworkSpec

// NetworkSpec is a stub. Please expand based on requirements.
type NetworkSpec struct {
}

types.NetConfig

// NetConfig is a stub. Please expand it or completely change it based on requirements.
type NetConfig struct {
	NetworkSpec SpecConfig `json:"network_spec"` // Network specification
}

types.NetConfig struct will implement a GetNetworkConfig method which returns network configuration parameters.

func (nc *NetConfig) GetNetworkConfig() *SpecConfig {
	return &nc.NetworkSpec
}

types.NetworkStats

// NetworkStats should contain all network info the user is interested in.
// for now there's only peerID and listening address but reachability, local and remote addr etc...
// can be added when necessary.
type NetworkStats struct {
	ID         string `json:"id"`
	ListenAddr string `json:"listen_addr"`
}

types.MessageInfo

// MessageInfo is a stub. Please expand it or completely change it based on requirements.
type MessageInfo struct {
	Info string `json:"info"` // Message information
}

proposed Network configuration data type

types.MessageEnvelope

type MessageType string

type MessageEnvelope struct {
	Type MessageType
	Data []byte
}

types.NetworkConfig

type NetworkType string

type NetworkConfig struct {
	Type NetworkType

	// libp2p
	types.Libp2pConfig

	// nats
	NATSUrl string
}

types.Libp2pConfig

// Libp2pConfig holds the libp2p configuration type Libp2pConfig struct { DHTPrefix string //crypto is Go-libp2p package that implements various cryptographic utilities PrivateKey crypto.PrivKey BootstrapPeers []multiaddr.Multiaddr Rendezvous string Server bool Scheduler *bt.Scheduler CustomNamespace string ListenAddress []string PeerCountDiscoveryLimit int PrivateNetwork types.PrivateNetworkConfig GracePeriodMs int GossipMaxMessageSize int }

types.PrivateNetworkConfig

type PrivateNetworkConfig struct {
	// WithSwarmKey if true, DMS will try to fetch the key from
	// `<config_path>/swarm.key`.
	WithSwarmKey bool

	// ACL defines the access control list for the private network.
	ACL []multiaddr.Multiaddr
}

release

Device Management Service (DMS)

Table of Contents

About

Payment

Installation

Binary releases

Building from source

Permissions and features (for compute providers)

Installation on VMs

Installation on WSL

System Requirements

GPU Driver Installation

Usage

Quick Start

Provide Compute Power to the Network

Deploy Jobs on the Network

REST Endpoints

Configuration

Config file

Tests

Specification

Description

Design and Architecture

Functionality

Data Types

References

Class Diagram

actor

NuActor: Secure Actor Programming in Decentralized Systems

Secure Interactions in Decentralized Systems

Capabilities

Behaviors and the Capability Namespace

Capability Tokens

Anchors of Trust

Token Chain Verification

Programming NuActors

The actor package

The Security Context

Actor Handles

Messages

The Actor Interfaces

Sending Messages

Interactive Invocations

Broadcast

Discarding Capability Tokens

Behind the Scenes

Sending a Message

Invoking a Behavior

Verifying Capabilities

api

api

Specification

Description

Structure and Organisation

Functionality

cmd

Introduction

Top-Level Commands

Actor System

Capability Management

Configuration Management

Usage

Available Commands

Flags

Key Management

Main Command

Subcommands

Run Command

TAP Command

nunet tap

GPU Command

nunet gpu

actor

Nunet Actor System CLI

Basic Commands

nunet actor msg

nunet actor send

nunet actor invoke

nunet actor broadcast

The `actor` package

`nunet tap`

`nunet gpu`

`nunet actor msg`

`nunet actor send`

`nunet actor invoke`

`nunet actor broadcast`

`nunet actor cmd`

`nunet cap anchor`

`nunet cap delegate`

`nunet cap grant`

`nunet cap list`

`nunet cap new`

`nunet cap remove`