main
Last updated: 2024-09-27 10:47:18.717023 File source: link on GitLab
Device Management Service (DMS)
Table of Contents
About
Device Management Service or DMS enables a machine to join the decentralized NuNet network both as a compute provider, offering its resources to the network, or to leverage the compute power of other machines in the network for processing tasks. Eventually users with available hardware resources will get compensated whenever their machine is utilized for a computational job by other users in the network. The ultimate goal of the platorm is to create a decentralized compute economy that is able to sustain itself.
Payment
All transactions on the Nunet network are expected to be conducted using the platform's utility token NTX. However, DMS is currently in development, and payment isn't part of v0.5.0-boot
release. NTX payments are expected to be implemented in the Public Alpha Mainnet milestone within later release cycles.
Note: If you are a developer, please check out the DMS specifications and Building from Source sections of this document.
Installation
You can install Device Management Service (DMS) via binary releases or building it from source.
Binary releases
You can find all binary releases here and other builds in-between releases in the package registry. We currently support ARM and AMD64 architectures.
Ubuntu/Debian
Download the latest .deb package from the package registry
Install the Debian package with
apt
ordpkg
:
Some dependencies such as
docker
andlibsystemd-dev
might be missing so it's recommended to fix install by running:
Building from source
We currently support Linux and MacOS (Darwin).
Dependencies
iproute2 (linux only)
build-essential (linux only)s
libsystemd-dev (linux only)
go (v1.21.7 or later)
Clone the repository:
Build the CLI:
This will result in a binary file in builds/ folder named as dms_linux_amd64
or dms_darwin_arm64
depending on the platform.
To cross compile to arm, cross compilers need to be installed. In particular arm-linux-gnueabihf and aarch64-linux-gnu. For debian systems, install with:
You can add the compiled binary to a directory in your $PATH
. See the Usage section for more information.
Installation on VMs
Skip doing an unattended installation for the new Ubuntu VM as it might not add the user with administrative privileges.
Enable Guest Additions when installing the VM (VirtualBox only).
Always change the default NAT network setting to Bridged before booting the VM.
Install Extension Pack if using VirtualBox (recommended).
Install VMware Tools if using VMware (recommended).
ML on GPU jobs on VMs are not supported.
Installation on WSL
Install WSL through the Windows Store.
Install the Update KB5020030 (Windows 10 only).
Install Ubuntu 20.04 through WSL.
Enable systemd on Ubuntu WSL.
ML Jobs deployed on Linux cannot be resumed on WSL.
Though it is possible to run ML jobs on Windows machines with WSL, using Ubuntu 20.04 natively is highly recommended to avoid unpredictability and performance losses.
If you are using a dual-boot machine, make sure to use the wsl --shutdown
command before shutting down Windows and running Linux for ML jobs. Also, ensure your Windows machine is not in a hibernated state when you reboot into Linux.
System Requirements
CPU-only machines
Minimum System Requirements
We require you to specify CPU (MHz x no. of cores) and RAM, but your system must meet at least the following requirements before you decide to onboard it:
CPU: 2 GHz
RAM: 4 GB
Free Disk Space: 10 GB
Internet Download/Upload Speed: 4 Mbps / 0.5 MBps
If the above CPU has 4 cores, your available CPU would be around 8000 MHz. So if you want to onboard half your CPU and RAM on NuNet, you can specify 4000 MHz CPU and 2000 MB RAM.
Recommended System Requirements
CPU: 3.5 GHz
RAM: 8-16 GB
Free Disk Space: 20 GB
Internet Download/Upload Speed: 10 Mbps / 1.25 MBps
GPU Machines
Minimum System Requirements
CPU: 3 GHz
RAM: 8 GB
NVIDIA GPU: 4 GB VRAM
Free Disk Space: 50 GB
Internet Download/Upload Speed: 50 Mbps
Recommended System Requirements
CPU: 4 GHz
RAM: 16-32 GB
NVIDIA GPU: 8-12 GB VRAM
Free Disk Space: 100 GB
Internet Download/Upload Speed: 100 Mbps
Usage
Quick Start
This quick start guide will walk you through the process of setting up a Device Management Service (DMS) instance for the first time and getting it running. We'll cover creating identities, setting up capabilities, and running the DMS.
The NuNet CLI
The Nunet CLI is the command-line interface for interacting with the Nunet Device Management Service (DMS). It provides commands for managing keys, capabilities, configuration, running the DMS, and more. It's essential for setting up and administering your DMS instance.
Key Concepts
Actor: An independent entity in the Nunet system capable of performing actions and communicating with other actors.
Capability: Defines the permissions and restrictions granted to actors within the system.
Key: A cryptographic key pair used for authentication and authorization within the DMS.
You can find a detailed documentation here.
Creating identities
The first step is to generate identities/keys and capability contexts. It is recommended that two keys are setup: one for the user (default name user
) and another for the dms (default name dms
)
A capability context is created with the dms cap new <context>
command and it is anchored on a key with the context name.
To set up a new identity/create a new key, run the command:
Then, to initialize its capability context:
In this example, we are going to set up two identities:
First for the user
then for the dms instance.
We then setup the capability contexts for each identity:
First the user
then the dms instance.
If you use a ledger wallet for your personal key, you can create the user context as follows:
Create a new key for the user
Then create the capability context for the user
You can create as many identities as you want, specially if you want to manage multiple DMS instances.
The key new
command returns a DID key for the specified identity. Remember to secure your keys and capability contexts, as they control access to your NuNet resources. They are encrypted and stored under $HOME/.nunet
by default.
Each time a new identity is generated it will prompt the user for a passphrase. The passphrase is associated with the created identity, thus a different passphrase can be set up for each identity. If you prefer, it's possible to set a DMS_PASSPHRASE
environment variable to avoid the command prompt.
Using a Ledger Wallet
It is also possible to use a Ledger Wallet instead of creating a new key; this is recommended for user contexts, but you should not use it for the dms context as it needs the key to sign capability tokens.
To set up a user context with a Ledger Wallet, you need the ledger-cli
script from NuNet's ledger wallet tool. The tool uses the Eth application and specifically the first Eth account with signing of personal messages. Everything that needs to be signed (namely capability tokens) will be presented on your Nano's screen in plaintext so that you can inspect it.
You can get your Ledger wallet's DID with:
Setting up Capabilities
NuNet's network communication is powered by the NuActor System, a zero-trust system that utilizes fine-grained capabilities, anchored on DIDs, following the UCAN model.
Once both identities are created, you'll need to set up capabilities. Specifically:
Create capability contexts for both the user and each of your DMS instances.
Add the user's DID as a root anchor for the DMS capability context. This ensures that the DMS instance fully trusts the user, granting complete control over the DMS (the root capability).
If you want your DMS to participate in the public NuNet testnet (and eventually the mainnet), you'll need to set up capability anchors for your DMS:
Create a capability anchor to allow your DMS to accept public behavior invocations from authorized users and DMSs in the NuNet ecosystem.
Add this token to your DMS as a require anchor.
Request a capability token from NuNet to invoke public behaviors on the network.
Add the token as a provide anchor in your personal capability context.
Delegate to your DMS the ability to make public invocations using your token.
Add the delegation token as a provide anchor in your DMS.
Add a root anchor for your DMS context
You can do this by invoking the dms cap anchor
command:
Where <user-did>
is the user did created above in Creating identities and can be obtained by:
or if you are using a Ledger Wallet
Setup your DMS for the public testnet
The NuNet DID
Create a capability anchor for public behaviors
Create the grant
or if you are using a Ledger Wallet
And the granted token as a require anchor
The first command grants nunet authorized users the capability to invoke public behaviors until December 31, 2024, and outputs a token.
The second command consumes the token and adds the require anchor for your DMS
Ask NuNet for a public network capability token
To request tokens for participating in the testnet, please go to did.nunet.io and submit the did you generated along with your gitlab username and an email address to receive the token. It's highly recommended that you use a Ledger hardware wallet for your keys.
Use the NuNet granted token to authorize public behavior invocations in the public network
3.1 Add the provide anchor to your personal context
or if you are using a Ledger Wallet
3.2 Delegate to your DMS
or if you are using a Ledger Wallet
3.3 Add the delegation token as a require anchor in your DMS
The first command ingests the NuNet provided token and the last two commands use this token to delegate the public behavior capabilities to your DMS.
Running DMS
If everything was setup properly, you should be able to run:
By default, DMS runs on port 9999.
Onboarding
You don't necessarily need to onboard for development, but that depends on which part you're working on. To onboard during development, /etc/nunet
needs to be manually created since it is created with the package during installation.
Refer to dms/onboarding
package README for details of onboarding functionality for compute provider users.
REST Endpoints
Refer to the api
package README for the list of all endpoints. Head over to project's issue section and create an issue with your question.
Configuration
Config file
The DMS searches for a configuration file dms_config.json
in the following locations, in order of priority whenever it's started:
The current directory (
.
)$HOME/.nunet
/etc/nunet
The configuration file must be in JSON format and it does not support comments. It's recommended that only the parameters that need to be changed are included in the config file so that other parameters can retain their default values.
It's possible to manage configuration using the config
subcommand as well. nunet config set
allows setting each parameter individually and nunet config edit
will open the config file in the default editor from $EDITOR
Run Two DMS Instances Side by Side
As a developer, you might find yourself needing to run two DMS instances, one acting as an SP (Service Provider) and the other as a CP (Compute Provider).
Step 1:
Clone the repository to two different directories. You might want to use descriptive directory names to avoid confusion.
Step 2:
You need to modify some configurations so that both DMS instances do not end up trying to listen on the same port and use the same path for storage. For example, ports on p2p.listen_address
, rest.port
, general.user_dir
etc... neeed to be different for two instances on the same host.
The dms_config.json
file can be used to modify these settings. Here is a sample config file that can be modified to your preference:
Prefer to use absolute paths and have a look at the config structure for more info.
Tests
Some packages contain tests, and it is always best to run them to ensure there are no broken tests before submitting any changes. Before running the tests, the Firecracker executor requires some test data, such as a kernel file, which can be downloaded with:
After the download is complete, all unit tests can be run with the following command. It's necessary to include the unit
tag due to the existence of files that contain functional and integration tests.
Help in contributing tests is always appreciated :)
Specification
Description
NuNet is a computing platform that provides globally distributed and optimized computing power and storage for decentralized networks, by connecting data owners and computing resources with computational processes in demand of these resources. NuNet provides a layer of intelligent interoperability between computational processes and physical computing infrastructures in an ecosystem which intelligently harnesses latent computing resources of the community into the global network of computations.
Detailed information about the NuNet platform, concepts, architecture, models, stakeholders can be found in these two papers:
DMS (Device Management Service) acts as the foundation of the NuNet platform, orchestrating the complex interactions between various computing resources and users. DMS implementation is structured into packages, creating a more maintainable, scalable, and robust codebase that is easier to understand, test, and collaborate on. Here are the existing packages in DMS and their purposes:
dms
: Responsible for starting the whole application and core DMS functionality such as onboarding, job orchestration, job and resource management, etc.internal
: Code that will not be imported by any other packages and is used only on the running instance of DMS. This includes all configuration-related code, background tasks, etc.db
: Database used by the DMS.storage
: Disk storage management on each DMS for data related to DMS and jobs deployed by DMS. It also acts as an adapter to external storage services.api
: All API functionality (including REST API, etc.) to interact with the DMS.cmd
: Command line functionality and tools.network
: All network-related code such as p2p communication, IP over Libp2p, and other networks that might be needed in the future.executor
: Responsible for executing the jobs received by the DMS. Interface to various executors such as Docker, Firecracker, etc.telemetry
: Logs, traces, and everything related to telemetry.plugins
: Defined entry points and specs for third-party plugins, registration, and execution of plugin code.types
: Contains data models imported by various packages.utils
: Utility tools and functionalities.tokenomics
: Interaction with blockchain for the crypto-micropayments layer of the platform.
Design and Architecture
Conceptual Basis
Main concepts of the architecture of DMS, the main component of the NuNet platform, can be found in the Yellow Paper.
Ontology
The Nunet Ontology, which forms the basis of the design, is explained in the articles below:
Architecture
Refer to the following items to understand DMS architecture at a high level.
Research
Relevant research work that has informed the design of DMS can be found below:
Functionality
DMS is currently being refactored and new functionality will be added.
Data Types
Refer to the DMS global class diagram in this section and various packages for data models.
References
In addition to the relevant links added in the sections above, you can also find useful links here: NuNet Links.
Class Diagram
The global class diagram for the DMS is shown below.
Source File
Last updated