- Milestone: Device Management Service Version 0.5.x
- Release management
- Feature scope
- Infrastructure
- Documentation
- Implementation
- Actor model with Object/Security Capabilities
- Object/Security Capabilities (UCAN)
- Dynamic method dispatch/invocation
- Local access (API and CMD)
- Local database interface and implementation
- Executor interface and implementation
- Machine benchmarking
- p2p network and routing
- Storage interface
- IP over Libp2p
- Observability and Telemetry
- Definition of compute workflows/recursive jobs
- Job deployment and orchestration model
- Hardware capability model
- Supervision model
- Tokenomics interface
Milestone: Device Management Service Version 0.5.x
Context / links:
- Project management portal page for the milestone
- Technical dependencies / work packages graph
- Milestone goals / description
- Tracking issue
- DMS 0.5.0 - feature scope GitLab board
- DMS 0.5.0 - release process GitLab board
Release management
The goal of the process is to expose all developed functionalities with frozen scope, as described above, to a rigorous internal and community testing via succession of release candidate builds.
Branching strategy
In preparation for this release process we will be changing the branching strategy as discussed and agreed in this issue and tracked by this issue. In summary:
- We will operate on two branches:
main(for trunk development) andrelease(for minor / final releases and patches). - The public dms binaries will be built from
releasebranch tags in the form ofv{version_number}-{suffix}, e.g.v0.5.0-rc1, etc. and released via gitlab release page as usual. - Interim builds between scheduled release candidates will be tagged with suffix
-p{patch_number}, e.g.v0.5.0-rc1-p1, etc.
Release process
The release process of Device Management Service Version 0.5.x is scheduled to start September 15, 2024 and finish December 15, 2024. It involves the following steps in chronological order:
- Feature scope freeze
- Specification and documentation system launch
- Project management portal launch
- Test management system launch (feature environment, CI/CD pipeline and QA visibility)
- Release candidates testing and delivering full feature scope
The currently scheduled release candidates are:
v0.5.0-boot-- bootstrap release;v0.5.0-rc1-- first release candidate;v0.5.0-rc2-- second release candidate;v0.5.0-rc3-- third release candidate;v0.5.0-rc4-- fourth release candidate;v0.5.0-- final release
The internal latest updated document with release process dates and countdown is available to view here. Feature scope management is done via DMS 0.5 pubic issue board.
Feature scope freeze
We first define the final scope of the dms features that we are releasing within this milestone, which is done within this section of the document. The scope of each feature should defined by functional test cases / scenarios associated with each feature and linked in Acceptance tests row of each table. All required test cases and scenarios pertaining to the feature scope freeze shall be written by the time of the last last release candidate release. The work is tracked by the work package dms/tests integrated into the test management system. Until then we will be continuing to implement all the features as they are explained in this document.
Specification and documentation portal launch
Documentation portal was operating internally since end of 2023, but was not fully aligned with the specification and documentation process that involves the whole team and including updates of documentation into acceptance criteria of each pull request. Documentation portal shall be launched in August. The work involving finishing and launching documentation portal are tracked by this issue.
Project management portal launch
Project management portal has been operating internally since Q4 2023, but was not exposed publicly as of yet. It will be launched publicly by the end of August 2024. The work involved in finishing and launching the project management portal for the dms v.0.5 milestone is tracked by this issue
Test management system launch
NuNet test management system consists of three main components, which have been developed for months now and are close to completion, but need to be finally coordinated and aligned in the preparation for the start of the DMS v0.5 release process and release candidate testing internally as well as via the community contributions. The components of the test management system are:
- CI/CD pipeline that publishes and makes available for further publishing all testing artifacts and reports; These are further used by developers, QA engineers and shall be exposed to community developers publicly;
- Feature environment features a small network of geographically distributed virtual machines connected via public internet, and allows for the execution of selected CI/CD pipeline stages automatically on heterogenous hardware environments -- testing functionality of the fully built DMS; Most of the acceptance tests of the frozen feature scope will be running via the
- QA management portal is a web portal (Testmo) which exposes all test artifacts via single interface and provides all information for NuNet QA team to see which test are passing / failing for every built of the DMS.
All three components are tightly coupled and will be internally released in quick succession, the last week of August - first week of September, targeting finalizing the test management system launch at the end of first week of September, so that the quality of release candidates can be fully validated by QA team.
Both feature environment and CI/CD process are instrumental to reach release level quality of the developed features. Both have been in development for months prior to current moment and are close to completion. Both are moving targets as they will develop together with the platform. Nevertheless, we aim to launch them as described above in the first half of August.
Release testing start
Given that all prerequisites are launched and prepared, we aim at starting the release process September 15, 2024 by releasing the first release candidate and exposing it to testing. We will release at least 3/4 release candidates and then final release, but the process will be fluid as explained here and most probably fill feature more minor release candidate releases per the adopted branching strategy.
Release
After testing the frozen feature scope of the release, we aim at releasing the 0.5.0 version of device-management-service during the second half of December 2024. For current / updated schedule and details, see release process kick-start presenatation and constantly updated countdown document.
Feature scope
Infrastructure
Feature environment
| Feature name | Bootstrap feature environment composed of geographically distributed machines connected via public internet |
| Work package | deliverables tech dependencies |
| Code reference | https://gitlab.com/nunet/test-suite/-/tree/develop/environments/feature?ref_type=heads |
| Description / definition of done | 1) Selected stages of CI/CD pipeline are automatically executed on the fully functional feature environment consisting of geographically dispersed virtual machines 2) Developers are able to test functionalities requiring more than one machine running code; 3) testing artifacts are collected and made accessible from centralized place (GitLab, Testmo, etc) for all tested scenarios / configurations and machines. 4) Interface / process for configure and manage feature environment configurations and reason about complex scenarios involving multiple machines |
| Timing | 1) Capability exposed to development team as soon as possible along in the preparation of release process (could comply to definition of done partially); 2) Full definition of done met and all impacted funcionalities considered by the September 1, 2024 at the latest |
| Status | On track |
| Team | Lead: @gabriel.chamon; Supporting: @sam.lake, @abhishek.shukla3143438 |
| Strategic alignment | 1) Enables advanced testing and quality assurance on limited hardware; 2) Enables scaling the testing environment from internally onwed to community developers machines, and eventually into testnet; 3) Builds grounds for release management now and in the future (including release candidates, testnet configurations involving third party hardware, canary releases, etc) |
| Who it benefits | 1) NuNet development team ensuring next level quality of CI/CD process and released software 2) Community testers and early compute providers 3) Solution providers who are already considering using NuNet for their business processes and will start testing in release candidate phase 4) Eventually all platform users will benefit from the quality of software |
| User challenge | 1) Developers want to test software as early as possible and access testing data / different hardware / software configurations in real network setting; 2) Community compute providers want to contribute to the development by providing machines for testnet |
| Value score | n/a |
| Design | n/a |
| Impacted functionality | This does not affect any feature (except possibly ability to launch testnet in the future) but rather deals with quality assurance of the whole platform, therefore indirectly but fundamentally affects quality of all features now and in the future. |
| Acceptance tests | Developers are able to spawn and tear down testing networks on demand for testing custom platform builds and complex functionalities involving more than one machine; CI/CD pipeline incorporates complex functional and integration tests which run on automatically spawned and teared down small (but geographically distributed) real networks of virtual machines |
CI/CD pipeline
| Feature name | Set the CI/CD pipeline able to run full test suite on all environments as defined in text matrix |
| Work package | deliverables tech dependencies |
| Code reference | https://gitlab.com/nunet/test-suite/-/tree/develop/cicd?ref_type=heads |
| Description / definition of done | 1) CI/CD pipeline is structured as per general test management and test matrix based-approaches and executes the minimal configuration for this milestone; 2) The CI/CD process explained and documented so that it could be evolved further via developers contributions 3) The minimal configuration is: 3.1) unit tests run per each defined test matrix trigger / VCS event; 3.2) functional tests (the ones that require code build, but no network) are executed per each defined test matrix trigger / VCS event 4) Each stage of CI/CD pipeline collects and publishes full test artifacts for 4.1) developers for easy access via GitLab flow; 4.2) code reviewers for easy access via GitLab pull requests; 4.3) QA team via Testmo interface; 5) QA requirements and availabilities provided by CI/CD pipeline are integrated into issue Acceptance Criteria |
| Timing | 1, 2, 3 implemented within first half of August 2024; 4 implemented within August; 5 implemented as soon as all is finished, but not later than first week of September; |
| Status | On track |
| Team | Lead: @gabriel.chamon; Supporting: @abhishek.shukla3143438, @ssarioglunn |
| Strategic alignment | 1) CI/CD pipeline integrates to the development process by enabling automatic testing of code against formalized functional requirements / scope of each release / milestone; 2) CI/CD pipeline process integrates into the test management process (of integrating new tests into the pipeline and ensuring that all testing domain is covered with each build) 3) Proper CI/CD pipeline and its visibility is instrumental for reaching our business goals of delivering quality software that is evolvable |
| Who it benefits | 1) NuNet development team ensuring next level quality of CI/CD process and released software 2) Internal QA team having the interface to easily access all built errors / issues and update requirements as needed 3) Eventually all platform users will benefit from the quality of software |
| User challenge | device-management-service code with the same build will run in diverse scenarios and diverse machine / os configurations while providing robust functionality; CI/CD pipeline process (together with test management process) will need to ensure that a) all QA requirements for implemented functionality scope are properly formalized and placed in correct test stages for exposing each build to them b) updated and/or newly defined functionality scope (during release process), including tests included as a result of community bug reports, are properly integrated into QA scope abd the CI/CD pipeline |
| Value score | n/a |
| Design | n/a |
| Impacted functionality | This does not affect any feature (except possibly ability to launch testnet in the future) but rather deals with quality assurance of the whole platform, therefore indirectly but fundamentally affects quality of all features now and in the future. |
| Acceptance tests | n/a |
Test management
| Feature name | Set up test management system, visibility tools and integration with CI/CD and Security pipelines |
| Work package | deliverables tech dependencies |
| Code reference | https://gitlab.com/nunet/test-suite |
| Description / definition of done | As detailed in work package deliverables: https://gitlab.com/nunet/architecture/-/issues/207 |
| Status | With challenges |
| Team | Lead: @abhishek.shukla3143438; Supporting: @gabriel.chamon, @ssarioglunn |
| Strategic alignment | 1) Test management system will enable 1.1) Define quality requirements and functionalities formally 1.2) Release software that precisely corresponds to the defined quality / functional requirements; 2) Test management system will enable for the functional and quality requirements to evolve together with the code; 3) Test management system implementation will enable for the community developers and testers to be part of the development process and greatly enhance it; 4) Participation of community in test management process will ideally be related to contributors program |
| Who it benefits | 1) NuNet development team ensuring next level quality of the released software 2) Internal QA team having the interface to easily access all built errors / issues and update requirements as needed 3) Eventually all platform users will benefit from the quality of software |
| User challenge | 1) Test management process should be flexible to evolve together with the code and constantly get updated with test vectors / scenarios; These can come both from core team and community developers / testers; Our test management system is fairly complex, involving different stages and frameworks (go for unit tests; Gherkin for other stages) as well as multiple machine environments, simulating real network behavior. In order to reach our goals, careful coordination (fitting together) of all aspects is needed. 2) Special challenge is inclusion of manual tests into the same framework of visibility, but enabling QA team to coordinate manual testing schedules and inclusion of their results into the test artifacts; 3) Community developers and testers should be included into the testing process both by public access to the test suite definitions as well as manual testing process, which is related to contributors programme; |
| Value score | n/a |
| Design | as described https://gitlab.com/nunet/test-suite |
| Impacted functionality | This does not affect any feature (except possibly ability to launch testnet in the future) but rather deals with quality assurance of the whole platform, therefore indirectly but fundamentally affects quality of all features now and in the future. |
| Acceptance tests | n/a |
Documentation
Specification and documentation system
| Feature name | Set up specification and documentation system for all nunet repos |
| Work package | tech dependencies |
| Code reference | documentation portal important issue related to README.md MR converting marmaid to plantuml, containing description of the process |
| Description / definition of done | 1) The documentation process is set up and described, including 1.1) requirement to have README.md files in each package and subpackage, 1.2) general structure and required contents of README.md files; 2) All packages and subpackages contain class diagram written in plantuml and their hierarchy is correctly constructed; 3) Documentation system allows to include package / subpackage specific descriptions or extensions via links in README.md and possibly using additionally files (also in ./specs directory) as needed; 4) Updating specification and documentation of respective packages is included into acceptance criteria of each issue and checked during merge request reviews; 5) Documentation system includes process of requesting and proposing new functionalities and architecture; 6) Regular builds of documentation portal which is based on README.md files |
| Status | Close to completion |
| Team | Lead: @0xPravar; Supporting: @kabir.kbr, @janaina.senna |
| Strategic alignment | As explained in tech discussion 2024/07/25 the goals of specification and documentation system are: 1. Clarity and understanding of: 1.1 architecture; 1.2 functionality; 1.3 tests [i.e. functionality with some emphasis on quality]; 2. [by that enabling/helping] coordination [of development activities] by: 2.1 the core team; 2.2 community; 2.3 integration partners; 3. Making documentation part of the code, i.e. living documentation, which changes with the code and reflects both current status and planned / proposed directions; |
| Who it benefits | 1) NuNet technical board by having space where conceptual architecture and priorities are translated into architectural and functional descriptions 2) Internal platform development team as well as community developers have clearly defined structure architecture and proposed functionality guiding the work and collaboration 4) Internal integrations team (developing use-cases) and third party solution providers have clear documentation of the platform to build solutions on top 5) integration partners, using the system can understand the current and future integration potential on a deep technical level and propose required functionality directly into the development process |
| User challenge | In order for all stakeholder groups to benefit, the documentation and specification system should be 1) structured enough for all stakeholders to quickly find aspects that are specifically needed for them; 2) flexible enough to express package architecture and implementation specific for maximal clarity and understanding; 3) integrated into development process for easy access by and contribution from internal development team 4) conveniently accessible for communication to community developers, integration partners (for this milestone) and other stakeholders from broader ecosystem (in the future); |
| Value score | n/a |
| Design | as described during tech discussion 2024/07/25 |
| Impacted functionality | This does not affect any feature directly but fundamentally enables the quality and integrity of the whole platform functionality, alignment with the business goals, use-case based platform development model and evolvability of the software. |
| Acceptance tests | documentation portal launch issue and deliverables |
Project management portal
| Feature name | Prepare and launch project management portal |
| Work package | There is not specific work package in the current milestone, as project management portal was developed outside its scope; |
| Code reference | Drive Folder with related working documents and discussions; project management portal repository; gitbook team pages; gitbook public pages |
| Description / definition of done | 1) Project management pages always reflect the latest status of each milestone; 2) Introductory page explains project management portal and process for internal and external users |
| Status | Close to completion |
| Team | Lead: @janaina.senna; Supporting: @0xPravar, @kabir.kbr |
| Strategic alignment | Project management portal is a part of the Critical Chain Project management process that is the basis of our development flow and is integrated into the team process; The aim of CCPM process is to achieve the level where we are able to plan, schedule and implement milestones and releases without delay; The portal is designed to automatically generate and share all important updates about the process with the team as well as external contributors in order to make the process as efficient and transparent as possible; The process is used internally since mid 2023 and shall be included into the contribution program and community developers process; |
| Who it benefits | 1) Development team follows the CCPM process via the project management portal; 2) Marketing team uses project management portal (as well as other internal resources) to shape communications transparently; 3) Operations and business development team can always see the recent status in development process |
| User challenge | |
| Value score | n/a |
| Design | As described in project management portal repository readme |
| Impacted functionality | This does not affect any feature directly but fundamentally enables alignment with the business goals, use-case based platform development model and evolvability of the software. |
| Acceptance tests | Project management portal launch issue and deliverables |
Implementation
Actor model with Object/Security Capabilities
| Feature name | Actor model and interface; Node and Allocation actors implementations; feature::actor-model; feature::general |
| Work packages | dms node jobs network |
| Code reference | https://gitlab.com/nunet/test-suite/-/tree/develop/environments/feature?ref_type=heads |
| Description / definition of done | 1) Machine onboarded on NuNet via dms act as separate actors (via Node interface); 2) Jobs deployed and orchestrated on the platform act as separate Actors (via Allocation interface); 3) Nodes and Allocations (both implementing the Actor interface) communicate only via the immutable messages (via Message interface and network package's transport layer) and have no direct access to each other private state; |
| Timing | Actor, Node and Allocation interfaces are correctly implemented at the start of the release process; correct interface implementation means that the send messages to each other, can read and process received messages and initiate arbitrary behaviors via RPC model; |
| Status | With challenges |
| Team | Lead: @mobin.hosseini; Supporting: @gugachkr |
| Strategic alignment | 1) Enables the unlimited horizontal scalability of the platform with potentially minimal manual intervention; 2) Enables any eligible entity to join the platform without central control point; 3) Enables concurrency and non-locking by design which is independent of the scale of the platform at any point in time |
| Who it benefits | 1) Solution integrators (providers) who need to use Decentralized Physical Infrastructure; 2) Hardware owners who wish to utilize their infrastructure in a more flexible manner than possible with established orchestration and deployment technologies and without direct involvement with the compute users or building compute marketplaces |
| User challenge | 1) All compute users want to maximize efficient and fast access to optimal hardware for doing a computing job at hand and not to overpay for that; 2) Hardware resource owners want the maximal utilization of their hardware resources without idle time; |
| Value score | n/a |
| Design | DMS architecture |
| Impacted functionality | All functionality of the platform is fundamentally affected implementation of actor model; This is especially true for the future projected functionalities involving edge computing, IoT deployments and decentralized physical infrastructure in general. |
| Acceptance tests | Functional and integration tests defined in node package, dms package related to Actor interface and jobs package related to Allocation interface; tracking issue |