Last updated: 2024-11-07 21:05:55.068812 File source: link on GitLab
NuNet's CI/CD pipeline is an automation pipeline to support NuNet's products development. It is designed to reflect the testing processes and support testing programs, campaigns and environments. It is invaluable for the the internal development team and the public development community for it provides them with the necessary feedback and structure to increase confidence and reliability of the changes made to the codebase during the development lifecycle.
Continuous delivery and continuous integration is a broad topic that spans practically the entire lifecycle of the software product, from the first commit of a new feature to its deployment in production. The scope of the pipeline in NuNet is no different, so it isn't hard to imagine a pipeline that spans multiple repositories, each specialized in a specific part of the process, from code that is responsible for testing, to jobs that automate deployment and continuous monitoring of the application.
The goal of a CI/CD pipeline is to test every new version of a platform on a network that is as close to the production environment as possible, automating the reproduction of all possible behaviors of the platform on this network. This enables the development team to rerun them on each new modification of the codebase in order to identify and eventually eliminate errors before releasing a new version of the components that compose the production network.
The complication compared to a usual CI/CD pipelines comes with the fact that NuNet is a decentralized platform running on different, heterogenous computers around the world. Therefore, we need to spawn a testnet that already contains independent compute providers to pass CI/CD jobs quite early in the pipeline. Spawning the required testnet for the purposes of testing the platform involves the execution of different flavors of 'deploy' jobs, while actually testing these deployments require direct interaction with the deployed components. Additional complication is that NuNet platform isn't prescriptive about the hardware that it runs on, since it is owned by compute providers's machine to which NuNet will not have root access. Therefore, advanced stages of testing pipeline will invariably involve manual tests, which are encorage by community testers programs.
When it comes to the actual implementation of the CI/CD pipeline, we leverage Gitlab CI/CD pipelines. This enables us to easily define and reuse components of the pipeline in every repository as we see fit.
Each repository has it's own .gitlab-ci.yml file which defines the pipeline for that repository. The repository won't have to redefine the entire pipeline however. It is able to import and reuse, as described above, pieces of the pipeline from external repositories.
Currently, we have the entirety of the pipeline described in nunet/test-suite, which is the repository responsible for tracking the development of the testing stack.
Before it's able to inherit the pipeline from test-suite the repository, however, the repository must define some jobs, namely:
the docker build job, which produces an image that contains the necessary dependencies for compiling and running the code in that repo.
the build job, which is responsible for producing runnable and/or installable artifacts, which the pipeline uses to run functional tests.
For reference, you can refer to the actual pipeline implementation of nunet/device-management-service.
We also have nunet/nunet-infra which is responsible for tracking progress related to the deployment of the infrastructure at NuNet. Lines between test-suite and nunet-infra blur when you think that there is need for specific infrastrucutre deployments to support the testing of software, which should provide the necessary components that the software needs to interact (databases, object stores, container runtimes etc...). Therefore there is infrastructure code in test-suite, but it shouldn't be mistaken with code in nunet-infra, which is related to the operational part of the pipeline.
Below is the explanation of each state in the general pipeline considering above.
The NuNet CI/CD pipeline combines a number of stages, which are called from the template file. All templates are contained in nunet/test-suite/cicd. Note, that the CI/CD pipeline is being implemented iterativelly step by step, and also is constantly augmented -- to cover all new functionalities and aspects of the platform --, therefore some jobs and stages may still contain empty templates which will be implemented or considerably change in the future.
The states are dependent on each other and ordered by order of execution. However, due to differences of code in different repositories, they may run in different order.
Static Analysis / Code Quality (name static_analysis)
Contains static analysis of the code quality using a number of tools that check and report code quality level. Code quality results for each run of this job are displayed via the gitlab interface. Definition: Code-Quality.gitlab-ci.yml
Unit Tests (name unit_tests)
Runs units tests on the codebase for each language which exists in the codebase (since NuNet is a language agnostic platform, it may contain multiple language code). Coverage report is displayed via gitlab interface.
Definition: Unit-Tests.gitlab-ci.yml
Static Security tests (name security_tests_1)
Automated security testing (using third party tools) does not need live environments or a testnet, i.e. can be run on the static repository code.
Definition: Security-Tests-1.gitlab-ci.yml
Build (name build)
Builds all NuNet platform components that are needed for deploying the platform on a testnet (and eventually on the production network) including multi-architecture packages of device-management-service and centralized services as needed. The usual source is develop branches of each respective repository, which contains the bleeding edge and unstable version of the code for each component.
Definition: Build.gitlab-ci.yml
Feature environment / Functional tests / API tests (name functional_tests)
Tests each API call as defined by the NuNet Open API of the respective version that is being tested. The goal of this stage is to make sure that the released versions of the platform fully correspond to the released Open APIs, which will be used by core team, community developers and app integrators to build further.
This stage is responsible for provisioning the feature environment, deploying the respective DMS build to be tested in virtual machines hosted by a cluster of LXD providers, and running functional tests, as implemented in nunet/test-suite/stages/functional_tests against this deployed network.
Definition: Feature-Environment.gitlab-ci.yml
Automated security tests (name security_tests_2) -- to be implemented
Tests security of API calls that does need deployed testnetwork.
Definition: Security-Tests-2.gitlab-ci.yml
User Acceptance Tests (name user_acceptance_tests) -- to be implemented
Testing user behaviors from user perspective. Include all identify possible user behaviors and is constantly updated as soon as these behaviors are identified. The goal is to run most of the user acceptance tests automatically (describing scenarios BDD style), however, some of the tests will need to be run manually by the network of beta testers.
Definition: User-Acceptance-Tests.gitlab-ci.yml
Deploy Staging Network (name deploy_staging) -- to be implemented
As described in Git workflow -- branching strategy and NuNet test process and environments, a pre-release version of the platform with frozen features for release version will be separated to the release branches on each respective repository, which will be build and extensively tested with advanced tests, requiring large network of compute providers covering all possible hardware configurations and supported operating systems. The staging network will be organized via collaboration with the community.
Definition: Deploy-Staging.gitlab-ci.yml
Regression tests (name regression_tests) -- to be implemented
Regression tests will deploy and run all applications that are running on the platform. These tests may overlap or include user acceptance testing and testing behaviors on these applications as well as deployment behaviors. As well as user acceptance testing stage, regression tests may include manual beta testing phase.
Definition: Regression-Tests.gitlab-ci.yml
Performance and load testing (name load_tests) -- to be implemented
Define performance scenarios that would exhaust the system and automatically run them for checking confidentiality, availability and integrity of the platform.
Definition: Load-Tests.gitlab-ci.yml
Live Security Tests (name security_tests_3) -- to be implemented
Security Tests that need full platform and all applications running to test security aspects from user perspective. Will be done mostly manually and include 'red team' style penetration testing. All detected vulnerabilities will be included into security_tests_1 and security_tests_2 stages for automated testing of the further platform versions.
Deploy into production (name deploy_prod) -- to be implemented
If all tests pass on the staging network, the release branches are tagged and announced as per Git workflow -- branching strategy.