main

Last updated: 2025-06-26 01:05:46.350915 File source: link on GitLab

test-suite

This repository combines all tests and defines full testing pipeline of the platform. The test pipeline is rather complex and different stages are implemented in different repositories. This repository aims at holding all relevant information and definitions inrrespectively as well as provide the interface to test management framework, which displays results of each test run in the Gitlab CI pipeline.

CLI

There is a CLI that makes it easier to execute some functions of the test-suite locally. Please refer to the CLI Guide for more instructions.

Repository structure

├── .gitlab         # issue templates and other technical gitlab related artifacts
├── stages          # definitions of use cases and bahaviors of each test stage
├── environments    # definitions of environments

Testing matrix

Pre-build tests

Stage
Environment
Status
Comments

code_quality

CI/CD Pipeline via GitLab Runners

implemented

unit_tests

CI/CD Pipeline via GitLab Runners

implemented

compliance_check

n/a

not implemented

n/a

integration_tests

Options: Local on devs machines & CI/CD Pipeline via GitLab Runners

implemented

Local option: run make itest in dms root; CI/CD pipeline Integration-Tests.gitlab-ci.yml

security_tests

CI/CD Pipeline via GitLab Runners

implemented

Tests running in testnet environments

Stage
Environment
Status
Comments

End-to-end tests (also called functional_tests / e2e / feature)

Standalone Options: Local on devs machines & CI/CD Pipeline via GitLab Runners

implemented

Local option: via cli.sh as explained in test-suite/docs/cli; CI/CD pipeline Feature-Environment.gitlab-ci.yml

End-to-end tests (also called functional_tests / e2e / feature)

Distributed Options: Local on devs machines & CI/CD Pipeline via GitLab Runners

implemented

Local option: via cli.sh as explained in test-suite/docs/cli; CI/CD pipeline Feature-Environment.gitlab-ci.yml

integration_tests_live

live testnet (?)

not implemented

security_tests_live

live testnet (?)

not implemented

regression_tests

live testnet (?)

not implemented

performance_tests

live testnet (?)

not implemented

load_tests

live testnet (?)

not implemented

security_tests_3

live testnet (?)

not implemented

Tests running in production

TBD

How to read the table: columns are environments and rows are test stages; there is a build stage that is ommited and runs between pre-build tests and testnet environments:

  • y means that a corresponding environment is needed to run a stage (equivalent -- a stage runs in that environment);

  • n means the opposite -- a stage does not need that environment to run an is not executed in that environment;

The meaning of the testing matrix cells is explained below. Note, they may differ depending on the stage (e.g. manual execution of deploy stage means that at least part of the environment will need manual actions from community compute providers; likewise, manual acceptance tests consider involvement of beta testers). Furthermore, some test stages contain sub-stages and the whole matrix will be evolving together with the code base.

Branching strategy

The branching strategy is defined here.

Testing schedule

TBD: will define when and how we run advanced tests which need manual testing as well as heavy environments (we will not be able to run them very frequently, and that will need to be properly scheduled).

Last updated