LogoLogo
WebsiteTechnical Discussions
Developer documentation
Developer documentation
  • Public Technical Documentation
    • device-management-service
      • main
        • actor
        • api
        • cmd
          • actor
          • cap
        • db
          • repositories
            • clover
        • dms
          • behaviors
          • jobs
          • node
          • onboarding
          • resources
        • docs
          • deployments
          • onboarding
          • private_network
        • executor
          • docker
          • firecracker
          • background_tasks
          • config
          • hardware
        • maint-scripts
        • network
          • libp2p
        • observability
        • specs
          • basic_controller
          • s3
          • volume
          • integration
        • tokenomics
        • types
        • utils
          • validate
      • release
        • actor
        • api
        • cmd
          • actor
          • cap
        • db
          • repositories
            • clover
        • dms
          • behaviors
          • hardware
          • jobs
          • node
          • onboarding
          • resources
        • docs
          • deployments
          • onboarding
          • private_network
        • executor
          • docker
          • firecracker
        • internal
          • background_tasks
          • config
        • maint-scripts
        • network
          • libp2p
        • observability
        • plugins
        • specs
        • storage
          • basic_controller
          • s3
          • volume
        • test
        • tokenomics
        • types
        • utils
          • validate
    • solutions
      • asi-node
        • main
          • demos
            • ocap_auth
              • ansible
              • videos
          • did-auth-use-scenarios
            • create-a-hosting-deployment
            • make-a-payment
            • manage-dids-and-ocaps
            • register-an-ai-agent
            • submit-a-compute-job
          • pilots
            • asi-create-authentication-poc
        • release
      • nunet-appliance
        • main
        • release
    • test-suite
      • main
        • cicd
          • tests
            • feature_environment
          • cli
          • dms-on-lxd
            • local
        • environments
          • development
          • feature
          • production
          • staging
        • infrastructure
          • cloud-init
          • dms-on-lxd
          • nginx
        • lib
        • stages
          • dependency_scanning
          • functional_tests
          • integration_tests
          • load_tests
          • regression_tests
          • security_tests_1
          • security_tests_2
          • security_tests_live
          • unit_tests
          • user_acceptance_tests
      • release
        • cicd
          • tests
            • feature_environment
          • cli
          • dms-on-lxd
            • local
        • environments
          • development
          • feature
          • production
          • staging
        • infrastructure
          • cloud-init
          • dms-on-lxd
          • nginx
        • lib
        • stages
          • dependency_scanning
          • functional_tests
          • integration_tests
          • load_tests
          • regression_tests
          • security_tests_1
          • security_tests_2
          • security_tests_live
          • unit_tests
          • user_acceptance_tests
    • team-processes-and-guidelines
      • main
        • best_practices
        • ci_cd_pipeline
        • community_feedback_process
        • contributing_guidelines
        • git_workflows
        • nunet_test_process_and_environments
        • secure_coding_guidelines
        • specification_and_documentation
        • team_process
          • a_project_management
          • b_ceremonies_artifacts
          • c_drum_buffer_rope
          • d_development_process
          • e_culture_rules
          • f_mr_review
        • vulnerability_management
          • devsecops_maturity_models
          • nunet_security_pipeline
          • secret_management
          • sop_security_mr_review
Powered by GitBook
On this page
  • NuNet CI/CD Pipeline Documentation
  • Table of Contents
  • Project Structure
  • Pipeline Stages and Jobs
  • Configuration Variables
  • Maintenance Guidelines
  • Technical Components
Export as PDF
  1. Public Technical Documentation
  2. test-suite
  3. release

cicd

PreviousreleaseNexttests

Last updated 10 hours ago

Last updated: 2025-05-15 01:09:51.978810 File source:

NuNet CI/CD Pipeline Documentation

This document provides an overview of the CI/CD pipeline infrastructure for the NuNet project. It is intended for developers, DevOps engineers, and SREs who will maintain and extend this system.

Table of Contents


Project Structure

The CI/CD configuration is organized into modular files that define specific aspects of the pipeline:

cicd/
├── Auto-DevOps.gitlab-ci.yml          # Main pipeline definition
├── Feature-Environment.gitlab-ci.yml  # Feature environment provisioning/deprovisioning
├── Jobs/
│   ├── Code-Quality.gitlab-ci.yml     # Code quality checks
│   ├── Unit-Tests.gitlab-ci.yml       # Unit test execution
│   ├── Integration-Tests.gitlab-ci.yml# Integration tests
│   ├── SAST-Security-Tests.gitlab-ci.yml# Static application security testing
│   ├── Feature-Environment.gitlab-ci.yml# Feature environment management
│   └── ...                            # Additional job definitions
├── Templates/
│   ├── rsync.gitlab-ci.yml            # Rsync template for artifact distribution
└── README.md                          # This documentation file

Pipeline Stages and Jobs

The pipeline is divided into logical stages, each containing specific jobs:

1. Pre-Build Stages

  • Stages: .pre

  • Jobs:

    • Prep Semver Config: Prepares versioning configuration for the pipeline.

    • DefectDojo Check Product Exists: Ensures product exists in DefectDojo.

2. Code Quality and Linting

  • Stages: code_quality

  • Jobs:

    • golangci_lint: Runs Go linting with golangci-lint.

    • license_check: Verifies software licenses using go-licence-detector.

3. Unit Tests

  • Stages: unit_tests

  • Jobs:

    • Golang Unit Tests: Executes unit tests for Go codebase.

    • JavaScript Unit Tests: Runs JavaScript unit tests with Mocha.

    • Golang Coverage Report: Generates coverage reports for Go code.

4. Integration and E2E Tests

  • Stages: integration_tests_1, e2e_tests_1

  • Jobs:

    • go_integration_tests: Runs integration tests for Go services.

    • go-e2e-tests: Executes end-to-end tests for the system.

5. Security Testing

  • Stages: security_tests_1

  • Jobs:

    • semgrep-sast: Runs Semgrep security analysis.

    • container-scanning-defectdojo: Performs container image scanning with Trivy.

    • secret-detection-defectdojo: Detects secrets in source code.

6. Build Stages

  • Stages: build

  • Jobs:

    • Build: Builds Debian packages for Linux distributions.

    • build:osx: Builds Darwin binaries for macOS.

7. Feature Environment Management

  • Stages: test, functional_tests

  • Jobs:

    • run_feature_environment: Spins up feature environments for testing.

    • feature_environment_test_results: Collects and aggregates test results.

8. Post-Build Stages

  • Stages: .post

  • Jobs:

    • delete_build_artifacts: Cleans up temporary build artifacts.

    • update_code_coverage: Updates code coverage information in the repository.


Configuration Variables

The pipeline uses a variety of variables to control its behavior. These can be customized in your GitLab project settings under CI/CD > Variables.

Build Configuration

  • BUILD_DISABLED: Enable/disable builds (default: false).

  • ALLOW_BUILD_FAIL: Allow build failures without blocking the pipeline (default: false).

Testing Configuration

  • SKIP_UNIT_TESTS: Skip unit tests (default: false).

  • ALLOW_E2E_TESTS_FAIL: Allow E2E test failures (default: false).

  • E2E_TESTS_DISABLE: Disable E2E tests (default: false).

Security Configuration

  • DEFECTDOJO_DISABLED: Disable DefectDojo integration (default: false).

  • SAST_EXCLUDED_ANALYZERS: List of analyzers to exclude from SAST scans.

  • SECRET_DETECTION_DISABLED: Disable secret detection (default: false).

Feature Environment Configuration

  • FEATURE_ENVIRONMENT_BRANCH: Branch for feature environment builds (default: release).

  • FEATURE_ENV_CONFIG_VAULT_PATH: Path to feature environment configuration in Vault.

Reporting and Notifications

  • SLACK_WEBHOOK: Webhook URL for Slack notifications.

  • TESTMO_URL: URL for Testmo integration.

  • CI_REPORTS_SSH_KEY_B64: SSH key for report distribution.


Maintenance Guidelines

  1. Keep It DRY:

    • Avoid duplicating configuration across files. Use templates and shared variables wherever possible.

  2. Versioned Configuration:

    • Major changes to the pipeline should be versioned to ensure backward compatibility.

  3. Monitor Performance:

    • Regularly review job execution times and optimize long-running jobs.

  4. Security Best Practices:

    • Rotate credentials regularly.

    • Use scoped tokens with minimal permissions.

  5. Documentation:

    • Keep this README updated whenever significant changes are made to the pipeline.

  6. Testing Changes:

    • Test configuration changes in a staging environment before deploying them to production pipelines.


Technical Components

The CI/CD pipeline integrates with several key tools and services:

  1. GitLab CI/CD:

    • Core orchestration engine for the pipeline.

    • Manages stages, jobs, and dependencies between tasks.

  2. Allure:

    • Test reporting and visualization tool.

    • Provides detailed insights into test results and coverage.

  3. SonarQube:

    • Code quality analysis tool.

    • Detects code smells, bugs, and vulnerabilities.

  4. DefectDojo:

    • Vulnerability management platform.

    • Integrates with security testing tools to manage findings.

  5. Testmo:

    • Test result aggregation and reporting tool.

    • Provides a unified view of test results across the pipeline.

  6. Trivy:

    • Container image scanner for vulnerabilities and misconfigurations.

  7. Semgrep:

    • Static application security testing (SAST) tool.

  8. GolangCI-Lint:

    • Linter for Go codebases.

    • Enforces coding standards and best practices.


This documentation provides a comprehensive overview of the CI/CD pipeline structure, configuration options, and maintenance guidelines. For more detailed information about specific jobs or stages, refer to the individual .gitlab-ci.yml files in the cicd/Jobs directory.

link on GitLab
Project Structure
Pipeline Stages and Jobs
Configuration Variables
Maintenance Guidelines
Technical Components