WebsiteTechnical Discussions

cicd

Last updated: 2025-06-26 01:05:58.095952 File source: link on GitLab

NuNet CI/CD Pipeline Documentation

This document provides an overview of the CI/CD pipeline infrastructure for the NuNet project. It is intended for developers, DevOps engineers, and SREs who will maintain and extend this system.

Table of Contents

  1. Project Structure

  2. Pipeline Stages and Jobs

  3. Configuration Variables

  4. Maintenance Guidelines

  5. Technical Components

Project Structure

The CI/CD configuration is organized into modular files that define specific aspects of the pipeline:

cicd/
├── Auto-DevOps.gitlab-ci.yml          # Main pipeline definition
├── Feature-Environment.gitlab-ci.yml  # Feature environment provisioning/deprovisioning
├── Jobs/
│   ├── Code-Quality.gitlab-ci.yml     # Code quality checks
│   ├── Unit-Tests.gitlab-ci.yml       # Unit test execution
│   ├── Integration-Tests.gitlab-ci.yml# Integration tests
│   ├── SAST-Security-Tests.gitlab-ci.yml# Static application security testing
│   ├── Feature-Environment.gitlab-ci.yml# Feature environment management
│   └── ...                            # Additional job definitions
├── Templates/
│   ├── rsync.gitlab-ci.yml            # Rsync template for artifact distribution
└── README.md                          # This documentation file

Pipeline Stages and Jobs

The pipeline is divided into logical stages, each containing specific jobs:

1. Pre-Build Stages

  • Stages: .pre

  • Jobs:

    • Prep Semver Config: Prepares versioning configuration for the pipeline.

    • DefectDojo Check Product Exists: Ensures product exists in DefectDojo.

2. Code Quality and Linting

  • Stages: code_quality

  • Jobs:

    • golangci_lint: Runs Go linting with golangci-lint.

    • license_check: Verifies software licenses using go-licence-detector.

3. Unit Tests

  • Stages: unit_tests

  • Jobs:

    • Golang Unit Tests: Executes unit tests for Go codebase.

    • JavaScript Unit Tests: Runs JavaScript unit tests with Mocha.

    • Golang Coverage Report: Generates coverage reports for Go code.

4. Integration and E2E Tests

  • Stages: integration_tests_1, e2e_tests_1

  • Jobs:

    • go_integration_tests: Runs integration tests for Go services.

    • go-e2e-tests: Executes end-to-end tests for the system.

5. Security Testing

  • Stages: security_tests_1

  • Jobs:

    • semgrep-sast: Runs Semgrep security analysis.

    • container-scanning-defectdojo: Performs container image scanning with Trivy.

    • secret-detection-defectdojo: Detects secrets in source code.

6. Build Stages

  • Stages: build

  • Jobs:

    • Build: Builds Debian packages for Linux distributions.

    • build:osx: Builds Darwin binaries for macOS.

7. Feature Environment Management

  • Stages: test, functional_tests

  • Jobs:

    • run_feature_environment: Spins up feature environments for testing.

    • feature_environment_test_results: Collects and aggregates test results.

8. Post-Build Stages

  • Stages: .post

  • Jobs:

    • delete_build_artifacts: Cleans up temporary build artifacts.

    • update_code_coverage: Updates code coverage information in the repository.

Configuration Variables

The pipeline uses a variety of variables to control its behavior. These can be customized in your GitLab project settings under CI/CD > Variables.

Build Configuration

  • BUILD_DISABLED: Enable/disable builds (default: false).

  • ALLOW_BUILD_FAIL: Allow build failures without blocking the pipeline (default: false).

Testing Configuration

  • SKIP_UNIT_TESTS: Skip unit tests (default: false).

  • ALLOW_E2E_TESTS_FAIL: Allow E2E test failures (default: false).

  • E2E_TESTS_DISABLE: Disable E2E tests (default: false).

Security Configuration

  • DEFECTDOJO_DISABLED: Disable DefectDojo integration (default: false).

  • SAST_EXCLUDED_ANALYZERS: List of analyzers to exclude from SAST scans.

  • SECRET_DETECTION_DISABLED: Disable secret detection (default: false).

Feature Environment Configuration

  • FEATURE_ENVIRONMENT_BRANCH: Branch for feature environment builds (default: release).

  • FEATURE_ENV_CONFIG_VAULT_PATH: Path to feature environment configuration in Vault.

Reporting and Notifications

  • SLACK_WEBHOOK: Webhook URL for Slack notifications.

  • TESTMO_URL: URL for Testmo integration.

  • CI_REPORTS_SSH_KEY_B64: SSH key for report distribution.

Maintenance Guidelines

  1. Keep It DRY:

    • Avoid duplicating configuration across files. Use templates and shared variables wherever possible.

  2. Versioned Configuration:

    • Major changes to the pipeline should be versioned to ensure backward compatibility.

  3. Monitor Performance:

    • Regularly review job execution times and optimize long-running jobs.

  4. Security Best Practices:

    • Rotate credentials regularly.

    • Use scoped tokens with minimal permissions.

  5. Documentation:

    • Keep this README updated whenever significant changes are made to the pipeline.

  6. Testing Changes:

    • Test configuration changes in a staging environment before deploying them to production pipelines.

Technical Components

The CI/CD pipeline integrates with several key tools and services:

  1. GitLab CI/CD:

    • Core orchestration engine for the pipeline.

    • Manages stages, jobs, and dependencies between tasks.

  2. Allure:

    • Test reporting and visualization tool.

    • Provides detailed insights into test results and coverage.

  3. SonarQube:

    • Code quality analysis tool.

    • Detects code smells, bugs, and vulnerabilities.

  4. DefectDojo:

    • Vulnerability management platform.

    • Integrates with security testing tools to manage findings.

  5. Testmo:

    • Test result aggregation and reporting tool.

    • Provides a unified view of test results across the pipeline.

  6. Trivy:

    • Container image scanner for vulnerabilities and misconfigurations.

  7. Semgrep:

    • Static application security testing (SAST) tool.

  8. GolangCI-Lint:

    • Linter for Go codebases.

    • Enforces coding standards and best practices.

This documentation provides a comprehensive overview of the CI/CD pipeline structure, configuration options, and maintenance guidelines. For more detailed information about specific jobs or stages, refer to the individual .gitlab-ci.yml files in the cicd/Jobs directory.

PreviousreleaseNexttests

Last updated