EPIC: Modernise and Refactor Adoptium CI Pipelines and Infrastructure

[steelhead31]

Summary

This epic covers the strategic refactoring and modernisation of the Adoptium CI platform, with a focus on improving maintainability, scalability, reliability, and long-term sustainability.

The current CI environment has evolved over many years and contains a significant amount of Jenkins-specific pipeline logic, shared libraries, infrastructure configuration, and operational knowledge. While the existing platform continues to deliver successfully, increasing complexity makes future enhancements, maintenance, onboarding, and platform evolution more challenging.

As part of this initiative, a new Jenkins environment will be established as a prototype and validation platform. This environment will be used to design, test, and validate a more modular architecture before wider adoption. The new platform should be treated as disposable infrastructure that can be fully recreated from code, enabling rapid recovery, reproducibility, and simplified maintenance.

A key objective is to reduce coupling between Adoptium build processes and Jenkins-specific implementations. Build, packaging, testing, publishing, and orchestration logic should be extracted into reusable, platform-agnostic components wherever practical. This will improve maintainability while reducing barriers to future adoption of alternative CI/CD platforms.

Goals

• Refactor existing CI pipelines into modular, reusable components.
• Reduce reliance on Jenkins-specific Groovy and pipeline implementations.
• Separate business logic from CI orchestration logic.
• Improve pipeline maintainability, readability, and testability.
• Establish Infrastructure-as-Code (IaC) principles across the CI platform.
• Ensure CI infrastructure can be recreated from source-controlled definitions.
• Improve platform resilience, disaster recovery, and operational sustainability.
• Standardise pipeline patterns, tooling, and configuration management.
• Reduce operational overhead associated with Jenkins administration.
• Improve portability of build and release workflows across CI platforms.
• Create a reference implementation on a new Jenkins server to validate architectural decisions.

Scope of Work

CI Infrastructure Modernisation

• Build and configure a new Jenkins prototype environment.
• Define infrastructure using Infrastructure-as-Code practices.
• Automate provisioning, configuration, and deployment of Jenkins services.
• Document all infrastructure components and dependencies.
• Establish backup, recovery, and recreation procedures.
• Validate full environment rebuild from source-controlled configuration.

Pipeline Refactoring

• Review existing pipeline architecture and identify common patterns.
• Consolidate duplicated pipeline logic.
• Extract reusable functionality into shared tools, scripts, or services.
• Move business logic out of Jenkinsfiles where possible.
• Standardise pipeline structure and implementation approaches.
• Improve testing and validation of pipeline changes.

Platform Abstraction

• Identify Jenkins-specific dependencies throughout the CI ecosystem.
• Replace Jenkins-specific functionality with portable alternatives where practical.
• Evaluate use of external tooling, APIs, and command-line utilities to perform build orchestration tasks.
• Define clear interfaces between CI orchestration and build execution logic.
• Create architecture patterns that could be implemented on alternative CI platforms with minimal changes.

Sustainability and Operational Excellence

• Improve documentation and reduce reliance on tribal knowledge.
• Define operational standards and best practices.
• Reduce maintenance burden for infrastructure administrators.
• Improve observability, monitoring, and troubleshooting capabilities.
• Establish repeatable upgrade and migration processes.

Deliverables

• Prototype Jenkins environment deployed from code.
• Infrastructure-as-Code repository and documentation.
• Refactored modular pipeline framework.
• Reusable build and release automation components.
• CI architecture and design documentation.
• Migration and operational runbooks.
• Recommendations for future CI platform portability.
• Assessment of Jenkins-specific dependencies and proposed replacements.