diff --git a/.gitignore b/.gitignore index 828d595a94..b52e2cc37a 100644 --- a/.gitignore +++ b/.gitignore @@ -8,3 +8,7 @@ ansible/host_vars/* *.swp ansible/machine_audit/backend/collectedInfo ansible/machine_audit/backend/.env +jenkins-as-code/Vagrant-Scripts/ +jenkins-as-code/ansible/hosts +jenkins-as-code/Vagrantfile +jenkins-as-code/data/ diff --git a/SETUP.md b/SETUP.md new file mode 100644 index 0000000000..5b17413e8c --- /dev/null +++ b/SETUP.md @@ -0,0 +1,111 @@ +# Infrastructure Setup Guide + +## Prerequisites + +This guide covers the setup requirements for the Adoptium infrastructure automation using Ansible. + +## Python Requirements + +The infrastructure automation requires Python 3.x and several Python packages, particularly for managing Windows hosts via WinRM. + +### Installation + +Install the required Python packages: + +```bash +pip3 install --user -r requirements.txt +``` + +Or system-wide (requires sudo): + +```bash +sudo pip3 install -r requirements.txt +``` + +### Key Dependencies + +- **ansible** - Core automation framework +- **pywinrm** - Required for Windows host management via WinRM protocol +- **requests-ntlm** - NTLM authentication support for Windows +- **requests-credssp** - CredSSP authentication support + +## Ansible Collections + +The project uses Ansible Galaxy collections that must be installed: + +```bash +ansible-galaxy collection install -r collections/requirements.yml +``` + +To force reinstall/update collections: + +```bash +ansible-galaxy collection install -r collections/requirements.yml --force +``` + +### Required Collections + +- **community.general** - General purpose modules and plugins +- **community.windows** - Windows-specific modules (requires pywinrm) +- **ansible.windows** - Core Windows support + +## Verification + +Verify your setup: + +```bash +# Check Python packages +pip3 list | grep -E "(ansible|pywinrm|requests)" + +# Check Ansible collections +ansible-galaxy collection list | grep -E "(community.general|community.windows|ansible.windows)" + +# Verify pywinrm is accessible +python3 -c "import winrm; print(f'pywinrm version: {winrm.__version__}')" +``` + +## Troubleshooting + +### WinRM Dependency Error + +If you see an error like: +``` +Unable to resolve dependency: user requested 'winrm (= 2.3.6)' +``` + +This indicates the `pywinrm` Python package is not installed or not accessible. Solutions: + +1. Install Python requirements: `pip3 install --user -r requirements.txt` +2. Reinstall collections: `ansible-galaxy collection install -r collections/requirements.yml --force` +3. Verify pywinrm: `python3 -c "import winrm"` + +### Collection Installation Issues + +If collections fail to install: + +1. Check internet connectivity to galaxy.ansible.com +2. Clear Ansible cache: `rm -rf ~/.ansible/collections` +3. Reinstall with `--force` flag + +## Windows Host Configuration + +For managing Windows hosts, the target machines must be configured for WinRM. See the playbook comments in: +- `ansible/playbooks/AdoptOpenJDK_Windows_Playbook/windows_with_ssh.yml` +- `ansible/playbooks/AdoptOpenJDK_Windows_Playbook/windows_dockerhost.yml` + +Basic Windows setup: +```powershell +# On the Windows target machine (as Administrator) +wget https://raw.githubusercontent.com/ansible/ansible-documentation/devel/examples/scripts/ConfigureRemotingForAnsible.ps1 -OutFile .\ConfigureRemotingForAnsible.ps1 +.\ConfigureRemotingForAnsible.ps1 -CertValidityDays 9999 +.\ConfigureRemotingForAnsible.ps1 -EnableCredSSP +.\ConfigureRemotingForAnsible.ps1 -ForceNewSSLCert +.\ConfigureRemotingForAnsible.ps1 -SkipNetworkProfileCheck +``` + +## Additional Resources + +- [Main README](README.md) - Project overview and documentation +- [Ansible Documentation](ansible/README.md) - Detailed Ansible usage +- [FAQ](FAQ.md) - Common questions and operations +- [Contributing Guidelines](CONTRIBUTING.md) - How to contribute \ No newline at end of file diff --git a/jenkins-as-code/AGENT-CONTEXT.md b/jenkins-as-code/AGENT-CONTEXT.md new file mode 100644 index 0000000000..d7791cc602 --- /dev/null +++ b/jenkins-as-code/AGENT-CONTEXT.md @@ -0,0 +1,446 @@ +# jenkins-as-code — Agent Context & Design Reference + +This document is the **authoritative design reference** for the `jenkins-as-code/` sub-project. +It explains what was built, why each decision was made, the conventions to follow, and what +remains to do. It is written for both human engineers and AI coding agents working on this +sub-project. + +--- + +## Purpose + +`jenkins-as-code/` provisions a fully production-equivalent Adoptium Jenkins master from scratch +using Ansible. The goal is that any engineer (or automated process) can: + +1. Take a blank Ubuntu 24.04 server (or a Vagrant VM) +2. Run two Ansible playbooks +3. Have a running, hardened Jenkins instance whose configuration mirrors the current production + Hetzner server + +A companion backup/restore script pair captures and reproduces Jenkins application configuration +(plugins, jobs, credentials, nodes) independently of the OS-level provisioning. + +--- + +## Repository layout + +``` +jenkins-as-code/ +├── Vagrantfile # Dev VM definition (libvirt, ubuntu/24.04, 8 GB) +├── Vagrant-Scripts/ # libvirt network startup helpers +├── ansible/ +│ ├── group_vars/ +│ │ └── all.yml # ALL variable defaults live here — single source of truth +│ ├── hosts # Minimal INI inventory: localhost ansible_connection=local +│ ├── inventory-vagrant.yml # Dev overrides: heap=auto, listen=0.0.0.0 +│ ├── inventory-production.yml # Prod overrides: heap=19G, listen=127.0.0.1, IP whitelist +│ ├── inventory-example.yml # Template to copy for new environments +│ ├── setup-jenkins-host.yml # Step 1 playbook (OS prep) +│ ├── install-jenkins-server.yml # Step 2 playbook (Jenkins install) +│ ├── roles/ +│ │ ├── README.md # Role reference +│ │ ├── system_update/ # apt update + upgrade +│ │ ├── unattended_upgrades/ # Security-only auto-updates +│ │ ├── ntp_config/ # ntpsec with Ubuntu pool servers +│ │ └── fail2ban/ # SSH protection + IP whitelist +│ └── templates/ +│ ├── jenkins.service.j2 # systemd unit file +│ ├── jenkins-defaults.j2 # /etc/default/jenkins +│ └── jenkins-logrotate.j2 # /etc/logrotate.d/jenkins +├── jenkins-scripts/ +│ ├── backup-jenkins-app-config.sh # Captures JENKINS_HOME config elements +│ ├── restore-jenkins-app-config.sh # Restores backup with env-specific overrides +│ └── restore-config-overrides.env # Edit before cross-env restore (URLs, Slack, etc.) +├── data/ +│ ├── jenkins-app-backup-*.tar.gz # Backup tarballs (produced by backup script) +│ └── archive/ # Planning docs and older backups +├── docs/ # Operational documentation +│ ├── QUICK-START.md +│ ├── ENVIRONMENT-CONFIG.md +│ ├── PRODUCTION-CONFIG-NOTES.md +│ ├── CONFIG-ANALYSIS.md +│ ├── DEPLOYMENT-GUIDE.md +│ ├── VAGRANT-DEPLOYMENT.md +│ ├── JENKINS-INSTALL.md +│ └── IMPLEMENTATION-SUMMARY.md +└── AGENT-CONTEXT.md # ← this file +``` + +--- + +## Variable conventions + +### Single source of truth + +Every variable has a documented default in [`ansible/group_vars/all.yml`](ansible/group_vars/all.yml). +Playbooks and templates **never** contain hard-coded values — they always reference a variable. + +Inventory files (`inventory-vagrant.yml`, `inventory-production.yml`) only override values that +differ from those defaults for the specific environment. + +### Upgrading Jenkins + +When a new Jenkins LTS is released: + +1. Update `jenkins_version` in `group_vars/all.yml` +2. Fetch the new checksum: `curl -fsSL https://get.jenkins.io/war-stable//jenkins.war.sha256` +3. Update `jenkins_war_sha256` in `group_vars/all.yml` + +Both values must change together. The `install-jenkins-server.yml` playbook verifies the SHA-256 +at download time and fails hard if they do not match. + +### Upgrading Java + +Change `java_major_version` in `group_vars/all.yml`. The `java_package` and `java_home` variables +are derived from it automatically. The Adoptium APT repository (`packages.adoptium.net`) is used +and is configured by `setup-jenkins-host.yml`. + +--- + +## Two-step deployment model + +### Step 1 — `setup-jenkins-host.yml` + +Prepares the OS. Safe to run on a fresh or existing Ubuntu 24.04 host. + +Execution order: +1. `system_update` role — `apt update && apt upgrade` +2. `unattended_upgrades` role — installs package, writes `/etc/apt/apt.conf.d/50unattended-upgrades` and `20auto-upgrades` (disabled by default — enable manually after testing) +3. `ntp_config` role — installs `ntpsec`, writes `/etc/ntp.conf` with Ubuntu pool servers, enables service +4. `fail2ban` role — installs fail2ban, writes `/etc/fail2ban/jail.local`, enables service +5. Tasks: installs ~60 packages matching the production package list +6. Tasks: installs Temurin JDK (Adoptium APT repo, GPG-verified) +7. Tasks: creates `jenkins` OS user (UID/GID 1000 if available, otherwise system-assigned), SSH key, limits +8. Tasks: SSH hardening — disables password auth, enables pubkey-only, sets MaxAuthTries 3 + +### Step 2 — `install-jenkins-server.yml` + +Installs Jenkins. Requires Step 1 to have run first. + +Execution order: +1. Pre-flight assertions: Ubuntu 24.04, `jenkins` user exists, Java installed +2. Environment detection: if `/vagrant` is present → set `effective_listen_address=0.0.0.0` +3. Memory calculation: if `jenkins_heap_size=="auto"` → calculate based on `ansible_memtotal_mb` +4. Downloads `jenkins.war` from `get.jenkins.io`, verifies SHA-256 +5. Creates directory structure: `JENKINS_HOME`, `/var/cache/jenkins`, `/var/log/jenkins` +6. Renders templates: `jenkins.service`, `/etc/default/jenkins`, logrotate config +7. Reloads systemd, enables and starts `jenkins.service` +8. Polls `http://localhost:8080/login` until HTTP 200 (30 × 10 s retries) +9. Reads and prints `initialAdminPassword` + +--- + +## Heap auto-sizing + +`install-jenkins-server.yml` applies the following logic when `jenkins_heap_size == "auto"`: + +| System RAM | Heap | +|---|---| +| < 6 GB | 2G | +| 6 – 14 GB | 4G | +| 14 – 30 GB | 8G | +| 30 GB+ | 19G | + +The 30 GB+ tier matches the production Hetzner server. Staging/dev environments with smaller +VMs automatically receive appropriate smaller heaps. Set `jenkins_heap_size: "19G"` (or any +explicit value) in the inventory to pin the heap regardless of detected RAM. + +--- + +## Ansible roles in detail + +### `system_update` + +Runs `apt update && apt upgrade -y`. Simple, no handlers. Tagged `system_update`. + +### `unattended_upgrades` + +- Installs `unattended-upgrades` +- Writes `50unattended-upgrades` configured for `*-security` and ESM origins only +- Writes `20auto-upgrades` with both periodic values set to `"0"` (disabled by default) + +**⚠ Important:** Automatic updates are intentionally disabled by default. Enable by setting +both values to `"1"` in `/etc/apt/apt.conf.d/20auto-upgrades` after validating on the target +host. This matches production where the security-only cron runs from the root crontab. + +### `ntp_config` + +- Package: `ntpsec` (overridable via `ntp_package`) +- Service: `ntpsec` (overridable via `ntp_service`) +- Servers: `0-3.ubuntu.pool.ntp.org iburst` + `ntp.ubuntu.com` +- Handler: `restart ntp` fires on `/etc/ntp.conf` change + +### `fail2ban` + +- Writes `/etc/fail2ban/jail.local` (not `.conf` to avoid upgrade conflicts) +- SSH jail: `maxretry=3`, `findtime=10m`, `bantime=1h`, `backend=systemd` +- Recidive jail: 5+ bans in 24 h → 7-day ban +- Default `ignoreip`: loopback + RFC-1918 private ranges +- **Always override `fail2ban_ignoreip` in your inventory** with your management IPs before + deploying to any environment that you SSH into. Failure to do this can lock you out. +- Handler: `restart fail2ban` fires on `jail.local` change + +--- + +## systemd service design + +[`ansible/templates/jenkins.service.j2`](ansible/templates/jenkins.service.j2) renders to +`/etc/systemd/system/jenkins.service`. + +Key decisions: +- `ExecStart` directly invokes `java -jar jenkins.war` using `JAVA_ARGS` and `JENKINS_ARGS` + from `/etc/default/jenkins` (sourced via `EnvironmentFile`) +- `StandardOutput` and `StandardError` both append to `/var/log/jenkins/jenkins.log` +- `NoNewPrivileges=true` and `PrivateTmp=true` for systemd-level hardening +- `LimitNOFILE=8192` and `LimitNPROC=30654` match production values +- `Restart=on-failure` with `StartLimitBurst=3` in 60 s prevents restart storms + +--- + +## `/etc/default/jenkins` design + +[`ansible/templates/jenkins-defaults.j2`](ansible/templates/jenkins-defaults.j2) documents the +full history of JVM flag changes in comments for traceability. Current production settings: + +| Flag | Value | Reason | +|---|---|---| +| `-Xmx` | auto or fixed | See heap auto-sizing above | +| `RESULT_CACHE_ENABLED=false` | JUnit memory fix | issue #4364 (2026-05-28) | +| `PREVIOUS_TEST_RESULT_BACKTRACK_BUILDS_MAX=1` | JUnit memory fix | issue #4364 | +| `XStream2.collectionUpdateLimit=-1` | Prevents XStream limit errors | issue #4364 area | +| GC logging | 5×50 MB rotating | Performance analysis | +| `--sessionTimeout=720` | 12-hour sessions | Balance security/UX | +| `--sessionEviction=43200` | 12-hour eviction | Added 2024-05-09 | +| `--accessLoggerClassName=...` | Winstone access log | Added 2019-03-21 | +| `--httpListenAddress` | `127.0.0.1` (or `0.0.0.0` in Vagrant) | Require reverse proxy in prod | + +The `effective_listen_address` variable is set at run time by `install-jenkins-server.yml` +based on Vagrant detection, not statically in `group_vars/all.yml`. + +--- + +## Backup script design + +[`jenkins-scripts/backup-jenkins-app-config.sh`](jenkins-scripts/backup-jenkins-app-config.sh) + +### Philosophy + +Captures only **application configuration** — not OS-level config (that is `extract-jenkins-master-config.sh`'s job) and not build data (`jobs/`, `workspace/` are explicitly excluded). + +### Archive format + +Outer tarball: `jenkins-app-backup-YYYYMMDD-HHMMSS.tar.gz` +Contains inner named tarballs: +- `config.tar.gz` — all `*.xml` files in `JENKINS_HOME` root (config.xml, credentials.xml, etc.) +- `users.tar.gz` — `users/` subtree +- `secrets.tar.gz` — `secrets/`, `.key`, `secret.key`, `secret.key.not-so-secret` +- `plugins.tar.gz` — `plugins/` (enables exact version restore) +- `nodes.tar.gz` — `nodes/` (agent XML definitions) +- `crontab.txt` — jenkins user crontab, plain text + +The inner-tarball-per-element design allows the restore script to skip individual elements +(e.g. `--skip plugins` to keep existing plugins when only restoring config). + +### Environment variables + +| Variable | Default | +|---|---| +| `JENKINS_HOME` | `/home/jenkins/.jenkins` | +| `JENKINS_USER` | `jenkins` | + +--- + +## Restore script design + +[`jenkins-scripts/restore-jenkins-app-config.sh`](jenkins-scripts/restore-jenkins-app-config.sh) + +### Flow + +1. Parse `--skip` and `--blank-oauth` flags +2. Source `restore-config-overrides.env` if present +3. Pre-flight: root check, backup file exists, jenkins user exists +4. Stop Jenkins service (if running) +5. Extract outer tarball to a temp dir +6. For each element (config, users, secrets, plugins, nodes): + - `config` is special: extracted to a staging subdir → overrides applied → `cp -a` into JENKINS_HOME + - All others: `tar -xzf` directly into JENKINS_HOME +7. Restore crontab (skip if file is comment-only) +8. `chown -R jenkins:jenkins $JENKINS_HOME` +9. If `--blank-oauth`: create `users/admin/config.xml` with bcrypt-hashed random password +10. `systemctl daemon-reload && systemctl start jenkins` +11. Poll for HTTP 200 on port 8080 +12. Print summary including admin password if `--blank-oauth` was used + +### Config override mechanism + +`restore-config-overrides.env` contains shell variables that are loaded before restoring. +The `apply_override ` helper performs an in-place `sed` +replacement on the staged file. Empty values are no-ops — a blank env file is safe. + +Fields that are always updated from env variables: + +| Variable | XML element | File | +|---|---|---| +| `JENKINS_URL` | `jenkinsUrl` | `jenkins.model.JenkinsLocationConfiguration.xml` | +| `JENKINS_URL` | `hudsonUrl` | `hudson.tasks.Mailer.xml` | +| `JENKINS_URL` | `logoPath` (URL prefix only) | `CustomHeaderConfiguration.xml` | +| `JENKINS_ADMIN_EMAIL` | `adminAddress` | `jenkins.model.JenkinsLocationConfiguration.xml` | +| `THINBACKUP_PATH` | `backupPath` | `org.jvnet.hudson.plugins.thinbackup.ThinBackupPluginImpl.xml` | +| `SLACK_TEAM_DOMAIN` | `teamDomain` | `jenkins.plugins.slack.SlackNotifier.xml` | +| `SLACK_DEFAULT_ROOM` | `room` | `jenkins.plugins.slack.SlackNotifier.xml` | + +Fields that are always **blanked unconditionally** (no env variable): +- Ansible Tower: ``, `` in `org.jenkinsci.plugins.ansible_tower.AnsibleTower.xml` +- Build queue (`queue.xml`) is always cleared to avoid restoring stale queued jobs + +### `--blank-oauth` behaviour + +Used when restoring to a different environment where the production GitHub OAuth app cannot be +reused (different URL, different allowed-callback domain). + +What it does: +1. Replaces `` with + `` via Python multiline regex +2. Clears `` and `` in `config.xml` +3. Grants `USER:hudson.model.Hudson.Administer:admin` to the local `admin` user in the authz matrix +4. Creates `users/admin/config.xml` with a bcrypt-hashed random 16-char password +5. Uses `users/admin/` (not a hashed subdir) — Jenkins migrates this to the HMAC-keyed path on startup + +The local admin user's password is printed in a box at the end of the restore output. **Change it +after first login.** + +--- + +## Vagrant environment + +[`Vagrantfile`](Vagrantfile): +- Box: `bento/ubuntu-24.04` +- RAM: 8192 MB, CPUs: 2, cpu_mode: `host-passthrough` +- Provider: `libvirt` (not VirtualBox) +- Port forward: guest 8080 → host 8080 (bound to 127.0.0.1) +- Sync: rsync, excluding `.git/` and `.vagrant/` +- Bootstrap: `apt update && apt upgrade -y`, optional `id_rsa.pub` → `authorized_keys`, reboot + +### Network helpers + +`Vagrant-Scripts/` contains libvirt network setup helpers. If `vagrant up` fails with +network errors, run `start-vagrant-networks.sh` first. + +### Known issues with the libvirt provider + +- The VM reboots during provisioning (kernel update). Wait for the reboot to complete before + running Ansible. +- If the default libvirt network is not started, use `start-networks.sh`. +- Use `cleanup-vagrant.sh` to fully destroy and remove all libvirt resources if the VM gets + into a broken state. + +--- + +## What remains to be done (from `CONFIG-ANALYSIS.md`) + +### Phase 1 — In progress / pending + +| Item | Status | +|---|---| +| Additional JDKs (Temurin 11, 17, 21) | ⚠ Not yet implemented in playbook | +| Multiple JDK versions in `JAVA_HOME` alternates | ⚠ Not yet implemented | +| Wazuh agent | ⚠ Optional — add when monitoring is required | +| InstallBuilder PATH entry | ⚠ Review if needed | + +### Phase 2 — After new server has DNS/IP + +| Item | Notes | +|---|---| +| Hostname + `/etc/hosts` | Configure with new server's IP | +| Nginx reverse proxy | Extract from production, update domains/certs | +| Backup mount | `//u158991.your-backup.de/backup` — new credentials needed | +| SSL certificates | Required for HTTPS reverse proxy | + +### Phase 3 — After Jenkins is running + +| Item | Notes | +|---|---| +| Cron: diskspace check | `08 08 * * * /home/jenkins/diskspace/rundiskspace` | +| Cron: queue depth log | Every 15 min | +| Root security cron | `0 5 * * 6 /root/apt-security.sh` | +| Monitoring integration | Nagios plugins already installed | + +--- + +## Testing and validation + +### Linting (run before every commit) + +```bash +cd /path/to/infrastructure +yamllint . # ~2 seconds +ansible-lint --offline # ~60 seconds +``` + +### Ansible syntax check + +```bash +cd jenkins-as-code/ansible +ansible-playbook setup-jenkins-host.yml --syntax-check +ansible-playbook install-jenkins-server.yml --syntax-check +``` + +### Dry run + +```bash +ansible-playbook -i inventory-vagrant.yml setup-jenkins-host.yml --check --diff +ansible-playbook -i inventory-vagrant.yml install-jenkins-server.yml --check --diff +``` + +### Selective role deployment + +```bash +# Only security roles +ansible-playbook setup-jenkins-host.yml --tags security + +# Only NTP +ansible-playbook setup-jenkins-host.yml --tags system + +# Only Java installation +ansible-playbook setup-jenkins-host.yml --tags java + +# Only Jenkins install and start +ansible-playbook install-jenkins-server.yml --tags jenkins_install,jenkins_start +``` + +--- + +## Commit message conventions + +For changes to this sub-project, prefix commits: + +- `jenkins-as-code:` — general changes to this sub-project +- `jenkins-as-code: ansible:` — playbook or role changes +- `jenkins-as-code: backup:` — backup/restore script changes +- `jenkins-as-code: vagrant:` — Vagrantfile or Vagrant-Scripts changes +- `jenkins-as-code: docs:` — documentation only + +For cross-cutting infrastructure changes, follow the conventions in +[`.github/copilot-instructions.md`](../.github/copilot-instructions.md). + +--- + +## Quick reference: key file paths + +| Purpose | Path | +|---|---| +| All defaults | `jenkins-as-code/ansible/group_vars/all.yml` | +| Step 1 playbook | `jenkins-as-code/ansible/setup-jenkins-host.yml` | +| Step 2 playbook | `jenkins-as-code/ansible/install-jenkins-server.yml` | +| Prod inventory | `jenkins-as-code/ansible/inventory-production.yml` | +| Vagrant inventory | `jenkins-as-code/ansible/inventory-vagrant.yml` | +| systemd unit template | `jenkins-as-code/ansible/templates/jenkins.service.j2` | +| Jenkins defaults template | `jenkins-as-code/ansible/templates/jenkins-defaults.j2` | +| Backup script | `jenkins-as-code/jenkins-scripts/backup-jenkins-app-config.sh` | +| Restore script | `jenkins-as-code/jenkins-scripts/restore-jenkins-app-config.sh` | +| Restore overrides | `jenkins-as-code/jenkins-scripts/restore-config-overrides.env` | +| Vagrantfile | `jenkins-as-code/Vagrantfile` | + +--- + +*Made with Bob* diff --git a/jenkins-as-code/ansible/group_vars/all.yml b/jenkins-as-code/ansible/group_vars/all.yml new file mode 100644 index 0000000000..451a5790ae --- /dev/null +++ b/jenkins-as-code/ansible/group_vars/all.yml @@ -0,0 +1,94 @@ +--- +########################## +# adoptopenjdk_variables # +########################## + +# Domain for setting hostname +Domain: adoptopenjdk.net + +# Sudoers file +Sudoers_File: /etc/sudoers + +# Jenkins User Variables: +Jenkins_Username: jenkins + +# Superuser Variables: +Superuser_Account: Enabled + +# Nagios Variables: +Nagios_Plugins: Enabled +Nagios_Monitoring: Enabled +Nagios_Master_IP: 78.47.239.96 + +# Security Variables: +Security: Enabled + +# JCK Variables: +jckftp_Username: jckftp + +# Vendor Variables: +Vendor_File: Disabled +Vendor_Playbook: /Vendor_Files/Vendor_Playbook/Vendor.yml + +# Default BootJDK installed +bootjdk: hotspot + +# Version of Ant used +ant_version: 1.10.15 +ant_checksum: sha512:1de7facbc9874fa4e5a2f045d5c659f64e0b89318c1dbc8acc6aae4595c4ffaf90a7b1ffb57f958dd08d6e086d3fff07aa90e50c77342a0aa5c9b4c36bff03a9 + +# GPG Public Keys +key: + curl: 27EDEAF22F3ABCEB50DB9A125CC908FDB71E12C2 # Daniel Stenberg + apache_ant: 0A123C1ED3F13A6A0140E166C71FB765CD9DE313 # Jaikiran Pai + apache_maven: B02137D875D833D9B23392ECAE5A7FB608A0221C # Robert Scholte + autoconf: A7A16B4A2527436A # Eric Blake + cmake: EC8FEF3A7BFB4EDA # Brad King + gmake: 96B047156338B6D4 # Paul Smith (Mad Scientist) + adoptium: 3B04D753C9050D9A5D343F39843C48A565F8F04B # Adoptium GPG Key (DEB/RPM Signing Key) + +############################################### +# Jenkins Infrastructure Variables +############################################### + +# --- Jenkins version & integrity --- +# Single source of truth. Update both values together when upgrading. +# Retrieve checksum: curl -fsSL https://get.jenkins.io/war-stable//jenkins.war.sha256 +jenkins_version: "2.555.3" +jenkins_war_sha256: "5d19905e6c0f23aff89ff007de5564b96e0a05c13f4d1a92d0fdcb69b033bb9a" + +# --- Java --- +# Temurin major version. Must be supported by the Jenkins version above. +java_major_version: "25" +java_package: "temurin-{{ java_major_version }}-jdk" +java_home: "/usr/lib/jvm/temurin-{{ java_major_version }}-jdk-amd64" + +# --- Jenkins OS user & directory layout --- +jenkins_username: jenkins +jenkins_home: /home/jenkins +jenkins_data_dir: /home/jenkins/.jenkins + +# --- Jenkins service --- +jenkins_port: 8080 +jenkins_install_dir: /opt/jenkins +jenkins_war_url: "https://get.jenkins.io/war-stable/{{ jenkins_version }}/jenkins.war" +jenkins_war_path: "{{ jenkins_install_dir }}/jenkins.war" + +# --- Jenkins JVM tuning --- +# Override jenkins_heap_size per-host in inventory (e.g. "2G", "4G", "19G"). +# Default "auto" selects a heap based on available RAM at run time. +jenkins_heap_size: "auto" + +# --- Jenkins runtime arguments --- +jenkins_session_timeout: "--sessionTimeout=720 --sessionEviction=43200" +jenkins_accesslog: "--accessLoggerClassName=winstone.accesslog.SimpleAccessLogger --simpleAccessLogger.format=combined --simpleAccessLogger.file=/var/log/jenkins/access.log" + +# Default listen address used on non-Vagrant hosts. +# The playbook auto-detects Vagrant (/vagrant exists) and overrides this to 0.0.0.0. +jenkins_listen_address: "127.0.0.1" + +# --- Jenkins backup / restore --- +# Filename of the application-config backup tarball (relative to the data/ directory +# alongside this repository). Used by restore-jenkins-app-config.yml. +jenkins_backup_file: "jenkins-app-backup-20260706-113355.tar.gz" + diff --git a/jenkins-as-code/ansible/install-jenkins-server.yml b/jenkins-as-code/ansible/install-jenkins-server.yml new file mode 100644 index 0000000000..b2c727cdff --- /dev/null +++ b/jenkins-as-code/ansible/install-jenkins-server.yml @@ -0,0 +1,376 @@ +--- +############################################### +# Jenkins Server Installation - Ansible Playbook +############################################### +# This playbook installs Jenkins server on Ubuntu 24.04 +# with the Jenkins home directory at /home/jenkins/.jenkins +# to mirror the production Jenkins master configuration. +# +# Prerequisites: +# - Ubuntu 24.04 system +# - Jenkins user already created (run setup-jenkins-host.yml first) +# - Java 17 or later installed +# +# Usage: +# ansible-playbook install-jenkins-server.yml --connection=local + +- name: Install Jenkins Server + hosts: localhost + connection: local + become: yes + gather_facts: yes + + tasks: + ############################################### + # Pre-flight Checks + ############################################### + - name: Verify we're running on Ubuntu 24.04 + assert: + that: + - ansible_distribution == "Ubuntu" + - ansible_distribution_version == "24.04" + fail_msg: "This playbook is designed for Ubuntu 24.04" + success_msg: "Running on Ubuntu 24.04" + tags: always + + - name: Check if Jenkins user exists + command: id {{ jenkins_username }} + register: jenkins_user_check + failed_when: false + changed_when: false + tags: always + + - name: Fail if Jenkins user does not exist + fail: + msg: "Jenkins user does not exist. Please run setup-jenkins-host.yml first." + when: jenkins_user_check.rc != 0 + tags: always + + - name: Check Java installation + command: java -version + register: java_check + failed_when: false + changed_when: false + tags: always + + - name: Fail if Java is not installed + fail: + msg: "Java is not installed. Please run setup-jenkins-host.yml first to install Java." + when: java_check.rc != 0 + tags: always + + ############################################### + # Detect Environment (Vagrant vs Production) + ############################################### + - name: Check if running inside a Vagrant box + stat: + path: /vagrant + register: vagrant_dir + tags: always + + - name: Set listen address based on environment + set_fact: + effective_listen_address: "{{ '0.0.0.0' if vagrant_dir.stat.exists else jenkins_listen_address }}" + tags: always + + - name: Display environment detection + debug: + msg: "Environment: {{ 'Vagrant (listen: 0.0.0.0)' if vagrant_dir.stat.exists else 'Production (listen: ' + jenkins_listen_address + ')' }}" + tags: always + + ############################################### + # Detect System Memory and Calculate Heap Size + ############################################### + - name: Get total system memory in MB + set_fact: + total_memory_mb: "{{ (ansible_memtotal_mb | int) }}" + tags: always + + - name: Calculate appropriate heap size based on system memory + set_fact: + calculated_heap_size: >- + {%- if jenkins_heap_size != 'auto' -%} + {{ jenkins_heap_size }} + {%- elif total_memory_mb | int >= 30720 -%} + 19G + {%- elif total_memory_mb | int >= 14336 -%} + 8G + {%- elif total_memory_mb | int >= 6144 -%} + 4G + {%- else -%} + 2G + {%- endif -%} + tags: always + + - name: Display memory configuration + debug: + msg: + - "System Memory: {{ total_memory_mb }}MB" + - "Jenkins Heap Size: {{ calculated_heap_size }}" + - "Configuration: {{ 'Manual override' if jenkins_heap_size != 'auto' else 'Auto-detected' }}" + tags: always + + - name: Build Jenkins JAVA_ARGS with calculated heap size + set_fact: + jenkins_java_opts: "-Dhudson.tasks.junit.TestResultAction.RESULT_CACHE_ENABLED=false -Dhudson.tasks.junit.History\\$HistoryTableResult.PREVIOUS_TEST_RESULT_BACKTRACK_BUILDS_MAX=1 -Djava.awt.headless=true -Xmx{{ calculated_heap_size }} -Dhudson.util.XStream2.collectionUpdateLimit=-1 -Xlog:gc*,gc+heap=info,gc+age=trace,gc+phases=trace,safepoint:file=/var/log/jenkins/gc.log:time,uptime,level,tags:filecount=5,filesize=50m" + tags: always + + ############################################### + # Download and Verify Jenkins WAR + ############################################### + - name: Install required packages for Jenkins + apt: + name: + - curl + - fontconfig + - daemon + state: present + tags: jenkins_install + + - name: Create Jenkins installation directory + file: + path: "{{ jenkins_install_dir }}" + state: directory + owner: root + group: root + mode: '0755' + tags: jenkins_install + + - name: Download Jenkins WAR file + get_url: + url: "{{ jenkins_war_url }}" + dest: "{{ jenkins_war_path }}" + mode: '0644' + owner: root + group: root + timeout: 300 + register: jenkins_war_download + tags: jenkins_install + + - name: Calculate actual SHA256 checksum of downloaded WAR + stat: + path: "{{ jenkins_war_path }}" + checksum_algorithm: sha256 + get_checksum: yes + register: actual_checksum + tags: jenkins_install + + - name: Verify Jenkins WAR checksum + assert: + that: + - actual_checksum.stat.checksum == jenkins_war_sha256 + fail_msg: "Jenkins WAR checksum verification failed! Expected: {{ jenkins_war_sha256 }}, Got: {{ actual_checksum.stat.checksum }}" + success_msg: "Jenkins WAR checksum verified successfully" + tags: jenkins_install + + - name: Display Jenkins version information + debug: + msg: + - "Jenkins Version: {{ jenkins_version }}" + - "WAR Location: {{ jenkins_war_path }}" + - "SHA256 Checksum: {{ actual_checksum.stat.checksum }}" + tags: jenkins_install + + ############################################### + # Configure Jenkins Home Directory + ############################################### + - name: Stop Jenkins service if running + systemd: + name: jenkins + state: stopped + failed_when: false + tags: jenkins_config + + - name: Create Jenkins data directory + file: + path: "{{ jenkins_data_dir }}" + state: directory + owner: "{{ jenkins_username }}" + group: "{{ jenkins_username }}" + mode: '0755' + tags: jenkins_config + + - name: Create Jenkins cache directory + file: + path: /var/cache/jenkins + state: directory + owner: "{{ jenkins_username }}" + group: "{{ jenkins_username }}" + mode: '0755' + tags: jenkins_config + + - name: Create Jenkins log directory + file: + path: /var/log/jenkins + state: directory + owner: "{{ jenkins_username }}" + group: "{{ jenkins_username }}" + mode: '0755' + tags: jenkins_config + + - name: Install Jenkins logrotate config + template: + src: templates/jenkins-logrotate.j2 + dest: /etc/logrotate.d/jenkins + owner: root + group: root + mode: '0644' + tags: jenkins_config + + ############################################### + # Configure Jenkins Service + ############################################### + - name: Configure Jenkins defaults file + template: + src: templates/jenkins-defaults.j2 + dest: /etc/default/jenkins + owner: root + group: root + mode: '0644' + backup: yes + tags: jenkins_config + + - name: Create Jenkins systemd service file + template: + src: templates/jenkins.service.j2 + dest: /etc/systemd/system/jenkins.service + owner: root + group: root + mode: '0644' + tags: jenkins_config + + - name: Reload systemd daemon + systemd: + daemon_reload: yes + tags: jenkins_config + + ############################################### + # Configure Jenkins Plugins Directory + ############################################### + - name: Create Jenkins plugins directory + file: + path: "{{ jenkins_data_dir }}/plugins" + state: directory + owner: "{{ jenkins_username }}" + group: "{{ jenkins_username }}" + mode: '0755' + tags: jenkins_config + + - name: Create Jenkins updates directory + file: + path: "{{ jenkins_data_dir }}/updates" + state: directory + owner: "{{ jenkins_username }}" + group: "{{ jenkins_username }}" + mode: '0755' + tags: jenkins_config + + - name: Create Jenkins jobs directory + file: + path: "{{ jenkins_data_dir }}/jobs" + state: directory + owner: "{{ jenkins_username }}" + group: "{{ jenkins_username }}" + mode: '0755' + tags: jenkins_config + + - name: Create Jenkins workspace directory + file: + path: "{{ jenkins_data_dir }}/workspace" + state: directory + owner: "{{ jenkins_username }}" + group: "{{ jenkins_username }}" + mode: '0755' + tags: jenkins_config + + - name: Suppress Jenkins setup wizard (lastExecVersion) + copy: + content: "{{ jenkins_version }}" + dest: "{{ jenkins_data_dir }}/jenkins.install.InstallUtil.lastExecVersion" + owner: "{{ jenkins_username }}" + group: "{{ jenkins_username }}" + mode: '0644' + tags: jenkins_config + + - name: Suppress Jenkins setup wizard (UpgradeWizard state) + copy: + content: "done" + dest: "{{ jenkins_data_dir }}/jenkins.install.UpgradeWizard.state" + owner: "{{ jenkins_username }}" + group: "{{ jenkins_username }}" + mode: '0644' + tags: jenkins_config + + ############################################### + # Start Jenkins Service + ############################################### + - name: Enable and start Jenkins service + systemd: + name: jenkins + state: started + enabled: yes + tags: jenkins_start + + - name: Wait for Jenkins to start + wait_for: + port: "{{ jenkins_port }}" + delay: 10 + timeout: 300 + tags: jenkins_start + + - name: Wait for Jenkins to be fully ready + uri: + url: "http://localhost:{{ jenkins_port }}/login" + status_code: 200 + timeout: 5 + register: jenkins_ready + until: jenkins_ready.status == 200 + retries: 30 + delay: 10 + tags: jenkins_start + + ############################################### + # Retrieve Initial Admin Password + ############################################### + - name: Check if initial admin password file exists + stat: + path: "{{ jenkins_data_dir }}/secrets/initialAdminPassword" + register: initial_password_file + tags: jenkins_start + + - name: Read initial admin password + slurp: + src: "{{ jenkins_data_dir }}/secrets/initialAdminPassword" + register: initial_password + when: initial_password_file.stat.exists + tags: jenkins_start + + - name: Display Jenkins installation information + debug: + msg: + - "==========================================" + - "Jenkins Installation Complete!" + - "==========================================" + - "Jenkins URL: http://{{ ansible_default_ipv4.address }}:{{ jenkins_port }}" + - "Jenkins Home: {{ jenkins_data_dir }}" + - "Jenkins User: {{ jenkins_username }}" + - "" + - "Initial Admin Password: {{ initial_password.content | b64decode | trim if initial_password_file.stat.exists else 'not found' }}" + - "" + - "Note: Setup wizard has been suppressed." + - "Jenkins will start directly to the login page." + - "==========================================" + tags: jenkins_start + + - name: Display Jenkins service status + command: systemctl status jenkins --no-pager + register: jenkins_status + changed_when: false + tags: jenkins_start + + - name: Show Jenkins service status + debug: + var: jenkins_status.stdout_lines + tags: jenkins_start + +# Made with Bob \ No newline at end of file diff --git a/jenkins-as-code/ansible/inventory-example.yml b/jenkins-as-code/ansible/inventory-example.yml new file mode 100644 index 0000000000..5330a1b44f --- /dev/null +++ b/jenkins-as-code/ansible/inventory-example.yml @@ -0,0 +1,49 @@ +--- +# Example Inventory File for Jenkins Infrastructure +# Copy this to inventory.yml and customize for your environment + +all: + hosts: + jenkins-master: + ansible_host: 192.168.1.100 + ansible_user: root + ansible_ssh_private_key_file: ~/.ssh/jenkins_master_key + + # Fail2ban IP Whitelist Configuration + # Default includes production Jenkins master whitelist + # IMPORTANT: Review and update for your specific environment + fail2ban_ignoreip: >- + 127.0.0.1/8 + ::1 + 10.0.0.0/8 + 192.168.0.0/16 + 78.47.239.96 + 46.224.123.39 + 178.62.115.224 + 20.90.182.165 + + # NTP Configuration (optional overrides) + # ntp_service: ntpsec # Use 'ntp' for older Ubuntu versions + # ntp_package: ntpsec # Use 'ntp' for older Ubuntu versions + + vars: + # Global variables for all hosts + ansible_python_interpreter: /usr/bin/python3 + + # Jenkins user configuration + jenkins_username: jenkins + jenkins_home: /home/jenkins + jenkins_ssh_key: "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC... jenkins@adoptopenjdk" + +# Example: Multiple Jenkins servers with different whitelists +# jenkins_servers: +# hosts: +# jenkins-master-prod: +# ansible_host: 10.0.1.100 +# fail2ban_ignoreip: "127.0.0.1/8 ::1 10.0.0.0/8 203.0.113.0/24" +# +# jenkins-master-staging: +# ansible_host: 10.0.2.100 +# fail2ban_ignoreip: "127.0.0.1/8 ::1 10.0.0.0/8 198.51.100.0/24" + +# Made with Bob diff --git a/jenkins-as-code/ansible/inventory-production.yml b/jenkins-as-code/ansible/inventory-production.yml new file mode 100644 index 0000000000..6be04239f5 --- /dev/null +++ b/jenkins-as-code/ansible/inventory-production.yml @@ -0,0 +1,22 @@ +--- +# Production Jenkins Server Inventory +# Use this for production deployments with full resources + +all: + hosts: + jenkins-production: + ansible_host: localhost + ansible_connection: local + + # Fail2ban IP Whitelist - Production trusted IPs + fail2ban_ignoreip: "127.0.0.1/8 ::1 10.0.0.0/8 192.168.0.0/16 78.47.239.96 46.224.123.39 178.62.115.224 20.90.182.165" + + # NTP configuration + ntp_service: ntpsec + ntp_package: ntpsec + + vars: + # Environment identifier + jenkins_environment: production + +# Made with Bob diff --git a/jenkins-as-code/ansible/inventory-vagrant.yml b/jenkins-as-code/ansible/inventory-vagrant.yml new file mode 100644 index 0000000000..525b76c7ee --- /dev/null +++ b/jenkins-as-code/ansible/inventory-vagrant.yml @@ -0,0 +1,22 @@ +--- +# Vagrant/Development Jenkins Server Inventory +# Use this for local development and testing with limited resources + +all: + hosts: + jenkins-vagrant: + ansible_host: localhost + ansible_connection: local + + # Fail2ban IP Whitelist - Local networks only + fail2ban_ignoreip: "127.0.0.1/8 ::1 10.0.0.0/8 192.168.0.0/16 172.16.0.0/12" + + # NTP configuration + ntp_service: ntpsec + ntp_package: ntpsec + + vars: + # Environment identifier + jenkins_environment: development + +# Made with Bob diff --git a/jenkins-as-code/ansible/roles/README.md b/jenkins-as-code/ansible/roles/README.md new file mode 100644 index 0000000000..eeab4acf3d --- /dev/null +++ b/jenkins-as-code/ansible/roles/README.md @@ -0,0 +1,190 @@ +# Ansible Roles for Jenkins Infrastructure + +This directory contains Ansible roles for configuring Jenkins infrastructure based on the existing production Jenkins master configuration. + +## Available Roles + +### 1. unattended_upgrades +**Purpose:** Configure security-only automatic updates for Ubuntu systems. + +**What it does:** +- Installs the `unattended-upgrades` package +- Configures `/etc/apt/apt.conf.d/50unattended-upgrades` with security-only updates +- Sets up `/etc/apt/apt.conf.d/20auto-upgrades` (disabled by default) + +**Configuration:** +The role is configured to only install security updates from: +- `${distro_id}:${distro_codename}-security` +- Extended Security Maintenance (ESM) updates if available + +**Note:** Automatic updates are disabled by default in `20auto-upgrades`. To enable: +```bash +# Edit /etc/apt/apt.conf.d/20auto-upgrades and change: +APT::Periodic::Update-Package-Lists "1"; +APT::Periodic::Unattended-Upgrade "1"; +``` + +**Tags:** `security` + +--- + +### 2. ntp_config +**Purpose:** Configure time synchronization with Ubuntu NTP pools. + +**What it does:** +- Installs NTP package (ntpsec for Ubuntu 20.04+, ntp for older versions) +- Configures `/etc/ntp.conf` with Ubuntu pool servers +- Enables and starts the NTP service + +**NTP Servers configured:** +- `0.ubuntu.pool.ntp.org` +- `1.ubuntu.pool.ntp.org` +- `2.ubuntu.pool.ntp.org` +- `3.ubuntu.pool.ntp.org` +- `ntp.ubuntu.com` (fallback) + +**Variables:** +- `ntp_service`: Service name (default: `ntpsec`) +- `ntp_package`: Package name (default: `ntpsec`) + +**Tags:** `system` + +--- + +### 3. fail2ban +**Purpose:** Configure fail2ban for SSH protection with IP whitelisting. + +**What it does:** +- Installs fail2ban package +- Configures `/etc/fail2ban/jail.local` with SSH protection +- Sets up repeat offender detection (recidive jail) +- Enables and starts the fail2ban service + +**Features:** +- **SSH Protection:** Bans IPs after 3 failed login attempts within 10 minutes +- **Progressive Banning:** Ban time doubles for repeat offenders (1h → 2h → 4h, max 1 week) +- **Repeat Offender Jail:** IPs banned 5+ times in 24 hours get a 7-day ban +- **IP Whitelisting:** Trusted IPs/networks are never banned + +**Default Configuration:** +- `findtime`: 10 minutes +- `maxretry`: 3 attempts +- `bantime`: 1 hour (escalates for repeat offenders) +- `backend`: systemd (for modern Ubuntu) + +**Variables:** +- `fail2ban_ignoreip`: Space-separated list of IPs/networks to whitelist + - Default: `127.0.0.1/8 ::1 10.0.0.0/8 192.168.0.0/16` + +**IMPORTANT:** Update the `fail2ban_ignoreip` variable with your trusted IPs before deployment! + +**Example - Setting custom whitelist:** +```yaml +# In your playbook or inventory +vars: + fail2ban_ignoreip: "127.0.0.1/8 ::1 10.0.0.0/8 192.168.0.0/16 203.0.113.0/24 198.51.100.50" +``` + +**Tags:** `security` + +--- + +## Usage + +### In Playbook +```yaml +- name: Configure Jenkins Host + hosts: jenkins_servers + become: yes + roles: + - role: unattended_upgrades + tags: security + - role: ntp_config + tags: system + - role: fail2ban + tags: security + vars: + fail2ban_ignoreip: "127.0.0.1/8 ::1 10.0.0.0/8 YOUR_OFFICE_IP" +``` + +### Run Specific Roles +```bash +# Run only security roles +ansible-playbook setup-jenkins-host.yml --tags security + +# Run only NTP configuration +ansible-playbook setup-jenkins-host.yml --tags system + +# Run all roles +ansible-playbook setup-jenkins-host.yml +``` + +--- + +## Configuration Source + +These roles are based on the configuration extracted from the existing Jenkins master server: +- **Extracted on:** 2026-06-24 +- **Source:** `jenkins-as-code/data/jenkins-master-configs-20260624-180156/` + +The configurations match the production Jenkins master to ensure consistency across the infrastructure. + +--- + +## Verification + +### Check unattended-upgrades status +```bash +sudo systemctl status unattended-upgrades +sudo cat /etc/apt/apt.conf.d/50unattended-upgrades +sudo cat /etc/apt/apt.conf.d/20auto-upgrades +``` + +### Check NTP status +```bash +sudo systemctl status ntpsec # or ntp on older systems +ntpq -p # Show NTP peers +timedatectl status # Show time sync status +``` + +### Check fail2ban status +```bash +sudo systemctl status fail2ban +sudo fail2ban-client status # Show all jails +sudo fail2ban-client status sshd # Show SSH jail details +sudo fail2ban-client status recidive # Show repeat offender jail +``` + +--- + +## Security Notes + +1. **Unattended Upgrades:** Disabled by default. Enable only after testing in your environment. +2. **Fail2ban Whitelist:** Always include your management IPs to avoid locking yourself out. +3. **NTP:** Proper time synchronization is critical for security (SSL/TLS, Kerberos, logs). + +--- + +## Troubleshooting + +### Fail2ban not banning +- Check logs: `sudo tail -f /var/log/fail2ban.log` +- Verify backend: `sudo fail2ban-client get sshd backend` +- Test regex: `sudo fail2ban-regex /var/log/auth.log /etc/fail2ban/filter.d/sshd.conf` + +### NTP not syncing +- Check peers: `ntpq -p` +- Check system time: `timedatectl` +- Verify network connectivity to NTP servers + +### Unattended upgrades not running +- Check timer: `sudo systemctl status apt-daily-upgrade.timer` +- Check logs: `sudo cat /var/log/unattended-upgrades/unattended-upgrades.log` +- Verify configuration: `sudo unattended-upgrade --dry-run --debug` + +--- + +## Related Documentation + +- [CONFIG-ANALYSIS.md](../../CONFIG-ANALYSIS.md) - Analysis of extracted Jenkins master configuration +- [setup-jenkins-host.yml](../setup-jenkins-host.yml) - Main playbook using these roles \ No newline at end of file diff --git a/jenkins-as-code/ansible/roles/fail2ban/defaults/main.yml b/jenkins-as-code/ansible/roles/fail2ban/defaults/main.yml new file mode 100644 index 0000000000..88dc1ea253 --- /dev/null +++ b/jenkins-as-code/ansible/roles/fail2ban/defaults/main.yml @@ -0,0 +1,21 @@ +--- +# Default variables for fail2ban role + +# IP addresses and networks that should NEVER be banned +# Format: space-separated list of IPs/CIDR ranges +# This matches the production Jenkins master configuration +# Includes: +# - 127.0.0.1/8 ::1 (loopback) +# - 10.0.0.0/8 192.168.0.0/16 (private networks) +# - 78.47.239.96 46.224.123.39 178.62.115.224 20.90.182.165 (production trusted IPs) +# +# IMPORTANT: Review and update this list for your specific environment +fail2ban_ignoreip: "127.0.0.1/8 ::1 10.0.0.0/8 192.168.0.0/16 78.47.239.96 46.224.123.39 178.62.115.224 20.90.182.165" + +# NTP service name (varies by Ubuntu version) +# Ubuntu 20.04+: ntpsec +# Older versions: ntp +ntp_service: "ntpsec" +ntp_package: "ntpsec" + +# Made with Bob diff --git a/jenkins-as-code/ansible/roles/fail2ban/handlers/main.yml b/jenkins-as-code/ansible/roles/fail2ban/handlers/main.yml new file mode 100644 index 0000000000..1f7ddabf7e --- /dev/null +++ b/jenkins-as-code/ansible/roles/fail2ban/handlers/main.yml @@ -0,0 +1,10 @@ +--- +# Handlers for fail2ban configuration + +- name: restart fail2ban + systemd: + name: fail2ban + state: restarted + become: yes + +# Made with Bob diff --git a/jenkins-as-code/ansible/roles/fail2ban/tasks/main.yml b/jenkins-as-code/ansible/roles/fail2ban/tasks/main.yml new file mode 100644 index 0000000000..e70e6e9339 --- /dev/null +++ b/jenkins-as-code/ansible/roles/fail2ban/tasks/main.yml @@ -0,0 +1,122 @@ +--- +# Fail2ban configuration role - SSH protection with IP whitelisting +# Based on existing Jenkins master configuration + +- name: Install fail2ban package + apt: + name: fail2ban + state: present + update_cache: yes + become: yes + +- name: Create fail2ban jail.local configuration + copy: + dest: /etc/fail2ban/jail.local + content: | + ############################################################################### + # Fail2Ban local configuration + # + # This file overrides package defaults in jail.conf. + # It is safe from package upgrades and is the recommended place for + # site-specific configuration. + ############################################################################### + + [DEFAULT] + + # --------------------------------------------------------------------------- + # LOG BACKEND + # --------------------------------------------------------------------------- + # Use systemd/journald for log reading. + # Required on modern Ubuntu where authentication logs are in journald. + backend = systemd + + # --------------------------------------------------------------------------- + # FAILURE DETECTION + # --------------------------------------------------------------------------- + # Time window in which failures are counted + # Example: maxretry=3 means "3 failures within 10 minutes" + findtime = 10m + + # Number of failures allowed within findtime before banning + maxretry = 3 + + # --------------------------------------------------------------------------- + # BAN DURATION + # --------------------------------------------------------------------------- + # Initial ban time for first-time offenders + bantime = 1h + + # Escalate ban time for repeat offenders + # Each repeat ban multiplies bantime by bantime.factor + bantime.increment = true + bantime.factor = 2 + + # Maximum ban time cap to prevent runaway values + bantime.max = 1w + + # --------------------------------------------------------------------------- + # IP EXCLUSIONS + # --------------------------------------------------------------------------- + # IPs/networks that should NEVER be banned + # Always include loopback and trusted networks (VPNs, office IPs, etc.) + ignoreip = {{ fail2ban_ignoreip | default('127.0.0.1/8 ::1 10.0.0.0/8 192.168.0.0/16') }} + + # --------------------------------------------------------------------------- + # DNS HANDLING + # --------------------------------------------------------------------------- + # Disable reverse DNS lookups for speed and reliability + usedns = no + + ############################################################################### + # SSH PROTECTION + ############################################################################### + + [sshd] + + # Enable protection for the OpenSSH daemon + enabled = true + + # Protect the SSH service port + # "ssh" resolves to port 22 via /etc/services + # Change or extend if you use non-standard ports + port = ssh + + ############################################################################### + # REPEAT OFFENDER JAIL (RECIDIVE) + ############################################################################### + # Bans IPs that are repeatedly banned across any jails + + [recidive] + + # Enable repeat-offender detection + enabled = true + + # Fail2Ban's own log file + logpath = /var/log/fail2ban.log + + # If an IP is banned this many times... + maxretry = 5 + + # ...within this time window... + findtime = 1d + + # ...apply a longer ban + bantime = 7d + owner: root + group: root + mode: '0644' + become: yes + notify: restart fail2ban + +- name: Ensure fail2ban service is enabled and started + systemd: + name: fail2ban + enabled: yes + state: started + become: yes + +- name: Display fail2ban status + debug: + msg: "Fail2ban installed and configured with SSH protection and repeat offender detection" + +# Made with Bob diff --git a/jenkins-as-code/ansible/roles/ntp_config/defaults/main.yml b/jenkins-as-code/ansible/roles/ntp_config/defaults/main.yml new file mode 100644 index 0000000000..3ccaa2b34e --- /dev/null +++ b/jenkins-as-code/ansible/roles/ntp_config/defaults/main.yml @@ -0,0 +1,10 @@ +--- +# Default variables for NTP configuration role + +# NTP service name (varies by Ubuntu version) +# Ubuntu 20.04+: ntpsec +# Older versions: ntp +ntp_service: "ntpsec" +ntp_package: "ntpsec" + +# Made with Bob diff --git a/jenkins-as-code/ansible/roles/ntp_config/handlers/main.yml b/jenkins-as-code/ansible/roles/ntp_config/handlers/main.yml new file mode 100644 index 0000000000..313348c978 --- /dev/null +++ b/jenkins-as-code/ansible/roles/ntp_config/handlers/main.yml @@ -0,0 +1,10 @@ +--- +# Handlers for NTP configuration + +- name: restart ntp + systemd: + name: "{{ ntp_service | default('ntpsec') }}" + state: restarted + become: yes + +# Made with Bob diff --git a/jenkins-as-code/ansible/roles/ntp_config/tasks/main.yml b/jenkins-as-code/ansible/roles/ntp_config/tasks/main.yml new file mode 100644 index 0000000000..2cd5bc2ce4 --- /dev/null +++ b/jenkins-as-code/ansible/roles/ntp_config/tasks/main.yml @@ -0,0 +1,89 @@ +--- +# NTP configuration role - Time synchronization with Ubuntu NTP pools +# Based on existing Jenkins master configuration + +- name: Install NTP package (ntpsec or ntp) + apt: + name: "{{ ntp_package | default('ntpsec') }}" + state: present + update_cache: yes + become: yes + +- name: Configure NTP with Ubuntu pool servers + copy: + dest: /etc/ntp.conf + content: | + # /etc/ntp.conf, configuration for ntpd; see ntp.conf(5) for help + + driftfile /var/lib/ntp/ntp.drift + + # Leap seconds definition provided by tzdata + leapfile /usr/share/zoneinfo/leap-seconds.list + + # Enable this if you want statistics to be logged. + #statsdir /var/log/ntpstats/ + + statistics loopstats peerstats clockstats + filegen loopstats file loopstats type day enable + filegen peerstats file peerstats type day enable + filegen clockstats file clockstats type day enable + + # Specify one or more NTP servers. + + # Use servers from the NTP Pool Project. Approved by Ubuntu Technical Board + # on 2011-02-08 (LP: #104525). See http://www.pool.ntp.org/join.html for + # more information. + pool 0.ubuntu.pool.ntp.org iburst + pool 1.ubuntu.pool.ntp.org iburst + pool 2.ubuntu.pool.ntp.org iburst + pool 3.ubuntu.pool.ntp.org iburst + + # Use Ubuntu's ntp server as a fallback. + pool ntp.ubuntu.com + + # Access control configuration; see /usr/share/doc/ntp-doc/html/accopt.html for + # details. The web page + # might also be helpful. + # + # Note that "restrict" applies to both servers and clients, so a configuration + # that might be intended to block requests from certain clients could also end + # up blocking replies from your own upstream servers. + + # By default, exchange time with everybody, but don't allow configuration. + restrict -4 default kod notrap nomodify nopeer noquery limited + restrict -6 default kod notrap nomodify nopeer noquery limited + + # Local users may interrogate the ntp server more closely. + restrict 127.0.0.1 + restrict ::1 + + # Needed for adding pool entries + restrict source notrap nomodify noquery + + # Clients from this (example!) subnet have unlimited access, but only if + # cryptographically authenticated. + #restrict 192.168.123.0 mask 255.255.255.0 notrust + + + # If you want to provide time to your local subnet, change the next line. + # (Again, the address is an example only.) + #broadcast 192.168.123.255 + + # If you want to listen to time broadcasts on your local subnet, de-comment the + # next lines. Please do this only if you trust everybody on the network! + #disable auth + #broadcastclient + owner: root + group: root + mode: '0644' + become: yes + notify: restart ntp + +- name: Ensure NTP service is enabled and started + systemd: + name: "{{ ntp_service | default('ntpsec') }}" + enabled: yes + state: started + become: yes + +# Made with Bob diff --git a/jenkins-as-code/ansible/roles/system_update/tasks/main.yml b/jenkins-as-code/ansible/roles/system_update/tasks/main.yml new file mode 100644 index 0000000000..7ba24aeca8 --- /dev/null +++ b/jenkins-as-code/ansible/roles/system_update/tasks/main.yml @@ -0,0 +1,42 @@ +--- +############################################### +# System Update Role - Main Tasks +############################################### +# This role performs system updates using apt +# It runs apt update and apt upgrade as the first role + +- name: Update apt package cache + apt: + update_cache: yes + cache_valid_time: 0 + register: apt_update_result + retries: 3 + delay: 5 + until: apt_update_result is succeeded + tags: + - system_update + - apt_update + +- name: Upgrade all apt packages + apt: + upgrade: dist + update_cache: no + autoremove: yes + autoclean: yes + register: apt_upgrade_result + retries: 3 + delay: 5 + until: apt_upgrade_result is succeeded + tags: + - system_update + - apt_upgrade + +- name: Display system update results + debug: + msg: + - "System update completed successfully" + - "Packages updated: {{ apt_upgrade_result.changed }}" + tags: + - system_update + +# Made with Bob \ No newline at end of file diff --git a/jenkins-as-code/ansible/roles/unattended_upgrades/tasks/main.yml b/jenkins-as-code/ansible/roles/unattended_upgrades/tasks/main.yml new file mode 100644 index 0000000000..a0085ad382 --- /dev/null +++ b/jenkins-as-code/ansible/roles/unattended_upgrades/tasks/main.yml @@ -0,0 +1,179 @@ +--- +# Unattended-upgrades role - Security-only automatic updates +# Based on existing Jenkins master configuration + +- name: Install unattended-upgrades package + apt: + name: unattended-upgrades + state: present + update_cache: yes + become: yes + +- name: Configure unattended-upgrades - 50unattended-upgrades + copy: + dest: /etc/apt/apt.conf.d/50unattended-upgrades + content: | + // Automatically upgrade packages from these (origin:archive) pairs + // + // Note that in Ubuntu security updates may pull in new dependencies + // from non-security sources (e.g. chromium). By allowing the release + // pocket these get automatically pulled in. + Unattended-Upgrade::Allowed-Origins { + "${distro_id}:${distro_codename}"; + "${distro_id}:${distro_codename}-security"; + // Extended Security Maintenance; doesn't necessarily exist for + // every release and this system may not have it installed, but if + // available, the policy for updates is such that unattended-upgrades + // should also install from here by default. + "${distro_id}ESMApps:${distro_codename}-apps-security"; + "${distro_id}ESM:${distro_codename}-infra-security"; + // "${distro_id}:${distro_codename}-updates"; + // "${distro_id}:${distro_codename}-proposed"; + // "${distro_id}:${distro_codename}-backports"; + }; + + // Python regular expressions, matching packages to exclude from upgrading + Unattended-Upgrade::Package-Blacklist { + // The following matches all packages starting with linux- + // "linux-"; + + // Use $ to explicitely define the end of a package name. Without + // the $, "libc6" would match all of them. + // "libc6$"; + // "libc6-dev$"; + // "libc6-i686$"; + + // Special characters need escaping + // "libstdc\+\+6$"; + + // The following matches packages like xen-system-amd64, xen-utils-4.1, + // xenstore-utils and libxenstore3.0 + // "(lib)?xen(store)?"; + + // For more information about Python regular expressions, see + // https://docs.python.org/3/howto/regex.html + }; + + // This option controls whether the development release of Ubuntu will be + // upgraded automatically. Valid values are "true", "false", and "auto". + Unattended-Upgrade::DevRelease "auto"; + + // This option allows you to control if on a unclean dpkg exit + // unattended-upgrades will automatically run + // dpkg --force-confold --configure -a + // The default is true, to ensure updates keep getting installed + //Unattended-Upgrade::AutoFixInterruptedDpkg "true"; + + // Split the upgrade into the smallest possible chunks so that + // they can be interrupted with SIGTERM. This makes the upgrade + // a bit slower but it has the benefit that shutdown while a upgrade + // is running is possible (with a small delay) + //Unattended-Upgrade::MinimalSteps "true"; + + // Install all updates when the machine is shutting down + // instead of doing it in the background while the machine is running. + // This will (obviously) make shutdown slower. + // Unattended-upgrades increases logind's InhibitDelayMaxSec to 30s. + // This allows more time for unattended-upgrades to shut down gracefully + // or even install a few packages in InstallOnShutdown mode, but is still a + // big step back from the 30 minutes allowed for InstallOnShutdown previously. + // Users enabling InstallOnShutdown mode are advised to increase + // InhibitDelayMaxSec even further, possibly to 30 minutes. + //Unattended-Upgrade::InstallOnShutdown "false"; + + // Send email to this address for problems or packages upgrades + // If empty or unset then no email is sent, make sure that you + // have a working mail setup on your system. A package that provides + // 'mailx' must be installed. E.g. "user@example.com" + //Unattended-Upgrade::Mail ""; + + // Set this value to one of: + // "always", "only-on-error" or "on-change" + // If this is not set, then any legacy MailOnlyOnError (boolean) value + // is used to chose between "only-on-error" and "on-change" + //Unattended-Upgrade::MailReport "on-change"; + + // Remove unused automatically installed kernel-related packages + // (kernel images, kernel headers and kernel version locked tools). + //Unattended-Upgrade::Remove-Unused-Kernel-Packages "true"; + + // Do automatic removal of newly unused dependencies after the upgrade + //Unattended-Upgrade::Remove-New-Unused-Dependencies "true"; + + // Do automatic removal of unused packages after the upgrade + // (equivalent to apt-get autoremove) + //Unattended-Upgrade::Remove-Unused-Dependencies "false"; + + // Automatically reboot *WITHOUT CONFIRMATION* if + // the file /var/run/reboot-required is found after the upgrade + //Unattended-Upgrade::Automatic-Reboot "false"; + + // Automatically reboot even if there are users currently logged in + // when Unattended-Upgrade::Automatic-Reboot is set to true + //Unattended-Upgrade::Automatic-Reboot-WithUsers "true"; + + // If automatic reboot is enabled and needed, reboot at the specific + // time instead of immediately + // Default: "now" + //Unattended-Upgrade::Automatic-Reboot-Time "02:00"; + + // Use apt bandwidth limit feature, this example limits the download + // speed to 70kb/sec + //Acquire::http::Dl-Limit "70"; + + // Enable logging to syslog. Default is False + // Unattended-Upgrade::SyslogEnable "false"; + + // Specify syslog facility. Default is daemon + // Unattended-Upgrade::SyslogFacility "daemon"; + + // Download and install upgrades only on AC power + // (i.e. skip or gracefully stop updates on battery) + // Unattended-Upgrade::OnlyOnACPower "true"; + + // Download and install upgrades only on non-metered connection + // (i.e. skip or gracefully stop updates on a metered connection) + // Unattended-Upgrade::Skip-Updates-On-Metered-Connections "true"; + + // Verbose logging + // Unattended-Upgrade::Verbose "false"; + + // Print debugging information both in unattended-upgrades and + // in unattended-upgrade-shutdown + // Unattended-Upgrade::Debug "false"; + + // Allow package downgrade if Pin-Priority exceeds 1000 + // Unattended-Upgrade::Allow-downgrade "false"; + + // When APT fails to mark a package to be upgraded or installed try adjusting + // candidates of related packages to help APT's resolver in finding a solution + // where the package can be upgraded or installed. + // This is a workaround until APT's resolver is fixed to always find a + // solution if it exists. (See Debian bug #711128.) + // The fallback is enabled by default, except on Debian's sid release because + // uninstallable packages are frequent there. + // Disabling the fallback speeds up unattended-upgrades when there are + // uninstallable packages at the expense of rarely keeping back packages which + // could be upgraded or installed. + // Unattended-Upgrade::Allow-APT-Mark-Fallback "true"; + owner: root + group: root + mode: '0644' + become: yes + +- name: Configure auto-upgrades - 20auto-upgrades (disabled by default) + copy: + dest: /etc/apt/apt.conf.d/20auto-upgrades + content: | + APT::Periodic::Update-Package-Lists "0"; + APT::Periodic::Unattended-Upgrade "0"; + owner: root + group: root + mode: '0644' + become: yes + +- name: Display unattended-upgrades status + debug: + msg: "Unattended-upgrades installed and configured (currently disabled in 20auto-upgrades). Enable by setting Update-Package-Lists and Unattended-Upgrade to 1." + +# Made with Bob diff --git a/jenkins-as-code/ansible/setup-jenkins-host.yml b/jenkins-as-code/ansible/setup-jenkins-host.yml new file mode 100644 index 0000000000..9fe8ba395b --- /dev/null +++ b/jenkins-as-code/ansible/setup-jenkins-host.yml @@ -0,0 +1,484 @@ +--- +############################################### +# Ubuntu 24 Jenkins User Setup - Ansible Playbook # +############################################### +# This playbook configures an Ubuntu 24.04 box with Jenkins user and SSH key +# Usage: Run locally on the target host with: +# ansible-playbook ubuntu24-jenkins-setup.yml --connection=local + +- name: Configure Ubuntu 24 with Jenkins User + hosts: localhost + connection: local + become: yes + gather_facts: yes + + vars: + # SSH public key for Jenkins user - override via JENKINS_SSH_KEY env var or inventory + jenkins_ssh_key: "{{ lookup('env', 'JENKINS_SSH_KEY') | default('ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC... jenkins@adoptopenjdk', true) }}" + + roles: + - role: system_update + tags: system_update + - role: unattended_upgrades + tags: security + - role: ntp_config + tags: system + - role: fail2ban + tags: security + + tasks: + - name: Verify we're running on Ubuntu 24.04 + assert: + that: + - ansible_distribution == "Ubuntu" + - ansible_distribution_version == "24.04" + fail_msg: "This playbook is designed for Ubuntu 24.04" + success_msg: "Running on Ubuntu 24.04" + tags: always + + - name: Update apt cache + apt: + update_cache: yes + cache_valid_time: 3600 + tags: setup + + - name: Install essential packages from Jenkins master + apt: + name: + # Core utilities + - openssh-server + - sudo + - python3 + - python3-pip + - curl + - wget + - git + - rsync + - unzip + - zip + - bzip2 + - xz-utils + - pigz + + # Development tools + - build-essential + - gcc + - make + - autoconf + - patch + - ant + + # System monitoring and management + - htop + - ncdu + - lsof + - strace + - nmap + - netcat-openbsd + - net-tools + - traceroute + - pciutils + - usbutils + + # Text editors + - vim + - nano + - joe + + # Terminal multiplexers + - screen + - tmux + + # Security and authentication + - fail2ban + - openssh-sftp-server + - ssl-cert + + # Networking + - bind9-host + - isc-dhcp-client + - ethtool + - fping + + # File systems and storage + - cifs-utils + - davfs2 + - xfsprogs + - lvm2 + - mdadm + + # Compression and archiving + - sharutils + + # Monitoring + - monitoring-plugins + - nagios-plugins + - qstat + + # Web server + - nginx + + # Time synchronization + - ntp + - ntpdate + + # Scheduling + - at + - cron + + # System utilities + - haveged + - cpufrequtils + - bc + - jq + - file + - less + - gawk + - gdisk + + # Version control + - mercurial + + # Repository management + - reprepro + + # Python environment + - virtualenv + - dh-python + - python3-bcrypt + + # Libraries + - libffi-dev + - libssl-dev + - libaugeas0 + - augeas-lenses + + # Package management + - apt-transport-https + - software-properties-common + - unattended-upgrades + + # Misc utilities + - run-one + - speedtest-cli + - xauth + + state: present + tags: setup + + ############################################### + # Install Adoptium Temurin JDK + ############################################### + - name: Add Adoptium GPG key + apt_key: + url: https://packages.adoptium.net/artifactory/api/gpg/key/public + state: present + tags: java + + - name: Add Adoptium Temurin repository + apt_repository: + repo: "deb https://packages.adoptium.net/artifactory/deb {{ ansible_distribution_release }} main" + state: present + filename: adoptium + tags: java + + - name: Update apt cache after adding Temurin repo + apt: + update_cache: yes + tags: java + + - name: Install Temurin JDK + apt: + name: "{{ java_package }}" + state: present + tags: java + + - name: Set JAVA_HOME environment variable + lineinfile: + path: /etc/environment + regexp: '^JAVA_HOME=' + line: 'JAVA_HOME={{ java_home }}' + state: present + tags: java + + - name: Verify Java installation + command: java -version + register: java_version + changed_when: false + tags: java + + - name: Display Java version + debug: + msg: "{{ java_version.stderr_lines }}" + tags: java + + ############################################### + # Check if UID/GID 1000 is already in use + ############################################### + - name: Check if UID 1000 is in use + shell: "getent passwd | awk -F: '$3 == 1000 {print $1}'" + register: uid_1000_user + changed_when: false + tags: jenkins_user + + - name: Check if GID 1000 is in use + shell: "getent group | awk -F: '$3 == 1000 {print $1}'" + register: gid_1000_group + changed_when: false + tags: jenkins_user + + - name: Display warning if UID 1000 is already in use + debug: + msg: + - "WARNING: UID 1000 is already in use by user: {{ uid_1000_user.stdout }}" + - "Jenkins user will be created with a system-assigned UID instead" + when: uid_1000_user.stdout != "" + tags: jenkins_user + + - name: Display warning if GID 1000 is already in use + debug: + msg: + - "WARNING: GID 1000 is already in use by group: {{ gid_1000_group.stdout }}" + - "Jenkins group will be created with a system-assigned GID instead" + when: gid_1000_group.stdout != "" + tags: jenkins_user + + ############################################### + # Create Jenkins user and group + ############################################### + - name: Create Jenkins user group + group: + name: "{{ jenkins_username }}" + gid: "{{ 1000 if gid_1000_group.stdout == '' else omit }}" + state: present + tags: jenkins_user + + - name: Create Jenkins user + user: + name: "{{ jenkins_username }}" + uid: "{{ 1000 if uid_1000_user.stdout == '' else omit }}" + group: "{{ jenkins_username }}" + state: present + home: "{{ jenkins_home }}" + shell: /bin/bash + create_home: yes + tags: jenkins_user + + - name: Get Jenkins user UID + command: id -u {{ jenkins_username }} + register: jenkins_uid + changed_when: false + tags: jenkins_user + + - name: Get Jenkins group GID + command: getent group {{ jenkins_username }} + register: jenkins_gid_info + changed_when: false + tags: jenkins_user + + - name: Display Jenkins user ID information + debug: + msg: + - "Jenkins user created with UID: {{ jenkins_uid.stdout }}" + - "Jenkins group created with GID: {{ jenkins_gid_info.stdout.split(':')[2] }}" + tags: jenkins_user + + - name: Ensure .ssh directory exists for Jenkins user + file: + path: "{{ jenkins_home }}/.ssh" + state: directory + owner: "{{ jenkins_username }}" + group: "{{ jenkins_username }}" + mode: '0700' + tags: jenkins_user + + - name: Set authorized key for Jenkins user + authorized_key: + user: "{{ jenkins_username }}" + state: present + key: "{{ jenkins_ssh_key }}" + tags: jenkins_user + + - name: Add github.com to known_hosts + known_hosts: + name: github.com + key: "github.com ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAq2A7hRGmdnm9tUDbO9IDSwBK6TbQa+PXYPCPy6rbTrTtw7PHkccKrpp0yVhp5HdEIcKr6pLlVDBfOLX9QUsyCOV0wzfjIJNlGEYsdlLJizHhbn2mUjvSAHQqZETYP81eFzLQNnPHt4EVVUh7VfDESU84KezmD5QlWpXLmvU31/yMf+Se8xhHTvKSCZIFImWwoG6mbUoWf9nzpIoaSjB+weqqUUmpaaasXVal72J+UX2B+2RPW3RcT0eOzQgqlJL3RKrTJvdsjE3JEAvGq3lGHSZXy28G3skua2SmVi/w4yCE6gbODqnTWlg7+wC604ydGXA8VJiS5ap43JXiUFFAaQ==" + path: "{{ jenkins_home }}/.ssh/known_hosts" + state: present + tags: jenkins_user + + - name: Change ownership of jenkins' ~/.ssh/known_hosts + file: + path: "{{ jenkins_home }}/.ssh/known_hosts" + owner: "{{ jenkins_username }}" + group: "{{ jenkins_username }}" + mode: '0644' + tags: jenkins_user + + - name: Unset password expiry for Jenkins user + command: chage -M -1 -E -1 {{ jenkins_username }} + tags: jenkins_user + + - name: Ensure proper limits are set in /etc/security/limits.conf + lineinfile: + path: /etc/security/limits.conf + line: "{{ jenkins_username }} {{ item.limit_type }} {{ item.limit_name }} {{ item.limit_value }}" + state: present + with_items: + - {limit_type: 'hard', limit_name: 'nofile', limit_value: '1048576'} + - {limit_type: 'soft', limit_name: 'nofile', limit_value: '1048576'} + - {limit_type: 'hard', limit_name: 'nproc', limit_value: 'unlimited'} + - {limit_type: 'soft', limit_name: 'nproc', limit_value: 'unlimited'} + - {limit_type: 'hard', limit_name: 'core', limit_value: 'unlimited'} + - {limit_type: 'soft', limit_name: 'core', limit_value: 'unlimited'} + tags: jenkins_user + + ############################################### + # Nagios plugins symlink + ############################################### + - name: Ensure /usr/local/nagios directory exists + file: + path: /usr/local/nagios + state: directory + owner: root + group: root + mode: '0755' + tags: setup + + - name: Create nagios libexec symlink to /usr/lib/nagios/plugins + file: + path: /usr/local/nagios/libexec + src: /usr/lib/nagios/plugins + state: link + owner: root + group: root + tags: setup + + ############################################### + # SSH service + ############################################### + - name: Ensure SSH service is enabled and running + systemd: + name: ssh + state: started + enabled: yes + tags: setup + + ############################################### + # Security Hardening - SSH Configuration + ############################################### + - name: Disable password authentication in SSH + lineinfile: + path: /etc/ssh/sshd_config + regexp: '^#?PasswordAuthentication' + line: 'PasswordAuthentication no' + state: present + backup: yes + notify: restart ssh + tags: security + + - name: Allow root login via SSH with key-based authentication only + lineinfile: + path: /etc/ssh/sshd_config + regexp: '^#?PermitRootLogin' + line: 'PermitRootLogin prohibit-password' + state: present + notify: restart ssh + tags: security + + - name: Disable empty passwords + lineinfile: + path: /etc/ssh/sshd_config + regexp: '^#?PermitEmptyPasswords' + line: 'PermitEmptyPasswords no' + state: present + notify: restart ssh + tags: security + + - name: Enable public key authentication + lineinfile: + path: /etc/ssh/sshd_config + regexp: '^#?PubkeyAuthentication' + line: 'PubkeyAuthentication yes' + state: present + notify: restart ssh + tags: security + + - name: Disable challenge-response authentication + lineinfile: + path: /etc/ssh/sshd_config + regexp: '^#?ChallengeResponseAuthentication' + line: 'ChallengeResponseAuthentication no' + state: present + notify: restart ssh + tags: security + + - name: Disable keyboard-interactive authentication + lineinfile: + path: /etc/ssh/sshd_config + regexp: '^#?KbdInteractiveAuthentication' + line: 'KbdInteractiveAuthentication no' + state: present + notify: restart ssh + tags: security + + - name: Set SSH protocol to 2 only + lineinfile: + path: /etc/ssh/sshd_config + regexp: '^#?Protocol' + line: 'Protocol 2' + state: present + notify: restart ssh + tags: security + + - name: Disable X11 forwarding + lineinfile: + path: /etc/ssh/sshd_config + regexp: '^#?X11Forwarding' + line: 'X11Forwarding no' + state: present + notify: restart ssh + tags: security + + - name: Set maximum authentication attempts + lineinfile: + path: /etc/ssh/sshd_config + regexp: '^#?MaxAuthTries' + line: 'MaxAuthTries 3' + state: present + notify: restart ssh + tags: security + + - name: Set login grace time + lineinfile: + path: /etc/ssh/sshd_config + regexp: '^#?LoginGraceTime' + line: 'LoginGraceTime 60' + state: present + notify: restart ssh + tags: security + + - name: Validate SSH configuration + command: sshd -t + changed_when: false + tags: security + + - name: Display Jenkins user information + debug: + msg: + - "Jenkins user created successfully!" + - "Username: {{ jenkins_username }}" + - "Home directory: {{ jenkins_home }}" + - "SSH key configured: Yes" + - "Sudo access: No (production security)" + - "Security hardening: SSH password auth disabled, SSH-only access enforced" + tags: always + + handlers: + - name: restart ssh + systemd: + name: ssh + state: restarted + tags: security + +# Made with Bob \ No newline at end of file diff --git a/jenkins-as-code/ansible/templates/jenkins-defaults.j2 b/jenkins-as-code/ansible/templates/jenkins-defaults.j2 new file mode 100644 index 0000000000..8cd345240a --- /dev/null +++ b/jenkins-as-code/ansible/templates/jenkins-defaults.j2 @@ -0,0 +1,104 @@ +# defaults for Jenkins automation server + +# pulled in from the init script; makes things easier. +NAME=jenkins + +# location of java +JAVA={{ java_home }}/bin/java + +# arguments to pass to java + +# Allow graphs etc. to work even when an X server is present +#JAVA_ARGS="-Djava.awt.headless=true " + +# Production JVM tuning based on adoptium/infrastructure Jenkins master +# History of memory settings (for reference): +# JAVA_ARGS="-Xmx20g -Xms8g" +# JAVA_ARGS="-Xmx16g -Xms8g -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintTenuringDistribution -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=5 -XX:GCLogFileSize=2M -Xloggc:/var/log/jenkins/gc.log -XX:+UseG1GC" +# JAVA_ARGS="-Xmx16g -Xms12g" +# JAVA_ARGS="-Xmx16g" - Changed to -Xmx18G on 2023-01-09 by sxa: https://github.com/adoptium/infrastructure/issues/2875 +# JAVA_ARGS="-Xmx18G" +# JAVA_ARGS="-Xmx20G -XX:+UseG1GC -XX:MaxGCPauseMillis=100 -Dhudson.model.Fingerprint.enableFingerprintCleanup=false -Dhudson.util.XStream2.collectionUpdateLimit=-1" +# JAVA_ARGS="-Djava.awt.headless=true -Xmx22G -Dhudson.util.XStream2.collectionUpdateLimit=-1 -Xlog:gc*,gc+heap=info,gc+age=trace,gc+phases=trace,safepoint:file=/var/log/jenkins/gc.log:time,uptime,level,tags:filecount=5,filesize=50m" + +# Current production settings (28/May/2026 by sxa & aleonard - https://github.com/adoptium/infrastructure/issues/4364) +# JUnit memory optimizations + GC logging +# Heap size is set dynamically by Ansible based on available system memory (see install-jenkins-server.yml) +JAVA_ARGS="{{ jenkins_java_opts }}" + +# make jenkins listen on IPv4 address +#JAVA_ARGS="-Djava.net.preferIPv4Stack=true" + +PIDFILE=/var/run/$NAME/$NAME.pid + +# user and group to be invoked as (default to jenkins) +JENKINS_USER=$NAME +JENKINS_GROUP=$NAME + +# location of the jenkins war file +JENKINS_WAR={{ jenkins_war_path }} + +# jenkins home location +#JENKINS_HOME=/var/lib/$NAME +JENKINS_HOME={{ jenkins_data_dir }} + +# set this to false if you don't want Jenkins to run by itself +# in this set up, you are expected to provide a servlet container +# to host jenkins. +RUN_STANDALONE=true + +# log location. this may be a syslog facility.priority +JENKINS_LOG=/var/log/$NAME/$NAME.log +#JENKINS_LOG=daemon.info + +# OS LIMITS SETUP +# comment this out to observe /etc/security/limits.conf +# this is on by default because http://github.com/jenkinsci/jenkins/commit/2fb288474e980d0e7ff9c4a3b768874835a3e92e +# reported that Ubuntu's PAM configuration doesn't include pam_limits.so, and as a result the # of file +# descriptors are forced to 1024 regardless of /etc/security/limits.conf +MAXOPENFILES=8192 + +# set the umask to control permission bits of files that Jenkins creates. +# 027 makes files read-only for group and inaccessible for others, which some security sensitive users +# might consider benefitial, especially if Jenkins runs in a box that's used for multiple purposes. +# Beware that 027 permission would interfere with sudo scripts that run on the master (JENKINS-25065.) +# +# Note also that the particularly sensitive part of $JENKINS_HOME (such as credentials) are always +# written without 'others' access. So the umask values only affect job configuration, build records, +# that sort of things. +# +# If commented out, the value from the OS is inherited, which is normally 022 (as of Ubuntu 12.04, +# by default umask comes from pam_umask(8) and /etc/login.defs + +# UMASK=027 + +# port for HTTP connector (default 8080; disable with -1) +HTTP_PORT={{ jenkins_port }} + +# servlet context, important if you want to use apache proxying +PREFIX=/$NAME + +# Enable the Jenkins Access log - Added by Martijn 21st March 2019 +JENKINS_ACCESSLOG="--accessLoggerClassName=winstone.accesslog.SimpleAccessLogger --simpleAccessLogger.format=combined --simpleAccessLogger.file=/var/log/jenkins/access.log" + +# Session timeout for users in minutes - Added by Martijn 21st March 2019 +# sessionEviction added 9th May 2024 by sxa +JENKINS_SESSION_TIMEOUT="--sessionTimeout=720 --sessionEviction=43200" + +# This was added on 28/May/2026 by sxa as part of https://github.com/adoptium/infrastructure/issues/4364 +#JENKINS_JUNIT_MEMORY_FIXES="-Dhudson.tasks.junit.TestResultAction.RESULT_CACHE_ENABLED=false" # -Dhudson.tasks.junit.History\$HistoryTableResult.PREVIOUS_TEST_RESULT_BACKTRACK_BUILDS_MAX=1" + +# arguments to pass to jenkins. +# --javahome=$JAVA_HOME +# --httpPort=$HTTP_PORT (default 8080; disable with -1) +# --httpsPort=$HTTP_PORT +# --argumentsRealm.passwd.$ADMIN_USER=[password] +# --argumentsRealm.roles.$ADMIN_USER=admin +# --webroot=~/.jenkins/war +# --prefix=$PREFIX + +# Note: EnvironmentFile does not support shell variable interpolation, so session timeout and +# access log args are expanded inline here rather than referencing $JENKINS_SESSION_TIMEOUT/$JENKINS_ACCESSLOG. +JENKINS_ARGS="{{ jenkins_session_timeout }} {{ jenkins_accesslog }} --webroot=/var/cache/jenkins/war --httpPort={{ jenkins_port }} --httpListenAddress={{ effective_listen_address }}" + +# Made with Bob \ No newline at end of file diff --git a/jenkins-as-code/ansible/templates/jenkins-logrotate.j2 b/jenkins-as-code/ansible/templates/jenkins-logrotate.j2 new file mode 100644 index 0000000000..a0c16367cf --- /dev/null +++ b/jenkins-as-code/ansible/templates/jenkins-logrotate.j2 @@ -0,0 +1,10 @@ +/var/log/jenkins/jenkins.log +/var/log/jenkins/access.log { + daily + rotate 14 + compress + delaycompress + missingok + notifempty + copytruncate +} diff --git a/jenkins-as-code/ansible/templates/jenkins.service.j2 b/jenkins-as-code/ansible/templates/jenkins.service.j2 new file mode 100644 index 0000000000..696e7190bd --- /dev/null +++ b/jenkins-as-code/ansible/templates/jenkins.service.j2 @@ -0,0 +1,41 @@ +[Unit] +Description=Jenkins Automation Server +Documentation=https://www.jenkins.io/doc/ +After=network.target + +[Service] +Type=simple +User={{ jenkins_username }} +Group={{ jenkins_username }} + +# Environment +Environment="JENKINS_HOME={{ jenkins_data_dir }}" +EnvironmentFile=-/etc/default/jenkins + +# Working directory +WorkingDirectory={{ jenkins_data_dir }} + +# Start Jenkins +ExecStart={{ java_home }}/bin/java $JAVA_ARGS -jar {{ jenkins_war_path }} $JENKINS_ARGS + +# Restart policy +Restart=on-failure +RestartSec=10 +StartLimitBurst=3 +StartLimitInterval=60 + +# Security settings +NoNewPrivileges=true +PrivateTmp=true + +# Resource limits +LimitNOFILE=8192 +LimitNPROC=30654 + +# Logging +StandardOutput=append:/var/log/jenkins/jenkins.log +StandardError=append:/var/log/jenkins/jenkins.log +SyslogIdentifier=jenkins + +[Install] +WantedBy=multi-user.target \ No newline at end of file diff --git a/jenkins-as-code/docs/CONFIG-ANALYSIS.md b/jenkins-as-code/docs/CONFIG-ANALYSIS.md new file mode 100644 index 0000000000..c0012ab03e --- /dev/null +++ b/jenkins-as-code/docs/CONFIG-ANALYSIS.md @@ -0,0 +1,193 @@ +# Jenkins Master Configuration Analysis + +## Overview +Analysis of extracted configuration from production Jenkins master (jenkins-hetzner-ubuntu2004) running Ubuntu 24.04.4 LTS. + +--- + +## ✅ IMPLEMENT NOW - Generic/Portable Configurations + +These configurations are environment-independent and should be implemented immediately in the Ansible playbook: + +### 1. **SSH Security Configuration** ✅ ALREADY IMPLEMENTED +- PasswordAuthentication: no +- KbdInteractiveAuthentication: no +- PermitRootLogin: prohibit-password (key-based only) +- **Status**: Already in playbook + +### 2. **System Limits** ⚠️ PARTIALLY IMPLEMENTED +Current production has default limits.conf (no custom limits). +- **Action**: Keep current playbook implementation with jenkins user limits +- **Status**: Already configured in playbook for jenkins user + +### 3. **Unattended Upgrades** 🔧 NEEDS IMPLEMENTATION +Production config shows security-only updates enabled. +``` +Allowed-Origins: +- ${distro_id}:${distro_codename} +- ${distro_id}:${distro_codename}-security +- ${distro_id}ESMApps:${distro_codename}-apps-security +- ${distro_id}ESM:${distro_codename}-infra-security +``` +- **Action**: Add unattended-upgrades configuration to playbook +- **Priority**: HIGH (security) + +### 4. **NTP Configuration** 🔧 NEEDS IMPLEMENTATION +Production uses standard Ubuntu NTP pool servers. +``` +pool 0.ubuntu.pool.ntp.org iburst +pool 1.ubuntu.pool.ntp.org iburst +pool 2.ubuntu.pool.ntp.org iburst +pool 3.ubuntu.pool.ntp.org iburst +pool ntp.ubuntu.com +``` +- **Action**: Add NTP configuration task +- **Priority**: MEDIUM (time sync important for builds) + +### 5. **Additional JDK Versions** 🔧 NEEDS IMPLEMENTATION +Production has multiple JDK versions: +- temurin-11-jdk +- temurin-17-jdk +- temurin-21-jdk +- temurin-25-jdk ✅ (already in playbook) +- **Action**: Add tasks to install JDK 11, 17, 21 +- **Priority**: HIGH (needed for multi-version builds) + +### 6. **Wazuh Agent** 🔧 OPTIONAL +Production has wazuh-agent installed (security monitoring). +- **Action**: Add wazuh-agent installation if monitoring is required +- **Priority**: LOW (can be added later) + +### 7. **Environment PATH** 🔧 NEEDS REVIEW +Production has custom PATH with InstallBuilder: +``` +PATH="/opt/installbuilder-17.7.0/bin:..." +``` +- **Action**: Review if InstallBuilder or other custom tools needed +- **Priority**: MEDIUM (depends on build requirements) + +--- + +## ⏳ IMPLEMENT LATER - Environment-Specific Configurations + +These configurations contain server-specific details and should be configured AFTER the new server has its domain/IP: + +### 1. **Hosts File** 🚫 DO NOT COPY +Production hosts file contains: +- Hostname: jenkins-hetzner-ubuntu2004 +- IPv4: 172.31.1.100, 78.47.239.97 +- IPv6: 2a01:4f8:c0c:1804::2 +- **Action**: Configure with new server's hostname and IPs +- **When**: After server provisioning + +### 2. **Network Configuration** 🚫 DO NOT COPY +Production network shows: +- eth0 with specific MAC and IPs +- Hetzner-specific network setup +- **Action**: Let new server use its own network config +- **When**: Automatic during provisioning + +### 3. **Nginx Configuration** 🚫 EXTRACT & REVIEW LATER +Needs analysis of: +- Virtual hosts +- SSL certificates +- Proxy configurations +- Domain names +- **Action**: Extract nginx configs, update domains/certs for new server +- **When**: After DNS and SSL certificates are ready + +### 4. **Fail2ban Configuration** 🚫 EXTRACT & REVIEW LATER +May contain IP whitelists or server-specific rules. +- **Action**: Extract and review, update IP whitelists +- **When**: After new server is accessible + +### 5. **Backup Mounts** 🚫 DO NOT COPY +Production has: +- //u158991.your-backup.de/backup mounted at /mnt/backup-server +- Additional disks: /dev/sdb, /dev/sdc for Jenkins workspace/jobs +- **Action**: Configure new backup solution with new credentials +- **When**: After backup infrastructure is ready + +### 6. **Cron Jobs** 🚫 REVIEW & ADAPT +**Jenkins user crontab:** +``` +08 08 * * * /home/jenkins/diskspace/rundiskspace +0,15,30,45 * * * * bash -c "date; grep label queue.xml..." >> /home/jenkins/queuedepth.log +``` + +**Root crontab:** +``` +0 5 * * 6 /root/apt-security.sh +``` +- **Action**: Review scripts, adapt paths for new server +- **When**: After Jenkins is installed and scripts are available + +--- + +## 📋 IMPLEMENTATION PRIORITY + +### Phase 1: Initial Server Setup (NOW) +1. ✅ Base packages installation (already in playbook) +2. ✅ SSH hardening (already in playbook) +3. ✅ Jenkins user creation with UID/GID handling (already in playbook) +4. ✅ Temurin 25 JDK (already in playbook) +5. 🔧 Add Temurin 11, 17, 21 JDKs +6. 🔧 Configure unattended-upgrades +7. 🔧 Configure NTP +8. ✅ System limits (already in playbook) + +### Phase 2: Post-Provisioning (AFTER new server has domain/IP) +1. Configure hostname and hosts file +2. Extract and configure nginx with new domains +3. Extract and configure fail2ban with new IPs +4. Set up backup mounts with new credentials +5. Configure monitoring (wazuh-agent if needed) + +### Phase 3: Jenkins-Specific (AFTER Jenkins installation) +1. Review and adapt cron jobs +2. Set up disk space monitoring +3. Configure queue depth monitoring +4. Set up Jenkins-specific scripts + +--- + +## 🔍 ADDITIONAL FINDINGS + +### Disk Usage Concerns +Production server shows: +- Root partition: 93% full (494G/564G used) +- Jobs partition: 76% full (1.5T/2.0T used) +- **Recommendation**: Plan for adequate storage on new server + +### Memory +- 30GB RAM, 15GB swap +- **Recommendation**: Match or exceed for new server + +### Multiple JDK Versions Required +Production uses 4 different JDK versions (11, 17, 21, 25). +- **Recommendation**: Install all versions in playbook + +### Custom Tools +- InstallBuilder 17.7.0 in PATH +- **Action**: Determine if needed for new server + +--- + +## 📝 NEXT STEPS + +1. **Immediate**: Update playbook with Phase 1 items +2. **Document**: Create separate configs for Phase 2 (domain-specific) +3. **Review**: Nginx and fail2ban configs in detail +4. **Plan**: Storage requirements for new server +5. **Identify**: All custom scripts and tools needed + +--- + +## 🚨 WARNINGS + +- **DO NOT** copy hosts file directly +- **DO NOT** copy network configurations +- **DO NOT** copy backup mount credentials +- **DO NOT** copy nginx configs without updating domains +- **DO NOT** copy fail2ban configs without reviewing IP whitelists +- **REVIEW** all cron jobs before implementing diff --git a/jenkins-as-code/docs/DEPLOYMENT-GUIDE.md b/jenkins-as-code/docs/DEPLOYMENT-GUIDE.md new file mode 100644 index 0000000000..4daef84876 --- /dev/null +++ b/jenkins-as-code/docs/DEPLOYMENT-GUIDE.md @@ -0,0 +1,294 @@ +# Deployment Guide - Security & System Configuration + +This guide covers deploying the security and system configuration roles (unattended-upgrades, NTP, fail2ban) to Jenkins infrastructure. + +## Prerequisites + +- Ansible 2.9 or higher installed +- SSH access to target servers +- Root or sudo privileges on target servers +- Python 3 installed on target servers + +## Quick Start + +### 1. Configure Inventory + +Copy the example inventory and customize it: + +```bash +cd jenkins-as-code/ansible +cp inventory-example.yml inventory.yml +``` + +Edit `inventory.yml` and update: +- Server IP addresses +- SSH keys +- **IMPORTANT:** Fail2ban IP whitelist with your trusted IPs + +### 2. Test Connectivity + +```bash +ansible all -i inventory.yml -m ping +``` + +### 3. Run the Playbook + +**Dry run (check mode):** +```bash +ansible-playbook -i inventory.yml setup-jenkins-host.yml --check --diff +``` + +**Full deployment:** +```bash +ansible-playbook -i inventory.yml setup-jenkins-host.yml +``` + +**Deploy only security roles:** +```bash +ansible-playbook -i inventory.yml setup-jenkins-host.yml --tags security +``` + +**Deploy only NTP:** +```bash +ansible-playbook -i inventory.yml setup-jenkins-host.yml --tags system +``` + +## Configuration Details + +### Unattended Upgrades + +**Default State:** Installed but disabled + +**To Enable Automatic Updates:** +```bash +# On the target server +sudo nano /etc/apt/apt.conf.d/20auto-upgrades + +# Change these values from 0 to 1: +APT::Periodic::Update-Package-Lists "1"; +APT::Periodic::Unattended-Upgrade "1"; +``` + +**What Gets Updated:** +- Security updates only (`${distro_id}:${distro_codename}-security`) +- Extended Security Maintenance (ESM) updates if available +- NO regular updates, proposed, or backports + +**Verify Configuration:** +```bash +sudo unattended-upgrade --dry-run --debug +``` + +### NTP Configuration + +**Configured Servers:** +- 0.ubuntu.pool.ntp.org +- 1.ubuntu.pool.ntp.org +- 2.ubuntu.pool.ntp.org +- 3.ubuntu.pool.ntp.org +- ntp.ubuntu.com (fallback) + +**Verify Time Sync:** +```bash +# Check NTP service +sudo systemctl status ntpsec + +# Check NTP peers +ntpq -p + +# Check system time +timedatectl status +``` + +### Fail2ban Configuration + +**Default Protection:** +- SSH brute force protection enabled +- 3 failed attempts within 10 minutes = 1 hour ban +- Progressive banning (doubles each time, max 1 week) +- Repeat offender jail (5+ bans in 24h = 7 day ban) + +**CRITICAL: IP Whitelist** + +Before deployment, update your inventory with trusted IPs: + +```yaml +fail2ban_ignoreip: >- + 127.0.0.1/8 + ::1 + 10.0.0.0/8 + 192.168.0.0/16 + YOUR_OFFICE_IP/32 + YOUR_VPN_NETWORK/24 +``` + +**Verify Configuration:** +```bash +# Check fail2ban status +sudo systemctl status fail2ban + +# List all jails +sudo fail2ban-client status + +# Check SSH jail +sudo fail2ban-client status sshd + +# Check repeat offender jail +sudo fail2ban-client status recidive + +# View banned IPs +sudo fail2ban-client get sshd banned + +# Unban an IP (if needed) +sudo fail2ban-client set sshd unbanip 203.0.113.50 +``` + +## Post-Deployment Verification + +### 1. Check All Services + +```bash +# Run on target server +sudo systemctl status ntpsec +sudo systemctl status fail2ban +sudo systemctl status unattended-upgrades +``` + +### 2. Verify Configurations + +```bash +# Check unattended-upgrades config +cat /etc/apt/apt.conf.d/50unattended-upgrades +cat /etc/apt/apt.conf.d/20auto-upgrades + +# Check NTP config +cat /etc/ntp.conf +ntpq -p + +# Check fail2ban config +cat /etc/fail2ban/jail.local +sudo fail2ban-client status +``` + +### 3. Test Fail2ban (Optional) + +From a non-whitelisted IP, attempt multiple failed SSH logins to verify banning works: + +```bash +# From test machine (will get banned!) +ssh wronguser@jenkins-server # Try 3+ times with wrong password + +# On Jenkins server, check if IP was banned +sudo fail2ban-client status sshd +``` + +## Troubleshooting + +### Fail2ban Not Starting + +```bash +# Check configuration syntax +sudo fail2ban-client -t + +# Check logs +sudo tail -f /var/log/fail2ban.log + +# Restart service +sudo systemctl restart fail2ban +``` + +### NTP Not Syncing + +```bash +# Check if NTP can reach servers +sudo ntpdate -q 0.ubuntu.pool.ntp.org + +# Restart NTP service +sudo systemctl restart ntpsec + +# Check system time settings +timedatectl +``` + +### Locked Out by Fail2ban + +If you accidentally get banned: + +1. Access server via console (not SSH) +2. Unban your IP: + ```bash + sudo fail2ban-client set sshd unbanip YOUR_IP + ``` +3. Add your IP to whitelist in inventory +4. Re-run playbook + +## Rollback + +To remove configurations: + +```bash +# Stop services +sudo systemctl stop fail2ban +sudo systemctl stop ntpsec + +# Remove packages (optional) +sudo apt remove fail2ban ntpsec unattended-upgrades + +# Restore original configs from backups +sudo cp /etc/ssh/sshd_config.backup /etc/ssh/sshd_config +``` + +## Maintenance + +### Update Fail2ban Whitelist + +1. Edit inventory.yml +2. Update `fail2ban_ignoreip` variable +3. Re-run playbook: + ```bash + ansible-playbook -i inventory.yml setup-jenkins-host.yml --tags security + ``` + +### Monitor Fail2ban Activity + +```bash +# View recent bans +sudo tail -100 /var/log/fail2ban.log | grep Ban + +# View all currently banned IPs +sudo fail2ban-client status sshd | grep "Banned IP" + +# Statistics +sudo fail2ban-client status sshd +``` + +### Check for Security Updates + +```bash +# List available security updates +sudo apt list --upgradable | grep -i security + +# Run unattended-upgrades manually +sudo unattended-upgrade --dry-run +``` + +## Security Best Practices + +1. **Always whitelist your management IPs** before enabling fail2ban +2. **Test in staging** before deploying to production +3. **Monitor fail2ban logs** regularly for suspicious activity +4. **Keep NTP synchronized** for accurate security logs +5. **Enable unattended-upgrades** only after testing in your environment +6. **Document all whitelisted IPs** and review regularly + +## Support + +For issues or questions: +- Check role documentation: `jenkins-as-code/ansible/roles/README.md` +- Review extracted configs: `jenkins-as-code/data/jenkins-master-configs-*/` +- See configuration analysis: `jenkins-as-code/CONFIG-ANALYSIS.md` + +--- + +**Last Updated:** 2026-06-24 +**Based on:** Production Jenkins master configuration \ No newline at end of file diff --git a/jenkins-as-code/docs/ENVIRONMENT-CONFIG.md b/jenkins-as-code/docs/ENVIRONMENT-CONFIG.md new file mode 100644 index 0000000000..732d5ed78c --- /dev/null +++ b/jenkins-as-code/docs/ENVIRONMENT-CONFIG.md @@ -0,0 +1,297 @@ +# Environment-Based Configuration Guide + +This guide explains how to deploy Jenkins with different configurations for production and development environments. + +## Overview + +The Jenkins installation playbook automatically detects system memory and adjusts heap size accordingly. You can also use environment-specific inventory files to override settings. + +## Automatic Memory Detection + +By default, the playbook automatically calculates appropriate heap size based on available system memory: + +| System RAM | Jenkins Heap | Use Case | +|------------|--------------|----------| +| < 6GB | 2G | Small dev/test VMs | +| 6-14GB | 4G | Medium dev environments | +| 14-30GB | 8G | Staging environments | +| 30GB+ | 19G | Production (matches current setup) | + +## Deployment Methods + +### Method 1: Auto-Detection (Recommended for Dev) + +Let the playbook automatically detect and configure based on system resources: + +```bash +# Uses auto-detection +ansible-playbook setup-jenkins-host.yml --connection=local +ansible-playbook install-jenkins-server.yml --connection=local +``` + +**Best for:** Vagrant VMs, development environments, testing + +### Method 2: Environment-Specific Inventory (Recommended for Production) + +Use pre-configured inventory files for consistent deployments: + +#### Production Deployment + +```bash +# Production with 19GB heap, localhost-only binding +ansible-playbook -i inventory-production.yml setup-jenkins-host.yml +ansible-playbook -i inventory-production.yml install-jenkins-server.yml +``` + +**Configuration:** +- Heap: 19GB (fixed) +- Listen: 127.0.0.1 (requires reverse proxy) +- Fail2ban: Production IP whitelist +- Environment: production + +#### Vagrant/Development Deployment + +```bash +# Development with auto-detected heap, all-interface binding +ansible-playbook -i inventory-vagrant.yml setup-jenkins-host.yml +ansible-playbook -i inventory-vagrant.yml install-jenkins-server.yml +``` + +**Configuration:** +- Heap: Auto-detected (2G/4G/8G based on VM RAM) +- Listen: 0.0.0.0 (direct access, no proxy needed) +- Fail2ban: Local network whitelist +- Environment: development + +### Method 3: Environment Variables + +Override specific settings using environment variables: + +```bash +# Custom heap size +export JENKINS_HEAP_SIZE="8G" +ansible-playbook install-jenkins-server.yml --connection=local + +# Custom listen address (for dev without proxy) +export JENKINS_LISTEN_ADDRESS="0.0.0.0" +ansible-playbook install-jenkins-server.yml --connection=local + +# Both +export JENKINS_HEAP_SIZE="4G" +export JENKINS_LISTEN_ADDRESS="0.0.0.0" +ansible-playbook install-jenkins-server.yml --connection=local +``` + +### Method 4: Playbook Variables + +Override in the playbook or via command line: + +```bash +# Command line override +ansible-playbook install-jenkins-server.yml --connection=local \ + -e "jenkins_heap_size=8G" \ + -e "jenkins_listen_address=0.0.0.0" +``` + +## Configuration Comparison + +### Production Configuration + +```yaml +# inventory-production.yml +jenkins_heap_size: "19G" # Fixed for production +jenkins_listen_address: "127.0.0.1" # Reverse proxy required +fail2ban_ignoreip: "127.0.0.1/8 ::1 10.0.0.0/8 192.168.0.0/16 78.47.239.96 46.224.123.39 178.62.115.224 20.90.182.165" +``` + +**Use when:** +- Deploying to production servers +- System has 32GB+ RAM +- Using Nginx/Apache reverse proxy +- Need consistent, predictable configuration + +### Vagrant/Development Configuration + +```yaml +# inventory-vagrant.yml +jenkins_heap_size: "auto" # Auto-detect based on VM RAM +jenkins_listen_address: "0.0.0.0" # Direct access +fail2ban_ignoreip: "127.0.0.1/8 ::1 10.0.0.0/8 192.168.0.0/16 172.16.0.0/12" +``` + +**Use when:** +- Testing in Vagrant VMs +- Local development +- Limited system resources +- No reverse proxy setup + +## Vagrant Integration + +### Example Vagrantfile Configuration + +```ruby +Vagrant.configure("2") do |config| + config.vm.box = "ubuntu/jammy64" + + # Small dev VM (4GB RAM) - will use 2G heap + config.vm.provider "virtualbox" do |vb| + vb.memory = "4096" + vb.cpus = 2 + end + + # Medium dev VM (8GB RAM) - will use 4G heap + # config.vm.provider "virtualbox" do |vb| + # vb.memory = "8192" + # vb.cpus = 4 + # end + + # Provision with Ansible + config.vm.provision "ansible_local" do |ansible| + ansible.playbook = "ansible/setup-jenkins-host.yml" + ansible.inventory_path = "ansible/inventory-vagrant.yml" + end + + config.vm.provision "ansible_local" do |ansible| + ansible.playbook = "ansible/install-jenkins-server.yml" + ansible.inventory_path = "ansible/inventory-vagrant.yml" + end +end +``` + +## Verification + +After deployment, verify the configuration: + +```bash +# Check heap size in use +ps aux | grep jenkins | grep Xmx + +# Check listen address +sudo netstat -tlnp | grep 8080 +# or +sudo ss -tlnp | grep 8080 + +# Check Jenkins configuration +sudo cat /etc/default/jenkins | grep JAVA_ARGS + +# View systemd environment +sudo systemctl show jenkins | grep JENKINS_HOME +``` + +## Memory Sizing Guidelines + +### Development/Testing + +**4GB VM (2G heap):** +- Basic testing +- Small projects +- Limited concurrent builds + +**8GB VM (4G heap):** +- Medium projects +- Multiple concurrent builds +- Plugin testing + +### Staging + +**16GB System (8G heap):** +- Pre-production testing +- Load testing +- Multiple concurrent builds + +### Production + +**32GB+ System (19G heap):** +- Large-scale deployments +- Many concurrent builds +- Extensive plugin usage +- Large test suites + +## Troubleshooting + +### Heap Size Too Large + +**Symptom:** Jenkins fails to start, OOM errors + +**Solution:** +```bash +# Check available memory +free -h + +# Override with smaller heap +export JENKINS_HEAP_SIZE="4G" +ansible-playbook install-jenkins-server.yml --connection=local +``` + +### Can't Access Jenkins (127.0.0.1 binding) + +**Symptom:** Can't access Jenkins from browser + +**Solution for Dev:** +```bash +# Use vagrant inventory or override +export JENKINS_LISTEN_ADDRESS="0.0.0.0" +ansible-playbook install-jenkins-server.yml --connection=local +``` + +**Solution for Production:** +Set up reverse proxy (Nginx/Apache) - see JENKINS-INSTALL.md + +### Auto-Detection Not Working + +**Symptom:** Unexpected heap size + +**Solution:** +```bash +# Check detected memory +ansible localhost -m setup -a 'filter=ansible_memtotal_mb' + +# Override manually +ansible-playbook install-jenkins-server.yml --connection=local \ + -e "jenkins_heap_size=4G" +``` + +## Best Practices + +1. **Production:** Always use `inventory-production.yml` with fixed heap size +2. **Development:** Use `inventory-vagrant.yml` with auto-detection +3. **Testing:** Test with production-like heap sizes before deploying +4. **Monitoring:** Monitor memory usage and adjust if needed +5. **Documentation:** Document any custom heap sizes in your inventory + +## Examples + +### Quick Dev Setup (4GB VM) + +```bash +cd jenkins-as-code/ansible +ansible-playbook -i inventory-vagrant.yml setup-jenkins-host.yml +ansible-playbook -i inventory-vagrant.yml install-jenkins-server.yml +# Result: 2G heap, accessible on 0.0.0.0:8080 +``` + +### Production Deployment (32GB+ Server) + +```bash +cd jenkins-as-code/ansible +ansible-playbook -i inventory-production.yml setup-jenkins-host.yml +ansible-playbook -i inventory-production.yml install-jenkins-server.yml +# Result: 19G heap, accessible on 127.0.0.1:8080 (needs reverse proxy) +``` + +### Custom Staging Setup (16GB Server) + +```bash +cd jenkins-as-code/ansible +ansible-playbook setup-jenkins-host.yml --connection=local \ + -e "jenkins_heap_size=8G" \ + -e "jenkins_listen_address=127.0.0.1" +ansible-playbook install-jenkins-server.yml --connection=local \ + -e "jenkins_heap_size=8G" \ + -e "jenkins_listen_address=127.0.0.1" +# Result: 8G heap, accessible on 127.0.0.1:8080 +``` + +--- + +**Made with Bob** \ No newline at end of file diff --git a/jenkins-as-code/docs/IMPLEMENTATION-SUMMARY.md b/jenkins-as-code/docs/IMPLEMENTATION-SUMMARY.md new file mode 100644 index 0000000000..40f2d0da23 --- /dev/null +++ b/jenkins-as-code/docs/IMPLEMENTATION-SUMMARY.md @@ -0,0 +1,259 @@ +# Implementation Summary - Security & System Configuration + +## Overview + +Successfully implemented Ansible roles for three critical system configurations based on the existing Jenkins master server: + +1. **Unattended Upgrades** - Security-only automatic updates +2. **NTP Configuration** - Time synchronization with Ubuntu NTP pools +3. **Fail2ban** - SSH protection with IP whitelisting + +## What Was Created + +### Ansible Roles + +#### 1. `roles/unattended_upgrades/` +- **Purpose:** Configure security-only automatic updates +- **Files:** + - `tasks/main.yml` - Installation and configuration tasks +- **Configuration:** + - Installs `unattended-upgrades` package + - Configures `/etc/apt/apt.conf.d/50unattended-upgrades` (security updates only) + - Sets up `/etc/apt/apt.conf.d/20auto-upgrades` (disabled by default) +- **Based on:** Extracted configs from production Jenkins master + +#### 2. `roles/ntp_config/` +- **Purpose:** Configure time synchronization with Ubuntu NTP pools +- **Files:** + - `tasks/main.yml` - Installation and configuration tasks + - `handlers/main.yml` - Service restart handler + - `defaults/main.yml` - Default variables +- **Configuration:** + - Installs NTP package (ntpsec for Ubuntu 20.04+) + - Configures `/etc/ntp.conf` with Ubuntu pool servers + - Enables and starts NTP service +- **NTP Servers:** 0-3.ubuntu.pool.ntp.org + ntp.ubuntu.com + +#### 3. `roles/fail2ban/` +- **Purpose:** SSH protection with progressive banning and IP whitelisting +- **Files:** + - `tasks/main.yml` - Installation and configuration tasks + - `handlers/main.yml` - Service restart handler + - `defaults/main.yml` - Default variables and IP whitelist +- **Configuration:** + - Installs fail2ban package + - Configures `/etc/fail2ban/jail.local` with SSH protection + - Sets up repeat offender detection (recidive jail) + - Enables progressive banning (1h → 2h → 4h, max 1 week) +- **Features:** + - 3 failed attempts in 10 minutes = ban + - Ban time doubles for repeat offenders + - 5+ bans in 24 hours = 7-day ban + - Configurable IP whitelist + +### Documentation + +#### 1. `roles/README.md` +Comprehensive documentation covering: +- Role descriptions and features +- Configuration options +- Usage examples +- Verification commands +- Troubleshooting guides + +#### 2. `DEPLOYMENT-GUIDE.md` +Step-by-step deployment guide including: +- Prerequisites +- Quick start instructions +- Configuration details for each role +- Post-deployment verification +- Troubleshooting procedures +- Security best practices + +#### 3. `inventory-example.yml` +Example inventory file showing: +- Server configuration +- Fail2ban IP whitelist setup +- Variable overrides +- Multiple server examples + +### Playbook Updates + +#### `setup-jenkins-host.yml` +- Added three new roles to the playbook +- Fixed YAML syntax issues (quoted shell commands) +- Roles are tagged for selective execution: + - `unattended_upgrades`: `security` tag + - `ntp_config`: `system` tag + - `fail2ban`: `security` tag + +## Configuration Source + +All configurations are based on the production Jenkins master server: +- **Extracted:** 2026-06-24 +- **Source Directory:** `jenkins-as-code/data/jenkins-master-configs-20260624-180156/` +- **Files Used:** + - `20auto-upgrades` + - `50unattended-upgrades` + - `ntp.conf` + - `fail2ban.tar.gz` (jail.local) + +## Key Features + +### Security +- ✅ Security-only automatic updates (no regular updates) +- ✅ SSH brute force protection with progressive banning +- ✅ IP whitelisting to prevent lockouts +- ✅ Repeat offender detection +- ✅ Configurable ban times and thresholds + +### Reliability +- ✅ Time synchronization for accurate logs and security +- ✅ Ubuntu NTP pool servers with fallback +- ✅ Service monitoring and automatic restart + +### Maintainability +- ✅ Ansible roles for easy deployment +- ✅ Comprehensive documentation +- ✅ Example configurations +- ✅ Verification commands +- ✅ Troubleshooting guides + +## Deployment + +### Quick Start +```bash +cd jenkins-as-code/ansible + +# Copy and customize inventory +cp inventory-example.yml inventory.yml +# Edit inventory.yml with your servers and IPs + +# Test connectivity +ansible all -i inventory.yml -m ping + +# Deploy (dry run) +ansible-playbook -i inventory.yml setup-jenkins-host.yml --check + +# Deploy for real +ansible-playbook -i inventory.yml setup-jenkins-host.yml +``` + +### Selective Deployment +```bash +# Deploy only security roles +ansible-playbook -i inventory.yml setup-jenkins-host.yml --tags security + +# Deploy only NTP +ansible-playbook -i inventory.yml setup-jenkins-host.yml --tags system +``` + +## Important Notes + +### Unattended Upgrades +- **Disabled by default** - Enable after testing +- Only installs security updates +- No automatic reboots configured +- To enable: Edit `/etc/apt/apt.conf.d/20auto-upgrades` and set values to "1" + +### Fail2ban IP Whitelist +- **CRITICAL:** Update `fail2ban_ignoreip` in inventory before deployment +- Default whitelist includes only loopback and private networks +- Add your management IPs to prevent lockout +- Example from production: `78.47.239.96 46.224.123.39 178.62.115.224 20.90.182.165` + +### NTP Service +- Uses `ntpsec` for Ubuntu 20.04+ +- Uses `ntp` for older Ubuntu versions +- Override with `ntp_service` and `ntp_package` variables if needed + +## Verification + +After deployment, verify each component: + +```bash +# Unattended upgrades +sudo systemctl status unattended-upgrades +cat /etc/apt/apt.conf.d/50unattended-upgrades + +# NTP +sudo systemctl status ntpsec +ntpq -p +timedatectl status + +# Fail2ban +sudo systemctl status fail2ban +sudo fail2ban-client status +sudo fail2ban-client status sshd +``` + +## Testing Status + +- ✅ Ansible syntax check passed +- ✅ Role structure verified +- ✅ YAML syntax validated +- ⚠️ Deployment testing pending (requires target server) + +## Next Steps + +1. **Test in staging environment** + - Deploy to a test server + - Verify all services start correctly + - Test fail2ban banning/unbanning + - Verify NTP synchronization + +2. **Update IP whitelist** + - Add all management IPs to inventory + - Add VPN networks if applicable + - Document all whitelisted IPs + +3. **Enable unattended-upgrades** (after testing) + - Edit `/etc/apt/apt.conf.d/20auto-upgrades` + - Set both values to "1" + - Monitor for issues + +4. **Monitor and maintain** + - Check fail2ban logs regularly + - Verify NTP sync status + - Review security updates + +## Files Created + +``` +jenkins-as-code/ansible/ +├── roles/ +│ ├── unattended_upgrades/ +│ │ └── tasks/ +│ │ └── main.yml +│ ├── ntp_config/ +│ │ ├── tasks/ +│ │ │ └── main.yml +│ │ ├── handlers/ +│ │ │ └── main.yml +│ │ └── defaults/ +│ │ └── main.yml +│ ├── fail2ban/ +│ │ ├── tasks/ +│ │ │ └── main.yml +│ │ ├── handlers/ +│ │ │ └── main.yml +│ │ └── defaults/ +│ │ └── main.yml +│ └── README.md +├── setup-jenkins-host.yml (updated) +├── inventory-example.yml +└── DEPLOYMENT-GUIDE.md +``` + +## References + +- [CONFIG-ANALYSIS.md](CONFIG-ANALYSIS.md) - Analysis of extracted Jenkins master configuration +- [roles/README.md](ansible/roles/README.md) - Detailed role documentation +- [DEPLOYMENT-GUIDE.md](ansible/DEPLOYMENT-GUIDE.md) - Step-by-step deployment guide +- [inventory-example.yml](ansible/inventory-example.yml) - Example inventory configuration + +--- + +**Implementation Date:** 2026-06-24 +**Based on:** Production Jenkins master configuration +**Status:** Ready for testing and deployment \ No newline at end of file diff --git a/jenkins-as-code/docs/JENKINS-INSTALL.md b/jenkins-as-code/docs/JENKINS-INSTALL.md new file mode 100644 index 0000000000..c2d1a00c14 --- /dev/null +++ b/jenkins-as-code/docs/JENKINS-INSTALL.md @@ -0,0 +1,418 @@ +# Jenkins Server Installation Guide + +This guide covers the installation of Jenkins server using the `install-jenkins-server.yml` playbook. + +## Overview + +The `install-jenkins-server.yml` playbook installs Jenkins LTS on Ubuntu 24.04 with the Jenkins home directory configured at `/home/jenkins/.jenkins` to mirror the production Jenkins master setup. + +## Prerequisites + +Before running this playbook, you must: + +1. **Run the host setup playbook first:** + ```bash + ansible-playbook setup-jenkins-host.yml --connection=local + ``` + This creates the Jenkins user, installs Java, and configures the base system. + +2. **Ensure you have:** + - Ubuntu 24.04 system + - Root/sudo access + - Internet connectivity for package downloads + +## Installation + +### Quick Start + +```bash +# 1. First, set up the Jenkins host (if not already done) +ansible-playbook setup-jenkins-host.yml --connection=local + +# 2. Install Jenkins server +ansible-playbook install-jenkins-server.yml --connection=local +``` + +### What Gets Installed + +The playbook performs the following: + +1. **Pre-flight Checks:** + - Verifies Ubuntu 24.04 + - Confirms Jenkins user exists + - Confirms Java is installed + +2. **Jenkins Installation:** + - Adds Jenkins official repository + - Installs Jenkins LTS package + - Installs required dependencies + +3. **Directory Structure:** + ``` + /home/jenkins/.jenkins/ # Jenkins home (JENKINS_HOME) + ├── plugins/ # Jenkins plugins + ├── jobs/ # Jenkins jobs + ├── workspace/ # Build workspaces + ├── updates/ # Update center data + └── secrets/ # Secrets (created by Jenkins) + └── initialAdminPassword # Initial admin password + + /var/cache/jenkins/ # Jenkins cache + └── war/ # Exploded WAR files + + /var/log/jenkins/ # Jenkins logs + └── jenkins.log # Main log file + ``` + +4. **Service Configuration:** + - Configures Jenkins to run as the `jenkins` user + - Sets `JENKINS_HOME=/home/jenkins/.jenkins` + - Configures systemd service with proper security settings + - Sets resource limits (file descriptors, processes) + +5. **Startup:** + - Enables Jenkins service to start on boot + - Starts Jenkins and waits for it to be ready + - Retrieves the initial admin password + +## Configuration Variables + +You can customize the installation by modifying variables in the playbook: + +```yaml +vars: + jenkins_username: jenkins # Jenkins system user + jenkins_home: /home/jenkins # User home directory + jenkins_data_dir: /home/jenkins/.jenkins # Jenkins home directory + jenkins_version: "lts" # Jenkins version (lts or specific version) + jenkins_port: 8080 # HTTP port + jenkins_java_opts: "-Djava.awt.headless=true -Xmx2048m -Xms512m" # JVM options + jenkins_args: "--webroot=/var/cache/jenkins/war --httpPort=8080" # Jenkins args +``` + +### Customizing JVM Memory + +To adjust Jenkins memory allocation, modify `jenkins_java_opts`: + +```yaml +# For a system with 8GB RAM, allocate 4GB to Jenkins +jenkins_java_opts: "-Djava.awt.headless=true -Xmx4096m -Xms1024m" +``` + +### Changing the Port + +To run Jenkins on a different port: + +```yaml +jenkins_port: 9090 +jenkins_args: "--webroot=/var/cache/jenkins/war --httpPort=9090" +``` + +## Post-Installation + +### 1. Access Jenkins + +After installation completes, the playbook displays: +- Jenkins URL (e.g., `http://192.168.1.100:8080`) +- Initial admin password + +Access Jenkins in your web browser using the provided URL. + +### 2. Unlock Jenkins + +Use the initial admin password displayed by the playbook, or retrieve it manually: + +```bash +sudo cat /home/jenkins/.jenkins/secrets/initialAdminPassword +``` + +### 3. Install Plugins + +Choose one of: +- **Install suggested plugins** (recommended for most users) +- **Select plugins to install** (for custom setups) + +### 4. Create Admin User + +Create your first administrator account with: +- Username +- Password +- Full name +- Email address + +### 5. Configure Jenkins URL + +Confirm or update the Jenkins URL for your environment. + +## Service Management + +### Check Jenkins Status + +```bash +sudo systemctl status jenkins +``` + +### Start/Stop/Restart Jenkins + +```bash +sudo systemctl start jenkins +sudo systemctl stop jenkins +sudo systemctl restart jenkins +``` + +### View Jenkins Logs + +```bash +# Real-time log viewing +sudo journalctl -u jenkins -f + +# View recent logs +sudo journalctl -u jenkins -n 100 + +# View logs in file +sudo tail -f /var/log/jenkins/jenkins.log +``` + +### Check Jenkins Configuration + +```bash +# View systemd service configuration +sudo systemctl cat jenkins + +# View environment variables +sudo cat /etc/default/jenkins + +# View systemd override +sudo cat /etc/systemd/system/jenkins.service.d/override.conf +``` + +## Verification + +### Verify Installation + +```bash +# Check Jenkins is running +sudo systemctl is-active jenkins + +# Check Jenkins is listening on port 8080 +sudo netstat -tlnp | grep 8080 +# or +sudo ss -tlnp | grep 8080 + +# Test HTTP access +curl -I http://localhost:8080 +``` + +### Verify Directory Structure + +```bash +# Check Jenkins home directory +ls -la /home/jenkins/.jenkins/ + +# Check ownership +ls -ld /home/jenkins/.jenkins/ +# Should show: drwxr-xr-x jenkins jenkins + +# Check Jenkins is using correct home +sudo systemctl show jenkins | grep JENKINS_HOME +``` + +## Troubleshooting + +### Jenkins Won't Start + +1. **Check logs:** + ```bash + sudo journalctl -u jenkins -n 50 + ``` + +2. **Check Java:** + ```bash + java -version + ``` + +3. **Check permissions:** + ```bash + ls -la /home/jenkins/.jenkins/ + sudo chown -R jenkins:jenkins /home/jenkins/.jenkins/ + ``` + +4. **Check port availability:** + ```bash + sudo netstat -tlnp | grep 8080 + ``` + +### Port Already in Use + +If port 8080 is already in use: + +1. Find what's using it: + ```bash + sudo lsof -i :8080 + ``` + +2. Either stop that service or change Jenkins port in the playbook. + +### Permission Denied Errors + +```bash +# Fix ownership of Jenkins directories +sudo chown -R jenkins:jenkins /home/jenkins/.jenkins/ +sudo chown -R jenkins:jenkins /var/cache/jenkins/ +sudo chown -R jenkins:jenkins /var/log/jenkins/ +``` + +### Jenkins Slow to Start + +Jenkins can take 1-2 minutes to fully start, especially on first run. Be patient and check logs: + +```bash +sudo journalctl -u jenkins -f +``` + +## Security Considerations + +### Initial Setup + +1. **Change admin password immediately** after first login +2. **Enable security realm** (Jenkins' own user database or LDAP) +3. **Configure authorization strategy** (Matrix-based security recommended) +4. **Install security plugins:** + - OWASP Markup Formatter + - Matrix Authorization Strategy + - Role-based Authorization Strategy + +### Firewall Configuration + +If using a firewall, allow Jenkins port: + +```bash +# UFW +sudo ufw allow 8080/tcp + +# iptables +sudo iptables -A INPUT -p tcp --dport 8080 -j ACCEPT +``` + +### Reverse Proxy (Recommended for Production) + +For production, use Nginx or Apache as a reverse proxy with HTTPS: + +```nginx +# Nginx example +server { + listen 80; + server_name jenkins.example.com; + return 301 https://$server_name$request_uri; +} + +server { + listen 443 ssl http2; + server_name jenkins.example.com; + + ssl_certificate /etc/ssl/certs/jenkins.crt; + ssl_certificate_key /etc/ssl/private/jenkins.key; + + location / { + proxy_pass http://localhost:8080; + proxy_set_header Host $host; + proxy_set_header X-Real-IP $remote_addr; + proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; + proxy_set_header X-Forwarded-Proto $scheme; + } +} +``` + +## Backup and Restore + +### Backup Jenkins Home + +```bash +# Stop Jenkins +sudo systemctl stop jenkins + +# Backup Jenkins home directory +sudo tar -czf jenkins-backup-$(date +%Y%m%d).tar.gz -C /home/jenkins .jenkins/ + +# Start Jenkins +sudo systemctl start jenkins +``` + +### Restore Jenkins Home + +```bash +# Stop Jenkins +sudo systemctl stop jenkins + +# Restore backup +sudo tar -xzf jenkins-backup-20260625.tar.gz -C /home/jenkins/ + +# Fix permissions +sudo chown -R jenkins:jenkins /home/jenkins/.jenkins/ + +# Start Jenkins +sudo systemctl start jenkins +``` + +## Upgrading Jenkins + +### Via Package Manager (Recommended) + +```bash +# Update package list +sudo apt update + +# Upgrade Jenkins +sudo apt upgrade jenkins + +# Restart Jenkins +sudo systemctl restart jenkins +``` + +### Manual Upgrade + +1. Download new jenkins.war +2. Stop Jenkins +3. Replace /usr/share/java/jenkins.war +4. Start Jenkins + +## Uninstallation + +To completely remove Jenkins: + +```bash +# Stop Jenkins +sudo systemctl stop jenkins +sudo systemctl disable jenkins + +# Remove package +sudo apt remove --purge jenkins + +# Remove directories (CAUTION: This deletes all Jenkins data!) +sudo rm -rf /home/jenkins/.jenkins/ +sudo rm -rf /var/cache/jenkins/ +sudo rm -rf /var/log/jenkins/ + +# Remove repository +sudo rm /etc/apt/sources.list.d/jenkins.list +sudo apt-key del $(apt-key list | grep -B 1 "Jenkins" | head -n 1 | awk '{print $2}') +``` + +## Related Documentation + +- [setup-jenkins-host.yml](setup-jenkins-host.yml) - Host preparation playbook +- [DEPLOYMENT-GUIDE.md](DEPLOYMENT-GUIDE.md) - Overall deployment guide +- [roles/README.md](roles/README.md) - Ansible roles documentation +- [Official Jenkins Documentation](https://www.jenkins.io/doc/) + +## Support + +For issues or questions: +1. Check Jenkins logs: `sudo journalctl -u jenkins -f` +2. Review this documentation +3. Consult [Jenkins Documentation](https://www.jenkins.io/doc/) +4. Check [Jenkins Community Forums](https://community.jenkins.io/) + +--- + +**Made with Bob** \ No newline at end of file diff --git a/jenkins-as-code/docs/PRODUCTION-CONFIG-NOTES.md b/jenkins-as-code/docs/PRODUCTION-CONFIG-NOTES.md new file mode 100644 index 0000000000..4233c96633 --- /dev/null +++ b/jenkins-as-code/docs/PRODUCTION-CONFIG-NOTES.md @@ -0,0 +1,244 @@ +# Production Jenkins Configuration Notes + +This document details the production Jenkins configuration settings used in the `install-jenkins-server.yml` playbook, based on the adoptium/infrastructure Jenkins master. + +## Configuration Source + +These settings are extracted from the production Jenkins master at Hetzner running Ubuntu 20.04, and have been adapted for Ubuntu 24.04 deployment. + +## Key Configuration Settings + +### Jenkins Home Directory + +```bash +JENKINS_HOME=/home/jenkins/.jenkins +``` + +**Rationale:** Mirrors production setup. This differs from the default `/var/lib/jenkins` to keep all Jenkins data under the jenkins user's home directory for easier management and backup. + +### JVM Memory Settings + +```bash +JAVA_ARGS="-Xmx19G ..." +``` + +**Current Setting:** 19GB maximum heap +**System Requirement:** 32GB+ RAM recommended + +**History of Memory Tuning:** +1. Initial: `-Xmx20g -Xms8g` +2. Added GC tuning: `-Xmx16g -Xms8g` with G1GC and detailed logging +3. Increased to `-Xmx18G` (2023-01-09, issue #2875) +4. Further tuned to `-Xmx20G` with G1GC optimizations +5. Adjusted to `-Xmx22G` with XStream limits +6. **Current:** `-Xmx19G` (2026-05-28, issue #4364) with JUnit optimizations + +**Recommendation for Different System Sizes:** +- 4GB RAM: `-Xmx2G` +- 8GB RAM: `-Xmx4G` +- 16GB RAM: `-Xmx8G` +- 32GB+ RAM: `-Xmx19G` (production) + +### JUnit Memory Optimizations + +```bash +-Dhudson.tasks.junit.TestResultAction.RESULT_CACHE_ENABLED=false +-Dhudson.tasks.junit.History\$HistoryTableResult.PREVIOUS_TEST_RESULT_BACKTRACK_BUILDS_MAX=1 +``` + +**Added:** 2026-05-28 by sxa & aleonard +**Issue:** https://github.com/adoptium/infrastructure/issues/4364 +**Purpose:** Reduce memory consumption from JUnit test result caching + +### GC Logging Configuration + +```bash +-Xlog:gc*,gc+heap=info,gc+age=trace,gc+phases=trace,safepoint:file=/var/log/jenkins/gc.log:time,uptime,level,tags:filecount=5,filesize=50m +``` + +**Features:** +- Detailed GC logging with heap info, age tracking, and phase details +- Rotating log files: 5 files × 50MB each = 250MB total +- Includes timestamps and log levels +- Logs safepoint information for performance analysis + +**Log Location:** `/var/log/jenkins/gc.log` + +### XStream Security + +```bash +-Dhudson.util.XStream2.collectionUpdateLimit=-1 +``` + +**Purpose:** Removes the default limit on XStream collection updates +**Caution:** This is a security-related setting. The default limit prevents potential DoS attacks via large XML payloads. Set to -1 only if you trust all XML sources. + +### Network Configuration + +```bash +JENKINS_ARGS="... --httpListenAddress=127.0.0.1" +``` + +**Setting:** Listen on localhost only (127.0.0.1) +**Rationale:** Jenkins should be accessed through a reverse proxy (Nginx/Apache) with HTTPS +**Security:** Prevents direct external access to Jenkins + +### Session Management + +```bash +--sessionTimeout=720 --sessionEviction=43200 +``` + +**Session Timeout:** 720 minutes (12 hours) +**Session Eviction:** 43,200 seconds (12 hours) + +**Added by:** Martijn (2019-03-21) and sxa (2024-05-09) +**Purpose:** Balance security with user convenience for long-running operations + +### Access Logging + +```bash +--accessLoggerClassName=winstone.accesslog.SimpleAccessLogger +--simpleAccessLogger.format=combined +--simpleAccessLogger.file=/var/log/jenkins/access.log +``` + +**Added by:** Martijn (2019-03-21) +**Format:** Combined log format (Apache-style) +**Location:** `/var/log/jenkins/access.log` +**Purpose:** Audit trail and troubleshooting + +### File Descriptor Limits + +```bash +MAXOPENFILES=8192 +``` + +**Setting:** 8,192 open files +**Rationale:** Jenkins can have many concurrent connections and file operations +**Note:** Also set in systemd service override (`LimitNOFILE=8192`) + +### Java Location + +```bash +JAVA=/usr/lib/jvm/temurin-25-jdk-amd64/bin/java +``` + +**JDK:** Eclipse Temurin 25 (Adoptium) +**Architecture:** amd64 +**Source:** Adoptium APT repository + +## Directory Structure + +``` +/home/jenkins/.jenkins/ # Jenkins home (JENKINS_HOME) +├── config.xml # Main Jenkins configuration +├── plugins/ # Installed plugins +├── jobs/ # Job configurations +├── workspace/ # Build workspaces +├── updates/ # Update center data +├── secrets/ # Secrets and credentials +│ └── initialAdminPassword # Initial setup password +└── logs/ # Jenkins internal logs + +/var/cache/jenkins/ # Jenkins cache +└── war/ # Exploded WAR files + +/var/log/jenkins/ # Jenkins logs +├── jenkins.log # Main application log +├── access.log # HTTP access log +└── gc.log # Garbage collection log +``` + +## Security Considerations + +### 1. Localhost-Only Binding + +Jenkins listens only on 127.0.0.1, requiring a reverse proxy for external access. This provides: +- HTTPS termination at the proxy +- Additional security layer +- Better logging and monitoring +- DDoS protection + +### 2. File Permissions + +```bash +# UMASK not set (defaults to 022) +# Sensitive files (credentials) always written with restricted permissions +``` + +### 3. Systemd Security + +```ini +NoNewPrivileges=true # Prevents privilege escalation +PrivateTmp=true # Isolated /tmp directory +``` + +## Performance Tuning Notes + +### Memory Allocation Strategy + +The production system uses 19GB heap on a 32GB+ RAM system, leaving approximately: +- 10-12GB for OS and other processes +- 2-3GB for file system cache +- Headroom for memory spikes + +### GC Strategy + +Using default G1GC (Java 9+) with: +- Detailed logging for performance analysis +- Rotating logs to prevent disk space issues +- Safepoint logging for identifying pause causes + +### JUnit Optimization + +Disabling JUnit result caching significantly reduces memory usage for projects with extensive test suites, at the cost of slightly slower test result page loads. + +## Monitoring Recommendations + +### 1. GC Logs + +Monitor `/var/log/jenkins/gc.log` for: +- Long GC pauses (>1 second) +- Frequent full GCs +- Heap exhaustion warnings + +### 2. Access Logs + +Monitor `/var/log/jenkins/access.log` for: +- Unusual access patterns +- Failed authentication attempts +- Performance issues (slow requests) + +### 3. System Resources + +Monitor: +- Memory usage (should stay below 90% of allocated heap) +- CPU usage during builds +- Disk I/O for workspace operations +- Network throughput + +## Upgrade Considerations + +When upgrading Jenkins or Java: + +1. **Backup First:** Always backup `/home/jenkins/.jenkins/` before upgrades +2. **Test GC Settings:** New Java versions may have different GC defaults +3. **Review Logs:** Check GC logs after upgrade for performance changes +4. **Monitor Memory:** Watch for memory leaks or increased usage +5. **Plugin Compatibility:** Verify all plugins work with new Jenkins version + +## References + +- [Adoptium Infrastructure Issue #2875](https://github.com/adoptium/infrastructure/issues/2875) - Memory increase to 18G +- [Adoptium Infrastructure Issue #4364](https://github.com/adoptium/infrastructure/issues/4364) - JUnit memory optimizations +- [Jenkins Performance Tuning](https://www.jenkins.io/doc/book/scaling/hardware-recommendations/) +- [G1GC Tuning Guide](https://docs.oracle.com/en/java/javase/17/gctuning/garbage-first-g1-garbage-collector1.html) + +--- + +**Last Updated:** 2026-06-25 +**Based on:** Production Jenkins master configuration (Ubuntu 20.04) +**Target:** Ubuntu 24.04 deployment + +**Made with Bob** \ No newline at end of file diff --git a/jenkins-as-code/docs/QUICK-START.md b/jenkins-as-code/docs/QUICK-START.md new file mode 100644 index 0000000000..a4d20d558c --- /dev/null +++ b/jenkins-as-code/docs/QUICK-START.md @@ -0,0 +1,275 @@ +# Jenkins Infrastructure Quick Start Guide + +This guide provides a quick reference for deploying a complete Jenkins infrastructure using the Ansible playbooks. + +## Overview + +The Jenkins infrastructure deployment consists of two main playbooks: + +1. **`setup-jenkins-host.yml`** - Prepares the host system +2. **`install-jenkins-server.yml`** - Installs and configures Jenkins + +## Prerequisites + +- Ubuntu 24.04 system +- Root/sudo access +- Internet connectivity +- Ansible installed + +## Complete Deployment (Two-Step Process) + +### Step 1: Prepare the Host + +This playbook sets up the base system with: +- Jenkins user (UID/GID 1000 if available) +- Java 25 (Temurin JDK) +- Essential packages +- Security hardening (fail2ban, SSH configuration) +- System services (NTP, unattended upgrades) + +```bash +cd jenkins-as-code/ansible +ansible-playbook setup-jenkins-host.yml --connection=local +``` + +**Duration:** ~5-10 minutes + +### Step 2: Install Jenkins Server + +This playbook installs Jenkins with: +- Jenkins LTS from official repository +- Jenkins home at `/home/jenkins/.jenkins` +- Systemd service configuration +- Proper directory structure and permissions + +```bash +ansible-playbook install-jenkins-server.yml --connection=local +``` + +**Duration:** ~3-5 minutes + +### Step 3: Access Jenkins + +After installation completes, the playbook displays: +- Jenkins URL (e.g., `http://192.168.1.100:8080`) +- Initial admin password + +Open the URL in your browser and use the password to unlock Jenkins. + +## One-Command Deployment + +To run both playbooks sequentially: + +```bash +ansible-playbook setup-jenkins-host.yml --connection=local && \ +ansible-playbook install-jenkins-server.yml --connection=local +``` + +## Directory Structure + +After deployment, your Jenkins installation will have: + +``` +/home/jenkins/ +├── .jenkins/ # Jenkins home directory (JENKINS_HOME) +│ ├── config.xml # Jenkins configuration +│ ├── plugins/ # Installed plugins +│ ├── jobs/ # Jenkins jobs +│ ├── workspace/ # Build workspaces +│ ├── updates/ # Update center data +│ ├── secrets/ # Secrets directory +│ │ └── initialAdminPassword +│ └── logs/ # Jenkins logs +│ +├── .ssh/ # SSH configuration +│ ├── authorized_keys # SSH public keys +│ └── known_hosts # Known SSH hosts +│ +/var/cache/jenkins/ # Jenkins cache +└── war/ # Exploded WAR files + +/var/log/jenkins/ # Jenkins logs +└── jenkins.log # Main log file +``` + +## Key Configuration Details + +### Jenkins User +- **Username:** `jenkins` +- **UID/GID:** 1000 (if available, otherwise system-assigned) +- **Home:** `/home/jenkins` +- **Shell:** `/bin/bash` +- **Sudo:** No (security best practice) + +### Jenkins Installation +- **Home Directory:** `/home/jenkins/.jenkins` +- **HTTP Port:** 8080 +- **Service User:** `jenkins` +- **Java Version:** Temurin 25 JDK +- **JVM Memory:** 2GB max, 512MB min (configurable) + +### Security Features +- SSH password authentication disabled +- Fail2ban protecting SSH (3 attempts, 10-minute window) +- Unattended security updates configured (disabled by default) +- NTP time synchronization +- Proper file permissions and ownership + +## Post-Installation Checklist + +- [ ] Access Jenkins web interface +- [ ] Complete initial setup wizard +- [ ] Install recommended plugins +- [ ] Create admin user +- [ ] Configure Jenkins URL +- [ ] Set up backup strategy +- [ ] Configure firewall rules (if needed) +- [ ] Set up reverse proxy with HTTPS (recommended for production) + +## Common Commands + +### Service Management +```bash +# Check Jenkins status +sudo systemctl status jenkins + +# Restart Jenkins +sudo systemctl restart jenkins + +# View logs +sudo journalctl -u jenkins -f +``` + +### File Operations +```bash +# View initial admin password +sudo cat /home/jenkins/.jenkins/secrets/initialAdminPassword + +# Check Jenkins home ownership +ls -la /home/jenkins/.jenkins/ + +# View Jenkins configuration +sudo cat /etc/default/jenkins +``` + +### Verification +```bash +# Check Jenkins is listening +sudo netstat -tlnp | grep 8080 + +# Test HTTP access +curl -I http://localhost:8080 + +# Check Java version +java -version +``` + +## Customization + +### Change Jenkins Port + +Edit `install-jenkins-server.yml`: +```yaml +vars: + jenkins_port: 9090 # Change from 8080 + jenkins_args: "--webroot=/var/cache/jenkins/war --httpPort=9090" +``` + +### Adjust JVM Memory + +Edit `install-jenkins-server.yml`: +```yaml +vars: + jenkins_java_opts: "-Djava.awt.headless=true -Xmx4096m -Xms1024m" +``` + +### Customize SSH Key + +Before running `setup-jenkins-host.yml`, set environment variable: +```bash +export JENKINS_SSH_KEY="ssh-rsa AAAAB3... your-key-here" +ansible-playbook setup-jenkins-host.yml --connection=local +``` + +Or edit the playbook directly: +```yaml +vars: + jenkins_ssh_key: "ssh-rsa AAAAB3... your-key-here" +``` + +## Troubleshooting + +### Jenkins Won't Start +```bash +# Check logs +sudo journalctl -u jenkins -n 50 + +# Check Java +java -version + +# Fix permissions +sudo chown -R jenkins:jenkins /home/jenkins/.jenkins/ +``` + +### Port Already in Use +```bash +# Find what's using port 8080 +sudo lsof -i :8080 + +# Change Jenkins port in playbook or stop conflicting service +``` + +### Can't Access Jenkins +```bash +# Check firewall +sudo ufw status +sudo ufw allow 8080/tcp + +# Check Jenkins is running +sudo systemctl status jenkins + +# Check network binding +sudo netstat -tlnp | grep 8080 +``` + +## Next Steps + +After successful deployment: + +1. **Complete Jenkins Setup:** + - Install plugins + - Configure security + - Set up credentials + - Create jobs + +2. **Configure Backups:** + - Set up automated backups of `/home/jenkins/.jenkins/` + - Test restore procedures + +3. **Production Hardening:** + - Set up HTTPS with reverse proxy + - Configure firewall rules + - Enable unattended security updates + - Set up monitoring + +4. **Integration:** + - Connect to version control (Git, GitHub, etc.) + - Configure build agents + - Set up notifications + +## Documentation + +- [JENKINS-INSTALL.md](JENKINS-INSTALL.md) - Detailed Jenkins installation guide +- [DEPLOYMENT-GUIDE.md](DEPLOYMENT-GUIDE.md) - Complete deployment documentation +- [roles/README.md](roles/README.md) - Ansible roles documentation +- [VAGRANT-DEPLOYMENT.md](VAGRANT-DEPLOYMENT.md) - Vagrant testing guide + +## Support Resources + +- [Jenkins Documentation](https://www.jenkins.io/doc/) +- [Jenkins Community](https://community.jenkins.io/) +- [Ansible Documentation](https://docs.ansible.com/) + +--- + +**Made with Bob** \ No newline at end of file diff --git a/jenkins-as-code/docs/README-ubuntu24-jenkins-setup.md b/jenkins-as-code/docs/README-ubuntu24-jenkins-setup.md new file mode 100644 index 0000000000..ec6807ca43 --- /dev/null +++ b/jenkins-as-code/docs/README-ubuntu24-jenkins-setup.md @@ -0,0 +1,135 @@ +# Ubuntu 24.04 Jenkins User Setup - Ansible Playbook + +This playbook configures an Ubuntu 24.04 system with a Jenkins user account, SSH access, and proper permissions for Jenkins automation. + +## Overview + +The playbook performs the following tasks: +- Verifies the target system is running Ubuntu 24.04 +- Updates the apt package cache +- Installs essential packages (openssh-server, sudo, python3, python3-pip) +- Creates a Jenkins user with home directory (no sudo access for security) +- Configures SSH access with authorized keys +- Configures system limits for the Jenkins user +- Adds GitHub to known_hosts + +## Prerequisites + +- Ansible installed on the Ubuntu 24.04 host +- Root or sudo access on the host +- SSH key for Jenkins user (set via environment variable or update in playbook) + +## Files + +- `ubuntu24-jenkins-setup.yml` - Main Ansible playbook (runs locally on the host) + +## Usage + +### 1. Install Ansible on the host + +```bash +sudo apt update +sudo apt install -y ansible +``` + +### 2. Set up your SSH key + +Export your Jenkins SSH public key as an environment variable: + +```bash +export JENKINS_SSH_KEY="ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC... jenkins@adoptopenjdk" +``` + +Or edit the playbook directly to set the `jenkins_ssh_key` variable. + +### 3. Run the playbook locally + +Execute the playbook on the host itself: + +```bash +sudo ansible-playbook ubuntu24-jenkins-setup.yml --connection=local +``` + +Or without sudo if you're already root: + +```bash +ansible-playbook ubuntu24-jenkins-setup.yml --connection=local +``` + +### 4. Run specific tags (optional) + +You can run specific parts of the playbook using tags: + +```bash +# Only setup system packages +sudo ansible-playbook ubuntu24-jenkins-setup.yml --connection=local --tags setup + +# Only configure Jenkins user +sudo ansible-playbook ubuntu24-jenkins-setup.yml --connection=local --tags jenkins_user +``` + +## Configuration Variables + +The playbook uses the following variables (defined in the playbook): + +- `jenkins_username`: Username for the Jenkins user (default: `jenkins`) +- `jenkins_home`: Home directory path (default: `/home/jenkins`) +- `jenkins_ssh_key`: SSH public key for authentication + +## Security Considerations + +- **Production-Ready Security**: The Jenkins user has NO sudo access (runs with minimal privileges) +- SSH key authentication is required (no password authentication) +- Password expiry is disabled for the Jenkins user to prevent service interruption +- System limits are set to allow Jenkins to handle many processes and open files +- If elevated privileges are needed for specific tasks, use a separate admin account + +## Verification + +After running the playbook, you can verify the setup: + +```bash +# SSH into the VM as Jenkins user +ssh -i /path/to/jenkins/private/key jenkins@ + +# Verify user has no sudo access (should show "not allowed") +sudo -l + +# Verify limits +ulimit -a + +# Check user groups (should NOT include sudo) +groups jenkins +``` + +## Troubleshooting + +### Playbook Fails on Ubuntu Version Check + +If the playbook fails on the Ubuntu version assertion: +- Verify you're running Ubuntu 24.04: `lsb_release -a` +- If using a different version, update the assertion in the playbook + +### SSH Key Not Working + +If SSH key authentication fails: +1. Verify the public key is correctly set in the playbook or environment variable +2. Check the key format (should start with `ssh-rsa`, `ssh-ed25519`, etc.) +3. Ensure the private key has correct permissions: `chmod 600 /path/to/private/key` + +## Tags Reference + +- `always` - Tasks that always run (verification, display info) +- `setup` - System setup tasks (packages, SSH service) +- `jenkins_user` - Jenkins user creation and configuration + +## Related Files + +This playbook is part of the Jenkins-as-Code infrastructure setup located in `jenkins-as-code/ansible/playbooks/`. + +## License + +This playbook is part of the AdoptOpenJDK infrastructure project. + +--- +*Made with Bob* \ No newline at end of file diff --git a/jenkins-as-code/docs/VAGRANT-DEPLOYMENT.md b/jenkins-as-code/docs/VAGRANT-DEPLOYMENT.md new file mode 100644 index 0000000000..0f6abd7e62 --- /dev/null +++ b/jenkins-as-code/docs/VAGRANT-DEPLOYMENT.md @@ -0,0 +1,207 @@ +# Vagrant Dev Server Deployment Guide + +Quick guide for deploying security and system configuration to your Vagrant dev server running on localhost. + +## Your Configuration + +Your `hosts` file is already configured with: +- **Connection:** localhost via local connection +- **Fail2ban Whitelist:** Production Jenkins master IPs included +- **Group:** `local` group for localhost + +## Quick Commands + +### 1. Check Configuration +```bash +cd jenkins-as-code/ansible + +# Verify inventory +ansible-inventory -i hosts --list + +# Check connectivity +ansible -i hosts local -m ping +``` + +### 2. Deploy Everything (Dry Run) +```bash +ansible-playbook -i hosts setup-jenkins-host.yml --check --diff +``` + +### 3. Deploy Everything (For Real) +```bash +ansible-playbook -i hosts setup-jenkins-host.yml +``` + +### 4. Deploy Only Security Roles +```bash +# Unattended-upgrades + Fail2ban +ansible-playbook -i hosts setup-jenkins-host.yml --tags security +``` + +### 5. Deploy Only NTP +```bash +ansible-playbook -i hosts setup-jenkins-host.yml --tags system +``` + +### 6. Deploy Specific Role +```bash +# Just fail2ban +ansible-playbook -i hosts setup-jenkins-host.yml --tags security --skip-tags unattended_upgrades + +# Just NTP +ansible-playbook -i hosts setup-jenkins-host.yml --tags system + +# Just unattended-upgrades +ansible-playbook -i hosts setup-jenkins-host.yml --tags security --skip-tags fail2ban +``` + +## Post-Deployment Verification + +### Check Services +```bash +# On your Vagrant server +sudo systemctl status fail2ban +sudo systemctl status ntpsec +sudo systemctl status unattended-upgrades +``` + +### Verify Fail2ban Configuration +```bash +# Check fail2ban status +sudo fail2ban-client status + +# Check SSH jail +sudo fail2ban-client status sshd + +# Verify your IP whitelist is loaded +sudo cat /etc/fail2ban/jail.local | grep ignoreip +``` + +### Verify NTP +```bash +# Check NTP peers +ntpq -p + +# Check time sync status +timedatectl status +``` + +### Verify Unattended Upgrades +```bash +# Check configuration +cat /etc/apt/apt.conf.d/50unattended-upgrades +cat /etc/apt/apt.conf.d/20auto-upgrades + +# Test (dry run) +sudo unattended-upgrade --dry-run --debug +``` + +## Your Fail2ban Whitelist + +Your `hosts` file includes the production whitelist: +``` +127.0.0.1/8 ::1 10.0.0.0/8 192.168.0.0/16 78.47.239.96 46.224.123.39 178.62.115.224 20.90.182.165 +``` + +This means these IPs will NEVER be banned: +- **127.0.0.1/8, ::1** - Loopback (localhost) +- **10.0.0.0/8, 192.168.0.0/16** - Private networks (your Vagrant network) +- **78.47.239.96, 46.224.123.39, 178.62.115.224, 20.90.182.165** - Production trusted IPs + +## Troubleshooting + +### Playbook Fails with Permission Denied +The playbook uses `become: yes` for privilege escalation. Make sure your user can sudo: +```bash +# Test sudo access +sudo -v +``` + +### Fail2ban Not Starting +```bash +# Check logs +sudo journalctl -u fail2ban -n 50 + +# Test configuration +sudo fail2ban-client -t + +# Restart service +sudo systemctl restart fail2ban +``` + +### NTP Not Syncing +```bash +# Check if service is running +sudo systemctl status ntpsec + +# Check network connectivity to NTP servers +ping -c 3 0.ubuntu.pool.ntp.org + +# Restart service +sudo systemctl restart ntpsec +``` + +## Modifying Configuration + +### Change Fail2ban Whitelist +Edit `hosts` file and update the `fail2ban_ignoreip` line: +```ini +[local:vars] +fail2ban_ignoreip=127.0.0.1/8 ::1 10.0.0.0/8 192.168.0.0/16 YOUR_NEW_IP +``` + +Then re-run: +```bash +ansible-playbook -i hosts setup-jenkins-host.yml --tags security +``` + +### Enable Unattended Upgrades +After testing, enable automatic updates: +```bash +# On Vagrant server +sudo nano /etc/apt/apt.conf.d/20auto-upgrades + +# Change both values to "1": +APT::Periodic::Update-Package-Lists "1"; +APT::Periodic::Unattended-Upgrade "1"; +``` + +## Complete Deployment Example + +```bash +# Navigate to ansible directory +cd jenkins-as-code/ansible + +# 1. Verify configuration +ansible-inventory -i hosts --list + +# 2. Test connectivity +ansible -i hosts local -m ping + +# 3. Dry run to see what will change +ansible-playbook -i hosts setup-jenkins-host.yml --check --diff + +# 4. Deploy everything +ansible-playbook -i hosts setup-jenkins-host.yml + +# 5. Verify services are running +ansible -i hosts local -m shell -a "systemctl status fail2ban ntpsec" --become + +# 6. Check fail2ban status +ansible -i hosts local -m shell -a "fail2ban-client status" --become +``` + +## Notes + +- Your Vagrant server is accessed via `localhost` with local connection (no SSH) +- All roles use `become: yes` for privilege escalation +- The playbook is idempotent - safe to run multiple times +- Configuration matches production Jenkins master exactly + +--- + +**Quick Reference:** +- Inventory file: `hosts` +- Playbook: `setup-jenkins-host.yml` +- Roles: `unattended_upgrades`, `ntp_config`, `fail2ban` +- Tags: `security`, `system` \ No newline at end of file diff --git a/jenkins-as-code/jenkins-scripts/backup-jenkins-app-config.sh b/jenkins-as-code/jenkins-scripts/backup-jenkins-app-config.sh new file mode 100755 index 0000000000..b9cd055cf4 --- /dev/null +++ b/jenkins-as-code/jenkins-scripts/backup-jenkins-app-config.sh @@ -0,0 +1,198 @@ +#!/bin/bash +################################################################################ +# Jenkins Application Configuration Backup Script +# +# Captures Jenkins application-level configuration from JENKINS_HOME. +# Explicitly excludes jobs/, workspace/, cache/, logs/ and fingerprints/ — +# these are either too large, ephemeral, or managed separately. +# +# Each element is archived as its own named tarball inside the outer archive, +# allowing the restore script to selectively omit individual elements: +# +# config.tar.gz — config.xml + credentials.xml + root *.xml +# users.tar.gz — users/ +# secrets.tar.gz — secrets/, .key, secret.key, secret.key.not-so-secret +# plugins.tar.gz — plugins/ +# nodes.tar.gz — nodes/ +# crontab.txt — jenkins user crontab (plain text) +# +# Usage: +# sudo bash backup-jenkins-app-config.sh +# +# Override defaults via environment variables: +# JENKINS_HOME=/path/to/jenkins sudo bash backup-jenkins-app-config.sh +# JENKINS_USER=jenkins sudo bash backup-jenkins-app-config.sh +################################################################################ + +set -e + +# Configuration — override via environment variables if needed +JENKINS_HOME="${JENKINS_HOME:-/home/jenkins/.jenkins}" +JENKINS_USER="${JENKINS_USER:-jenkins}" + +TIMESTAMP=$(date +%Y%m%d-%H%M%S) +BACKUP_DIR="jenkins-app-backup-${TIMESTAMP}" +TARBALL="${BACKUP_DIR}.tar.gz" + +# Colors for output +RED='\033[0;31m' +GREEN='\033[0;32m' +YELLOW='\033[1;33m' +NC='\033[0m' + +# Check if running as root +if [[ $EUID -ne 0 ]]; then + echo -e "${RED}Error: This script must be run as root (use sudo)${NC}" + exit 1 +fi + +# Check JENKINS_HOME exists +if [ ! -d "$JENKINS_HOME" ]; then + echo -e "${RED}Error: JENKINS_HOME not found at $JENKINS_HOME${NC}" + echo -e "${YELLOW}Override with: JENKINS_HOME=/path/to/.jenkins sudo bash $0${NC}" + exit 1 +fi + +echo -e "${GREEN}========================================${NC}" +echo -e "${GREEN}Jenkins Application Config Backup${NC}" +echo -e "${GREEN}========================================${NC}" +echo "" +echo -e "JENKINS_HOME : ${YELLOW}${JENKINS_HOME}${NC}" +echo -e "JENKINS_USER : ${YELLOW}${JENKINS_USER}${NC}" +echo -e "Output : ${YELLOW}${TARBALL}${NC}" +echo "" + +mkdir -p "$BACKUP_DIR" + +# Helper — create a named tarball from one or more source paths. +# Files/dirs that don't exist are silently skipped; if nothing exists the +# tarball is not created and the element is marked as skipped. +# +# Usage: pack_element [ ...] +pack_element() { + local dest="$1" + local description="$2" + shift 2 + local sources=("$@") + + local existing=() + for src in "${sources[@]}"; do + [ -e "$src" ] && existing+=("$src") + done + + if [[ ${#existing[@]} -eq 0 ]]; then + echo -e " ${YELLOW}Skipping ${description} (nothing found)${NC}" + return + fi + + echo -n " Backing up ${description}... " + + # Build tar args: each source is expressed as -C + local tar_args=() + for src in "${existing[@]}"; do + tar_args+=(-C "$(dirname "$src")" "$(basename "$src")") + done + + tar -czf "$dest" "${tar_args[@]}" 2>/dev/null \ + && echo -e "${GREEN}✓${NC}" \ + || echo -e "${RED}✗${NC}" +} + +# --------------------------------------------------------------------------- +# Core config XML files +# --------------------------------------------------------------------------- +echo -e "${GREEN}--- Core configuration + root-level XMLs ---${NC}" + +# Collect every *.xml in JENKINS_HOME root +xml_sources=() +for xml_file in "$JENKINS_HOME"/*.xml; do + [ -e "$xml_file" ] && xml_sources+=("$xml_file") +done + +pack_element \ + "$BACKUP_DIR/config.tar.gz" \ + "config XMLs (config.xml, credentials.xml, all root *.xml)" \ + "${xml_sources[@]}" + +# --------------------------------------------------------------------------- +# Users +# --------------------------------------------------------------------------- +echo "" +echo -e "${GREEN}--- Users ---${NC}" +pack_element \ + "$BACKUP_DIR/users.tar.gz" \ + "users/" \ + "$JENKINS_HOME/users" + +# --------------------------------------------------------------------------- +# Secrets and encryption keys +# --------------------------------------------------------------------------- +echo "" +echo -e "${GREEN}--- Secrets and encryption keys ---${NC}" +pack_element \ + "$BACKUP_DIR/secrets.tar.gz" \ + "secrets/, .key, secret.key, secret.key.not-so-secret" \ + "$JENKINS_HOME/secrets" \ + "$JENKINS_HOME/.key" \ + "$JENKINS_HOME/secret.key" \ + "$JENKINS_HOME/secret.key.not-so-secret" + +# --------------------------------------------------------------------------- +# Plugins +# --------------------------------------------------------------------------- +echo "" +echo -e "${GREEN}--- Plugins ---${NC}" +pack_element \ + "$BACKUP_DIR/plugins.tar.gz" \ + "plugins/ (this may take a moment)" \ + "$JENKINS_HOME/plugins" + +# --------------------------------------------------------------------------- +# Nodes (agents) +# --------------------------------------------------------------------------- +echo "" +echo -e "${GREEN}--- Nodes (agents) ---${NC}" +pack_element \ + "$BACKUP_DIR/nodes.tar.gz" \ + "nodes/" \ + "$JENKINS_HOME/nodes" + +# --------------------------------------------------------------------------- +# Jenkins user crontab +# --------------------------------------------------------------------------- +echo "" +echo -e "${GREEN}--- Jenkins crontab ---${NC}" +echo -n " Backing up jenkins user crontab... " +crontab -u "$JENKINS_USER" -l > "$BACKUP_DIR/crontab.txt" 2>/dev/null \ + || echo "# No crontab for $JENKINS_USER" > "$BACKUP_DIR/crontab.txt" +echo -e "${GREEN}✓${NC}" + +# --------------------------------------------------------------------------- +# Package everything into the final outer tarball +# --------------------------------------------------------------------------- +echo "" +echo -e "${GREEN}--- Creating outer tarball ---${NC}" +tar -czf "$TARBALL" "$BACKUP_DIR" +rm -rf "$BACKUP_DIR" + +FILESIZE=$(du -h "$TARBALL" | cut -f1) + +echo "" +echo -e "${GREEN}========================================${NC}" +echo -e "${GREEN}Backup Complete!${NC}" +echo -e "${GREEN}========================================${NC}" +echo "" +echo -e "Archive : ${GREEN}${TARBALL}${NC}" +echo -e "Size : ${GREEN}${FILESIZE}${NC}" +echo "" +echo "Contents of the archive:" +echo " tar -tzf $TARBALL" +echo "" +echo "To extract:" +echo " tar -xzf $TARBALL" +echo "" +echo -e "${YELLOW}Note: secrets/ and key files are included in secrets.tar.gz.${NC}" +echo -e "${YELLOW}Store this archive securely — it contains encryption keys.${NC}" +echo "" + +# Made with Bob diff --git a/jenkins-as-code/jenkins-scripts/restore-jenkins-app-config.sh b/jenkins-as-code/jenkins-scripts/restore-jenkins-app-config.sh new file mode 100755 index 0000000000..0c932d6ffc --- /dev/null +++ b/jenkins-as-code/jenkins-scripts/restore-jenkins-app-config.sh @@ -0,0 +1,613 @@ +#!/bin/bash +################################################################################ +# Jenkins Application Configuration Restore Script +# +# Restores a backup produced by backup-jenkins-app-config.sh into JENKINS_HOME. +# Each element is stored as its own named tarball inside the outer archive; +# use --skip flags to omit any element during restore. +# +# Elements: +# config — root-level XML files (config.xml, credentials.xml, *.xml) +# users — users/ +# secrets — secrets/, .key, secret.key, secret.key.not-so-secret +# plugins — plugins/ +# nodes — nodes/ +# crontab — jenkins user crontab +# +# Environment-specific XML overrides are applied from restore-config-overrides.env +# (located alongside this script). Any variable left empty in that file is skipped. +# Override the path via: OVERRIDES_FILE=/path/to/overrides.env +# +# Flags: +# --skip [ ...] Omit one or more elements during restore. +# --blank-oauth Clear the GitHub OAuth client ID and secret +# from config.xml and switch the security realm +# to Jenkins' own user database. Use when +# restoring to a different environment where a +# new OAuth app must be registered. Omit this +# flag when restoring to the same environment. +# +# Usage: +# sudo bash restore-jenkins-app-config.sh [--skip ...] [--blank-oauth] +# +# Cross-environment restore (new OAuth app needed): +# sudo bash restore-jenkins-app-config.sh jenkins-app-backup-20260706-113355.tar.gz --blank-oauth +# +# Same-environment restore (keep OAuth config intact): +# sudo bash restore-jenkins-app-config.sh jenkins-app-backup-20260706-113355.tar.gz +# +# Override defaults via environment variables: +# JENKINS_HOME=/path/to/.jenkins sudo bash restore-jenkins-app-config.sh +# JENKINS_USER=jenkins sudo bash restore-jenkins-app-config.sh +################################################################################ + +set -e + +# Configuration — override via environment variables if needed +JENKINS_HOME="${JENKINS_HOME:-/home/jenkins/.jenkins}" +JENKINS_USER="${JENKINS_USER:-jenkins}" +JENKINS_PORT="${JENKINS_PORT:-8080}" + +# Path to the env file containing environment-specific XML overrides. +# Defaults to a sibling file in the same directory as this script. +OVERRIDES_FILE="${OVERRIDES_FILE:-$(dirname "$0")/restore-config-overrides.env}" + +# Colors for output +RED='\033[0;31m' +GREEN='\033[0;32m' +YELLOW='\033[1;33m' +NC='\033[0m' + +############################################### +# Argument parsing +############################################### + +TARBALL="${1:-}" +if [[ -z "$TARBALL" ]]; then + echo -e "${RED}Error: No backup tarball specified.${NC}" + echo "" + echo "Usage: sudo bash $0 [--skip [ ...]] [--blank-oauth]" + echo "" + echo "Elements: config users secrets plugins nodes crontab" + echo "" + echo " Same-env restore : sudo bash $0 jenkins-app-backup-20260706-113355.tar.gz" + echo " Cross-env restore : sudo bash $0 jenkins-app-backup-20260706-113355.tar.gz --blank-oauth" + exit 1 +fi +shift + +# Collect --skip values and other flags from remaining args. +# --skip supports space-separated elements: --skip nodes plugins +# or repeated flags: --skip nodes --skip plugins +declare -A SKIP +BLANK_OAUTH=0 +VALID_ELEMENTS=("config" "users" "secrets" "plugins" "nodes" "crontab") +while [[ $# -gt 0 ]]; do + case "$1" in + --skip) + shift + if [[ -z "${1:-}" || "${1:-}" == --* ]]; then + echo -e "${RED}Error: --skip requires at least one element name.${NC}" + exit 1 + fi + # Consume all following non-flag words that are valid element names. + # Stop at any word starting with '--' OR any word that is not a + # recognised element (e.g. a bare "blank-oauth" typo). + consumed=0 + while [[ $# -gt 0 && "${1:-}" != --* ]]; do + elem_valid=0 + for v in "${VALID_ELEMENTS[@]}"; do [[ "$1" == "$v" ]] && elem_valid=1 && break; done + if [[ $elem_valid -eq 0 ]]; then + echo -e "${RED}Error: Unknown element '${1}' passed to --skip.${NC}" + echo -e "${YELLOW}Valid elements: ${VALID_ELEMENTS[*]}${NC}" + echo -e "${YELLOW}Flags like --blank-oauth must use the -- prefix.${NC}" + exit 1 + fi + SKIP["$1"]=1 + consumed=$((consumed + 1)) + shift + done + if [[ $consumed -eq 0 ]]; then + echo -e "${RED}Error: --skip requires at least one element name.${NC}" + exit 1 + fi + ;; + --blank-oauth) + BLANK_OAUTH=1 + shift + ;; + *) + echo -e "${RED}Error: Unknown argument: $1${NC}" + exit 1 + ;; + esac +done + +############################################### +# Load config overrides (if present) +############################################### + +# Initialise all override variables to empty so they are always defined, +# even if the env file is absent or a variable is omitted from it. +JENKINS_URL="" +JENKINS_ADMIN_EMAIL="" +THINBACKUP_PATH="" +SLACK_TEAM_DOMAIN="" +SLACK_DEFAULT_ROOM="" + +if [[ -f "$OVERRIDES_FILE" ]]; then + # shellcheck source=/dev/null + source "$OVERRIDES_FILE" +fi + +############################################### +# Pre-flight Checks +############################################### + +# Must be run as root +if [[ $EUID -ne 0 ]]; then + echo -e "${RED}Error: This script must be run as root (use sudo)${NC}" + exit 1 +fi + +if [[ ! -f "$TARBALL" ]]; then + echo -e "${RED}Error: Backup file not found: ${TARBALL}${NC}" + exit 1 +fi + +# Verify the Jenkins user exists +if ! id "$JENKINS_USER" &>/dev/null; then + echo -e "${RED}Error: Jenkins user '${JENKINS_USER}' does not exist.${NC}" + echo -e "${YELLOW}Create the user before running this script.${NC}" + exit 1 +fi + +# Resolve to an absolute path so it stays valid after any cd +TARBALL="$(realpath "$TARBALL")" +TARBALL_BASENAME="$(basename "$TARBALL")" +# Strip .tar.gz to get the inner directory name produced by the backup script +BACKUP_STEM="${TARBALL_BASENAME%.tar.gz}" + +RESTORE_TMP="$(mktemp -d /tmp/jenkins-restore-XXXXXX)" +# Ensure temp dir is cleaned up on exit (normal or error) +trap 'rm -rf "$RESTORE_TMP"' EXIT + +echo -e "${GREEN}========================================${NC}" +echo -e "${GREEN}Jenkins Application Config Restore${NC}" +echo -e "${GREEN}========================================${NC}" +echo "" +echo -e "Backup file : ${YELLOW}${TARBALL}${NC}" +echo -e "JENKINS_HOME : ${YELLOW}${JENKINS_HOME}${NC}" +echo -e "JENKINS_USER : ${YELLOW}${JENKINS_USER}${NC}" +if [[ ${#SKIP[@]} -gt 0 ]]; then + echo -e "Skipping : ${YELLOW}${!SKIP[*]}${NC}" +fi +if [[ -f "$OVERRIDES_FILE" ]]; then + echo -e "Overrides : ${YELLOW}${OVERRIDES_FILE}${NC}" +else + echo -e "Overrides : ${YELLOW}none (${OVERRIDES_FILE} not found)${NC}" +fi +if [[ "$BLANK_OAUTH" -eq 1 ]]; then + echo -e "OAuth : ${YELLOW}will be blanked (--blank-oauth)${NC}" +else + echo -e "OAuth : ${YELLOW}restored as-is (pass --blank-oauth for cross-env restore)${NC}" +fi +echo "" + +############################################### +# Stop Jenkins +############################################### +echo -e "${GREEN}--- Stopping Jenkins service ---${NC}" +if systemctl is-active --quiet jenkins; then + systemctl stop jenkins + echo -e " Jenkins stopped ${GREEN}✓${NC}" +else + echo -e " ${YELLOW}Jenkins was not running — continuing${NC}" +fi +echo "" + +############################################### +# Extract outer backup tarball +############################################### +echo -e "${GREEN}--- Extracting backup ---${NC}" +chmod 700 "$RESTORE_TMP" +tar -xzf "$TARBALL" -C "$RESTORE_TMP" +echo -e " Extracted to ${RESTORE_TMP} ${GREEN}✓${NC}" +echo "" + +BACKUP_ROOT="${RESTORE_TMP}/${BACKUP_STEM}" + +if [[ ! -d "$BACKUP_ROOT" ]]; then + echo -e "${RED}Error: Expected directory '${BACKUP_STEM}' not found inside tarball.${NC}" + echo -e "${YELLOW}Is this a valid backup produced by backup-jenkins-app-config.sh?${NC}" + exit 1 +fi + +mkdir -p "$JENKINS_HOME" + +# --------------------------------------------------------------------------- +# Helper — apply a sed replacement to a single XML element in a file. +# Replaces the text content of ... (single-line form). +# No-op if new_value is empty. +# +# Usage: apply_override +# --------------------------------------------------------------------------- +apply_override() { + local file="$1" + local element="$2" + local new_value="$3" + + [[ -z "$new_value" ]] && return 0 + [[ ! -f "$file" ]] && return 0 + + sed -i "s|<${element}>[^<]*|<${element}>${new_value}|g" "$file" +} + +# --------------------------------------------------------------------------- +# Helper — restore a single element from its inner tarball into JENKINS_HOME. +# The 'config' element has special handling: it is extracted to a staging +# directory first so overrides can be applied before copying to JENKINS_HOME. +# +# Usage: restore_element +# --------------------------------------------------------------------------- +restore_element() { + local name="$1" + local description="$2" + local inner_tar="${BACKUP_ROOT}/${name}.tar.gz" + + echo -e "${GREEN}--- ${description} ---${NC}" + + if [[ -n "${SKIP[$name]+x}" ]]; then + echo -e " ${YELLOW}Skipped (--skip ${name})${NC}" + echo "" + return + fi + + if [[ ! -f "$inner_tar" ]]; then + echo -e " ${YELLOW}Not present in backup — skipped${NC}" + echo "" + return + fi + + if [[ "$name" == "config" ]]; then + # Extract to a staging area, apply overrides, then copy into JENKINS_HOME + local staging="${RESTORE_TMP}/config-staging" + mkdir -p "$staging" + echo -n " Extracting ${description}... " + tar -xzf "$inner_tar" -C "$staging" 2>/dev/null \ + && echo -e "${GREEN}✓${NC}" \ + || { echo -e "${RED}✗${NC}"; echo ""; return; } + + _apply_config_overrides "$staging" + + echo -n " Copying ${description} to JENKINS_HOME... " + cp -a "${staging}/." "${JENKINS_HOME}/" 2>/dev/null \ + && echo -e "${GREEN}✓${NC}" \ + || echo -e "${RED}✗${NC}" + else + echo -n " Restoring ${description}... " + tar -xzf "$inner_tar" -C "$JENKINS_HOME" 2>/dev/null \ + && echo -e "${GREEN}✓${NC}" \ + || echo -e "${RED}✗${NC}" + fi + echo "" +} + +# --------------------------------------------------------------------------- +# Apply all environment-specific overrides to the staged config XML files. +# Called after config.tar.gz is extracted into the staging directory. +# --------------------------------------------------------------------------- +_apply_config_overrides() { + local staging="$1" + + local config="${staging}/config.xml" + local loc_cfg="${staging}/jenkins.model.JenkinsLocationConfiguration.xml" + local mailer="${staging}/hudson.tasks.Mailer.xml" + local header="${staging}/io.jenkins.plugins.customizable_header.CustomHeaderConfiguration.xml" + local thinbackup="${staging}/org.jvnet.hudson.plugins.thinbackup.ThinBackupPluginImpl.xml" + local slack="${staging}/jenkins.plugins.slack.SlackNotifier.xml" + local ansible="${staging}/org.jenkinsci.plugins.ansible_tower.AnsibleTower.xml" + local queue="${staging}/queue.xml" + + echo -e " ${GREEN}Applying config overrides:${NC}" + + # JENKINS_URL → jenkinsUrl, hudsonUrl, and the logoPath prefix + if [[ -n "$JENKINS_URL" ]]; then + apply_override "$loc_cfg" "jenkinsUrl" "${JENKINS_URL}" + apply_override "$mailer" "hudsonUrl" "${JENKINS_URL}" + # logoPath has a suffix (path) appended to the base URL — replace only the URL prefix + if [[ -f "$header" ]]; then + local logo_suffix + logo_suffix=$(grep -oP '(?<=)[^<]*' "$header" | sed 's|.*/userContent|/userContent|' || true) + if [[ -n "$logo_suffix" ]]; then + local new_logo="${JENKINS_URL%/}${logo_suffix}" + apply_override "$header" "logoPath" "${new_logo}" + else + # No userContent suffix found — replace the whole value + apply_override "$header" "logoPath" "${JENKINS_URL}" + fi + fi + echo -e " JENKINS_URL → ${JENKINS_URL} ${GREEN}✓${NC}" + else + echo -e " JENKINS_URL — ${YELLOW}skipped (empty)${NC}" + fi + + # JENKINS_ADMIN_EMAIL → adminAddress + if [[ -n "$JENKINS_ADMIN_EMAIL" ]]; then + apply_override "$loc_cfg" "adminAddress" "${JENKINS_ADMIN_EMAIL}" + echo -e " JENKINS_ADMIN_EMAIL → ${JENKINS_ADMIN_EMAIL} ${GREEN}✓${NC}" + else + echo -e " JENKINS_ADMIN_EMAIL — ${YELLOW}skipped (empty)${NC}" + fi + + # THINBACKUP_PATH → backupPath + if [[ -n "$THINBACKUP_PATH" ]]; then + apply_override "$thinbackup" "backupPath" "${THINBACKUP_PATH}" + echo -e " THINBACKUP_PATH → ${THINBACKUP_PATH} ${GREEN}✓${NC}" + else + echo -e " THINBACKUP_PATH — ${YELLOW}skipped (empty)${NC}" + fi + + # SLACK_TEAM_DOMAIN → teamDomain + if [[ -n "$SLACK_TEAM_DOMAIN" ]]; then + apply_override "$slack" "teamDomain" "${SLACK_TEAM_DOMAIN}" + echo -e " SLACK_TEAM_DOMAIN → ${SLACK_TEAM_DOMAIN} ${GREEN}✓${NC}" + else + echo -e " SLACK_TEAM_DOMAIN — ${YELLOW}skipped (empty)${NC}" + fi + + # SLACK_DEFAULT_ROOM → room + if [[ -n "$SLACK_DEFAULT_ROOM" ]]; then + apply_override "$slack" "room" "${SLACK_DEFAULT_ROOM}" + echo -e " SLACK_DEFAULT_ROOM → ${SLACK_DEFAULT_ROOM} ${GREEN}✓${NC}" + else + echo -e " SLACK_DEFAULT_ROOM — ${YELLOW}skipped (empty)${NC}" + fi + + # Ansible Tower — always blanked (not used in this environment) + if [[ -f "$ansible" ]]; then + sed -i 's|[^<]*||g' "$ansible" + sed -i 's|[^<]*||g' "$ansible" + echo -e " Ansible Tower fields → blanked ${GREEN}✓${NC}" + fi + + # Build queue — always cleared on restore to avoid stale queued jobs + if [[ -f "$queue" ]]; then + printf '\n\n \n \n \n \n\n' > "$queue" + echo -e " Build queue → cleared ${GREEN}✓${NC}" + fi + + # GitHub OAuth — only blanked when --blank-oauth is passed. + # Clears clientID and clientSecret from config.xml and replaces the + # GitHub security realm with Jenkins' own user database so the instance + # is accessible immediately while a new OAuth app is registered. + if [[ "$BLANK_OAUTH" -eq 1 ]] && [[ -f "$config" ]]; then + # The securityRealm block spans multiple lines so use python for a + # reliable multiline replacement rather than fragile sed tricks. + python3 - "$config" <<'PYEOF' +import re, sys +path = sys.argv[1] +content = open(path).read() +# Blank OAuth credentials +content = re.sub(r'[^<]*', '', content) +content = re.sub(r'[^<]*', '', content) +# Replace the GitHub security realm block with Jenkins' own user database +content = re.sub( + r']*>.*?', + '' + 'false' + 'false' + '', + content, + flags=re.DOTALL +) +# Grant Hudson.Administer to the local admin user if not already present. +# The backup's authorization matrix only grants admin to GitHub groups — +# the local admin account needs an explicit USER permission to see Manage Jenkins. +administer_perm = 'USER:hudson.model.Hudson.Administer:admin' +if administer_perm not in content: + content = content.replace( + 'GROUP:hudson.model.Hudson.Administer:AdoptOpenJDK*jenkins-admins', + 'GROUP:hudson.model.Hudson.Administer:AdoptOpenJDK*jenkins-admins\n ' + administer_perm, + 1 + ) + # Fallback: if that exact group line isn't present, insert before + if administer_perm not in content: + content = content.replace( + '', + ' ' + administer_perm + '\n ', + 1 + ) +open(path, 'w').write(content) +PYEOF + echo -e " GitHub OAuth → blanked, switched to Jenkins user DB ${GREEN}✓${NC}" + echo -e " admin permissions → Hudson.Administer granted to local admin ${GREEN}✓${NC}" + echo -e " ${YELLOW}Note: log in at /login and reconfigure OAuth under Manage Jenkins.${NC}" + elif [[ "$BLANK_OAUTH" -eq 0 ]]; then + echo -e " GitHub OAuth — restored as-is (--blank-oauth not set)" + fi +} + +############################################### +# Restore individual elements +############################################### +restore_element "config" "Core configuration + root-level XMLs" +restore_element "users" "Users" +restore_element "secrets" "Secrets and encryption keys" +restore_element "plugins" "Plugins" +restore_element "nodes" "Nodes (agents)" + +############################################### +# Restore Jenkins user crontab +############################################### +echo -e "${GREEN}--- Jenkins crontab ---${NC}" +if [[ -n "${SKIP[crontab]+x}" ]]; then + echo -e " ${YELLOW}Skipped (--skip crontab)${NC}" +elif [[ ! -f "${BACKUP_ROOT}/crontab.txt" ]]; then + echo -e " ${YELLOW}Not present in backup — skipped${NC}" +else + CRONTAB_FILE="${BACKUP_ROOT}/crontab.txt" + # Skip installing if the file only contains a "no crontab" comment + if grep -qvE '^\s*#|^\s*$' "$CRONTAB_FILE"; then + crontab -u "$JENKINS_USER" "$CRONTAB_FILE" + echo -e " Crontab restored ${GREEN}✓${NC}" + else + echo -e " ${YELLOW}Crontab file is empty/comment-only — skipped${NC}" + fi +fi +echo "" + +############################################### +# Fix ownership across all restored files +############################################### +echo -e "${GREEN}--- Fixing ownership ---${NC}" +chown -R "${JENKINS_USER}:${JENKINS_USER}" "$JENKINS_HOME" +echo -e " Ownership set to ${JENKINS_USER} ${GREEN}✓${NC}" +echo "" + +############################################### +# Create local admin user (--blank-oauth only) +############################################### +# When --blank-oauth is used the GitHub OAuth realm is replaced with Jenkins' +# own user database. The backup only contains GitHub OAuth users (no local +# password hashes), so a fresh local admin account must be created from scratch. +RESET_PASSWORD="" +if [[ "$BLANK_OAUTH" -eq 1 ]]; then + echo -e "${GREEN}--- Creating local admin user ---${NC}" + + # Generate a random 16-character alphanumeric password and write it to a + # temp file — never interpolated into shell strings to avoid quoting issues. + RESET_PASSWORD=$(tr -dc 'A-Za-z0-9' < /dev/urandom | head -c 16) + PASS_TMP=$(mktemp) + printf '%s' "$RESET_PASSWORD" > "$PASS_TMP" + + # Produce a bcrypt hash (cost 10). Jenkins requires the $2a$ variant. + # python3-bcrypt produces $2b$; htpasswd produces $2y$ — both normalised. + HASHED="" + if python3 -c "import bcrypt" 2>/dev/null; then + HASHED=$(python3 - "$PASS_TMP" <<'PYEOF' +import bcrypt, sys +password = open(sys.argv[1], 'rb').read() +hashed = bcrypt.hashpw(password, bcrypt.gensalt(10)).decode() +# Jenkins requires $2a$ — python bcrypt emits $2b$ +print(hashed.replace('$2b$', '$2a$', 1)) +PYEOF +) + elif command -v htpasswd &>/dev/null; then + HASHED=$(htpasswd -bnBC 10 "" "$(cat "$PASS_TMP")" \ + | tr -d ':\n' \ + | sed 's/\$2y\$/\$2a\$/; s/\$2b\$/\$2a\$/') + else + echo -e " ${YELLOW}Warning: neither python3-bcrypt nor htpasswd found.${NC}" + echo -e " ${YELLOW}Install python3-bcrypt or apache2-utils and re-run.${NC}" + RESET_PASSWORD="" + fi + rm -f "$PASS_TMP" + + if [[ -n "$RESET_PASSWORD" && -n "$HASHED" ]]; then + # Write the user into the legacy plain-name directory (users/admin/). + # Jenkins on startup auto-migrates this to the HMAC-keyed hashed + # directory name and writes users.xml itself — this is more reliable + # than trying to predict the HMAC hash externally, which requires + # reading Jenkins' per-instance secret from secrets/. + # + # Remove any stale hashed admin dir left from a previous attempt so + # there is no conflict during migration. + find "${JENKINS_HOME}/users" -maxdepth 1 -name 'admin_*' -exec rm -rf {} + 2>/dev/null || true + rm -f "${JENKINS_HOME}/users/users.xml" + ADMIN_USER_DIR="${JENKINS_HOME}/users/admin" + mkdir -p "$ADMIN_USER_DIR" + + # Write user config.xml via Python — avoids shell expansion of the + # bcrypt hash ($2a$10$...) which would corrupt it in a heredoc. + python3 - "$ADMIN_USER_DIR/config.xml" "$HASHED" <<'PYEOF' +import sys +user_xml_path, hashed = sys.argv[1], sys.argv[2] +with open(user_xml_path, 'w') as f: + f.write( + "\n" + "\n" + " 10\n" + " admin\n" + " admin\n" + " \n" + " \n" + " #jbcrypt:" + hashed + "\n" + " \n" + " \n" + " \n" + " \n" + " \n" + " \n" + "\n" + ) +PYEOF + + chown -R "${JENKINS_USER}:${JENKINS_USER}" "$ADMIN_USER_DIR" + echo -e " local admin user created (users/admin/ — Jenkins will migrate on startup) ${GREEN}✓${NC}" + fi + echo "" +fi + +############################################### +# Start Jenkins +############################################### +echo -e "${GREEN}--- Starting Jenkins service ---${NC}" +systemctl daemon-reload +systemctl enable --quiet jenkins +systemctl start jenkins +echo -e " Jenkins started ${GREEN}✓${NC}" +echo "" + +echo -e "${GREEN}--- Waiting for Jenkins to be ready (port ${JENKINS_PORT}) ---${NC}" +RETRIES=30 +DELAY=10 +for ((i = 1; i <= RETRIES; i++)); do + if curl -sf "http://localhost:${JENKINS_PORT}/login" -o /dev/null; then + echo -e " Jenkins is ready ${GREEN}✓${NC}" + break + fi + if [[ $i -eq $RETRIES ]]; then + echo -e "${YELLOW}Warning: Jenkins did not respond after $((RETRIES * DELAY))s.${NC}" + echo -e "${YELLOW}Check: systemctl status jenkins${NC}" + break + fi + echo -e " Waiting... (attempt ${i}/${RETRIES})" + sleep "$DELAY" +done +echo "" + +############################################### +# Summary +############################################### +echo -e "${GREEN}========================================${NC}" +echo -e "${GREEN}Restore Complete!${NC}" +echo -e "${GREEN}========================================${NC}" +echo "" +echo -e "Backup file : ${GREEN}${TARBALL_BASENAME}${NC}" +echo -e "Jenkins Home : ${GREEN}${JENKINS_HOME}${NC}" +echo -e "Jenkins URL : ${GREEN}http://localhost:${JENKINS_PORT}${NC}" +if [[ ${#SKIP[@]} -gt 0 ]]; then + echo -e "Skipped : ${YELLOW}${!SKIP[*]}${NC}" +fi +echo "" +echo -e "${YELLOW}Note: If secrets were restored, ensure the Jenkins URL and${NC}" +echo -e "${YELLOW}credentials configuration still matches this environment.${NC}" +if [[ -n "$JENKINS_URL" ]]; then + echo -e "${YELLOW}Jenkins URL override was applied — register a matching GitHub OAuth app${NC}" + echo -e "${YELLOW}at ${JENKINS_URL}securityRealm/finishLogin if using GitHub auth.${NC}" +fi +if [[ -n "$RESET_PASSWORD" ]]; then + echo "" + echo -e "${YELLOW}┌─────────────────────────────────────────────────┐${NC}" + echo -e "${YELLOW}│ Admin password was reset (--blank-oauth) │${NC}" + echo -e "${YELLOW}│ │${NC}" + echo -e "${YELLOW}│ Username : admin │${NC}" + echo -e "${YELLOW}│ Password : ${GREEN}${RESET_PASSWORD}${YELLOW} │${NC}" + echo -e "${YELLOW}│ │${NC}" + echo -e "${YELLOW}│ Change this password after first login. │${NC}" + echo -e "${YELLOW}└─────────────────────────────────────────────────┘${NC}" +fi +echo "" + +# Made with Bob diff --git a/requirements.txt b/requirements.txt new file mode 100644 index 0000000000..3780d60249 --- /dev/null +++ b/requirements.txt @@ -0,0 +1,21 @@ +# Python requirements for Ansible control node +# Required for managing Windows hosts via WinRM + +# Core Ansible requirement +ansible>=2.9 + +# Windows management via WinRM +pywinrm>=0.4.1 + +# Additional WinRM dependencies for better compatibility +requests>=2.25.0 +requests-ntlm>=1.1.0 +requests-credssp>=2.0.0 + +# For Kerberos authentication (optional but recommended) +# Uncomment if you need Kerberos support: +# pywinrm[kerberos]>=0.4.1 + +# For CredSSP authentication (optional) +# Uncomment if you need CredSSP support: +# pywinrm[credssp]>=0.4.1 \ No newline at end of file