Skip to content

CI playbook to recreate CI test clusters #349

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
justinc1 merged 1 commit into from
Jan 17, 2025
Merged

CI playbook to recreate CI test clusters #349

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 11 additions & 0 deletions DEVELOPMENT.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -200,3 +200,14 @@ Steps:
- ensure ip address is added to entraAD (azure) app registration for OIDC integration (ask Dave if needed)
- "app_display_name": "Scale Computing HC3",
- "app_id": "d2298ec0-0596-49d2-9554-840a2fe20603",

## Reset VSNS test cluster

After enough testing VSNS test clusters get slow.
CI integration testing takes say 8 hours instead of 1 hour.
At that time we recreate VSNS from template VM.
Run:

```bash
ansible-playbook ci-infra/helpers/ci_hosts_recreate.yml -v
```
64 changes: 64 additions & 0 deletions ci-infra/helpers/ci_hosts_recreate.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,64 @@
---
# On CI physical HyperCore host run multiple VSNS CI HyperCore hosts.
# The VSNS needs a periodic "refresh". Seems like after many many API calls,
# the VSNS API gets slow. Daily CI integration test last for 8 hours instead of 1..2 hours.
# Refresh means remove old VSNS, and create new from VSNS template VM.
# VSNS template VMs were prepared manually

- name: Recreate CI VSNS test hosts/VMs
hosts: localhost
connection: local
# gather_facts: false
vars:
vsns_spec_all:
vsns914-ci:
source: vsns914-template
ip: 10.5.11.200
vsns9213-ci:
source: vsns9213-template
ip: 10.5.11.201
# vsns931-ci: # not used
# source: vsns931-template
# ip: 10.5.11.202
# vsns935-ci: # no template VM
# source: ?
# ip: 10.5.11.203
vsns940-ci:
# Is not used in CI tests
source: vsns940-template
ip: 10.5.11.204
vsns9417-ci:
source: vsns9417-template
ip: 10.5.11.205

tasks:
# ------------------------------------------------------
- name: Show info
ansible.builtin.debug:
msg: source_vm={{ source_vm }} dest_vm={{ dest_vm }} ip={{ vm_ip }} url=https://{{ vm_ip }}
loop: "{{ vsns_spec_all.keys() }}"
loop_control:
loop_var: dest_vm
vars:
source_vm: "{{ vsns_spec_all[dest_vm].source }}"
vm_ip: "{{ vsns_spec_all[dest_vm].ip }}"

- name: Recreate CI VSNSs
loop: "{{ vsns_spec_all.keys() }}"
loop_control:
loop_var: dest_vm
vars:
source_vm: "{{ vsns_spec_all[dest_vm].source }}"
vm_ip: "{{ vsns_spec_all[dest_vm].ip }}"
ansible.builtin.include_tasks:
file: ci_hosts_recreate_one.yml

- name: Show URLs
ansible.builtin.debug:
msg: source_vm={{ source_vm }} dest_vm={{ dest_vm }} ip={{ vm_ip }} url=https://{{ vm_ip }}
loop: "{{ vsns_spec_all.keys() }}"
loop_control:
loop_var: dest_vm
vars:
source_vm: "{{ vsns_spec_all[dest_vm].source }}"
vm_ip: "{{ vsns_spec_all[dest_vm].ip }}"
51 changes: 51 additions & 0 deletions ci-infra/helpers/ci_hosts_recreate_one.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,51 @@
---
- name: Show input params
ansible.builtin.debug:
msg: source_vm={{ source_vm }} dest_vm={{ dest_vm }} vm_ip={{ vm_ip }} url=https://{{ vm_ip }}

- name: Source VM info

Check warning on line 6 in ci-infra/helpers/ci_hosts_recreate_one.yml

View workflow job for this annotation

GitHub Actions / sanity-test

args[module] 

missing required arguments: host, password, username found in cluster_instance
scale_computing.hypercore.vm_info:
vm_name: "{{ source_vm }}"
register: source_vm_result

- name: Fail if source VM is not present
ansible.builtin.assert:
that:
- source_vm_result.records | length == 1
fail_msg: "Source VM {{ source_vm }} does not exist."

- name: Old VM info

Check warning on line 17 in ci-infra/helpers/ci_hosts_recreate_one.yml

View workflow job for this annotation

GitHub Actions / sanity-test

args[module] 

missing required arguments: host, password, username found in cluster_instance
scale_computing.hypercore.vm_info:
vm_name: "{{ dest_vm }}"
register: vm_result

- name: Force-shutdown old VM if exists

Check warning on line 22 in ci-infra/helpers/ci_hosts_recreate_one.yml

View workflow job for this annotation

GitHub Actions / sanity-test

args[module] 

missing required arguments: host, password, username found in cluster_instance
scale_computing.hypercore.vm_params:
vm_name: "{{ dest_vm }}"
power_state: stop
force_reboot: true
shutdown_timeout: 10
when: vm_result.records | length

- name: Delete old VM

Check warning on line 30 in ci-infra/helpers/ci_hosts_recreate_one.yml

View workflow job for this annotation

GitHub Actions / sanity-test

args[module] 

missing required arguments: host, password, username found in cluster_instance
scale_computing.hypercore.vm:
vm_name: "{{ dest_vm }}"
state: absent

- name: Clone template VM

Check warning on line 35 in ci-infra/helpers/ci_hosts_recreate_one.yml

View workflow job for this annotation

GitHub Actions / sanity-test

args[module] 

missing required arguments: host, password, username found in cluster_instance
scale_computing.hypercore.vm_clone:
vm_name: "{{ dest_vm }}"
source_vm_name: "{{ source_vm }}"
preserve_mac_address: true

- name: Set VM resources
scale_computing.hypercore.vm_params:
vm_name: "{{ dest_vm }}"
description: "https://{{ vm_ip }} created={{ ansible_date_time.date }}"
vcpu: 8
memory: "{{ '16 GB' | human_to_bytes }}"

- name: Start VM

Check warning on line 48 in ci-infra/helpers/ci_hosts_recreate_one.yml

View workflow job for this annotation

GitHub Actions / sanity-test

args[module] 

missing required arguments: host, password, username found in cluster_instance
scale_computing.hypercore.vm_params:
vm_name: "{{ dest_vm }}"
power_state: start
Loading