CI Templates Server
The CI templates server is a static web server which stores VM disk templates used by CI of several projects. The user-facing URL is https://templates.ovirt.org yet it is served by a CentOS 8 VM hosted in the PHX oVirt cluster - templates02.phx.ovirt.org
Images are spread accross four main directories:
Path | Description | URL |
---|---|---|
/var/www/html/repo | used for storing lago images | https://templates.ovirt.org/repo/ |
/var/www/html/bundles | contains the 'ovirt-demo-tool' | https://templates.ovirt.org/bundles/ |
/var/www/html/yum | mainly for ost-images | https://templates.ovirt.org/yum/ |
/var/www/html/kubevirt | stores images for kubevirt ci | https://templates.ovirt.org/kubevirt/ |
Please note that kubevirt directory should be accessible only from restricted number of allowed CI systems. These restrictions are enforced by a configuration file containing specific whitelisted subnets. The file is located at /etc/httpd/conf.d/templates.conf of templates02 VM.
At the time of writing this document there are no offical backups for the contents of these directories, however recovering from data loss/corruption is feasible in a timely manner.
Disks Layout
templates02.phx.ovirt.org VM has three disks that are separate from it and stored on storage02-data1
The Disks are:
- boot
- templates02-PV01
- templates02-PV02
The first disk stores the OS and some specific configuration. The two remaining disks form a volume group for storing the images.
Data Recovery
Lago Templates
Most lago images are cached under /var/lib/lago/store/ across different nodes, but require some steps before it is possible to copy them back to the templates machine:
-
Remove the 'phx_repo:' prefix and the ':v1' from suffix. Note that *.lock files are not needed. For instance, phx_repo:el8.2-base:v1.hash --> el8.2-base.hash
-
Compress the image files only using the following command:
xz --compress --keep --threads=0 --best --force --verbose --block-size=16777216 /path/to/image/file
-
Make sure the hashes match the compressed images. If not just create a new hash.
-
Copy back the files to /var/www/html/repo. Note that for each image, three files have to be copied:
- image_name.xz
- image_name.hash
- image_name.metadata
OST Images
Regarding ost-images, due to recent efforts they can be recreated and published back to their relevant path. Make sure /var/www/html/yum holds sufficient permissions for the images owner to be able to reupload them.
Kubevirt
Similar to Lago tempalates, images for kubevirt are cached under /var/lib/stdci/shared/kubevirt-images/ accross bare metal nodes used by KubeVirt. This path contains directories for different operating systems. Files under each sub-directory have to be renamed before they are copied back to the server.
Before renaming the files, verify the image names KubeVirt actually uses:
For instance, the rhel8/ dir contains these files:
- disk.img
- disk.img.sha1
Make sure to rename them to:
- rhel8.qcow2
- rhel8.qcow2.sha1
Also make sure the hashes match the original files.