coordination #89278
Updated by okurz about 3 years ago
## Observation As brought up by @gracewang "did anybody face the problem that the qcow2 in osd got damaged? What I mean damaged is that the qcow2 type is changed and then can not boot to desktop https://openqa.nue.suse.com/tests/5564738#step/boot_to_desktop/1" we don't calculate a checksum for the generated qcow images so we don't know any initial hash. The image worked to boot initially so it seems like it was damaged while it was just sitting on osd which of course is weird. ``` okurz@openqa:~> ls -lh /var/lib/openqa/share/factory/hdd/SLE-15-SP3-x86_64-Build156.3-sled-gnome.qcow2 -rw-r--r-- 1 geekotest nogroup 2.5G Feb 27 13:34 /var/lib/openqa/share/factory/hdd/SLE-15-SP3-x86_64-Build156.3-sled-gnome.qcow2 okurz@openqa:~> file /var/lib/openqa/share/factory/hdd/SLE-15-SP3-x86_64-Build156.3-sled-gnome.qcow2 /var/lib/openqa/share/factory/hdd/SLE-15-SP3-x86_64-Build156.3-sled-gnome.qcow2: data ``` https://openqa.nue.suse.com/tests/5555304#dependencies is the original job that created the qcow image at 2021-02-26 19:47:35 +0000, from log: ``` [2021-02-26T20:46:35.0065 CET] [info] [pid:17912] SLE-15-SP3-x86_64-Build156.3-sled-gnome.qcow2: Processing chunk 2602/2602, avg. speed ~440.938 KiB/s ``` so as visible from the timestamps the qcow image was clearly modified multiple hours after the initial job uploaded the image. https://openqa.nue.suse.com/tests/5555310# is the first job within the scenario sle-15-SP3-Online-x86_64-wayland-desktopapps-documentation@64bit-virtio-vga for build 156.3 and it was able to boot the qcow image correctly, later tests in this scenario then failed to find a valid qcow image. ## Problem Looking into the database I can find: ``` openqa=# select jobs.id,t_finished,state,result,result_dir from jobs,job_settings where jobs.id = job_settings.job_id and key = 'PUBLISH_HDD_1' and value = 'SLE-15-SP3-x86_64-Build156.3-sled-gnome.qcow2' order by t_finished; id | t_finished | state | result | result_dir ---------+---------------------+---------+--------+----------------------------------------------------------------------------------------------------------------------------------------- ------ 5554094 | 2021-02-26 18:25:43 | done | failed | 05554094-sle-15-SP3-Online-x86_64-Build156.3-create_hdd_sled_gnome@64bit-virtio-vga 5555304 | 2021-02-26 19:47:35 | done | passed | 05555304-sle-15-SP3-Online-x86_64-Build156.3-create_hdd_sled_gnome@64bit-virtio-vga 5555252 | 2021-02-27 12:27:22 | done | passed | 05555252-sle-15-SP3-Online-x86_64-create_hdd_sled_gnome:investigate:retry@64bit-virtio-vga 5555253 | 2021-02-27 12:35:54 | done | passed | 05555253-sle-15-SP3-Online-x86_64-create_hdd_sled_gnome:investigate:last_good_tests:a24328b7173f02f5c4cf54f74f6799abcc9ff839@64bit-virti o-vga ``` so I suspect that the *two* "investigate:*" job overwrote the qcow image at the same time. The logs * https://openqa.suse.de/tests/5555252/file/autoinst-log.txt * https://openqa.suse.de/tests/5555253/file/autoinst-log.txt confirm that as an upload happens to the same file during the same time period. ## Acceptance criteria * **AC1:** jobs are prevented from uploading to the same asset file during the same time * **AC2:** Assets with incorrect checksum are not used in other jobs * **AC3:** Investigation jobs don't override production assets ## Suggestions * Calculate checksum of big assets on worker side before uploading * Upload checksum along with big assets * Check checksum on webUI side after receiving upload * Prevent multiple concurrent uploads to the same file, e.g. abort any additional job as incomplete when trying to upload to the same file * Prevent investigation jobs to do any asset uploads ## Workaround Retrigger the parent image creation job