QA » openQA Project

https://github.com/os-autoinst/openQA/pull/5909 Move some tests out of 14-grutasks.t

Draft: https://github.com/os-autoinst/openQA/pull/5910

Todo: schedule an update regularly, independent from a running test

Open questions: Should this be enabled with the same git_auto_clone feature which handles CASEDIR/NEEDLES_DIR repos, or should it be a feature that needs to be enabled additionally?

https://github.com/os-autoinst/openQA/pull/5910 Automatically update git for jobs without CASEDIR/NEEDLES_DIR

tinita wrote in #note-18:

https://github.com/os-autoinst/openQA/pull/5910 Automatically update git for jobs without CASEDIR/NEEDLES_DIR

Merged.

• Can /etc/cron.d/openqa-update-git which calls fetchneedles now be removed on o3?
• Can etc/master/cron.d/SLES.CRON also be removed accordingly to stop calling fetchneedles on osd?
• Anything else needed to fulfill AC1?

My plan is:

• Wait for deployment on o3 (DONE)
• Enable the feature on o3 and comment out the fetchneedles cronjob
• Monitor for a while

If things work out, improve documentation and enable it on osd as well.

Since it has been deployed on o3, I will enable it now and closely monitor.

Found a typo: https://github.com/os-autoinst/openQA/pull/5945

Please handle the symptoms of incomplete GRU git clone related jobs from yesterday and today.

Yesterday I restarted all incomplete jobs starting 2024-09-24 16:00 UTC

I enabled the feature again and scheduled an openQA build:
https://openqa.opensuse.org/tests/overview?version=Tumbleweed&distri=openqa&build=Build%3ATW.31611-tinita
It's failing because there is no image, but the git_clone minion job passed:
https://openqa.opensuse.org/minion/jobs?id=4355020

Now I will monitor for other scheduled products and possible incompletes.

There is some problem with the hourly timer.
It's retrying every 30something seconds, without an error message, so we assume it's having problems aquiring minion guards.
Here is the minion job:
https://openqa.opensuse.org/minion/jobs?id=4355236
I deleted it now and copied the YAML here:
Minion jobMinion job

---
args:
- /var/lib/openqa/share/tests/alp: ~
  /var/lib/openqa/share/tests/alp/products/alp/needles: ~
  /var/lib/openqa/share/tests/alp/products/d-installer/needles: ~
  /var/lib/openqa/share/tests/alp/products/kubic/needles: ~
  /var/lib/openqa/share/tests/alp/products/leap-micro/needles: ~
  /var/lib/openqa/share/tests/alp/products/microos/needles: ~
  /var/lib/openqa/share/tests/alp/products/opensuse/needles: ~
  /var/lib/openqa/share/tests/alp/products/sle-micro/needles: ~
  /var/lib/openqa/share/tests/alp/products/windows/needles: ~
  /var/lib/openqa/share/tests/d-installer: ~
  /var/lib/openqa/share/tests/d-installer/products/alp/needles: ~
  /var/lib/openqa/share/tests/d-installer/products/d-installer/needles: ~
  /var/lib/openqa/share/tests/d-installer/products/kubic/needles: ~
  /var/lib/openqa/share/tests/d-installer/products/leap-micro/needles: ~
  /var/lib/openqa/share/tests/d-installer/products/microos/needles: ~
  /var/lib/openqa/share/tests/d-installer/products/opensuse/needles: ~
  /var/lib/openqa/share/tests/d-installer/products/sle-micro/needles: ~
  /var/lib/openqa/share/tests/d-installer/products/windows/needles: ~
  /var/lib/openqa/share/tests/example: ~
  /var/lib/openqa/share/tests/kubic: ~
  /var/lib/openqa/share/tests/kubic/products/alp/needles: ~
  /var/lib/openqa/share/tests/kubic/products/d-installer/needles: ~
  /var/lib/openqa/share/tests/kubic/products/kubic/needles: ~
  /var/lib/openqa/share/tests/kubic/products/leap-micro/needles: ~
  /var/lib/openqa/share/tests/kubic/products/microos/needles: ~
  /var/lib/openqa/share/tests/kubic/products/opensuse/needles: ~
  /var/lib/openqa/share/tests/kubic/products/sle-micro/needles: ~
  /var/lib/openqa/share/tests/kubic/products/windows/needles: ~
  /var/lib/openqa/share/tests/leap-micro: ~
  /var/lib/openqa/share/tests/leap-micro/products/alp/needles: ~
  /var/lib/openqa/share/tests/leap-micro/products/d-installer/needles: ~
  /var/lib/openqa/share/tests/leap-micro/products/kubic/needles: ~
  /var/lib/openqa/share/tests/leap-micro/products/leap-micro/needles: ~
  /var/lib/openqa/share/tests/leap-micro/products/microos/needles: ~
  /var/lib/openqa/share/tests/leap-micro/products/opensuse/needles: ~
  /var/lib/openqa/share/tests/leap-micro/products/sle-micro/needles: ~
  /var/lib/openqa/share/tests/leap-micro/products/windows/needles: ~
  /var/lib/openqa/share/tests/microos: ~
  /var/lib/openqa/share/tests/microos/products/alp/needles: ~
  /var/lib/openqa/share/tests/microos/products/d-installer/needles: ~
  /var/lib/openqa/share/tests/microos/products/kubic/needles: ~
  /var/lib/openqa/share/tests/microos/products/leap-micro/needles: ~
  /var/lib/openqa/share/tests/microos/products/microos/needles: ~
  /var/lib/openqa/share/tests/microos/products/opensuse/needles: ~
  /var/lib/openqa/share/tests/microos/products/sle-micro/needles: ~
  /var/lib/openqa/share/tests/microos/products/windows/needles: ~
  /var/lib/openqa/share/tests/obs: ~
  /var/lib/openqa/share/tests/openQA: ~
  /var/lib/openqa/share/tests/openQA/products/openQA/needles: ~
  /var/lib/openqa/share/tests/openQA/products/openqa/needles: ~
  /var/lib/openqa/share/tests/openqa: ~
  /var/lib/openqa/share/tests/openqa/products/openQA/needles: ~
  /var/lib/openqa/share/tests/openqa/products/openqa/needles: ~
  /var/lib/openqa/share/tests/opensuse: ~
  /var/lib/openqa/share/tests/opensuse/products/alp/needles: ~
  /var/lib/openqa/share/tests/opensuse/products/d-installer/needles: ~
  /var/lib/openqa/share/tests/opensuse/products/kubic/needles: ~
  /var/lib/openqa/share/tests/opensuse/products/leap-micro/needles: ~
  /var/lib/openqa/share/tests/opensuse/products/microos/needles: ~
  /var/lib/openqa/share/tests/opensuse/products/opensuse/needles: ~
  /var/lib/openqa/share/tests/opensuse/products/sle-micro/needles: ~
  /var/lib/openqa/share/tests/opensuse/products/windows/needles: ~
  /var/lib/openqa/share/tests/windows: ~
  /var/lib/openqa/share/tests/windows/products/alp/needles: ~
  /var/lib/openqa/share/tests/windows/products/d-installer/needles: ~
  /var/lib/openqa/share/tests/windows/products/kubic/needles: ~
  /var/lib/openqa/share/tests/windows/products/leap-micro/needles: ~
  /var/lib/openqa/share/tests/windows/products/microos/needles: ~
  /var/lib/openqa/share/tests/windows/products/opensuse/needles: ~
  /var/lib/openqa/share/tests/windows/products/sle-micro/needles: ~
  /var/lib/openqa/share/tests/windows/products/windows/needles: ~
attempts: 1
children: []
created: 2024-09-26T11:00:04.004942Z
delayed: 2024-09-26T13:17:24.742090Z
expires: ~
finished: ~
id: 4355236
lax: 0
notes:
  gru_id: 20606345
parents: []
priority: 10
queue: default
result: ~
retried: 2024-09-26T13:16:48.742090Z
retries: 225
started: 2024-09-26T13:16:48.701831Z
state: inactive
task: git_clone
time: 2024-09-26T13:16:59.526746Z
worker: 2603

https://github.com/os-autoinst/openQA/pull/5951 for making the systemd script work.

next step: also enqueue git_clone for job restarts

Updated https://github.com/os-autoinst/openQA/pull/5951

I had a closer look at OpenQA::Task::Needle::Save now and will change it so that it creates a limit_needle_task and a git_clone_${needledir}_task guard, so that the git_clone task only needs the git_clone guard for every path, but not the limit_needle_task guard.

https://github.com/os-autoinst/openQA/pull/5961 Improve minion guards for needle tasks

Apparently the delete_needles task is also doing git operations, just hidden in the result class.

While trying to change the guard for that as well, I noticed that t/ui/21-admin-needles.t is failing for me locally.

ok 6 - last use is right                                                                                 
not ok 7 - last match is right                                                                           

#   Failed test 'last match is right'                                                                    
#   at t/ui/21-admin-needles.t line 76.                                                                  
#          got: 'about 12 hours ago'                                                                     
#     expected: 'about 14 hours ago'        

I did further checks: Going to the needle admin interface on o3, osd and in my local instance I noticed that every reported time of last seen or last match is two hours older than it should be, so it's a timezone issue.
The server returns a timestamp without an offset to the client.

Currently writing a fix for this.

I need this test to pass so I can test my actual feature.

While adding code for the new minion guards in OpenQA::Task::Needle::Delete I saw that we have insufficient tests in that area, so I added tests first:
https://github.com/os-autoinst/openQA/pull/5969

https://github.com/os-autoinst/openQA/pull/5961 ready for review

next step (again): also enqueue git_clone for job restarts

Ready: https://github.com/os-autoinst/openQA/pull/5953 Trigger git_clone also for Job restart

I enabled and started openqa-enqueue-git-auto-update.timer but saw another problem again:
https://openqa.opensuse.org/minion/jobs?id=4405061

---
args:
- /var/lib/openqa/share/tests/example: ~
  /var/lib/openqa/share/tests/obs: ~
  /var/lib/openqa/share/tests/openqa: ~
  /var/lib/openqa/share/tests/openqa/products/openqa/needles: ~
  /var/lib/openqa/share/tests/opensuse: ~
  /var/lib/openqa/share/tests/opensuse/products/alp/needles: ~
  /var/lib/openqa/share/tests/opensuse/products/kubic/needles: ~
  /var/lib/openqa/share/tests/opensuse/products/microos/needles: ~
  /var/lib/openqa/share/tests/opensuse/products/opensuse/needles: ~
  /var/lib/openqa/share/tests/opensuse/products/sle-micro/needles: ~
attempts: 1
children: []
created: 2024-10-09T10:36:06.786822Z
delayed: 2024-10-09T10:52:04.955126Z
expires: ~
finished: ~
id: 4405061
lax: 0
notes:
  gru_id: 20646663
parents: []
priority: 10
queue: default
result: ~
retried: 2024-10-09T10:51:26.955126Z
retries: 26
started: 2024-10-09T10:51:26.906912Z
state: inactive
task: git_clone
time: 2024-10-09T10:52:03.132277Z
worker: 2644

Could not get guard for git_clone_/var/lib/openqa/share/tests/opensuse/products/opensuse/needles_task, retrying in 31s

The needle directories also need to be skipped if they are symlinks. I only checked the product dirs.

https://github.com/os-autoinst/openQA/pull/5991 Skip all symlinks for git_auto_update service

Disabling the timer again for now, but the feature can stay active.

Merged and deployed on o3.

systemctl enable openqa-enqueue-git-auto-update.timer
systemctl start openqa-enqueue-git-auto-update.timer

It immediately started and finished very fast, according to logfile everything ok:
https://openqa.opensuse.org/minion/jobs?id=4405459

One observation:
There can be cases when a lot of jobs are created. One case is openqa-investigate.
I can see many git_clone jobs created at the same time and see many investigate jobs in the scheduled table.
In that case the git_clone jobs are waiting for each other and often doing the same thing (although sometimes with different refs of course).
(This was also the case before this ticket as in that case we have an explicit CASEDIR.)

I think it might be better suited for the scheduler instead of the individual job creation/restart/schedule product events.

The scheduler regularly looks for new jobs. There could be an additional step where it collects all CASEDIR/NEEDLE_DIR settings from new jobs and creates one minion job per CASEDIR/NEEDLE_DIR.
But maybe this would require a new status.

Not sure what to do now. We should have a discussion whether it makes sense to move the code to the scheduler.

I disabled the feature again because people were reporting that they are getting Another git task is ongoing. Try again later. when trying to save needles.
I checked the database:

select id, created, result from minion_jobs where result::text like '%Another git task is ongoing%' and created >= '2024-10-09' order by id limit 1000;
   id    |            created            |                           result                           
---------+-------------------------------+------------------------------------------------------------
 4407213 | 2024-10-09 17:51:44.048423+00 | {"error": "Another git task is ongoing. Try again later."}
...
 4412348 | 2024-10-10 08:24:35.936075+00 | {"error": "Another git task is ongoing. Try again later."}
 4412895 | 2024-10-10 09:06:21.612128+00 | {"error": "Another git task is ongoing. Try again later."}
(28 rows)

It was happening 28 times since yesterday morning.
I will check later how often we still see this. I turned of the feature at 08:57 UTC, so there was at least one occasion after that https://openqa.opensuse.org/minion/jobs?id=4412895

We discussed today that it might work to check for existing GruTasks with the same args and assign the openqa jobs to those tasks.

I created a proof of concept:
https://github.com/os-autoinst/openQA/pull/6001

Draft: https://github.com/os-autoinst/openQA/pull/6001 Reuse existing GruTask

https://github.com/os-autoinst/openQA/pull/6001 merged

I enabled the feature on o3 again and disabled the cronjob. Monitoring

disabled again. I think I found a bug

https://github.com/os-autoinst/openQA/pull/6015 Fix handling of job array in enqueue_git_clones

https://github.com/os-autoinst/openQA/pull/6015 merged, enabled feature again on o3