action #166433
closed[alert] Waves of emails due to manual changes in /opt/openqa-trigger-from-obs size:S
Added by livdywan 2 months ago. Updated 2 months ago.
0%
Description
Observation¶
Many emails with the subject Cron <geekotest@ariel> git -C /opt/openqa-trigger-from-obs pull --quiet --rebase origin master and the contents
error: cannot pull with rebase: You have unstaged changes.
error: please commit or stash them.
Suggestions¶
- Lookup ways to deduplicate or throttle cron emails
- Replace such cron jobs with systemd timers
- Add the timer definition to the git repo
- Copy the service/timer to avoid it being changed if the git repo is rolled back
Updated by livdywan 2 months ago
Seems like this case was caused by #166430. For reference:
diff --git a/t/obs/systemsmanagement:Agama:Devel/s390x/files_iso.lst b/t/obs/systemsmanagemen
t:Agama:Devel/s390x/files_iso.lst
index 7141d0d..063d298 100644
--- a/t/obs/systemsmanagement:Agama:Devel/s390x/files_iso.lst
+++ b/t/obs/systemsmanagement:Agama:Devel/s390x/files_iso.lst
@@ -2,3 +2,7 @@ agama-installer.aarch64-9.0.0-openSUSE-Build17.9.iso
agama-installer.ppc64le-9.0.0-openSUSE-Build17.9.iso
agama-installer.s390x-9.0.0-openSUSE-Build17.9.iso
agama-installer.x86_64-9.0.0-openSUSE-Build17.9.iso
+agama-installer.aarch64-9.0.0-openSUSE-Build17.19.iso
+agama-installer.ppc64le-9.0.0-openSUSE-Build17.19.iso
+agama-installer.s390x-9.0.0-openSUSE-Build17.19.iso
+agama-installer.x86_64-9.0.0-openSUSE-Build17.19.iso
diff --git a/xml/obs/systemsmanagement:Agama:Devel.xml b/xml/obs/systemsmanagement:Agama:Deve
l.xml
index a398145..972c3aa 100644
--- a/xml/obs/systemsmanagement:Agama:Devel.xml
+++ b/xml/obs/systemsmanagement:Agama:Devel.xml
@@ -6,7 +6,7 @@
<batch name="base" archs="86_64 aarch64 ppc64le">
<flavor name="agama-installer" folder="images/*/agama-installer:openSUSE/" iso="1" m
edia1="0"/>
</batch>
- <batch name="s390x" archs="s390x" iso_path="iso">
+ <batch name="s390x" archs="s390x">
<flavor name="agama-installer" folder="images/*/agama-installer:openSUSE/" iso="extra
ct_as_repo" media1="0"/>
</batch>
</openQA>
Updated by tinita 2 months ago
- Related to action #166430: [tools] Keep a hardlink to iso in repo folder for agama test added
Updated by livdywan 2 months ago
- Status changed from In Progress to Feedback
- Replace such cron jobs with systemd timers
So I replaced /etc/cron.d/openqa-trigger-from-obs-update-git (-*/3 * * * * geekotest git -C /opt/openqa-trigger-from-obs pull --quiet --rebase origin master) with openqa-trigger-from-obs-update.{service,timer}
Updated by livdywan 2 months ago
livdywan wrote in #note-4:
- Replace such cron jobs with systemd timers
So I replaced /etc/cron.d/openqa-trigger-from-obs-update-git (
-*/3 * * * * geekotest git -C /opt/openqa-trigger-from-obs pull --quiet --rebase origin master) withopenqa-trigger-from-obs-update.{service,timer}
@mkittler suggested to add the timer/service to git and maybe just link it, so I'll look into that before resolving the ticket.
Updated by livdywan 2 months ago
- Copied to action #166721: [alert] Waves of emails due to kex_exchange_identification: Connection closed by remote host errors added
Updated by livdywan 2 months ago
- Status changed from Feedback to Resolved
livdywan wrote in #note-7:
@mkittler suggested to add the timer/service to git and maybe just link it, so I'll look into that before resolving the ticket.
https://github.com/os-autoinst/openqa-trigger-from-obs/pull/268
sudo ln -s systemd/openqa-trigger-from-obs-update.* /etc/systemd/system
Updated by livdywan 2 months ago
- Copied to action #166739: Consistent alerts for failed systemd services on o3 size:S added
Updated by livdywan 2 months ago
tinita wrote in #note-10:
Just mentioning that with this we won't get alerted for o3 anymore if the timer fails.
I guess this wasn't covered explicitly by the estimation. To my mind this is out of scope. Errors end up in the journal. Another existing example would be openqa-continuous-update, which can be checked via sudo journalctl -r -u openqa-continuous-update | grep rror to see similar errors.
We should have a follow-up ticket on that, though, to actually consider the status quo. So thank you for raising that. I filed #166739.