Project

General

Profile

Roadmap

QA (public, currently private due to #173521) - QE-Core: Ready

open

Used by QE Core to track sprint items

93%

945 issues   (873 closed — 72 open)

Related issues
action #113716: [qe-core] proxy-scc is down Actions
action #127523: [qe-core][s390x][kvm] Make use of generic "s390-kvm" class to prevent too long waiting for s390x worker ressources Actions
action #134363: [qe-tools] New InfluxDB database for QE-Core metrics Actions
action #151331: [qe-core][s390x][zvm] Make use of generic "s390x-zVM" class instead of s390x-zVM-vswitch-l2+l3+etc. Actions

openQA Project (public) - Ready

open

Next tasks to be picked up by the QE tools team

98%

2681 issues   (2642 closed — 39 open)

Related issues
action #12506: openqa-scripts on osd/o3 should be deployed automatically (CD) Actions
action #18142: [labs] Create wiki with up to date information (linklist included) Actions
action #19190: make use of ix64ph1014, e.g. for proxymode Actions
action #19238: setup pool devices+mounts+folders with salt(was: ext2 on workers busted) Actions
action #29015: [o3][monitoring] too often alert from nagios about swap space depleted; alarm notification disabled -> increase/disable swap? Actions
action #37644: [tools] osd SSL certificate is only valid for openqa.suse.de, not for openqa.nue.suse.com Actions
action #44078: Implement proper backups for o3 size:M Actions
action #44612: Do we want to update http://tumblesle.qa.suse.de/ or decommision it? Actions
action #49694: openqaworker7 lost one NVMe Actions
action #51836: Manage (parts) of s390 kvm instances (formerly s390p7 and s390p8) with salt Actions
action #53471: machine aarch64.o.o often unresponsive and needs power-cycle Actions
action #57233: Properly handle transport errors in ObsRsync plugin Actions
action #57680: o3: optimize fs utilization and resize root disk (was: change /var/log to bind mount to prevent out-of-space) Actions
action #59300: auto_review:"DBus.*: The name org.opensuse.os_autoinst.switch was not provided":retry GRE tunnel settings not applied on initial setup / after reboot Actions
action #59621: osd: Sporadically high CPU and IO load (vdd), grafana alerts "Disk I/O time for /dev/vdd" and "CPU usage", also other disks Actions
action #59933: Prevent depletion of space on / with extra partition or quota on /home on o3 size:M Actions
action #61221: osd: unable to save needles, minion fails with "fatal: Unable to create '/var/lib/openqa/.../needles/.git/index.lock'" Actions
action #61994: VNC console corruption on aarch64 Actions
action #63142: Upgrade firmware of ppc9 machine redcurrant Actions
action #64279: [virtualization][OS upgrade] upgrade xen host openqaw5-xen.qa.suse.de Actions
action #64685: openqaworker1 showing NVMe problems "kernel: nvme nvme0: Abort status: 0x0" Actions
action #64700: setup o3 workers openqaworker4 and openqaworker7 for multi-machine tests size:S Actions
action #64941: after every reboot openqaworker7 is missing var-lib-openqa-share.mount , check dependencies of service with openqaworker1 Actions
action #64983: Staging machine openqa-staging-1.qa.suse.de does not boot after update to Leap 15.1 Actions
action #65130: Upgrade of firmware(s) for cloudberry (power9 machine) Actions
action #65154: root partition on osd exceeds alert threshold, 90%, after osd deployment -> apply automatic reboots to OSD machines Actions
action #65178: Drop rsync.pl config from salt for osd and o3 Actions
action #65975: Monitoring for "scheduled but not executed" (was: perl-Mojolicious-8.37 broke WS connection for (some?) workers) Actions
action #66709: Storage server for OSD and monitoring Actions
action #67804: use non-personal account and key for pushing needles on osd to gitlab.suse.de Actions
action #68053: powerqaworker-qam-1 fails to come up on reboot (repeatedly) Actions
action #68077: alert about too many failed minion jobs but https://openqa.suse.de/minion/jobs?state=failed shows none Actions
action #68095: Migrate osd workers from SuSEfirewall2 to firewalld Actions
action #68161: alert raised about "Minion workers down" after osd-deployment 2020-06-17 Actions
action #68410: repeated alerts about "no data" that are not actionable and recover themselves often enough Actions
action #68785: [monitoring] Setup of QA generic monitoring instance Actions
action #68872: job age max exceeds alarm threshold Actions
action #69202: icinga alert "openqaworker3.suse.de/Number of threads" Actions
action #69475: [tools] openQA child task fails to download asset created by parent job Actions
action #69523: lessons learned: osd did not come up after reboot 2020-08-02 Actions
action #69577: Handle installation of the new "Storage Server" Actions
action #69610: ipmi management interface of openqaworker-arm-3 is inaccessible Actions
action #69613: osd-pre-deployment checks fail due to invalid certificates for stats.openqa-monitor.qa.suse.de Actions
action #69664: [osd][alert] CPU usage alert: IOwait too high Actions
action #69667: missing monitoring data for vde after partitions where reordered Actions
action #69694: openqa-worker systemd services running in osd which should not be enabled at all and have no tap-device configured auto_review:"backend died:.*tap.*is not connected to bridge.*br1":retry Actions
action #69718: harmonize timezone used for our machines size:M Actions
action #69721: autoinst.qa is pingable and resolves to snipe.qa.suse.de, a VM, whereas there is a physical machine as well Actions
action #69727: reduce heat in NUE-SRV2 Actions
action #70768: obs_rsync_run and obs_rsync_update_builds_text Minion tasks fail frequently Actions
action #70834: [alert] Refine I/O time alerts for OSD Actions
action #70843: OSD automatic deployment failed in Sept, 02, 2020 Actions
action #70885: [osd][alert] flaky file system alert: /assets Actions
action #70909: salt CI jobs fail due to (now) missing python3 package Actions
action #70918: gitlab pillar merge requests try to execute CI tests and fail because tests from states are not compatible Actions
action #70939: [alert] ** PROBLEM Service Alert: ariel-opensuse.suse.de/root partition is WARNING ** Actions
action #70966: ipmi management interface of openqaworker-arm-3 is inaccessible Actions
action #70969: openqaworker-arm-2 stuck in system management menu after reboot Actions
action #70975: [alert] too many failed minion jobs Actions
action #70978: automatic reboots on o3 to activate new kernel versions Actions
action #71011: [alert] workers alert Actions
action #71098: openqaworker3 down but no alert was raised Actions
action #71182: OBS package os-autoinst blocked since multiple days on "blocked: downloading 6 dod packages" Actions
action #71191: inform EngInfra automatically if the IPMI interfaces are not accessible Actions
action #71224: [osd][alert] NTP offset alert Actions
action #71332: [alert] failed systemd service on openqaworker6: "display-manager" Actions
action #71575: [osd][alert] limited /assets - idea: ask EngInfra for slow+cheap storage from central server for /assets/fixed only Actions
action #71635: [osd][alert] QA-Power8-4-kvm: CPU Usage alert - alert needs to be refined Actions
action #71893: can not reach openqaworker-arm-2-ipmi.suse.de DNS anymore from a gitlab CI pipeline Actions
action #71908: [osd][alert] CPU usage alert (High IO wait on OSD) Actions
action #71962: [osd-admins] openqaworker-arm-1: MergePoint email alert " Voltage sensor, warning event was asserted" Actions
action #72136: [osd-admins] [Alerting] Workers alert during osd deployment, then "ok" after 1 minute, should not alert Actions
action #72139: openQA services on OSD failed to connect to database Actions
action #73165: [osd] Consolidate "expensive+fast" and "cheap+slow" storage after realizing vdc is "cheap+slow" as well Actions
action #73174: [osd][alert] Job age (scheduled) (median) alert Actions
action #73183: Extend vdc (/assets) Actions
action #73189: Upgrade o3 workers to openSUSE Leap 15.2 after openqa-aarch64 already done Actions
action #73228: [labs] Create guide for new machines (from the "customer perspective") Actions
action #73234: ipmi management interface of openqaworker7 is inaccessible ("no matching cipher suite") and not pingable Actions
action #73246: [osd-admins][alert] openqaworker8: Memory usage alert Actions
action #73333: Failed systemd services alert (workers) flaky Actions
action #73342: all jobs run on openqaworker8 incomplete: "Cache service status error from API: Minion job .*failed: .*(database disk image is malformed|not a database)":retry Actions
action #73348: The worker openqaworker-arm-2.suse.de is down, so the OSD deployment fails Actions
action #73360: merging MRs in openqa/salt-states-openqa does not trigger pipeline in master Actions
action #73387: Cleanup of old needles from os-autoinst-needles-opensuse and os-autoinst-needles-sle Actions
action #73399: [osd][alert] Failed systemd services alert (workers) - many alert messages Actions
action #73405: job incompletes with "(?s)openqaworker8.*terminated prematurely.*OpenCV Error: Insufficient memory" Actions
action #73483: ipmitool from openSUSE Leap 15.2 and TW to our openQA workers fails with "no matching cipher suite", works on login.suse.de (SLES15SP1) Actions
action #73501: Bind mounts of fixed assets is racy Actions
action #73633: OSD partially unresponsive, triggering 500 responses, spotty response visible in monitoring panels but no alert triggered (yet) Actions
action #75016: [osd-admins][alert] Failed systemd services alert (workers): os-autoinst-openvswitch.service (and var-lib-openqa-share.mount) on openqaworker-arm-2 and others Actions
action #75055: grenache-1 can't connect to webui's over IPv4 only Actions
action #75067: save_needle minion task fails because "Your account has been blocked" Actions
action #75220: all jobs run on openqaworker8 incomplete: "Cache service status error from API: Minion job .*failed: .*(database disk image is malformed|not a database)":retry Actions
action #75235: container image devel:openQA:ci/base is unresolvable with "nothing provides this-is-only-for-build-envs needed by cmake-mini" Actions
action #75238: Upgrade osd workers and openqa-monitor to openSUSE Leap 15.2 Actions
action #75241: Upgrade o3 webUI host to openSUSE Leap 15.2 Actions
action #75244: Upgrade osd webUI host to openSUSE Leap 15.2 Actions
action #75259: 100% of powerpc tests incomplete auto_review:"(?s)Running on power8.*qemu-system-ppc64: Requested safe cache capability level not supported by kvm":retry Actions
action #75274: [osd-admins][alert][learning] Failed systemd services alert (workers): os-autoinst-openvswitch.service aborts retries after 60s and is not easily configurable Actions
action #75277: enable "postfix" service again after #73633 resolved Actions
action #75403: salt "cheat sheet" for common commands to change/fix osd infrastructure Actions
action #75445: unknown dashboards for "linux-fwcx" and "localhost" reappearing on monitor.qa Actions
action #75448: OSD deployment fails because openqaworker-arm-3.suse.de is not connected Actions
action #76774: auto-review/openqa-label-known-issues does not conclude within GitLab CI Actions
action #76783: research how hostnames with systemd work and make them static for all OSD related machines Actions
action #76786: Configure static hostnames with salt for all salt nodes Actions
action #76804: openqaworker-arm-2 was detected as offline but no grafana webhook action was triggered Actions
action #76822: Fix /results over-usage on osd (was: sudden increase in job group results for SLE 15 SP2 Incidents) Actions
action #76828: big job queue for ppc as powerqaworker-qam-1.qa and malbec.arch and qa-power8-5-kvm were not active Actions
action #76876: Find a better (automated) way to inform infra about hanging (arm) workers Actions
action #76903: openQA test modules failure statistics collection Actions
action #76924: Upgrade postgresql database version on o3 to default of Leap 15.2, i.e. postgres12 size:M Actions
action #76927: Upgrade postgresql database version on osd to default of Leap 15.2, like on o3 size:M Actions
action #76951: Check if new firmware for kerosene (aka. power8.o.o) exists and remove os-autoinst workarounds again when according machine settings are applied when necessary size:M Actions
action #76987: re-encode some videos from existing results to save space Actions
action #77017: Add "how to do rollback of deployment" to https://gitlab.suse.de/openqa/osd-deployment/ Actions
action #77089: [osd][retrospective] multiple unattended alerts, unattended gitlab CI pipeline fails, all osd aarch64 workers offline Actions
action #77101: fix selection of gitlab CI runners Actions
action #77191: [easy][learning] ** PROBLEM Service Alert: openqaworker12.suse.de/Interface br1 is CRITICAL ** Actions
action #77209: workers on o3 machine rebel provide no "WORKER_HOSTNAME" value anymore but it shows up in journal of worker service Actions
action #77218: gitlab CI pipeline openqa/grafana-webhook-actions failed with "You have reached your pull rate limit" for docker hub but should not use docker hub at all Actions
action #77827: asg-qe.maintenance reported problems about too aggressive asset cleanup - bumping some settings Actions
action #77836: login to aarch64.o.o fails with ssh keys and password, also not working over IPMI SoL Actions
action #77839: Use external videoencoder in production on one o3 worker Actions
action #77842: Use external videoencoder in production on all o3 machines size:M Actions
action #77845: Use external videoencoder in production on all osd machines size:M Actions
action #77848: No more workarounds in OSD due to inefficient video encoder size:M Actions
action #77887: [tools][openqa] Enable automatic openQA investigation jobs for osd as well Actions
action #77890: [easy] Extend OSD storage space for "results" to make bug investigation and failure archeology easier Actions
action #78010: unreliable reboots on openqaworker3, likely due do openqa_nvme_format (was: [alert] PROBLEM Host Alert: openqaworker3.suse.de is DOWN) Actions
action #78058: [Alerting] Incomplete jobs of last 24h alert - again many incompletes due to corrupted cache, on openqaworker8 Actions
action #78064: failing logrotate on monitor.qa.suse.de due to mariadb/mysql? Actions
action #78127: follow-up to #73633 - lessons learned and suggestions Actions
action #78165: infrastructure task: After osd deployment 2020-11-18 many jobs incomplete with auto_review:"Cache service (status error from API|.*error 500: Internal Server Error)":retry Actions
action #78218: [openQA][worker] Almost all openQA workers become offline Actions
action #78438: openQA webui entry "Assigned worker" shows ip instead of names as formerly - manual cleanup work Actions
action #80106: corrupted worker cache sqlite: Enlarge systemd service kill timeout temporarily Actions
action #80128: openqaworker-arm-2 fails to download from openqa Actions
action #80166: IPMI SOL does not show me anything anymore during bootup until the linux getty login prompt, can anybody confirm? Actions
action #80178: gitlab.suse.de CI shared runners are unable to resolve addresses within suse.de domain Actions
action #80380: salt deploy often or always fails in test-webui and deploy due to postgres files not found Actions
action #80398: [openQA] increase storage space for baremetal_support VM on qamaster Actions
action #80408: revert longer timeout override for openQA services as we could not see less problems with corrupted worker cache Actions
action #80538: flaky and misleading alerts about "openQA minion workers alert" as well as "Minion Jobs alert" Actions
action #80542: Configure "automatic power-on" after power loss for openqaworker1 (and all others) Actions
action #80544: Ensure that IPMI for powerqaworker-qam works reliably Actions
action #80594: Needles are not pushed from o3 to github repo Actions
action #80656: OSD deployment failed at 2020-12-02 because 'malbec.arch.suse.de' is down Actions
action #80688: Upgrade IO firmware for powerqaworker-qam-1 Actions
action #80734: GitLab pipeline trigger via Grafana fails due to TLS errors Actions
action #80768: All workers in grenache-1 are broken at 2020-12-07 Actions
action #80770: "host up" alerts alerting on Sunday around 03:10Z and all going back to ok 30m later, should be prevented Actions
action #80812: Fix mail sending on o3 size:S Actions
action #80834: [alert] osd reported 502 errors, unresponsive, failing alerts for CPU usage since 2020-12-08 0840Z and minion jobs failed Actions
action #80956: update osd VM root disk VM config and resize root disk on osd Actions
action #81020: QA-Power8-4-kvm start failed since reboot on 2020-12-13 Actions
action #81026: many jobs incomplete with auto_review:"(?s)Running on openqaworker-arm-2.*failed: 521 Connect timeout.*Result: setup failure":retry Actions
action #81046: openqaworker-arm-2.suse.de unreachable Actions
action #81058: [tracker-ticket] Power machines can't find installed OS. Automatic reboots disabled for now Actions
action #81198: [tracker-ticket] openqaworker-arm-{1..3} have network problems (cacheservice, OSD reachability). IPv6 disabled for now Actions
action #81220: overdrive2.arch wants to be accepted with salt key on osd, what to do with this machine? Actions
action #81228: many o3 workers not working on jobs as of 2020-12-21 (w4 seems to be ok) Actions
action #81232: salt high state triggers multiple errors Actions
action #81274: consider disabling emergency mode on our machines Actions
action #81884: openqa-webui should automatically restart on config updates Actions
action #87838: *alert* osd: Open database connections by user alert Actions
action #87856: [alert] openqaworker3.suse.de/NRPE is CRITICAL Actions
action #87883: osd infrastructure: services like "telegraf" are not enabled to start immediately on boot Actions
action #87970: tumbleweed container images in gitlab CI fail on salt calls, e.g. "KeyError: 'cmd.run_all'" Actions
action #87979: Requirements for access to the openqa VM Actions
action #88189: Firmware upgrade of qa-power8-4.qa.suse.de Actions
action #88191: openqaworker2 boot ends in emergency shell Actions
action #88225: osd infrastructure: Many failed systemd services on various machines Actions
action #88385: openqaworker3 host up alert is flaky Actions
action #88450: Flaky NTP offset alert Actions
action #88474: All workers on powerqaworker-qam-1 are offline Actions
action #88546: Make use of the new "Storage Server", e.g. complete OSD backup Actions
action #88807: Open vSwitch command 'set_vlan' with arguments 'tap41 13' failed: 'tap41' is not connected to bridge 'br1' at /usr/lib/os-autoinst/backend/qemu.pm line 152. Actions
action #88900: openqaworker13 was unreachable Actions
action #88912: Assets cleaned up too early for Tumbleweed aarch64 Actions
action #89047: Failed to commit needles, gitlab account blocked 2021-02-24 Actions
action #89050: OSD deployment blocked since two weeks - salt nodes failing Actions
action #89113: [alert] PROBLEM Service Alert: openqa.suse.de/NTP Time is CRITICAL Actions
action #89275: Flaky Incomplete jobs (not restarted) of last 24h alert and New incompletes alert triggered and back to OK Actions
action #89419: Incomplete jobs after OSD deployment Actions
action #89497: flaky Failed systemd services alert (except openqa.suse.de) Actions
action #89551: NFS mount fails after boot (reproducible on some OSD workers) Actions
action #89815: osd-deployment blocked by openqaworker-arm-3 offline and not recovered automatically Actions
action #89821: alert: PROBLEM Service Alert: openqa.suse.de/fs_/srv is WARNING (flaky, partial recovery with OK messages) Actions
action #89993: OSD deployment rollback failed finding "before" and "osd_deployment_rpm_q" files Actions
action #90161: [Alerting] malbec: Memory usage alert triggered briefly and turned OK within the next minute Actions
action #90170: Service for purging old kernels might run while system management is locked and fail Actions
action #90275: Replacement openQA OSD aarch64 hardware (was: Dedicated non-rpi aarch64 hardware for manual testing) Actions
action #90629: administration of the new "Storage Server" Actions
action #90635: NTP alerts coinciding with reboots of ppc64le host Actions
action #90746: OSD deployment fails at 2021-04-07 because 'storage.qa.suse.de: Minion did not return' Actions
action #90755: aarch64 openQA jobs on osd not properly processed since 2021-04-05 Actions
action #90857: Add redundancy for SAP multi machines tests - Extend openQA worker config to accomodate for upgraded RAM Actions
action #90875: powerqaworker-qam-1 is online but https://monitor.qa.suse.de/d/4KkGdvvZk/osd-status-overview shows "No Data" Actions
action #90920: Restore IPMI access to malbec.arch.suse.de Actions
action #90968: [alert] Multiple flaky incomplete job alerts on Sunday Actions
action #91530: Severe performance problems on malbec Actions
action #91779: Add monitoring for storage.qa.suse.de Actions
action #91803: openqaworker8 failed to reboot: [Alerting] openqaworker8: host up alert on 2021-04-25 Actions
action #92034: Re-enable openqa-investigate options after the black certificate now only shows properly "reviewed" jobs Actions
action #92086: http://monitor.qa should redirect to https://monitor.qa.suse.de Actions
action #92110: Several Job age (scheduled) alerts on Sunday Actions
action #92113: [Alerting] openqaworker-arm-3: NTP offset alert Actions
action #92167: remove workaround for PowerPC workers after boo#1174166 fixed and updates available Actions
action #92176: [alert] openqaworker-arm-3 offline and CI pipeline unable to send email but stating "passed" Actions
action #92185: Our PowerPC worker machines need to be configured for automatic start after a power outage Actions
action #92302: NFS mount var-lib-openqa-share.mount often fails after boot of some workers Actions
action #92338: [Alerting] File systems alert, / on osd Actions
action #92467: Unit has `iscsid.socket` failed on some OSD workers since today's nightly reboot Actions
action #92701: backup of etc/ from both o3 was not working since some days due to OOM on backup.qa.suse.de (was: … and osd not updated anymore since 2019) Actions
action #92770: openqa.opensuse.org down, o3 VM reachable, no failed service Actions
action #92915: OSD deployment fails at 2021-05-21 because ' openqaworker-arm-2.suse.de Minion did not return' Actions
action #92969: Failing service os-autoinst-openvswitch after boot of some workers Actions
action #92978: salt failed to write files on grenache-1.qa.suse.de 'workers.ini': Operation not permitted Actions
action #92996: OSD deployment fails at 2021-05-24 because ' openqaworker-arm-3.suse.de Minion did not return' Actions
action #93050: Proposal: Use openqaworker11 and openqaworker12 as normal workers and only pull out from production when necessary Actions
action #93071: o3 s390x worker machines unreachable Actions
action #93195: [Alerting] Failed systemd services alert (except openqa.suse.de) on 2021-05-28, logrotate.service on openqaworker-arm-1 Actions
action #93381: [O3]request to add an IPMI SUT to O3 size:M Actions
action #93612: Several unhandled alerts triggered regarding incompletes and running out of space Actions
action #93650: alert: PROBLEM Service Alert: openqa.suse.de/fs_/assets is WARNING Actions
action #93683: osd-deployment failed due to storage.qa.suse.de not reachable by salt Actions
action #93919: salt gitlab CI pipeline fails applying openqa-webui restart state with "The following requisites were not found: watch: file: /etc/openqa/openqa.ini" Actions
action #93922: grafana dashboard for "approximate result size by job group" fails to render any data with "InfluxDB Error: unsupported mean iterator type: *query.stringInterruptIterator" Actions
action #93961: Add redundancy for SAP multi machines tests - extend RAM on machines Actions
action #93964: salt-states CI pipeline deploy step fails on some workers with "Unable to unmount /var/lib/openqa/share: umount.nfs: /var/lib/openqa/share: device is busy." Actions
action #94015: proper backup for osd Actions
action #94237: No alert about too many scheduled tests size:S Actions
action #94312: [Alerting] web UI: Too many Minion job failures alert - likely due to openqa-client declared deprecated Actions
action #94318: http://jenkins.qa.suse.de not reachable Actions
action #94438: OSD deployment fails at 2021-06-21 because ' openqaworker (arm-3 and arm-2) Minion did not return' Actions
action #94456: no data from any arm host on https://stats.openqa-monitor.qa.suse.de/d/1bNU0StZz/automatic-actions?orgId=1 Actions
action #94492: Configure retention/downsampling policy for monitoring data stored within InfluxDB size:M Actions
action #94513: openqaworker-arm-3 not reachable and not recoverable over usual ways Actions
action #94555: backup.qa.suse.de needs to be upgraded to Leap 15.2 (size:S) Actions
action #94576: alert: PROBLEM Service Alert: openqa.suse.de/fs_/results is WARNING Actions
action #94747: broken RPM database on openqaworker-arm-* during osd deployment Actions
action #94919: All arm workers down 2021-06-30 , NUE SRV2 Rack A8 was switched off by EngInfra size:S Actions
action #94940: multiple network related problems, gitlab CI pipelines not working, workers not reachable, proxySCC not reachable Actions
action #94949: Failed systemd services alert for openqaworker3 var-lib-openqa-share.automount Actions
action #94994: Tracebacks in the journal for salt-minion size:M Actions
action #95140: regression: salt state fails on one worker with "Passed invalid arguments to state.highstate: expected str, bytes or os.PathLike object, not list" size:M Actions
action #95167: Bring openqaworker12 into production including multi-machine test support Actions
action #95293: Monitoring alerts on errors in logs on o3 (was: followup to: error on "Next & previous results": ajax error message and no results showing up) size:M Actions
action #95443: Variants of Job age (scheduled) alerts on Grafana on Sunday and Monday size:S Actions
action #95482: openqaworker-arm-3 offline and not automatically recovered due to gitlab CI failures Actions
action #95980: "no data" alert for "Incomplete jobs (not restarted)" but there should be no alert for "no data" size:S Actions
action #95983: alert about "minion workers", alert triggered two times and turned green again Actions
action #96089: alert about "minion workers" - make meaning of grafana panel clear size:S Actions
action #96242: [alert] Disk I/O time for /dev/vde (/space-slow) alert 2021-07-28 size:M Actions
action #96269: Define what a "complete OSD backup" should or can include Actions
action #96272: Test failed incomplete: "Reason: backend died: QEMU exited unexpectedly, see log for details" size:M Actions
action #96380: "minion workers" alert shows <1 total minion workers if active == 1 size:M Actions
action #96551: Persistent records of systemd journal size:S Actions
action #96554: Mitigate on-going disk I/O alerts size:M Actions
action #96618: The machine is unable to boot from PXE in NUE QA network due to TFTP open timeout Actions
action #96710: Error `Can't call method "write" on an undefined value` shows up in worker log leading to incompletes Actions
action #96713: Slow grep in openqa-label-known-issues leads to high CPU usage Actions
action #96719: recover imagetester with broken filesystem/hardware (was: automatic updates on imagetester don't work and it failed to come up after reboot) Actions
action #96789: File systems alert 90.256 assets used size:M Actions
action #96795: CPU Load alert and telegraf going between 41%, 98.5% and 115% CPU Actions
action #96807: Web UI is slow and Apache Response Time alert got triggered Actions
action #96938: openqaworker10+13 are offline, reason unknown, let's fix other problems first size:M Actions
action #96947: Availability of openQA webUI is above 100% size:S Actions
action #97043: job queue hitting new record 14k jobs Actions
action #97136: [alert] multiple unhandled alerts about "broken workers" size:M Actions
action #97139: [alert] multiple unhandled alerts about "malbec: Memory usage alert" size:M Actions
action #97244: openqaworker-arm-3 is offline and EngInfra wants us to create JiraSD tickets instead of infra size:M Actions
action #97364: openqaworker-arm-2 and openqaworker-arm-3 seem to be offline, alerts had been triggered size:S Actions
action #97382: ARM automatic reboot pipeline does not fail if ipmitool fails size:S Actions
action #97406: gitlab emails to osd-admins@suse.de are not accepted as the ML is not in "To:" field but CC/BCC Actions
action #97415: handle request about machines in SRV1 from SUSE-IT/Accenture to fill a spreadsheet with system properties Actions
action #97418: Pipeline in salt-states-openqa can fail occasionally size:M Actions
action #97502: osd deployment failed due to openqaworker-arm-3 being down, needs to be worked around size:M Actions
action #97574: deployment failed in gitlab job with "ERROR: Job failed: execution took longer than 1h0m0s seconds" Actions
action #97583: [spike] Configure retention/downsampling policy for monitoring data stored within InfluxDB on newer influxdb version Actions
action #97658: many (maybe all) jobs on rebel within o3 run into timeout_exceeded "setup exceeded MAX_SETUP_TIME" size:M Actions
action #97751: replacement setup for o3 s390x openQA workers size:M Actions
action #97862: More openQA worker hardware for OSD size:M Actions
action #97943: Increase number of CPU cores on OSD VM due to high usage size:S Actions
action #98243: salt-states-openqa pipeline/job "test-worker" fails because zypper reports an error size:M Actions
action #98307: Many jobs in o3 fail with timeout_exceeded on openqaworker1 auto_review:"timeout: setup exceeded MAX_SETUP_TIME":retry size:M Actions
action #98499: [alert] web UI: Too many Minion job failures alert size:S Actions
action #98661: Tweak worker numbers for atomkraft and wasserstoff Actions
action #98922: Run asset cleanup concurrently to results based on config Actions
action #98979: monitor-post-deployment failed while arm3 was being rebooted by our automatic recovery Actions
action #99045: osd deployment fails due to zypper package resolution problem Actions
action #99117: malbec 🍷️ is not reachable via ssh or ipmi Actions
action #99189: Upgrade o3 workers to openSUSE Leap 15.3 size:M Actions
action #99192: Upgrade osd workers and openqa-monitor to openSUSE Leap 15.3 size:M Actions
action #99195: Upgrade o3 webUI host to openSUSE Leap 15.3 size:M Actions
action #99198: Upgrade osd webUI host to openSUSE Leap 15.3 size:M Actions
action #99201: Upgrade postgresql database version on o3 to default of Leap 15.3, i.e. postgres14 size:M Actions
action #99204: Upgrade postgresql database version on osd to default of Leap 15.3, like on o3 Actions
action #99240: Upgrade CI container image versions to Leap 15.3 size:M Actions
action #99243: Upgrade qam hosts maintained by us to latest stable, i.e. Leap 15.3 Actions
action #99288: [Alerting] openqaworker-arm-5 and openqaworker-arm-4: host up alert on 2021-09-26 size:M Actions
action #99333: qa-maintenance/openQABot CI job fails after max retries to reach germ159.suse.cz Actions
action #99741: Minion jobs for job hooks failed silently on o3 size:M Actions
action #100712: Investigate what broke git checkouts on o3 Actions
action #100859: investigate how to optimize /srv data utilization on OSD size:S Actions
action #100862: Failed systemd services alert: session-50177.scope failed - but why? Actions
action #101006: Provide unique non-dictionary passwords for all our IPMI/HMC interfaces size:S Actions
action #101033: openqaworker13: Too many Minion job failures alert - sqlite failed: database is locked size:M Actions
action #101166: o3 was not auto-updated lately, due to unresolvables in devel:openQA for Leap 15.3 Actions
action #101271: Try Kernel:stable on arm4+arm5 and compare failure rate size:M Actions
action #101373: worker openqa-aarch64 fails on cache Actions
action #101962: update BMC/IPMI firmware on openQA hosts Actions
action #102062: check state of packages fails with test '' = clean Actions
action #102143: o3 ran out of disk space Actions
action #102284: openQABot pipeline failed with simplejson.errors.JSONDecodeError: Expecting value: line 1 column 1 (char 0) Actions
action #102575: Prevent false-positive ticket reporting for openqaworker-arm-3 Actions
action #102650: Organize labs move to new building and SRV2 size:M Actions
action #102713: Implement proper backups for o3 - list of installed packages Actions
action #102716: Disable job triggering during the weekly SUSE IT maintenance window Actions
action #102942: Failed systemd services alert: snapper-cleanup on QA-Power8-4-kvm fails size:M Actions
action #103128: gitlab CI pipelines sporadically fail with "Could not resolve host: gitlab.suse.de", e.g. Recovery pipelines for ARM workers might fail during the maintenance window size:M Actions
action #103149: Salt the dehydrated setup Actions
action #103380: Configure retention/downsampling policy for specific monitoring data stored within InfluxDB Actions
action #103518: opencv update broke os-autoinst (was: Tests on Raspberry Pi 2/3/4 are broken) Actions
action #103524: OW1: performance loss size:M Actions
action #103530: failed systemd services alert - openqaworker-arm-3 - ovsdb-server size:M Actions
action #103539: Update expired SSL certificate on monitor.qa.suse.de with dehydrated and salt, same as on OSD size:M Actions
action #103554: o3 s390x worker instances 102+103 down whereas 101+104 are up Actions
action #103575: [virtualization][3rd party hypervisor] Worker openqaw8-vmware.qa.suse.de is not reachable Actions
action #103602: Reply to gitlab@suse.de pipeline email fails silently Actions
action #103683: [tools][sle][x86_64][aarch64][QEMUTPM] install package "swtpm" on x86_64 and aarch64 workers Actions
action #103736: Make aarch64 machine chan-1 and chow up and running after it is broken size:M Actions
action #103848: No permission to view SD tickets filed for broken arm workers Actions
action #103954: Run asset cleanup concurrently to results based on config on o3 as well Actions
action #104085: openQABot pipeline failed with terminating connection due to administrator command size:S Actions
action #104088: bot-ng pipeline consistently fails with KeyError: 'test_issues' Actions
action #104106: [qe-core] test fails in await_install - Network peformace for ppc installations is decreasing size:S Actions
action #104142: osd-deployment pipeline failed: File ... not found on medium Actions
action #104172: osd service ca-certificates failed with "p11-kit: couldn't complete writing of file: /var/lib/ca-certificates/ca-bundle.pem.tmp: File exists" Actions
action #104217: Ask eng infra why thruk.suse.de stopped working Actions
action #104304: Crosscheck results of https://github.com/os-autoinst/os-autoinst#verifying-a-runtime-environment on arm-1/2/3 vs. arm-4/5 to find out if arm-4/5 are "typing stable" size:M Actions
action #104344: Inconsistent systemd default target in OSD infrastructure Actions
action #104347: Lost salt connection from some hosts in OSD infrastructure due to no route to host "salt" Actions
action #104607: Verify fix for https://bugzilla.suse.com/show_bug.cgi?id=1192126 regarding QEMU+OVMF, ensure updated packages on o3 size:M Actions
action #104673: Access to o3 workers is not well-documented and not automated Actions
action #104835: Failed systemd service "systemd-journal-flush" on OSD Actions
action #104869: Packages not building on Leap 15.2 and SLE15-SP3 Actions
action #104965: openqaworker10 restarted into maintenance mode - reason unknown Actions
action #104967: File systems alert repeatedly triggering on and off Actions
action #104970: Add two OSD workers (openqaworker14+openqaworker15) specifically for sap-application testing size:M Actions
action #104992: [virtualization][3rd party hypervisor] vmware test run failed with credential errors Actions
action #105169: Pipeline of openQABot project fails with "urllib.error.HTTPError: HTTP Error 503: Service Unavailable" causing alert/notification Actions
action #105274: Workers constantly logging websocket connection "finished by remote side with code 1006, no reason" Actions
action #105373: Ask to increase OSD /srv so that we can save enough logs+DB Actions
action #105594: Two new machines for OSD and o3, meant for bare-metal virtualization size:M Actions
action #105603: openQABot pipeline failed: "ERROR:root:Something bad happended during reading MR data from SMELT/IBS: Expecting value: line 4 column 1 (char 3)" size:M Actions
action #105618: [Alerting] CPU Load alert size:S Actions
action #105621: [Alerting] Failed systemd services alert Actions
action #105828: 4-7 logreport emails a day cause alert fatigue size:M Actions
action #105960: Dehydrated fails on OSD size:M Actions
action #106017: [ipmi backend]VNC stalled, no update for 5.38 seconds Actions
action #106035: [qe-tools] dehydrated service fails on osd Actions
action #106365: Improve security for OSD worker credentials broke Gitlab CI/CD deploy of salt in OSD size:M Actions
action #106538: lessons learned "five whys" for "All OSD PPC64LE workers except malbec appear to have horribly broken cache service" size:S Actions
action #106540: Mitigate/resolve All OSD PPC64LE workers except malbec appear to have horribly broken cache service Actions
action #106543: Conduct rollback steps and check impact for "All OSD PPC64LE workers except malbec appear to have horribly broken cache service" size:M Actions
action #106594: [tools] openqaworker-arm-3 periodically fails os-autoinst-openvswitch service Actions
action #106598: Redcurrant has a broken HDD Actions
action #106607: GitHub review notifications are sent to osd-admins+os-autoinst-obs@suse.de Actions
action #106666: Improve worker startup in our salt states or "openqa-worker-auto-restart repeatedly failing on grenache-1.qa.suse.de" Actions
action #106751: Update machines and passwords in the monitor-o3 repository Actions
action #106753: openqa-worker-auto-restart repeatedly failing on grenache-1.qa.suse.de Actions
action #106771: imagetester missing in action Actions
action #106832: Monitor masked units on our infrastructure Actions
action #106846: Junk messages posted on osd-admins@suse.de Actions
action #106880: Job template name ... is already used in job group error logged on o3 size:M Actions
action #106919: openqa_install+publish fails in start_test size:M Actions
action #106925: [timeboxed:10h][research] What are best practices and options in the salt and GitLab community to handle secrets Actions
action #106933: Use PSU capabilites to power cycle openqaworker-arm-[1-3] instead of infra tickets size:M Actions
action #107017: Random asset download (cache service) failures on openqaworker1 Actions
action #107074: error on openqaworker-arm-2 failing osd-deployment size:M Actions
action #107077: gitlab.suse.de/openqa/scripts CI checks fail due to xargs missing, update or decommission Actions
action #107083: SUSE QE Tools team must learn about switch administration and get access Actions
action #107086: Ask for volunteers in SUSE QE Tools that would be able to visit the Nbg server rooms, e.g. as second person accompanying nsinger or any potential new admin Actions
action #107089: Make SUSE QE Tools team aware that we need to support EngInfra due to limited capacity Actions
action #107152: [osd] failing systemd services on "grenache-1": "openqa-reload-worker-auto-restart@10, openqa-reload-worker-auto-restart@21, openqa-reload-worker-auto-restart@22, openqa-reload-worker-auto-restart@23, openqa-reload-worker-auto-restart@25, …" size:M Actions
action #107158: [osd] failing systemd service on "storage": "systemd-udev-settle" size:M Actions
action #107257: [alert][osd] Apache Response Time alert size:M Actions
action #107437: [alert] Recurring "no data" alerts with only few minutes of outages since SUSE Nbg QA labs move size:M Actions
action #107515: [Alerting] web UI: Too many Minion job failures alert size:S Actions
action #107638: OSD deployment failed due to openqaworker-arm-1 with "rpmdb2solv: inconsistent rpm database" despite our repair attempts Actions
action #107875: [alert][osd] Apache Response Time alert size:M Actions
action #107917: Recovery of imagetester via IPMI failed size:M Actions
action #107932: Handling broken RPM databases does not handle certain cases Actions
action #107989: CPU-specific worker classes Actions
action #108401: Can't call method "id" on an undefined value at V1/Job.pm Actions
action #108533: o3 logreports DBI Exception: DBD::Pg::st execute failed: ERROR: invalid input syntax for type integer Actions
action #108665: openqa_from_containers repeatedly failing at build Actions
action #108668: Failed systemd services alert (except openqa.suse.de) for < 60 min Actions
action #108671: Resilient IPMI recovery of o3 machines in monitor-o3 size:M Actions
action #108740: qa-power8-5-kvm minions alert is heart-broken 💔️ Actions
action #108743: qa-power8-5-kvm minions alert is heart-broken Actions
action #108845: Network performance problems, DNS, DHCP, within SUSE QA network auto_review:"(Error connecting to VNC server.*qa.suse.*Connection timed out|ipmitool.*qa.suse.*Unable to establish)":retry but also other symptoms size:M Actions
action #108872: Outdated information on openqaw5-xen https://racktables.suse.de/index.php?page=object&tab=default&object_id=3468 Actions
action #108896: [ppc64le] auto_review:"(?s)Size of.*differs, expected.*but downloaded.*Download.*failed: 521 Connect timeout":retry Actions
action #108998: openqa_install+publish fails in start_test Actions
action #109028: [openqa][worker][sut] Very severe stability and connectivity issues of openqa workers and suts Actions
action #109052: openqa_install+publish fails in dashboard Actions
action #109055: Broken workers alert Actions
action #109241: Prefer to use domain names rather than IPv4 in salt pillars size:M Actions
action #109253: Add monitoring for SUSE QA network infrastructure size:M Actions
action #109298: salt-minion on grenache-1 is not working preventing OSD deployment Actions
action #109301: openqaworker14 + openqaworker15 sporadically get stuck on boot Actions
action #109494: Restore network connection of arm-4/5 size:M Actions
action #109746: Improve QA related server room management, consistent naming and tagging size:M Actions
action #109969: s390zp19 - out of disk space Actions
action #109971: Service `systemd-journal-flush` timed out on OSD size:M Actions
action #110170: Request for s390x worker class with SIE support Actions
action #110266: [alert] Disk I/O time for sr0 size:S Actions
action #110269: [alert] QA-Power8-4-kvm + QA-Power8-5-kvm: Disk I/O time alert size:M Actions
action #110293: Can not ping s.qa.suse.de Actions
action #110296: Machines within .qa.suse.de unavailable (was: Some ipmi workers become offline which affects PublicRC-202204 candidate Build137.1 test run) Actions
action #110301: alert about o3: ** PROBLEM Service Alert: ariel-opensuse.suse.de/swap is CRITICAL ** size:S Actions
action #110494: alert: openqaworker5 host up size:M Actions
action #110521: Improve QA related server room management, network topology and configuration size:M Actions
action #110539: Ask OBS team if they would like to swap ARM workers with us Actions
action #110920: Emails from o3 are rejected by mx2.suse.de for certain sender/recipients size:S Actions
action #111063: Ping monitoring for our s390z mainframes size:S Actions
action #111149: Recover openqaworker-arm-3 Actions
action #111156: No effect of remote PDU controls on openqaworker-arm-4.qa and openqaworker-arm-5.qa, check power connections Actions
action #111159: Check if we can give away our left-over 10G cisco switch Actions
action #111171: Handle installation of new FC switch size:M Actions
action #111440: Avoid OSD deployment monitoring to fail due to WIP dashboard alerts Actions
action #111473: Get replacements for imagetester and openqaworker1 size:M Actions
action #111578: Recover openqaworker-arm-4/5 after "bricking" in #110545 size:M Actions
action #111755: power8 fails to execute jobs successfully, no kvm, but also no sshd auto_review:"(?s)power8.*no kvm-img/qemu-img found":retry Actions
action #111758: o3 jobs exceeding MAX_SETUP_TIME auto_review:"(?s)openqaworker4.*timeout: setup exceeded MAX_SETUP_TIME":retry size:M Actions
action #111863: Upgrade o3 workers to openSUSE Leap 15.4 size:M Actions
action #111866: Upgrade osd workers and openqa-monitor to openSUSE Leap 15.4 Actions
action #111869: Upgrade o3 webUI host to openSUSE Leap 15.4 size:S Actions
action #111872: Upgrade osd webUI host to openSUSE Leap 15.4 Actions
action #111884: Upgrade qam hosts maintained by us to latest stable, i.e. Leap 15.4 size:M Actions
action #111926: osd-deployment pipeline failed: test 481 -le 0, due to job age alert, likely just the raspberry pi based tests stuck in schedule Actions
action #111986: Ensure uno.openqanet.opensuse.org is properly used Actions
action #112193: [alert][osd] web UI: Too many Minion job failures alert size:S Actions
action #112196: [alert][sporadic] QA-Power8-4-kvm: Disk I/O time alert size:M Actions
action #112346: [alert] multiple alerts about "Download rate" and "Job age" on OSD 2022-06-12 size:M Actions
action #112553: [osd][amd][zen3][network][sriov] New AMD Zen3 machine on OSD lost its nework connection with p3p1 interface Actions
action #112583: [alert] "Job age" on OSD 2022-06-12 Actions
action #112673: Cannot send emails from o3 size:M Actions
action #112733: Webui Summary dashboard in Grafana is missing I/O panels size:M Actions
action #112781: Connect and switch on autobot for Anton Actions
action #112835: All "developers" in progress QA project hierarchy should be able to change the status of "openqa-force-result" tickets same as for "action" Actions
action #112916: postgresql.conf is invalid after recent salt changes size:M Actions
action #113366: Add three more Prague located OSD workers size:M Actions
action #113477: Get replacements for o3+osd top of rack switch Actions
action #113498: [alert] QA network infrastructure Ping time alert for several hosts Actions
action #113561: failed pipelines for openQABot and bot-ng because of an expired cert Actions
action #113662: Ensure upgrade of vanilla Leap 15.3 os-autoinst can be automatically upgraded to Leap 15.4 size:S Actions
action #113671: [timeboxed][10h] Configure write of I/O panels to be on the negative Y-axis again once we're on grafana 8.4 size:S Actions
action #113746: monitoring: The grafana "ping time" panel does not list all hosts size:S Actions
action #114397: glibc regression causes cron to crash Actions
action #114448: malbec.arch.suse.de is unreachable via IPMI Actions
action #114484: Failed service "transactional-update.service" on openqaworker4 Actions
action #114523: Deal with QEMU and OVMF default resolution being 1280x800, affecting (at least) qxl, but on aarch64 size:M Actions
action #114526: recover openqaworker14 Actions
action #114565: recover qa-power8-4+qa-power8-5 size:M Actions
action #114586: fix openqaworker-arm-1+2+3 recovery pipeline (was: likely stuck in reboot loop) size:M Actions
action #114685: powerqaworker-qam-1 seems to have just gone unresponsive due to unknown reason Actions
action #114697: What are orion and andromeda.o.o Actions
action #114733: openqaworker-arm-3 not consistently reachable Actions
action #114802: Handle "QA network infrastructure Package loss alert" introduced by #113746 size:M Actions
action #114817: salt-states-openqa deployment failed due to SUSE ca issue on openqaworker-arm-2 Actions
action #114908: [tools] https://stats.openqa-monitor.qa.suse.de not responding Actions
action #114914: Container devel/openqa/containers/isotovideo:qemu-x86 is outdated Actions
action #114923: We lost multi-machine capabilities within o3 due to openqaworker1 being replaced Actions
action #114941: repository names inconsistent on o3 workers hence continuous update failed to update as "devel_openQA" as repo name is not found everywhere Actions
action #115094: [tools] test fails in bootloader_start: redcurrant is down size:M Actions
action #115208: failed-systemd-services: logrotate-openqa alerting on and off size:M Actions
action #115226: Use ext4 (instead of ext2) for /var/lib/openqa on qa-power8 workers Actions
action #115418: Setup ow19+20 to be able to run MM tests size:M Actions
action #115484: [alert] OSD deployment failed on 18.08.22 size:M Actions
action #115547: openqaworker20 fails to boot, broken hardware size:M Actions
action #115553: salt-states-openqa pipeline failed: Authentication failed for 'https://gitlab.suse.de/openqa/salt-pillars-openqa/' Actions
action #115580: Reason: abandoned: associated worker openqaworker3:13 re-connected but abandoned the job size:M Actions
action #115733: bot-ng pipeline fails because of empty dictionary in data returned by smelt size:M Actions
action #116060: Recover openqaworker-arm-1 size:M Actions
action #116078: Recover o3 worker kerosene formerly known as power8, restore IPMI access size:M Actions
action #116113: salt responses timing out some of the time size:M Actions
action #116344: openqaw9-hyperv.qa.suse.de (flexo.qa.suse.cz) can not be reached size:M Actions
action #116377: openQABot pipeline failing with KeyError: 'project' Actions
action #116437: Recover qa-power8-5 size:M Actions
action #116494: Too many Minion job failures alert because needle-pusher is blocked on GitLab Actions
action #116563: `salt storage\* test.ping` times out Actions
action #116566: salt-states-openqa: Failed pipeline for master Actions
action #116689: Do not rely on statically configured IPv4 addresses for the salt master in /etc/hosts size:S Actions
action #116722: openqa.suse.de is not reachable 2022-09-18, no ping response, postgreSQL OOM and kernel panics size:M Actions
action #116740: [alert] openqaworker14: host up alert Actions
action #116743: [alert] QA-Power8-5-kvm: host up alert Actions
action #116746: [alert] openqaworker9: host up alert Actions
action #116752: [alert] powerqaworker-qam-1: host up alert Actions
action #116758: Help with adding monitoring for the SLE maintenance update queue size:M Actions
action #116782: o3 s390 workers are offline Actions
action #116794: Bring back grenache.qa.suse.de + grenache-1.qa.suse.de Actions
action #116845: salt-states-openqa CI complains about "fatal: Authentication failed for 'https://gitlab.suse.de/openqa/salt-pillars-openqa/'" but no useful hint size:M Actions
action #116848: Ensure kdump is enabled and working on all OSD machines Actions
action #116911: [openQA][needle] Can not commit new needle for test suite on openqa.suse.de Actions
action #117172: Flaky alert about infrastructure packet loss Actions
action #117205: Some boot_from_pxe failed from assigned worker: grenache-1:17 (kermit) and also openqaworker2:17 (quinn) size:M Actions
action #117229: [tools] openqa failing on worker QA-Power8-5-kvm Actions
action #117262: [alert] failed systemd service: ca-certificates on openqa.suse.de, "p11-kit: couldn't complete writing of file: /var/lib/ca-certificates/ca-bundle.pem.tmp: Unknown error 17" size:M Actions
action #117526: [alert] Dehydrated fails again on OSD Actions
action #117580: web interface of qanet02nue.qa.suse.de and qanet03nue.qa.suse.de can not be reached over neither IPv4 nor IPv6 size:M Actions
action #117625: Fix IPMI connection to openqaworker1.suse.de size:M Actions
action #117631: Failed systemd service transactional-update on openqaworker1 - system is no longer reachable after reboot size:M Actions
action #118024: Ensure all PPC workers are upgraded after kernel regression resolved size:M Actions
action #118885: No cable connection or link to the 2nd network card in amd-zen3-gpu-sut1-1 Actions
action #119179: [openQA][iso] Fail to mount nfs share openqa.suse.de:/var/lib/openqa/share/factory/iso/fixed Actions
action #119215: [openQA][repos][aarch64][15-SP5[Full Media] 15-SP5 Full media is missing from http://openqa.suse.de/assets/repo/ Actions
action #119290: [alert] Packet loss between worker hosts and other hosts alert Actions
action #119488: The QEM dashboard is empty Actions
action #119521: [alert] jenkins: partitions usage (%) alert Actions
action #119557: [qem-dashboard] Show a link to Smelt if there are no active incidents size:S Actions
action #119767: Failed pipeline for "openqa-worker" in salt-states-openqa size:M Actions
action #119938: openQABot | Failed pipeline for master | 9b1b1857: ERROR: Connection error during reading from IBS: HTTP Error 403: Forbidden size:M Actions
action #120004: [alert] Host powerqaworker-qam-1.qa.suse.de is down size:M Actions
action #120007: [alert] Many systemd alerts triggered on 06.11.22 size:S Actions
action #120025: [openQA][ipmi][worker] Worker host hostname changed and broken networking connection Actions
action #120073: [qem-dashboard] Loading incident... indefinitely for non-existing incident Actions
action #120112: worker worker2.oqa.suse.de auto_review:"Error connecting to <root@win2k19.qa.suse.cz>: Connection timed out":retry size:M Actions
action #120163: Use salt grains instead of manually specifying IPs in "bridge_ip" size:M Actions
action #120261: tests should try to access worker by WORKER_HOSTNAME FQDN but sometimes get 'worker2' or something auto_review:".*curl.*worker\d+:.*failed at.*":retry size:meow Actions
action #120270: Conduct the migration of SUSE openQA systems IPMI from Nbg SRV1 to new security zones size:M Actions
action #120339: QEMU DNS fails to resolve openqa.suse.de via IP address Actions
action #120372: OSD deployment fails presumably when handling the changelog Actions
action #120441: OSD parallel jobs failed with "get_job_autoinst_url: No worker info for job xxx available" size:meow Actions
action #120564: [qem-dashboard] Assets are not cleaned up on deployment Actions
action #120615: [alert] "Packet loss between worker hosts and other hosts" i.e. between osd and worker12.oqa.suse.de and worker5.oqa.suse.de Actions
action #120648: [alert] qa-power8-4-kvm host up alert Actions
action #120675: openqaworker-arm-1 not bootable via IPMI Actions
action #120744: [alert] QA-Power8-5-kvm: Too many Minion job failures alert size:M Actions
action #120780: [Alerting] InfluxDB not reachable, turned ok after some minutes Actions
action #120783: [Alerting] failed systemd service on worker11, os-autoinst-openvswitch. Failed at system boot, turned ok after some hours size:M Actions
action #120807: [alert] openqa.suse.de - worker12.oqa.suse.de 100% packet loss due to outdated AAAA record Actions
action #120813: [alert] Salt states fail to apply on `scriptgen.py` invocation Actions
action #120874: openqa_service VM is down size:M Actions
action #120886: "PXE-E32: TFTP open timeout" on ipmi machines in O3 size:S Actions
action #120921: [alert] Salt states fail to compile with "Rendering SLS 'base:openqa.openvswitch' failed: Jinja error: argument of type 'NoneType' is not iterable" size:M Actions
action #120939: [alert] Pipeline for scheduling incidents runs into timeout size:M Actions
action #120967: [alert] o3 webUI down due to regression from https://github.com/os-autoinst/openQA/pull/4922 Actions
action #120973: [qem-dashboard] 500 internal server errors reported by qem-bot Actions
action #121282: Recover storage.qa.suse.de size:S Actions
action #121594: Extend OSD storage space for "results" to make bug investigation and failure archeology easier - 2022 Actions
action #121771: openqaworker20 has no heartbeat Actions
action #121789: MultiMachine tests lose ability to communicate Actions
action #121816: Cannot access installation media on updates.suse.com - maintenance tests broken size:S Actions
action #122158: [alert] qa-power8-4-kvm host up alert - machine not up, nothing obvious on SoL but IPMI works size:M Actions
action #122302: Support SD-105827 "PowerPC often fails to boot from network with 'error: time out opening'" Actions
action #122575: [alert] jenkins host up alert Actions
action #122653: Ask SUSE-IT network admins to REJECT packets instead of DROP so that we get more clear results size:S Actions
action #122656: Ask SUSE-IT network admins to *not* block this traffic which we need for tests regarding s390x within SUSE network size:M Actions
action #122743: Rotate secrets used in CircleCI as per recommendation Actions
action #122746: Publishing opensuse.openqa.job.done failed: SSL connect attempt failed Actions
action #122842: Configure I/O alerts again for the webui after migrating to the "unified alerting" in grafana size:M Actions
action #122845: Migrate our Grafana setup to "unified alerting" Actions
action #122848: Configure grouped alerts in Grafana correctly size:M Actions
action #122983: [alert] openqa/monitor-o3 failing because openqaworker1 is down size:M Actions
action #122998: o3 worker rebel is down; was: inconsistent package database or filesystem corruption size:M Actions
action #123004: Downgrade kernel on o3+osd x86_64 machines as workaround for boo#1206616 size:M Actions
action #123025: o3 worker openqaworker4 is down; boots to emergency shell only Actions
action #123028: A/C broken in TAM lab size:M Actions
action #123082: backup of o3 to storage.qa.suse.de was not conducted by rsnapshot since 2021-12 size:M Actions
action #123151: [Alerting] Failed systemd services alert Actions
action #123232: [Alerting] failed pipelines for openQABot/ bot-ng/ os-autoinst-needles-opensuse-mirror on gitlab staging instance size:S Actions
action #123382: repurpose openqaworker-arm-3 as baremetal worker Actions
action #123421: [alert] dehydrated is failing on monitor.qa.suse.de Actions
action #123490: o3 logreports DBI Exception: DBD::Pg::st execute failed: ERROR: invalid input syntax for type bigint size:M Actions
action #123493: [Alerting] Failed systemd services alert Actions
action #123825: Ensure proper o3 monitoring after shutdown of thruk/icinga by SUSE-IT Eng-Infra Actions
action #123933: [worker][ipmi][bmc] Some worker can not be reached via BMC Actions
action #123984: [boot][pxe][sut] Machine fozzie can not boot from pxe Actions
action #124119: Conduct the migration of remaining SUSE openQA systems IPMI to new security zones Actions
action #124146: [alert] Incomplete jobs (not restarted) of last 24h Actions
action #124391: test fails in bootloader_start - The command server of powerhmc1 cannot be connected size:M Actions
action #124398: salt gitlab CI jobs fail due to exceeding log length of 500kB size:M Actions
action #124412: [alert] logrotate services failed on openqa-piworker.qa.suse.de and OSD size:M Actions
action #124655: [openQA][infra][pxe] Physical SUT machine can not boot from pxe and mismatch hostname Actions
action #124658: [osd-deployment] missing os-autoinst changes in deployment and changelog emails , build failure on ppc64le in OBS size:M Actions
action #124661: [qe-tools] tftp server and directory mount issue on qanet.qa Actions
action #124685: [qe-tools] Make sure Power8 and Power9 machines can be used with *any* usable HMC (was: move to new powerhmc3.arch.suse.de) size:M Actions
action #124715: Failing pipelines because of unreachable machine openqaworker-arm-1 Actions
action #124877: Failing pipelines because of unreachable machine openqaworker-arm-1 Actions
action #125132: [alert] logrotate failed on OSD Actions
action #125207: worker11 host up alert - similar as for worker13 Actions
action #125210: worker13 host up alert - kernel crash size:M Actions
action #125213: Failed systemd services alert due do crash dumps on worker11 and worker13 (except openqa.suse.de) Actions
action #125216: Use qa-power8 for ppc tests in o3 - try one of the suggestions size:M Actions
action #125303: prevent confusing "no data" alerts size:M Actions
action #125306: rebootmgr/D-Bus-related problem on backup.qa.suse.de Actions
action #125468: [alert] [FIRING:1] (Apache Response Time alert J5M8aX04z) then resolved itself so flaky? size:M Actions
action #125531: salt-pillar C pipeline runs into 1h timeout Actions
action #125534: Consolidate the installation of openqaw5-xen with SUSE QE Tools maintained machines size:M Actions
action #125642: Manage "unified alerting" via salt size:M Actions
action #125735: [openQA][infra][pxe] Some machines can not boot from pxe due to "TFTP open timeout" Actions
action #125750: In salt-states-openqa support machines requiring ssh password login for root user size:M Actions
action #125765: Make Telegraf errors visible in alert handling Actions
action #125798: Visual differences in GRUB menu on different x86_64 UEFI workers Actions
action #125810: [openqa][infra] Some SUT machines can not upload logs to worker machine size:S Actions
action #125885: worker10 crashed triggering systemd-services alert and host-up alert size:M Actions
action #126212: openqa.suse.de response times very slow. No alert fired size:M Actions
action #126290: Recover thincsus.qe.nue2.suse.org size:M Actions
action #126674: openqa-review pipeline failed to produce a report with no test results Actions
action #126821: [openQA][infra][worker] ppc64 fails to load grub2 completely over tftp/pxe on qanet and PXE load timeout issues size:M Actions
action #126872: bot-ng pipeline(s) fail(s) to pull openSUSE container images Actions
action #126962: Use templating for all provisioned "unified alerts" were the original alerts were part of templated dashboards Actions
action #127049: gitlab runner system failure: Cannot connect to the Docker daemon Actions
action #127052: [alert] Apache Response Time alert followed by DatasourceNoData for min_apache_response size:M Actions
action #127055: [alert] Download rate alert openQA (openqaworker{14,16,17,18}) Actions
action #127097: [alert] Failed systemd services alert Actions
action #127256: missing nameservers in dhcp response for baremetal machines in NUE-FC-B 2 size:M Actions
action #127274: [alert] Usage of partition mmcblk0p3 on openqa-piworker exceeds threshold size:M Actions
action #127337: Some s390x workers have been failing for all jobs since 11 months ago Actions
action #127754: osd nfs-server needed to be restarted but we got no alerts size:M Actions
action #127877: [alert] Mail "DatasourceNoData Salt" triggered after rebooting OSD-VM Actions
action #127961: FC office access for mkittler Actions
action #127982: [alert][grafana] DatasourceNoData e.g. openQA QA-Power8-5-kvm+openqaw5-xen Disk I/O time alert worker Actions
action #127985: [alert][grafana] Handle various problems with slow monitoring https requests and such (was: DatasourceNoData HTTP Response alert) size:M Actions
action #127991: Fix DNS entry or IPv6 connectivity of qe-jumpy.suse.de Actions
action #128030: IPMI of ix64ph1075.qa.suse.de is not accessible size:M Actions
action #128090: [alert] SUSE nbg network outage 2023-04-20 Actions
action #128273: [alert] openqaworker-arm-1+2+ failed to recover, problem in name resolution, network connection? size:M Actions
action #128417: [alert][grafana] openqaw5-xen: partitions usage (%) alert fired and quickly after recovered again size:M Actions
action #128420: [alert][grafana] 100% packet loss from qa-power8-4-kvm, grenache-1 and powerqaworker-qam-1 to s390zp{11,15,17}.suse.de size:M Actions
action #128561: salt managed host being down does not trigger any alert (was: jenkins.qa.suse.de stuck in emergency mode but no alert) size:M Actions
action #128654: [sporadic] Fail to create an ipmi session to worker grenache-1:16 (ix64ph1075) in its vlan Actions
action #128669: gitlab CI jobs fail when using docker executor with "ERROR: Failed to remove network for build" Actions
action #128786: worker instances on rebel (o3 s390x worker) were not running, services disabled, except for rebel:5 Actions
action #128789: [alert] Apache Response Time alert size:M Actions
action #128927: OSD deployment changelog repeatedly mentions old os-autoinst changes, is a worker outdated and unable to install newer packages? size:M Actions
action #128942: [alert][grafana][openqa-piworker] NTP offset alert Generic (openqa-piworker ntp_offset_alert_openqa-piworker generic) size:M Actions
action #128945: [alert][grafana] web UI: Too many Minion job failures alert Salt (liA25iB4k) Actions
action #128969: [alert][grafana] Failed systemd services alert (except openqa.suse.de) Salt (Uk02cifVkz) Actions
action #128999: openQA workers salt recipes should ensure that also developer mode works size:M Actions
action #129065: [alert] HTTP Response alert fired, OSD loads slow size:M Actions
action #129127: Trying to login on o3 fails with Forbidden Actions
action #129241: [alert] Many alert notification for "QA network infrastructure Ping time alert openQA" received on 12.05.23 13:10 CEST Actions
action #129244: [alert][grafana] File systems alert for WebUI /results size:M Actions
action #129484: high response times on osd - Move OSD workers to o3 to prevent OSD overload size:M Actions
action #129493: high response times on osd - better nice level for velociraptor Actions
action #129604: evil backtick character in ipmi password Actions
action #130132: jenkins.qa.suse.de seems down Actions
action #130201: openqa_bootstrap: fetchneedles not called during openqa-bootstrap size:S Actions
action #130210: [FIRING:1] Packet loss between worker hosts and other hosts alert Salt (2Z025iB4km) Actions
action #130375: Automatically update jenkins plugins on jenkins.qa.suse.de Actions
action #130633: Better documentation on jenkins.qa.suse.de alerts and recovery Actions
action #130790: [alert] failed systemd alert openqa-staging-2 velociraptor-client size:M Actions
action #130835: salt high state fails after recent merge requests in salt pillars size:M Actions
action #130952: [alert][thursday] gitlab.suse.de CI jobs fail with "error: RPC failed; HTTP 500 curl 22 The requested URL returned error: 500" as of 2023-06-15 size:M Actions
action #131021: [O3 repo]Missing openSUSE-Tumbleweed-oss-x86_64-CURRENT directory in /var/lib/openqa/share/factory/repo size:M Actions
action #131096: [alert] Service `ca-certificates` can fail size:M Actions
action #131123: qa-jump.qe.nue2.suse.org is not reachable since 2023-06-19 Actions
action #131141: [alert] Packet loss between qa-jump.qe.nue2.suse.org and other hosts Actions
action #131147: Reduce /assets usage on o3 Actions
action #131150: Add alarms for partition usage on o3 size:M Actions
action #131201: [alert][grafana] HTTP Response alert, alerted at 10:46L, back to green 10:56L size:M Actions
action #131249: [alert][ci][deployment] OSD deployment failed, grenache-1, worker5, worker2 salt-minion does not return, error message "No response" size:M Actions
action #131264: OSD deployment fails to install package os-autoinst-distri-opensuse size:M Actions
action #131276: SUSE Summer 2023 - AC failure in NUE1-SRV1 Actions
action #131303: [alert] Packet loss between worker hosts and other hosts (tumblesle.qa.suse.de) Actions
action #131309: [alert] NFS mount can fail due to hostname resolution error size:M Actions
action #131318: User management via salt does not work on backup.qa.suse.de Actions
action #131321: salt-states-openqa | Failed pipeline for master Actions
action #131459: [openQA][infra] OSD ran out of inodes without triggering a notification size:M Actions
action #131540: openqa-piworker fails to upgrade many packages. vendor change is not enabled as our salt states so far only do that for openQA machines, not generic machines size:M Actions
action #131543: We have machines with both auto-update&auto-upgrade deployed, we should have only one at a time size:M Actions
action #132134: Setup new PRG2 multi-machine openQA worker for o3 size:M Actions
action #132137: Setup new PRG2 openQA worker for osd size:M Actions
action #132143: Migration of o3 VM to PRG2 - 2023-07-19 size:M Actions
action #132218: Conduct lessons learned for "openQA is not accessible" on 2023-07-02 Actions
action #132278: Basic o3 http response alert on zabbix size:M Actions
action #132311: osd-deployment failed at least on qa-power8-5 due to home:tiwai:kernel:5.6 missing Actions
action #132383: FC Basement OSD hosts not reachable since 2023-07-06 01:50 CEST Actions
action #132437: Ensure everybody in SUSE QE Tools knows how to silence alerts in various monitoring systems size:M Actions
action #132461: manage tls certificates on o3/ariel directly with dehydrated size:M Actions
action #132470: salt states fail to apply due to glibc error on storage.oqa.suse.de Actions
action #132500: NUE1-SRV2, .qa.suse.de, aarch64 workers offline due to heat-related SRV2 shutdown size:M Actions
action #132512: Handover manual raspberry pi testing size:S Actions
action #132647: Migration of o3 VM to PRG2 - bare-metal tests size:M Actions
action #132671: Ensure everybody in SUSE QE Tools knows how to access netbox size:M Actions
action #132752: Use proper bot account for notifications in zabbix.suse.de size:M Actions
action #132773: fail to set iPXE boot for a baremetal machine in FC basement Actions
action #132788: [alert][flaky] QA-Power8-5-kvm: QA network infrastructure Ping time alert Actions
action #132812: [alert] openqaw5-xen host up alert + infrastructure ping size:M Actions
action #132815: [alert][flaky][o3] Multiple flaky zabbix alerts related to o3 Actions
action #132818: salt state for worker in CI test does not apply anymore, "ID: /var/lib/openqa, Function: mount.mounted" size:M Actions
action #132827: [tools][qe-core]test fails in rsync_client/salt-master, DNS resolve issue with workers "sapworker*" on multi-machine tests size:M Actions
action #132860: openqa-piworker is unstable and needs regular power-cycles size:M Actions
action #132893: [alert] failed systemd services on jenkins: jenkins-plugins-update, snapper-cleanup Actions
action #132902: Check and document PDU connection of nibali.qe.nue2.suse.org Actions
action #132947: Bring back ada.qe.suse.de and fix it properly Actions
action #133097: cron on OSD (date; fetch_openqa_bugs /etc/openqa/bugfetcher_o3.conf) > /tmp/fetch_openqa_bugs_o3.log failed Actions
action #133127: Frankencampus network broken + GitlabCi failed --> uploading artefacts Actions
action #133130: Lots of alerts for a single cause. Can we group and de-duplicate? Actions
action #133142: 4 baremetal SUTs in FC basement are unreachable Actions
action #133154: osd-deployment failed because unreachable workers Actions
action #133160: Setup a modern UEFI httpboot setup on o3 with dnsmasq size:M Actions
action #133181: Migration of o3 VM to PRG2 - Fix https://openqa.opensuse.org/snapshot-changes/opensuse/Tumbleweed/ Actions
action #133250: gitlab.suse.de unusable "Failed to write to log, write /srv/www/vhosts/gitlab-ce/log/gitlab-shell.log: no space left on device" on remote operations Actions
action #133322: qa-jump.qe.nue2.suse.org is not reachable - take 3 Actions
action #133325: osd http response alerts - bump threshold further up Actions
action #133328: Notification settings API | GitLab: I wonder if we can loop over all and disable/enable size:M Actions
action #133358: Migration of o3 VM to PRG2 - Ensure IPv6 is fully working Actions
action #133364: Migration of o3 VM to PRG2 - Decommission old-ariel in NUE1 as soon as we do not need it anymore Actions
action #133367: Evaluate if we have hardware alternatives for Windows Server 2016+ testing Actions
action #133385: Problem: Interface tun5: Link down alerting and autoresolving shortly size:S Actions
action #133397: HTTP Response alert Salt alerting and autoresolving shortly size:M Actions
action #133403: Login on o3 does not work Actions
action #133469: [alert] Salt states don't apply sometimes on individual workers size:M Actions
action #133475: Migration of o3 VM to PRG2 - connection to rabbit.opensuse.org Actions
action #133490: Migration of o3 VM to PRG2 - Fix o3 bare metal hosts iPXE booting size:M Actions
action #133511: [spike solution][timeboxed:10h] Prevent memory over-commits in openQA worker service definitions size:S Actions
action #133553: OSD deployment fails due to unreachable openqaworker1 Actions
action #133580: Be able to bind openQA tests to x86_64 architecture sub-versions Actions
action #133793: salt-pillars-openqa failing to apply within 2h and it is not clear which minion(s) are missing size:M Actions
action #133892: [alert] arm-worker2 (arm-worker2: host up alert openQA host_up_alert_arm-worker2 worker size:M Actions
action #133928: salt-states-openqa | Failed pipeline for master Actions
action #133985: [alert] Backup VM not reachable via FQDN size:M Actions
action #134018: [alert] Multiple alerts with "used > 80%" size:S Actions
action #134042: auto-update on OSD does not install updates due to "Problem: nothing provides 'libwebkit2gtk3 ..." but service does not fail and we do not get an alert size:M Actions
action #134051: Eng-Infra maintained DNS server for .qa.suse.de taking over from qanet size:M Actions
action #134087: Fix ix64ph1075 bare metal openQA test size:M Actions
action #134123: Setup new PRG2 openQA worker for o3 - two new arm workers size:M Actions
action #134126: Setup new PRG2 openQA worker for o3 - bare-metal testing size:M Actions
action #134132: Bare-metal control openQA worker in NUE2 size:M Actions
action #134243: fozzie not responsive via ipmi Actions
action #134282: [tools] network protocols failures on multimachine tests on HA/SAP size:S auto_review:"no candidate.*iscsi-target-overview-service-tab|yast2.+firewall.+services.+add.+zone":retry Actions
action #134453: backup.qam.suse.de is Failed according to netbox and not creating backups size:M Actions
action #134489: backup.qa.suse.de does not create backups Actions
action #134519: We were not notified that backup.qa.suse.de did not create backups size:M Actions
action #134522: [alert] Certificate renewal on monitor.qa.suse.de might not be working causing alerts size:M Actions
action #134723: Conduct lessons learned for making new-lines in script_run fatal size:S Actions
action #134735: [alert] openQA piworker openqa-piworker: host up alert Actions
action #134816: [tools] grafana dashboard for `OpenQA Jobs test` partially without any data from OSD migration size:M Actions
action #134822: Migration of osd VM to PRG2 - Decommission old-osd in NUE1 as soon as we do not need it anymore size:M Actions
action #134861: https://stats.openqa-monitor.qa.suse.de/ reports "502 Bad Gateway", https://monitor.qa.suse.de works fine Actions
action #134879: reverse DNS resolution PTR for openqa.oqa.prg2.suse.org. yields "3(NXDOMAIN)" for PRG1 workers (NUE1+PRG2 are fine) size:M Actions
action #134900: salt states fail to apply due to "Pillar openqa.oqa.prg2.suse.org.key does not exist" Actions
action #134906: osd-deployment failed due to openqaworker1 showing "No response" in salt size:M Actions
action #134912: Gradually phase out NUE1 based openQA workers size:M Actions
action #134927: OSD throws 503, unresponsive for some minutes size:M Actions
action #134939: [OSD][Worker] openqaworker1:7 is unworkable Actions
action #134948: Ensure IPv6 is working in the OSD setup (since we have workers in PRG2 and the VM has been migrated) size:M Actions
action #135026: Oli shows up with 50% resolved tickets despite having been on hols for two weeks Actions
action #135029: Many unhandled alert messages while users report problems Actions
action #135032: Communication about vacation within the team could be better Actions
action #135047: retro feedback: This week was very exhausting - too much infra work Actions
action #135137: Bring back imagetester size:M Actions
action #135152: Zabbix agent is not available Actions
action #135185: Zabbix reports high swap space usage > 50% Actions
action #135191: Migration of o3 VM to PRG2 - Use direct zabbix connection size:M Actions
action #135206: [tools] GitlabCI telegraf step on salt-states-openqa failed Actions
action #135230: salt pillars pipelines failing due to Temporary failure in name resolution Actions
action #135236: https://monitor.qa.suse.de/d/WebuiDb/webui-summary?orgId=1&viewPanel=78 shows no data Actions
action #135260: zabbix - o3 High CPU utilization (over 90% for 5m) size:M Actions
action #135329: s390x work demand exceeds available workers Actions
action #135335: [tools] gitlabci salt-pillars-openqa deploy failed on imagetester and other hosts size:M Actions
action #135380: A significant number of scheduled jobs with one or two running triggers an alert Actions
action #135404: openqaworker-arm-2.suse.de minion not returning Actions
action #135467: aarch64.openqanet.opensuse.org 100 packet loss Actions
action #135470: Grafana: Average Ping time (ms) alert with unexpanded variable "${tag_url}", which machine is this about? size:M Actions
action #135491: fozzie and quinn unable to access PXE server or iPXE server (TFTP open timeout) Actions
action #135509: monitor.qa.suse.de yields 502 Bad Gateway from nginx/1.21.5 size:M Actions
action #135515: malbec.arch.suse.de not reachable anymore Actions
action #135578: Long job age and jobs not executed for long size:M Actions
action #135632: "Mojo::File::spurt is deprecated in favor of Mojo::File::spew" breaking os-autoinst OBS build and osd-deployment size:M Actions
action #135737: [alert] Munin - network eth errors - opensuse.org :: openqa.opensuse.org size:M Actions
action #135740: [alert] Munin - minion hook failed - opensuse.org :: openqa.opensuse.org - only "label_known_issues" hook scriptssize:M Actions
action #135773: [tools] many multi-machine test failures in "ovs-client+ovs-server" test scenario when tests are run across different workers size:M Actions
action #135833: false-positive inode and disk usage alert on windows image Actions
action #135848: Icinga alarm about Postfix mail queue since July 26 Actions
action #135890: [alert][FIRING:1] (Apache Workers alert Salt MW025mB4z) 2023-09-18 00:11 Actions
action #135893: FQDN (long host name) is unavailable for fozzie Actions
action #135941: openqa-piworker offline and no alert Actions
action #136007: Conduct "lessons learned" with Five Why analysis for network protocols failures on multimachine tests on HA/SAP size:S Actions
action #136052: bot-ng in gitlab CI runner + monitor.qe.nue2.suse.org unable to access smelt.suse.de, "No route to host" Actions
action #136115: https://racktables.nue.suse.com/ yields "Database write failed - LDAP caching error" on connection attempts after about 1m waiting time Actions
action #136121: Repurpose PowerPC hardware in FC Basement - deagol Actions
action #136133: Migrate aarch64.openqanet.opensuse.org to FC Basement size:M Actions
action #136238: test incompletes with auto_review:"qemu-system-.*Address already in use":retry size:M Actions
action #136274: Failing DNS resolution on o3 for hosts like github.com Actions
action #136304: [tools] please provide a machine for testing sle-micro 5.5 (virtualization testing) Actions
action #136325: salt deploy fails due to multiple offline workers in qe.nue2.suse.org+prg2.suse.org Actions
action #136370: systemd service rsnapshot@beta on backup-vm.qe.nue2.suse.org failed due to process conflict Actions
action #136946: https://monitor.qa.suse.de/d/WebuiDb/webui-summary shows multiple panels with "no data" size:M Actions
action #137075: Fail to login to the osd, 'Forbidden' error is returned due to DNS server change within SUSE *and* auto_review:"Bugzilla query failed: Network is unreachable":retry size:M Actions
action #137231: [tools] qem-bot and others can not execute scheduled jobs due to registry.opensuse.org outage Actions
action #137270: [FIRING:1] host_up (malbec: host up alert openQA malbec host_up_alert_malbec worker) and similar about malbec Actions
action #137306: Check unreal6 cabling, SP and system not reachable over network size:M Actions
action #137408: Support move of s390x mainframe(s) to PRG2 - o3 size:M Actions
action #137504: Lots of emails of the form Switched Rack PDU: Outlet .* (openqaworker-arm-.*) .* Actions
action #137519: [alert] Failed systemd services - openqaworker1 - proc-sys-fs-binfmt_misc.mount, kernel modules already removed with old kernel still running size:M Actions
action #137522: [alert] alerts about "host: sushil-linux-tw-kde" that tools team should not be notified about, e.g. Inode utilization inside the OSD infrastructure is too high size:M Actions
action #137600: [alert] Packet loss between worker hosts and other hosts size:S Actions
action #137603: [alert] Queue: State (SUSE) - too few jobs executed alert size:S Actions
action #137615: [alert] Failed systemd services alert - s390zl12,s390zl13 - kdump-early, kdump, smartd Actions
action #137744: [alert] gitlab pipeline fails in deploy step for salt-pillars-openqa: Data failed to compile Actions
action #137747: NFS and download.opensuse.org related outage handling Actions
action #137756: Re-enable worker31 for multi-machine tests in production auto_review:"tcpdump.+check.log.+timed out at" Actions
action #137771: Configure o3 ppc64le multi-machine worker size:M Actions
action #137813: [alert] Failed systemd services - qamaster - logrotate fails on /var/log/messages with "/usr/bin/xz: (stdin): Read error: Input/output error" size:S Actions
action #137984: salt "refresh" job full of errors but CI job passes size:M Actions
action #138005: grafana panel "Packet loss between worker hosts and other hosts" shows more than just ping to "other hosts" and hence becomes slow and triggers redundant alerts size:M Actions
action #138038: diesel+petrol missing network, IPMI still reachable Actions
action #138044: Grouped seemingly unrelated alert emails are confusing size:M Actions
action #138320: test fail hook in openqa_webui is failing: script/opencli api jobs - Can't locate Mojo::Base in @INC Actions
action #138350: worker31 and likely more OSD machines get stuck on boot in grub command line Actions
action #138377: bot-ng and qamops pipelines fail to pull containers from registry.suse.de size:M Actions
action #138515: foobar host up alert size:S Actions
action #138518: unreal6 partition usage alert Actions
action #138527: Zabbix agent on ariel.dmz-prg2.suse.org reported no data for 30m and there is nothing in the journal size:S Actions
action #138536: Alert Worker .* has no heartbeat (900 seconds), restarting (see FAQ for more) on o3 size:S Actions
action #138545: Munin - minion hook failed - opensuse.org :: openqa.opensuse.org size:S Actions
action #138551: DNS outage of 2023-10-25, e.g. Cron <root@openqa-service> (date; fetch_openqa_bugs)> /tmp/fetch_openqa_bugs_osd.log Max retries exceeded with url size:S Actions
action #138650: partition usage panels show a long list of undefined and no reasonable graphs at least for some generic machines size:M Actions
action #138746: [tools] s390x VM randomly fails to open QCOW disk image: Permission denied Actions
action #138773: Trying to login on o3 or osd fails with Forbidden - again Actions
action #139004: Cron <root@openqa-service> (date; fetch_openqa_bugs)> /tmp/fetch_openqa_bugs_osd.log - OSError: [Errno 101] Network is unreachable Actions
action #139100: Long OSD ppc64le job queue - Move nue3 power8 machines to nue2 Actions
action #139103: Long OSD ppc64le job queue - Decrease number of x86_64 worker slots on osd to give ppc64le jobs a better chance to be assigned jobs size:M Actions
action #139145: dehydrated on monitor.qe.nue2.suse.org aka. monitor.qa.suse.de fails with "EXPECTED value GOT EOF" size:M Actions
action #139148: [alert] openqa-review repeatedly failed to read job group overview data running into "Read timed out" Actions
action #139271: Repurpose PowerPC hardware in FC Basement - mania Power8 PowerPC size:M Actions
action #139307: openQA for s390x does not work at the moment Actions
action #150815: unable to login over ssh to o3 (gate.opensuse.org:2214) size:M Actions
action #150824: monitor-pre-deploy from osd-deployment is failing with Build status for devel:openQA openSUSE_Leap_15.5 arch x86_64 is not successful Actions
action #150830: Two new ARM servers 2023-11 for openqa.suse.de bare-metal testing size:M Actions
action #150845: openqaworker-arm22 broken due to packages automatically removed size:M Actions
action #150869: Ensure multi-machine tests work on aarch64-o3 (or another but single machine only) size:M Actions
action #150887: [alert] [FIRING:1] s390zl12 (s390zl12: partitions usage (%) alert Generic partitions_usage_alert_s390zl12 generic), also s390zl13 size:M Actions
action #150908: o3 "Unable to fetch build results" and "Internal server error" on some pages size:M Actions
action #150920: openqaworker-arm22 is unable to join download.opensuse.org in parallel tests = tap mode size:M Actions
action #150938: [openQA][sut][ipmi] No ipmi sol output with ix64ph1075 size:M Actions
action #150956: o3 cannot send e-mails via smtp relay size:M Actions
action #150965: At least diesel+petrol+mania fail to auto-update due to kernel locks preventing patches size:M Actions
action #150983: CPU Load and usage alert for openQA workers size:S Actions
action #151013: o3 yielding "502 Bad Gateway" from nginx 2023-11-19, why was the config overwritten? size:M Actions
action #151130: IPv6 for openqa.opensuse.org and open.qa size:S Actions
action #151165: salt-pillars-openqa pipeline fails Actions
action #151231: package loss between o3 machines and download.opensuse.org size:M Actions
action #151390: Brute-force salt osiris so that we enable self-management of VMs for users size:M Actions
action #151396: After osiris is now in salt decide about the fate of seth Actions
action #151588: [potential-regression] Our salt node up check in osd-deployment never fails size:M Actions
action #151597: [alert] osiris-1 (osiris-1: partitions usage (%) alert Generic partitions_usage_alert_osiris-1 generic Actions
action #151696: Evaluate use of https://itpe.io.suse.de/open-platform/docs/ size:M Actions
action #151807: [alert] o3 zabbix: Problem: /var/lib/snapshot-changes: Disk space is critically low (used > 94%) size:M Actions
action #152008: s390zl13 redirecting http to https because of .wget-hsts but we don't install ca-certificates-suse on all salt-controlled machines Actions
action #152092: Handle all package downgrades in OSD infrastructure properly in salt size:M Actions
action #152095: [spike solution][timeboxed:8h] Ping over GRE tunnels and TAP devices and openvswitch outside a VM with differing packet sizes size:S Actions
action #152101: Allow salt to properly configure non-production multi-machine workers size:M Actions
action #152377: [tools] SLE-15-SP2 and SLE-15-SP3 install target medias for x86_64 in the ...../assets/repo are purged Actions
action #152386: [alert] "Apache Workers" and "HTTP Response" alerts fired shortly on 2023-12-11 size:S Actions
action #152446: openqaworker-arm21 is broken and produces lots of incomplete jobs Actions
action #152503: [FIRING:1] worker38 (worker38: partitions usage (%) alert openQA partitions_usage_alert_worker38 worker) Actions
action #152557: unexpected routing between PRG1/NUE2+PRG2 Actions
action #152578: Many incompletes with "Error connecting to VNC server <unreal6.qe.nue2.suse.org:...>" size:M Actions
action #152599: [alert] `rsnapshot@alpha.service` failed on `backup.qa.suse.de` Actions
action #152643: https://monitor.qa.suse.de/d/nRDab3Jiz/openqa-jobs-test showing two entries for x86_64, one with quotes and one without Actions
action #152649: [alert] `rsnapshot@alpha.service` failed on `backup.qa.suse.de` size:M Actions
action #152673: [alert] `systemctl status iscsid.socket` failed on `s390zl12.oqa.prg2.suse.org` size:S Actions
action #152741: [tools] gitlab CI - openqa_review failed with connection timeout on osd Actions
action #152811: ada.qe.suse.de is not responding to salt commands Actions
action #152813: openqaw5-xen.qa.suse.de is not responding to salt commands Actions
action #152827: [tools] cron service updating clamav database failing on OSD + O3 size:S Actions
action #152857: [tools] alert ping between hosts timeout proxy.scc.suse.de Actions
action #152887: Setup of Ampere Altra Q32-17 for bare-metal tests in openQA size:M Actions
action #152941: circleCI job runs into 20m timeout due to slow download from registry.opensuse.org Actions
action #152981: monitoring: Update deprecated angular grafana panels Actions
action #153023: [FIRING:1] (Packet loss between worker hosts and other hosts alert Salt 2Z025iB4km) Actions
action #153325: osd-deployment | Failed pipeline, Digest verification failed for openQA-common size:M Actions
action #153328: jenkins fails in submit-openQA-TW-to-oS_Fctry, Server returned an error: HTTP Error 400: Bad Request size:M Actions
action #153418: http based health check against proxy.scc.suse.de size:M Actions
action #153544: os-autoinst:os-autoinst-openvswitch-test not building on Leap_15.5 x86_64 Actions
action #153925: Support YAM squad to get backlogger running in our salt states (and fix our pipelines again) Actions
action #153958: [alert] s390zl12: Memory usage alert Generic memory_usage_alert_s390zl12 generic Actions
action #153961: Grafana doesn't show values for inode utilization for s390zl12 Actions
action #154018: [alert] Failed systemd services alert: backup-vm postfix Actions
action #154177: File systems alert Salt: One of the file systems is too full size:M Actions
action #154345: Incomplete jobs (not restarted) of last 24h alert Salt Actions
action #154426: HTTP Response alert Salt alerting and autoresolving shortly size:M Actions
action #154546: Cron fetch_openqa_bugs refused or timed out trying to fetch individual tickets Actions
action #154549: Certain queries on poo are not accessible Actions
action #154624: Periodically running simple ping-check multi-machine tests on x86_64 covering multiple physical hosts on OSD alerting tools team on failures size:M Actions
action #154627: [potential-regression] Ensure that our "host up" alert alerts on not host-up conditions size:M Actions
action #154927: [alert] Broken workers alert was firing several hours after weekly reboot Actions
action #154939: Clarify use of quake2 and ensure it does not waste power size:S Actions
action #154942: Clarify use of quake6 and ensure it does not waste power size:S Actions
action #155074: salt-states-pipeline fails trying to install influxdb Actions
action #155080: jenkins is no longer producing GNOME:Next test runs: http://jenkins.qa.suse.de/job/gnome_next-openqa/8670/console Actions
action #155326: [alert] "HTTP Response" alert fired shortly on 2024-02-12 and 2024-03-04 size:M Actions
action #155659: [openQA][infra][sut] Failed to establish connnection to ix64ph1075-sp.qe.nue2.suse.org Actions
action #155689: bot-ng pipelines fails to schedule incidents Actions
action #155725: [openQA][infra][sut] Failed to establish connnection to fozzie-sp and quinn-sp Actions
action #155737: Salt pillars pipelines fail due to refused connection errors on telegraf Actions
action #155740: Scripts CI pipelines fail due to timeout after many Job state of job ID xyz: scheduled, waiting messages Actions
action #155824: Support IPv6 SLAAC in our infrastructure size:M Actions
action #155848: Firewalld is logging many errors and sometimes restarting on worker29, possibly related to MM failures size:M Actions
action #155929: Try out rstp_enable=True in openqa/openvswitch.sls size:M Actions
action #156016: [openQA][sle-micro][virtualization] Test slem_virtualization@uefi with Default-encrypted image was not triggered correctly size:M Actions
action #156049: [alert] Scripts CI pipeline failing with No reason for devel:openQA:Leap:15.6/local-npm-registry Actions
action #156064: Handle planned datacenter shutdown PRG1 2024-02-28 size:M Actions
action #156076: bot-ng gitlab CI pipeline fails with "ERROR: Job failed (system failure): Error response from daemon: mkdir /var/lib/docker/overlay2/…/merged: no space left on device (exec.go:78:0s)" Actions
action #156130: Install Intel GPU on one of the servers in FC size:M Actions
action #156226: [bot-ng] Pipeline failed / failed to pulled image / no space left on device Actions
action #156301: [bot-ng] Pipeline failed / KeyError: 'priority' / execution took longer than 4h0m0s seconds Actions
action #156304: [salt-pillars] Pipeline failed / Passed invalid arguments to state.highstate Actions
action #156322: zabbix-proxy.dmz-prg2.suse.org not reachable from ariel.suse-dmz.opensuse.org Actions
action #156331: [gitlab] New pipeline schedules cannot be created, you have exceeded the maximum number of pipeline schedules for your plan Actions
action #156460: Potential FS corruption on osd due to 2 VMs accessing the same disk Actions
action #156481: cron -> (fetch_openqa_bugs)> /tmp/fetch_openqa_bugs_osd.log failed / No route to host / openqa.suse.de Actions
action #156514: Cron <root@openqa-service> (date; fetch_openqa_bugs)> /tmp/fetch_openqa_bugs_osd.log failed / simplejson.errors.JSONDecodeError: Expecting value: line 1 column 1 (char 0) / bugzilla_issue.py Actions
action #156517: Can't call method "script" on an undefined value at lib/OpenQA/WebAPI/Controller/Step.pm size:S Actions
action #156532: lessons learned about "Potential FS corruption on osd due to 2 VMs accessing the same disk" size:S Actions
action #156535: Handle unfinished SLE maintenance tests due to FS corruption on OSD 2024-03-01 Actions
action #156913: Remove its=off setting in global QEMUMACHINE for worker-arm{1,2} size:M Actions
action #156934: RPi realhw tests fail with # Test died: Error connecting to <root@10.168.192.112> Actions
action #157081: OSD unresponsive or significantly slow for some minutes 2024-03-12 08:30Z Actions
action #157243: Update HMC with vMF68994 Actions
action #157432: parted /dev/sda disk got error at powerVM worker size:M Actions
action #157438: Failed systemd services alert (jenkins-plugins-update, snapper-cleanup) Actions
action #157441: osd-deployment | Failed pipeline for master (qesapworker-prg5.qa.suse.cz) Actions
action #157453: [FIRING:1] host_up (qesapworker-prg5: host up alert openQA qesapworker-prg5 host_up_alert_qesapworker-prg5 worker) Actions
action #157468: Handle internal test machines with compromised root password size:M Actions
action #157528: Remove redundant ASM connections for powerPC machines size:S Actions
action #157615: [alert] osd-deployment failed in post-deploy , telegraf errors size:M Actions
action #157666: OSD unresponsive and then not starting any more jobs on 2024-03-21 Actions
action #157726: osd-deployment | Failed pipeline for master (worker3[6-9].oqa.prg2.suse.org) Actions
action #157750: Better secure the networks to have s390kvm… (and others) less accessible Actions
action #157834: [openQA][ipmi] IPMI backend machines in NUE2 can not be reached auto_review:"Reason: backend died: ipmitool.*Address lookup for.*qe.nue2.suse.org":retry Actions
action #157975: Upgrade osd workers to openSUSE Leap 15.6 size:S Actions
action #158020: salt-states-openqa pipeline times out Actions
action #158023: salt-states-openqa pipeline invalid arguments to state.highstate on monitor.qe.nue2.suse.org Actions
action #158026: osd-deployment exceeds 2h maximum runtime during package installation Actions
action #158041: grenache needs upgrade to 15.5 Actions
action #158059: OSD unresponsive or significantly slow for some minutes 2024-03-26 13:34Z Actions
action #158104: typing issue on ppc64 worker size:S Actions
action #158113: typing issue on ppc64 worker - make CPU load alert more strict size:M Actions
action #158170: Increase resources for s390x kvm size:M Actions
action #158185: parallel job failed to get the vars from its pair size:S Actions
action #158242: Prevent ssh access to test VMs on svirt hypervisor hosts with firewall size:M Actions
action #158266: openQA jobs on diesel ppc64le fail due to auto_review:"QEMU: This is probably because your SMT is enabled." Actions
action #158383: Crosscheck which machines marked as "unused" in racktables are still pingable (as they should not be powered on at all) size:M Actions
action #158404: [qe-core][jeos]test fails in prepare_firstboot Actions
action #158419: osiris-1.qe.nue2.suse.org not responsive over virt-manager and "virsh list" hangs Actions
action #158502: [FIRING:1] host_up (petrol: host up alert openQA petrol host_up_alert_petrol worker) Actions
action #158505: Failed systemd services alert for jenkins-plugins-update size:S Actions
action #158526: Apply the latest firmware+BIOS upgrade for diesel as well size:S Actions
action #158556: Single-value SLI of OSD HTTP response code successful vs. all size:S Actions
action #158559: Single-value SLI of OSD HTTP response time size:S Actions
action #158907: Automated check for machines marked as "unused" in racktables but still pingable (as they should not be powered on at all) size:M Actions
action #158913: Fix salt warning observed on petrol "State for file: /etc/fstab - Neither 'source' nor 'contents' nor 'contents_pillar' nor 'contents_grains' was defined, yet 'replace' was set to 'True'" Actions
action #159048: Setup new Power10 machine for QE LSG in PRG2 (S/N 7882391) size:M Actions
action #159066: network-level firewall preventing direct ssh+vnc access to openQA test VMs size:M Actions
action #159069: network-level firewall preventing direct ssh+vnc access to all machines within the oqa.prg2.suse.org network if needed Actions
action #159165: Ensure that rsyncd is properly covered in salt states size:S Actions
action #159186: [alert] Systemd-services alert failing due to unit "rsnapshot@alpha" on host "storage" Actions
action #159264: Discard Ampere ARM server packaging if we still have it + reserve thunderx21 machine in FC Basement for afaerber size:S Actions
action #159270: openqaworker-arm-1 is Unreachable size:S Actions
action #159303: [alert] osd-deployment pre-deploy pipeline failed because openqaworker-arm-1.qe.nue2.suse.org was offline size:S Actions
action #159318: openqa-piworker host up alert Actions
action #159396: Repeated HTTP Response alert for /tests and unresponsiveness due to potential detrimental impact of pg_dump (was: HTTP Response alert for /tests briefly going up to 15.7s) size:M Actions
action #159402: "SSL expiration" panel on https://stats.openqa-monitor.qa.suse.de/d/WebuiDb/webui-summary shows problems size:S Actions
action #159555: IPMI access over IPv6 doesn't work on imagetester - try to update BIOS with physical access size:S Actions
action #159639: [alert] "web UI: Too many 5xx HTTP responses alert" size:S Actions
action #159669: No new openQA data on metrics.opensuse.org since o3 migration to PRG2 size:S Actions
action #159840: Munin - minion hook failed - see openqa-gru service logs for details - opensuse.org :: openqa.opensuse.org size:M Actions
action #159966: [FIRING:1] DatasourceError Salt (000000001 A External http responses) Actions
action #160083: client gets a redirect and downloads an HTML page from microsoft instead of the proper windows .qcow2 image Actions
action #160098: After the upgrade to Leap 15.6 osiris showed no proper mount points again for libvirt VMs size:S Actions
action #160171: [openQA][assets] Access to openQA assets forbidden size:S Actions
action #160239: [alert] External http responses Salt (https://openqa.suse.de/health) due to "Too many open files" after switch to nginx Actions
action #160284: grafana server fails to start due to "alert rules: invalid alert rule\ncannot create rule with UID 'too_many_minion_job_failures_alert_s390zl12': UID is longer than 40 symbols" size:M Actions
action #160293: wicked_basic_ref@ppc64le reliably fails when run on qa-power8 Actions
action #160343: [alert][o3] opensuse.org :: openqa.opensuse.org :: hook failed - see openqa-gru service logs for details WARNINGs: rc_failed_per_5min is 10.00 (outside range [:5]). size:S Actions
action #160367: After switch to nginx on OSD let's investigate how system performance was impacted Actions
action #160424: o3: arm21 & arm22 show up as "localhost", (almost) all jobs incomplete size:M Actions
action #160442: Recover IPMI/SOL for qa-power8-3 Actions
action #160451: Fix and monitor SLE needle mirorring Actions
action #160478: Try out higher global openQA job limit on OSD again after switch to nginx size:S Actions
action #160481: backup-vm: partitions usage (%) alert & systemd services alert size:S Actions
action #160484: Failed systemd services alert (monitor run-rcfb4bb8f767f4aa8b7c3022d9ffd2d26.scope) Actions
action #160487: Failed systemd services alert (unreal6 kdump) size:S Actions
action #160493: Failed systemd services alert (osiris-1 drbd) Actions
action #160514: qamaster is down, i.e. also no monitoring from monitor.qe.nue2.suse.org Actions
action #160598: [alert] s390zl12: CPU load alert openQA s390zl12 salt cpu_load_alert_s390zl12 worker size:S Actions
action #160646: multiple multi-machine test failures, no GRE tunnels are setup between machines anymore at all size:M Actions
action #160652: Secondary TAP worker class in different zones size:S Actions
action #160730: [FIRING:1] s390zl12 (s390zl12: CPU load alert openQA s390zl12 salt cpu_load_alert_s390zl12 worker) Actions
action #160802: bot-ng and openQABot pipelines fail to pull containers from registry.suse.de - again Actions
action #160883: salt-states-openqa CI complains about "Authentication failed for 'https://gitlab.suse.de/openqa/salt-pillars-openqa.git/'" size:S Actions
action #161138: Fix graphs/data on job age dashbaord Actions
action #161309: osd not accessible, 502 Bad Gateway Actions
action #161318: Ensure we have a consistent racktables entry for OSD Actions
action #161324: Conduct "lessons learned" with Five Why analysis for "osd not accessible, 502 Bad Gateway" Actions
action #161381: multi-machine test network issues reported 2024-06-03 due to missing content in the salt mine size:S Actions
action #161396: gitlab CI jobs do not fail on invalid YAML size:S Actions
action #161423: [timeboxed:10h] Incomplete config files on OSD due to salt - Improve salt state application from remotely accessible salt master size:S Actions
action #161756: IPMI access over IPv6 doesn't work on blackbauhinia size:S Actions
action #161786: QE-Performance team request 2TB storage for performance test data backup size:S Actions
action #161930: Upgrade scooter BIOS to latest version 3.2 Actions
action #162038: No HTTP Response on OSD on 10-06-2024 - auto_review:".*timestamp mismatch - check whether clocks on the local host and the web UI host are in sync":retry size:S Actions
action #162044: Ensure proper indentation yamllint checks in our salt-states-openqa+salt-pillars-openqa size:S Actions
action #162062: No space left on device causing GitLab pipelines to fail Actions
action #162260: auto-update.service fails on various workers due to a package conflict Actions
action #162272: [qe-tools][RPi3] test fails in prepare_firstboot - use retries with exponential back-off in os-autoinst size:S Actions
action #162293: SMART errors on bootup of worker31, worker32 and worker34 size:M Actions
action #162332: 2024-06-15 osd not accessible size:M Actions
action #162353: Ensure consistent known root password on all OSD webUI+workers size:S Actions
action #162356: Treat OSD non-root mounts as non-critical for boot size:M Actions
action #162359: Change OSD root to more modern filesystem mount options size:S Actions
action #162362: 2024-06-15 osd not accessible - ensure healthy filesystems size:S Actions
action #162365: OSD can fail on xfs_repair OOM conditions size:S Actions
action #162374: Limit number of OSD PRG2 x86_64 tap multi-machine workers until stabilized Actions
action #162377: incomplete config files on OSD due to salt - Prevent conflicting state applications on OSD "fstab" size:S Actions
action #162380: 2024-06-15 osd not accessible - causing false alerts for other hosts size:S Actions
action #162443: Setup of dedicated HMC within qe.prg2.suse.org size:S Actions
action #162455: Secondary TAP worker class instead of "tap_poo…" on closed tickets size:S Actions
action #162485: [alert] failed systemd service: openqa-worker-cacheservice on worker40.oqa.prg2.suse.org "Database has been corrupted: DBD::SQLite::db commit failed: disk I/O error" size:S Actions
action #162590: NFS mounts are stuck on OSD workers if partitions on OSD fail to come up properly on boot size:S Actions
action #162596: [FIRING:1] worker40 (worker40: partitions usage (%) alert openQA partitions_usage_alert_worker40 worker) Actions
action #162602: [FIRING:1] worker40 (worker40: CPU load alert openQA worker40 salt cpu_load_alert_worker40 worker) size:S Actions
action #162605: [FIRING:1] CPU load alert, should be "system load" Actions
action #162641: Prevent redundant salt state.apply actions that are executed in every call - openqa-trigger-from-ibs-plugin Actions
action #162719: Ensure w40 has more space for worker pool directories size:S Actions
action #162725: After w40 reconsider storage use for other OSD workers size:S Actions
action #163052: jenkins.qa.suse.de no longer reachable via web browser (but responsive to SSH) Actions
action #163067: [alert] Munin - minion Minion Jobs - see https://openqa.opensuse.org/minion/jobs?state=failed - opensuse.org :: openqa.opensuse.org Actions
action #163097: Share mount not working on openqaworker-arm-1 and other workers size:M Actions
action #163100: Scripts CI pipeline failing with openQA jobs ending incomplete Actions
action #163340: OBSRSync regularily fails minion jobs - nobody cares, tools gets alerted (e.g. "Munin - minion Minion Jobs") size:M Actions
action #163469: Upgrade a single o3 worker to openSUSE Leap 15.6 Actions
action #163529: redcurrant is unable to boot from PXE server (was: ppc64le pvm_hmc backend workers lost OS / grub access) size:S Actions
action #163544: [virtualization] [SL Micro] Replace USB disk on scooter for uefi boot entry issue verification size:S Actions
action #163592: [alert] (HTTP Response alert Salt tm0h5mf4k) size:M Actions
action #163595: [alert] (Web proxy Response Time alert Salt IeChie2He) Actions
action #163610: Conduct "lessons learned" with Five Why analysis for "[alert] (HTTP Response alert Salt tm0h5mf4k)" Actions
action #163622: Scripts CI pipeline failing with invalid numeric literal error size:S Actions
action #163745: [tools] tests on worker31 time out on yast2 firewall services add zone=EXT service=service:target Actions
action #163766: Scripts CI | Failed pipeline for master (asset failure: Failed to download ....qcow2) size:S Actions
action #163772: [openQA][ipmi][worker35:x] Assigned jobs hang and actually can not run size:M Actions
action #163775: Conduct "lessons learned" with Five Why analysis about many alerts, e.g. alerts not silenced for known issues size:S Actions
action #163778: [alert] host_up & Average Ping time (ms) alert for s390zl12&s390zl13 size:S Actions
action #163790: OSD openqa.ini is corrupted, invalid characters size:M Actions
action #163840: openqaworker18 shows "unable to upgrade ws to command server" size:S Actions
action #163928: [alert] Openqa HTTP Response lost on 15-07-24 size:S Actions
action #164207: The SAN endpoints on openqa.oqa.prg2.suse.org is nearing expiration Actions
action #164222: Client error messages look like http response codes - we should phrase them better - was: asset download fails with "521 Connect timeout" size:S Actions
action #164284: [FIRING:1] worker-arm1 (worker-arm1: System load alert openQA worker-arm1 salt system_load_alert_worker-arm1 worker) size:S Actions
action #164379: [tools] Enable setting YAML_SCHEDULE_DEFAULT in configuration of openQA instance for its value to be shown as link to source code Actions
action #164412: jenkins.qa.suse.de fails to run its jobs Actions
action #164415: SUSE IT storage is broken (was: Broken SUSE:CA.repo causes os-autoinst-needles-opensuse-mirror pipelines to fail) Actions
action #164427: HTTP response alert every Monday 01:00 CET/CEST due to fstrim size:M Actions
action #164490: [tools] We need to find a way to remind us about the process which we gave ourselves: How about a checklist that could be followed based on what's already mentioned in https://progress.opensuse.org/projects/qa/wiki/Tools#Process ? size:S Actions
action #164676: [alert][o3] test fails in setup_multimachine to reach other machines/instances from w20 size:S Actions
action #164709: OpenQA logreport for o3 logging livestream connection refused errors Actions
action #164853: [alert][FIRING:1] s390zl13 (s390zl13: Memory usage alert Generic memory_usage_alert_s390zl13 generic) size:S Actions
action #164895: o3 had corrupted needles git repo, lost uncommitted needles between 2024-07-31 and 2024-08-02 Actions
action #164907: [alert][FIRING:1] host_up (qesapworker-prg5: host up alert openQA, qesapworker-prg5-mgmt.qa.suse.cz not reachable, failing osd-deployment Actions
action #164979: [alert][grafana] File systems alert for WebUI /results size:S Actions
action #165027: (Cron Daemon) sending email about /usr/bin/dehydrated --cron Actions
action #165030: (Cron Daemon) sending email about /opt/openqa-scripts/dump-psql Actions
action #165033: Alert Worker .* has no heartbeat (900 seconds), restarting (see FAQ for more) on o3 - again Actions
action #165096: [osd] Extend /assets + /space-slow to allow to store more assets size:S Actions
action #165105: o3 stopped committing new needles Actions
action #165126: error from cron "Cron <geekotest@ariel> find /var/lib/snapshot-changes/opensuse/Tumbleweed/ -maxdepth 1 -mindepth 1 -type f -printf '" Actions
action #165195: [alert] Failed systemd services alert Actions
action #165267: Check power supply and racktable information on squiddlydiddly Actions
action #165348: Handle questionable grafana notification policy "alert=host_up" Actions
action #165434: OSD SSL certificates not always refreshed within expected time, probably only after system reboots size:S Actions
action #165512: [alert] worker-arm2: System load alert active for 5 minutes Actions
action #165521: grafana-server failed to start with "Error: ✗ failure to parse file alerts_to_delete.yaml: open /etc/grafana/provisioning/alerting/alerts_to_delete.yaml: permission denied" Actions
action #165611: [openQA][infra][sut][aarch64] Power supply failure on squiddlydiddly arm64 machine size:M Actions
action #165617: [alert] openqa-trigger-from-obs pull: RPC failed, expected flush after ref listing size:S Actions
action #165620: [alert] OSD deployment pipeline failing with invalid numeric literal error Actions
action #165806: [alert] (Cron Daemon) fetchneedles/git pull connection timed out Actions
action #165860: wicked_basic_ref fails on ppc64le: Open vSwitch command 'set_vlan' with arguments 'tap3 8' failed Actions
action #166136: s390 LPAR s390ZL12 down and unable to boot - potential corrupted filesystem Actions
action #166169: Failed systemd services on worker31 / osd size:M Actions
action #166172: [FIRING:1] ada (ada: partitions usage (%) alert Generic partitions_usage_alert_ada generic) size:S Actions
action #166202: Unable to login to Grafana aka monitor.qa.suse.de with valid credentials Actions
action #166280: Setup of Nvidia Orin for bare-metal tests in openQA Actions
action #166313: Failed systemd services on multiple workers due to logrotate errors size:S Actions
action #166394: iPXE service is unavailable for baremetal SUTs on OSD worker pool size:S Actions
action #166430: [tools] Keep a hardlink to iso in repo folder for agama test Actions
action #166433: [alert] Waves of emails due to manual changes in /opt/openqa-trigger-from-obs size:S Actions
action #166511: [tools] Could not resolve host: workerX.mshome.net. seems something wrong with domain "mshome.net" Actions
action #166520: [alert][FIRING:1] qesapworker-prg5 (qesapworker-prg5: host up alert host_up openQA host_up_alert_qesapworker-prg5 worker) size:S Actions
action #166721: [alert] Waves of emails due to kex_exchange_identification: Connection closed by remote host errors size:S Actions
action #166739: Consistent alerts for failed systemd services on o3 size:S Actions
action #166742: Prevent security email for ariel Actions
action #166775: Support the move of haldir Actions
action #166844: Multiple machines in 10.100.192.0/24 network are unreachable Actions
action #166976: dehydrated error on o3: "JWS has no anti-replay nonce" Actions
action #166979: Grafana aka monitor.qa.suse.de reporting Bad Gateway error Actions
action #167051: https://gitlab.suse.de/openqa/salt-pillars-openqa/-/jobs/3109145 failed due to telegraf errors on monitor.qa.suse.de size:S Actions
action #167057: Run more standard, qemu OSD openQA jobs in CC-compliant PRG2 and none in NUE2 size:S Actions
action #167081: test fails in support_server/setup on osd worker37 size:S Actions
action #167164: osd-deployment | Minions returned with non-zero exit code (qesapworker-prg5.qa.suse.cz) size:M Actions
action #167257: Grafana aka monitor.qa.suse.de reporting Bad Gateway error - again size:S Actions
action #167266: openQA log reports "Failed to login" with no details in logs size:S Actions
action #167458: openqa.oqa.prg2.suse.org SAN validity alert Actions
action #167485: salt-states-openqa deploy pipeline can cause monitor.qe to fail to render dashboard templates correctly size:M Actions
action #167557: OSD not starting new jobs on 2024-09-28 due to >1k worker instances connected, overloading websocket server Actions
action #167584: grafana-server on monitor.qe.nue2.suse.org yields "502 Bad Gateway", fails to start since 2024-09-28 03:57Z Actions
action #167719: No new data in monitor.qe.nue2.suse.org due to influxdb failing to write with ""error opening new segment file for wal (1): write /var/lib/influxdb/….wal: no space left on device" Actions
action #167722: Efficient use of monitoring data within influxdb on monitor.qe.nue2.suse.org size:M Actions
action #167728: grafana dashboard for monitor.qe.nue2.suse.org size:S Actions
action #167830: salt-states-openqa static pipeline checks fail size:S Actions
action #167833: openqa/scripts-ci pipeline fails - "jq: parse error: Invalid numeric literal at line 1, column 8 (rc: 5 Input: >>>Request failed, hit error 502" while running openqa-schedule-mm-ping-test Actions
action #167911: Scripts CI | Failed pipeline - openqa-schedule-mm-ping-test incompletes on o3 Actions
action #167974: bare-metal worker `unreal3` lost in OSD webUI Actions
action #168106: QE PXE server in PRG2 Actions
action #168177: Migrate critical VM based services needing access to CC-services to CC areas Actions
action #168214: [alert] Average Ping time (ms) alert size:S Actions
action #168337: [tools]test fails in bootloader_zkvm - auto_review:"qemu-img.*Failed to get shared.*No locks available" Actions
action #168358: [alert] Failed systemd services alert: var-lib-openqa-share-factory*.mount Actions
action #168376: Enable automatic openQA git clone instead of fetchneedles on OSD size:S Actions
action #168454: `openqaworker21` fails with no `Qemu/KVM found` Actions
action #168460: salt-states-openqa pipeline fails due to missing XML file expected by scriptgen.py Actions
action #168475: salt-states-openqa telegraf pipeline failing with error in libcrypto Actions
action #168544: [alert] Failed systemd services alert: check-for-kernel-crash, kdump-notify Actions
action #168556: Support setup of bare-metal5+6 size:S Actions
action #168811: baremetal-support in PRG2 size:M Actions
action #168865: salt-states-openqa | Failed pipeline for master | pkg.installed package vagrant not found Actions
action #168901: Support SUSE PRG office datacenter "PRG1" move to a new location "PRG3" while ensuring business continuity - pre-planning size:M Actions
action #168916: find out what's the current state of openqaworker27 size:S Actions
action #168967: Support setup network for new bare-metal in PRG2e - D10 (SD-169875) size:S Actions
action #169078: dashboard.qam.suse.de SSL certificate not deployed within expiry size:S Actions
action #169147: Unable to login at Grafana (monitor.qa.suse.de or any other domain) Actions
action #169294: Failed systemd services alert - squid service is failing on osd Actions
action #169348: Custom, non-IT-provided wireguard tunnels to connect NUE2 OSD openQA workers to OSD Actions
action #169549: START_AFTER_TEST seems doesn't work for some certain test suites size:s Actions
action #169555: Setup new development laptop (rrichardson) Actions
action #169564: Configure wireguard tunnels on OSD production hosts needed for openQA located in the NUE2 server room size:S Actions
action #169585: [alert] Adapt "Packet loss between worker hosts and other hosts" alert (and potentially other related alerts) after CC-related network separation - split "required_external_networks" monitoring by source location size:S Actions
action #169684: [alert] (Failed systemd services alert (except openqa.suse.de) Salt Uk02cifVkz) Actions
action #169690: logwarn repeatedly sending openssl related problems. Is the state file stuck? Actions
action #169699: [alert] opensuse.org openqa.opensuse.org openqa_minion_jobs_hook_rc_failed minion 'hook failed - see openqa-gru service logs for details' Actions
action #169744: [alert] Adapt "Packet loss between worker hosts and other hosts" alert (and potentially other related alerts) after CC-related network separation - fix mirror.nue2 monitoring from NUE2 size:S Actions
action #169750: [alert] backup-vm (backup-vm: partitions usage (%) alert Generic partitions_usage_alert_backup-vm generic) Actions
action #169918: [security] tests never get executed on tyrion and coppi Actions
action #169945: Multimachine seems to be broken on openqaworker-arm22 Actions
action #170026: [QA][tools][monitor] monitor.qa.suse.de is down Actions
action #170038: power outage in NUE2 2024-11-17 Actions
action #170041: Configure wireguard tunnels on hosts located in the NUE2 server room - at least one KVM@PowerNV host size:S Actions
action #170077: Put more storage into qamaster "to make our lives easier in general" size:M Actions
action #170083: [FIRING:1] (Failed systemd services alert (except openqa.suse.de) Salt Uk02cifVkz) backup-qam failing unit "ypbind" since 2024-11-19 0818Z Actions
action #170104: Use nessberry as KVM@PowerNV host Actions
action #170122: [alert][FIRING:1] s390zl13 (s390zl13: partitions usage (%) alert Generic partitions_usage_alert_s390zl13 generic) Actions
action #170257: salt-pillars-openqa | Failed deployment on openqa-piworker.qe.nue2.suse.org size:S Actions
action #170323: [openQA][infra][http(s)] Many baremetal SUT machine fails to get autoyast profile from worker machine via HTTP(s) Server returned code 404 Actions
action #170338: No monitoring data from OSD since 2024-11-25 1449Z size:M Actions
action #170347: NUE2 DHCP server dhcpd says "host unknown" (red), possibly related to squidbilly Actions
action #170464: Enable automatic openQA git clone instead of fetchneedles on o3 Actions
action #170473: k2.qe.suse.de not reachable from mania:2 size:S Actions
action #170479: tftp in francencampus basement no longer working Actions
action #170494: nginx.service on monitor failed because of: "No such file or directory:calling fopen(/etc/dehydrated/certs/loki.qa.suse.de/fullchain.pem" Actions
action #170506: qa-jump-configs | Failed pipeline for master Actions
action #173347: Ensure we have a current backup of qamaster VMs, VM config, jenkins data, data from backup-vm itself, etc. size:S Actions
action #173353: physically label slots 10+11 on qamaster size:S Actions
action #173359: salt-pillars-openqa | Failed pipeline for master, conflicting ID 'net.ipv6.conf.br0.accept_ra Actions
action #173500: Support setup network for new bare-metal in PRG2e - D10 (SD-173361) Actions
action #173674: qamaster-independent backup size:S Actions

QA (public, currently private due to #173521) - Tools - Next

open

Staging for next tasks to be picked up by the QE tools team after "Ready"

27%

57 issues   (11 closed — 46 open)

Related issues
action #40196: [monitoring] monitor internal port 9526, port 80, external port 443 accessibility of o3 and response times size:M Actions
action #108974: Loan Fujitsu server for OpenQA FIPS testing needs size:S Actions
action #135023: Consider moving the retro out of the weekly or alternating either one size:S Actions
action #155743: OBSRSync fails to sync openSUSE:Factory:PowerPC:ToTest (was: WARNINGs: failed is 452.00 in Munin - minion Minion Jobs on o3) Actions
action #157606: Prevent missing gre tunnel connections in our salt states due to misconfiguration Actions
action #158116: typing issue on ppc64 worker - crosscheck performance impact of ffmpeg on ppc64le (Power8 kvm) size:M Actions
action #158377: Detect from monitoring data which monitored machines show a too low system usage over time size:M Actions
action #160826: Optimize gre_tunnel_preup.sh generation jinja template size:S Actions
action #161393: gitlab CI jobs do not execute salt-lint and do not fail on missing "find" executable Actions
action #162347: Invalid journald configuration on OSD (any maybe other hosts) size:S Actions
action #162494: telegraf error on some OSD controlled machines "W! [inputs.diskio] Error gathering disk info: no such file or directory" size:S Actions
action #162734: Simple script detecting gre_tunnel_preup.sh with only empty remote_ip= statements during salt CI pipelines size:M Actions
action #162737: Put #161735-4, #161735-5, #161735-6 about "Better error detection in GRE tunnel config" into tasks size:S Actions
action #166523: worker33 log entry "Couldn't write '1' to 'net/ipv4/conf/br1/forwarding', ignoring: No such file or directory" size:S Actions
action #167725: Correct worker class and racktables entry for Power9 machines haldir+legolas size:S Actions
action #168148: hackweek idea: use loki to monitor our log files and explore alerting possibilites based on these size:S Actions
action #168361: "Too many 5xx HTTP responses alert"-alert does not (or too slow) recover automatically again Actions
action #168718: The "response codes" panel takes a considerable time to load or even runs into timeouts size:M Actions
action #169357: monitoring+alerting for dashboard.qam.suse.de SSL certificate not deployed within expiry size:S Actions
action #169399: Update name of machine "migration-smt2" size:S Actions
action #170260: Help others (or ourselves) to configure wireguard tunnels on other hosts needing wireguard to PRG2 in the NUE2 server room size:M Actions

QA (public, currently private due to #173521) - future

open

a "virtual" milestone that is never reached. Everything that is currently out of scope can go here. Place for new feature ideas, wishes, proposed improvements and changes which we could work on in the future but are not currently planned for.

41%

1632 issues   (619 closed — 1013 open)

Related issues
action #10516: [functional][u] Use jenkins as extension for openQA Actions
action #17816: [functional][u][ipmi] Error: Unable to establish IPMI v2 / RMCP+ session Actions
action #18608: [qe-core][tools][sle][functional][research][medium] Implement proper clean up for images on jump hosts, e.g. s390pb, hyperv host, svirt and a proper monitoring Actions
action #30595: [ppc64le] Deploy bcache on tmpfs workers Actions
action #38516: [tools] Disable memory autoballooning on openqaw5-xen Actions
action #39008: [tools][pvm] Redundant powerVM production workers Actions
action #41867: [devops][tools] Replace get-metrics script by telegraf Actions
action #41975: Evaluate graphite vs prometheus Actions
action #42233: Production-grade jenkins.qa.suse.de for QA SLE (EMEA) Actions
action #42467: http://jenkins.qa.suse.de/job/monitor-openQA_in_openQA-TW fails to write comments in OBS project, monitor-openqa_job expects package name for OBS api, not project Actions
action #44183: Add checks on transactional-update workers using "health-checker" Actions
action #54785: tap devices not in any zone, error reported by firewalld Actions
action #55316: monitoring alerts for too long running database queries Actions
action #57239: Add/fix openqa_logwarn for o3 and osd sending to o3-admins@suse.de and osd-admins@suse.de respectively Actions
action #58325: Update firmware of our powerKVM machines (QA-Power8-3, QA-Power8-4, QA-Power8-5) Actions
action #60962: Enable multi-machine capability for all configured workers Actions
action #62666: Move openqa.opensuse.org into opensuse private network Actions
action #63382: /usr/share/qemu/ovmf-x86_64-staging{,-code,-vars}.bin on workers is not installed by any package, e.g. missing on imagetester Actions
action #63874: ensure openqa worker instances are disabled and stopped when "numofworkers" is reduced in salt pillars, e.g. causing non-obvious multi-machine failures Actions
action #64123: Expose database revert as a manual trigger Actions
action #64541: Issues with DNS resolving in QEMU Actions
action #64580: Detect and recover from I/O blocked worker machines, e.g. openqaworker-arm-{1,2,3}.suse.de Actions
action #64803: harmonize IPMI username and password auto_review:"(sonic|tails)-sp.qa.suse.de .*Unable to establish IPMI" Actions
action #65019: ppc64 debuginfo repo is not stripped for Tumbleweed Actions
action #65555: [qe-core][functional][core] html5test.opensuse.org includes non-existing file Actions
action #66019: Configure downsampling and a retention policy for InfluxDB Actions
action #66610: Prevent misleading login failing over http on https-enabled instances, e.g. disable non-SSL in production openQA instances using OpenID (OSD, o3) at least for login attempts Actions
action #68633: alert if there is no worker active for any existant machine class Actions
action #71590: [osd][alert] Implement proper monitoring of needed resources of workers Actions
action #72079: Upgrade o3 worker openqa-aarch64 to openSUSE Leap 15.2 (to use newer packages specifically needed for aarch64 and as precursor), also problem auto_review:"(?s)starting: /usr/bin/qemu-system-aarch64.*backend died: Migrate to file failed" Actions
action #73297: auto_review:"(?s)Running on openqa-aarch64:.*considering VNC stalled.*THERE IS NOTHING TO READ" Actions
action #73345: [u] Cleanup of old needles from os-autoinst-needles-opensuse and os-autoinst-needles-sle Actions
action #76768: HDD images are cleaned-up too early for aarch64 Tumbleweed Actions
action #76792: /var/lib/openqa in QA-Power8-5-kvm reports Structure needs cleaning (auto_review:"(?s)Running on QA-Power8-5-kvm.*cannot remove directory for testresults":retry) Actions
action #77011: openqaworker7 (o3) is stuck in "recovery mode" as visible over IPMI SoL Actions
action #77317: chat bot to conduct daily checks, alerts, reminders, etc. Actions
action #78061: [Alerting] openQA minion workers alert - alert turned "OK" again after 20 minutes and we don't know what was wrong Actions
action #78222: Persistent postgres setup and configuration in salt Actions
action #78232: [ipmi|PXE]OSD repo seems to be having mount issue Actions
action #78314: Order 2 VMs for the LinuxONE Community Cloud Actions
action #78348: osd: Some job group obsolete definitions should be removed Actions
action #80130: grenache salt is spilling errors about "Failed to load grains defined in grain file ppc_powervm.check_output" Actions
action #80482: qa-power8-5-kvm has been down for days, use more robust filesystem setup Actions
action #80540: idea: Conduct "power outage drills", e.g. once every half-year? Actions
action #80732: Monitoring on log alerts, also on grafana log alerts, maybe grafana-loki? Actions
action #80824: o3: Migrate from SuSEfirewall2 to firewalld Actions
action #80990: some jobs incomplete with auto_review:"(?s)Running on[^\n]*Linux 5.3.18.*Migrate to file failed, it has been running for more than 240 seconds":retry , seems to happen since upgrade to Leap 15.2 Actions
action #81308: Show actual percentage in partitions usage (%) alert Actions
action #88195: Full test of salt states including DBus on CI Actions
action #88197: Workflow for staged deployments Actions
action #88299: [virtualization] Worker openqaw5-xen-1.qa.suse.de is not reachable (xen-hvm/xen-pv failing) Actions
action #89398: OSD deployment stopped prematurely due to "System management is locked by the application with pid 29165 (zypper)" Actions
action #90062: osd deployment fails in "record package versions pre-deploy" maybe because a kernel update is pending? Actions
action #91626: Deployment emails should cover rollback Actions
action #93119: [s390x] Update of s390x Test infrastructure after shutdown of Mainframe zEC12 Actions
action #93138: openqa-reload-worker-auto-restart and openqa-worker-auto-restart fail if numofworkers is reduced Actions
action #94399: No alert when arm workers are offline, alert if telegraf throws errors size:M Actions
action #94495: Migrate to InfluxDB 2 when needed (at least when our current OS base has a package) Actions
action #94765: Bring openqaworker12 into production (w/o multi-machine test support) size:M Actions
action #94783: Bring openqaworker11 into production including multi-machine test support (same as w12) size:M Actions
action #96257: Tests timeout with MAX_SETUP_TIME on osd - Apply a higher MAX_SETUP_TIME applicable for the *complete* OSD infrastructure, e.g. add to every worker config entry Actions
action #96552: Persistent records of I/O usage by process size:M Actions
action #96683: Reducing the number of worker slots leads to failing systemd units Actions
action #97409: Re-use existing filesystems on workers after reboot if possible to prevent full worker asset cache re-syncing Actions
action #97412: Reduce I/O load on OSD by using more cache size on workers with using free disk space when available instead of hardcoded space Actions
action #97640: have a secondary backup openQA webUI instance in another location than OSD, e.g. PRG Actions
action #97745: [virtualization][hyperv] ensure_serialdev_permissions fails for hyperv Actions
action #97859: Improve network for OSD Actions
action #97976: [alert] OSD file systems - assets Actions
action #98682: jobs run powerqaworker-qam-1 fail with auto_review:"(?s)powerqaworker-qam-1.*Can't write to file (.*): No space left on device at .*":retry size:M Actions
action #98835: arm jobs failing (again?) with auto_review:"backend died: Open vSwitch command 'set_vlan' with arguments .*was not provided by any .service files":retry Actions
action #99138: Provide ratio of tests by result in monitoring - by job group Actions
action #100841: Improve logging for libvirt hosts Actions
action #100976: specific alert about the size of our database, maybe even database tables? Actions
action #100979: configure better auto-vacuum for our database(s) Actions
action #100985: Come up with a way to regularly check job group configs for outliers and misconfiguration, e.g. overly long result retention periods Actions
action #100988: ask for OSD /srv to be enlarged (was: investigate how to optimize /srv data utilization on OSD) size:S Actions
action #102167: Disk monitoring for s390x z/VM backend Actions
action #102269: openQABot pipeline failed with HTTP 403 Actions
action #104091: Try `aarch64_cpu_max` tests on `openqaworkerX` Actions
action #105013: backup o3 worker config files size:S Actions
action #105867: OpenQA bot schedules jobs with incomplete INCIDENT_REPO Actions
action #106756: cmd returned 32768 repeatedly reported on o3 Actions
action #106760: DBI Exception: DBD::Pg::st execute failed: number of parameters must be between 0 and 65535 repeatedly reported on o3 Actions
action #106766: DBI Exception: DBD::Pg::st execute failed: ERROR: invalid input syntax for type integer reported on o3 Actions
action #106841: Try using tmpfs on openqaworker1 to use RAM more efficiently Actions
action #107023: cmd returned 31744 repeatedly reported on o3 Actions
action #107794: openqa-logwarn is missing messages Actions
action #107980: openqa-aarch64 - kernel traces in dmesg "watchdog: BUG: soft lockup - CPU#.* stuck for .*s! [qemu-system-aar:.*]" Actions
action #108266: grenache: script_run() commands randomly time out since server room move Actions
action #109112: Improve os-autoinst sshXtermVt.pm connection error handling (was: "Test died: Error connecting to <root@redcurrant-4.qa.suse.de>: No route to host") size:M Actions
action #109635: Check grafana monitoring host performance size:M Actions
action #110191: CI checks should ensure that apparmor changes are correct Actions
action #110467: Establish reliable tap setup on ow14 Actions
action #110545: Investigate and fix higher instability of openqaworker-arm-4/5 vs. arm-1/2/3 - further things to try size:M Actions
action #112769: errors from telegraf on osd webUI "[inputs.procstat] Error in plugin: open /sys/fs/cgroup/systemd/system.slice/salt-master.service/cgroup.procs: no such file or directory", same for openqa-scheduler.service Actions
action #112787: Use different users for our database accesses so that we can debug where load comes from Actions
action #112850: Make pgstatstatements persistent on OSD (in salt) Actions
action #112943: Consider running explicitly triggered `vacuum ANALYZE` after every database migration Actions
action #113357: UEFI PXE or "network boot" support within .qa.suse.de size:M Actions
action #113701: [qe-core] Move workers back to grenache Actions
action #113800: Setup o3 to run rootless containers on worker hosts Actions
action #114400: [qe-core] Rebalance OpenQA worker distribution Actions
action #114622: Identification of unused/idle machines by alarming when there is no traffic on the corresponding switch ports for some time Actions
action #114866: Handle a trailing slash in NEEDLES_DIR correctly Actions
action #114992: Broken MM machines don't appear as available workers Actions
action #115763: Add new feature to create a dependency that could be derived automatically from asset settings Actions
action #115925: aarch64: Random QEMU failures while retrieving host CPU features Actions
action #116473: Add OSD PowerPC workers to automatic recovery we already have for ARM workers Actions
action #116716: Repurpose ix64ph1079, ix64ph1080, ix64ph1081, e.g. as openQA workers Actions
action #116968: separate email or mailing list address for os-autoinst related development, e.g. circleCI bot account Actions
action #118375: Do not alert about "packet loss" if hosts are down Actions
action #118891: Make alerts depend on each other Actions
action #119059: Use qa-power8 for ppc tests in o3 - network connected? Actions
action #119479: openqABot pipeline failed after runner getting stuck for 1h0m0s size:M Actions
action #119755: Use a PowerVM machine to serve both PowerVM LPARs for testing as well as one VM running qemu tests Actions
action #120064: SSD SAN storage (NetApp) for O3 would improve user experience (can not just buy drives on our own) Actions
action #120115: [qem-dashboard] Use new openQA job settings API to show more data Actions
action #120145: Where can we see which services are blocked ourselves in SUSE networks? Actions
action #120169: Make s390x kvm workers also use FQDN instead of IPv4 in salt pillars for VIRSH_GUEST Actions
action #120279: Proper maintainership for openqa_service VM size:M Actions
action #120651: [openQA][infra][ipmi][worker][api] The expected pattern CMD_FINISHED-xxxxx returned but did not show up in serial log (wait_serial timed out) size:M Actions
action #121171: Consider using pngquant for speedup Actions
action #121306: [virtualization][hyperv] worker7-hyperv.oqa.suse.de can not be reached Actions
action #121507: Iscsi issue on OSD worker Actions
action #121597: Implement automated racktables sanity check Actions
action #121600: Add monitoring for connected machines inside "our" network which are not present in racktables Actions
action #121672: [virtualization] Connectivity issues on worker8-vmware.oqa.suse.de Actions
action #122776: containers_tw_image_on_ubuntu_host: error 403 (forbidden) downloading the ubuntu qcow2 file Actions
action #123226: Temperature monitoring in SUSE QE lab size:M Actions
action #123712: PowerVM HMC within o3, e.g. on VM Actions
action #123960: s390x tests fail to log into VNC console on worker2 due to already present VM with same mac address Actions
action #124562: Re-install at least one of the new OSD workers located in Prague Actions
action #125054: Review https://github.com/alexellis/awesome-baremetal Actions
action #125141: Salt state security-sensor.repo fails regularly due to invalid repo contents from the velociraptor project size:M Actions
action #125219: Use qa-power8 for ppc tests in o3 - try the other suggestions Actions
action #125744: [tools][alert][FIRING:1] (Failed systemd services alert (except openqa.suse.de) QDG8aXAVz) due to openqa-piworker.qa.suse.de unable to reach openqa.suse.de Actions
action #126677: salt-states-openqa fails with 0 errors and result False messages buried in one of several places Actions
action #126941: [spike] Evaluate a move of the osd database to its own VM Actions
action #128222: [virtualization] The Xen specific host configuration on openqaw5-xen can be re-created from salt size:M Actions
action #128513: Apache Response Time monitoring dashboard is slow to render and should be made lighter Actions
action #128870: [alert] DatasourceError for "InfluxDB not reachable" size:M Actions
action #129032: ipmitool monitoring Actions
action #129121: performance: various tests fail receiving data at usable speed over the internet Actions
action #130378: Integration of extra salt repositories as salt formulas size:M Actions
action #130513: Add o3 to grafana Actions
action #130778: Treat some openqa-clone-job failures non-fatal in openqa-investigate hook Actions
action #131549: [spike][timeboxed:20h] Additional redundancy for OSD virtualization testing - Hyperv 2016 worker host size:M Actions
action #132293: Ensure cleaned up tenants in netbox Actions
action #132380: Multiple empty folders in grafana linked to alerts Actions
action #132440: Research productive workflows for reviews on GitHub Actions
action #132926: OSD cron -> (fetch_openqa_bugs)> /tmp/fetch_openqa_bugs_osd.log failed size:M Actions
action #132998: [alert] [FIRING:1] openqaworker-arm-3: Memory usage alert openQA (openqaworker-arm-3 memory_usage_alert_openqaworker-arm-3 worker) size:M Actions
action #133247: Additional redundancy for OSD virtualization testing - Hyperv 2019 and 2022 (or 2012r2) worker host size:M Actions
action #133388: Unavailable developer mode on ow18 Actions
action #133700: Network bandwidth graphs per switch, like https://mrtg.suse.de/qanet13nue, for all current top-of-rack switches (TORs) that we are connected to size:M Actions
action #133907: Improve monitoring for http(s?) reachable on jenkins.qa.suse.de size:M Actions
action #133991: Cover same metric for different hosts with a single alert rule Actions
action #134048: openqa-piworker does not restart openQA worker processes after upgrades as it is not a salt minion in "worker" role Actions
action #134135: openqa-monitor.qa.suse.de salt CI deploy telegraf check shows error about "could not find file: [/etc/dehydrated/certs/monitor.qe.nue2.suse.org/fullchain.pem]" Actions
action #134492: [tools] QA-Power8-2-8247-22L-SN1010D5A is not working size:M Actions
action #134585: Display the job age per architecture Actions
action #134819: Errors in salt minion and master log on osd Actions
action #134846: Old NFS share mount is keeping processes stuck and openQA workers seem up but do not work on jobs Actions
action #134852: gitlab CI job fails in telegraf check with unsupported option since telegraf package on monitor.qa.suse.de was downgraded due to invalid repo metadata Actions
action #135005: Reduce duplication in salt-pillars-openqa openqa/workerconf.sls with advanced YAML/jinja features Actions
action #135008: Max job age graphs use mean aggregation when max would make more sense Actions
action #135056: MM Test fails in a connection to an address outside of the worker Actions
action #135662: Make the state of deployments (and rollbacks) more discoverable Actions
action #135944: Implement a constantly running monitoring/debugging VM for the multi-machine network Actions
action #136952: Consider lowering the threshold for job age and job queue alerts Actions
action #136958: Add Grafana panel for number of workers Actions
action #136967: Monitor number of uploading jobs in grafana Actions
action #137114: openQA workers fail to register after bootup due to unable to resolve openqa.suse.de but manage to do so immediately when restarting worker services Actions
action #137810: Add openqa_logwarn for osd sending to osd-admins@suse.de respectively Actions
action #138275: Ensure that there is proper ownership and maintainership for qanet.qa.suse.cz Actions
action #138374: Ensure all salt managed machines show grub and boot log and Getty on ipmi sol size:M Actions
action #138683: https://metrics.opensuse.org/d/osrt_openqa/osrt-openqa?orgId=1&from=now-7d&to=now should show current results … which apparently it doesn't Actions
action #138707: Re-enable worker32 for multi-machine tests in production Actions
action #139070: Re-enable worker34 for multi-machine tests in production Actions
action #139154: Re-enable worker33 for multi-machine tests in production Actions
action #139202: Consolidate the current use or make use of PRG2-J : Rack J11 - OpenQA Rack : H03-ch1 https://racktables.nue.suse.com/index.php?page=object&object_id=23373 Actions
action #150986: [timeboxed:10h] Prevent cpu over-allocation in openQA worker service definitions Actions
action #150995: Fix MM setup on diesel so test scenarios like ha_ctdb_supportservertest-ppc-mm work Actions
action #151180: Automatic reboot on o3 openQA workers broken Actions
action #151307: [grafana] migrate deprecated angular plugins Actions
action #151570: [qe-core] Cleanup openQA jobgroups Actions
action #151606: Ensure that the tap class is fully enabled on petrol Actions
action #152098: [research][timeboxed:10h] Learn more about openvswitch with experimenting together size:S Actions
action #154180: Proper kvm asset cleanup for s390x kvm backend (svirt) and tests Actions
action #155089: Automatically comment on tickets not using one of our templates Actions
action #155200: Periodically running simple ping-check multi-machine tests on ppc64le covering multiple physical hosts on OSD alerting tools team on failures size:M Actions
action #155482: [qe-sap] Update contact person for racktable entries with Lee Martin Actions
action #155764: Consider switching to safer filesystems than ext2 in osd+o3 Actions
action #157636: remove NOVIDEO=1 from ppc64le workers Actions
action #157738: Use rstp_enable=True on o3 as well Actions
action #158380: Detect and switch off from monitoring data which monitored machines show a too low CPU usage over time Actions
action #158386: Crosscheck which machines marked as "unused" in racktables still draw power according to ePDU data (as they should not be powered on and wasting significant power at all) - NUE2 size:M Actions
action #158709: typing issue on ppc64 worker - with automatic CPU load based limiting in place let's increase the instances on mania again Actions
action #159063: s390x qemu backend host within SUSE networks Actions
action #159234: ignore this one Actions
action #159534: Make use of all 12 redcurrant LPARs Actions
action #159657: [alert] about "web UI: Too many 5xx HTTP responses alert" hidden behind grafana alert grouping Actions
action #159666: Lack of alerts about error messages related to InfluxDB and SSL expiration panel Actions
action #159963: [sporadic] salt-pillars-openqa deploy pipeline fails because file.manage complains about »name '__env__' is not defined« Actions
action #160436: Use s390zl19+s390zl1a in production size:S Actions
action #160655: No persistent journal on unreal6 Actions
action #160739: Support ssh keys with special characters like `@` in the name in our infrastructure size:S Actions
action #160985: nginx on osd returns 503 Service Temporarily Unavailable when accessing assets due to rate limiting Actions
action #161321: OSD status overview in monitoring shows osd is online despite the whole machine being down Actions
action #161426: incomplete config files on OSD due to salt - introduce post-deploy monitoring steps like in osd-deployment but in salt-states-openqa Actions
action #161429: incomplete config files on OSD due to salt - create annotations in grafana on the time of the osd deployment as well as salt-states-openqa deployments Actions
action #162245: Setup new PRG2 openQA worker for o3 - w27 Actions
action #162518: Unable to consistently login at stats.openqa-monitor.qa.suse.de Actions
action #162521: Reconsider the global job limit on o3, try higher than 170 Actions
action #163394: Consider extending our logging of broken workers in grafana (Better understand "Broken workers alert" retroactively) Actions
action #163478: [sporadic] sometimes in salt deploy pipelines "Unable to manage file: name '__env__' is not defined" Actions
action #163541: [virtualization] worker8-vmware IPMI interface cannot be connected Actions
action #164042: Consider using https://github.com/openSUSE/os-update instead of custom auto-update Actions
action #164400: Feature: Continue failed downloads without starting from the beginning in cacheservice Actions
action #164472: Access log statistics Actions
action #164478: Monitoring of idle/busy webUI/liveview handler workers Actions
action #164484: [tools] Investigation helper, e.g. commands in a bash script to collect useful logs, systemd journal, etc. Actions
action #164487: [tools] Research about a status page or maintenance fallback mode information page which we could redirect to. That could be known issues on openQA itself as well as redirect on the level of the web proxy Actions
action #164493: [tools] Research the meaning and implications of the number of workers as a tuning parameter. Come up with an understanding on how to select the value, where the limits are, what suggestions to openQA admins are -> add that in openQA documentation Actions
action #164496: [tools] Follow-up on A4-1 but select a value dynamically based on nr. of requests vs. available system resources Actions
action #164499: [tools] Limit the number of live views handled and present a "busy" message to users when there are no more free liveview handler workers available Actions
action #164502: [tools] Investigate if there are more potentially long running blocking requests which should be treated similar as the live view ones, i.e. being handled by a different process (group) Actions
action #164874: osd-deployment failed due to openqaworker-arm-1 salt-minion yielding "no response" but the machine is up and reachable over ssh Actions
action #165192: Enable all OSD PRG2 x86_64 machines for multi-machine use again Actions
action #165345: [spike][timeboxed:20h] Custom qa-tools team managed low-maintenance platform for hosting team-owned containerized workload Actions
action #165782: [openQA][infra][ipxe][uefi][initrd] UEFI iPXE Machine fails to load initrd size:S Actions
action #166415: [alert] diesel could not reach qa-jump.qe.nue2.suse.org causing ping alert to fire Actions
action #166802: Recover worker37, worker38, worker39 size:S Actions
action #167015: [SLM][infra]Ask for SLE Micro 6.2 pxe boot contents to be published as http source size:M Actions
action #167263: OpenQA logreport regularly reports Failed to login in email size:M Actions
action #168145: implement telegraf health check and adjust according pipelines Actions
action #168301: After w40 related problems reconsider storage use for all PRG2 based OSD workers Actions
action #168922: Look into better documentation and possible automation of upgrades on o3 Actions
action #168949: Deploy OSD from SLE+packagehub instead of custom OBS repo devel:openQA Actions
action #170101: Use grenache-1 as KVM@PowerNV host Actions
action #170353: Bare metal machines have no links on network interfaces Actions
action #170365: Understand and eventually correct why enginfra thinks we're using their mysql cluster Actions
action #173344: Extend iPXE in qe/oqa.*.suse.org to also display on local console Actions
action #173350: Migrate VMs from qamaster to modern hypervisor solution Actions