Project

General

Profile

Roadmap

QA - QE-Core: Ready

open

Used by QE Core to track sprint items

92%

800 issues   (730 closed — 70 open)

Related issues
action #113716: [qe-core] proxy-scc is down Actions
action #127523: [qe-core][s390x][kvm] Make use of generic "s390-kvm" class to prevent too long waiting for s390x worker ressources Actions
action #134363: [qe-tools] New InfluxDB database for QE-Core metrics Actions
action #151331: [qe-core][s390x][zvm] Make use of generic "s390x-zVM" class instead of s390x-zVM-vswitch-l2+l3+etc. Actions

openQA Project - Ready

open

Next tasks to be picked up by the QE tools team

98%

2590 issues   (2544 closed — 46 open)

Related issues
action #12506: openqa-scripts on osd/o3 should be deployed automatically (CD) Actions
action #18142: [labs] Create wiki with up to date information (linklist included) Actions
action #19190: make use of ix64ph1014, e.g. for proxymode Actions
action #19238: setup pool devices+mounts+folders with salt(was: ext2 on workers busted) Actions
action #29015: [o3][monitoring] too often alert from nagios about swap space depleted; alarm notification disabled -> increase/disable swap? Actions
action #37644: [tools] osd SSL certificate is only valid for openqa.suse.de, not for openqa.nue.suse.com Actions
action #44078: Implement proper backups for o3 size:M Actions
action #44612: Do we want to update http://tumblesle.qa.suse.de/ or decommision it? Actions
action #49694: openqaworker7 lost one NVMe Actions
action #51836: Manage (parts) of s390 kvm instances (formerly s390p7 and s390p8) with salt Actions
action #53471: machine aarch64.o.o often unresponsive and needs power-cycle Actions
action #57233: Properly handle transport errors in ObsRsync plugin Actions
action #57680: o3: optimize fs utilization and resize root disk (was: change /var/log to bind mount to prevent out-of-space) Actions
action #59300: auto_review:"DBus.*: The name org.opensuse.os_autoinst.switch was not provided":retry GRE tunnel settings not applied on initial setup / after reboot Actions
action #59933: Prevent depletion of space on / with extra partition or quota on /home on o3 size:M Actions
action #61994: VNC console corruption on aarch64 Actions
action #63142: Upgrade firmware of ppc9 machine redcurrant Actions
action #64279: [virtualization][OS upgrade] upgrade xen host openqaw5-xen.qa.suse.de Actions
action #64685: openqaworker1 showing NVMe problems "kernel: nvme nvme0: Abort status: 0x0" Actions
action #64700: setup o3 workers openqaworker4 and openqaworker7 for multi-machine tests size:S Actions
action #64941: after every reboot openqaworker7 is missing var-lib-openqa-share.mount , check dependencies of service with openqaworker1 Actions
action #64983: Staging machine openqa-staging-1.qa.suse.de does not boot after update to Leap 15.1 Actions
action #65130: Upgrade of firmware(s) for cloudberry (power9 machine) Actions
action #65154: root partition on osd exceeds alert threshold, 90%, after osd deployment -> apply automatic reboots to OSD machines Actions
action #65178: Drop rsync.pl config from salt for osd and o3 Actions
action #65975: Monitoring for "scheduled but not executed" (was: perl-Mojolicious-8.37 broke WS connection for (some?) workers) Actions
action #66709: Storage server for OSD and monitoring Actions
action #67804: use non-personal account and key for pushing needles on osd to gitlab.suse.de Actions
action #68053: powerqaworker-qam-1 fails to come up on reboot (repeatedly) Actions
action #68077: alert about too many failed minion jobs but https://openqa.suse.de/minion/jobs?state=failed shows none Actions
action #68095: Migrate osd workers from SuSEfirewall2 to firewalld Actions
action #68161: alert raised about "Minion workers down" after osd-deployment 2020-06-17 Actions
action #68410: repeated alerts about "no data" that are not actionable and recover themselves often enough Actions
action #68785: [monitoring] Setup of QA generic monitoring instance Actions
action #68872: job age max exceeds alarm threshold Actions
action #69202: icinga alert "openqaworker3.suse.de/Number of threads" Actions
action #69475: [tools] openQA child task fails to download asset created by parent job Actions
action #69523: lessons learned: osd did not come up after reboot 2020-08-02 Actions
action #69577: Handle installation of the new "Storage Server" Actions
action #69610: ipmi management interface of openqaworker-arm-3 is inaccessible Actions
action #69613: osd-pre-deployment checks fail due to invalid certificates for stats.openqa-monitor.qa.suse.de Actions
action #69664: [osd][alert] CPU usage alert: IOwait too high Actions
action #69667: missing monitoring data for vde after partitions where reordered Actions
action #69694: openqa-worker systemd services running in osd which should not be enabled at all and have no tap-device configured auto_review:"backend died:.*tap.*is not connected to bridge.*br1":retry Actions
action #69721: autoinst.qa is pingable and resolves to snipe.qa.suse.de, a VM, whereas there is a physical machine as well Actions
action #69727: reduce heat in NUE-SRV2 Actions
action #70768: obs_rsync_run and obs_rsync_update_builds_text Minion tasks fail frequently Actions
action #70834: [alert] Refine I/O time alerts for OSD Actions
action #70843: OSD automatic deployment failed in Sept, 02, 2020 Actions
action #70885: [osd][alert] flaky file system alert: /assets Actions
action #70909: salt CI jobs fail due to (now) missing python3 package Actions
action #70918: gitlab pillar merge requests try to execute CI tests and fail because tests from states are not compatible Actions
action #70939: [alert] ** PROBLEM Service Alert: ariel-opensuse.suse.de/root partition is WARNING ** Actions
action #70966: ipmi management interface of openqaworker-arm-3 is inaccessible Actions
action #70969: openqaworker-arm-2 stuck in system management menu after reboot Actions
action #70975: [alert] too many failed minion jobs Actions
action #70978: automatic reboots on o3 to activate new kernel versions Actions
action #71011: [alert] workers alert Actions
action #71098: openqaworker3 down but no alert was raised Actions
action #71182: OBS package os-autoinst blocked since multiple days on "blocked: downloading 6 dod packages" Actions
action #71191: inform EngInfra automatically if the IPMI interfaces are not accessible Actions
action #71224: [osd][alert] NTP offset alert Actions
action #71332: [alert] failed systemd service on openqaworker6: "display-manager" Actions
action #71575: [osd][alert] limited /assets - idea: ask EngInfra for slow+cheap storage from central server for /assets/fixed only Actions
action #71635: [osd][alert] QA-Power8-4-kvm: CPU Usage alert - alert needs to be refined Actions
action #71893: can not reach openqaworker-arm-2-ipmi.suse.de DNS anymore from a gitlab CI pipeline Actions
action #71908: [osd][alert] CPU usage alert (High IO wait on OSD) Actions
action #71962: [osd-admins] openqaworker-arm-1: MergePoint email alert " Voltage sensor, warning event was asserted" Actions
action #72136: [osd-admins] [Alerting] Workers alert during osd deployment, then "ok" after 1 minute, should not alert Actions
action #72139: openQA services on OSD failed to connect to database Actions
action #73165: [osd] Consolidate "expensive+fast" and "cheap+slow" storage after realizing vdc is "cheap+slow" as well Actions
action #73174: [osd][alert] Job age (scheduled) (median) alert Actions
action #73183: Extend vdc (/assets) Actions
action #73189: Upgrade o3 workers to openSUSE Leap 15.2 after openqa-aarch64 already done Actions
action #73228: [labs] Create guide for new machines (from the "customer perspective") Actions
action #73234: ipmi management interface of openqaworker7 is inaccessible ("no matching cipher suite") and not pingable Actions
action #73246: [osd-admins][alert] openqaworker8: Memory usage alert Actions
action #73333: Failed systemd services alert (workers) flaky Actions
action #73342: all jobs run on openqaworker8 incomplete: "Cache service status error from API: Minion job .*failed: .*(database disk image is malformed|not a database)":retry Actions
action #73348: The worker openqaworker-arm-2.suse.de is down, so the OSD deployment fails Actions
action #73360: merging MRs in openqa/salt-states-openqa does not trigger pipeline in master Actions
action #73387: Cleanup of old needles from os-autoinst-needles-opensuse and os-autoinst-needles-sle Actions
action #73399: [osd][alert] Failed systemd services alert (workers) - many alert messages Actions
action #73405: job incompletes with "(?s)openqaworker8.*terminated prematurely.*OpenCV Error: Insufficient memory" Actions
action #73483: ipmitool from openSUSE Leap 15.2 and TW to our openQA workers fails with "no matching cipher suite", works on login.suse.de (SLES15SP1) Actions
action #73501: Bind mounts of fixed assets is racy Actions
action #73633: OSD partially unresponsive, triggering 500 responses, spotty response visible in monitoring panels but no alert triggered (yet) Actions
action #75016: [osd-admins][alert] Failed systemd services alert (workers): os-autoinst-openvswitch.service (and var-lib-openqa-share.mount) on openqaworker-arm-2 and others Actions
action #75055: grenache-1 can't connect to webui's over IPv4 only Actions
action #75067: save_needle minion task fails because "Your account has been blocked" Actions
action #75220: all jobs run on openqaworker8 incomplete: "Cache service status error from API: Minion job .*failed: .*(database disk image is malformed|not a database)":retry Actions
action #75235: container image devel:openQA:ci/base is unresolvable with "nothing provides this-is-only-for-build-envs needed by cmake-mini" Actions
action #75238: Upgrade osd workers and openqa-monitor to openSUSE Leap 15.2 Actions
action #75241: Upgrade o3 webUI host to openSUSE Leap 15.2 Actions
action #75244: Upgrade osd webUI host to openSUSE Leap 15.2 Actions
action #75259: 100% of powerpc tests incomplete auto_review:"(?s)Running on power8.*qemu-system-ppc64: Requested safe cache capability level not supported by kvm":retry Actions
action #75274: [osd-admins][alert][learning] Failed systemd services alert (workers): os-autoinst-openvswitch.service aborts retries after 60s and is not easily configurable Actions
action #75277: enable "postfix" service again after #73633 resolved Actions
action #75403: salt "cheat sheet" for common commands to change/fix osd infrastructure Actions
action #75445: unknown dashboards for "linux-fwcx" and "localhost" reappearing on monitor.qa Actions
action #75448: OSD deployment fails because openqaworker-arm-3.suse.de is not connected Actions
action #76774: auto-review/openqa-label-known-issues does not conclude within GitLab CI Actions
action #76783: research how hostnames with systemd work and make them static for all OSD related machines Actions
action #76786: Configure static hostnames with salt for all salt nodes Actions
action #76804: openqaworker-arm-2 was detected as offline but no grafana webhook action was triggered Actions
action #76822: Fix /results over-usage on osd (was: sudden increase in job group results for SLE 15 SP2 Incidents) Actions
action #76828: big job queue for ppc as powerqaworker-qam-1.qa and malbec.arch and qa-power8-5-kvm were not active Actions
action #76876: Find a better (automated) way to inform infra about hanging (arm) workers Actions
action #76903: openQA test modules failure statistics collection Actions
action #76924: Upgrade postgresql database version on o3 to default of Leap 15.2, i.e. postgres12 size:M Actions
action #76927: Upgrade postgresql database version on osd to default of Leap 15.2, like on o3 size:M Actions
action #76951: Check if new firmware for kerosene (aka. power8.o.o) exists and remove os-autoinst workarounds again when according machine settings are applied when necessary size:M Actions
action #76987: re-encode some videos from existing results to save space Actions
action #77017: Add "how to do rollback of deployment" to https://gitlab.suse.de/openqa/osd-deployment/ Actions
action #77089: [osd][retrospective] multiple unattended alerts, unattended gitlab CI pipeline fails, all osd aarch64 workers offline Actions
action #77101: fix selection of gitlab CI runners Actions
action #77191: [easy][learning] ** PROBLEM Service Alert: openqaworker12.suse.de/Interface br1 is CRITICAL ** Actions
action #77209: workers on o3 machine rebel provide no "WORKER_HOSTNAME" value anymore but it shows up in journal of worker service Actions
action #77218: gitlab CI pipeline openqa/grafana-webhook-actions failed with "You have reached your pull rate limit" for docker hub but should not use docker hub at all Actions
action #77827: asg-qe.maintenance reported problems about too aggressive asset cleanup - bumping some settings Actions
action #77836: login to aarch64.o.o fails with ssh keys and password, also not working over IPMI SoL Actions
action #77839: Use external videoencoder in production on one o3 worker Actions
action #77842: Use external videoencoder in production on all o3 machines size:M Actions
action #77845: Use external videoencoder in production on all osd machines size:M Actions
action #77848: No more workarounds in OSD due to inefficient video encoder size:M Actions
action #77887: [tools][openqa] Enable automatic openQA investigation jobs for osd as well Actions
action #77890: [easy] Extend OSD storage space for "results" to make bug investigation and failure archeology easier Actions
action #78010: unreliable reboots on openqaworker3, likely due do openqa_nvme_format (was: [alert] PROBLEM Host Alert: openqaworker3.suse.de is DOWN) Actions
action #78058: [Alerting] Incomplete jobs of last 24h alert - again many incompletes due to corrupted cache, on openqaworker8 Actions
action #78064: failing logrotate on monitor.qa.suse.de due to mariadb/mysql? Actions
action #78127: follow-up to #73633 - lessons learned and suggestions Actions
action #78165: infrastructure task: After osd deployment 2020-11-18 many jobs incomplete with auto_review:"Cache service (status error from API|.*error 500: Internal Server Error)":retry Actions
action #78218: [openQA][worker] Almost all openQA workers become offline Actions
action #78438: openQA webui entry "Assigned worker" shows ip instead of names as formerly - manual cleanup work Actions
action #80106: corrupted worker cache sqlite: Enlarge systemd service kill timeout temporarily Actions
action #80128: openqaworker-arm-2 fails to download from openqa Actions
action #80166: IPMI SOL does not show me anything anymore during bootup until the linux getty login prompt, can anybody confirm? Actions
action #80178: gitlab.suse.de CI shared runners are unable to resolve addresses within suse.de domain Actions
action #80380: salt deploy often or always fails in test-webui and deploy due to postgres files not found Actions
action #80398: [openQA] increase storage space for baremetal_support VM on qamaster Actions
action #80408: revert longer timeout override for openQA services as we could not see less problems with corrupted worker cache Actions
action #80538: flaky and misleading alerts about "openQA minion workers alert" as well as "Minion Jobs alert" Actions
action #80542: Configure "automatic power-on" after power loss for openqaworker1 (and all others) Actions
action #80544: Ensure that IPMI for powerqaworker-qam works reliably Actions
action #80594: Needles are not pushed from o3 to github repo Actions
action #80656: OSD deployment failed at 2020-12-02 because 'malbec.arch.suse.de' is down Actions
action #80688: Upgrade IO firmware for powerqaworker-qam-1 Actions
action #80734: GitLab pipeline trigger via Grafana fails due to TLS errors Actions
action #80768: All workers in grenache-1 are broken at 2020-12-07 Actions
action #80770: "host up" alerts alerting on Sunday around 03:10Z and all going back to ok 30m later, should be prevented Actions
action #80812: Fix mail sending on o3 size:S Actions
action #80834: [alert] osd reported 502 errors, unresponsive, failing alerts for CPU usage since 2020-12-08 0840Z and minion jobs failed Actions
action #80956: update osd VM root disk VM config and resize root disk on osd Actions
action #81020: QA-Power8-4-kvm start failed since reboot on 2020-12-13 Actions
action #81026: many jobs incomplete with auto_review:"(?s)Running on openqaworker-arm-2.*failed: 521 Connect timeout.*Result: setup failure":retry Actions
action #81046: openqaworker-arm-2.suse.de unreachable Actions
action #81058: [tracker-ticket] Power machines can't find installed OS. Automatic reboots disabled for now Actions
action #81198: [tracker-ticket] openqaworker-arm-{1..3} have network problems (cacheservice, OSD reachability). IPv6 disabled for now Actions
action #81220: overdrive2.arch wants to be accepted with salt key on osd, what to do with this machine? Actions
action #81228: many o3 workers not working on jobs as of 2020-12-21 (w4 seems to be ok) Actions
action #81232: salt high state triggers multiple errors Actions
action #81274: consider disabling emergency mode on our machines Actions
action #81884: openqa-webui should automatically restart on config updates Actions
action #87838: *alert* osd: Open database connections by user alert Actions
action #87856: [alert] openqaworker3.suse.de/NRPE is CRITICAL Actions
action #87883: osd infrastructure: services like "telegraf" are not enabled to start immediately on boot Actions
action #87970: tumbleweed container images in gitlab CI fail on salt calls, e.g. "KeyError: 'cmd.run_all'" Actions
action #87979: Requirements for access to the openqa VM Actions
action #88189: Firmware upgrade of qa-power8-4.qa.suse.de Actions
action #88191: openqaworker2 boot ends in emergency shell Actions
action #88225: osd infrastructure: Many failed systemd services on various machines Actions
action #88385: openqaworker3 host up alert is flaky Actions
action #88450: Flaky NTP offset alert Actions
action #88474: All workers on powerqaworker-qam-1 are offline Actions
action #88546: Make use of the new "Storage Server", e.g. complete OSD backup Actions
action #88807: Open vSwitch command 'set_vlan' with arguments 'tap41 13' failed: 'tap41' is not connected to bridge 'br1' at /usr/lib/os-autoinst/backend/qemu.pm line 152. Actions
action #88900: openqaworker13 was unreachable Actions
action #88912: Assets cleaned up too early for Tumbleweed aarch64 Actions
action #89047: Failed to commit needles, gitlab account blocked 2021-02-24 Actions
action #89050: OSD deployment blocked since two weeks - salt nodes failing Actions
action #89113: [alert] PROBLEM Service Alert: openqa.suse.de/NTP Time is CRITICAL Actions
action #89275: Flaky Incomplete jobs (not restarted) of last 24h alert and New incompletes alert triggered and back to OK Actions
action #89419: Incomplete jobs after OSD deployment Actions
action #89497: flaky Failed systemd services alert (except openqa.suse.de) Actions
action #89551: NFS mount fails after boot (reproducible on some OSD workers) Actions
action #89815: osd-deployment blocked by openqaworker-arm-3 offline and not recovered automatically Actions
action #89821: alert: PROBLEM Service Alert: openqa.suse.de/fs_/srv is WARNING (flaky, partial recovery with OK messages) Actions
action #89993: OSD deployment rollback failed finding "before" and "osd_deployment_rpm_q" files Actions
action #90161: [Alerting] malbec: Memory usage alert triggered briefly and turned OK within the next minute Actions
action #90170: Service for purging old kernels might run while system management is locked and fail Actions
action #90275: Replacement openQA OSD aarch64 hardware (was: Dedicated non-rpi aarch64 hardware for manual testing) Actions
action #90629: administration of the new "Storage Server" Actions
action #90635: NTP alerts coinciding with reboots of ppc64le host Actions
action #90746: OSD deployment fails at 2021-04-07 because 'storage.qa.suse.de: Minion did not return' Actions
action #90755: aarch64 openQA jobs on osd not properly processed since 2021-04-05 Actions
action #90857: Add redundancy for SAP multi machines tests - Extend openQA worker config to accomodate for upgraded RAM Actions
action #90875: powerqaworker-qam-1 is online but https://monitor.qa.suse.de/d/4KkGdvvZk/osd-status-overview shows "No Data" Actions
action #90920: Restore IPMI access to malbec.arch.suse.de Actions
action #90968: [alert] Multiple flaky incomplete job alerts on Sunday Actions
action #91530: Severe performance problems on malbec Actions
action #91779: Add monitoring for storage.qa.suse.de Actions
action #91803: openqaworker8 failed to reboot: [Alerting] openqaworker8: host up alert on 2021-04-25 Actions
action #92034: Re-enable openqa-investigate options after the black certificate now only shows properly "reviewed" jobs Actions
action #92086: http://monitor.qa should redirect to https://monitor.qa.suse.de Actions
action #92110: Several Job age (scheduled) alerts on Sunday Actions
action #92113: [Alerting] openqaworker-arm-3: NTP offset alert Actions
action #92167: remove workaround for PowerPC workers after boo#1174166 fixed and updates available Actions
action #92176: [alert] openqaworker-arm-3 offline and CI pipeline unable to send email but stating "passed" Actions
action #92185: Our PowerPC worker machines need to be configured for automatic start after a power outage Actions
action #92302: NFS mount var-lib-openqa-share.mount often fails after boot of some workers Actions
action #92338: [Alerting] File systems alert, / on osd Actions
action #92467: Unit has `iscsid.socket` failed on some OSD workers since today's nightly reboot Actions
action #92701: backup of etc/ from both o3 was not working since some days due to OOM on backup.qa.suse.de (was: … and osd not updated anymore since 2019) Actions
action #92770: openqa.opensuse.org down, o3 VM reachable, no failed service Actions
action #92915: OSD deployment fails at 2021-05-21 because ' openqaworker-arm-2.suse.de Minion did not return' Actions
action #92969: Failing service os-autoinst-openvswitch after boot of some workers Actions
action #92978: salt failed to write files on grenache-1.qa.suse.de 'workers.ini': Operation not permitted Actions
action #92996: OSD deployment fails at 2021-05-24 because ' openqaworker-arm-3.suse.de Minion did not return' Actions
action #93050: Proposal: Use openqaworker11 and openqaworker12 as normal workers and only pull out from production when necessary Actions
action #93071: o3 s390x worker machines unreachable Actions
action #93195: [Alerting] Failed systemd services alert (except openqa.suse.de) on 2021-05-28, logrotate.service on openqaworker-arm-1 Actions
action #93381: [O3]request to add an IPMI SUT to O3 size:M Actions
action #93612: Several unhandled alerts triggered regarding incompletes and running out of space Actions
action #93650: alert: PROBLEM Service Alert: openqa.suse.de/fs_/assets is WARNING Actions
action #93683: osd-deployment failed due to storage.qa.suse.de not reachable by salt Actions
action #93919: salt gitlab CI pipeline fails applying openqa-webui restart state with "The following requisites were not found: watch: file: /etc/openqa/openqa.ini" Actions
action #93922: grafana dashboard for "approximate result size by job group" fails to render any data with "InfluxDB Error: unsupported mean iterator type: *query.stringInterruptIterator" Actions
action #93961: Add redundancy for SAP multi machines tests - extend RAM on machines Actions
action #93964: salt-states CI pipeline deploy step fails on some workers with "Unable to unmount /var/lib/openqa/share: umount.nfs: /var/lib/openqa/share: device is busy." Actions
action #94015: proper backup for osd Actions
action #94237: No alert about too many scheduled tests size:S Actions
action #94312: [Alerting] web UI: Too many Minion job failures alert - likely due to openqa-client declared deprecated Actions
action #94318: http://jenkins.qa.suse.de not reachable Actions
action #94438: OSD deployment fails at 2021-06-21 because ' openqaworker (arm-3 and arm-2) Minion did not return' Actions
action #94456: no data from any arm host on https://stats.openqa-monitor.qa.suse.de/d/1bNU0StZz/automatic-actions?orgId=1 Actions
action #94492: Configure retention/downsampling policy for monitoring data stored within InfluxDB size:M Actions
action #94513: openqaworker-arm-3 not reachable and not recoverable over usual ways Actions
action #94555: backup.qa.suse.de needs to be upgraded to Leap 15.2 (size:S) Actions
action #94576: alert: PROBLEM Service Alert: openqa.suse.de/fs_/results is WARNING Actions
action #94747: broken RPM database on openqaworker-arm-* during osd deployment Actions
action #94919: All arm workers down 2021-06-30 , NUE SRV2 Rack A8 was switched off by EngInfra size:S Actions
action #94940: multiple network related problems, gitlab CI pipelines not working, workers not reachable, proxySCC not reachable Actions
action #94949: Failed systemd services alert for openqaworker3 var-lib-openqa-share.automount Actions
action #94994: Tracebacks in the journal for salt-minion size:M Actions
action #95140: regression: salt state fails on one worker with "Passed invalid arguments to state.highstate: expected str, bytes or os.PathLike object, not list" size:M Actions
action #95167: Bring openqaworker12 into production including multi-machine test support Actions
action #95293: Monitoring alerts on errors in logs on o3 (was: followup to: error on "Next & previous results": ajax error message and no results showing up) size:M Actions
action #95443: Variants of Job age (scheduled) alerts on Grafana on Sunday and Monday size:S Actions
action #95482: openqaworker-arm-3 offline and not automatically recovered due to gitlab CI failures Actions
action #95980: "no data" alert for "Incomplete jobs (not restarted)" but there should be no alert for "no data" size:S Actions
action #95983: alert about "minion workers", alert triggered two times and turned green again Actions
action #96089: alert about "minion workers" - make meaning of grafana panel clear size:S Actions
action #96242: [alert] Disk I/O time for /dev/vde (/space-slow) alert 2021-07-28 size:M Actions
action #96269: Define what a "complete OSD backup" should or can include Actions
action #96272: Test failed incomplete: "Reason: backend died: QEMU exited unexpectedly, see log for details" size:M Actions
action #96380: "minion workers" alert shows <1 total minion workers if active == 1 size:M Actions
action #96551: Persistent records of systemd journal size:S Actions
action #96554: Mitigate on-going disk I/O alerts size:M Actions
action #96618: The machine is unable to boot from PXE in NUE QA network due to TFTP open timeout Actions
action #96710: Error `Can't call method "write" on an undefined value` shows up in worker log leading to incompletes Actions
action #96713: Slow grep in openqa-label-known-issues leads to high CPU usage Actions
action #96719: recover imagetester with broken filesystem/hardware (was: automatic updates on imagetester don't work and it failed to come up after reboot) Actions
action #96789: File systems alert 90.256 assets used size:M Actions
action #96795: CPU Load alert and telegraf going between 41%, 98.5% and 115% CPU Actions
action #96807: Web UI is slow and Apache Response Time alert got triggered Actions
action #96938: openqaworker10+13 are offline, reason unknown, let's fix other problems first size:M Actions
action #96947: Availability of openQA webUI is above 100% size:S Actions
action #97043: job queue hitting new record 14k jobs Actions
action #97136: [alert] multiple unhandled alerts about "broken workers" size:M Actions
action #97139: [alert] multiple unhandled alerts about "malbec: Memory usage alert" size:M Actions
action #97244: openqaworker-arm-3 is offline and EngInfra wants us to create JiraSD tickets instead of infra size:M Actions
action #97364: openqaworker-arm-2 and openqaworker-arm-3 seem to be offline, alerts had been triggered size:S Actions
action #97382: ARM automatic reboot pipeline does not fail if ipmitool fails size:S Actions
action #97406: gitlab emails to osd-admins@suse.de are not accepted as the ML is not in "To:" field but CC/BCC Actions
action #97415: handle request about machines in SRV1 from SUSE-IT/Accenture to fill a spreadsheet with system properties Actions
action #97418: Pipeline in salt-states-openqa can fail occasionally size:M Actions
action #97502: osd deployment failed due to openqaworker-arm-3 being down, needs to be worked around size:M Actions
action #97574: deployment failed in gitlab job with "ERROR: Job failed: execution took longer than 1h0m0s seconds" Actions
action #97583: [spike] Configure retention/downsampling policy for monitoring data stored within InfluxDB on newer influxdb version Actions
action #97658: many (maybe all) jobs on rebel within o3 run into timeout_exceeded "setup exceeded MAX_SETUP_TIME" size:M Actions
action #97751: replacement setup for o3 s390x openQA workers size:M Actions
action #97862: More openQA worker hardware for OSD size:M Actions
action #97943: Increase number of CPU cores on OSD VM due to high usage size:S Actions
action #98243: salt-states-openqa pipeline/job "test-worker" fails because zypper reports an error size:M Actions
action #98307: Many jobs in o3 fail with timeout_exceeded on openqaworker1 auto_review:"timeout: setup exceeded MAX_SETUP_TIME":retry size:M Actions
action #98499: [alert] web UI: Too many Minion job failures alert size:S Actions
action #98922: Run asset cleanup concurrently to results based on config Actions
action #98979: monitor-post-deployment failed while arm3 was being rebooted by our automatic recovery Actions
action #99045: osd deployment fails due to zypper package resolution problem Actions
action #99117: malbec 🍷️ is not reachable via ssh or ipmi Actions
action #99189: Upgrade o3 workers to openSUSE Leap 15.3 size:M Actions
action #99192: Upgrade osd workers and openqa-monitor to openSUSE Leap 15.3 size:M Actions
action #99195: Upgrade o3 webUI host to openSUSE Leap 15.3 size:M Actions
action #99198: Upgrade osd webUI host to openSUSE Leap 15.3 size:M Actions
action #99201: Upgrade postgresql database version on o3 to default of Leap 15.3, i.e. postgres14 size:M Actions
action #99204: Upgrade postgresql database version on osd to default of Leap 15.3, like on o3 Actions
action #99240: Upgrade CI container image versions to Leap 15.3 size:M Actions
action #99243: Upgrade qam hosts maintained by us to latest stable, i.e. Leap 15.3 Actions
action #99288: [Alerting] openqaworker-arm-5 and openqaworker-arm-4: host up alert on 2021-09-26 size:M Actions
action #99333: qa-maintenance/openQABot CI job fails after max retries to reach germ159.suse.cz Actions
action #99741: Minion jobs for job hooks failed silently on o3 size:M Actions
action #100712: Investigate what broke git checkouts on o3 Actions
action #100859: investigate how to optimize /srv data utilization on OSD size:S Actions
action #100862: Failed systemd services alert: session-50177.scope failed - but why? Actions
action #101006: Provide unique non-dictionary passwords for all our IPMI/HMC interfaces size:S Actions
action #101033: openqaworker13: Too many Minion job failures alert - sqlite failed: database is locked size:M Actions
action #101166: o3 was not auto-updated lately, due to unresolvables in devel:openQA for Leap 15.3 Actions
action #101271: Try Kernel:stable on arm4+arm5 and compare failure rate size:M Actions
action #101373: worker openqa-aarch64 fails on cache Actions
action #101962: update BMC/IPMI firmware on openQA hosts Actions
action #102062: check state of packages fails with test '' = clean Actions
action #102143: o3 ran out of disk space Actions
action #102284: openQABot pipeline failed with simplejson.errors.JSONDecodeError: Expecting value: line 1 column 1 (char 0) Actions
action #102575: Prevent false-positive ticket reporting for openqaworker-arm-3 Actions
action #102650: Organize labs move to new building and SRV2 size:M Actions
action #102713: Implement proper backups for o3 - list of installed packages Actions
action #102716: Disable job triggering during the weekly SUSE IT maintenance window Actions
action #102942: Failed systemd services alert: snapper-cleanup on QA-Power8-4-kvm fails size:M Actions
action #103128: gitlab CI pipelines sporadically fail with "Could not resolve host: gitlab.suse.de", e.g. Recovery pipelines for ARM workers might fail during the maintenance window size:M Actions
action #103149: Salt the dehydrated setup Actions
action #103518: opencv update broke os-autoinst (was: Tests on Raspberry Pi 2/3/4 are broken) Actions
action #103524: OW1: performance loss size:M Actions
action #103530: failed systemd services alert - openqaworker-arm-3 - ovsdb-server size:M Actions
action #103539: Update expired SSL certificate on monitor.qa.suse.de with dehydrated and salt, same as on OSD size:M Actions
action #103554: o3 s390x worker instances 102+103 down whereas 101+104 are up Actions
action #103575: [virtualization][3rd party hypervisor] Worker openqaw8-vmware.qa.suse.de is not reachable Actions
action #103602: Reply to gitlab@suse.de pipeline email fails silently Actions
action #103683: [tools][sle][x86_64][aarch64][QEMUTPM] install package "swtpm" on x86_64 and aarch64 workers Actions
action #103736: Make aarch64 machine chan-1 and chow up and running after it is broken size:M Actions
action #103848: No permission to view SD tickets filed for broken arm workers Actions
action #103954: Run asset cleanup concurrently to results based on config on o3 as well Actions
action #104085: openQABot pipeline failed with terminating connection due to administrator command size:S Actions
action #104088: bot-ng pipeline consistently fails with KeyError: 'test_issues' Actions
action #104106: [qe-core] test fails in await_install - Network peformace for ppc installations is decreasing size:S Actions
action #104142: osd-deployment pipeline failed: File ... not found on medium Actions
action #104172: osd service ca-certificates failed with "p11-kit: couldn't complete writing of file: /var/lib/ca-certificates/ca-bundle.pem.tmp: File exists" Actions
action #104217: Ask eng infra why thruk.suse.de stopped working Actions
action #104304: Crosscheck results of https://github.com/os-autoinst/os-autoinst#verifying-a-runtime-environment on arm-1/2/3 vs. arm-4/5 to find out if arm-4/5 are "typing stable" size:M Actions
action #104344: Inconsistent systemd default target in OSD infrastructure Actions
action #104347: Lost salt connection from some hosts in OSD infrastructure due to no route to host "salt" Actions
action #104607: Verify fix for https://bugzilla.suse.com/show_bug.cgi?id=1192126 regarding QEMU+OVMF, ensure updated packages on o3 size:M Actions
action #104673: Access to o3 workers is not well-documented and not automated Actions
action #104835: Failed systemd service "systemd-journal-flush" on OSD Actions
action #104869: Packages not building on Leap 15.2 and SLE15-SP3 Actions
action #104965: openqaworker10 restarted into maintenance mode - reason unknown Actions
action #104967: File systems alert repeatedly triggering on and off Actions
action #104970: Add two OSD workers (openqaworker14+openqaworker15) specifically for sap-application testing size:M Actions
action #104992: [virtualization][3rd party hypervisor] vmware test run failed with credential errors Actions
action #105169: Pipeline of openQABot project fails with "urllib.error.HTTPError: HTTP Error 503: Service Unavailable" causing alert/notification Actions
action #105274: Workers constantly logging websocket connection "finished by remote side with code 1006, no reason" Actions
action #105373: Ask to increase OSD /srv so that we can save enough logs+DB Actions
action #105594: Two new machines for OSD and o3, meant for bare-metal virtualization size:M Actions
action #105603: openQABot pipeline failed: "ERROR:root:Something bad happended during reading MR data from SMELT/IBS: Expecting value: line 4 column 1 (char 3)" size:M Actions
action #105618: [Alerting] CPU Load alert size:S Actions
action #105621: [Alerting] Failed systemd services alert Actions
action #105828: 4-7 logreport emails a day cause alert fatigue size:M Actions
action #105960: Dehydrated fails on OSD size:M Actions
action #106017: [ipmi backend]VNC stalled, no update for 5.38 seconds Actions
action #106035: [qe-tools] dehydrated service fails on osd Actions
action #106365: Improve security for OSD worker credentials broke Gitlab CI/CD deploy of salt in OSD size:M Actions
action #106538: lessons learned "five whys" for "All OSD PPC64LE workers except malbec appear to have horribly broken cache service" size:S Actions
action #106540: Mitigate/resolve All OSD PPC64LE workers except malbec appear to have horribly broken cache service Actions
action #106543: Conduct rollback steps and check impact for "All OSD PPC64LE workers except malbec appear to have horribly broken cache service" size:M Actions
action #106594: [tools] openqaworker-arm-3 periodically fails os-autoinst-openvswitch service Actions
action #106598: Redcurrant has a broken HDD Actions
action #106607: GitHub review notifications are sent to osd-admins+os-autoinst-obs@suse.de Actions
action #106666: Improve worker startup in our salt states or "openqa-worker-auto-restart repeatedly failing on grenache-1.qa.suse.de" Actions
action #106751: Update machines and passwords in the monitor-o3 repository Actions
action #106753: openqa-worker-auto-restart repeatedly failing on grenache-1.qa.suse.de Actions
action #106771: imagetester missing in action Actions
action #106832: Monitor masked units on our infrastructure Actions
action #106846: Junk messages posted on osd-admins@suse.de Actions
action #106880: Job template name ... is already used in job group error logged on o3 size:M Actions
action #106919: openqa_install+publish fails in start_test size:M Actions
action #106925: [timeboxed:10h][research] What are best practices and options in the salt and GitLab community to handle secrets Actions
action #106933: Use PSU capabilites to power cycle openqaworker-arm-[1-3] instead of infra tickets size:M Actions
action #107017: Random asset download (cache service) failures on openqaworker1 Actions
action #107074: error on openqaworker-arm-2 failing osd-deployment size:M Actions
action #107077: gitlab.suse.de/openqa/scripts CI checks fail due to xargs missing, update or decommission Actions
action #107083: SUSE QE Tools team must learn about switch administration and get access Actions
action #107086: Ask for volunteers in SUSE QE Tools that would be able to visit the Nbg server rooms, e.g. as second person accompanying nsinger or any potential new admin Actions
action #107089: Make SUSE QE Tools team aware that we need to support EngInfra due to limited capacity Actions
action #107152: [osd] failing systemd services on "grenache-1": "openqa-reload-worker-auto-restart@10, openqa-reload-worker-auto-restart@21, openqa-reload-worker-auto-restart@22, openqa-reload-worker-auto-restart@23, openqa-reload-worker-auto-restart@25, …" size:M Actions
action #107158: [osd] failing systemd service on "storage": "systemd-udev-settle" size:M Actions
action #107257: [alert][osd] Apache Response Time alert size:M Actions
action #107437: [alert] Recurring "no data" alerts with only few minutes of outages since SUSE Nbg QA labs move size:M Actions
action #107515: [Alerting] web UI: Too many Minion job failures alert size:S Actions
action #107638: OSD deployment failed due to openqaworker-arm-1 with "rpmdb2solv: inconsistent rpm database" despite our repair attempts Actions
action #107875: [alert][osd] Apache Response Time alert size:M Actions
action #107917: Recovery of imagetester via IPMI failed size:M Actions
action #107932: Handling broken RPM databases does not handle certain cases Actions
action #107989: CPU-specific worker classes Actions
action #108401: Can't call method "id" on an undefined value at V1/Job.pm Actions
action #108533: o3 logreports DBI Exception: DBD::Pg::st execute failed: ERROR: invalid input syntax for type integer Actions
action #108665: openqa_from_containers repeatedly failing at build Actions
action #108668: Failed systemd services alert (except openqa.suse.de) for < 60 min Actions
action #108671: Resilient IPMI recovery of o3 machines in monitor-o3 size:M Actions
action #108740: qa-power8-5-kvm minions alert is heart-broken 💔️ Actions
action #108743: qa-power8-5-kvm minions alert is heart-broken Actions
action #108845: Network performance problems, DNS, DHCP, within SUSE QA network auto_review:"(Error connecting to VNC server.*qa.suse.*Connection timed out|ipmitool.*qa.suse.*Unable to establish)":retry but also other symptoms size:M Actions
action #108896: [ppc64le] auto_review:"(?s)Size of.*differs, expected.*but downloaded.*Download.*failed: 521 Connect timeout":retry Actions
action #108998: openqa_install+publish fails in start_test Actions
action #109028: [openqa][worker][sut] Very severe stability and connectivity issues of openqa workers and suts Actions
action #109052: openqa_install+publish fails in dashboard Actions
action #109055: Broken workers alert Actions
action #109241: Prefer to use domain names rather than IPv4 in salt pillars size:M Actions
action #109253: Add monitoring for SUSE QA network infrastructure size:M Actions
action #109298: salt-minion on grenache-1 is not working preventing OSD deployment Actions
action #109301: openqaworker14 + openqaworker15 sporadically get stuck on boot Actions
action #109494: Restore network connection of arm-4/5 size:M Actions
action #109746: Improve QA related server room management, consistent naming and tagging size:M Actions
action #109969: s390zp19 - out of disk space Actions
action #109971: Service `systemd-journal-flush` timed out on OSD size:M Actions
action #110170: Request for s390x worker class with SIE support Actions
action #110266: [alert] Disk I/O time for sr0 size:S Actions
action #110269: [alert] QA-Power8-4-kvm + QA-Power8-5-kvm: Disk I/O time alert size:M Actions
action #110293: Can not ping s.qa.suse.de Actions
action #110296: Machines within .qa.suse.de unavailable (was: Some ipmi workers become offline which affects PublicRC-202204 candidate Build137.1 test run) Actions
action #110301: alert about o3: ** PROBLEM Service Alert: ariel-opensuse.suse.de/swap is CRITICAL ** size:S Actions
action #110494: alert: openqaworker5 host up size:M Actions
action #110521: Improve QA related server room management, network topology and configuration size:M Actions
action #110539: Ask OBS team if they would like to swap ARM workers with us Actions
action #110920: Emails from o3 are rejected by mx2.suse.de for certain sender/recipients size:S Actions
action #111063: Ping monitoring for our s390z mainframes size:S Actions
action #111149: Recover openqaworker-arm-3 Actions
action #111156: No effect of remote PDU controls on openqaworker-arm-4.qa and openqaworker-arm-5.qa, check power connections Actions
action #111159: Check if we can give away our left-over 10G cisco switch Actions
action #111171: Handle installation of new FC switch size:M Actions
action #111440: Avoid OSD deployment monitoring to fail due to WIP dashboard alerts Actions
action #111473: Get replacements for imagetester and openqaworker1 size:M Actions
action #111578: Recover openqaworker-arm-4/5 after "bricking" in #110545 size:M Actions
action #111755: power8 fails to execute jobs successfully, no kvm, but also no sshd auto_review:"(?s)power8.*no kvm-img/qemu-img found":retry Actions
action #111758: o3 jobs exceeding MAX_SETUP_TIME auto_review:"(?s)openqaworker4.*timeout: setup exceeded MAX_SETUP_TIME":retry size:M Actions
action #111863: Upgrade o3 workers to openSUSE Leap 15.4 size:M Actions
action #111866: Upgrade osd workers and openqa-monitor to openSUSE Leap 15.4 Actions
action #111869: Upgrade o3 webUI host to openSUSE Leap 15.4 size:S Actions
action #111872: Upgrade osd webUI host to openSUSE Leap 15.4 Actions
action #111884: Upgrade qam hosts maintained by us to latest stable, i.e. Leap 15.4 size:M Actions
action #111926: osd-deployment pipeline failed: test 481 -le 0, due to job age alert, likely just the raspberry pi based tests stuck in schedule Actions
action #111986: Ensure uno.openqanet.opensuse.org is properly used Actions
action #112193: [alert][osd] web UI: Too many Minion job failures alert size:S Actions
action #112196: [alert][sporadic] QA-Power8-4-kvm: Disk I/O time alert size:M Actions
action #112346: [alert] multiple alerts about "Download rate" and "Job age" on OSD 2022-06-12 size:M Actions
action #112553: [osd][amd][zen3][network][sriov] New AMD Zen3 machine on OSD lost its nework connection with p3p1 interface Actions
action #112583: [alert] "Job age" on OSD 2022-06-12 Actions
action #112673: Cannot send emails from o3 size:M Actions
action #112733: Webui Summary dashboard in Grafana is missing I/O panels size:M Actions
action #112781: Connect and switch on autobot for Anton Actions
action #112835: All "developers" in progress QA project hierarchy should be able to change the status of "openqa-force-result" tickets same as for "action" Actions
action #112916: postgresql.conf is invalid after recent salt changes size:M Actions
action #113366: Add three more Prague located OSD workers size:M Actions
action #113477: Get replacements for o3+osd top of rack switch Actions
action #113498: [alert] QA network infrastructure Ping time alert for several hosts Actions
action #113561: failed pipelines for openQABot and bot-ng because of an expired cert Actions
action #113662: Ensure upgrade of vanilla Leap 15.3 os-autoinst can be automatically upgraded to Leap 15.4 size:S Actions
action #113671: [timeboxed][10h] Configure write of I/O panels to be on the negative Y-axis again once we're on grafana 8.4 size:S Actions
action #113746: monitoring: The grafana "ping time" panel does not list all hosts size:S Actions
action #114397: glibc regression causes cron to crash Actions
action #114448: malbec.arch.suse.de is unreachable via IPMI Actions
action #114484: Failed service "transactional-update.service" on openqaworker4 Actions
action #114526: recover openqaworker14 Actions
action #114565: recover qa-power8-4+qa-power8-5 size:M Actions
action #114586: fix openqaworker-arm-1+2+3 recovery pipeline (was: likely stuck in reboot loop) size:M Actions
action #114685: powerqaworker-qam-1 seems to have just gone unresponsive due to unknown reason Actions
action #114697: What are orion and andromeda.o.o Actions
action #114733: openqaworker-arm-3 not consistently reachable Actions
action #114802: Handle "QA network infrastructure Package loss alert" introduced by #113746 size:M Actions
action #114817: salt-states-openqa deployment failed due to SUSE ca issue on openqaworker-arm-2 Actions
action #114908: [tools] https://stats.openqa-monitor.qa.suse.de not responding Actions
action #114914: Container devel/openqa/containers/isotovideo:qemu-x86 is outdated Actions
action #114923: We lost multi-machine capabilities within o3 due to openqaworker1 being replaced Actions
action #114941: repository names inconsistent on o3 workers hence continuous update failed to update as "devel_openQA" as repo name is not found everywhere Actions
action #115094: [tools] test fails in bootloader_start: redcurrant is down size:M Actions
action #115208: failed-systemd-services: logrotate-openqa alerting on and off size:M Actions
action #115226: Use ext4 (instead of ext2) for /var/lib/openqa on qa-power8 workers Actions
action #115418: Setup ow19+20 to be able to run MM tests size:M Actions
action #115484: [alert] OSD deployment failed on 18.08.22 size:M Actions
action #115547: openqaworker20 fails to boot, broken hardware size:M Actions
action #115553: salt-states-openqa pipeline failed: Authentication failed for 'https://gitlab.suse.de/openqa/salt-pillars-openqa/' Actions
action #115580: Reason: abandoned: associated worker openqaworker3:13 re-connected but abandoned the job size:M Actions
action #115733: bot-ng pipeline fails because of empty dictionary in data returned by smelt size:M Actions
action #116060: Recover openqaworker-arm-1 size:M Actions
action #116078: Recover o3 worker kerosene formerly known as power8, restore IPMI access size:M Actions
action #116113: salt responses timing out some of the time size:M Actions
action #116344: openqaw9-hyperv.qa.suse.de (flexo.qa.suse.cz) can not be reached size:M Actions
action #116377: openQABot pipeline failing with KeyError: 'project' Actions
action #116437: Recover qa-power8-5 size:M Actions
action #116494: Too many Minion job failures alert because needle-pusher is blocked on GitLab Actions
action #116563: `salt storage\* test.ping` times out Actions
action #116566: salt-states-openqa: Failed pipeline for master Actions
action #116689: Do not rely on statically configured IPv4 addresses for the salt master in /etc/hosts size:S Actions
action #116722: openqa.suse.de is not reachable 2022-09-18, no ping response, postgreSQL OOM and kernel panics size:M Actions
action #116740: [alert] openqaworker14: host up alert Actions
action #116743: [alert] QA-Power8-5-kvm: host up alert Actions
action #116746: [alert] openqaworker9: host up alert Actions
action #116752: [alert] powerqaworker-qam-1: host up alert Actions
action #116758: Help with adding monitoring for the SLE maintenance update queue size:M Actions
action #116782: o3 s390 workers are offline Actions
action #116794: Bring back grenache.qa.suse.de + grenache-1.qa.suse.de Actions
action #116845: salt-states-openqa CI complains about "fatal: Authentication failed for 'https://gitlab.suse.de/openqa/salt-pillars-openqa/'" but no useful hint size:M Actions
action #116848: Ensure kdump is enabled and working on all OSD machines Actions
action #116911: [openQA][needle] Can not commit new needle for test suite on openqa.suse.de Actions
action #117172: Flaky alert about infrastructure packet loss Actions
action #117205: Some boot_from_pxe failed from assigned worker: grenache-1:17 (kermit) and also openqaworker2:17 (quinn) size:M Actions
action #117229: [tools] openqa failing on worker QA-Power8-5-kvm Actions
action #117262: [alert] failed systemd service: ca-certificates on openqa.suse.de, "p11-kit: couldn't complete writing of file: /var/lib/ca-certificates/ca-bundle.pem.tmp: Unknown error 17" size:M Actions
action #117526: [alert] Dehydrated fails again on OSD Actions
action #117580: web interface of qanet02nue.qa.suse.de and qanet03nue.qa.suse.de can not be reached over neither IPv4 nor IPv6 size:M Actions
action #117625: Fix IPMI connection to openqaworker1.suse.de size:M Actions
action #117631: Failed systemd service transactional-update on openqaworker1 - system is no longer reachable after reboot size:M Actions
action #118024: Ensure all PPC workers are upgraded after kernel regression resolved size:M Actions
action #118885: No cable connection or link to the 2nd network card in amd-zen3-gpu-sut1-1 Actions
action #119179: [openQA][iso] Fail to mount nfs share openqa.suse.de:/var/lib/openqa/share/factory/iso/fixed Actions
action #119215: [openQA][repos][aarch64][15-SP5[Full Media] 15-SP5 Full media is missing from http://openqa.suse.de/assets/repo/ Actions
action #119290: [alert] Packet loss between worker hosts and other hosts alert Actions
action #119488: The QEM dashboard is empty Actions
action #119521: [alert] jenkins: partitions usage (%) alert Actions
action #119557: [qem-dashboard] Show a link to Smelt if there are no active incidents size:S Actions
action #119767: Failed pipeline for "openqa-worker" in salt-states-openqa size:M Actions
action #119938: openQABot | Failed pipeline for master | 9b1b1857: ERROR: Connection error during reading from IBS: HTTP Error 403: Forbidden size:M Actions
action #120004: [alert] Host powerqaworker-qam-1.qa.suse.de is down size:M Actions
action #120007: [alert] Many systemd alerts triggered on 06.11.22 size:S Actions
action #120025: [openQA][ipmi][worker] Worker host hostname changed and broken networking connection Actions
action #120073: [qem-dashboard] Loading incident... indefinitely for non-existing incident Actions
action #120112: worker worker2.oqa.suse.de auto_review:"Error connecting to <root@win2k19.qa.suse.cz>: Connection timed out":retry size:M Actions
action #120163: Use salt grains instead of manually specifying IPs in "bridge_ip" size:M Actions
action #120261: tests should try to access worker by WORKER_HOSTNAME FQDN but sometimes get 'worker2' or something auto_review:".*curl.*worker\d+:.*failed at.*":retry size:meow Actions
action #120270: Conduct the migration of SUSE openQA systems IPMI from Nbg SRV1 to new security zones size:M Actions
action #120339: QEMU DNS fails to resolve openqa.suse.de via IP address Actions
action #120372: OSD deployment fails presumably when handling the changelog Actions
action #120441: OSD parallel jobs failed with "get_job_autoinst_url: No worker info for job xxx available" size:meow Actions
action #120564: [qem-dashboard] Assets are not cleaned up on deployment Actions
action #120615: [alert] "Packet loss between worker hosts and other hosts" i.e. between osd and worker12.oqa.suse.de and worker5.oqa.suse.de Actions
action #120648: [alert] qa-power8-4-kvm host up alert Actions
action #120675: openqaworker-arm-1 not bootable via IPMI Actions
action #120744: [alert] QA-Power8-5-kvm: Too many Minion job failures alert size:M Actions
action #120780: [Alerting] InfluxDB not reachable, turned ok after some minutes Actions
action #120783: [Alerting] failed systemd service on worker11, os-autoinst-openvswitch. Failed at system boot, turned ok after some hours size:M Actions
action #120807: [alert] openqa.suse.de - worker12.oqa.suse.de 100% packet loss due to outdated AAAA record Actions
action #120813: [alert] Salt states fail to apply on `scriptgen.py` invocation Actions
action #120874: openqa_service VM is down size:M Actions
action #120886: "PXE-E32: TFTP open timeout" on ipmi machines in O3 size:S Actions
action #120921: [alert] Salt states fail to compile with "Rendering SLS 'base:openqa.openvswitch' failed: Jinja error: argument of type 'NoneType' is not iterable" size:M Actions
action #120939: [alert] Pipeline for scheduling incidents runs into timeout size:M Actions
action #120967: [alert] o3 webUI down due to regression from https://github.com/os-autoinst/openQA/pull/4922 Actions
action #120973: [qem-dashboard] 500 internal server errors reported by qem-bot Actions
action #121282: Recover storage.qa.suse.de size:S Actions
action #121594: Extend OSD storage space for "results" to make bug investigation and failure archeology easier - 2022 Actions
action #121771: openqaworker20 has no heartbeat Actions
action #121789: MultiMachine tests lose ability to communicate Actions
action #121816: Cannot access installation media on updates.suse.com - maintenance tests broken size:S Actions
action #122158: [alert] qa-power8-4-kvm host up alert - machine not up, nothing obvious on SoL but IPMI works size:M Actions
action #122302: Support SD-105827 "PowerPC often fails to boot from network with 'error: time out opening'" Actions
action #122575: [alert] jenkins host up alert Actions
action #122653: Ask SUSE-IT network admins to REJECT packets instead of DROP so that we get more clear results size:S Actions
action #122656: Ask SUSE-IT network admins to *not* block this traffic which we need for tests regarding s390x within SUSE network size:M Actions
action #122743: Rotate secrets used in CircleCI as per recommendation Actions
action #122746: Publishing opensuse.openqa.job.done failed: SSL connect attempt failed Actions
action #122842: Configure I/O alerts again for the webui after migrating to the "unified alerting" in grafana size:M Actions
action #122845: Migrate our Grafana setup to "unified alerting" Actions
action #122848: Configure grouped alerts in Grafana correctly size:M Actions
action #122983: [alert] openqa/monitor-o3 failing because openqaworker1 is down size:M Actions
action #122998: o3 worker rebel is down; was: inconsistent package database or filesystem corruption size:M Actions
action #123004: Downgrade kernel on o3+osd x86_64 machines as workaround for boo#1206616 size:M Actions
action #123025: o3 worker openqaworker4 is down; boots to emergency shell only Actions
action #123028: A/C broken in TAM lab size:M Actions
action #123082: backup of o3 to storage.qa.suse.de was not conducted by rsnapshot since 2021-12 size:M Actions
action #123151: [Alerting] Failed systemd services alert Actions
action #123232: [Alerting] failed pipelines for openQABot/ bot-ng/ os-autoinst-needles-opensuse-mirror on gitlab staging instance size:S Actions
action #123382: repurpose openqaworker-arm-3 as baremetal worker Actions
action #123421: [alert] dehydrated is failing on monitor.qa.suse.de Actions
action #123490: o3 logreports DBI Exception: DBD::Pg::st execute failed: ERROR: invalid input syntax for type bigint size:M Actions
action #123493: [Alerting] Failed systemd services alert Actions
action #123825: Ensure proper o3 monitoring after shutdown of thruk/icinga by SUSE-IT Eng-Infra Actions
action #123933: [worker][ipmi][bmc] Some worker can not be reached via BMC Actions
action #123984: [boot][pxe][sut] Machine fozzie can not boot from pxe Actions
action #124119: Conduct the migration of remaining SUSE openQA systems IPMI to new security zones Actions
action #124146: [alert] Incomplete jobs (not restarted) of last 24h Actions
action #124391: test fails in bootloader_start - The command server of powerhmc1 cannot be connected size:M Actions
action #124398: salt gitlab CI jobs fail due to exceeding log length of 500kB size:M Actions
action #124412: [alert] logrotate services failed on openqa-piworker.qa.suse.de and OSD size:M Actions
action #124655: [openQA][infra][pxe] Physical SUT machine can not boot from pxe and mismatch hostname Actions
action #124658: [osd-deployment] missing os-autoinst changes in deployment and changelog emails , build failure on ppc64le in OBS size:M Actions
action #124661: [qe-tools] tftp server and directory mount issue on qanet.qa Actions
action #124685: [qe-tools] Make sure Power8 and Power9 machines can be used with *any* usable HMC (was: move to new powerhmc3.arch.suse.de) size:M Actions
action #124715: Failing pipelines because of unreachable machine openqaworker-arm-1 Actions
action #124877: Failing pipelines because of unreachable machine openqaworker-arm-1 Actions
action #125132: [alert] logrotate failed on OSD Actions
action #125207: worker11 host up alert - similar as for worker13 Actions
action #125210: worker13 host up alert - kernel crash size:M Actions
action #125213: Failed systemd services alert due do crash dumps on worker11 and worker13 (except openqa.suse.de) Actions
action #125216: Use qa-power8 for ppc tests in o3 - try one of the suggestions size:M Actions
action #125303: prevent confusing "no data" alerts size:M Actions
action #125306: rebootmgr/D-Bus-related problem on backup.qa.suse.de Actions
action #125468: [alert] [FIRING:1] (Apache Response Time alert J5M8aX04z) then resolved itself so flaky? size:M Actions
action #125531: salt-pillar C pipeline runs into 1h timeout Actions
action #125534: Consolidate the installation of openqaw5-xen with SUSE QE Tools maintained machines size:M Actions
action #125642: Manage "unified alerting" via salt size:M Actions
action #125735: [openQA][infra][pxe] Some machines can not boot from pxe due to "TFTP open timeout" Actions
action #125750: In salt-states-openqa support machines requiring ssh password login for root user size:M Actions
action #125765: Make Telegraf errors visible in alert handling Actions
action #125798: Visual differences in GRUB menu on different x86_64 UEFI workers Actions
action #125810: [openqa][infra] Some SUT machines can not upload logs to worker machine size:S Actions
action #125885: worker10 crashed triggering systemd-services alert and host-up alert size:M Actions
action #126212: openqa.suse.de response times very slow. No alert fired size:M Actions
action #126290: Recover thincsus.qe.nue2.suse.org size:M Actions
action #126674: openqa-review pipeline failed to produce a report with no test results Actions
action #126821: [openQA][infra][worker] ppc64 fails to load grub2 completely over tftp/pxe on qanet and PXE load timeout issues size:M Actions
action #126872: bot-ng pipeline(s) fail(s) to pull openSUSE container images Actions
action #126962: Use templating for all provisioned "unified alerts" were the original alerts were part of templated dashboards Actions
action #127049: gitlab runner system failure: Cannot connect to the Docker daemon Actions
action #127052: [alert] Apache Response Time alert followed by DatasourceNoData for min_apache_response size:M Actions
action #127055: [alert] Download rate alert openQA (openqaworker{14,16,17,18}) Actions
action #127097: [alert] Failed systemd services alert Actions
action #127256: missing nameservers in dhcp response for baremetal machines in NUE-FC-B 2 size:M Actions
action #127274: [alert] Usage of partition mmcblk0p3 on openqa-piworker exceeds threshold size:M Actions
action #127337: Some s390x workers have been failing for all jobs since 11 months ago Actions
action #127754: osd nfs-server needed to be restarted but we got no alerts size:M Actions
action #127877: [alert] Mail "DatasourceNoData Salt" triggered after rebooting OSD-VM Actions
action #127961: FC office access for mkittler Actions
action #127982: [alert][grafana] DatasourceNoData e.g. openQA QA-Power8-5-kvm+openqaw5-xen Disk I/O time alert worker Actions
action #127985: [alert][grafana] Handle various problems with slow monitoring https requests and such (was: DatasourceNoData HTTP Response alert) size:M Actions
action #127991: Fix DNS entry or IPv6 connectivity of qe-jumpy.suse.de Actions
action #128030: IPMI of ix64ph1075.qa.suse.de is not accessible size:M Actions
action #128090: [alert] SUSE nbg network outage 2023-04-20 Actions
action #128273: [alert] openqaworker-arm-1+2+ failed to recover, problem in name resolution, network connection? size:M Actions
action #128417: [alert][grafana] openqaw5-xen: partitions usage (%) alert fired and quickly after recovered again size:M Actions
action #128420: [alert][grafana] 100% packet loss from qa-power8-4-kvm, grenache-1 and powerqaworker-qam-1 to s390zp{11,15,17}.suse.de size:M Actions
action #128561: salt managed host being down does not trigger any alert (was: jenkins.qa.suse.de stuck in emergency mode but no alert) size:M Actions
action #128654: [sporadic] Fail to create an ipmi session to worker grenache-1:16 (ix64ph1075) in its vlan Actions
action #128669: gitlab CI jobs fail when using docker executor with "ERROR: Failed to remove network for build" Actions
action #128786: worker instances on rebel (o3 s390x worker) were not running, services disabled, except for rebel:5 Actions
action #128789: [alert] Apache Response Time alert size:M Actions
action #128927: OSD deployment changelog repeatedly mentions old os-autoinst changes, is a worker outdated and unable to install newer packages? size:M Actions
action #128942: [alert][grafana][openqa-piworker] NTP offset alert Generic (openqa-piworker ntp_offset_alert_openqa-piworker generic) size:M Actions
action #128945: [alert][grafana] web UI: Too many Minion job failures alert Salt (liA25iB4k) Actions
action #128969: [alert][grafana] Failed systemd services alert (except openqa.suse.de) Salt (Uk02cifVkz) Actions
action #128999: openQA workers salt recipes should ensure that also developer mode works size:M Actions
action #129065: [alert] HTTP Response alert fired, OSD loads slow size:M Actions
action #129127: Trying to login on o3 fails with Forbidden Actions
action #129241: [alert] Many alert notification for "QA network infrastructure Ping time alert openQA" received on 12.05.23 13:10 CEST Actions
action #129244: [alert][grafana] File systems alert for WebUI /results size:M Actions
action #129484: high response times on osd - Move OSD workers to o3 to prevent OSD overload size:M Actions
action #129493: high response times on osd - better nice level for velociraptor Actions
action #129604: evil backtick character in ipmi password Actions
action #130132: jenkins.qa.suse.de seems down Actions
action #130201: openqa_bootstrap: fetchneedles not called during openqa-bootstrap size:S Actions
action #130210: [FIRING:1] Packet loss between worker hosts and other hosts alert Salt (2Z025iB4km) Actions
action #130375: Automatically update jenkins plugins on jenkins.qa.suse.de Actions
action #130633: Better documentation on jenkins.qa.suse.de alerts and recovery Actions
action #130790: [alert] failed systemd alert openqa-staging-2 velociraptor-client size:M Actions
action #130835: salt high state fails after recent merge requests in salt pillars size:M Actions
action #130952: [alert][thursday] gitlab.suse.de CI jobs fail with "error: RPC failed; HTTP 500 curl 22 The requested URL returned error: 500" as of 2023-06-15 size:M Actions
action #131021: [O3 repo]Missing openSUSE-Tumbleweed-oss-x86_64-CURRENT directory in /var/lib/openqa/share/factory/repo size:M Actions
action #131096: [alert] Service `ca-certificates` can fail size:M Actions
action #131123: qa-jump.qe.nue2.suse.org is not reachable since 2023-06-19 Actions
action #131141: [alert] Packet loss between qa-jump.qe.nue2.suse.org and other hosts Actions
action #131147: Reduce /assets usage on o3 Actions
action #131150: Add alarms for partition usage on o3 size:M Actions
action #131201: [alert][grafana] HTTP Response alert, alerted at 10:46L, back to green 10:56L size:M Actions
action #131249: [alert][ci][deployment] OSD deployment failed, grenache-1, worker5, worker2 salt-minion does not return, error message "No response" size:M Actions
action #131264: OSD deployment fails to install package os-autoinst-distri-opensuse size:M Actions
action #131276: SUSE Summer 2023 - AC failure in NUE1-SRV1 Actions
action #131303: [alert] Packet loss between worker hosts and other hosts (tumblesle.qa.suse.de) Actions
action #131309: [alert] NFS mount can fail due to hostname resolution error size:M Actions
action #131318: User management via salt does not work on backup.qa.suse.de Actions
action #131321: salt-states-openqa | Failed pipeline for master Actions
action #131459: [openQA][infra] OSD ran out of inodes without triggering a notification size:M Actions
action #131540: openqa-piworker fails to upgrade many packages. vendor change is not enabled as our salt states so far only do that for openQA machines, not generic machines size:M Actions
action #131543: We have machines with both auto-update&auto-upgrade deployed, we should have only one at a time size:M Actions
action #132134: Setup new PRG2 multi-machine openQA worker for o3 size:M Actions
action #132137: Setup new PRG2 openQA worker for osd size:M Actions
action #132143: Migration of o3 VM to PRG2 - 2023-07-19 size:M Actions
action #132218: Conduct lessons learned for "openQA is not accessible" on 2023-07-02 Actions
action #132278: Basic o3 http response alert on zabbix size:M Actions
action #132311: osd-deployment failed at least on qa-power8-5 due to home:tiwai:kernel:5.6 missing Actions
action #132383: FC Basement OSD hosts not reachable since 2023-07-06 01:50 CEST Actions
action #132437: Ensure everybody in SUSE QE Tools knows how to silence alerts in various monitoring systems size:M Actions
action #132461: manage tls certificates on o3/ariel directly with dehydrated size:M Actions
action #132470: salt states fail to apply due to glibc error on storage.oqa.suse.de Actions
action #132500: NUE1-SRV2, .qa.suse.de, aarch64 workers offline due to heat-related SRV2 shutdown size:M Actions
action #132512: Handover manual raspberry pi testing size:S Actions
action #132671: Ensure everybody in SUSE QE Tools knows how to access netbox size:M Actions
action #132752: Use proper bot account for notifications in zabbix.suse.de size:M Actions
action #132773: fail to set iPXE boot for a baremetal machine in FC basement Actions
action #132788: [alert][flaky] QA-Power8-5-kvm: QA network infrastructure Ping time alert Actions
action #132812: [alert] openqaw5-xen host up alert + infrastructure ping size:M Actions
action #132815: [alert][flaky][o3] Multiple flaky zabbix alerts related to o3 Actions
action #132818: salt state for worker in CI test does not apply anymore, "ID: /var/lib/openqa, Function: mount.mounted" size:M Actions
action #132860: openqa-piworker is unstable and needs regular power-cycles size:M Actions
action #132893: [alert] failed systemd services on jenkins: jenkins-plugins-update, snapper-cleanup Actions
action #132902: Check and document PDU connection of nibali.qe.nue2.suse.org Actions
action #132947: Bring back ada.qe.suse.de and fix it properly Actions
action #133097: cron on OSD (date; fetch_openqa_bugs /etc/openqa/bugfetcher_o3.conf) > /tmp/fetch_openqa_bugs_o3.log failed Actions
action #133127: Frankencampus network broken + GitlabCi failed --> uploading artefacts Actions
action #133130: Lots of alerts for a single cause. Can we group and de-duplicate? Actions
action #133142: 4 baremetal SUTs in FC basement are unreachable Actions
action #133154: osd-deployment failed because unreachable workers Actions
action #133160: Setup a modern UEFI httpboot setup on o3 with dnsmasq size:M Actions
action #133181: Migration of o3 VM to PRG2 - Fix https://openqa.opensuse.org/snapshot-changes/opensuse/Tumbleweed/ Actions
action #133250: gitlab.suse.de unusable "Failed to write to log, write /srv/www/vhosts/gitlab-ce/log/gitlab-shell.log: no space left on device" on remote operations Actions
action #133322: qa-jump.qe.nue2.suse.org is not reachable - take 3 Actions
action #133325: osd http response alerts - bump threshold further up Actions
action #133328: Notification settings API | GitLab: I wonder if we can loop over all and disable/enable size:M Actions
action #133358: Migration of o3 VM to PRG2 - Ensure IPv6 is fully working Actions
action #133364: Migration of o3 VM to PRG2 - Decommission old-ariel in NUE1 as soon as we do not need it anymore Actions
action #133367: Evaluate if we have hardware alternatives for Windows Server 2016+ testing Actions
action #133385: Problem: Interface tun5: Link down alerting and autoresolving shortly size:S Actions
action #133397: HTTP Response alert Salt alerting and autoresolving shortly size:M Actions
action #133403: Login on o3 does not work Actions
action #133469: [alert] Salt states don't apply sometimes on individual workers size:M Actions
action #133490: Migration of o3 VM to PRG2 - Fix o3 bare metal hosts iPXE booting size:M Actions
action #133511: [spike solution][timeboxed:10h] Prevent memory over-commits in openQA worker service definitions size:S Actions
action #133553: OSD deployment fails due to unreachable openqaworker1 Actions
action #133580: Be able to bind openQA tests to x86_64 architecture sub-versions Actions
action #133793: salt-pillars-openqa failing to apply within 2h and it is not clear which minion(s) are missing size:M Actions
action #133892: [alert] arm-worker2 (arm-worker2: host up alert openQA host_up_alert_arm-worker2 worker size:M Actions
action #133928: salt-states-openqa | Failed pipeline for master Actions
action #133985: [alert] Backup VM not reachable via FQDN size:M Actions
action #134018: [alert] Multiple alerts with "used > 80%" size:S Actions
action #134042: auto-update on OSD does not install updates due to "Problem: nothing provides 'libwebkit2gtk3 ..." but service does not fail and we do not get an alert size:M Actions
action #134051: Eng-Infra maintained DNS server for .qa.suse.de taking over from qanet size:M Actions
action #134087: Fix ix64ph1075 bare metal openQA test size:M Actions
action #134123: Setup new PRG2 openQA worker for o3 - two new arm workers size:M Actions
action #134132: Bare-metal control openQA worker in NUE2 size:M Actions
action #134243: fozzie not responsive via ipmi Actions
action #134282: [tools] network protocols failures on multimachine tests on HA/SAP size:S auto_review:"no candidate.*iscsi-target-overview-service-tab|yast2.+firewall.+services.+add.+zone":retry Actions
action #134453: backup.qam.suse.de is Failed according to netbox and not creating backups size:M Actions
action #134489: backup.qa.suse.de does not create backups Actions
action #134519: We were not notified that backup.qa.suse.de did not create backups size:M Actions
action #134522: [alert] Certificate renewal on monitor.qa.suse.de might not be working causing alerts size:M Actions
action #134723: Conduct lessons learned for making new-lines in script_run fatal size:S Actions
action #134735: [alert] openQA piworker openqa-piworker: host up alert Actions
action #134816: [tools] grafana dashboard for `OpenQA Jobs test` partially without any data from OSD migration size:M Actions
action #134822: Migration of osd VM to PRG2 - Decommission old-osd in NUE1 as soon as we do not need it anymore size:M Actions
action #134861: https://stats.openqa-monitor.qa.suse.de/ reports "502 Bad Gateway", https://monitor.qa.suse.de works fine Actions
action #134879: reverse DNS resolution PTR for openqa.oqa.prg2.suse.org. yields "3(NXDOMAIN)" for PRG1 workers (NUE1+PRG2 are fine) size:M Actions
action #134900: salt states fail to apply due to "Pillar openqa.oqa.prg2.suse.org.key does not exist" Actions
action #134906: osd-deployment failed due to openqaworker1 showing "No response" in salt size:M Actions
action #134912: Gradually phase out NUE1 based openQA workers size:M Actions
action #134927: OSD throws 503, unresponsive for some minutes size:M Actions
action #134939: [OSD][Worker] openqaworker1:7 is unworkable Actions
action #134948: Ensure IPv6 is working in the OSD setup (since we have workers in PRG2 and the VM has been migrated) size:M Actions
action #135026: Oli shows up with 50% resolved tickets despite having been on hols for two weeks Actions
action #135029: Many unhandled alert messages while users report problems Actions
action #135032: Communication about vacation within the team could be better Actions
action #135047: retro feedback: This week was very exhausting - too much infra work Actions
action #135137: Bring back imagetester size:M Actions
action #135152: Zabbix agent is not available Actions
action #135185: Zabbix reports high swap space usage > 50% Actions
action #135191: Migration of o3 VM to PRG2 - Use direct zabbix connection size:M Actions
action #135206: [tools] GitlabCI telegraf step on salt-states-openqa failed Actions
action #135230: salt pillars pipelines failing due to Temporary failure in name resolution Actions
action #135236: https://monitor.qa.suse.de/d/WebuiDb/webui-summary?orgId=1&viewPanel=78 shows no data Actions
action #135260: zabbix - o3 High CPU utilization (over 90% for 5m) size:M Actions
action #135329: s390x work demand exceeds available workers Actions
action #135335: [tools] gitlabci salt-pillars-openqa deploy failed on imagetester and other hosts size:M Actions
action #135380: A significant number of scheduled jobs with one or two running triggers an alert Actions
action #135404: openqaworker-arm-2.suse.de minion not returning Actions
action #135467: aarch64.openqanet.opensuse.org 100 packet loss Actions
action #135470: Grafana: Average Ping time (ms) alert with unexpanded variable "${tag_url}", which machine is this about? size:M Actions
action #135491: fozzie and quinn unable to access PXE server or iPXE server (TFTP open timeout) Actions
action #135509: monitor.qa.suse.de yields 502 Bad Gateway from nginx/1.21.5 size:M Actions
action #135515: malbec.arch.suse.de not reachable anymore Actions
action #135578: Long job age and jobs not executed for long size:M Actions
action #135632: "Mojo::File::spurt is deprecated in favor of Mojo::File::spew" breaking os-autoinst OBS build and osd-deployment size:M Actions
action #135737: [alert] Munin - network eth errors - opensuse.org :: openqa.opensuse.org size:M Actions
action #135740: [alert] Munin - minion hook failed - opensuse.org :: openqa.opensuse.org - only "label_known_issues" hook scriptssize:M Actions
action #135773: [tools] many multi-machine test failures in "ovs-client+ovs-server" test scenario when tests are run across different workers size:M Actions
action #135833: false-positive inode and disk usage alert on windows image Actions
action #135848: Icinga alarm about Postfix mail queue since July 26 Actions
action #135890: [alert][FIRING:1] (Apache Workers alert Salt MW025mB4z) 2023-09-18 00:11 Actions
action #135893: FQDN (long host name) is unavailable for fozzie Actions
action #135941: openqa-piworker offline and no alert Actions
action #136007: Conduct "lessons learned" with Five Why analysis for network protocols failures on multimachine tests on HA/SAP size:S Actions
action #136052: bot-ng in gitlab CI runner + monitor.qe.nue2.suse.org unable to access smelt.suse.de, "No route to host" Actions
action #136115: https://racktables.nue.suse.com/ yields "Database write failed - LDAP caching error" on connection attempts after about 1m waiting time Actions
action #136121: Repurpose PowerPC hardware in FC Basement - deagol Actions
action #136133: Migrate aarch64.openqanet.opensuse.org to FC Basement size:M Actions
action #136238: test incompletes with auto_review:"qemu-system-.*Address already in use":retry size:M Actions
action #136274: Failing DNS resolution on o3 for hosts like github.com Actions
action #136304: [tools] please provide a machine for testing sle-micro 5.5 (virtualization testing) Actions
action #136325: salt deploy fails due to multiple offline workers in qe.nue2.suse.org+prg2.suse.org Actions
action #136370: systemd service rsnapshot@beta on backup-vm.qe.nue2.suse.org failed due to process conflict Actions
action #136946: https://monitor.qa.suse.de/d/WebuiDb/webui-summary shows multiple panels with "no data" size:M Actions
action #137075: Fail to login to the osd, 'Forbidden' error is returned due to DNS server change within SUSE *and* auto_review:"Bugzilla query failed: Network is unreachable":retry size:M Actions
action #137231: [tools] qem-bot and others can not execute scheduled jobs due to registry.opensuse.org outage Actions
action #137270: [FIRING:1] host_up (malbec: host up alert openQA malbec host_up_alert_malbec worker) and similar about malbec Actions
action #137306: Check unreal6 cabling, SP and system not reachable over network size:M Actions
action #137408: Support move of s390x mainframe(s) to PRG2 - o3 size:M Actions
action #137504: Lots of emails of the form Switched Rack PDU: Outlet .* (openqaworker-arm-.*) .* Actions
action #137519: [alert] Failed systemd services - openqaworker1 - proc-sys-fs-binfmt_misc.mount, kernel modules already removed with old kernel still running size:M Actions
action #137522: [alert] alerts about "host: sushil-linux-tw-kde" that tools team should not be notified about, e.g. Inode utilization inside the OSD infrastructure is too high size:M Actions
action #137600: [alert] Packet loss between worker hosts and other hosts size:S Actions
action #137603: [alert] Queue: State (SUSE) - too few jobs executed alert size:S Actions
action #137615: [alert] Failed systemd services alert - s390zl12,s390zl13 - kdump-early, kdump, smartd Actions
action #137744: [alert] gitlab pipeline fails in deploy step for salt-pillars-openqa: Data failed to compile Actions
action #137747: NFS and download.opensuse.org related outage handling Actions
action #137756: Re-enable worker31 for multi-machine tests in production auto_review:"tcpdump.+check.log.+timed out at" Actions
action #137771: Configure o3 ppc64le multi-machine worker size:M Actions
action #137813: [alert] Failed systemd services - qamaster - logrotate fails on /var/log/messages with "/usr/bin/xz: (stdin): Read error: Input/output error" size:S Actions
action #137984: salt "refresh" job full of errors but CI job passes size:M Actions
action #138005: grafana panel "Packet loss between worker hosts and other hosts" shows more than just ping to "other hosts" and hence becomes slow and triggers redundant alerts size:M Actions
action #138038: diesel+petrol missing network, IPMI still reachable Actions
action #138044: Grouped seemingly unrelated alert emails are confusing size:M Actions
action #138320: test fail hook in openqa_webui is failing: script/opencli api jobs - Can't locate Mojo::Base in @INC Actions
action #138350: worker31 and likely more OSD machines get stuck on boot in grub command line Actions
action #138377: bot-ng and openQABot pipelines fail to pull containers from registry.suse.de size:M Actions
action #138515: foobar host up alert size:S Actions
action #138518: unreal6 partition usage alert Actions
action #138527: Zabbix agent on ariel.dmz-prg2.suse.org reported no data for 30m and there is nothing in the journal size:S Actions
action #138536: Alert Worker .* has no heartbeat (900 seconds), restarting (see FAQ for more) on o3 size:M Actions
action #138545: Munin - minion hook failed - opensuse.org :: openqa.opensuse.org size:S Actions
action #138551: DNS outage of 2023-10-25, e.g. Cron <root@openqa-service> (date; fetch_openqa_bugs)> /tmp/fetch_openqa_bugs_osd.log Max retries exceeded with url size:S Actions
action #138650: partition usage panels show a long list of undefined and no reasonable graphs at least for some generic machines size:M Actions
action #138746: [tools] s390x VM randomly fails to open QCOW disk image: Permission denied Actions
action #138773: Trying to login on o3 or osd fails with Forbidden - again Actions
action #139004: Cron <root@openqa-service> (date; fetch_openqa_bugs)> /tmp/fetch_openqa_bugs_osd.log - OSError: [Errno 101] Network is unreachable Actions
action #139100: Long OSD ppc64le job queue - Move nue3 power8 machines to nue2 Actions
action #139145: dehydrated on monitor.qe.nue2.suse.org aka. monitor.qa.suse.de fails with "EXPECTED value GOT EOF" size:M Actions
action #139148: [alert] openqa-review repeatedly failed to read job group overview data running into "Read timed out" Actions
action #139271: Repurpose PowerPC hardware in FC Basement - mania Power8 PowerPC size:M Actions
action #139307: openQA for s390x does not work at the moment Actions
action #150824: monitor-pre-deploy from osd-deployment is failing with Build status for devel:openQA openSUSE_Leap_15.5 arch x86_64 is not successful Actions
action #150830: Two new ARM servers 2023-11 for openqa.suse.de bare-metal testing size:M Actions
action #150845: openqaworker-arm22 broken due to packages automatically removed size:M Actions
action #150887: [alert] [FIRING:1] s390zl12 (s390zl12: partitions usage (%) alert Generic partitions_usage_alert_s390zl12 generic), also s390zl13 size:M Actions
action #150908: o3 "Unable to fetch build results" and "Internal server error" on some pages size:M Actions
action #150920: openqaworker-arm22 is unable to join download.opensuse.org in parallel tests = tap mode size:M Actions
action #150938: [openQA][sut][ipmi] No ipmi sol output with ix64ph1075 size:M Actions
action #150956: o3 cannot send e-mails via smtp relay size:M Actions
action #150965: At least diesel+petrol+mania fail to auto-update due to kernel locks preventing patches size:M Actions
action #150983: CPU Load and usage alert for openQA workers size:S Actions
action #151013: o3 yielding "502 Bad Gateway" from nginx 2023-11-19, why was the config overwritten? size:M Actions
action #151165: salt-pillars-openqa pipeline fails Actions
action #151231: package loss between o3 machines and download.opensuse.org size:M Actions
action #151390: Brute-force salt osiris so that we enable self-management of VMs for users size:M Actions
action #151396: After osiris is now in salt decide about the fate of seth Actions
action #151588: [potential-regression] Our salt node up check in osd-deployment never fails size:M Actions
action #151597: [alert] osiris-1 (osiris-1: partitions usage (%) alert Generic partitions_usage_alert_osiris-1 generic Actions
action #151696: Evaluate use of https://itpe.io.suse.de/open-platform/docs/ size:M Actions
action #151807: [alert] o3 zabbix: Problem: /var/lib/snapshot-changes: Disk space is critically low (used > 94%) size:M Actions
action #152008: s390zl13 redirecting http to https because of .wget-hsts but we don't install ca-certificates-suse on all salt-controlled machines Actions
action #152092: Handle all package downgrades in OSD infrastructure properly in salt size:M Actions
action #152095: [spike solution][timeboxed:8h] Ping over GRE tunnels and TAP devices and openvswitch outside a VM with differing packet sizes size:S Actions
action #152101: Allow salt to properly configure non-production multi-machine workers size:M Actions
action #152377: [tools] SLE-15-SP2 and SLE-15-SP3 install target medias for x86_64 in the ...../assets/repo are purged Actions
action #152386: [alert] "Apache Workers" and "HTTP Response" alerts fired shortly on 2023-12-11 size:S Actions
action #152446: openqaworker-arm21 is broken and produces lots of incomplete jobs Actions
action #152503: [FIRING:1] worker38 (worker38: partitions usage (%) alert openQA partitions_usage_alert_worker38 worker) Actions
action #152557: unexpected routing between PRG1/NUE2+PRG2 Actions
action #152578: Many incompletes with "Error connecting to VNC server <unreal6.qe.nue2.suse.org:...>" size:M Actions
action #152599: [alert] `rsnapshot@alpha.service` failed on `backup.qa.suse.de` Actions
action #152643: https://monitor.qa.suse.de/d/nRDab3Jiz/openqa-jobs-test showing two entries for x86_64, one with quotes and one without Actions
action #152649: [alert] `rsnapshot@alpha.service` failed on `backup.qa.suse.de` size:M Actions
action #152673: [alert] `systemctl status iscsid.socket` failed on `s390zl12.oqa.prg2.suse.org` size:S Actions
action #152741: [tools] gitlab CI - openqa_review failed with connection timeout on osd Actions
action #152811: ada.qe.suse.de is not responding to salt commands Actions
action #152813: openqaw5-xen.qa.suse.de is not responding to salt commands Actions
action #152827: [tools] cron service updating clamav database failing on OSD + O3 size:S Actions
action #152857: [tools] alert ping between hosts timeout proxy.scc.suse.de Actions
action #152887: Setup of Ampere Altra Q32-17 for bare-metal tests in openQA size:M Actions
action #152941: circleCI job runs into 20m timeout due to slow download from registry.opensuse.org Actions
action #152981: monitoring: Update deprecated angular grafana panels Actions
action #153023: [FIRING:1] (Packet loss between worker hosts and other hosts alert Salt 2Z025iB4km) Actions
action #153325: osd-deployment | Failed pipeline, Digest verification failed for openQA-common size:M Actions
action #153328: jenkins fails in submit-openQA-TW-to-oS_Fctry, Server returned an error: HTTP Error 400: Bad Request size:M Actions
action #153418: http based health check against proxy.scc.suse.de size:M Actions
action #153544: os-autoinst:os-autoinst-openvswitch-test not building on Leap_15.5 x86_64 Actions
action #153925: Support YAM squad to get backlogger running in our salt states (and fix our pipelines again) Actions
action #153958: [alert] s390zl12: Memory usage alert Generic memory_usage_alert_s390zl12 generic Actions
action #153961: Grafana doesn't show values for inode utilization for s390zl12 Actions
action #154018: [alert] Failed systemd services alert: backup-vm postfix Actions
action #154177: File systems alert Salt: One of the file systems is too full size:M Actions
action #154345: Incomplete jobs (not restarted) of last 24h alert Salt Actions
action #154426: HTTP Response alert Salt alerting and autoresolving shortly size:M Actions
action #154546: Cron fetch_openqa_bugs refused or timed out trying to fetch individual tickets Actions
action #154549: Certain queries on poo are not accessible Actions
action #154624: Periodically running simple ping-check multi-machine tests on x86_64 covering multiple physical hosts on OSD alerting tools team on failures size:M Actions
action #154627: [potential-regression] Ensure that our "host up" alert alerts on not host-up conditions size:M Actions
action #154927: [alert] Broken workers alert was firing several hours after weekly reboot Actions
action #154939: Clarify use of quake2 and ensure it does not waste power size:S Actions
action #154942: Clarify use of quake6 and ensure it does not waste power size:S Actions
action #155074: salt-states-pipeline fails trying to install influxdb Actions
action #155080: jenkins is no longer producing GNOME:Next test runs: http://jenkins.qa.suse.de/job/gnome_next-openqa/8670/console Actions
action #155326: [alert] "HTTP Response" alert fired shortly on 2024-02-12 and 2024-03-04 size:M Actions
action #155659: [openQA][infra][sut] Failed to establish connnection to ix64ph1075-sp.qe.nue2.suse.org Actions
action #155689: bot-ng pipelines fails to schedule incidents Actions
action #155725: [openQA][infra][sut] Failed to establish connnection to fozzie-sp and quinn-sp Actions
action #155737: Salt pillars pipelines fail due to refused connection errors on telegraf Actions
action #155740: Scripts CI pipelines fail due to timeout after many Job state of job ID xyz: scheduled, waiting messages Actions
action #155824: Support IPv6 SLAAC in our infrastructure size:M Actions
action #155848: Firewalld is logging many errors and sometimes restarting on worker29, possibly related to MM failures size:M Actions
action #155929: Try out rstp_enable=True in openqa/openvswitch.sls size:M Actions
action #156016: [openQA][sle-micro][virtualization] Test slem_virtualization@uefi with Default-encrypted image was not triggered correctly size:M Actions
action #156049: [alert] Scripts CI pipeline failing with No reason for devel:openQA:Leap:15.6/local-npm-registry Actions
action #156064: Handle planned datacenter shutdown PRG1 2024-02-28 size:M Actions
action #156076: bot-ng gitlab CI pipeline fails with "ERROR: Job failed (system failure): Error response from daemon: mkdir /var/lib/docker/overlay2/…/merged: no space left on device (exec.go:78:0s)" Actions
action #156130: Install Intel GPU on one of the servers in FC size:M Actions
action #156226: [bot-ng] Pipeline failed / failed to pulled image / no space left on device Actions
action #156301: [bot-ng] Pipeline failed / KeyError: 'priority' / execution took longer than 4h0m0s seconds Actions
action #156304: [salt-pillars] Pipeline failed / Passed invalid arguments to state.highstate Actions
action #156322: zabbix-proxy.dmz-prg2.suse.org not reachable from ariel.suse-dmz.opensuse.org Actions
action #156331: [gitlab] New pipeline schedules cannot be created, you have exceeded the maximum number of pipeline schedules for your plan Actions
action #156460: Potential FS corruption on osd due to 2 VMs accessing the same disk Actions
action #156481: cron -> (fetch_openqa_bugs)> /tmp/fetch_openqa_bugs_osd.log failed / No route to host / openqa.suse.de Actions
action #156514: Cron <root@openqa-service> (date; fetch_openqa_bugs)> /tmp/fetch_openqa_bugs_osd.log failed / simplejson.errors.JSONDecodeError: Expecting value: line 1 column 1 (char 0) / bugzilla_issue.py Actions
action #156517: Can't call method "script" on an undefined value at lib/OpenQA/WebAPI/Controller/Step.pm size:S Actions
action #156532: lessons learned about "Potential FS corruption on osd due to 2 VMs accessing the same disk" size:S Actions
action #156535: Handle unfinished SLE maintenance tests due to FS corruption on OSD 2024-03-01 Actions
action #156913: Remove its=off setting in global QEMUMACHINE for worker-arm{1,2} size:M Actions
action #156934: RPi realhw tests fail with # Test died: Error connecting to <root@10.168.192.112> Actions
action #157081: OSD unresponsive or significantly slow for some minutes 2024-03-12 08:30Z Actions
action #157243: Update HMC with vMF68994 Actions
action #157438: Failed systemd services alert (jenkins-plugins-update, snapper-cleanup) Actions
action #157441: osd-deployment | Failed pipeline for master (qesapworker-prg5.qa.suse.cz) Actions
action #157453: [FIRING:1] host_up (qesapworker-prg5: host up alert openQA qesapworker-prg5 host_up_alert_qesapworker-prg5 worker) Actions
action #157468: Handle internal test machines with compromised root password size:M Actions
action #157528: Remove redundant ASM connections for powerPC machines size:S Actions
action #157615: [alert] osd-deployment failed in post-deploy , telegraf errors size:M Actions
action #157666: OSD unresponsive and then not starting any more jobs on 2024-03-21 Actions
action #157834: [openQA][ipmi] IPMI backend machines in NUE2 can not be reached auto_review:"Reason: backend died: ipmitool.*Address lookup for.*qe.nue2.suse.org":retry Actions
action #158020: salt-states-openqa pipeline times out Actions
action #158023: salt-states-openqa pipeline invalid arguments to state.highstate on monitor.qe.nue2.suse.org Actions
action #158026: osd-deployment exceeds 2h maximum runtime during package installation Actions
action #158041: grenache needs upgrade to 15.5 Actions
action #158059: OSD unresponsive or significantly slow for some minutes 2024-03-26 13:34Z Actions
action #158104: typing issue on ppc64 worker size:S Actions
action #158113: typing issue on ppc64 worker - make CPU load alert more strict size:M Actions
action #158170: Increase resources for s390x kvm size:M Actions
action #158185: parallel job failed to get the vars from its pair size:S Actions
action #158242: Prevent ssh access to test VMs on svirt hypervisor hosts with firewall size:M Actions
action #158266: openQA jobs on diesel ppc64le fail due to auto_review:"QEMU: This is probably because your SMT is enabled." Actions
action #158377: Detect from monitoring data which monitored machines show a too low CPU usage over time size:M Actions
action #158383: Crosscheck which machines marked as "unused" in racktables are still pingable (as they should not be powered on at all) size:M Actions
action #158404: [qe-core][jeos]test fails in prepare_firstboot Actions
action #158419: osiris-1.qe.nue2.suse.org not responsive over virt-manager and "virsh list" hangs Actions
action #158502: [FIRING:1] host_up (petrol: host up alert openQA petrol host_up_alert_petrol worker) Actions
action #158505: Failed systemd services alert for jenkins-plugins-update size:S Actions
action #158526: Apply the latest firmware+BIOS upgrade for diesel as well size:S Actions
action #158556: Single-value SLI of OSD HTTP response code successful vs. all size:S Actions
action #158559: Single-value SLI of OSD HTTP response time size:S Actions
action #158907: Automated check for machines marked as "unused" in racktables but still pingable (as they should not be powered on at all) size:M Actions
action #158913: Fix salt warning observed on petrol "State for file: /etc/fstab - Neither 'source' nor 'contents' nor 'contents_pillar' nor 'contents_grains' was defined, yet 'replace' was set to 'True'" Actions
action #159048: Setup new Power10 machine for QE LSG in PRG2 (S/N 7882391) Actions
action #159066: network-level firewall preventing direct ssh+vnc access to openQA test VMs size:M Actions
action #159165: Ensure that rsyncd is properly covered in salt states size:S Actions
action #159186: [alert] Systemd-services alert failing due to unit "rsnapshot@alpha" on host "storage" Actions
action #159264: Discard Ampere ARM server packaging if we still have it + reserve thunderx21 machine in FC Basement for afaerber size:S Actions
action #159270: openqaworker-arm-1 is Unreachable size:S Actions
action #159303: [alert] osd-deployment pre-deploy pipeline failed because openqaworker-arm-1.qe.nue2.suse.org was offline size:S Actions
action #159318: openqa-piworker host up alert Actions
action #159396: Repeated HTTP Response alert for /tests and unresponsiveness due to potential detrimental impact of pg_dump (was: HTTP Response alert for /tests briefly going up to 15.7s) size:M Actions
action #159402: "SSL expiration" panel on https://stats.openqa-monitor.qa.suse.de/d/WebuiDb/webui-summary shows problems size:S Actions
action #159555: IPMI access over IPv6 doesn't work on imagetester - try to update BIOS with physical access size:S Actions
action #159639: [alert] "web UI: Too many 5xx HTTP responses alert" size:S Actions
action #159669: Missing openQA data on metrics.opensuse.org since o3 migration to PRG2 Actions

QA - Tools - Next

open

Staging for next tasks to be picked up by the QE tools team after "Ready"

22%

65 issues   (6 closed — 59 open)

Related issues
action #40196: [monitoring] monitor internal port 9526, port 80, external port 443 accessibility of o3 and response times size:M Actions
action #134126: Setup new PRG2 openQA worker for o3 - bare-metal testing size:M Actions
action #135023: Consider moving the retro out of the weekly or alternating either one size:S Actions
action #139103: Long OSD ppc64le job queue - Decrease number of x86_64 worker slots on osd to give ppc64le jobs a better chance to be assigned jobs size:M Actions
action #150815: unable to login over ssh to o3 (gate.opensuse.org:2214) size:M Actions
action #150869: Ensure multi-machine tests work on aarch64-o3 (or another but single machine only) size:M Actions
action #155743: OBSRSync fails to sync openSUSE:Factory:PowerPC:ToTest (was: WARNINGs: failed is 452.00 in Munin - minion Minion Jobs on o3) Actions
action #157750: Better secure the networks to have s390kvm… (and others) less accessible Actions
action #158116: typing issue on ppc64 worker - crosscheck performance impact of ffmpeg on ppc64le (Power8 kvm) size:M Actions
action #159063: s390x qemu backend host within SUSE networks Actions
action #159069: network-level firewall preventing direct ssh+vnc access to all machines within the oqa.prg2.suse.org network if needed Actions

QA - future

open

a "virtual" milestone that is never reached. Everything that is currently out of scope can go here. Place for new feature ideas, wishes, proposed improvements and changes which we could work on in the future but are not currently planned for.

41%

1637 issues   (624 closed — 1013 open)

Related issues
action #10516: [functional][u] Use jenkins as extension for openQA Actions
action #17816: [functional][u][ipmi] Error: Unable to establish IPMI v2 / RMCP+ session Actions
action #18608: [qe-core][tools][sle][functional][research][medium] Implement proper clean up for images on jump hosts, e.g. s390pb, hyperv host, svirt and a proper monitoring Actions
action #30595: [ppc64le] Deploy bcache on tmpfs workers Actions
action #38516: [tools] Disable memory autoballooning on openqaw5-xen Actions
action #39008: [tools][pvm] Redundant powerVM production workers Actions
action #41867: [devops][tools] Replace get-metrics script by telegraf Actions
action #41975: Evaluate graphite vs prometheus Actions
action #42233: Production-grade jenkins.qa.suse.de for QA SLE (EMEA) Actions
action #42467: http://jenkins.qa.suse.de/job/monitor-openQA_in_openQA-TW fails to write comments in OBS project, monitor-openqa_job expects package name for OBS api, not project Actions
action #44183: Add checks on transactional-update workers using "health-checker" Actions
action #54785: tap devices not in any zone, error reported by firewalld Actions
action #55316: monitoring alerts for too long running database queries Actions
action #57239: Add/fix openqa_logwarn for o3 and osd sending to o3-admins@suse.de and osd-admins@suse.de respectively Actions
action #58325: Update firmware of our powerKVM machines (QA-Power8-3, QA-Power8-4, QA-Power8-5) Actions
action #59621: osd: Sporadically high CPU and IO load (vdd), grafana alerts "Disk I/O time for /dev/vdd" and "CPU usage", also other disks Actions
action #60962: Enable multi-machine capability for all configured workers Actions
action #61221: osd: unable to save needles, minion fails with "fatal: Unable to create '/var/lib/openqa/.../needles/.git/index.lock'" Actions
action #62666: Move openqa.opensuse.org into opensuse private network Actions
action #63382: /usr/share/qemu/ovmf-x86_64-staging{,-code,-vars}.bin on workers is not installed by any package, e.g. missing on imagetester Actions
action #63874: ensure openqa worker instances are disabled and stopped when "numofworkers" is reduced in salt pillars, e.g. causing non-obvious multi-machine failures Actions
action #64123: Expose database revert as a manual trigger Actions
action #64541: Issues with DNS resolving in QEMU Actions
action #64580: Detect and recover from I/O blocked worker machines, e.g. openqaworker-arm-{1,2,3}.suse.de Actions
action #64803: harmonize IPMI username and password auto_review:"(sonic|tails)-sp.qa.suse.de .*Unable to establish IPMI" Actions
action #65019: ppc64 debuginfo repo is not stripped for Tumbleweed Actions
action #65555: [qe-core][functional][core] html5test.opensuse.org includes non-existing file Actions
action #66019: Configure downsampling and a retention policy for InfluxDB Actions
action #66610: Prevent misleading login failing over http on https-enabled instances, e.g. disable non-SSL in production openQA instances using OpenID (OSD, o3) at least for login attempts Actions
action #68633: alert if there is no worker active for any existant machine class Actions
action #69718: harmonize timezone used for our machines Actions
action #71590: [osd][alert] Implement proper monitoring of needed resources of workers Actions
action #72079: Upgrade o3 worker openqa-aarch64 to openSUSE Leap 15.2 (to use newer packages specifically needed for aarch64 and as precursor), also problem auto_review:"(?s)starting: /usr/bin/qemu-system-aarch64.*backend died: Migrate to file failed" Actions
action #73297: auto_review:"(?s)Running on openqa-aarch64:.*considering VNC stalled.*THERE IS NOTHING TO READ" Actions
action #73345: [u] Cleanup of old needles from os-autoinst-needles-opensuse and os-autoinst-needles-sle Actions
action #76768: HDD images are cleaned-up too early for aarch64 Tumbleweed Actions
action #76792: /var/lib/openqa in QA-Power8-5-kvm reports Structure needs cleaning (auto_review:"(?s)Running on QA-Power8-5-kvm.*cannot remove directory for testresults":retry) Actions
action #77011: openqaworker7 (o3) is stuck in "recovery mode" as visible over IPMI SoL Actions
action #77317: chat bot to conduct daily checks, alerts, reminders, etc. Actions
action #78061: [Alerting] openQA minion workers alert - alert turned "OK" again after 20 minutes and we don't know what was wrong Actions
action #78222: Persistent postgres setup and configuration in salt Actions
action #78232: [ipmi|PXE]OSD repo seems to be having mount issue Actions
action #78314: Order 2 VMs for the LinuxONE Community Cloud Actions
action #78348: osd: Some job group obsolete definitions should be removed Actions
action #80130: grenache salt is spilling errors about "Failed to load grains defined in grain file ppc_powervm.check_output" Actions
action #80482: qa-power8-5-kvm has been down for days, use more robust filesystem setup Actions
action #80540: idea: Conduct "power outage drills", e.g. once every half-year? Actions
action #80732: Monitoring on log alerts, also on grafana log alerts, maybe grafana-loki? Actions
action #80824: o3: Migrate from SuSEfirewall2 to firewalld Actions
action #80990: some jobs incomplete with auto_review:"(?s)Running on[^\n]*Linux 5.3.18.*Migrate to file failed, it has been running for more than 240 seconds":retry , seems to happen since upgrade to Leap 15.2 Actions
action #81308: Show actual percentage in partitions usage (%) alert Actions
action #88195: Full test of salt states including DBus on CI Actions
action #88197: Workflow for staged deployments Actions
action #88299: [virtualization] Worker openqaw5-xen-1.qa.suse.de is not reachable (xen-hvm/xen-pv failing) Actions
action #89398: OSD deployment stopped prematurely due to "System management is locked by the application with pid 29165 (zypper)" Actions
action #90062: osd deployment fails in "record package versions pre-deploy" maybe because a kernel update is pending? Actions
action #91626: Deployment emails should cover rollback Actions
action #93119: [s390x] Update of s390x Test infrastructure after shutdown of Mainframe zEC12 Actions
action #93138: openqa-reload-worker-auto-restart and openqa-worker-auto-restart fail if numofworkers is reduced Actions
action #94399: No alert when arm workers are offline, alert if telegraf throws errors size:M Actions
action #94495: Migrate to InfluxDB 2 when needed (at least when our current OS base has a package) Actions
action #94765: Bring openqaworker12 into production (w/o multi-machine test support) size:M Actions
action #94783: Bring openqaworker11 into production including multi-machine test support (same as w12) size:M Actions
action #96257: Tests timeout with MAX_SETUP_TIME on osd - Apply a higher MAX_SETUP_TIME applicable for the *complete* OSD infrastructure, e.g. add to every worker config entry Actions
action #96552: Persistent records of I/O usage by process size:M Actions
action #96683: Reducing the number of worker slots leads to failing systemd units Actions
action #97409: Re-use existing filesystems on workers after reboot if possible to prevent full worker asset cache re-syncing Actions
action #97412: Reduce I/O load on OSD by using more cache size on workers with using free disk space when available instead of hardcoded space Actions
action #97640: have a secondary backup openQA webUI instance in another location than OSD, e.g. PRG Actions
action #97745: [virtualization][hyperv] ensure_serialdev_permissions fails for hyperv Actions
action #97859: Improve network for OSD Actions
action #97976: [alert] OSD file systems - assets Actions
action #98661: Tweak worker numbers for openqaworker-arm-4 and arm-5 Actions
action #98682: jobs run powerqaworker-qam-1 fail with auto_review:"(?s)powerqaworker-qam-1.*Can't write to file (.*): No space left on device at .*":retry size:M Actions
action #98835: arm jobs failing (again?) with auto_review:"backend died: Open vSwitch command 'set_vlan' with arguments .*was not provided by any .service files":retry Actions
action #99138: Provide ratio of tests by result in monitoring - by job group Actions
action #100841: Improve logging for libvirt hosts Actions
action #100976: specific alert about the size of our database, maybe even database tables? Actions
action #100979: configure better auto-vacuum for our database(s) Actions
action #100985: Come up with a way to regularly check job group configs for outliers and misconfiguration, e.g. overly long result retention periods Actions
action #100988: ask for OSD /srv to be enlarged (was: investigate how to optimize /srv data utilization on OSD) size:S Actions
action #102167: Disk monitoring for s390x z/VM backend Actions
action #102269: openQABot pipeline failed with HTTP 403 Actions
action #103380: Configure retention/downsampling policy for specific monitoring data stored within InfluxDB Actions
action #104091: Try `aarch64_cpu_max` tests on `openqaworkerX` Actions
action #105013: backup o3 worker config files Actions
action #105867: OpenQA bot schedules jobs with incomplete INCIDENT_REPO Actions
action #106756: cmd returned 32768 repeatedly reported on o3 Actions
action #106760: DBI Exception: DBD::Pg::st execute failed: number of parameters must be between 0 and 65535 repeatedly reported on o3 Actions
action #106766: DBI Exception: DBD::Pg::st execute failed: ERROR: invalid input syntax for type integer reported on o3 Actions
action #106841: Try using tmpfs on openqaworker1 to use RAM more efficiently Actions
action #107023: cmd returned 31744 repeatedly reported on o3 Actions
action #107794: openqa-logwarn is missing messages Actions
action #107980: openqa-aarch64 - kernel traces in dmesg "watchdog: BUG: soft lockup - CPU#.* stuck for .*s! [qemu-system-aar:.*]" Actions
action #108266: grenache: script_run() commands randomly time out since server room move Actions
action #108872: Outdated information on openqaw5-xen https://racktables.suse.de/index.php?page=object&tab=default&object_id=3468 Actions
action #108974: Loan Fujitsu server for OpenQA FIPS testing needs Actions
action #109112: Improve os-autoinst sshXtermVt.pm connection error handling (was: "Test died: Error connecting to <root@redcurrant-4.qa.suse.de>: No route to host") size:M Actions
action #109635: Check grafana monitoring host performance size:M Actions
action #110191: CI checks should ensure that apparmor changes are correct Actions
action #110467: Establish reliable tap setup on ow14 Actions
action #110545: Investigate and fix higher instability of openqaworker-arm-4/5 vs. arm-1/2/3 - further things to try size:M Actions
action #112769: errors from telegraf on osd webUI "[inputs.procstat] Error in plugin: open /sys/fs/cgroup/systemd/system.slice/salt-master.service/cgroup.procs: no such file or directory", same for openqa-scheduler.service Actions
action #112787: Use different users for our database accesses so that we can debug where load comes from Actions
action #112850: Make pgstatstatements persistent on OSD (in salt) Actions
action #112943: Consider running explicitly triggered `vacuum ANALYZE` after every database migration Actions
action #113357: UEFI PXE or "network boot" support within .qa.suse.de size:M Actions
action #113701: [qe-core] Move workers back to grenache Actions
action #113800: Setup o3 to run rootless containers on worker hosts Actions
action #114400: [qe-core] Rebalance OpenQA worker distribution Actions
action #114523: Deal with QEMU and OVMF default resolution being 1280x800, affecting (at least) qxl, but on aarch64 size:M Actions
action #114622: Identification of unused/idle machines by alarming when there is no traffic on the corresponding switch ports for some time Actions
action #114866: Handle a trailing slash in NEEDLES_DIR correctly Actions
action #114992: Broken MM machines don't appear as available workers Actions
action #115763: Add new feature to create a dependency that could be derived automatically from asset settings Actions
action #115925: aarch64: Random QEMU failures while retrieving host CPU features Actions
action #116473: Add OSD PowerPC workers to automatic recovery we already have for ARM workers Actions
action #116716: Repurpose ix64ph1079, ix64ph1080, ix64ph1081, e.g. as openQA workers Actions
action #116968: separate email or mailing list address for os-autoinst related development, e.g. circleCI bot account Actions
action #118375: Do not alert about "packet loss" if hosts are down Actions
action #118891: Make alerts depend on each other Actions
action #119059: Use qa-power8 for ppc tests in o3 - network connected? Actions
action #119479: openqABot pipeline failed after runner getting stuck for 1h0m0s size:M Actions
action #119755: Use a PowerVM machine to serve both PowerVM LPARs for testing as well as one VM running qemu tests Actions
action #120064: SSD SAN storage (NetApp) for O3 would improve user experience (can not just buy drives on our own) Actions
action #120115: [qem-dashboard] Use new openQA job settings API to show more data Actions
action #120145: Where can we see which services are blocked ourselves in SUSE networks? Actions
action #120169: Make s390x kvm workers also use FQDN instead of IPv4 in salt pillars for VIRSH_GUEST Actions
action #120279: Proper maintainership for openqa_service VM size:M Actions
action #120651: [openQA][infra][ipmi][worker][api] The expected pattern CMD_FINISHED-xxxxx returned but did not show up in serial log (wait_serial timed out) size:M Actions
action #121171: Consider using pngquant for speedup Actions
action #121306: [virtualization][hyperv] worker7-hyperv.oqa.suse.de can not be reached Actions
action #121507: Iscsi issue on OSD worker Actions
action #121597: Implement automated racktables sanity check Actions
action #121600: Add monitoring for connected machines inside "our" network which are not present in racktables Actions
action #121672: [virtualization] Connectivity issues on worker8-vmware.oqa.suse.de Actions
action #122776: containers_tw_image_on_ubuntu_host: error 403 (forbidden) downloading the ubuntu qcow2 file Actions
action #123226: Temperature monitoring in SUSE QE lab size:M Actions
action #123712: PowerVM HMC within o3, e.g. on VM Actions
action #123960: s390x tests fail to log into VNC console on worker2 due to already present VM with same mac address Actions
action #124562: Re-install at least one of the new OSD workers located in Prague Actions
action #125054: Review https://github.com/alexellis/awesome-baremetal Actions
action #125141: Salt pillars deployment pipeline failed on "tumblesle" due to invalid security sensor repo size:M Actions
action #125219: Use qa-power8 for ppc tests in o3 - try the other suggestions Actions
action #125744: [tools][alert][FIRING:1] (Failed systemd services alert (except openqa.suse.de) QDG8aXAVz) due to openqa-piworker.qa.suse.de unable to reach openqa.suse.de Actions
action #126677: salt-states-openqa fails with 0 errors and result False messages buried in one of several places Actions
action #126941: [spike] Evaluate a move of the osd database to its own VM Actions
action #128222: [virtualization] The Xen specific host configuration on openqaw5-xen can be re-created from salt size:M Actions
action #128513: Apache Response Time monitoring dashboard is slow to render and should be made lighter Actions
action #128870: [alert] DatasourceError for "InfluxDB not reachable" size:M Actions
action #129032: ipmitool monitoring Actions
action #129121: performance: various tests fail receiving data at usable speed over the internet Actions
action #130378: Integration of extra salt repositories as salt formulas size:M Actions
action #130513: Add o3 to grafana Actions
action #130778: Treat some openqa-clone-job failures non-fatal in openqa-investigate hook Actions
action #131549: [spike][timeboxed:20h] Additional redundancy for OSD virtualization testing - Hyperv 2016 worker host size:M Actions
action #132293: Ensure cleaned up tenants in netbox Actions
action #132380: Multiple empty folders in grafana linked to alerts Actions
action #132440: Research productive workflows for reviews on GitHub Actions
action #132647: Migration of o3 VM to PRG2 - bare-metal tests size:M Actions
action #132827: [tools][qe-core]test fails in rsync_client/salt-master, DNS resolve issue with workers "sapworker*" on multi-machine tests size:M Actions
action #132926: OSD cron -> (fetch_openqa_bugs)> /tmp/fetch_openqa_bugs_osd.log failed size:M Actions
action #132998: [alert] [FIRING:1] openqaworker-arm-3: Memory usage alert openQA (openqaworker-arm-3 memory_usage_alert_openqaworker-arm-3 worker) size:M Actions
action #133247: Additional redundancy for OSD virtualization testing - Hyperv 2019 and 2022 (or 2012r2) worker host size:M Actions
action #133388: Unavailable developer mode on ow18 Actions
action #133475: Migration of o3 VM to PRG2 - connection to rabbit.opensuse.org Actions
action #133700: Network bandwidth graphs per switch, like https://mrtg.suse.de/qanet13nue, for all current top-of-rack switches (TORs) that we are connected to size:M Actions
action #133907: Improve monitoring for http(s?) reachable on jenkins.qa.suse.de size:M Actions
action #133991: Cover same metric for different hosts with a single alert rule Actions
action #134048: openqa-piworker does not restart openQA worker processes after upgrades as it is not a salt minion in "worker" role Actions
action #134081: Setup new PRG2 openQA hardware https://racktables.suse.de/index.php?page=object&object_id=23373 Actions
action #134135: openqa-monitor.qa.suse.de salt CI deploy telegraf check shows error about "could not find file: [/etc/dehydrated/certs/monitor.qe.nue2.suse.org/fullchain.pem]" Actions
action #134492: [tools] QA-Power8-2-8247-22L-SN1010D5A is not working size:M Actions
action #134585: Display the job age per architecture Actions
action #134819: Errors in salt minion and master log on osd Actions
action #134846: Old NFS share mount is keeping processes stuck and openQA workers seem up but do not work on jobs Actions
action #134852: gitlab CI job fails in telegraf check with unsupported option since telegraf package on monitor.qa.suse.de was downgraded due to invalid repo metadata Actions
action #135005: Reduce duplication in salt-pillars-openqa openqa/workerconf.sls with advanced YAML/jinja features Actions
action #135008: Max job age graphs use mean aggregation when max would make more sense Actions
action #135056: MM Test fails in a connection to an address outside of the worker Actions
action #135662: Make the state of deployments (and rollbacks) more discoverable Actions
action #135944: Implement a constantly running monitoring/debugging VM for the multi-machine network Actions
action #136952: Consider lowering the threshold for job age and job queue alerts Actions
action #136958: Add Grafana panel for number of workers Actions
action #136967: Monitor number of uploading jobs in grafana Actions
action #137114: openQA workers fail to register after bootup due to unable to resolve openqa.suse.de but manage to do so immediately when restarting worker services Actions
action #137810: Add openqa_logwarn for osd sending to osd-admins@suse.de respectively Actions
action #138275: Ensure that there is proper ownership and maintainership for qanet.qa.suse.cz Actions
action #138374: Ensure all salt managed machines show grub and boot log and Getty on ipmi sol size:M Actions
action #138683: https://metrics.opensuse.org/d/osrt_openqa/osrt-openqa?orgId=1&from=now-7d&to=now should show current results … which apparently it doesn't Actions
action #138707: Re-enable worker32 for multi-machine tests in production Actions
action #139070: Re-enable worker34 for multi-machine tests in production Actions
action #139154: Re-enable worker33 for multi-machine tests in production Actions
action #139202: Consolidate the current use or make use of PRG2-J : Rack J11 - OpenQA Rack : H03-ch1 https://racktables.nue.suse.com/index.php?page=object&object_id=23373 Actions
action #150986: [timeboxed:10h] Prevent cpu over-allocation in openQA worker service definitions Actions
action #150995: Fix MM setup on diesel so test scenarios like ha_ctdb_supportservertest-ppc-mm work Actions
action #151130: IPv6 for openqa.opensuse.org and open.qa Actions
action #151180: Automatic reboot on o3 openQA workers broken Actions
action #151307: [grafana] migrate deprecated angular plugins Actions
action #151570: [qe-core] Cleanup openQA jobgroups Actions
action #151606: Ensure that the tap class is fully enabled on petrol Actions
action #152098: [research][timeboxed:10h] Learn more about openvswitch with experimenting together size:S Actions
action #154180: Proper kvm asset cleanup for s390x kvm backend (svirt) and tests Actions
action #155089: Automatically comment on tickets not using one of our templates Actions
action #155200: Periodically running simple ping-check multi-machine tests on ppc64le covering multiple physical hosts on OSD alerting tools team on failures size:M Actions
action #155482: [qe-sap] Update contact person for racktable entries with Lee Martin Actions
action #155764: Consider switching to safer filesystems than ext2 in osd+o3 Actions
action #157606: Prevent missing gre tunnel connections in our salt states due to misconfiguration Actions
action #157636: remove NOVIDEO=1 from ppc64le workers Actions
action #157726: osd-deployment | Failed pipeline for master (worker3[6-9].oqa.prg2.suse.org) Actions
action #157738: Use rstp_enable=True on o3 as well Actions
action #158380: Detect and switch off from monitoring data which monitored machines show a too low CPU usage over time Actions
action #158386: Crosscheck which machines marked as "unused" in racktables still draw power according to ePDU data (as they should not be powered on and wasting significant power at all) - NUE2 size:M Actions
action #158709: typing issue on ppc64 worker - with automatic CPU load based limiting in place let's increase the instances on mania again Actions
action #159234: ignore this one Actions
action #159534: Make use of all 12 redcurrant LPARs Actions
action #159657: [alert] about "web UI: Too many 5xx HTTP responses alert" hidden behind grafana alert grouping Actions
action #159666: Lack of alerts about error messages related to InfluxDB and SSL expiration panel Actions