Project

General

Profile

Ready

Next tasks to be picked up by the QE tools team

94%

1100 issues   (1021 closed — 79 open)

Time tracking
Estimated time 389.00 hours
Issues by
action

990/1045
coordination

30/54
tickets

1/1
Related issues
openQA Infrastructure - action #12506: openqa-scripts on osd/o3 should be deployed automatically (CD)
action #17886: [dashboard] Create full screen view for openQA dashboard
openQA Infrastructure - action #18142: [labs] Create wiki with up to date information (linklist included)
openQA Infrastructure - action #19238: setup pool devices+mounts+folders with salt(was: ext2 on workers busted)
action #27955: Allow the worker_bridge to sync job status from a slaveUI to a masterUI
action #29634: Enable QEMU snapshots function for virtio-gpu
action #32545: Provide warning when user creates/triggers misconfigured multi-machine clusters
action #34486: database of "test cases" or how to search for tests we have in openQA
action #36250: Add automated check for isotovideo::INTERFACE
action #39905: Job trying to download worker local file "aavmf-aarch64-vars.bin" into cache and fails with 404
action #43619: Improve workflow for dealing with openQA's dependencies
action #43712: Update upstream dockerfiles to provide an easy to use docker image of openQA-webui
action #43715: Update upstream dockerfiles to provide an easy to use docker image of workers
action #43718: Docker image for webui and workers are versioned and uploaded to obs registry
openQA Infrastructure - action #44078: Implement proper backups for o3 size:M
action #45062: Better visualization of incompletes - show module in which incomplete happens
openQA Tests - action #46502: [tools] Support "record_info" in serial_failure_detection
action #48389: self-tests in os-autoinst-distri-opensuse executing a simple (staging) test using isotovideo
action #49202: Make audit events more accessible to testers
openQA Infrastructure - action #49694: openqaworker7 lost one NVMe
action #52532: Harmonisation for team text messaging communication structure
QA - action #52655: [epic] Move openqa-review from cron-jobs on lord.arch to a more sustainable long-term solution
action #53264: Show actual "version" in job groups for builds when configured + multiple versions are available
openQA Infrastructure - action #53471: machine aarch64.o.o often unresponsive and needs power-cycle
action #54557: All openQA tests incomplete but neither package build nor unit tests fail when a new file is added to os-autoinst without mentioning in Makefile.am
action #56534: support list of machines for YAML job templates
action #56591: Improve feedback for jobs in "Uploading"
openQA Infrastructure - action #57233: Properly handle transport errors in ObsRsync plugin
openQA Infrastructure - action #57680: o3: optimize fs utilization and resize root disk (was: change /var/log to bind mount to prevent out-of-space)
action #57689: asset cleanup jobs do not run on o3 (results cleanup works), workaround: unlock locks manually
action #58304: A personal activity view for developers
action #58823: os-autoinst is too slow pressing F2 causing ARM tests to fail in "boot_to_desktop"
action #59043: Fix unstable/flaky full-stack test, i.e. remove sleep, and ui tests
action #59273: module result missing for incompleting job
openQA Infrastructure - action #59300: auto_review:"DBus.*: The name org.opensuse.os_autoinst.switch was not provided":retry GRE tunnel settings not applied on initial setup / after reboot
openQA Infrastructure - action #59933: Prevent depletion of space on / with extra partition or quota on /home on o3 size:M
action #59984: unstable test: t/05-scheduler-full.t
action #60029: OBS builds of os-autoinst fail for SLE12SP5 but not for SLE12SP3-4
action #60443: job incomplete with "(?s)process exited: 0.*isotovideo failed.*EXIT 1":retry but no further details what is wrong
action #61164: Use of uninitialized value / unhandled debug output in os-autoinst t/17-basetest.t
openQA Infrastructure - action #61994: VNC console corruption on aarch64
action #62159: Asset GRU download not done by web UI host if job scheduled by `isos post`, fails to download and then cloned (was: … using the Web UI)
openQA Infrastructure - action #63142: Upgrade firmware of ppc9 machine redcurrant
action #63451: Improve openqa-monitor-incompletes and openqa-label-known-issues to not report about incompletes with clone
action #63718: incomplete reason with just "quit"/"died" could provide more information
action #64120: Make weekly meetings available to community members
action #64520: Deal with jobs stuck in assigned state
openQA Infrastructure - action #64685: openqaworker1 showing NVMe problems "kernel: nvme nvme0: Abort status: 0x0"
action #64776: [cache][worker] cache service suddenly stopped to download assets, all subsequent jobs needing download incomplete auto_review:"setup failure: Cache service status error: Premature connection close":retry
openQA Infrastructure - action #64941: after every reboot openqaworker7 is missing var-lib-openqa-share.mount , check dependencies of service with openqaworker1
openQA Infrastructure - action #64983: Staging machine openqa-staging-1.qa.suse.de does not boot after update to Leap 15.1
action #65004: The asset generated by FORCE_PUBLISH works fine but does not show up in "Logs & Assets" and is also not mentioned in logs
openQA Infrastructure - action #65130: Upgrade of firmware(s) for cloudberry (power9 machine)
openQA Infrastructure - action #65154: root partition on osd exceeds alert threshold, 90%, after osd deployment -> apply automatic reboots to OSD machines
openQA Infrastructure - action #65975: Monitoring for "scheduled but not executed" (was: perl-Mojolicious-8.37 broke WS connection for (some?) workers)
action #66066: incomplete with reason "died: terminated prematurely" but log shows error 404 failing to download asset into cache auto_review:"(?s)Download.*failed: 404.*No scripts"
action #66376: MM tests fail in obscure way when tap device is not present
action #66664: circleci jobs do not retry, are aborted with "Too long with no output (exceeded 10m0s): context deadline exceeded"
action #66682: `apache2.service` and `openqa-webui.service` wrongly started by `openqa-worker.service`
openQA Infrastructure - action #66709: Storage server for OSD and monitoring
action #66718: unstable/flaky/sporadic test: t/ui/27-plugin_obs_rsync_status_details.t
action #66721: Use GitHub actions for os-autoinst
action #67000: Job incompletes due to malformed worker cache database disk image with auto_review:"Cache service status error.*(database disk image is malformed|Specified job ID is invalid).*":retry
action #67039: [sporadic] os-autoinst tests stuck in OBS, likely t/18-backend-qemu.t ?
action #67573: "OpenID Connect" support in openQA
openQA Infrastructure - action #67804: use non-personal account and key for pushing needles on osd to gitlab.suse.de
action #67810: trigger OBS builds of openQA github pull requests
action #67972: one module failed to upload results ending an otherwise fine job as failed (was: Some tests are flagged as failed whereas they passed)
openQA Infrastructure - action #68053: powerqaworker-qam-1 fails to come up on reboot (repeatedly)
openQA Infrastructure - action #68077: alert about too many failed minion jobs but https://openqa.suse.de/minion/jobs?state=failed shows none
openQA Infrastructure - action #68095: Migrate osd workers from SuSEfirewall2 to firewalld
action #68131: [cache] Cache service misses requests from workers
openQA Infrastructure - action #68161: alert raised about "Minion workers down" after osd-deployment 2020-06-17
action #68167: [tests][services] Tests for systemd services and/or the daemon wrapper scripts and wrong arguments should exit services with failure
action #68260: [packages] prevent submission of failed packages from devel:openQA:tested to Factory
action #68350: Improve package and base OS version support
action #68362: Click on link to *.qcow2 file display raw content inside the web browser
openQA Infrastructure - action #68410: repeated alerts about "no data" that are not actionable and recover themselves often enough
action #68473: The test job is stuck on BIOS.
action #68684: Port os-autoinst's build system to CMake
openQA Infrastructure - action #68785: [monitoring] Setup of QA generic monitoring instance
action #68851: Allow qemu host IP to be something other than 10.0.2.2
openQA Infrastructure - action #68872: job age max exceeds alarm threshold
action #68938: Try to reduce waiting time in case of qemu (early-)exits auto_review:"QEMU terminated before QMP connection could be established at /usr/lib/os-autoinst/OpenQA/Qemu/Proc.pm line 443":retry
action #68956: Restart the parent and child jobs of a test in a START_DIRECTLY_AFTER_TEST test chain
action #69058: openqa-review fails to download job module detail step text info files since around 2020-07-08
action #69082: javascript lint tests
action #69085: Make "last good" a link to a job instead of plain job ID
action #69088: Present changes between packages on openQA worker machines in "investigation"
action #69133: os-autoinst self-tests fail to remove tempdir on test failure
action #69148: regression: output of self-tests on circle CI is missing from log view, "tests" tab shows nothing (due to timeout)
action #69154: Improve package and base OS version support: Bump versions to Leap 15.2
action #69157: Improve package and base OS version support: Update contribution hints or review checklists for package impact
action #69160: Improve package and base OS version support: Enable tests for more repos
action #69178: workaround for #64776 using https://github.com/os-autoinst/scripts/blob/master/openqa-label-known-issues
openQA Infrastructure - action #69202: icinga alert "openqaworker3.suse.de/Number of threads"
action #69313: When using refspec for ppc for a particular job, PRODUCTDIR is set wrong
QA - action #69322: Automatic check for SUSE QA tools WIP-Limit based on tickets
openQA Tests - action #69328: [o3][s390x] Early fail on s390x workers: connection refused
action #69343: all jobs on o3 incomplete with auto_review:"Can.t locate object method.*pid.*"
action #69346: flaky/unstable os-autoinst test "22-svirt.t"
action #69355: [spike] redundant/load-balancing webui deployments of openQA
openQA Tests - action #69432: test fails with no module details after boot_ltp, broken run-time scheduling?
action #69439: not possible to delete expiration for api keys anymore over UI
action #69460: Video not available anymore from WebUI for some tests
openQA Infrastructure - action #69475: [tools] openQA child task fails to download asset created by parent job
action #69484: perl warning in os-autoinst t/99-full-stack.t "Subroutine OpenQA::Isotovideo::Utils::diag redefined at …os-autoinst/t/data/tests/main.pm line 19." not caught
action #69490: openqa webui sometimes displays the test suite instead of the needles in the needles drop down
openQA Infrastructure - action #69523: lessons learned: osd did not come up after reboot 2020-08-02
action #69553: job incompletes with "Failed to rsync tests: exit code 10":retry, improve user feedback
openQA Infrastructure - action #69577: Handle installation of the new "Storage Server"
openQA Infrastructure - action #69610: ipmi management interface of openqaworker-arm-3 is inaccessible
openQA Infrastructure - action #69613: osd-pre-deployment checks fail due to invalid certificates for stats.openqa-monitor.qa.suse.de
action #69637: Faster localhost uploads from worker to webui
openQA Infrastructure - action #69664: [osd][alert] CPU usage alert: IOwait too high
openQA Infrastructure - action #69667: missing monitoring data for vde after partitions where reordered
openQA Infrastructure - action #69694: openqa-worker systemd services running in osd which should not be enabled at all and have no tap-device configured auto_review:"backend died:.*tap.*is not connected to bridge.*br1":retry
openQA Infrastructure - action #69721: autoinst.qa is pingable and resolves to snipe.qa.suse.de, a VM, whereas there is a physical machine as well
openQA Infrastructure - action #69727: reduce heat in NUE-SRV2
action #69784: Workers not considered offline after ungraceful disconnect; stale job detection has no effect in that case
action #69820: os-autoinst: unstable/flaky/sporadic test 28-signalblocker.t
action #69946: [tools] when needle does not exist, test is paused until MAX_JOB_TIME
action #69979: Advanced job restarting via the web UI
action #69997: Streamline "restart" and "duplicate" routes of the REST-API
action #70120: os-autoinst: failed 18-qemu.t, prevents submission to Factory
action #70189: openQA-common package broken on Tumbleweed
action #70252: Dependency problem: Parent shown as passed instead of failed
action #70384: Align tools/tidy scripts
action #70648: Show serial tests in a more feasible way
action #70654: Create git-subrepo for tools/update-deps
action #70687: Download gru is attached to all scheduled jobs when doing 'isos post'
action #70720: Unable to restart a child from START_DIRECTLY_AFTER_TEST chain if another child has been restarted already
action #70723: Fix tests not to rely on `/var/lib/openqa/share` mountpoint
openQA Infrastructure - action #70768: obs_rsync_run and obs_rsync_update_builds_text Minion tasks fail frequently
action #70792: Optional link to fixed timestamp for tests overview
openQA Infrastructure - action #70834: [alert] Refine I/O time alerts for OSD
openQA Infrastructure - action #70843: OSD automatic deployment failed in Sept, 02, 2020
action #70873: Test fails because auto_review:"Encoder not accepting data":retry, video is missing
openQA Infrastructure - action #70885: [osd][alert] flaky file system alert: /assets
openQA Infrastructure - action #70909: salt CI jobs fail due to (now) missing python3 package
openQA Infrastructure - action #70918: gitlab pillar merge requests try to execute CI tests and fail because tests from states are not compatible
openQA Infrastructure - action #70939: [alert] ** PROBLEM Service Alert: ariel-opensuse.suse.de/root partition is WARNING **
openQA Infrastructure - action #70966: ipmi management interface of openqaworker-arm-3 is inaccessible
openQA Infrastructure - action #70969: openqaworker-arm-2 stuck in system management menu after reboot
action #70972: failed minion jobs with ""malformed JSON string, neither tag, array, object, number, string or atom, at character offset 0 (before \"(end of string)\") at /usr/lib/perl5/vendor_perl/5.26.1/Mojo/JSON.pm line 31.\n","
openQA Infrastructure - action #70975: [alert] too many failed minion jobs
openQA Infrastructure - action #70978: automatic reboots on o3 to activate new kernel versions
openQA Infrastructure - action #71011: [alert] workers alert
action #71095: [ci][os-autoinst] unable to read files in codecov reports, probably due to "opt" prefix
openQA Infrastructure - action #71098: openqaworker3 down but no alert was raised
action #71110: Reduce waiting time in case of os-autoinst shutdown
action #71137: osd deployment fails due to python-openqa_client failing in OBS
action #71176: Fail to show job details on OSD when using Firefox
openQA Infrastructure - action #71182: OBS package os-autoinst blocked since multiple days on "blocked: downloading 6 dod packages"
action #71185: job incompletes with auto_review:"setup failure: Cache service status error: Premature connection close":retry and does not retry, should we just automatically retry the connection?
openQA Infrastructure - action #71191: inform EngInfra automatically if the IPMI interfaces are not accessible
openQA Infrastructure - action #71224: [osd][alert] NTP offset alert
action #71236: job incompletes with auto_review:"backend died: Error connecting to VNC server <openqaw5-xen.qa.suse.de:5901>: IO::Socket::INET: connect: Connection refused"
action #71251: Forbid duplicate mapping keys in Job Templates YAML files
action #71323: outdated references to "coveralls" in os-autoinst
openQA Infrastructure - action #71332: [alert] failed systemd service on openqaworker6: "display-manager"
action #71368: 25-cache-service.t fails after passing all tests
action #71386: Stale job detection fails with "Can't locate object method "gru" via package "OpenQA::Scheduler" at /usr/share/openqa/script/../lib/OpenQA/Schema/Result/Jobs.pm"
action #71449: 25-cache-service.t fails repeatedly but circleCI receives the status as "success"
action #71464: Show output of failed tests within circleci log view, like on local prove calls, not only "Tests" tab
action #71467: add specific timeout values to all applicable openQA tests in t/
action #71476: Migrate from AssetPack to Webpack
action #71500: Potential optimization by skipping deployment checks in our tests (all except explicit deployment check tests)
action #71536: Unhandled perl warnings in t/ui/15-admin-workers.t, not failing tests as expected (possibly other test modules as well)
action #71551: unstable/flaky/sporadic t/04-scheduler.t test failing
action #71554: unstable/flaky/sporadic t/full-stack.t test failing in script waits on CircleCI
openQA Infrastructure - action #71575: [osd][alert] limited /assets - idea: ask EngInfra for slow+cheap storage from central server for /assets/fixed only
openQA Infrastructure - action #71635: [osd][alert] QA-Power8-4-kvm: CPU Usage alert - alert needs to be refined
action #71644: job stuck in "uploading", can not be cancelled, after what should have incompleted early
action #71758: [spike][timeboxed:20h] complete test definition from a yaml schedule file in local test distribution folder
action #71827: test incompletes with auto_review:"(?s)Failed to download.*Asset was pruned immediately after download":retry because worker cache prunes the asset it just downloaded
action #71845: tests fail in circleci with "Non-zero wait status: <non-negative number>" but "All <X> subtests passed"
action #71857: flaky/unstable/sporadic test coverage from t/34-developer_mode-unit.t
openQA Infrastructure - action #71893: can not reach openqaworker-arm-2-ipmi.suse.de DNS anymore from a gitlab CI pipeline
openQA Infrastructure - action #71908: [osd][alert] CPU usage alert (High IO wait on OSD)
openQA Infrastructure - action #71962: [osd-admins] openqaworker-arm-1: MergePoint email alert " Voltage sensor, warning event was asserted"
action #72055: download_asset tasks are not reliable
action #72082: Reduce test runtime, e.g. less reliance on test fixtures or test database instances
action #72127: check and/or reduce runtime of t/44-scripts.t
openQA Infrastructure - action #72136: [osd-admins] [Alerting] Workers alert during osd deployment, then "ok" after 1 minute, should not alert
openQA Infrastructure - action #72139: openQA services on OSD failed to connect to database
action #72196: t/24-worker-jobs.t fails in OBS
action #72238: websocket connection retry on flaky connections (was: SLE15-SP2 on AWS M6g (aarch64) machine fails to run a worker properly due to lots of Websocket connections lose)
action #72289: web-ui - test display is incomplete
action #72292: openqa-review fails to reference bugs from soft-fail references
action #72316: [tests][ci] circleci can fail in `zypper ref` due to temporary repository problems
action #72319: unstable test: t/ui/15-comments.t "got: 'Demo wrote less than a minute ago (last edited less than a minute ago)'" expected: 'Demo wrote less than a minute ago' (2nd try)
action #73123: t/14-grutasks.t shows errors but still succeeds
action #73126: No tests show unhandled output in main test summary log
action #73156: test not failing but error showing up t/25-downloader.t: "Cannot determine file type for '/tmp/DptzoOZjyt/test"
action #73162: t/01-test-utilities.t fails in "test would have failed"
openQA Infrastructure - action #73165: [osd] Consolidate "expensive+fast" and "cheap+slow" storage after realizing vdc is "cheap+slow" as well
openQA Infrastructure - action #73174: [osd][alert] Job age (scheduled) (median) alert
openQA Infrastructure - action #73183: Extend vdc (/assets)
openQA Infrastructure - action #73189: Upgrade o3 workers to openSUSE Leap 15.2 after openqa-aarch64 already done
action #73231: [microos]job incompletes with auto_review:"backend died: Virtio terminal and svirt serial terminal do not support send_key. Use"
openQA Infrastructure - action #73234: ipmi management interface of openqaworker7 is inaccessible ("no matching cipher suite") and not pingable
openQA Infrastructure - action #73246: [osd-admins][alert] openqaworker8: Memory usage alert
action #73285: test incompletes with auto_review:"(?s)Download of.*processed[^:].*Failed to download":retry , not helpful details about reason of error
action #73321: all jobs run on openqaworker8 incomplete:"Cache service status error from API: Minion job #46203 failed: Couldn't add download: DBD::SQLite::st execute failed: database disk image is malformed*"
openQA Infrastructure - action #73333: Failed systemd services alert (workers) flaky
action #73339: auto_review:"setup failure: Cache service status error from API: Minion job.* failed: Can't use an undefined value as a HASH reference at.*"
openQA Infrastructure - action #73342: all jobs run on openqaworker8 incomplete: "Cache service status error from API: Minion job .*failed: .*(database disk image is malformed|not a database)":retry
openQA Infrastructure - action #73348: The worker openqaworker-arm-2.suse.de is down, so the OSD deployment fails
openQA Infrastructure - action #73360: merging MRs in openqa/salt-states-openqa does not trigger pipeline in master
action #73366: auto-review: Improve output
openQA Infrastructure - action #73387: Cleanup of old needles from os-autoinst-needles-opensuse and os-autoinst-needles-sle
action #73396: job incompletes with auto_review:"setup failure: Failed to rsync tests: exit code 23":retry
openQA Infrastructure - action #73399: [osd][alert] Failed systemd services alert (workers) - many alert messages
openQA Infrastructure - action #73405: job incompletes with "(?s)openqaworker8.*terminated prematurely.*OpenCV Error: Insufficient memory"
action #73447: POC: Create openQA Web Application container image (feature)
action #73450: POC: Create openQA worker container image (feature)
QA - action #73468: SUSE QA tools team ticket process helpers: Set due date on tickets in redmine based on SLOs
openQA Infrastructure - action #73483: ipmitool from openSUSE Leap 15.2 and TW to our openQA workers fails with "no matching cipher suite", works on login.suse.de (SLES15SP1)
action #73486: Package devel:openQA:Leap:15.2/perl-DBD-Pg failed to build in openSUSE_Leap_15.2/ppc64le
openQA Infrastructure - action #73501: Bind mounts of fixed assets is racy
action #73567: openQA tour does not reliably save its state
openQA Infrastructure - action #73633: OSD partially unresponsive, triggering 500 responses, spotty response visible in monitoring panels but no alert triggered (yet)
openQA Infrastructure - action #75016: [osd-admins][alert] Failed systemd services alert (workers): os-autoinst-openvswitch.service (and var-lib-openqa-share.mount) on openqaworker-arm-2 and others
openQA Infrastructure - action #75055: grenache-1 can't connect to webui's over IPv4 only
openQA Infrastructure - action #75067: save_needle minion task fails because "Your account has been blocked"
action #75073: finalize_job_results minion task fails because 'Job xxx does not exist.'
action #75214: openqa-review fails to post reminder comments on bugzilla, errors in log "Encountered error trying to post a reminder comment on issue"
openQA Infrastructure - action #75220: all jobs run on openqaworker8 incomplete: "Cache service status error from API: Minion job .*failed: .*(database disk image is malformed|not a database)":retry
action #75232: error message when worker has no network (yet): Unable to serialize fatal error: Can't open file "base_state.json": Permission denied at /usr/lib/os-autoinst/bmwqemu.pm line 86."
openQA Infrastructure - action #75235: container image devel:openQA:ci/base is unresolvable with "nothing provides this-is-only-for-build-envs needed by cmake-mini"
openQA Infrastructure - action #75238: Upgrade osd workers and openqa-monitor to openSUSE Leap 15.2
openQA Infrastructure - action #75241: Upgrade o3 webUI host to openSUSE Leap 15.2
openQA Infrastructure - action #75244: Upgrade osd webUI host to openSUSE Leap 15.2
action #75256: Try out AV1 video codec as potential new default
openQA Infrastructure - action #75259: 100% of powerpc tests incomplete auto_review:"(?s)Running on power8.*qemu-system-ppc64: Requested safe cache capability level not supported by kvm":retry
action #75265: sporadic errors in test suite of perl-Mojo-IOLoop-ReadWriteProcess
openQA Infrastructure - action #75274: [osd-admins][alert][learning] Failed systemd services alert (workers): os-autoinst-openvswitch.service aborts retries after 60s and is not easily configurable
openQA Infrastructure - action #75277: enable "postfix" service again after #73633 resolved
action #75346: t/api/08-jobtemplates.t started failing in OBS checks
action #75370: unstable/flaky/sporadic t/full-stack.t failing on master (circleCI) "worker did not propagate URL for os-autoinst cmd srv within 1 minute"
openQA Tests - action #75373: Reserve repo links of SLE-15-SP2-Online-s390x-GM-Media1 and SLE-15-SP2-Online-x86_64-Build209.2-Media1
openQA Infrastructure - action #75403: salt "cheat sheet" for common commands to change/fix osd infrastructure
openQA Infrastructure - action #75445: unknown dashboards for "linux-fwcx" and "localhost" reappearing on monitor.qa
openQA Infrastructure - action #75448: OSD deployment fails because openqaworker-arm-3.suse.de is not connected
action #75454: sometimes clone job is incomplete because of api failure
action #76741: help popup next to "Submit comment" does not work anymore
openQA Infrastructure - action #76774: auto-review/openqa-label-known-issues does not conclude within GitLab CI
openQA Infrastructure - action #76783: research how hostnames with systemd work and make them static for all OSD related machines
openQA Infrastructure - action #76786: Configure static hostnames with salt for all salt nodes
openQA Infrastructure - action #76804: openqaworker-arm-2 was detected as offline but no grafana webhook action was triggered
openQA Infrastructure - action #76822: Fix /results over-usage on osd (was: sudden increase in job group results for SLE 15 SP2 Incidents)
openQA Infrastructure - action #76828: big job queue for ppc as powerqaworker-qam-1.qa and malbec.arch and qa-power8-5-kvm were not active
openQA Infrastructure - action #76876: Find a better (automated) way to inform infra about hanging (arm) workers
action #76900: unstable/flaky/sporadic t/full-stack.t test failing in CircleCI "worker did not propagate URL for os-autoinst cmd srv within 1 minute"
openQA Infrastructure - action #76903: openQA test modules failure statistics collection
action #76912: OpenQA::Script::Client throws perl warning "Wide character in print", should not be there
openQA Infrastructure - action #76924: Upgrade postgresql database version on o3 to default of Leap 15.2, i.e. postgres12 size:M
openQA Infrastructure - action #76927: Upgrade postgresql database version on osd to default of Leap 15.2, like on o3 size:M
openQA Infrastructure - action #76987: re-encode some videos from existing results to save space
action #76990: Improve documentation for redundant/load-balancing webui deployments of openQA
action #77008: Conduct openQA-in-openQA test on the latest *published* TW snapshot
openQA Infrastructure - action #77017: Add "how to do rollback of deployment" to https://gitlab.suse.de/openqa/osd-deployment/
action #77032: Improve t/25-cache.t runtime
openQA Infrastructure - action #77089: [osd][retrospective] multiple unattended alerts, unattended gitlab CI pipeline fails, all osd aarch64 workers offline
openQA Infrastructure - action #77101: fix selection of gitlab CI runners
openQA Infrastructure - action #77191: [easy][learning] ** PROBLEM Service Alert: openqaworker12.suse.de/Interface br1 is CRITICAL **
openQA Infrastructure - action #77209: workers on o3 machine rebel provide no "WORKER_HOSTNAME" value anymore but it shows up in journal of worker service
action #77215: In jobs API data reference a parent job group name as well
openQA Infrastructure - action #77218: gitlab CI pipeline openqa/grafana-webhook-actions failed with "You have reached your pull rate limit" for docker hub but should not use docker hub at all
action #77320: why does this show leap 15.1? https://build.opensuse.org/package/view_file/devel:openQA/os-autoinst_dev/_service:download_url:Dockerfile?expand=1u
openQA Infrastructure - action #77827: asg-qe.maintenance reported problems about too aggressive asset cleanup - bumping some settings
openQA Infrastructure - action #77836: login to aarch64.o.o fails with ssh keys and password, also not working over IPMI SoL
openQA Infrastructure - action #77839: Use external videoencoder in production on one o3 worker
openQA Infrastructure - action #77887: [tools][openqa] Enable automatic openQA investigation jobs for osd as well
openQA Infrastructure - action #77890: [easy] Extend OSD storage space for "results" to make bug investigation and failure archeology easier
action #77905: CI pipeline proof-of-concept running isotovideo
action #77929: UI build bar alignment broken on index page
QA - action #77944: Run "auto-review" more often but alarm less
openQA Infrastructure - action #78010: unreliable reboots on openqaworker3, likely due do openqa_nvme_format (was: [alert] PROBLEM Host Alert: openqaworker3.suse.de is DOWN)
action #78019: [sporadic] os-autoinst t/18-backend-qemu.t timed out in OBS checks after 10s
action #78043: unstable/flaky/inconsistent statement coverage in t/lib/OpenQA/SeleniumTest
action #78052: system stalled and then no output for three seconds, then test shuts down with no explicit message about reason and ends up incomplete
openQA Infrastructure - action #78058: [Alerting] Incomplete jobs of last 24h alert - again many incompletes due to corrupted cache, on openqaworker8
openQA Infrastructure - action #78064: failing logrotate on monitor.qa.suse.de due to mariadb/mysql?
openQA Infrastructure - action #78127: follow-up to #73633 - lessons learned and suggestions
action #78163: After OSD upgrade, many jobs incomplete with "Cache service status error 500: Internal Server Error"
openQA Infrastructure - action #78165: infrastructure task: After osd deployment 2020-11-18 many jobs incomplete with auto_review:"Cache service (status error from API|.*error 500: Internal Server Error)":retry
action #78169: after osd-deploy 2020-11-18 incompletes with auto_review:"Cache service (status error from API|.*error 500: Internal Server Error)":retry
QA - action #78187: Import and enable Testreport data into SMELT
openQA Infrastructure - action #78218: [openQA][worker] Almost all openQA workers become offline
action #78390: Worker is stuck in "broken" state due to unavailable cache service (was: and even continuously fails to (re)connect to some configured web UIs)
openQA Infrastructure - action #78438: openQA webui entry "Assigned worker" shows ip instead of names as formerly - manual cleanup work
openQA Infrastructure - action #80106: corrupted worker cache sqlite: Enlarge systemd service kill timeout temporarily
action #80108: HDD images not available for aarch64 Tumbleweed (cleaned-up too early?)
action #80118: test incompletes with auto_review:"(?s)Failed to download.*Asset was pruned immediately after download":retry, not effective on osd, or second fix needed
openQA Infrastructure - action #80128: openqaworker-arm-2 fails to download from openqa
openQA Infrastructure - action #80166: IPMI SOL does not show me anything anymore during bootup until the linux getty login prompt, can anybody confirm?
openQA Infrastructure - action #80178: gitlab.suse.de CI shared runners are unable to resolve addresses within suse.de domain
action #80202: jobs incomplete with auto_review:"setup failure: No workers active in the cache service":retry
action #80264: multimachine tests unable to get vars from its pair job
action #80268: Fix flaky coverage - t/05-scheduler-full.t
action #80274: Fix flaky coverage - t/lib/OpenQA/Test/Utils.pm size:M
action #80298: Fix flaky coverage - lib/OpenQA/WebSockets.pm
action #80334: job incompletes with auto_review:"(?s)terminated prematurely with corrupted state file.*No space left on device":retry , should automatically retrigger
action #80374: unstable/flaky/sporadic test t/ui/25-developer_mode.t
openQA Infrastructure - action #80380: salt deploy often or always fails in test-webui and deploy due to postgres files not found
openQA Infrastructure - action #80398: [openQA] increase storage space for baremetal_support VM on qamaster
openQA Infrastructure - action #80408: revert longer timeout override for openQA services as we could not see less problems with corrupted worker cache
action #80412: tests fail with auto_review:"(?s)version is 4\.6\.1606298538\.191b5988.*Can.*t locate object method.*code.*via package":retry
QA - action #80414: [proof-of-concept] Extend "auto-review" for failed jobs as well, start with o3
QA - action #80418: [learning] Fix parse errors in "openqa-investigate" "parse error: Invalid numeric literal at line 1, column 10"
action #80492: [easy] API help document says "Deletes …" for routes like "test_suites/:id" PUT and POST and looks inconsistent
openQA Infrastructure - action #80538: flaky and misleading alerts about "openQA minion workers alert" as well as "Minion Jobs alert"
openQA Infrastructure - action #80542: Configure "automatic power-on" after power loss for openqaworker1 (and all others)
openQA Infrastructure - action #80544: Ensure that IPMI for powerqaworker-qam works reliably
action #80576: [learning] perl warning in openqa-livehandler "Use of uninitialized value $ret in exit at /usr/share/openqa/script/openqa-livehandler line 31.", similar for openqa-websockets
openQA Infrastructure - action #80594: Needles are not pushed from o3 to github repo
openQA Infrastructure - action #80656: OSD deployment failed at 2020-12-02 because 'malbec.arch.suse.de' is down
action #80662: uninitialized value in string eq at .../load_templates line 137
openQA Tests - action #80670: [opensuse][gnuhealth] test fails in gnuhealth_client_preconfigure due to incorrect needle about "host field selected" when it is not selected
action #80682: Automatic tests for our openQA containers - worker only
action #80686: wrong label carry-over to a job where an additional module failed before the one failing in before
openQA Infrastructure - action #80688: Upgrade IO firmware for powerqaworker-qam-1
QA - action #80690: [qem] Testing of packages in different codestreams but in one incident
openQA Infrastructure - action #80734: GitLab pipeline trigger via Grafana fails due to TLS errors
action #80736: Trigger 'auto-review' from within openQA when jobs incomplete (or fail) , for testing: auto_review:"tests died: unable to load main.pm, check the log for the cause"
action #80746: test fails in await_install - openQA hasn't inherit the comments for incomplete job
openQA Infrastructure - action #80768: All workers in grenache-1 are broken at 2020-12-07
openQA Infrastructure - action #80770: "host up" alerts alerting on Sunday around 03:10Z and all going back to ok 30m later, should be prevented
action #80800: flaky/unstable t/full-stack.t, Failed test 'test 1 is running', Bailout called. Further testing stopped: URL for os-autoinst cmd srv not available
QA - action #80806: Extend "auto-review" for failed jobs as well - Generalize openqa-monitor-investigation-candidates to look at more than just one job group
QA - action #80808: Extend "auto-review" for failed jobs as well - enable same as on o3 but on osd
openQA Infrastructure - action #80812: Fix mail sending on o3 size:S
action #80826: Trigger 'auto-review' from within openQA when jobs incomplete on osd as well
action #80830: Trigger 'openqa-investigate' from within openQA when jobs fail on o3
openQA Infrastructure - action #80834: [alert] osd reported 502 errors, unresponsive, failing alerts for CPU usage since 2020-12-08 0840Z and minion jobs failed
action #80910: openQA workers read updated configuration, e.g. WORKER_CLASS, whenever they are ready to pick up new jobs
openQA Infrastructure - action #80956: update osd VM root disk VM config and resize root disk on osd
action #80962: os-autoinst development container image is still based on 15.1 but repo is switched to 15.2 already
action #80986: terminate worker process after executing all currently assigned jobs based on config/env variable
openQA Infrastructure - action #81020: QA-Power8-4-kvm start failed since reboot on 2020-12-13
openQA Infrastructure - action #81026: many jobs incomplete with auto_review:"(?s)Running on openqaworker-arm-2.*failed: 521 Connect timeout.*Result: setup failure":retry
action #81038: Avoid using Mojolicious::Routes::Route::route which is DEPRECATED in favor of Mojolicious::Routes::Route::any in asset pack
openQA Infrastructure - action #81046: openqaworker-arm-2.suse.de unreachable
openQA Infrastructure - action #81058: [tracker-ticket] Power machines can't find installed OS. Automatic reboots disabled for now
action #81060: Create a helm chart to deploy web UI in kubernetes
action #81068: openQA tests fail on OBS Tumbleweed
QA - action #81106: test out chat service notifications, e.g. matrix, from github actions size:M
action #81116: openqa-webui is failed to start because /var/lib/openqa/db/db.lock owned by vnc user
action #81118: automatic container tests for os-autoinst
action #81150: sporadic fail of os-autoins t/10-virtio_terminal.t
action #81180: fetchneedles fails on uncommited needles
action #81206: Trigger 'openqa-investigate' from within openQA when jobs fail on osd
openQA Infrastructure - action #81220: overdrive2.arch wants to be accepted with salt key on osd, what to do with this machine?
openQA Infrastructure - action #81228: many o3 workers not working on jobs as of 2020-12-21 (w4 seems to be ok)
openQA Infrastructure - action #81232: salt high state triggers multiple errors
action #81386: python-openqa_client fails to build on OBS for SLE 15/ 15 SP1
action #81492: openQA-in-openQA fails with 'function' object has no attribute 'func_name' and label:non_existing asset, candidate for removal or wrong settings
action #81703: The values of 'ISO' and 'HDD' includes absolute path in vars.json
action #81816: OSD deployment failed at 2021-01-06
action #81828: Jobs run into timeout_exceeded after the 'chattr' call, no output until timeout, auto_review:"(?s)Refusing to save an empty state file to avoid overwriting a useful one.*Result: timeout":retry
action #81859: openqa-investigate triggers incomplete sets for multi-machine scenarios
openQA Infrastructure - action #81884: openqa-webui should automatically restart on config updates
action #82067: Unable to use openqa-clone-custom-git-refspec - Returns 403 Not authorized
action #87698: openQA jobs can be triggered with single curl calls
openQA Infrastructure - action #87838: *alert* osd: Open database connections by user alert
openQA Infrastructure - action #87856: [alert] openqaworker3.suse.de/NRPE is CRITICAL
openQA Infrastructure - action #87883: osd infrastructure: services like "telegraf" are not enabled to start immediately on boot
action #87898: Add grafana alert for "broken workers" as reported by openQA
openQA Infrastructure - action #87970: tumbleweed container images in gitlab CI fail on salt calls, e.g. "KeyError: 'cmd.run_all'"
openQA Infrastructure - action #87979: Requirements for access to the openqa VM
action #88121: Trigger cleanup of results (or assets) if not enough free space based on configuration limit
action #88125: Get feedback and adapt the population scripts for testreports
QA - action #88127: [tools][qem] Test coverage DB for maintenance updates
QA - action #88183: [spike][timeboxed:20h] rancher: Create a simple selenium test for the UI as proof of concept
action #88187: Set the addresses in the "internal clients" configurable
openQA Infrastructure - action #88189: Firmware upgrade of qa-power8-4.qa.suse.de
openQA Infrastructure - action #88191: openqaworker2 boot ends in emergency shell
openQA Infrastructure - action #88225: osd infrastructure: Many failed systemd services on various machines
action #88247: manage password for multiple test users in os-autoinst for openQA tests
action #88347: upload_asset function fails for larger files
action #88363: openqa-client cannot return job info
openQA Infrastructure - action #88385: openqaworker3 host up alert is flaky
openQA Infrastructure - action #88450: Flaky NTP offset alert
action #88451: Static validation for containers
action #88452: script_run syntax error when & is at the end of command
action #88459: low-prio single-machine jobs can starve out high-prio multi-machine tests
openQA Infrastructure - action #88474: All workers on powerqaworker-qam-1 are offline
action #88482: Two absolute paths concatenated to form a default needle dir when PRODUCT_DIR/needles doesn't exist
QA - action #88485: [teregen] Fetch and store coverage info for each incident
action #88496: openQA "t" tests time out (again) and take long
action #88538: Since 2021-02-11 all openQA CI tests fail in codecov "406 Not Acceptable {'detail': ErrorDetail(string='Could not satisfy the request Accept header.', code='not_acceptable')}"
openQA Infrastructure - action #88546: Make use of the new "Storage Server", e.g. complete OSD backup
action #88564: text field for git commit details in needle editor
action #88594: codecov reports for individual files yield "forbidden" and files can not be shown
action #88603: Two identical jobs are created from one abandoned test
action #88609: Mojolicious 9.0 compatibility
action #88696: Executing /usr/share/openqa/script/openqa-gru fails
action #88745: [easy] Clicking on "Untracked" on /admin/assets yields "Not Found" page, better have no clickable link at all
action #88754: openQA-in-openQA tests always fail and results do not impact submission pipeline
openQA Infrastructure - action #88807: Open vSwitch command 'set_vlan' with arguments 'tap41 13' failed: 'tap41' is not connected to bridge 'br1' at /usr/lib/os-autoinst/backend/qemu.pm line 152.
openQA Infrastructure - action #88900: openqaworker13 was unreachable
openQA Infrastructure - action #88912: Assets cleaned up too early for Tumbleweed aarch64
action #88915: Codecov always fails with 404 Not Found - Build has already finished, uploads rejected
openQA Infrastructure - action #89047: Failed to commit needles, gitlab account blocked 2021-02-24
openQA Infrastructure - action #89050: OSD deployment blocked since two weeks - salt nodes failing
action #89056: incomplete jobs with auto_review:"isotovideo died: isotovideo received signal HUP":retry
action #89059: package checks fail due to timeout in openSUSE:Factory:ARM/aarch64
action #89077: os-autoinst Makefile is missing symlinks configuration
QA - action #89080: Flatpak OBS builds fail on tumbleweed
openQA Infrastructure - action #89113: [alert] PROBLEM Service Alert: openqa.suse.de/NTP Time is CRITICAL
action #89200: Switch OSD deployment to two-daily deployment
action #89206: openqa-review CI failure about "ImportError: No module named enum" in python 2.7 tests
action #89221: Statistics about the time it takes developers from QA to close tickets related to failing tests (creation - closing)
action #89224: Limit execution time of hook scripts run within Minion
openQA Infrastructure - action #89275: Flaky Incomplete jobs (not restarted) of last 24h alert and New incompletes alert triggered and back to OK
action #89281: Prevent investigation jobs to do any asset uploads to prevent overriding production assets
openQA Infrastructure - action #89419: Incomplete jobs after OSD deployment
QA - action #89491: Add flatpak build test to obs-build CI
openQA Infrastructure - action #89497: flaky Failed systemd services alert (except openqa.suse.de)
action #89548: presence on discord
openQA Infrastructure - action #89551: NFS mount fails after boot (reproducible on some OSD workers)
action #89557: presence on matrix
action #89620: openqa-clone-custom-git-refspec fails with: env: git: Permission denied
action #89710: Add redirection after login compatible to the new IDP login system size:M
action #89719: docker-compose up fails on master
action #89722: Need automatic check for docker-compose
action #89731: containers: The deploy using docker-compose is not stable and eventually fails
action #89752: containers: Add a worker service as part of the docker-compose
openQA Infrastructure - action #89815: osd-deployment blocked by openqaworker-arm-3 offline and not recovered automatically
openQA Infrastructure - action #89821: alert: PROBLEM Service Alert: openqa.suse.de/fs_/srv is WARNING (flaky, partial recovery with OK messages)
action #89899: Fix flaky coverage - t/ui/27-plugin_obs_rsync_status_details.t
action #89935: t/ui/27-plugin_obs_rsync_status_details.t fails in circleCI, master branch even
openQA Tests - action #89990: Error on tests/kernel/run_ltp.pm: Can't locate sle/tests/kernel/run_ltp.pm
openQA Infrastructure - action #89993: OSD deployment rollback failed finding "before" and "osd_deployment_rpm_q" files
action #90038: Better error handling when reading API key+secret from ~/.config/openqa/client.conf
QA - action #90047: Improve dependency detection in cpanspec
action #90131: don't obsolete jobs with the same build number in different job groups when using `_OBSOLETE`
action #90152: module results missing on quick job (on auto-restarting worker)
openQA Infrastructure - action #90161: [Alerting] malbec: Memory usage alert triggered briefly and turned OK within the next minute
action #90164: Make gitlab.suse.de/openqa/salt-states-openqa public
openQA Infrastructure - action #90170: Service for purging old kernels might run while system management is locked and fail
openQA Infrastructure - action #90275: Replacement openQA OSD aarch64 hardware (was: Dedicated non-rpi aarch64 hardware for manual testing)
action #90290: Relative paths for CASEDIR and others as default to be not bound to specific workers
action #90293: Optional relative paths for CASEDIR and others to be not bound to specific workers
action #90299: 414 failure when cloning job with very long job setting values
action #90302: Remote openQA worker fails to run tests from openqa-clone-custom-git-refspec due to differing paths
action #90362: Allow to customize worker engine by configuration
action #90371: Warnings "Subroutine JSON::PP::Boolean::(0+ redefined"
QA - action #90401: [teregen] Integrate coverage information in a presentable way into test template
QA - action #90404: [teregen] Update TeReGen for deployment on qam2
QA - action #90428: proposal: Plan to demo features in weekly meetings depending on due-date
QA - action #90441: Only set due date on tickets in progress
QA - action #90443: Reset due date on tickets when it doesn't apply / only apply due-date when assignee actually works on tickets
action #90614: CI test webui-docker-compose failed but PR was merged anyway
openQA Infrastructure - action #90629: administration of the new "Storage Server"
openQA Infrastructure - action #90635: NTP alerts coinciding with reboots of ppc64le host
openQA Infrastructure - action #90746: OSD deployment fails at 2021-04-07 because 'storage.qa.suse.de: Minion did not return'
openQA Infrastructure - action #90755: aarch64 openQA jobs on osd not properly processed since 2021-04-05
action #90761: os-autoinst-distri-opensuse CI checks fail due to `cpanm --installdeps` failing on Inline::Python
QA - action #90764: auto-merge github feature can be enabled for os-autoinst-distri-opensuse
action #90767: containers: Fix github test "webui-docker-compose" timeout
action #90818: [openqa][tool] Not able to get group_overview json output.
openQA Infrastructure - action #90857: Add redundancy for SAP multi machines tests - Extend openQA worker config to accomodate for upgraded RAM
action #90872: openQA / os-autoinst 'either does not dequeue its messages, or exhibits some other buggy client-behavior'
openQA Infrastructure - action #90875: powerqaworker-qam-1 is online but https://monitor.qa.suse.de/d/4KkGdvvZk/osd-status-overview shows "No Data"
openQA Infrastructure - action #90920: Restore IPMI access to malbec.arch.suse.de
action #90929: get OAuth2 to work with salsa.debian.org (gitlab)
openQA Infrastructure - action #90968: [alert] Multiple flaky incomplete job alerts on Sunday
action #90974: Make it obvious if qemu gets terminated unexpectedly due to out-of-memory
action #91097: CI: "webui-docker-compose" eventually fails building images
action #91157: [Alerting] web UI: Too many Minion job failures alert: limit_results_and_logs failed
action #91163: Many jobs on OSD and o3 are incomplete because of auto_review:"backend died: missing input at /usr/lib/os-autoinst/bmwqemu.pm line 202"
action #91232: Test t/ui/25-developer_mode.t failed in CI
action #91250: handle codecov Bash Uploader Security Update
action #91257: try out python backend for production tests in a new test distribution or os-autoinst-distri-openQA
action #91284: Prevent recursive apparmor profile inclusion
openQA Tests - action #91341: [qac][jeos][opensuse] test fails in vim: ssh test use the wrong console font
action #91347: [spike][timeboxed:18h] Support for archived jobs
QA - action #91356: Save openqa-review reports as gitlab CI artifacts
action #91377: CI: fix static-check-containers
action #91461: Test is missing webui results and fail despite all tests passed
action #91488: containers: openqa test "single_container_webui" eventually fails
action #91521: link to "first bad" in investigation tab
action #91527: Cleanup logging in autoinst-log.txt
openQA Infrastructure - action #91530: Severe performance problems on malbec
action #91542: openQA API what jobs were/are testing X incident/package
action #91578: [openQA][environmen][module] Missing perl use NetAddr::IP on openQA environmen
action #91601: Add "return to top" button on openQA pages, e.g. job details, index, group overview
action #91638: Ensure standard javascript code
action #91647: Making option to filter by flavor, test name on /tests/overview more prominent
action #91650: Resolve the most recent builds per job group on /tests/overview when showing multiple job groups
action #91652: Remind about the use of openqa-review in squads
action #91653: Python tests fail with generic error message regardless of the problem size:M
action #91658: Make "black certificate" stricter to only show when /tests/overview?todo=1 is empty, i.e. no unlabeled failures
action #91659: Add Playwright to devel:languages:perl
QA - action #91665: Add projects that we maintain to "SUSE Projects" list
QA - action #91674: SUSE QE Tools process proposal: Go further on regression fixing
action #91710: openQA API documentation mentions non-functional route /groups
action #91752: jenkins: Multiple missing fields and errors in configuration of openQA-in-openQA
action #91764: openqaworker13 has active jobs but not doing anything useful for hours
action #91773: Automatic replacement of openQA job URLs preview of openQA size:M
openQA Infrastructure - action #91779: Add monitoring for storage.qa.suse.de
action #91782: Add support for archived jobs
action #91785: UI representation for archived jobs
openQA Infrastructure - action #91803: openqaworker8 failed to reboot: [Alerting] openqaworker8: host up alert on 2021-04-25
action #91902: Tests incomplete with reason "Failed modules: …"
action #91971: crosscheck if testapi::eject_cd actually works
QA - action #92007: proposal: Move SUSE QE Tools workshop to 0700 UTC after multiple conflicts
openQA Infrastructure - action #92034: Re-enable openqa-investigate options after the black certificate now only shows properly "reviewed" jobs
openQA Infrastructure - action #92086: http://monitor.qa should redirect to https://monitor.qa.suse.de
openQA Infrastructure - action #92110: Several Job age (scheduled) alerts on Sunday
openQA Infrastructure - action #92113: [Alerting] openqaworker-arm-3: NTP offset alert
QA - action #92125: Move "MR" on submission tests into a separate job group
QA - action #92149: crosscheck status/goal/maintainership of a machine "qam2"
action #92161: CI: build-docs-nightly broken since May 3
action #92164: t/01-test-utilities.t fails in 'test would have failed' in unrelated PR
openQA Infrastructure - action #92167: remove workaround for PowerPC workers after boo#1174166 fixed and updates available
openQA Infrastructure - action #92176: [alert] openqaworker-arm-3 offline and CI pipeline unable to send email but stating "passed"
action #92179: `make coverage` on openSUSE Leap 15.2 can fail with "Bad Sereal header: Not a valid Sereal document"
openQA Infrastructure - action #92185: Our PowerPC worker machines need to be configured for automatic start after a power outage
action #92188: test reviewers are pointed to the "first bad vs. last good" comparison if current job is not already the first bad
QA - action #92221: support for SLE-Module-NVIDIA-Compute_15
QA - action #92287: Document access to raw openqa database
openQA Infrastructure - action #92302: NFS mount var-lib-openqa-share.mount often fails after boot of some workers
openQA Infrastructure - action #92338: [Alerting] File systems alert, / on osd
QA - action #92341: fix potential leak of tempfiles of openqa-label-known-issues
action #92344: many temp-folders left over from live openQA jobs, regression?
openQA Infrastructure - action #92467: Unit has `iscsid.socket` failed on some OSD workers since today's nightly reboot
action #92497: Perl test execution in os-autoinst runs succesfully but then fails with error
action #92584: Leap 15.3 support for os-autoinst+openQA
action #92665: Automatically validate code style for python code
openQA Infrastructure - action #92701: backup of etc/ from both o3 was not working since some days due to OOM on backup.qa.suse.de (was: … and osd not updated anymore since 2019)
action #92746: Log viewer in openQA webUI with color parsing
QA - action #92755: Investigate problems with post-commit hook for testreports
action #92761: devel:openQA/openqa_dev failed to build in containers15.2/x86_64 "no space left on device"
action #92764: Possible spam from check_for_unsent_reports.sh, repeated sending of "PASSED" and "FAILED" emails
openQA Infrastructure - action #92770: openqa.opensuse.org down, o3 VM reachable, no failed service
action #92788: Use openQA archiving feature on osd size:S
action #92800: tests: Can't open file "t/data/openqa/share/tests/opensuse/needles/inst-timezone-text.json"
action #92833: containers: Web UI cannot connect to scheduler
QA - action #92851: Workshop series proposal "How SUSE QE teams review openQA test results"
action #92893: containers, docker-compose: Ensure that the scheduler can connect to the websockets container size:M
action #92905: openqa-investigate creates jobs with wrong Git hash when triggered for jobs which already have a CASEDIR set to a git repo
openQA Infrastructure - action #92915: OSD deployment fails at 2021-05-21 because ' openqaworker-arm-2.suse.de Minion did not return'
action #92957: Add option to openqa-review to skip displaying all passed results
openQA Infrastructure - action #92969: Failing service os-autoinst-openvswitch after boot of some workers
openQA Infrastructure - action #92978: salt failed to write files on grenache-1.qa.suse.de 'workers.ini': Operation not permitted
openQA Infrastructure - action #92996: OSD deployment fails at 2021-05-24 because ' openqaworker-arm-3.suse.de Minion did not return'
openQA Infrastructure - action #93050: Proposal: Use openqaworker11 and openqaworker12 as normal workers and only pull out from production when necessary
action #93065: "parallel_failed" jobs show up on /tests/overview?todo=1 but these do not need a label
openQA Infrastructure - action #93071: o3 s390x worker machines unreachable
action #93083: unstable test in os-autoinst master
action #93086: unstable test in openQA master t/10-jobs.t exceeding runtime of 280s
action #93101: investigation jobs scheduled against specific git hashes load wrong product dir files
openQA Infrastructure - action #93195: [Alerting] Failed systemd services alert (except openqa.suse.de) on 2021-05-28, logrotate.service on openqaworker-arm-1
action #93345: Source URLs ending in .pm/steps/1/src.txt produce 404
openQA Infrastructure - action #93381: [O3]request to add an IPMI SUT to O3
action #93423: Migrate openqa_review CI to supported solution after travis-ci.org shutdown expected at or after 2021-06-15, e.g. Github Actions
action #93453: openQA CI: Certain Selenium tests are failing
QA - action #93522: [tools][qem] auto-approval of kgrafts and live-patches
openQA Infrastructure - action #93612: Several unhandled alerts triggered regarding incompletes and running out of space
openQA Infrastructure - action #93650: alert: PROBLEM Service Alert: openqa.suse.de/fs_/assets is WARNING
action #93677: t/full-stack.t runs into 9m make-level timeout exceeding test module internal timeout size:S
openQA Infrastructure - action #93683: osd-deployment failed due to storage.qa.suse.de not reachable by salt
QA - action #93710: Reference individual openqa-review reports in gitlab CI artifacts, e.g. using gitlab pages
action #93713: openqa-in-OpenQA fails in openqa-from-containers
action #93724: openqa-review: "Unassigned bugs" includes soft-fails to bugs with assignees
action #93727: Publish openqa-review reports with "--skip-passed"
action #93811: [timeboxed:20h] vanastasiadis: Learn about Perl and Perl development
openQA Tests - action #93826: test fails in openqa_webui - openqa_from_containers - package installation times out, just seems to be slow due to recent opensuse mirror problems
action #93853: os-autoinst t/01-test_needle.t fails in OBS on aarch64 in float number comparison
action #93880: os-autoinst "ci extended" CI job fails due to boo#1182451
QA - action #93916: idea from SUSE QE Tools retro 2021-06-11: Ask stakeholders for their list of priorities regarding our tasks
openQA Infrastructure - action #93919: salt gitlab CI pipeline fails applying openqa-webui restart state with "The following requisites were not found: watch: file: /etc/openqa/openqa.ini"
openQA Infrastructure - action #93922: grafana dashboard for "approximate result size by job group" fails to render any data with "InfluxDB Error: unsupported mean iterator type: *query.stringInterruptIterator"
action #93925: Optimize SQL on /tests/overview
QA - action #93931: [tools][qem] MTUI support multiple versions of package in incident
QA - action #93934: [tools][qem] template generator - multiple package version in incident
action #93940: text thumbnail preview feels inconsistent to other screenshots size:M
QA - action #93943: openqa-review pipeline fails accessing OSD test overview pages sometimes, more retries?
openQA Infrastructure - action #93961: Add redundancy for SAP multi machines tests - extend RAM on machines
openQA Infrastructure - action #93964: salt-states CI pipeline deploy step fails on some workers with "Unable to unmount /var/lib/openqa/share: umount.nfs: /var/lib/openqa/share: device is busy."
QA - action #93988: build openqa-review container within OBS with ca-certificates-suse package
openQA Infrastructure - action #94015: proper backup for osd
action #94060: unstable test t/ui/18-tests-details.t
action #94231: lessons learned from "make needle matching less forgiving"
openQA Infrastructure - action #94237: No alert about too many scheduled tests size:S
QA - action #94246: Collect data on update/packages and maintenance workload for 2020/2021
openQA Infrastructure - action #94312: [Alerting] web UI: Too many Minion job failures alert - likely due to openqa-client declared deprecated
openQA Infrastructure - action #94318: http://jenkins.qa.suse.de not reachable
action #94354: Optimize /dashboard_build_results and /group_overview/* pages
openQA Infrastructure - action #94438: OSD deployment fails at 2021-06-21 because ' openqaworker (arm-3 and arm-2) Minion did not return'
openQA Infrastructure - action #94456: no data from any arm host on https://stats.openqa-monitor.qa.suse.de/d/1bNU0StZz/automatic-actions?orgId=1
openQA Tests - action #94465: [tools] zkvm tests are scheduled by retriggering month old jobs even though we do not have any "svirt" workers anymore
QA - action #94486: Complete and deploy testreport db and page on smelt
openQA Infrastructure - action #94492: Configure retention/downsampling policy for monitoring data stored within InfluxDB size:M
openQA Infrastructure - action #94513: openqaworker-arm-3 not reachable and not recoverable over usual ways
action #94531: OpenQA worker randomly skips uploading artefacts for whole test modules size:M
openQA Infrastructure - action #94555: backup.qa.suse.de needs to be upgraded to Leap 15.2 (size:S)
openQA Infrastructure - action #94576: alert: PROBLEM Service Alert: openqa.suse.de/fs_/results is WARNING
QA - action #94661: [teregen] template generator - make sure to skip SUSE:Updates:openSUSE-* release targets for test report generation
action #94678: Multiple errors from hook scripts in openqa-gru service log (was: Missing openqa-investigate jobs on osd) size:M
openQA Tests - action #94714: [tools] Prevent rogue SCC servers to break maintenance tests auto_review:"2021-06-25.*no candidate needle.*scc-invalid-url":retry
action #94732: Provide link to /tests/overview of latest builds of all job groups within a parent job group size:M
action #94735: needles not found in `needles` subdirectory when CASEDIR is a git repository
openQA Infrastructure - action #94747: broken RPM database on openqaworker-arm-* during osd deployment
action #94753: Try out munin on o3
action #94762: openqa-review: Add mode of single-line todo lists size:M
action #94774: bug referral links only point to bugzilla.suse.com, not the specific bug size:M
action #94792: Also show "investigation" tabs for incomplete jobs
action #94838: Make qem-dashboard a proper public open source project size:M
action #94850: QEMU 6.0 fails to start if job has QEMU_NUMA=1
action #94880: Carry-over despite mismatch of failing job modules
action #94901: Simple validation of parameters on /tests/overview
openQA Infrastructure - action #94919: All arm workers down 2021-06-30 , NUE SRV2 Rack A8 was switched off by EngInfra size:S
action #94937: Distinguish comment types on jobs on /tests (maybe optional) size:S
openQA Infrastructure - action #94940: multiple network related problems, gitlab CI pipelines not working, workers not reachable, proxySCC not reachable
openQA Infrastructure - action #94949: Failed systemd services alert for openqaworker3 var-lib-openqa-share.automount
action #94991: os-autoinst OBS checks fail in t/10-virtio_terminal.t (https://github.com/os-autoinst/os-autoinst/issues/1686), sporadic failure or only in OBS size:S
openQA Infrastructure - action #94994: Tracebacks in the journal for salt-minion size:M
QA - action #95000: Remove SUSE-internal links to line teams in our team roster which should not be needed size:XS
action #95006: jenkins job trigger-openQA_in_openQA-TW fails with "/opt/openqa-scripts/trigger-openqa_in_openqa: line 68: client_prefix: unbound variable" size:S
action #95009: unstable/flaky test t/ui/12-needle-edit.t size:M
action #95024: openQA test t/ui/26-jobs_restart.t very unstable (already marked as unstable) size:M
QA - action #95033: openqa-review fails upon trying to access openqa with no-urlencoded addresses
action #95075: Find jobs matching search parameters over /api/v1/jobs (especially documentation) size:S
action #95105: osd-deployment pipelines fail and alerts are not handled size:M
openQA Infrastructure - action #95140: regression: salt state fails on one worker with "Passed invalid arguments to state.highstate: expected str, bytes or os.PathLike object, not list" size:M
action #95164: OBS Package build in github fails to provide updates, "OBS Package Build Expected — Waiting for status to be reported" size:M
action #95170: Increase code coverage of critical component OpenQA::Worker::Job without introducing slow-down due to subprocess coverage collection
action #95179: [sporadic] containers: eventually the tests fails on single_container_webui step with the error "Mojolicious::Plugin::AssetPack::Pipe::Sass requires"
action #95185: openqa-bootstrap ignores errors when systemd is not available
action #95188: Document how to properly configure GitLab pipeline notifications size:M
QA - action #95221: Provide more people with administrative access to services on qam2.suse.de, at least qa-maintenance/openQABot, i.e. increase bus factor size:M
QA - action #95254: [teregen] Out of memory exception during template generation
action #95281: error on "Next & previous results": ajax error message and no results showing up
action #95290: error in openqa-webui systemd service log "Mojo::Reactor::Poll: Timer failed: Can't open file "/tmp/RbB4AGlDAq/": No such file or directory at /usr/share/openqa/script/../lib/OpenQA/WebAPI/Controller/Running.pm line 211."
openQA Infrastructure - action #95293: Monitoring alerts on errors in logs on o3 (was: followup to: error on "Next & previous results": ajax error message and no results showing up) size:M
action #95296: openQA-in-openQA container tests fail with "/root/run_openqa.sh: line 6: This: command not found"
action #95299: Tests timeout with reason 'setup exceeded MAX_SETUP_TIME' on osd ppc64le workers auto_review:"Result: timeout":retry size:M
action #95317: openqa-bootstrap on Leap 15.3 from devel:openQA passes
action #95320: openqa-bootstrap on Leap 15.3 from official repos passes
action #95323: openqa-bootstrap support on current version of Leap - automatic test size:S
QA - action #95335: openqa-review: Only send the openqa_suse_de_status to openqa-suse-status@suse.de, ignore all others size:S
action #95374: All pull request openQA CI checks fail now in "webui-docker-compose": "Container is unhealthy" size:M
openQA Infrastructure - action #95443: Variants of Job age (scheduled) alerts on Grafana on Sunday and Monday size:S
openQA Infrastructure - action #95482: openqaworker-arm-3 offline and not automatically recovered due to gitlab CI failures
QA - action #95503: qam.suse.de api returns 500(Internal Server Error) for requests with valid logs
action #95581: ci: Use a git commit message style checker size:S
action #95662: Codecov: Green/red markers are off by one or more lines sometimes size:S
action #95715: Investigate non-fatal openqa-review pipeline error "ERROR:openqa_review.openqa_review:Could not find any soft failure reference within details of soft-failed job" size:M
action #95721: [Sporadic] containers: tests fail with "Test died: no candidate needle with tag(s) 'inst-console' matched" size:M
action #95730: on o3 in openqa-webui journal: "Jul 20 08:06:54 ariel openqa-webui-daemon[31559]: Use of uninitialized value in concatenation (.) or string at /usr/share/openqa/script/../lib/OpenQA/Schema/Result/Jobs.pm line 661." size:S
QA - action #95742: In openqa-investigate jobs add URL to original job as setting
QA - action #95746: Identify likely "sporadic" openQA tests with "openqa-investigate" size:M
QA - action #95765: Provide more people with administrative access to services on qam2.suse.de, adding ssh keys for existing tools team members
action #95830: Errors from hook scripts in openqa-gru service log ("parse error: Invalid numeric literal at...", JSON error) size:M
action #95836: [sporadic] CircleCI: openQA t/full-stack.t flaky size:M
action #95839: [sporadic] CircleCI: openQA t/ui/14-dashboard.t flaky size:M
action #95842: URLs containing "#" should be parsed as complete URLs size:S
action #95845: IP address for accessing host network from SUT no longer automatically assigned (under Tumbleweed) size:M
action #95848: [sporadic] CircleCI: openQA t/05-scheduler-full.t flaky size:M
action #95851: [sporadic] CircleCI: openQA t/43-scheduling-and-worker-scalability.t flaky size:M
QA - action #95854: Grafana doesn't show information during some minutes, but also we got alerts on the CPU after the recovery
openQA Infrastructure - action #95980: "no data" alert for "Incomplete jobs (not restarted)" but there should be no alert for "no data" size:S
openQA Infrastructure - action #95983: alert about "minion workers", alert triggered two times and turned green again
QA - action #95989: openqa-review gitlab CI pipeline jobs fail with "AttributeError: 'NoneType' object has no attribute 'group'"
action #95995: [sporadic][openqa-in-openqa] Test openqa_from_git eventually fails because of a timeout waiting for webui service size:M
action #96007: OpenQA jobs randomly time out during setup phase
action #96010: [qem] test fails in hawk_gui acquiring a lock as the support server ended prematurely after a '503 response: Service Unavailable; URL was http://openqa.suse.de/api/v1/mm/children'
QA - action #96016: [oscqam] Traceback on running ibs qam assigned -G qam-manager
action #96019: incomplete jobs with no logs get labeled with poo#61922 which is closed for long
action #96058: [spike] Filter test results on /tests or /tests/overview by regex match in modules size:M
openQA Infrastructure - action #96089: alert about "minion workers" - make meaning of grafana panel clear size:S
action #96122: Errors from hook scripts in openqa-gru service log ("grep: range out of order in character class") size:M
action #96191: Provide "fail-rate" of tests, especially multi-machine, in grafana size:M
action #96197: [alert] web UI: Too many Minion job failures alert size:M
openQA Infrastructure - action #96242: [alert] Disk I/O time for /dev/vde (/space-slow) alert 2021-07-28 size:M
action #96260: Failed to add GRE tunnel to openqaworker10 on most OSD workers, recent regression explaining multi-machine errors? size:M
openQA Infrastructure - action #96269: Define what a "complete OSD backup" should or can include
openQA Infrastructure - action #96272: Test failed incomplete: "Reason: backend died: QEMU exited unexpectedly, see log for details" size:M
action #96311: qemu error message is still "debug", should be "warn" or more severe size:S
action #96317: Unexpected warnings fail 25-cache.t if run with recent perl-DBD-SQLite
QA - action #96350: Improve openqa-review generation: Add date to index size:S
QA - action #96353: Improve openqa-review generation: If individual reports can not be generated, mention that as well with an explicit message
QA - action #96356: Improve openqa-review generation: Try to preserve old reports if individual new ones can not be generated
QA - action #96362: Improve openqa-review generation: Passthrough all vars and use git managed file for config
openQA Infrastructure - action #96380: "minion workers" alert shows <1 total minion workers if active == 1 size:M
action #96519: Fix flaky coverage - lib/OpenQA/Scheduler.pm size:M
action #96545: t/43-scheduling-and-worker-scalability.t fails in multiple OBS checks size:S
openQA Infrastructure - action #96551: Persistent records of systemd journal size:S
openQA Infrastructure - action #96554: Mitigate on-going disk I/O alerts size:M
action #96557: jobs run into MAX_SETUP_TIME, one hour between 'Downloading' and 'Download processed' and no useful output in between auto_review:"timeout: setup exceeded MAX_SETUP_TIME":retry
QA - action #96560: openqa-review gitlab CI pipeline failed to access remote repo
action #96561: Speed up `t/25-cache-service.t` by avoiding forking to run Minion jobs
action #96564: Speed up `t/ui/12-needle-edit.t` and `t/ui/21-admin-needles.t` by avoiding forking to run Minion jobs size:M
openQA Infrastructure - action #96618: The machine is unable to boot from PXE in NUE QA network due to TFTP open timeout
action #96623: Let workers declare themselves as broken if asset downloads are piling up size:M
action #96632: A lot of time is being spent stabilizing existing tests
action #96636: static-check-containers is flaky on GHA size:M
action #96707: Failed systemd services alert for osd in session-c26076.scope
openQA Infrastructure - action #96710: Error `Can't call method "write" on an undefined value` shows up in worker log leading to incompletes
openQA Infrastructure - action #96713: Slow grep in openqa-label-known-issues leads to high CPU usage
openQA Infrastructure - action #96719: recover imagetester with broken filesystem/hardware (was: automatic updates on imagetester don't work and it failed to come up after reboot)
action #96744: Report Product Bug from 15SP4 jobs failed due to wrong project name
QA - action #96752: 'openSUSE-SLE' product schedule jobs to OSD
openQA Infrastructure - action #96789: File systems alert 90.256 assets used size:M
QA - action #96792: openqa-review index page doesn't fail when a report is missing
openQA Infrastructure - action #96795: CPU Load alert and telegraf going between 41%, 98.5% and 115% CPU
openQA Infrastructure - action #96807: Web UI is slow and Apache Response Time alert got triggered
QA - action #96827: gitlab CI pipeline failed with Job failed: pod status is Failed size:S
QA - action #96899: jenkins.qa.suse.de is unavailable, error from /var/log/jenkins/jenkins.log "java.io.IOException: NSS initialization failed" size:M
openQA Infrastructure - action #96938: openqaworker10+13 are offline, reason unknown, let's fix other problems first size:M
openQA Infrastructure - action #96947: Availability of openQA webUI is above 100% size:S
action #96959: Job hooks trigger investigate jobs for passed/soft-failed size:M
QA - action #96968: qem-dashboard jobs api extension
QA - action #96998: Increase bus factor for bot-ng size:M
openQA Infrastructure - action #97043: job queue hitting new record 14k jobs
action #97046: Fix flaky coverage - t/05-scheduler-full.t size:M
QA - action #97091: teach SUSE QE Tools team about "Extreme Programming" and Kanban size:M
openQA Infrastructure - action #97136: [alert] multiple unhandled alerts about "broken workers" size:M
openQA Infrastructure - action #97139: [alert] multiple unhandled alerts about "malbec: Memory usage alert" size:M
action #97241: build failure of os-autoinst in 18-backend-qemu.t on OBS (timeout) size:S
openQA Infrastructure - action #97244: openqaworker-arm-3 is offline and EngInfra wants us to create JiraSD tickets instead of infra size:M
action #97304: Assets deleted even if there are still pending jobs size:M
openQA Infrastructure - action #97364: openqaworker-arm-2 and openqaworker-arm-3 seem to be offline, alerts had been triggered size:S
QA - action #97367: Update CI image QA:Maintenance/openSUSE-Leap-Container to Leap 15.3
openQA Infrastructure - action #97382: ARM automatic reboot pipeline does not fail if ipmitool fails size:S
QA - action #97403: openqa-review: Polish job group section titles in todo-only mode size:S
openQA Infrastructure - action #97406: gitlab emails to osd-admins@suse.de are not accepted as the ML is not in "To:" field but CC/BCC
openQA Infrastructure - action #97415: handle request about machines in SRV1 from SUSE-IT/Accenture to fill a spreadsheet with system properties
openQA Infrastructure - action #97418: Pipeline in salt-states-openqa can fail occasionally size:M
QA - action #97481: qam dashboard including development job group
openQA Infrastructure - action #97502: osd deployment failed due to openqaworker-arm-3 being down, needs to be worked around size:M
action #97508: openqa-label-known-issues not triggered correctly on o3 size:S
action #97541: Monitoring alerts on errors in logs on o3
openQA Infrastructure - action #97574: deployment failed in gitlab job with "ERROR: Job failed: execution took longer than 1h0m0s seconds"
action #97580: Automatic check for qa-tools backlog limits in Github Actions
openQA Infrastructure - action #97583: [spike] Configure retention/downsampling policy for monitoring data stored within InfluxDB on newer influxdb version
openQA Infrastructure - action #97658: many (maybe all) jobs on rebel within o3 run into timeout_exceeded "setup exceeded MAX_SETUP_TIME" size:M
QA - action #97661: [retro] find a better time for "estimation" meeting which is APAC friendly size:S
QA - action #97733: Bot fails on Failed to query latest publiccloud tools image using {settings['PUBLICCLOUD_TOOLS_IMAGE_QUERY']} and no aggregates are scheduled
openQA Infrastructure - action #97751: replacement setup for o3 s390x openQA workers size:M
action #97763: Event-based cleanup jobs triggered based on quota size:M
QA - action #97847: Feature request: Ability for bot-ng to schedule test runs on custom openQA instance
action #97856: [sporadic] openqa-review pipeline failed: ConnectionResetError size:M
openQA Infrastructure - action #97862: More openQA worker hardware for OSD size:M
openQA Infrastructure - action #97943: Increase number of CPU cores on OSD VM due to high usage size:S
action #97952: Handle "unable to upgrade ws to command server" better size:S
QA - action #97955: [openqabot] Possible TypeError during execution "'NoneType' object is not subscriptable"
action #97979: Asset cleanup takes very long to process 60k files in "other" size:M
action #98087: jobs on s390 fail with "Too few arguments for subroutine 'consoles::sshVirtsh::define_and_start'" after OSD deployment on 2021-09-03
action #98117: qemu-img calls fail with qemu 6.1.0 (need to specify format for backing file)
action #98153: OSD deployment fails in check state of packages at 2021-09-06
action #98186: Backlog checker does not take `QA` project issues into account for untriaged issues
QA - action #98198: [bot-ng] Fix auto-approve on updates without Incidents size:M
openQA Infrastructure - action #98243: salt-states-openqa pipeline/job "test-worker" fails because zypper reports an error size:M
action #98258: No results on /tests/overview w/o build
QA - action #98262: MTUI fails to export version of the package prior installation of the update
openQA Infrastructure - action #98307: Many jobs in o3 fail with timeout_exceeded on openqaworker1 auto_review:"timeout: setup exceeded MAX_SETUP_TIME":retry size:M
action #98388: Non-existing asset "uefi-vars" is still shown up on #downloads
QA - action #98394: Fix MicroOS scheduling to have incidents
action #98403: [easy][beginner] Make sections on build bars clickable size:S
action #98445: improve description for "Test module" UI element as followup to #96058
action #98448: As followup to #98087 tell mergify to only accept changes with 100% patch statement coverage
action #98460: Filter actual test results on /tests or /tests/overview by regex match in modules
QA - action #98475: links in https://progress.opensuse.org/projects/qa/wiki#Definition-of-DONE are dead size:S
action #98496: nightly circle CI job fails in "Install cached packages" with "rpm: no packages given for install" size:M
openQA Infrastructure - action #98499: [alert] web UI: Too many Minion job failures alert size:S
action #98577: Unknown ARRAY( variables matching HDD_1 or ISO in job settings
action #98604: Provide data about ratio of automatically approved SLE Maintenance incidents size:M
QA - action #98637: [timeboxed:20h] try to enable comments on IBS (and smelt) again from SUSE QA maintenance openQA test results size:M
action #98655: Fix flaky coverage - lib/OpenQA/Worker/Job.pm size:M
action #98664: Hiding test data
QA - action #98667: Unhandled [Alerting] Queue: State (SUSE) alert for > 4h size:M
QA - action #98673: [retro] Unhandled alert about job queue for nearly a day, users brought it up in chat, should have been picked up sooner size:S
action #98727: [tools][sle][aarch64] the published hdd can't be booted up due to wrong format
action #98769: Unresolvables in devel:openQA Leap 15.3 size:S
action #98841: qemu randomly fails to start on QA-Power8-5-kvm auto_review:"(?s)QA-Power8.*Failed to allocate KVM HPT of order 25 (try smaller maxmem?): Cannot allocate memory":retry size:M
action #98898: `t/05-scheduler-full.t` sometimes fails in CircleCI size:M
action #98901: [alert] Incomplete jobs (not restarted) of last 24h alert
QA - action #98916: Improve alert handling - weekly alert duty
openQA Infrastructure - action #98922: Run asset cleanup concurrently to results based on config
action #98940: mmapi calls can still fail despite retries
action #98946: Argument "4.2.1" isn't numeric in numeric ge (>=) at /usr/lib/os-autoinst/backend/qemu.pm line 686.
openQA Infrastructure - action #98979: monitor-post-deployment failed while arm3 was being rebooted by our automatic recovery
QA - action #98997: [tools] [smelt] [python3.6] Replace all older text formatting with f-strings
action #99009: Occasional "Unhandled rejected promise" failure when publishing AMQP messages size:M
openQA Infrastructure - action #99045: osd deployment fails due to zypper package resolution problem
action #99108: Automatically retry openQA bare-metal tests size:S
action #99111: Confirm or disprove that openQA bare-metal test loses SSH connection due to package updates size:M
openQA Infrastructure - action #99117: malbec 🍷️ is not reachable via ssh or ipmi
action #99123: ssh based backends can run into timeout if ssh connection is stuck
action #99126: os-autoinst CI GHA fails with weird C++ error?
action #99135: Provide ratio of tests by result in monitoring - by worker
action #99153: [Alerting] Incomplete jobs (not restarted) of last 24h alert on 2021-9-24
action #99168: drop unsupported distributions from devel:openQA
openQA Infrastructure - action #99189: Upgrade o3 workers to openSUSE Leap 15.3 size:M
openQA Infrastructure - action #99192: Upgrade osd workers and openqa-monitor to openSUSE Leap 15.3 size:M
openQA Infrastructure - action #99195: Upgrade o3 webUI host to openSUSE Leap 15.3 size:M
openQA Infrastructure - action #99198: Upgrade osd webUI host to openSUSE Leap 15.3 size:M
openQA Infrastructure - action #99201: Upgrade postgresql database version on o3 to default of Leap 15.3, i.e. postgres14 size:M
openQA Infrastructure - action #99204: Upgrade postgresql database version on osd to default of Leap 15.3, like on o3
action #99234: Proposal: Remove https://build.opensuse.org/project/show/devel:openQA:stable and https://build.opensuse.org/project/show/devel:openQA:V2
openQA Infrastructure - action #99240: Upgrade CI container image versions to Leap 15.3 size:M
openQA Infrastructure - action #99243: Upgrade qam hosts maintained by us to latest stable, i.e. Leap 15.3
action #99246: Published QCOW images appear to be uncompressed
openQA Infrastructure - action #99288: [Alerting] openqaworker-arm-5 and openqaworker-arm-4: host up alert on 2021-09-26 size:M
action #99327: openqa-webui-daemon: DBIx::Class::Storage::DBI::_dbh_execute(): DBI Exception: DBD::Pg::st execute failed: ERROR: syntax error at or near "AND"
openQA Infrastructure - action #99333: qa-maintenance/openQABot CI job fails after max retries to reach germ159.suse.cz
QA - action #99339: Find out with SUSE-IT what is the best way to collaborate based on tickets
action #99396: Incompletes with auto_review:"api failure: Failed to register .* 503":retry should be restarted automatically
action #99402: Incompletes with "backend died: Error connecting to VNC server.*: IO::Socket::INET: connect: Connection timed out":retry should be restarted automatically
QA - action #99411: openqa-review report openqa_suse_de_status.html missing from https://openqa.io.suse.de/openqa-review/openqa_suse_de_status.html, page is 404
action #99426: Asset cleanup takes very long to process 60k files in "other" - suboptimal logging?
action #99519: Investigation jobs triggered but missing comment on original job size:M
QA - action #99534: qa-maintenance / openQABot fails trying to access download.suse.de, we should provide certificates already in the container image size:S
action #99594: Fix flaky coverage - t/lib/OpenQA/Test/FullstackUtils.pm size:M
action #99597: Fix flaky coverage - lib/OpenQA/Worker/WebUIConnection.pm size:M
action #99654: Revisit decision in https://gitlab.suse.de/openqa/salt-states-openqa/-/merge_requests/545 regarding I/O alerts size:S
action #99663: Use more perl signatures - os-autoinst
action #99672: [openqa][tools] Non-existing `…-uefi-vars.qcow2` asset disturbs openqa-clone-job workflow
action #99675: Useless use of a constant ("can load python test module at a"...) in void context at ./08-autotest.t line 342.
action #99678: Speedup tidy
action #100503: Identify all "finalize_job_results" failures and handle them (report ticket or fix)
action #100709: openqa-review pipeline failed because details-* JSON is empty
openQA Infrastructure - action #100712: Investigate what broke git checkouts on o3
action #100850: Chromedriver crashes in openQA's CI
openQA Infrastructure - action #100859: investigate how to optimize /srv data utilization on OSD size:S
openQA Infrastructure - action #100862: Failed systemd services alert: session-50177.scope failed - but why?
action #100967: Use more perl signatures - openQA
action #100973: Cancel any scheduled jobs after a configurable timeout, e.g. days size:M
QA - action #100982: openqa-review: Do not post reminder comment if comment would be exactly the same as the last size:M
openQA Infrastructure - action #101006: Provide unique non-dictionary passwords for all our IPMI/HMC interfaces size:S
action #101015: [tools][sle][x86_64][aarch64][QEMUTPM] can openqa create swtpm device automatically? size:M
action #101030: Typing problems on aarch64
openQA Infrastructure - action #101033: openqaworker13: Too many Minion job failures alert - sqlite failed: database is locked size:M
QA - action #101045: presenter needed for SUSE QE Tools roadmap workshop 2021-11 on 2021-11-05
QA - action #101073: Smoke test of basic rust/cargo functionality size:S
openQA Infrastructure - action #101166: o3 was not auto-updated lately, due to unresolvables in devel:openQA for Leap 15.3
action #101262: Document running os-autoinst full-stack.t on OSD workers size:M
action #101265: Upgrade arm3 to Leap 15.3 and compare failure rate size:M
openQA Infrastructure - action #101271: Try Kernel:stable on arm4+arm5 and compare failure rate size:M
openQA Infrastructure - action #101373: worker openqa-aarch64 fails on cache
action #101379: Reduce amount of unhelpful log messages at debug level
action #101478: openqa-review pipeline failed because details-* JSON contains non-UTF8 char ? size:S
action #101520: [bot-ng] Stop very frequent scheduling of single incident jobs size:M
action #101533: Make text thumbnails easily distinguishable from info thumbnails
QA - action #101722: openqa-review: Do not post reminder comment in progress.opensuse.org if comment would be exactly the same as the last size:M
action #101734: Prevent "called on undefined file descriptor" in myjsonrpc.pm:40 in openQA t/full-stack.t
action #101779: osd deployment failed with non-zero status openqa-worker5
action #101884: openqa-review: list index out of range
action #101950: open.qa document need update base latest opensuse release for trace/debug firewalld size:S
openQA Infrastructure - action #101962: update BMC/IPMI firmware on openQA hosts
QA - action #102059: Integrate the Slack feed notifications feature for progress queries
openQA Infrastructure - action #102062: check state of packages fails with test '' = clean
openQA Infrastructure - action #102143: o3 ran out of disk space
action #102146: Deprecate os-autoinst backend::pvm
QA - action #102200: openqa-review pipeline failed: 'NoneType' object is not subscriptable, or failed with `assert self.issue_type == "bugzilla"` size:M
action #102206: Make bot-ng a proper public open source project size:M
action #102221: t/25-cache-service.t fails exceeding 90s timeout consistently size:M
openQA Infrastructure - action #102284: openQABot pipeline failed with simplejson.errors.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
action #102332: Unable to read *.json: Can't open file in o3 openQA logs /var/log/openqa size:M
QA - action #102335: qa-tools-backlog-assistant: Automated runs of the workflow are making pull requests impossible to accept
action #102347: bot-ng: repohash calculation
QA - action #102350: Move openqa-review CI to Github Actions
action #102374: Support use of force_result via ticket title in auto-review size:M
action #102428: Provide "fail-rate" alerting with ratio_mm_failed 5.360 size:M
action #102437: Job age alert median followed by max size:S
action #102440: openqa-review pipeline failed with assert self.issue_type == "bugzilla"
action #102464: Upgrade OBS package CI checks to Leap 15.3 (os-autoinst+openQA) size:M
action #102467: test fails in reconnect_mgmt_console with auto_review:"Test died: unexpected end of data at /usr/lib/os-autoinst/consoles/VNC.pm line 190."
action #102539: Non-fatal test modules aborting the whole job size:M
openQA Infrastructure - action #102575: Prevent false-positive ticket reporting for openqaworker-arm-3
action #102578: [sporadic] t/full-stack.t Failed test 'Expected result for job 1 not found' size:M
action #102584: openqa_review: Cannot run tox successfully locally
action #102641: openqa_review: Cannot build doctests successfully size:M
openQA Infrastructure - action #102650: Organize labs move to new building and SRV2
openQA Infrastructure - action #102713: Implement proper backups for o3 - list of installed packages
openQA Infrastructure - action #102716: Disable job triggering during the weekly SUSE IT maintenance window
action #102786: Text results "unable to read" when showing result page during test execution, i.e. while an openQA test is running size:M
openQA Infrastructure - action #102882: All OSD PPC64LE workers except malbec appear to have horribly broken cache service
openQA Infrastructure - action #102942: Failed systemd services alert: snapper-cleanup on QA-Power8-4-kvm fails size:M
action #102957: Better network performance monitoring - up-/download speed from cache service, e.g. in log file size:M
action #102975: Fix missing openqa.o.o data on metrics.o.o size:M
action #103029: Fix problems with os-autoinst's thread creation and the latest TBB version
action #103032: openQA "investigation" does not show "diff_to_last_good" content anymore?
action #103125: compile tests in os-autoinst-distri-openQA defined in template project size:S
openQA Infrastructure - action #103128: gitlab CI pipelines sporadically fail with "Could not resolve host: gitlab.suse.de", e.g. Recovery pipelines for ARM workers might fail during the maintenance window size:M
openQA Infrastructure - action #103149: Salt the dehydrated setup
QA - action #103197: qem-dashboard: Failed deployment pipeline
action #103287: Ensure openQA within standard repos of Leap 15.3 is working and no problem with object method "route" size:M
action #103329: os-autoinst: unhandled and confusing CI test ouput from git calls size:M
action #103398: Send an AMQP message on comments taken over size:M
action #103416: Better handle minion tasks failing with "Job terminated unexpectedly" - "limit_results_and_logs" size:M
action #103422: [sporadic] os-autoinst: 13-osutils.t:167 Failed test 'Exit code appear in log' in GHA size:M
action #103425: Ratio of multi-machine tests alerting with ratio_mm_failed 5.280 size:M
action #103467: [sporadic] os-autoinst: 14-isotovideo.t:133 'no fatal error recorded' size:M
action #103485: [sporadic] os-autoinst: flaky test t/13-osutils.t
openQA Infrastructure - action #103518: opencv update broke os-autoinst (was: Tests on Raspberry Pi 2/3/4 are broken)
openQA Infrastructure - action #103524: OW1: performance loss
action #103527: osd-deployment pipelines fail and alerts are not handled size:M
openQA Infrastructure - action #103530: failed systemd services alert - openqaworker-arm-3 - ovsdb-server size:M
openQA Infrastructure - action #103539: Update expired SSL certificate on monitor.qa.suse.de with dehydrated and salt, same as on OSD size:M
openQA Infrastructure - action #103554: o3 s390x worker instances 102+103 down whereas 101+104 are up
openQA Infrastructure - action #103575: [virtualization][3rd party hypervisor] Worker openqaw8-vmware.qa.suse.de is not reachable
action #103581: Many jobs on openqa.opensuse.org incomplete in ' timeout: setup exceeded MAX_SETUP_TIME'
action #103584: job incompletes with exception in OpenCV code "Assertion failed) ksize.width > 0 && ksize.width % 2 == 1 && ksize.height > 0 && ksize.height % 2 == 1 in function 'createGaussianKernels'"
openQA Infrastructure - action #103602: Reply to gitlab@suse.de pipeline email fails silently
action #103605: Most tests time out with perl-Mojo-IOLoop-ReadWriteProcess 0.28
action #103611: test failure: os-autoinst: 29-backend-driver.t:31 Failed test 'exit logged'
action #103617: Cover unhandled output in openQA "t" tests size:M
openQA Infrastructure - action #103683: [tools][sle][x86_64][aarch64][QEMUTPM] install package "swtpm" on x86_64 and aarch64 workers
action #103692: Only upgrade o3 workers if package checks are good, same as for o3 webui size:M
QA - action #103762: gitlab CI pipeline failed with Error cleaning up pod: Delete ... connect: connection refused Job failed (system failure): prepare environment: waiting for pod running ... i/o timeout. Check ... for more information
action #103765: Support for "todo" query parameter on /tests, same as /tests/overview size:M
action #103788: Run openQA full-stack test as part of os-autoinst CI tests size:M
action #103791: After module failure, the console is broken size:M
openQA Infrastructure - action #103848: No permission to view SD tickets filed for broken arm workers
action #103864: Support arch as filter parameter in jobs/overview API route (was: The openqa-cli api command seems not support …)
action #103953: Use openQA archiving feature on o3 size:S
openQA Infrastructure - action #103954: Run asset cleanup concurrently to results based on config on o3 as well
QA - action #103999: Include Mojo-IOLoop-ReadWriteProcess in our team wiki for usual processes size:S
QA - action #104004: Subscribe to Mojo-IOLoop-ReadWriteProcess github issues + pull requests size:S
action #104007: Support retry of openQA jobs based on test variables
QA - action #104025: Grafana: grenache-1: partitions usage (%) alert
QA - action #104031: bot-ng: Provide manual openQA trigger commands for retrying/retriggering/triggering special tests
action #104037: Can not save description in job group unless other values are changed as well
action #104077: backend died: Can't syswrite(IO::Socket::UNIX=GLOB(0x558d9dd5cb68), <BUFFER>): Broken pipe at /usr/lib/os-autoinst/backend/qemu.pm line 985 size:M
openQA Infrastructure - action #104085: openQABot pipeline failed with terminating connection due to administrator command size:S
openQA Infrastructure - action #104088: bot-ng pipeline consistently fails with KeyError: 'test_issues'
openQA Infrastructure - action #104106: [qe-core] test fails in await_install - Network peformace for ppc installations is decreasing size:S
action #104116: Better handle minion tasks failing with "Job terminated unexpectedly" - "scan_needles" size:M
action #104136: "archive_job_results" fail with "Unable to copy ... File exists" size:M
openQA Infrastructure - action #104142: osd-deployment pipeline failed: File ... not found on medium
action #104149: openqa_bugfetcher fails to build size:S
action #104164: The openSUSE package perl-App-cpanminus was suggested for removal but we rely on it within openQA
openQA Infrastructure - action #104172: osd service ca-certificates failed with "p11-kit: couldn't complete writing of file: /var/lib/ca-certificates/ca-bundle.pem.tmp: File exists"
action #104178: Increase OSD deployment rate from every second day to daily
action #104199: Prevent confusion when openQA comments look like both a bugref as well as label at the same time size:M
openQA Infrastructure - action #104217: Ask eng infra why thruk.suse.de stopped working
action #104220: openQA-inopenQA tests failing in boot step
action #104262: o3 responds with 500 on softfailed for job that "failed to load modules"
openQA Infrastructure - action #104304: Crosscheck results of https://github.com/os-autoinst/os-autoinst#verifying-a-runtime-environment on arm-1/2/3 vs. arm-4/5 to find out if arm-4/5 are "typing stable" size:M
openQA Infrastructure - action #104344: Inconsistent systemd default target in OSD infrastructure
openQA Infrastructure - action #104347: Lost salt connection from some hosts in OSD infrastructure due to no route to host "salt"
action #104350: [alert] failed systemd service on grenache-1, os-autoinst-openvswitch, turned to "ok" automatically size:M
action #104499: s390 tests fail with backend died: unable to extract assets: Too many arguments for subroutine 'backend::baseclass::do_extract_assets
action #104517: Connections to download.opensuse.org affecting openQA CI and o3
action #104520: Move svirt extract_asset code from os-autoinst-distri-opensuse to os-autoinst/backend/svirt.pm size:M
action #104577: Autoreview script pick up BSC references from softfail message but ignores WONTFIX tags
openQA Infrastructure - action #104607: Verify fix for https://bugzilla.suse.com/show_bug.cgi?id=1192126 regarding QEMU+OVMF, ensure updated packages on o3 size:M
action #104616: Make openQA labels clearly visible
action #104619: openqa_from_containers needs adapted post_fail_hook (was: tests fails due to openSUSE mirror problems with can't locate Mojo/Base.pm in INC in script/client)
openQA Infrastructure - action #104673: Access to o3 workers is not well-documented and not automated
action #104751: Extend "_SECRET_" variable handling to os-autoinst backend password variables
action #104827: openQA documentation generation deletes test API content
openQA Infrastructure - action #104835: Failed systemd service "systemd-journal-flush" on OSD
action #104836: [easy] perl warning about uninitialized value in pattern match when localhost not reachable size:S
action #104841: Prevent empty changelog messages from osd-deployment when there are no changes size:M
openQA Infrastructure - action #104869: Packages not building on Leap 15.2 and SLE15-SP3
action #104917: Raw (escaped) html shown in developer mode error message box
openQA Infrastructure - action #104965: openqaworker10 restarted into maintenance mode - reason unknown
openQA Infrastructure - action #104967: File systems alert repeatedly triggering on and off
openQA Infrastructure - action #104970: Add two OSD workers (openqaworker14+openqaworker15) specifically for sap-application testing
action #104971: [sporadic] os-autoinst t/99-full-stack.t sporadically fails in "Result in testresults/result-reload_needles.json is ok" size:M
action #104986: tests incomplete with: auto_review:"backend died: Too many arguments for subroutine.*consoles::vnc_base::get_last_mouse_set":retry
openQA Infrastructure - action #104992: [virtualization][3rd party hypervisor] vmware test run failed with credential errors
action #104995: Add UI element and help text for "todo" query parameter on /tests, similar as /tests/overview
action #105001: Add doc for "todo" query parameter on /tests, similar to /tests/overview size:S
action #105019: Make the number of previous restarts of a job more discoverable size:M
openQA Infrastructure - action #105040: [tools][sle][s390x] handle select_console 'root-console' failure if root ssh is not permitted in system with Common Criteria role
action #105061: [sporadic] os-autoinst: 29-backend-driver.t: Failed test 'log output for backend driver creation' size:S
action #105064: Reduce verbosity of openQA logging to improve performance and reduce storage requirements size:M
action #105127: Use more perl signatures - openQA - some simple classes size:S
action #105145: osd-deployment pipelines fail because ContainersNotInitialized
openQA Infrastructure - action #105169: Pipeline of openQABot project fails with "urllib.error.HTTPError: HTTP Error 503: Service Unavailable" causing alert/notification
coordination #12876: [epic] Offer a way for jobs to dynamically schedule children
coordination #19720: [epic] Simplify investigation of job failures
coordination #37958: [epic] self-tests in os-autoinst-distri-opensuse for impact on staging test schedule
coordination #39719: [saga][epic] Detection of "known failures" for stable tests, easy test results review and easy tracking of known issues
coordination #55364: [epic] Let's make codecov reports reliable
coordination #62420: [epic] Distinguish all types of incompletes
coordination #64746: [saga][epic] Scale up: Efficient handling of large storage to be able to run current tests efficiently but keep big archives of old results
coordination #65118: [epic] multimachine test fails with symptoms "websocket refusing connection" and other unclear reasons
coordination #66727: [epic] Define structure to define test suites not in openQA database
coordination #67723: [epic] Remote openQA worker fails to run tests from openqa-clone-custom-git-refspec
QA - coordination #69310: [epic] SUSE QA tools team ticket process helpers
openQA Infrastructure - coordination #69478: [epic] Upgrade o3+osd workers+webui to openSUSE Leap 15.2
openQA Infrastructure - coordination #75400: [epic] Improve documentation about how to manage infrastructure
QA - coordination #77899: [epic] Extend "auto-review" for failed jobs as well
openQA Infrastructure - coordination #78206: [epic] 2020-11-18 nbg power outage aftermath
coordination #80142: [saga][epic] Scale out: Redundant/load-balancing deployments of openQA, easy containers, containers on kubernetes
coordination #80150: [epic] Scale out openQA: Easier openQA setup
coordination #80372: [epic] Cleanup vars.json as initial information container between openQA worker and isotovideo
coordination #80546: [epic] Scale up: Enable to store more results
coordination #80828: [epic] Trigger 'auto-review' and 'openqa-investigate' from within openQA when jobs incomplete or fail on o3+osd
coordination #89062: [epic] Simplify review for SUSE QAM
coordination #90758: [epic] python bindings for openQA
coordination #91467: [epic] Surface openQA failures per squad in a single place
coordination #91518: [epic] Provide 'first bad' vs. 'last good' difference in investigation info
QA - coordination #91646: [saga][epic] SUSE Maintenance QA workflows with fully automated testing, approval and release
coordination #91914: [epic] Make reviewing openQA results per squad easier
coordination #93246: [epic] List all unreviewed failed (or incomplete) jobs on /tests on request size:M
coordination #93501: openqa-bootstrap fails to guess a value for platform on Leap 15.3
coordination #93609: [epic] openqa-bootstrap support on Leap 15.3
coordination #93883: [epic] Speedup openQA coverage tests with running minion jobs synchronously using new upstream "perform_jobs_in_foreground" mojo function
QA - coordination #94105: [epic] Use feedback from openqa-investigate to automatically inform on github pull requests, open tickets, weed out automatically failed tests
coordination #94258: [epic] deployment pipeline failed, alerts not handled
coordination #96185: [epic] Multimachine failure rate increased
coordination #96263: [epic] Exclude certain Minion tasks from "Too many Minion job failures alert" alert
openQA Infrastructure - coordination #96447: [epic] Failed systemd services and job age alerts
openQA Infrastructure - coordination #96974: [epic] Improve/reconsider thresholds for skipping cleanup
QA - coordination #97121: [epic] enable qa-maintenance/openQABot comments on smelt again
QA - coordination #98457: [epic] Handle openqa-review reminder comments on very old jobs better
coordination #98472: [epic] Scale out: Disaster recovery deployments of existing openQA infrastructures
coordination #98952: [epic] t/full-stack.t sporadically fails "clickElement: element not interactable" and other errors
coordination #99030: [epic] openQA bare-metal test dies due to lost SSH connection auto_review:"backend died: Lost SSH connection to SUT: Failure while draining incoming flow":retry
openQA Infrastructure - coordination #99183: [epic] Upgrade all our infrastructure, e.g. o3+osd workers+webui, to openSUSE Leap 15.3
coordination #99579: [epic][retro] Follow-up to "Published QCOW images appear to be uncompressed"
coordination #99660: [epic] Use more perl signatures in our perl projects
coordination #99831: [epic] Better handle minion tasks failing with "Job terminated unexpectedly"
coordination #101048: [epic] Investigate and fix higher instability of openqaworker-arm-4/5 vs. arm-1/2/3
openQA Infrastructure - coordination #102266: [epic] o3 ran out of disk space
coordination #102581: Proof of concept of t/full-stack.t on GitHubActions
coordination #102710: [epic] Improve our backup
coordination #102951: [epic] Better network performance monitoring
coordination #103122: [epic] Test distribution template for openQA tests to use, e.g. for automatic PR tests in os-autoinst-distri-openQA
QA - coordination #103845: [epic] Get involved with Mojo-IOLoop-ReadWriteProcess
coordination #103947: [saga][epic] Scale up: Future proof backup of o3+osd
coordination #105142: [epic] More statistics in job results
openSUSE admin - tickets #88751: Problem login into openqa.suse.de and openqa.opensuse.org