Project

General

Profile

Ready

Next tasks to be picked up by the QE tools team

84%

471 issues   (380 closed — 91 open)

Time tracking
Estimated time 30.00 hours
Issues by
action

376/447
coordination

4/23
tickets

0/1
Related issues
openQA Infrastructure - action #12506: openqa-scripts on osd/o3 should be deployed automatically (CD)
action #17886: [dashboard] Create full screen view for openQA dashboard
openQA Infrastructure - action #18142: [labs] Create wiki with up to date information (linklist included)
openQA Infrastructure - action #19238: setup pool devices+mounts+folders with salt(was: ext2 on workers busted)
action #27955: Allow the worker_bridge to sync job status from a slaveUI to a masterUI
action #29634: Enable QEMU snapshots function for virtio-gpu
action #32545: Provide warning when user creates/triggers misconfigured multi-machine clusters
action #34486: database of "test cases" or how to search for tests we have in openQA
action #36250: Add automated check for isotovideo::INTERFACE
action #38807: Make sure all used assets are accounted for in tests on osd (#38804) but not as assets which people need to download on cloning the job as the ISO is not actually used
action #39905: Job trying to download worker local file "aavmf-aarch64-vars.bin" into cache and fails with 404
action #43619: Improve workflow for dealing with openQA's dependencies
action #43712: Update upstream dockerfiles to provide an easy to use docker image of openQA-webui
action #43715: Update upstream dockerfiles to provide an easy to use docker image of workers
action #43718: Docker image for webui and workers are versioned and uploaded to obs registry
openQA Infrastructure - action #43934: Manage o3 infrastructure with salt again
openQA Infrastructure - action #44078: proper backup for o3
action #45062: Better visualization of incompletes - show module in which incomplete happens
action #48641: [epic] Trigger openQA tests in pull requests of any product github pull request
action #49202: Make audit events more accessible to testers
openQA Infrastructure - action #49694: openqaworker7 lost one NVMe
action #52532: Harmonisation for team text messaging communication structure
openQA Infrastructure - action #53471: machine aarch64.o.o often unresponsive and needs power-cycle
action #54557: All openQA tests incomplete but neither package build nor unit tests fail when a new file is added to os-autoinst without mentioning in Makefile.am
action #56534: support list of machines for YAML job templates
action #56591: Improve feedback for jobs in "Uploading"
openQA Infrastructure - action #57233: Properly handle transport errors in ObsRsync plugin
openQA Infrastructure - action #57680: o3: optimize fs utilization and resize root disk (was: change /var/log to bind mount to prevent out-of-space)
action #57689: asset cleanup jobs do not run on o3 (results cleanup works), workaround: unlock locks manually
action #58304: A personal activity view for developers
action #58823: os-autoinst is too slow pressing F2 causing ARM tests to fail in "boot_to_desktop"
action #59043: Fix unstable/flaky full-stack test, i.e. remove sleep, and ui tests
action #59273: module result missing for incompleting job
openQA Infrastructure - action #59300: auto_review:"DBus.*: The name org.opensuse.os_autoinst.switch was not provided":retry GRE tunnel settings not applied on initial setup / after reboot
action #59984: unstable test: t/05-scheduler-full.t
action #60029: OBS builds of os-autoinst fail for SLE12SP5 but not for SLE12SP3-4
action #60443: job incomplete with "(?s)process exited: 0.*isotovideo failed.*EXIT 1":retry but no further details what is wrong
action #61164: Use of uninitialized value / unhandled debug output in os-autoinst t/17-basetest.t
openQA Infrastructure - action #61994: VNC console corruption on aarch64
action #62159: Asset GRU download not done by web UI host if job scheduled by `isos post`, fails to download and then cloned (was: … using the Web UI)
action #62441: openqa-worker systemd service can timeout when stopping
openQA Infrastructure - action #63142: Upgrade firmware of ppc9 machine redcurrant
action #63451: Improve openqa-monitor-incompletes and openqa-label-known-issues to not report about incompletes with clone
action #63718: incomplete reason with just "quit"/"died" could provide more information
action #64120: Make weekly meetings available to community members
action #64520: Deal with jobs stuck in assigned state
openQA Infrastructure - action #64685: openqaworker1 showing NVMe problems "kernel: nvme nvme0: Abort status: 0x0"
action #64776: [cache][worker] cache service suddenly stopped to download assets, all subsequent jobs needing download incomplete auto_review:"setup failure: Cache service status error: Premature connection close":retry
action #64881: Reconsider triggering cleanup jobs
openQA Infrastructure - action #64983: Staging machine openqa-staging-1.qa.suse.de does not boot after update to Leap 15.1
action #65004: The asset generated by FORCE_PUBLISH works fine but does not show up in "Logs & Assets" and is also not mentioned in logs
action #65082: flaky/sporadic unstable failure in t/43-scheduling-and-worker-scalability.t
openQA Infrastructure - action #65130: Upgrade of firmware(s) for cloudberry (power9 machine)
openQA Infrastructure - action #65154: root partition on osd exceeds alert threshold, 90%, after osd deployment -> apply automatic reboots to OSD machines
openQA Infrastructure - action #65975: Monitoring for "scheduled but not executed" (was: perl-Mojolicious-8.37 broke WS connection for (some?) workers)
action #66066: incomplete with reason "died: terminated prematurely" but log shows error 404 failing to download asset into cache auto_review:"(?s)Download.*failed: 404.*No scripts"
action #66376: MM tests fail in obscure way when tap device is not present
action #66664: circleci jobs do not retry, are aborted with "Too long with no output (exceeded 10m0s): context deadline exceeded"
action #66682: `apache2.service` and `openqa-webui.service` wrongly started by `openqa-worker.service`
openQA Infrastructure - action #66709: Storage server for OSD and monitoring
action #66718: unstable/flaky/sporadic test: t/ui/27-plugin_obs_rsync_status_details.t
action #66721: Use GitHub actions for os-autoinst
action #67000: Job incompletes due to malformed worker cache database disk image with auto_review:"Cache service status error.*(database disk image is malformed|Specified job ID is invalid).*":retry
action #67039: [sporadic] os-autoinst tests stuck in OBS, likely t/18-backend-qemu.t ?
action #67573: "OpenID Connect" support in openQA
action #67723: Remote openQA worker fails to run tests from openqa-clone-custom-git-refspec
openQA Infrastructure - action #67804: use non-personal account and key for pushing needles on osd to gitlab.suse.de
action #67810: trigger OBS builds of openQA github pull requests
action #67972: one module failed to upload results ending an otherwise fine job as failed (was: Some tests are flagged as failed whereas they passed)
openQA Infrastructure - action #68053: powerqaworker-qam-1 fails to come up on reboot (repeatedly)
openQA Infrastructure - action #68077: alert about too many failed minion jobs but https://openqa.suse.de/minion/jobs?state=failed shows none
openQA Infrastructure - action #68095: Migrate osd workers from SuSEfirewall2 to firewalld
action #68131: [cache] Cache service misses requests from workers
openQA Infrastructure - action #68161: alert raised about "Minion workers down" after osd-deployment 2020-06-17
action #68167: [tests][services] Tests for systemd services and/or the daemon wrapper scripts and wrong arguments should exit services with failure
action #68260: [packages] prevent submission of failed packages from devel:openQA:tested to Factory
action #68350: Improve package and base OS version support
action #68362: Click on link to *.qcow2 file display raw content inside the web browser
action #68473: The test job is stuck on BIOS.
action #68684: Port os-autoinst's build system to CMake
openQA Infrastructure - action #68785: [monitoring] Setup of QA generic monitoring instance
action #68851: Allow qemu host IP to be something other than 10.0.2.2
openQA Infrastructure - action #68872: job age max exceeds alarm threshold
action #68938: Try to reduce waiting time in case of qemu (early-)exits auto_review:"QEMU terminated before QMP connection could be established at /usr/lib/os-autoinst/OpenQA/Qemu/Proc.pm line 443":retry
action #68956: Restart the parent and child jobs of a test in a START_DIRECTLY_AFTER_TEST test chain
action #69058: openqa-review fails to download job module detail step text info files since around 2020-07-08
action #69082: javascript lint tests
action #69085: Make "last good" a link to a job instead of plain job ID
action #69088: Present changes between packages on openQA worker machines in "investigation"
action #69133: os-autoinst self-tests fail to remove tempdir on test failure
action #69148: regression: output of self-tests on circle CI is missing from log view, "tests" tab shows nothing (due to timeout)
action #69154: Improve package and base OS version support: Bump versions to Leap 15.2
action #69157: Improve package and base OS version support: Update contribution hints or review checklists for package impact
action #69160: Improve package and base OS version support: Enable tests for more repos
action #69178: workaround for #64776 using https://github.com/os-autoinst/scripts/blob/master/openqa-label-known-issues
openQA Infrastructure - action #69202: icinga alert "openqaworker3.suse.de/Number of threads"
action #69313: When using refspec for ppc for a particular job, PRODUCTDIR is set wrong
QA - action #69322: Automatic check for SUSE QA tools WIP-Limit based on tickets
openQA Tests - action #69328: [o3][s390x] Early fail on s390x workers: connection refused
action #69343: all jobs on o3 incomplete with auto_review:"Can.t locate object method.*pid.*"
action #69346: flaky/unstable os-autoinst test "22-svirt.t"
action #69355: [spike] redundant/load-balancing webui deployments of openQA
openQA Tests - action #69432: test fails with no module details after boot_ltp, broken run-time scheduling?
action #69439: not possible to delete expiration for api keys anymore over UI
action #69460: Video not available anymore from WebUI for some tests
openQA Infrastructure - action #69475: [tools] openQA child task fails to download asset created by parent job
action #69484: perl warning in os-autoinst t/99-full-stack.t "Subroutine OpenQA::Isotovideo::Utils::diag redefined at …os-autoinst/t/data/tests/main.pm line 19." not caught
action #69490: openqa webui sometimes displays the test suite instead of the needles in the needles drop down
openQA Infrastructure - action #69523: lessons learned: osd did not come up after reboot 2020-08-02
action #69553: job incompletes with "Failed to rsync tests: exit code 10":retry, improve user feedback
openQA Infrastructure - action #69577: Handle installation of the new "Storage Server"
openQA Infrastructure - action #69610: ipmi management interface of openqaworker-arm-3 is inaccessible
openQA Infrastructure - action #69613: osd-pre-deployment checks fail due to invalid certificates for stats.openqa-monitor.qa.suse.de
action #69637: Faster localhost uploads from worker to webui
openQA Infrastructure - action #69664: [osd][alert] CPU usage alert: IOwait too high
openQA Infrastructure - action #69667: missing monitoring data for vde after partitions where reordered
openQA Infrastructure - action #69694: openqa-worker systemd services running in osd which should not be enabled at all and have no tap-device configured auto_review:"backend died:.*tap.*is not connected to bridge.*br1":retry
openQA Infrastructure - action #69721: autoinst.qa is pingable and resolves to snipe.qa.suse.de, a VM, whereas there is a physical machine as well
openQA Infrastructure - action #69727: reduce heat in NUE-SRV2
action #69784: Workers not considered offline after ungraceful disconnect; stale job detection has no effect in that case
action #69820: os-autoinst: unstable/flaky/sporadic test 28-signalblocker.t
action #69946: [tools] when needle does not exist, test is paused until MAX_JOB_TIME
action #69997: Streamline "restart" and "duplicate" routes of the REST-API
action #70120: os-autoinst: failed 18-qemu.t, prevents submission to Factory
action #70189: openQA-common package broken on Tumbleweed
action #70252: Dependency problem: Parent shown as passed instead of failed
action #70384: Align tools/tidy scripts
action #70648: Show serial tests in a more feasible way
action #70654: Create git-subrepo for tools/update-deps
action #70687: Download gru is attached to all scheduled jobs when doing 'isos post'
action #70720: Unable to restart a child from START_DIRECTLY_AFTER_TEST chain if another child has been restarted already
action #70723: Fix tests not to rely on `/var/lib/openqa/share` mountpoint
openQA Infrastructure - action #70768: obs_rsync_run and obs_rsync_update_builds_text Minion tasks fail frequently
action #70792: Optional link to fixed timestamp for tests overview
openQA Infrastructure - action #70834: [alert] Refine I/O time alerts for OSD
openQA Infrastructure - action #70843: OSD automatic deployment failed in Sept, 02, 2020
action #70873: Test fails because auto_review:"Encoder not accepting data":retry, video is missing
openQA Infrastructure - action #70885: [osd][alert] flaky file system alert: /assets
openQA Infrastructure - action #70909: salt CI jobs fail due to (now) missing python3 package
openQA Infrastructure - action #70918: gitlab pillar merge requests try to execute CI tests and fail because tests from states are not compatible
openQA Infrastructure - action #70939: [alert] ** PROBLEM Service Alert: ariel-opensuse.suse.de/root partition is WARNING **
openQA Infrastructure - action #70966: ipmi management interface of openqaworker-arm-3 is inaccessible
openQA Infrastructure - action #70969: openqaworker-arm-2 stuck in system management menu after reboot
action #70972: failed minion jobs with ""malformed JSON string, neither tag, array, object, number, string or atom, at character offset 0 (before \"(end of string)\") at /usr/lib/perl5/vendor_perl/5.26.1/Mojo/JSON.pm line 31.\n","
openQA Infrastructure - action #70975: [alert] too many failed minion jobs
openQA Infrastructure - action #70978: automatic reboots on o3 to activate new kernel versions
openQA Infrastructure - action #71011: [alert] workers alert
action #71095: [ci][os-autoinst] unable to read files in codecov reports, probably due to "opt" prefix
openQA Infrastructure - action #71098: openqaworker3 down but no alert was raised
action #71110: Reduce waiting time in case of os-autoinst shutdown
action #71137: osd deployment fails due to python-openqa_client failing in OBS
action #71176: Fail to show job details on OSD when using Firefox
openQA Infrastructure - action #71182: OBS package os-autoinst blocked since multiple days on "blocked: downloading 6 dod packages"
action #71185: job incompletes with auto_review:"setup failure: Cache service status error: Premature connection close":retry and does not retry, should we just automatically retry the connection?
openQA Infrastructure - action #71191: inform EngInfra automatically if the IPMI interfaces are not accessible
openQA Infrastructure - action #71224: [osd][alert] NTP offset alert
action #71236: job incompletes with auto_review:"backend died: Error connecting to VNC server <openqaw5-xen.qa.suse.de:5901>: IO::Socket::INET: connect: Connection refused"
action #71251: Forbid duplicate mapping keys in Job Templates YAML files
action #71323: outdated references to "coveralls" in os-autoinst
openQA Infrastructure - action #71332: [alert] failed systemd service on openqaworker6: "display-manager"
action #71368: 25-cache-service.t fails after passing all tests
action #71386: Stale job detection fails with "Can't locate object method "gru" via package "OpenQA::Scheduler" at /usr/share/openqa/script/../lib/OpenQA/Schema/Result/Jobs.pm"
action #71449: 25-cache-service.t fails repeatedly but circleCI receives the status as "success"
action #71464: Show output of failed tests within circleci log view, like on local prove calls, not only "Tests" tab
action #71467: add specific timeout values to all applicable openQA tests in t/
action #71500: Potential optimization by skipping deployment checks in our tests (all except explicit deployment check tests)
action #71536: Unhandled perl warnings in t/ui/15-admin-workers.t, not failing tests as expected (possibly other test modules as well)
action #71551: unstable/flaky/sporadic t/04-scheduler.t test failing
action #71554: unstable/flaky/sporadic t/full-stack.t test failing in script waits on CircleCI
openQA Infrastructure - action #71575: [osd][alert] limited /assets - idea: ask EngInfra for slow+cheap storage from central server for /assets/fixed only
openQA Infrastructure - action #71635: [osd][alert] QA-Power8-4-kvm: CPU Usage alert - alert needs to be refined
action #71644: job stuck in "uploading", can not be cancelled, after what should have incompleted early
action #71758: [spike][timeboxed:20h] complete test definition from a yaml schedule file in local test distribution folder
action #71827: test incompletes with auto_review:"(?s)Failed to download.*Asset was pruned immediately after download":retry because worker cache prunes the asset it just downloaded
action #71845: tests fail in circleci with "Non-zero wait status: <non-negative number>" but "All <X> subtests passed"
action #71857: flaky/unstable/sporadic test coverage from t/34-developer_mode-unit.t
openQA Infrastructure - action #71893: can not reach openqaworker-arm-2-ipmi.suse.de DNS anymore from a gitlab CI pipeline
openQA Infrastructure - action #71908: [osd][alert] CPU usage alert (High IO wait on OSD)
openQA Infrastructure - action #71962: [osd-admins] openqaworker-arm-1: MergePoint email alert " Voltage sensor, warning event was asserted"
action #72055: download_asset tasks are not reliable
action #72082: Reduce test runtime, e.g. less reliance on test fixtures or test database instances
action #72127: check and/or reduce runtime of t/44-scripts.t
openQA Infrastructure - action #72136: [osd-admins] [Alerting] Workers alert during osd deployment, then "ok" after 1 minute, should not alert
openQA Infrastructure - action #72139: openQA services on OSD failed to connect to database
action #72196: t/24-worker-jobs.t fails in OBS
action #72238: websocket connection retry on flaky connections (was: SLE15-SP2 on AWS M6g (aarch64) machine fails to run a worker properly due to lots of Websocket connections lose)
action #72289: web-ui - test display is incomplete
action #72292: openqa-review fails to reference bugs from soft-fail references
action #72316: [tests][ci] circleci can fail in `zypper ref` due to temporary repository problems
action #72319: unstable test: t/ui/15-comments.t "got: 'Demo wrote less than a minute ago (last edited less than a minute ago)'" expected: 'Demo wrote less than a minute ago' (2nd try)
action #73123: t/14-grutasks.t shows errors but still succeeds
action #73126: No tests show unhandled output in main test summary log
action #73156: test not failing but error showing up t/25-downloader.t: "Cannot determine file type for '/tmp/DptzoOZjyt/test"
action #73162: t/01-test-utilities.t fails in "test would have failed"
openQA Infrastructure - action #73165: [osd] Consolidate "expensive+fast" and "cheap+slow" storage after realizing vdc is "cheap+slow" as well
openQA Infrastructure - action #73174: [osd][alert] Job age (scheduled) (median) alert
openQA Infrastructure - action #73183: Extend vdc (/assets)
openQA Infrastructure - action #73189: Upgrade o3 workers to openSUSE Leap 15.2 after openqa-aarch64 already done
action #73231: [microos]job incompletes with auto_review:"backend died: Virtio terminal and svirt serial terminal do not support send_key. Use"
openQA Infrastructure - action #73234: ipmi management interface of openqaworker7 is inaccessible ("no matching cipher suite") and not pingable
openQA Infrastructure - action #73246: [osd-admins][alert] openqaworker8: Memory usage alert
action #73285: test incompletes with auto_review:"(?s)Download of.*processed[^:].*Failed to download":retry , not helpful details about reason of error
action #73321: all jobs run on openqaworker8 incomplete:"Cache service status error from API: Minion job #46203 failed: Couldn't add download: DBD::SQLite::st execute failed: database disk image is malformed*"
openQA Infrastructure - action #73333: Failed systemd services alert (workers) flaky
action #73339: auto_review:"setup failure: Cache service status error from API: Minion job.* failed: Can't use an undefined value as a HASH reference at.*"
openQA Infrastructure - action #73342: all jobs run on openqaworker8 incomplete: "Cache service status error from API: Minion job .*failed: .*(database disk image is malformed|not a database)":retry
openQA Infrastructure - action #73348: The worker openqaworker-arm-2.suse.de is down, so the OSD deployment fails
openQA Infrastructure - action #73360: merging MRs in openqa/salt-states-openqa does not trigger pipeline in master
action #73366: auto-review: Improve output
openQA Infrastructure - action #73387: Cleanup of old needles from os-autoinst-needles-opensuse and os-autoinst-needles-sle
action #73396: job incompletes with auto_review:"setup failure: Failed to rsync tests: exit code 23":retry
openQA Infrastructure - action #73399: [osd][alert] Failed systemd services alert (workers) - many alert messages
openQA Infrastructure - action #73405: job incompletes with "(?s)openqaworker8.*terminated prematurely.*OpenCV Error: Insufficient memory"
action #73447: POC: Create openQA Web Application container image (feature)
action #73450: POC: Create openQA worker container image (feature)
QA - action #73468: SUSE QA tools team ticket process helpers: Set due date on tickets in redmine based on SLOs
openQA Infrastructure - action #73483: ipmitool from openSUSE Leap 15.2 and TW to our openQA workers fails with "no matching cipher suite", works on login.suse.de (SLES15SP1)
action #73486: Package devel:openQA:Leap:15.2/perl-DBD-Pg failed to build in openSUSE_Leap_15.2/ppc64le
openQA Infrastructure - action #73501: Bind mounts of fixed assets is racy
action #73567: openQA tour does not reliably save its state
openQA Infrastructure - action #73633: OSD partially unresponsive, triggering 500 responses, spotty response visible in monitoring panels but no alert triggered (yet)
openQA Infrastructure - action #75016: [osd-admins][alert] Failed systemd services alert (workers): os-autoinst-openvswitch.service (and var-lib-openqa-share.mount) on openqaworker-arm-2 and others
openQA Infrastructure - action #75055: grenache-1 can't connect to webui's over IPv4 only
openQA Infrastructure - action #75067: save_needle minion task fails because "Your account has been blocked"
action #75073: finalize_job_results minion task fails because 'Job xxx does not exist.'
action #75214: openqa-review fails to post reminder comments on bugzilla, errors in log "Encountered error trying to post a reminder comment on issue"
openQA Infrastructure - action #75220: all jobs run on openqaworker8 incomplete: "Cache service status error from API: Minion job .*failed: .*(database disk image is malformed|not a database)":retry
action #75232: error message when worker has no network (yet): Unable to serialize fatal error: Can't open file "base_state.json": Permission denied at /usr/lib/os-autoinst/bmwqemu.pm line 86."
openQA Infrastructure - action #75235: container image devel:openQA:ci/base is unresolvable with "nothing provides this-is-only-for-build-envs needed by cmake-mini"
openQA Infrastructure - action #75238: Upgrade osd workers and openqa-monitor to openSUSE Leap 15.2
openQA Infrastructure - action #75241: Upgrade o3 webUI host to openSUSE Leap 15.2
openQA Infrastructure - action #75244: Upgrade osd webUI host to openSUSE Leap 15.2
action #75256: Try out AV1 video codec as potential new default
openQA Infrastructure - action #75259: 100% of powerpc tests incomplete auto_review:"(?s)Running on power8.*qemu-system-ppc64: Requested safe cache capability level not supported by kvm":retry
action #75265: sporadic errors in test suite of perl-Mojo-IOLoop-ReadWriteProcess
openQA Infrastructure - action #75274: [osd-admins][alert][learning] Failed systemd services alert (workers): os-autoinst-openvswitch.service aborts retries after 60s and is not easily configurable
openQA Infrastructure - action #75277: enable "postfix" service again after #73633 resolved
action #75346: t/api/08-jobtemplates.t started failing in OBS checks
action #75370: unstable/flaky/sporadic t/full-stack.t failing on master (circleCI) "worker did not propagate URL for os-autoinst cmd srv within 1 minute"
openQA Tests - action #75373: Reserve repo links of SLE-15-SP2-Online-s390x-GM-Media1 and SLE-15-SP2-Online-x86_64-Build209.2-Media1
openQA Infrastructure - action #75403: salt "cheat sheet" for common commands to change/fix osd infrastructure
openQA Infrastructure - action #75445: unknown dashboards for "linux-fwcx" and "localhost" reappearing on monitor.qa
openQA Infrastructure - action #75448: OSD deployment fails because openqaworker-arm-3.suse.de is not connected
action #75454: sometimes clone job is incomplete because of api failure
action #76741: help popup next to "Submit comment" does not work anymore
openQA Tests - action #76747: [tools] Needle masks do not handle matches with offsets
openQA Infrastructure - action #76774: auto-review/openqa-label-known-issues does not conclude within GitLab CI
openQA Infrastructure - action #76783: research how hostnames with systemd work and make them static for all OSD related machines
openQA Infrastructure - action #76786: Configure static hostnames with salt for all salt nodes
openQA Infrastructure - action #76804: openqaworker-arm-2 was detected as offline but no grafana webhook action was triggered
openQA Infrastructure - action #76822: Fix /results over-usage on osd (was: sudden increase in job group results for SLE 15 SP2 Incidents)
openQA Infrastructure - action #76828: big job queue for ppc as powerqaworker-qam-1.qa and malbec.arch and qa-power8-5-kvm were not active
openQA Infrastructure - action #76876: Find a better (automated) way to inform infra about hanging (arm) workers
action #76900: unstable/flaky/sporadic t/full-stack.t test failing in CircleCI "worker did not propagate URL for os-autoinst cmd srv within 1 minute"
openQA Infrastructure - action #76903: openQA test modules failure statistics collection
action #76912: OpenQA::Script::Client throws perl warning "Wide character in print", should not be there
openQA Infrastructure - action #76924: Upgrade postgresql database version on o3 to default of Leap 15.2, i.e. postgres12
action #76978: How to run an openQA test in 5 minutes
action #76984: [epic] Automatically remove assets+results based on available free space
openQA Infrastructure - action #76987: re-encode some videos from existing results to save space
action #76990: Improve documentation for redundant/load-balancing webui deployments of openQA
action #77008: Conduct openQA-in-openQA test on the latest *published* TW snapshot
openQA Infrastructure - action #77017: Add "how to do rollback of deployment" to https://gitlab.suse.de/openqa/osd-deployment/
action #77032: Improve t/25-cache.t runtime
openQA Infrastructure - action #77089: [osd][retrospective] multiple unattended alerts, unattended gitlab CI pipeline fails, all osd aarch64 workers offline
openQA Infrastructure - action #77101: fix selection of gitlab CI runners
openQA Infrastructure - action #77191: [easy][learning] ** PROBLEM Service Alert: openqaworker12.suse.de/Interface br1 is CRITICAL **
openQA Infrastructure - action #77209: workers on o3 machine rebel provide no "WORKER_HOSTNAME" value anymore but it shows up in journal of worker service
action #77215: In jobs API data reference a parent job group name as well
openQA Infrastructure - action #77218: gitlab CI pipeline openqa/grafana-webhook-actions failed with "You have reached your pull rate limit" for docker hub but should not use docker hub at all
openQA Infrastructure - action #77317: chat bot to conduct daily checks, alerts, reminders, etc.
action #77320: why does this show leap 15.1? https://build.opensuse.org/package/view_file/devel:openQA/os-autoinst_dev/_service:download_url:Dockerfile?expand=1u
action #77728: Test incompletes with auto_review:"(?s)Error in .*isotovideo: backen.* corrupted size vs. prev_size":retry
openQA Infrastructure - action #77827: asg-qe.maintenance reported problems about too aggressive asset cleanup - bumping some settings
openQA Infrastructure - action #77836: login to aarch64.o.o fails with ssh keys and password, also not working over IPMI SoL
openQA Infrastructure - action #77839: Use external videoencoder in production on one o3 worker
openQA Infrastructure - action #77842: Use external videoencoder in production on all o3 machines
openQA Infrastructure - action #77887: [tools][openqa] Enable automatic openQA investigation jobs for osd as well
openQA Infrastructure - action #77890: [easy] Extend OSD storage space for "results" to make bug investigation and failure archeology easier
action #77905: CI pipeline proof-of-concept running isotovideo
action #77929: UI build bar alignment broken on index page
QA - action #77944: Run "auto-review" more often but alarm less
openQA Infrastructure - action #78010: unreliable reboots on openqaworker3, likely due do openqa_nvme_format (was: [alert] PROBLEM Host Alert: openqaworker3.suse.de is DOWN)
action #78019: [sporadic] os-autoinst t/18-backend-qemu.t timed out in OBS checks after 10s
action #78043: unstable/flaky/inconsistent statement coverage in t/lib/OpenQA/SeleniumTest
action #78052: system stalled and then no output for three seconds, then test shuts down with no explicit message about reason and ends up incomplete
openQA Infrastructure - action #78058: [Alerting] Incomplete jobs of last 24h alert - again many incompletes due to corrupted cache, on openqaworker8
openQA Infrastructure - action #78064: failing logrotate on monitor.qa.suse.de due to mariadb/mysql?
openQA Infrastructure - action #78127: follow-up to #73633 - lessons learned and suggestions
action #78163: After OSD upgrade, many jobs incomplete with "Cache service status error 500: Internal Server Error"
openQA Infrastructure - action #78165: infrastructure task: After osd deployment 2020-11-18 many jobs incomplete with auto_review:"Cache service (status error from API|.*error 500: Internal Server Error)":retry
action #78169: after osd-deploy 2020-11-18 incompletes with auto_review:"Cache service (status error from API|.*error 500: Internal Server Error)":retry
QA - action #78187: Import and enable Testreport data into SMELT
openQA Infrastructure - action #78218: [openQA][worker] Almost all openQA workers become offline
action #78390: Worker is stuck in "broken" state due to unavailable cache service (was: and even continuously fails to (re)connect to some configured web UIs)
openQA Infrastructure - action #78438: openQA webui entry "Assigned worker" shows ip instead of names as formerly - manual cleanup work
openQA Infrastructure - action #80106: corrupted worker cache sqlite: Enlarge systemd service kill timeout temporarily
action #80108: HDD images not available for aarch64 Tumbleweed (cleaned-up too early?)
action #80118: test incompletes with auto_review:"(?s)Failed to download.*Asset was pruned immediately after download":retry, not effective on osd, or second fix needed
openQA Infrastructure - action #80128: openqaworker-arm-2 fails to download from openqa
openQA Infrastructure - action #80166: IPMI SOL does not show me anything anymore during bootup until the linux getty login prompt, can anybody confirm?
openQA Infrastructure - action #80178: gitlab.suse.de CI shared runners are unable to resolve addresses within suse.de domain
action #80202: jobs incomplete with auto_review:"setup failure: No workers active in the cache service":retry
action #80226: job incomplete with autoinst-log.txt ending just in the middle
action #80264: multimachine tests unable to get vars from its pair job
action #80268: Fix flaky coverage - t/05-scheduler-full.t
action #80274: Fix flaky coverage - t/lib/OpenQA/Test/Utils.pm
action #80298: Fix flaky coverage - lib/OpenQA/WebSockets.pm
action #80334: job incompletes with auto_review:"(?s)terminated prematurely with corrupted state file.*No space left on device":retry , should automatically retrigger
action #80372: [epic] Cleanup vars.json as initial information container between openQA worker and isotovideo
openQA Infrastructure - action #80380: salt deploy often or always fails in test-webui and deploy due to postgres files not found
openQA Infrastructure - action #80398: [openQA] increase storage space for baremetal_support VM on qamaster
openQA Infrastructure - action #80408: revert longer timeout override for openQA services as we could not see less problems with corrupted worker cache
action #80412: tests fail with auto_review:"(?s)version is 4\.6\.1606298538\.191b5988.*Can.*t locate object method.*code.*via package":retry
QA - action #80414: [proof-of-concept] Extend "auto-review" for failed jobs as well, start with o3
QA - action #80418: [learning] Fix parse errors in "openqa-investigate" "parse error: Invalid numeric literal at line 1, column 10"
openQA Infrastructure - action #80482: qa-power8-5-kvm has been down for days, use more robust filesystem setup
action #80492: [easy] API help document says "Deletes …" for routes like "test_suites/:id" PUT and POST and looks inconsistent
openQA Infrastructure - action #80538: flaky and misleading alerts about "openQA minion workers alert" as well as "Minion Jobs alert"
openQA Infrastructure - action #80544: Ensure that IPMI for powerqaworker-qam works reliably
action #80576: [learning] perl warning in openqa-livehandler "Use of uninitialized value $ret in exit at /usr/share/openqa/script/openqa-livehandler line 31.", similar for openqa-websockets
openQA Infrastructure - action #80594: Needles are not pushed from o3 to github repo
openQA Infrastructure - action #80656: OSD deployment failed at 2020-12-02 because 'malbec.arch.suse.de' is down
action #80662: uninitialized value in string eq at .../load_templates line 137
openQA Tests - action #80670: [opensuse][gnuhealth] test fails in gnuhealth_client_preconfigure due to incorrect needle about "host field selected" when it is not selected
action #80682: Automatic tests for our openQA containers - worker only
openQA Infrastructure - action #80688: Upgrade IO firmware for powerqaworker-qam-1
openQA Infrastructure - action #80734: GitLab pipeline trigger via Grafana fails due to TLS errors
action #80736: Trigger 'auto-review' from within openQA when jobs incomplete (or fail) , for testing: auto_review:"tests died: unable to load main.pm, check the log for the cause"
action #80746: test fails in await_install - openQA hasn't inherit the comments for incomplete job
openQA Infrastructure - action #80768: All workers in grenache-1 are broken at 2020-12-07
openQA Infrastructure - action #80770: "host up" alerts alerting on Sunday around 03:10Z and all going back to ok 30m later, should be prevented
action #80800: flaky/unstable t/full-stack.t, Failed test 'test 1 is running', Bailout called. Further testing stopped: URL for os-autoinst cmd srv not available
QA - action #80806: Extend "auto-review" for failed jobs as well - Generalize openqa-monitor-investigation-candidates to look at more than just one job group
QA - action #80808: Extend "auto-review" for failed jobs as well - enable same as on o3 but on osd
action #80826: Trigger 'auto-review' from within openQA when jobs incomplete on osd as well
action #80830: Trigger 'openqa-investigate' from within openQA when jobs fail on o3
openQA Infrastructure - action #80834: [alert] osd reported 502 errors, unresponsive, failing alerts for CPU usage since 2020-12-08 0840Z and minion jobs failed
action #80908: [epic] Continuous deployment (package upgrade or config update) without interrupting currently running openQA jobs
action #80910: openQA workers read updated configuration, e.g. WORKER_CLASS, whenever they are ready to pick up new jobs
openQA Infrastructure - action #80956: update osd VM root disk VM config and resize root disk on osd
action #80962: os-autoinst development container image is still based on 15.1 but repo is switched to 15.2 already
action #80986: terminate worker process after executing all currently assigned jobs based on config/env variable
openQA Infrastructure - action #81020: QA-Power8-4-kvm start failed since reboot on 2020-12-13
openQA Infrastructure - action #81026: many jobs incomplete with auto_review:"(?s)Running on openqaworker-arm-2.*failed: 521 Connect timeout.*Result: setup failure":retry
action #81038: Avoid using Mojolicious::Routes::Route::route which is DEPRECATED in favor of Mojolicious::Routes::Route::any in asset pack
openQA Infrastructure - action #81046: openqaworker-arm-2.suse.de unreachable
openQA Infrastructure - action #81058: [tracker-ticket] Power machines can't find installed OS. Automatic reboots disabled for now
action #81060: Create a helm chart to deploy web UI in kubernetes
action #81068: openQA tests fail on OBS Tumbleweed
QA - action #81106: test out rocket chat notifications from github actions
action #81116: openqa-webui is failed to start because /var/lib/openqa/db/db.lock owned by vnc user
action #81118: automatic container tests for os-autoinst
action #81150: sporadic fail of os-autoins t/10-virtio_terminal.t
action #81180: fetchneedles fails on uncommited needles
action #81206: Trigger 'openqa-investigate' from within openQA when jobs fail on osd
openQA Infrastructure - action #81220: overdrive2.arch wants to be accepted with salt key on osd, what to do with this machine?
openQA Infrastructure - action #81228: many o3 workers not working on jobs as of 2020-12-21 (w4 seems to be ok)
openQA Infrastructure - action #81232: salt high state triggers multiple errors
action #81386: python-openqa_client fails to build on OBS for SLE 15/ 15 SP1
action #81492: openQA-in-openQA fails with 'function' object has no attribute 'func_name' and label:non_existing asset, candidate for removal or wrong settings
action #81816: OSD deployment failed at 2021-01-06
action #81859: openqa-investigate triggers incomplete sets for multi-machine scenarios
action #82067: Unable to use openqa-clone-custom-git-refspec - Returns 403 Not authorized
action #86063: [epic] Add possibility to trigger openQA API calls, e.g. single "jobs", without the need of the client / over the webUI / with curl
action #87698: openQA jobs with arbitrary parameters can be triggered over the webUI for authenticated users with right permissions (operator+)
openQA Infrastructure - action #87838: *alert* osd: Open database connections by user alert
openQA Infrastructure - action #87856: [alert] openqaworker3.suse.de/NRPE is CRITICAL
openQA Infrastructure - action #87883: osd infrastructure: services like "telegraf" are not enabled to start immediately on boot
action #87898: Add grafana alert for "broken workers" as reported by openQA
openQA Infrastructure - action #87970: tumbleweed container images in gitlab CI fail on salt calls, e.g. "KeyError: 'cmd.run_all'"
openQA Infrastructure - action #87979: Requirements for access to the openqa VM
action #88121: Trigger cleanup of results (or assets) if not enough free space based on configuration limit
action #88125: Get feedback and adapt the population scripts for testreports
QA - action #88127: [tools][qem] Test coverage DB for maintenance updates
QA - action #88183: [spike][timeboxed:20h] rancher: Create a simple selenium test for the UI as proof of concept
action #88187: Set the addresses in the "internal clients" configurable
openQA Infrastructure - action #88189: Firmware upgrade of qa-power8-4.qa.suse.de
openQA Infrastructure - action #88191: openqaworker2 boot ends in emergency shell
openQA Infrastructure - action #88195: Full test of salt states including DBus on CI
openQA Infrastructure - action #88225: osd infrastructure: Many failed systemd services on various machines
action #88247: manage password for multiple test users in os-autoinst for openQA tests
action #88347: upload_asset function fails for larger files
action #88363: openqa-client cannot return job info
openQA Infrastructure - action #88385: openqaworker3 host up alert is flaky
openQA Infrastructure - action #88450: Flaky NTP offset alert
action #88452: script_run syntax error when & is at the end of command
action #88459: low-prio single-machine jobs can starve out high-prio multi-machine tests
openQA Infrastructure - action #88474: All workers on powerqaworker-qam-1 are offline
action #88482: Two absolute paths concatenated to form a default needle dir when PRODUCT_DIR/needles doesn't exist
QA - action #88485: Fetch and store coverage info for each incident
action #88496: openQA "t" tests time out (again) and take long
action #88538: Since 2021-02-11 all openQA CI tests fail in codecov "406 Not Acceptable {'detail': ErrorDetail(string='Could not satisfy the request Accept header.', code='not_acceptable')}"
openQA Infrastructure - action #88546: Make use of the new "Storage Server", e.g. complete OSD backup
action #88564: text field for git commit details in needle editor
action #88594: codecov reports for individual files yield "forbidden" and files can not be shown
action #88603: Two identical jobs are created from one abandoned test
action #88609: Mojolicious 9.0 compatibility
action #88696: Executing /usr/share/openqa/script/openqa-gru fails
action #88745: [easy] Clicking on "Untracked" on /admin/assets yields "Not Found" page, better have no clickable link at all
action #88754: openQA-in-openQA tests always fail and results do not impact submission pipeline
openQA Infrastructure - action #88807: Open vSwitch command 'set_vlan' with arguments 'tap41 13' failed: 'tap41' is not connected to bridge 'br1' at /usr/lib/os-autoinst/backend/qemu.pm line 152.
openQA Infrastructure - action #88900: openqaworker13 was unreachable
openQA Infrastructure - action #88912: Assets cleaned up too early for Tumbleweed aarch64
action #88915: Codecov always fails with 404 Not Found - Build has already finished, uploads rejected
openQA Infrastructure - action #89047: Failed to commit needles, gitlab account blocked 2021-02-24
openQA Infrastructure - action #89050: OSD deployment blocked since two weeks - salt nodes failing
action #89056: incomplete jobs with auto_review:"isotovideo died: isotovideo received signal HUP":retry
action #89059: package checks fail due to timeout in openSUSE:Factory:ARM/aarch64
action #89077: os-autoinst Makefile is missing symlinks configuration
QA - action #89080: Flatpak OBS builds fail on tumbleweed
openQA Infrastructure - action #89113: [alert] PROBLEM Service Alert: openqa.suse.de/NTP Time is CRITICAL
action #89200: Switch OSD deployment to two-daily deployment
action #89206: openqa-review CI failure about "ImportError: No module named enum" in python 2.7 tests
action #89221: Statistics about the time it takes developers from QA to close tickets related to failing tests (creation - closing)
action #89224: Limit execution time of hook scripts run within Minion
openQA Infrastructure - action #89275: Flaky Incomplete jobs (not restarted) of last 24h alert and New incompletes alert triggered and back to OK
action #89281: Prevent investigation jobs to do any asset uploads to prevent overriding production assets
openQA Infrastructure - action #89419: Incomplete jobs after OSD deployment
QA - action #89491: Add flatpak build test to obs-build CI
openQA Infrastructure - action #89497: flaky Failed systemd services alert (except openqa.suse.de)
action #89545: [epic] Presence on openSUSE discord? Or matrix channel? reddit?
action #89548: presence on discord
openQA Infrastructure - action #89551: NFS mount fails after boot (reproducible on some OSD workers)
action #89557: presence on matrix
coordination #12876: [epic] Offer a way for jobs to dynamically schedule children
coordination #39719: [saga][epic] Detect "known failures" and mark jobs as such to make tests more stable, reviewing test results and tracking known issues easier
coordination #41057: [epic] Make reviewing results easier
coordination #44360: [epic] Parameterize test suites within job groups
coordination #45302: [epic] smarter fetchneedles (was: fetchneedles should ensure we are always on a branch (and try to self-repair))
coordination #55364: [epic] Let's make codecov reports reliable
coordination #58184: [saga][epic][use case] full version control awareness within openQA, e.g. user forks and branches, fully versioned test schedules and configuration settings
coordination #62420: [epic] Distinguish all types of incompletes
coordination #64746: [saga][epic] Scale up: Handle large storage efficiently to be able to run current tests efficiently but keep big archives of old results
coordination #65118: [epic] multimachine test fails with symptoms "websocket refusing connection" and other unclear reasons
coordination #66727: [epic] Define structure to define test suites not in openQA database
openQA Infrastructure - coordination #68923: [epic] Use external videoencoder in production auto_review:"External encoder not accepting data"
QA - coordination #69310: [epic] SUSE QA tools team ticket process helpers
coordination #69451: [epic] test incompletes with "(?s)Download.*successful.*Failed to download":retry, not helpful details
openQA Infrastructure - coordination #69478: [epic] Upgrade o3+osd workers+webui to openSUSE Leap 15.2
openQA Infrastructure - coordination #75400: [epic] Improve documentation about how to manage infrastructure
QA - coordination #77899: [epic] Extend "auto-review" for failed jobs as well
openQA Infrastructure - coordination #78206: [epic] 2020-11-18 nbg power outage aftermath
coordination #80142: [saga][epic] Scale out openQA: Redundant/load-balancing deployments of openQA, easy containers, containers on kubernetes
coordination #80150: [epic] Scale out openQA: Easier openQA setup
coordination #80546: [epic] Scale up: Enable to store more results
coordination #80828: [epic] Trigger 'auto-review' and 'openqa-investigate' from within openQA when jobs incomplete or fail on o3+osd
coordination #88229: [epic] Prevent unintended test coverage decrease
openSUSE admin - tickets #88751: Problem login into openqa.suse.de and openqa.opensuse.org