# Done

Milestone to track done tasks (Tools team, originally used for "tickets to demo")

* action #7086: add some worker version check
* action #8082: calculate overall test result in os-autoinst or worker?
* action #9906: Show added/updated needles in editor
* action #10474: Check supported version of os-autoinst
* action #10514: [tools][sprint 201711.2] Prevent test module name clashes (problem on loading test module twice / multiple times / more than one)
* action #12092: [tools][dashboard][Feature] - make openQA show when a bug is marked as "RESOLVED FIXED" but still in the product
* action #13524: Update current test module details when cancelling a running job
* action #13920: Showing context builds test result in the view of current "Previous Results" 
* action #14580: Add ability to backup selected jobs/tests results outside of o.s.d/o.o.o
* action #14674: ppc: os-autoinst uses the wrong qemu emulation
* action #14904: log warn message: Can't call method "DISTRI" on an undefined value
* action #15030: highlight actually matched tag in multi-tag needle tag info popup
* action #15088: Build overview should display dependency relations between jobs
* action #15192: [tools]DB exception popup while trying to add Test Suite with same name 
* action #15612: UI improvements in jobs templates
* action #16076: parse_junit_log is crashing with xunit from Avocado / proper xunit parsing in openQA
* action #16282: [tools][sprint 201711.2] documentation: Better API documentation
* action #16418: [tools][sprint 201712.2] svirt: openQA won't perform another query_isotovideo check when VM is down -> incompletes
* action #16496: [tools][sprint 201711.2] display current disk space consumption of job groups
* action #16556: Need to display "incomplete" jobs in "Previous results"
* action #17706: error creating needle "wrong parameters: imageversion"
* action #17788: [tools]Uploading images chksum check relies on global /var/lib/openqa/share
* action #17846: [tools][sprint 201712.2] Allow same group name within different parent groups
* action #17874: test fails with DIE No map for '\r' in VNC.pm
* action #17902: Check if qemu-img finished correctly and die if not
* action #18460: Add full screen diff to needle preview
* action #18462: Move GRU tasks into Minion jobs
* action #18464: Display quotes for run_cmd in the log if there is whitespace
* action #19408: interactive mode: stopped live view shows current needle, needle editor shows old needle
* action #19424: [tools] logwarn: [websockets:error] Worker not found for given connection during connection close
* action #19542: avoid type_string taking "too long"
* action #19672: GRU may delete assets while jobs are registered
* action #19730:  [tools][openqa-monitoring] "can't remove <needle_path>"
* action #19790: Broken links to the openQA documentation from github
* action #19806: eth0 address of one node is sometime in use by other test in multimachine tests within same ovs VLAN
* action #19996: [tools][sprint 201712.2] Show only online workers by default on the workers page `/admin/workers`
* action #23318: Limit gru tasks
* action #23422: [tools][sprint 201712.1] Same miniature for 2 steps
* action #25290: Why do we now have two separate mechanisms by which workers report status to the server?
* action #25412: Clone_job: support global variables
* action #25572: [tools][needles]needles pushing can interfer with "fetchneedles"
* action #25688: [tools][sprint 201710.1][sprint 201710.2][infrastructure] Upgrade openqa.suse.de and remote workers to sle12sp3 and 42.3
* action #25702: add option in tty console to change key
* action #25766: [tools][sprint 201712.1][bonus] Displaying all Worker status by calling Websocket DBus service in the WebUI list is overkill
* action #25892: Scheduling parallel jobs
* action #27120: [tools][sprint 201711.2] Make worker upload part of its log to WebUI
* action #27409: [tools][sprint 201711.2] Make an option for barriers to get destroyed if one of the jobs belonging to it fails
* action #27454: [tools][scheduling] Worker's seen DB field is ignored by WebSocket server when checking for stale jobs
* coordination #27832: [epic][tools][sprint 201712.2][bonus] Revamping of /admin/workers
* action #27958: [tools][Sprint 201711.2] Allow filtering of jobs by worker_class.
* action #28018: [tools] Regression: Page refresh of a liveview displays no image
* action #28027: Add function to account for the unexpected in serial output
* action #28192: opencv update in TW broke os-autoinst test suite
* action #28325: [tools][sprint 201712.2][aarch64][bonus] worker incompletes job with timeout after ~9h on seattle14 - no action in log for more than 8h
* action #28429: [tools][sprint 201711.2] Sqlite support busted
* action #28705: [tools][sprint 201712.1] Fix travis problems with pull requests
* action #28857: [tools][sprint 201712.2] t/25-serverstartup.t fails without MOJO_LOG_LEVEL=debug
* action #28961: openQA in openQA tests are broken
* action #29110: [tools][sprint 201712.2] Can't update job group settings
* action #29241: Sorting scheduled jobs is broken
* action #29304: [tools][Sprint 201711.2] logging timestamp lost sub-second precision (+ longer debug lines with not much helpful content)
* action #29378: worker overview page changing column sort order when clicking on status pulldown menue
* action #29529: openqa-websockets defaults to debug
* action #29535: Regression: Can't delete jobs "holding" assets 
* action #30002: Provide currently installed os-autoinst version in openQA worker page and log
* action #30124: don't set mail adress for system user
* action #30267: [tools][Sprint 201711.2] Workaround memory dumps taking too long
* action #30388: [qam]  openqaworker10:4 - worker fail all tests - qemu instances are left behind
* action #31045: Invalid URL gives 500 - should give 404
* action #31771: 0ac2c9f6cd1bd5f2d7091e1c08096152ed02a23f failing on chattr
* action #31777: [tools][kernel] Better document serial terminal console (virtio_console)
* action #31933: Uploaded asset fails moving file to final destination 
* action #32260: Improve worker version checking.
* action #32263: [tools] Implement sub-reaper support in workers when starting isotovideo
* action #32281: Can't locate images in Xen jobs
* action #32587: mouse_hide(1) does not seems to work on non-qemu backends
* action #32593: Multiple ttySx consoles for qemu
* action #32602: HDD upload fails with 502: Proxy Error
* action #32632: With caching enabled worker still requires /var/lib/openqa/share
* action #32653: workers stall and no longer grab jobs
* action #32725: [tools] Scheduler job_grab/filter_jobs refactoring
* action #32887: Worker version is reset -> no jobs scheduled
* action #33037: dead children uncaught (again)
* action #33514: [tools] Use of uninitialized value $status in right bitshift (>>) 
* action #33529: isotovideo: backend takes 100 % of CPU when driving svirt job
* action #33856: openQA needle editor causes internal server error at lib/OpenQA/WebAPI/Controller/Step.pm
* action #33892: Allow the client to trigger a build for a jobgroup
* action #34042: [tools] Worker goes to infinite loop during upload of screenshots in case of writing failure
* action #34195: Replace "candidate needles combo box" in test details with popup containing tags/needles
* action #34267: osd instance unresponsive (HTTP 502)
* action #34282: implement catching errors in all our log_* functions
* action #34396: Show bug name on mouseover
* action #34504: [tools][sporadic] Job's auto_duplicate fails to duplicate job dependencies
* action #34507: Document requirements for external test suite results to be rendered by openQA
* action #34510: Add logrotate for logs in OSD
* action #34519: Start migration to Boostrap 4
* action #34522: OBS openQA tests failing: DIE Can't call method "add_console" on unblessed reference
* action #34525: Improve handling of screenshot inserts into the database
* action #34531: Improvements to the job_module_needles table
* action #35200: bootstrap 4: needle popup a little unaligned
* action #35203: bootstrap 4: job group navigation misses divider
* action #35206: Needle selection disfunctional on screenshots without tags
* action #35221: Result boxes on test pages are hard to distinguish since bootstrap 4
* action #35224: bootstrap 4: The message-box (pop-up) that shows failing needles stays forever
* action #35299: Potential corrupted file causing installation tests to fail
* action #35374: Rendering of external harness output in test results as rows
* action #35377: dump-templates renders versions not always as string
* action #35380: Allow filtering of test steps  for jobs that are finished
* action #35383: Create a docker container to run unit and integration tests
* action #35389: Improve performance of /admin/assets 
* action #35407: [kernel][tools] QEMU Refactor - Serialise state and reimplement SKIPTO
* action #35431: [kernel][tools] QEMU Refactor - Clean up miscellaneous weird stuff
* action #35434: [kernel][tools] QEMU Refactor - Ensure consistent use of List::Util, map and grep
* action #35437: [kernel][tools] QEMU Refactor - Publish disk
* action #35440: [kernel][tools] QEMU Refactor - Code format and rebase
* action #35443: [kernel][tools] QEMU Refactor - Acceptance testing
* action #35749: Ignore insert errors in limit_assets
* action #35895: [os-autoinst-openvswitch][theory] problem with assigining VLANs tags - more testsuites are using the same VLAN?
* action #35914: Changes to Job::duplicate
* action #36264: Add _source field to every detail generated by parsers implementations
* action #36307: Remove limit option in Next & previous results tab UI
* action #36442: Access to running SUTs for System Developers
* action #36454: Move 'Minimal developer mode' to openQA web UI
* action #36565: parallel parents aren't killed when chained parents fail
* action #36574: Display instructions how to connect when a job is in status waiting
* action #36613: Implement WS connection from web UI to command server
* action #36697: SLE-12-SP3-Server installation medium missing on openqaworker6:5 - iso was present in cache but then not in pool when starting VM
* action #36727: job_grab does not cope with parallel cycles
* action #36823: weird line break in failed module box - regression?
* action #36826: popover on audit log is not clickable because of layout
* action #36985: Allow _GROUP_ID for jobs posting
* action #37009: openqa-scripts repo sync on OSD disabled and tainted
* action #37375: Create UI elements for developer sessions and update them via WS connection
* action #37420: Try OBS registry (registry.opensuse.org)
* action #37423: Make parser tests less verbose
* action #37453: Mark all restarted (cluster) jobs in UI
* action #37519: horizontal wrapping of test steps displays broken, near-infinite horizontal scrolling bar instead
* action #37692: Verbosity of isotovideo command server is too high by default
* action #37973: Move os-autoinst Github repo to run tests within a Docker environment in Travis
* action #38015: [qam] [sle15] Missing SLE5 iso
* action #38351: openQA client requires "OpenQA::UserAgent" which is not provided in openSUSE package openQA-client
* action #38447: "developer mode" on o3 does not seem to ever update the "current test module" and is stuck in "retrieving status"
* action #38510: Allow os-autoinst to pause on next assert_screen timeout
* action #38522: Updating Job Group default priority does not have effect on new tests added to that group
* action #38555: Create OpenQA parser for IPA file format
* action #38558: Add openqaworker13 to osd
* action #38606: proceed_on_failure option in script_output is broken
* action #38813: Qemu backend rewrite fallout
* action #38822: Qemu: Could not open backing file: Cannot reference an existing block device with additional options or a new filename
* action #39035: PFLASH files handling
* action #39068: Webui killed by out of memory in o3 (triggered by postgresql)
* action #39227: Handle the job being finished more nicely in developer mode
* action #39275: Up and down keys don't work
* action #39347: Upload of qcow2 fails with Mojolicious 7.88*
* action #39494: Flaky exit status code on ppc64le
* action #39503: svirt tests fail with unsupported update encoding -1733194013 at /.../consoles/VNC.pm line 988
* action #39515: videoencoder crashed on aarch64
* action #39560: Tests for blocked_by and loops inside of it
* action #39629: openQA Scheduler refactor fallout
* action #39692: openqaworker-arm-3 is outdated - "mutex_wait" is not exported by the lockapi module
* action #39707: Constant subroutine * redefined at /usr/lib/perl5/vendor_perl/5.18.2/*-linux-thread-multi/_h2ph_pre.ph
* action #39743: [o3][tools] o3 unusable, often responds with 504 Gateway Time-out
* action #39761: [labs] openqaw7-hyperv.qa.suse.de not accessible
* action #39776: Mojo::Reactor::Poll: I/O watcher failed: DBD::SQLite::db do failed: table "assets" already exists  error should be recoverable
* action #39806: [OpenQA][workers] 4/6 64bit-ipmi workers are down
* action #39818: Enhance openQA docker image for tests
* action #39833: [tools] When a worker is abruptly killed, jobs get blocked - CACHE: Being downloaded by another worker, sleeping
* action #39881: Worker dies during file upload (Mojo bug)
* action #39917: Remove "Jump to video" icon from "needle view" when video is unavailable
* action #39980: Cache locks assets when worker dies in critical section
* action #40004: worker continues to work on job which he as well as the webui considers dead
* action #40088: sorting in worker page jobs table seems to be broken
* action #40103: [o3] openqaworker4 not able to finish any jobs
* action #40106: [functional][u][tools] /liveviewhandler/ route causes a lot of "502" errors
* action #40148: [OpenQA][64bit-ipmi worker] Three online 64bit-ipmi workers do not take scheduled jobs for over 10 hours.
* action #40157: Running out of space in openqa.suse.de for /var/lib/openqa
* action #40223: Document GRU cleanup strategy
* action #40352: Unexpected end of data from the backend
* action #40415: Concurrent jobs with dependencies don't work if they are on different machines.
* action #40583: Provide job stats for telegraf to poll
* action #40652: AMQP on openqa.opensuse.org disfunctional
* action #40661: Make it possible to overwrite/manipulate job template priority on isos posts
* action #40772: Proper display of cluster/dependencies
* action #40793: Prevent isotovideo <=> command server IPC to occasionally lose messages
* action #40811: Single Machine jobs starve clusters
* action #40823: get_version broken (broken in refactoring)
* action #40871: Asset removed during job runtime
* action #40898: openQA-client-4.6.1534873283.be929573-1.2.noarch missing dependency for JSON.pm
* action #40904: Scheduler service is not started on package openQA-4.6.1536726988.de30ae3a-778.1.noarch
* action #40973: worker retry seems to be bogus
* action #41015: Don't use livehandler if no developer looks at it
* action #41024: dashboard misses one state
* action #41027: worker disconnects during cleanup
* action #41048: job_networks not reliably deleted
* action #41054: /tests is superslow if there are a lot of scheduled jobs
* action #41066: Scheduling jobs for IPMI (bare metal) on the same worker (aka FOLLOW_TEST_DIRECTLY aka START_DIRECTLY_AFTER_TEST).
* action #41075: Sync newly created needles with os-autoinst for developer mode
* action #41102: Stale qemu processes are left again
* action #41174: Treat jobs without job group with default timeouts
* action #41399: Using 'Limit Match' filter on /admin/needles results in error 500
* action #41483: [tools] medium that should belong to job group was deleted after just 2 minutes making our SLE15 tests in build 47.1 useless ... and more
* action #41486: Port openQA to Mojolicious 8.0
* action #41510: CSS class mismatch resborder_softfail vs. resborder_softfailed
* action #41543: parallel_failed jobs miss t_finished
* action #41756: module "failed_module" on /tests/overview does not show latest in each build / does not resolve latest build
* action #41891: Worker stalled during flock
* action #42089: Reduce debug impact of needle matching
* action #42407: enable needeling using developer mode
* action #42560: Progress bars on 'All tests' page broken
* action #42563: Hide cloned jobs from job dependency graph
* action #42569: Unexpected error when setting QEMURAM is specified twice on workers.ini
* action #42584: Priority field in /tests is broken
* action #42614: /admin/workers/X shows 'Never Started' as finished
* action #42779: [tools] Make table headers more visible in openQA test result matrix
* action #42947: Job group name can be updated with empty input then no link clickable
* action #43031: Fix failing t/ui/13-admin.t
* action #43100: Test result failed should not be overriden by softfailed
* action #43112: Enhancement for overview page(s): show blocked in /tests/overview as well; rethink color choice for blocked
* action #43229: Could openQA automatic takeover more than 1 known bsc/poo
* action #43931: Message about deprecated calls "Mojo::Transaction::success is DEPRECATED in favor of Mojo::Transaction::result and Mojo::Transaction::error"
* action #43952: developer mode: expand commands block on timeout
* action #43991: Scheduler stays busy after restarting/clonging a job
* action #44039: /admin/test_suites is broken without login
* action #46967: [functional][u][tools] warning in bootloader_zkvm: Calling Net::SSH2::Channel::readline in non-blocking mode is usually a programming error
* action #49946: NEW Mojolicious-8.13 breaks openqa build
* action #50531: docs/images/architecture.svg is not colorblind friendly
* action #51464: New (machine, test, medium ...) button does not work when search is used
* action #52487: [webui] osd: audit log page takes 5 seconds to render a page
* action #52928: scheduling: Cyclic dependencies created on sap jobs
* action #53420: [tools] Bug title escaping on test result matrix page
* action #53603: unstable test "t/33-developer_mode.t"
* action #53858: flaky test: 25-cache-service.t
* action #54203: workers with "--no-cleanup" empty the pool, most likely regression from "worker restructuring"
* action #54464: qemu-img convert failed
* action #54485: worker can not read out worker class and reverts to default
* action #54680: Worker re-registration after websocket disconnects causes incompletes
* action #54809: PUBLISH_HDD_1 fails with '/' in variable name
* action #54827: [cache] downloading assets with `:` or `#` in filenames using the caching fails
* action #55112: Load dashboard contents on demand
* action #55241: Prevent spamming cleanup jobs
* action #55328: job is considered incomplete by openQA but worker still pushes updates so that "job is not considered dead"
* action #55373: Worker::Cache thinks no space is used when cache resides in symlinked folder pointing to other partition
* action #55460: flaky test: t/33-developer_mode.t - not ok 308 - Wait for (?^:\"set_pause_on_screen_mismatch\":\"assert_screen\") timed out
* action #55526: detection for still running qemu instance in worker only works with "--no-cleanup"
* action #55529: job incompletes when it can not reach the openqa webui host just for a single time aka. retry on 521 connect timeout in cache
* action #55541: from o3: DBIx::Class::Storage::DBI::select_single(): Query returned more than one row.  SQL that returns multiple rows is DEPRECATED for ->find and ->single at /usr/share/openqa/script/../lib/OpenQA/WebAPI/Controller/Running.pm line 64
* action #55751: Formatting for <br> and <code> tags in job description and group comments broken
* action #55835: Support running multiple jobs directly in sequence on the same machine
* action #55907: Deleting job results is too slow
* action #56453: Asset limit for parent group
* action #56813: START_AFTER_TEST dependency error should use `@$MACHINE` syntax
* action #56876: YAML setting overwrite each other
* action #56987: Polish UI of openQA's OBS plugin
* action #56993: left-over qemu processes cause incomplete jobs with "Failed to find an available port: Address already in use", previous jobs failed but did not tear down qemu
* action #57017: websocket service stalls for 10 minutes when a worker misbehaves
* action #57113: livehandler service tries to load plugins and fails on obs rsync
* action #57266: Forbid test suite name changes if used in job groups
* action #57287: Limit assets task fails violating database key constraint
* action #57338: unable to successfully restart a job because /media.1/media not found on medium
* action #57413: sort order of running tests takes 'not yet' as very old
* action #57662: monitoring considered harmful
* action #57707: isotovideo fails to terminate cleanly, message "isotovideo: unable to inform websocket clients about stopping command server: Request timeout", regression from 4cd4af2b
* action #57785: Remove all assets in iso
* action #58097: Script openqa-load-templates dose not support --group, openQA document gives a wrong example.
* action #58124: When re-adding an existed job group in a parent job group, the web page stays in the waiting result.
* action #58133: Hide old job templates editor for new groups 
* coordination #58166: EPIC: Continue tests after failures on !qemu
* action #58319: [openqa-client] make --form visible in the help output
* action #58367: vnc port information is wrong for > 99
* action #58484: jobs are stuck in downloading while cache service died
* action #58625: add new medium faild if you first search item
* action #59008: 900GB of assets not tracked ('other' assets?)
* action #59082: Dependency graph shows just test name (misses machine)
* action #59094: Allow hiding/collapsing the left-hand yaml guide
* action #59151: Move sha256 files to 'other'
* action #59184: Research about testing with a custom git ref
* action #59187: unstable/flaky test: ui/15-admin-workers.t
* action #59190: Broken SSH serial console
* action #60140: job incompletes failing on initial asset download with "Asset was already requested by another job"
* action #60191: Proposal: Drop support for Leap 42.3 + SLE12
* action #60389: Use human readable sizes in Cache Service logs
* action #60413: status favicon for job details shows "scheduled" instead of running on firefox
* action #60539: openQA remote-back-ends crashes with Xvnc >= 1.9.80
* action #60560: Self-investigate potential reasons for failures in openQA
* action #60866: Periodic stale job detection keeps scheduler busy producing a massive number of incomplete jobs (was: Again many incomplete jobs with no logs at all)
* action #60884: openqa-clone-custom-git-refspec has issue with NEEDLES_DIR 'null'
* action #61019: new needle can not download during development mode
* action #61064: testapi.pm should sleep before clicking
* action #61143: Show asset table also when cleanup is ongoing
* action #61167: "Possible attempt to separate words with commas" in os-autoinst t/18-qemu.t
* action #62015: jobs incomplete without logs as some workers are rejected (was: Scheduler does not work)
* action #62021: Add test for gh#os-autoinst/openQA#2655
* action #62237: many incompletes with just "setup failure" and no further information
* action #62243: After latest updates, openQA has problematic behavior on Dell Precision 5810
* action #15668: [kernel][LTP][OpenQA] hyperthreading ht_interrupt won't run
* action #23612: [tools]/usr/bin/isotovideo: backen: symbol lookup error: /usr/lib/perl5/vendor_perl/5.18.2/x86_64-linux-thread-multi/auto/Net/SSH2/SSH2.so: undefined symbol: libssh2_session_set_last_error
* action #25286: [sle][functional][u][qam][tools] Wrong URL for repositories -> repo symlinks deleted on osd prematurely?
* action #25964: [tools][aarch64] test ends incomplete after first_boot - Fail to make snapshot
* action #29637: [tools][kernel] test suddenly print login words -> script_output used wrong
* action #33919: [kernel] nvmftests fail for Leap 15
* action #34123: [kernel]wicked_basic: invalid bash syntax
* action #34297: [kernel][hpc] Need provide separate reg key for SLE 15 HPC  aarch/x86 install jobs
* action #37674: [QAM] [SLE 15] test fails in evolution_prepare_servers - test does not support SLE 15
* action #37925: [QAM] [SLE 15] Make SLE-HPC product installation compatible with QAM update flow
* action #38000: [sle15] [qam] test fails in bootloader_s390 - Could not retrieve required variable REPO_0
* action #38108: [openqa][kernel] Power down needs to have longer timeout due longer shutdown of BTRFS related service
* action #38135: [qam] [sle15] test fails in userspace_bind - qa_test_bind is not updated for SLE15
* action #38261: [qam] test fails in keymap_or_locale - tested text not written in console
* action #39536: [qam] test fails in patch_and_reboot -  add missing variables for qam test issues
* action #39749: [kernel] wrong "start-after" reference kills testcase
* action #39887: [kernel] [sle] [network] test fails in before_test - network failure
* action #40181: [network] test fails in before_test - wicked basic test fails on 15SP1
* action #40547: [qam][opensuse] test fails in updates_packagekit_kde - job timeout
* action #43895: [kernel] test fails in enable_kdump - yast bug but no logs recorded
* action #47336: [migration] orphaned_packages_check.pm - use for other products?
* action #50615: [functional][y] test fails in await_install - does not catch rebootnow
* action #53534: [opensuse][kde] test fails in await_install - timeout not working properly
* coordination #55556: [epic][osd][functional][u] osd: Untracked assets that most likely should be tracked
* action #58013: [aarch64] Some incompletes due to slow migration
* action #58361: [opensuse] test fails in gnucash
* action #60467: test fails in first_boot - Undefined subroutine &opensusebasetest::is_desktop_installed called at /var/lib/openqa/cache/openqa1-opensuse/tests/opensuse/lib/opensusebasetest.pm line 820.
* action #60740: test fails in gnuhealth_setup
* action #60743: [qam] Migrate to file failed, it has been running for more than 240 at /usr/lib/os-autoinst/backend/qemu.pm line 263.
* action #61149: test fails in gnuhealth_setup
* action #61910: [aarch64][tools] test fails in boot_to_desktop to select boot device for aarch64 using the tianocore menu
* action #62009: [openqa-in-openqa] test fails in openqa_worker during a zypper install
* action #38552: Add infinibad servers to osd
* action #38609: reconfigure openqaworker11 openqaworker12 and both staging instances
* action #41336: Create a monitoring dashboard for openqa.suse.de
* action #41846: Link version to a possible deployed delta
* action #53234: all jobs on aarch64.o.o incompleted with "Permission denied" on /dev/hugepages, "others" had no r/w
* action #53570: Failed service "logrotate" on aarch64.o.o and openqaworker1.o.o
* action #53573: Failed service "irqbalance" on aarch64.o.o
* action #54308: Failed to open /var/lib/openqa/.config/openqa/client.conf: Permission denied at /usr/share/openqa/script/../lib/OpenQA/UserAgent.pm line 42
* action #56819: worker cacheservice on *arm* does not seem to be reboot safe (race condition with nvme prepare?)
* action #57476: Recurring partitions full and logrotate fails, possibly due to disabling /var/log/openqa as log target
* action #57539: imagetester is incompleting all jobs with existing but empty logs, as /var/lib/openqa/pool is full
* action #58373: Proposal: Poll test and needle repos every minute to make people wait less for updates in tests or needles
* action #59855: openqaworker-arm-1 seems to be under serious distress "kernel:[93903.692361] BUG: workqueue lockup - pool cpus=32 node=0 flags=0x0 nice=0 stuck for 42657s!"
* action #60653: salt patch for "host not found" does not apply cleanly anymore on all x86_64 workers, ppc64le and aarch64 still have old versions
* action #60923: [alert] /srv about to run full, postgres logs very big due to repeated error "duplicate key value violates unique constraint "screenshots_filename", Key (filename)=(8ca/3c9/98a00d8bb2ccba5a2de1d403b5.png) already exists. INSERT INTO screenshots …"
* action #60959: Seems we have many incompletes since 2019-12-12 on o3 , failing just in the middle of test modules with no logs uploaded
* action #61218: ** PROBLEM Service Alert: ariel-opensuse.suse.de/SSH is CRITICAL **
* action #61913: rollback of CI pipeline fails to read old package version from non-existing file "before"
* action #61916: rollback of OSD CI deployment pipeline can only be triggered if all previous steps are complete
* action #61985: Clarification of "openqa" and "infra" queues in https://infra.nue.suse.com/SelfService/CreateTicketInQueue.html
* action #62048: monitor incompletes
* action #62063: Monitor failed systemd services
* action #62198: failed services os-autoinst-openvswitch.service on powerqaworker-qam-1 and malbec
* action #62201: auto-update failed on osd arm1
* action #62228: powerqaworker-qam-1 is down
* action #62312: auto-update failed on osd arm1