action #64809
closedWorker uploads some text results possibly multiple times wasting resources
0%
Description
Observation¶
Worker uploads some text results possibly multiple times.
I've created the following change track result uploads (and their size): https://github.com/os-autoinst/openQA/compare/master...Martchus:log-size-accounting?expand=1
With this modification one can see lots of duplicated uploads in the web UI log. I used the following command with the web UI log from a local test run:
sort webui-log-job-1260-opensuse-15.1-DVD-Updates-x86_64-Build20200321-1-gnome@uefi.log | uniq -d | grep -i accounting
(The log file is attached.)
Problem¶
This wastes resources mainly network bandwidth, increased disk I/O on the web UI host and time.
Expected result¶
Every result is only uploaded once unless a retry is necessary due to transfer problems.
Suggestions¶
- Ensure there is no duplicate upload, e.g. – if necessary – apply the same hash lookup algorithm as for screenshots
Notes / suggestion¶
For screenshots the web UI provides the worker with a list of known screenshot MD5 sums to avoid this. A similar mechanism could be implemented for text results.
Files