action #12246
closed
[tools]upload of log files can fail sometimes (was: https://openqa.suse.de/tests/412464 has no X-related log files)
Added by okurz almost 8 years ago.
Updated almost 7 years ago.
Category:
Regressions/Crashes
Description
observation¶
As stated by fcrozat in https://bugzilla.suse.com/show_bug.cgi?id=983425 the linked openQA job has no X-related log files
problem¶
maxlin found out that curl fails to upload command 'curl --form upload=@/tmp/loadavg --form upname=first_boot-loadavg http://10.0.2.2:20013/pkeE1q3bP_Bw8ogL/uploadlog/loadavg'
reason for that is so far unknown
As curl is being used on the SUT to communicate with isotovideo, one can think that for some reason, it was impossible for the SUT to stablish a communication channel to upload via curl.
suggestion¶
- If an upload of a file fails, retry (e.g. up to 7 times), because the isotovideo can just be busy
- Ensure events are logged into autoinst-log.txt
- Optional: try even harder on autoinst-log.txt because we need to know the other errors from there
further details¶
okurz looked into os-autoinst/commands.pm . In there there is upload_file but I don't know how to properly reproduce the issue to test it accordingly. This comment explains a bit more
- Related to action #14068: [tools] Gather more system information and logs in case of boot/reboot times out added
- Project changed from openQA Tests to openQA Project
- Subject changed from https://openqa.suse.de/tests/412464 has no X-related log files to upload of log files can fail sometimes (was: https://openqa.suse.de/tests/412464 has no X-related log files)
- Description updated (diff)
- Category changed from Enhancement to existing tests to Regressions/Crashes
- Assignee set to szarate
@szarate I updated the description to better reflect the idea that we discussed, i.e. just retry uploading files on problems. Can you test this properly?
@okurz yes, i can test it, one way to quickly test it, is having the server down (that leads to other problems but baby steps first here)... anywho i'm trying to figure out now how to get quickly the asset's url to see if it's really there.
- Related to action #13876: Error 502: Bad gateway on uploading added
I will provide a PR for this during next days if necessary at all, but actually.. curl shouldn't fail to do the upload.
The upload_file method is actually a Mojo::Upload handler (which is just meant to handle what comes from the SUT), the actual magic for the SUT happens in the testapi functions upload_logs and upload_asset
There are two uploads: One from the SUT to the worker, and then one from the worker to the webUI, curl is in the SUT side, i took a look at this and got something workable, but i'm actually not 100% sure if we want the retry on both places.
I can say that on the webUI side it makes sense, but on the curl/isotovideo side... means that something went really wrong (and there should be the autoinst-log.txt to check out, since it belongs to the webui part of the uploads).
this is mixing isotovideo and worker - I would just close this. The worker upload is handled in a different issue
You want to close this issue just because we got confused a bit? #13876 is caring about server side, this about client. Everything normal :-)
no, this is about curl in SUT.
- Priority changed from Normal to Low
Lowering the priority, until we find a new test with no logs (Other than autoinst-log.txt), for now the curl shouldn't fail.
- Description updated (diff)
- Description updated (diff)
- Has duplicate action #14104: [Build 2162] test fails to upload logs in time in https://openqa.suse.de/tests/603376#step/logs/1 added
- Subject changed from upload of log files can fail sometimes (was: https://openqa.suse.de/tests/412464 has no X-related log files) to [tools]upload of log files can fail sometimes (was: https://openqa.suse.de/tests/412464 has no X-related log files)
- Status changed from New to In Progress
- Status changed from In Progress to Resolved
This has not been seen for a while, setting to resolved.
Also available in: Atom
PDF