script_output sometimes fail on virtio console
It seams randomly happen in openQA, that tests fail, cause "cat -" never finish.
Steps to reproduce¶
Observations on openQA
I was able to bring my openQA instance in such a state. I'm actually not sure, if this
is the same problem as we have in osd, but it looks similar. The big different is, that if it happen once,
it happen always for that worker.
What I did so far:
- Start a test which is using virtio console
- restart openQA while the test is running
- run tests again
A call like this:
cat - > /tmp/script8RI3l.sh; echo 8RI3l-$?-
Doesn't get the EOT and so we never reach the prompt again.
We need deeper investigations.
#1 Updated by rpalethorpe over 3 years ago
Assuming that EOT is handled specially by the terminal and converted into a signal at some early stage, then it may be that the signal is raised before the cat process is ready to receive signals. I'm not sure if we wait long enough for the shell to start cat before sending the data. For the text content this probably doesn't matter because it is just buffered somewhere and eventually gets delivered to cat.
Sending EOT multiple times is dangerous because it might cause the shell to exit IIRC. We need to know when cat has started and is ready receive data, but I don't see a way of doing that.
#2 Updated by riafarov over 3 years ago
Such a call is harder to handle even on non virtio consoles, will basically work only if we type everything. But can you just replace it with some other call? To be honest I would expect that main use case for hyphen is when we want to input from file and from keyboard. In this case seems we could simply use echo or if script is too big, just download it from the worker.
#3 Updated by cfconrad over 3 years ago
riafarov Yes I think using a temp file and just downloading it, would avoid that problem. And if someone really need "type_command" we could go the echo approach.
rpalethorpe This seems to be the problem. I played with a delay and the problem didn't appear anymore.
I will remove my reproduce steps, as this didn't turned out to be correct. If I just run the tests in a loop, they fail from time to time...
#5 Updated by cfconrad over 3 years ago
We could just wait for the script being echoed back from the virtio terminal (https://github.com/cfconrad/os-autoinst/commit/1bfeca458c2c9a8eae3d7ea7f4aaa82027923968).
But unfortunately there seems to be a other problem, that sometimes the echo is missing
#6 Updated by cfconrad over 3 years ago
- Status changed from New to Resolved
We went the "here tag" approach, which anton cam up with, thx.
PR https://github.com/os-autoinst/os-autoinst/pull/1029 is merged.
#9 Updated by okurz over 3 years ago
Sure, that is certainly an exception. The normal flow should be like this that someone picks up a ticket whenever when that person starts the development and sets the ticket to "In Progress" together with assigning him-/herself and then just set to "Resolved" whenever done.