Project

General

Profile

action #113282

Many incompletes due to VNC error "backend died: unexpected end of data at /usr/lib/os-autoinst/consoles/VNC.pm line 183.", especially on o3/aarch64 size:M

Added by mkittler about 1 month ago. Updated about 1 month ago.

Status:
Resolved
Priority:
Normal
Assignee:
Category:
Concrete Bugs
Target version:
Start date:
2022-07-05
Due date:
2022-07-20
% Done:

0%

Estimated time:
Difficulty:

Description

Observation

Especially on o3/aarch64 many jobs incomplete with backend died: unexpected end of data at /usr/lib/os-autoinst/consoles/VNC.pm line 183.. It looks like the first occurrence the problem 2357381 | 2022-05-19 20:09:39 | incomplete | backend died: unexpected end of data … and since then the log of incompletes on the worker openqa-aarch64 is significantly dominated by this error.

Likely the culprit is https://github.com/os-autoinst/os-autoinst/commit/d1adda78adc34c5ac02b5040a2bc0e97eaa83827 (and by extension https://github.com/os-autoinst/os-autoinst/commit/93ff454deae61e573a9cbf88f172304002fb83a4). In my tests/investigation with svirt jobs this change was an overall improvement. However, I can imagine that in certain cases it would be better to rather block longer on reads instead of giving up and possibly not being able to recover. I suppose the timeouts should be handled more sensibly. We should create a separate ticket for that problem.

Further details

Suggestion

  • Attempt to re-connect instead of just stopping the backend process.
  • As a workaround, increase VNC_TIMEOUT_LOCAL and VNC_TIMEOUT_REMOTE to a very high value on affected workers (and also set VNC_CONNECT_TIMEOUT_LOCAL/VNC_CONNECT_TIMEOUT_REMOTE explicitly so these stay the same). This way the old behavior (very long timeout on reads on the VNC socket) is restored. (If it doesn't help then https://github.com/os-autoinst/os-autoinst/commit/d1adda78adc34c5ac02b5040a2bc0e97eaa83827 was not the culprit after all).
  • Maybe it makes sense to increase the default timeout for VNC_TIMEOUT_LOCAL because 10 seconds might not be much on a busy worker. (I suppose VNC_CONNECT_TIMEOUT_LOCAL should be kept to be 10 seconds by default.)

Related issues

Related to openQA Project - action #111004: Timeout of test API functions not enforced if backend gets stuck, e.g. on the VNC socket size:MResolved2022-05-122022-05-28

History

#1 Updated by okurz about 1 month ago

  • Related to action #111004: Timeout of test API functions not enforced if backend gets stuck, e.g. on the VNC socket size:M added

#2 Updated by okurz about 1 month ago

  • Category set to Concrete Bugs
  • Target version set to Ready

#3 Updated by mkittler about 1 month ago

I've been editing /etc/workers.ini on aarch64 (see second suggestion) and will have a look whether it makes a difference.

#4 Updated by mkittler about 1 month ago

  • Status changed from New to In Progress
  • Assignee set to mkittler

#5 Updated by openqa_review about 1 month ago

  • Due date set to 2022-07-20

Setting due date based on mean cycle time of SUSE QE Tools

#6 Updated by mkittler about 1 month ago

  • Status changed from In Progress to Feedback

So far it looks better, the last incomplete is 2451865 | 2022-07-05 14:36:22 | incomplete | backend died: unexpected end of data at /usr/lib/os-autoinst/consoles/VNC.pm line 183.. I'll have a look again in the next days. If it helps I'll adjust the defaults on os-autoinst level because it would mean that 10 seconds is not a good default timeouts for reads on the local VNC connections.

#7 Updated by ggardet_arm about 1 month ago

mkittler wrote:

So far it looks better, the last incomplete is 2451865 | 2022-07-05 14:36:22 | incomplete | backend died: unexpected end of data at /usr/lib/os-autoinst/consoles/VNC.pm line 183.. I'll have a look again in the next days. If it helps I'll adjust the defaults on os-autoinst level because it would mean that 10 seconds is not a good default timeouts for reads on the local VNC connections.

Let's wait to have a new Tumbleweed snapshot to have a real load on workers.

#8 Updated by okurz about 1 month ago

  • Subject changed from Many incompletes due to VNC error "backend died: unexpected end of data at /usr/lib/os-autoinst/consoles/VNC.pm line 183.", especially on o3/aarch64 to Many incompletes due to VNC error "backend died: unexpected end of data at /usr/lib/os-autoinst/consoles/VNC.pm line 183.", especially on o3/aarch64 size:M
  • Description updated (diff)

#9 Updated by mkittler about 1 month ago

  • Description updated (diff)

Yes, let's wait until the next snapshot. At least so far it looks still good.

#10 Updated by mkittler about 1 month ago

We've got no further VNC errors:

openqa=# select id, t_finished, result, reason from jobs where (select host from workers where id = assigned_worker_id) = 'openqa-aarch64' and result = 'incomplete' and reason like '%VNC%' and t_finished >= '2022-07-06T00:00:00' order by t_finished asc;
 id | t_finished | result | reason 
----+------------+--------+--------
(0 Zeilen)

There were 582 jobs running on the worker:

openqa=# select count(id) from jobs where (select host from workers where id = assigned_worker_id) = 'openqa-aarch64' and t_finished >= '2022-07-06T00:00:00';
 count 
-------
   582
(1 Zeile)

I'll check again next week but I've already prepared a PR to change the defaults: https://github.com/os-autoinst/os-autoinst/pull/2112

#11 Updated by mkittler about 1 month ago

  • Status changed from Feedback to Resolved

The query still gives no results (now after 818 jobs were running). So I guess we're good again. The PR has been merged and deployed as well. So I removed the custom config on aarch64 and I think we can consider this resolved.

Also available in: Atom PDF