action #32290
closed[QAM][CaaSP] timeouts due to slow MF infobloxx
0%
Description
It's been over a month that Maintenance faces failed jobs in CaaSP due to timeouts. In our scenario we are downloading the updates (updates.suse.com) by registering CaaSP during the installation. Quickly enough we found out that the speed against updates.suse.com is very slow. To mitigate this, we decided to increase the timeout of the job. This kind of works, however, last week, we started seeing again similar issues. According to the latest testing-plan changes, we are going to run the upstream kubernetes conformance tests -- which will add an additional hour to the test. Given the situation with internet speed is problematic, this is an issue which blocks us of doing so.
Proper fix: We have filed a ticket (1 month ago) to Microfocus IT, but this is still not resolved. Last week, we escalated this up to department level as blocker. After talking with #infra, the problem is located at Microfocus info-bloxx cluster. For some reason, sometimes it thinks that connections from NUE office are in US and thus it redirects us through Washington DC. As a result we end-up with ping ~90-100ms. They are working to solve this issue.
Workaround: After having discussion with #infra we found that the proper Edgecast IP Address for Europe (infoblxx uses several Geo-located DNS servers) is: 68.232.34.211 . As a result, I've tried to add this at /etc/hosts of our DNS in CaaSP and the results were astonishing.
Before: https://openqa.suse.de/tests/1493854#step/installation/69 < 100kiB/s
After : http://skyrim.qam.suse.de/tests/2140#step/installation/3 > 30 MiB/s
Updated by mkravec about 6 years ago
Good job finding source of the problem
To use it in openqa.suse.de - https://gitlab.suse.de/openqa/salt-states-openqa/merge_requests/31
Updated by pgeorgiadis about 6 years ago
Can you review also: https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/4498
Updated by pgeorgiadis about 6 years ago
- Status changed from In Progress to Resolved
Thanks everybody for the quick review on this one :)
Updated by mkravec about 6 years ago
- Status changed from Resolved to In Progress
I modified support-server DNS to rewrite updates.suse.com IP. Using /etc/hosts does not work (it would have to be set on every node, which might be a problem on autoyast installations)
PR: https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/4537
Updated by pgeorgiadis about 6 years ago
- Status changed from In Progress to Resolved