action #54791
closed[functional][y][research][timeboxed:24h] Test installation with network issues during the process
0%
Description
Motivation¶
We have explored tooling which will help us simulate different network issues. In #50855 we have found http://man7.org/linux/man-pages/man8/tc-netem.8.html
This tool allows to introduce packet losses, increasing network delays, etc.
As SLE 15 installer heavily depends on the network, we should know problematic areas and make sure that YaST handles this properly.
Acceptance criteria¶
- SLE 15 SP1 or SP2 installer is tested with simulated network issues
- Ticket for automation with detailed description of the suggested scenario is created in case worth it
Suggestions¶
- Check if installer can properly handle network disconnection (and reconnect)
- Check if installer can properly handle packet losses
- Check if installer can properly handle delays in responses
These scenarios are relevant for the packages installation (with remote addons) and registration, as those depend on the network mostly.
Another area is NTP configuration, as we have options to sync time and test connection to the server.
Files
| half-backed_modules.png (57.6 KB) half-backed_modules.png | cannot_access_installation_media | ||
| half-backed_error_modules.png (62.3 KB) half-backed_error_modules.png | cannot_access_installation_media_error_in_abort | ||
| updater_fail.png (78 KB) updater_fail.png |
Updated by riafarov over 4 years ago
- Subject changed from [functional][y][research] Test installation with network issues during the process to [functional][y][research][timeboxed:24h] Test installation with network issues during the process
- Description updated (diff)
- Status changed from New to Workable
Updated by ybonatakis over 4 years ago
- Status changed from Workable to In Progress
- Assignee set to ybonatakis
Updated by ybonatakis over 4 years ago
- File half-backed_modules.png half-backed_modules.png added
So far i havent managed to find any way to load tc before the installation.
A default Qdisc is present with cant be configured.
Also despite iptables command is available which also can be used to drop packets, it does work.
So i tried an installation manually, using tc in the host to manipulate the traffic. i experiment with 50% and 30% destruction. With some patient the second choice was affordable in some way.
With problematic network, the problems appear mainly in
- Updating the installer when it runs the 'Download the Packages' ** with 50% destructed network the installer could not continue ** not even when the network was reset.
- Registration ** it takes enough time to give you a error message. The error is "fail to open TCP connection to the scc address. ** if you pick some modules to be included the installation can take for ever trying to contacting the registration Server. After a long period of time the process can throw a notification [half-backed_modules.png]. When I chose "abort" the next thing i saw was the error message [half-backed_error_modules.png]. When i press the OK button, the installer continue to contacting the Registration server of the next module. and pretty much is imposible to stop the process from the installer
Performing the actual installation after all the steps can take real long time but i think it walks forward slowly
Updated by ybonatakis over 4 years ago
Updated by ybonatakis over 4 years ago
- File updater_fail.png updater_fail.png added
the goal today was to see if we can run a full installation with network issues.
Any attempts again was in two groups. in one we set the packet loss in 30% and in the second in 50%.
In general the installer can run a full installation (having destructive incoming networking in the whole process), with the following actions:
- When in "Updating the Installer", and if you have given the regurl during the boot, an error shows up to tell you that dns has failed for the scc. you can continue from there. The process will try to download a few other things. Every now and then and depending from the loss the network has, there are other warning to retry again [see updater_fail.png], or skip. Skip works also, for continue the process in this case.
- Skip registration
- Do not select to install any extra module
- have a lot of time for the actual installation
The 50% loss was really slow process. for both loss rates the actuall installation was insane slow to consider automation, unless the netword is restored to finish the final step.
Updated by ybonatakis over 4 years ago
- Status changed from In Progress to Feedback
Updated by riafarov over 4 years ago
- Status changed from Feedback to Resolved
ybonatakis will file a bug regarding Updating the installer when it runs the 'Download the Packages' ** with 50% destructed network the installer could not continue ** not even when the network was reset.
I will schedule follow-up ticket to check some scenarios with autoyast, as there we might not be able to proceed with installation.
Updated by ybonatakis over 4 years ago
@riafarov i run the case again to get some logs to include and the problem didnt appear. i tried a few times to make this sure(even with bigger loss of 50%). So no need for a bugzilla ticket. looks good