Project

General

Profile

Actions
Copy link

action #55757

closed

OSD ftp service has problem: NFS mount on qanet did not handle move of assets folder on osd (or reboot) gracefully

Added by xlai over 4 years ago. Updated over 4 years ago.

Status:
Resolved
Priority:
High
Assignee:
okurz
Category:
-
Target version:
-
Start date:
2019-08-20
Due date:
% Done:

0%

Estimated time:

Description

From around 4 hours ago, pxe installation on osd always fail at boot_from_pxe with error "Could not find kernel image xxx", https://openqa.suse.de/tests/3270212#step/boot_from_pxe/6.

We tested ftp service, it is not accessible for images, but http works well.

Please help recover. It blocks virtualization test for beta4 candidate.

Log:

FTP STATUS:
'wget ftp://openqa.suse.de/assets/repo/fixed/SLE-12-SP4-Server-DVD-x86_64-GM-DVD1/boot/x86_64/loader/linux
--2019-08-20 06:15:32-- ftp://openqa.suse.de/assets/repo/fixed/SLE-12-SP4-Server-DVD-x86_64-GM-DVD1/boot/x86_64/loader/linux
=> ‘linux.1’
Resolving openqa.suse.de (openqa.suse.de)... 10.160.0.207, 2620:113:80c0:8080:10:160:0:207
Connecting to openqa.suse.de (openqa.suse.de)|10.160.0.207|:21... connected.
Logging in as anonymous ... Logged in!
==> SYST ... done. ==> PWD ... done.
==> TYPE I ... done. ==> CWD (1) /assets/repo/fixed/SLE-12-SP4-Server-DVD-x86_64-GM-DVD1/boot/x86_64/loader ...
No such directory ‘assets/repo/fixed/SLE-12-SP4-Server-DVD-x86_64-GM-DVD1/boot/x86_64/loader’.
'
HTTP:
'wget http://openqa.suse.de/assets/repo/fixed/SLE-12-SP4-Server-DVD-x86_64-GM-DVD1/boot/x86_64/loader/linux
--2019-08-20 06:16:35-- http://openqa.suse.de/assets/repo/fixed/SLE-12-SP4-Server-DVD-x86_64-GM-DVD1/boot/x86_64/loader/linux
Resolving openqa.suse.de (openqa.suse.de)... 10.160.0.207, 2620:113:80c0:8080:10:160:0:207
Connecting to openqa.suse.de (openqa.suse.de)|10.160.0.207|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 6856480 (6.5M)
Saving to: ‘linux.1’

100%[===============================================================>] 6,856,480 --.-K/s in 0.08s

2019-08-20 06:16:35 (80.0 MB/s) - ‘linux.1’ saved [6856480/6856480]
'

'# curl ftp://openqa.suse.de/assets/repo/fixed/SLE-12-SP4-Server-DVD-x86_64-GM-DVD1/boot/x86_64/loader/linux
curl: (9) Server denied you to change to the given directory
'

Actions
Copy link
 #1

Updated by okurz over 4 years ago

  • Assignee changed from nicksinger to okurz

I can see from journalctl -u vsftpd on osd:

Aug 20 13:44:19 openqa vsftpd[29454]: [ftp] FAIL DOWNLOAD: Client "::ffff:10.163.1.98", "/assets/repo/fixed/SLE-12-SP4-Server-DVD-x86_64-GM-DVD1/boot/x86_64/loader/linux", 0.00Kbyte/sec
Aug 20 13:44:25 openqa vsftpd[29617]: [ftp] OK LOGIN: Client "::ffff:10.160.1.100", anon password "yast@17.11.4"
Aug 20 13:44:25 openqa vsftpd[29623]: [ftp] OK DOWNLOAD: Client "::ffff:10.160.1.100", "/SLE-15-SP2-Staging:A-Module-Basesystem-POOL-x86_64-Build27.1-Media1/repodata/repomd.xml", 9125 bytes, 9350.61Kbyte/sec

so downloading works but the path /assets/repo/fixed/… doesn't . I wonder how that ever worked because the vsftpd config simply has anon_root=/var/lib/openqa/factory/repo/

Actions
Copy link
 #2

Updated by okurz over 4 years ago

On qanet:

qanetnue:~ # cat /etc/fstab
…
openqa.suse.de:/var/lib/openqa/share/factory    /mnt/openqa nfs ro,defaults 0 0 
qanetnue:~ # ls /mnt/openqa 
ls: cannot access /mnt/openqa: Stale NFS file handle
qanetnue:~ # umount /mnt/openqa
qanetnue:~ # mount -a
qanetnue:~ # ls /mnt/openqa/
hdd  iso  other  repo  tmp

retriggered job -> https://openqa.suse.de/tests/3271515

Seems like on qanet the nfs mount is not handling a reboot of OSD gracefully. As qanet is SLE11SP3 I would start with an upgrade first :)

Actions
Copy link
 #3

Updated by okurz over 4 years ago

  • Subject changed from OSD ftp service has problem to OSD ftp service has problem: NFS mount on qanet did not handle osd reboot gracefully
  • Status changed from New to In Progress
Actions
Copy link
 #4

Updated by okurz over 4 years ago

  • Subject changed from OSD ftp service has problem: NFS mount on qanet did not handle osd reboot gracefully to OSD ftp service has problem: NFS mount on qanet did not handle move of assets folder on osd (or reboot) gracefully
  • Status changed from In Progress to Resolved

test has passed this stage again: https://openqa.suse.de/tests/3271515 . Found one other test that failed in the same step, retriggered.

Actions
Copy link
 #5

Updated by xlai over 4 years ago

@okurz, thank you for the quick fix. It is working quite well now from virtualization tests.

Actions
Copy link

Also available in: Atom PDF