action #77836
login to aarch64.o.o fails with ssh keys and password, also not working over IPMI SoL
0%
Description
Observation¶
login to aarch64.o.o fails with ssh keys and password, also not working over IPMI SoL. Was reported by mkittler and reproduced by ggardet_arm
Acceptance criteria¶
- AC1: login works again for QE Tools members
- AC2: login works for ggardet_arm
- AC3: We understood how it came to that, e.g. if we have been hacked, system corruption, user error, etc.
Suggestions¶
- If you do not yet have IPMI aliases I suggest https://gitlab.suse.de/openqa/salt-pillars-openqa#get-ipmi-definition-aliases
- Connect over IPMI SoL
- try to boot previous snapshots
- If they don't work boot with
init=/bin/bash
or a rescue system or installer if that is available over PXE, and chroot into the installed system - investigate, recover, fix
History
#1
Updated by okurz over 2 years ago
- Description updated (diff)
#2
Updated by favogt over 2 years ago
With IPMI I checked the graphical console and on tty10 (I think) rsyslog was spamming write error: Read-only file system
.
I triggered a reboot, as it was inaccessible anyway.
#3
Updated by favogt over 2 years ago
It's up again, apparently running fine.
At "2020-11-12T03:38:14.930211+01:00", logging stopped, probably due to /var becoming read only or whatever the root cause was.
#4
Updated by okurz over 2 years ago
there is also https://infra.nue.suse.com/SelfService/Display.html?id=179857 which I created. We should coordinate with EngInfra.
#5
Updated by nicksinger over 2 years ago
- Assignee set to nicksinger
#6
Updated by nicksinger over 2 years ago
- Status changed from Workable to In Progress
Machine was shut of now. I started it again, currently booting
#7
Updated by nicksinger over 2 years ago
After I did an explicit "power on" the machine came back online and I was able to login with the expected credentials. As favogt mentioned (thanks for checking!) errors about /var
I fired up smart to see if any errors where reported in the past:
openqa-aarch64:/home/opensuse # smartctl -i /dev/sda2 smartctl 7.0 2019-05-21 r4917 [aarch64-linux-5.8.15-1.gc680e93-default] (SUSE RPM) Copyright (C) 2002-18, Bruce Allen, Christian Franke, www.smartmontools.org === START OF INFORMATION SECTION === Model Family: Intel 730 and DC S35x0/3610/3700 Series SSDs Device Model: INTEL SSDSC2BP480G4 Serial Number: BTJR507404WJ480BGN LU WWN Device Id: 5 5cd2e4 04b7929ce Firmware Version: L2010420 User Capacity: 480.103.981.056 bytes [480 GB] Sector Size: 512 bytes logical/physical Rotation Rate: Solid State Device Form Factor: 2.5 inches Device is: In smartctl database [for details use: -P show] ATA Version is: ATA8-ACS T13/1699-D revision 4 SATA Version is: SATA 2.6, 6.0 Gb/s (current: 6.0 Gb/s) Local Time is: Fri Nov 13 13:31:56 2020 CET SMART support is: Available - device has SMART capability. SMART support is: Enabled openqa-aarch64:/home/opensuse # smartctl -a /dev/sda2 smartctl 7.0 2019-05-21 r4917 [aarch64-linux-5.8.15-1.gc680e93-default] (SUSE RPM) Copyright (C) 2002-18, Bruce Allen, Christian Franke, www.smartmontools.org === START OF INFORMATION SECTION === Model Family: Intel 730 and DC S35x0/3610/3700 Series SSDs Device Model: INTEL SSDSC2BP480G4 Serial Number: BTJR507404WJ480BGN LU WWN Device Id: 5 5cd2e4 04b7929ce Firmware Version: L2010420 User Capacity: 480.103.981.056 bytes [480 GB] Sector Size: 512 bytes logical/physical Rotation Rate: Solid State Device Form Factor: 2.5 inches Device is: In smartctl database [for details use: -P show] ATA Version is: ATA8-ACS T13/1699-D revision 4 SATA Version is: SATA 2.6, 6.0 Gb/s (current: 6.0 Gb/s) Local Time is: Fri Nov 13 13:32:30 2020 CET SMART support is: Available - device has SMART capability. SMART support is: Enabled === START OF READ SMART DATA SECTION === SMART overall-health self-assessment test result: PASSED General SMART Values: Offline data collection status: (0x02) Offline data collection activity was completed without error. Auto Offline Data Collection: Disabled. Self-test execution status: ( 0) The previous self-test routine completed without error or no self-test has ever been run. Total time to complete Offline data collection: ( 2) seconds. Offline data collection capabilities: (0x79) SMART execute Offline immediate. No Auto Offline data collection support. Suspend Offline collection upon new command. Offline surface scan supported. Self-test supported. Conveyance Self-test supported. Selective Self-test supported. SMART capabilities: (0x0003) Saves SMART data before entering power-saving mode. Supports SMART auto save timer. Error logging capability: (0x01) Error logging supported. General Purpose Logging supported. Short self-test routine recommended polling time: ( 1) minutes. Extended self-test routine recommended polling time: ( 2) minutes. Conveyance self-test routine recommended polling time: ( 2) minutes. SCT capabilities: (0x003d) SCT Status supported. SCT Error Recovery Control supported. SCT Feature Control supported. SCT Data Table supported. SMART Attributes Data Structure revision number: 1 Vendor Specific SMART Attributes with Thresholds: ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE 5 Reallocated_Sector_Ct 0x0032 097 097 000 Old_age Always - 0 9 Power_On_Hours 0x0032 100 100 000 Old_age Always - 17587 12 Power_Cycle_Count 0x0032 100 100 000 Old_age Always - 86 170 Available_Reservd_Space 0x0033 100 100 010 Pre-fail Always - 0 171 Program_Fail_Count 0x0032 100 100 000 Old_age Always - 0 172 Erase_Fail_Count 0x0032 100 100 000 Old_age Always - 0 174 Unsafe_Shutdown_Count 0x0032 100 100 000 Old_age Always - 82 175 Power_Loss_Cap_Test 0x0033 100 100 010 Pre-fail Always - 619 (90 17) 183 SATA_Downshift_Count 0x0032 100 100 000 Old_age Always - 0 184 End-to-End_Error 0x0033 100 100 090 Pre-fail Always - 0 187 Reported_Uncorrect 0x0032 100 100 000 Old_age Always - 0 190 Temperature_Case 0x0022 083 083 000 Old_age Always - 17 (Min/Max 12/18) 192 Unsafe_Shutdown_Count 0x0032 100 100 000 Old_age Always - 82 194 Temperature_Internal 0x0022 100 100 000 Old_age Always - 17 197 Current_Pending_Sector 0x0032 100 100 000 Old_age Always - 0 199 CRC_Error_Count 0x003e 100 100 000 Old_age Always - 0 225 Host_Writes_32MiB 0x0032 100 100 000 Old_age Always - 18321456 226 Workld_Media_Wear_Indic 0x0032 100 100 000 Old_age Always - 11776 227 Workld_Host_Reads_Perc 0x0032 100 100 000 Old_age Always - 35 228 Workload_Minutes 0x0032 100 100 000 Old_age Always - 126853 232 Available_Reservd_Space 0x0033 100 100 010 Pre-fail Always - 0 233 Media_Wearout_Indicator 0x0032 043 043 000 Old_age Always - 0 234 Thermal_Throttle 0x0032 100 100 000 Old_age Always - 0/0 241 Host_Writes_32MiB 0x0032 100 100 000 Old_age Always - 18321456 242 Host_Reads_32MiB 0x0032 100 100 000 Old_age Always - 7116970 SMART Error Log Version: 1 No Errors Logged SMART Self-test log structure revision number 1 Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error # 1 Short offline Completed without error 00% 17578 - # 2 Short offline Completed without error 00% 17554 - # 3 Short offline Completed without error 00% 17530 - # 4 Short offline Completed without error 00% 17506 - # 5 Short offline Completed without error 00% 17482 - # 6 Short offline Completed without error 00% 17458 - # 7 Short offline Completed without error 00% 17434 - # 8 Short offline Completed without error 00% 17410 - # 9 Short offline Completed without error 00% 17386 - #10 Short offline Completed without error 00% 17362 - #11 Short offline Completed without error 00% 17338 - #12 Short offline Completed without error 00% 17314 - #13 Short offline Completed without error 00% 17290 - #14 Extended offline Completed without error 00% 17288 - #15 Short offline Completed without error 00% 17266 - #16 Short offline Completed without error 00% 17242 - #17 Short offline Completed without error 00% 17218 - #18 Short offline Completed without error 00% 17194 - #19 Short offline Completed without error 00% 17170 - #20 Short offline Completed without error 00% 17146 - #21 Short offline Completed without error 00% 17122 - SMART Selective self-test log data structure revision number 1 SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS 1 0 0 Not_testing 2 0 0 Not_testing 3 0 0 Not_testing 4 0 0 Not_testing 5 0 0 Not_testing Selective self-test flags (0x0): After scanning selected spans, do NOT read-scan remainder of disk. If Selective self-test is pending on power-up, resume after 0 minute delay.
So as you can see the last 21 self-tests where successful without errors. I will dig a little bit more if I can find the root cause but for now the main issue (nobody can login) is resolved.
#8
Updated by nicksinger over 2 years ago
- Status changed from In Progress to Feedback
#9
Updated by okurz over 2 years ago
nicksinger thanks for taking the ticket . fvogt and guillaume_g are following, discussing about it in irc://chat.freenode.net/opensuse-factory but too shy to update you :D But of course they are happy that you fixed it :)
I updated
https://infra.nue.suse.com/SelfService/Display.html?id=179857
and asked to close it.
#10
Updated by nicksinger over 2 years ago
- Status changed from Feedback to Resolved
Closing for now as it is really hard to debug and the disk itself looks good. All files in /var/log are (obviously) truncated. If this happens again soon and we need to debug further we could think about setting up a central rsyslogd-server (e.g. on o3 itself) to collect the root cause there. I skipped this for now as it would require close monitoring (to not fill up o3 with all the worker logs) and I'm not up for this over the weekend :)