coordination #61203: [kernel][ltp][epic][grub] General solution for handling kernel parameters (debug_pagealloc=on) - openQA Tests - openSUSE Project Management Tool

Custom queries

All open Feature tests
openQA Infrastructure Project
openqa-review - Closed tickets last updated by openqa-review, last 30 days
QA roadmap long-term
QA SLE functional
QA SLE Functional - closed in last 14 days
QA SLE Functional - High, need to be refined
QA SLE Functional - over cycle time median
QA SLE u
QA SLE y
QA tools (tag not necessary in openQA and subprojects)
QA tools tag (tag not necessary in openQA and subprojects; excluding tickets in "Ready" version as they are already on the backlog)
QAC - Backlog
QAM
QE tools team - backlog (ready issues)
QE tools team - backlog (w/o infra)
QE tools team - backlog SLA high
QE tools team - backlog SLA immediate
QE tools team - backlog SLA no immediate/urgent in feedback/blocked
QE tools team - backlog SLA normal
QE tools team - backlog SLA urgent
QE tools team - backlog SLO high
QE tools team - backlog SLO normal
QE tools team - backlog SLO urgent
QE tools team - backlog, high-level view (epics and higher)
QE tools team - backlog, non-reactive work, needs parent
QE tools team - backlog, top-level view (all sagas)
QE tools team - closed within last 14 days
QE tools team - closed within last 60 days
QE tools team - closed yesterday
QE Tools Team - Collaborative Session
QE tools team - due date forecast
QE tools team - exceeding due-date
QE tools team - infrastructure backlog
QE tools team - next - sorted by update time
QE tools team - next issues
QE tools team - non-estimated (unblocked) issues (w/o infra)
QE tools team - ready issues - Workable
QE tools team - ready, not assigned/blocked/low
QE tools team - update forecast
QE tools team - updated by priority
QE tools team - what members of the team are working on - Feedback (not-low)
QE Tools Team Backlog By Assignee
SLE15 Migration Open Tickets
SLE15 SP1 Migration Open Tickets
SLE15SP3 Migration open ticket
SLE15SP3 Security open ticket
Tools Team Retrospective
Tools Team Retrospective (not estimated or assigned)

Actions

Copy link

coordination #61203

closed

[kernel][ltp][epic][grub] General solution for handling kernel parameters (debug_pagealloc=on)

Added by pvorel over 4 years ago. Updated over 3 years ago.

Status:

Resolved

Priority:

Normal

Assignee:

pvorel

Category:

Target version:

QE Kernel - QE Kernel Done

Start date:

2020-01-08

Due date:

% Done:

100%

Estimated time:

(Total: 0.00 h)

Difficulty:

Description

We should decide which kernel params are needed and adapt our solution for it.

we had
1) general parameters in grub (for all tests, added via add_grub_cmdline_settings() into GRUB_CMDLINE_LINUX_DEFAULT grub variable in /etc/default/grub)
2) and ima_policy=tcb added via GRUB_PARAM openqa test variable and add_custom_grub_entries() as separate boot option in grub.

Recently we added debug_pagealloc=on among general parameters
https://github.com/os-autoinst/os-autoinst-distri-opensuse/commit/ba40311d3219f511408bd86a271044e08e574801
(as https://jira.suse.com/browse/SLE-9743; + some options are needed due https://jira.suse.com/browse/SLE-9505)
which found a problems with it in some QAM (ppc64le), QA (s390) and Tumbleweed tests (ppc64le)
https://bugzilla.suse.com/show_bug.cgi?id=1159455
https://bugzilla.suse.com/show_bug.cgi?id=1159096

While there is PR to temporarily revert debug_pagealloc=on
https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/9206
we should decide what params we need and how to achieve it.

If we need more parameters, we can achieve it via either
1) type some params (editing grub config before boot, solution we have never used for LTP)
2) or extend add_custom_grub_entries() to support more special entries than just one
3) or drop IMA parameter and reuse GRUB_PARAM for something else
4) or generate yet another LTP image with different parameters (waste of disc space)

Subtasks 1 (0 open — 1 closed)

action #61895: [kernel][ltp][grub][s390x] Implement solution for handling kernel parameters also for on svirt (S390_ZKVM)

Resolved

pvorel

2020-01-08

Actions

Related issues 1 (0 open — 1 closed)

Related to openQA Tests - action #51743: [openqa] All LTP tests are failing on boot_ltp for openSUSE (o3) on [x86_64]

Resolved

pvorel

2019-05-21

Actions

Issue # Delay: days Cancel

History
Notes
Property changes

Actions

Copy link

Updated by pvorel over 4 years ago

Description updated (diff)

Actions

Copy link

Updated by pvorel over 4 years ago

Related to action #51743: [openqa] All LTP tests are failing on boot_ltp for openSUSE (o3) on [x86_64] added

Actions

Copy link

Updated by MDoucha over 4 years ago

I propose turning GRUB_PARAMS into a semicolon-separated list of extra boot options. Each list entry will represent one Grub menu choice.

Actions

Copy link

Updated by pvorel over 4 years ago

Assignee set to pvorel
Target version changed from 454 to 445

+1 to this option. I'll implement it this sprint as it's quite urgent.

Next step would be to define desired setup we want. I suggest 1) default as it was, 2) debug_pagealloc=on for syscalls and CVE and 3) IMA.
IMHO booting non-default should be as a default set in boot_ltp (as it's already now for IMA), but I also add openqa variable LTP_GRUB_ITEM and LTP_GRUB_ITEM2 to optionally select different setup during runtime.

So the only thing which will not be supported is to add completely different parameter / delete parameter. Workaround for adding new parameter would be a custom build with GRUB_IMA (run test with INSTALL_LTP and GRUB_PARAM). Hope removing any parameter should never be needed.

Actions

Copy link

Updated by okurz over 4 years ago

There are already multiple test variables available for help, e.g. EXTRABOOTPARAMS_BOOT_LOCAL to add variables, multiples if you like, to use during boot of an existing image, EXTRABOOTPARAMS_DELETE_CHARACTERS to delete characters at a certain position defined by EXTRABOOTPARAMS_DELETE_NEEDLE_TARGET . I used that to turn VIDEO=1024x768-16 defined on older, created qcow images into VIDEO=1024x768 to be compatible with non-cirrus graphics adapters.

Actions

Copy link

Updated by pvorel over 4 years ago

Oliver, thanks for tips, good to know about them. We'd prefer to avoid needles, that's why I wrote add_custom_grub_entries() and boot_grub_item().
But I'm thinking about moving boot_grub_item() under certain variable into handle_grub().

Actions

Copy link

Updated by pvorel over 4 years ago

PR for general solution select boot item: https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/9217

Actions

Copy link

Updated by pvorel over 4 years ago

Status changed from New to In Progress

Actions

Copy link

Updated by pvorel over 4 years ago

PR implement mdoucha's suggestion turning GRUB_PARAMS into a semicolon-separated list of extra boot options
https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/9218

Actions

Copy link

#10

Updated by pvorel over 4 years ago

Both PRs are merged, now we should change setup. I suggest some tests with debug_pagealloc=on (probably syscalls and CVE), the rest without it.

Actions

Copy link

#11

Updated by pvorel over 4 years ago

GRUB_PARAM changed on both osd and o3 to GRUB_PARAM=debug_pagealloc=on;ima_policy=tcb

For tests using debug_pagealloc=on it's needed to add GRUB_BOOT_NONDEFAULT=1 (as there are more of them, for ltp_ima (single test) add GRUB_SELECT_FIRST_MENU=5

Added these test suites:

osd

kotd_ltp_cve_debug_pagealloc
ltp_syscalls_debug_pagealloc kotd_ltp_syscalls_debug_pagealloc
ltp_cve_debug_pagealloc
ltp_cve_spvm_debug_pagealloc
ltp_syscalls_spvm_debug_pagealloc

ltp_cve_debug_pagealloc
ltp_syscalls_debug_pagealloc

Actions

Copy link

#12

Updated by pvorel over 4 years ago

Status changed from In Progress to Feedback

Actions

Copy link

#13

Updated by pvorel over 4 years ago

Status changed from Feedback to Resolved

Broken ppc64le builds on Tumbleweed were fixed, by removing debug_pagealloc=on (and these using it are still failing, see
https://openqa.opensuse.org/tests/overview?distri=opensuse&version=Tumbleweed&build=20200104&groupid=32). There haven't been any SLES build yet, but it'll be fixed as well. This should also fix problems on s390x on SLES.

Problems with boot error on intel on Tumbleweed (#51743) still persists (no surprise).

Actions

Copy link

#14

Updated by MDoucha over 4 years ago

FYI: ltp_syscalls_debug_pagealloc and ltp_cve_debug_pagealloc added to QAM kernel job groups for SLE-12SP4+ and SLE-15* (all archs).

Actions

Copy link

#15

Updated by pvorel over 4 years ago

Subject changed from [kernel][ltp][epic] General solution for handling kernel parameters (debug_pagealloc=on) to [kernel][ltp][epic][grub] General solution for handling kernel parameters (debug_pagealloc=on)

Actions

Copy link

#16