Running Playwright tests in openQA¶

Note: At the time of writing (2023-06-29) this we're still experementing with an Agama Live-CD, so things might change over time.

Building blocks¶

e2e-agama-playwright repo on GitHub¶

The e2e-agama-playwright repo was created by Joaquin and is supposed to be the repository that provides the Playwright tests.

yupdate¶

This is a script provided with the Agama Live-CD. The main purpose (at least for the Yam squad) is to provide a method of updating the playwright code in the Live-CD which is located under /usr/share/agama-playwright. For this use case we use yupdate patch <github_repo> <branch> which uses the branch on GitHub to update the Playwright and agama configuration.

Integration of yupdate into openQA¶

PR 17228 implements test modules to call yupdate from inside openQA.
The test module is stored under yam/agama/patch_agama.pm and uses a setting YUPDATE_GIT which is describred as follow:

YUPDATE_GIT | string | | Github link used by yast help script yupdate, format is repo#branch such as yast/agama#main.

Test module agama_installation¶

There is a test module yam/agama/agama-installation that has the purpose to run the installation. For that it uses the following code:

 assert_script_run('RUN_INSTALLATION=1 playwright test --trace on --project chromium --config /usr/share/agama-playwright take_screenshots', timeout => 600);

The take_screenshots test for Playwright has some specialties that we need to know:

    // check for multiple texts in parallel, avoid waiting for timeouts
    let action = await Promise.any([
      page.getByText("Product selection").waitFor().then(() => actions.setProduct),
      page.getByText("None authentication method").waitFor().then(() => actions.setPassword),
      page.getByText("Root authentication set").waitFor().then(() => actions.done),
    ]);

    // optional product selection
    if (action === actions.setProduct) {
      await test.step("Select the product", async () => {
        // select openSUSE Tumbleweed
        await page.getByText("openSUSE Tumbleweed").click();
        await page.screenshot({ path: "screenshots/product-selection.png" });
        await page.getByRole("button", { name: "Select" }).click();
      });

This means the test is able to determine if we're still on the product selection page and then select Tumbleweed.

    if (process.env.RUN_INSTALLATION === "1") {
      await test.step("Run installation",  async () => {
        // start the installation
        await page.getByRole("button", { name: "Install", exact: true }).click();
        await expect(page.getByText("Confirm Installation")).toBeVisible();
        await page.getByRole("button", { name: "Continue" }).click();

Here we check if we run the test with RUN_INSTALLATION=1 (as our openQA test module agama_installation does, so then we launch the installation by making Playwright click on the install button.

Keep that in mind if your adapted installation test module will call another Playwright test!

Experiments made so far¶

Clean tumbleweed installation¶

Test 3267958 on O3 executes a clean Tumbleweed installation using what is in the e2e-agama-playwright repo without updating it. The test run is still take screenshots.

Install tumbleweed after yupdate¶

Test 3374624 attempts a Tumbleweed installation, but Playwright reports the error, that there is no install button and the click() method runs into a timeout. The test makes use of yupdate by using the patch_agama test module, but in fact there shouldn't be updates since YUPDATE_GIT is still set to jknphy/e2e-agama-playwright#main. ** This needs to be investigated further**, differences to the above clean installation that worked:

• we now use the Agama-Live CD from staging, version "agama-2.1-staging".

Install tumbleweed with LVM settings¶

Test 3370827 by Lemon does a Tumbleweed installation with a specific lvm Playwright test module.
For this the agama_installation test module got tweaked and uses a direct run of yupdate instead of using the patch_agama test module in the YAML schedule:

assert_script_run("yupdate patch lemon-suse/e2e-agama-playwright add-lvm-installation", timeout => 300);

Open questions so far¶

Page Object Model¶

Both Lemon and Rainer have made Playwright tests with POM. Lemon places the pages directory inside the root of his repo, Rainer has stored it under tests/. Which place shall we use to have a common understanding?

What is the long term approach?¶

On the experiments above we practically launch exactly one Playwright test to do an installation. On the other hand wie have tickets that say "Implement XYZ using Page Object Model". So in the long run we should have pages and tests that e.g configure LVM or setup disk encrpytion, but we are lacking of a sort of "Playwright scheduler" that could make use of this in a similar way as we now use the diffeent pages to configure settings from the YaST installation overview when we run install tests in openQA. Shall we provide individual 'agama_installation_xzy' test modules where 'xyz' stands for a combination of different options or shall we think of a method to e.g. submit a YAML file that tells the "Playwright scheduler" to execute the tests in a defined order and then do the installation?