[Bug]: Playwright-Junit lifecycle incompatible with maven retries 'surefire.rerunFailingTestsCount'

[rtretyak]

Version

1.50.0

Steps to reproduce

1. Have a basic Playwright test that fails
2. Try running it with maven retries: ./mvnw test -Dtest=MyTest -Dsurefire.rerunFailingTestsCount=2

Expected behavior

System is able to retry the failing test, system can work with Playwright context (browser, page etc.)

Actual behavior

First attempt is a legit failure, subsequent rerun attempts have broken Playwright context

[ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.002 s <<< FAILURE! -- in fm.seller.onboardingweb.test.MyTest
[ERROR] fm.seller.onboardingweb.test.MyTest.test(Page) -- Time elapsed: 0.001 s <<< ERROR!
org.junit.jupiter.api.extension.ParameterResolutionException: Failed to resolve parameter [com.microsoft.playwright.Page page] in method [public void fm.seller.onboardingweb.test.MyTest.test(com.microsoft.playwright.Page)]: Playwright connection closed
        at java.base/java.util.ArrayList.forEach(ArrayList.java:1511)
        at java.base/java.util.ArrayList.forEach(ArrayList.java:1511)
Caused by: com.microsoft.playwright.PlaywrightException: Playwright connection closed
        at com.microsoft.playwright.impl.PipeTransport.send(PipeTransport.java:50)
        at com.microsoft.playwright.impl.Connection.internalSendMessage(Connection.java:168)
        at com.microsoft.playwright.impl.Connection.sendMessageAsync(Connection.java:134)
        at com.microsoft.playwright.impl.ChannelOwner.sendMessageAsync(ChannelOwner.java:109)
        at com.microsoft.playwright.impl.SelectorsImpl.setTestIdAttributeName(SelectorsImpl.java:42)
        at com.microsoft.playwright.impl.SharedSelectors.lambda$setTestIdAttribute$2(SharedSelectors.java:72)
        at java.base/java.util.ArrayList.forEach(ArrayList.java:1511)
        at com.microsoft.playwright.impl.SharedSelectors.setTestIdAttribute(SharedSelectors.java:72)
        at com.microsoft.playwright.impl.junit.ExtensionUtils.setTestIdAttribute(ExtensionUtils.java:53)
        at com.microsoft.playwright.impl.junit.BrowserContextExtension.getOrCreateBrowserContext(BrowserContextExtension.java:60)
        at com.microsoft.playwright.impl.junit.PageExtension.getOrCreatePage(PageExtension.java:51)
        at com.microsoft.playwright.impl.junit.PageExtension.resolveParameter(PageExtension.java:35)
        ... 2 more

[ERROR] Errors: 
[ERROR] fm.seller.onboardingweb.test.MyTest.test(Page)
[ERROR]   Run 1: MyTest.test:13
[ERROR]   Run 2: MyTest.test(Page) » ParameterResolution Failed to resolve parameter [com.microsoft.playwright.Page page] in method [public void fm.seller.onboardingweb.test.MyTest.test(com.microsoft.playwright.Page)]: Playwright connection closed
[ERROR]   Run 3: MyTest.test(Page) » ParameterResolution Failed to resolve parameter [com.microsoft.playwright.Page page] in method [public void fm.seller.onboardingweb.test.MyTest.test(com.microsoft.playwright.Page)]: Playwright connection closed

Additional context

Minimal test:

@UsePlaywright(OptionsFactory.class)
public class MyTest {
    @Test
    public void test(Page page) {
        page.navigate("https://google.com/");
        assert false;
    }
}

Environment

• OS name: "mac os x", version: "15.3", arch: "aarch64", family: "mac"
• Java version: 17.0.2, vendor: Oracle Corporation
• Apache Maven 3.8.5 (3599d3414f046de2324203b78ddcf9b5e4388aa0)
• playwright 1.50.0
• Junit 5.11.4
• surefire 3.5.2

[yury-s]

• Have a basic Playwright test that fails

Please share a complete example. I was not able to reproduce it with a simple test.

uchagani (@uchagani) can you have a look?

[uchagani]

I cannot reproduce this. Please provide a complete example. Here is mine that works as expected:

import com.microsoft.playwright.Page;
import com.microsoft.playwright.assertions.PageAssertions;
import com.microsoft.playwright.junit.Options;
import com.microsoft.playwright.junit.OptionsFactory;
import com.microsoft.playwright.junit.UsePlaywright;
import org.junit.jupiter.api.Test;

import static com.microsoft.playwright.assertions.PlaywrightAssertions.assertThat;

@UsePlaywright(SampleTests.PlaywrightOptions.class)
public class SampleTests {
    public static class PlaywrightOptions implements OptionsFactory {
        @Override
        public Options getOptions() {
            return new Options().setBaseUrl("https://www.playwright.dev");
        }
    }

    @Test
    void testPageTitle(Page page) {
        page.navigate("/");
        assertThat(page).hasTitle("foo", new PageAssertions.HasTitleOptions().setTimeout(1000));
    }
}

pom.xml

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <groupId>org.example</groupId>
    <artifactId>playwright-java-sample</artifactId>
    <version>1.0-SNAPSHOT</version>

    <properties>
        <maven.compiler.source>11</maven.compiler.source>
        <maven.compiler.target>11</maven.compiler.target>
        <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
    </properties>

    <dependencies>
        <dependency>
            <groupId>com.microsoft.playwright</groupId>
            <artifactId>playwright</artifactId>
            <version>1.50.0</version>
        </dependency>

        <dependency>
            <groupId>org.junit.jupiter</groupId>
            <artifactId>junit-jupiter</artifactId>
            <version>5.11.4</version>
            <scope>test</scope>
        </dependency>
    </dependencies>
    <build>
        <pluginManagement>
            <plugins>
                <plugin>
                    <groupId>org.apache.maven.plugins</groupId>
                    <artifactId>maven-surefire-plugin</artifactId>
                    <version>3.5.2</version>
                </plugin>
            </plugins>
        </pluginManagement>
    </build>
</project>

[rtretyak]

pom:

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <groupId>org.example</groupId>
    <artifactId>playwrightRetries</artifactId>
    <version>1.0-SNAPSHOT</version>
    <parent>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-parent</artifactId>
        <version>3.4.2</version>
        <relativePath/>
    </parent>

    <properties>
        <maven.compiler.source>17</maven.compiler.source>
        <maven.compiler.target>17</maven.compiler.target>
        <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
    </properties>
    <build>
        <plugins>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-surefire-plugin</artifactId>
                <version>3.5.2</version>
            </plugin>
        </plugins>
    </build>
    <dependencies>
        <dependency>
            <groupId>com.microsoft.playwright</groupId>
            <artifactId>playwright</artifactId>
            <version>1.50.0</version>
        </dependency>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-test</artifactId>
            <scope>test</scope>
        </dependency>
    </dependencies>
</project>

test:

import com.microsoft.playwright.Page;
import com.microsoft.playwright.junit.UsePlaywright;
import org.junit.jupiter.api.Test;

@UsePlaywright
public class MyTest {
    @Test
    public void test(Page page) {
        page.navigate("https://google.com/");
        assert false;
    }
}

command:
mvn clean test -Dsurefire.rerunFailingTestsCount=1

output:

[INFO] -------------------------------------------------------
[INFO]  T E S T S
[INFO] -------------------------------------------------------
[INFO] Running MyTest
[ERROR] Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 3.671 s <<< FAILURE! -- in MyTest
[ERROR] MyTest.test(Page) -- Time elapsed: 3.640 s <<< FAILURE!
java.lang.AssertionError
        at MyTest.test(MyTest.java:10)
        at java.base/java.lang.reflect.Method.invoke(Method.java:580)
        at java.base/java.util.ArrayList.forEach(ArrayList.java:1597)
        at java.base/java.util.ArrayList.forEach(ArrayList.java:1597)

[INFO] Running MyTest
[ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.004 s <<< FAILURE! -- in MyTest
[ERROR] MyTest.test(Page) -- Time elapsed: 0.002 s <<< ERROR!
org.junit.jupiter.api.extension.ParameterResolutionException: Failed to resolve parameter [com.microsoft.playwright.Page page] in method [public void MyTest.test(com.microsoft.playwright.Page)]: Playwright connection closed
        at java.base/java.util.ArrayList.forEach(ArrayList.java:1597)
        at java.base/java.util.ArrayList.forEach(ArrayList.java:1597)
Caused by: com.microsoft.playwright.PlaywrightException: Playwright connection closed
        at com.microsoft.playwright.impl.PipeTransport.send(PipeTransport.java:50)
        at com.microsoft.playwright.impl.Connection.internalSendMessage(Connection.java:168)
        at com.microsoft.playwright.impl.Connection.sendMessageAsync(Connection.java:134)
        at com.microsoft.playwright.impl.ChannelOwner.sendMessageAsync(ChannelOwner.java:109)
        at com.microsoft.playwright.impl.SelectorsImpl.setTestIdAttributeName(SelectorsImpl.java:42)
        at com.microsoft.playwright.impl.SharedSelectors.lambda$setTestIdAttribute$2(SharedSelectors.java:72)
        at java.base/java.util.ArrayList.forEach(ArrayList.java:1597)
        at com.microsoft.playwright.impl.SharedSelectors.setTestIdAttribute(SharedSelectors.java:72)
        at com.microsoft.playwright.impl.junit.ExtensionUtils.setTestIdAttribute(ExtensionUtils.java:53)
        at com.microsoft.playwright.impl.junit.BrowserContextExtension.getOrCreateBrowserContext(BrowserContextExtension.java:60)
        at com.microsoft.playwright.impl.junit.PageExtension.getOrCreatePage(PageExtension.java:51)
        at com.microsoft.playwright.impl.junit.PageExtension.resolveParameter(PageExtension.java:35)
        ... 2 more

[INFO] 
[INFO] Results:
[INFO] 
[ERROR] Errors: 
[ERROR] MyTest.test(Page)
[ERROR]   Run 1: MyTest.test:10
[ERROR]   Run 2: MyTest.test(Page) » ParameterResolution Failed to resolve parameter [com.microsoft.playwright.Page page] in method [public void MyTest.test(com.microsoft.playwright.Page)]: Playwright connection closed
[INFO] 
[INFO] 
[ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0
[INFO] 
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE

[rtretyak]

uchagani (@uchagani) I tried your project and I'm getting the same problem. The retry is not able to start properly:

[INFO] -------------------------------------------------------
[INFO]  T E S T S
[INFO] -------------------------------------------------------
[INFO] Running MyTest
[ERROR] Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 5.565 s <<< FAILURE! -- in MyTest
[ERROR] MyTest.testPageTitle(Page) -- Time elapsed: 5.528 s <<< FAILURE!
org.opentest4j.AssertionFailedError: 
Page title expected to be: foo
Received: Fast and reliable end-to-end testing for modern web apps | Playwright

Call log:
  - Locator.expect with timeout 1000ms
  - waiting for locator(":root")
    5 × locator resolved to <html lang="en" dir="ltr" data-theme="light" data-has-hydrated="true" class="plugin-pages plugin-id-default" data-rh="lang,dir,class,data-has-hydrated">…</html>
      - unexpected value "Fast and reliable end-to-end testing for modern web apps | Playwright"
        at com.microsoft.playwright.impl.AssertionsBase.expectImpl(AssertionsBase.java:74)
        at com.microsoft.playwright.impl.AssertionsBase.expectImpl(AssertionsBase.java:50)
        at com.microsoft.playwright.impl.AssertionsBase.expectImpl(AssertionsBase.java:42)
        at com.microsoft.playwright.impl.PageAssertionsImpl.hasTitle(PageAssertionsImpl.java:45)
        at MyTest.testPageTitle(MyTest.java:22)
        at java.base/java.lang.reflect.Method.invoke(Method.java:580)
        at java.base/java.util.ArrayList.forEach(ArrayList.java:1597)
        at java.base/java.util.ArrayList.forEach(ArrayList.java:1597)

[INFO] Running MyTest
[ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.002 s <<< FAILURE! -- in MyTest
[ERROR] MyTest.testPageTitle(Page) -- Time elapsed: 0.002 s <<< ERROR!
org.junit.jupiter.api.extension.ParameterResolutionException: Failed to resolve parameter [com.microsoft.playwright.Page arg0] in method [void MyTest.testPageTitle(com.microsoft.playwright.Page)]: Playwright connection closed
        at java.base/java.util.ArrayList.forEach(ArrayList.java:1597)
        at java.base/java.util.ArrayList.forEach(ArrayList.java:1597)
Caused by: com.microsoft.playwright.PlaywrightException: Playwright connection closed
        at com.microsoft.playwright.impl.PipeTransport.send(PipeTransport.java:50)
        at com.microsoft.playwright.impl.Connection.internalSendMessage(Connection.java:168)
        at com.microsoft.playwright.impl.Connection.sendMessageAsync(Connection.java:134)
        at com.microsoft.playwright.impl.ChannelOwner.sendMessageAsync(ChannelOwner.java:109)
        at com.microsoft.playwright.impl.SelectorsImpl.setTestIdAttributeName(SelectorsImpl.java:42)
        at com.microsoft.playwright.impl.SharedSelectors.lambda$setTestIdAttribute$2(SharedSelectors.java:72)
        at java.base/java.util.ArrayList.forEach(ArrayList.java:1597)
        at com.microsoft.playwright.impl.SharedSelectors.setTestIdAttribute(SharedSelectors.java:72)
        at com.microsoft.playwright.impl.junit.ExtensionUtils.setTestIdAttribute(ExtensionUtils.java:53)
        at com.microsoft.playwright.impl.junit.BrowserContextExtension.getOrCreateBrowserContext(BrowserContextExtension.java:60)
        at com.microsoft.playwright.impl.junit.PageExtension.getOrCreatePage(PageExtension.java:51)
        at com.microsoft.playwright.impl.junit.PageExtension.resolveParameter(PageExtension.java:35)
        ... 2 more

[INFO] 
[INFO] Results:
[INFO] 
[ERROR] Errors: 
[ERROR] MyTest.testPageTitle(Page)
[ERROR]   Run 1: MyTest.testPageTitle:22 Page title expected to be: foo
Received: Fast and reliable end-to-end testing for modern web apps | Playwright

Call log:
  - Locator.expect with timeout 1000ms
  - waiting for locator(":root")
    5 × locator resolved to <html lang="en" dir="ltr" data-theme="light" data-has-hydrated="true" class="plugin-pages plugin-id-default" data-rh="lang,dir,class,data-has-hydrated">…</html>
      - unexpected value "Fast and reliable end-to-end testing for modern web apps | Playwright"
[ERROR]   Run 2: MyTest.testPageTitle(Page) » ParameterResolution Failed to resolve parameter [com.microsoft.playwright.Page arg0] in method [void MyTest.testPageTitle(com.microsoft.playwright.Page)]: Playwright connection closed
[INFO] 
[INFO] 
[ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0
[INFO] 
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE

Here is also my mvn version output:

 % mvn -version
Apache Maven 3.8.5 (3599d3414f046de2324203b78ddcf9b5e4388aa0)
Maven home: /opt/homebrew/Cellar/maven/3.8.5/libexec
Java version: 23, vendor: Homebrew, runtime: /opt/homebrew/Cellar/openjdk/23/libexec/openjdk.jdk/Contents/Home
Default locale: en_UA, platform encoding: UTF-8
OS name: "mac os x", version: "15.3", arch: "aarch64", family: "mac"

And I also tried with this with the same result:

 % ./mvnw -version
Apache Maven 3.8.5 (3599d3414f046de2324203b78ddcf9b5e4388aa0)
Maven home: /Users/user/.m2/wrapper/dists/apache-maven-3.8.5-bin/67203e94/apache-maven-3.8.5
Java version: 17.0.2, vendor: Oracle Corporation, runtime: /Users/user/Library/Java/JavaVirtualMachines/openjdk-17.0.2/Contents/Home
Default locale: en_UA, platform encoding: UTF-8
OS name: "mac os x", version: "15.3", arch: "aarch64", family: "mac"

[uchagani]

I'm not sure what could be causing this and I can't seem to reproduce it.

[rtretyak]

I have also tried it on my home Windows PC which is completely independent from the previous environment and was able to reproduce it with your project.

mvn -version
Apache Maven 3.8.4 (9b656c72d54e5bacbed989b64718c159fe39b537)
Maven home: E:\workspace\apache-maven-3.8.4
Java version: 23.0.2, vendor: Oracle Corporation, runtime: C:\Users\38097\.jdks\openjdk-23.0.2
Default locale: en_US, platform encoding: UTF-8
OS name: "windows 10", version: "10.0", arch: "amd64", family: "windows"

Let me know if there is any other debug information I can provide you with...

[uchagani]

Are you configuring your Java or maven in any specific way?

Can you check and post your settings.xml file (without any private info).

Are you running these tests from inside of an IDE or just the command line/terminal?

[rtretyak]

Are you configuring your Java or maven in any specific way?

No, nothing special. I tried with multiple different Javas as you can see above. Not doing anyhting special to them. 1 was installed with homebrew, other 2 just a default installation by IntelliJ IDEA

settings.xml file

default one, just empty tags, no additional configurations

Are you running these tests from inside of an IDE or just the command line/terminal?

I'm running from terminal. Tried both in terminal inside IntelliJ and native terminal apps on windows/mac, same result.

[rtretyak]

You can also reproduce it in the official playwright-java image..
docker run --rm -v "$(pwd):/app" -w /app mcr.microsoft.com/playwright/java:v1.50.0-noble mvn clean test -Dsurefire.rerunFailingTestsCount=1

[Develop-KIM]

I hit this too and dug into it. PlaywrightExtension keeps the Playwright in a static ThreadLocal, and the PlaywrightRegistry in the root store closes it when the launcher finishes the first run, but the ThreadLocal is never cleared. Surefire's rerun happens in the same JVM on the same thread, so getOrCreatePlaywright hands back the closed instance and BrowserExtension fails on launch():

ParameterResolutionException: Failed to resolve parameter [com.microsoft.playwright.Page arg0] ...: Playwright connection closed
  at com.microsoft.playwright.impl.BrowserTypeImpl.launch(BrowserTypeImpl.java:48)
  at com.microsoft.playwright.impl.junit.BrowserExtension.getOrCreateBrowser(BrowserExtension.java:82)

Repro on 1.62.0 is one @UsePlaywright test that always fails plus mvn test -Dsurefire.rerunFailingTestsCount=1. One thing to be aware of: since #1860 (1.57) the registry is AutoCloseable rather than CloseableResource, and JUnit only closes AutoCloseable store values from 5.13 on. With junit-jupiter 5.11 the first Playwright is simply never closed, so the retry reuses it and the problem is invisible; with 5.14 it fails every time.

The fix I have in mind is to only reuse the ThreadLocal instance when it belongs to the current run's registry and create a fresh one otherwise (clearing the ThreadLocal from close() can't reach other threads). Happy to send a PR if that direction works for you.

[Develop-KIM]

Opened #1988 with the fix and a regression test. If you'd rather handle this differently I'm fine closing it.