Upgrading PHPUnit and speeding up API Platform unit tests

I’m removing legacy code on the API Platform codebase to prepare for the API Platform 4 release. While doing that I also updated many dependency requirements such as PHP 8.2 (previously PHP 8.1 was allowed), Symfony 7.1 (as 7.0 is now EOL) and PHPUnit to 11 which is the main topic of this blog post. If you’re only interested in how I sped up tests click here.

# The Symfony PHPUnit Bridge

The Symfony PHPUnit Bridge is an attempt at extending PHPUnit to improve testing with Symfony. I’d resume its functionalities to:

# 1. an entrypoint to run tests differently

This not very useful and their are many ways to avoid using the bridge even on older PHPUnit versions. In my opinion features like “separating the dependencies of your app from those of phpunit” or “running tests in parallel when a test suite is split in several phpunit.xml files” should’ve stayed internal and be used only on the source code of the Symfony framework. Most of the features this adds needs messy code.

# 2. a test framework (mocking clocks and dates)

Mocks and helpers don’t really mess with PHPUnit and are just nice to have.

# 3. an error handler that catches deprecations and allows to ignore, trace, test deprecations

The most useful feature for API Platform.

The problem is that the bridge does not support PHPUnit 10.

# PHPUnit 10

PHPUnit 10 is:

the most significant release in the history of the PHPUnit project (which began 23 years ago). This release is to PHPUnit what PHP 7 was to PHP: a massive cleanup, refactoring, and modernisation that lays the groundwork for future development.

The most impacted users are PHPUnit extensions as some PHPUnit developers achieved a new system for extending PHPUnit based on events at the EU-FOSSA 2 Hackathon in 2019. The Symfony bridge relies a lot on the deprecated (and removed) TestListener, and one of the main issue is that:

The (new) event system is “read only”. Developers can write extensions for PHPUnit’s test runner and subscribe to events. The Event objects passed to subscribers as well as any value object aggregated in such an Event object is immutable.

Indeed, the bridge was relying on the ability to:

mutate the TestSuite (adding or removing tests)
change the result of a test (TestResult)

Nowadays, extending PHPUnit 10 is mostly about displaying or measuring things that happen during testing, not changing a test output or skipping a test which should be left to PHPUnit itself. The Pest framework for example, is even overriding some code, it uses the PHPUnit\TextUI\Application (although documented as not meant to be (re)used by developers). PHPUnit 11 also documents that one can wrap PHPUnit’s TestRunner eventhough it’s marked as @internal. This is what I do to run the tests on API Platform’s guides using our php-documentation-generator tool.

Therefore, it’s hard to maintain compatibility with new PHPUnit versions if you use things that changed drastically between PHPUnit 9 and 10. This also shows that there are ways to extend, and maybe provide the PHPUnit bridge features on top of PHPUnit 10. Feel free to read comments on the #49069 issue to know more about a potential compatibility.

# Not using the PHPUnit bridge

I’m mostly interested in deprecation management as API Platform does not rely on other features provided by the PHPUnit bridge. Although this was harder to manage using older PHPUnit versions, PHPUnit 11 now provides everything needed. To make this work, a few details need to be taken care of such as:

remove the PHPUnit bridge as it registers a PHP error_handler that conflicts with PHPUnit’s error handler
when using the Symfony Deprecation Contracts, add ignoreSuppressionOfDeprecations="true" as Symfony suppresses the triggered error using the Error Control Operator @
use --fail-on-deprecation to mimic SYMFONY_DEPRECATIONS_HELPER=max[direct]=0
use --display-deprecations

This is an example of API Platform’s source configuration:

<source ignoreSuppressionOfDeprecations="true" ignoreIndirectDeprecations="true" baseline="phpunit.baseline.xml">
	<include>
		<directory>.</directory>
	</include>
	<exclude>
		<directory>features</directory>
		<directory>vendor</directory>
		<file>.php-cs-fixer.dist.php</file>
	</exclude>
</source>

To ignore some tests we need to create a phpunit baseline as ignoring tests programmatically won’t work. As of today, using --generate-baseline doesn’t save deprecations, I proposed a patch to make this work. You can still write the baseline by hand (it’s more verbose then Symfony’s):

<?xml version="1.0"?>
<files version="1">
 <file path="vendor/symfony/deprecation-contracts/function.php">
  <line number="25" hash="c6af5d66288d0667e424978000f29571e4954b81">
   <issue><![CDATA[Since symfony/validator 7.1: Not passing a value for the "requireTld" option to the Url constraint is deprecated. Its default value will change to "true".]]></issue>
  </line>
 </file>
</files>

API Platform’s baseline, we ignore this error as the default value is just fine in our tests

# Speeding up tests

You can imagine that by doing the above, I had to run the API Platform’s test suite a bunch of time. Since we splitted our components, each component has its tests and at the root directory we have functional tests using: ApiTestCase, KernelTestCase or WebTestCase and some unit tests leftovers. My recent tweets show a massive improvement: from more then 2 minutes to a few seconds. Yes, there were less tests in the performant version but these were not the slow tests. I’m working on removing legacy code on API Platform, hence the reduced amount of tests.

I knew from the PHPUnit progress dots that some tests were taking too much time and to get the details I used the --debug option of PHPUnit. When it looked significantly slower, I noted the test to check. Without surprise the slow tests were the functional tests and I knew why. We have additional Behat tests with around 900 scenarios and almost as many ApiResource classes in our codebase. Every functional test works by loading all these fixtures which is not really an issue unless:

when Doctrine entities are used, we looped over all the classes to drop and recreate the schema
generating an OpenAPI specification over thousands of resources is slow (each also has a JSON schema that’s computed)

# Doctrine

Instead of recreating a whole Database schema, let’s load only what we need by creating a $this->recreateSchema([Dummy::class]) function:

trait RecreateSchemaTrait
{
    /**
     * @param class-string[] $classes
     */
    private function recreateSchema(array $classes = []): void
    {
        $manager = $this->getManager();

        /** @var ClassMetadata[] $cl */
        $cl = [];
        foreach ($classes as $c) {
            $cl[] = $manager->getMetadataFactory()->getMetadataFor($c);
        }

        $schemaTool = new SchemaTool($manager);
        @$schemaTool->dropSchema($cl);
        @$schemaTool->createSchema($cl);
    }

    private function getManager(): EntityManagerInterface
    {
        return static::getContainer()->get('doctrine')->getManager();
    }
}

This works great, each test case can pick whether to load the schema once setUpBeforeClass, or directly inside a test. This can be improved to create the schema before a test and dropping the schema when the test is done.

# API Resources

The second part of the improvement is where we gain even more performances. Tests like the OpenAPI tests only check a subset of the generated specification and we don’t need to generate a specification for all the fixtures. Selecting only the metadata we test is a good idea to speed up tests as it reduces the amount of data to generate. There are two problems with selecting resources:

metadata is cached and we need to force the regeneration, or avoid caching some parts of the metadata system
routes are based on what resources are loaded, and the router cache in Symfony is quite hard to get around

We definitely want to avoid regenerating the whole cache, as you’d find out browsing the web, an easy (but not performant) solution is to remove the var directory after each test.

To skip caching API Platform metadata, I replaced the ResourceNameCollectionFactory service by decorated the api_platform.metadata.resource.name_collection_factory.cached (it’s not calling the decoration chain as we know what resources we want to load).

phpunit_resource_name_collection:
    class: ApiPlatform\Tests\PhpUnitResourceNameCollectionFactory
    decorates: 'api_platform.metadata.resource.name_collection_factory.cached'
    public: true

Then, we could before each test call something like:

$container->get('phpunit_resource_name_collection')->setClasses([Dummy::class])

This is a good first step but while working on that I hit an issue with the Symfony router: the RouteCollection would not be computed again after each test. Indeed, there’s a caching mechanism and you need to create a ConfigCacheFactoryInterface that instructs whether the cache is fresh or stale:

use Symfony\Component\Config\ConfigCache;
use Symfony\Component\Config\ConfigCacheFactoryInterface;
use Symfony\Component\Config\ConfigCacheInterface;

final class ConfigCacheFactory implements ConfigCacheFactoryInterface
{
    public function cache(string $file, callable $callback): ConfigCacheInterface
    {
        $configCache = new TestSuiteConfigCache(new ConfigCache($file, false));

        if (!$configCache->isFresh()) {
            $callback($configCache);
        }

        return $configCache;
    }
}

Let’s also decorate the ConfigCache:

use Symfony\Component\Config\ConfigCacheInterface;

final readonly class TestSuiteConfigCache implements ConfigCacheInterface
{
    public function __construct(private ConfigCacheInterface $decorated) {}

    public function getPath(): string
    {
        return $this->decorated->getPath();
    }

    public function isFresh(): bool
    {
        return false;
    }

    public function write(string $content, ?array $metadata = null): void
    {
        $this->decorated->write($content, $metadata);
    }
}

And replace the service used by the Router:

config_cache_factory:
    class: ApiPlatform\Tests\ConfigCacheFactory

This almost works but I’m not satisfied as returning false forces the Router to be computed many times during a single Request. The solution I choose is to dump the resources I want to load into a file, then use the require function of environment variables to make the parameter dynamic. Indeed, as the Symfony container gets dumped (cache:warmup) the Container and its parameters are hard-written inside the var directory. If I use a simple parameter, I won’t be able to change it without clearing the cache. Note that when you work on Symfony code, the configuration has a resource mechanism that detects whether you changed a parameter or a file, and clears that cache for you.

Let code speak for itself:

parameters:
    env(RESOURCES): '%kernel.project_dir%/var/resources.php'
services:
    phpunit_resource_name_collection:
        class: ApiPlatform\Tests\PhpUnitResourceNameCollectionFactory
        decorates: 'api_platform.metadata.resource.name_collection_factory.cached'
        arguments:
            $classes: '%env(require:RESOURCES)%'
    config_cache_factory:
        class: ApiPlatform\Tests\ConfigCacheFactory

use ApiPlatform\Metadata\Resource\Factory\ResourceNameCollectionFactoryInterface;
use ApiPlatform\Metadata\Resource\ResourceNameCollection;
use ApiPlatform\State\ApiResource\Error;
use ApiPlatform\Validator\Exception\ValidationException;

final readonly class PhpUnitResourceNameCollectionFactory implements ResourceNameCollectionFactoryInterface
{
    /**
     * @param class-string[] $classes
     */
    public function __construct(private array $classes) {}

    public function create(): ResourceNameCollection
    {
        return new ResourceNameCollection($this->classes);
    }
}

So far so good, now this allows me to know whether my cache is fresh or not:

use Symfony\Component\Config\ConfigCacheInterface;

final readonly class TestSuiteConfigCache implements ConfigCacheInterface
{
    /** @var array<string, string> */
    public static $md5 = [];

    public function __construct(private ConfigCacheInterface $decorated) {}

    public function isFresh(): bool
    {
        $p = $this->getPath();
        if (!isset(static::$md5[$p]) || static::$md5[$p] !== $this->getHash()) {
            static::$md5[$p] = $this->getHash();
            return false;
        }

        return $this->decorated->isFresh();
    }

    private function getHash(): string
    {
        return md5_file(__DIR__.'/Fixtures/app/var/resources.php');
    }
}

The resources.php file is written/removed before each test case (setUpBeforeClass and tearDownAfterClass):

use Symfony\Component\Routing\Router;

trait SetupClassResourcesTrait
{
    public static function setUpBeforeClass(): void
    {
        file_put_contents(
            __DIR__.'/Fixtures/app/var/resources.php', 
            sprintf('<¿php return [%s];', implode(',', array_map(fn ($v) => $v.'::class', self::getResources())))
        );
    }

    public static function tearDownAfterClass(): void
    {
        file_put_contents(__DIR__.'/Fixtures/app/var/resources.php', '<¿php return [];');
        $reflectionClass = new \ReflectionClass(Router::class);
        $reflectionClass->setStaticPropertyValue('cache', []);
    }

    /**
     * @return class-string[]
     */
    abstract public static function getResources(): array;
}

Inside each test case we implement the getResources method to return our list of resources. Last but not least we need to clear the static property Router::cache as it holds our route collection in-memory. An other solution would be to run tests using --process-isolation but I know from past experiences that forking is slow no matter what and I find this an acceptable solution.

To conclude, here are time comparison on the same test: almost 10 seconds before improvements to a a few milliseconds after.

phpunit performance improvements

Applied to the whole test suite I see a huge improvement as all the tests take less then 10 seconds whereas before it’d take almost 3 minutes.

Thanks for reading, I work a lot on my private time to bring you the best optimization and developer experience on API Platform if you like my work consider donating or starring one of my projects (links below).

soyuka.me