Compile hot PHP classes into the turbo extension

.claude/skills/rebase-branch/SKILL.md

@@ -85,10 +85,13 @@ Decide these before replaying (ask the user if the answer isn't obvious):
 
 ## 3. Turbo mirrors
 
-Classes carrying `#[ShadowedByTurboExtension(turboClass: ..., implementation: .../turbo-ext/src/X.cpp)]`
-have a native C++ twin. Vendored twins live in `VENDORED_PAIRS` in
+Classes carrying `#[ImplementedInTurboExtension(implementation: .../turbo-ext/src/X.cpp)]` (named
+`ShadowedByTurboExtension` on bases from before the rename) have a hand-written native C++ twin;
+classes carrying `#[CompiledIntoTurboExtension]` have one generated by `turbo-ext/bin/compile-twins.php`.
+Vendored twins live in `VENDORED_PAIRS` and `VENDORED_COMPILED` in
 `build/PHPStan/Build/TurboAttributeCollector.php` and change when their package's version in
-`composer.lock` changes. The PHP class and the C++ class must behave identically after every replayed
+`composer.lock` changes. Never resolve a conflict inside a generated file: resolve the PHP class, then
+rerun `compile-twins.php` and take its output. The PHP class and the C++ class must behave identically after every replayed
 commit. A mirror change goes into the same commit as the PHP change it mirrors, never into a trailing
 "port" commit.
 

.claude/skills/rebase-branch/rebase-plan.php

@@ -3,7 +3,7 @@
 // Read-only report that prepares a rebase of a feature branch onto a fresh base
 // tip: both sides' commits, commits the base already contains, files both sides
 // touch, the conflicts a merge of the two tips predicts, and the turbo twins
-// (#[ShadowedByTurboExtension] PHP class <-> turbo-ext .cpp) whose other half
+// (#[ImplementedInTurboExtension] / #[CompiledIntoTurboExtension] PHP class <-> turbo-ext .cpp) whose other half
 // needs a port during the replay.
 //
 // Usage: php rebase-plan.php <new-base-ref> [<feature-ref>=HEAD]
@@ -137,9 +137,17 @@ function short(string $sha): string
 function shadowMap(string $rev): array
 {
 	$map = [];
-	foreach (gitLines('grep', '-e', 'ShadowedByTurboExtension(', $rev, '--', 'src') as $line) {
+	// ShadowedByTurboExtension is the hand-written attribute's old name, still
+	// on bases from before the rename; a compiled class's .cpp follows from
+	// its short name (TurboAttributeCollector::COMPILED_DIRECTORY)
+	foreach (gitLines('grep', '-e', 'ShadowedByTurboExtension(', '-e', 'ImplementedInTurboExtension(', '-e', '#\\[CompiledIntoTurboExtension', $rev, '--', 'src') as $line) {
 		$line = substr($line, strlen($rev) + 1);
 		[$path, $content] = explode(':', $line, 2);
+		if (str_contains($content, 'CompiledIntoTurboExtension')) {
+			$class = 'PHPStan\\' . str_replace('/', '\\', substr($path, 4, -4));
+			$map[$class] = ['php' => $path, 'cpp' => 'turbo-ext/src/compiled/' . basename($path, '.php') . '.cpp'];
+			continue;
+		}
 		if (preg_match("~implementation:\\s*__DIR__\\s*\\.\\s*'([^']+)'~", $content, $m) !== 1) {
 			continue;
 		}

.github/workflows/phar.yml

@@ -238,6 +238,9 @@ jobs:
       - name: "Check the shadowed PHP and C++ implementations are in sync (method parity)"
         run: "php turbo-ext/bin/side-by-side.php"
 
+      - name: "Check the compiled shadows are generated from the current PHP twins"
+        run: "php turbo-ext/bin/compile-twins.php --check"
+
       - name: "Check the vendored php-parser matches the version the native parser engine was ported against"
         run: |
           INSTALLED="$(jq -r '.packages[] | select(.name == "nikic/php-parser") | .version' composer.lock)"

CLAUDE.md

@@ -260,11 +260,16 @@ Bugs in PHPStan's type system rarely live alone. A fix in one accessory type usu
 
 Every threshold, fan-out cap, recursion bound, or collection-size limit must be a class constant whose name ends in `_LIMIT`, declared at the top of the owning class. Inline numerics in conditionals (`if (count($this->optionalKeys) <= 10)`) are not acceptable — extract to `self::OPTIONAL_KEYS_LIMIT` (or similar) so the threshold is named, discoverable, and tunable in one place.
 
-### Classes marked with `#[ShadowedByTurboExtension]` have a native C++ mirror
+### Classes marked with `#[ImplementedInTurboExtension]` or `#[CompiledIntoTurboExtension]` have a native C++ mirror
 
-Classes carrying the `PHPStan\Turbo\ShadowedByTurboExtension` attribute are reimplemented natively by the phpstan_turbo extension; the attribute names the `.cpp` file implementing it (`build/generate-turbo-manifest.php` derives the full map into `vendor/turbo-shadowed-classes.json` on composer dump-autoload). The two implementations must behave identically — analysis output is bit-for-bit identical with the extension on or off. When editing such a class, always check whether the `.cpp` mirror in `turbo-ext/` needs the same change; if it does, port it following `turbo-ext/CLAUDE.md` (method parity check, differential smoke test, full test suite with the extension loaded). Any commit touching `turbo-ext/src/` requires the follow-up commit bumping `TurboExtensionEnabler::EXPECTED_EXTENSION_VERSION` to that commit's short SHA — run `make bump-turbo`.
+Classes carrying one of these attributes (`PHPStan\Turbo\`) are reimplemented natively by the phpstan_turbo extension (`build/generate-turbo-manifest.php` derives the full map into `vendor/turbo-shadowed-classes.json` on composer dump-autoload). The two implementations must behave identically — analysis output is bit-for-bit identical with the extension on or off. What an edit of such a class requires depends on the attribute:
 
-New introduced classes in these categories should be shadowed with a native C++ mirror in the same commit:
+* `#[ImplementedInTurboExtension(implementation: ...)]` — the C++ is written by hand in the named `.cpp`. Check whether it needs the same change; if it does, port it following `turbo-ext/CLAUDE.md` (method parity check, differential smoke test, full test suite with the extension loaded).
+* `#[CompiledIntoTurboExtension]` — the C++ is generated from the PHP class. Rerun `php turbo-ext/bin/compile-twins.php` (plus `php turbo-ext/bin/generate-declarations.php` when properties or signatures changed) and commit the regenerated files with the change; never edit them by hand. CI fails while they are stale.
+
+Any commit touching `turbo-ext/src/` requires the follow-up commit bumping `TurboExtensionEnabler::EXPECTED_EXTENSION_VERSION` to that commit's short SHA — run `make bump-turbo`.
+
+New introduced classes in these categories should be shadowed with a native C++ mirror in the same commit — compiled (`#[CompiledIntoTurboExtension]`) when the compiler handles them, hand-written otherwise:
 
 * `PHPStan\Type\Type` implementations
 * `PHPStan\Analyser\StmtHandler` and `PHPStan\Analyser\ExprHandler` implementations

build/PHPStan/Build/TurboAttributeCollector.php

@@ -4,6 +4,7 @@
 
 use FilesystemIterator;
 use JsonException;
+use Nette\Utils\Random;
 use Nette\Utils\RegexpException;
 use Nette\Utils\Strings;
 use PhpParser\Comment\Doc;
@@ -187,25 +188,34 @@
 use PHPStan\PhpDocParser\Ast\Type\ThisTypeNode;
 use PHPStan\PhpDocParser\Ast\Type\UnionTypeNode;
 use PHPStan\PhpDocParser\Printer\Printer;
+use PHPStan\Turbo\CompiledIntoTurboExtension;
+use PHPStan\Turbo\ImplementedInTurboExtension;
 use PHPStan\Turbo\ReferencedByTurboExtension;
-use PHPStan\Turbo\ShadowedByTurboExtension;
 use RecursiveDirectoryIterator;
 use RecursiveIteratorIterator;
 use ReflectionClass;
 use RuntimeException;
+use Symfony\Component\Finder\Finder;
+use Symfony\Component\Finder\Glob;
 use Throwable;
+use function array_keys;
+use function basename;
 use function class_exists;
 use function count;
 use function file_get_contents;
 use function interface_exists;
 use function json_decode;
 use function json_encode;
 use function ksort;
+use function preg_replace;
 use function realpath;
 use function sprintf;
 use function str_replace;
 use function str_starts_with;
 use function strlen;
+use function strpos;
+use function strrpos;
+use function strtolower;
 use function strtr;
 use function substr;
 use function var_export;
@@ -215,8 +225,8 @@
 use const JSON_UNESCAPED_SLASHES;
 
 /**
- * Collects the ShadowedByTurboExtension and ReferencedByTurboExtension
- * attributes with runtime reflection and renders the two generated files:
+ * Collects the ImplementedInTurboExtension, CompiledIntoTurboExtension and
+ * ReferencedByTurboExtension attributes with runtime reflection and renders the two generated files:
  * vendor/turbo-shadowed-classes.json and vendor/turbo-class-map.php. Shared
  * by build/generate-turbo-manifest.php (which writes the artifacts on every
  * autoloader dump) and turbo-ext/bin/side-by-side.php (which re-derives and
@@ -227,12 +237,39 @@
 final class TurboAttributeCollector
 {
 
-	// Shadowed classes living in vendor/ cannot carry the attribute, so
-	// their pairs are hardcoded. Class name => .cpp
+	// Shadowed classes living in vendor/ cannot carry the attributes, so
+	// they are hardcoded. Hand-written ports: class name => .cpp
+	// where turbo-ext/bin/compile-twins.php writes the compiled classes
+	public const COMPILED_DIRECTORY = 'turbo-ext/src/compiled';
+
 	private const VENDORED_PAIRS = [
 		NodeTraverser::class => 'turbo-ext/src/NodeTraverser.cpp',
 	];
 
+	// Compiled vendored classes: class name => the stem of the generated
+	// file, which the short names would make ambiguous (ReflectionClass,
+	// Standard). Their compile settings are CompiledIntoTurboExtension's
+	// defaults.
+	private const VENDORED_COMPILED = [
+		'PhpParser\\PrettyPrinterAbstract' => 'PhpParserPrettyPrinterAbstract',
+		'PhpParser\\PrettyPrinter\\Standard' => 'PhpParserStandardPrinter',
+		'PhpParser\\NameContext' => 'NameContext',
+		'PHPStan\\BetterReflection\\Identifier\\Identifier' => 'BetterReflectionIdentifier',
+		'PHPStan\\BetterReflection\\Identifier\\IdentifierType' => 'BetterReflectionIdentifierType',
+		'PHPStan\\BetterReflection\\Reflection\\Adapter\\ReflectionMethod' => 'BetterReflectionAdapterMethod',
+		'PHPStan\\BetterReflection\\Reflection\\Adapter\\ReflectionNamedType' => 'BetterReflectionAdapterNamedType',
+		'PHPStan\\BetterReflection\\Reflection\\Adapter\\ReflectionType' => 'BetterReflectionAdapterType',
+		'PHPStan\\BetterReflection\\Reflection\\Annotation\\AnnotationHelper' => 'BetterReflectionAnnotationHelper',
+		'PHPStan\\BetterReflection\\Reflection\\Attribute\\ReflectionAttributeHelper' => 'BetterReflectionAttributeHelper',
+		'PHPStan\\BetterReflection\\Reflection\\Deprecated\\DeprecatedHelper' => 'BetterReflectionDeprecatedHelper',
+		'PHPStan\\BetterReflection\\Reflection\\ReflectionAttribute' => 'BetterReflectionAttribute',
+		'PHPStan\\BetterReflection\\Reflection\\ReflectionClass' => 'BetterReflectionClass',
+		'PHPStan\\BetterReflection\\Reflection\\ReflectionNamedType' => 'BetterReflectionNamedType',
+		'PHPStan\\BetterReflection\\Reflection\\ReflectionParameter' => 'BetterReflectionParameter',
+		'PHPStan\\BetterReflection\\SourceLocator\\Located\\LocatedSource' => 'BetterReflectionLocatedSource',
+		'PHPStan\\BetterReflection\\Util\\FileHelper' => 'BetterReflectionFileHelper',
+	];
+
 	// Classes the native code references that live in vendor/ cannot carry
 	// the ReferencedByTurboExtension attribute either; their class-map
 	// entries are hardcoded. Key (pt_class_refs in turbo-ext/src/support.cpp)
@@ -270,6 +307,9 @@ final class TurboAttributeCollector
 		'constExprStringNode' => ConstExprStringNode::class,
 		'netteStrings' => Strings::class,
 		'netteRegexpException' => RegexpException::class,
+		'netteRandom' => Random::class,
+		'symfonyFinder' => Finder::class,
+		'symfonyGlob' => Glob::class,
 		'constExprFloatNode' => ConstExprFloatNode::class,
 		'thisTypeNode' => ThisTypeNode::class,
 		'objectShapeNode' => ObjectShapeNode::class,
@@ -487,15 +527,24 @@ public function __construct(string $root)
 	 * in CI by the name collision detector, so the class name follows from
 	 * the path) and merges the hardcoded vendored entries.
 	 *
+	 * Compiled classes get their generated file derived from the class name
+	 * (see CompiledIntoTurboExtension) and their compile settings returned
+	 * under `compiled`, for turbo-ext/bin/compile-twins.php.
+	 *
 	 * @return array{
-	 *     manifest: array<string, array{php: string, cpp: string, turboClass: string, final: bool, parent: string|null, interfaces: list<class-string>, vendored?: bool}>,
+	 *     manifest: array<string, array{php: string, cpp: string, turboClass: string, final: bool, parent: string|null, interfaces: list<class-string>, compiled?: true, vendored?: bool}>,
 	 *     classMap: array<string, string>,
 	 *     referenced: array<string, string>,
+	 *     compiled: array<class-string, array{stem: string, ce: string, defineCe: bool, new?: array{string, string}, dispatch?: array{function: string, enum: string, prefix: string, receiver: string, fallback?: string}}>,
 	 * }
 	 */
 	public function collect(): array
 	{
 		$pairs = self::VENDORED_PAIRS;
+		$compiledAttributes = [];
+		foreach (array_keys(self::VENDORED_COMPILED) as $className) {
+			$compiledAttributes[$className] = new CompiledIntoTurboExtension();
+		}
 		$classMap = self::VENDORED_CLASS_MAP;
 		$referenced = [];
 
@@ -523,11 +572,18 @@ public function collect(): array
 			}
 			$reflection = new ReflectionClass($className);
 
-			$attributes = $reflection->getAttributes(ShadowedByTurboExtension::class);
+			$attributes = $reflection->getAttributes(ImplementedInTurboExtension::class);
 			if (count($attributes) > 0) {
 				$attribute = $attributes[0]->newInstance();
 				$pairs[$className] = $this->relativize($attribute->implementation);
 			}
+			$compiledAttribute = $reflection->getAttributes(CompiledIntoTurboExtension::class);
+			if (count($compiledAttribute) > 0) {
+				if (isset($pairs[$className])) {
+					throw new RuntimeException(sprintf('%s is both implemented in and compiled into the turbo extension', $className));
+				}
+				$compiledAttributes[$className] = $compiledAttribute[0]->newInstance();
+			}
 
 			foreach ($reflection->getAttributes(ReferencedByTurboExtension::class) as $referencedAttribute) {
 				$key = $referencedAttribute->newInstance()->key;
@@ -539,12 +595,24 @@ public function collect(): array
 			}
 		}
 
+		$compiled = $this->compiledSettings($compiledAttributes, $pairs);
+		foreach ($compiled as $className => $settings) {
+			$pairs[$className] = self::COMPILED_DIRECTORY . '/' . $settings['stem'] . '.cpp';
+		}
+
 		ksort($pairs);
 		ksort($classMap);
 		ksort($referenced);
+		ksort($compiled);
+
+		// how many shadowed classes have each short name
+		$shortNameCounts = [];
+		foreach (array_keys($pairs) as $className) {
+			$shortName = substr($className, (int) strrpos('\\' . $className, '\\'));
+			$shortNameCounts[$shortName] = ($shortNameCounts[$shortName] ?? 0) + 1;
+		}
 
 		$manifest = [];
-		$byShortName = [];
 		foreach ($pairs as $className => $cppFile) {
 			$reflection = new ReflectionClass($className);
 			$fileName = $reflection->getFileName();
@@ -553,15 +621,12 @@ public function collect(): array
 			}
 
 			// The name the differential tests declare the native class under
-			// next to its twin, derived the way the extension derives it -
-			// so two shadowed classes sharing a short name would arrive at
-			// one native name and silently shadow each other there.
+			// next to its twin, derived the way the extension derives it
+			// (Shadow.cpp, pt_shadow_declared_name()): the short name, or —
+			// when another shadowed class has the same one — the real name
+			// past its first namespace segment, so the two stay apart.
 			$shortName = $reflection->getShortName();
-			if (isset($byShortName[$shortName])) {
-				throw new RuntimeException(sprintf('%s and %s share the short name %s, so both would be declared as PHPStanTurbo\\%s next to their twins', $byShortName[$shortName], $className, $shortName, $shortName));
-			}
-			$byShortName[$shortName] = $className;
-			$turboClass = 'PHPStanTurbo\\' . $shortName;
+			$turboClass = 'PHPStanTurbo\\' . ($shortNameCounts[$shortName] > 1 ? substr($className, (int) strpos($className, '\\') + 1) : $shortName);
 
 			$parent = $reflection->getParentClass();
 			$phpFile = $this->relativize($fileName);
@@ -575,13 +640,73 @@ public function collect(): array
 				'parent' => $parent === false ? null : $parent->getName(),
 				'interfaces' => $reflection->getInterfaceNames(),
 			];
+			if (isset($compiled[$className])) {
+				$entry['compiled'] = true;
+			}
 			if (str_starts_with($phpFile, 'vendor/')) {
 				$entry['vendored'] = true;
 			}
 			$manifest[$className] = $entry;
 		}
 
-		return ['manifest' => $manifest, 'classMap' => $classMap, 'referenced' => $referenced];
+		return ['manifest' => $manifest, 'classMap' => $classMap, 'referenced' => $referenced, 'compiled' => $compiled];
+	}
+
+	/**
+	 * The stem of each compiled class's generated file and its compile
+	 * settings with the defaults filled in. The stem is the short class
+	 * name, unless another compiled class has the same one or a hand-written
+	 * port's file already has it — then the name past its first namespace
+	 * segment, without the separators.
+	 *
+	 * @param array<class-string, CompiledIntoTurboExtension> $attributes
+	 * @param array<string, string> $implemented class => hand-written .cpp
+	 * @return array<class-string, array{stem: string, ce: string, defineCe: bool, new?: array{string, string}, dispatch?: array{function: string, enum: string, prefix: string, receiver: string, fallback?: string}}>
+	 */
+	private function compiledSettings(array $attributes, array $implemented): array
+	{
+		$takenStems = [];
+		foreach ($implemented as $cppFile) {
+			$takenStems[strtolower(basename($cppFile, '.cpp'))] = true;
+		}
+		$shortNameCounts = [];
+		foreach (array_keys($attributes) as $className) {
+			if (isset(self::VENDORED_COMPILED[$className])) {
+				continue;
+			}
+			$shortName = strtolower(substr($className, (int) strrpos('\\' . $className, '\\')));
+			$shortNameCounts[$shortName] = ($shortNameCounts[$shortName] ?? 0) + 1;
+		}
+
+		$settings = [];
+		$stems = [];
+		foreach ($attributes as $className => $attribute) {
+			$stem = self::VENDORED_COMPILED[$className] ?? null;
+			if ($stem === null) {
+				$shortName = substr($className, (int) strrpos('\\' . $className, '\\'));
+				$stem = $shortNameCounts[strtolower($shortName)] > 1 || isset($takenStems[strtolower($shortName)])
+					? str_replace('\\', '', substr($className, (int) strpos($className, '\\') + 1))
+					: $shortName;
+			}
+			if (isset($takenStems[strtolower($stem)]) || isset($stems[strtolower($stem)])) {
+				throw new RuntimeException(sprintf('The generated file of %s, %s.cpp, would overwrite another class\'s', $className, $stem));
+			}
+			$stems[strtolower($stem)] = true;
+			$entry = [
+				'stem' => $stem,
+				'ce' => $attribute->ce ?? 'pt_ce_' . strtolower((string) preg_replace('/([a-z0-9])([A-Z])/', '$1_$2', $stem)),
+				'defineCe' => $attribute->defineCe,
+			];
+			if ($attribute->new !== null) {
+				$entry['new'] = $attribute->new;
+			}
+			if ($attribute->dispatch !== null) {
+				$entry['dispatch'] = $attribute->dispatch;
+			}
+			$settings[$className] = $entry;
+		}
+
+		return $settings;
 	}
 
 	/**

build/PHPStan/Build/TurboCodegen/ClassUnit.php

@@ -0,0 +1,35 @@
+<?php declare(strict_types = 1);
+
+namespace PHPStan\Build\TurboCodegen;
+
+use PhpParser\Node\Stmt\Class_;
+use ReflectionClass;
+
+/** One class of the compile set. */
+final class ClassUnit
+{
+
+	/** @var array<string, MethodInfo> lowercase name => method */
+	public array $methods = [];
+
+	/**
+	 * @param ReflectionClass<object> $reflection
+	 */
+	public function __construct(
+		public readonly string $className,
+		public readonly string $stem,
+		public readonly string $ce,
+		public readonly ReflectionClass $reflection,
+		public readonly TypeOracle $oracle,
+		public readonly Class_ $node,
+		public readonly string $implementation,
+		public readonly bool $defineCe = false,
+		/** @var array{string, string}|null */
+		public readonly ?array $newExport = null,
+		/** @var array{function: string, enum: string, prefix: string, receiver: string, fallback?: string}|null */
+		public readonly ?array $dispatchExport = null,
+	)
+	{
+	}
+
+}

build/PHPStan/Build/TurboCodegen/CompileError.php

@@ -0,0 +1,11 @@
+<?php declare(strict_types = 1);
+
+namespace PHPStan\Build\TurboCodegen;
+
+use RuntimeException;
+
+/** A construct outside the compiled subset, or a twin the compiler cannot prove safe to lower. */
+final class CompileError extends RuntimeException
+{
+
+}