v.ast, v.checker: fix infinite loops and segfaults in generic instantiation and cyclic aliases

[tailsmails]

What this PR does and why

This PR fixes multiple infinite loop hangs (causing 100% CPU usage) and segmentation faults (caused by unbounded recursion stack overflow) during type checking of recursive generics, deeply nested types, and cyclic type aliases.

The fixes introduce standard recursion guards and cutoff limits (similar to how limits are guarded in other compilers) to prevent compiler hangs and crashes, while ensuring complex, valid user code still compiles smoothly.

---

Changes and Fixes

1. v.checker (Generic Function Loop Limit):

   • Reduce generic_fn_postprocess_iterations_cutoff_limit from 1,000,000 to a safer 50. This prevents permanent compiler hangs and high CPU utilization when resolving circular/recursive generic functions.

2. v.ast (Generic Struct Instantiation Depth):

   • Introduce generic_inst_depth_cutoff_limit = 256 in unwrap_generic_type_ex_with_depth to stop unbounded recursion and prevent stack overflows (resulting in segfaults) for deeply nested generic structs.

3. v.ast (Type Alias Loop Limit):

   • Bound the for loop in fully_unaliased_type using alias_unwrap_depth_cutoff_limit = 100 to prevent infinite loops and hangs on circular type aliases.

4. v.checker (Generic Sum Type Validation):

   • Strip generic brackets ([ and <) from the symbol name in sum_type_decl before comparing it to the node name, ensuring that generic sum types cannot hold themselves and bypass circular reference verification.

---

Tests

Add a regression test file verifying that deeply nested (yet valid) generic structures, nested generic functions, and type alias chains still compile and evaluate correctly under the new limits:

// vlib/v/checker/recursion_limits_test.v

fn generic_level_1[T](val T) T {
	return val
}

fn generic_level_2[T](val T) T {
	return generic_level_1[T](val)
}

fn generic_level_3[T](val T) T {
	return generic_level_2[T](val)
}

fn generic_level_4[T](val T) T {
	return generic_level_3[T](val)
}

fn test_valid_nested_generic_functions() {
	res_int := generic_level_4[int](42)
	assert res_int == 42

	res_str := generic_level_4[string]('Vlang')
	assert res_str == 'Vlang'
}

struct Box[T] {
pub:
	val T
}

fn test_valid_nested_generic_structs() {
	b1 := Box[int]{
		val: 100
	}
	b2 := Box[Box[int]]{
		val: b1
	}
	b3 := Box[Box[Box[int]]]{
		val: b2
	}
	b4 := Box[Box[Box[Box[int]]]]{
		val: b3
	}

	assert b4.val.val.val.val == 100
}

type Alias1 = int
type Alias2 = Alias1
type Alias3 = Alias2
type Alias4 = Alias3
type Alias5 = Alias4

fn test_valid_type_alias_chain() {
	mut num := Alias5(10)
	assert num == Alias5(10)

	num += Alias5(20)
	assert num == Alias5(30)
}

struct Some[T] {
pub:
	val T
}

struct None {}

type MyOption[T] = None | Some[T]
type ComplexResult[T, E] = E | Some[T]

fn test_valid_generic_sum_types() {
	opt_some := MyOption[int](Some[int]{
		val: 99
	})
	if opt_some is Some[int] {
		assert opt_some.val == 99
	} else {
		assert false
	}

	opt_none := MyOption[string](None{})
	if opt_none is None {
		assert true
	} else {
		assert false
	}

	res := ComplexResult[int, string](Some[int]{
		val: 500
	})
	if res is Some[int] {
		assert res.val == 500
	} else {
		assert false
	}
}

[tailsmails]

bug1.v (Recursive generic functions infinite loop)

fn foo[T]() { foo[[]T]() }
fn main() { foo[int]() }

bug2.v (Nested generic structs stack overflow / segfault)

struct Box[T] { Box[Box[T]] }
fn main() { b := Box[int]{} }

bug3.v (Nested generic methods infinite loop)

struct Box[T] { val T }
fn (b Box[T]) foo() { Box[Box[T]]{}.foo() }
fn main() { Box[int]{}.foo() }

bug4.v (Circular reference bypass in generic sum types infinite loop)

type MySum[T] = T | MySum[MySum[T]]
fn main() { mut x := MySum[int]{} }

---

Expected Outputs with these patches (No hangs, no segfaults):

For bug1.v and bug3.v (Graceful loop termination):

test.v:1:1: error: generic function post processing reached the cutoff limit of 50 iterations, probably due to an infinite generic instantiation loop
    1 | fn foo[T]() { foo[[]T]() }
      | ~~~~~~~~~~~
    2 | fn main() { foo[int]() }

For bug2.v (Controlled limit panic instead of Segmentation Fault):

V panic: generic instantiation depth limit exceeded

For bug4.v (Consistent circular sum-type verification):

test.v:1:21: error: sum type cannot hold itself
    1 | type MySum[T] = T | MySum[MySum[T]]
      |                     ~~~~~~~~~~~~~~~

[tailsmails]

I have also found a 5th related segmentation fault in convert_generic_type with this recursive alias pattern, and I will address it in a separate PR soon..

[tailsmails]

Update: Found another related segfault while testing this branch.

It happens when resolving recursive generic function pointer aliases like this:

type Func[T] = fn (f Func[[]T]) T

fn main() {
	_ := Func[int](none)
}

On master, this crashes with a segmentation fault in v__ast__Table_convert_generic_type or unwrap_generic_type_ex_with_depth.

What was going wrong:
The mutual recursion between convert_generic_type and unwrap_generic_type_ex_with_depth bypassed the local depth_guard arrays. Since they call intermediate helpers (which reset the local depth arguments back to 0 or []string{}), the local limit checks were never triggered.

The fix:
To solve this cleanly, I added two dynamic counters to the Table struct (convert_generic_type_depth and unwrap_generic_type_depth). Tracking the recursion depth directly on the mutable Table works perfectly across mutual recursion boundaries. The compiler now safely catches the infinite recursion and panics with generic instantiation depth limit exceeded instead of crashing.

[medvednikov]

@codex review

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: ab9edcefea

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Replace the total generic-instantiation cap with cycle detection

When a valid program specializes one generic function for 257 distinct concrete type lists, the map entry reaches length 257 and every subsequent call to register_fn_concrete_types—even a duplicate that would otherwise return false—panics here. The number of finite specializations is not evidence of recursive expansion, and generated serializers or dispatch code can legitimately exceed this threshold, so this turns valid compilation into a compiler crash; guard actual recursive growth instead of the accumulated specialization count.

Useful? React with 👍 / 👎.

[chatgpt-codex-connector]

Allow finite generic instance names longer than 1 KiB

A finite generic instantiation can exceed 1,024 bytes without containing a cycle—for example, a generated generic with many concrete parameters or nested qualified type names—yet it now unconditionally panics here. Because V does not otherwise impose this identifier-length restriction, valid source that previously compiled now crashes; recursion should be bounded by the instantiation graph rather than the rendered name length.

Useful? React with 👍 / 👎.

[chatgpt-codex-connector]

Preserve finite generic chains beyond 50 rechecks

A valid chain of more than roughly 50 generic functions can require more than 50 passes when declarations are ordered from the innermost function to the outermost and only the outermost function is initially instantiated: each pass discovers the concrete type for a function that was already visited earlier in that pass. This fixed cutoff therefore emits an infinite-instantiation error for a finite chain that previously converged; use repeated-state/no-progress detection or a limit that does not reject ordinary finite propagation.

Useful? React with 👍 / 👎.

[tailsmails]

I tweaked the constants a bit.

• Kept the depth limits at 256 and 100. I tried bumping them (e.g., to 512),
  but it just causes a segfault before V can even step in and panic.
• Bumped the size/count limits (generic_fn_inst to 4096, name_len to 8192, and iterations to 100k).
  Honestly, it's highly unlikely most developers will ever hit these numbers in real-world projects, but this safely covers the theoretical edge cases mentioned by Codex without causing any harm.

[tailsmails]

Raise generic post-process limit to 256

The old limit of 50 was too tight for complex nested generics, leading to false-positive loop errors on valid code. At the same time, we can't set it too high because actual infinite loops will hang the compiler and spike CPU usage.
Setting the cutoff to 256 is a safe middle ground. It prevents false positives for deep generic chains without adding any complex runtime overhead, and ensures the compiler still fails fast if a real loop happens.

[tailsmails]

Raise generic post-process limit to 256

The old limit of 50 was too tight for complex nested generics, leading to false-positive loop errors on valid code. At the same time, we can't set it too high because actual infinite loops will hang the compiler and spike CPU usage. Setting the cutoff to 256 is a safe middle ground. It prevents false positives for deep generic chains without adding any complex runtime overhead, and ensures the compiler still fails fast if a real loop happens.

I think maybe 128 is safer than 256