Skip to content

Fix generated-class source attribution in MAL, LAL and OAL #13971

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
wu-sheng merged 1 commit into from
Aug 9, 2026
Merged

Fix generated-class source attribution in MAL, LAL and OAL #13971

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions docs/en/changes/changes.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -255,6 +255,12 @@
admin-host only" entry above for the public REST retirement.

#### OAP Server
* Fix MAL generated-class line attribution, which conflated three different coordinate spaces.
- The `_L<n>_` segment in a generated class name was the rules-list index, not a line, so every rule file's first rule was reported as `L0`. It now carries the rule's real YAML line (and the `filter:` line for filter classes).
- The `SourceFile` attribute named a file that was never written — it used the metric name while the generated source file is named after the class — so IDE source-attach could never resolve a MAL stack frame regardless of the line. It now equals the generated `.java` file.
- `LineNumberTable` held statement ordinals matching neither the YAML nor the generated source; it now points at real statements in the generated `.java`.
- Closure companion classes now get their own `.java` and their own line. Javassist always stamps a companion's `SourceFile` as `<class>.java`, but that file was never written, so every closure frame named a source file that did not exist. Each companion is a separate class file — Javassist cannot emit lambdas, so one rule with two closures produces three classes — and each now carries its own source file plus a single `LineNumberTable` entry at its SAM signature. One entry rather than a per-statement table is deliberate: the statement scanner detects stores to a result slot, which a closure body never makes, so a per-statement table would be invented rather than derived.
- The same `SourceFile` defect is fixed in LAL and OAL, which stamped the rule's YAML provenance (`(execution-basic.yaml:304)auto-layer-not-set.java`) while writing `execution_basic_L304_auto_layer_not_set.java`, so no generated frame in either DSL could ever resolve. Their `.java` sidecars are also now written as UTF-8 with an ASCII header instead of through a platform-default `FileWriter`, which is the pairing that produced a `MalformedInputException` for MAL on a non-UTF-8 JVM. Hierarchy writes no sidecar at all, so its `SourceFile` still names nothing — tracked separately, since adding one is a feature rather than a fix.
* Support runtime rule hot-update and DSL debugging for the `meter-analyzer-config` catalog, bringing native meter (`MeterReportService`) rules to parity with `otel-rules`.
- Meter rules now load through the same `Rules`/`Rule` pipeline `otel-rules` uses, so they participate in `RuleSetMerger`, are recorded in `StaticRuleRegistry`, support the optional `layerDefinitions` block, and generate source-named expression classes instead of falling back to `MalExpr_<N>`.
- `MeterProcessService` now implements `MalConverterRegistry` and publishes debug holders at boot, so a meter rule can be added / overridden / inactivated at runtime, and attached to a DSL debug session, without restarting the OAP.
Expand Down
59 changes: 43 additions & 16 deletions docs/en/operation/dynamic-code-generation-debugging.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -98,17 +98,39 @@ When a runtime error occurs inside a generated class, the JVM prints a stack tra
at <package>.<ClassName>.<method>(SourceFile:LineNumber)
```

The `SourceFile` attribute encodes the original DSL configuration file in parentheses:
The `.java` sidecar is written ON DEMAND only — when `SW_DYNAMIC_CLASS_ENGINE_DEBUG` is set.
Javassist compiles from an in-memory string, so a default deployment produces classes with no
sidecar on disk at all.

When a sidecar IS written, `SourceFile` names it, so an IDE can resolve the frame to real source:

```
(<dsl_source_file>:<rule_line_or_index>)<GeneratedClassName>.java
<GeneratedClassName>.java
```

When none is written, the frame's usefulness depends on the DSL. MAL and LAL encode the rule's line
in the CLASS NAME (`vm_L38_cpu_total`, `execution_basic_L304_…`), so the location survives regardless.
An OAL class is named after its metric (`ServiceRespTimeMetrics`) and carries no line elsewhere, so
its `SourceFile` keeps a `(core.oal:20)` prefix outside debug mode — and always, for the
metrics-builder and Hierarchy classes, which write no sidecar in any mode. A dispatcher is shared by
every metric of a scope, so it names no line at all rather than borrow one metric's.

For MAL and LAL it no longer embeds the DSL location in parentheses. That form described where a rule came from but
addressed no file on disk, so source-attach could never resolve a frame. The DSL location is instead
carried by the generated class NAME (`vm_L38_cpu_total` is line 38 of `vm.yaml`).

Line numbers differ by DSL. MAL attaches a `LineNumberTable` mapping each statement to its line in
the sidecar, and one entry at the SAM signature for closure companions. LAL and OAL attach no line
table: theirs numbered statements 1, 2, 3 rather than addressing sidecar lines, and once `SourceFile`
started resolving, those ordinals resolved too — to the sidecar's comment header. An absent table
makes the JVM report an unknown line, which is honest; real lines there require the sidecar geometry
to be derived and tested, which is not yet done.

### Example Stack Trace

```
java.lang.ArithmeticException: / by zero
at ...metrics.generated.ServiceRespTimeMetrics.id0((core.oal:20)ServiceRespTimeMetrics.java:3)
at ...metrics.generated.ServiceRespTimeMetrics.id0(ServiceRespTimeMetrics.java:3)
at ...worker.MetricsStreamProcessor.in(MetricsStreamProcessor.java:...)
...
```
Expand All @@ -122,17 +144,18 @@ Reading this:

| DSL | SourceFile Example | Generated Class Name | How to Read |
|-----|-------------------|---------------------|-------------|
| OAL | `(core.oal:20)ServiceRespTimeMetrics.java` | `ServiceRespTimeMetrics` | OAL file `core.oal`, line 20 defines this metric |
| MAL | `(vm.yaml:25)cpu_total_percentage.java` | `vm_L25_cpu_total_percentage` | YAML file `vm.yaml`, line 25, rule `cpu_total_percentage` |
| MAL filter | `(vm.yaml:20)filter.java` | `vm_L20_filter` | YAML file `vm.yaml`, line 20, filter expression |
| LAL | `(default.yaml:3)default.java` | `default_L3_default` | YAML file `default.yaml`, line 3, rule `default` |
| Hierarchy | `(hierarchy-definition.yml:88)name.java` | `hierarchy_definition_L88_name` | Rule `name` at line 88 in `hierarchy-definition.yml` |
| OAL | `ServiceRespTimeMetrics.java` | `ServiceRespTimeMetrics` | OAL file `core.oal`, line 20 defines this metric |
| MAL | `vm_L25_cpu_total_percentage.java` | `vm_L25_cpu_total_percentage` | YAML file `vm.yaml`, line 25, rule `cpu_total_percentage`. The `SourceFile` equals the generated class file so an IDE can resolve the frame; the YAML provenance is carried by the class name. |
| MAL filter | `vm_L20_filter.java` | `vm_L20_filter` | YAML file `vm.yaml`, line 20, filter expression |
| LAL | `default.java` | `default_L3_default` | YAML file `default.yaml`, line 3, rule `default` |
| Hierarchy | `name.java` | `hierarchy_definition_L88_name` | Rule `name` at line 88 in `hierarchy-definition.yml` |

**Notes:**
- The class name pattern is `{yamlFileName}_L{lineNo}_{ruleName}` for all DSLs (except OAL).
The yaml file name and line number from `yamlSource` are combined with the rule name or `filter`.
- The number after `:` in the SourceFile prefix is the line number in the YAML file where the rule is defined (for MAL in production, this may be a 0-based rule index instead of a line number).
- When source information is unavailable, the class name falls back to `MalExpr_<N>` / `LalExpr_<N>` / `HierarchyRule_<N>` and the SourceFile to just `ClassName.java` without the parenthesized prefix.
- The number after `_L` in the class name is the 1-based line in the YAML file where the rule (or the `filter:`) is defined.
- For MAL, the line number in a stack frame (`SourceFile:LineNumber`) is a line in the GENERATED `.java` file, not in the YAML. The two are different coordinate spaces: use the class name for YAML provenance, and the frame line to locate the generated statement. The per-stage YAML line is reported separately by the [DSL debugging API](../setup/backend/admin-api/dsl-debugging-mal.md) as `sourceLine`.
- When source information is unavailable, the class name falls back to `MalExpr_<N>` / `LalExpr_<N>` / `HierarchyRule_<N>`. For MAL, a line that was expected but could not be resolved renders as `_Lunknown_` rather than being silently omitted, so the failure stays visible.

### Mapping Back to DSL Source

Expand All @@ -142,13 +165,17 @@ Reading this:
- `...log.analyzer.v2.compiler.rt.*` → LAL (class name: `{yamlName}_L{lineNo}_{ruleName}` or `LalExpr_<N>`)
- `...hierarchy.rule.rt.*` → Hierarchy (class name: `{yamlName}_L{lineNo}_{ruleName}` or `HierarchyRule_<N>`)

2. **Find the source file** from the parenthesized prefix in the SourceFile attribute (e.g., `core.oal`, `vm.yaml`),
or from the class name prefix (e.g., `vm_L25_...` → `vm.yaml`). These files are in the `config/` directory.
2. **Find the source file** from the class name prefix (e.g., `vm_L25_...` → `vm.yaml`). These files are
in the `config/` directory. OAL classes are named after the metric rather than the file, so their
`SourceFile` keeps a `(core.oal:20)` prefix when no sidecar was written — that is the only place an
OAL frame carries its rule location.

3. **Locate the rule** using the line number from the class name (`_L25_`) or the SourceFile prefix (`:25`).
3. **Locate the rule** using the line number in the class name (`_L25_`), or the `(file:line)` prefix
for OAL.

4. **Use the statement number** (after the last `:`) as a rough indicator of which operation within the
generated method failed. Dump the class (see above) and use `javap -v` to see the exact mapping.
4. **Use the line number after the last `:`** to find the statement inside the generated `.java`. MAL
frames carry a real line there; LAL and OAL frames carry none, so they identify the method but not
the statement. Dump the class (see above) and use `javap -v` to inspect the mapping.

## Generating All DSL Classes Offline

Expand Down Expand Up @@ -207,7 +234,7 @@ MAL and LAL errors during metric processing are caught and logged per-expression
```
ERROR o.a.s.o.m.a.v.MetricConvert - Analyze Analyzer{...} error
java.lang.NullPointerException
at ...vm_L25_cpu_total_percentage.run((vm.yaml:25)cpu_total_percentage.java:5)
at ...vm_L25_cpu_total_percentage.run(cpu_total_percentage.java:5)
```

This tells you: the error is in `vm.yaml`, line 25, metric `cpu_total_percentage`,
Expand Down
4 changes: 3 additions & 1 deletion docs/en/setup/backend/admin-api/dsl-debugging-mal.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,9 @@ nodes[]
sourceText — verbatim DSL fragment for this probe
continueOn — true (MAL captures kept-only; see overview)
payload — SampleFamily.toJson() at this probe stage
sourceLine — omitted for MAL (no per-line mapping)
written on. Per STAGE, not per rule: a stage from the
file-level `expSuffix:` reports the suffix's line, not
the rule's. Omitted when it could not be resolved.
```

Sample types and the probes that emit them:
Expand Down
99 changes: 21 additions & 78 deletions ...r/src/main/java/org/apache/skywalking/oap/log/analyzer/v2/compiler/LALClassGenerator.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,9 @@
import java.io.File;
import java.io.FileOutputStream;
import java.io.IOException;
import java.io.OutputStreamWriter;
import java.io.Writer;
import java.nio.charset.StandardCharsets;
import java.util.ArrayList;
import java.util.Collections;
import java.util.HashMap;
Expand Down Expand Up @@ -373,64 +376,6 @@ private void writeClassFile(final CtClass ctClass) {
}
}

/**
* Adds a {@code LineNumberTable} attribute by scanning bytecode for
* store instructions to local variable slots &ge; {@code firstResultSlot}.
*/
private void addLineNumberTable(final javassist.CtMethod method,
final int firstResultSlot) {
try {
final javassist.bytecode.MethodInfo mi = method.getMethodInfo();
final javassist.bytecode.CodeAttribute code = mi.getCodeAttribute();
if (code == null) {
return;
}

final List<int[]> entries = new ArrayList<>();
int line = 1;
boolean nextIsNewLine = true;

final javassist.bytecode.CodeIterator ci = code.iterator();
while (ci.hasNext()) {
final int pc = ci.next();
if (nextIsNewLine) {
entries.add(new int[]{pc, line++});
nextIsNewLine = false;
}
final int op = ci.byteAt(pc) & 0xFF;
int slot = -1;
if (op >= 59 && op <= 78) {
slot = (op - 59) % 4;
} else if (op >= 54 && op <= 58) {
slot = ci.byteAt(pc + 1) & 0xFF;
}
if (slot >= firstResultSlot) {
nextIsNewLine = true;
}
}

if (entries.isEmpty()) {
return;
}

final javassist.bytecode.ConstPool cp = mi.getConstPool();
final byte[] info = new byte[2 + entries.size() * 4];
info[0] = (byte) (entries.size() >> 8);
info[1] = (byte) entries.size();
for (int i = 0; i < entries.size(); i++) {
final int off = 2 + i * 4;
info[off] = (byte) (entries.get(i)[0] >> 8);
info[off + 1] = (byte) entries.get(i)[0];
info[off + 2] = (byte) (entries.get(i)[1] >> 8);
info[off + 3] = (byte) entries.get(i)[1];
}
code.getAttributes().add(
new javassist.bytecode.AttributeInfo(cp, "LineNumberTable", info));
} catch (Exception e) {
log.warn("Failed to add LineNumberTable: {}", e.getMessage());
}
}

private static void setSourceFile(final CtClass ctClass, final String name) {
try {
final javassist.bytecode.ClassFile cf = ctClass.getClassFile();
Expand All @@ -445,19 +390,6 @@ private static void setSourceFile(final CtClass ctClass, final String name) {
}
}

/**
* Builds the SourceFile name for a generated class. When YAML source info
* is available, produces {@code "default(ruleName.java)"};
* otherwise falls back to {@code "ruleName.java"}.
*/
private String formatSourceFileName(final String ruleName) {
final String classFile = ruleName + ".java";
if (yamlSource != null) {
return "(" + yamlSource + ")" + classFile;
}
return classFile;
}

private void addLocalVariableTable(final javassist.CtMethod method,
final String className,
final String[][] vars) {
Expand Down Expand Up @@ -569,7 +501,6 @@ public LalExpression compileFromModel(final LALScriptModel model) throws Excepti
final javassist.CtMethod ctMethod = CtNewMethod.make(pm.source, ctClass);
ctClass.addMethod(ctMethod);
addLocalVariableTable(ctMethod, className, pm.lvtVars);
addLineNumberTable(ctMethod, pm.lvtVars.length + 1); // after this + params
}

final javassist.CtMethod execMethod = CtNewMethod.make(executeBody, ctClass);
Expand All @@ -590,10 +521,19 @@ public LalExpression compileFromModel(final LALScriptModel model) throws Excepti
execLvt.addAll(genCtx.localVarLvtVars);
addLocalVariableTable(execMethod, className,
execLvt.toArray(new String[0][]));
addLineNumberTable(execMethod, 3); // slot 0=this, 1=filterSpec, 2=ctx

setSourceFile(ctClass, formatSourceFileName(
classNameHint != null ? classNameHint : className));
// No LineNumberTable. It numbered bytecode boundaries 1, 2, 3 -- statement ORDINALS,
// not lines in any file. That was harmless only while SourceFile named a file that did
// not exist, so no frame could resolve and act on the wrong number. Now that SourceFile
// names the sidecar actually written, an ordinal RESOLVES: line 1 of that file is the
// synthetic comment header, which an IDE presents as the frame's source. Emitting real
// lines needs the sidecar's geometry (preamble + package + class decl + preceding private
// methods, which vary per rule) the way MAL derives it; until that is computed and
// tested, an absent attribute reports an unknown line, which is honest.

// Must equal the .java written below: a SourceFile naming a file that does not exist
// reads as correct to an IDE right up until it fails to open it. YAML provenance
// belongs in the debug API, not in this attribute.
setSourceFile(ctClass, ctClass.getSimpleName() + ".java");

writeClassFile(ctClass);
writeSourceFile(ctClass, genCtx, executeBody);
Expand Down Expand Up @@ -623,7 +563,7 @@ private void writeSourceFile(final CtClass ctClass,
}
final File file = new File(classOutputDir, ctClass.getSimpleName() + ".java");
final StringBuilder sb = new StringBuilder();
sb.append("// Synthetic source — Javassist compile input for ")
sb.append("// Synthetic source - Javassist compile input for ")
.append(ctClass.getSimpleName()).append("\n")
.append("// Written when SW_DYNAMIC_CLASS_ENGINE_DEBUG is on; used by IDE\n")
.append("// source-attach to render the bytecode without FernFlower.\n\n");
Expand All @@ -648,7 +588,10 @@ private void writeSourceFile(final CtClass ctClass,
}
sb.append(" ").append(executeBody.replace("\n", "\n ")).append("\n");
sb.append("}\n");
try (java.io.FileWriter w = new java.io.FileWriter(file)) {
// UTF-8 explicitly, NOT the platform default: a FileWriter encodes in whatever charset
// the JVM happens to default to, while every reader of these files opens them as UTF-8.
try (Writer w = new OutputStreamWriter(
new FileOutputStream(file), StandardCharsets.UTF_8)) {
w.write(sb.toString());
} catch (Exception e) {
log.warn("Failed to write source file {}: {}", file, e.getMessage());
Expand Down
Loading
Loading