uLangs - Universal Language Server Toolkit

A toolkit that simplifies language server development by automating grammar and semantic analysis generation. Given sample code from any domain-specific language, uLangs generates the necessary parsing and analysis components to create a LSP server implementation. The system uses LLMs to automate the traditionally manual tasks of grammar definition and semantic rule creation.

Development Status

This project represents completed Master's thesis research in automated language server development. While the core research is finished, ongoing refinement and feature additions may occur.

Master's defense presentation slides

2026 update: newer models

In October 2026 we ran the Lua case study from the thesis again with newer models. We used the same 99 snippets (test-dsl/lua-med) and the same LSP evaluation against lua-language-server on GameObject.lua. The results are in test-dsl/lua-med/solution/2026/.

Grammar

	Thesis (o3-mini)	2026 (gpt-6.1-sol, high)
Snippets that pass	99/99	99/99
LLM calls	14	1
Tokens	433,671	19,118
Cost	$0.39	≈ $0.10

The new model wrote a full Lua grammar from the first snippet. All 99 snippets passed without repair.

Semantic adapter

All adapters below use the thesis grammar. Only the model is different. The 2026 runs use DeepSeek V4.1 Flash (high) as the generator and as the judge.

	Thesis (o3, o3-mini judge)	2026 best run	2026 current code	Naive regex
Snippets that pass	92/99	99/99	99/99	–
Go to definition	8/10	7/10	6/10	–
References precision	71.9%	91.9%	100%	72.4%
References recall	97.6%	81.0%	66.7%	100%
References F1 (micro)	82.8%	86.1%	80.0%	83.9%
References F1 (macro)	74.9%	69.0%	69.2%	84.9%
Cost	$6.85	≈ $1.25	≈ $0.85	–

• 2026 best run: two runs in sequence. The second run started from the adapter of the first run, with a stricter judge prompt.
• 2026 current code: one run from scratch with the code in this repository.
• The 2026 costs are estimates from token counts at OpenRouter list prices.

What we learned

• Synthesis is now much cheaper. The grammar costs about 4 times less, and the adapter about 8 times less, for similar quality.
• The LLM judge is the weak part. A model can satisfy the judge and still give a bad LSP. Our first DeepSeek run passed 99/99, and a separate judge scored it 97/100 on average. But its F1 was only 72.7%, because it did not find fields and methods. A stricter judge prompt (clear symbol rules, numbered lines, symbols before the AST) fixed most of this.
• A rule without an example gives untested code. We told the generator to handle methods such as function A:b(), but no training snippet uses them. The adapter now declares methods, but it also declares the class name again, so 6 of 7 references to GameObject are lost.
• The adapter is more precise, but it finds fewer references. The thesis adapter finds almost all references, but also many wrong ones. The 2026 adapters find fewer references, but almost all of them are correct.
• The naive regex baseline still has the best macro F1. One evaluation file with few symbols makes macro F1 noisy, but this shows the limit of the current judge.

Code changes in this update:

• The judge is stateless: each score uses one new chat. Before, the judge replies were added to the generator chat.
• The repair loop stops trying a snippet after the maximum number of retries, and continues with the next snippets. Before, one snippet that could not be solved blocked all later repairs.
• Reasoning effort and the structured-output mode are configurable (see Configuration).

Limits: we did one run for each configuration, and we evaluated the LSP on one file. Lua is in the training data of all models. The final run used a different API provider than the other runs.

Features

• Automated LSP Generation: Creates complete language servers from code samples without manual grammar writing
• Semantic Analysis: Extracts language rules and patterns to provide intelligent code assistance
• Code Intelligence: Delivers syntax highlighting, completions, diagnostics, and symbol navigation
• Extensible Pipeline: Modular design supporting both traditional (ANTLR/Prolog) and LLM-based analysis
• Language Agnostic: Works with any textual DSL or programming language given sufficient code examples
• Universal Client: Includes ready-to-use VSCode extension that automatically integrates with any generated language server

Project Structure

• antlr-grammar-checker/: Contains grammar definitions and Java/Kotlin code for ANTLR parsing and error handling.
• lsp-server/: Implements the server-side logic for handling LSP requests.
• actions/: Contains scripts for grammar inference and query generation.
• rules/: Prolog rules and queries for semantic analysis.
• test-dsl/: Provides test files for various languages to validate grammar and server functionality.

Getting Started

Prerequisites

• Node.js: Ensure that Node.js is installed on your system.
• Java: Required for running ANTLR grammar checks. Build the checker with JDK 21 (the Kotlin compiler fails on JDK 25).
• ANTLR 4.13.2: The antlr4 command must be on your PATH (e.g. brew install antlr, then link antlr as antlr4).
• SWI-Prolog: Needed for executing Prolog queries.

Installation

1. Clone the Repository:

   git clone <repository-url>
   cd ulangs

2. Install Dependencies:

   pnpm install

3. Build antlr-grammar-checker

cd antlr-grammar-checker/
mvn clean install

Configuration

Set these variables in a .env file. Any OpenAI-compatible API works.

Variable	Description
OPENAI_COMPATIBLE_BASEURL	API base URL, e.g. https://openrouter.ai/api/v1
OPENAI_COMPATIBLE_API_KEY	API key
OPENAI_COMPATIBLE_MODEL	Generator model
OPENAI_COMPATIBLE_SO_MODEL	Judge model (optional, the generator model is the default)
OPENAI_COMPATIBLE_REASONING_EFFORT	low, medium or high (optional)
OPENAI_COMPATIBLE_STRUCTURED_OUTPUT	json_schema (default), or json_object for providers without JSON-schema support

Running the Language Server

1. Build the universal-language-server (vscode extension):

   pnpm run build:lsp

2. Launch the Server: Use the provided VSCode launch configurations to start the language server.

Command-Line Interface

• Generate Prolog Query:

  pnpm run cli query <target> <lexer> <parser> [options]

• Check Grammar:

  pnpm run cli check <directory> <extension> <lexer> <parser>

• Infer Grammar:

  pnpm run cli infer-grammar <directory> <extension> [outputDir] [options]

• Infer Semantic Adapter:

  pnpm run cli infer-adapter <directory> <extension> [outputDir] [options]