How much improvement can be achieved on reasoning heavy tasks (games/puzzles) through transfer learning?
I set out to try and answer this question through training an LLM on poker reasoning traces, then evaluating its performance on Connect 4 vs the base model. The results weren't exactly what I expected (no consistent clear performance transfer learning), data for the 3 LoRAs I created are below (same seeds).
The 1k Poker LoRA boosted the RedHatAI/gemma-3-12b-it-quantized.w4a16 model's performance against the base version so I scaled the data up to 10k and found that the new LoRA consistently lowered win rate against the base model. The amount of performance degradation on the 10k sample LoRA also isn't consistent across the two runs I made.
Data is on hf at nacknuck/synthetic-poker-traces
I'd expect to get more consistent/better performance if I created data for more games/puzzles and also used a larger model. I tracked the experiments and also found that a large amount of the Connect 4 games played were unique. The LoRA also clearly raises invalid moves (fail to parse/place a piece in a full col) compared to the base model across all matches it's involved in.
Below results show win rate of the fine tuned and base models as both p0 (first) and p1 across 1k and 1.5k trials. Making one of these tables takes ~24 hours and generating the 10k data samples once takes ~30 hours.
1k samples, lr 2e-4, bsz 8, rank 64, 1k trials each
| Matchup (P0 v P1) | P0 win rate | P1 win rate | Avg invalid moves (per match) |
|---|---|---|---|
| Base v Base | 62.6% | 37.3% | 0.33 |
| Base v SFT | 58.3% (-4.3) | 41.6% (+4.3) | 0.48 |
| SFT v Base | 63.8% (+1.2) | 36.1% (-1.2) | 0.56 |
| SFT v SFT | 60.7% (-1.9) | 39.1% (+1.8) | 1.17 |
10k samples, lr 2e-4, bsz 8, rank 64, 1.5k trials each
| Matchup (P0 v P1) | P0 win rate | P1 win rate | Avg invalid moves (per match) |
|---|---|---|---|
| Base v Base | 60.6% | 39.1% | 0.36 |
| Base v SFT | 61.1% (+0.5) | 38.7% (-0.4) | 0.73 |
| SFT v Base | 55.7% (-4.9) | 44.1% (+5.0) | 0.91 |
| SFT v SFT | 62.7% (+2.1) | 37.0% (-2.1) | 2.31 |
10k samples, lr 1e-4, bsz 8, rank 64, 1.5k trials each
| Matchup (P0 v P1) | P0 win rate | P1 win rate | Avg invalid moves (per match) |
|---|---|---|---|
| Base v Base | 61.1% | 38.5% | 0.38 |
| Base v SFT | 62.9% (+1.8) | 36.9% (-1.6) | 0.51 |
| SFT v Base | 59.1% (-2.0) | 40.7% (+2.2) | 0.57 |
| SFT v SFT | 65.4% (+4.3) | 34.2% (-4.3) | 1.12 |
Looking at the results makes me wonder whether the 1k data sample was randomly good, or that training on a very small amount of reasoning data (relative to the base SFT) in a new domain generally helps model performance. I'd be surprised if this was true because 1k samples is a very small amount (in this case ~1M tokens) compared to a domain specific SFT which commonly uses 100M+ to billions of tokens.
TL:DR - LoRA SFT raises invalid moves made. Small amount of data boosts performance; upping it 10x lowers performance. Need more retrains (different seeds) to tell whether 1k data consistently boosts performance and 10k degrades.
Train and eval scripts are targetted for a single 4090 for throughput and vram.
Go into src
Run python -m multipuzzle.cli play connect4 --players human,human to test out Connect 4
Run an LM match once w/ turns logged: python -m multipuzzle.cli eval connect4 --players lm:gemma-3-27b-qat,lm:gemma-3-27b-qat --trials 1 --dump-lm runs/lm_debug
LM names come from configs/players
Go into src
python -m multipuzzle.cli eval connect4 --players random,random --trials 1000
python -m multipuzzle.cli eval poker --players lm:gemma-3-27b-qat,lm:gemma-3-27b-qat --trials 500
To replot results, run from src/ (sometimes long poker runs crash the vLLM)
python -m multipuzzle.cli replot runs/poker_20260608_073053
Use the stock vLLM image vllm/vllm-openai:latest and pass in one of the configs under serve/:
docker run --rm -it --gpus all -p 8000:8000 \
-v ~/.cache/huggingface:/root/.cache/huggingface \
-v $(pwd)/serve:/config \
-e VLLM_ATTENTION_BACKEND=FLASHINFER -e VLLM_USE_V1=1 \
vllm/vllm-openai:latest \
--config /config/gemma-3-27b-qat-w4a16.yaml
Swap gemma-3-27b-qat-w4a16.yaml for glm-4.7-flash.yaml to serve GLM instead.
Evaluations use w4a16 over bits and bytes config - dequant fused into matmul, faster then dequant -> matmul.
python serve/test_request.py to check its up
Build the uv venv here w/ uv sync
Afterwards, activate it and run the following to create the SFT dataset
-
Setup a vLLM instance, look in serve/*.yaml for more info
-
python scripts/poker_sft_1_transform.py --limit 12000 --concurrency 3 --out runs/poker_sft_data_creation/pokerbench_dev_12k
- creates two jsonl files in given outdir
- this uses
leon-se/gemma-3-27b-it-qat-W4A16-G128by default, if your vLLM serves something else, update this arg - goal of this script is to take HF dataset of optimal Poker move, and convert it into our template
python scripts/poker_sft_2_reason.py --n 20
I got the dataset from https://huggingface.co/datasets/RZ412/PokerBench
I also used the reasoning template from ToolPoker (https://arxiv.org/pdf/2602.00528) as inspiration, although my template heavily differs from theirs due to no solver being used and I do behavioral cloning whereas their goal was tool integration
See src/multipuzzle/poker_sft/train/train.py for more info, example run command below.
You'll need to generate the data first, which expects a vLLM instance to be up.
python3 -m multipuzzle.poker_sft.train.train \
runs/pokerbench_dev_1k_reasoned/reasoned.jsonl
uv sync installs base deps only. To pull in optional groups:
uv sync --extra datagen— datasets + langgraph (SFT data generation)uv sync --extra train— transformers/torch/trl/peft/etc. (SFT training)uv sync --extra dev— pytestuv sync --all-extras— everything
On Windows/CUDA, torch may resolve to a CPU build; if so install it from the
PyTorch index, e.g. uv pip install torch --index-url https://download.pytorch.org/whl/cu124.
If two players play random legal moves in Connect 4, P1 has a ~55% win chance compared to P2's ~44% w/ less than 1% draw chance
- calculated using MC estimation at n=1e6
LoRA w/o Regret (Thinking Machines Sept 29 2025)
- Amazing starter blog post on how to configure LoRAs + what to expect
- thought LoRA was A.T@B before reading this, it's B@A
- empirically shows RL works well on LoRAs, maybe due to desired training effect using same assumption as small batch of SFT data (new task is weight perturbation in lower dimension)
Tool Poker (ICLR 2026)
- shows that SFT alone (they term behavior cloning) and RL afterwards isn't enough to make a poker LLM play SOTA
- helped me reevaluate what I considered good for my pipeline, and focus on designing data that shows clear reasoning process rather than optimal play
- shows a path forward for reasoning heavy domains where LLMs are integrated w/ specific tools and aligned to use them rather than internalize complex reasoning processes
Below links are for understanding/playing poker recommended to me from a friend. Not directly useful for the synthetic data pipeline construction in this project, but I'd still like to document them.
Easy Game Volume II - Andrew Seidman
100 Hands - Peter Clarke
The Grinder's Manual - Peter Clarke