Better CodeLlama

Fine-tuning CodeLlama 7B for enhanced Python software engineering capabilities using LoRA (Low-Rank Adaptation).

🎯 Project Overview

This project implements a comprehensive fine-tuning pipeline to enhance CodeLlama 7B for Python software engineering tasks:

• Architectural patterns and design principles (SOLID, Observer, etc.)
• Code review and refactoring best practices
• Debugging and testing strategies (pytest, mocking, etc.)
• Professional development practices and clean code principles

🔧 Requirements

System Requirements

• Python: 3.12
• GPU: RTX 4080 Super (16GB VRAM) or equivalent
• RAM: 32GB system RAM recommended
• Storage: ~50GB for models, datasets, and checkpoints
• Platform: Linux/WSL2 (tested on WSL2 Ubuntu)

🚀 Quick Start

⚠️ CRITICAL: CUDA Setup First

bitsandbytes only supports CUDA versions up to 12.8. If you have CUDA 13.0+, the installation will fail. You must install the correct CUDA version BEFORE installing Python dependencies.

1. CUDA Environment Setup (Required First!)

Step 1: Create fresh conda environment

conda create -n better-codellama python=3.12
conda activate better-codellama

Step 2: Install CUDA 12.8 specifically (if you have CUDA 13.0+)

# Download CUDA 12.8 from NVIDIA (WSL2/Linux)
wget https://developer.download.nvidia.com/compute/cuda/12.8.0/local_installers/cuda_12.8.0_560.28.03_linux.run
sudo sh cuda_12.8.0_560.28.03_linux.run

# Add to PATH (add to ~/.bashrc for persistence)
export PATH=/usr/local/cuda-12.8/bin:$PATH
export LD_LIBRARY_PATH=/usr/local/cuda-12.8/lib64:$LD_LIBRARY_PATH

Step 3: Verify CUDA installation

nvcc --version  # Should show CUDA 12.8
nvidia-smi      # Should show your GPU

2. Install PyTorch with Matching CUDA

pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu121

3. Install Project Dependencies

make dev-requirements

# Verify bitsandbytes works
python -m bitsandbytes

4. Download Data & Models

# Download datasets and models (handled automatically by training scripts)
make train  # This will download required assets on first run

🚀 Usage

All training and evaluation workflows are available through make commands:

Dependency Management

make uv-update            # Update uv package manager to latest version
make requirements         # Install production dependencies only
make dev-requirements     # Install all dependencies including development tools

Training Pipeline

# Pre-training validation (recommended before long runs)
python scripts/sanity_check.py          # Validate training setup quickly

# Using make commands (recommended)
make setup-wandb          # Configure experiment tracking
make preprocess           # Preprocess datasets for training
make train                # Complete two-phase LoRA training
make train-p1             # Phase 1: General instruction tuning
make train-p2             # Phase 2: Specialized SE training

Resuming Training from Checkpoints

If training is interrupted, you can resume from the latest checkpoint:

# Using make command (recommended)
make train-resume CHECKPOINT=checkpoints/phase1/checkpoint-2000

# Using CLI directly
python -m tuner train --resume_from_checkpoint checkpoints/phase1/checkpoint-2000

# Resume with specific phase
python -m tuner train --phase 1 --resume_from_checkpoint checkpoints/phase1/checkpoint-2000

# Resume with custom config
python -m tuner train --config configs/codellama_lora.yaml --resume_from_checkpoint checkpoints/phase1/checkpoint-2000

Finding Checkpoints: Training automatically saves checkpoints every 500 steps in the output directory:

• Phase 1: checkpoints/phase1/checkpoint-{step}/
• Phase 2: checkpoints/phase2/checkpoint-{step}/

Each checkpoint contains all necessary files to resume training, including model weights, optimizer state, and training progress.

Evaluation Pipeline

# Using make commands (recommended)
make eval-baseline        # HumanEval/MBPP baseline measurement
make eval-se              # Software engineering evaluation
make eval-all             # Comprehensive evaluation

Development & Code Quality

⚠️ CRITICAL: All code changes must pass make lint-all before acceptance

# Primary quality gate - run this before any code submission
make lint-all            # Complete quality check suite (format, lint, type-check, pylint, security)

# Individual quality checks
make format              # Auto-format code with ruff
make lint                # Run ruff linter (tuner/ strict, scripts/ warnings only)
make lint-fix            # Auto-fix linting issues + format
make type-check          # Static type analysis with pyright
make pylint              # Additional linting with pylint
make security            # Security scan with bandit
make check               # Quick validation (lint + type-check + security, no formatting)

Quality Standards: This project enforces strict code quality through multiple linters:

• ruff: Code formatting and linting (Google Python Style Guide)
• pyright: Static type checking
• pylint: Additional code analysis
• bandit: Security vulnerability scanning

The lint-all target runs all quality checks and must pass completely before any code changes can be accepted.

🎯 Goals & Results

• Baseline: CodeLlama 7B (~29% HumanEval Pass@1, ~38% MBPP Pass@1)
• Target: 35-40% HumanEval Pass@1 with enhanced software engineering capabilities
• Approach: Two-phase LoRA fine-tuning with custom SE datasets

⚠️ Troubleshooting

CUDA/bitsandbytes Compatibility Issues

Problem: RuntimeError: CUDA Setup failed despite GPU being available

Root Cause: bitsandbytes only supports CUDA versions up to 12.8. If you have CUDA 13.0+, bitsandbytes cannot find compatible precompiled libraries.

Solution (tested on WSL2):

1. Create fresh environment with specific CUDA version:

   conda create -n better-codellama-cuda12 python=3.12
   conda activate better-codellama-cuda12

2. Install CUDA 12.8 specifically (follow NVIDIA's WSL2 instructions):

   # Download and install CUDA 12.8 from NVIDIA
   wget https://developer.download.nvidia.com/compute/cuda/12.8.0/local_installers/cuda_12.8.0_560.28.03_linux.run
   sudo sh cuda_12.8.0_560.28.03_linux.run

3. Install PyTorch with matching CUDA version:

   pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu121

4. Install bitsandbytes from PyPI:

   pip install bitsandbytes

5. Verify setup:

   python -c "import bitsandbytes; print('✅ bitsandbytes working!')"

Reference: bitsandbytes CUDA compatibility chart

Other Common Issues

• Out of Memory: Reduce batch size in configs/codellama_lora.yaml
• Download Issues: Ensure sufficient disk space (~50GB) and stable internet
• Import Errors: Run uv pip install -e ".[dev]" to ensure all dependencies