docs: add Tesla T4/Turing hardware notes (attn-impl, dtype, quantization)

[moduvoice]

Motivation

Ran the GPA-v1.5 native infer Quick Start (GPA_1.5/docs/infer.md) verbatim on a real Tesla T4 16GB (Turing, sm_75; driver 550.163.01 / CUDA 12.4, torch 2.6.0+cu124, transformers 5.13.0) and hit a crash the docs don't mention, plus found a couple of undocumented behaviors worth calling out for anyone on Turing-class GPUs (T4, similarly L4).

Changes

Doc-only additions to GPA_1.5/docs/infer.md — no code changes:

1. --attn-impl auto crashes on Turing GPUs. inference/model_loader.py::normalize_attn_impl returns flash_attention_2 for any cuda* device with no torch.cuda.get_device_capability() check and no check that flash-attn is even installed (it's commented out as an optional extra in pyproject.toml, not in requirements.txt). Following the Quick Start as-is on T4 fails at model load with:

   ImportError: FlashAttention2 has been toggled on, but it cannot be used due to the following error:
   the package for FlashAttention2 doesn't seem to be installed.
   

   Documented the fix: pass --attn-impl sdpa (already a supported CLI value, no code change) — both ASR and TTS complete normally. Also corrected the "Fast Troubleshooting Data preparation files are missing #5" section, which previously only suggested falling back to --device cpu; now it points to --attn-impl sdpa first so users don't need to give up the GPU.

2. No --dtype CLI flag. --device and --attn-impl are exposed as CLI args, but dtype is hardcoded to torch.bfloat16 on CUDA in model_loader.py. Documented that this is measurably suboptimal on T4: PyTorch's SDPA EFFICIENT_ATTENTION kernel rejects bfloat16 and silently falls back to the slower MATH path, while fp16 reaches EFFICIENT_ATTENTION directly (per-token latency was within noise in our runs, but only fp16 hits the faster kernel on this GPU class).

3. bitsandbytes int8 quantization trade-off. Not mentioned anywhere in the docs today. Measured BitsAndBytesConfig(load_in_8bit=True) reducing peak VRAM by ~43% (2317 MB → 1327 MB, reproduced twice) but making batch=1 autoregressive decode ~3.2x slower (≈36–39 ms/token → ≈121–125 ms/token, reproduced twice). Documented as a real trade-off rather than a free win.

4. VRAM headroom note. The native infer path uses a 0.6B backbone; full-pipeline peak VRAM measured ≈4.0–4.2 GB (bf16/fp16) / ≈3.0–3.1 GB (int8) — roughly 74–81% of a T4's 16GB unused. Noted this so T4/L4 users know OOM isn't a practical concern here.

Testing

• Ran GPA_1.5/infer.py --task asr and --task tts end-to-end on a Tesla T4 16GB with --attn-impl sdpa: both complete successfully (ASR transcript correct, TTS produces a valid wav).
• Reproduced the flash_attention_2 ImportError with the default --attn-impl auto on the same box (2 runs, deterministic).
• Measured peak VRAM and per-token decode time for bf16/fp16/int8 via the repo's own AutoModelForCausalLM/BitsAndBytesConfig APIs (no repo code modified), each reproduced twice.
• This PR only touches GPA_1.5/docs/infer.md; no source files changed.

[Alichua]

Hi @moduvoice ,

Thanks for testing this on a real T4. The core finding is useful: the current auto implementation selects flash_attention_2 for every CUDA device, and --attn-impl sdpa is the correct workaround on T4.

Before merging, could you please address these documentation issues?

1. Please remove L4 from the Turing scope. NVIDIA L4 is an Ada GPU (sm_89), not Turing, and the official FlashAttention 2 implementation supports Ada. The T4-specific bf16/SDPA and VRAM conclusions therefore should not be generalized to L4.

2. Please separate the two failure modes:

   • If flash-attn is not installed, --attn-impl auto can fail on any CUDA GPU.
   • On T4/sm_75, the official FlashAttention 2 implementation is unsupported even when installed, so --attn-impl sdpa should be used.

3. Please limit the benchmark and VRAM conclusions to the tested “Tesla T4 16GB + stated software versions/workload” environment. They should not be described as applying to every Turing-class GPU.

4. Minor wording issue: flash-attn is commented out from the project dependency list in pyproject.toml; [tool.uv.extra-build-dependencies] is not an installable optional extra.

Once these points are corrected, this documentation change looks good to merge.