[NPU] Migrate Megatron training to Core 0.18 with MegatronAdaptor and TENPU

[addsubmuldiv]

PR type

• Bug Fix
• New Feature
• Document Updates
• More Models or Datasets Support

Summary

Migrate the Ascend NPU Megatron training integration from MindSpeed to Megatron Core 0.18 + MegatronAdaptor + TransformerEngineNPU (TENPU), including the NPU compatibility needed by the existing Megatron-FSDP checkpoint flow.

Companion bridge change: modelscope/mcore-bridge#189

Changes

Initialization and attention

• Import megatron_adaptor before MCore/bridge initialization on NPU, with an installation hint for a missing adaptor module.
• Remove the MindSpeed initialization/repatch chain, its internal exports, and npu_patch/mindspeed.py.
• Use the standard attention_backend contract instead of injecting MindSpeed's use_flash_attn. Retain task/padding-free guards and Qwen3.5's 2D attention-mask handling.

Optimizer and checkpoint compatibility

• Delegate optimizer loading to MCore's native load_state_dict; remove the MindSpeed loader detection, manual reshard loading, and tensor-step restoration. Keep save-time NPU step normalization.
• Add an NPU Megatron-FSDP AdamW adapter because the tested TENPU FusedAdam rejects DTensor parameters. Delegate updates to native PyTorch AdamW and reject unsupported optimizer options rather than silently changing their semantics.
• Complete empty local optimizer shards for DCP and refresh their step values after resume, preventing inconsistent checkpoint state on subsequent saves.
• Complete FSDP parameter synchronization and restore raw parameters before the subsequent bridge HF export, avoiding exposure of empty local shards.
• Explicitly reject NPU FSDP virtual pipeline parallelism.

Scope

• Update Chinese and English NPU setup documentation for the new stack.
• Keep the generic NPU checkpoint-reader and pipeline P2P compatibility paths.
• Do not change GPU installation requirements or add production CLI switches. No validation scripts or unit-test files are included in this PR.

Validation

The submitted Swift and bridge snapshots were rechecked on 8×Ascend 910B3, Python 3.12, CANN 9.1.0, PyTorch 2.10.0, torch_npu 2.10.0.post2, Transformers 5.12.1, and vLLM 0.23. Backend snapshots: MCore ba7b5ebce12a (0.18.0), MegatronAdaptor 2a164d4548e2, and TENPU 0a389a677586.

Check	Result / boundary
Qwen3-0.6B full, TP2/PP2, continuous training and two successive resumes	Matching logged loss and model tensors; optimizer comparison within approximately 1e-9. CPU global RNG was not identical.
Qwen3-0.6B full, fully-reshardable TP2/DP4 → TP4/DP2	Save, load, and continued training exited successfully; functional reshard validation, not exact cross-topology state equivalence.
Four-layer Qwen3.5 GDN full/LoRA, TP2/PP2/CP2, packing	Training/checkpoint smoke completed.
Four-layer Qwen3.5 MoE full/LoRA, TP2/PP2/EP2	Training/checkpoint smoke completed; full used no_save_optim, so MoE optimizer resume is not established by this check.
Dense Megatron-FSDP full/LoRA, DP2, successive resumes	Matching logged loss for full and for LoRA with merge_lora=false.
Dense HF export and reload	Maximum logits difference 7.82e-5.
GKD LoRA, TP2/PP2, top-k teacher logits	Two-step colocated smoke with sleep/offload completed.
GRPO LoRA, TP2, vLLM adapter loading enabled	Three cycles completed with an additional dataset-column fix outside this PR; not an unmodified-PR end-to-end pass.

Known limitations / follow-ups

• BF16 LoRA in-place merge/unmerge can perturb same-process training after saving. This PR does not fix that bridge behavior; use merge_lora=false for resume-equivalence checks.
• FSDP+PP is not claimed as supported by these results. VPP is explicitly rejected.
• Full server-mode RL, numerical comparison of post-update vLLM synchronized tensors, and GDN/MoE HF logits alignment are not established by this validation.
• Checks are bounded 2–4-step runs with reduced-layer GDN/MoE assets, not long-run stability or full-model capacity tests. GPU regression testing remains pending.