Unsloth, a library for adapting open models to a task, has added a simpler route into multi-GPU training in unslothai/unsloth. Its October 26 Python release lets supported fine-tuning scripts request tensor parallelism when they start: several GPUs cooperate on the same model, sharing parts of its layers. The script can keep its usual loading and training calls.
The useful change is in the training software around the model. Someone using LoRA—a method that learns a smaller set of adjustments to existing weights—can try this arrangement without wiring the distributed-training API into their Python script. The release also connects it to optimizer state and checkpoint recovery.
The model changes its arrangement
Follow the same two GPUs down the plan. The shapes show where weights go; their areas do not measure memory.
Train together
Released training pathBoth devices work on the same training examples. Selected layer weights are split across them.
Generate afterward
Later merged codeEach GPU needs room for a full copy, gathering workspace and headroom alongside retained training allocations. The code checks before gathering.
Export a model
Later merged codethen waits
The writer can gather onto its GPU or use host RAM when the required space fits. Saving just the LoRA adapter gathers its adjustment tensors instead.
Loading has its own rule. Supported 16-bit checkpoints can load sharded slices. The 4-bit path first loads the whole model on each GPU, then shards it.
Placement follows the launch and loading code and the generation and export implementation. This is a schematic of supported paths, not a measured run.
The launch setting carries more than a device count
The October 25 baseline already offered tensor-parallel loading and training through explicit Python calls. The new work makes that existing machinery reachable from ordinary Unsloth loader and Trainer calls. It is an integration gain, rather than the invention of multi-GPU training.
A two-process launch looks like this for a supported script:
UNSLOTH_TP_SIZE=2 torchrun --nproc-per-node 2 train.pyOn one shell line, that starts two cooperating processes. Unsloth checks the requested arrangement before loading the model, applies sharding around the existing calls, and connects supported optimizers to local weight shards. Checkpoints save each process’s local training state. Recovery expects the same arrangement; changing the number or grouping of devices is refused.
These details matter because a useful training run needs to survive interruption and leave something another run can use. A flag that merely spreads computation would leave much of that work to the person writing the script.
The next cells took a second change
The shipped launch path could train, then stop when a notebook reached generation. A follow-up merged late on October 26 adds supported generation, adapter saving, full-model exports and reload handling. Our downloaded 2026.10.9 package contains the launch implementation but lacks this follow-up. The inspected GitHub beta release has the same boundary.
The follow-up returns to the ordinary generation machinery by gathering a complete model copy on every participating GPU. It retains the original sharded parameters so training can resume. That explains why a training arrangement can fit while its next generation call cannot: the output step asks for additional memory in a different shape.
Saving takes another route. Adapter weights gather at the writer; full-model exports gather the model there, with a host-memory option. Every process must join the collective model-save call even though only one writes. A script that calls that save only inside a rank-zero branch can leave it waiting for the missing participants.
A usable workflow is the test
The checkpoint tests compare resumed training with uninterrupted state. The output tests check gathered-model generation, saved artifacts and collective refusal when memory is insufficient. The maintainer also reports short two-T4 GPU trials, including matching exports from the same trained adapter.
We inspected the code, test definitions and released package; we did not reproduce training. The reported trials do not establish general speed, cost or fine-tuning quality. They also disclose a Qwen tokenizer padding-side difference. Full fine-tuning and evaluation during training remain outside this supported route; combining tensor parallelism with native FSDP2 still does not gain the follow-up’s generation and export path.
For someone with a supported LoRA script and several GPUs, Unsloth now removes a layer of custom distributed-training integration. The next useful demonstration is a released, pinned workflow that loads, trains, resumes and produces a reusable model—measuring memory at each phase and checking the result on the task it was trained to do.
