Custom LoRA for HiDream-O1-Image: Train visual enhancements without VAE

A groundbreaking model with no training tools

HiDream-O1-Image has rapidly climbed the Artificial Analysis T2I Arena leaderboard, securing a top ten position among open-weight text-to-image systems. Yet despite its performance, the model shipped exclusively with inference code, leaving creators without standard tools for customization. Unlike conventional models that rely on VAE and UNet architectures, HiDream-O1-Image uses a Pixel-level Unified Transformer (UiT) without separate encoders—making existing LoRA trainers incompatible.

This challenge led to the development of one of the first publicly documented LoRA training methods for HiDream-O1-Image. The resulting LoRA enhances rendering quality, lighting, and stylization across anime and semi-realistic subjects using a simple trigger phrase. No prior LoRA exists that matches this general-purpose capability, making this workflow a pioneering effort in adapting the model for creative workflows.

Why standard LoRA trainers fail on HiDream-O1-Image

Most LoRA training tools are designed for models with UNet or Diffusion Transformer architectures, where a VAE compresses images into latent space and a text encoder processes prompts separately. These tools patch LoRA adapters into attention layers of the UNet or DiT, freeze other components, and handle the training pipeline automatically.

HiDream-O1-Image breaks this pattern entirely. It uses a Qwen3-VL-based transformer that processes raw pixel patches directly—no VAE, no latent space, and no separate text encoder. Images are divided into 32x32 pixel patches, each represented as a 3072-dimensional vector. The model predicts clean pixel values at each denoising step, not latent noise. This unified architecture eliminates the standard pathways that trainers expect, rendering tools like kohya_ss, ai-toolkit, and SimpleTuner ineffective.

Reverse-engineering the training loop from inference code

With no official training pipeline available, the only path forward was to reconstruct the training process from the model’s inference code. The inference pipeline revealed critical insights: the model predicts the clean image x0 at each denoising step, and the noise schedule uses a scale factor of 8.0 when generating noisy inputs.

Key observations from the inference logic:

• The model receives a timestep expressed as 1 minus the noise level (t = 1 - σ).
• The noisy input is constructed as z_t = (1 - σ) x0 + σ (8.0 * ε), where ε is standard Gaussian noise scaled to the pixel range.
• The output x_pred represents the model’s estimate of the clean image x0 at the image token positions.

These findings directly inform the training loss: mean squared error between the predicted clean image and the ground truth. The loss is computed only on image token positions, ignoring text tokens. This approach aligns with the model’s training objective and ensures that LoRA adapters learn to improve visual quality without altering language understanding.

Building a minimal 150-line LoRA trainer for HiDream-O1-Image

The solution leverages Hugging Face’s PEFT library, which natively supports adding LoRA adapters to transformer models. Since the backbone is a standard Qwen3-VL transformer, LoRA injection is straightforward—only the training loop requires custom implementation.

The training recipe follows a clean and minimal structure:

import torch
from transformers import AutoModelForConditionalGeneration
from peft import LoraConfig, get_peft_model

# Load the model in inference mode (no gradients)
model = AutoModelForConditionalGeneration.from_pretrained(
    "HiDream-ai/HiDream-O1-Image",
    torch_dtype=torch.float16,
    device_map="auto",
    low_cpu_mem_usage=True
)

# Configure LoRA for linear layers in the transformer
lora_config = LoraConfig(
    r=64,
    lora_alpha=128,
    target_modules=["q_proj", "k_proj", "v_proj", "o_proj"],
    lora_dropout=0.05,
    bias="none",
    task_type="CAUSAL_LM"
)

# Apply LoRA to the model
model = get_peft_model(model, lora_config)

# Training loop (simplified excerpt)
for batch in dataloader:
    x0 = batch["image_pixels"].to(device)  # Clean image patches in [-1, 1]
    prompt_ids = batch["input_ids"].to(device)
    
    sigma = torch.rand(1).item() * (1.0 - 1e-4) + 1e-4
    eps = torch.randn_like(x0)
    z_t = (1.0 - sigma) * x0 + sigma * (8.0 * eps)  # Noise scale 8.0
    t = torch.tensor([1.0 - sigma], device=device)
    
    # Forward pass with noisy input and timestep
    outputs = model(
        input_ids=prompt_ids,
        vinputs=z_t,
        timestep=t,
        token_types=torch.ones_like(prompt_ids)
    )
    
    # Compute loss only on image token positions
    x_pred = outputs.x_pred[0, vinput_mask[0]]
    loss = torch.nn.functional.mse_loss(x_pred, x0)
    
    loss.backward()
    optimizer.step()

The full trainer spans approximately 150 lines of code and handles patch-based image inputs, noise scheduling, and token masking. Early experiments revealed subtle gotchas—such as incorrect noise scaling and token type mismatches—that could derail training. Addressing these required careful alignment with the model’s inference behavior.

What this LoRA delivers—and what it doesn’t

The resulting LoRA is designed as a general-purpose visual enhancement adapter for anime and semi-realistic styles. It improves rendering quality, lighting consistency, and stylization without targeting specific characters or single artistic styles. A trigger phrase activates the enhancement, making it versatile for creative workflows.

Important limitations include:

• The LoRA is not a character-specific adapter, so it won’t replicate faces or styles with high fidelity.
• It does not function as a model distillation artifact—it learns visual enhancements from external image datasets.
• Training requires careful tuning of learning rate, batch size, and noise schedule to avoid overfitting or instability.

The path forward for HiDream-O1-Image customization

HiDream-O1-Image represents a new frontier in open-weight image generation, but its architectural uniqueness has left creators without off-the-shelf tools. This first public LoRA training workflow opens the door to broader experimentation by demonstrating that custom adapters are feasible with modest engineering effort.

As the community continues to explore HiDream-O1-Image, we can expect more general-purpose and specialized LoRAs to emerge. The release of training data, datasets, and refined tooling will further democratize access to high-quality customization. For now, this LoRA stands as a proof of concept—and a foundation for future innovation in unified transformer-based image generation.