One model, one image reference channel, producing real detail on two different tasks —
but failing in completely different ways: for video super-resolution the problem is instability, for black-and-white film colourisation it is the price paid.
This page puts every input of each group (reference video / reference image / text prompt) side by side with the output, comparable frame by frame.
Super-resolution is not ineffective — the two reference channels are dragging each other down. With the three-arm ablation run on all ten UDM10 clips, the division of labour is very clean:
the image reference (super-resolved first frame) carries quality — the R3 arm, which gets only that, reaches MUSIQ 56–78 on 10/10 clips, the highest anywhere on this page;
the video reference (the LR video) carries motion — remove it and motion collapses below 30% on 8/10 clips.
But the video reference also makes the model reproduce the input blur as if it were content: on the two failure cases udm10_008 / 009,
removing it lifts MUSIQ from 14.0 / 15.1 to 64.8 / 56.0.
So what is really missing is a channel that carries motion only, not appearance. Counting “quality improves + motion retention 85–110%” together,
5 of the ten clips pass today. udm10_002 is also the only reference-frame control on this site:
replacing a single hallucinated reference with a regenerated one gives PSNR +5.54 dB and motion 35% → 92% —
the reliability of the reference frame is itself one of the failure sources.
The difference in the colourisation section comes from whether the target information exists in the input at all: the high frequencies super-resolution must restore are already hidden inside the input blur,
so the model easily reproduces the blur as content; the colours colourisation must invent simply do not exist in a black-and-white input, so the “copy the input” shortcut does not apply,
and Ref2VA’s bias toward generation turns from a defect into exactly what the task needs.
| Item | Notes |
|---|---|
| Built-in limit of the reference video | _resolve_reference_video_shape force-shrinks any video reference to short side 768 / area ≤768×1344, so it cannot possibly carry high-resolution information; measured, its role is to supply motion (motion retention 31% → 79%) |
| The reference-image channel | An image reference is kept at short side 2048 and is encoded twice — by both Qwen3-VL (semantics) and the video VAE (spatial tokens) |
| Evaluating the colourisation task | There is no colour GT, but the luma channel is GT by construction: PSNR-Y is computed against gray(bicubic input). Two de-confounding steps are applied — downsampling the output back to the original resolution to remove the super-resolution penalty, and matching per-frame luma mean/std to separate out tone shifts |
| Temporal metrics | adj = mean absolute difference between adjacent frames, drift = deviation of the last frame from the first. No-reference metrics alone give a directionally wrong ranking: one variant was best in the field on NIQE/MUSIQ while actually being a still image — the reference frame copied 32 times |
| Sample size | The super-resolution conclusions rest on all 10 UDM10 clips; colourisation has only 2, so it can only be stated as a hypothesis to be verified. This project has already had three extrapolations from n≤2 overturned |
Weights, code and dependency versions for every experiment. The 2026-05 batch of SparkVSR results originally left no checkpoint record;
its provenance was only confirmed here by a deterministic-mode frame-by-frame comparison (47.7 dB) — every new artefact since then ships with a run_config.json.
| Component | Version / provenance | Used for |
|---|---|---|
| MiniMax-H3 | MiniMax/MiniMax-H3 (ModelScope, 269 GB local)two partitions: Ref2VA/ and FL2VA/, each with DiT 33.12B + text encoder 25.75B + video VAE 2.60B + audio VAE 0.151Bfull bf16, no quantisation |
every generated result on this site |
| H3 · Ref2VA | The DiT differs from FL2VA; text encoder / VAE / processor are shard-for-shard identical to FL2VA (measured — contrary to what the upstream docs say) | all arms of super-resolution / colourisation / style / faces |
| H3 · FL2VA | As above; keyframe_indices only accepts 0 or −1 | the region-generation and first/last-keyframe tasks of the combined benchmark |
| DiffSynth-Studio | upstream 3f3df52 (2026-08-25) + two patches from this project, +56/−10 lines in total:① denoising_strength opens the v2v entry point (originally blocked by a one-line training gate)② inpaint_mask_video adds a spatial-mask entry point — H3 officially only has temporal retake, no in/outpainting interfaceboth were verified bit-identical to the original when =None |
inference framework |
| nano-banana-pro | fal-ai/nano-banana-pro/edit (fal.ai, paid API) |
generates every image reference frame. Its outputs are not reproducible (stochastic sampling) |
| SparkVSR | checkpoints/sparkvsr-s2/ckpt-500-sft (S2)Provenance verified: re-run in deterministic mode with --ref_mode no_ref and compared frame by frame against the 2026-05 batch, PSNR 47.7 dB (the magnitude expected for the same ckpt; a different ckpt would land below 30 dB) |
dedicated-VSR-model baseline |
| Qwen3-VL-4B-Instruct | local weights | generates scene descriptions and per-face <Subject N> descriptions |
| Metric models | pyiqa 0.1.15 (LPIPS / DISTS / NIQE / MUSIQ / CLIPIQA) · CLIP ViT-B/32 (style adherence) · facexlib 0.3.0 RetinaFace-R50 (face detection) | evaluation |
| Runtime | Python 3.11.15 · torch 2.12.1+cu130 · transformers 5.15.1 (generation) torch 2.5.0+cu124 (evaluation env) | |
| Hardware | a single NVIDIA H200 (143 GB). Peak VRAM 63–77 GB, no multi-GPU and no quantisation needed; neither pipeline supports multi-GPU inference, extra GPUs are only used to run different experiments in parallel | |
| Sampling parameters | Unless noted otherwise: 50 steps · flow_shift 12 · seed 0 · cfg 1.0. Constraints: height/width must be multiples of 32, num_frames must be 17n+5 |