arXiv 2611.25457

CANDORContrastive Self-Denoising Rewards for
Label-Free Text-to-Image Post-Training

A label-free reward for Flow-GRPO that scores how much the prompt helps a text-to-image model denoise its own sample, not how well the model denoises it.

Junhyeok Lee1Byung-Hoon Kim2Kyu Sung Choi1

1Seoul National University2EverEx

Five samples of Stable Diffusion 3.5 Medium after CANDOR post-training
The same five prompts from the base model
Base model+ CANDOR
a photo of an elephant below a surfboard
a photo of two trains
“Abandon All Hope”
“Bestsellers of 2023”
“No Liquids Allowed”
Scroll sideways to see all five prompts.
Base model before and after CANDOR post-training. Drag to compare. GenEval (left two) and OCR text-rendering (right three) test prompts with shared initial noise; OCR labels: text to render. Hand-picked to show differences.
0.60 / 0.63
Concordance of CANDOR with correctness within same-prompt groups, GenEval / OCR. Absolute self-confidence: 0.48 / 0.51 (chance 0.5).
0.63 → 0.70
OCR text-rendering score of SD3.5 Medium with CANDOR as the only reward. ImageReward 0.91 → 1.09.
1.12 → 1.28
ImageReward of the public Flow-GRPO (GenEval) adapter after further post-training with CANDOR. PickScore 22.53 → 22.73.
0 labels
No annotations and no reward model: the empty-prompt branch that classifier-free guidance already has gives the contrast.
Abstract

Post-training text-to-image generators with reinforcement learning usually relies on external reward models that need annotated preferences or auxiliary evaluators and are prone to reward hacking. We show that the generator's own denoising accuracy, used by a recent intrinsic reward, is a poor proxy for prompt fidelity. Within a same-prompt group, this absolute self-confidence is dominated by image complexity and ranks correct samples above incorrect ones near chance. We introduce CANDOR (Contrastive Alignment from Null-referenced Denoising lOg-Ratio), the log-ratio of unconditional to conditional denoising errors under shared noise probes, which measures how much the prompt helps the model denoise its own sample. It cancels prompt-independent difficulty that scales both errors and ranks the better of two same-prompt samples higher in 60% and 63% of pairs on compositional generation and text rendering. As the only Flow-GRPO reward for a flow-matching model, CANDOR improves text rendering, aesthetic and human-preference scores over the base model and a prior absolute self-confidence reward, with text-rendering and preference gains that stronger classifier-free guidance does not reproduce. On public Flow-GRPO adapters trained on external rewards, CANDOR further raises PickScore and ImageReward, so it can complement those rewards.

Method

Read self-denoising contrastively

A sample can be easy to denoise because it is simple, such as a flat background or a texture-free object, not because it follows the prompt. Image complexity makes a sample harder to denoise with or without the prompt, whereas prompt fidelity should only help the prompt-conditioned prediction.

CANDOR therefore rewards the contrast between the two predictions on the same re-noised latent and noise:

D=logMSE(∅)−logMSE(c)
MSE(∅) and MSE(c) are the errors of the model's noise estimate without and with the prompt. The log-ratio cancels any difficulty factor that scales both errors, so D is large when the prompt explains the sample.
Overview of CANDOR: rollout and re-noising, the contrastive denoising log-ratio, and GRPO
Overview of CANDOR. Rollout & re-noising: policy samples re-noised with antithetic probes. Contrastive denoising log-ratio: Di,t of empty-prompt to prompt errors (CFG off). GRPO: per-sample rewards (timestep mean of z-scored Di,t) group-normalized into advantages. Error maps: latent error of this sample at t = 0.60, averaged over probes and channels, one color scale; other thumbnails: illustrative pixel-space renderings.
1

Rollout & re-noising

Sample a group of images per prompt, then re-noise each final latent at several timesteps with antithetic noise pairs (+ε, −ε).

2

Contrastive log-ratio

Predict the noise with and without the prompt on the same latent and noise. D is the log-ratio of the two errors.

3

GRPO update

Average the z-scored D over timesteps into one reward per sample, normalize rewards within the group into advantages, and update the policy with Flow-GRPO.

Analysis

Absolute self-confidence tracks simplicity

GRPO centers rewards within each prompt's group, so only their relative values drive the update. Within these groups, absolute self-confidence S is dominated by image complexity and ranks correct samples above incorrect ones near chance.

  • S follows complexity (r = −0.75 and −0.74), and within a group it scores samples almost exactly like the unconditional confidence (correlation 0.999).
  • D is nearly uncorrelated with complexity and with S, and is the only score clearly above chance, also with complexity regressed out.
  • At the training CFG scale of 4.5, on training prompts, D reaches 0.615 on both benchmarks, against 0.550 and 0.522 for S.
ScoreGenEvalOCR
Within-group correlation
Absolute S vs. complexity−0.75−0.74
CANDOR D vs. complexity0.120.11
CANDOR D vs. absolute S−0.11−0.09
Within-group concordance with correctness ↑
Complexity (log PNG size)0.530.50
Absolute S0.480.51
CANDOR D0.600.63
Diagnostic
CANDOR D (w/o complexity)0.600.63

Offline analysis of intrinsic rewards. Base-model test-prompt rollouts at CFG 7.0. Concordance: fraction of same-prompt pairs ranked correctly (0.5 = chance); w/o complexity: log PNG size regressed out. Bold: best; underline: second best in the concordance block.

Results

Better text rendering, aesthetics and preference

As the only reward, CANDOR improves text rendering, aesthetic and human-preference scores over the base model and over SOLACE, a prior absolute self-confidence reward, on external metrics that the reward never sees. On public Flow-GRPO adapters trained on external rewards, it further raises PickScore and ImageReward.

Task-specificAlignment & aestheticsHuman preference
ModelGenEval ↑OCR ↑CLIPScore ↑Aesthetic ↑PickScore ↑HPSv2.1 ↑ImageReward ↑UnifiedReward ↑
Reference models (not ranked)
SDXL0.540.130.2785.6022.300.2650.552.84
SD3.5-L0.700.700.2905.5322.790.2890.973.25
Base model: SD3.5 Medium
Base0.690.630.2845.3822.370.2830.913.11
+ SOLACE0.690.630.2855.3522.360.2760.863.09
+ CANDOR0.710.700.2855.4522.560.2971.093.20
On top of the public Flow-GRPO (GenEval) adapter
+ Flow-GRPO (GenEval)0.940.710.2945.3322.530.2801.123.27
+ SOLACE0.920.680.2925.3322.470.2701.003.25
+ CANDOR0.920.730.3015.5222.730.3051.283.20
On top of the public Flow-GRPO (OCR) adapter
+ Flow-GRPO (OCR)0.700.920.2905.3122.450.2840.993.16
+ SOLACE0.690.890.2895.3022.380.2770.923.13
+ CANDOR0.740.920.2935.2422.550.2931.133.11
On top of the public Flow-GRPO (PickScore) adapter
+ Flow-GRPO (PickScore)0.750.730.2805.9123.470.3211.313.41
+ SOLACE0.720.680.2875.5422.700.2811.043.25
+ CANDOR0.740.760.2815.9723.550.3221.363.43

Post-training results. Measured in this work. GenEval: official task-averaged; OCR: Flow-GRPO test set; other metrics: DrawBench. Flow-GRPO (X): public adapter for X. + Y: the model above, post-trained with reward Y. Bold: best; underline: second best per block; ties share the mark. SOLACE: absolute self-confidence reward, evaluated from its public checkpoint.

Citation

BibTeX

lee2026candor
@article{lee2026candor,
  title   = {{CANDOR}: Contrastive Self-Denoising Rewards for Label-Free Text-to-Image Post-Training},
  author  = {Lee, Junhyeok and Kim, Byung-Hoon and Choi, Kyu Sung},
  journal = {arXiv preprint arXiv:2611.25457},
  year    = {2026}
}