Image Editing

Instruction-driven editing rewrites photos from plain commands like make it sunset, changing pixels through language instead of masks and brushes.

Instruction editing blends photo conditioning at strength 1.5 with command conditioning at strength 7.5, so one guidance step pulls the noise estimate from 0.1 to 2.5 toward sunset

Why Does This Exist?

Masks are tedious and imprecise for global edits. Asking for sunset over a lake should not require painting the sky pixel by pixel, yet classical inpainting demands exactly that kind of selection. Language already describes the change, so the editor should accept the sentence.

Instruction editing trains on triplets of original photo, written command, and edited result, generated at scale from paired captioning plus image models. At inference the photo and the command condition the denoiser jointly, and two guidance strengths balance obedience to each. The mask-based machinery behind selections is taught separately under image editing and inpainting, and the generative backbone it edits with belongs to image generation.

Think of It Like This

A retoucher who takes dictation

A photographer dictates to a retoucher: same portrait, golden-hour light. The retoucher holds the original under one hand and the instruction in mind, changing only what the sentence requires. No masks, no selections, just understanding of both inputs at once.

The two guidance strengths are how loudly each input speaks. The sitting fails on contradictory dictation, change nothing but change everything, and the retoucher must pick a middle the photographer never wanted.

How It Actually Works

The denoiser sees the noisy latent plus the encoded original photo concatenated as extra channels, and the command text through cross-attention. Sampling uses dual classifier-free guidance with separate scales for image and text:

\hat{\epsilon} = \epsilon_\varnothing + s_I (\epsilon_I - \epsilon_\varnothing) + s_T (\epsilon_T - \epsilon_\varnothing)

Here $\epsilon_\varnothing$ is the unconditional estimate, $\epsilon_I$ is the photo-conditioned estimate, and $\epsilon_T$ is the fully conditioned estimate. Typical values are $s_I = 1.5$ holding the photo and $s_T = 7.5$ pushing the command. Training triplets come from prompting a language model for edit commands plus a text-to-image model for before-after pairs, filtered for consistency.

Worked example

Daylight lake, command make it sunset. Plug one scalar step through the formula with $e_\varnothing = 0.1$ , $e_I = 0.2$ , $e_T = 0.4$ : the image term contributes $1.5 \times (0.2 - 0.1) = 0.15$ and the text term contributes $7.5 \times (0.4 - 0.1) = 2.25$ , so $\hat{e} = 0.1 + 0.15 + 2.25 = 2.5$ . The command pull dominates the photo pull fifteen to one, which is why skies burn orange while the dock geometry holds. Drop $s_I$ toward $1.0$ and the dock drifts into a generic pier; push $s_T$ past $10$ and the scene oversaturates into kitsch. The two-knob balance is the entire craft of the method.

Code

Dual guidance blending, the two-knob update inside each step. Pure standard library.

1def dual_guidance(e_uncond: float, e_img: float, e_full: float, s_i: float = 1.5, s_t: float = 7.5) -> float:2    """Combine photo obedience with command obedience."""3    return e_uncond + s_i * (e_img - e_uncond) + s_t * (e_full - e_uncond)4
5
6print(round(dual_guidance(0.1, 0.2, 0.4), 3))  # -> 2.5 strong command pull

Watch Out For

Identity drift on faces at high text strength

Commands like make him smile at $s_T = 7.5$ reshape identity along with expression: jawlines shift, moles vanish. Face edits need lower text strength plus identity checks, because likeness lives in details the command never mentions.

Pin faces with $s_T$ near $4$ and verify with face similarity, not eyeballing. What reads as a good smile may be a different person.

Training-triplet artefacts inherited wholesale

Synthetic triplets carry their generator's biases: same skies, same skin smoothing. The editor learns the triplet factory's house style on top of real edits.

Filter triplets for diversity and mix real edit pairs where available. Audit outputs for repeated stylistic tics across unrelated commands.

The Quick Version

Instruction editing rewrites photos from commands, no masks required.
The denoiser conditions on the photo plus the command with dual guidance scales.
Typical balance is $s_I = 1.5$ for the photo and $s_T = 7.5$ for the command.
Our worked step pulls $0.1$ to $2.5$ under strong command pressure.
Faces drift at high text strength, and synthetic triplets inject house styles.