# Ask for the shirt, get the vest: extracting one object from a photograph of several

*How the Inspired pipeline learned that "extract the shirt on a white
background" is not a specification.*

---

There is a class of bug that only exists because the tool is good. Image editing
models got competent enough to isolate an object from a photograph, which made
the obvious instruction work often enough to ship — and then fail in a way that
is almost impossible to see in aggregate.

Here is the failure. A photograph of someone in a navy knitted vest over a light
blue shirt. Ask for the shirt:

```
Extract the shirt from the image on a white background.
```

What comes back is a shirt. On white. Centred. Well lit. Still wearing the vest.

The model did what was asked. It found the shirt, it isolated *the shirt region*
— and the shirt region, as photographed, contains a vest. Nothing in the
instruction said the vest was the problem, so nothing removed it. The result
passes every automated check you would think to write: one garment, white
background, no person. It is simply the wrong garment.

Every object photograph has this problem. A sofa with a cushion on it, a lamp
behind a chair, a serum bottle in front of a jar. The Inspired pipeline cuts up
to six objects out of one photograph, one at a time, which means it hits the
problem six times per read. What follows is what the instruction turned into,
and why each part of it is there.

## The reader has to describe the occlusion, not just the object

The first thing that changes is upstream. If the extraction step is going to
delete the vest, something has to tell it there is a vest.

So the reading pass does not return a list of names and descriptions. Per
object, it returns the relationships:

- `layer` — which layer this garment is, from `outer` down to `footwear` and
  `accessory`. (For non-fashion subjects the same field becomes `placement`:
  `foreground`, `on-top-of`, `inside`, `standalone`.)
- `covered_by` — the names of the other listed objects that physically hide part
  of this one.
- `covers` — the names of the objects this one hides.
- `visible_parts` — which parts the camera can actually see: *collar, upper
  chest, both sleeves, left cuff, hem below the vest*.
- `hidden_parts` — which parts are hidden and would have to be inferred: *torso
  front, buttons, placket, pockets*.
- `disambiguator` — one noun phrase that points at this object and nothing else
  in the frame: *the light blue shirt whose collar and long sleeves show from
  under the navy knitted vest*.

Two fields are not about occlusion at all, and turn out to matter as much:

- `fabric_finish` (or `material_finish`) — *soft lightweight cotton, visibly
  creased and unpressed, matte*.
- `silhouette` (or `form`) — *cropped, boxy with dropped shoulders, no collar
  and no lapel, straight ribbed hem, full-length sleeves with ribbed cuffs*.

The schema is strict and every property is required, so the reader cannot
quietly skip the awkward ones. An empty list is expressible — a photograph with
nothing shoppable in it returns zero objects rather than an invented garment —
but a *partial* object is not.

### Two rules that came out of the reading pass

**Commit to one reading.** The instruction forbids hedged alternatives:
"collarless or very small collar" is not allowed, because a hedge handed to an
image model is reproduced as the wrong option roughly half the time. Decide
which it is.

**Describe only this object's own edges.** When a vest frames a shirt in a V,
that V is the shape of the vest's opening, not the shirt's neckline. The reader
is told to look at the shirt's own collar edge and buttons instead. Without that
clause, layered garments inherit the shape of whatever is on top of them.

## Lead with the deletion

Now the extraction instruction. The single most load-bearing discovery:

> Open with the edit to perform, not with the description of the object.

Every description-first phrasing lost to this one. "A light blue shirt with a
button placket, isolated on white, without the vest" leaves the vest in place a
lot of the time. This does not:

```
Remove the navy knitted vest and keep only the shirt worn underneath it.
```

The occluder is named as a thing to delete, in the first clause, before anything
else. The description comes after, as support.

The mirror case is the outer layer. Asking for the vest has the opposite
failure: the shirt underneath survives, showing through the armholes and above
the neckline. So the instruction inverts:

```
Keep only the vest and remove the shirt worn underneath it.
```

And when a top layer hangs open, one more clause is required, because "remove
the shirt" alone leaves the model free to fill the gap with fabric:

> Show the vest on its own with nothing worn under it: through its neckline,
> front opening and any gap, only the plain white background is visible — no
> shirt, no collar, no cuff, no second fabric of any kind.

### Recovering an occluder the reader forgot to declare

`covered_by` and `covers` are two views of one relationship, and models fill
them asymmetrically: the vest reliably reports that it `covers` the shirt, while
the shirt sometimes reports an empty `covered_by`. That asymmetry would silently
drop the opening clause — the sentence that does all the work.

So the occluder list is reconstructed. If an object declares no `covered_by`,
the pipeline scans its siblings for anything whose `covers` names it, and uses
that instead. One derived list, two chances to populate it.

## Rebuild what was hidden, or get a hole

Deleting the vest leaves a vest-shaped absence in the middle of the shirt. Left
alone, models resolve that absence in one of three ways: a literal hole, a
smooth guess that does not match the visible fabric, or — most often — a
*different* shirt that happens to be complete.

The instruction names the absence and how to fill it, using the reader's
`hidden_parts` and `visible_parts`:

> Where the navy knitted vest hid part of the shirt (torso front, buttons,
> placket, pockets), continue the same fabric, colour and weave so the whole
> garment is shown, complete and symmetrical, and keep those rebuilt areas as
> plain as the visible ones.

"As plain as the visible ones" is doing specific work. A rebuilt torso is an
invitation to invent: a chest pocket, a contrast placket, a logo. Plainness has
to be requested.

## The two upgrades nobody asks for

Image models make things nicer. It is what they were trained to do, and for
product extraction it is a defect.

**They upgrade the finish.** A soft, creased, unpressed cotton shirt comes back
crisp, pressed and faintly glossy — a nicer shirt than the one in the
photograph, and therefore the wrong one to search for. The counter is the
reader's `fabric_finish`, stated positively and then defended:

> Its fabric is soft lightweight cotton, visibly creased and unpressed, matte.
> Reproduce that exact weight, texture and finish across the whole garment,
> copied from the collar, upper chest, both sleeves, left cuff, hem below the
> vest. Do not make it crisper, stiffer, glossier, newer or more formal than the
> photo: keep the same drape, softness and creasing, and do not press or restyle
> it.

**They upgrade the shape.** A boxy cropped knit drifts toward a longer, more
tailored, more flattering version of itself. "Keep the silhouette as
photographed" carries no information a model can act on. The named cut does:

> Its cut and shape: cropped, boxy with dropped shoulders, no collar and no
> lapel, straight ribbed hem, full-length sleeves with ribbed cuffs. Match that
> shape exactly — the same length, the same shoulder line, the same collar or
> neckline, the same sleeve and hem shape. Do not lengthen or shorten it, do not
> add a lapel, a pointed collar, a stand collar or a hood, do not make it
> oversized or gathered, and do not curve a straight hem.

Naming what must be *absent* — no lapel, no hood — matters as much as naming
what is present. Absences are what get filled in.

## Hardware is a ceiling, never an assertion

This is the subtlest rule in the pipeline, and the one most worth stealing.

The reader returns a `hardware` field: the fastenings, pockets and trims it can
unambiguously see. It is instructed to be conservative and to write "none
clearly visible" rather than guess — and, specifically, not to credit a garment
with a detail belonging to the layer over or under it, because a placket seen
through an open jacket belongs to the shirt.

The extraction step then treats that field as an **upper bound**:

```
Show no fastening, pocket, trim or hardware beyond what the photo shows
(button placket with six matching buttons, buttoned cuffs); anything not
listed there must be left off entirely.
```

Not "add a button placket with six buttons". The difference is everything. If
the reader is wrong — and on a half-hidden garment it sometimes is — a ceiling
produces a plainer garment, which searches acceptably. An assertion produces a
garment with six buttons that were never there, which searches for the wrong
product entirely.

And when the field is empty or hedged, the instruction flips to an explicit
prohibition rather than falling silent:

> Add no zipper, buttons, buckles, pockets, drawstring, logo or trim — none is
> clearly visible in the photo, so the garment must be shown plain.

Silence is not neutral. An unmentioned zipper is a zipper the model may add.

## Name the objects that must not appear

Last clause, and it catches a whole family of near-misses:

> None of these other garments may appear, whole or in part: navy knitted vest,
> wide-leg trousers, white sneakers.

Every sibling object, listed by name, forbidden. "Remove everything else" is a
weaker instruction than an enumeration, because the enumeration gives the model
specific things to check its output against.

## The shape of the whole instruction

Assembled, in this order, and the order is not decorative:

1. **The deletion.** Remove the occluders; keep only this object.
2. **The frame.** One isolated e-commerce product photo, alone, centred on plain
   solid white.
3. **The pointer.** This object is *the light blue shirt whose collar and long
   sleeves show from under the navy knitted vest*.
4. **The description.** What it is.
5. **The shape**, matched exactly, with the specific distortions forbidden.
6. **The finish**, reproduced from the visible areas, with the upgrades
   forbidden.
7. **The rebuild**, if something was hidden — same fabric, complete, and plain.
8. **The see-through clause**, if this is a top layer.
9. **The hardware ceiling**, or the prohibition when there is nothing to see.
10. **Invent nothing**: no print, panel, seam or styling detail.
11. **No person, no background, no props, no text.**
12. **The named exclusions.**
13. **Even studio lighting, one faint contact shadow.**

Thirteen clauses to replace "extract the shirt on a white background." Each one
exists because its absence produced a specific, reproducible wrong answer.

## What carried over, and what did not

When the pipeline stopped being fashion-only, the question was how much of this
was about clothes.

Almost none of it. Layering is occlusion; a cushion on a sofa hides the seat the
way a vest hides a shirt. Every structural rule transferred: lead with the
deletion, rebuild the hidden parts, hold the finish and the shape, treat details
as a ceiling, enumerate the exclusions, an empty list is a valid answer.

What did not transfer was vocabulary. `layer` becomes `placement`, and its
values change from `outer`/`mid`/`base` to `foreground`/`on-top-of`/`inside`,
because a lamp is not a base layer. `hardware` becomes `details`,
`fabric_finish` becomes `material_finish`, `silhouette` becomes `form`. And
`front_opening` — whether a top layer hangs open and what shows through the gap
— has no general equivalent, so it was dropped rather than faked into something
meaningless for a coffee table.

That ratio is the useful finding. The thirteen clauses are about photographs and
about how image models behave, not about garments. If you are extracting one
object from a picture of several, most of this is yours to reuse.

## Three things to take away

**An isolation instruction is a deletion instruction.** Name what must go, first,
before describing what must stay.

**Uncertainty must be encoded as a ceiling.** "Show nothing beyond X" degrades
into a plainer object when the upstream description is wrong. "Add X" degrades
into a fabricated one. Same information, opposite failure modes.

**Absence has to be named.** No lapel, no hood, no logo, no second fabric
through the gap. Anything you do not forbid is something the model may
helpfully provide.

---

*The prompts described here are in `src/domains.js` and `src/pruna.js` in the
Inspired repository. The fashion path is asserted character for character by
`test/api-platform.test.js`, so the general object path could be added without
any risk of changing what the consumer app has always produced.*