Ask for the shirt, get the vest: extracting one object from a photograph of several

How the Inspired pipeline learned that "extract the shirt on a white background" is not a specification.

There is a class of bug that only exists because the tool is good. Image editing models got competent enough to isolate an object from a photograph, which made the obvious instruction work often enough to ship — and then fail in a way that is almost impossible to see in aggregate.

Here is the failure. A photograph of someone in a navy knitted vest over a light blue shirt. Ask for the shirt:

Extract the shirt from the image on a white background.

What comes back is a shirt. On white. Centred. Well lit. Still wearing the vest.

The model did what was asked. It found the shirt, it isolated the shirt region — and the shirt region, as photographed, contains a vest. Nothing in the instruction said the vest was the problem, so nothing removed it. The result passes every automated check you would think to write: one garment, white background, no person. It is simply the wrong garment.

Every object photograph has this problem. A sofa with a cushion on it, a lamp behind a chair, a serum bottle in front of a jar. The Inspired pipeline cuts up to six objects out of one photograph, one at a time, which means it hits the problem six times per read. What follows is what the instruction turned into, and why each part of it is there.

The reader has to describe the occlusion, not just the object

The first thing that changes is upstream. If the extraction step is going to delete the vest, something has to tell it there is a vest.

So the reading pass does not return a list of names and descriptions. Per object, it returns the relationships:

Two fields are not about occlusion at all, and turn out to matter as much:

The schema is strict and every property is required, so the reader cannot quietly skip the awkward ones. An empty list is expressible — a photograph with nothing shoppable in it returns zero objects rather than an invented garment — but a partial object is not.

Two rules that came out of the reading pass

Commit to one reading. The instruction forbids hedged alternatives: "collarless or very small collar" is not allowed, because a hedge handed to an image model is reproduced as the wrong option roughly half the time. Decide which it is.

Describe only this object's own edges. When a vest frames a shirt in a V, that V is the shape of the vest's opening, not the shirt's neckline. The reader is told to look at the shirt's own collar edge and buttons instead. Without that clause, layered garments inherit the shape of whatever is on top of them.

Lead with the deletion

Now the extraction instruction. The single most load-bearing discovery:

Open with the edit to perform, not with the description of the object.

Every description-first phrasing lost to this one. "A light blue shirt with a button placket, isolated on white, without the vest" leaves the vest in place a lot of the time. This does not:

Remove the navy knitted vest and keep only the shirt worn underneath it.

The occluder is named as a thing to delete, in the first clause, before anything else. The description comes after, as support.

The mirror case is the outer layer. Asking for the vest has the opposite failure: the shirt underneath survives, showing through the armholes and above the neckline. So the instruction inverts:

Keep only the vest and remove the shirt worn underneath it.

And when a top layer hangs open, one more clause is required, because "remove the shirt" alone leaves the model free to fill the gap with fabric:

Show the vest on its own with nothing worn under it: through its neckline, front opening and any gap, only the plain white background is visible — no shirt, no collar, no cuff, no second fabric of any kind.

Recovering an occluder the reader forgot to declare

covered_by and covers are two views of one relationship, and models fill them asymmetrically: the vest reliably reports that it covers the shirt, while the shirt sometimes reports an empty covered_by. That asymmetry would silently drop the opening clause — the sentence that does all the work.

So the occluder list is reconstructed. If an object declares no covered_by, the pipeline scans its siblings for anything whose covers names it, and uses that instead. One derived list, two chances to populate it.

Rebuild what was hidden, or get a hole

Deleting the vest leaves a vest-shaped absence in the middle of the shirt. Left alone, models resolve that absence in one of three ways: a literal hole, a smooth guess that does not match the visible fabric, or — most often — a different shirt that happens to be complete.

The instruction names the absence and how to fill it, using the reader's hidden_parts and visible_parts:

Where the navy knitted vest hid part of the shirt (torso front, buttons, placket, pockets), continue the same fabric, colour and weave so the whole garment is shown, complete and symmetrical, and keep those rebuilt areas as plain as the visible ones.

"As plain as the visible ones" is doing specific work. A rebuilt torso is an invitation to invent: a chest pocket, a contrast placket, a logo. Plainness has to be requested.

The two upgrades nobody asks for

Image models make things nicer. It is what they were trained to do, and for product extraction it is a defect.

They upgrade the finish. A soft, creased, unpressed cotton shirt comes back crisp, pressed and faintly glossy — a nicer shirt than the one in the photograph, and therefore the wrong one to search for. The counter is the reader's fabric_finish, stated positively and then defended:

Its fabric is soft lightweight cotton, visibly creased and unpressed, matte. Reproduce that exact weight, texture and finish across the whole garment, copied from the collar, upper chest, both sleeves, left cuff, hem below the vest. Do not make it crisper, stiffer, glossier, newer or more formal than the photo: keep the same drape, softness and creasing, and do not press or restyle it.

They upgrade the shape. A boxy cropped knit drifts toward a longer, more tailored, more flattering version of itself. "Keep the silhouette as photographed" carries no information a model can act on. The named cut does:

Its cut and shape: cropped, boxy with dropped shoulders, no collar and no lapel, straight ribbed hem, full-length sleeves with ribbed cuffs. Match that shape exactly — the same length, the same shoulder line, the same collar or neckline, the same sleeve and hem shape. Do not lengthen or shorten it, do not add a lapel, a pointed collar, a stand collar or a hood, do not make it oversized or gathered, and do not curve a straight hem.

Naming what must be absent — no lapel, no hood — matters as much as naming what is present. Absences are what get filled in.

Hardware is a ceiling, never an assertion

This is the subtlest rule in the pipeline, and the one most worth stealing.

The reader returns a hardware field: the fastenings, pockets and trims it can unambiguously see. It is instructed to be conservative and to write "none clearly visible" rather than guess — and, specifically, not to credit a garment with a detail belonging to the layer over or under it, because a placket seen through an open jacket belongs to the shirt.

The extraction step then treats that field as an upper bound:

Show no fastening, pocket, trim or hardware beyond what the photo shows
(button placket with six matching buttons, buttoned cuffs); anything not
listed there must be left off entirely.

Not "add a button placket with six buttons". The difference is everything. If the reader is wrong — and on a half-hidden garment it sometimes is — a ceiling produces a plainer garment, which searches acceptably. An assertion produces a garment with six buttons that were never there, which searches for the wrong product entirely.

And when the field is empty or hedged, the instruction flips to an explicit prohibition rather than falling silent:

Add no zipper, buttons, buckles, pockets, drawstring, logo or trim — none is clearly visible in the photo, so the garment must be shown plain.

Silence is not neutral. An unmentioned zipper is a zipper the model may add.

Name the objects that must not appear

Last clause, and it catches a whole family of near-misses:

None of these other garments may appear, whole or in part: navy knitted vest, wide-leg trousers, white sneakers.

Every sibling object, listed by name, forbidden. "Remove everything else" is a weaker instruction than an enumeration, because the enumeration gives the model specific things to check its output against.

The shape of the whole instruction

Assembled, in this order, and the order is not decorative:

  1. The deletion. Remove the occluders; keep only this object.
  2. The frame. One isolated e-commerce product photo, alone, centred on plain solid white.
  3. The pointer. This object is the light blue shirt whose collar and long sleeves show from under the navy knitted vest.
  4. The description. What it is.
  5. The shape, matched exactly, with the specific distortions forbidden.
  6. The finish, reproduced from the visible areas, with the upgrades forbidden.
  7. The rebuild, if something was hidden — same fabric, complete, and plain.
  8. The see-through clause, if this is a top layer.
  9. The hardware ceiling, or the prohibition when there is nothing to see.
  10. Invent nothing: no print, panel, seam or styling detail.
  11. No person, no background, no props, no text.
  12. The named exclusions.
  13. Even studio lighting, one faint contact shadow.

Thirteen clauses to replace "extract the shirt on a white background." Each one exists because its absence produced a specific, reproducible wrong answer.

What carried over, and what did not

When the pipeline stopped being fashion-only, the question was how much of this was about clothes.

Almost none of it. Layering is occlusion; a cushion on a sofa hides the seat the way a vest hides a shirt. Every structural rule transferred: lead with the deletion, rebuild the hidden parts, hold the finish and the shape, treat details as a ceiling, enumerate the exclusions, an empty list is a valid answer.

What did not transfer was vocabulary. layer becomes placement, and its values change from outer/mid/base to foreground/on-top-of/inside, because a lamp is not a base layer. hardware becomes details, fabric_finish becomes material_finish, silhouette becomes form. And front_opening — whether a top layer hangs open and what shows through the gap — has no general equivalent, so it was dropped rather than faked into something meaningless for a coffee table.

That ratio is the useful finding. The thirteen clauses are about photographs and about how image models behave, not about garments. If you are extracting one object from a picture of several, most of this is yours to reuse.

Three things to take away

An isolation instruction is a deletion instruction. Name what must go, first, before describing what must stay.

Uncertainty must be encoded as a ceiling. "Show nothing beyond X" degrades into a plainer object when the upstream description is wrong. "Add X" degrades into a fabricated one. Same information, opposite failure modes.

Absence has to be named. No lapel, no hood, no logo, no second fabric through the gap. Anything you do not forbid is something the model may helpfully provide.

The prompts described here are in src/domains.js and src/pruna.js in the Inspired repository. The fashion path is asserted character for character by test/api-platform.test.js, so the general object path could be added without any risk of changing what the consumer app has always produced.