Generative data for an optical sorter

A machine that sorts potatoes has to recognize defects it has almost never seen. Rotten, green, misshapen and cut potatoes are a small fraction of what comes off a field, harvests happen twice a year, and a model learns from labeled photographs of exactly the thing nobody has enough of.

CropVision is an optical sorter built for unwashed potatoes. It runs up to a hundred tonnes an hour, films each object from every side as it travels along a spiral conveyor, classifies it on embedded computers, and then a row of pneumatic fingers pushes it toward one of three exits, keep, discard, or the middle grade the operator configures.

My 2024 R&D internship at Downs was about the photographs that were missing. I fine-tuned a diffusion model on the defects the company did have, generated more, and trained the sorter on the mixture. The work is half of a paper at AGRITECH DAY 2024. The other half compares it against active learning and belongs to Noura Mounif, who was first author.

Sample frames from the CropVision cameras, grouped by class: good, green, dirt, stone, deformed, rotten, cut, and four mixed scenes. — What the cameras hand to the model. Seven classes, on spirals and on the belt.

Ten images per background

Before any generation, the dataset decides what is possible. A diffusion model needs to see a concept in at least ten images before it can produce it, and it needs to see it in more than one setting, because a defect photographed only on the spirals is a defect it will only know how to draw on spirals.

The images also come out too small. Stable Diffusion was trained at 512 pixels and above, ours sit between 224 and 384, so each one goes through an upscaling model first and then into size buckets for training. The picture below is the same potato before and after, shown at the same size, and the softness on the left is what the model would otherwise have learned to reproduce.

I wrote all the annotations by hand. The automatic captioning models I tried were built for ordinary photographs and had nothing useful to say about a bruised potato lit from four sides on an industrial belt.

The same potato twice at equal size, blurry on the left before upscaling and sharper on the right after. — One potato before and after upscaling, both shown at the size the training pipeline uses.

Naming the background so the model stops learning it

Here is the problem a caption solves. A potato sitting on the machine's spirals and the same potato on the conveyor belt are two very different pictures, and most of the difference is behind the potato. If the caption says only "potato", the model has no reason to separate the two, and what it learns is that a potato is a lumpy thing with spirals behind it.

So the captions name both parts. [Potato, Coils] for one, [Potato, Conveyor Belt] for the other. The word that varies is the background, which gives the model something to attach the background to, and the word that stays is the object.

Then I checked whether my own wording was doing that. Attention heatmaps show which region of a generated image each token is pulling on, so I could take a caption apart and look. In the figure below, "potato" lands on the potato, "placed on industrial coils" lands on everything except the potato, and "cut" lands on the cut. That is the separation working, and when it was not working the heatmap said so and I rewrote the caption. I tuned the wording on a small set that way and then applied the same convention to the rest.

A generated cut potato, then three attention heatmaps: the word potato lights up the potato, placed on industrial coils lights up the background, cut lights up the cut surface. — One generated potato and where three parts of its caption are looking.

What it did to the sorter

The generated images reached a Fréchet Inception Distance of 0.40, measured over 89 images, 45 real and 44 generated. That number says the two sets sit close together in the feature space of an Inception network, and it says nothing about whether the sorter gets better, which is the only reason any of this existed.

On the sorting task itself, adding synthetic images moved global precision from 0.67 to 0.68 over four folds. The clearest effect was on the cut potatoes, where the model recovered classifications it had been losing. Active learning, the other half of the paper, moved the same number from 0.67 to 0.78. The paper reports both, concludes that generation works as a supplementary tool next to a selection strategy rather than instead of one, and says that this was a first exploration and that more of it is needed.

Read the paper, AGRITECH DAY 2024 proceedings, pages 63 to 72, with Mounif, Dufaye and Wannous.