Trentingrana Sensory Captioning

A tasting panel scores Trentingrana wheels on seven sensory attributes and leaves comments in Italian. Photographs of those same wheels sit in a separate archive, organised by year and session. The project builds the bridge between the two, then tests whether a model can learn to write those descriptions from the images.

The first part took the most care. Free-text comments are noisy, four Excel workbooks from different years use different schemas, and the data does not support a one-to-one match between an individual wheel and a comment: the join happens at dairy level, on session date and product code. One principle drove everything: do deterministically whatever can be done deterministically, and use a language model only for the genuinely hard rewriting.

So normalisation, typo maps, the per-attribute controlled vocabulary and the conversion of crust measurements into qualitative bands are all ordinary code. The lemmatiser is custom-written, because standard NLP libraries consistently mis-handle sensory vocabulary. Only then does an LLM rewrite each unique caption into consistent form, under a hard zero-invention rule with an explicit escape when the text carries no information. The final dataset holds 38,437 rows, and the entire rewriting cost under six dollars in API calls.

For modelling we defined six encoder–decoder architectures across two families: decoders trained from scratch on one side, a pretrained Italian language model on the other. The three evaluated so far show a clear result: seven per-attribute specialists reach 0.40 BLEU-4, while a single global model collapses to 0.13.

The most interesting finding, though, is a negative one, and it is what we would pursue next. CLIPScore, which measures how well a caption fits the image while ignoring the reference text, stays identical between trained models and random baselines. With a frozen encoder the models are doing language modelling over the caption distribution: they are not really looking at the cheese.