Which part of the body is this slice?

Scrolling the CT of a PET/CT from the skull to the mid-thigh while the panel and the body figure follow the slice.

I saw the Clef announcement from Cloudflare on X. Clef is also a classifier like Jev which answers multiple choice questions, but it can also look at images, so I wanted to try it on medical images and see if it can tell which part of the body a slice is from.

DICOM already has a tag for this, BodyPartExamined, but it is often missing or wrong. In the Imaging Data Commons, 11.5% of 267,168 CT series do not have it and another 6.9% have an organ name in it instead. Even when it is right, it is for the whole scan and not for the slice you are looking at, so a whole-body PET/CT is "whole body" from the first slice to the last. So if Clef can classify each slice from its pixels, OHIF can show the body part while you scroll.

What you see

A panel is added to the top right corner of every axial viewport which shows the body part of the current slice, one of head, neck, chest, abdomen, pelvis, legs or other, with the probability of each, and lights up the same region on a full-body figure.

A chest slice with the Clef panel — A chest slice which clef-flash on Workers AI classified as 95% chest in 169 ms.

How it works

Rendering of the current slice is done in the browser, where it is windowed with a soft-tissue window for CT or a percentile window for MR and shrunk to 128 × 128.
Classification is done via Clef on Workers AI in a Cloudflare worker which receives the image and is asked one multiple choice question with the instruction "Classify this image." and the seven body parts as the choices, without any description or example.
Result is the probability that Clef returns for each body part, which the panel shows, since Clef does not write any text.

I tested two simplifications on 19 slices, the bare prompt against a long descriptive prompt and 128 pixels against 512 pixels, and both gave the same answers on all of them. The smaller image also means fewer tokens, which is 251 tokens for each slice or about two thousandths of a cent with clef-flash, and the classification takes about 200 ms in the worker.

Waiting for the scroll to stop

Since scrolling through a stack would send a request for every slice, the panel waits until the scrolling stops, with a wait between 60 and 300 ms based on how fast you scroll, and then sends one request and caches the answer. On the live site, scrolling 20 slices in 0.6 seconds sends one request 93 ms after the scrolling stops, and the answer is on the screen about a third of a second after the scrolling stops.

PET reads its CT

PET slices alone did not work well, and on the slices I tested Clef got only two or three out of six right from the PET image but all of them from the matching CT. So for PET and SPECT, the panel finds the co-registered CT slice at the same position, which is in the same study and frame of reference and within 10 mm, and classifies that one instead.

A PET slice labelled from its CT — A PET slice which is classified from the CT slice at the same position as 92% abdomen.

How good is Clef

On this task it works well, and in the scroll above the body part goes from head to neck, chest, abdomen, pelvis and legs over 86 slices with only one back and forth at the hip. To get a real number, I also ran Clef on all 17,778 test images of OrganAMNIST, which is a harder benchmark with 11 organs, with the same kind of one-line prompt and no training, and the small clef-flash got 45% right while the larger clef got 34%. More on that in the next post.

Try it

You can try it at ohif-clef.alirz.dev by opening a CT, MR or PET/CT and scrolling through it while watching the top right corner. It is a research demo on public datasets and not a medical device.