How your phone measures food: depth sensors, LiDAR, and portion size
Identifying food from a picture is close to solved. Estimating how much of it is on the plate is not, and it is the larger source of error. Here is how an iPhone can measure the plate instead of guessing, and what the measurement does and does not fix.
Every photo-based calorie app has to answer two questions at once: what is this, and how much of it is there. Modern models are genuinely good at the first. The second is where the error lives — and until recently every app solved it the same way, by guessing.
A camera records angles, not distances
A lens measures direction. It has no idea how far away anything is. A large bowl photographed from across the table and a small bowl photographed up close land on the sensor as the same shape at the same size, and no amount of model training resolves that, because the information was never captured.
People get around this constantly without noticing. You know roughly how big a dinner plate is, so your brain uses the plate to scale everything on it. Take the plate away — shoot a bowl of rice against a plain worktop — and your own estimate collapses too.
This is why early food-photo apps asked you to put a coin, a credit card, or your thumb in the frame. It worked, and nobody did it, because it turned a one-second action into a small chore.
Two ways a phone can measure depth
Every iPhone with more than one rear camera can infer depth from parallax: the two lenses see the scene from slightly different positions, and the disparity between those views encodes distance. It is the same principle as your two eyes, and it is what powers portrait mode. In our code this source is called stereo depth.
Pro models since the iPhone 12 Pro add LiDAR, which does something different: it projects a grid of infrared dots and times how long each one takes to come back. That is a direct measurement rather than an inference, and it stays reliable in low light and on low-texture surfaces — a white plate on a white table, exactly the case where stereo depth struggles most.
0xCal exposes these as two tiers, because the hardware genuinely differs: Ruler uses camera depth and is rated 80%+ accurate, Ruler+ uses LiDAR and is rated 90%+. Those figures describe how well the phone measures the dish, not how accurate the resulting calorie number is. They are separate things, and conflating them is the easiest way to oversell this feature.
Why almost nobody else does this
The capability has been sitting in these phones for years, so the obvious question is why it is not everywhere.
Most calorie trackers ship one product across iOS and Android from a shared codebase. Depth APIs are platform-specific and the hardware varies wildly between Android devices, so anything built on them has to be written twice and will behave differently on each. For a team optimising for reach, that is an easy feature to defer forever.
LiDAR narrows it further: it is a Pro-only sensor, so it can never be the baseline experience. Building on it means building two paths — one for phones that have it and one for phones that do not — and then explaining the difference to users without sounding like you are upselling hardware you do not sell.
And the payoff is uneven, as the table above shows. A feature that helps decisively on curries and hurts on dumplings is a hard thing to put on a marketing page. It is a much easier thing to publish a benchmark about.
What actually gets sent with the photo
The measurement is not a vague hint. When the phone succeeds, the photo is accompanied by a small structured payload:
- A shape and its size. Round dishes are described by diameter in centimetres; everything else by a bounding box in width and height.
- Peak height above the supporting surface, when the depth map is good enough to estimate it — how far the food rises off the table.
- Where the dish sits in the frame, as normalised coordinates, so the model knows which part of the image the measurement refers to.
- Which sensor produced it — LiDAR or stereo depth — plus a confidence score and an estimated error percentage.
That last item matters more than it looks. The phone does not just report a number, it reports how much it trusts the number. A measurement taken at a bad angle in poor light arrives flagged as low confidence, and can be weighed accordingly instead of being treated as ground truth.
Height matters more than width
This is the part that surprises people, and it explains most of what follows.
Knowing a plate is 28 cm across tells you about area. Calories track volume. A 28 cm plate holding a flat salad and a 28 cm plate holding a heaped mound of rice have identical diameters and can differ by five hundred calories. Diameter alone constrains only one dimension of a three-dimensional problem.
Height is the dimension that converts an outline into a quantity, and it is also the one depth sensors find hardest. A liquid surface is smooth and well-behaved. A pile of irregular objects with air gaps between them is not: the sensor measures the envelope around the pile, and the envelope contains a lot of nothing.
Volume still is not calories
Even a perfect measurement leaves the hardest variable untouched, and it is worth stating plainly: volume tells you nothing about energy density.
A hundred millilitres of olive oil is about 884 calories. The same hundred millilitres of clear broth is around fifteen. That is a sixtyfold spread at an identical measured volume, and no depth sensor in the world distinguishes them — they occupy the same space and reflect infrared much the same way.
So measurement and identification are not competing approaches, they are two halves of one calculation. The camera and the model decide what is on the plate, which supplies the density. The depth sensor decides how much space it occupies. Get either wrong and the answer is wrong; the measurement simply removes one of the two guesses.
This is also why the feature runs automatically rather than as a mode you switch on. If it were a button, people would press it expecting a better number and be disappointed on the dishes where it does not help. Running it silently, weighting it by its own confidence score, and falling back when the framing is poor is the honest way to ship something that works most of the time rather than always.
Where measuring helps, and where it does not
We publish this rather than assert it. From our photo benchmark, comparing the same dishes scanned with and without a scale reference:
| Dish (true kcal) | Measured | Photo alone |
|---|---|---|
| Green curry with chicken and rice (502) | 86.3% | 69.9% |
| Chicken Caesar sandwich (528) | 78.8% | 79.9% |
| Pan-fried pierogi with onion (784) | 63.0% | 77.6% |
The curry is the case measurement was built for. Liquid conforms to its container, so the dish diameter plus the fill height genuinely determines the volume, and the error more than halved.
The pierogi are the opposite case, and they lost. Dumplings stack irregularly with gaps, so the measured envelope is substantially larger than the food inside it, and a bigger envelope pushes the estimate the wrong way. The sandwich barely moved, because its size was never the uncertain part.
Three dishes, five runs each, is a small sample and we say so. We publish the losses because a benchmark you only ever win is marketing rather than measurement. The live benchmark page re-runs weekly, so the numbers above will move as the dish set grows.
Framing decides whether it works at all
Depth sensing needs to see where the dish ends. Leave a little room around the plate — that margin is the reference the measurement is taken against. A dish cropped at two edges has no measurable boundary, and the sensible response is to skip the measurement rather than to invent one, which is what 0xCal does by default. Automatic mode only measures when the framing is clear, and falls back to a plain photo estimate otherwise.
That fallback is deliberate. A confident wrong measurement is worse than no measurement, because it displaces the model's own prior with something false.
What this means when you actually log a meal
Measurement narrows one error source. It does nothing for the others, and being clear about that is the difference between a useful feature and a marketing claim.
- It does not identify food. If the model reads a cream sauce as a tomato one, a perfect plate measurement makes the wrong answer more precise, not more correct.
- It cannot see what is invisible. The oil the vegetables were fried in and the butter in the sauce carry real calories and leave no trace in a depth map.
- It helps most on liquid and uniform dishes — soups, curries, bowls, anything where the vessel genuinely defines the portion.
Which is why, for anything you cooked yourself, typing still wins. Our benchmark puts text logging around ten points ahead of photo logging overall, for the simple reason that a sentence carries information no camera can capture. We took that apart in detail in photo vs. text calorie logging, and the practical answer is unchanged: describe what you know, photograph what you do not, and add a short note to the photo when you know something the camera cannot see.
If you want the broader picture on how much any of this can be trusted, our weekly accuracy benchmark covers the whole pipeline, and the 20% rule on nutrition labels is a useful reminder that even the packaged foods you are measuring against carry their own error bars.
Try 0xCal on iPhone
Snap a photo or type what you ate — calories and macros logged in seconds, synced with Apple Health.
