Multimodal Models for PMs: When the AI Can Actually 'See'

A multimodal model is an AI that embeds images and text into the same mathematical space, so it can reason across both at once — reading a chart, describing a screenshot, or answering a question about a photo. This is different from bolting OCR onto a chatbot: the model genuinely relates pixels to language, which is what unlocks image Q&A, document understanding, and visual debugging inside a product.

Quick Answer: Multimodal LLMs (like GPT-4o, Claude, and Gemini) convert images into the same token-based representation as text, letting one model answer questions about a screenshot, receipt, or chart. They're strong at reading layout and describing content, weak at fine detail and precise spatial reasoning, and meaningfully more expensive per request than text-only calls.

What "multimodal" actually means under the hood

A multimodal model turns an image into a sequence of tokens the same way it turns a sentence into tokens, then lets both flows attend to each other inside one transformer. The model isn't running a separate "vision plugin" — it has learned, during training, to place image content and text content into a shared representational space.

That shared space is the same idea covered in how embeddings work for PMs: points that are semantically close sit near each other in high-dimensional space, whether the input was a word or a patch of pixels. A photo of a dog and the word "dog" end up near each other in that space — which is precisely what lets a model answer "what's in this image?" in plain language.

From pixels to patches to tokens

1. The image is split into a grid of fixed-size patches (commonly 14x14 or 16x16 pixels).
2. Each patch is run through a vision encoder and converted into an embedding vector.
3. Those vectors are projected into the same token space the language model already uses.
4. The transformer processes image tokens and text tokens together, attending across both.

This is why resolution matters more than file size. A high-resolution image is chopped into more patches, which means more tokens — and, as covered in why tokens are the AI billing unit, more tokens is the direct driver of both cost and latency. A 4K screenshot isn't just "a bigger file" to a multimodal model — it's a much longer input sequence.

What multimodal models are genuinely good at

Multimodal models are reliably strong at three PM-relevant tasks: describing what's broadly present in an image, reading printed or typed text inside it, and summarizing structured visuals like charts and tables. All three stem from the same underlying skill — relating spatial layout to language — rather than three separate capabilities.

Image Q&A is the most direct use case: point the model at a screenshot and ask "what does this error say?" or "is the checkout button visible above the fold?" The model answers in natural language, grounded in what it can see.

OCR-like reading goes further than a traditional OCR engine because it understands context, not just characters. It can read a receipt and tell you the tax line specifically, or read a form and tell you which field is empty — tasks that require understanding structure, not just extracting text.

Chart and document interpretation is where multimodal models earn their keep for PMs specifically:

• Reading a bar chart and stating which category is highest, without needing the underlying CSV
• Summarizing a dense product roadmap slide into three sentences
• Extracting rows from a screenshotted table into structured text
• Flagging a UI element that looks broken or misaligned in a screenshot
• Comparing two mockup screenshots and describing what changed

Where "good" quietly breaks down

Performance on these tasks isn't uniform — it degrades with image quality, chart complexity, and text density. A clean, high-contrast screenshot of a dashboard is a very different task from a blurry photo of a whiteboard, even though both are "just an image" to the person prompting.

What multimodal models still get wrong

Multimodal models are unreliable at fine visual detail, precise counting, and spatial or geometric reasoning — the exact opposite of their strength at holistic description. Ask a model to count the dots in a scatter plot or measure the angle between two lines, and accuracy drops noticeably compared to describing the chart's general trend.

This connects to the broader reliability caveat in why LLM outputs aren't fully reproducible: the same image and the same prompt can produce a slightly different chart reading on two separate calls, particularly on borderline cases like a value sitting right at a gridline. Treat multimodal output as a strong first read, not a verified extraction — especially for anything that will drive a number in a report.

The pattern behind the failures

Most failure modes trace back to one root cause: the model reasons about images the way it reasons about text — by pattern-matching against what it has seen before, not by running geometry. It hasn't "measured" anything; it has generated a plausible-sounding description. That's a useful mental model whenever you're deciding how much to trust a specific answer.

What multimodal calls actually cost

Images are token-expensive, often costing the equivalent of hundreds to well over a thousand tokens per image depending on resolution and provider-specific tiling rules. A single high-resolution screenshot can consume more tokens than several paragraphs of text — which matters directly for both latency and API spend.

Providers handle this differently, and the differences are large enough to change a build decision:

The most practical cost lever for a product team is simple: resize images before sending them, rather than uploading full-resolution screenshots by default. If a feature only needs to read a headline and a button label, a downscaled image often does the job at a fraction of the token cost — the same "does this token spend earn its keep" question that should govern every LLM feature, per the complete guide to how LLMs actually work.

Good and bad use cases for image-in features

Multimodal features work best when the task tolerates an approximate, human-readable answer and fails when the task demands a precise, auditable number. This single distinction should decide whether an image-in workflow belongs in your roadmap at all.

Good fits:

• Customer support triage: "what's wrong in this screenshot the user sent?"
• Accessibility review: describing a UI for a visually impaired user
• Rough competitive scans: summarizing a competitor's pricing page screenshot
• First-pass document intake: pulling a vendor name and rough total off a scanned invoice

Bad fits:

• Financial reconciliation that requires exact totals from a receipt, unverified
• Any workflow where a wrong pixel-level measurement has real consequences (medical, legal, structural)
• Replacing a real OCR pipeline where 99.9% character accuracy is a hard requirement
• Counting-heavy tasks (inventory counts from a photo, precise chart value extraction) without a human check

This good/bad split maps cleanly onto the "job" framing from the complete guide to Jobs to Be Done: the job a customer is hiring the feature for determines the acceptable error tolerance, not the model's capability alone. "Help me understand roughly what's in this screenshot" and "give me the exact number from this invoice" are different jobs with wildly different tolerance for the same underlying weakness.

A quick gut-check before scoping a vision feature

Ask three questions before committing engineering time: Does the task need exact numbers, or an approximate read? Is the image likely to be clean and high-contrast, or noisy? And does a wrong answer get caught downstream, or does it ship silently? A "yes" to precision, noise, and silent failure together is the profile of a feature that needs a human in the loop, not full automation.

Pressure-testing an image-in idea before you build it

Before committing engineering time to a vision feature, it helps to sketch the actual flow — what image comes in, what question gets asked of it, what the output looks like — and stress-test it against the failure modes above. Prodinja's Wireframing composer lets you sketch that kind of image-in workflow early, so a team can walk through screens and data flow together and surface an over-ambitious precision requirement before a single API call gets written. It's a sketching and thinking tool for the workflow, not a live vision model — the value is in forcing the "what does this feature actually need the model to get right" conversation before scoping starts.

Mapping the surrounding experience is worth doing at the same time. Prodinja's Customer Journey tool can help lay out the emotion curve around a support-triage or document-intake flow, so a team can see exactly where an approximate multimodal answer is good enough and where it isn't — before the feature reaches customers. That distinction — covered in more depth in the complete guide to mapping customer journeys — is often the difference between a multimodal feature that delights and one that quietly erodes trust.

Key Takeaways

• Multimodal models embed images and text into one shared space, letting a single model reason across both rather than running separate OCR and language systems.
• Resolution drives cost, because images are converted into patches and then tokens — a high-res screenshot can cost far more than a paragraph of text.
• Holistic description is strong; precise counting and spatial measurement are weak — the gap between "what does this chart show" and "how many dots are above the line."
• Good use cases tolerate an approximate answer; bad use cases need an exact, auditable number without a human check.
• Downscaling images before sending them is the simplest, most direct lever a product team has over multimodal API cost.
• Sketching the image-in workflow before building it — what comes in, what's asked, what ships — surfaces precision requirements the model may not actually be able to meet.