A model that can read, see, hear, and speak is called multimodal. This guide starts from a phone pointed at a foreign menu, builds up the shared space that lets one model mix its senses, tours the tools doing it today, and then draws the line most people get wrong: multimodal is not the same as multi-model.
To answer it, something has to see the menu, read the words on it in a script you cannot, understand that a little leaf icon next to a dish means meat-free, reason about which prices clear your budget, and then speak the reply so you can keep your eyes on the street. Seeing, reading, reasoning, speaking. Four different kinds of skill, braided into one ordinary request.
A text-only assistant cannot begin. It has no eyes. You would have to type out a menu you cannot read, which is the whole problem. The question only works if one system can take in a picture and a voice, and hand back a spoken answer. That capability has a name, and this guide is about what is really going on behind it.
Start with the root word. A modality is a type of information, defined by its nature and not its file format. Text is one modality. Images are another. So are audio, and video, and less obvious ones: a depth map from a camera, a table of numbers, a stream of sensor readings, even source code. Hover each to see what makes it its own modality and how a machine takes it in.
Hold onto one fact that the rest of the guide leans on. A computer cannot work with any of these directly. It turns every one of them into numbers first: long lists of numbers called vectors. A word, a patch of a photo, a slice of sound, all become vectors. That shared fate is the seed of everything multimodal.
Before one model could do it all, engineers chained specialists together. The menu problem became an assembly line: a vision tool reads text off the image, a translator converts it, a language model reasons about it, a speech tool voices the result. Each box is good at its one job. The trouble is the gaps between them. Step through the line.
The pipeline works, and for years it was the only option. But every arrow between boxes is a place where meaning drains out. The reasoning model never sees the photo, so it cannot use the little leaf icon that a human reads instantly. The words arrive stripped of layout, colour, and emphasis. Information that lived in the image dies at the first arrow, because after step one the rest of the line is working with text about the menu, not the menu.
The leap is to stop whispering down a line and put all the senses in one head. A multimodal model is a single model trained so that different kinds of data live in the same internal space of vectors. The photo of the menu, the words on it, and your spoken question are not translated into each other. They are all turned into vectors of the same shape, in one shared space, and the one model reasons over all of them together.
Because the picture and the words share a space, the model can notice that the leaf icon sits next to a particular dish and that the dish is priced at twelve dollars, in a single thought, without anything being whispered. Nothing is lost to a hand-off, because there are no hand-offs. That shared space is the whole trick, and the next part lets you play with it directly.
Here is the idea made concrete. Every piece of data, whatever its modality, becomes a point in one high-dimensional space. The model is trained so that things which mean the same thing land near each other, no matter which sense they came from. A photo of a dog, the written word "dog", and the sound of a bark all cluster in the same neighbourhood.
That is not loose talk. A point in that space is a real, concrete thing: a short list of numbers. Turning a word into that list is called making an embedding. Here is the word "dog" as a tiny four-number embedding. Each number is one coordinate, one learned dimension of meaning.
It is literal geometry you can measure, so in the next chapter you can drag a point through it and watch which neighbours light up.
Below is a flattened view of one shared space. Each dot is a real item: some are images, some are words, some are sounds, coloured by modality. Grab the ringed query point and drag it anywhere. The page measures, live, which items sit nearest and lists them. Notice that the nearest neighbours are almost never all the same colour. That mixing is multimodality, made visible.
This is what "understanding across modalities" actually means underneath the marketing. Not that the model has a soul that grasps dogs, but that it has learned to place a dog photo and the word dog close together, so a question asked in one sense can be answered with material from another. Search your photo library by typing words, caption an image, find music that matches a mood: all of it is a query landing in this shared space and reading off its neighbours.
How does a photo end up in the same space as a sentence? The model turns each modality into a sequence of tokens, the same currency it already uses for text, then runs the whole mixed sequence through one network. Scroll to watch a sentence, an image, and a clip of sound converge into a single stream.
A sentence, a photo, and two seconds of audio. Three modalities, three completely different kinds of raw data. Right now they have nothing in common.
The sentence is chopped into pieces (roughly words), and each piece is turned into a short list of numbers, a vector. Here, four numbers each, shown as four little bars. That list is the embedding from Figure 3.
The photo is cut into a grid of small patches, and each patch becomes its own four-number vector. Look at the bars: a patch is the exact same shape as a word. That is the quiet miracle.
The audio is sliced into short frames, and each frame becomes four numbers as well. Now every sense, however different it started, has been rewritten as vectors of one shape.
Because they are all the same shape, the vectors simply line up into a single row and flow through one network. From here the model does not care which numbers came from a word, a patch, or a sound. That is fusion.
People argue about when to merge the streams: early, right at the input, or late, after each sense is processed on its own for a while. That is the early-fusion versus late-fusion debate, and it is in the weeds. The headline holds either way: make the senses share a notation, and one model can reason across all of them.
The idea is abstract; the tools are not. Each modality has both native multimodal assistants that handle it inside one model and specialist tools built for it alone. Hover each modality to see representative examples. The roster shifts every few months, so treat these as landmarks, not a leaderboard.
"Multimodal" hides a distinction worth making. A model can be multimodal on the way in, the way out, or both. Many assistants read images and hear speech but only ever reply in text. Others are any-to-any: they can also emit an image, a voice, a clip. Click through the three shapes.
There is a second nuance underneath: real-time versus turn-based. Early voice assistants were a pipeline in disguise, speech-to-text then a text model then text-to-speech, with a pause you could feel. Newer native-voice models process the audio directly and reply while you are still talking, which is why they can be interrupted and can hear a laugh. Same word, "voice", very different machinery.
Here is the confusion this guide exists to clear up. The words rhyme, so people swap them, but they describe different things. Multimodal is about senses: one model, many kinds of data. Multi-model is about architecture: many separate models, each a specialist, with something routing work between them. Take one request through both and the difference is plain.
A single request that mixes a picture, a description task, and a coding task. Watch how each architecture handles it.
A multimodal model takes the whole thing at once. The same model sees the photo, describes it, and writes the code. One brain, several senses, no routing.
A multi-model system starts with a router. It inspects the request and decides which specialist should handle which part.
The router dispatches: a vision model for the photo, a code model for the sorting, maybe a third to stitch the pieces. Separate models, each expert at one job.
Multimodal counts senses inside one model. Multi-model counts models inside one system. A multimodal model can even be one of the specialists a multi-model system routes to. They stack; they are not rivals.
One more trap. A "mixture of experts" model sounds like a multi-model system but is not. Its experts live inside one model and are switched between token by token, invisibly. That is an efficiency trick within a single model, not a router calling separate products. The weeds untangle it.
A model with more senses is not a model with fewer flaws. Some of the gaps are specific to mixing modalities, and they matter in production.
The senses are uneven. Most multimodal models are strongest in text, decent at images, weaker at audio, and shakiest on video. A confident answer about a picture can still be wrong in ways a caption never reveals. Trust each sense only as far as your own tests show it earns.
Grounding fails across senses too. A model can describe a stop sign that is not in the photo, or mishear a word that changes a whole instruction. Hallucination did not go away when we added eyes; it got new places to happen.
Other senses are expensive. An image can cost hundreds or thousands of tokens; video multiplies that by every frame. Multimodal requests are heavier and slower than text, and the bill and the latency both grow with how much you feed in.
Privacy changes shape. Sending a photo or your voice is not like sending a sentence. It can carry a face, a location, a room, a person who never agreed to be uploaded. More senses in means more of the world leaving your device.
A modality is a sense.
Multimodal is one model with several.
Multi-model is several models with a router.
The menu needed the first; a whole product often needs both.
Put it together. Choose which senses come in, which go out, and whether you build it as one multimodal model or a multi-model system with a router. Watch capability, cost, and complexity move against each other. The presets rebuild the pipeline era, a modern native assistant, and a full product.
Seven rabbit holes the main path stepped around. Each stands alone.