There's a term out there in the AI community today that everybody's using: Multimodal RAG.

Let's Start from Scratch: What's This RAG Thing Anyway?

Click to zoom

Let's get this "RAG" out of the way before we dive into Multimodal RAG. RAG stands for Retrieval-Augmented Generation. Think of it like this: "Go find some helpful info, mix it with what you know, and then give me a smart answer."

Here's the deal - you know those AI chatbots like ChatGPT that seem super smart?

Well, they've learned tons of stuff, but they have some problems. Sometimes their knowledge is outdated, or when you ask about something really specific, they just shrug and say "Sorry, no idea." That's where RAG comes to the rescue. Before the AI answers you, it acts like a detective.

It searches through fresh databases, company documents, maybe even the internet, to find the most up-to-date and relevant information about what you asked. Then it blends this new info with what it already knows and gives you a much better, more current answer. Pretty cool, right?

Okay, But Is Text-Enough? What's With Old-School RAG?

Traditional RAG systems primarily work with text. The query is text, the information it retrieves is text, and the answer it generates is text. This approach works well for many scenarios, but it has limitations. Our world contains much more than just text. Think about diagrams in instruction manuals, charts in presentations, or important details in medical images.

These visual elements contain valuable information that's difficult to capture in words alone. This is where traditional text-based RAG systems fall short - they simply can't process or understand non-textual information effectively.

And Stage Left Enters: Multimodal RAG

Click to zoom

Just at the moment when we can say, "Text is not enough!" Multimodal RAG comes storming in. "Multimodal" means simply "many channels" or "many forms." Thus, this friend here is not all text; it's a RAG system that is aware of and uses images, audio, video, those less-than-famous Excel spreadsheets, graphs, etc. Just like us! When we learn, we read, we scan through images, we watch videos, don't we?

All because of this new-gen RAG, AI is now able to "read" that graph in the PDF you uploaded ahead, "see" that hairline scratch on the product photo, and "hear" that critical emphasis in the record of that meeting. Then it gathers all this data tweezed out from everywhere and gives you an answer so comprehensive, it's like it had a better grasp of the subject than me or you.

Multimodal RAG Explorer

Select Input Type:

Your Query:

Or try one of these examples:

Why Should I Care About Multimodal RAG?

Okay, so you're thinking now, "Sounds cool, but what am I gonna do with it?"

That's all around us is not just bland plain text. Slideshows, articles, tweets, scientific papers. They're a complete mess: there's text, there's pictures somewhere, a video somewhither and there's another graph somewhere.

And that is where the alchemy of Multimodal RAG begins:

• Deep Water Swimming: Instead of just reading through the text on the surface, it recognizes objects from pictures, reads graphs for trends, and works out hidden relations in tables. Thus, it does not scratch the surface but goes deep.
• Answers Are More on Point: Particularly for questions that require visual or audio information (such as "What is the car model in this photo?"), it can provide on-the-dot answers since it handles that data natively.
• Making Complex Stuff Simple: It can take those pages and pages of number-laden tables nobody wants to read, or those immensely complex diagrams, and say, "Buddy, here's the bottom line."
• Using with Real-World Data: It handles mixed-format data, just like the real world displays, much more effectively.

Isn't There a Catch? What Are the Hard Places?

Of course, every rose has a thorn, and every tech has a "but" or two. Constructing Multimodal RAG is not quite a walk in the park:

• Every Modality is an Individual Things: Text analysis is one thing, interpreting an image or decoding a sound recording something else again. Every form of data has its quirks. Think about it, achieving that "vibe" in a holiday photo is different from being able to pick up a millimeter detail on an architect's plan, isn't it?
• Combining Different Worlds: The integration of diverse data modalities presents a significant technical challenge. Effectively synthesizing semantic information from disparate sources—such as correlating graphical elements with their textual explanations—requires sophisticated algorithms that can create coherent, unified representations while preserving the unique contextual value of each modality.

So How Do the Pros Construct These Systems? Basic Strategies

Our engineering wizards have gotten together and come up with a few broad strategies:

1. "All for One" (Common Embedding Space): All data types get thrown into one shared mathematical space using clever models like CLIP. It's like putting apples and oranges in the same fruit basket to compare them. Makes using existing RAG stuff easier, but you need beefy models that can handle all the details.

2. "Let Me Speak Your Language" (Translating One Modality to Another): Just turn everything into text first. Like, "This image shows a cat on grass under clear sky." Then feed these descriptions to a regular RAG. Works when text does the job, saves you from building new models. Downside? You lose some of the original image's magic.

3. "Everyone on Their Own Team, Meet in the Finals" (Separate Stores and Re-ranking): Use different storage boxes for different data types. When asked something, each box grabs its best stuff. Then a smart filter (re-ranker) picks the most relevant bits. Makes specializing easier but adds complexity at the filtering stage.

4. "The Mixtape" (Hybrid Approaches): Mix and match these approaches for best results. Like making your own custom playlist.

What Do We Need if We're Building a Multimodal RAG?

For a Multimodal RAG system, generally, one must use the following on the pitch:

• Multimodal Large Language Models (MLLMs): These are the operation's brainy super-geniuses. Special LLMs that can understand text, images, sound, etc., and generate useful responses from all of them. When you see names like LLaVa, GPT-4V, Qwen-VL, realize that these are what they refer to.
• Embedding Models: These are the translators. They convert text or images into a semantic equivalent that can be used by computers (i.e., vectors full of numbers). CLIP and Sentence-BERT are the masters of this trick.
• Vector Databases: Special storage that stores these numerical equivalents (vectors) and allows us to query through them at lightning speed. Think Chroma DB, Milvus, FAISS.
• Data Parsing/Extraction Tools: You see those PDFs, Word documents, etc.? Those are little programs that pull out the text, images, and tables from them. Unstructured.io, for example, is quite good at it.
• The Orchestra Conductor (Orchestration Tools): Tools that orchestrate the workflow and make all these different pieces play together in harmony without stepping on each other's toes. LangChain is a widely used conductor for that.

Let's Get Practical: Step-by-Step, How Does Multimodal RAG Work?

Theory aside, if you ask, "How does this stuff actually work in real life?" it generally goes like this:

1. The Warm-Up (Data Preprocessing):

   • Extraction Operation: Texts on one side, images on the other – they're separated from the documents we have (like those notorious PDFs).
   • Who's Who? (Classification): Images are taken into consideration and a categorization is created, e.g., "Is it a graph, or is it a picture of our friend Necati's holidays?"
   • Giving Images a Voice (Summarization/Captioning): Short descriptions are generated for images, e.g., "This image has X, doing Y." Especially for graphs, models like DePlot can translate the figures and lines to text.
   • Everybody Gets an ID (Embedding): Semantic ID cards (embeddings) in machine-readable form are generated for images (or their image descriptions) and text passages.

2. Storage Strike Time: These created ID cards (vectors) are dumped into a dedicated vector database where they can be quickly found when needed. Every now and then, a reference to the original image file is also stored, labeled "True Copy."

3. Question In, Brains On! (Retrieval and Generation):

   • The user's query is taken in, and an ID card (vector) is created for it too.
   • The database is queried and the most appropriate text and/or image IDs are fetched and called in along with their respective owners.
   • Those most suitable texts and (if any) images are provided to our super-brain Multimodal LLM (MLLM) as "Here's your material."
   • The MLLM uses the question and this extensive content laid out before it to create a rich, satisfying answer for the user. If it is simply inquiring outright about an image (like "How many individuals are in this photo?"), then the MLLM flaunts its visual question answering (VQA) capabilities.

Uses Cases of Multi Modal Rag

Actually, you should be asking, "Where can't we use it?" But for a couple of well-known examples, here they are anyway:

• Reports Such as Mixed Nuts: Perfect for breaking down those large reports, financial reports, or market research reports that contain graphs, tables, and lots and lots of text.
• Seeing Chatbots: Smart aides that are able to answer questions like, "What is the function of this button on the screen?" or "What type of architecture is this in the picture?" – ones that are able to see the same thing you do.
• Dancing with Manuals: You recognize those instruction manuals full of pictures, or user guides for high-tech devices? Guiding users who ask, "Where do I put this screw?" through them immediately.
• News from the Sectors: Familiarizing physicians in health care with X-rays, identifying patterns from live stock market charts in finance, providing interactive, multi-channel course materials for students in education. And many more!

What's on the Horizon? Where Is This Headed?

This Multimodal RAG topic is still super fresh. But it's already giving us a glimpse of what we can expect in the future:

• Information will be retrieved not just by typing, but by asking, "Hey assistant, who is the person in this picture?" or sending an audio file.
• AI output won't be text anymore either. Maybe it'll graph something out for you, or possibly show you what it's talking about in a picture.
• We'll see "multimodal agents" – systems that can plan and execute much more advanced tasks in a step-by-step manner, working with various forms of data simultaneously.
• Complaints like "This image resolution is too low, I can't make anything out" will recede, as AI becomes better at understanding all sorts of visuals.