The Filter Returned Everything

The Filter Returned Everything

What a few weeks on Google's managed RAG service actually taught me, told the way it happened.

The model was wrong with total confidence.

It told a user that a plan covered something it did not cover, and it said it the way you would state your own birthday. No hedging. No doubt. A clean, fluent, completely invented fact, pulled out of nothing, because the model had never seen our documents and was filling the silence with the most plausible thing it could assemble.

That is where every RAG project actually begins. Not with a diagram. With a lie you cannot afford.

The fix is old news by now. You stop asking the model to know things and you start handing it the right pages at the right moment. Retrieval first, generation second. The idea is simple. The plumbing is not. Parsers, chunking, an embedding model, a vector database, a retrieval layer, and the glue between all of it. Build that from scratch and you will spend a month on infrastructure before you answer one question well.

So I did not build it from scratch. I reached for Google's RAG Engine.

Google gives you three doors. Behind the first is Vertex AI Search, fully managed, opinionated, high quality out of the box, deciding most things for you. Behind the third is do-it-yourself, every component exposed, every decision yours, every bug yours too. RAG Engine is the middle door. Managed enough that you skip the plumbing, open enough that you still pick the parser, the chunk size, the embedding model, the vector store, the model. I wanted the middle door. Most people who try this do.

The first afternoon was a pleasure. You create a corpus. You point it at a Cloud Storage bucket. You wait while it imports and embeds. Then you ask a question and it hands back the real paragraphs from your real documents, sources attached. No API keys to babysit, because it rides on the cloud's own identity. No vector database to stand up. It just worked, and working felt like progress.

Then I read the answers closely. They were fine. Fine was the problem. The retrieval pulled things that were sort of about the question, vaguely on topic, not precisely right. I had assumed "managed" meant "good." It means "set up." By default the engine runs plain dense vector search, and dense search alone goes mushy on exact terms, codes, policy numbers, the things people actually type.

The fix was sitting right there. Hybrid search, which blends keyword matching with meaning. A reranker, which takes the first messy pile of results and reorders them by real relevance before anything reaches the model. I switched both on and the answers sharpened. First lesson: this tool gives you the parts of a good retrieval system, not a good retrieval system. You assemble the quality yourself.

The documents taught the next lesson. Contracts and benefit summaries are full of tables and two-column layouts, and an ordinary parser reads those left to right and turns a fee schedule into a paragraph of scrambled noise. The engine has an answer here too, a layout parser that looks at the page the way a person does, keeps the tables as tables, respects the columns. For messy documents it is the single biggest quality lever in the whole product. It also has a temper. It ignores your chunk settings and chunks the way it wants, and if you imported files before turning it on, it will not go back and fix them. You delete and re-import. I re-imported a lot.

And then I hit the wall, and the wall is the reason I am writing this.

I needed to filter. A small thing. Every document had a category, and a query should only ever see documents in its own category. The SDK has a filter argument that takes a metadata filter. I set it. I ran the query.

It returned everything. No error. No warning. Just the whole corpus, calmly handed back.

Here is what no quickstart tells you. That high-level filter does nothing at all unless you have already built a metadata schema on the corpus, through a different and lower-level version of the API. It does not read the metadata you set on your files in Cloud Storage. It does not read a metadata file in your bucket. It wants a ritual. Declare a schema for each field you plan to filter on. Attach the values to every file by its internal file ID. Then query with a filter written in a specific expression syntax, against the beta API, and not while you are using the new serverless storage mode, and not at all if your project switched on certain security controls before the feature existed.

The filter argument I had been using was real. It just quietly does nothing when the ritual is incomplete. No exception. No "you skipped a step." It returns everything and lets you find out on your own.

I want to be fair. Once you know the ritual, it works. But "it works once you know" is carrying a lot of weight, because the failure mode is silence, and silence is the most expensive kind of bug. Plenty of teams never finish the ritual. They give up on native filtering and scope each query to an explicit list of file IDs instead, pulled from a database they already trust. Honestly, that is what I would tell you to plan for from the start.

After that, the tool stopped being mysterious. The pieces settled into the shape they had been in all along. RAG Engine takes the genuinely tedious parts of RAG, the storage, the embedding orchestration, the corpus bookkeeping, and makes them vanish. Then it hands you a box of good components and steps back. Hybrid search, reranking, a real layout parser, your choice of model, even your choice of vector store. All of it solid. None of it automatic.

It costs the way a box of parts costs. There is no single number. You pay to embed at import, and again to embed every query. You pay to store, to retrieve, to rerank, for the grounding, and for the model tokens, and if the retrieved context grows long enough you cross a line and the per-token price steps up. The bill is fair. It is also a row of small meters, and you should model one real query end to end before you promise anyone a budget.

The limits are tighter than the marketing implies. A few concurrent imports, modest per-minute ceilings. A large first ingestion is an exercise in patience and batching, not a button. And the platform keeps moving under you. While I worked, serverless storage arrived so you can skip even the database setup, a cross-corpus mode arrived so you can ask across several knowledge bases at once, and the whole family got rebranded at Google Cloud Next, which changed every documentation URL and absolutely nothing about the engine itself.

Use it when you live on Google Cloud, you want Gemini doing the grounding, and you want the plumbing gone but the knobs in reach. Turn on hybrid search and reranking on day one. Use the layout parser for anything with a table. Walk in knowing quality is something you build, not something you are handed.

Reconsider when your whole design leans on filtering, where the right answer depends on showing one user only their documents. Prove the schema ritual actually filters before you spend a sprint here. The engine will not stop you when it is wrong. It will just return everything.

That symmetry is the whole story. We adopt these systems to end the confident wrongness. The trick is remembering they can be confidently wrong about themselves too.