Search & Indexing

Search & Indexing#

Vector search in Redis works differently from traditional databases. Understanding the underlying model helps you design better schemas and write more effective queries.

How Redis Indexes Work#

Redis search indexes are secondary structures that sit alongside your data. When you create an index, you’re telling Redis: “Watch all keys that match this prefix, and build a searchable structure from these specific fields.”

The index doesn’t store your data—it references it. Your documents live as Redis Hash or JSON objects, and the index maintains pointers and optimized structures for fast lookup. When you write a document, Redis automatically updates the index. When you delete a document, the index entry is removed.

This design means searches are fast (the index is optimized for queries) while writes remain efficient (only the affected index entries are updated). It also means you can have multiple indexes over the same data with different field configurations.

Field Types and Their Purpose#

Each field type serves a different search use case.

Text fields enable full-text search. Redis tokenizes the content, applies stemming (so “running” matches “run”), and builds an inverted index. Text search finds documents containing specific words or phrases, ranked by relevance.

Tag fields are for exact-match filtering. Unlike text fields, tags are not tokenized or stemmed. A tag value is treated as an atomic unit. This is ideal for categories, statuses, IDs, and other discrete values where you want exact matches.

Numeric fields support range queries and sorting. You can filter for values greater than, less than, or between bounds. Numeric fields are also used for sorting results.

Geo fields enable location-based queries. You can find documents within a radius of a point or within a bounding box.

Vector fields enable similarity search. Each document has an embedding vector, and queries find documents whose vectors are closest to a query vector. This is the foundation of semantic search.

Vector Indexing Algorithms#

Vector similarity search requires specialized data structures. Redis offers three algorithms, each with different trade-offs.

Flat indexing performs exact nearest-neighbor search by comparing the query vector against every indexed vector. This guarantees finding the true closest matches, but search time grows linearly with dataset size. Use flat indexing for small datasets (under ~100K vectors) where exact results matter.

HNSW (Hierarchical Navigable Small World) is an approximate algorithm that builds a multi-layer graph structure. Queries navigate this graph to find approximate nearest neighbors in logarithmic time. HNSW typically achieves 95-99% recall (meaning it finds 95-99% of the true nearest neighbors) while being orders of magnitude faster than flat search on large datasets. This is the default choice for most applications.

SVS (Scalable Vector Search) is designed for very large datasets with memory constraints. It supports vector compression techniques that reduce memory footprint at the cost of some recall. SVS is useful when you have millions of vectors and memory is a limiting factor.

The algorithm choice is made at index creation and cannot be changed without rebuilding the index.

Distance Metrics#

When comparing vectors, you need a way to measure how “close” two vectors are. Redis supports three distance metrics.

Cosine distance measures the angle between vectors, ignoring their magnitude. Two vectors pointing in the same direction have distance 0; opposite directions have distance 2. Cosine is widely used because most embedding models produce vectors where direction encodes meaning and magnitude is less important. Similarity equals 1 minus distance.

Euclidean distance (L2) measures the straight-line distance between vector endpoints. Unlike cosine, it considers magnitude. Euclidean distance ranges from 0 to infinity.

Inner product (IP) is the dot product of two vectors. It combines both direction and magnitude. When vectors are normalized (magnitude 1), inner product equals cosine similarity. Inner product can be negative and ranges from negative infinity to positive infinity.

Choose your metric based on how your embedding model was trained. Most text embedding models use cosine.

Storage: Hash vs JSON#

Redis offers two storage formats for documents.

Hash storage is a flat key-value structure where each field is a top-level key. It’s simple, fast, and works well when your documents don’t have nested structures. Field names in your schema map directly to hash field names.

JSON storage supports nested documents. You can store complex objects and use JSONPath expressions to index nested fields. This is useful when your data is naturally hierarchical or when you want to store the original document structure without flattening it.

The choice affects how you structure data and how you reference fields in schemas. Hash is simpler; JSON is more flexible.

Index Lifecycle#

Indexes have a straightforward lifecycle: create, use, and eventually delete.

Creating an index registers the schema with Redis. The create() method sends the schema definition to Redis, which builds the necessary data structures. If an index with the same name exists, you can choose to overwrite it (optionally dropping existing data) or raise an error.

Checking existence with exists() tells you whether an index is registered in Redis. This is useful before creating (to avoid errors) or before querying (to ensure the index is ready).

Connecting to existing indexes is possible with from_existing(). If an index was created elsewhere—by another application, a previous deployment, or the CLI—you can connect to it by name. RedisVL fetches the index metadata from Redis and reconstructs the schema automatically.

# Connect to an index that already exists in Redis
index = SearchIndex.from_existing("my-index", redis_url="redis://localhost:6379")

Deleting an index with delete() removes the index definition from Redis. By default, this also deletes all documents associated with the index. Pass drop=False to keep the documents while removing only the index structure.

Clearing data with clear() removes all documents from the index without deleting the index itself. The schema remains intact, ready for new data.

Bulk Delete and Update#

Redis has no server-side “delete/update by query”, so mutating many documents traditionally means scanning for keys and issuing writes yourself. RedisVL wraps that pattern behind two filter-driven methods (available on both SearchIndex and AsyncSearchIndex).

Deleting by filter resolves every document matching a filter expression and removes it with non-blocking UNLINK, in batches:

from redisvl.query.filter import Tag, Num

# Remove every archived document from before 2020
index.drop_by_filter((Tag("status") == "archived") & (Num("year") < 2020))

Updating by filter applies a partial field update to every match. Fields you don’t mention are left untouched—hash fields are written with HSET, and JSON documents are merged at the root with JSON.MERGE (RFC 7396: nested objects merge recursively, arrays are replaced wholesale, and a None value deletes that path):

# Mark all draft documents as published, leaving other fields intact
index.update_by_filter(Tag("status") == "draft", {"status": "published"})

Two caveats for update_by_filter values:

  • No schema validation. Unlike load(), values are written as-is—pre-encode vectors/bytes and format numerics as your schema expects, or you may corrupt query results.

  • JSON keys must match the document layout, not the schema field name. Values merge at the document root ($). If a field is indexed at a nested path (e.g. $.metadata.status), pass the correspondingly nested mapping ({"metadata": {"status": "published"}}); passing the flat field name writes the wrong path and leaves the indexed field unchanged. Only static values are supported (no callable/expression transforms).

Both methods share the same safety-oriented options and return a BulkResult:

  • The return value carries matched (documents matching the filter), processed (documents actually affected), completed (False if a delete stopped early at its runaway backstop; always True for update), and dry_run. Read the fields explicitly—result.processed, result.completed, etc.

  • dry_run=True reports how many documents would be affected—via a count query—without changing anything.

  • A match-all filter (empty, "*", or None) is refused unless you pass allow_all=True; use clear() when you intentionally want to empty the index. This guard catches the canonical match-all forms only—it is a convenience backstop, not a security control.

  • on_progress(processed, matched) is called after each write batch for observability on large operations (it runs synchronously—don’t pass a coroutine—and raising from it aborts the run). Note update_by_filter resolves all matching keys before it writes, so progress callbacks begin only once the write phase starts. batch_size tunes how many documents are processed per round-trip.

The related drop_documents() and drop_keys() helpers delete by document ID or full Redis key and batch large inputs. On Redis Cluster, drop_keys() unlinks per-key so it works across hash slots, while drop_documents() requires the target keys to share a hash tag and raises ValueError otherwise.

Durability and partial failure. Because Redis has no server-side delete/update-by-query, these operations run as a series of batched writes rather than a single transaction—they are not atomic across the match set and there is no rollback. The unit of atomicity is a single document: each key is deleted with one UNLINK, and each document is updated with one HSET or JSON.MERGE, so you never get a half-deleted key or a half-updated document. But batches are applied incrementally, so a crash or connection error mid-run leaves some documents changed and the rest untouched.

The intended recovery is simply to re-run the same call: both operations are idempotent, so a repeat pass removes (or re-sets) only what still needs it and converges on the desired state. There is no built-in checkpoint or resume token—on_progress reports live progress but is not a restart point. update_by_filter resolves keys before it writes, so a document can be deleted by another client in between; each write is applied only if the key still exists (atomically), so such a document is skipped, not recreated as a partial document—it just isn’t counted in processed, which is why processed can be less than matched under concurrent deletion. Note this guard is existence-based, not identity-based: if a key is deleted and a different document is recreated at the same key mid-run, the update can land on that new document. Running against a quiescent partition (below) avoids all of these concurrency cases.

Operational considerations at scale. These are single-threaded, foreground operations that issue one round-trip per batch; a very large match set is a long-running job. Keep the following in mind for production use:

  • Live-traffic impact. Every delete/update also updates the search index synchronously. Running a large bulk job against a node that is serving queries competes for CPU and reindex bandwidth and can raise query latency—prefer off-peak windows, or narrow the filter and run in waves.

  • Memory. drop_by_filter holds only one batch of keys at a time (O(batch_size)). update_by_filter must resolve all matching keys before it can write (an open aggregation cursor can’t be read while the index is being written), so its client memory grows with the match count—roughly the total size of the matched keys. For very large match sets, partition the filter (see below) rather than updating everything in one call.

  • Redis Cluster. Cross-slot multi-key commands aren’t allowed, so writes are issued per key (no pipelining across slots)—expect this to be slower on Cluster for large match sets.

  • Resumability. There is no checkpoint; recovery is a full re-run. For very large corpora, partition the filter (e.g. by a tag or numeric range) and process partition-by-partition. This is the recommended pattern at scale: it bounds memory and per-run time, keeps each call independently retryable, and lets you spread load across off-peak windows.

Data Validation#

RedisVL can validate data against your schema before loading it to Redis. This catches type mismatches, missing required fields, and invalid values early—before they cause problems in production.

Enable validation by setting validate_on_load=True when creating the index:

index = SearchIndex.from_dict(schema, redis_url="redis://localhost:6379", validate_on_load=True)

When validation is enabled, each object is checked against the schema during load(). If validation fails, a SchemaValidationError is raised with details about which field failed and why.

Validation adds overhead, so it’s typically enabled during development and testing, then disabled in production once you’re confident in your data pipeline.

Schema Evolution#

Redis doesn’t support modifying an existing index schema. Once an index is created, its field definitions are fixed.

To change a schema, you create a new index with the updated configuration, reindex your data into it, update your application to use the new index, and then delete the old index. This pattern—create new, migrate, switch, drop old—is the standard approach for schema changes in production.

Planning your schema carefully upfront reduces the need for migrations, but the capability exists when requirements evolve.

RedisVL now includes a dedicated migration workflow for this lifecycle:

  • drop_recreate for document-preserving rebuilds, including vector quantization (float32 → float16)

That means schema evolution is no longer only a manual operational pattern. It is also a product surface in RedisVL with a planner, CLI, and validation artifacts.

Related concepts: Field Attributes explains how to configure field options like sortable and index_missing. Query Types covers the different query types available. Index Migrations explains migration modes, supported changes, and architecture.

Learn more: Getting Started walks through building your first index. Choose a Storage Type compares storage options in depth. Query and Filter Data covers query composition. Migrate an Index shows how to use the migration CLI in practice.