diff --git a/content/develop/clients/ruby/vecsets.md b/content/develop/clients/ruby/vecsets.md new file mode 100644 index 0000000000..164cf3592f --- /dev/null +++ b/content/develop/clients/ruby/vecsets.md @@ -0,0 +1,171 @@ +--- +categories: +- docs +- develop +- stack +- oss +- rs +- rc +- oss +- kubernetes +- clients +description: Index and query embeddings with Redis vector sets +linkTitle: Vector set embeddings +title: Vector set embeddings +weight: 40 +scope: example +relatedPages: +- /develop/clients/ruby/vecsearch +topics: +- vector sets +- vectors +--- + + + +A Redis [vector set](/content/develop/data-types/vector-sets/_index.md) lets +you store a set of unique keys, each with its own associated vector. +You can then retrieve keys from the set according to the similarity between +their stored vectors and a query vector that you specify. + +You can use vector sets to store any type of numeric vector but they are +particularly optimized to work with text embedding vectors (see +[Redis for AI](/content/develop/ai/_index.md) to learn more about text +embeddings). The example below shows how to use the +[`informers`](https://github.com/ankane/informers) gem to generate +vector embeddings and then store and retrieve them using a vector set with +`redis-rb`. + +## Initialize + +Start by installing `redis-rb` and `informers`. In your `Gemfile`: + +```ruby +gem 'redis', git: 'https://github.com/redis/redis-rb', branch: 'master' +gem 'informers' +``` + +Then require them: + +{{< clients-example set="home_vecsets" step="import" lang_filter="Ruby" description="Foundational: Import required libraries for vector sets, embeddings, and Redis operations" difficulty="beginner" >}} +{{< /clients-example >}} + +`informers` is a Ruby port of Hugging Face transformers that runs the +ONNX-exported [`all-MiniLM-L6-v2`](https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2) +model locally through `onnxruntime`. This model generates vectors with 384 +dimensions, regardless of the length of the input text, but note that the +input is truncated to 256 tokens (see +[Word piece tokenization](https://huggingface.co/learn/nlp-course/en/chapter6/6) +at the [Hugging Face](https://huggingface.co/) docs to learn more about the +way tokens are related to the original text). + +{{< clients-example set="home_vecsets" step="model" lang_filter="Ruby" description="Foundational: Initialize an embedding model to generate vector embeddings from text" difficulty="beginner" >}} +{{< /clients-example >}} + +## Create the data + +The example data is a hash with some brief descriptions of famous people: + +{{< clients-example set="home_vecsets" step="data" lang_filter="Ruby" description="Foundational: Define sample data with text descriptions for vector embedding and storage" difficulty="beginner" >}} +{{< /clients-example >}} + +## Add the data to a vector set + +The next step is to connect to Redis and add the data to a new vector set. + +The code below iterates through the hash and adds a corresponding element to +a vector set called `famousPeople` for each entry. + +Call the model with `pooling: 'mean', normalize: true` to generate the +embedding as an `Array` that matches the pooling and normalization +`sentence-transformers` applies by default. Pass the array directly as the +vector argument to [`vadd()`](/content/commands/vadd.md), along with the +`born` and `died` values as attribute data. You can access this attribute +data during a query or by using the +[`vgetattr()`](/content/commands/vgetattr.md) method. + +{{< clients-example set="home_vecsets" step="add_data" lang_filter="Ruby" description="Foundational: Add vector embeddings and attributes to a vector set using VADD command" difficulty="beginner" >}} +{{< /clients-example >}} + +## Query the vector set + +You can now query the data in the set. The basic approach is to call the +model again to generate another embedding vector for the query text, then +pass the query vector to [`vsim()`](/content/commands/vsim.md) to return +elements of the set, ranked in order of similarity to the query. + +Start with a simple query for "actors": + +{{< clients-example set="home_vecsets" step="basic_query" lang_filter="Ruby" description="Vector similarity search: Find semantically similar items in a vector set using VSIM command" difficulty="intermediate" >}} +{{< /clients-example >}} + +This returns the following list of elements: + +``` +'actors': ["Masako Natsume", "Chaim Topol", "Linus Pauling", +"Marie Fredriksson", "Maryam Mirzakhani", "Marie Curie", +"Freddie Mercury", "Paul Erdos"] +``` + +The first two people in the list are the two actors, as expected, but none of +the people from Linus Pauling onward was especially well-known for acting +(and there certainly isn't any information about that in the short +description text). As it stands, the search attempts to rank all the +elements in the set, based on the information contained in the embedding +model. You can use the `count` parameter of `vsim()` to limit the list of +elements to just the most relevant few items: + +{{< clients-example set="home_vecsets" step="limited_query" lang_filter="Ruby" description="Vector similarity search with limits: Restrict results to the top K most similar items using the count parameter" difficulty="intermediate" >}} +{{< /clients-example >}} + +The reason for using text embeddings rather than simple text search is that +the embeddings represent semantic information. This allows a query to find +elements with a similar meaning even if the text is different. For example, +the word "entertainer" doesn't appear in any of the descriptions but if you +use it as a query, the actors and musicians are ranked highest in the results +list: + +{{< clients-example set="home_vecsets" step="entertainer_query" lang_filter="Ruby" description="Semantic search: Leverage text embeddings to find semantically similar items even when exact keywords don't match" difficulty="intermediate" >}} +{{< /clients-example >}} + +Similarly, if you use "science" as a query, you get the following results: + +``` +'science': ["Marie Curie", "Linus Pauling", "Maryam Mirzakhani", +"Paul Erdos", "Marie Fredriksson", "Freddie Mercury", "Masako Natsume", +"Chaim Topol"] +``` + +The scientists are ranked highest but they are then followed by the +mathematicians. This seems reasonable given the connection between +mathematics and science. + +You can also use +[filter expressions](/content/develop/data-types/vector-sets/filtered-search.md) +with `vsim()` to restrict the search further. For example, repeat the +"science" query, but this time limit the results to people who died before +the year 2000: + +{{< clients-example set="home_vecsets" step="filtered_query" lang_filter="Ruby" description="Filtered vector search: Combine vector similarity with attribute filters to narrow results based on metadata conditions" difficulty="advanced" >}} +{{< /clients-example >}} + +Note that the boolean filter expression is applied to items in the list +before the vector distance calculation is performed. Items that don't pass +the filter test are removed from the results completely, rather than just +reduced in rank. This can help to improve the performance of the search +because there is no need to calculate the vector distance for elements that +have already been filtered out of the search. + +## More information + +See the [vector sets](/content/develop/data-types/vector-sets/_index.md) +docs for more information and code examples. See the +[Redis for AI](/content/develop/ai/_index.md) section for more details about +text embeddings and other AI techniques you can use with Redis. + +You may also be interested in +[vector search](/content/develop/clients/ruby/vecsearch.md). This is a +feature of [Redis Search](/content/develop/ai/search-and-query/_index.md) +that lets you retrieve [JSON](/content/develop/data-types/json/_index.md) +and [hash](/content/develop/data-types/hashes.md) documents based on vector +data stored in their fields. diff --git a/local_examples/client-specific/ruby/home_vecsets.rb b/local_examples/client-specific/ruby/home_vecsets.rb new file mode 100644 index 0000000000..4e1001ead8 --- /dev/null +++ b/local_examples/client-specific/ruby/home_vecsets.rb @@ -0,0 +1,187 @@ +# EXAMPLE: home_vecsets +# STEP_START import +require 'redis' +require 'informers' +# STEP_END + +# REMOVE_START +def assert_equal(expected, actual) + raise "Expected #{expected.inspect}, got #{actual.inspect}" unless actual == expected +end +# REMOVE_END + +# STEP_START model +# `informers` is a Ruby port of Hugging Face transformers that runs the +# ONNX-exported `all-MiniLM-L6-v2` encoder locally through `onnxruntime`. +# `Informers.pipeline("embedding", ...)` returns a callable that maps a +# string to a 384-element `Array`. +model = Informers.pipeline('embedding', 'sentence-transformers/all-MiniLM-L6-v2') +# STEP_END + +# STEP_START data +people_data = { + 'Marie Curie' => { + born: 1867, died: 1934, + description: 'Polish-French chemist and physicist. The only person ' \ + 'ever to win two Nobel prizes for two different sciences.' + }, + 'Linus Pauling' => { + born: 1901, died: 1994, + description: 'American chemist and peace activist. One of only two ' \ + 'people to win two Nobel prizes in different fields ' \ + '(chemistry and peace).' + }, + 'Freddie Mercury' => { + born: 1946, died: 1991, + description: 'British musician, best known as the lead singer of the ' \ + 'rock band Queen.' + }, + 'Marie Fredriksson' => { + born: 1958, died: 2019, + description: 'Swedish multi-instrumentalist, mainly known as the lead ' \ + 'singer and keyboardist of the band Roxette.' + }, + 'Paul Erdos' => { + born: 1913, died: 1996, + description: 'Hungarian mathematician, known for his eccentric ' \ + 'personality almost as much as his contributions to many different ' \ + 'fields of mathematics.' + }, + 'Maryam Mirzakhani' => { + born: 1977, died: 2017, + description: 'Iranian mathematician. The first woman ever to win the ' \ + 'Fields medal for her contributions to mathematics.' + }, + 'Masako Natsume' => { + born: 1957, died: 1985, + description: 'Japanese actress. She was very famous in Japan but was ' \ + 'primarily known elsewhere in the world for her portrayal of ' \ + 'Tripitaka in the TV series Monkey.' + }, + 'Chaim Topol' => { + born: 1935, died: 2023, + description: "Israeli actor and singer, usually credited simply as " \ + "'Topol'. He was best known for his many appearances as Tevye in " \ + 'the musical Fiddler on the Roof.' + } +} +# STEP_END + +# STEP_START connect +r = Redis.new +# STEP_END + +# REMOVE_START +r.del('famousPeople') +# REMOVE_END + +# STEP_START add_data +people_data.each do |name, details| + embedding = model.(details[:description], pooling: 'mean', normalize: true) + + r.vadd( + 'famousPeople', + embedding, + name, + attributes: { born: details[:born], died: details[:died] } + ) +end +# STEP_END + +# STEP_START basic_query +query_value = 'actors' + +actors_results = r.vsim( + 'famousPeople', + vector: model.(query_value, pooling: 'mean', normalize: true) +) + +puts "'actors': #{actors_results}" +# >>> 'actors': ["Masako Natsume", "Chaim Topol", "Linus Pauling", +# "Marie Fredriksson", "Maryam Mirzakhani", "Marie Curie", +# "Freddie Mercury", "Paul Erdos"] +# REMOVE_START +assert_equal( + ['Masako Natsume', 'Chaim Topol', 'Linus Pauling', 'Marie Fredriksson', + 'Maryam Mirzakhani', 'Marie Curie', 'Freddie Mercury', 'Paul Erdos'], + actors_results +) +# REMOVE_END +# STEP_END + +# STEP_START limited_query +query_value = 'actors' + +two_actors_results = r.vsim( + 'famousPeople', + vector: model.(query_value, pooling: 'mean', normalize: true), + count: 2 +) + +puts "'actors (2)': #{two_actors_results}" +# >>> 'actors (2)': ["Masako Natsume", "Chaim Topol"] +# REMOVE_START +assert_equal(['Masako Natsume', 'Chaim Topol'], two_actors_results) +# REMOVE_END +# STEP_END + +# STEP_START entertainer_query +query_value = 'entertainer' + +entertainer_results = r.vsim( + 'famousPeople', + vector: model.(query_value, pooling: 'mean', normalize: true) +) + +puts "'entertainer': #{entertainer_results}" +# >>> 'entertainer': ["Chaim Topol", "Freddie Mercury", +# "Marie Fredriksson", "Linus Pauling", "Masako Natsume", "Paul Erdos", +# "Maryam Mirzakhani", "Marie Curie"] +# REMOVE_START +assert_equal( + ['Chaim Topol', 'Freddie Mercury', 'Marie Fredriksson', 'Linus Pauling', + 'Masako Natsume', 'Paul Erdos', 'Maryam Mirzakhani', 'Marie Curie'], + entertainer_results +) +# REMOVE_END +# STEP_END + +query_value = 'science' + +science_results = r.vsim( + 'famousPeople', + vector: model.(query_value, pooling: 'mean', normalize: true) +) + +puts "'science': #{science_results}" +# >>> 'science': ["Marie Curie", "Linus Pauling", "Maryam Mirzakhani", +# "Paul Erdos", "Marie Fredriksson", "Freddie Mercury", "Masako Natsume", +# "Chaim Topol"] +# REMOVE_START +assert_equal( + ['Marie Curie', 'Linus Pauling', 'Maryam Mirzakhani', 'Paul Erdos', + 'Marie Fredriksson', 'Freddie Mercury', 'Masako Natsume', 'Chaim Topol'], + science_results +) +# REMOVE_END + +# STEP_START filtered_query +query_value = 'science' + +science2000_results = r.vsim( + 'famousPeople', + vector: model.(query_value, pooling: 'mean', normalize: true), + filter: '.died < 2000' +) + +puts "'science2000': #{science2000_results}" +# >>> 'science2000': ["Marie Curie", "Linus Pauling", "Paul Erdos", +# "Freddie Mercury", "Masako Natsume"] +# REMOVE_START +assert_equal( + ['Marie Curie', 'Linus Pauling', 'Paul Erdos', 'Freddie Mercury', + 'Masako Natsume'], + science2000_results +) +# REMOVE_END +# STEP_END diff --git a/local_examples/vecset_tutorial/ruby/dt_vec_set.rb b/local_examples/vecset_tutorial/ruby/dt_vec_set.rb new file mode 100644 index 0000000000..8ce5a31f41 --- /dev/null +++ b/local_examples/vecset_tutorial/ruby/dt_vec_set.rb @@ -0,0 +1,281 @@ +# EXAMPLE: vecset_tutorial +# HIDE_START +require 'redis' + +r = Redis.new +# HIDE_END + +# REMOVE_START +def assert_equal(expected, actual) + raise "Expected #{expected.inspect}, got #{actual.inspect}" unless actual == expected +end + +def assert(condition) + raise 'Assertion failed' unless condition +end + +r.del( + 'points', 'quantSetQ8', 'quantSetNoQ', + 'quantSetBin', 'setNotReduced', 'setReduced' +) +# REMOVE_END + +# STEP_START vadd +res1 = r.vadd('points', [1.0, 1.0], 'pt:A') +puts res1 # >>> true + +res2 = r.vadd('points', [-1.0, -1.0], 'pt:B') +puts res2 # >>> true + +res3 = r.vadd('points', [-1.0, 1.0], 'pt:C') +puts res3 # >>> true + +res4 = r.vadd('points', [1.0, -1.0], 'pt:D') +puts res4 # >>> true + +res5 = r.vadd('points', [1.0, 0], 'pt:E') +puts res5 # >>> true + +res6 = r.type('points') +puts res6 # >>> vectorset +# STEP_END +# REMOVE_START +assert_equal(true, res1) +assert_equal(true, res2) +assert_equal(true, res3) +assert_equal(true, res4) +assert_equal(true, res5) + +assert_equal('vectorset', res6) +# REMOVE_END + +# STEP_START vcardvdim +res7 = r.vcard('points') +puts res7 # >>> 5 + +res8 = r.vdim('points') +puts res8 # >>> 2 +# STEP_END +# REMOVE_START +assert_equal(5, res7) +assert_equal(2, res8) +# REMOVE_END + +# STEP_START vemb +res9 = r.vemb('points', 'pt:A') +p res9 # >>> [0.9999999403953552, 0.9999999403953552] + +res10 = r.vemb('points', 'pt:B') +p res10 # >>> [-0.9999999403953552, -0.9999999403953552] + +res11 = r.vemb('points', 'pt:C') +p res11 # >>> [-0.9999999403953552, 0.9999999403953552] + +res12 = r.vemb('points', 'pt:D') +p res12 # >>> [0.9999999403953552, -0.9999999403953552] + +res13 = r.vemb('points', 'pt:E') +p res13 # >>> [1.0, 0.0] +# STEP_END +# REMOVE_START +assert(1 - res9[0] < 0.001) +assert(1 - res9[1] < 0.001) +assert(1 + res10[0] < 0.001) +assert(1 + res10[1] < 0.001) +assert(1 + res11[0] < 0.001) +assert(1 - res11[1] < 0.001) +assert(1 - res12[0] < 0.001) +assert(1 + res12[1] < 0.001) +assert_equal([1.0, 0.0], res13) +# REMOVE_END + +# STEP_START attr +res14 = r.vsetattr('points', 'pt:A', { + 'name' => 'Point A', + 'description' => 'First point added' +}) +puts res14 # >>> true + +res15 = r.vgetattr('points', 'pt:A') +puts res15 +# >>> {"name"=>"Point A", "description"=>"First point added"} + +res16 = r.vsetattr('points', 'pt:A', '') +puts res16 # >>> true + +res17 = r.vgetattr('points', 'pt:A') +puts res17.inspect # >>> nil +# STEP_END +# REMOVE_START +assert_equal(true, res14) +assert_equal({ 'name' => 'Point A', 'description' => 'First point added' }, res15) +assert_equal(true, res16) +assert_equal(nil, res17) +# REMOVE_END + +# STEP_START vrem +res18 = r.vadd('points', [0, 0], 'pt:F') +puts res18 # >>> true + +res19 = r.vcard('points') +puts res19 # >>> 6 + +res20 = r.vrem('points', 'pt:F') +puts res20 # >>> true + +res21 = r.vcard('points') +puts res21 # >>> 5 +# STEP_END +# REMOVE_START +assert_equal(true, res18) +assert_equal(6, res19) +assert_equal(true, res20) +assert_equal(5, res21) +# REMOVE_END + +# STEP_START vsim_basic +res22 = r.vsim('points', vector: [0.9, 0.1]) +p res22 +# >>> ["pt:E", "pt:A", "pt:D", "pt:C", "pt:B"] +# STEP_END +# REMOVE_START +assert_equal(['pt:E', 'pt:A', 'pt:D', 'pt:C', 'pt:B'], res22) +# REMOVE_END + +# STEP_START vsim_options +res23 = r.vsim( + 'points', element: 'pt:A', + with_scores: true, + count: 4 +) +puts res23 +# >>> {"pt:A"=>1.0, "pt:E"=>0.8535534143447876, "pt:D"=>0.5, "pt:C"=>0.5} +# STEP_END +# REMOVE_START +assert_equal(1.0, res23['pt:A']) +assert_equal(0.5, res23['pt:D']) +assert_equal(0.5, res23['pt:C']) +assert(res23['pt:E'] - 0.85 < 0.005) +# REMOVE_END + +# STEP_START vsim_filter +res24 = r.vsetattr('points', 'pt:A', { + 'size' => 'large', + 'price' => 18.99 +}) +puts res24 # >>> true + +res25 = r.vsetattr('points', 'pt:B', { + 'size' => 'large', + 'price' => 35.99 +}) +puts res25 # >>> true + +res26 = r.vsetattr('points', 'pt:C', { + 'size' => 'large', + 'price' => 25.99 +}) +puts res26 # >>> true + +res27 = r.vsetattr('points', 'pt:D', { + 'size' => 'small', + 'price' => 21.00 +}) +puts res27 # >>> true + +res28 = r.vsetattr('points', 'pt:E', { + 'size' => 'small', + 'price' => 17.75 +}) +puts res28 # >>> true + +# Return elements in order of distance from point A whose +# `size` attribute is `large`. +res29 = r.vsim( + 'points', element: 'pt:A', + filter: '.size == "large"' +) +p res29 # >>> ["pt:A", "pt:C", "pt:B"] + +# Return elements in order of distance from point A whose size is +# `large` and whose price is greater than 20.00. +res30 = r.vsim( + 'points', element: 'pt:A', + filter: '.size == "large" && .price > 20.00' +) +p res30 # >>> ["pt:C", "pt:B"] +# STEP_END +# REMOVE_START +assert_equal(true, res24) +assert_equal(true, res25) +assert_equal(true, res26) +assert_equal(true, res27) +assert_equal(true, res28) + +assert_equal(['pt:A', 'pt:C', 'pt:B'], res29) +assert_equal(['pt:C', 'pt:B'], res30) +# REMOVE_END + +# STEP_START add_quant +res31 = r.vadd( + 'quantSetQ8', [1.262185, 1.958231], + 'quantElement', + quantization: :q8 +) +puts res31 # >>> true + +res32 = r.vemb('quantSetQ8', 'quantElement') +puts "Q8: #{res32}" +# >>> Q8: [1.2643694877624512, 1.958230972290039] + +res33 = r.vadd( + 'quantSetNoQ', [1.262185, 1.958231], + 'quantElement', + quantization: :noquant +) +puts res33 # >>> true + +res34 = r.vemb('quantSetNoQ', 'quantElement') +puts "NOQUANT: #{res34}" +# >>> NOQUANT: [1.262184977531433, 1.958230972290039] + +res35 = r.vadd( + 'quantSetBin', [1.262185, 1.958231], + 'quantElement', + quantization: :bin +) +puts res35 # >>> true + +res36 = r.vemb('quantSetBin', 'quantElement') +puts "BIN: #{res36}" +# >>> BIN: [1.0, 1.0] +# STEP_END +# REMOVE_START +assert_equal(true, res31) +# REMOVE_END + +# STEP_START add_reduce +# Create a list of 300 arbitrary values. +values = (0...300).map { |x| x / 299.0 } + +res37 = r.vadd( + 'setNotReduced', + values, + 'element' +) +puts res37 # >>> true + +res38 = r.vdim('setNotReduced') +puts res38 # >>> 300 + +res39 = r.vadd( + 'setReduced', + values, + 'element', + reduce: 100 +) +puts res39 # >>> true + +res40 = r.vdim('setReduced') +puts res40 # >>> 100 +# STEP_END