Similarity Seach#

After fitting your database, you can perform similarity searches in two ways: Based on an existing index of your already included model or using new data.

Using existing index#

You can query itens that already have been inputed by their ids. The below example shows how to find the five most similar values for ids 0 and 1.

>>> results = j.similar(name, [0, 1], top_k=5)

To find the 20 most similar values for every id from [0, 99].

>>> ids = list(range(100))
>>> results = j.similar(name, ids, top_k=20)

Now, finding the 100 most similar values for every input value can be done like the example below.

>>> results = j.similar(name, data.index, top_k=100, batch_size=1024)

Using new data#

(All data should be in pandas.DataFrame or pandas.Series format)

Find the 100 most similar values for every new_data.

>>> results = j.similar(name, new_data, top_k=100, batch_size=1024)

The output will be a list of dictionaries with 'query_id' being the id of the value you want to find similars and 'results') a list with top_k dictionaries with the 'id' and the 'distance' between 'query_id' and 'id'.

[
    {
        'query_id': 0,
        'results':
        [
        {'id': 0, 'distance': 0.0},
        {'id': 3836, 'distance': 2.298321008682251},
        {'id': 9193, 'distance': 2.545339584350586},
        {'id': 832, 'distance': 2.5819168090820312},
        {'id': 6162, 'distance': 2.638622283935547},
        ...
        ]
    },
    ...,
    {
        'query_id': 9,
        'results':
        [
        {'id': 9, 'distance': 0.0},
        {'id': 54, 'distance': 5.262974262237549},
        {'id': 101, 'distance': 5.634262561798096},
        ...
        ]
    },
    ...
]

Note

The method similar has a default batch_size=16384, which will result in ceil(n_samples/batch_size) + 2 requests. We DON’T recommend changing the default value as it could reduce the performance of the API.

Output formating#

There are two possible output formats for the similarity search. You can change which format you wish to use by changing the parameter orient.

orient: “nested” or “flat”

Changes the output format. Default is “nested”.

Here are some examples for each of the possible formats bellow:

  • nested:

[
    {
        'query_id': 0,
        'results':
        [
        {'id': 0, 'distance': 0.0},
        {'id': 3836, 'distance': 2.298321008682251},
        {'id': 9193, 'distance': 2.545339584350586},
        {'id': 832, 'distance': 2.5819168090820312},
        {'id': 6162, 'distance': 2.638622283935547},
        ...
        ]
    },
    ...,
    {
        'query_id': 9,
        'results':
        [
        {'id': 9, 'distance': 0.0},
        {'id': 54, 'distance': 5.262974262237549},
        {'id': 101, 'distance': 5.634262561798096},
        ...
        ]
    },
    ...
]
  • flat:

[
    {'query_id': 0, 'id': 0, 'distance': 0.0},
    {'query_id': 0, 'id': 3836, 'distance': 2.298321008682251},
    {'query_id': 0, 'id': 9193, 'distance': 2.545339584350586},
    {'query_id': 0, 'id': 832, 'distance': 2.5819168090820312},
    {'query_id': 0, 'id': 6162, 'distance': 2.638622283935547},
    ...
    {'query_id': 9, 'id': 9, 'distance': 0.0},
    {'query_id': 9, 'id': 54, 'distance': 5.262974262237549},
    {'query_id': 9, 'id': 101, 'distance': 5.634262561798096},
    ...
]