Unlock 3x Storage Savings on your OpenAI Embeddings with Embaas Reduce

Introduction

We are happy to introduce our new product Embaas Reduce! The Reduce API helps to reduce the number of dimensions of your embeddings with minimal performance loss for downstream tasks. To get started, we offer the reduce API for OpenAI’s text-embedding-ada-002. You generate the embeddings with OpenAI’s API, send those to our Reduce API, and receive a new, embedding with 460 dimensions - more than 3x smaller than the original size.

Benefits of Embaas Reduce

We know that vector embeddings are currently the best method to turn unstructured data, such as documents, images, audio, or video, into a numerical representation. Neural networks learn to create vectors that maximise the amount of information extracted from the data given a chosen vector size.

For many models, the output embedding can be an over 500+ element array of 32-bit floating point numbers. OpenAI’s embeddings are high-dimensional, consisting of 1536-element vectors. While these high-dimensional embeddings are powerful, they can also be storage-intensive and computationally expensive.

Embaas Reduce takes these 1536-dimensional vectors and produces new 460-dimensional vectors that retain most of the semantic information, allowing for more efficient storage and faster computation. OpenAI’s embeddings also suffer from a problem called “dimension collapse” - this causes all vectors to be predicted in a similar space, which also explains why you never get cosine similarity below 0.70.

Here our Reduce API comes in; we take the 1536 dimension embedding and produce a new 460 dimension embedding, which preserves most of the semantic information and overcomes the dimension collapse issue.

Let’s look into the benchmarks

The to-go benchmark for embeddings is the Massive Text Embeddings Benchmark. We use a random subset of the benchmark to demonstrate that the reduced embeddings have minimal performance loss. Let’s look at the following table.

As you can see from the table above, the results are impressive. Even after reducing the dimensions from the original embeddings, Embaas Reduce maintains minimal performance loss across a diverse set of benchmark tasks. In fact, in some cases, it even outperforms the original embeddings, demonstrating its ability to preserve semantic information effectively while overcoming the dimension collapse issue.

These benchmark results clearly show that Embaas Reduce is a powerful solution for those looking to save money and reduce the dimensionality of their embeddings without sacrificing performance. Whether you're working on retrieval, clustering, or other downstream tasks, our Reduce API can significantly improve efficiency without compromising the quality of your results.

How to use Embaas Reduce

Using Embaas Reduce is straightforward and can greatly enhance the efficiency of your data processing workflows. Here's a step-by-step guide on how to integrate Embaas Reduce into your projects using Python. But before you start, you'll need to sign up for an Embaas account and get your API key. You can do that here.

Step 1: Generate Embeddings

1
2
3
4
5
6
7
8
9
10
11
12
13
import requests
import openai

openai.api_key = {OPENAI_API_KEY}

sentences = [
"Why in India do we not have one on one political debate as in USA?",
"Why cant we have a public debate between politicians in India like the one in US?",
"Can people on Quora stop India Pakistan debate? We are sick and tired seeing this everyday in bulk?"
]

embeddings = openai.Embedding.create(model="text-embedding-ada-002", input=sentences)
embeddings = [embedding.embedding for embedding in embeddings["data"]]

Step 2: Embaas Reduce

Next, you'll send the generated embeddings to Embaas Reduce API to reduce their dimensionality. This step significantly reduces the size of the embeddings while preserving most of the semantic information.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
import requests

# Define headers and data for the Embaas Reduce API request

headers = {'Content-Type': 'application/json', 'Authorization': f'Bearer {EMBAAS_API_KEY}'}
data = {
'model': 'text-embedding-ada-002',
'embeddings': embeddings
}

# Send a POST request to Embaas Reduce API

response = requests.post("https://api.embaas.io/v1/reduce/", json=data, headers=headers)
reduced_embeddings = response.json()
reduced_embeddings = [embedding["embedding"] for embedding in reduced_embeddings["data"]]

Step 3: Compare Cosine Similarity

Now let's compare the cosine similarity between the original embeddings and the reduced embeddings.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
from sklearn.metrics.pairwise import cosine_similarity

original_cosine_similarity = cosine_similarity(embeddings)[0]
reduced_cosine_similarity = cosine_similarity(reduced_embeddings)[0]

print(f'Similarity between query and positive passage: {original_cosine_similarity[1]} and negative passage: {original_cosine_similarity[2]}')
print(f'Similarity between query and positive passage: {reduced_cosine_similarity[1]} and negative passage: {reduced_cosine_similarity[2]}')

# Output 1:

# Similarity between query and positive passage: 0.956616428355283 and negative passage: 0.8343663037634148

# Output 2:

# Similarity between query and positive passage: 0.8809468945903628 and negative passage: 0.46931625115618464

In the provided code, we calculate and print the cosine similarity between query and positive/negative passages for both the original and reduced embeddings. You can observe how well Embaas Reduce preserves similarity relationships even better than the original embeddings.

Things to keep in mind

Presently, our Reduce API caters exclusively to OpenAI’s text-embedding-ada-002 model. We will add more models in the future based on demand. You can find the list of supported models here. And remember, with the transition to reduced embeddings, you might need slight tweaks in your vector database, such as modifying the Faiss index dimension.

Pricing

Starting your journey with Embaas Reduce won't break the bank. On registration, we gift you a $1 in free credit! Subsequent charges are a mere $0.0001 per embedding.

Conclusion

The digital age demands efficiency, and the Embaas Reduce API is designed to meet that need. With support for OpenAI's models, you don't have to choose between performance and reduced storage space. Start using Embaas Reduce to cost-effectively scale your projects while maintaining top-tier performance. Welcome to the future of embeddings, welcome to Embaas!

Find out more about Embaas Reduce here.
Share your thoughts! We'd love to hear from you.