Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add vector store endpoints #472

Merged
merged 16 commits into from
Jun 10, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
162 changes: 159 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -39,7 +39,12 @@ Stream text with GPT-4, transcribe and translate audio with Whisper, or create i
- [Embeddings](#embeddings)
- [Batches](#batches)
- [Files](#files)
- [For fine-tuning purposes](#for-fine-tuning-purposes)
- [For assistant purposes](#for-assistant-purposes)
- [Finetunes](#finetunes)
- [Vector Stores](#vector-stores)
- [Vector Store Files](#vector-store-files)
- [Vector Store File Batches](#vector-store-file-batches)
- [Assistants](#assistants)
- [Threads and Messages](#threads-and-messages)
- [Runs](#runs)
Expand Down Expand Up @@ -585,7 +590,7 @@ These files are in JSONL format, with each line representing the output or error
If a request fails with a non-HTTP error, the error object will contain more information about the cause of the failure.

### Files

#### For fine-tuning purposes
Put your data in a `.jsonl` file like this:

```json
Expand All @@ -603,6 +608,24 @@ client.files.content(id: "file-123")
client.files.delete(id: "file-123")
```

#### For assistant purposes

You can send a file path:

```ruby
client.files.upload(parameters: { file: "path/to/file.pdf", purpose: "assistants" })
```

or a File object

```ruby
my_file = File.open("path/to/file.pdf", "rb")
client.files.upload(parameters: { file: my_file, purpose: "assistants" })
```


See supported file types on [API documentation](https://platform.openai.com/docs/assistants/tools/file-search/supported-files).

### Finetunes

Upload your fine-tuning data in a `.jsonl` file as above and get its ID:
Expand Down Expand Up @@ -655,6 +678,135 @@ You can also capture the events for a job:
client.finetunes.list_events(id: fine_tune_id)
```

### Vector Stores
Vector Store objects give the File Search tool the ability to search your files.

You can create a new vector store:

```ruby
response = client.vector_stores.create(
parameters: {
name: "my vector store",
file_ids: ["file-abc123", "file-def456"]
}
)

vector_store_id = response["id"]
```

Given a `vector_store_id` you can `retrieve` the current field values:

```ruby
client.vector_stores.retrieve(id: vector_store_id)
```

You can get a `list` of all vector stores currently available under the organization:

```ruby
client.vector_stores.list
```

You can modify an existing vector store, except for the `file_ids`:

```ruby
response = client.vector_stores.modify(
id: vector_store_id,
parameters: {
name: "Modified Test Vector Store",
}
)
```

You can delete vector stores:

```ruby
client.vector_stores.delete(id: vector_store_id)
```

### Vector Store Files
Vector store files represent files inside a vector store.

You can create a new vector store file by attaching a File to a vector store.

```ruby
response = client.vector_store_files.create(
vector_store_id: "vector-store-abc123",
parameters: {
file_id: "file-abc123"
}
)

vector_store_file_id = response["id"]
```

Given a `vector_store_file_id` you can `retrieve` the current field values:

```ruby
client.vector_store_files.retrieve(
vector_store_id: "vector-store-abc123",
id: vector_store_file_id
)
```

You can get a `list` of all vector store files currently available under the vector store:

```ruby
client.vector_store_files.list(vector_store_id: "vector-store-abc123")
```

You can delete a vector store file:

```ruby
client.vector_store_files.delete(
vector_store_id: "vector-store-abc123",
id: vector_store_file_id
)
```
Note: This will remove the file from the vector store but the file itself will not be deleted. To delete the file, use the delete file endpoint.

### Vector Store File Batches
Vector store file batches represent operations to add multiple files to a vector store.

You can create a new vector store file batch by attaching multiple Files to a vector store.

```ruby
response = client.vector_store_file_batches.create(
vector_store_id: "vector-store-abc123",
parameters: {
file_ids: ["file-abc123", "file-def456"]
}
)

file_batch_id = response["id"]
```

Given a `file_batch_id` you can `retrieve` the current field values:

```ruby
client.vector_store_file_batches.retrieve(
vector_store_id: "vector-store-abc123",
id: file_batch_id
)
```

You can get a `list` of all vector store files in a batch currently available under the vector store:

```ruby
client.vector_store_file_batches.list(
vector_store_id: "vector-store-abc123",
id: file_batch_id
)
```

You can cancel a vector store file batch (This attempts to cancel the processing of files in this batch as soon as possible):

```ruby
client.vector_store_file_batches.cancel(
vector_store_id: "vector-store-abc123",
id: file_batch_id
)
```

### Assistants

Assistants are stateful actors that can have many conversations and use tools to perform tasks (see [Assistant Overview](https://platform.openai.com/docs/assistants/overview)).
Expand All @@ -670,10 +822,14 @@ response = client.assistants.create(
instructions: "You are a Ruby dev bot. When asked a question, write and run Ruby code to answer the question",
tools: [
{ type: "code_interpreter" },
{ type: "file_search" }
],
tool_resources: {
"code_interpreter": {
"file_ids": [] # See Files section above for how to upload files
code_interpreter: {
file_ids: [] # See Files section above for how to upload files
},
file_search: {
vector_store_ids: [] # See Vector Stores section above for how to add vector stores
}
},
"metadata": { my_internal_version_id: "1.0.0" }
Expand Down
3 changes: 3 additions & 0 deletions lib/openai.rb
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,9 @@
require_relative "openai/messages"
require_relative "openai/runs"
require_relative "openai/run_steps"
require_relative "openai/vector_stores"
require_relative "openai/vector_store_files"
require_relative "openai/vector_store_file_batches"
require_relative "openai/audio"
require_relative "openai/version"
require_relative "openai/batches"
Expand Down
12 changes: 12 additions & 0 deletions lib/openai/client.rb
Original file line number Diff line number Diff line change
Expand Up @@ -78,6 +78,18 @@ def run_steps
@run_steps ||= OpenAI::RunSteps.new(client: self)
end

def vector_stores
@vector_stores ||= OpenAI::VectorStores.new(client: self)
end

def vector_store_files
@vector_store_files ||= OpenAI::VectorStoreFiles.new(client: self)
end

def vector_store_file_batches
@vector_store_file_batches ||= OpenAI::VectorStoreFileBatches.new(client: self)
end

def batches
@batches ||= OpenAI::Batches.new(client: self)
end
Expand Down
29 changes: 29 additions & 0 deletions lib/openai/vector_store_file_batches.rb
Original file line number Diff line number Diff line change
@@ -0,0 +1,29 @@
module OpenAI
class VectorStoreFileBatches
def initialize(client:)
@client = client.beta(assistants: OpenAI::Assistants::BETA_VERSION)
end

def list(vector_store_id:, id:, parameters: {})
@client.get(
path: "/vector_stores/#{vector_store_id}/file_batches/#{id}/files",
parameters: parameters
)
end

def retrieve(vector_store_id:, id:)
@client.get(path: "/vector_stores/#{vector_store_id}/file_batches/#{id}")
end

def create(vector_store_id:, parameters: {})
@client.json_post(
path: "/vector_stores/#{vector_store_id}/file_batches",
parameters: parameters
)
end

def cancel(vector_store_id:, id:)
@client.post(path: "/vector_stores/#{vector_store_id}/file_batches/#{id}/cancel")
end
end
end
23 changes: 23 additions & 0 deletions lib/openai/vector_store_files.rb
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
module OpenAI
class VectorStoreFiles
def initialize(client:)
@client = client.beta(assistants: OpenAI::Assistants::BETA_VERSION)
end

def list(vector_store_id:, parameters: {})
@client.get(path: "/vector_stores/#{vector_store_id}/files", parameters: parameters)
end

def retrieve(vector_store_id:, id:)
@client.get(path: "/vector_stores/#{vector_store_id}/files/#{id}")
end

def create(vector_store_id:, parameters: {})
@client.json_post(path: "/vector_stores/#{vector_store_id}/files", parameters: parameters)
end

def delete(vector_store_id:, id:)
@client.delete(path: "/vector_stores/#{vector_store_id}/files/#{id}")
end
end
end
27 changes: 27 additions & 0 deletions lib/openai/vector_stores.rb
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
module OpenAI
class VectorStores
def initialize(client:)
@client = client.beta(assistants: OpenAI::Assistants::BETA_VERSION)
end

def list(parameters: {})
@client.get(path: "/vector_stores", parameters: parameters)
end

def retrieve(id:)
@client.get(path: "/vector_stores/#{id}")
end

def create(parameters: {})
@client.json_post(path: "/vector_stores", parameters: parameters)
end

def modify(id:, parameters: {})
@client.json_post(path: "/vector_stores/#{id}", parameters: parameters)
end

def delete(id:)
@client.delete(path: "/vector_stores/#{id}")
end
end
end
79 changes: 79 additions & 0 deletions spec/fixtures/cassettes/vector_store_file_batches_cancel.yml

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.