Gradio Local Editor cookbook #826
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Draft
saqadri
force-pushed
the
pr826
branch
3 times, most recently
from
2f2f846
to
8ba46eb
Compare
This was referenced Jan 10, 2024
saqadri
force-pushed
the
pr826
branch
3 times, most recently
from
602e134
to
5eca3c6
Compare
The HF prompt schema has been updated in #850 to support a "model" property, which can specify the model to use for a given HF task. This change updates the model parsers to use this setting. Test Plan: Tested with the following prompt: ``` { "name": "image_caption", "input": { "attachments": [ { "data": "https://s3.amazonaws.com/lastmileai.aiconfig.public/uploads/2024_1_10_18_41_31/1742/bear_with_honey.png", "mime_type": "image/png" }, { "data": "https://s3.amazonaws.com/lastmileai.aiconfig.public/uploads/2024_1_10_18_41_31/7275/fox_in_forest.png", "mime_type": "image/png" } ] }, "metadata": { "model": { "name": "HuggingFaceImage2TextTransformer", "settings": { "max_new_tokens": 10, "model": "Salesforce/blip-image-captioning-base" } }, "parameters": {} }, "outputs": [ { "output_type": "execute_result", "execution_count": 0, "data": "a bear sitting on a rock eating honey", "metadata": {} }, { "output_type": "execute_result", "execution_count": 1, "data": "a red fox in the woods", "metadata": {} } ] }, ``` Validated the the model setting was respected and worked.
saqadri
requested review from
rholinshead,
suyoglastmileai,
Ankush-lastmile,
jonathanlastmileai and
rossdanlm
as code owners
rholinshead
reviewed
Jan 10, 2024
|
|
||
| ## Prompt IDE with LastMile AI | ||
|
|
||
| This cookbook is meant to demonstrate the capabilities of the AI Workbook editor. It can run inference against locally hosted or remote models from many inference providers, including Hugging Face, OpenAI and others. |
There was a problem hiding this comment.
Should "AI Workbook editor" be "local AIConfig Editor"?
|
|
||
| It supports text, image and audio model formats, allowing you to easily chain them together in a single notebook session! | ||
|
|
||
| With `aiconfig`, it lets you save the state in a single json config file which you can version control or share with others. |
There was a problem hiding this comment.
nit, "it lets you" --> "you can"
| **Instructions**: | ||
|
|
||
| - Clone https://github.com/lastmile-ai/aiconfig | ||
| - Go back to top-level directory: `cd <aiconfig>` |
There was a problem hiding this comment.
nit, just "Go", no "back"
Comment on lines
+52
to
+56
| ## TODO | ||
|
|
||
| - Publish new version of aiconfig_extension_hugging_face package | ||
| - Update huggingface.aiconfig.json with clean examples | ||
| - Add video demo |
Now that we have non-text input model parsers, we can see some issues in our parameterization.
1) ASR and Image-to-Text modelparsers should NOT be `ParameterizedModelParser` instances, since their inputs cannot be parameterized.
2) Parameter resolution logic shouldn't throw errors in the case where it's parsing prompts belonging to regular model parsers.
Test Plan:
Ran the translation prompt for this aiconfig:
```
prompts: [
{
"name": "image_caption",
"input": {
"attachments": [
{
"data": "https://s3.amazonaws.com/lastmileai.aiconfig.public/uploads/2024_1_10_18_41_31/1742/bear_with_honey.png",
"mime_type": "image/png"
},
{
"data": "https://s3.amazonaws.com/lastmileai.aiconfig.public/uploads/2024_1_10_18_41_31/7275/fox_in_forest.png",
"mime_type": "image/png"
}
]
},
"metadata": {
"model": {
"name": "HuggingFaceImage2TextTransformer",
"settings": {
"max_new_tokens": 10,
"model": "Salesforce/blip-image-captioning-base"
}
},
"parameters": {}
},
"outputs": [
{
"output_type": "execute_result",
"execution_count": 0,
"data": "a bear sitting on a rock eating honey",
"metadata": {}
},
{
"output_type": "execute_result",
"execution_count": 1,
"data": "a red fox in the woods",
"metadata": {}
}
]
},
{
"name": "translation",
"input": "Once upon a time, in a lush and vibrant forest, there lived a magnificent creature known as the Quick Brown Fox. This fox was unlike any other, possessing incredible speed and agility that awed all the animals in the forest. With its fur as golden as the sun and its eyes as sharp as emeralds, the Quick Brown Fox was admired by everyone, from the tiniest hummingbird to the mightiest bear. The fox had a kind heart and would often lend a helping paw to those in need. The Quick Brown Fox had a particular fondness for games and challenges. It loved to test its skills against others, always seeking new adventures to satisfy its boundless curiosity. Its favorite game was called \"The Great Word Hunt,\" where it would embark on a quest to find hidden words scattered across the forest. \n\nOne day it got very old and died",
"metadata": {
"model": {
"name": "HuggingFaceTextTranslationTransformer",
"settings": {
"model": "Helsinki-NLP/opus-mt-en-fr",
"max_length": 100,
"min_length": 50,
"num_beams": 1
}
},
"parameters": {}
}
}
]
```
Before:
```
File "/opt/homebrew/lib/python3.11/site-packages/aiconfig/util/params.py", line 235, in get_prompt_template
raise Exception(f"Cannot get prompt template string from prompt: {prompt.input}")
Exception: Cannot get prompt template string from prompt: attachments=[Attachment(data='https://s3.amazonaws.com/lastmileai.aiconfig.public/uploads/2024_1_10_18_41_31/1742/bear_with_honey.png', mime_type='image/png', metadata=None), Attachment(data='https://s3.amazonaws.com/lastmileai.aiconfig.public/uploads/2024_1_10_18_41_31/7275/fox_in_forest.png', mime_type='image/png', metadata=None)] data=None
```
After:
* Beautiful translation
rholinshead
approved these changes
Jan 10, 2024
|
LGTM, just nits |
saqadri
added a commit
that referenced
this pull request
Jan 10, 2024
Fix parameterization issues
Now that we have non-text input model parsers, we can see some issues in
our parameterization.
1) ASR and Image-to-Text modelparsers should NOT be
`ParameterizedModelParser` instances, since their inputs cannot be
parameterized.
2) Parameter resolution logic shouldn't throw errors in the case where
it's parsing prompts belonging to regular model parsers.
Test Plan:
Ran the translation prompt for this aiconfig:
```
prompts: [
{
"name": "image_caption",
"input": {
"attachments": [
{
"data": "https://s3.amazonaws.com/lastmileai.aiconfig.public/uploads/2024_1_10_18_41_31/1742/bear_with_honey.png",
"mime_type": "image/png"
},
{
"data": "https://s3.amazonaws.com/lastmileai.aiconfig.public/uploads/2024_1_10_18_41_31/7275/fox_in_forest.png",
"mime_type": "image/png"
}
]
},
"metadata": {
"model": {
"name": "HuggingFaceImage2TextTransformer",
"settings": {
"max_new_tokens": 10,
"model": "Salesforce/blip-image-captioning-base"
}
},
"parameters": {}
},
"outputs": [
{
"output_type": "execute_result",
"execution_count": 0,
"data": "a bear sitting on a rock eating honey",
"metadata": {}
},
{
"output_type": "execute_result",
"execution_count": 1,
"data": "a red fox in the woods",
"metadata": {}
}
]
},
{
"name": "translation",
"input": "Once upon a time, in a lush and vibrant forest, there lived a magnificent creature known as the Quick Brown Fox. This fox was unlike any other, possessing incredible speed and agility that awed all the animals in the forest. With its fur as golden as the sun and its eyes as sharp as emeralds, the Quick Brown Fox was admired by everyone, from the tiniest hummingbird to the mightiest bear. The fox had a kind heart and would often lend a helping paw to those in need. The Quick Brown Fox had a particular fondness for games and challenges. It loved to test its skills against others, always seeking new adventures to satisfy its boundless curiosity. Its favorite game was called \"The Great Word Hunt,\" where it would embark on a quest to find hidden words scattered across the forest. \n\nOne day it got very old and died",
"metadata": {
"model": {
"name": "HuggingFaceTextTranslationTransformer",
"settings": {
"model": "Helsinki-NLP/opus-mt-en-fr",
"max_length": 100,
"min_length": 50,
"num_beams": 1
}
},
"parameters": {}
}
}
]
```
Before:
```
File "/opt/homebrew/lib/python3.11/site-packages/aiconfig/util/params.py", line 235, in get_prompt_template
raise Exception(f"Cannot get prompt template string from prompt: {prompt.input}")
Exception: Cannot get prompt template string from prompt: attachments=[Attachment(data='https://s3.amazonaws.com/lastmileai.aiconfig.public/uploads/2024_1_10_18_41_31/1742/bear_with_honey.png', mime_type='image/png', metadata=None), Attachment(data='https://s3.amazonaws.com/lastmileai.aiconfig.public/uploads/2024_1_10_18_41_31/7275/fox_in_forest.png', mime_type='image/png', metadata=None)] data=None
```
After:
* Beautiful translation
---
Stack created with [Sapling](https://sapling-scm.com). Best reviewed
with
[ReviewStack](https://reviewstack.dev/lastmile-ai/aiconfig/pull/866).
* #826
* __->__ #866
* #863
|
Will clean up readme afterwards. Thanks @rholinshead |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Gradio Local Editor cookbook
Stack created with Sapling. Best reviewed with ReviewStack.