[Easy] Thoughts on preventing send additional prompts when the length of a chat is longer than 5.

[haopengsong]

Hello,
Correct me if I'm wrong.
Since when interacting with the openai api, the whole chat in the context window would be sent. Hence, the size of the context tokens could grow way larger than the generated tokens (pic below). In my case, it may have inccured a huge unwanted costs.
It would be nice to limit the length of each chat (could be 5 or etc).

Need some help in providing throughts on where to make changes that may allow such feature...

[enricoros]

Thanks for this request. It's a good idea indeed.

[haopengsong]

Hey, so I did some hack last month and get some progresses.
added some condition check on history.length in src/apps/chat/AppChat.tsx within _handleExecute.
When it is too long (current limitation is 12), just alert( ) with warnings and return , hence no api calls ...

[enricoros]

@haopengsong I've released today the newest version of big-AGI, which has cost estimation for various models.
Good job to edit _handleExecute.

Let's look at the options:

• which is the perfect UI for you to auto-activate this mode? (selector in each chat, in all chats (preferences))

• which is the perfect manual selected decimation algorithm for you, examples:

  • last N messages (e.g. last 12)
  • first System prompt + last N-1 messages
  • first System prompt + first user Message + last N-1 messages

• shall we use AI for an automatic decimation (e.g. it will look at all the messages of the chat and decide which ones to hide and which to keep, and it needs to do that with a smaller and cheaper model)

• do you need a manual "visible / invisible" (e.g. eye icon) button to manually decide to exclude messages in the history (keep them in the history but not send them to GPT)

• what else?

[haopengsong]

THanks for the update! Great progresses on the new features. Really like the cost estimation.
For the options, I would love to provide my personal opinions:

which is the perfect UI for you to auto-activate this mode? (selector in each chat, in all chats (preferences))

• I'd say default to all chats as this might be the most strightforward way.

which is the perfect manual selected decimation algorithm for you, examples:
last N messages (e.g. last 12)
first System prompt + last N-1 messages
first System prompt + first user Message + last N-1 messages

• Probably choose 'last N messages' as this sounds easier for casual users.

shall we use AI for an automatic decimation (e.g. it will look at all the messages of the chat and decide which ones to hide and which to keep, and it needs to do that with a smaller and cheaper model)

• This sounds exciting. Probably better if it can first notify the user which messages will be deleted then proceed according to the user confirmation.

do you need a manual "visible / invisible" (e.g. eye icon) button to manually decide to exclude messages in the history (keep them in the history but not send them to GPT)

• This would be great as this gives the user the option to fine-tune their chats with the AI model.

what else?
So far so good.

[enricoros]

Thanks for the responses, I'm referencing #480 to keep track of those.
Sounds like the app shall have some Chat Preferences, with the window policy for sending messages (e.g. none, last-N, AI-aided (using a cheap model)). And the messages should have a visibility flag that the user can toggle, and another visual indication on whether they were actually submitted in the last chat request (i.e. automatic exclusion).