What's most optimal context size for the average workflow to work well ?

[ayoubachak]

I tried to add LM_STUDIO internal server as a model to be in the options, and I only tried it with 2000 context using the google gemma 7b model, I didn't get any results, even upping the number of tokens to 8192 which is the maximum supported by the model, it took so much time and didn't reach the goal, I asked it to create the game of life using python and matplotlib :

once it surpasses the number of tokens it keeps looping to the void, I want to know if there's a way that we can make the most out of the 8192 context size, if not what is the optimal the one should have ?

[ayoubachak]

It suddenly changed its mind and started looking arround for SQL content which is completly non-related with the topic, It must be because of gemma's non ability to perform coding tasks due to its small size and it not being optimized for coding :

[ayoubachak]

UPDATE : I've managed to get one of the models to work, I've used mistral instruc 7B, and asked the same question, but it gave me the results in rust instead of python, using python libraries that I don't think are yet available in rust like matplotlib and numpy.

The context size I've used was 32768 tokens, which I don't think was necessary to set as the whole process took 5522 tokens as shown in the image above .

[obliviousz]

Umm what i feel is it at first didn't work because of the wrong response from the LLM on the prompts which are there. Because it shows "Invalid response from the model, trying again". This comes when the output from the LLM is a bit wrong and doesn't get validated due to its wrong format (as per the prompt).

[ayoubachak]

Umm what i feel is it at first didn't work because of the wrong response from the LLM on the prompts which are there. Because it shows "Invalid response from the model, trying again". This comes when the output from the LLM is a bit wrong and doesn't get validated due to its wrong format (as per the prompt).

I think it's because gemma wasn't good for task, however the result using mistral was a little promising given that it's a 7b model, I tried codellama 34B but it was too much for my computer, I created a pull request for the LM Studio option, maybe someone can try it out and play with the context size : #389