Run_any_gguf_model_in_TextGen_webui.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "private_outputs": true,
      "provenance": [],
      "gpuType": "T4",
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/seyf1elislam/LocalLLM_OneClick_Colab/blob/main/Run_any_gguf_model_in_TextGen_webui.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# $\\color{#19ABEA}{\\text{Run any gguf quantized models in Latest Version TextGen-webui :}}$\n",
        "### `version 05/2024`\n",
        "> changes : uses hf client  (5/2024)\n",
        "\n",
        "this [Notebook](https://github.com/seyf1elislam/LocalLLM_OneClick_Colab) allow u run  any quantized gguf file from any Hugging face repository as long as it fits the colab Vram and ram.\n",
        "\n",
        ">follow this [githubRepository](https://github.com/seyf1elislam/LocalLLM_OneClick_Colab) to get the updates of this noteboook.\n",
        "\n",
        "> Created by :\n",
        "> 😇 github :   [@seyf1elislam](https://github.com/seyf1elislam)\n",
        "> 🤗 Huggingface : [@seyf1elislam](https://huggingface.co/seyf1elislam)\n",
        "\n",
        "## where can i find gguf quantized models ?\n",
        "> you can check those 🤗 huggingface repos :\n",
        "- [Thebloke](https://huggingface.co/TheBloke) #1 best gguf source ,contains hundreds of quantized models(gptq ,gguf ,awQ).\n",
        "- [LongStriker](https://huggingface.co/LoneStriker) #2 best gguf source ,contains hundreds of quantized models.(exl2,gguf)\n",
        "- [seyf1elislam](https://huggingface.co/seyf1elislam) contains some gguf models.  \n",
        "- [using search](https://huggingface.co/models?sort=trending&search=gguf) here u can find all gguf files on  huggingface.\n",
        "\n",
        "## need good models to try ?\n",
        "> you can try those :  \n",
        "- [Meta-Llama-3-8B-Instruct-GGUF](https://huggingface.co/QuantFactory/Meta-Llama-3-8B-Instruct-GGUF)   (recommended  🔥) .\n",
        "- [seyf1elislam/WestKunai-Hermes-7b-GGUF](https://huggingface.co/seyf1elislam/WestKunai-Hermes-7b-GGUF)\n",
        "- [TheBloke/SOLAR-10.7B-v1.0-GGUF](https://huggingface.co/TheBloke/SOLAR-10.7B-v1.0-GGUF)\n",
        "- [TheBloke/Kunoichi-7B-GGUF](https://huggingface.co/TheBloke/Kunoichi-7B-GGUF)\n",
        "- [TheBloke/Starling-LM-7B-alpha-GGUF](https://huggingface.co/TheBloke/Starling-LM-7B-alpha-GGUF)\n",
        "-[seyf1elislam/KuTrix-7b-GGUF](https://huggingface.co/seyf1elislam/KuTrix-7b-GGUF)\n",
        "-[seyf1elislam/KunaiBeagle-Hermes-7b](https://huggingface.co/seyf1elislam/KunaiBeagle-Hermes-7b)\n",
        "\n",
        "## some Tips ?\n",
        "> most mistral models goes with 8k context length if u want to use longer context u need to make sure the models support longer context.\n",
        "\n",
        "> if u want to run model higher then 13B (such as 20B,4x7b..) on colab u may need to reduce the offloaded gpu models to split the ram usage between gpu vram and system ram. (slower but it works 😉)"
      ],
      "metadata": {
        "id": "KW0lK7R8yp_i"
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "H_bLc7-6yVQS"
      },
      "outputs": [],
      "source": [
        "#@title # $\\color{#19ABEA}{\\text{Download and install requirements}}$ {display-mode: \"form\"}\n",
        "#@markdown $\\color{#19ABEA}{\\text{run this only once at the start}}$\n",
        "\n",
        "## @markdown `Note :no need to check these checkboxes unless u facing a problem`\n",
        "#@markdown ---\n",
        "#@markdown `check this if u need to change the version only`\n",
        "delete_existing_texgen_webui_folder =False #@param {type:\"boolean\"}\n",
        "if delete_existing_texgen_webui_folder :\n",
        "  !rm -rf /content/text-generation-webui\n",
        "#@markdown ---\n",
        "#@markdown ### TextGen webui version :\n",
        "#@markdown `note : gemma models are supported in snapshot-2024-02-25 or newer`\n",
        "%cd /content\n",
        "branch = \"dev\" # @param [\"dev\", \"snapshot-2024-01-07\", \"snapshot-2024-01-14\", \"snapshot-2024-01-28\", \"snapshot-2024-02-11\", \"snapshot-2024-02-25\", \"snapshot-2024-03-03\", \"snapshot-2024-03-10\", \"snapshot-2024-03-17\", \"snapshot-2024-03-24\", \"snapshot-2024-03-31\", \"snapshot-2024-04-07\", \"snapshot-2024-04-14\", \"snapshot-2024-04-21\", \"snapshot-2024-04-28\"]\n",
        "branch = f\"-b {branch}\"\n",
        "#@markdown `to download the latest version of TextGen Webui use use_latest ,this will ignore the branch name.`\n",
        "use_latest =False #@param {type:\"boolean\"}\n",
        "\n",
        "if use_latest :\n",
        "  branch = \"\"\n",
        "!git clone {branch} https://github.com/oobabooga/text-generation-webui\n",
        "\n",
        "%cd /content/text-generation-webui\n",
        "!pip install -r requirements.txt\n",
        "!pip install git+https://github.com/UWUplus/flask-cloudflared\n",
        "!pip install -r extensions/openai/requirements.txt --upgrade\n",
        "!apt-get -y install -qq aria2\n",
        "#@markdown ----\n",
        "reinstall_llama_cpp_python = False # @param {type:\"boolean\"}\n",
        "if reinstall_llama_cpp_python :\n",
        "  !pip uninstall -y llama-cpp-python\n",
        "  !CMAKE_ARGS=\"-DLLAMA_CUBLAS=on\" FORCE_CMAKE=1 pip install llama-cpp-python --no-cache-dir\n",
        "\n",
        "\n",
        "reinstall_flash_attention = False # @param {type:\"boolean\"}\n",
        "if reinstall_flash_attention:\n",
        "  !pip uninstall -y flash-attn\n",
        "  !pip install --no-build-isolation flash-attn==2.3.0\n",
        "#@markdown ----\n",
        "\n",
        "from IPython.display import clear_output\n",
        "clear_output()\n",
        "print(\"Done\")"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "#@title #  $\\color{#19ABEA}{Download Model}$ {display-mode: \"form\"}\n",
        "#@markdown ## $\\color{#19ABEA}{\\text{Copy the RepoName here :}}$\n",
        "\n",
        "#----------\n",
        "from huggingface_hub import HfFileSystem\n",
        "fs = HfFileSystem()\n",
        "\n",
        "def get_donwloadlink_and_filename(repo_name,quant='Q5_K_M'):\n",
        "  gguf_files = fs.glob(f\"{repo_name}/**.gguf\")\n",
        "  filtered_files = [file for file in gguf_files if (quant in file or (quant.lower()) or  (quant.upper()) in file)]\n",
        "\n",
        "  if len(filtered_files) ==0 : # quant not existed\n",
        "    print( \"=\" * 10 +\"\\navailable quants:\\n\",gguf_files)\n",
        "    return None,None\n",
        "  file_path= filtered_files[0]\n",
        "  *repo_name , file_name = file_path.split(\"/\");\n",
        "  repo_name='/'.join(repo_name)\n",
        "  return f\"https://huggingface.co/{repo_name}/resolve/main/{file_name}?download=true\" ,file_name\n",
        "\n",
        "#----------\n",
        "\n",
        "#@markdown  Example :`seyf1elislam/WestKunai-Hermes-7b-GGUF`\n",
        "repo_name = \"QuantFactory/Meta-Llama-3-8B-Instruct-GGUF\" # @param [\"seyf1elislam/WestKunai-Hermes-7b-GGUF\", \"seyf1elislam/KuTrix-7b-GGUF\", \"TheBloke/Mistral-7B-OpenOrca-GGUF\", \"TheBloke/zephyr-7B-alpha-GGUF\", \"TheBloke/zephyr-7B-beta-GGUF\", \"TheBloke/neural-chat-7B-v3-1-GGUF\", \"TheBloke/mistral-ft-optimized-1218-GGUF\", \"TheBloke/xDAN-L1-Chat-RL-v1-GGUF\", \"TheBloke/openchat-3.5-1210-starling-slerp-GGUF\", \"TheBloke/Loyal-Macaroni-Maid-7B-GGUF\", \"TheBloke/Valkyrie-V1-GGUF\", \"TheBloke/Sonya-7B-GGUF\", \"TheBloke/Starling-LM-7B-alpha-GGUF\", \"TheBloke/SOLAR-10.7B-v1.0-GGUF\", \"QuantFactory/Meta-Llama-3-8B-Instruct-GGUF\"] {allow-input: true}\n",
        "\n",
        "#@markdown  in colab gpu (15Gvram) you can use :\n",
        "#@markdown  - 13b model quantized upto Q5_K_M(`context up to 4K`)\n",
        "#@markdown  - 7b model quantized upto Q8_0(`context up to 16K or even more if ur model stable with long context`)\n",
        "quant =\"Q5_K_M\" #@param ['Q2_K', 'Q3_K_L', 'Q3_K_M', 'Q3_K_S', 'Q4_0', 'Q4_1', 'Q4_K_M', 'Q4_K_S', 'Q5_0', 'Q5_1', 'Q5_K_M', 'Q5_K_S', 'Q6_K', 'Q8_0']\n",
        "model_download_url,model_file_name =get_donwloadlink_and_filename(repo_name,quant=quant)\n",
        "\n",
        "if model_download_url is not None and model_file_name is not None :\n",
        "  !aria2c --console-log-level=error -c -x 16 -s 16 -k 1M {model_download_url} -d /content/text-generation-webui/models/ -o {model_file_name}\n",
        "else :\n",
        "  print(\"======================\")\n",
        "  print(\"please check the repo name or check availability of the quantized file inside the repo\")"
      ],
      "metadata": {
        "id": "9AsZFjNe0Y7_"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "#@title ##  $\\color{#19ABEA}{RunOnly}$ {display-mode: \"form\"}\n",
        "# %cd /content/text-generation-webui\n",
        "#@markdown  $\\color{#19ABEA}{\\text{Enter ur Context size or let it unchecked to use the default size}}$\n",
        "params =\" \"\n",
        "#@markdown _____\n",
        "api =True # @param {type:\"boolean\"}\n",
        "if api :params +=\" --api --public-api --listen \"\n",
        "api_only_mode_nowebui =False # @param {type:\"boolean\"}\n",
        "if api_only_mode_nowebui : params+=\" --nowebui \"\n",
        "#@markdown _____\n",
        "use_custom_context_len = True # @param {type:\"boolean\"}\n",
        "context_len=8192 #@param\n",
        "if use_custom_context_len : params += f\" --n_ctx  {context_len} \"\n",
        "#@markdown _____\n",
        "use_custom_params = True # @param {type:\"boolean\"}\n",
        "custom_params = \"\" # @param {type:\"string\"}\n",
        "if use_custom_params : params +=custom_params\n",
        "\n",
        "#@markdown _____\n",
        "# params\n",
        "!python server.py   --share  --n-gpu-layers 100000  \t--model $model_file_name   $params"
      ],
      "metadata": {
        "id": "8jBKRxrXVceJ"
      },
      "execution_count": null,
      "outputs": []
    }
  ]
}