GitHub - rpehkone/Chat-With-RTX-python-api: Chat With RTX Python API

Python api for calling Chat With RTX local server

import rtx_api_3_5 as rtx_api

response = rtx_api.send_message("write fire emoji")
print(response)

Chat With RTX builds int4 (W4A16 AWQ) tensortRT engines for mistral 7b and llama2 13b

On my 4090
  mistral 130 tok/s
  lama 75 tok/s

LICENSE: CC0

Name		Name	Last commit message	Last commit date
Latest commit History 13 Commits
readme.md		readme.md
rtx_api_2_11.py		rtx_api_2_11.py
rtx_api_3_5.py		rtx_api_3_5.py
rtx_api_4_24.py		rtx_api_4_24.py
test.py		test.py
test_chat.py		test_chat.py