Release v1.4.1 🚀

Updating Patch Version

This patch version Introduces explicit resource management to prevent unexpected resource leaks.
By default, the library closes underlying HTTP and gRPC connections when the client is garbage-collected. However, you can now manually close the Friendli or AsyncFriendli client using the .close() method or utilize a context manager to ensure proper closure when exiting a with block.

Usage examples

import asyncio
from friendli import AsyncFriendli

client = AsyncFriendli(base_url="0.0.0.0:8000", use_grpc=True)

async def run():
    async with client:
        stream = await client.completions.create(
            prompt="Explain what gRPC is. Also give me a Python code snippet of gRPC client.",
            stream=True,
            top_k=1,
        )

        async for chunk in stream:
            print(chunk.text, end="", flush=True)

asyncio.run(run())

Release v1.4.0 🚀

• gRPC client support for completions API.

Release v1.3.7 🚀

• Minor: add a default value for the "index" and "text" fields of the completion stream's chunk.

Release v1.3.6 🚀

• Support Phi3 FP8 conversion.
• Hotfix for safetensor checkpoint saver.

Release v1.3.5 🚀

• Optimize CPU RAM usage during quantization with offloading
• Support FP8 conversion for DBRX, Mixtral, and Command R+

Release v1.3.4 🚀

• Hotfix for LoRA checkpoint saving error.

Release v1.3.3 🚀

New Features

• FP8 Checkpoint Conversion: We've introduced a new feature for FP8 checkpoint conversion.
• Sharded Safetensors Checkpoint Saving: Added the ability to save sharded safetensors checkpoints.
• LoRA Support on Mistral Model: We have added support for LoRA (Low-Rank Adaptation) on the Mistral model.

Bug Fixes

• BF16 Hotfix: Addressed an urgent issue with bf16 processing.
• BFloat Safetensors Conversion: Fixed an issue related to bfloat conversion for safetensors.
• Automatic Token Refresh: Resolved a bug affecting automatic token refresh.

Release v1.3.2 🚀

• Add base_model_name_or_path option to friendli model convert-adapter.
• Remove stale dependencies.

Release v1.3.1 🚀

• Update protobuf schema.
• Patch sending API requests with content type application/protobuf.

Release v1.3.0 🚀

• Now resources of Friendli Dedicated Endpoints can be managed with CLI and SDK. The available resources are endpoint, model, team, and project.
• Login with CLI is now available. SSO login is also available.
• Update on Multi-LoRA checkpoint conversion.