Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

community[patch]: support alibaba_tongyi stream output #5271

Merged
merged 3 commits into from
May 14, 2024
Merged
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
108 changes: 107 additions & 1 deletion libs/langchain-community/src/chat_models/alibaba_tongyi.ts
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
import { type CallbackManagerForLLMRun } from "@langchain/core/callbacks/manager";
import {
BaseChatModel,
type BaseChatModelParams,
Expand All @@ -6,10 +7,12 @@ import {
AIMessage,
type BaseMessage,
ChatMessage,
AIMessageChunk,
} from "@langchain/core/messages";
import { type ChatResult } from "@langchain/core/outputs";
import { type CallbackManagerForLLMRun } from "@langchain/core/callbacks/manager";
jacoblee93 marked this conversation as resolved.
Show resolved Hide resolved
import { ChatGenerationChunk } from "@langchain/core/outputs";
import { getEnvironmentVariable } from "@langchain/core/utils/env";
import { IterableReadableStream } from "@langchain/core/utils/stream";

/**
* Type representing the role of a message in the Tongyi chat model.
Expand Down Expand Up @@ -493,6 +496,109 @@ export class ChatAlibabaTongyi
return this.caller.call(makeCompletionRequest);
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey there! I noticed that the new code introduces a net-new HTTP request using the fetch method. I've flagged this for your review to ensure it aligns with the project's architecture and requirements. Let me know if you have any questions or need further clarification!

}

async *_streamResponseChunks(
messages: BaseMessage[],
options?: this["ParsedCallOptions"],
runManager?: CallbackManagerForLLMRun
): AsyncGenerator<ChatGenerationChunk> {
const parameters = {
...this.invocationParams(),
stream: true,
incremental_output: true,
};

const messagesMapped: TongyiMessage[] = messages.map((message) => ({
role: messageToTongyiRole(message),
content: message.content as string,
}));

const stream = await this.caller.call(async () =>
this.createTongyiStream(
{
model: this.model,
parameters,
input: {
messages: messagesMapped,
},
},
options?.signal
)
);

for await (const chunk of stream) {
const { text, finish_reason } = chunk.output;
yield new ChatGenerationChunk({
text,
message: new AIMessageChunk({ content: text }),
generationInfo:
finish_reason === "stop"
? {
finish_reason,
request_id: chunk.request_id,
usage: chunk.usage,
}
: undefined,
});
await runManager?.handleLLMNewToken(text);
}
}

private async *createTongyiStream(
request: ChatCompletionRequest,
signal?: AbortSignal
) {
const response = await fetch(this.apiUrl, {
method: "POST",
headers: {
Authorization: `Bearer ${this.alibabaApiKey}`,
Accept: "text/event-stream",
"Content-Type": "application/json",
},
body: JSON.stringify(request),
signal,
});

if (!response.ok) {
let error;
const responseText = await response.text();
try {
const json = JSON.parse(responseText);
error = new Error(
`Tongyi call failed with status code ${response.status}: ${json.error}`
);
} catch (e) {
error = new Error(
`Tongyi call failed with status code ${response.status}: ${responseText}`
);
}
error.response = response;
throw error;
}
if (!response.body) {
throw new Error(
"Could not begin Tongyi stream. Please check the given URL and try again."
);
}
const stream = IterableReadableStream.fromReadableStream(response.body);
const decoder = new TextDecoder();
let extra = "";
for await (const chunk of stream) {
const decoded = extra + decoder.decode(chunk);
const lines = decoded.split("\n");
extra = lines.pop() || "";
for (const line of lines) {
if (!line.startsWith("data:")) {
continue;
}
try {
yield JSON.parse(line.slice("data:".length).trim());
} catch (e) {
console.warn(`Received a non-JSON parseable chunk: ${line}`);
}
}
}
}

_llmType(): string {
return "alibaba_tongyi";
}
Expand Down