ArgumentException: Buffer Too Small in UploadFileAsync Method

[Chinayuesir]

Bug Report

Overview
I encountered an ArgumentException related to the byte buffer's size while using the UploadFileAsync method from mApi.FilesEndpoint. The complete error message is as follows:

System.ArgumentException: The output byte buffer is too small to contain the encoded data, encoding 'Unicode (UTF-8)' fallback 'System.Text.EncoderReplacementFallback'.

This issue arises when attempting to upload a .docx file containing both Chinese and English text. It appears that the buffer allocated for the encoding process is insufficient for the encoded data's size.

Steps to Reproduce
To reproduce the behavior:
Attempt to upload a .docx file that contains Chinese and English text.
Observe the ArgumentException.
Note: When uploading a .cs script file, the upload process completes successfully.

[StephenHodgson]

Please provide full stack trace and example file

[Chinayuesir]

2023 中文测试中文测试文测试.docx
I provide a minmum example.

Here is the full stack:

ArgumentException: The output byte buffer is too small to contain the encoded data, encoding 'Unicode (UTF-8)' fallback 'System.Text.EncoderReplacementFallback'.
Parameter name: bytes
System.Text.Encoding.ThrowBytesOverflow () (at <17d9ce77f27a4bd2afb5ba32c9bea976>:0)
System.Text.Encoding.ThrowBytesOverflow (System.Text.EncoderNLS encoder, System.Boolean nothingEncoded) (at <17d9ce77f27a4bd2afb5ba32c9bea976>:0)
System.Text.UTF8Encoding.GetBytes (System.Char* chars, System.Int32 charCount, System.Byte* bytes, System.Int32 byteCount, System.Text.EncoderNLS baseEncoder) (at <17d9ce77f27a4bd2afb5ba32c9bea976>:0)
System.Text.UTF8Encoding.GetBytes (System.Char* chars, System.Int32 charCount, System.Byte* bytes, System.Int32 byteCount) (at <17d9ce77f27a4bd2afb5ba32c9bea976>:0)
UnityEngine.WWWTranscoder.SevenBitClean (System.String s, System.Text.Encoding e) (at <b3175cf5e68b4630a894aa95aa711ad1>:0)
UnityEngine.WWWForm.get_data () (at <b3175cf5e68b4630a894aa95aa711ad1>:0)
UnityEngine.Networking.UnityWebRequest.SetupPost (UnityEngine.Networking.UnityWebRequest request, UnityEngine.WWWForm formData) (at <b3175cf5e68b4630a894aa95aa711ad1>:0)
UnityEngine.Networking.UnityWebRequest.Post (System.String uri, UnityEngine.WWWForm formData) (at <b3175cf5e68b4630a894aa95aa711ad1>:0)
Utilities.WebRequestRest.Rest.PostAsync (System.String query, UnityEngine.WWWForm formData, Utilities.WebRequestRest.RestParameters parameters, System.Threading.CancellationToken cancellationToken) (at ./Library/PackageCache/[email protected]/Runtime/Rest.cs:157)
OpenAI.Files.FilesEndpoint.UploadFileAsync (OpenAI.Files.FileUploadRequest request, System.IProgress`1[T] uploadProgress, System.Threading.CancellationToken cancellationToken) (at ./Library/PackageCache/[email protected]/Runtime/Files/FilesEndpoint.cs:87)
OpenAI.Files.FilesEndpoint.UploadFileAsync (System.String filePath, System.String purpose, System.IProgress`1[T] uploadProgress, System.Threading.CancellationToken cancellationToken) (at ./Library/PackageCache/[email protected]/Runtime/Files/FilesEndpoint.cs:68)
TeamFlow.Utilities.OpenAIUtility.UploadFileAsync (System.String filePath, System.Action`1[T] callback) (at Assets/Scripts/TeamFlow/Utilities/OpenAIUtility.cs:178)
Cysharp.Threading.Tasks.UniTaskCompletionSourceCore`1[TResult].GetResult (System.Int16 token) (at Assets/Plugins/UniTask/Runtime/UniTaskCompletionSource.cs:244)
Cysharp.Threading.Tasks.CompilerServices.AsyncUniTask`2[TStateMachine,T].GetResult (System.Int16 token) (at Assets/Plugins/UniTask/Runtime/CompilerServices/StateMachineRunner.cs:342)
Cysharp.Threading.Tasks.UniTask`1+Awaiter[T].GetResult () (at Assets/Plugins/UniTask/Runtime/UniTask.cs:653)
TeamFlow.TeamFlowFile.UploadFile () (at Assets/Scripts/TeamFlow/Nodes/Architecture/TeamFlowFile.cs:42)
System.Runtime.CompilerServices.AsyncMethodBuilderCore+<>c.<ThrowAsync>b__7_0 (System.Object state) (at <17d9ce77f27a4bd2afb5ba32c9bea976>:0)
UnityEngine.UnitySynchronizationContext+WorkRequest.Invoke () (at <f7237cf7abef49bfbb552d7eb076e422>:0)
UnityEngine.UnitySynchronizationContext.Exec () (at <f7237cf7abef49bfbb552d7eb076e422>:0)
UnityEngine.UnitySynchronizationContext.ExecuteTasks () (at <f7237cf7abef49bfbb552d7eb076e422>:0)

By the way,I find if I shorten the filename,this issue cannot happen.

[Chinayuesir]

If the file whose name includes Chinese and is uploaded successfully,it will cause garbled text.
Now i am trying to find the solution.

[StephenHodgson]

Yeah this seems to be an issue with the web request uploader itself.

I wonder if it helps to change the encoding of the file.

[Chinayuesir]

Fine，thanks.
It seems like the web version of ChatGPT can handle my file correctly.So I am confused.

[StephenHodgson]

I suspect it is a problem with the Unity webrequest uploader.

Try uploading it with my DotNet package.

[Chinayuesir]

Nice，I will try.