Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[speech-to-text] Connection immediately closed in Android build, works fine in editor #691

Open
jpingen opened this issue Jul 29, 2022 · 1 comment
Open

[speech-to-text] Connection immediately closed in Android build, works fine in editor #691

jpingen opened this issue Jul 29, 2022 · 1 comment

Comments

@jpingen
Copy link

jpingen commented Jul 29, 2022 •
edited

Hi, I'm having issues with getting speech-to-text to work in an Android build. Everything works fine in the Unity Editor.

Unity version: 2021.3.6f1
Watson unity-sdk: v6.0.0
IBM unity-sdk-core: v1.2.4
Scripting backend: IL2CPP
Api compatibility level: .NET Framework (I tried switching to .NET Standard 2.1, but it results in a compilation error IBMSdkCore\Connection\RESTConnector.cs(668,24): error CS0656: Missing compiler required member 'Microsoft.CSharp.RuntimeBinder.CSharpArgumentInfo.Create')

So everything works fine in the Editor, but when I run the Android build, the websocket connection is immediately closed. Similar to #542, but I'm using the Frankfurt endpoint (eu-de).
Edit: also tested on London (eu-gb) and Dallas (us-south), problem still the same.

Logs:

: RESTConnector: URL: https://iam.cloud.ibm.com/identity/token
: IBM.Cloud.SDK.Connection.<ProcessRequestQueue>d__33:MoveNext()
: IBM.Cloud.SDK.Utilities.Routine:MoveNext()
: UnityEngine.SetupCoroutine:InvokeMoveNext(IEnumerator, IntPtr)
:
: RESTConnector: URL: https://iam.cloud.ibm.com/identity/token
: IBM.Cloud.SDK.Connection.<ProcessRequestQueue>d__33:MoveNext()
: IBM.Cloud.SDK.Utilities.Routine:MoveNext()
: UnityEngine.SetupCoroutine:InvokeMoveNext(IEnumerator, IntPtr)
:
: RESTConnector: URL: https://api.eu-de.assistant.watson.cloud.ibm.com/v2/assistants/[redacted-assistant-id]/sessions?version=2021-11-27
: IBM.Cloud.SDK.Connection.<ProcessRequestQueue>d__33:MoveNext()
: IBM.Cloud.SDK.Utilities.Routine:MoveNext()
: UnityEngine.SetupCoroutine:InvokeMoveNext(IEnumerator, IntPtr)
:
: [07-29-2022 08:55:14][WSConnector][WARNING] No case for URL for wss://. Replacing https:// with wss://.
: IBM.Cloud.SDK.Debug.DebugReactor:ProcessLog(LogRecord)
: IBM.Cloud.SDK.LogSystem:ProcessLog(LogRecord)
: IBM.Cloud.SDK.Log:Warning(String, String, Object[])
: IBM.Cloud.SDK.Connection.WSConnector:FixupURL(String)
: IBM.Cloud.SDK.Connection.WSConnector:CreateConnector(Authenticator, String, String, String)
: IBM.Watson.SpeechToText.V1.SpeechToTextService:CreateListenConnector()
: IBM.Watson.SpeechToText.V1.SpeechToTextService:StartListening(OnRecognize, OnRecognizeSpeaker)
: 
: SpeechToText.CreateListenConnector(): Created listen socket. Model: nl-NL_Telephony, parsedParams:
: IBM.Watson.SpeechToText.V1.SpeechToTextService:CreateListenConnector()
: IBM.Watson.SpeechToText.V1.SpeechToTextService:StartListening(OnRecognize, OnRecognizeSpeaker)
:
: WSConnector.Send(): Sending TextMessage message: {"action":"start","content-type":"audio/l16;rate=16000;channels=1;","inactivity_timeout":-1,"interim_results":true,"max_alternatives":1,"profanity_filter":false,"smart_formatting":true,"speaker_labels":false,"timestamps":true,"word_confidence":true,"speech_detector_sensitivity":0,5,"background_audio_suppression":0,"processing_metrics":false}
: IBM.Cloud.SDK.Connection.WSConnector:Send(Message, Boolean)
: IBM.Watson.SpeechToText.V1.SpeechToTextService:SendStart()
: IBM.Watson.SpeechToText.V1.SpeechToTextService:OnListen(AudioData)
:
: SpeechToText.SendStart(): SendStart() with the following params: {"action":"start","content-type":"audio/l16;rate=16000;channels=1;","inactivity_timeout":-1,"interim_results":true,"max_alternatives":1,"profanity_filter":false,"smart_formatting":true,"speaker_labels":false,"timestamps":true,"word_confidence":true,"speech_detector_sensitivity":0,5,"background_audio_suppression":0,"processing_metrics":false}
: IBM.Watson.SpeechToText.V1.SpeechToTextService:SendStart()
: IBM.Watson.SpeechToText.V1.SpeechToTextService:OnListen(AudioData)
:
: WSConnector.OnWSClose(): Reason: An exception has occurred while receiving. - Code: 1006 - WasClean: False
: IBM.Cloud.SDK.Connection.WSConnector:OnWSClose(Object, CloseEventArgs)
: IBM.Cloud.SDK.Plugins.WebSocketSharp.WebSocket:close(PayloadData, Boolean, Boolean, Boolean)
: System.Threading.Tasks.TaskToApm:Begin(Task, AsyncCallback, Object)
: IBM.Cloud.SDK.Plugins.WebSocketSharp.<>c__DisplayClass57_0:<ReadBytesAsync>b__0(IAsyncResult)
: System.Threading.Tasks.TaskToApm:Begin(Task, AsyncCallback, Object)
: IBM.Cloud.SDK.Plugins.WebSocketSharp.<>c__DisplayClass57_0:<ReadBytesAsync>b__0(IAsyncResult)
: System.Threading.Tasks.TaskToApm:Begin(Task, AsyncCallback, Object)
: IBM.Cloud.SDK.Plugins.WebSocketSharp.<>c__DisplayClass57_0:<ReadBytesAsync>b__0(IAsyncResult)
: System.Threading.Tasks.TaskToApm:Begin(Task, AsyncCallback, Object)
: IBM.Cloud.SDK.Plugins.WebSocketSharp.<>c__DisplayClass57_0:<ReadBytesAsync>b__0(IAsyncResult)
: System.Threading.Tasks.TaskToApm:Begin(Task, AsyncCallback, Object)
: IBM.Cloud.SDK.Plugins.WebSocketSharp
: SpeechToText.OnListenClosed(): OnListenClosed(), State = DISCONNECTED
: IBM.Watson.SpeechToText.V1.SpeechToTextService:OnListenClosed(WSConnector)
: IBM.Cloud.SDK.Connection.<ProcessReceiveQueue>d__49:MoveNext()
: IBM.Cloud.SDK.Utilities.Routine:MoveNext()
: UnityEngine.SetupCoroutine:InvokeMoveNext(IEnumerator, IntPtr)
:
: SpeechToText.KeepAlive(): KeepAlive exited.
: IBM.Watson.SpeechToText.V1.<KeepAlive>d__242:MoveNext()
: IBM.Cloud.SDK.Utilities.Routine:MoveNext()
: UnityEngine.SetupCoroutine:InvokeMoveNext(IEnumerator, IntPtr)

I have also enabled logging on the Websocket in WSConnector:

29-7-2022 10:55:15|Warn |WSConnector.SendMessages|The current logging level has been changed to Trace.
29-7-2022 10:55:15|Debug|WebSocket.sendHandshakeRequest|A request to the server:
                         GET /v1/recognize?model=nl-NL_Telephony HTTP/1.1
                         User-Agent: websocket-sharp/1.0
                         Host: api.eu-de.speech-to-text.watson.cloud.ibm.com
                         Upgrade: websocket
                         Connection: Upgrade
                         Sec-WebSocket-Key: [redacted]
                         Sec-WebSocket-Version: 13
                         Authorization: Bearer [redacted]
29-7-2022 10:55:16|Debug|WebSocket.sendHandshakeRequest|A response to this request:
                         HTTP/1.1 101 Switching Protocols
                         sec-websocket-accept: [redacted]
                         Content-Type: application/octet-stream
                         Server: watson-gateway
                         strict-transport-security: max-age=31536000; includeSubDomains;
                         x-dp-watson-tran-id: 36a4772e-ae7f-4e2a-a996-1e37e6931fd0
                         x-request-id: 36a4772e-ae7f-4e2a-a996-1e37e6931fd0
                         x-global-transaction-id: 36a4772e-ae7f-4e2a-a996-1e37e6931fd0
                         X-EdgeConnect-MidMile-RTT: 0
                         X-EdgeConnect-Origin-MEX-Latency: 534
                         Date: Fri, 29 Jul 2022 08:55:15 GMT
                         Upgrade: websocket
                         Connection: Upgrade
29-7-2022 10:55:17|Fatal|TaskToApm.Begin|IBM.Cloud.SDK.Plugins.WebSocketSharp.WebSocketException: The header part of a frame could not be read.
                           at IBM.Cloud.SDK.Plugins.WebSocketSharp.WebSocketFrame.processHeader (System.Byte[] header) [0x00000] in <00000000000000000000000000000000>:0 
                           at IBM.Cloud.SDK.Plugins.WebSocketSharp.WebSocketFrame+<>c__DisplayClass73_0.<readHeaderAsync>b__0 (System.Byte[] bytes) [0x00000] in <00000000000000000000000000000000>:0 
                           at IBM.Cloud.SDK.Plugins.WebSocketSharp.Ext+<>c__DisplayClass57_0.<ReadBytesAsync>b__0 (System.IAsyncResult ar) [0x00000] in <00000000000000000000000000000000>:0 
                           at System.Threading.Tasks.TaskToApm.Begin (System.Threading.Tasks.Task task, System.AsyncCallback callback, System.Object state) [0x00000] in <00000000000000000000000000000000>:0 
                           at IBM.Cloud.SDK.Plugins.WebSocketSharp.Ext+<>c__DisplayClass57_0.<ReadBytesAsync>b__0 (System.IAsyncResult ar) [0x00000] in <00000000000000000000000000000000>:0 
                           at System.Threading.Tasks.TaskToApm.Begin (System.Threading.Tasks.Task task, System.AsyncCallback callback, System.Object state) [0x00000] in <00000000000000000000000000000000>:0 
                           at IBM.Cloud.SDK.Plugins.WebSocketSharp.Ext+<>c__DisplayClass57_0.<ReadBytesAsync>b__0 (System.IAsyncResult ar) [0x00000] in <00000000000000000000000000000000>:0 
                           at System.Threading.Tasks.TaskToApm.Begin (System.Threading.Tasks.Task task, System.AsyncCallback callback, System.Object state) [0x00000] in <00000000000000000000000000000000>:0 
                           at IBM.Cloud.SDK.Plugins.WebSocketSharp.Ext+<>c__DisplayClass57_0.<ReadBytesAsync>b__0 (System.IAsyncResult ar) [0x00000] in <00000000000000000000000000000000>:0 
                           at System.Threading.Tasks.TaskToApm.Begin (System.Threading.Tasks.Task task, System.AsyncCallback callback, System.Object state) [0x00000] in <00000000000000000000000000000000>:0 
                           at IBM.Cloud.SDK.Plugins.WebSocketSharp.Ext+<>c__DisplayClass57_0.<ReadBytesAsync>b__0 (System.IAsyncResult ar) [0x00000] in <00000000000000000000000000000000>:0 
                           at System.Threading.Tasks.TaskToApm.Begin (System.Threading.Tasks.Task task, System.AsyncCallback callback, System.Object state) [0x00000] in <00000000000000000000000000000000>:0 
                           at IBM.Cloud.SDK.Plugins.WebSocketSharp.Ext+<>c__DisplayClass57_0.<ReadBytesAsync>b__0 (System.IAsyncResult ar) [0x00000] in <00000000000000000000000000000000>:0 
                           at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction (System.Action action, System.Boolean allowInlining, System.Threading.Tasks.Task& currentTask) [0x00000] in <00000000000000000000000000000000>:0 
                           at System.Threading.Tasks.Task.FinishContinuations () [0x00000] in <00000000000000000000000000000000>:0 
                           at System.Threading.Tasks.Task`1[TResult].TrySetResult (TResult result) [0x00000] in <00000000000000000000000000000000>:0 
                           at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[TResult].SetResult (TResult result) [0x00000] in <00000000000000000000000000000000>:0 
                           at Mono.Net.Security.MobileAuthenticatedStream.StartOperation (Mono.Net.Security.MobileAuthenticatedStream+OperationType type, Mono.Net.Security.AsyncProtocolRequest asyncRequest, System.Threading.CancellationToken cancellationToken) [0x00000] in <00000000000000000000000000000000>:0 
                           at System.Threading.ExecutionContext.RunInternal (System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, System.Object state, System.Boolean preserveSyncCtx) [0x00000] in <00000000000000000000000000000000>:0 
                           at System.Runtime.CompilerServices.AsyncMethodBuilderCore+MoveNextRunner.Run () [0x00000] in <00000000000000000000000000000000>:0 
                           at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction (System.Action action, System.Boolean allowInlining, System.Threading.Tasks.Task& currentTask) [0x00000] in <00000000000000000000000000000000>:0 
                           at System.Threading.Tasks.Task.FinishContinuations () [0x00000] in <00000000000000000000000000000000>:0 
                           at System.Threading.Tasks.Task`1[TResult].TrySetResult (TResult result) [0x00000] in <00000000000000000000000000000000>:0 
                           at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[TResult].SetResult (TResult result) [0x00000] in <00000000000000000000000000000000>:0 
                           at Mono.Net.Security.AsyncProtocolRequest.StartOperation (System.Threading.CancellationToken cancellationToken) [0x00000] in <00000000000000000000000000000000>:0 
                           at System.Threading.ExecutionContext.RunInternal (System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, System.Object state, System.Boolean preserveSyncCtx) [0x00000] in <00000000000000000000000000000000>:0 
                           at System.Runtime.CompilerServices.AsyncMethodBuilderCore+MoveNextRunner.Run () [0x00000] in <00000000000000000000000000000000>:0 
                           at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction (System.Action action, System.Boolean allowInlining, System.Threading.Tasks.Task& currentTask) [0x00000] in <00000000000000000000000000000000>:0 
                           at System.Threading.Tasks.Task.FinishContinuations () [0x00000] in <00000000000000000000000000000000>:0 
                           at System.Threading.Tasks.Task`1[TResult].TrySetResult (TResult result) [0x00000] in <00000000000000000000000000000000>:0 
                           at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[TResult].SetResult (TResult result) [0x00000] in <00000000000000000000000000000000>:0 
                           at Mono.Net.Security.AsyncProtocolRequest.ProcessOperation (System.Threading.CancellationToken cancellationToken) [0x00000] in <00000000000000000000000000000000>:0 
                           at System.Threading.ExecutionContext.RunInternal (System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, System.Object state, System.Boolean preserveSyncCtx) [0x00000] in <00000000000000000000000000000000>:0 
                           at System.Runtime.CompilerServices.AsyncMethodBuilderCore+MoveNextRunner.Run () [0x00000] in <00000000000000000000000000000000>:0 
                           at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction (System.Action action, System.Boolean allowInlining, System.Threading.Tasks.Task& currentTask) [0x00000] in <00000000000000000000000000000000>:0 
                           at System.Threading.Tasks.Task.FinishContinuations () [0x00000] in <00000000000000000000000000000000>:0 
                           at System.Threading.Tasks.Task`1[TResult].TrySetResult (TResult result) [0x00000] in <00000000000000000000000000000000>:0 
                           at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[TResult].SetResult (TResult result) [0x00000] in <00000000000000000000000000000000>:0 
                           at Mono.Net.Security.AsyncProtocolRequest.InnerRead (System.Threading.CancellationToken cancellationToken) [0x00000] in <00000000000000000000000000000000>:0 
                           at System.Threading.ExecutionContext.RunInternal (System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, System.Object state, System.Boolean preserveSyncCtx) [0x00000] in <00000000000000000000000000000000>:0 
                           at System.Runtime.CompilerServices.AsyncMethodBuilderCore+MoveNextRunner.Run () [0x00000] in <00000000000000000000000000000000>:0 
                           at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction (System.Action action, System.Boolean allowInlining, System.Threading.Tasks.Task& currentTask) [0x00000] in <00000000000000000000000000000000>:0 
                           at System.Threading.Tasks.Task.FinishContinuations () [0x00000] in <00000000000000000000000000000000>:0 
                           at System.Threading.Tasks.Task`1[TResult].TrySetResult (TResult result) [0x00000] in <00000000000000000000000000000000>:0 
                           at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[TResult].SetResult (TResult result) [0x00000] in <00000000000000000000000000000000>:0 
                           at Mono.Net.Security.MobileAuthenticatedStream.InnerRead (System.Boolean sync, System.Int32 requestedSize, System.Threading.CancellationToken cancellationToken) [0x00000] in <00000000000000000000000000000000>:0 
                           at System.Threading.ExecutionContext.RunInternal (System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, System.Object state, System.Boolean preserveSyncCtx) [0x00000] in <00000000000000000000000000000000>:0 
                           at System.Runtime.CompilerServices.AsyncMethodBuilderCore+MoveNextRunner.Run () [0x00000] in <00000000000000000000000000000000>:0 
                           at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction (System.Action action, System.Boolean allowInlining, System.Threading.Tasks.Task& currentTask) [0x00000] in <00000000000000000000000000000000>:0 
                           at System.Threading.Tasks.Task.FinishContinuations () [0x00000] in <00000000000000000000000000000000>:0 
                           at System.Threading.Tasks.Task`1[TResult].TrySetResult (TResult result) [0x00000] in <00000000000000000000000000000000>:0 
                           at System.Threading.Tasks.ValueTask`1+ValueTaskSourceAsTask+<>c[TResult].<.cctor>b__4_0 (System.Object state) [0x00000] in <00000000000000000000000000000000>:0 
                           at System.Net.Sockets.Socket+<>c.<.cctor>b__367_10 (System.IAsyncResult ares) [0x00000] in <00000000000000000000000000000000>:0 
                           at System.Threading.ThreadPoolWorkQueue.Dispatch () [0x00000] in <00000000000000000000000000000000>:0 
29-7-2022 10:55:17|Trace|TaskToApm.Begin|Begin closing the connection.
29-7-2022 10:55:17|Debug|WebSocket.close|Was clean?: False
                           sent: False
                           received: False
29-7-2022 10:55:17|Trace|TaskToApm.Begin|End closing the connection.
29-7-2022 10:55:17|Info |ExecutionContext.RunInternal|The connection has already been closed.
@jpingen
Copy link
Author

jpingen commented Aug 8, 2022

It appears to run fine as an Android app for mobile.
However, when building an Android apk for the Oculus Quest (same setup, only added an XR rig so it runs on VR) the problem occurs.

@mediumTaj ?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@jpingen