iOS,Android: Dropbox sync broken

[personalizedrefrigerator]

Operating system

iOS

Joplin version

13.0.1 (dev, ios)

Desktop version info

No response

Current behaviour

1. Enable Dropbox sync
2. Add a new file
3. Sync
4. Observe that a "Network request failed" is logged and the item isn't uploaded.

Expected behaviour

Dropbox sync should work.

Logs

No response

[personalizedrefrigerator]

Additional details (originally posted on the forum):

I'm able to reproduce this on an iOS simulator. For me, the failing fetch is requesting /info.json from Dropbox. If I use the requestToCurl_ debug method to convert the request to a CURL command¹. If I then run this from a terminal, the request is successful.

The failing request seems to be using React Native's fetch (on this line) (rather than fetchBlob or uploadBlob).

Other users are reporting this on older versions of Joplin.

Footnotes

1. The CURL request is similar to (may have typos): curl -X GET -H 'Dropbox-API-Arg: {"path": "/info.json"}' -H 'Authorization: Bearer TOKEN_HERE' -H 'Content-Type: application/octet-stream' https://content.dropboxapi.com/2/files/download ↩

[laurent22]

At some point we had this "Network request failed" error because we weren't passing the "content-type" header (even when it wasn't needed), but in that case it looks like it's specified. Or maybe it should be application/json in this particular request?

[personalizedrefrigerator]

A Content-Type of application/json doesn't work either. With this, however, the error message is more descriptive:

Synchronizer: [Error: GET files/download: Error (400): Error in call to API function "files/download": Bad HTTP "Content-Type" header: "application/json".  Expecting one of "text/plain", "text/plain; charset=utf-8", "application/octet-stream", "application/octet-stream; charset=utf-8".]

Using text/plain again gives the error,

 ERROR  17:40:01: Synchronizer: [TypeError: Network request failed]

Replacing shim.fetch with shim.debugFetch (which uses XMLHttpRequest) and using text/plain for the content type gives

 INFO  ======================== XHR ERROR
 INFO  null
 INFO  -------------------------------------
 INFO  The network connection was lost.
 INFO  ======================== XHR ERROR

These all fail when fetching /info.json.

Here are a few other things that could be tried:

• Manually checking whether files/download works for other files in the Joplin folder.
• Overwriting info.json with other content, then trying to fetch again.
• Upgrading React Native (we use v0.71, the latest version is v0.74).
  • Upgrading to RN 0.73.8 doesn't seem to help.
• Running an identical fetch within a WebView.
  • This results in a similar error:

  > await fetch( 'https://content.dropboxapi.com/2/files/download' ,  {"headers":{"Dropbox-API-Arg":"{\"path\":\"/info.json\"}","Authorization":"Bearer TOKEN_HERE","Content-Type":"application/octet-stream"},"method":"GET"} )
  [Error] Failed to load resource: The network connection was lost. (download, line 0)
  [Error] TypeError: Load failed
  

  • (WebView only) Changing method from GET to POST results in a successful Response. If this change is made to the Dropbox API, sync still fails on iOS.
    • The historical reason for using GET instead of POST is explained in this comment:
      

      joplin/packages/lib/file-api-driver-dropbox.js

      Lines 126 to 135 in 8e93f09

      	try {
      	// IMPORTANT:
      	//
      	// We cannot use POST here, because iOS (as of version 14?) doesn't
      	// support POST requests with an empty body:
      	//
      	// https://www.dropboxforum.com/t5/Dropbox-API-Support-Feedback/Error-1017-quot-cannot-parse-response-quot/td-p/589595
      	const response = await this.api().exec(
      	'GET',

[personalizedrefrigerator]

I've also noticed this issue on an Android 14 emulator (I've only tested with the React Native 0.73.8 upgrade branch, however).

[laurent22]

I've also noticed this issue on an Android 14 emulator (I've only tested with the React Native 0.73.8 upgrade branch, however).

With Android maybe it's easier to debug - did you try running adb logcat at the same time to see if there's more info in the low-level error messages?

[laurent22]

Also maybe compare the response between running curl on /info.json and another file. Maybe Dropbox started adding some more headers to the JSON responses that makes the RN network lib fail.

[personalizedrefrigerator]

Also maybe compare the response between running curl on /info.json and another file. Maybe Dropbox started adding some more headers to the JSON responses that makes the RN network lib fail.

I've done additional testing on Android.

fetch seems to work for a resource file. When running the following from the Hermes development tools, the following is successful (prints the content of a note)¹:

fetch("https://content.dropboxapi.com/2/files/download", {"headers":{"Dropbox-API-Arg":"{\"path\":\"/faf15418f15e410da0000ac96d8c4c47.md\"}","Authorization":"Bearer TOKEN_HERE","Content-Type":"application/octet-stream"},"method":"GET"})
    .then(e => e.text()).then(t => console.log(t)).catch(e => console.error(e))

Edit: If I run the above fetch twice (rather than just once), the second and subsequent requests fail.

Changing the path to /info.json is still not successful:

fetch("https://content.dropboxapi.com/2/files/download", {"headers":{"Dropbox-API-Arg":"{\"path\":\"/info.json\"}","Authorization":"Bearer TOKEN_HERE","Content-Type":"application/octet-stream"},"method":"GET"})
    .then(e => e.text()).then(t => console.log(t)).catch(e => console.error(e))

Interestingly, sending a POST request works:

// First, fetch with "method": "POST"
fetch("https://content.dropboxapi.com/2/files/download", {"headers":{"Dropbox-API-Arg":"{\"path\":\"/info.json\"}","Authorization":"Bearer TOKEN_HERE","Content-Type":"application/octet-stream"},"method":"POST"})
    .then(e => e.text()).then(t => console.log(t))
    // Fetching a second time, this time with "method": "GET" **also works**:
    .then(() =>
        fetch("https://content.dropboxapi.com/2/files/download", {"headers":{"Dropbox-API-Arg":"{\"path\":\"/info.json\"}","Authorization":"Bearer TOKEN_HERE","Content-Type":"application/octet-stream"},"method":"GET"})
    ).then(e => e.text()).then(t => console.log(t))

Adding a third .then(() => fetch( ... )).then(e => e.text()).then(t => console.log(t)) results in an Unhandled Promise Rejection (id: 3): TypeError: Network request failed, after printing two copies of info.json.

With Android maybe it's easier to debug - did you try running adb logcat at the same time to see if there's more info in the low-level error messages?

When I run adb logcat, I get a large number of unrelated errors per second:

05-04 09:10:44.720   539   679 E ClipboardService: Denying clipboard access to com.android.chrome, application is not in focus nor is it a system service for user 0
05-04 09:10:44.720   539   679 E JavaBinder: !!! FAILED BINDER TRANSACTION !!!  (parcel size = 108)
05-04 09:10:44.720   539   679 E ClipboardService: Denying clipboard access to com.android.chrome, application is not in focus nor is it a system service for user 0
05-04 09:10:44.720   539   679 E JavaBinder: !!! FAILED BINDER TRANSACTION !!!  (parcel size = 108)
05-04 09:10:44.721   539   679 E ClipboardService: Denying clipboard access to com.android.chrome, application is not in focus nor is it a system service for user 0
05-04 09:10:44.721   539   679 E JavaBinder: !!! FAILED BINDER TRANSACTION !!!  (parcel size = 108)
05-04 09:10:44.721   539   679 E ClipboardService: Denying clipboard access to com.android

If I search for request in the output (using adb logcat | grep -i 'request' -B 3), I get output that doesn't seem particularly helpful:

...
--
05-04 09:17:04.231   539   679 E ClipboardService: Denying clipboard access to com.android.chrome, application is not in focus nor is it a system service for user 0
05-04 09:17:04.232   539   653 W PackageManager: Not removing package com.google.android.trichromelibrary hosting lib com.google.android.trichromelibrary version 631209938 used by [VersionedPackage[com.android.chrome/631209938]] for user 0
05-04 09:17:04.232   539   653 W PackageManager: Not removing package com.google.android.trichromelibrary hosting lib com.google.android.trichromelibrary version 631211838 used by [VersionedPackage[com.google.android.webview/631211838]] for user 0
05-04 09:17:04.236   455  9722 D installd: Device /data has 429649920 free; requested 624066560; needed 194416640
--
05-04 09:17:04.264   539   679 E ClipboardService: Denying clipboard access to com.android.chrome, application is not in focus nor is it a system service for user 0
05-04 09:17:04.267   455  9722 D installd: Refusing to clear cached data in reserved space
05-04 09:17:04.267   455  9722 E installd: Failed to free up 624066560 on /data; final free space 429649920: Success
05-04 09:17:04.271  1365  1476 I ExternalStorageServiceImpl: Free cache requested for 194416640 bytes
05-04 09:17:04.272   539   679 E ClipboardService: Denying clipboard access to com.android.chrome, application is not in focus nor is it a system service for user 0
05-04 09:17:04.278   539   679 E ClipboardService: Denying clipboard access to com.android.chrome, application is not in focus nor is it a system service for user 0
05-04 09:17:04.286   539   679 E ClipboardService: Denying clipboard access to com.android.chrome, application is not in focus nor is it a system service for user 0
05-04 09:17:04.286  1365  1476 I ExternalStorageServiceImpl: Free cache requested for 194416640 bytes
--
05-04 09:17:23.771   539   679 E JavaBinder: !!! FAILED BINDER TRANSACTION !!!  (parcel size = 108)
05-04 09:17:23.771   539   679 E ClipboardService: Denying clipboard access to com.android.chrome, application is not in focus nor is it a system service for user 0
05-04 09:17:23.774   539   679 E ClipboardService: Denying clipboard access to com.android.chrome, application is not in focus nor is it a system service for user 0
05-04 09:17:23.776 13576 20973 E ReactNativeJS: '09:17:23: Synchronizer:', [TypeError: Network request failed]

Footnotes

1. Note that Hermes uses code preprocessing to support async and await, so .then is used above. ↩

[personalizedrefrigerator]

I've raised this issue on the Dropbox forum (Edit: The post was marked as spam. Edit 2: It should now be visible).

[rickwendy]

Syncing is working on Mac, but not on any of my iOS devices.

[rickwendy]

To add some symptoms that may help in diagnosis, if I completely delete the iOS app, re-connect the Dropbox link, it syncs fine. Loads all the existing data from the Dropbox vault. But then will not perform any updates after that. Adding or deleting an item on a Mac does not sync to the iOS device. Or in the other direction.

[personalizedrefrigerator]

I've opened a pull request with a workaround for the sync issue. It seems that if Joplin tries to download the same file multiple times with a GET request, the second and subsequent requests fail. The pull request works around the issue by, when a download request fails, requesting another file, then re-trying the original request.

It's strange that repeated GET requests fail on mobile but not on desktop. Maybe this is related to cookies or other information (e.g. a user-agent string) passed with the request?

To add some symptoms that may help in diagnosis, if I completely delete the iOS app, re-connect the Dropbox link, it syncs fine. Loads all the existing data from the Dropbox vault. But then will not perform any updates after that. Adding or deleting an item on a Mac does not sync to the iOS device. Or in the other direction.

For me, the sync errors happen first when requesting info.json. One possibility is that uploading a file causes Joplin to make multiple requests for this file, which breaks future sync (Joplin might request info.json at the beginning of a sync?).

An alternate workaround could be to cache the last request to /files/download. However, I'm unsure how long cached responses would need to be kept and whether this would break downloading changes to files.

[rickwendy]

I don't have a lot of experience with Dropbox programming (I'm a real time embedded engineer) but could the difference between mobile and desktop be that on desktop computers Dropbox is a daemon and the app writes directly to the local Joplin folder for requests? Versus on mobile, the GET requests have to be issued to the Dropbox servers directly?

[laurent22]

I don't have a lot of experience with Dropbox programming (I'm a real time embedded engineer) but could the difference between mobile and desktop be that on desktop computers Dropbox is a daemon and the app writes directly to the local Joplin folder for requests? Versus on mobile, the GET requests have to be issued to the Dropbox servers directly?

No this is not it. There are three reasons for this issue:

• Dropbox changed something to their API (not really a bug, but that's the most likely reason)

• React Native networking library is terrible and fails on basic things that work fine everywhere else.

• React Native error reporting is terrible. HTTP has been around for 33 years and is very well understood, yet RN can't figure out how to display a proper error message and defaults to "Network request failed" for every error, leaving us clueless as to what's happening.

[personalizedrefrigerator]

Re-opening — users are still experiencing Dropbox sync issues: https://discourse.joplinapp.org/t/ios-sync-broke-due-to-network-request-failing-with-dropbox/37915/24.