OneDrive Client Version v2.5.0-rc2 is now available for testing

[abraunegg]

Summary of Changes

Important

OneDrive Client for Linux v2.5.0 contains a database schema change to support the additional functionality and bug fixes in the v2.5.x branch. As such, on your first use of this client version, the database schema will be updated and the client will perform an entire re-validation of your data - both locally and online which will take an extended period of time.

Caution

OneDrive Client for Linux v2.5.0 is not backwards compatible. Do not use v2.5.0 and v2.4.x applications at the same time against the same Microsoft OneDrive account. If using v2.5.0, you must upgrade all your clients to v2.5.0

Caution

OneDrive Business Shared Folder Sync has been 100% re-written in v2.5.0. If you are using this feature, please read the new documentation carefully.

Feature Requests

• Implement Feature Request: Multi-threaded uploading/downloading of files FR-232
• Implement Feature Request: Renaming/Relocation of OneDrive Business shared folders FR-1251
• Implement Feature Request: Support the syncing of individual business shared files FR-1300
• Implement Feature Request: Log when manual Authorization is required when using --auth-files FR-2475
• Implement Feature Request: Add cmdline parameter to display (human readable) quota status FR-2359
• Implement Feature Request: Ability to set --disable-download-validation from Docker environment variable FR-2559
• Implement Feature Request: Ability to set --sync-shared-files from Docker environment variable FR-2680

Bug Fixes

• Fix Bug: Business shared folders with same basename get merged BUG-2434
• Fix Bug: --dry-run prevents authorization BUG-2451
• Fix Bug: Log timestamps lacking trailing zeros, leading to poor log file output alignment BUG-2479
• Fix Bug: Subscription ID already exists when using webhooks BUG-2501
• Fix Bug: Not all files being downloaded when API data includes HTML ASCII Control Sequences BUG-2562
• Fix Bug: vim backups when editing files cause edited file to be deleted rather than the edited file being uploaded BUG-2586
• Fix Bug: skip_dir does not always work as intended for all directory entries BUG-2590
• Fix Bug: Online date being changed in download-only mode BUG-2591
• Fix Bug: Resolve that upload session are not canceled with resync option BUG-2624
• Fix Bug: Local files should be safely backed up when the item is not in sync locally to prevent data loss when they are deleted online BUG-2625
• Fix Bug: Synced file is removed when updated on the remote while being processed by onedrive BUG-2699

Other Changes | Fixes

• Implement buffered logging so that all logging from all upload & download activities are handled correctly
• Replace polling monitor loop with blocking wait
• Update how the application utilises curl to fix socket reuse
• Various performance enhancements with many thanks to @JC-comp
• Implement refactored OneDrive API logic with many thanks to @JC-comp
• Enforcement of operational conflicts
• Enforcement of application configuration defaults and minimums
• Implement application output to detail upload|download failures at the end of a sync process
• Utilise threadsafe sqlite DB access methods
• Various code cleanup and optimisations

Documentation

• Most documentation has been totally overhauled

Documentation List

• Advanced Usage
• Application Config Options
• Application Security
• Build RPM How-To
• Business Shared Items
• Client Architecture
• Docker Installation
• Install
• Known Issues
• National Cloud Deployments
• Podman Installation
• Privacy Policy
• SharePoint Libraries
• Terms of Service
• Ubuntu Package Install
• Usage

How can you get v2.5.0-rc2 ?

You need to build this client from source. Follow the high level directions here to get the development requirements installed first: https://github.com/abraunegg/onedrive/blob/master/docs/INSTALL.md#building-from-source---high-level-requirements

Next clone and checkout the PR:

git clone https://github.com/abraunegg/onedrive.git
cd onedrive
git fetch origin pull/2709/head:pr2709
git checkout pr2709
./configure --enable-debug; make clean; make;

To run the PR, you need to run the client from the PR build directory:

./onedrive <any options needed>

When running the PR, your version should be as per below or greater:

onedrive v2.5.0-rc2

The important point is onedrive v2.5.0-rc2 as this will change as new commits are put into the Release Candidate PR branch, whilst things are added/fixed/updated.

[bpozdena]

I've been getting the same crashes on every run since upgrade to RC2.
The workaround for me is to disable sync of business shared folders with sync_business_shared_items = "false"

./onedrive --version
onedrive v2.5.0-rc2-6-g12631f2

 ./onedrive --confdir='/home/bob/.config/onedrive/accounts/[email protected]' --monitor -vvv
DEBUG: Application started
DEBUG: Process ID: 7DFD6B2EF200
DEBUG: User UID:   1000
DEBUG: User GID:   1000
DEBUG: User Name:  bob
DEBUG: Application Version: onedrive v2.5.0-rc2-6-g12631f2
DEBUG: Passed in 'cliArgs': ["./onedrive", "--monitor"]
DEBUG: Note: --confdir and --verbose are not listed in 'cliArgs' array
DEBUG: Passed in --confdir if present: /home/bob/.config/onedrive/accounts/[email protected]
DEBUG: Passed in --verbose count if present: 3
DEBUG: runtime_environment: HOME environment variable detected, expansion of '~' should be possible
DEBUG: runtime_environment: Calculated defaultHomePath: /home/bob
DEBUG: configDirName: CLI override to set configDirName to: /home/bob/.config/onedrive/accounts/[email protected]
DEBUG: refreshTokenFilePath =   /home/bob/.config/onedrive/accounts/[email protected]/refresh_token
DEBUG: deltaLinkFilePath =      /home/bob/.config/onedrive/accounts/[email protected]/delta_link
DEBUG: databaseFilePath =       /home/bob/.config/onedrive/accounts/[email protected]/items.sqlite3
DEBUG: databaseFilePathDryRun = /home/bob/.config/onedrive/accounts/[email protected]/items-dryrun.sqlite3
DEBUG: uploadSessionFilePath =  /home/bob/.config/onedrive/accounts/[email protected]/session_upload
DEBUG: userConfigFilePath =     /home/bob/.config/onedrive/accounts/[email protected]/config
DEBUG: syncListFilePath =       /home/bob/.config/onedrive/accounts/[email protected]/sync_list
DEBUG: systemConfigFilePath =   config
DEBUG: configBackupFile =       /home/bob/.config/onedrive/accounts/[email protected]/.config.backup
DEBUG: configHashFile =         /home/bob/.config/onedrive/accounts/[email protected]/.config.hash
DEBUG: syncListHashFile =       /home/bob/.config/onedrive/accounts/[email protected]/.sync_list.hash
DEBUG: Reading configuration file: /home/bob/.config/onedrive/accounts/[email protected]/config
DEBUG: Configuration file successfully loaded
DEBUG: Using 'user' configuration path for application state data: /home/bob/.config/onedrive/accounts/[email protected]
DEBUG: Disabling GUI notifications as per user configuration
DEBUG: Configuring Client Side Filtering (Selective Sync)
DEBUG: Configuring skip_dir ...
DEBUG: skip_dir: __pycache__|.git|venv|venv2|env|cache|env-linux|uploads|qqqqqqqqqq
DEBUG: Selective Sync Directory Mask: Regex!char("^__pycache__$|^\\.git$|^venv$|^venv2$|^env$|^cache$|^env-linux$|^uploads$|^qqqqqqqqqq$", "i")
DEBUG: Configuring skip_dir_strict_match ...
DEBUG: skip_dir_strict_match: false
DEBUG: Configuring skip_dotfiles ...
DEBUG: skip_dotfiles: true
DEBUG: Configuring skip_file ...
DEBUG: skip_file: ~*|.~*|*.tmp|*.swp|*.partial
DEBUG: Selective Sync File Mask: Regex!char("^~.*$|^\\.~.*$|^.*\\.tmp$|^.*\\.swp$|^.*\\.partial$", "i")
DEBUG: sync_dir: Setting runtimeSyncDirectory from config value 'sync_dir'
DEBUG: sync_dir: Using configured 'sync_dir' path as-is as however SHELL or USER environment variable configuration detected - should be placed in USER home directory
DEBUG: use 'sync_dir' as is - no touch
DEBUG: sync_dir: runtimeSyncDirectory set to: /home/bob/OneDrive
DEBUG: Configuring default new folder permissions as: 700
DEBUG: Configuring default new file permissions as: 600
DEBUG: Application configuration file has been updated, checking if --resync needed
DEBUG: Using this configBackupFile: /home/bob/.config/onedrive/accounts/[email protected]/.config.backup
DEBUG: Backup Config Key: sync_dir
DEBUG: Backup Config Key: disable_notifications
DEBUG: Backup Config Key: local_first
DEBUG: Backup Config Key: skip_dotfiles
DEBUG: Backup Config Key: sync_business_shared_items
DEBUG: Backup Config Key: data_timeout
DEBUG: Backup Config Key: threads
DEBUG: Backup Config Key: skip_dir
DEBUG: Backup Config Key: display_memory
DEBUG: Creating a backup of the applicable config file
DEBUG: Updating applicable config file hash
DEBUG: Updating sync_list hash
DEBUG: Using IPv4 and IPv6 (if configured) for all network operations
DEBUG: Testing network to ensure network connectivity to Microsoft OneDrive Service
DEBUG: Attempting to contact Microsoft OneDrive Login Service
DEBUG: Successfully reached Microsoft OneDrive Login Service
DEBUG: Checking Application Version ...
DEBUG: applicationVersion:       2.5.0
DEBUG: latestVersion:            2.4.25
DEBUG: publishedDate:            2023-Jun-21 05:30:55Z
DEBUG: currentTime:              2024-May-06 06:10:18Z
DEBUG: releaseGracePeriod:       2023-Jul-21 05:30:55Z
DEBUG: Attempting to initialise the OneDrive API ...
DEBUG: CURL ENGINE AVAILABLE POOL SIZE: 0
DEBUG: CURL ENGINE POOL EMPTY - CONSTRUCTING A NEW CURL ENGINE INSTANCE
DEBUG: http.dnsTimeout = 60
DEBUG: http.connectTimeout = 10
DEBUG: http.dataTimeout = 5
DEBUG: http.operationTimeout = 3600
DEBUG: http.maxRedirects = 5
DEBUG: http.CurlOption.ipresolve = 0
DEBUG: http.header.Connection.keepAlive = true
DEBUG: Configuring Global Azure AD Endpoints
DEBUG: Authorised State: true
DEBUG: The OneDrive API was initialised successfully
DEBUG: Configured clientId          d50ca740-c83f-4d1b-b616-12c519384f0c
DEBUG: Configured userAgent         ISV|abraunegg|OneDrive Client for Linux/v2.5.0-rc2-6-g12631f2
DEBUG: Configured authScope:        &scope=Files.ReadWrite%20Files.ReadWrite.All%20Sites.ReadWrite.All%20offline_access&response_type=code&prompt=login&redirect_uri=
DEBUG: Configured authUrl:          https://login.microsoftonline.com/common/oauth2/v2.0/authorize
DEBUG: Configured redirectUrl:      https://login.microsoftonline.com/common/oauth2/nativeclient
DEBUG: Configured tokenUrl:         https://login.microsoftonline.com/common/oauth2/v2.0/token
DEBUG: Configured driveUrl:         https://graph.microsoft.com/v1.0/me/drive
DEBUG: Configured driveByIdUrl:     https://graph.microsoft.com/v1.0/drives/
DEBUG: Configured sharedWithMeUrl:  https://graph.microsoft.com/v1.0/me/drive/sharedWithMe
DEBUG: Configured itemByIdUrl:      https://graph.microsoft.com/v1.0/me/drive/items/
DEBUG: Configured itemByPathUrl:    https://graph.microsoft.com/v1.0/me/drive/root:/
DEBUG: Configured siteSearchUrl:    https://graph.microsoft.com/v1.0/sites?search
DEBUG: Configured siteDriveUrl:     https://graph.microsoft.com/v1.0/sites/
DEBUG: OneDrive API releaseCurlEngine() CALLED
DEBUG: CurlEngine releaseEngine() CALLED
DEBUG: CURRENT CURL ENGINE AVAILABLE POOL SIZE: 0
DEBUG: CurlEngine cleanup() CALLED
DEBUG: CURL ENGINE POOL SIZE AFTER RELEASE BACK TO POOL: 1
DEBUG: Opening the item database ...
DEBUG: Threadsafe database value: 1
DEBUG: PROCESS POOL WORKER THREADS: 16
DEBUG: Using Curl defaults for HTTP operational protocol version (potentially HTTP/2)
DEBUG: CURL ENGINE AVAILABLE POOL SIZE: 1
DEBUG: CURL ENGINE WAS VALID - RETURNED AN EXISTING CURL ENGINE INSTANCE
DEBUG: CURL ENGINE ID: ukAvn36tN9jrJc4X
DEBUG: http.dnsTimeout = 60
DEBUG: http.connectTimeout = 10
DEBUG: http.dataTimeout = 5
DEBUG: http.operationTimeout = 3600
DEBUG: http.maxRedirects = 5
DEBUG: http.CurlOption.ipresolve = 0
DEBUG: http.header.Connection.keepAlive = true
DEBUG: Read token from appConfig
DEBUG: Authorised State: true
DEBUG: CURL ENGINE AVAILABLE POOL SIZE: 0
DEBUG: CURL ENGINE POOL EMPTY - CONSTRUCTING A NEW CURL ENGINE INSTANCE
DEBUG: http.dnsTimeout = 60
DEBUG: http.connectTimeout = 10
DEBUG: http.dataTimeout = 5
DEBUG: http.operationTimeout = 3600
DEBUG: http.maxRedirects = 5
DEBUG: http.CurlOption.ipresolve = 0
DEBUG: http.header.Connection.keepAlive = true
DEBUG: Read token from appConfig
DEBUG: Authorised State: true
DEBUG: Getting Account Default Drive Details
DEBUG: Request URL = https://graph.microsoft.com/v1.0/me/drive
DEBUG: Microsoft OneDrive Access Token has EXPIRED. Must generate a new Microsoft OneDrive Access Token
DEBUG: Need to generate a new access token for Microsoft OneDrive
DEBUG: Request URL = https://login.microsoftonline.com/common/oauth2/v2.0/token
...
* Connection #0 to host graph.microsoft.com left intact
DEBUG: CurlEngine cleanup() CALLED
DEBUG: OneDrive API Response:
----
{"@odata.context":"https://graph.microsoft.com/v1.0/$metadata#Collection(driveItem)/$entity","@microsoft.graph.Decorator":"decorator has been deprecated. Refer to folder.decorator","createdBy":{"application":{"id":"c58637bb-e2e1-4312-8a00-04b5ffcd3403","displayName":"SharePoint Online Client Extensibility"},"user":{"email":"[email protected]","id":"b932219d-e08c-497a-8968-FFFFFFFFFF","displayName":"FName FLastname"}},"createdDateTime":"2024-03-18T16:10:03Z","eTag":"\"{310CED80-0B00-4D29-891D-CCCCCCCCCC},13\"","id":"EEEEEEEEEE","lastModifiedBy":{"user":{"email":"[email protected]","id":"b932219d-e08c-497a-8968-FFFFFFFFFF","displayName":"FName FLastname"}},"lastModifiedDateTime":"2024-03-18T16:10:02Z","name":"9148836","parentReference":{"driveType":"business","driveId":"b!AAAAAAAAAA","id":"DDDDDDDDDD","name":"Documents","path":"/drives/b!AAAAAAAAAA/root:","siteId":"a53f7d31-4abd-4849-bb1a-BBBBBBBBBB"},"cTag":"\"c:{310CED80-0B00-4D29-891D-CCCCCCCCCC},1\"","fileSystemInfo":{"createdDateTime":"2024-03-18T16:10:03Z","lastModifiedDateTime":"2024-03-18T16:10:02Z"},"remoteItem":{"folder":{"childCount":10},"id":"YYYYYYYYYY","parentReference":{"driveType":"documentLibrary","driveId":"XXXXXXXXXX"},"sharepointIds":{"listId":"967f2e82-4f50-480b-aaf6-b8e87b4040aa","listItemUniqueId":"304c22c6-f0e7-44bb-84c9-e2c049ed4eeb","siteId":"09a383f2-68ce-45b2-88c1-95c0dc2b4601","siteUrl":"https://company.sharepoint.com/sites/Test-FILES","webId":"f6a3e32c-cfe6-458e-8b85-1e7724494770","webRelativeListUrl":"Test_Test_Shared%20Files"},"size":14949427},"size":0}
----

* Found bundle for host: 0x5fb7cc1d4fe0 [can multiplex]
* Re-using existing connection with host graph.microsoft.com
DEBUG: onlinePathData: {"@microsoft.graph.Decorator":"decorator has been deprecated. Refer to folder.decorator","@odata.context":"https:\/\/graph.microsoft.com\/v1.0\/$metadata#Collection(driveItem)\/$entity","cTag":"\"c:{310CED80-0B00-4D29-891D-CCCCCCCCCC},1\"","createdBy":{"application":{"displayName":"SharePoint Online Client Extensibility","id":"c58637bb-e2e1-4312-8a00-04b5ffcd3403"},"user":{"displayName":"FName FLastname","email":"[email protected]","id":"b932219d-e08c-497a-8968-FFFFFFFFFF"}},"createdDateTime":"2024-03-18T16:10:03Z","eTag":"\"{310CED80-0B00-4D29-891D-CCCCCCCCCC},13\"","fileSystemInfo":{"createdDateTime":"2024-03-18T16:10:03Z","lastModifiedDateTime":"2024-03-18T16:10:02Z"},"id":"EEEEEEEEEE","lastModifiedBy":{"user":{"displayName":"FName FLastname","email":"[email protected]","id":"b932219d-e08c-497a-8968-FFFFFFFFFF"}},"lastModifiedDateTime":"2024-03-18T16:10:02Z","name":"9148836","parentReference":{"driveId":"b!AAAAAAAAAA","driveType":"business","id":"DDDDDDDDDD","name":"Documents","path":"\/drives\/b!AAAAAAAAAA\/root:","siteId":"a53f7d31-4abd-4849-bb1a-BBBBBBBBBB"},"remoteItem":{"folder":{"childCount":10},"id":"YYYYYYYYYY","parentReference":{"driveId":"XXXXXXXXXX","driveType":"documentLibrary"},"sharepointIds":{"listId":"967f2e82-4f50-480b-aaf6-b8e87b4040aa","listItemUniqueId":"304c22c6-f0e7-44bb-84c9-e2c049ed4eeb","siteId":"09a383f2-68ce-45b2-88c1-95c0dc2b4601","siteUrl":"https:\/\/company.sharepoint.com\/sites\/Test-FILES","webId":"f6a3e32c-cfe6-458e-8b85-1e7724494770","webRelativeListUrl":"Test_Test_Shared%20Files"},"size":14949427},"size":0}* [HTTP/2] [2343] OPENED stream for https://graph.microsoft.com/v1.0/drives/XXXXXXXXXX/items/YYYYYYYYYY

* [HTTP/2] [2343] [:method: GET]
DEBUG: Existing Remote Online Folder is most likely a OneDrive Shared Business Folder Link added by 'Add shortcut to My files'*
[HTTP/2] [2343] [:scheme: https]
DEBUG: Request URL = https://graph.microsoft.com/v1.0/drives/XXXXXXXXXX/items/YYYYYYYYYY*
[HTTP/2] [2343] [:authority: graph.microsoft.com]
DEBUG: Existing Microsoft OneDrive Access Token Expires: 2024-May-06 10:29:57.2973988*
[HTTP/2] [2343] [:path: /v1.0/drives/XXXXXXXXXX/items/YYYYYYYYYY]
* [HTTP/2] [2343] [user-agent: ISV|abraunegg|OneDrive Client for Linux/v2.5.0-rc2-6-g12631f2]
* [HTTP/2] [2343] [accept: */*]
* [HTTP/2] [2343] [authorization: bearer ZZZZZZZZZZ.ZZZZZZZZZZ.ZZZZZZZZZZ-ZZZZZZZZZZ-ZZZZZZZZZZ-ZZZZZZZZZZ-ZZZZZZZZZZ-FqoYkfn453JrOnPztzlF2KHW7buaZ3buFuz6DZZZZZ
> GET /v1.0/drives/XXXXXXXXXX/items/YYYYYYYYYY HTTP/2
Host: graph.microsoft.com
User-Agent: ISV|abraunegg|OneDrive Client for Linux/v2.5.0-rc2-6-g12631f2
Accept: */*
Authorization: bearer ZZZZZZZZZZ.ZZZZZZZZZZ.ZZZZZZZZZZ-ZZZZZZZZZZ-ZZZZZZZZZZ-ZZZZZZZZZZ-ZZZZZZZZZZ-FqoYkfn453JrOnPztzlF2KHW7buaZ3buFuz6DO2gpUf-rm0C4w1XPxoaG8bZiL0K0D-UtSQiV4iqnxUxiUGhv_GTDUqhPUcwEbpGGtse23SSNWF1kCSlMpXVr1kuFs278YjJoeRZSlj51kNayVLcQ8zgljuuZZZZZ

* Request completely sent off
< HTTP/2 404
< cache-control: no-store, no-cache
< content-type: application/json; odata.metadata=minimal; odata.streaming=true; IEEE754Compatible=false; charset=utf-8
< strict-transport-security: max-age=31536000
< request-id: 86ab3d6b-be72-4842-9191-640e268cee60
< client-request-id: 86ab3d6b-be72-4842-9191-640e268cee60
< x-ms-ags-diagnostic: {"ServerInfo":{"DataCenter":"Germany West Central","Slice":"E","Ring":"4","ScaleUnit":"001","RoleInstance":"FR2PEPF000003EB"}}
< date: Mon, 06 May 2024 07:08:32 GMT
<
* Connection #0 to host graph.microsoft.com left intact
DEBUG: CurlEngine cleanup() CALLED
DEBUG: OneDrive API Response:
----
{"error":{"code":"itemNotFound","message":"The resource could not be found."}}
----

DEBUG: Handling a OneDrive API exception:
DEBUG: Curl debugging:
< get https://graph.microsoft.com/v1.0/drives/XXXXXXXXXX/items/YYYYYYYYYY
< 404  (2.0)
Curl response:

----
{"error":{"code":"itemNotFound","message":"The resource could not be found."}}
----

DEBUG: onedrive.OneDriveApi.getPathDetailsById() - Generated a OneDriveException
DEBUG: Configuring libcurl to use a fresh connection for re-try
DEBUG: Failure scope was called
DEBUG: performSynchronisedExitProcess called by: failureScope
DEBUG: Shutdown Client Side Filtering instance
DEBUG: DESTROY ALL CURL ENGINES
DEBUG: CURL ENGINES TO RELEASE: 2
DEBUG: CurlEngine cleanup() CALLED
DEBUG: HTTP SHUTDOWN CALLED ...
DEBUG: CurlEngine cleanup() CALLED
DEBUG: HTTP SHUTDOWN CALLED ...
DEBUG: Shutdown Sync Engine instance
DEBUG: SYNC-ENGINE: Waiting for all internal threads to complete
DEBUG: COMPILER VERSION: 2107
DEBUG: COMPILER VERSION: 2107
DEBUG: Shutdown Filesystem Monitoring instance
DEBUG: Monitored directory removed: Python/junk/OneDriveGUI_POC/resources/images/
DEBUG: Monitored directory removed: Python/FAC_Reader_public/bin/openssl-1.0.2o/crypto/bn/asm/
...
DEBUG: Monitored directory removed: Python/junk/FAC_Reader_public - Copy/system/static/
DEBUG: Monitored directory removed: Python/FAC_Reader_public/bin/openssl-1.0.2o/crypto/md5/asm/
DEBUG: Shutdown Database instance
DEBUG: Attempting to perform a database vacuum to merge any temporary data
DEBUG: Database vacuum is complete
DEBUG: Shutdown Application Configuration instance
DEBUG: Application is exiting.
[1]    73662 segmentation fault (core dumped)  ./onedrive  --monitor -vvv --resync

[abraunegg]

@bpozdena
So when I add a new folder to a Business Shared Folder, this is what I get:

Processing: Sub Folder 2/Documents/keepass-minipc-windows.kdbx
The file has not changed
Scanning the local file system '~/OneDrive' for new data to upload
Skipping item - excluded by sync_list config: ./random_25k_files
New directories to create on Microsoft OneDrive: 1
OneDrive Client requested to create this directory online: ./Sub Folder 2/new_directory
OneDrive Client requested to create this directory online: ./Sub Folder 2/new_directory
OneDrive Client requested to create this directory online: ./Sub Folder 2/new_directory
OneDrive Client requested to create this directory online: ./Sub Folder 2/new_directory
OneDrive Client requested to create this directory online: ./Sub Folder 2/new_directory
OneDrive Client requested to create this directory online: ./Sub Folder 2/new_directory
OneDrive Client requested to create this directory online: ./Sub Folder 2/new_directory
OneDrive Client requested to create this directory online: ./Sub Folder 2/new_directory
OneDrive Client requested to create this directory online: ./Sub Folder 2/new_directory
OneDrive Client requested to create this directory online: ./Sub Folder 2/new_directory
OneDrive Client requested to create this directory online: ./Sub Folder 2/new_directory
OneDrive Client requested to create this directory online: ./Sub Folder 2/new_directory
OneDrive Client requested to create this directory online: ./Sub Folder 2/new_directory
OneDrive Client requested to create this directory online: ./Sub Folder 2/new_directory
OneDrive Client requested to create this directory online: ./Sub Folder 2/new_directory
OneDrive Client requested to create this directory online: ./Sub Folder 2/new_directory
OneDrive Client requested to create this directory online: ./Sub Folder 2/new_directory
OneDrive Client requested to create this directory online: ./Sub Folder 2/new_directory
OneDrive Client requested to create this directory online: ./Sub Folder 2/new_directory
OneDrive Client requested to create this directory online: ./Sub Folder 2/new_directory
OneDrive Client requested to create this directory online: ./Sub Folder 2/new_directory
OneDrive Client requested to create this directory online: ./Sub Folder 2/new_directory
OneDrive Client requested to create this directory online: ./Sub Folder 2/new_directory
OneDrive Client requested to create this directory online: ./Sub Folder 2/new_directory
OneDrive Client requested to create this directory online: ./Sub Folder 2/new_directory
^C
Received termination signal, initiating cleanup
Waiting for any existing upload|download process to complete

The online folder never gets created .. so there is an issue. This was working before this was merged in: https://github.com/abraunegg/onedrive/pull/2686/commits

I will go back and work out what is up with this tomorrow.

However .. I get zero crash or failure on exit .......... as I said I am parking that one for the moment. I suspect that this has to do with something compiler related, and, will need to build an ArchLinux system and test there - so if you could advise what compiler you are using on your system so I can try and replicate that would be greatly appreciated.

[aothmane-control]

I am using ldc2-1.28.0-linux-x86_64 I will try to post a bug report later today, if I have time

[abraunegg]

As suspected, this is a regression with 'rc2' I had not yet picked up as rolling back to 'rc1' yields this:

The file has not changed
Processing: Sub Folder 2/Documents/keepass-minipc-windows.kdbx
The file has not changed
Scanning the local file system '~/OneDrive' for new data to upload
Skipping item - excluded by sync_list config: ./random_25k_files
OneDrive Client requested to create this directory online: ./Sub Folder 2/new_directory
The requested directory to create was not found on OneDrive - creating remote directory: ./Sub Folder 2/new_directory
Successfully created the remote directory ./Sub Folder 2/new_directory on Microsoft OneDrive
New items to upload to OneDrive: 1
Total New Data to Upload:        12 KB
WARNING: Shared Folder OneDrive quota information is being restricted or providing a zero value. Please fix by speaking to your OneDrive / Office 365 Administrator.
Uploading new file: ./Sub Folder 2/new_directory/dummy_file.docx ... done.
Performing a last examination of the most recent online data within Microsoft OneDrive to complete the reconciliation process
Fetching /delta response from the OneDrive API for Drive ID: b!bO8V7s9SSk6r7mWHpIjURotN33W1W2tEv3OXV_oFIdQimEdOHR-1So7CqeT1MfHA
Processing API Response Bundle: 1 - Quantity of 'changes|items' in this bundle to process: 0
Finished processing /delta JSON response from the OneDrive API
No additional changes or items that can be applied were discovered while processing the data received from Microsoft OneDrive
Syncing this OneDrive Business Shared Folder: Sub Folder 2
Fetching /delta response from the OneDrive API for Drive ID: b!fZgJhK-pU0eTQpylvmoYCkE4YgH_KRNDlxjRx9OWNqmV9Q_E_uWdRJKIB5L_ruPN
Processing API Response Bundle: 1 - Quantity of 'changes|items' in this bundle to process: 3
Finished processing /delta JSON response from the OneDrive API
Processing 3 applicable changes and items received from Microsoft OneDrive
Processing OneDrive JSON item batch [1/1] to ensure consistent local state

Sync with Microsoft OneDrive is complete
Waiting for all internal threads to complete before exiting application

[abraunegg]

So I have traced back to where the problem lies, at it is to do with the 'retry' handling of API calls introduced by the API refactoring that @JC-comp frameworked as part of #2608

In the rc1 , when the API throws a 404, this tells the application that this item is not there, thus, create or upload to OneDrive.

For OneDrive Business Shared Folders, this mechanism cannot be used, thus the new item has to be searched for, and if it cannot be found, a 404 is manually being thrown. This manual throw in rc1 works without fault, but in rc2 it is triggering a API retry - which causes the loop.

So the fix here is to rework how the search failure is being handled - that is going to take some time.

[abraunegg]

@bpozdena , @aothmane-control
The issue with OneDrive Business Shared Folders should be fixed. I have tested the PR specifically against creating new local directories in the Shared Folder + adding data to those new directories - the regression is fixed and this operates as per rc1

Please rebuild your client so you are running onedrive v2.5.0-rc2-15-ge408145 or greater.

[mozram]

I got this error when renaming file:

DEBUG: Curl debugging: 
< patch https://graph.microsoft.com/v1.0/drives/a51e76518c7ead4f/items/A51E76518C7EAD4F!278715
< If-Match: aQTUxRTc2NTE4QzdFQUQ0RiEyNzg3MTUuMQ
< Content-Type: application/json

----
{"fileSystemInfo":{"lastModifiedDateTime":"2024-05-05T01:52:04Z"},"name":"20240430 - Farmasi.pdf","parentReference":{"id":"A51E76518C7EAD4F!264953"}}
----
< 412  (2.0)
> request-id: cae9a38c-8356-4b97-a41b-58734edebcf4
> x-ms-ags-diagnostic: {"ServerInfo":{"DataCenter":"Southeast Asia","Slice":"E","Ring":"5","ScaleUnit":"002","RoleInstance":"SG1PEPF000011F0"}}
> strict-transport-security: max-age=31536000
> client-request-id: cae9a38c-8356-4b97-a41b-58734edebcf4
> date: Tue, 07 May 2024 22:42:43 GMT
> content-type: application/json
Curl response: 

----
{"error":{"code":"notAllowed","message":"ETag does not match current item's value","innerError":{"code":"preconditionFailed","date":"2024-05-07T22:42:43","request-id":"cae9a38c-8356-4b97-a41b-58734edebcf4","client-request-id":"cae9a38c-8356-4b97-a41b-58734edebcf4"}}}
----

DEBUG: onedrive.OneDriveApi.updateById() - Generated a OneDriveException

• Running this on RPi5 Raspberry Pi OS
• Running via onedrive -vv --monitor
• File modified over FUSE.sshfs, running Fedora 40 KDE, in Dolphin

After this error, this file changes will not sync, but after some time it will do full scan, and will apply the changes correctly. Not sure why it fail when try to do immediately.

[abraunegg]

@mozram
What account type are you using?

[mozram]

Personal OneDrive

[abraunegg]

@mozram
Circling back on this - this is not an error or problem.

What is occurring is the client will rename the file, and because of the file type, Microsoft will 'change' the file online post the rename operation. As the client records the rename response, this 'back end change' that Microsoft does also updates the 'eTag' element - thus when the client attempts to ensure that the time stamp of the file is correct, the client is using the 'eTag' of the rename response (which is now invalid) and the client is unable to update the timestamp using that eTag. The client then drops the 'eTag' and makes the request again which is successful.

Due to you using debug logging, you will be seeing all of this extra to-from occurring.

In a non-verbose scenario, this looks like the following:

[M] Local item moved: pdf/rv215w_admin_en.pdf -> pdf/rv215w_admin_en_renamed.pdf
Moving pdf/rv215w_admin_en.pdf to pdf/rv215w_admin_en_renamed.pdf
OneDrive returned a 'HTTP 412 - Precondition Failed' when attempting to move the file - gracefully handling error

So as far as this query goes - this is 100% normal and not a problem with RC2.

[abraunegg]

@bpozdena , @aothmane-control
Regarding segfault on exit:

[1]    73662 segmentation fault (core dumped)  ./onedrive  --monitor -vvv --resync

I have tracked this down to the synchronised logging that was implemented:

[Thread 0x7ffff62006c0 (LWP 11340) exited]

Thread 1 "onedrive" received signal SIGSEGV, Segmentation fault.
0x00005555555d2f25 in _D3log11addLogEntryFAyaAQeZv (message=..., levels=...) at src/log.d:176
176             logBuffer.logThisMessage(message, levels);

It is not being cleanly shutdown. Will look at this next.

[abraunegg]

@bpozdena , @aothmane-control
Scratch that .. when using ./onedrive --monitor -vvv --resync and then CTRL-C I am still getting a segfault ...

[aothmane-control]

@abraunegg I can confirm that with the new config there is no segfault when doing a sync. When I do ./onedrive -m -v and then CTRL-C I do not get a segfault. I am on Ubuntu.

[abraunegg]

@aothmane-control
Very interesting feedback ...

So .. on my ArchLinux install (using to specifically test @bpozdena issues) ... if I use the distribution provided LDC package pacman -S ldc I get segfaults.

If I manually install ldc 1.37.0 - I do not get a segfault on exit ......

The other interesting item is that the ArchLinux LDC package + manually installed LDC package - all have a problem accessing the HTTP Response Headers (see: ldc-developers/ldc#4654 for details)

Older LDC versions or older|newer DMD version do not exhibit this issue .....

@bpozdena
At the moment I feel like I am chasing a ghost regarding segfaults on ArchLinux.

I can reproduce the segfaults using the ArchLinux distribution package of LDC , but I cannot reproduce any segfault when using that exact same version but manually installed via curl -fsS https://dlang.org/install.sh | bash -s ldc-1.37.0. I did this post @aothmane-control feedback.

Note:
You may need to download install.sh and temporarily comment out the gpg check as there are expired keys:

alex@archlinux ~]$ curl -fsS https://dlang.org/install.sh | bash -s ldc-1.37.0
Downloading https://dlang.org/d-keyring.gpg
######################################################################## 100.0%
Downloading https://dlang.org/install.sh
######################################################################## 100.0%
gpg: directory '/home/alex/.gnupg' created
gpg: /home/alex/.gnupg/trustdb.gpg: trustdb created
gpg: Signature made Tue 02 Apr 2024 07:02:30 AM AEDT
gpg:                using RSA key E22EC04C82780970381402F4A7D4D42F8EC6A355
gpg: Can't check signature: No public key
Invalid signature https://dlang.org/install.sh.sig

I have notified Iain about these expired keys

[JC-comp]

@abraunegg

The other interesting item is that the ArchLinux LDC package + manually installed LDC package - all have a problem accessing the HTTP Response Headers (see: ldc-developers/ldc#4654 for details)

Just FYI, this was a known bug and has been fixed.
https://issues.dlang.org/show_bug.cgi?id=24458

Scratch that .. when using ./onedrive --monitor -vvv --resync and then CTRL-C I am still getting a segfault ...

By the way, the following will trigger a segfault if you exit before initialization.

onedrive/src/main.d

Line 1369 in 2b8d1ad

syncEngineInstance.shutdown();

[abraunegg]

@JC-comp
Thanks for the information

[abraunegg]

@Lyncredible , @bpozdena , @JC-comp
Regarding this specific error:

Aborting from core/sync/mutex.d(149) Error: pthread_mutex_destroy failed.Aborted

This error only occurs when using CTRL-C when using --monitor.

I have traced this back to the following specific PRs:

• OneDrive Client Version v2.5.0-alpha-4 + @JC-comp #2519: Replace polling monitor loop with blocking wait #2582
  • This was a manual addition of Support blocking monitor loop and full scan separation #2519
  • This PR (2582 )introduced this specific error output when using CTRL-C
• Support blocking monitor loop and full scan separation #2519
  • This PR segfaults on exit in the same CTRL-C scenario. Whilst the logging output is not occurring - it is still segfaulting on exit.

As such, PR#2724 removes the following PR's: #2582, #2519, #2609 and #2628. With these removed, there is no segfault on exit when using CTRL-C in --monitor mode.

@Lyncredible , @bpozdena

Please can you test #2724 against your use cases - specifically webhooks as code has modified that touches the webook functionality as part of this revert.

@JC-comp
If you would like to have another shot at introducing that capability to remove the constant polling in --monitor mode, then please do so, otherwise this will not be included as the original PR (#2519) is not clean.

[JC-comp]

Correct - all of which right now should not occur with #2724 ..

Both cases above were tested with #2724 and the errors are still there. But the original case that interrupt ing after "Sync with Microsoft OneDrive is complete" can exit without segfaults.

[abraunegg]

Right .. but can you look at the original RC2 as per above and determine what is causing the actual segfault?

[abraunegg]

@JC-comp

1. Unrecognized option

Are you 100% sure you are doing this correctly? I cannot reproduce even your error on my test system:

alex@debian-12-x64:~/onedrive-pr2724$ ./onedrive -n
Unrecognized option -n
Try 'onedrive -h' for more information
alex@debian-12-x64:~/onedrive-pr2724$ ./onedrive -n

No segfault. Are you actually using #2724 for your testing on a clean branch ?

2. Interrupt during sync

alex@debian-12-x64:~/onedrive-pr2724$ ./onedrive -s
Using IPv4 and IPv6 (if configured) for all network operations
Attempting to contact Microsoft OneDrive Login Service
Successfully reached Microsoft OneDrive Login Service
Configuring Global Azure AD Endpoints
Fetching items from the OneDrive API for Drive ID: 66d53be8a5056eca ..
No changes or items that can be applied were discovered while processing the data received from Microsoft OneDrive
Performing a database consistency and integrity check on locally stored data .
Deleted local items to delete on Microsoft OneDrive: 1
Deleting item from Microsoft OneDrive: newfile_renamed.txt
Scanning the local file system '~/OneDrive' for new data to upload ..
Performing a last examination of the most recent online data within Microsoft OneDrive to complete the reconciliation process
Fetching items from the OneDrive API for Drive ID: 66d53be8a5056eca .^C
Received termination signal, initiating application cleanup
Waiting for any existing upload|download process to complete
Segmentation fault
alex@debian-12-x64:~/onedrive-pr2724$ 

This I can reproduce

[JC-comp]

This is how I build the environment with docker for case 1:

FROM debian:12

RUN apt update && apt install build-essential libcurl4-openssl-dev libsqlite3-dev pkg-config git curl -y
RUN apt install ldc -y

RUN git clone https://github.com/abraunegg/onedrive.git && \
    cd onedrive && \
    git fetch origin pull/2724/head:pr2724 && \
    git checkout pr2724

WORKDIR /onedrive
RUN ./configure --enable-debug; make clean; make;

ENTRYPOINT ./onedrive -n

$ docker build -t onedrive-pr2724 .
$ docker run onedrive-pr2724
Unrecognized option -n
Try 'onedrive -h' for more information
Aborting from core/sync/mutex.d(149) Error: pthread_mutex_destroy failed.Aborted (core dumped)

[abraunegg]

So not replicating the environment fully ... as without Docker that exit crash does not occur.

[abraunegg]

@JC-comp

Please can you test #2728 if you are able to.

This is the alpha-4 'monitor' code , with a revamped logging code.

The focus here is the logging aspect - ensuring that this is working and clean.

On my Debian 12 test system (not using Docker ... a VM install) in all of the tests I have done in comparison to #2724 - I am not getting any segfault at any point despite trying.

Any feedback is greatly appreciated.