Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Publishing to ria store #7536

Closed
dickscheid opened this issue Dec 2, 2023 · 2 comments · Fixed by datalad/datalad-next#653 · May be fixed by #7549
Closed

Publishing to ria store #7536

dickscheid opened this issue Dec 2, 2023 · 2 comments · Fixed by datalad/datalad-next#653 · May be fixed by #7549

Comments

@dickscheid
Copy link

What is the problem?

I created a remote RIA store using ria+ssh protocol.
Creation of the remote works without errors and results in the appropriate structure on the ssh remote.
When actually publishing to the ria store however, datalad fails due to invalid permission settings on the remote side:

Unable to remove <my-remote-path>/datastore/bbc/62b4a-cdad-4b17-ba6e-47bb04bb12f0/ora-remote-ec45211b-96b8-48ce-8386-907725f6fac1/transfer/MD5E-s305407--4ed88684b9e0d8db36e433fc771ff7e3.png or to obtain write permission in parent directory. -caused by- stat -f%Dp <my-remote-path>/datastore/bbc/62b4a-cdad-4b17-ba6e-47bb04bb12f0/ora-remote-ec45211b-96b8-48ce-8386-907725f6fac1/transfer failed:
copy: 1 failed

Indeed, the png file in the transfer folder on the target, as well as its parent folder transfer have permissions -r--r--r--. The ore-remote- ... folder has permissions drwxrwxr-x. I have no influence on those permissions since the complete tree is auto-created by datalad.

Using DataLad version 0.19.3 and git version 2.43.0 on a MacBook M1.
I tried different ssh remotes, including the JUDAC system in Jülich and a local ubuntu box.
Done the same with ria+file to a local folder works fine.

What steps will reproduce the problem?

  1. Create test datalad repository
    datalad create test

  2. Add a test file to the repository and datalad save

  3. Create ria remote on ssh machine
    datalad create-sibling-ria -s datastore --new-store-ok "ria+ssh://<user>@<server>/path/to/datastore"

  4. Try to push
    datalad push --to datastore

DataLad information

0.19.3
darwin/23.1.0 14.1.1/arm64

Additional context

No response

Have you had any success using DataLad before?

No response
@dickscheid
Copy link
Author

Just to add, I also tried -f checkdatapresent for the push, even after generating everything from scratch (including the local repo and the remote datastore folder), with the same result.

@mslw
Copy link
Contributor

mslw commented Jan 22, 2024

I think this is indeed a bug in DataLad, where the format of the remotely executed stat command gets chosen based on the local OS:

Inside SSHRemoteIO there's this (scroll to see the conditional):

datalad/datalad/distributed/ora_remote.py

Lines 463 to 492 in 08315c1

def ensure_writeable(self, path):
"""Context manager to get write permission on `path` and restore
original mode afterwards.
If git-annex ever touched the key store, the keys will be in mode 444
directories, and we need to obtain permission first.
Parameters
----------
path: Path
path to the target file
"""
path = sh_quote(str(path))
# remember original mode -- better than to prescribe a fixed mode
if on_osx:
format_option = "-f%Dp"
# on macOS this would return decimal representation of mode (same
# as python's stat().st_mode
conversion = int
else: # win is currently ignored anyway
format_option = "--format=\"%f\""
# in opposition to the above form for macOS, on debian this would
# yield the hexadecimal representation of the mode; hence conversion
# needed.
conversion = functools.partial(int, base=16)
output = self._run(f"stat {format_option} {path}",
no_output=False, check=True)

Note that the "Unable to remove" message contains:

-caused by- stat -f%Dp

Digging a bit deeper, I see that it has been reported previously in a slightly more involved context (issue since transferred to another repo) datalad/datalad-ria#40 and discussed in psychoinformatics-de/knowledge-base#5

Thinking loud about a fix or workaround: I think the simplest way would be to only keep the "else" part of the statement, at the cost of breaking the (arguably rare) cause of mac-to-mac. A general solution requires a method to detect the remote OS.

@mslw mslw mentioned this issue Jan 23, 2024
Open
mih added a commit to mih/datalad-next that referenced this issue Apr 17, 2024
@mih
Add test to document broken RIA SSH access from Mac clients
2089edd 
This is essentially an implementation of the reproducer from
datalad/datalad#7536
@mih mih mentioned this issue Apr 17, 2024
Merged
mih added a commit to mih/datalad-next that referenced this issue Apr 17, 2024
@mih
Add patch to rectify ORA remote remote platform detection
11ab3a1 
Analysis and patch are taken more-or-less exactly from
datalad/datalad#7549. Thanks @mslw!

The patch is slightly modified to be more compact, and work by replacing
`SSHRemoteIO.ensure_writeable()` alone.

Fixes datalad/datalad#7536
mih added a commit to mih/datalad-next that referenced this issue Apr 17, 2024
@mih
Add test to document broken RIA SSH access from Mac clients
2e7221c 
This is essentially an implementation of the reproducer from
datalad/datalad#7536
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
2 participants
@dickscheid @mslw