The role fails if the iSCSI client has a session opened with another iSCSI target

[jesusserranost]

tl;dr;
When running the role against an iSCSI client which is already connected to a different target than we want to connect now, it fails consistently unless you log out from the session with the other iSCSI target.

Long version:
When there is an already configured cluster with shared storage coming from an iSCSI targeta and you want to re-run the role in order to present a new shared storage from another iSCSI targetb, the role fails with the following (verbose) error message:

failed: [192.168.42.150] (item={'path': '/var/lib/iscsi/send_targets/192.168.42.191,3260/iqn.1994-05.com.redhat:targetb,192.168.42.191,3260,1,default', 'mode': '0777', 'isdir': False, 'ischr': False, 'isblk': False, 'isreg': False, 'isfifo': False, 'islnk': True, 'issock': False, 'uid': 0, 'gid': 0, 'size': 73, 'inode': 13535614, 'dev': 64768, 'nlink': 1, 'atime': 1708355481.779, 'mtime': 1708355481.391, 'ctime': 1708355481.391, 'gr_name': 'root', 'pw_name': 'root', 'wusr': True, 'rusr': True, 'xusr': True, 'wgrp': True, 'rgrp': True, 'xgrp': True, 'woth': True, 'roth': True, 'xoth': True, 'isuid': False, 'isgid': False}) => {
    "ansible_loop_var": "item",
    "changed": false,
    "cmd": "grep -c -E \"^iqn.1994-05.com.redhat:targetb$\" /sys/class/iscsi_session/*/targetname",
    "delta": "0:00:00.003392",
    "end": "2024-02-19 16:41:52.010114",
    "invocation": {
        "module_args": {
            "_raw_params": "grep -c -E \"^iqn.1994-05.com.redhat:targetb$\" /sys/class/iscsi_session/*/targetname",
            "_uses_shell": true,
            "argv": null,
            "chdir": null,
            "creates": null,
            "executable": null,
            "removes": null,
            "stdin": null,
            "stdin_add_newline": true,
            "strip_empty_ends": true
        }
    },
    "item": {
        "atime": 1708355481.779,
        "ctime": 1708355481.391,
        "dev": 64768,
        "gid": 0,
        "gr_name": "root",
        "inode": 13535614,
        "isblk": false,
        "ischr": false,
        "isdir": false,
        "isfifo": false,
        "isgid": false,
        "islnk": true,
        "isreg": false,
        "issock": false,
        "isuid": false,
        "mode": "0777",
        "mtime": 1708355481.391,
        "nlink": 1,
        "path": "/var/lib/iscsi/send_targets/192.168.42.191,3260/iqn.1994-05.com.redhat:targetb,192.168.42.191,3260,1,default",
        "pw_name": "root",
        "rgrp": true,
        "roth": true,
        "rusr": true,
        "size": 73,
        "uid": 0,
        "wgrp": true,
        "woth": true,
        "wusr": true,
        "xgrp": true,
        "xoth": true,
        "xusr": true
    },
    "msg": "non-zero return code",
    "rc": 1,
    "start": "2024-02-19 16:41:52.006722",
    "stderr": "",
    "stderr_lines": [],
    "stdout": "0",
    "stdout_lines": [
        "0"
    ]
}
<192.168.42.151> (0, b'', b'')
failed: [192.168.42.151] (item={'path': '/var/lib/iscsi/send_targets/192.168.42.191,3260/iqn.1994-05.com.redhat:targetb,192.168.42.191,3260,1,default', 'mode': '0777', 'isdir': False, 'ischr': False, 'isblk': False, 'isreg': False, 'isfifo': False, 'islnk': True, 'issock': False, 'uid': 0, 'gid': 0, 'size': 73, 'inode': 1270206, 'dev': 64768, 'nlink': 1, 'atime': 1708355480.908, 'mtime': 1708355480.537, 'ctime': 1708355480.537, 'gr_name': 'root', 'pw_name': 'root', 'wusr': True, 'rusr': True, 'xusr': True, 'wgrp': True, 'rgrp': True, 'xgrp': True, 'woth': True, 'roth': True, 'xoth': True, 'isuid': False, 'isgid': False}) => {
    "ansible_loop_var": "item",
    "changed": false,
    "cmd": "grep -c -E \"^iqn.1994-05.com.redhat:targetb$\" /sys/class/iscsi_session/*/targetname",
    "delta": "0:00:00.004205",
    "end": "2024-02-19 16:41:51.150517",
    "invocation": {
        "module_args": {
            "_raw_params": "grep -c -E \"^iqn.1994-05.com.redhat:targetb$\" /sys/class/iscsi_session/*/targetname",
            "_uses_shell": true,
            "argv": null,
            "chdir": null,
            "creates": null,
            "executable": null,
            "removes": null,
            "stdin": null,
            "stdin_add_newline": true,
            "strip_empty_ends": true
        }
    },
    "item": {
        "atime": 1708355480.908,
        "ctime": 1708355480.537,
        "dev": 64768,
        "gid": 0,
        "gr_name": "root",
        "inode": 1270206,
        "isblk": false,
        "ischr": false,
        "isdir": false,
        "isfifo": false,
        "isgid": false,
        "islnk": true,
        "isreg": false,
        "issock": false,
        "isuid": false,
        "mode": "0777",
        "mtime": 1708355480.537,
        "nlink": 1,
        "path": "/var/lib/iscsi/send_targets/192.168.42.191,3260/iqn.1994-05.com.redhat:targetb,192.168.42.191,3260,1,default",
        "pw_name": "root",
        "rgrp": true,
        "roth": true,
        "rusr": true,
        "size": 73,
        "uid": 0,
        "wgrp": true,
        "woth": true,
        "wusr": true,
        "xgrp": true,
        "xoth": true,
        "xusr": true
    },
    "msg": "non-zero return code",
    "rc": 1,
    "start": "2024-02-19 16:41:51.146312",
    "stderr": "",
    "stderr_lines": [],
    "stdout": "0",
    "stdout_lines": [
        "0"
    ]
}

I think the error comes from this line - shouldn't the role rather evaluate the output (because the grep uses -c) rather than the return code?:

ansible.iscsiadm/tasks/connect_to_target.yml

Line 24 in 5687c77

shell: grep -c -E "^{{ (item.path|basename).split(',')[0] }}$" /sys/class/iscsi_session/*/targetname

Taking node1 as example, we can see that there is a session opened but with another target (targeta) different than the one we want to define now (targetb), yet the output of the command returns 1:

[root@fastvm-rhel-8-4-150 ~]# cat /sys/class/iscsi_session/session1/targetname
iqn.1994-05.com.redhat:targeta
[root@fastvm-rhel-8-4-150 ~]# grep -c -E \"^iqn.1994-05.com.redhat:targetb$\" /sys/class/iscsi_session/*/targetname
0
[root@fastvm-rhel-8-4-150 ~]# echo $?
1

If I manually logout from that session on node1 and re-run the playbook, it works and the task only fails for the other node (where there is still a session opened with a different iSCSI target targeta):

[root@fastvm-rhel-8-4-150 ~]# cat /sys/class/iscsi_session/session1/targetname
iqn.1994-05.com.redhat:targeta
[root@fastvm-rhel-8-4-150 ~]# cat /sys/class/iscsi_session/session1/state 
LOGGED_IN
[root@fastvm-rhel-8-4-150 ~]# iscsiadm --mode node --targetname iqn.1994-05.com.redhat:targeta --portal 192.168.42.190:3260 --logout
Logging out of session [sid: 1, target: iqn.1994-05.com.redhat:targeta, portal: 192.168.42.190,3260]
Logout of [sid: 1, target: iqn.1994-05.com.redhat:targeta, portal: 192.168.42.190,3260] successful.

The (non-verbose) output this time after re-running the role is the following:

TASK [ondrejhome.iscsiadm : Find out if we are already logged in on the target] ********************************************************************************************************************************
skipping: [192.168.42.150] => (item={'path': '/var/lib/iscsi/send_targets/192.168.42.191,3260/iqn.1994-05.com.redhat:targetb,192.168.42.191,3260,1,default', 'mode': '0777', 'isdir': False, 'ischr': False, 'isblk': False, 'isreg': False, 'isfifo': False, 'islnk': True, 'issock': False, 'uid': 0, 'gid': 0, 'size': 73, 'inode': 13535614, 'dev': 64768, 'nlink': 1, 'atime': 1708355481.779, 'mtime': 1708355481.391, 'ctime': 1708355481.391, 'gr_name': 'root', 'pw_name': 'root', 'wusr': True, 'rusr': True, 'xusr': True, 'wgrp': True, 'rgrp': True, 'xgrp': True, 'woth': True, 'roth': True, 'xoth': True, 'isuid': False, 'isgid': False}) 
skipping: [192.168.42.150]
failed: [192.168.42.151] (item={'path': '/var/lib/iscsi/send_targets/192.168.42.191,3260/iqn.1994-05.com.redhat:targetb,192.168.42.191,3260,1,default', 'mode': '0777', 'isdir': False, 'ischr': False, 'isblk': False, 'isreg': False, 'isfifo': False, 'islnk': True, 'issock': False, 'uid': 0, 'gid': 0, 'size': 73, 'inode': 1270206, 'dev': 64768, 'nlink': 1, 'atime': 1708355480.908, 'mtime': 1708355480.537, 'ctime': 1708355480.537, 'gr_name': 'root', 'pw_name': 'root', 'wusr': True, 'rusr': True, 'xusr': True, 'wgrp': True, 'rgrp': True, 'xgrp': True, 'woth': True, 'roth': True, 'xoth': True, 'isuid': False, 'isgid': False}) => {"ansible_loop_var": "item", "changed": false, "cmd": "grep -c -E \"^iqn.1994-05.com.redhat:targetb$\" /sys/class/iscsi_session/*/targetname", "delta": "0:00:00.003386", "end": "2024-02-19 16:58:31.884001", "item": {"atime": 1708355480.908, "ctime": 1708355480.537, "dev": 64768, "gid": 0, "gr_name": "root", "inode": 1270206, "isblk": false, "ischr": false, "isdir": false, "isfifo": false, "isgid": false, "islnk": true, "isreg": false, "issock": false, "isuid": false, "mode": "0777", "mtime": 1708355480.537, "nlink": 1, "path": "/var/lib/iscsi/send_targets/192.168.42.191,3260/iqn.1994-05.com.redhat:targetb,192.168.42.191,3260,1,default", "pw_name": "root", "rgrp": true, "roth": true, "rusr": true, "size": 73, "uid": 0, "wgrp": true, "woth": true, "wusr": true, "xgrp": true, "xoth": true, "xusr": true}, "msg": "non-zero return code", "rc": 1, "start": "2024-02-19 16:58:31.880615", "stderr": "", "stderr_lines": [], "stdout": "0", "stdout_lines": ["0"]}

And we can confirm that the node1 is now logged into targetb (because of this role) and not to targeta (because we manually logged out from that):

[root@fastvm-rhel-8-4-150 ~]# cat /sys/class/iscsi_session/session2/targetname 
iqn.1994-05.com.redhat:targetb
[root@fastvm-rhel-8-4-150 ~]# cat /sys/class/iscsi_session/session2/state
LOGGED_IN

[OndrejHome]

Hi @jesusserranost, and thank you for very detailed description of the issue!

Looking into code the first obvious problem is that I can see is that next task depends on previous one failing to work in some situations - the check_for_existing_login.results[loop_index].rc != 0 is checking if the previous task finished with non-zero exit code. This obviously needs the previous task to continue when error occurs so the fix for this thing will be failed_when: false for that task. I have pushed this small change with commit c4a9d23. If you get into testing this let me know if that helped your use case.

However the above is "not a great solution(tm)" and I will be looking into this further when there is time. At present the role really just assumes that all sessions will be related to server towards which you are trying to discover and login. The multiple iscsi targets use case was not considered when this role was made in past and code reflects that - there is no checking that target name is for the target you are trying to get into, so if there is other target with same target name it will just assume that is okay - which is wrong. It will take me some time to figure out how to address this situation, but at least I have a good example use case from you on which I can test. Will keep you updated on progress once there is some.