Issue on running ansible.windows.win_updates module

[HaomouJi]

SUMMARY

Received error message while using ansible.windows.win_updates. Ansible 1.13.0.

ISSUE TYPE

• Bug Report

COMPONENT NAME

ANSIBLE VERSION

 Ansible 2.15.0

COLLECTION VERSION

Ansible.windows 1.14.0

CONFIGURATION

OS / ENVIRONMENT

windows 2012-2019

STEPS TO REPRODUCE

- name: WIN PATCHING - PRE-INSTALL - Search for and Download Updates
      ansible.windows.win_updates:
        category_names: '*'
        state: downloaded
      register: winpatch_download_status

EXPECTED RESULTS

list available KBs

ACTUAL RESULTS

TASK [win_patching : WIN PATCHING - PRE-INSTALL - Search for and Download Updates] ***
task path: /runner/requirements_roles/win_patching/tasks/main.yml:20
The full traceback is:
Exception calling "NativeCreateProcess" with "9" argument(s): "CreateProcessW() failed (Access is denied, Win32ErrorCode 5 - 0x00000005)"
At line:388 char:9
+         $pi = [Ansible.Windows.Process.ProcessUtil]::NativeCreateProc ...
+         ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (:) [], MethodInvocationException
    + FullyQualifiedErrorId : Win32Exception

ScriptStackTrace:
at Invoke-AsBatchLogon, <No file>: line 388
at <ScriptBlock>, <No file>: line 1534
fatal: [dwa620802135802.btfin-dev.com]: FAILED! => {
    "changed": false,
    "failed_update_count": 0,
    "filte…

[jborean93]

Are you able to run - win_whoami and share the output? I'm curious whether your connection user holds certain privileges. What connection plugin are you using to communicate with the Windows server?

[HaomouJi]

Hi Jordan, thanks for the reply. Pretty sure the cred I put in has local administrator access on remote server. Same cred is working with an ealier version of win_updates module. I will try to revert my fix in the envionrment and add this win_whoami for another test.

[jborean93]

I am more curious whether the account has the SeBatchLogonRight right. This controls whether the newly spawned process is running as the same user or as SYSTEM. I'm unsure why else you would get the access denied error so trying to look at the rights the connection user has might show some more info.

[imedaouidene]

@HaomouJi have you find a solution/workaround for this issue you are having ?

[JamesHaomou-Ji-Westpac]

Hi guys, this is Haomou Ji. Sorry for being late, just did the test from working envrionment.
@jborean93 , here is the out come of win_whoami
"rights": [ "SeInteractiveLogonRight", "SeNetworkLogonRight", "SeServiceLogonRight", "SeBatchLogonRight", "SeRemoteInteractiveLogonRight" ], "privileges": { "SeChangeNotifyPrivilege": "enabled-by-default", "SeAssignPrimaryTokenPrivilege": "enabled-by-default", "SeRemoteShutdownPrivilege": "enabled-by-default", "SeBackupPrivilege": "enabled-by-default", "SeLoadDriverPrivilege": "enabled-by-default", "SeSystemProfilePrivilege": "enabled-by-default", "SeShutdownPrivilege": "enabled-by-default", "SeCreatePagefilePrivilege": "enabled-by-default", "SeSystemEnvironmentPrivilege": "enabled-by-default", "SeSystemtimePrivilege": "enabled-by-default", "SeSecurityPrivilege": "enabled-by-default", "SeDelegateSessionUserImpersonatePrivilege": "enabled-by-default", "SeIncreaseWorkingSetPrivilege": "enabled-by-default", "SeTakeOwnershipPrivilege": "enabled-by-default", "SeIncreaseQuotaPrivilege": "enabled-by-default", "SeCreateGlobalPrivilege": "enabled-by-default", "SeCreateSymbolicLinkPrivilege": "enabled-by-default", "SeRestorePrivilege": "enabled-by-default", "SeUndockPrivilege": "enabled-by-default", "SeImpersonatePrivilege": "enabled-by-default", "SeProfileSingleProcessPrivilege": "enabled-by-default", "SeDebugPrivilege": "enabled-by-default", "SeIncreaseBasePriorityPrivilege": "enabled-by-default", "SeManageVolumePrivilege": "enabled-by-default", "SeTimeZonePrivilege": "enabled-by-default" },

[JamesHaomou-Ji-Westpac]

@imedaouidene Yes, I got a workaround. I have copied old version of both win_updates.ps1 and win_updates.py files back to the controller host via a task:
- name: WIN PATCHING - PRE-INSTALL - Copy Win_Updates 1.13.0 Module delegate_to: localhost run_once: true ansible.builtin.copy: mode: '755' src: "files/{{ item }}" dest: "/usr/share/ansible/collections/ansible_collections/ansible/windows/plugins/modules/{{ item }}" loop: - "win_updates.ps1" - "win_updates.py"

[HypaActive]

Thanks a lot for your workaround @JamesHaomou-Ji-Westpac, it was really helpful. win_updates stopped working for me too after upgrading the collection. Weirdly it is still working for a few servers (especially those that are freshly deployed from template), I can't find the difference between them though.
Btw, I had to replace win_updates.py under .../plugins/actions/ too to get it working again, coming from latest version of the collection.

[jborean93]

We believe we have narrowed down what part of the code is causing the access is denied error but we haven't figured out why. As I cannot replicate this issue it's very hard to figure out a fix for it. Essentially part of the new win_updates process is to spawn a new subprocess that does the update work. It is started with the CREATE_NEW_CONSOLE flag to ensure it isn't tied to the Ansible process which exits shortly after. For whatever reason the presence of the CREATE_NEW_CONSOLE flag is causing this error.

It would be great if whoever is affected by this could copy the script at https://gist.github.com/jborean93/94b3eea93dbca15e60c51247c3a7b399 to one of the affected servers and then run the following from another Windows host.

Invoke-Command -ComputerName target-host -Command { & "C:\path\where\process_new_console_test.ps1" } 

winrs -r:http://target-host:5985/wsman powershell.exe -File "C:\path\where\process_new_console_test.ps1"

The Invoke-Command example replicates how things run with the psrp connection plugin while the winrs command replicates the winrm connection plugin. It would be good to know if none, one, or both fail in an environment that is affected.

[HypaActive]

Here the output from your tests:

Invoke-Command -ComputerName test-computer -Command { & "C:\process_new_console_test.ps1" }
Done, exited with rc 1
OperationStopped: Process has exited, so the requested information is not available.

and

winrs -r:http://test-computer:5985/wsman powershell.exe -File "C:\process_new_console_test.ps1"
Exception calling "NativeCreateProcess" with "9" argument(s):
"CreateProcessW() failed (Access is denied, Win32ErrorCode 5 - 0x00000005)"
At C:\process_new_console_test.ps1:644 char:5
+     $procInfo = [Ansible.Windows.Process.ProcessUtil]::NativeCreateProcess(
+     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (:) [], ParentContainsErrorRecordE
   xception
    + FullyQualifiedErrorId : Win32Exception

[jborean93]

Thanks that at least confirms it's a problem with the way that the winrm connection plugin works. You could try the win_updates with the psrp connection plugin. This plugin also uses WinRM as the transport but uses a different shell mechanism on top. It's akin to PowerShell's Invoke-Command vs the winrs command from my example.

Now as to why this is happening I'm honestly not sure. It's not something I can replicate or even explain why it might be happening. Knowing any differences between hosts that work might help narrow it down, like:

• Do some have different AVs installed
• Have you installed Windows Terminal or played with conhost.exe in any way
• Any major env var differences with winrs -r:http://target-host:5985/wsman powershell.exe 'gci env: | Sort-object Name'
• Could the user potentially be using a roaming profile in cases or not
• Anything else you might think is relevant

You could try and use procmon and filter by the winrs process id for testing and see if there's anything that might jump out at you.

[calciosoccer1]

I'm not sure if my posting will help at all or confuse things further. Last month I successfully ran win_updates on my servers. Since then I have updated ansible and as a result the win_updates module was updated as well. I am now unable to run win_updates on some servers but on others it still completes successfully. I uninstalled our Anti-virus solution from a server that was failing and now win_updates completed successfully. I asked our security team to verify why the anti-virus solution is blocking the execution of the module. They responded that the issue is most likely not the anti-virus considering nothing changed in the last couple of months on their end and I was provided the following information.

"I’m seeing some differing behavior. When you ran it this morning, I see you connect and create a scheduled task called ‘ansible-ansible.windows.win_updates’. This scheduled task opens cmd.exe, with no parameters. Then you run the task, which opens a command prompt, and then delete the task. All within a few milliseconds of each other. Basically all you’re doing is opening a command prompt. This aligns with the log you posted below in that the task is so short lived, you can’t even get the name of the process running.

I looked further back to when the updates ran on Sept 27th. That time, I see you connect, but instead of creating scheduled tasks, you run an encoded powershell command. I didn’t take the time to decode it, but it’s enough to see that on the 27th, it ran via powershell commands, today, it’s running an empty scheduled task. So, why the difference in behavior? On Sept 27th there’s no scheduled tasks being created, only powershell scripts, but today, it’s creating empty tasks. So, I’m not sure why the change, or what prompted that, but I’d start looking there."

Hope the snippet above adds more clarity on what is occurring in our environment.

[jborean93]

@calciosoccer1 thanks for sharing. Essentially the change in behaviour is down to the complexity of this module. What happens is that the Windows Update code needs to run in a context outside of the WinRM network logon as the API doesn't allow this. In the past we used to run the powershell code directly as a scheduled task using -EncodedCommand or a -File (depending on your version). This had a bunch of problems around getting the output back reliably or debugging failures in the bootstrapping process that was very hard to diagnose. To counter this we now spawn just a cmd.exe process that does nothing and use that ne process as the parent of our own powershell process we control. This has a few advantages as now we have a far simpler and secure way to send data to the new powershell process (through stdin) and can easily capture any unreported failures (through stdout/stderr) which we could not do before with having the scheduled task spawn our process. Unfortunately this brings us to the problem in this report where for some reason this is failing and it's not something I can replicate myself.

To try and provide an alternative method we now just spawn a detached process if you are running win_updates without become removing the scheduled task method altogether but it still uses the parent process spawning step to get working. From what I can gather the problem exists when trying to start with the flag CREATE_NEW_CONSOLE but I've been unable to find out why it would be failing and even have been unable to replicate the issue on my test hosts so I'm very limited with what I can try out. From what I've seen using the psrp connection plugin vs the winrm one can help some people but I've yet to get a confirmation that is actually the case.

[HypaActive]

I can confirm, it works as expected when I use psrp instead of winrm.
Does anyone know if I can use psrp in general instead of winrm?

[jborean93]

Yes it should be mostly interchangeable, in fact you would probably notice some slight speed improvements, especially with loops. I can’t say it is 100% the same but they both use WinRM as the underlying transport.

[calciosoccer1]

Thank you so much for the quick reply. I managed to switch to psrp and it does work flawlessly. Once again. Thank you so much.

[prof79]

I'm using open source AWX Tower via awx-operator/Kubernetes and stumbled upon this first suspecting an awx-ee issue (which is indirectly true, since from 22.2.0 on they're using collections 1.14.0+.

I do, however, see some integrative issues still there. Trying to switch my project's inventory transport from winrm to psrp I found these:

Internal domain servers (Kerberos/ansible_psrp_auth=kerberos):

An exception occurred during task execution. To see the full traceback, use -vvv. The error was: ImportError: GSSAPIProxy requires the Python gssapi library: No module named 'gssapi' fatal: [server.domain.tld]: FAILED! => {"changed": false, "failed_update_count": 0, "filtered_updates": {}, "found_update_count": 0, "installed_update_count": 0, "msg": "GSSAPIProxy requires the Python gssapi library: No module named 'gssapi'", "updates": {}}

-> probably just missing gssapi library in awx-ee container

DMZ/externally hosted servers (NTLM/ansible_psrp_auth=ntlm): AWX/Ansible automatically tries to "become" which seems to automatically switch to winrm as the error resembles the original one (even though ansible_connection=psrp is set):

An exception occurred during task execution. To see the full traceback, use -vvv. The error was: ansible_collections.ansible.windows.plugins.action.win_updates._ReturnResultException: Failed to start new win_updates task with Ansible Become: Ausnahme beim Aufrufen von "NativeCreateProcess" mit 9 Argument(en): "CreateProcessW() failed (Zugriff verweigert, Win32ErrorCode 5 - 0x00000005)" fatal: [server.domain.tld]: FAILED! => {"changed": false, "failed_update_count": 0, "filtered_updates": {}, "found_update_count": 0, "installed_update_count": 0, "msg": "Failed to start new win_updates task with Ansible Become: Ausnahme beim Aufrufen von \"NativeCreateProcess\" mit 9 Argument(en): \"CreateProcessW() failed (Zugriff verweigert, Win32ErrorCode 5 - 0x00000005)\"", "updates": {}}