Timeout script remains in upcoming activities without displaying and BLOCKS other scripts to be executed

[primo-vincent]

Fleet version: script sent in 4.49, but instance updated in 4.49.3 since

Web browser and operating system: Windows 11 Professionnel 23H2 10.0.22631.3593

---

💥  Actual behavior

A script that times out remains blocked and continues to be counted in "upcoming activities" but does not appear in the display. Others script in the queue are no more executed.

🧑‍💻  Steps to reproduce

1. prerequisites:
   • 1password should not be install on the device
   • run script to install chocolatey :

Set-ExecutionPolicy Bypass -Scope Process -Force; [System.Net.ServicePointManager]::SecurityProtocol = [System.Net.ServicePointManager]::SecurityProtocol -bor 3072; iex ((New-Object System.Net.WebClient).DownloadString('https://community.chocolatey.org/install.ps1'))

2. via the API /api/v1/fleet/scripts/run/sync run the following script :

C:\ProgramData\chocolatey\bin\choco.exe install --yes --no-progress --no-color 1password

3. the api call should timeout, and no script in the queue ran after that

🕯️ More info

• it also appears with the /api/v1/fleet/scripts/run endpoint
• activities given by the api also contains 4 items but a count of 5

{
    "meta": {
        "has_next_results": false,
        "has_previous_results": false
    },
    "activities": [
        {
            "created_at": "2024-05-15T10:12:53Z",
            "uuid": "c1095aa4-7558-4417-a955-d951756ac4f2",
            "actor_full_name": "xxx",
            "actor_id": 2,
            "actor_gravatar": "",
            "actor_email": "xxx",
            "type": "ran_script",
            "details": {...}
        },
        {
            "created_at": "2024-05-15T10:13:11Z",
            "uuid": "ce00d244-23a9-4946-9f67-c045b37defd4",
            "actor_full_name": "xxx",
            "actor_id": 2,
            "actor_gravatar": "",
            "actor_email": "xxx",
            "type": "ran_script",
            "details": {...}
        },
        {
            "created_at": "2024-05-15T10:13:15Z",
            "uuid": "a460f397-a71b-4453-b8ef-fb032ae9f050",
            "actor_full_name": "xxx",
            "actor_id": 2,
            "actor_gravatar": "",
            "actor_email": "xxx",
            "type": "ran_script",
            "details": {...}
        },
        {
            "created_at": "2024-05-15T10:13:19Z",
            "uuid": "cbb86d45-9b6f-4dc1-a574-77a7d1d823cb",
            "actor_full_name": "xxx",
            "actor_id": 2,
            "actor_gravatar": "",
            "actor_email": "xxx",
            "type": "ran_script",
            "details": {...}
        }
    ],
    "count": 5
}

• the script result of the timed out script is

{
    "script_contents": "choco install --yes --no-color --no-progress 1password\n",
    "script_id": null,
    "exit_code": null,
    "output": "",
    "message": "Fleet didn’t hear back from the host in under 5 minutes (timeout for live scripts). Fleet doesn’t know if the script ran because it didn’t receive the result. Please try again.",
    "hostname": "PC",
    "host_timeout": true,
    "host_id": 108,
    "execution_id": "68d71e10-53f0-4250-ab85-e108fcabbb59",
    "runtime": 0
}

• In the database, the script result has sync_request=1
  
• On device reboot, it executed 1 script of the queue but not the others, the ghost timeout script still remains

[zayhanlon]

@georgekarrv Do we think this will be likely to make it into 4.51.0 release? Requesting a status update on behalf of the customer as their urgency is high