[BUG] 504 errors and other bash issues

[gmenziesint]

Describe the bug
Runbook is failing after what appears to be two 504 errors, there was also another issue with another run where bash failed on a temp file with too many arguments. Wanted to report these in case they were fixable or there was a way to reduce the likelihood of them occurring (504 errors are probably just MS Graph acting up) but I'd like to improve the reliability of the runs where possible.

To Reproduce
Steps to reproduce the behavior:

Unable to reproduce reliably currently but has happened on and off previously in roughly one in every 7 runs.

Expected behavior
Runbook completes without error or recovers after multiple failures.

Screenshots
If applicable, add screenshots to help explain your problem.

Run type (please complete the following information):

• Mode: [e.g. 0 or 1] - 1
• Client [e.g. Pipeline, local machine] - Pipeline
• Version [e.g. 1.0.2] - 2.2.0

Additional context

First run fails on Backup creation

Traceback (most recent call last):
  File "/home/vsts/.local/bin/IntuneCD-startbackup", line 8, in <module>
    sys.exit(start())
  File "/home/vsts/.local/lib/python3.10/site-packages/IntuneCD/run_backup.py", line 313, in start
    run_backup(
  File "/home/vsts/.local/lib/python3.10/site-packages/IntuneCD/run_backup.py", line 260, in run_backup
    backup_intune(results, path, output, exclude, token, prefix, append_id, args)
  File "/home/vsts/.local/lib/python3.10/site-packages/IntuneCD/backup_intune.py", line 55, in backup_intune
    savebackup(path, output, exclude, token, append_id, args.audit, scope_tags)
  File "/home/vsts/.local/lib/python3.10/site-packages/IntuneCD/backup/Intune/backup_applications.py", line 58, in savebackup
    data = makeapirequest(ENDPOINT, token, q_param)
  File "/home/vsts/.local/lib/python3.10/site-packages/IntuneCD/intunecdlib/graph_request.py", line 89, in makeapirequest
    raise requests.exceptions.HTTPError(
requests.exceptions.HTTPError: Request failed with 504 - {"error":{"code":"UnknownError","message":"{\"Message\":\"{\\r\\n  \\\"_version\\\": 3,\\r\\n  \\\"Message\\\": \\\"An error has occurred - Operation ID (for customer support): 00000000-0000-0000-0000-000000000000 - Activity ID: 571adc8b-1c23-4b22-8602-87cde5f2a4e6 - Url: [https://fef.msua02.manage.microsoft.com/TrafficGateway/TrafficRoutingService/AppLifecycle/StatelessAppMetadataFEService/deviceAppManagement/mobileApps?api-version=5024-01-23\\\",\\r\\n](https://fef.msua02.manage.microsoft.com/TrafficGateway/TrafficRoutingService/AppLifecycle/StatelessAppMetadataFEService/deviceAppManagement/mobileApps?api-version=5024-01-23\\\%22,\\r\\n)  \\\"CustomApiErrorPhrase\\\": \\\"\\\",\\r\\n  \\\"RetryAfter\\\": null,\\r\\n  \\\"ErrorSourceService\\\": \\\"\\\",\\r\\n  \\\"HttpHeaders\\\": \\\"{}\\\"\\r\\n}\"}","innerError":{"date":"2024-03-13T00:06:57","request-id":"571adc8b-1c23-4b22-8602-87cde5f2a4e6","client-request-id":"571adc8b-1c23-4b22-8602-87cde5f2a4e6"}}}

##[error]Bash exited with code '1'.
##[error]Bash wrote one or more lines to the standard error stream.
##[error]Traceback (most recent call last):
  File "/home/vsts/.local/bin/IntuneCD-startbackup", line 8, in <module>
    sys.exit(start())
  File "/home/vsts/.local/lib/python3.10/site-packages/IntuneCD/run_backup.py", line 313, in start

##[error]    run_backup(
  File "/home/vsts/.local/lib/python3.10/site-packages/IntuneCD/run_backup.py", line 260, in run_backup
    backup_intune(results, path, output, exclude, token, prefix, append_id, args)
  File "/home/vsts/.local/lib/python3.10/site-packages/IntuneCD/backup_intune.py", line 55, in backup_intune
    savebackup(path, output, exclude, token, append_id, args.audit, scope_tags)
  File "/home/vsts/.local/lib/python3.10/site-packages/IntuneCD/backup/Intune/backup_applications.py", line 58, in savebackup
    data = makeapirequest(ENDPOINT, token, q_param)
  File "/home/vsts/.local/lib/python3.10/site-packages/IntuneCD/intunecdlib/graph_request.py", line 89, in makeapirequest
    raise requests.exceptions.HTTPError(
requests.exceptions.HTTPError: Request failed with 504 - {"error":{"code":"UnknownError","message":"{\"Message\":\"{\\r\\n  \\\"_version\\\": 3,\\r\\n  \\\"Message\\\": \\\"An error has occurred - Operation ID (for customer support): 00000000-0000-0000-0000-000000000000 - Activity ID: 571adc8b-1c23-4b22-8602-87cde5f2a4e6 - Url: [https://fef.msua02.manage.microsoft.com/TrafficGateway/TrafficRoutingService/AppLifecycle/StatelessAppMetadataFEService/deviceAppManagement/mobileApps?api-version=5024-01-23\\\",\\r\\n](https://fef.msua02.manage.microsoft.com/TrafficGateway/TrafficRoutingService/AppLifecycle/StatelessAppMetadataFEService/deviceAppManagement/mobileApps?api-version=5024-01-23\\\%22,\\r\\n)  \\\"CustomApiErrorPhrase\\\": \\\"\\\",\\r\\n  \\\"RetryAfter\\\": null,\\r\\n  \\\"ErrorSourceService\\\": \\\"\\\",\\r\\n  \\\"HttpHeaders\\\": \\\"{}\\\"\\r\\n}\"}","innerError":{"date":"2024-03-13T00:06:57","request-id":"571adc8b-1c23-4b22-8602-87cde5f2a4e6","client-request-id":"571adc8b-1c23-4b22-8602-87cde5f2a4e6"}}}

Second run fails on Markdown documentation creation

Starting: Generate markdown document
==============================================================================
Task         : Bash
Description  : Run a Bash script on macOS, Linux, or Windows
Version      : 3.236.1
Author       : Microsoft Corporation
Help         : https://docs.microsoft.com/azure/devops/pipelines/tasks/utility/bash
==============================================================================
Generating script.
========================== Starting Command Output ===========================
/usr/bin/bash /home/vsts/work/_temp/153698cd-ffd4-499d-946a-deafbfd109cc.sh
no configuration backup change detected in the last commit, documentation will not be created
/home/vsts/work/_temp/153698cd-ffd4-499d-946a-deafbfd109cc.sh: line 1: [: too many arguments

##[error]Bash wrote one or more lines to the standard error stream.
##[error]/home/vsts/work/_temp/153698cd-ffd4-499d-946a-deafbfd109cc.sh: line 1: [: too many arguments

Finishing: Generate markdown document

Below is our YML file for the runbook

# Starter pipeline
# Start with a minimal pipeline that you can customize to build and deploy your code.
# Add steps that build, run tests, deploy, and more:
# https://aka.ms/yaml

trigger:
  none
schedules:
  - cron: '0 */6 * * *'
    displayName: "Every 6 hours"
    branches:
      include:
      - main
    always: true

pool:
  vmImage: ubuntu-latest

jobs:
  - job: backup_document
    displayName: Backup Intune configuration
    pool:
      vmImage: ubuntu-latest
    continueOnError: false
    steps:
    - checkout: self
      persistCredentials: true
    
#Remove existing prod-backup directory
    - task: Bash@3
      displayName: Remove existing prod-backup directory
      inputs:
        targetType: 'inline'
        script: |
          rm -rfv "$(Build.SourcesDirectory)/prod-backup"
        workingDirectory: '$(Build.SourcesDirectory)'
        failOnStderr: false
        
# Install IntuneCD
# https://github.com/almenscorner/IntuneCD
    - task: Bash@3
      displayName: Install IntuneCD Package V2.2.0
      inputs:
        targetType: 'inline'
        script: |
          pip3 install IntuneCD==2.2.0
        workingDirectory: '$(Build.SourcesDirectory)'
        failOnStderr: true

# Backup the latest configuration, using the current directory
    - task: Bash@3
      displayName: IntuneCD backup Start
      inputs:
        targetType: 'inline'
        script: |
          mkdir -p "$(Build.SourcesDirectory)/prod-backup"
          IntuneCD-startbackup \
          --mode=1 \
          --path "$(Build.SourcesDirectory)/prod-backup" \
          --exclude ConditionalAccess \
          --ignore-omasettings \
          --append-id \
          --autopilot=True \
          --audit
        workingDirectory: '$(Build.SourcesDirectory)'
        failOnStderr: true
      env:
        TENANT_NAME: $(TENANT_NAME)
        CLIENT_ID: $(CLIENT_ID)
        CLIENT_SECRET: $(CLIENT_SECRET)
        
# Set git settings
    - task: Bash@3
      displayName: Configure Git Global Settings
      inputs:
        targetType: 'inline'
        script: |
          git config user.name $(USER_NAME)
          git config user.email $(USER_EMAIL)
        workingDirectory: '$(Build.SourcesDirectory)'
        failOnStderr: true
# Commit changes and push to Azure DevOps repo
    - task: Bash@3
      displayName: Commit changes to Azure DevOps repo
      name: commitAndsetVariable
      inputs:
        targetType: 'inline'
        script: |
          DATEF=`date +%Y.%m.%d`
          git add --all
          # modified files in folder prod-backup
          var=$(git diff --name-only --staged -- prod-backup)
          echo "##vso[task.setVariable variable=CHANGE_DETECTED;isOutput=true;]$var"
          git commit -m "Intune config backup $DATEF"
          git push origin HEAD:main
        workingDirectory: '$(Build.SourcesDirectory)'
        failOnStderr: false
        
# Create markdown documentation
    - task: Bash@3
      displayName: Generate markdown document
      inputs:
        targetType: 'inline'
        script: |
          if [ ! -z "$(commitAndsetVariable.CHANGE_DETECTED)" ]
          then
            INTRO="Intune backup and documentation generated at $(Build.Repository.Uri) <img align=\"right\" width=\"96\" height=\"96\" src=\"./logo.png\">"
            IntuneCD-startdocumentation \
                --path="$(Build.SourcesDirectory)/prod-backup" \
                --outpath="$(Build.SourcesDirectory)/prod-as-built.md" \
                --tenantname=$TENANT_NAME \
                --intro="$INTRO" \
                #--split=Y
          else
            echo "no configuration backup change detected in the last commit, documentation will not be created"
          fi
        workingDirectory: '$(Build.SourcesDirectory)'
        failOnStderr: true
      env:
        TENANT_NAME: $(TENANT_NAME)

# Commit changes and push to repo
    - task: Bash@3
      displayName: Commit changes
      inputs:
        targetType: 'inline'
        script: |
          DATEF=`date +%Y.%m.%d`
          git add --all
          git commit -m "Intune config as-built $DATEF"
          git push origin HEAD:main
        workingDirectory: '$(Build.SourcesDirectory)'
        failOnStderr: false

  - job: tag
    displayName: Tag repo
    dependsOn: backup_document
    condition: and(succeeded(), ne(dependencies.backup_document.outputs['commitAndsetVariable.CHANGE_DETECTED'], ''))
    pool:
      vmImage: ubuntu-latest
    continueOnError: false
    steps:
    - checkout: self
      persistCredentials: true

# Set git global settings
    - task: Bash@3
      displayName: Configure Git
      inputs:
        targetType: 'inline'
        script: |
          git config user.name $(USER_NAME)
          git config user.email $(USER_EMAIL)
        workingDirectory: '$(Build.SourcesDirectory)'
        failOnStderr: true

    - task: Bash@3
      displayName: Pull origin
      inputs:
        targetType: 'inline'
        script: |
          git pull origin main
        workingDirectory: '$(Build.SourcesDirectory)'
        failOnStderr: false

    - task: PowerShell@2
      displayName: Git tag
      inputs:
        targetType: 'inline'
        script: |
          # change in configuration backup folder detected, create TAG
          $DATEF= Get-Date -format "yyyy-MM-dd_THH-mm"
          git tag -a "v$DATEF" -m "Microsoft Intune configuration snapshot $DATEF"
          git push origin "v$DATEF" *> $null # even status information goes to stderr :(
        failOnStderr: true
        pwsh: false
        workingDirectory: '$(Build.SourcesDirectory)'

  - job: publish
    displayName: Publish as-built artifacts
    dependsOn: tag
    condition: and(succeeded(), ne(dependencies.backup_document.outputs['commitAndsetVariable.CHANGE_DETECTED'], ''))
    pool:
      vmImage: ubuntu-latest
    continueOnError: false
    steps:
    - checkout: self
      persistCredentials: true

[almenscorner]

Currently if a 504 is hit it will retry 3 times, question is how many retires we want to add here before we should just give up and try again later.

For the config step, I would guess that something is not setting the commitAndsetVariable correctly so the if statement fails