Unable to apply Custom Rule against Yaml File

[nkreiger]

Describe the bug

Applying a custom rule against a YAML file, results in no matches, however, it should be matching correctly.

[
  {
    "name": "Network Connection: Serving API Routes",
    "description": "Service exposes RESTful API Routes",
    "id": "test_exposed_api_routes",
    "severity": "moderate",
    "applies_to": [
      "go"
    ],
    "tags": [
      "Test.OS.Network.Connection.Routes.API.Path"
    ],
    "patterns": [
      {
        "pattern": "(?<=(?:Post|Get|Put|Delete|Connect|Patch|Options|Head|Trace)\\(\")([^\"]*\\/[^\"]+)(?=\")",
        "type": "regex",
        "scopes": [
          "code"
        ],
        "confidence": "high"
      },
      {
        "pattern": "http.NewServeMux|fiber.New",
        "type": "regexword",
        "scopes": [
          "code"
        ],
        "confidence": "high"
      }
    ]
  },
  {
    "name": "Test",
    "description": "Test",
    "id": "test_internet_facing_public",
    "severity": "moderate",
    "applies_to": [
      "yaml"
    ],
    "tags": [
      "Test.OS.Network.DNS.InternetFacing.Public"
    ],
    "scopes": [
      "config",
      "code",
      "comments"
    ],
    "patterns": [
      {
        "pattern": "host",
        "type": "regexword",
        "confidence": "high"
      },
      {
        "pattern": "investments-unlimited.mortgages.io",
        "type": "regexword",
        "confidence": "high"
      },
      {
        "pattern": "investments-unlimited.mortgages.io",
        "type": "string",
        "confidence": "high"
      },
      {
        "pattern": "investments-unlimited.mortgages.io",
        "type": "regex",
        "confidence": "high"
      }
    ]
  }
]

appinspector analyze -s ./xd-trading-api -f json -r .test/custom_rules.json -o test.json

File in repository:

host: investments-unlimited.mortgages.io
bamboo: bamboo

Expected behavior

Rule is matched.

Screenshots
If applicable, add screenshots to help explain your problem.

demos/xd-trading-app » cat test.json | grep Internet -A 10
demos/xd-trading-app » cat test.json | grep Internet -A 10

Additional context

I was able to match the same value in a code file .go. But not with the yaml extension.

[nkreiger]

Seeming same issue with any non-code file (json, yaml, .config).

Copied the default rules in case my syntax was incorrect, no luck.

[nkreiger]

It looks like the tag needed to have Metadata in it for some reason. Not sure why that is.

Feel free to close if that's expected.

[gfs]

This is the currently the intended behavior, you need the -A/--allow-all-tags-in-build-files argument to the cli to match non 'metadata' tags in 'build' files like yml.

https://github.com/microsoft/ApplicationInspector/wiki/1.-CLI-Usage

For writing rules to match structured data files I can also recommend using the respective Path based queries for much higher fidelity than regex.

https://github.com/microsoft/ApplicationInspector/wiki/3.6-Structured-Data-Queries-(XPath,-JSONPath,-YamlPath)

Let me know if that helps. Would be interested in any other feedback as well on rule authoring.