bug: Array properties aren't handled correctly when Filter Policy Scope is set to MessageBody for SNS subscription

[ABAG603]

Is there an existing issue for this?

• I have searched the existing issues

Current Behavior

Messages are not not being routed to the queues based on the Filter Policy (see steps to reproduce for more details)

Expected Behavior

Messages are being correctly routed. This is the behaviour when tested against "real" SQS/SNS.

How are you starting LocalStack?

With the localstack script

Steps To Reproduce

Setup

• SNS topic: topic1
• SQS queues: queue1, queue2

Both queues are being subscribed to the topic1 with Filter Policy Scope set to MessageBody

Filter policy for queue1 topic subscription:

{
    "headers": {
      "route-to": ["queue1"]  
    }
}

Filter policy for queue2 topic subscription:

{
    "headers": {
      "route-to": ["queue2"]  
    }
}

Scenario 1
Send message below to the topic1

{
    "headers": {
      "route-to": ["queue1"]  
    }
}

Expected: message to appear in the queue1 (this is exactly what is happening in AWS SNS\SQS)
Actual: no messages in the queue1

Scenario 2
Send message below to the topic1

{
    "headers": {
      "route-to": ["queue2"]  
    }
}

Expected: message to appear in the queue2 (this is exactly what is happening in AWS SNS\SQS)
Actual: no messages in the queue2

Scenario 3
Send message below to the topic1

{
    "headers": {
      "route-to": ["queue1", "queue2"]  
    }
}

Expected: message to appear in both queue1 and queue2 (this is exactly what is happening in AWS SNS\SQS)
Actual: message is neither in queue1 nor queue2

Environment

- OS: Mac OS Sonoma 14.4.1
- LocalStack: 3.2.0

Anything else?

No response

[localstack-bot]

Welcome to LocalStack! Thanks for reporting your first issue and our team will be working towards fixing the issue for you or reach out for more background information. We recommend joining our Slack Community for real-time help and drop a message to LocalStack Pro Support if you are a Pro user! If you are willing to contribute towards fixing this issue, please have a look at our contributing guidelines and our contributing guide.

[ABAG603]

Hi @bentsku . It looks like this fix introduced a regression. Now filter policy works only when header is an array. So,

this works - message will be delivered to queue1

{
    "headers": {
      "route-to": ["queue1"]  
    }
}

this doesn't

{
    "headers": {
      "route-to": "queue1"
    }
}

I'm testing with LocalStack 3.4.0. I've also tested in AWS and it works in both cases above.

[bentsku]

Interesting, in the AWS documentation, I don't think I can find any example of a policy setting its value in something else than an array. The example you shared above is the filter policy, did I understood that right?

edit: looking at your message, it's not very clear if you're talking about the sent message or the filter policy. Could you explain a bit more?

If what you're specifying is the message, then this should work. We have specific test cases:
This policy:

{
  "f2": ["v5"]
}

Yield the following results:

• does not match this

{
  "f1": "v1",
  "f2": "v4"
}

• matches this (this looks like your use case above)

{
  "f1": "v2",
  "f2": "v5"
}

• matches this (this looks like your use case above)

{
  "f1": "v3",
  "f2": "v5"
}

[ABAG603]

@bentsku sorry, now I realize that it was not clear.

So, filter policy is:

{
    "headers": {
      "route-to": ["queue1"]  
    }
}

If we send this message it won't be delivered to queue1 with LocalStack 3.4.0

{
    "messageProp1": "value1",
    "messageProp2": "value2",
    "headers": {
      "route-to": "queue1"
    }
}

this message will be delivered:

{
    "messageProp1": "value1",
    "messageProp2": "value2",
    "headers": {
      "route-to": ["queue1"]
    }
}

When tested against AWS both messages are being delivered.

[bentsku]

@ABAG603 thanks, I see. Maybe the issue is with the fact that the value is nested. I'm currently working on a PR (#10691) which will add missing features to SNS Filter Policies, and I'll add a specific test for your use case and see if it fixes it. Thanks for the report, I'll reopen!

[bentsku]

Hi @ABAG603, I've just gave it a try with the example you've given and I'm currently unable to reproduce, it works both on AWS and LocalStack.

This what I'm using, this is a test based on this issue scenario:

filter_policy_1 = {"headers": {"route-to": ["queue1"]}}
sns_create_sqs_subscription_with_filter_policy(
    topic_arn=topic_arn,
    queue_url=queue_url_1,
    filter_scope="MessageBody",
    filter_policy=filter_policy_1,
)

filter_policy_2 = {"headers": {"route-to": ["queue2"]}}
sns_create_sqs_subscription_with_filter_policy(
    topic_arn=topic_arn,
    queue_url=queue_url_2,
    filter_scope="MessageBody",
    filter_policy=filter_policy_2,
)

# publish messages that satisfies the filter policy, assert that messages are received
messages = [
    {"headers": {"route-to": ["queue3"]}},
    {"headers": {"route-to": ["queue1"]}},
    {"headers": {"route-to": "queue1"}},
    {"headers": {"route-to": ["queue2"]}},
    {"headers": {"route-to": "queue2"}},
    {"headers": {"route-to": ["queue1", "queue2"]}},
]
for message in messages:
    aws_client.sns.publish(
        TopicArn=topic_arn,
        Message=json.dumps(message),
    )

For Queue 1, I got the following message bodies:

{
  "headers": {
    "route-to": "queue1"
  }
}

{
  "headers": {
    "route-to": [
      "queue1",
      "queue2"
    ]
  }
}

{
  "headers": {
    "route-to": [
      "queue1"
    ]
  }
}

And for Queue2:

{
  "headers": {
    "route-to": "queue2"
  }
}

{
  "headers": {
    "route-to": [
      "queue1",
      "queue2"
    ]
  }
}

{
  "headers": {
    "route-to": [
      "queue2"
    ]
  }
}

Could you please share a sample which is currently failing against LocalStack? Maybe we do not have the same setup and something is different, but with what is currently provided, I'm not able to reproduce.

[ABAG603]

Hi @bentsku . It took quite some time but it looks like I've found it
Please try with this message - it won't be delivered (at least this is what I see in my setup)

{
   "arrayProp": [
     "arrayValue"
   ],
   "headers": {
     "route-to": "queue1"
   }
}

[bentsku]

Hello @ABAG603 and thanks again for the report! #10691 will finally supported the use-case reported here. Once merged, it will close this issue and should be available in the next hour or so after this issue being closed.