Message tags might not always be reflected in the History tab

[roisec]

Is your feature request related to a problem? Please describe.

Hi
I encourage some use cases that http request contains json requests, and there is no tag on the relevant message.
I think there are missing scenarios that I send content-type: application/json and there is no JSON tag on the message

Describe the solution you'd like

Cover all the http requests with json

Describe alternatives you've considered

Filter out by myself .

Screenshots

No response

Additional context

No response

Would you like to help fix this issue?

• Yes

[thc202]

The tagging mechanism works on requests and responses but the default tag is only for responses (as the name indicates response_json).

[roisec]

@thc202
I saw in history some HTTP requests and responses without the JSON tag, but they contained the header content-type: application/JSON.
Do you know why?

[kingthorin]

Tagging is based on passive scan which uses a queue, so they just may not have been processed/tagged yet.

Also the current pattern seems to be case sensitive.

[kingthorin]

Correction the patterns are all handled case insensitively in the end.

[kingthorin]

Should the tag also cover other JSON'ish types?

zaproxy/docker/scripts/scripts/httpsender/Alert_on_Unexpected_Content_Types.js

Lines 12 to 25 in 02b5c4f

	"application/hal+json",
	"application/health+json",
	"application/json",
	"application/octet-stream",
	"application/problem+json",
	"application/problem+xml",
	"application/soap+xml",
	"application/vnd.api+json",
	"application/xml",
	"application/x-ndjson",
	"application/x-yaml",
	"application/yaml",
	"text/x-json",
	"text/json",

[kingthorin]

@psiinon @thc202 @ricekot ☝ thoughts?

[kingthorin]

This was discussed via IRC. The decision was to not modify the existing pattern at this point. Others can be added, initially disabled.

Without further evidence of something actually misbehaving this issue will be closed.

[roisec]

Hi
I have several examples of HTTP responses with content-type: application/json, and there is no JSON tag on the request in ZAP.

[kingthorin]

That's not evidence. Unless you can provide the details to back it up there's nothing we can do for you.

[roisec]

In the following screenshots you are able to see some requests tagged by JSON and original without.
The first request is not tagged by JSON.
When I send the same request again manually for this use case it works fine.

Maybe you should support: application/json; charset=utf-8
or adding with a wildcard? like "applicaiton/json*"

[kingthorin]

As discussed above the existing pattern won't be updated in case anyone is depending on it in the current form. However there is already a PR in flight adding a json_extended pattern which would cover more circumstances.

[roisec]

@kingthorin thanks for your response,
Do you mean that there is no current solution for this bug?
Because this form does not cover tag JSON requests properly. (I can't depend on this)

[kingthorin]

The solution would be to update the regex in your options.
In future versions of ZAP a new regex will be available to everyone (who chooses to enable it), but it won't replace the existing one.

[roisec]

Can I edit the regex currently?if yes,where?
Can I add custom tag and add custom regex for this?

[kingthorin]

Yes: https://www.zaproxy.org/docs/desktop/ui/dialogs/options/pscan/
Yes.

[roisec]

Ok, I checked this and configured my regex, and still, there is no automatic assignment of the relevant tag.
It's works only if I manually send the same request but not detected automatically by proxy requests.
@kingthorin wdyt?

[kingthorin]

It's not retro active, it'll only tag new traffic (proxy, spider, manual), and only if your regex actually matches.
You can test with regexplanet (select Java).

[roisec]

I created the tag regex and then generated the traffic .
if you want I able to share with you an screen recording

[kingthorin]

🤷‍♂️ There's a ton of variables here. Your regex could be wrong/poor, you might not be waiting for the passive scan queue, you might have "only in scope" on, etc. To name a few. Unless you can provide the tag/regex and a public example of it not working then we can't recreate/troubleshoot.

[roisec]

The regex work perfect ,when I do the same request manually through zap the tag works properly
Tomorrow I will send a screen recording that proves it

[kingthorin]

Make sure it includes the rule config and passive scan settings. Plus the message details of what is tagged and what's not.

[thc202]

#8399 (comment)

The existing pattern is not a/the problem anyway, it already handles the case application/json; charset=utf-8.

[roisec]

Screen.Recording.2024-04-11.at.10.19.06.mov

Let's see the video,
My tag and JSON tag are recognized only in manual, but through the proxy it's not detected automatically.
This issue occurs in many sites

[thc202]

That's a problem with the UI not the passive scanner/regex.

[kingthorin]

Things you can try:

• On one/some of the entry(ies) after the passive scan queue has emptied (the Eyeball counter in the lower right), right click on an un-tagged entry and select "Manage History Tags" (do you see the tag there even if it wasn't shown in the table?).
• Run this standalone community-script after the passive scan queue is empty.

[thc202]

There's no need to try, it's a confirmed bug in the UI (or more accurately on how the HistoryReferences are managed).

[roisec]

@kingthorin I did 2 things you said, and still this is not tagged with JSON, also in "Manage History Tags".
Anyway, when the filter you implement in the PR through the API ZAP (#8422), is should be working properly?

[kingthorin]

I haven't talked to @thc202 about the issue but he seems to have confirmed a UI issue, so no there will still be a piece outstanding it seems.