Add support for MediaPackage metrics

[theunissenne]

As the metrics are currently very sparse, we can't use a resource or dimension filter yet.

[cristiangreco]

As the metrics are currently very sparse,

Hi @theunissenne thanks for this contribution. What do you mean with the above?

[theunissenne]

As the metrics are currently very sparse,

Hi @theunissenne thanks for this contribution. What do you mean with the above?

Well I checked if I could filter on dimension etc, but currently MediaPackage only has a few metricsEgressRequestCount etc.
The Dimensions are things like Channel, StatusRangeCode, and I did not know if I had to create a dimensionfilter for them?

[theunissenne]

{ "Namespace": "AWS/MediaPackage", "MetricName": "EgressRequestCount", "Dimensions": [ { "Name": "PackagingConfiguration", "Value": "dash-cenc-multikey-prod" }, { "Name": "StatusCodeRange", "Value": "5xx" } ] }, { "Namespace": "AWS/MediaPackage", "MetricName": "EgressBytes", "Dimensions": [ { "Name": "PackagingConfiguration", "Value": "dash-cenc-singlekey-prod" } ] }, { "Namespace": "AWS/MediaPackage", "MetricName": "EgressBytes", "Dimensions": [] }, { "Namespace": "AWS/MediaPackage", "MetricName": "EgressResponseTime", "Dimensions": [ { "Name": "PackagingConfiguration", "Value": "dash-cenc-multikey-prod" } ] }, { "Namespace": "AWS/MediaPackage", "MetricName": "EgressRequestCount", "Dimensions": [ { "Name": "Channel", "Value": "POC_Castlabs_VTM" }, { "Name": "StatusCodeRange", "Value": "2xx" } ] } ] }

[cristiangreco]

As the metrics are currently very sparse,

Hi @theunissenne thanks for this contribution. What do you mean with the above?

Well I checked if I could filter on dimension etc, but currently MediaPackage only has a few metricsEgressRequestCount etc. The Dimensions are things like Channel, StatusRangeCode, and I did not know if I had to create a dimensionfilter for them?

Yes, if resources in this namespace can be tagged, you should add a ResourceFilter and DimensionRegexps value for them. The former should be the namespace of the resources (guess it's mediapackage), the latter should be a list of regex for ARNs of the resources in the namespace -- looking at AWS docs should be something like:

":channels/(?P<Channel>[^/]+)"
":origin_endpoints/(?P<OriginEndpoint>[^/]+)"
":harvest_jobs/(?P<HarvestJob>[^/]+)"

[theunissenne]

As the metrics are currently very sparse,

Hi @theunissenne thanks for this contribution. What do you mean with the above?

Well I checked if I could filter on dimension etc, but currently MediaPackage only has a few metricsEgressRequestCount etc. The Dimensions are things like Channel, StatusRangeCode, and I did not know if I had to create a dimensionfilter for them?

Yes, if resources in this namespace can be tagged, you should add a ResourceFilter and DimensionRegexps value for them. The former should be the namespace of the resources (guess it's mediapackage), the latter should be a list of regex for ARNs of the resources in the namespace -- looking at AWS docs should be something like:

":channels/(?P<Channel>[^/]+)"
":origin_endpoints/(?P<OriginEndpoint>[^/]+)"
":harvest_jobs/(?P<HarvestJob>[^/]+)"

So, what exactly do you use these dimension for?
Cause the dimension filter in here, should then represent all resource for e.g. mediapackage, or mediapackagevod.
But this has no connection to the actual Dimensions on the metrics?

[theunissenne]

I have been testing a bit, and it seems that MediaPackage has a dimension Channel, but we can't derive it from the arn.
A mediapackage arn contains the id, while it seems the metric dimenions contains the channel name.
So I suppose I just do not filter on those "Channel". Cause if I do and test it locally, I don't not get back any metrics for the channel dimension.

[theunissenne]

aws resourcegroupstaggingapi get-resources --resource-type-filters "mediapackage" --no-cli-pager { "ResourceTagMappingList": [ { "ResourceARN": "arn:aws:mediapackage:eu-west-1:638618404227:channels/24ec33efd6c04dbba4291ad1c4edfcf8", "Tags": [ { "Key": "Test", "Value": "senne" } ] }, { "ResourceARN": "arn:aws:mediapackage:eu-west-1:638618404227:channels/816eee87589e46f2b11801804ddd6ab4", "Tags": [] } ] }

{ "Namespace": "AWS/MediaPackage", "MetricName": "EgressRequestCount", "Dimensions": [ { "Name": "Channel", "Value": "POC_Elemental_SCTE_Test_Senne" }, { "Name": "OriginEndpoint", "Value": "test-senne" }, { "Name": "StatusCodeRange", "Value": "2xx" } ] },

[cristiangreco]

@theunissenne normally the regexps on the arn (e.g. ":channels/(?P<Channel>[^/]+)") corresponds to the name of one or more dimensions on metrics. In this case, one would expect that if the arn is "arn:aws:mediapackage:eu-west-1:638618404227:channels/24ec33efd6c04dbba4291ad1c4edfcf8" then the metric should at least have a dimension Channel with value 24ec33efd6c04dbba4291ad1c4edfcf8. But it doesn't seem the case here :(

What about other resources like OriginEndpoint? Do they have tags assigned? Do you have example arns and metrics with dimensions?

[theunissenne]

@theunissenne normally the regexps on the arn (e.g. ":channels/(?P<Channel>[^/]+)") corresponds to the name of one or more dimensions on metrics. In this case, one would expect that if the arn is "arn:aws:mediapackage:eu-west-1:638618404227:channels/24ec33efd6c04dbba4291ad1c4edfcf8" then the metric should at least have a dimension Channel with value 24ec33efd6c04dbba4291ad1c4edfcf8. But it doesn't seem the case here :(

What about other resources like OriginEndpoint? Do they have tags assigned? Do you have example arns and metrics with dimensions?

Origin endpoints is the same case as mediapackage channels.
e.g.
{ "Namespace": "AWS/MediaPackage", "MetricName": "ActiveInput", "Dimensions": [ { "Name": "OriginEndpoint", "Value": "channel-a" }, { "Name": "IngestEndpoint", "Value": "24ec33efd6c04dbba4291ad1c4edfcf8" } ] },

and if you check the resourcearn:
{ "ResourceARN": "arn:aws:mediapackage:eu-west-1:638618404227:origin_endpoints/7a62e0f5a5754941a0f1f90a0bb34cdb", "Tags": [ { "Key": "Test", "Value": "senne" }, { "Key": "Name", "Value": "channel-a" } ] }

[cristiangreco]

@theunissenne interesting, could you paste the complete output of the get-resources and list-metrics commands?

[theunissenne]

mediapackage_metrics.txt
mediapackage_resource_tagging.txt
mediapackage_vod_resource_tagging.txt

I think we either need to leave filtering blank, so that mediapackage-live metrics also appaer.
For mediapackage vod, it's working a the packaging configuration is in the arn and this will work, however, channels, origin-endpoints are harder, due to their arn not matching any actual dimension

[theunissenne]

Another option ofcourse is to somewhere in the code consider MediaPackage a special case, but this would polute the code right

[cristiangreco]

@theunissenne thanks for the files! The mediapackage namespace looks quite complex, with a lot of auto-generated ids in the dimensions. I've suggested an initial setup, could you please check it out? We can expand furter after some testing.

[theunissenne]

@theunissenne thanks for the files! The mediapackage namespace looks quite complex, with a lot of auto-generated ids in the dimensions. I've suggested an initial setup, could you please check it out? We can expand furter after some testing.

Yeah sure.
I'll also try to go over it a bit more in the upcoming days to look for simularities. The packaging configuration is already okay, was already doing that locally, tested it and confirmed it works. I'll add your config for the ingest endpoint (good catch btw) and do some tests.

[theunissenne]

@cristiangreco

So I just tested a bit, but still there is something off if we just parse using IngestEndpoint.
So your assumption was correct, the first ingestendpoint can be parse used channels/, however MediaPackage has two IngestPoints, and the second one has a different ID.

E.g.
`
  | channel-a | 539ccc8624f943e18b6cd2181d0769fc | IngressBytes
  | channel-a | 539ccc8624f943e18b6cd2181d0769fc | IngressResponseTime
| channel-a | 24ec33efd6c04dbba4291ad1c4edfcf8 | IngressBytes
  | channel-a | 24ec33efd6c04dbba4291ad1c4edfcf8 | IngressResponseTime

`
 
So, let's say that I would tag both ingest endpoints, it would mean that the second one (channel-a arn contains 539ccc8624f943e18b6cd2181d0769fc ) so I guess this one would be filtered out: 24ec33efd6c04dbba4291ad1c4edfcf8

I'll verify it to be sure though

[cristiangreco]

@theunissenne sorry I lost momentum on this PR. Are you still interested in testing it and getting it merged?

[sennetheunispersgroep]

Hi @cristiangreco

I tested it out and it seemed to work. However, our cloudwatch costs was immense due to using YACE (aws knows how to bill you for metrics). So we switch to the cloudwatch plugin.

It can still be nice to merge it maybe for other users, otherwise it's not needed for my use case anymore :)