should I install aws cli in the otel-collector service to enable cloudwatch?

[AllanOricil]

I'm kind lost with this tutorial and need your help. My first question raised when it told me to install otel-collector. I believe I don't need to install otel-collector, because it is already running as a container.

I think it is properly configured because I was able to run telemetrygen test command without a problem as you can see below

Then while continuing with the tutorial I could not figure out where exactly to setp aws. So, these are my question regarding the following step:

Should I install and configure AWS cli in the container?
How can I test that I configured cloudwatch receiver properly?

Thanks in advance for the time you spent trying to help me!

[chiragasawa]

@AllanOricil how did you achieve this, please share your findings?

[AllanOricil]

1. Create an IAM role to attach to the EC2 instance you will create in step 2. This IAM role must have all policies related to cloud watch logs (e.g. list log groups, log streams ...). Just add all permissions related to cloud watch and give access to all log groups arns you want to let the otel-collector to extract.
2. Create an EC2 in aws using Amazon Linux 2023 image or any other Linux distro that the otel-collector binary supports. Don't forget to use use the IAM role you created in step 1.
3. ssh into this EC2.
4. Download the otel-collector binary for your Linux distribution
5. setup the otel-collector config.yaml like explained in Signoz docs.
6. Install aws cli if your ami does not have it already
7. Test if you can list log groups using the aws cli. If not, then run aws configure and set it up with a IAM User credentials. I don't remember if the IAM role attached to the instance is enought. If you still need to create an IAM User update this "guide" in a comment to help others.
8. Now start the otel-collector binary with the config.yaml you created in step 5 as a background service using systemd. I have to google how to do it all the time because I never remember. So do it too, please. Configure this service to restart when the machine restarts.

I will share my config.yaml here later.

Obs: the cloudwatch receiver is not working properly. It does not scrape all logs and it only get the newer ones. Also, when it gets the newer ones, it does not retrieve all events from that log stream. For example, In my case I had a log stream with more than 15 log events, but it just got the first 15.

Read more about this issue here: open-telemetry/opentelemetry-collector-contrib#32231

[AllanOricil]

If you want to use the otel-collector that is shipped with Signoz as a docker container called "signoz-otel-collector", then you have to:

1. go inside that running container (e.g. docker exec -it CONTAINER_ID /bin/bash or /bin/sh)
2. install and configure aws cli
3. update the otel collector config.yaml that is located somewhere inside the container to include the cloudwatch receiver and a pipeline for extracting cloudwatch logs
4. then finally restart the container.