Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add prometheus metrics support #1859

Open
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

songjiaxun
Copy link
Contributor

@songjiaxun songjiaxun commented Apr 23, 2024

Description

Add Prometheus metrics support. Add a new flag prometheus-port for users to specify prometheus metrics endpoint port.

Link to the issue in case of a bug fix.

NA

Testing details

  1. Manual - Done
  • Build the code using the command go run tools/build_gcsfuse/main.go . . v3.0.0.
  • Use the following config file:
monitoring:
  prometheus-port: "8080"
  • Mount a bucket: ./bin/gcsfuse --config-file <config_file_path> <bucket-name> <mount-path>.
  • Pull the metrics: curl http://localhost:8080/metrics
  • See the output example in the docs/metrics.md.
  1. Unit tests - Done
  2. Integration tests - NA
  3. Performance tests

@songjiaxun songjiaxun requested a review from a team as a code owner April 23, 2024 07:15
@songjiaxun songjiaxun requested a review from sethiay April 23, 2024 07:15
@songjiaxun songjiaxun force-pushed the master branch 3 times, most recently from 9772ba2 to 77d5d33 Compare April 24, 2024 17:31
flags.go Outdated Show resolved Hide resolved
MaxHeaderBytes: 1 << 20,
}

go func() {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Though this should be lightweight but we should do run perf testing of GCSfuse with promotheus server running and also ensure that this doesn't take extra memory and CPU

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @sethiay, thanks for the review! I updated the PR to move the mount flag to config file. Do you happen to have a guidance to do the GCSFuse perf test and memory/CPU resource tests?

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey @songjiaxun We have presubmit performance tests (I have enabled for this PR with the labels) that do some basic performance testing without capturing CPU and memory. That is good to ensure that performance doesn't degrade incase some business logic is changed or dependencies are upgraded. However, this change requires more testing and monitoring CPU and memory, so I recommend running all the tests pipeline that we have and monitor the memory, CPU and performance numbers for the tests.

@songjiaxun songjiaxun force-pushed the master branch 3 times, most recently from 2610032 to d21c254 Compare May 25, 2024 13:01
Copy link

codecov bot commented May 25, 2024

Codecov Report

Attention: Patch coverage is 8.10811% with 34 lines in your changes are missing coverage. Please review.

Project coverage is 61.09%. Comparing base (89392be) to head (1afeaac).
Report is 2 commits behind head on master.

Files Patch % Lines
internal/monitor/exporter.go 0.00% 31 Missing ⚠️
main.go 0.00% 3 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##           master    #1859      +/-   ##
==========================================
- Coverage   61.27%   61.09%   -0.19%     
==========================================
  Files         130      130              
  Lines       12458    12495      +37     
==========================================
  Hits         7634     7634              
- Misses       4481     4517      +36     
- Partials      343      344       +1     
Flag Coverage Δ
unittests 61.09% <8.10%> (-0.19%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@sethiay sethiay added execute-perf-test Execute performance test in PR execute-integration-tests Run only integration tests labels May 27, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
execute-integration-tests Run only integration tests execute-perf-test Execute performance test in PR
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants