Prometheus metrics

[xoxys]

Hi, any chance to get some backup metrics in Prometheus native format to file as an alternative to the status file? Maybe a template feature would be useful so status fields can be added to whatever file format is required.

[creativeprojects]

Yes that's something I could look into. To be honest I'm not familiar with prometheus, hence why it's not already done 😄

I quite like the idea of template as well actually 🤔

[xoxys]

Cool! If I could help somehow or test something, let me know.

[creativeprojects]

Maybe a template feature would be useful so status fields can be added to whatever file format is required.

Come to think of it, I'm not sure if it's a good idea to implement a templating system at that point:

• it would work great for a short lived status file, like prometheus: you send it to the push gateway and you're done with it
• it would not work very well for a status.json file if you have more than one profile: each profile writes their new details into the status file, but leave the information of the other profiles intact

[xoxys]

but leave the information of the other profiles intact

Is that the expected behavior or not? How does the current status file work? It will append information from tha last run to the status file and keep the history forever?

[creativeprojects]

It updates the details of the current profile, as such it doesn't keep any history of it. But it leaves the information of the other profiles in place.

[creativeprojects]

So I was reading a bit about prometheus and I'm not sure what is the best approach here, maybe you can help me:

• it looks like, as a batch job, we could send the results to a push gateway once finished. I believe this is what you're asking for, but as I was reading, it has drawbacks.
• as a process that can take a significant amount of time (depending on the size of the backups) we could also provide a HTTP endpoint where prometheus can get the metrics and progress of the backup in real time. That would work nicely when the backup is running, but then that would trigger a down event when the backup is finished.

As a prometheus user, do you think both approaches are valid?
Is it going to be the case that some users would prefer the push method and some others the standard method for services?

[creativeprojects]

For a first version, I'm saving the metrics to a file (to send manually) and/or push to a URL.

It adds two configuration parameters per profile:

profile:
    inherit: default
    prometheus-save-to-file: "test.prom"
    prometheus-push: "http://push-gateway:9091"
    backup:
        extended-status: true
        source: /

Here's a sample file:

# HELP resticprofile_backup_added_bytes Total number of bytes added to the repository.
# TYPE resticprofile_backup_added_bytes gauge
resticprofile_backup_added_bytes 56493
# HELP resticprofile_backup_dir_changed Number of directories with changes.
# TYPE resticprofile_backup_dir_changed gauge
resticprofile_backup_dir_changed 8
# HELP resticprofile_backup_dir_new Number of new directories added to the backup.
# TYPE resticprofile_backup_dir_new gauge
resticprofile_backup_dir_new 0
# HELP resticprofile_backup_dir_unmodified Number of directories unmodified since last backup.
# TYPE resticprofile_backup_dir_unmodified gauge
resticprofile_backup_dir_unmodified 314
# HELP resticprofile_backup_duration_seconds The duration of backup in seconds.
# TYPE resticprofile_backup_duration_seconds gauge
resticprofile_backup_duration_seconds 0.938584574
# HELP resticprofile_backup_files_changed Number of files with changes.
# TYPE resticprofile_backup_files_changed gauge
resticprofile_backup_files_changed 2
# HELP resticprofile_backup_files_new Number of new files added to the backup.
# TYPE resticprofile_backup_files_new gauge
resticprofile_backup_files_new 1
# HELP resticprofile_backup_files_processed Total number of files scanned by the backup for changes.
# TYPE resticprofile_backup_files_processed gauge
resticprofile_backup_files_processed 3864
# HELP resticprofile_backup_files_unmodified Number of files unmodified since last backup.
# TYPE resticprofile_backup_files_unmodified gauge
resticprofile_backup_files_unmodified 3861
# HELP resticprofile_backup_processed_bytes Total number of bytes scanned for changes.
# TYPE resticprofile_backup_processed_bytes gauge
resticprofile_backup_processed_bytes 3.5324565e+07
# HELP resticprofile_backup_status Backup status: 0=fail, 1=warning, 2=success.
# TYPE resticprofile_backup_status gauge
resticprofile_backup_status 2

Before going further, I wanted to know if the file looks like what you'd expect 👍🏻

[creativeprojects]

Right, so here's a simple file running a single profile:

# HELP resticprofile_backup_added_bytes Total number of bytes added to the repository.
# TYPE resticprofile_backup_added_bytes gauge
resticprofile_backup_added_bytes{profile="prom"} 35746
# HELP resticprofile_backup_dir_changed Number of directories with changes.
# TYPE resticprofile_backup_dir_changed gauge
resticprofile_backup_dir_changed{profile="prom"} 9
# HELP resticprofile_backup_dir_new Number of new directories added to the backup.
# TYPE resticprofile_backup_dir_new gauge
resticprofile_backup_dir_new{profile="prom"} 0
# HELP resticprofile_backup_dir_unmodified Number of directories unmodified since last backup.
# TYPE resticprofile_backup_dir_unmodified gauge
resticprofile_backup_dir_unmodified{profile="prom"} 314
# HELP resticprofile_backup_duration_seconds The backup duration (in seconds).
# TYPE resticprofile_backup_duration_seconds gauge
resticprofile_backup_duration_seconds{profile="prom"} 0.946567354
# HELP resticprofile_backup_files_changed Number of files with changes.
# TYPE resticprofile_backup_files_changed gauge
resticprofile_backup_files_changed{profile="prom"} 3
# HELP resticprofile_backup_files_new Number of new files added to the backup.
# TYPE resticprofile_backup_files_new gauge
resticprofile_backup_files_new{profile="prom"} 0
# HELP resticprofile_backup_files_processed Total number of files scanned by the backup for changes.
# TYPE resticprofile_backup_files_processed gauge
resticprofile_backup_files_processed{profile="prom"} 3925
# HELP resticprofile_backup_files_unmodified Number of files unmodified since last backup.
# TYPE resticprofile_backup_files_unmodified gauge
resticprofile_backup_files_unmodified{profile="prom"} 3922
# HELP resticprofile_backup_processed_bytes Total number of bytes scanned for changes.
# TYPE resticprofile_backup_processed_bytes gauge
resticprofile_backup_processed_bytes{profile="prom"} 3.8524672e+07
# HELP resticprofile_backup_status Backup status: 0=fail, 1=warning, 2=success.
# TYPE resticprofile_backup_status gauge
resticprofile_backup_status{profile="prom"} 1
# HELP resticprofile_build_info resticprofile build information.
# TYPE resticprofile_build_info gauge
resticprofile_build_info{goversion="go1.16.6",version="0.15.0-dev"} 1

A label "group" is added when the profile is running in a group (not shown here).

Are you familiar with compilation of Go programs? Would you be able to give it a try from the prometheus branch?

[xoxys]

Awesome! Sure, I'll give it a shot.

[xoxys]

I've compiled and deployed it now to one of my hosts, works like a charm in general, but it looks like some values were not updated:

2021/08/04 13:57:16 using configuration file: /etc/restic/profiles.yaml
2021/08/04 13:57:16 profile 'traefik': initializing repository (if not existing)
2021/08/04 13:57:18 profile 'traefik': checking repository consistency
using temporary cache in /tmp/restic-check-cache-173969411
repository 463b1642 opened successfully, password is correct
created new cache in /tmp/restic-check-cache-173969411
create exclusive lock for repository
load indexes
check all packs
check snapshots, trees and blobs
[0:01] 100.00%  10 / 10 snapshots
no errors were found
2021/08/04 13:57:22 profile 'traefik': starting 'backup'
repository 463b1642 opened successfully, password is correct
using parent snapshot cd8922f8

Files:           1 new,     1 changed,     2 unmodified
Dirs:            1 new,     2 changed,     0 unmodified
Added to the repo: 2.491 KiB

processed 4 files, 3.596 KiB in 0:25
snapshot 8bbf22a9 saved
2021/08/04 13:57:49 profile 'traefik': finished 'backup'

But metrics still show:

# HELP resticprofile_backup_added_bytes Total number of bytes added to the repository.
# TYPE resticprofile_backup_added_bytes gauge
resticprofile_backup_added_bytes{group="full-backup",profile="traefik"} 0
# HELP resticprofile_backup_dir_changed Number of directories with changes.
# TYPE resticprofile_backup_dir_changed gauge
resticprofile_backup_dir_changed{group="full-backup",profile="traefik"} 0
# HELP resticprofile_backup_dir_new Number of new directories added to the backup.
# TYPE resticprofile_backup_dir_new gauge
resticprofile_backup_dir_new{group="full-backup",profile="traefik"} 0
# HELP resticprofile_backup_dir_unmodified Number of directories unmodified since last backup.
# TYPE resticprofile_backup_dir_unmodified gauge
resticprofile_backup_dir_unmodified{group="full-backup",profile="traefik"} 0
# HELP resticprofile_backup_duration_seconds The backup duration (in seconds).
# TYPE resticprofile_backup_duration_seconds gauge
resticprofile_backup_duration_seconds{group="full-backup",profile="traefik"} 27.026780902
# HELP resticprofile_backup_files_changed Number of files with changes.
# TYPE resticprofile_backup_files_changed gauge
resticprofile_backup_files_changed{group="full-backup",profile="traefik"} 0
# HELP resticprofile_backup_files_new Number of new files added to the backup.
# TYPE resticprofile_backup_files_new gauge
resticprofile_backup_files_new{group="full-backup",profile="traefik"} 0
# HELP resticprofile_backup_files_processed Total number of files scanned by the backup for changes.
# TYPE resticprofile_backup_files_processed gauge
resticprofile_backup_files_processed{group="full-backup",profile="traefik"} 0
# HELP resticprofile_backup_files_unmodified Number of files unmodified since last backup.
# TYPE resticprofile_backup_files_unmodified gauge
resticprofile_backup_files_unmodified{group="full-backup",profile="traefik"} 0
# HELP resticprofile_backup_processed_bytes Total number of bytes scanned for changes.
# TYPE resticprofile_backup_processed_bytes gauge
resticprofile_backup_processed_bytes{group="full-backup",profile="traefik"} 0
# HELP resticprofile_backup_status Backup status: 0=fail, 1=warning, 2=success.
# TYPE resticprofile_backup_status gauge
resticprofile_backup_status{group="full-backup",profile="traefik"} 2
# HELP resticprofile_build_info resticprofile build information.
# TYPE resticprofile_build_info gauge
resticprofile_build_info{goversion="go1.16.6",version="0.15.0-dev"} 1

[creativeprojects]

It's because you need to add the option extended-status: true to your backup configuration.
The reason why it's not activated by default is that it's hiding the progress when running on the command line.

See Extended Status section

[xoxys]

Ah, thanks. Missed that completely. Ok works now, hope to see it in a release soon 😃

[creativeprojects]

Support for generating a file + send via push gateway: #57