[DOC] Adding missing metric-generator keys

[WesselAtWork]

What this PR does:

Adds missing keys to the metrics-generator

Which issue(s) this PR fixes:
Fixes #3642

Sources/References

override_ring_key

---

tempo/modules/generator/config.go

Line 49 in 251bf5a

cfg.QueryTimeout = 30 * time.Second

override_ring_key

---

tempo/modules/generator/config.go

Line 50 in 251bf5a

cfg.OverrideRingKey = generatorRingKey

tempo/modules/generator/config.go

Lines 18 to 19 in 251bf5a

	// generatorRingKey is the default key under which we store the metric-generator's ring in the KVStore.
	generatorRingKey = "metrics-generator"

processor.service_graphs.peer_attributes

tempo/modules/generator/processor/servicegraphs/config.go

Lines 54 to 57 in 251bf5a

	peerAttr := make([]string, 0, len(defaultPeerAttributes))
	for _, attr := range defaultPeerAttributes {
	peerAttr = append(peerAttr, string(attr))
	}

tempo/modules/generator/processor/servicegraphs/servicegraphs.go

Lines 57 to 59 in 251bf5a

	var defaultPeerAttributes = []attribute.Key{
	semconv.PeerServiceKey, semconv.DBNameKey, semconv.DBSystemKey,
	}

processor.span_metrics.subprocessors

This one did not work

tempo/modules/generator/processor/spanmetrics/config.go

Lines 59 to 62 in 251bf5a

	cfg.Subprocessors = make(map[Subprocessor]bool)
	cfg.Subprocessors[Latency] = true
	cfg.Subprocessors[Count] = true
	cfg.Subprocessors[Size] = true

storage.trace.pool

tempo/tempodb/pool/config.go

Lines 11 to 12 in 854caca

	MaxWorkers: 30,
	QueueDepth: 10000,

[CLAassistant]

All committers have signed the CLA.

[WesselAtWork]

Unsure if I should put the WAL config in it's own block.
The loki docs usually separate out common config into typed blocks
e.g. https://grafana.com/docs/loki/latest/configure/#s3_storage_config

They postfix these blocks with _config

[WesselAtWork]

Needed a way to show that this particular component is overriding values from the base component block. (WAL Config)

Wanted to keep it light for the future, so no comments where none would add additional value.
If there were comments we would need to keep both synchronized.

This way if you are making changes to the query-frontend you won't need to change the base block because it isn't dependant on the query-frontend
Same in reverse, if you made changes to the base WAL Config you wouldn't need to propagate them to every component that use it. (Unless you made changes to them too)

Feel free to dispute the omission of the newline breaks.

I felt a need to keep this visually distinct, but I am not married to the idea.
Might be better to keep the format consistent

[WesselAtWork]

Not quite sure on the placement of this (Configuration blocks) section

Feel free to dispute

[WesselAtWork]

I didn't find a default value for the path is set in the code, around the metrics-generator
But If I look at the wal code, it should except if you do not set it, which feels like an oversight?
It feels like it should set a default value?

tempo/tempodb/wal/wal.go

Lines 52 to 54 in 251bf5a

	if c.Filepath == "" {
	return nil, fmt.Errorf("please provide a path for the WAL")
	}

[WesselAtWork]

Just discovered something:
missing the localblocks key

tempo/modules/generator/config.go

Lines 53 to 57 in 251bf5a

	type ProcessorConfig struct {
	ServiceGraphs servicegraphs.Config `yaml:"service_graphs"`
	SpanMetrics spanmetrics.Config `yaml:"span_metrics"`
	LocalBlocks localblocks.Config `yaml:"local_blocks"`
	}

and most of the ring keys:

tempo/modules/generator/generator_ring.go

Lines 18 to 31 in 251bf5a

	type RingConfig struct {
	KVStore kv.Config `yaml:"kvstore"`
	HeartbeatPeriod time.Duration `yaml:"heartbeat_period"`
	HeartbeatTimeout time.Duration `yaml:"heartbeat_timeout"`
	InstanceID string `yaml:"instance_id"`
	InstanceInterfaceNames []string `yaml:"instance_interface_names"`
	InstanceAddr string `yaml:"instance_addr"`
	InstancePort int `yaml:"instance_port"`
	EnableInet6 bool `yaml:"enable_inet6"`
	// Injected internally
	ListenPort int `yaml:"-"`
	}

[WesselAtWork]

Is this correct?

tempo/modules/generator/processor/spanmetrics/config.go

Line 54 in 251bf5a

cfg.HistogramBuckets = prometheus.ExponentialBuckets(0.002, 2, 14)

Should be 14 entries correct?

[joe-elliott]

Howdy, @WesselAtWork. Appreciate all the details cleaned up here!

@kvrhdn is going to take a look at this in the next few days. Just didn't want you to think we were ignoring you.

[knylander-grafana]

I love all the clean up you've done on this doc. It really helps readability. (I'll defer to developer review for this PR.)

[kvrhdn]

Thank you for this work! 🙏🏻

Sorry it took a while to get through it, overall changes look good I've just left some smaller comments. Once those are addressed this is good to go 🙂

[WesselAtWork]

Help with these descriptions please

[knylander-grafana]

Happy to try to help. What do you mean by server kinds?

[WesselAtWork]

tempo/modules/generator/processor/localblocks/processor.go

Lines 886 to 920 in 56a0b8c

	// filterBatches to only root spans or kind==server. Does not modify the input
	// but returns a new struct referencing the same input pointers. Returns nil
	// if there were no matching spans.
	func filterBatches(batches []*v1.ResourceSpans) []*v1.ResourceSpans {
	keep := make([]*v1.ResourceSpans, 0, len(batches))
	for _, batch := range batches {
	var keepSS []*v1.ScopeSpans
	for _, ss := range batch.ScopeSpans {
	var keepSpans []*v1.Span
	for _, s := range ss.Spans {
	if s.Kind == v1.Span_SPAN_KIND_SERVER || len(s.ParentSpanId) == 0 {
	keepSpans = append(keepSpans, s)
	}
	}
	if len(keepSpans) > 0 {
	keepSS = append(keepSS, &v1.ScopeSpans{
	Scope: ss.Scope,
	Spans: keepSpans,
	})
	}
	}
	if len(keepSS) > 0 {
	keep = append(keep, &v1.ResourceSpans{
	Resource: batch.Resource,
	ScopeSpans: keepSS,
	})
	}
	}
	return keep
	}

It's the span kind(?) client/server kinds I think.

[kvrhdn]

My suggestions would be:

            # How often to run flush checks (??)
            [flush_check_period: <duration> | default = 10s]

How often to run the flush loop to cut idle traces and blocks

            # After this idle period, a trace is closed. (??)
            [trace_idle_period: <duration> | default = 10s]

Duration after which to consider a trace complete if no spans have been received

(this is copied from the flag definition)

            # Maximum (???) (lifespan for any given block?)
            [max_block_duration: <duration> | default = 1m]

Maximum duration which the head block can be appended to before cutting it

            # Maximum block size before it is closed (?)
            [max_block_bytes: <uint64> | default = 500000000]

Maximum size of the head block before cutting it

            # Maximum (???)
            [complete_block_timeout: <duration> | default = 1h]

Duration to keep blocks in the ingester after they have been flushed

            # (Unused?)
            # Max number of "live" traces
            [max_live_traces: <uint64>]

Maximum amount of live traces, if this is exceeded traces will be dropped with reason live_traces_exceeded.

            # Whether server spans should be filtered in or not.
            # true means keep the server kinds (???)
            # false means drop the server kinds (???)
            [filter_server_spans: <bool> | default = true]

This code iterates through all spans and only keeps spans that either:

• have span.kind == SPAN_KIND_SERVER
• don't have a parent span

If enabled, only parent spans or spans with SpanKind server will be retained

[WesselAtWork]

Out of curiosity, why only retain parents and Server spans by default?
What about other spans or Client types?

[WesselAtWork]

I think I found a bug?

I described what the code does here, but I don't think this is the intended behaviour...

Please verify :

tempo/tempodb/wal/wal.go

Lines 70 to 75 in 3a4f309

	p := filepath.Join(c.Filepath, blocksDir)
	err = os.MkdirAll(p, os.ModePerm)
	if err != nil {
	return nil, err
	}
	c.BlocksFilepath = p

I think it's supposed to look like this (but with c.BlocksFilepath == ""):

tempo/tempodb/wal/wal.go

Lines 56 to 67 in 3a4f309

	// The /completed/ folder is now obsolete and no new data is written,
	// but it needs to be cleared out one last time for any files left
	// from a previous version.
	if c.CompletedFilepath == "" {
	completedFilepath := filepath.Join(c.Filepath, completedDir)
	err = os.RemoveAll(completedFilepath)
	if err != nil {
	return nil, err
	}
	c.CompletedFilepath = completedFilepath
	}