Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fix panic in query-frontend when combining results #3683

Merged
merged 2 commits into from
May 16, 2024
Merged

Fix panic in query-frontend when combining results #3683

merged 2 commits into from
May 16, 2024

Conversation

mapno
Copy link
Member

@mapno mapno commented May 16, 2024

What this PR does:

Fixes a panic when combining results in the query frontend.

Panic stack trace 
2024-05-15 13:01:13.396		/Users/mapno/workspace/github.com/grafana/tempo/vendor/google.golang.org/grpc/server.go:1030 +0x125
2024-05-15 13:01:13.396	created by google.golang.org/grpc.(*Server).serveStreams.func2 in goroutine 1386
2024-05-15 13:01:13.396		/Users/mapno/workspace/github.com/grafana/tempo/vendor/google.golang.org/grpc/server.go:1019 +0x8b
2024-05-15 13:01:13.396	google.golang.org/grpc.(*Server).serveStreams.func2.1()
2024-05-15 13:01:13.396		/Users/mapno/workspace/github.com/grafana/tempo/vendor/google.golang.org/grpc/server.go:1784 +0xe3a
2024-05-15 13:01:13.396	google.golang.org/grpc.(*Server).handleStream(0xc000725600, {0x3161420, 0xc0006455f0}, 0xc046aef680)
2024-05-15 13:01:13.396		/Users/mapno/workspace/github.com/grafana/tempo/vendor/google.golang.org/grpc/server.go:1670 +0x11e7
2024-05-15 13:01:13.396	google.golang.org/grpc.(*Server).processStreamingRPC(0xc000725600, {0x3144358, 0xc046afebd0}, {0x3161420, 0xc0006455f0}, 0xc046aef680, 0xc04193e780, 0x484a9e0, 0x0)
2024-05-15 13:01:13.396		/Users/mapno/workspace/github.com/grafana/tempo/vendor/google.golang.org/grpc/server.go:1506 +0x85
2024-05-15 13:01:13.396	google.golang.org/grpc.NewServer.chainStreamServerInterceptors.chainStreamInterceptors.func2({0x268c620, 0xc0416ca5b0}, {0x3153a00, 0xc0469ff0e0}, 0xc040e11f50, 0xc046ab0c60?)
2024-05-15 13:01:13.396		/Users/mapno/workspace/github.com/grafana/tempo/vendor/github.com/grafana/dskit/middleware/grpc_logging.go:77 +0xab
2024-05-15 13:01:13.396	github.com/grafana/dskit/middleware.GRPCServerLog.StreamServerInterceptor({{0x31201e0?, 0xc000642b90?}, 0x60?, 0xc2?}, {0x268c620, 0xc0416ca5b0}, {0x3153a00, 0xc0469ff0e0}, 0xc040e11f50, 0xc046af0d00)
2024-05-15 13:01:13.396		/Users/mapno/workspace/github.com/grafana/tempo/vendor/google.golang.org/grpc/server.go:1515 +0xb2
2024-05-15 13:01:13.396	google.golang.org/grpc.getChainStreamHandler.func1({0x268c620, 0xc0416ca5b0}, {0x3153a00, 0xc0469ff0e0})
2024-05-15 13:01:13.396		/Users/mapno/workspace/github.com/grafana/tempo/vendor/github.com/opentracing-contrib/go-grpc/server.go:114 +0x34a
2024-05-15 13:01:13.396	github.com/opentracing-contrib/go-grpc.OpenTracingStreamServerInterceptor.func1({0x268c620, 0xc0416ca5b0}, {0x3153a00, 0xc0469ff0e0}, 0xc040e11f50, 0xc046af0d40)
2024-05-15 13:01:13.396		/Users/mapno/workspace/github.com/grafana/tempo/vendor/google.golang.org/grpc/server.go:1515 +0xb2
2024-05-15 13:01:13.396	google.golang.org/grpc.getChainStreamHandler.func1({0x268c620, 0xc0416ca5b0}, {0x3154030, 0xc0474ae000})
2024-05-15 13:01:13.396		/Users/mapno/workspace/github.com/grafana/tempo/vendor/github.com/grafana/dskit/middleware/grpc_instrumentation.go:44 +0xa4
2024-05-15 13:01:13.395	github.com/grafana/dskit/middleware.StreamServerInstrumentInterceptor.func1({0x268c620, 0xc0416ca5b0}, {0x3154030, 0xc0474ae000}, 0xc040e11f50, 0xc046af0f00)
2024-05-15 13:01:13.395		/Users/mapno/workspace/github.com/grafana/tempo/vendor/google.golang.org/grpc/server.go:1515 +0xb2
2024-05-15 13:01:13.395	google.golang.org/grpc.getChainStreamHandler.func1({0x268c620, 0xc0416ca5b0}, {0x3154030, 0xc0474ae000})
2024-05-15 13:01:13.395		/Users/mapno/workspace/github.com/grafana/tempo/cmd/tempo/app/app.go:156 +0x9c
2024-05-15 13:01:13.395	github.com/grafana/tempo/cmd/tempo/app.(*App).setupAuthMiddleware.func2({0x268c620, 0xc0416ca5b0}, {0x3154030, 0xc0474ae000}, 0xc040e11f50, 0x2a51190)
2024-05-15 13:01:13.395		/Users/mapno/workspace/github.com/grafana/tempo/vendor/github.com/grafana/dskit/middleware/grpc_auth.go:53 +0xb2
2024-05-15 13:01:13.395	github.com/grafana/dskit/middleware.StreamServerUserHeaderInterceptor({0x268c620, 0xc0416ca5b0}, {0x3154030, 0xc0474ae000}, 0x7e9dc30c8ab0?, 0x2a51190)
2024-05-15 13:01:13.395		/Users/mapno/workspace/github.com/grafana/tempo/pkg/tempopb/tempo.pb.go:4100 +0x107
2024-05-15 13:01:13.395	github.com/grafana/tempo/pkg/tempopb._StreamingQuerier_Search_Handler({0x268c620, 0xc0416ca5b0}, {0x3153898, 0xc0474ae020})
2024-05-15 13:01:13.395		/Users/mapno/workspace/github.com/grafana/tempo/modules/frontend/frontend.go:171 +0x2d
2024-05-15 13:01:13.395	github.com/grafana/tempo/modules/frontend.(*QueryFrontend).Search(0x481c1c0?, 0x268c620?, {0x3157330?, 0xc046ab0db0?})
2024-05-15 13:01:13.395		/Users/mapno/workspace/github.com/grafana/tempo/modules/frontend/search_handlers.go:60 +0x8f1
2024-05-15 13:01:13.395	github.com/grafana/tempo/modules/frontend.newSearchStreamingGRPCHandler.func1(0xc046af0f40, {0x3157330, 0xc046ab0db0})
2024-05-15 13:01:13.395		/Users/mapno/workspace/github.com/grafana/tempo/modules/frontend/pipeline/collector_grpc.go:53 +0x207
2024-05-15 13:01:13.395	github.com/grafana/tempo/modules/frontend/pipeline.GRPCCollector[...].RoundTrip(0x20?, 0xc046aefd40)
2024-05-15 13:01:13.395		/Users/mapno/workspace/github.com/grafana/tempo/modules/frontend/combiner/common.go:77 +0x357
2024-05-15 13:01:13.395	github.com/grafana/tempo/modules/frontend/combiner.(*genericCombiner[...]).AddResponse(0x31701c0, {0x31318d8, 0xc047167230})
2024-05-15 13:01:13.395		/Users/mapno/workspace/github.com/grafana/tempo/modules/frontend/combiner/search.go:31 +0xd4
2024-05-15 13:01:13.395	github.com/grafana/tempo/modules/frontend/combiner.NewSearch.func2(0xc04194b2a0, 0xc0474ae060, {0x3139c60?, 0xc04194b2a0?})
2024-05-15 13:01:13.395		/Users/mapno/workspace/github.com/grafana/tempo/pkg/traceql/combine.go:24 +0x95
2024-05-15 13:01:13.395	github.com/grafana/tempo/pkg/traceql.(*MetadataCombiner).AddMetadata(0xc00049f9a0, 0xc0005ac230)
2024-05-15 13:01:13.395		/Users/mapno/workspace/github.com/grafana/tempo/pkg/traceql/combine.go:84 +0x1e5
2024-05-15 13:01:13.395	github.com/grafana/tempo/pkg/traceql.combineSearchResults(0xc000564460, 0xc0005ac230)
2024-05-15 13:01:13.395	goroutine 1390 [running]:
2024-05-15 13:01:13.395	
2024-05-15 13:01:13.395	panic: assignment to entry in nil map

Checklist

  • Tests updated
  • Documentation added
  • CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX]

electron0zero
electron0zero approved these changes May 16, 2024
View reviewed changes
Copy link
Member

@electron0zero electron0zero left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

tested, it works :shipit:

@mapno mapno merged commit 805b3f5 into grafana:main May 16, 2024
18 checks passed
@mapno mapno deleted the combine-panic branch May 16, 2024 13:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@electron0zero electron0zero electron0zero approved these changes

@joe-elliott joe-elliott Awaiting requested review from joe-elliott joe-elliott is a code owner

@annanay25 annanay25 Awaiting requested review from annanay25 annanay25 is a code owner

@mdisibio mdisibio Awaiting requested review from mdisibio mdisibio is a code owner

@kvrhdn kvrhdn Awaiting requested review from kvrhdn kvrhdn is a code owner

@zalegrala zalegrala Awaiting requested review from zalegrala zalegrala is a code owner

@ie-pham ie-pham Awaiting requested review from ie-pham ie-pham is a code owner

@stoewer stoewer Awaiting requested review from stoewer stoewer is a code owner

Assignees
No one assigned
Labels
component/query-frontend type/bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
2 participants
@mapno @electron0zero