DAPR Sidecar fails as Pub Sub container takes time to initiate

[spicycoder]

I have an Aspire project, with 2 Web API Projects

• Publisher
• Subscriber

My AppHost's Program.cs looks like

using Aspire.Hosting.Dapr;

var builder = DistributedApplication.CreateBuilder(args);

var rabbitMQ = builder
    .AddContainer("rabbitMQ", "rabbitmq", "3-management")
    .WithEndpoint(scheme: "amqp", port: 5672, targetPort: 5672)
    .WithEndpoint(scheme: "http", port: 15672, targetPort: 15672)
    .WithEnvironment("RABBITMQ_DEFAULT_USER", "guest")
    .WithEnvironment("RABBITMQ_DEFAULT_PASS", "guest");

var pubsub = builder.AddDaprPubSub(
    "pubsub",
    new DaprComponentOptions
    {
        LocalPath = Path.Combine("..", "components", "pubsub.yaml")
    });

var publisher = builder.AddProject<Projects.Publisher>("publisher")
    .WithDaprSidecar()
    .WithReference(pubsub);

var subscriber = builder.AddProject<Projects.Subscriber>("subscriber")
    .WithDaprSidecar()
    .WithReference(pubsub);

builder.Build().Run();

And components/pubsub.yaml looks like

apiVersion: dapr.io/v1alpha1
kind: Component
metadata:
  name: pubsub
  namespace: default
spec:
  type: pubsub.rabbitmq
  version: v1
  metadata:
  - name: protocol
    value: amqp
  - name: hostname
    value: localhost
  - name: username
    value: guest
  - name: password
    value: guest

When I disable the statement from Aspire, to spin up an instance of RabbitMQ and use docker run instead, everything works just fine

docker run -d --name rabbit -e RABBITMQ_DEFAULT_USER=guest -e RABBITMQ_DEFAULT_PASS=guest -p 5672:5672 -p 15672:15672 rabbitmq:3-management

But when I use Aspire to spin up an instance of RabbitMQ, with same setup (5672, 15672, guest\guest), the DAPR sidecars fail (both Publisher & Subscriber)

Error Message

Quick


Resources
Console
Structured
Traces
Metrics
Console logs
Watching logs...

The daprd process exited with error code: exit status 1
Error exiting Dapr: exit status 1
WARNING: no application command found.
Starting Dapr with id publisher. HTTP Port: 52809. gRPC Port: 52808
time="2024-05-14T14:30:57.554847+05:30" level=info msg="Starting Dapr Runtime -- version 1.13.2 -- commit f09b193d8ac62bef6c0fd5452a00d2a49cd0d134" app_id=publisher instance=DevPC scope=dapr.runtime type=log ver=1.13.2
time="2024-05-14T14:30:57.5553546+05:30" level=info msg="Log level set to: info" app_id=publisher instance=DevPC scope=dapr.runtime type=log ver=1.13.2
time="2024-05-14T14:30:57.5553546+05:30" level=warning msg="mTLS is disabled. Skipping certificate request and tls validation" app_id=publisher instance=DevPC scope=dapr.runtime.security type=log ver=1.13.2
time="2024-05-14T14:30:57.560073+05:30" level=warning msg="The default value for 'spec.metric.http.increasedCardinality' will change to 'false' in Dapr 1.14" app_id=publisher instance=DevPC scope=dapr.runtime type=log ver=1.13.2
time="2024-05-14T14:30:57.5610981+05:30" level=info msg="standalone mode configured" app_id=publisher instance=DevPC scope=dapr.runtime type=log ver=1.13.2
time="2024-05-14T14:30:57.5616414+05:30" level=info msg="app id: publisher" app_id=publisher instance=DevPC scope=dapr.runtime type=log ver=1.13.2
time="2024-05-14T14:30:57.5616414+05:30" level=info msg="metrics server started on :52810/" app_id=publisher instance=DevPC scope=dapr.runtime type=log ver=1.13.2
time="2024-05-14T14:30:57.5627352+05:30" level=info msg="Dapr trace sampler initialized: DaprTraceSampler(P=1.000000)" app_id=publisher instance=DevPC scope=dapr.runtime type=log ver=1.13.2
time="2024-05-14T14:30:57.6113844+05:30" level=info msg="local service entry announced: publisher -> 192.168.1.124:52815" app_id=publisher component="nr (mdns/v1)" instance=DevPC scope=dapr.contrib type=log ver=1.13.2
time="2024-05-14T14:30:57.6119132+05:30" level=info msg="Initialized name resolution to mdns" app_id=publisher instance=DevPC scope=dapr.runtime type=log ver=1.13.2
time="2024-05-14T14:30:57.6119132+05:30" level=info msg="Loading components…" app_id=publisher instance=DevPC scope=dapr.runtime type=log ver=1.13.2
time="2024-05-14T14:30:57.6150179+05:30" level=info msg="Waiting for all outstanding components to be processed…" app_id=publisher instance=DevPC scope=dapr.runtime type=log ver=1.13.2
time="2024-05-14T14:30:57.6150179+05:30" level=info msg="rabbitmq pub/sub: connectionCount: current=0 reference=0" app_id=publisher component="pubsub (pubsub.rabbitmq/v1)" instance=DevPC scope=dapr.contrib type=log ver=1.13.2
time="2024-05-14T14:31:00.9264882+05:30" level=error msg="Failed to init component pubsub (pubsub.rabbitmq/v1): [INIT_COMPONENT_FAILURE]: initialization error occurred for pubsub (pubsub.rabbitmq/v1): Exception (501) Reason: \"EOF\"" app_id=publisher instance=DevPC scope=dapr.runtime.processor type=log ver=1.13.2
time="2024-05-14T14:31:00.9270042+05:30" level=warning msg="Error processing component, daprd will exit gracefully" app_id=publisher instance=DevPC scope=dapr.runtime.processor type=log ver=1.13.2
time="2024-05-14T14:31:00.9270042+05:30" level=info msg="All outstanding components processed" app_id=publisher instance=DevPC scope=dapr.runtime type=log ver=1.13.2
time="2024-05-14T14:31:00.9270042+05:30" level=info msg="Loading endpoints…" app_id=publisher instance=DevPC scope=dapr.runtime type=log ver=1.13.2
time="2024-05-14T14:31:00.9275336+05:30" level=info msg="Waiting for all outstanding http endpoints to be processed…" app_id=publisher instance=DevPC scope=dapr.runtime type=log ver=1.13.2
time="2024-05-14T14:31:00.9275336+05:30" level=info msg="All outstanding http endpoints processed" app_id=publisher instance=DevPC scope=dapr.runtime type=log ver=1.13.2
time="2024-05-14T14:31:00.9280803+05:30" level=info msg="gRPC server listening on TCP address: :52808" app_id=publisher instance=DevPC scope=dapr.runtime.grpc.api type=log ver=1.13.2
time="2024-05-14T14:31:00.9280803+05:30" level=info msg="Enabled gRPC tracing middleware" app_id=publisher instance=DevPC scope=dapr.runtime.grpc.api type=log ver=1.13.2
time="2024-05-14T14:31:00.9280803+05:30" level=info msg="Enabled gRPC metrics middleware" app_id=publisher instance=DevPC scope=dapr.runtime.grpc.api type=log ver=1.13.2
time="2024-05-14T14:31:00.9280803+05:30" level=info msg="Registering workflow engine for gRPC endpoint: [::]:52808" app_id=publisher instance=DevPC scope=dapr.runtime.grpc.api type=log ver=1.13.2
time="2024-05-14T14:31:00.9280803+05:30" level=info msg="API gRPC server is running on port 52808" app_id=publisher instance=DevPC scope=dapr.runtime type=log ver=1.13.2
time="2024-05-14T14:31:00.9280803+05:30" level=info msg="Enabled max body size HTTP middleware with size 4 MB" app_id=publisher instance=DevPC scope=dapr.runtime.http type=log ver=1.13.2
time="2024-05-14T14:31:00.928608+05:30" level=info msg="Enabled tracing HTTP middleware" app_id=publisher instance=DevPC scope=dapr.runtime.http type=log ver=1.13.2
time="2024-05-14T14:31:00.928608+05:30" level=info msg="Enabled metrics HTTP middleware" app_id=publisher instance=DevPC scope=dapr.runtime.http type=log ver=1.13.2
time="2024-05-14T14:31:00.9291527+05:30" level=info msg="HTTP server listening on TCP address: :52809" app_id=publisher instance=DevPC scope=dapr.runtime.http type=log ver=1.13.2
time="2024-05-14T14:31:00.9291527+05:30" level=info msg="HTTP server is running on port 52809" app_id=publisher instance=DevPC scope=dapr.runtime type=log ver=1.13.2
time="2024-05-14T14:31:00.9291527+05:30" level=info msg="The request body size parameter is: 4" app_id=publisher instance=DevPC scope=dapr.runtime type=log ver=1.13.2
time="2024-05-14T14:31:00.9291527+05:30" level=info msg="gRPC server listening on TCP address: :52815" app_id=publisher instance=DevPC scope=dapr.runtime.grpc.internal type=log ver=1.13.2
time="2024-05-14T14:31:00.9291527+05:30" level=info msg="Enabled gRPC tracing middleware" app_id=publisher instance=DevPC scope=dapr.runtime.grpc.internal type=log ver=1.13.2
time="2024-05-14T14:31:00.9291527+05:30" level=info msg="Enabled gRPC metrics middleware" app_id=publisher instance=DevPC scope=dapr.runtime.grpc.internal type=log ver=1.13.2
time="2024-05-14T14:31:00.9291527+05:30" level=info msg="Internal gRPC server is running on port 52815" app_id=publisher instance=DevPC scope=dapr.runtime type=log ver=1.13.2
time="2024-05-14T14:31:00.9296734+05:30" level=info msg="application protocol: http. waiting on port 5162.  This will block until the app is listening on that port." app_id=publisher instance=DevPC scope=dapr.runtime type=log ver=1.13.2
time="2024-05-14T14:31:00.9296734+05:30" level=info msg="Dapr is shutting down" app_id=publisher instance=DevPC scope=dapr.runtime type=log ver=1.13.2
time="2024-05-14T14:31:00.9296734+05:30" level=info msg="Dapr runtime stopped" app_id=publisher instance=DevPC scope=dapr.runtime type=log ver=1.13.2
time="2024-05-14T14:31:00.9296734+05:30" level=info msg="Closing HTTP server [::]:52809…" app_id=publisher instance=DevPC scope=dapr.runtime.http type=log ver=1.13.2
time="2024-05-14T14:31:00.9296734+05:30" level=info msg="Shutting down workflow engine" app_id=publisher instance=DevPC scope=dapr.runtime type=log ver=1.13.2
time="2024-05-14T14:31:00.9296734+05:30" level=info msg="stopping mDNS server for app id: publisher" app_id=publisher component="nr (mdns/v1)" instance=DevPC scope=dapr.contrib type=log ver=1.13.2
time="2024-05-14T14:31:00.9329754+05:30" level=fatal msg="Fatal error from runtime: process component pubsub error: [INIT_COMPONENT_FAILURE]: initialization error occurred for pubsub (pubsub.rabbitmq/v1): [INIT_COMPONENT_FAILURE]: initialization error occurred for pubsub (pubsub.rabbitmq/v1): Exception (501) Reason: \"EOF\"" app_id=publisher instance=DevPC scope=dapr.runtime type=log ver=1.13.2
Could not update sidecar metadata for cliPID: PUT http://127.0.0.1:52809/v1.0/metadata/cliPID giving up after 5 attempt(s): Put "http://127.0.0.1:52809/v1.0/metadata/cliPID";: dial tcp 127.0.0.1:52809: connectex: No connection could be made because the target machine actively refused it.
You're up and running! Dapr logs will appear here.
 
 
terminated signal received: shutting down

Are there any known workaround?

Thanks

[davidfowl]

Looks like the same issue as #2484 (comment). There's no real workaround if the dapr component isn't resilient. I'm not sure if you can configure retries.

[DawnDevelop]

Sadly there is no way to configure any type of resiliency (in the init) for dapr components right now. There is the v1.14 release planning where the component resiliency is mentioned as "Not prioritized", so i am not getting my hopes up. @spicycoder i don't think there is much we can do besides making sure rabbitmq is already up and running before starting the AppHost.

I also tried your repo @davidfowl WaitForDependenciesAspire but the sidecar starts regardless of what you configure (works great for the projects / containers though).

In theory im guessing you could try to start the dapr sidecar with custom args inside a container and then set the WaitFor like: AddContainer().WithArgs().WaitFor(rabbitmq)? But this seems so verbose to configure for all projects.

[davidfowl]

Yea, there's no way to get a handle on the side car. I think if we exposed the sidecar resource it would be possible to use WaitFor