cds: update picker synchronously upon receipt of configuration update

[easwars]

#5469 recommends an audit of existing LB policies to ensure that they update their pickers synchronously upon receipt of a configuration update.

cds_experimental does not update its picker synchronously.

• Configuration updates are handled here:

  grpc-go/xds/internal/balancer/cdsbalancer/cdsbalancer.go

  Line 281 in b4f7947

  func (b *cdsBalancer) UpdateClientConnState(state balancer.ClientConnState) error {

• In the above function, the LB policy
  • initially checks for some error conditions
  • returns early if the cluster name in the configuration has not changed (the cluster name is the only field in its configuration currently)
  • if the cluster name has changed
    • schedules a serializer callback in a blocking fashion to create a new watcher for the cluster name:

      grpc-go/xds/internal/balancer/cdsbalancer/cdsbalancer.go

      Line 313 in b4f7947

      ok = b.serializer.Schedule(func(context.Context) {

      • if the new cluster turns out to be an aggregate cluster, more watches will be registered
• Anyways, at the end of the UpdateClientConnState , it does not return a picker inline
  • It does not even push the child policy configuration inline, because that has to wait for the cluster graph to be resolved completely

This needs fixing in at least a few places, and we need answers for these:

• Should the cds LB policy return a picker at build time? It currently does not.
• Should the cds LB policy return an erroring picker when cluster graph resolution is incomplete? It currently does not.