Ability to add multiple nodes to sunspot.yml (e.g. when using SolrCloud)

[heaven]

Hello,

We have SolrCloud setup and a few physical servers, ones run shards and others — replicas for them. The problem is in that we can only add a single IP to the sunspot.yml and if we need to reboot that Solr node search becomes unavailable.

It would be really great to have an ability to specify an array of nodes. For SolrCloud there's no difference which node receives requests so both reads and updates could be sent to any. The benefit is in that we can specify IPs of different physical servers and in case if one of them fails (with a timeout) we can try another node and raise an error if none of specified nodes respond in time.

Best,
Alexander

[iRonin]

+1

[itsmechlark]

+1

[heaven]

New pull: #796

[stex]

+1

@heaven: Thanks for you work here regarding solrcloud!

Probably a stupid question, but how would a sunspot configuration currently have to look when using 1 shard with 4 nodes regarding the port sunspot uses? I haven't really found anything in the documentation.

From my understanding, I may simply point it to the master node which will then balance the requests, but I'm not sure whether it's actually working.
Thanks in advance!

[heaven]

Hey, here's our production example:

production:
  solr:
    hosts:
      - 'solr1.devops:8080/solr/crm-prod'
      - 'solr1.devops:8081/solr/crm-prod'
      - 'solr1.devops:8082/solr/crm-prod'
      - 'solr1.devops:8083/solr/crm-prod'
      - 'solr2.devops:8080/solr/crm-prod'
      - 'solr2.devops:8081/solr/crm-prod'
      - 'solr2.devops:8082/solr/crm-prod'
      - 'solr2.devops:8083/solr/crm-prod'
    read_timeout: 60
    open_timeout: 3
    log_level: WARNING

I didn't add this to README because this depends of RSolr patch I've also submitted (rsolr/rsolr#138) and it wasn't merged yet.

You can also specify master_hosts array the same way.

I will close this pull for now, here's the new one with resolved conflicts: #796

[stex]

Sorry to bother you (and for misusing this ticket), but is it currently (= without your PRs here and for rsolr) possible to use multiple nodes with sunspot at all?

From my understanding, it should be possible by specifying one of the nodes in the sunspot configuration. This node would then get all requests and distribute them across the other nodes - with the disadvantage that it may never be down (this is fixed by your PRs by supporting multiple solr endpoints)

I am currently pointing sunspot to the master node, but I am not sure that requests are actually distributed correctly automatically.

[heaven]

Sure, you can use Solr Cloud without this patch, and you're correct about the distribution – for Solr Cloud it doesn't matter to which node requests were sent.

This patch solves the problem of availability. for example you may have 4 nodes and just one specified in your config file. If you ever need to shud down this node, e.g. for maintenance, you will end up with some downtime. With this patch you can specify all your 4 nodes and reboot them one by one, without downtime. When one node stopped responding sunspot will send requests to another available. You will see an error only if all nodes failed.

The actual implementation is in the RSolr gem, this patch here only allows to configure Sunspot to use multiple addresses and pass them to RSolr.

Please note this will work if these nodes are all replicas, if you split your index on shards, saying you have 2 shards with one replica each – 4 nodes in total, then you need at least 2 nodes representing both shards to be running for Solr Cloud to work correctly.

[waterbeach]

I still don't konw how to add multiple nodes to sunspot.yml? have some document to reference? thanks