Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feature request: Handle immutable solr schema with flag #299

Open
vrindavda opened this issue May 26, 2020 · 0 comments
Open

Feature request: Handle immutable solr schema with flag #299

vrindavda opened this issue May 26, 2020 · 0 comments

Comments

@vrindavda
Copy link

Using spark-solr to write CSV data into solr collection.

Sometimes this CSV might consists of additional fields that they do not want to index to solr collection. Hence we made the solr schema as immutable:
<schemaFactory class="ManagedIndexSchemaFactory"> ​<bool name="mutable">false</bool> ​<str name="managedSchemaResourceName">managed-schema</str> ​</schemaFactory>
This work alright until we have all the columns in CSV defined in Solr(manually). The moment we have some unexpected fields in CSV. Solr-spark tries to create them and the job fails as there is no check if the schema is immutable. As I understand CSV data( all columns) will be loaded as is with
scala> var csvDF= spark.read.format("com.databricks.spark.csv").option("header", "true").load(csvFileLocation)

Feature request: config flag in Solr-spark that says collection schema is immutable.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant