Databricks Error SparkLinker

[Mac-delValle]

Hi all,

I am new to databricks and spark/python and very excited to use Splink. I am trying to run Splink in databricks. I am getting an error 'NoneType' object has no attribute 'sparkContext' when trying to execute the command linker = SparkLinker(df, settings). I am using the default settings provided in the example repo. Any help is much appreciated. Thank you in advance!

Here are my cluster configurations:

• 10.4 LTS (includes Apache Spark 3.2.1, Scala 2.12)

When I did the pip install, I realized the version has been updated since the demo repo was last published so I specified version 3.9.3.

Here is the full error message provided by databricks:

AttributeError Traceback (most recent call last)
in
1 # from splink.spark.linker import SparkLinker
----> 2 linker = SparkLinker(df, settings)
3 deterministic_rules = [
4 "l.first_name = r.first_name and levenshtein(r.dob, l.dob) <= 1",
5 "l.surname = r.surname and levenshtein(r.dob, l.dob) <= 1",

/local_disk0/.ephemeral_nfs/envs/pythonEnv-xxx-xx-xxxxxx-xxxx/lib/python3.8/site-packages/splink/spark/linker.py in init(self, input_table_or_tables, settings_dict, break_lineage_method, set_up_basic_logging, input_table_aliases, spark, validate_settings, catalog, database, repartition_after_blocking, num_partitions_on_repartition)
182 self.in_databricks = "DATABRICKS_RUNTIME_VERSION" in os.environ
183 if self.in_databricks:
--> 184 enable_splink(spark)
185
186 self._set_default_break_lineage_method()

/local_disk0/.ephemeral_nfs/envs/pythonEnv-xxx-xx-xxxxxx-xxxx/lib/python3.8/site-packages/splink/databricks/enable_splink.py in enable_splink(spark)
13 None
14 """
---> 15 sc = spark.sparkContext

[RobinL]

Hiya,

The error is being created from:

splink/splink/databricks/enable_splink.py

Line 15 in 31d4d0e

sc = spark.sparkContext

which is run automatically for you since you're in a databricks environment, see here

The spark context is obtained from your input Spark dataframes using this function

Specifically:
self.spark = t.sql_ctx.sparkSession

So, things for you to check:

What happens if you run df.sql_ctx.sparkSession. Do you get a SparkSession back. If not , what do you get

Another option is to pass Spark explicitly into the SparkLinker see here, which I think should avoid the problem.

I'm afraid I don't have access to databricks so I'm unable to check this easily.

[sthamodh]

Hi @RobinL and @beedrumms,

I can confirm a couple of things from above:

1. I have used 12.2 LTS when testing the splink solution on a compute cluster with a service principal and it worked without any errors
2. When I tried to use 13.3 LTS, it yielded me errors
3. Specifically, blacklist errors were raised when the splink libary tried to create a spark context with the scalaudfs from splink

import splink
from splink.spark.jar_location import similarity_jar_location
conf = SparkConf()
conf.set("spark.driver.memory", "32g")
conf.set("spark.default.parallelism", "32")
path = similarity_jar_location()
conf.set("spark.jars", path)

sc = SparkContext.getOrCreate(conf=conf)

When our organization spoke to databricks about this, we were able to confirm that the issue here is that Scala (and other lower level libraries) run directly on the JVM and isolation between a given user's data and another user's data segment in memory cannot be guaranteed. This is because the notebook that houses the splink code is running on a compute cluster that is shared between multiple users or has passthrough enabled.

Two of the following options worked for me:

1. Use a no-isolation shared cluster with a service principal when such a cluster is meant to serve multiple users
2. Use a single user cluster i.e. each user gets their own personal cluster with 12.2 LTS

As for your suggestion @RobinL, I believe one can install jars from Maven according to this: https://learn.microsoft.com/en-us/azure/databricks/libraries/package-repositories

Hope this helps!

[jericksonclinicaloptions]

My understanding is you must use runtime 12.2 LTS (I use the ML variant).
To install the jar file in Databricks, I downloaded it locally and then installed to DBFS on the cluster:

Also, you can set the spark configuration under Advance options on the cluster since Databricks automatically sets up the spark context when the cluster starts:

[beedrumms]

@RobinL apologies for the delay and @sthamodh thanks for the response! Unfortunately, manual registration of the jar did not work, but there is small progress:

the 'enable_splink' module used to fail when it uses java method to write to the following location:
file:/local_disk0/.ephemeral_nfs/cluster_libraries/python/lib/python3.10/site-packages/splink/files/spark_jars/scala-udf-similarity-0.1.1_spark3.x.jar

Without the jar manually registered, the module would break at JarURI = JavaURI.create("file:" + _jar_path)

Now with the scala-udf jar manually registered on cluster via dbfs, new error point is just below, at:

JavaJarId(JarURI, ManagedLibraryId.defaultOrganization(), NoVersionModule.simpleString() )

The new error is below:

This was all tested between a 12.2 RT and a 14.0 RT (as 13.3 is depreciated). Parallelism and partition configurations have no baring on the outcome. I have been discussing options w/ our data eng team w/ respect to writing the jar to the unity catalog or azure blob storage.

[RobinL]

@beedrumms. thanks very much for the additional info. Apologies if you'd already seen it, but we recently had an external contribution (not yet released, but on master) which fixes this for some users. I wondered if that helps at all for you?

See here: #1973

[beedrumms]

@RobinL thanks so much for this -- this code looks like it would have worked perfectly to address the RT issue -- However, we did solve the RT issue by:

1. uploading splink as a .whl file to a shared databricks workspace
2. then installing the library on our cluster by referencing the workspace location,

This was tested on a runtime of 14.2 and worked perfectly. I discovered this while coming into compliance w/ databricks depreciation of init scripts in dbfs so the azure docs to use a workspace for library was helpful in implementing this solution. I will note that this does not solve the shared access cluster compatibility issue -- this seems to be a whole other can of worms, but for now, its working great for us! thanks so much