Logistic Regression with LBFGS in Spark 1.6 and 2.1

[colbec]

@jadianes Nice tutorial on Logistic Regression, thankyou.
I ran the tutorial on Spark 1.6.2 and 2.1.0 - both ran fine and I could repeat your results perfectly in 1.6.2, but I would like to offer the following observation re 2.1.0. In 2.1.0 the process takes about 3 times longer to run and produces a different answer than that produced by 1.6.2. I thought this was strange and found that in the list of Spark tasks 2.1.0 was calling a non-LBFGS algorithm. I raised this issue in a JIRA question (https://issues.apache.org/jira/browse/SPARK-16768). It seems that even though a user can import the LBFGS version into pyspark and you can call help on it and actually call it, I don't think it is actually an LBFGS version.
http://spark.apache.org/docs/latest/mllib-optimization.html has some other information on LBFGS in Spark.
Later when 2.1.0 becomes the standard your readers may find that they don't get your results for accuracy. Or maybe I just missed something, can anyone confirm my observations?

[jadianes]

Hi @colbec,

Sorry about the late reply, and thanks for having a look at the repos. I'm afraid I didn't have the time to update the repos and user a more recent version of Spark, so I can't reproduce your results. They look ver interesting indeed!

Hopefully somebody else will be able to do it and will let us know :)

But please, keep us up to date with your findings.

[colbec]

Hi @jadianes

No problem, Sir - unfortunately I have not been able to show any merit in the results I get with the Spark community. My post with Apache JIRA mentioned above has been dismissed as "not a problem." The result is that I for one will be using 1.6.2 for Logistic Regression since (for me, anyway) it is 3 times faster than 2.1.0 and produces a more attractive result.

[colbec]

For those following along, I have found some discussions related to mllib (http://apache-spark-developers-list.1001551.n3.nabble.com/Switch-RDD-based-MLlib-APIs-to-maintenance-mode-in-Spark-2-0-td17033.html) which indicate that from the Spark 2.x onwards, there is to be a preference for the newer "ml" library over the "mllib" which will be going to maintenance only mode. With much legacy tutorial support out in the wild using mllib there will inevitably be a period of transition where we just have to be a bit careful.

[Swarup17]

interactions_df = sqlContext.createDataFrame(row_data)

when i run the above in pyspark shell , I get the below error.Not able to figure out the issue here.
Please advise.

---

Caused by: ERROR XJ040: Failed to start database 'metastore_db' with class loade
r org.apache.spark.sql.hive.client.IsolatedClientLoader$$anon$1@2734d50, see the
next exception for details.
at org.apache.derby.iapi.error.StandardException.newException(Unknown So
urce)
at org.apache.derby.impl.jdbc.SQLExceptionFactory.wrapArgsForTransportAc
rossDRDA(Unknown Source)
... 97 more
Caused by: ERROR XSDB6: Another instance of Derby may have already booted the da
tabase C:\spark\metastore_db.
at org.apache.derby.iapi.error.StandardException.newException(Unknown So
urce)
at org.apache.derby.iapi.error.StandardException.newException(Unknown So
urce)
at org.apache.derby.impl.store.raw.data.BaseDataFileFactory.privGetJBMSL
ockOnDB(Unknown Source)
at org.apache.derby.impl.store.raw.data.BaseDataFileFactory.run(Unknown
Source)
at java.security.AccessController.doPrivileged(Native Method)
at org.apache.derby.impl.store.raw.data.BaseDataFileFactory.getJBMSLockO
nDB(Unknown Source)
at org.apache.derby.impl.store.raw.data.BaseDataFileFactory.boot(Unknown
Source)
at org.apache.derby.impl.services.monitor.BaseMonitor.boot(Unknown Sourc
e)
at org.apache.derby.impl.services.monitor.TopService.bootModule(Unknown
Source)
at org.apache.derby.impl.services.monitor.BaseMonitor.startModule(Unknow
n Source)
at org.apache.derby.impl.services.monitor.FileMonitor.startModule(Unknow
n Source)
at org.apache.derby.iapi.services.monitor.Monitor.bootServiceModule(Unkn
own Source)
at org.apache.derby.impl.store.raw.RawStore.boot(Unknown Source)
at org.apache.derby.impl.services.monitor.BaseMonitor.boot(Unknown Sourc
e)
at org.apache.derby.impl.services.monitor.TopService.bootModule(Unknown
Source)
at org.apache.derby.impl.services.monitor.BaseMonitor.startModule(Unknow
n Source)
at org.apache.derby.impl.services.monitor.FileMonitor.startModule(Unknow
n Source)
at org.apache.derby.iapi.services.monitor.Monitor.bootServiceModule(Unkn
own Source)
at org.apache.derby.impl.store.access.RAMAccessManager.boot(Unknown Sour
ce)
at org.apache.derby.impl.services.monitor.BaseMonitor.boot(Unknown Sourc
e)
at org.apache.derby.impl.services.monitor.TopService.bootModule(Unknown
Source)
at org.apache.derby.impl.services.monitor.BaseMonitor.startModule(Unknow
n Source)
at org.apache.derby.impl.services.monitor.FileMonitor.startModule(Unknow
n Source)
at org.apache.derby.iapi.services.monitor.Monitor.bootServiceModule(Unkn
own Source)
at org.apache.derby.impl.db.BasicDatabase.bootStore(Unknown Source)
at org.apache.derby.impl.db.BasicDatabase.boot(Unknown Source)
at org.apache.derby.impl.services.monitor.BaseMonitor.boot(Unknown Sourc
e)
at org.apache.derby.impl.services.monitor.TopService.bootModule(Unknown
Source)
at org.apache.derby.impl.services.monitor.BaseMonitor.bootService(Unknow
n Source)
at org.apache.derby.impl.services.monitor.BaseMonitor.startProviderServi
ce(Unknown Source)
at org.apache.derby.impl.services.monitor.BaseMonitor.findProviderAndSta
rtService(Unknown Source)
at org.apache.derby.impl.services.monitor.BaseMonitor.startPersistentSer
vice(Unknown Source)
at org.apache.derby.iapi.services.monitor.Monitor.startPersistentService
(Unknown Source)
... 94 more