Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Submitting a 32-bit application to Spark #681

Open
LesiaCM opened this issue Feb 13, 2018 · 0 comments
Open

Submitting a 32-bit application to Spark #681

LesiaCM opened this issue Feb 13, 2018 · 0 comments

Comments

@LesiaCM
Copy link

LesiaCM commented Feb 13, 2018

I've asked this question on stackoverflow (https://stackoverflow.com/questions/48323609/submitting-a-32-bit-application-to-spark-via-mobius):

I would like to submit to Spark a 32-bit driver program. The driver program is the .NET Framework console app:

the app is rather simple and features a very simple testing code:

public static void Main(string[] args)
{

        var conf = new SparkConf(); 
        var sparkContext = new SparkContext(conf);

        var rdd = sparkContext.Parallelize(new List<string>() { "a", "b", "c", "d" });

        var response = rdd.Map(s => s).Collect();

}

After I properly submit this app to sparkclr (that is, to mobius which then passes it to Spark itself), I get the following exception:

System.Exception: JVM method execution failed: Static method collectAndServe failed for class 
org.apache.spark.api.python.PythonRDD when called with 1 parameters ([Index=1, 
Type=JvmObjectReference, Value=11], )
   at Microsoft.Spark.CSharp.Interop.Ipc.JvmBridge.CallJavaMethod(Boolean isStatic, Object 
classNameOrJvmObjectReference, String methodName, Object[] parameters)
   at Microsoft.Spark.CSharp.Interop.Ipc.JvmBridge.CallStaticJavaMethod(String className, String 
methodName, Object[] parameters)
   at Microsoft.Spark.CSharp.Proxy.Ipc.RDDIpcProxy.CollectAndServe()
   at Microsoft.Spark.CSharp.Core.RDD`1.Collect()

The exception goes away when I build the app in x64. Also, the exception is not there when the app is built in x86 but does nothing (i.e. has no

  var rdd = sparkContext.Parallelize(new List<string>() { "a", "b", "c", "d" });
  var response = rdd.Map(s => s).Collect();

lines).

Is there any workaround to submit 32bit app to Spark?
(In the original question I thought the issue could be with Java, but later found this was not the case).

With kind regards,
Lesia

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant