Uploaded image for project: 'MariaDB Connector/J'
  1. MariaDB Connector/J
  2. CONJ-421

Spark error: java.sql.SQLException: Out of range value for column 'i' : value i is not in Integer range

    Details

    • Type: Task
    • Status: Open (View Workflow)
    • Priority: Minor
    • Resolution: Unresolved
    • Affects Version/s: 1.5.7
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      Install Spark 2.0 using docker for simplicity following this image:
      https://hub.docker.com/r/singularities/spark/

      copy the sample docker-compose.yml locally and run:
      docker-compose up

      After a while the master and worker will both start and logging stops. Now:

      docker exec -it sparkdocker_master_1 bash
      curl -O https://downloads.mariadb.com/Connectors/java/connector-java-1.5.7/mariadb-java-client-1.5.7.jar
      pyspark --driver-class-path mariadb-java-client-1.5.7.jar --jars  mariadb-java-client-1.5.7.jar
       
      from pyspark.sql import DataFrameReader
      url = 'jdbc:mariadb://172.21.21.2:3306/test?useServerPrepStmts=false'
      properties = {'user': 'root', 'driver': 'org.mariadb.jdbc.Driver', 'useServerPrepStmts':'false'}
      df = DataFrameReader(sqlContext).jdbc(url='%s' % url, table='tmp1', properties=properties)
      df.show()
      

      the df.show will result in a stack trace with the error:

      Caused by: java.sql.SQLException: Out of range value for column 'i' : value i is not in Integer range
              at org.mariadb.jdbc.internal.queryresults.resultset.MariaSelectResultSet.parseInt(MariaSelectResultSet.java:3233)
              at org.mariadb.jdbc.internal.queryresults.resultset.MariaSelectResultSet.getInt(MariaSelectResultSet.java:992)
              at org.mariadb.jdbc.internal.queryresults.resultset.MariaSelectResultSet.getInt(MariaSelectResultSet.java:969)
              at org.apache.spark.sql.execution.datasources.jdbc.JDBCRDD$$anon$1.getNext(JDBCRDD.scala:446)
              at org.apache.spark.sql.execution.datasources.jdbc.JDBCRDD$$anon$1.hasNext(JDBCRDD.scala:544)
              at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIterator.processNext(Unknown Source)
              at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
              at org.apache.spark.sql.execution.WholeStageCodegenExec$$anonfun$8$$anon$1.hasNext(WholeStageCodegenExec.scala:370)
              at org.apache.spark.sql.execution.SparkPlan$$anonfun$4.apply(SparkPlan.scala:246)
              at org.apache.spark.sql.execution.SparkPlan$$anonfun$4.apply(SparkPlan.scala:240)
              at org.apache.spark.rdd.RDD$$anonfun$mapPartitionsInternal$1$$anonfun$apply$24.apply(RDD.scala:803)
              at org.apache.spark.rdd.RDD$$anonfun$mapPartitionsInternal$1$$anonfun$apply$24.apply(RDD.scala:803)
              at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
              at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:319)
              at org.apache.spark.rdd.RDD.iterator(RDD.scala:283)
              at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:70)
              at org.apache.spark.scheduler.Task.run(Task.scala:86)
              at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:274)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
              ... 1 more
      

      If instead i use the mysql connector (mysql-connector-java-5.1.40.tar.gz) it works.

      This docker image use java 8, however i modified this to use Java 7 and rebuilt and the error still happens so not a java 8 specific issue.

      Table definition is very simple:
      create table tmp1 (i int, ip int);
      insert into tmp1 values (1,1);

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                diego dupin Diego Dupin
                Reporter:
                dthompson David Thompson (Inactive)
              • Votes:
                1 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated: