Uploaded image for project: 'MariaDB ColumnStore'
  1. MariaDB ColumnStore
  2. MCOL-3323

Pentaho plugin strange behavior when load data into table

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Confirmed (View Workflow)
    • Priority: Critical
    • Resolution: Unresolved
    • Affects Version/s: 1.2.3
    • Fix Version/s: Icebox
    • Component/s: Pentaho Kettle Adapter
    • Labels:
      None
    • Environment:
      Vmware environment 6.7
      all vm with Oracle Linux 7.6
      MariadDB columnstore 1.2.3 :
      3 PM
      2 UM
      1 Pentaho 8.2 with pentaho kettle adapter 1.2.3

      Description

      Hi,
      we've a lot job that load with plugin in mariadb columnstore, but one job in particular is creating problem.
      There are jobs that load about 500k records into the Columnstore tables without problems, This job instead fails to enter even 100 records (with limit a 10 records the job go in success).
      it seems that the plugin is not able to commit the inserted records, in fact after several minutes it goes into error.

      This is the table where I can load about 500 thousand records at a time.
      CREATE TABLE `c6_messaggio_in` (
      `PROG_MESSAGGIO_IN_D` BIGINT(20) UNSIGNED NULL DEFAULT NULL,
      `PROG_TIPO_MSG_ID` TINYINT(3) UNSIGNED NULL DEFAULT NULL,
      `PROG_STATO_ELABORAZIONE_ID` TINYINT(3) UNSIGNED NULL DEFAULT NULL,
      `MESSAGGIO` TEXT NULL DEFAULT NULL,
      `DATA_RICEZIONE` DATETIME NULL DEFAULT NULL,
      `IP_ADDRESS` VARCHAR(50) NULL DEFAULT NULL,
      `ID_MSG_APP` VARCHAR(50) NULL DEFAULT NULL,
      `MSG_OPER` TEXT NULL DEFAULT NULL,
      `COD_MITT` VARCHAR(1) NULL DEFAULT NULL,
      `USER_INSERT` SMALLINT(5) UNSIGNED NULL DEFAULT NULL,
      `DATE_INSERT` DATETIME NULL DEFAULT NULL,
      `USER_MODIFY` SMALLINT(5) UNSIGNED NULL DEFAULT NULL,
      `DATE_MODIFY` DATETIME NULL DEFAULT NULL,
      `USER_DELETE` SMALLINT(5) UNSIGNED NULL DEFAULT NULL,
      `DATE_DELETE` DATETIME NULL DEFAULT NULL,
      `GATEWAY` VARCHAR(1) NULL DEFAULT 'N',
      `CODEID` VARCHAR(50) NULL DEFAULT NULL,
      `ID_MSG_APP_REQ` VARCHAR(50) NULL DEFAULT NULL
      )
      COLLATE='utf8mb4_general_ci'
      ENGINE=Columnstore
      ;

      This is the table where I can load about 10 records at a time but with 100 record it crash.
      CREATE TABLE `c6_messaggio_out` (
      `PROG_MESSAGGIO_OUT_ID` BIGINT(20) UNSIGNED NULL DEFAULT NULL,
      `PROG_TIPO_MSG_ID` TINYINT(3) UNSIGNED NULL DEFAULT NULL,
      `MESSAGGIO` TEXT NULL DEFAULT NULL,
      `DATA_TRASMISSIONE` DATETIME NULL DEFAULT NULL,
      `ID_MSG_APP` VARCHAR(50) NULL DEFAULT NULL,
      `MSG_OPER` TEXT NULL DEFAULT NULL,
      `COD_MITT` VARCHAR(1) NULL DEFAULT NULL,
      `PROG_STATO_ELABORAZIONE_ID` TINYINT(3) UNSIGNED NULL DEFAULT NULL,
      `USER_INSERT` TINYINT(3) UNSIGNED NULL DEFAULT NULL,
      `DATE_INSERT` DATETIME NULL DEFAULT NULL,
      `USER_MODIFY` TINYINT(3) UNSIGNED NULL DEFAULT NULL,
      `DATE_MODIFY` DATETIME NULL DEFAULT NULL,
      `USER_DELETE` TINYINT(3) UNSIGNED NULL DEFAULT NULL,
      `DATE_DELETE` DATETIME NULL DEFAULT NULL,
      `CODEID` VARCHAR(50) NULL DEFAULT NULL,
      `GATEWAY` VARCHAR(1) NULL DEFAULT 'N'
      )
      COLLATE='utf8mb4_general_ci'
      ENGINE=Columnstore
      ;

      I suspect that clob (text) fields can create difficulties when they are loaded by the plugin, but at the same time the c6_messaggio_in table which also has clob fields has no problems.

      Sorry for my bad english.
      Thanks,
      Regards
      Nicola Battista

        Attachments

          Issue Links

            Activity

              People

              Assignee:
              Unassigned Unassigned
              Reporter:
              nbattista89 Nicola
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

                Dates

                Created:
                Updated:

                  Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.