Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-12477

MariaDB can't connect 100MB JSON file

    XMLWordPrintable

Details

    Description

      When trying to load a JSON file produce by mongodump in the format where each row is a separate JSON document as opposed to a single array I hit an out of memory error.

      I am using the following syntax;

      CREATE TABLE test
      ENGINE=CONNECT
      table_type=JSON
      file_name='test.json'
      lrecl=128
      option_list='level=2, PRETTY=0'
      catfunc= COLUMNS;

      I have experimented with lrecl sizes of up to 4096 but it made no difference and eventually crashed the database engine. Since crashing the database engine I now get a different error that I can't get rid of even with smaller files now.

      Below is some configuration detail and the stack trace from the error logs;

      Server version: 10.1.21-MariaDB
      key_buffer_size=134217728
      read_buffer_size=131072
      max_used_connections=1
      max_threads=1001
      thread_count=1

      ha_connect.dll!_setjmp()[setjmp.asm:95]
      ha_connect.dll!ParseJson()[json.cpp:62]
      ha_connect.dll!TDBJSN::ReadDB()[tabjson.cpp:745]
      ha_connect.dll!JSONColumns()[tabjson.cpp:180]
      ha_connect.dll!TDBCAT::Initialize()[table.cpp:436]
      ha_connect.dll!TDBCAT::OpenDB()[table.cpp:493]
      ha_connect.dll!CntOpenTable()[connect.cc:360]
      ha_connect.dll!ha_connect::OpenTable()[ha_connect.cc:1877]
      ha_connect.dll!ha_connect::rnd_init()[ha_connect.cc:3761]
      mysqld.exe!handler::ha_rnd_init_with_error()[handler.cc:2754]
      mysqld.exe!init_read_record()[records.cc:281]
      mysqld.exe!join_init_read_record()[sql_select.cc:19222]
      mysqld.exe!sub_select()[sql_select.cc:18309]
      mysqld.exe!do_select()[sql_select.cc:17967]
      mysqld.exe!JOIN::exec_inner()[sql_select.cc:3220]
      mysqld.exe!JOIN::exec()[sql_select.cc:2511]
      mysqld.exe!mysql_select()[sql_select.cc:3446]
      mysqld.exe!handle_select()[sql_select.cc:384]
      mysqld.exe!execute_sqlcom_select()[sql_parse.cc:5896]
      mysqld.exe!mysql_execute_command()[sql_parse.cc:2971]
      mysqld.exe!mysql_parse()[sql_parse.cc:7324]
      mysqld.exe!dispatch_command()[sql_parse.cc:1491]
      mysqld.exe!do_command()[sql_parse.cc:1109]
      mysqld.exe!threadpool_process_request()[threadpool_common.cc:271]
      mysqld.exe!io_completion_callback()[threadpool_win.cc:568]
      KERNEL32.DLL!TermsrvOpenRegEntry()
      ntdll.dll!RtlReleasePebLock()
      ntdll.dll!RtlReleaseSRWLockExclusive()
      KERNEL32.DLL!BaseThreadInitThunk()
      ntdll.dll!RtlUserThreadStart()

      Attachments

        Activity

          People

            bertrandop Olivier Bertrand
            SimonTulett Simon Tulett
            Votes:
            1 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:

              Git Integration

                Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.