Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-9228

CONNECT: Can not load JSON files over 12MB (roughly)

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Closed (View Workflow)
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 10.1.9, 10.0, 10.1
    • Fix Version/s: 10.0.23, 10.1.10
    • Labels:
      None
    • Environment:
      Windows Server 2012 R2 (x64) with 16GB of memory

      Description

      CONNECT is not parsing JSON files over roughly 12MB. I receive the following error when I try to select from the table-

      _Error Code: 1296. Got error 174 'ParseString: Out of memory' from CONNECT 0.406 sec_

      I've tried every option available, and also tried increasing the amount of memory available. I am not sure if there is a way to increase the memory blocks available to 'Sarea' through either a patch or server variable?

      I have tried each option, including-

      _SET GLOBAL connect_work_size = 4199999999;
      SET GLOBAL connect_use_tempfile=YES;
      SET GLOBAL connect_json_grp_size=2147483647;_

      I tried to use multiple files using a wildcard (this is what I really need - I have 2GB of files, smallest being under 1MB and largest being over 100MB in size.) My use case is around parsing some operations data from a custom application and just pull some counts. CONNECT seemed like it was fit for this.

      My data is an array of objects. My table creation statement looks like the following (I tried everything, from the most simple autodetection to manual mapping, and also trying to force the row size to see if I could get it to allocate more memory - nothing works)-

      CREATE TABLE `raw_sr` (
      `userid` varchar(60) field_format='userid:*' DEFAULT NULL,
      `fromgoogle_message_id` varchar(45) field_format='fromgoogle_message_id:*' DEFAULT NULL,
      `fromswift_container_id` int field_format='fromswift_container_id:*' DEFAULT NULL,
      `fromswift_object_id` int field_format='fromswift_object_id:*' DEFAULT NULL
      )
      ENGINE=CONNECT DEFAULT CHARSET=latin1 `TABLE_TYPE`='JSON'
      MULTIPLE=1 MAPPED=1 HUGE=1 READONLY=1
      `FILE_NAME`='ALL/*_SR.json'
      option_list='Pretty=2,Jmode=1' lrecl=1024;

      It failed loading multiple, it also fails loading a single large file. None of the options make a difference. Is CONNECT JSON designed for this use case? Is there a way to get it to allocate more memory for JSON parsing? I have 16GB, MySQL usage is around ~2GB. I don't care if it uses all the memory, I just want the data to be exposed in the database engine. Please advise. Thanks you.

        Attachments

          Activity

            People

            Assignee:
            bertrandop Olivier Bertrand
            Reporter:
            nicholas.a.fries@seagate.com Nick Fries
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved:

                Git Integration