Uploaded image for project: 'MariaDB ColumnStore'
  1. MariaDB ColumnStore
  2. MCOL-4694

Distribute primitive jobs messages b/w TCP connections provided by DEC

Details

    • Task
    • Status: Closed (View Workflow)
    • Major
    • Resolution: Fixed
    • 5.5.2
    • 6.1.1
    • None
    • None
    • 2021-7, 2021-8, 2021-9

    Description

      As of now the meta primitive messages:

      • BATCH_PRIMITIVE_DESTROY
      • BATCH_PRIMITIVE_ADD_JOINER
      • BATCH_PRIMITIVE_END_JOINER
      • BATCH_PRIMITIVE_ABORT
      • DICT_CREATE_EQUALITY_FILTER
      • DICT_DESTROY_EQUALITY_FILTER
        are always sent of over the first DEC connections to PMs. This fact results in a contention given that BATCH_PRIMITIVE_ADD_JOINER streams a lot of small side table data with it.
        Low level scanning/filtering primitives:
      • BATCH_PRIMITIVE_RUN
      • DICT_TOKEN_BY_SCAN_COMPARE
        Are effectively distributed b/w first min(N, M) connections where N is a number of connections to every PP and M is a number of primitive jobs sent(effectively produced by TupleBPS::makeJobs)

      MCS needs to distribute workload b/w connections using round-robin algorithm.
      Future investigation might point to a finer-grained distribution algo.

      Attachments

        Issue Links

          Activity

            drrtuy Roman added a comment -

            Waiting for MCOL-563 to be merged into develop.

            drrtuy Roman added a comment - Waiting for MCOL-563 to be merged into develop.
            drrtuy Roman added a comment -

            4QA there is no way to properly test the distribution now however MCS must work with non-default value for PrimitiveServers.ConnectionsPerPrimProc in Columnstore.xml.

            drrtuy Roman added a comment - 4QA there is no way to properly test the distribution now however MCS must work with non-default value for PrimitiveServers.ConnectionsPerPrimProc in Columnstore.xml.

            Build verified: 6.1.1 ( Drone #2587 )

            Tested ConnectionsPerPrimProc with value set to 4. Ran regression test suite.

            dleeyh Daniel Lee (Inactive) added a comment - Build verified: 6.1.1 ( Drone #2587 ) Tested ConnectionsPerPrimProc with value set to 4. Ran regression test suite.

            People

              dleeyh Daniel Lee (Inactive)
              drrtuy Roman
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.