[MDEV-27106] Spider: specify connection to data node by engine-defined attributes Created: 2020-05-13  Updated: 2023-10-27  Resolved: 2022-01-24

Status: Closed
Project: MariaDB Server
Component/s: Storage Engine - Spider
Fix Version/s: 10.8.1

Type: Task Priority: Blocker
Reporter: Geoff Montee (Inactive) Assignee: Nayuta Yanagisawa (Inactive)
Resolution: Done Votes: 0
Labels: Preview_10.8

Issue Links:
PartOf
is part of MDEV-27373 Q1 2022 release merge Closed
Problem/Incident
causes MDEV-27521 SIGSEGV in spider_parse_connect_info ... Closed
causes MDEV-27676 Assertion `str.alloced_length() >= st... Closed
Relates
relates to MDEV-27169 Change default values of Spider plugi... Closed
relates to MDEV-27211 Spider: deprecate table option specif... Closed
relates to MDEV-28856 Spider: Implement more engine-defined... Closed
relates to MDEV-28861 Spider: Deprecate table options by CO... Closed
relates to MDEV-33388 Dedicated table options for Spider+ODBC Open
relates to MDEV-5271 Support engine-defined attributes per... Closed
relates to MDEV-22541 CREATE SERVER does not treat SOCKET a... Open
relates to MDEV-27400 Spider attempts localhost TCP/IP conn... Confirmed
relates to MDEV-27564 Clearly specify/indicate in the parti... Confirmed
relates to MDEV-31146 Spider: Remove table options by COMMENT Open

 Description   

The Spider storage engine provides two ways to specify remote data nodes. One is by the COMMENT clause and the other is by the CONNECTION clause (see below links for existing options). However, writing every specification in a single connection string is not very user-friendly.

We introduce new engine-defined attributes to specify remote data nodes. The engine attributes do not cover all the existing DSN parameters because most of the parameters need not be specified at the table level. So, we first introduce engine-defined attributes that correspond to essential parameters. If users complain that a parameter they need is missing, we can introduce it later.

I believe that the following attributes are enough:

engine-defined attribute corresponding DNS parameter
REMOTE_SERVER srv
REMOTE_DATABASE database
REMOTE_TABLE tbl

Writing a raw password in the COMMENT or engine-defined option is not preferable from the viewpoint of security because the password might be logged in slow log or general log. Therefore, we should encourage the use of the CREATE SERVER statement, and not introduce attributes such as REMOTE_USER and REMOTE_PASSWORD.

SSL-related parameters might be essential but I believe that they should be covered by CREATE SERVER statement rather than per-table or per-partition attributes.

If, for example, both COMMENT='table "t1"' and REMOTE_TABLE="t2" are specified for a single table or a single partition, the Spider returns an error.

Example:

CREATE TABLE `users` (
    `id` INT,
    `name` VARCHAR(255)
) ENGINE=SPIDER REMOTE_TABLE="users"
PARTITION BY HASH(id)
(
    PARTITION `pt1` REMOTE_SERVER="s1" REMOTE_DATABASE="db1",
    PARTITION `pt2` REMOTE_SERVER="s1" REMOTE_DATABASE="db2",
    PARTITION `pt3` REMOTE_SERVER="s2" REMOTE_DATABASE="db1"
);

TODO:

  • Introduce engine-defined attributes described above;
  • Make Spider to use the attributes;
  • Write comprehensive test cases;
  • Refactor entire patch.


 Comments   
Comment by Nayuta Yanagisawa (Inactive) [ 2021-11-30 ]

Please review. The patch depends on one for MDEV-5271.
https://github.com/MariaDB/server/commit/ae9e72ca06d95947e44ef0801c94fbac6882b38e

Comment by Alexey Botchkov [ 2021-12-10 ]

see minor comments to the patch.

Comment by Nayuta Yanagisawa (Inactive) [ 2021-12-10 ]

holyfoot Thank you for your review. I fixed the patch according to your comments. Please check it again. https://github.com/MariaDB/server/commit/384f9d34ff29fe81de234c48516ccf257b9b10f5

Comment by Alexey Botchkov [ 2021-12-13 ]

Ok to push.

Comment by Roel Van de Paar [ 2022-01-01 ]

Logged MDEV-27400 Spider attempts localhost TCP/IP connection, even when none is specified.

Comment by Roel Van de Paar [ 2022-01-16 ]

Logged MDEV-27521 SIGSEGV in spider_parse_connect_info in MDEV-27106 branch

Comment by Roel Van de Paar [ 2022-01-22 ]

Confirming this comment, it is interesting to note that Spider already supported MDEV-5271-like functionality (i.e. per partition engine-defined attributes) using the comment field. iow, in the comment field one could specify per-partition options even before MDEV-5271 was implemented.

Comment by Roel Van de Paar [ 2022-01-22 ]

As the COMMENT field functionality was not changed in any way, and as any new SQL syntax is 10.8-only, I did not do any upgrade testing for this issue. nayuta-yanagisawa and serg If you think any should be done, please let me know your thoughts.

Comment by Roel Van de Paar [ 2022-01-22 ]

OK to push. However, I am still running a generic crash test against the feature branch (including the MDEV-27521 patch). Results from that will be available around next week Wednesday.

Comment by Roel Van de Paar [ 2022-01-22 ]

From the same run, logged MDEV-27575 SIGSEGV in intern_plugin_lock on SHUTDOWN when setting Spider as default storage engine (temporary or global)

Comment by Roel Van de Paar [ 2022-01-29 ]

Logged MDEV-27676 Assertion `str.alloced_length() >= str.length() + data_len' failed in spider_string::q_append

Comment by Yuchen Pei [ 2023-04-26 ]

Hi greenman, how much work do you think is needed to document the nice feature introduced in this ticket in the KB?

Comment by Yuchen Pei [ 2023-06-28 ]

> Hi Ian Gilfillan, how much work do you think is needed to document the nice feature introduced in this ticket in the KB?

I added to the kb two weeks ago: https://mariadb.com/kb/en/spider-storage-engine-overview/#basic-usage

Generated at Thu Feb 08 09:50:22 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.