[MDEV-377] Name support for dynamic columns - Jira

Oleksandr Byelkin created issue - 2012-07-02 10:39

Oleksandr Byelkin made changes - 2012-07-02 10:39

Field	Original Value	New Value
Assignee		Oleksandr Byelkin [ sanja ]

Oleksandr Byelkin made changes - 2012-07-02 10:39

Status

Open [ 1 ]

In Progress [ 3 ]

Oleksandr Byelkin made changes - 2012-07-02 12:26

Status

In Progress [ 3 ]

Open [ 1 ]

Oleksandr Byelkin made changes - 2012-07-02 22:33

Description

Dynamic columns should be referred by names (now only numeric referrences supported).

Dynamic columns should be referred by names (now only numeric referrences supported).

Numeric (current) format of dynamic columns string is:

-- fixed part
1 byte flags (now stores <offset size> - 1 in first 3 bits)
2 bytes column counter
-- fixed size column headers sorted by column number, each entry contains of:
  2 bytes - column number
  1-4 bytes combined offset from begning of the data segnebt + 3 bit type
-- data of above columns size of data and length depend on type

For columns with names proposed following format. I think in this format numeric references could be written as strings.

-- fixed part
1 byte flags (now stores <offset size> - 1 in first 3 bits, and 1 bit to set
this new format)
2 bytes column counter
2 bytes size of stored names pool
-- fixed size column headers sorted by names, each consists of
  2 bytes - column number
  1 byte length of the name
  2 bytes offset of name
-- names stored one after another
-- data of above columns size of data and length depend on type

there was other ideas which I think is not so good:
1) store names just in header part (then it looses its array properties and binary search)
2) store both headers numbers and names (I doubts that both way of references could be used simultaneously).

Sergei Petrunia made changes - 2012-07-04 19:44

Description

Dynamic columns should be referred by names (now only numeric referrences supported).

Numeric (current) format of dynamic columns string is:

-- fixed part
1 byte flags (now stores <offset size> - 1 in first 3 bits)
2 bytes column counter
-- fixed size column headers sorted by column number, each entry contains of:
  2 bytes - column number
  1-4 bytes combined offset from begning of the data segnebt + 3 bit type
-- data of above columns size of data and length depend on type

For columns with names proposed following format. I think in this format numeric references could be written as strings.

-- fixed part
1 byte flags (now stores <offset size> - 1 in first 3 bits, and 1 bit to set
this new format)
2 bytes column counter
2 bytes size of stored names pool
-- fixed size column headers sorted by names, each consists of
  2 bytes - column number
  1 byte length of the name
  2 bytes offset of name
-- names stored one after another
-- data of above columns size of data and length depend on type

there was other ideas which I think is not so good:
1) store names just in header part (then it looses its array properties and binary search)
2) store both headers numbers and names (I doubts that both way of references could be used simultaneously).

Dynamic columns should be referred by names (now only numeric referrences supported).

Numeric (current) format of dynamic columns string is:

-- fixed part
1 byte flags (now stores <offset size> - 1 in first 3 bits)
2 bytes column counter
-- fixed size column headers sorted by column number, each entry contains of:
  2 bytes - column number
  1-4 bytes combined offset from begning of the data segnebt + 3 bit type
-- data of above columns size of data and length depend on type

For columns with names proposed following format. I think in this format numeric references could be written as strings.

-- fixed part
1 byte flags (now stores <offset size> - 1 in first 3 bits, and 1 bit to set
this new format)
2 bytes column counter
2 bytes size of stored names pool
-- fixed size column headers sorted by names, each consists of
  2 bytes - column number
  1 byte length of the name
  2 bytes offset of name
-- names stored one after another
-- data of above columns size of data and length depend on type

there was other ideas which I think is not so good:
1) store names just in header part (then it looses its array properties and binary search)
2) store both headers numbers and names (I doubts that both way of references could be used simultaneously). (We have agreed this is not possible, because name<->number mapping must be the same for every record in the table, and there is no way we could efficiently maintain this mapping. For example, HBase does not provide any fast way to get a full list of column names).

Sergei Petrunia made changes - 2012-07-11 14:09

Link

This issue relates to ~~MDEV-122~~ [ ~~MDEV-122~~ ]

Oleksandr Byelkin made changes - 2012-07-27 11:26

Description

Dynamic columns should be referred by names (now only numeric referrences supported).

Numeric (current) format of dynamic columns string is:

-- fixed part
1 byte flags (now stores <offset size> - 1 in first 3 bits)
2 bytes column counter
-- fixed size column headers sorted by column number, each entry contains of:
  2 bytes - column number
  1-4 bytes combined offset from begning of the data segnebt + 3 bit type
-- data of above columns size of data and length depend on type

For columns with names proposed following format. I think in this format numeric references could be written as strings.

-- fixed part
1 byte flags (now stores <offset size> - 1 in first 3 bits, and 1 bit to set
this new format)
2 bytes column counter
2 bytes size of stored names pool
-- fixed size column headers sorted by names, each consists of
  2 bytes - column number
  1 byte length of the name
  2 bytes offset of name
-- names stored one after another
-- data of above columns size of data and length depend on type

there was other ideas which I think is not so good:
1) store names just in header part (then it looses its array properties and binary search)
2) store both headers numbers and names (I doubts that both way of references could be used simultaneously). (We have agreed this is not possible, because name<->number mapping must be the same for every record in the table, and there is no way we could efficiently maintain this mapping. For example, HBase does not provide any fast way to get a full list of column names).

Dynamic columns should be referred by names (now only numeric referrences supported).

Numeric (current) format of dynamic columns string is:

-- fixed part
1 byte flags (now stores <offset size> - 1 in first 3 bits)
2 bytes column counter
-- fixed size column headers sorted by column number, each entry contains of:
  2 bytes - column number
  1-4 bytes combined offset from begning of the data segnebt + 3 bit type
-- data of above columns size of data and length depend on type

For columns with names proposed following format. I think in this format numeric references could be written as strings.

-- fixed part
1 byte flags (now stores <offset size> - 1 in first 3 bits, and 1 bit to set
this new format)
2 bytes column counter
2 bytes size of stored names pool
-- fixed size column headers sorted by names, each consists of
  1 byte length of the name
  2 bytes offset of name
  1-4 bytes combined offset from begning of the data segnebt + 3 bit type
-- names stored one after another
-- data of above columns size of data and length depend on type

there was other ideas which I think is not so good:
1) store names just in header part (then it looses its array properties and binary search)
2) store both headers numbers and names (I doubts that both way of references could be used simultaneously). (We have agreed this is not possible, because name<->number mapping must be the same for every record in the table, and there is no way we could efficiently maintain this mapping. For example, HBase does not provide any fast way to get a full list of column names).

Oleksandr Byelkin made changes - 2012-08-13 09:53

Description

Dynamic columns should be referred by names (now only numeric referrences supported).

Numeric (current) format of dynamic columns string is:

-- fixed part
1 byte flags (now stores <offset size> - 1 in first 3 bits)
2 bytes column counter
-- fixed size column headers sorted by column number, each entry contains of:
  2 bytes - column number
  1-4 bytes combined offset from begning of the data segnebt + 3 bit type
-- data of above columns size of data and length depend on type

For columns with names proposed following format. I think in this format numeric references could be written as strings.

-- fixed part
1 byte flags (now stores <offset size> - 1 in first 3 bits, and 1 bit to set
this new format)
2 bytes column counter
2 bytes size of stored names pool
-- fixed size column headers sorted by names, each consists of
  1 byte length of the name
  2 bytes offset of name
  1-4 bytes combined offset from begning of the data segnebt + 3 bit type
-- names stored one after another
-- data of above columns size of data and length depend on type

there was other ideas which I think is not so good:
1) store names just in header part (then it looses its array properties and binary search)
2) store both headers numbers and names (I doubts that both way of references could be used simultaneously). (We have agreed this is not possible, because name<->number mapping must be the same for every record in the table, and there is no way we could efficiently maintain this mapping. For example, HBase does not provide any fast way to get a full list of column names).

Dynamic columns should be referred by names (now only numeric referrences supported).

Numeric (current) format of dynamic columns string is:

-- fixed part
1 byte flags (now stores <offset size> - 1 in first 3 bits)
2 bytes column counter
-- fixed size column headers sorted by column number, each entry contains of:
  2 bytes - column number
  1-4 bytes combined offset from begning of the data segnebt + 3 bit type
-- data of above columns size of data and length depend on type

For columns with names proposed following format. I think in this format numeric references could be written as strings.

-- fixed part
1 byte flags (now stores <offset size> - 1 in first 3 bits, and 1 bit to set
this new format)
2 bytes column counter
2 bytes size of stored names pool
-- fixed size column headers sorted by names, each consists of
  1 byte length of the name
  2 bytes offset of name
  1-4 bytes combined offset from begning of the data segnebt + 3 bit type
-- names stored one after another
-- data of above columns size of data and length depend on type

there was other ideas which I think is not so good:
1) store names just in header part (then it looses its array properties and binary search)
2) store both headers numbers and names (I doubts that both way of references could be used simultaneously). (We have agreed this is not possible, because name<->number mapping must be the same for every record in the table, and there is no way we could efficiently maintain this mapping. For example, HBase does not provide any fast way to get a full list of column names).

Limitations:
* each name not more then 255 symbols
* sum of all names length cant be more then 65535

New format used if
* at least one of dynamic column references has non integer type.
or
* dynamic column string on which operation performed has new format

Oleksandr Byelkin made changes - 2012-08-13 10:24

Description

Dynamic columns should be referred by names (now only numeric referrences supported).

Numeric (current) format of dynamic columns string is:

-- fixed part
1 byte flags (now stores <offset size> - 1 in first 3 bits)
2 bytes column counter
-- fixed size column headers sorted by column number, each entry contains of:
  2 bytes - column number
  1-4 bytes combined offset from begning of the data segnebt + 3 bit type
-- data of above columns size of data and length depend on type

For columns with names proposed following format. I think in this format numeric references could be written as strings.

-- fixed part
1 byte flags (now stores <offset size> - 1 in first 3 bits, and 1 bit to set
this new format)
2 bytes column counter
2 bytes size of stored names pool
-- fixed size column headers sorted by names, each consists of
  1 byte length of the name
  2 bytes offset of name
  1-4 bytes combined offset from begning of the data segnebt + 3 bit type
-- names stored one after another
-- data of above columns size of data and length depend on type

there was other ideas which I think is not so good:
1) store names just in header part (then it looses its array properties and binary search)
2) store both headers numbers and names (I doubts that both way of references could be used simultaneously). (We have agreed this is not possible, because name<->number mapping must be the same for every record in the table, and there is no way we could efficiently maintain this mapping. For example, HBase does not provide any fast way to get a full list of column names).

Limitations:
* each name not more then 255 symbols
* sum of all names length cant be more then 65535

New format used if
* at least one of dynamic column references has non integer type.
or
* dynamic column string on which operation performed has new format

Dynamic columns should be referred by names (now only numeric referrences supported).

Numeric (current) format of dynamic columns string is:

-- fixed part
1 byte flags (now stores <offset size> - 1 in first 3 bits)
2 bytes column counter
-- fixed size column headers sorted by column number, each entry contains of:
  2 bytes - column number
  1-4 bytes combined offset from begning of the data segnebt + 3 bit type
-- data of above columns size of data and length depend on type

For columns with names proposed following format. I think in this format numeric references could be written as strings.

-- fixed part
1 byte flags (now stores <offset size> - 1 in first 3 bits, and 1 bit to set
this new format)
2 bytes column counter
2 bytes size of stored names pool
-- fixed size column headers sorted by names, each consists of
  1 byte length of the name
  2 bytes offset of name
  1-4 bytes combined offset from begning of the data segnebt + 3 bit type
-- names stored one after another
-- data of above columns size of data and length depend on type

there was other ideas which I think is not so good:
1) store names just in header part (then it looses its array properties and binary search)
2) store both headers numbers and names (I doubts that both way of references could be used simultaneously). (We have agreed this is not possible, because name<->number mapping must be the same for every record in the table, and there is no way we could efficiently maintain this mapping. For example, HBase does not provide any fast way to get a full list of column names).

Limitations:
* each name not more then 255 symbols
* sum of all names length cant be more then 65535

New format used if
* at least one of dynamic column references has non integer type.
or
* dynamic column string on which operation performed has new format

New low level interface functions:

Parameters of creaing/changimg functions:
  DYNAMIC_COLUMN *str - the dynamic column.
  uchar *column_keys - array of uint or LEX_STRING keys depends on 'names'
  DYNAMIC_COLUMN_VALUE *values - array of volues
  uint column_count - number of above arrays
  names - TRUE if names are character strings

enum enum_dyncol_func_result
dynamic_column_create_many_fmt(DYNAMIC_COLUMN *str,
                               uint column_count,
                               uchar *column_keys,
                               DYNAMIC_COLUMN_VALUE *values,
                               my_bool names);
enum enum_dyncol_func_result
dynamic_column_update_many_fmt(DYNAMIC_COLUMN *str,
                               uint add_column_count,
                               void *column_keys,
                               DYNAMIC_COLUMN_VALUE *values,
                               my_bool string_keys);
enum enum_dyncol_func_result

Check existance of column (not null) (2 functions, one is addition to old
function, other is universal)

dynamic_column_exists_str(DYNAMIC_COLUMN *str, LEX_STRING *name);
enum enum_dyncol_func_result
dynamic_column_exists_fmt(DYNAMIC_COLUMN *str, void *key, my_bool string_keys);
enum enum_dyncol_func_result

Get list of of columns (works for both formats):

dynamic_column_list_str(DYNAMIC_COLUMN *str, DYNAMIC_ARRAY *array_of_lexstr);
enum enum_dyncol_func_result

Get column value (2 functions, one is addition to old
function, other is universal)

dynamic_column_get_str(DYNAMIC_COLUMN *str, LEX_STRING *name,
                           DYNAMIC_COLUMN_VALUE *store_it_here);
enum enum_dyncol_func_result
dynamic_column_get_fmt(DYNAMIC_COLUMN *str, void *key,
                           DYNAMIC_COLUMN_VALUE *store_it_here,
                           my_bool string_key)

Check the string format:

my_bool dynamic_column_has_names(DYNAMIC_COLUMN *str);

Elena Stepanova made changes - 2012-08-16 02:31

Link

This issue relates to TODO-253 [ TODO-253 ]

Elena Stepanova made changes - 2012-08-16 02:37

Link

This issue relates to ~~MDEV-458~~ [ ~~MDEV-458~~ ]

Elena Stepanova made changes - 2012-08-16 03:28

Link

This issue relates to ~~MDEV-459~~ [ ~~MDEV-459~~ ]

Elena Stepanova made changes - 2012-08-16 03:32

Link

This issue relates to ~~MDEV-460~~ [ ~~MDEV-460~~ ]

Elena Stepanova made changes - 2012-08-16 04:13

Link

This issue relates to ~~MDEV-461~~ [ ~~MDEV-461~~ ]

Oleksandr Byelkin made changes - 2012-08-21 11:47

Status

Open [ 1 ]

In Progress [ 3 ]

Oleksandr Byelkin made changes - 2012-08-24 12:07

Status

In Progress [ 3 ]

Open [ 1 ]

Oleksandr Byelkin made changes - 2012-08-24 12:10

Status

Open [ 1 ]

In Progress [ 3 ]

Oleksandr Byelkin made changes - 2012-08-24 13:34

Status

In Progress [ 3 ]

Open [ 1 ]

Elena Stepanova made changes - 2012-08-26 04:55

Link

This issue relates to ~~MDEV-488~~ [ ~~MDEV-488~~ ]

Elena Stepanova made changes - 2012-08-26 19:51

Link

This issue relates to ~~MDEV-489~~ [ ~~MDEV-489~~ ]

Elena Stepanova made changes - 2012-08-26 21:25

Link

This issue relates to ~~MDEV-490~~ [ ~~MDEV-490~~ ]

Elena Stepanova made changes - 2012-08-26 21:41

Link

This issue relates to ~~MDEV-491~~ [ ~~MDEV-491~~ ]

Sergei Golubchik made changes - 2012-09-18 15:41

Fix Version/s		10.0.1 [ 11400 ]
Fix Version/s	10.0.0 [ 10000 ]

Elena Stepanova made changes - 2012-09-27 18:01

Link

This issue relates to ~~MDEV-558~~ [ ~~MDEV-558~~ ]

Elena Stepanova made changes - 2012-09-28 00:19

Link

This issue relates to ~~MDEV-560~~ [ ~~MDEV-560~~ ]

Elena Stepanova made changes - 2012-09-28 02:19

Link

This issue relates to ~~MDEV-561~~ [ ~~MDEV-561~~ ]

Elena Stepanova made changes - 2012-09-28 23:56

Link

This issue relates to ~~MDEV-565~~ [ ~~MDEV-565~~ ]

Sergei Golubchik made changes - 2012-12-11 20:01

Link

This issue is part of ~~MDEV-431~~ [ ~~MDEV-431~~ ]

Sergei Golubchik made changes - 2012-12-11 20:01

Link

This issue is part of ~~MDEV-506~~ [ ~~MDEV-506~~ ]

Sergei Golubchik made changes - 2012-12-11 20:02

Assignee

Oleksandr Byelkin [ sanja ]

Sergei Golubchik [ serg ]

Sergei Golubchik made changes - 2012-12-11 20:04

Priority

Major [ 3 ]

Critical [ 2 ]

Sergei Golubchik made changes - 2012-12-12 17:38

Description

Dynamic columns should be referred by names (now only numeric referrences supported).

Numeric (current) format of dynamic columns string is:

-- fixed part
1 byte flags (now stores <offset size> - 1 in first 3 bits)
2 bytes column counter
-- fixed size column headers sorted by column number, each entry contains of:
  2 bytes - column number
  1-4 bytes combined offset from begning of the data segnebt + 3 bit type
-- data of above columns size of data and length depend on type

For columns with names proposed following format. I think in this format numeric references could be written as strings.

-- fixed part
1 byte flags (now stores <offset size> - 1 in first 3 bits, and 1 bit to set
this new format)
2 bytes column counter
2 bytes size of stored names pool
-- fixed size column headers sorted by names, each consists of
  1 byte length of the name
  2 bytes offset of name
  1-4 bytes combined offset from begning of the data segnebt + 3 bit type
-- names stored one after another
-- data of above columns size of data and length depend on type

there was other ideas which I think is not so good:
1) store names just in header part (then it looses its array properties and binary search)
2) store both headers numbers and names (I doubts that both way of references could be used simultaneously). (We have agreed this is not possible, because name<->number mapping must be the same for every record in the table, and there is no way we could efficiently maintain this mapping. For example, HBase does not provide any fast way to get a full list of column names).

Limitations:
* each name not more then 255 symbols
* sum of all names length cant be more then 65535

New format used if
* at least one of dynamic column references has non integer type.
or
* dynamic column string on which operation performed has new format

New low level interface functions:

Parameters of creaing/changimg functions:
  DYNAMIC_COLUMN *str - the dynamic column.
  uchar *column_keys - array of uint or LEX_STRING keys depends on 'names'
  DYNAMIC_COLUMN_VALUE *values - array of volues
  uint column_count - number of above arrays
  names - TRUE if names are character strings

enum enum_dyncol_func_result
dynamic_column_create_many_fmt(DYNAMIC_COLUMN *str,
                               uint column_count,
                               uchar *column_keys,
                               DYNAMIC_COLUMN_VALUE *values,
                               my_bool names);
enum enum_dyncol_func_result
dynamic_column_update_many_fmt(DYNAMIC_COLUMN *str,
                               uint add_column_count,
                               void *column_keys,
                               DYNAMIC_COLUMN_VALUE *values,
                               my_bool string_keys);
enum enum_dyncol_func_result

Check existance of column (not null) (2 functions, one is addition to old
function, other is universal)

dynamic_column_exists_str(DYNAMIC_COLUMN *str, LEX_STRING *name);
enum enum_dyncol_func_result
dynamic_column_exists_fmt(DYNAMIC_COLUMN *str, void *key, my_bool string_keys);
enum enum_dyncol_func_result

Get list of of columns (works for both formats):

dynamic_column_list_str(DYNAMIC_COLUMN *str, DYNAMIC_ARRAY *array_of_lexstr);
enum enum_dyncol_func_result

Get column value (2 functions, one is addition to old
function, other is universal)

dynamic_column_get_str(DYNAMIC_COLUMN *str, LEX_STRING *name,
                           DYNAMIC_COLUMN_VALUE *store_it_here);
enum enum_dyncol_func_result
dynamic_column_get_fmt(DYNAMIC_COLUMN *str, void *key,
                           DYNAMIC_COLUMN_VALUE *store_it_here,
                           my_bool string_key)

Check the string format:

my_bool dynamic_column_has_names(DYNAMIC_COLUMN *str);

Dynamic columns should be referred by names (now only numeric references supported).

Numeric (current) format of dynamic columns string is:

|| fixed part
| 1 byte flags (now stores <offset size> - 1 in first 3 bits)
| 2 bytes column counter
|| fixed size column headers sorted by column number, each entry contains of:
| 2 bytes column number
| 1-4 bytes combined offset from beginning of the data segment + 3 bit type
|| data of above columns size of data and length depend on type

For columns with names proposed following format. I think in this format numeric references could be written as strings.

|| fixed part
| 1 byte flags (now stores <offset size> - 1 in first 3 bits, and 1 bit to set this new format)
| 2 bytes column counter
| 2 bytes size of stored names pool
|| fixed size column headers sorted by names, each consists of
| 1 byte length of the name
| 2 bytes offset of name
| 1-4 bytes combined offset from beginning of the data segment + 3 bit type
|| names stored one after another
|| data of above columns size of data and length depend on type

there was other ideas which I think is not so good:
# store names just in header part (then it looses its array properties and binary search)
# store both headers numbers and names (I doubts that both way of references could be used simultaneously). (We have agreed this is not possible, because name↔number mapping must be the same for every record in the table, and there is no way we could efficiently maintain this mapping. For example, HBase does not provide any fast way to get a full list of column names).

Limitations:
* each name not more then 255 symbols
* sum of all names length cant be more then 65535

New format used if
* at least one of dynamic column references has non integer type.
or
* dynamic column string on which operation performed has new format

New low level interface functions:

Parameters of creating/changing functions:
| {{DYNAMIC_COLUMN *str}} | the dynamic column.
| {{uchar *column_keys}} | array of {{uint}} or {{LEX_STRING}} keys depends on {{names}}
| {{DYNAMIC_COLUMN_VALUE *values}} | array of values
| {{uint column_count}} | number of above arrays
| {{my_bool names}} | TRUE if names are character strings

{code}
enum enum_dyncol_func_result
dynamic_column_create_many_fmt(DYNAMIC_COLUMN *str,
                               uint column_count,
                               uchar *column_keys,
                               DYNAMIC_COLUMN_VALUE *values,
                               my_bool names);
enum enum_dyncol_func_result
dynamic_column_update_many_fmt(DYNAMIC_COLUMN *str,
                               uint add_column_count,
                               void *column_keys,
                               DYNAMIC_COLUMN_VALUE *values,
                               my_bool string_keys);
{code}

Check existence of column (not null) (2 functions, one is addition to old
function, other is universal)

{code}
enum enum_dyncol_func_result
dynamic_column_exists_str(DYNAMIC_COLUMN *str, LEX_STRING *name);
enum enum_dyncol_func_result
dynamic_column_exists_fmt(DYNAMIC_COLUMN *str, void *key, my_bool string_keys);
{code}

Get list of of columns (works for both formats):

{code}
enum enum_dyncol_func_result
dynamic_column_list_str(DYNAMIC_COLUMN *str, DYNAMIC_ARRAY *array_of_lexstr);
{code}

Get column value (2 functions, one is addition to old
function, other is universal)

{code}
enum enum_dyncol_func_result
dynamic_column_get_str(DYNAMIC_COLUMN *str, LEX_STRING *name,
                           DYNAMIC_COLUMN_VALUE *store_it_here);
enum enum_dyncol_func_result
dynamic_column_get_fmt(DYNAMIC_COLUMN *str, void *key,
                           DYNAMIC_COLUMN_VALUE *store_it_here,
                           my_bool string_key)
{code}

Check the string format:
{code}
my_bool dynamic_column_has_names(DYNAMIC_COLUMN *str);
{code}

Sergei Golubchik made changes - 2012-12-12 21:16

Description

Dynamic columns should be referred by names (now only numeric references supported).

Numeric (current) format of dynamic columns string is:

|| fixed part
| 1 byte flags (now stores <offset size> - 1 in first 3 bits)
| 2 bytes column counter
|| fixed size column headers sorted by column number, each entry contains of:
| 2 bytes column number
| 1-4 bytes combined offset from beginning of the data segment + 3 bit type
|| data of above columns size of data and length depend on type

For columns with names proposed following format. I think in this format numeric references could be written as strings.

|| fixed part
| 1 byte flags (now stores <offset size> - 1 in first 3 bits, and 1 bit to set this new format)
| 2 bytes column counter
| 2 bytes size of stored names pool
|| fixed size column headers sorted by names, each consists of
| 1 byte length of the name
| 2 bytes offset of name
| 1-4 bytes combined offset from beginning of the data segment + 3 bit type
|| names stored one after another
|| data of above columns size of data and length depend on type

there was other ideas which I think is not so good:
# store names just in header part (then it looses its array properties and binary search)
# store both headers numbers and names (I doubts that both way of references could be used simultaneously). (We have agreed this is not possible, because name↔number mapping must be the same for every record in the table, and there is no way we could efficiently maintain this mapping. For example, HBase does not provide any fast way to get a full list of column names).

Limitations:
* each name not more then 255 symbols
* sum of all names length cant be more then 65535

New format used if
* at least one of dynamic column references has non integer type.
or
* dynamic column string on which operation performed has new format

New low level interface functions:

Parameters of creating/changing functions:
| {{DYNAMIC_COLUMN *str}} | the dynamic column.
| {{uchar *column_keys}} | array of {{uint}} or {{LEX_STRING}} keys depends on {{names}}
| {{DYNAMIC_COLUMN_VALUE *values}} | array of values
| {{uint column_count}} | number of above arrays
| {{my_bool names}} | TRUE if names are character strings

{code}
enum enum_dyncol_func_result
dynamic_column_create_many_fmt(DYNAMIC_COLUMN *str,
                               uint column_count,
                               uchar *column_keys,
                               DYNAMIC_COLUMN_VALUE *values,
                               my_bool names);
enum enum_dyncol_func_result
dynamic_column_update_many_fmt(DYNAMIC_COLUMN *str,
                               uint add_column_count,
                               void *column_keys,
                               DYNAMIC_COLUMN_VALUE *values,
                               my_bool string_keys);
{code}

Check existence of column (not null) (2 functions, one is addition to old
function, other is universal)

{code}
enum enum_dyncol_func_result
dynamic_column_exists_str(DYNAMIC_COLUMN *str, LEX_STRING *name);
enum enum_dyncol_func_result
dynamic_column_exists_fmt(DYNAMIC_COLUMN *str, void *key, my_bool string_keys);
{code}

Get list of of columns (works for both formats):

{code}
enum enum_dyncol_func_result
dynamic_column_list_str(DYNAMIC_COLUMN *str, DYNAMIC_ARRAY *array_of_lexstr);
{code}

Get column value (2 functions, one is addition to old
function, other is universal)

{code}
enum enum_dyncol_func_result
dynamic_column_get_str(DYNAMIC_COLUMN *str, LEX_STRING *name,
                           DYNAMIC_COLUMN_VALUE *store_it_here);
enum enum_dyncol_func_result
dynamic_column_get_fmt(DYNAMIC_COLUMN *str, void *key,
                           DYNAMIC_COLUMN_VALUE *store_it_here,
                           my_bool string_key)
{code}

Check the string format:
{code}
my_bool dynamic_column_has_names(DYNAMIC_COLUMN *str);
{code}

Dynamic columns should be referred by names (now only numeric references supported).

Numeric (current) format of dynamic columns string is:

|| fixed part
| 1 byte flags (now stores <offset size> - 1 in first 2 bits)
| 2 bytes column counter
|| fixed size column headers sorted by column number, each entry contains of:
| 2 bytes column number
| 1-4 bytes combined offset from beginning of the data segment + 3 bit type
|| data of above columns size of data and length depend on type

For columns with names proposed following format. I think in this format numeric references could be written as strings.

|| fixed part
| 1 byte flags (now stores <offset size> - 1 in first 2 bits, and 1 bit to set this new format)
| 2 bytes column counter
| 2 bytes size of stored names pool
|| fixed size column headers sorted by names, each consists of
| 1 byte length of the name
| 2 bytes offset of name
| 1-4 bytes combined offset from beginning of the data segment + 3 bit type
|| names stored one after another
|| data of above columns size of data and length depend on type

there was other ideas which I think is not so good:
# store names just in header part (then it looses its array properties and binary search)
# store both headers numbers and names (I doubts that both way of references could be used simultaneously). (We have agreed this is not possible, because name↔number mapping must be the same for every record in the table, and there is no way we could efficiently maintain this mapping. For example, HBase does not provide any fast way to get a full list of column names).

Limitations:
* each name not more then 255 symbols
* sum of all names length cant be more then 65535

New format used if
* at least one of dynamic column references has non integer type.
or
* dynamic column string on which operation performed has new format

New low level interface functions:

Parameters of creating/changing functions:
| {{DYNAMIC_COLUMN *str}} | the dynamic column.
| {{uchar *column_keys}} | array of {{uint}} or {{LEX_STRING}} keys depends on {{names}}
| {{DYNAMIC_COLUMN_VALUE *values}} | array of values
| {{uint column_count}} | number of above arrays
| {{my_bool names}} | TRUE if names are character strings

{code}
enum enum_dyncol_func_result
dynamic_column_create_many_fmt(DYNAMIC_COLUMN *str,
                               uint column_count,
                               uchar *column_keys,
                               DYNAMIC_COLUMN_VALUE *values,
                               my_bool names);
enum enum_dyncol_func_result
dynamic_column_update_many_fmt(DYNAMIC_COLUMN *str,
                               uint add_column_count,
                               void *column_keys,
                               DYNAMIC_COLUMN_VALUE *values,
                               my_bool string_keys);
{code}

Check existence of column (not null) (2 functions, one is addition to old
function, other is universal)

{code}
enum enum_dyncol_func_result
dynamic_column_exists_str(DYNAMIC_COLUMN *str, LEX_STRING *name);
enum enum_dyncol_func_result
dynamic_column_exists_fmt(DYNAMIC_COLUMN *str, void *key, my_bool string_keys);
{code}

Get list of of columns (works for both formats):

{code}
enum enum_dyncol_func_result
dynamic_column_list_str(DYNAMIC_COLUMN *str, DYNAMIC_ARRAY *array_of_lexstr);
{code}

Get column value (2 functions, one is addition to old
function, other is universal)

{code}
enum enum_dyncol_func_result
dynamic_column_get_str(DYNAMIC_COLUMN *str, LEX_STRING *name,
                           DYNAMIC_COLUMN_VALUE *store_it_here);
enum enum_dyncol_func_result
dynamic_column_get_fmt(DYNAMIC_COLUMN *str, void *key,
                           DYNAMIC_COLUMN_VALUE *store_it_here,
                           my_bool string_key)
{code}

Check the string format:
{code}
my_bool dynamic_column_has_names(DYNAMIC_COLUMN *str);
{code}

Sergei Golubchik made changes - 2012-12-12 21:55

Description

Dynamic columns should be referred by names (now only numeric references supported).

Numeric (current) format of dynamic columns string is:

|| fixed part
| 1 byte flags (now stores <offset size> - 1 in first 2 bits)
| 2 bytes column counter
|| fixed size column headers sorted by column number, each entry contains of:
| 2 bytes column number
| 1-4 bytes combined offset from beginning of the data segment + 3 bit type
|| data of above columns size of data and length depend on type

For columns with names proposed following format. I think in this format numeric references could be written as strings.

|| fixed part
| 1 byte flags (now stores <offset size> - 1 in first 2 bits, and 1 bit to set this new format)
| 2 bytes column counter
| 2 bytes size of stored names pool
|| fixed size column headers sorted by names, each consists of
| 1 byte length of the name
| 2 bytes offset of name
| 1-4 bytes combined offset from beginning of the data segment + 3 bit type
|| names stored one after another
|| data of above columns size of data and length depend on type

there was other ideas which I think is not so good:
# store names just in header part (then it looses its array properties and binary search)
# store both headers numbers and names (I doubts that both way of references could be used simultaneously). (We have agreed this is not possible, because name↔number mapping must be the same for every record in the table, and there is no way we could efficiently maintain this mapping. For example, HBase does not provide any fast way to get a full list of column names).

Limitations:
* each name not more then 255 symbols
* sum of all names length cant be more then 65535

New format used if
* at least one of dynamic column references has non integer type.
or
* dynamic column string on which operation performed has new format

New low level interface functions:

Parameters of creating/changing functions:
| {{DYNAMIC_COLUMN *str}} | the dynamic column.
| {{uchar *column_keys}} | array of {{uint}} or {{LEX_STRING}} keys depends on {{names}}
| {{DYNAMIC_COLUMN_VALUE *values}} | array of values
| {{uint column_count}} | number of above arrays
| {{my_bool names}} | TRUE if names are character strings

{code}
enum enum_dyncol_func_result
dynamic_column_create_many_fmt(DYNAMIC_COLUMN *str,
                               uint column_count,
                               uchar *column_keys,
                               DYNAMIC_COLUMN_VALUE *values,
                               my_bool names);
enum enum_dyncol_func_result
dynamic_column_update_many_fmt(DYNAMIC_COLUMN *str,
                               uint add_column_count,
                               void *column_keys,
                               DYNAMIC_COLUMN_VALUE *values,
                               my_bool string_keys);
{code}

Check existence of column (not null) (2 functions, one is addition to old
function, other is universal)

{code}
enum enum_dyncol_func_result
dynamic_column_exists_str(DYNAMIC_COLUMN *str, LEX_STRING *name);
enum enum_dyncol_func_result
dynamic_column_exists_fmt(DYNAMIC_COLUMN *str, void *key, my_bool string_keys);
{code}

Get list of of columns (works for both formats):

{code}
enum enum_dyncol_func_result
dynamic_column_list_str(DYNAMIC_COLUMN *str, DYNAMIC_ARRAY *array_of_lexstr);
{code}

Get column value (2 functions, one is addition to old
function, other is universal)

{code}
enum enum_dyncol_func_result
dynamic_column_get_str(DYNAMIC_COLUMN *str, LEX_STRING *name,
                           DYNAMIC_COLUMN_VALUE *store_it_here);
enum enum_dyncol_func_result
dynamic_column_get_fmt(DYNAMIC_COLUMN *str, void *key,
                           DYNAMIC_COLUMN_VALUE *store_it_here,
                           my_bool string_key)
{code}

Check the string format:
{code}
my_bool dynamic_column_has_names(DYNAMIC_COLUMN *str);
{code}

Dynamic columns should be referred by names (now only numeric references supported).

Numeric (current) format of dynamic columns string is:

|| fixed part
| 1 byte flags (now stores <offset size> - 1 in first 2 bits)
| 2 bytes column counter
|| fixed size column headers sorted by column number, each entry contains of:
| 2 bytes column number
| 1-4 bytes combined offset from beginning of the data segment + 3 bit type
|| data of above columns size of data and length depend on type

For columns with names proposed following format. I think in this format numeric references could be written as strings.

|| fixed part
| 1 byte flags (now stores <offset size> - 1 in first 2 bits, and 1 bit to set this new format)
| 2 bytes column counter
| 2 bytes size of stored names pool
|| fixed size column headers sorted by names, each consists of
| 1 byte length of the name
| 2 bytes offset of name
| 1-4 bytes combined offset from beginning of the data segment + 3 bit type
|| names stored one after another
|| data of above columns size of data and length depend on type

there was other ideas which I think is not so good:
# store names just in header part (then it looses its array properties and binary search)
# store both headers numbers and names (I doubts that both way of references could be used simultaneously). (We have agreed this is not possible, because name↔number mapping must be the same for every record in the table, and there is no way we could efficiently maintain this mapping. For example, HBase does not provide any fast way to get a full list of column names).

Limitations:
* each name not more then 255 symbols
* sum of all names length cant be more then 65535

New format used if
* at least one of dynamic column references has non integer type.
or
* dynamic column string on which operation performed has new format

New low level interface functions:

Parameters of creating/changing functions:
| {{DYNAMIC_COLUMN *str}} | the dynamic column.
| {{uchar *column_keys}} | array of {{uint}} or {{LEX_STRING}} keys depends on {{names}}
| {{DYNAMIC_COLUMN_VALUE *values}} | array of values
| {{uint column_count}} | number of above arrays
| {{my_bool names}} | TRUE if names are character strings

{code}
enum enum_dyncol_func_result
dynamic_column_create_many_fmt(DYNAMIC_COLUMN *str,
                               uint column_count,
                               uchar *column_keys,
                               DYNAMIC_COLUMN_VALUE *values,
                               my_bool names);
enum enum_dyncol_func_result
dynamic_column_update_many_fmt(DYNAMIC_COLUMN *str,
                               uint add_column_count,
                               void *column_keys,
                               DYNAMIC_COLUMN_VALUE *values,
                               my_bool string_keys);
{code}

Check existence of column (not null) (2 functions, one is addition to old
function, other is universal)

{code}
enum enum_dyncol_func_result
dynamic_column_exists_str(DYNAMIC_COLUMN *str, LEX_STRING *name);
enum enum_dyncol_func_result
dynamic_column_exists_fmt(DYNAMIC_COLUMN *str, void *key, my_bool string_keys);
{code}

Get list of of columns (works for both formats):

{code}
enum enum_dyncol_func_result
dynamic_column_list_str(DYNAMIC_COLUMN *str, DYNAMIC_ARRAY *array_of_lexstr);
{code}

Get column value (2 functions, one is addition to old
function, other is universal)

{code}
enum enum_dyncol_func_result
dynamic_column_get_str(DYNAMIC_COLUMN *str, LEX_STRING *name,
                           DYNAMIC_COLUMN_VALUE *store_it_here);
enum enum_dyncol_func_result
dynamic_column_get_fmt(DYNAMIC_COLUMN *str, void *key,
                           DYNAMIC_COLUMN_VALUE *store_it_here,
                           my_bool names)
{code}

Check the string format:
{code}
my_bool dynamic_column_has_names(DYNAMIC_COLUMN *str);
{code}

Sergei Golubchik made changes - 2012-12-14 13:01

Status

Open [ 1 ]

In Progress [ 3 ]

Sergei Golubchik made changes - 2012-12-14 13:22

Status

In Progress [ 3 ]

Open [ 1 ]

Sergei Golubchik made changes - 2012-12-14 13:30

Assignee

Sergei Golubchik [ serg ]

Oleksandr Byelkin [ sanja ]

Oleksandr Byelkin made changes - 2012-12-19 15:38

Status

Open [ 1 ]

In Progress [ 3 ]

Oleksandr Byelkin made changes - 2012-12-19 23:11

Status

In Progress [ 3 ]

Open [ 1 ]

Oleksandr Byelkin made changes - 2013-01-14 13:47

Resolution		Fixed [ 1 ]
Status	Open [ 1 ]	Closed [ 6 ]

Sergei Golubchik made changes - 2014-06-13 15:06

Workflow

defaullt [ 12501 ]

MariaDB v2 [ 46289 ]

Rasmus Johansson (Inactive) made changes - 2015-05-18 17:51

Workflow

MariaDB v2 [ 46289 ]

MariaDB v3 [ 64336 ]

Sergei Golubchik made changes - 2021-12-06 21:22

Workflow

MariaDB v3 [ 64336 ]

MariaDB v4 [ 131947 ]

`DYNAMIC_COLUMN *str`	the dynamic column.
`uchar *column_keys`	array of `uint` or `LEX_STRING` keys depends on `names`
`DYNAMIC_COLUMN_VALUE *values`	array of values
`uint column_count`	number of above arrays
`my_bool names`	TRUE if names are character strings

MariaDB Server

Name support for dynamic columns

Details

Description

Attachments

Issue Links

Activity

People

Dates

Git Integration