[CONC-465] [client] default-character-set does not work Created: 2020-03-27  Updated: 2020-03-28  Resolved: 2020-03-27

Status: Closed
Project: MariaDB Connector/C
Component/s: None
Affects Version/s: 3.1.7
Fix Version/s: N/A

Type: Bug Priority: Major
Reporter: Olaf Behrendt Assignee: Georg Richter
Resolution: Not a Bug Votes: 0
Labels: None
Environment:

Linux 4.15.0-91-generic #92-Ubuntu SMP Fri Feb 28 11:09:48 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
mysql Ver 15.1 Distrib 10.3.22-MariaDB, for debian-linux-gnu (x86_64) using readline 5.2
mysqld Ver 10.3.22-MariaDB-1:10.3.22+maria~bionic-log for debian-linux-gnu on x86_64 (mariadb.org binary distribution)


 Description   

[client] default-character-set does not work

Summary

After switching from MariaDB 10.1.44 to MariaDB 10.3.22 on Debian 10 and on Ubuntu 18.04 using apt packages from MariaDB repository we observed that when using the C-API INSERTS of UTF8 encoded text is silently saved as latin1 encoding.

  • Source C-String is UTF8
  • Target table field has character set utf8mb4
  • mariadb.cnf has [client] default-character-set = utf8mb4

MariaDB [urldb]> show variables like '%character_set%';
 
+--------------------------+----------------------------+
 
| Variable_name            | Value                      |
 
+--------------------------+----------------------------+
 
| character_set_client     | utf8mb4                    |
 
| character_set_connection | utf8mb4                    |
 
| character_set_database   | utf8mb4                    |
 
| character_set_filesystem | binary                     |
 
| character_set_results    | utf8mb4                    |
 
| character_set_server     | utf8mb4                    |
 
| character_set_system     | utf8                       |
 
| character_sets_dir       | /usr/share/mysql/charsets/ |
 
+--------------------------+----------------------------+

Workaround: Since switch to MariaDB v10.3.22 we have to use mysql_set_character_set() on each connection.

Code to Reproduce Behaviour

#define _GNU_SOURCE
 
#include <stdlib.h>
 
#include <stdio.h>
 
#include <mariadb/mysql.h>
 
 
 
#define HANDLE_MYSQL_ERR(MYSQL) \
 
if (MYSQL && mysql_errno(MYSQL)) {\
 
    fprintf(stderr, "DB ERROR(%d) [%s] \"%s\"\n", mysql_errno(MYSQL), mysql_sqlstate(MYSQL), mysql_error(MYSQL));\
 
    exit(-1);\
 
}
 
 
 
/**
 
 * Compile: gcc -Wall main.c -lmariadb
 
 * Usage..: a.out <host> <user> <passwd> <db> <port> <unix_socket>
 
 * Example: $ ./a.out localhost harry 1234 mydb 0 /var/run/mysqld/mysqld.sock
 
 *          Default charachter set: latin1
 
 *          New     charachter set: utf8mb4
 
 */
 
int main (int argc, char *argv[])
 
{
 
    /* connect to MariaDB server */
 
    MYSQL *mysql =  mysql_init(NULL);
 
    // host, user, passwd, db, unix, port, socket
 
    if (!mysql_real_connect(mysql, argv[1], argv[2], argv[3], argv[4], atoi(argv[5]), argv[6], 0))
 
        HANDLE_MYSQL_ERR(mysql);
 
 
 
    printf("Default charachter set: %s\n", (char*)mysql_character_set_name(mysql));
 
    if (!mysql_set_character_set(mysql, "utf8mb4"))
 
        HANDLE_MYSQL_ERR(mysql);
 
    printf("New     charachter set: %s\n", (char*)mysql_character_set_name(mysql));
 
 
 
    mysql_close(mysql);
 
    return 0;
 
}



 Comments   
Comment by Georg Richter [ 2020-03-27 ]

The default character set in Connector/C is latin1 unless you build it with cmake option -DDEFAULT_CHARSET=another_charset.

Entries in configuration file have no effect, unless you force Connector/C to read the configuration.

From MariaDB Connector/C documentation:

MYSQL_READ_DEFAULT_FILE
Read options from named configuration file. To read from default my.cnf configuration file, a NULL pointer has to be passed instead of a file name. Note: MariaDB Connector/C will not read the configuration by default. If MYSQL_READ_DEFAULT_FILE is specified the following sections will be always processed:
....

Comment by Olaf Behrendt [ 2020-03-28 ]

Dear @georg, thank you for your immediate reply! You are absolutely right, we have to explicitly call mysql_optionsv() in order to enable parsing of config files. This is also documented in Configuring MariaDB Connector/C with Option Files.

Nevertheless the fact that reading config file options by C-client library is not done by default, seems counter-intuitive to me. That's probably the reason I did not read the documentation intensely enough. I try to do better next time

Comment by Olaf Behrendt [ 2020-03-28 ]

In order to test MYSQL_READ_DEFAULT_FILE I modified my little test program (see above) by adding immediately after mysql_init() (as documented here):

    if (!mysql_optionsv(mysql, MYSQL_READ_DEFAULT_FILE, NULL))
 
        HANDLE_MYSQL_ERR(mysql);

When running the modified a.out program I get

Default charachter set: latin1
 
New     charachter set: utf8mb4

Same result after I copied /etc/mysql/my.cnf to ~/.my.cnf, So seemingly setting mysql_optionsv(mysql, MYSQL_READ_DEFAULT_FILE, NULL) did not change the default client character set.

I also double checked by copying my.cnf to the current working directory (where a.out test program is located) and re-compiled and ran a.out with mysql_optionsv(mysql, MYSQL_READ_DEFAULT_FILE, "my.cnf"). That means I expect my.cnf in the current working directory to be read. And voila, now the output is as expected:

Default charachter set: utf8mb4
 
New     charachter set: utf8mb4

To check the default location of config files I ran mysql --help --verbose (see Configuring MariaDB with Option Files:

Default options are read from the following files in the given order:
 
/etc/my.cnf /etc/mysql/my.cnf ~/.my.cnf

.h2 Summary

  • calling mysql_optionsv(mysql, MYSQL_READ_DEFAULT_FILE, NULL) does not change default character set for default config file locations /etc/mysql/my.cnf, ~/.my.cnf.
  • calling mysql_optionsv(mysql, MYSQL_READ_DEFAULT_FILE, "my.cnf") correctly changes default character set for my.cnf in the current working directory.

Should I open a new ticket, or did I overlook anything in the documentation or other?

Comment by Georg Richter [ 2020-03-28 ]

You're right - thats a bug in documentation (already fixed). You need to pass NULL pointer or empty string to MYSQL_READ_DEFAULT_GROUP, not to MYSQL_READ_DEFAULT_FILE.

Reading configuration file is not enabled by default (also not in libmysql). Consider you have several configuration files, which contain a server section only. Now for every single connect the client library needs to search for various configuration files at different places and process these files. This wouldn't be effective nor performant.

Comment by Olaf Behrendt [ 2020-03-28 ]

Yes, MYSQL_READ_DEFAULT_GROUP worked as expected. Thanks, your feedback is much appreciated!

Generated at Thu Feb 08 03:05:31 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.