[MDEV-10191] non convertible chars convert() resulted in Null instead "?" on Windows Created: 2016-06-08  Updated: 2020-08-25  Resolved: 2017-09-15

Status: Closed
Project: MariaDB Server
Component/s: Character Sets
Affects Version/s: 10.1.4, 10.1, 10.2
Fix Version/s: 10.1.27, 10.2.9, 10.3.2

Type: Bug Priority: Major
Reporter: Richard Stracke Assignee: Alexander Barkov
Resolution: Fixed Votes: 1
Labels: None
Environment:

Windows, reproduced on Windows Server 2012 R2 Standard Edition 64bit



 Description   

All Linux Versions and Windows before MariaDB 10.1.4
select convert('ABCMogę jeśćDEF' using latin1)
->'ABCMog? je??DEF'

On Windows and all Versions starting from 10.1.4
select convert('ABCMogę jeśćDEF' using latin1)
--> NULL



 Comments   
Comment by Alexander Barkov [ 2016-06-08 ]

Can you please send results of these queries:

SHOW VARIABLES LIKE 'character_set_%';
SELECT HEX('ABCMogę jeśćDEF');

Thanks.

Comment by Richard Stracke [ 2016-06-08 ]

Certainly.

SHOW VARIABLES LIKE 'character_set_%';

"Variable_name" "Value"
"character_set_client" "utf8mb4"
"character_set_connection" "utf8mb4"
"character_set_database" "latin1"
"character_set_filesystem" "binary"
"character_set_results" "utf8mb4"
"character_set_server" "latin1"
"character_set_system" "utf8"
"character_sets_dir" "C:\Program Files\MariaDB 10.1\share\charsets\"

SELECT HEX('ABCMogę jeśćDEF');

4142434D6F67C499206A65C59BC487444546

Comment by Alexander Barkov [ 2016-06-08 ]

Thank you!
The client seems to send correct UTF8 sequences,
and values of character_set_client, character_set_connection and character_set_results variables look good.

Which client is it? Windows console does not support UTF8.
So I guess this is something else than "mysql.exe".

Also, I could not reproduce the problem on Linux.
I'm getting the expected result with 10.1.15 using the standard "mysql" client:

+--------------------------------------------+
| convert('ABCMogę jeśćDEF' using latin1)    |
+--------------------------------------------+
| ABCMog? je??DEF                            |
+--------------------------------------------+
1 row in set, 1 warning (0.00 sec)

Which client are you using on Linux?

Any other details which can help to reproduce the problem would be very appreciated.

Thanks!

Comment by Richard Stracke [ 2016-06-08 ]

Sorry, if I don't describe it clear enough.

The Issue onaly oocurs on Windows since Version 10.1.4 .
All Linux versions are fine.
All Windows before 10.1.4 are fine.
So the function on Windows is broken between 10.1.3 and 10.1.4

I use the official MariaDB client for Window: HeidiSQL

regards,

Richard

Comment by Alexander Barkov [ 2016-06-09 ]

Thanks for the additional info.
Just tried with MariaDB Server 10.1.15 on Linux with HeidiSQL 9.3.0.4984 on Windows 7.

Got expected results:

ABCMog? je??DEF

with a warning:

Cannot convert 'utf8mb4' character 0xC499 to 'latin1'

Comment by Alexander Barkov [ 2016-06-09 ]

Tried with MariaDB Server 10.1.15 on Windows 7, with HeidiSQL 9.3.0.4984 running on the same machine.
Got expected results.

Comment by Alexander Barkov [ 2016-06-09 ]

Perhaps the problem is on the HeidiSQL side...
According to the source code, CONVERT() should not return NULL, unless the input is NULL...

Can you please provide some more information:

  • Which HeidiSQL version are you using?
  • If you do "SHOW WARNINGS;" after the query with convert(), does it return any warnings?
  • Can you please try this query:

    select hex(convert('ABCMogę jeśćDEF' using latin1));
    

    What does it return?

Thanks!

Comment by Richard Stracke [ 2016-06-09 ]

It returns null.

Version is 9.1.0.4867.

Please consider, that a customer have the same results.

Valerii have checked on console as well.

I quote his note:

Let me share my test results. Windows first:

[openxs@fc23 maria10.1]$ bin/mysql -A -uroot -proot --host=192.168.0.60 --port=3316 test
Welcome to the MariaDB monitor. Commands end with ; or \g.
Your MariaDB connection id is 2
Server version: 10.1.13-MariaDB mariadb.org binary distribution
 
Copyright (c) 2000, 2016, Oracle, MariaDB Corporation Ab and others.
 
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
 
MariaDB [test]> show variables like 'char%';
+--------------------------+-----------------------------------------------------+
| Variable_name | Value |
+--------------------------+-----------------------------------------------------+
| character_set_client | utf8 |
| character_set_connection | utf8 |
| character_set_database | latin1 |
| character_set_filesystem | binary |
| character_set_results | utf8 |
| character_set_server | latin1 |
| character_set_system | utf8 |
| character_sets_dir | C:\Program Files (x86)\MariaDB 10.1\share\charsets\ |
+--------------------------+-----------------------------------------------------+
8 rows in set (0.01 sec)
 
MariaDB [test]> select convert('ABCMogę jeśćDEF' using latin1);
+--------------------------------------------+
| convert('ABCMogę jeśćDEF' using latin1) |
+--------------------------------------------+
| NULL |
+--------------------------------------------+
1 row in set, 1 warning (0.00 sec)

Richard

Comment by Richard Stracke [ 2016-06-09 ]

Hi,

i forgot the warning.
Code 1976
Cannot convert 'utf8mb4' character 0xC499 to 'latin1'

Comment by Alexander Barkov [ 2016-06-09 ]

Valerii helped to track the problem down.
The result depends only on sql_mode.

SET sql_mode='';
SELECT CONVERT(_utf8 0xC499 USING latin1);

returns

+------------------------------------+
| CONVERT(_utf8 0xC499 USING latin1) |
+------------------------------------+
| ?                                  |
+------------------------------------+
1 row in set, 1 warning (0.00 sec)

while this:

SET sql_mode='STRICT_TRANS_TABLES';
SELECT CONVERT(_utf8 0xC499 USING latin1);

returns

+------------------------------------+
| CONVERT(_utf8 0xC499 USING latin1) |
+------------------------------------+
| NULL                               |
+------------------------------------+
1 row in set, 1 warning (0.00 sec)

Comment by Vladislav Vaintroub [ 2017-08-21 ]

Windows, MSI installatiion is setting sql_mode to contain STRICT_TRANS_TABLES. This has its roots back to how MySQL handled it (since 4.1 or so), and also was discussed with QA engineer when MSI installer was implemented.

Also MariaDB 10.2 is using this bit STRICT_TRANS_TABLES by default cross-plattform.

Everyone can set sql_mode to another value, if defaults do not work well with the app.

Comment by Vladislav Vaintroub [ 2017-08-21 ]

reassigning to bar to check.

Generated at Thu Feb 08 07:40:21 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.