[MDEV-30639] Upgrade to 10.6 and later does not work on Windows Created: 2023-02-11  Updated: 2023-08-01  Resolved: 2023-06-29

Status: Closed
Project: MariaDB Server
Component/s: Platform Windows, Upgrades
Affects Version/s: 10.10.3
Fix Version/s: 10.6.15, 10.9.8, 10.10.6, 10.11.5, 11.0.3, 11.1.2, 11.2.1

Type: Bug Priority: Critical
Reporter: Mitchell Lee Assignee: Michael Widenius
Resolution: Fixed Votes: 1
Labels: None
Environment:

Windows 11 64bit 22H2


Attachments: PNG File error.png     Text File mysql_upgrade_service.MariaDB.log     Text File sample.txt    
Issue Links:
Duplicate
is duplicated by MDEV-31637 Impossible to upgrade MariaDB (either... Closed
Problem/Incident
is caused by MDEV-27105 --ssl option set as default for maria... Closed
Relates
relates to CONC-635 named pipe transport must skip ssl Closed

 Description   

1. Using 10.7.8 at the 'C:\Program Files\MariaDB'
2. To upgrade, execute mariadb-10.10.3-winx64.msi
3. choose 'do not create dateabase for upgrade' radio button
4 'Next' untill see the upgrade screen.
5. It do things some.
6. finally I see the popup screen in the attachment.

What's next?



 Comments   
Comment by Mitchell Lee [ 2023-02-11 ]
  • By seeing the log file,
    the DB using SSL by defining

[mysqld]
ssl_cert=
ssl_key=
ssl_ca=

Comment by Mitchell Lee [ 2023-02-11 ]

I need urgent workaround procedure (manual or patch whatever) to upgrade.

Comment by Marko Mäkelä [ 2023-02-11 ]

Can you try to get the server error log from somewhere? Something that would include both the messages from MariaDB Server 10.7.8 shutdown and the attempt to start up 10.10.3? Sorry, I am not familiar with the Microsoft Windows platform, so I can’t tell where these logs are stored. If they are in regular files, the file name should be something like mysqld.err or mariadbd.err.

Comment by Mitchell Lee [ 2023-02-11 ]
  • non of error log I can found in both folder while (or right after ) the upgrade program working.
  • after the error popup, manually try to start the service, and I can see the beelow error in the old version db's error.log.

2023-02-11 18:13:54 0 [Note] InnoDB: Buffer pool(s) load completed at 230211 18:13:54
2023-02-11 18:15:05 0 [Note] Starting MariaDB 10.7.8-MariaDB-log source revision bc656c4fa54c12ceabd857e8ae134f8979d82944 as process 16164
2023-02-11 18:15:05 0 [Note] InnoDB: Compressed tables use zlib 1.2.12
2023-02-11 18:15:05 0 [Note] InnoDB: Number of transaction pools: 1
2023-02-11 18:15:05 0 [Note] InnoDB: Using crc32 + pclmulqdq instructions
2023-02-11 18:15:05 0 [Note] InnoDB: Initializing buffer pool, total size = 21474836480, chunk size = 536870912
2023-02-11 18:15:05 0 [Note] InnoDB: Completed initialization of buffer pool
2023-02-11 18:15:05 0 [ERROR] InnoDB: Unsupported redo log format. The redo log was created with MariaDB 10.10.3.
2023-02-11 18:15:05 0 [ERROR] InnoDB: Plugin initialization aborted with error Generic error
2023-02-11 18:15:05 0 [Note] InnoDB: Starting shutdown...
2023-02-11 18:15:05 0 [ERROR] Plugin 'InnoDB' init function returned error.
2023-02-11 18:15:05 0 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
2023-02-11 18:15:05 0 [ERROR] Unknown/unsupported storage engine: InnoDB
2023-02-11 18:15:05 0 [ERROR] Aborting

Comment by Mitchell Lee [ 2023-02-11 ]

addtionally

[mysqld]
innodb_flush_method=unbuffered
innodb_doublewrite=ON

Comment by Marko Mäkelä [ 2023-02-11 ]

npdmailing@gmail.com, thank you. I changed the redo log format for one (hopefully last) time in MDEV-14425, in MariaDB Server 10.8.

The messages suggest that the ib_logfile0 is already in the 10.8+ format (the log had been upgraded by MariaDB Server 10.10.3), but for some reason, there is an attempt to start the old 10.7.8 server on it. A downgrade to an older server would be really tricky because you can’t simply delete the ib_logfile0 (see MDEV-27199). If you did that, you’d first have to update the FIL_PAGE_FILE_FLUSH_LSN field in the first page of the system tablespace in order to avoid any corruption due to "log sequence number is in the future".

In any case, I would think that you only want to start up the 10.10.3 server and not go back to the 10.7.8 server.

This feels like a bug in the installer. In your position, I’d try to locate the mariadbd.exe for the 10.10.3 server and manually start it up. Sorry, I can’t give more specific advice. I do not even know the exact steps to start or shut down the server on Windows.

Comment by Mitchell Lee [ 2023-02-11 ]

Well. Now I'm rolled back to 10.7.8 and all are good.
So, now I want
1. proper upgrade steps to 10.1x from 10.7.8 under my customized directory path
and
2. fix any of issue for further seemless upgrade from 10.1x to above for coming days.

Comment by Mitchell Lee [ 2023-02-11 ]

By getting hint from your comment,
I have updgraded the db somehow that by copy and overwrite all bin and lib and share folder into original one .
Seems working.
But totally inconsistant way I feel.
Fix the bug for people.

Comment by Mitchell Lee [ 2023-02-11 ]

and...one more.
below spit the error.

C:\Program Files\MariaDB\bin>mysql_upgrade_service --service=MySQL
FATAL ERROR: OpenService failed with 1060

Comment by Mitchell Lee [ 2023-02-11 ]

Damn....it is not.
the page https://mariadb.com/kb/en/upgrading-mariadb-on-windows/ has wrong information.

it must be 'mariadb-upgrade-service --service=MariaDB' or so.

Comment by Mitchell Lee [ 2023-02-11 ]

and...you should give me solution for below error which in C:\Users\zzzzzzzz\AppData\Local\Temp\mysql_upgrade_service.MariaDB.log

Executing "C:\Program Files\MariaDB\bin\mysqladmin.exe" "-protocol=pipe" "socket=mysql_upgrade_service_14272" "ping" "-no-beep"
C:\Program Files\MariaDB\bin\mysqladmin.exe: connect to server at 'localhost' failed
error: 'TLS/SSL error: The message received was unexpected or badly formatted. Error 0x80090326(SEC_E_ILLEGAL_MESSAGE)'

Comment by Mitchell Lee [ 2023-02-11 ]

Erm....by copy overwrite bin files after installer's mysql_upgrade_service failure, seems not working..
system keep spitting below logs. Going back to 10.7.8.
You guys must fix and release patchs.

2023-02-11 18:47:01 8 [ERROR] Incorrect definition of table mysql.column_stats: expected column 'hist_type' at position 9 to have type enum('SINGLE_PREC_HB','DOUBLE_PREC_HB','JSON_HB'), found type enum('SINGLE_PREC_HB','DOUBLE_PREC_HB').
2023-02-11 18:47:01 8 [ERROR] Incorrect definition of table mysql.column_stats: expected column 'histogram' at position 10 to have type longblob, found type varbinary(255).
2023-02-11 18:47:01 10 [ERROR] Incorrect definition of table mysql.column_stats: expected column 'hist_type' at position 9 to have type enum('SINGLE_PREC_HB','DOUBLE_PREC_HB','JSON_HB'), found type enum('SINGLE_PREC_HB','DOUBLE_PREC_HB').
2023-02-11 18:47:01 10 [ERROR] Incorrect definition of table mysql.column_stats: expected column 'histogram' at position 10 to have type longblob, found type varbinary(255).
2023-02-11 18:47:01 11 [ERROR] Incorrect definition of table mysql.column_stats: expected column 'hist_type' at position 9 to have type enum('SINGLE_PREC_HB','DOUBLE_PREC_HB','JSON_HB'), found type enum('SINGLE_PREC_HB','DOUBLE_PREC_HB').

Comment by Vladislav Vaintroub [ 2023-02-11 ]

Can you maybe share your C:\Program Files\MariaDB\data\my.ini, so we get the full picture.

Comment by Mitchell Lee [ 2023-02-11 ]

Well...
1. your request related to error contents mysql_upgrade_service.MariaDB.log ? (which is attached))
2. which part actually you want?

Comment by Vladislav Vaintroub [ 2023-02-11 ]

1. Yes, thank you, I see it is attached.
2. The config file, C:\Program Files\MariaDB\data\my.ini

You can remove sensitive information, e.g passwords and what not. I'm, interested in the SSL configuration and SSL parameters related to client and server

Comment by Mitchell Lee [ 2023-02-11 ]

I mean...which sections of config names you want. not all.

Comment by Vladislav Vaintroub [ 2023-02-11 ]

Thanks for the tip, I'm aware of backslash weirdness. Thus I'm using single forward slashes. I assume that
A:
Locate is not the actual name of the file, and that ssl_cert, ssl_key, ssl_ca refer to existing, different files, and ssl configuration would be valid. Do you have any ssl settings in any other section than [mysqld], that would refer to client program?

Comment by Mitchell Lee [ 2023-02-11 ]

that is all. sample.txt

Comment by Vladislav Vaintroub [ 2023-02-11 ]

The actual error is attempt to use named pipe with SSL.
Named pipe is used during upgrade, on some reason it "automagically" tries to connect with SSL, and it can't

Executing "C:\Program Files\MariaDB 10.10\bin\mysqladmin.exe" "--protocol=pipe" "--socket=mysql_upgrade_service_16476" "ping" "--no-beep"
C:\Program Files\MariaDB 10.10\bin\mysqladmin.exe: connect to server at 'localhost' failed
error: 'TLS/SSL error: The message received was unexpected or badly formatted. Error 0x80090326(SEC_E_ILLEGAL_MESSAGE)'

If pipe can't be used with SSL, and it can't currently, until then server should not tell the named pipe client it supports SSL. Alternatively, the client should ignore the server's SSL capacity if it connects via named pipe.

Comment by Vladislav Vaintroub [ 2023-02-11 ]

There is an unfortunate circumstance that leaves any MariaDB Server on Windows that has SSL in a non-startable state after attempted upgrade.

  • The server starts up with the new binaries. The startup works fine, and innodb creates redo log for the new version
  • mysql_upgrade_service does not know that startup worked fine, because mysqladmin.exe fails to ping described, so it decides not to change service configuration, i.e service will startup with old binaries.
  • old Innodb in the old binaries does not like new redo log , Innodb startup fails.
Comment by Mitchell Lee [ 2023-04-18 ]

Erm,
1. What does the actuall meaning of the status STALL? going to be fixed anyway?
2. Any proper upgrade manual/document from old version to new version which manually do for who using SSL?

Comment by Justus Greiber [ 2023-05-19 ]

Hi,

I think I am facing the same problem (although for different versions). I discovered that the "mysqladmin.exe" that is running to execute the ping has different behavior if the order of the arguments is switched. The first order works fine, while the order used by the update process (second) does not work.

C:\Windows\System32>mysqladmin.exe "--socket=mysql_upgrade_service_23516" --protocol=pipe -v PING
mysqld is alive
 
C:\Windows\System32>mysqladmin.exe --protocol=pipe "--socket=mysql_upgrade_service_23516" -v PING
mysqladmin.exe: connect to server at 'localhost' failed
error: 'Can't connect to server on 'localhost' (10061)'
Check that mariadbd is running and that the socket: 'mysql_upgrade_service_23516' exists!

Comment by Mitchell Lee [ 2023-06-16 ]

After long struggling, found out the root cause since the system uses custom plugin not that listed in standard MariaDB.

So,
Install new DB with instance(target) > install the custom UDF plugin binary and run the UDF installation sql script to register the plugin into the MariaDB > Turned off SSL configs and related configs in the old version MariaDB(source) > use mariadb-upgrade-wizard.exe > upgrade succeed.

So question and suggestion.
Question : Does the SSL config can be affected to the upgrade service process with vanilla MariaDB?
Sugession : Please include the copy and migrate process to copy 'Plugin' UDF binary and automatic copy/run UDF registry sql script.

Comment by Oleg Smirnov [ 2023-06-28 ]

JustusGreiber, thanks for your valuable observation! Indeed,

"--socket"

parameter resets previously set

"--protocol"

parameter value, that's why it works in one order and does not in another.

Comment by Justus Greiber [ 2023-06-28 ]

Glad I could help. Is it intended to behave like that or will this be changed? (If it is intended, then it is IMHO very hard to find in the documentation and maybe that could be improved.)

Comment by Oleg Smirnov [ 2023-06-28 ]

No, it is certainly unintended. We are already testing the patch to fix it.

Comment by Michael Widenius [ 2023-06-29 ]

Looks good. Ok to push.
Good clear commit comment!

Comment by Michael Widenius [ 2023-06-29 ]

ok to push

Comment by Oleg Smirnov [ 2023-06-29 ]

Pushed to 10.9, to be merged upstream.

Comment by Oleg Smirnov [ 2023-06-30 ]

This bug affects versions starting from 10.6, so the fix is pushed to 10.6 as well.

Generated at Thu Feb 08 10:17:46 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.