Debugging, I see that the crash happens inside this call:

pthread_mutex_init(&THR_LOCK_dbug, NULL);

pthread_mutex_init translates to InitializeCriticalSection on Windows. InitializeCriticalSection only requires that valid memory is passed to it (which is true).

My guess is that we're trying to initialize another critical section where the first critical section is already initialized. MSDN mentions that CRITICAL_SECTION objects cannot be moved in memory, so attempt to initialize one over another may be considered an invalid operation.

The following patch makes the crash go away:

=== modified file 'dbug/dbug.c'

--- dbug/dbug.c 2013-11-20 11:05:39 +0000

+++ dbug/dbug.c 2014-03-20 13:45:44 +0000

@@ -342,6 +342,7 @@ static CODE_STATE *code_state(void)

     sstdout->file= stdout;

     sstderr->file= stderr;

     pthread_mutex_init(&THR_LOCK_dbug, NULL);

+       fprintf(stderr, "psergey: initing THR_LOCK_dbug\n");

     bzero(&init_settings, sizeof(init_settings));

     init_settings.out_file= sstderr;

     init_settings.flags=OPEN_APPEND;

@@ -1642,6 +1643,9 @@ void _db_end_()

   cs->stack= &init_settings;

   FreeState(cs, 0);

+  //psergey:

+  fprintf(stderr, "psergey: freeing THR_LOCK_dbug\n");

+  pthread_mutex_destroy(&THR_LOCK_dbug);

   init_done= 0;

 }

When running with the patch, I see:

psergey: initing THR_LOCK_dbug

(standard messages about MariaDB startup)

psergey: freeing THR_LOCK_dbug
psergey: initing THR_LOCK_dbug

So, it could be that this particular Windows Server 2012 machine started being picky about programs placing one CRITICAL_SECTION object over another.

As Sergei found out, the machine had Application Verifier set up for mysqld.exe. It made the machine being picky about this critical section specifics.

The crash only happens under verifier on debug builds, both 5.5 and 10.0. I don't know whether the verifier points out at a real problem here – if it does, then it should probably be fixed. If it's just the verifier's whim which does not reveal a code flaw, I suppose it can be left as is.

How it happened:
Application verifier is used in buildbot tests on this machine.
We turn off appverif for mysqld.exe at the first test step, as the precaution measure, then turn it on for one test run on a non-debug build, and immediately turn it off again when the test finishes, and turn it off once again at the next step before collecting the data. So, it should always be off except for a single test run.
Apparently, the server got rebooted during this very test run, as it happens sometimes with Windows machines and their critical upgrades. It turns out that the appverif configuration is sticky – once turned on, it stays on until it is explicitly turned off. So, once it happened, the server started crashing on the build step, when bootstrap is run and the initial data is created; when buildbot fails on a build step, it doesn't go further, so it never reached the first test step when the verifier would be turned off. I will add switching it off at the very beginning of the factory as another precaution.

serg, please review the fix (assuming fprintfs will be deleted).