I've confirmed that without pam_user_map auth_pam works just fine, (ie the user with the same name exists in passwd and in the database.)

We're trying to use pam_user_map with this conf:

/etc/pam.d/mysql:

auth requisite pam_unix.so
account required pam_unix.so
auth required pam_user_map.so

/etc/security/user_map.conf

testuser: dbadmin

The user dbadmin exists in the database with [...] AUTHENTICATED VIA pam set.

Two scenarios can happen. First scenario is if the user dbadmin does not exist in /etc/passwd.
I run mysql -utestuser -p
PAM then logs this:

mysqld: pam_unix(mysql:account): could not identify user (from getpwnam(dbadmin))

The second scenario is if dbadmin actually exists for pam_unix (/etc/passwd), then I get this:

151118 12:24:33 [ERROR] mysqld got signal 11 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.

To report this bug, see http://kb.askmonty.org/en/reporting-bugs

We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.

Server version: 10.0.22-MariaDB-0+deb8u1-log
key_buffer_size=805306368
read_buffer_size=131072
max_used_connections=1
max_threads=502
thread_count=1
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1888682 K bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x0x7fe1e62ff008
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0x7fe3ad585e58 thread_stack 0x80000
/usr/sbin/mysqld(my_print_stacktrace+0x2e)[0xbf5afe]
/usr/sbin/mysqld(handle_fatal_signal+0x392)[0x7304d2]
/lib/x86_64-linux-gnu/libpthread.so.0(+0xf8d0)[0x7fe3aee2b8d0]
/lib/x86_64-linux-gnu/security/pam_user_map.so(+0xbd0)[0x7fe1def9cbd0]
/lib/x86_64-linux-gnu/security/pam_user_map.so(pam_sm_authenticate+0x2dd)[0x7fe1def9cee1]
/lib/x86_64-linux-gnu/libpam.so.0(+0x2f8f)[0x7fe34d9f2f8f]
/lib/x86_64-linux-gnu/libpam.so.0(pam_authenticate+0x2d)[0x7fe34d9f285d]
/usr/lib/mysql/plugin/auth_pam.so(+0xeae)[0x7fe3ad587eae]
/usr/sbin/mysqld[0x550206]
/usr/sbin/mysqld(_Z16acl_authenticateP3THDj+0x104d)[0x5709ad]
/usr/sbin/mysqld[0x6906c6]
/usr/sbin/mysqld(_Z16login_connectionP3THD+0x42)[0x691c82]
/usr/sbin/mysqld(_Z24do_handle_one_connectionP3THD+0x155)[0x692665]
/usr/sbin/mysqld(handle_one_connection+0x39)[0x6927b9]
/usr/sbin/mysqld[0x93c741]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x80a4)[0x7fe3aee240a4]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7fe3ada8604d]

Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (0x0): is an invalid pointer
Connection ID (thread ID): 17
Status: NOT_KILLED

Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=off,table_elimination=on,extended_keys=on,exists_to_in=on

The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.

I did some changes to squash a few implicit warnings I was getting from the compiler as well as the #define PAM_SM_AUTH to make sure PAM allowed it to act as an authentication module.
I'm pasting it below since I'm getting an error from JIRA when I tried to attach it as a patch.

diff --git a/plugin/auth_pam/mapper/pam_user_map.c b/plugin/auth_pam/mapper/pam_user_map.c
index 1c4bccc..535a146 100644
— a/plugin/auth_pam/mapper/pam_user_map.c
+++ b/plugin/auth_pam/mapper/pam_user_map.c
@@ -24,6 +24,8 @@ top: accounting

*/

+#include <string.h>
+#include <ctype.h>
#include <stdlib.h>
#include <stdio.h>
#include <syslog.h>
@@ -31,6 +33,8 @@ top: accounting
#include <pwd.h>

#include <security/pam_modules.h>
+#include <security/pam_ext.h>
+#define PAM_SM_AUTH

#define FILENAME "/etc/security/user_map.conf"
#define skip(what) while (*s && (what)) s++

Some other notes:
I had to get mariadb source for the debian package and compile it from scatch since dialog.so is not installed properly in the Debian packages, I did find a separate issue for this though, but it seems to have stalled.
It would be good to flesh out the documentation regarding this module if you intend to support it, especially with examples of how it will work when it is enabled, much like here:
https://dev.mysql.com/doc/refman/5.5/en/pam-authentication-plugin.html
Which makes it much easier to understand it conceptually.