|
Confirm. I've hit this while working on MDEV-9233. There, we've got a core file. It's possible to load it, but debugging information is not available.
RPM-based repositories actually have a special "debuginfo" packages. When loading a core into gdb, it says:
Missing separate debuginfos, use: debuginfo-install MariaDB-server-10.0.22-1.el6.x86_64
|
The process to install debuginfo packages is as follows:
- Edit /etc/yum.repos.d/CentOS-Debuginfo.repo (or what you have) and set enabled=1 there.
- yum install yum-utils - this is where debuginfo-install is.
- debuginfo-install $package_name
This gives:
# debuginfo-install MariaDB-server-10.0.22-1.el6.x86_64
|
Loaded plugins: fastestmirror, presto
|
Loading mirror speeds from cached hostfile
|
* base: repos.dfw.quadranet.com
|
* extras: mirror.thelinuxfix.com
|
* updates: centos.chi.host-engine.com
|
Could not find debuginfo for main pkg: MariaDB-server-10.0.22-1.el6.x86_64
|
Could not find debuginfo pkg for dependency package libaio-0.3.107-10.el6.x86_64
|
Could not find debuginfo pkg for dependency package libaio-0.3.107-10.el6.x86_64
|
Could not find debuginfo pkg for dependency package libaio-0.3.107-10.el6.x86_64
|
Could not find debuginfo pkg for dependency package MariaDB-server-10.0.22-1.el6.x86_64
|
--> Running transaction check
|
---> Package gcc-debuginfo.x86_64 0:4.4.7-4.el6 will be installed
|
---> Package glibc-debuginfo.x86_64 0:2.12-1.132.el6 will be installed
|
--> Processing Dependency: glibc-debuginfo-common = 2.12-1.132.el6 for package: glibc-debuginfo-2.12-1.132.el6.x86_64
|
---> Package openssl-debuginfo.x86_64 0:1.0.1e-15.el6 will be installed
|
---> Package pam-debuginfo.x86_64 0:1.1.1-17.el6 will be installed
|
...
|
The key point is:
Could not find debuginfo for main pkg: MariaDB-server-10.0.22-1.el6.x86_64
|
I am looking at the repository as I got it configured by tool at mariadb.org and I don't see debuginfo packages there.
|
|
Btw, CentOS themselves provide debuginfo packages: Looking at http://debuginfo.centos.org/6/x86_64/ I see:
 mariadb55-mariadb-debuginfo-5.5.32-1.el6.centos.alt.x86_64.rpm 21-Dec-2013 13:31 38M
|
 mariadb55-mariadb-debuginfo-5.5.35-1.1.el6.centos.alt.x86_64.rpm 26-Feb-2014 11:10 38M
|
 mariadb55-mariadb-debuginfo-5.5.37-1.3.el6.centos.alt.x86_64.rpm 21-May-2014 17:23 38M
|
 mariadb55-mariadb-debuginfo-5.5.37-9.el6.centos.alt.x86_64.rpm 25-Aug-2014 15:52 38M
|
 mariadb55-mariadb-debuginfo-5.5.40-10.el6.centos.alt.x86_64.rpm 19-Nov-2014 15:08 38M
|
 mariadb55-mariadb-debuginfo-5.5.41-12.el6.centos.alt.x86_64.rpm 06-Feb-2015 15:33 38M
|
which is good. Now, we need the same for packages from MariaDB.org. For 10.0 and 10.1.
EDIT: Fedora hs them too:
http://dl.fedoraproject.org/pub/fedora/linux/development/rawhide/x86_64/debug/m/ has:
 mariadb-connector-c-debuginfo-2.1.0-1.fc24.x86_64.rpm 2015-08-16 18:23 464K
|
 mariadb-debuginfo-10.1.8-3.fc24.x86_64.rpm 2015-12-08 02:46 88M
|
 mariadb-galera-debuginfo-10.0.17-5.fc23.x86_64.rpm 2015-08-16 18:24 37M
|
|
|
(gdb) file /usr/sbin/mysqld
|
Reading symbols from /usr/sbin/mysqld...(no debugging symbols found)...done.
|
|
## OK
|
|
(gdb) core /path/to/core
|
...
|
Reading symbols from /lib64/libaio.so.1...(no debugging symbols found)...done.
|
Loaded symbols for /lib64/libaio.so.1
|
Reading symbols from /lib64/libz.so.1.2.3...Reading symbols from /usr/lib/debug/lib64/libz.so.1.2.3.debug...done.
|
done.
|
Loaded symbols for /lib64/libz.so.1.2.3
|
Reading symbols from /lib64/librt.so.1...Reading symbols from /usr/lib/debug/lib64/librt-2.12.so.debug...done.
|
done.
|
Loaded symbols for /lib64/librt.so.1
|
## Some libraries have debuginfo, some dont.
|
...
|
|
Reading symbols from /usr/lib64/mysql/plugin/ha_tokudb.so...done.
|
Loaded symbols for /usr/lib64/mysql/plugin/ha_tokudb.so
|
|
## But ha_tokudb.so in our packages DOES HAVE debug info! It's not in a
|
## separate package, though
|
Now let's see. traces in TokuDB show line numbers:
Core was generated by `/usr/sbin/mysqld --basedir=/usr --datadir=/var/lib/mysql --plugin-dir=/usr/lib6'.
|
...
|
Thread 152 (Thread 0x7f00511d8700 (LWP 112507)):
|
#0 0x000000332d00b63c in pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:245
|
#1 0x00007f00708ca7f3 in toku_cond_wait (kidv=Unhandled dwarf expression opcode 0xf3
|
)
|
at /home/buildbot/buildbot/build/mariadb-10.0.22/storage/tokudb/ft-index/portability/toku_pthread.h:309
|
#2 kwait (kidv=Unhandled dwarf expression opcode 0xf3
|
) at /home/buildbot/buildbot/build/mariadb-10.0.22/storage/tokudb/ft-index/util/kibbutz.cc:157
|
#3 work_on_kibbutz (kidv=Unhandled dwarf expression opcode 0xf3
|
) at /home/buildbot/buildbot/build/mariadb-10.0.22/storage/tokudb/ft-index/util/kibbutz.cc:196
|
#4 0x000000332d007a51 in __nptl_deallocate_tsd (arg=0x7f00511d8700) at pthread_create.c:148
|
#5 start_thread (arg=0x7f00511d8700) at pthread_create.c:308
|
#6 0x000000332cce893d in ?? () from /lib64/libc.so.6
|
The rest of the server shows function names but not line numbers:
Thread 52 (Thread 0x7f00917f9700 (LWP 112432)):
|
#0 0x0000003ef1e00614 in ?? () from /lib64/libaio.so.1
|
#1 0x00000000008d6027 in os_aio_linux_handle(unsigned long, fil_node_t**, void**, unsigned long*, unsigned long*) ()
|
#2 0x00000000009f7915 in fil_aio_wait(unsigned long) ()
|
#3 0x00000000009443f5 in io_handler_thread ()
|
#4 0x000000332d007a51 in __nptl_deallocate_tsd (arg=0x7f00917f9700) at pthread_create.c:148
|
---Type <return> to continue, or q <return> to quit---
|
#5 start_thread (arg=0x7f00917f9700) at pthread_create.c:308
|
#6 0x000000332cce893d in ?? () from /lib64/libc.so.6
|
#7 0x0000000000000000 in ?? ()
|
|
|
I'll re-state the obvious just in case. This is not about stripped vs not-stripped binaries. All binaries are not stripped (and it's ok):
[root@ip-10-204-137-38 ~]# file /usr/lib64/mysql/plugin/ha_tokudb.so
|
/usr/lib64/mysql/plugin/ha_tokudb.so: ELF 64-bit LSB shared object, x86-64, version 1 (GNU/Linux), dynamically linked, not stripped
|
[root@ip-10-204-137-38 ~]# file /usr/sbin/mysqld
|
/usr/sbin/mysqld: ELF 64-bit LSB executable, x86-64, version 1 (GNU/Linux), dynamically linked (uses shared libs), for GNU/Linux 2.6.18, not stripped
|
However, ha_toku has debug info:
[root@ip-10-204-137-38 ~]# nm -a /usr/lib64/mysql/plugin/ha_tokudb.so | grep '\.debug'
|
0000000000000000 N .debug_abbrev
|
0000000000000000 N .debug_aranges
|
0000000000000000 N .debug_info
|
0000000000000000 N .debug_line
|
0000000000000000 N .debug_loc
|
0000000000000000 N .debug_ranges
|
0000000000000000 N .debug_str
|
while mysqld hasn't:
[root@ip-10-204-137-38 ~]# nm -a /usr/sbin/mysqld | grep '\.debug'
|
[root@ip-10-204-137-38 ~]#
|
|
|
Since CMake-3,7 - added to build infrastructure in MDEV-11258 has the added feature in CPackRPM that allows the creation of Debug info
https://cmake.org/cmake/help/v3.7/module/CPackRPM.html#variable:CPACK_RPM_DEBUGINFO_PACKAGE
Even with a text paste of a crash dump at least the addresses can be mapped to code without reporter intervention with the unpacked RPM and debug symbols:
for pkg in MariaDB-server-* ; do echo $pkg; rpm2cpio ${pkg} | (cd /tmp/m55; cpio -idmv ) ; done
|
gdb /tmp/m55/usr/sbin/mysqld
|
(gdb) set debug-file-directory /tmp/m55//usr/lib/debug
|
(gdb) path /tmp/m55
|
(gdb) symbol-file /tmp/m55//usr/sbin/mysqld
|
Reading symbols from /tmp/m55//usr/sbin/mysqld...Reading symbols from /tmp/m55/usr/lib/debug/usr/sbin/mysqld.debug...done.
|
done.
|
(gdb) disassemble /m mysql_load
|
Dump of assembler code for function mysql_load(THD*, sql_exchange*, TABLE_LIST*, List<Item>&, List<Item>&, List<Item>&, enum_duplicates, bool, bool):
|
110 ::end_io_cache(&cache);
|
0x000000000077ae83 <+3443>: mov 0x48(%rsp),%rax
|
0x000000000077ae88 <+3448>: lea 0x78(%rax),%rdi
|
0x000000000077ae8c <+3452>: callq 0x9a8220 <end_io_cache>
|
0x000000000077b080 <+3952>: mov 0x48(%rsp),%rax
|
0x000000000077b085 <+3957>: lea 0x78(%rax),%rdi
|
0x000000000077b089 <+3961>: callq 0x9a8220 <end_io_cache>
|
|
111 need_end_io_cache = 0;
|
0x000000000077ae99 <+3465>: movb $0x0,0x203(%rsp)
|
0x000000000077b096 <+3974>: movb $0x0,0x203(%rsp)
|
|
112 }
|
113 my_off_t file_length() { return cache.end_of_file; }
|
114 my_off_t position() { return my_b_tell(&cache); }
|
0x000000000077adac <+3228>: mov 0x258(%rsp),%rax
|
0x000000000077adba <+3242>: mov (%rax),%rsi
|
|
|
Thanks, I'll check it out!
|
|
With pull request and cmake-3.8.git-latest (required merged cmake commit and cmake merge request 620 (not merged yet)):
cmake /usr/src/mariadb-hopefully-this-is-a-really-really-long-path-big-enough-for-cmake-debug-package-builds/ -DRPM=fedora && make -j 10 package
|
...
|
ls -la *rpm
|
-rw-rw-r-- 1 dan dan 4578574 Mar 24 21:52 MariaDB-5.5.55-fedora-x86_64-client.rpm
|
-rw-rw-r-- 1 dan dan 27170 Mar 24 21:52 MariaDB-5.5.55-fedora-x86_64-common.rpm
|
-rw-rw-r-- 1 dan dan 1391578 Mar 24 21:52 MariaDB-5.5.55-fedora-x86_64-devel.rpm
|
-rw-rw-r-- 1 dan dan 14907622 Mar 24 21:52 MariaDB-5.5.55-fedora-x86_64-server.rpm
|
-rw-rw-r-- 1 dan dan 1038850 Mar 24 21:52 MariaDB-5.5.55-fedora-x86_64-shared.rpm
|
-rw-rw-r-- 1 dan dan 20348250 Mar 24 21:52 MariaDB-5.5.55-fedora-x86_64-test.rpm
|
-rw-rw-r-- 1 dan dan 3686154 Mar 24 21:52 MariaDB-client-debuginfo-5.5.55-1.fc25.x86_64.rpm
|
-rw-rw-r-- 1 dan dan 30874194 Mar 24 21:52 MariaDB-server-debuginfo-5.5.55-1.fc25.x86_64.rpm
|
-rw-rw-r-- 1 dan dan 2578694 Mar 24 21:52 MariaDB-test-debuginfo-5.5.55-1.fc25.x86_64.rpm
|
The package name isn't consistent however CMake issue 16715 refers it as a possible future release.
debug rpm contents:
rm -rf /tmp/xx ; mkdir /tmp/xx ;for pkg in *rpm ; do echo PACKAGE: $pkg; rpm2cpio $pkg| (cd /tmp/xx; cpio -idmv ) ; done2>&1 | tee /tmp/list-new.txt
|
...
|
PACKAGE: MariaDB-server-debuginfo-5.5.55-1.fc25.x86_64.rpm
|
./usr/lib/debug
|
./usr/lib/debug/.build-id
|
./usr/lib/debug/.build-id/01
|
./usr/lib/debug/.build-id/01/d91b7bdf07181be9147b18c5d62c4240f37e25
|
./usr/lib/debug/.build-id/01/d91b7bdf07181be9147b18c5d62c4240f37e25.debug
|
...
|
./usr/lib/debug/.build-id/fa/16afe4e85267197ca4b2d64d29ea1103f6733c.debug
|
./usr/lib/debug/.dwz
|
./usr/lib/debug/.dwz/MariaDB-server-5.5.55-1.fc25.x86_64
|
./usr/lib/debug/usr
|
./usr/lib/debug/usr/bin
|
./usr/lib/debug/usr/bin/aria_chk.debug
|
./usr/lib/debug/usr/bin/aria_dump_log.debug
|
./usr/lib/debug/usr/bin/aria_ftdump.debug
|
./usr/lib/debug/usr/bin/aria_pack.debug
|
.
|
...
|
./usr/src/debug/MariaDB-5.5.55-fedora-x86_64
|
./usr/src/debug/MariaDB-5.5.55-fedora-x86_64/server
|
./usr/src/debug/MariaDB-5.5.55-fedora-x86_64/server/src_0
|
./usr/src/debug/MariaDB-5.5.55-fedora-x86_64/server/src_0/client
|
./usr/src/debug/MariaDB-5.5.55-fedora-x86_64/server/src_0/client/client_priv.h
|
./usr/src/debug/MariaDB-5.5.55-fedora-x86_64/server/src_0/client/mysql_upgrade.c
|
./usr/src/debug/MariaDB-5.5.55-fedora-x86_64/server/src_0/extra
|
./usr/src/debug/MariaDB-5.5.55-fedora-x86_64/server/src_0/extra/innochecksum.c
|
./usr/src/debug/MariaDB-5.5.55-fedora-x86_64/server/src_0/extra/my_print_defaults.c
|
...
|
cd /tmp/xx
|
$ gdb usr/sbin/mysqld
|
..
|
Try: dnf --enablerepo='*debug*' install /usr/lib/debug/.build-id/3d/8222b0b73026809ab69161ec7459eba87cdd24.debug
|
Reading symbols from /tmp/xx/usr/sbin/mysqld...(no debugging symbols found)...done.
|
(no debugging symbols found)...done.
|
(gdb) file usr/lib/debug/.build-id/3d/8222b0b73026809ab69161ec7459eba87cdd24.debug
|
Reading symbols from usr/lib/debug/.build-id/3d/8222b0b73026809ab69161ec7459eba87cdd24.debug...done.
|
(gdb) list sql_kill
|
6922 only_kill_query Should it kill the query or the connection
|
6923 */
|
6924
|
6925 static
|
6926 void sql_kill(THD *thd, ulong id, killed_state state)
|
6927 {
|
6928 uint error;
|
6929 if (!(error= kill_one_thread(thd, id, state)))
|
6930 {
|
6931 if ((!thd->killed))
|
So to make it work:
- requires latest cmake - make package generates tar.gz and untar that to /usr/local and set PATH=/usr/local/cmake-..../bin:$PATH
- Also requires the source/build directories to be very log. This is because the debug info path is edited in-place in the file during the build and is replaces with ./usr/src/debug/MariaDB-5.5.55-fedora-x86_64/server/src_0/sql/..... - so the source/build path needs to be longer than this.
|
|
CMake-3.8.0 has been released containing all the fixes required to make this happen. Note quite the 3.7.0 required anticipated MDEV-11258. Still partially works with CMake-3.7.0 with the exception being the source code isn't saved in the package correctly. This can be worked around by checking out a source tree at the right release version.
For the build infrastructure it looks like a CMake-3.8.0 update will be required to generate proper debug packages for distribution.
|
|
I've just push a one-liner to set CPACK_RPM_DEBUGINFO_PACKAGE. Although I suspect it won't do anything for our packages, until distributions start to ship cmake 3.7+.
|
|
I think it will be good enough for mariadb package distribution to have debug symbol packages that can map to a source. The main reason for pushing this was so existing core dumps can be analysed without users to build debug versions or attempt to reproduce the bug. Developers can also use debug packages to analyse user core dumps. Fedora is still crafting its own spec file https://src.fedoraproject.org/cgit/rpms/mariadb.git/tree/ (and I'm not sure why _ maybe the rest of their build infrastrure depends on it)
|
|
danblack, I fully agree with that. But if we build with non-standard manually installed cmake, we will get builds that users won't be able to reproduce (on a stock distro installation). That might still be fine. But then there's a good chance we break the build for older cmake versions and won't even notice it.
Perhaps we should switch to using manually installed new cmake almost everywhere, but keep the oldest supported cmake at least one one builder, just to make sure it still works...
Another thought — this RPM-DEFAULT patch. I've pushed it under assumption that it'll work for all new distributions, not for old one. If we switch to a new cmake on old distros, we'll have to document that "mariadb rpm is named XXX for versions 5.5.x and below, 10.0.y and below, 10.1.z and below, and it's named YYY for newer 5.5, 10.0, 10.1 versions". Of course, we can have this in the manual, but I think we'd better not to.
|
|
Fedora-25 has cmake-3.8 up from 3.6 in FC24 so its gradually changing (as is openSUSE).
An distro cmake on one builder sounds very sane.
The RPM-DEFAULT patch defers the package name to the rpmbuilder in the distribution. I don't imagine any distribution has that broken.
Do you think cmake/cpack_rpm.cmake CPACK_RPM_XXX_PACKAGE_OBSOLETES needs to refer to the old mariadb rpm name?
|
|
No, I don't. I think the rpm package name was always MariaDB-server, MariaDB-client, etc. This didn't change. RPM-DEFAULT only changed file names, but MariaDB-server still means what it used to mean.
|
|
Good to know RPM names/filenames are distinguished.
I finally read your comments and MDEV-11258 correctly. You're saying a followup task is needed to use cmake-3.7+ on most builders (perhaps MDEV-12508). I agree.
On rpm documentation - seems that kb pages mentioning RPM could just mention the two forms of filename https://mariadb.com/kb/en/mariadb/about-the-mariadb-rpm-files/. Or like mentioned on https://mariadb.com/kb/en/mariadb/installing-mariadb-with-the-rpm-tool/ "look similar to ...". (or do a major cull of RPM documentation to mainly refer to yum repo based installs).
|