[#MDBF-143] buildbot - static analysis for Infer

Comment by Daniel Black [ 2020-07-01 ]

interesting. Of use would be:

sql
storage/* (broken down by engine due to developer responsibility)

Did you use the 10.5.2 release? or was it a specific git version. A latest 10.5 branch would be useful for results going forwards as the last few months of development has been rather high volume.

Comment by Diego Medina [ 2020-07-01 ]

Perfect, I'll take those folders under consideration for future examination.

Did you use the 10.5.2 release? or was it a specific git version. A latest 10.5 branch would be useful for results going forwards as the last few months of development has been rather high volume.

I used the released version 10.5.2, this can help to solve them?

Comment by Daniel Black [ 2020-07-02 ]

10.5.4 is the latest release https://github.com/MariaDB/server/releases. 10.5.2 is ok, however for some large churned files the line numbers will be different but maybe the context information is enough. Developers are going to be looking at the latest git version to fix these. So don't worry about redoing these attachments, but consider using a git clone of the public repository for future scanning.

I see infer has some good guides on use with CI like - https://fbinfer.com/docs/steps-for-ci, however if there are any non-obvious tips of how you've used it on the mariadb code base that would be useful to note.

Comment by Daniel Black [ 2020-09-30 ]

I had a small play. The 0.17 release is hard compiled against python2.7 and some of its library files aren't py3 compatable. With a small amount of 2to3 and removing encode/decode functions on strings I got it running.

   2to3 -w /opt/infer-linux64-v0.17.0/lib/infer/infer/lib/python/inferlib/issues.py

   2to3 -w /opt/infer-linux64-v0.17.0/lib/infer/infer/lib/python/inferlib/config.py

   2to3 -w /opt/infer-linux64-v0.17.0/lib/infer/infer/lib/python/inferlib/source.py

   2to3 -w /opt/infer-linux64-v0.17.0/lib/infer/infer/bin/../lib/python/report.py

   2to3 -w /opt/infer-linux64-v0.17.0/lib/infer/infer/lib/python/inferlib/utils.py

Following:

https://fbinfer.com/docs/next/analyzing-apps-or-projects/#cmake

10.5- infer run
mariadb-server-10.5]$ infer run --compilation-database ../build-mariadb-server-10.5/compile_commands.json
...
if (from->ctype)##############################################################.....] 93% 19min29s
~~~~ ^rage/test_sql_discovery/test_sql_discovery.cc
/home/dan/repos/mariadb-server-10.5/strings/conf_to_src.c:127:9: error: no member named 'ctype' in 'struct charset_info_st'
to->ctype= (uchar) mdup((char) from->ctype, MY_CS_CTYPE_TABLE_SIZE);
~~ ^ storage/spider/spd_sys_table.cc
/home/dan/repos/mariadb-server-10.5/strings/conf_to_src.c:127:44: error: no member named 'ctype' in 'struct charset_info_st'
to->ctype= (uchar) mdup((char) from->ctype, MY_CS_CTYPE_TABLE_SIZE);
~~~~ ^
/home/dan/repos/mariadb-server-10.5/strings/conf_to_src.c:163:14: error: no member named 'ctype' in 'struct charset_info_st'
if (refcs->ctype && cs->ctype &&
~~~~~ ^
/home/dan/repos/mariadb-server-10.5/strings/conf_to_src.c:163:27: error: no member named 'ctype' in 'struct charset_info_st'
if (refcs->ctype && cs->ctype &&
~~ ^
/home/dan/repos/mariadb-server-10.5/strings/conf_to_src.c:164:19: error: no member named 'ctype' in 'struct charset_info_st'
!memcmp(cs->ctype, refcs->ctype, MY_CS_CTYPE_TABLE_SIZE))
~~ ^
/home/dan/repos/mariadb-server-10.5/strings/conf_to_src.c:164:33: error: no member named 'ctype' in 'struct charset_info_st'
!memcmp(cs->ctype, refcs->ctype, MY_CS_CTYPE_TABLE_SIZE))
~~~~~ ^
/home/dan/repos/mariadb-server-10.5/strings/conf_to_src.c:165:9: error: no member named 'ctype' in 'struct charset_info_st'
cs->ctype= NULL;
~~ ^
/home/dan/repos/mariadb-server-10.5/strings/conf_to_src.c:200:48: error: no member named 'ctype' in 'struct charset_info_st'
return ((cs->csname && cs->tab_to_uni && cs->ctype && cs->to_upper &&
~~ ^
/home/dan/repos/mariadb-server-10.5/strings/conf_to_src.c:279:11: error: no member named 'ctype' in 'struct charset_info_st'
if (cs->ctype)
~~ ^
/home/dan/repos/mariadb-server-10.5/strings/conf_to_src.c:280:48: error: no member named 'ctype' in 'struct charset_info_st'
print_array(f, cs->name, "ctype", cs->ctype, MY_CS_CTYPE_TABLE_SIZE);
~~ ^
/home/dan/repos/mariadb-server-10.5/strings/conf_to_src.c:370:32: error: no member named 'ctype' in 'struct charset_info_st'
print_array_ref(f, cs, cs->ctype, srccs, srccs->ctype, "ctype");
~~ ^
/home/dan/repos/mariadb-server-10.5/strings/conf_to_src.c:370:53: error: no member named 'ctype' in 'struct charset_info_st'
print_array_ref(f, cs, cs->ctype, srccs, srccs->ctype, "ctype");
~~~~~ ^
13 errors generated.ider/spd_table.cc
Error: the following clang command did not run successfully:
/opt/infer-linux64-v0.17.0/lib/infer/facebook-clang-plugins/clang/install/bin/clang-8
@/tmp/clang_command_.tmp.3bb499.txt ++Contents of '/tmp/clang_command_.tmp.3bb499.txt':
"-cc1" "-load"
"/opt/infer-linux64-v0.17.0/lib/infer/infer/bin/../../facebook-clang-plugins/libtooling/build/FacebookClangPlugin.dylib"
"-add-plugin" "BiniouASTExporter"
"-plugin-arg-BiniouASTExporter"
"-"
"-plugin-arg-BiniouASTExporter"
"PREPEND_CURRENT_DIR=1"
"-plugin-arg-BiniouASTExporter"
"MAX_STRING_SIZE=65535" "-cc1"
"-triple"
"x86_64-unknown-linux-gnu"
"-fsyntax-only" "-disable-free"
"-disable-llvm-verifier"
"-discard-value-names"
"-main-file-name" "conf_to_src.c"
"-mrelocation-model" "pic"
"-pic-level" "2" "-mthread-model"
"posix" "-mdisable-fp-elim"
"-fmath-errno" "-masm-verbose"
"-mconstructor-aliases"
"-munwind-tables"
"-fuse-init-array" "-target-cpu"
"x86-64" "-dwarf-column-info"
"-debug-info-kind=limited"
"-dwarf-version=4"
"-debugger-tuning=gdb"
"-momit-leaf-frame-pointer"
"-resource-dir"
"/opt/infer-linux64-v0.17.0/lib/infer/facebook-clang-plugins/clang/install/lib/clang/8.0.0"
"-include"
"/opt/infer-linux64-v0.17.0/lib/infer/infer/bin/../lib/clang_wrappers/global_defines.h"
"-D" "DBUG_TRACE" "-D"
"DISABLE_MYSQL_THREAD_H" "-D"
"HAVE_CONFIG_H" "-D"
"_FILE_OFFSET_BITS=64" "-I"
"/home/dan/repos/mariadb-server-10.5/wsrep-lib/include"
"-I"
"/home/dan/repos/mariadb-server-10.5/wsrep-lib/wsrep-API/v26"
"-I"
"/home/dan/repos/build-mariadb-server-10.5/include"
"-I"
"/home/dan/repos/mariadb-server-10.5/include"
"-D" "NDEBUG" "-D"
"_FORTIFY_SOURCE=2" "-D"
"DBUG_OFF" "-internal-isystem"
"/usr/local/include"
"-internal-isystem"
"/opt/infer-linux64-v0.17.0/lib/infer/facebook-clang-plugins/clang/install/lib/clang/8.0.0/include"
"-internal-externc-isystem"
"/include"
"-internal-externc-isystem"
"/usr/include" "-O2" "-Wall"
"-Wdeclaration-after-statement"
"-Wextra" "-Wno-format-truncation"
"-Wno-init-self"
"-Wno-nonnull-compare"
"-Wno-unused-parameter" "-Wvla"
"-Wwrite-strings"
"-Wno-ignored-optimization-argument"
"-Wno-everything"
"-fconst-strings"
"-fdebug-compilation-dir"
"/home/dan/repos/build-mariadb-server-10.5/strings"
"-ferror-limit" "19"
"-fmessage-length" "0"
"-stack-protector" "1"
"-stack-protector-buffer-size" "4"
"-fobjc-runtime=gcc"
"-fdiagnostics-show-option"
"-vectorize-loops"
"-vectorize-slp" "-x" "c"
"/home/dan/repos/mariadb-server-10.5/strings/conf_to_src.c"
"-O0" "-include"
"/opt/infer-linux64-v0.17.0/lib/infer/infer/bin/../lib/clang_wrappers/global_defines.h"
"-Wno-everything"



External Error: Error running compilation for 'strings/conf_to_src.c': /opt/infer-linux64-v0.17.0/lib/infer/infer/bin/../lib/wrappers/clang
@/tmp/cdb_clang_args.tmp.dffcbe.txt
-fsyntax-only
++Contents of '/tmp/cdb_clang_args.tmp.dffcbe.txt':
-DDBUG_TRACE -DDISABLE_MYSQL_THREAD_H -DHAVE_CONFIG_H -D_FILE_OFFSET_BITS=64 -I/home/dan/repos/mariadb-server-10.5/wsrep-lib/include -I/home/dan/repos/mariadb-server-10.5/wsrep-lib/wsrep-API/v26 -I/home/dan/repos/build-mariadb-server-10.5/include -I/home/dan/repos/mariadb-server-10.5/include -pie -fPIC -fstack-protector --param=ssp-buffer-size=4 -O2 -g -DNDEBUG -fno-omit-frame-pointer -D_FORTIFY_SOURCE=2 -DDBUG_OFF -Wall -Wdeclaration-after-statement -Wextra -Wno-format-truncation -Wno-init-self -Wno-nonnull-compare -Wno-unused-parameter -Wvla -Wwrite-strings -o CMakeFiles/conf_to_src.dir/conf_to_src.c.o -c /home/dan/repos/mariadb-server-10.5/strings/conf_to_src.c

:
exited with code 1
...
Run the command again with `--keep-going` to try and ignore this error.
[dan@linux mariadb-server-10.5]$ infer run --compilation-database --keep-going ../build-mariadb-server-10.5/compile_commands.json
/opt/infer-linux64-v0.17.0/lib/infer/infer/bin/infer: Unexpected anonymous argument: '../build-mariadb-server-10.5/compile_commands.json'.
Infer version v0.17.0
Copyright 2009 - present Facebook. All Rights Reserved.
Usage: infer command [options]
See `infer --help` for more information.

hmm valid compulation bug 'ctype' in 'struct charset_info_st' is only defined for C++ and conf_to_src.c is C

cpp isn't enough
/home/dan/repos/mariadb-server-10.5/strings/conf_to_src.cpp:164:19: error: reference to non-static member function must be called
!memcmp(cs->ctype, refcs->ctype, MY_CS_CTYPE_TABLE_SIZE))
~~~~^~~~~

Comment by Diego Medina [ 2020-10-12 ]

I had tried to analize it with cmake, but I got this error:

diego@localhost:~/mariadbserver-10.5$ infer run --compilation-database ../build-mariadb-server-10.5/compile_commands.json

Capturing using compilation database...

WARNING: Failed to resolve file /home/diego/build-mariadb-server-10.5/sql/sql_yacc.cc with "No such file or directory"

WARNING: Failed to resolve file /home/diego/build-mariadb-server-10.5/sql/sql_yacc_ora.cc with "No such file or directory"

Starting translating and linting 1732 files

Usage Error: Failed to execute compilation command:..............] 0% 589ms

'/opt/infer-linux64-v0.17.0/lib/infer/infer/bin/../../facebook-clang-plugins/clang/install/bin/clang++' ild-mariadb-server-10.5/sql/sql_builtin.cc

  @/tmp/cdb_clang_args.tmp.9455ea.txt -fsyntax-only  ++Contents of '/tmp/cdb_clang_args.tmp.9455ea.txt':

                                                         -DDBUG_TRACE -DHAVE_CONFIG_H -DHAVE_EVENT_SCHEDULER -DHAVE_OPENSSL -DHAVE_POOL_OF_THREADS -DMYSQL_SERVER -D_FILE_OFFSET_BITS=64 -I/home/diego/mariadbserver-10.5/wsrep-lib/include -I/home/diego/mariadbserver-10.5/wsrep-lib/wsrep-API/v26 -I/home/diego/build-mariadb-server-10.5/include -I/home/diego/mariadbserver-10.5/include -I/home/diego/mariadbserver-10.5/sql -I/home/diego/build-mariadb-server-10.5/extra/pcre2/src/pcre2-build -I/home/diego/build-mariadb-server-10.5/extra/pcre2/src/pcre2/src -I/home/diego/build-mariadb-server-10.5/sql -I/home/diego/mariadbserver-10.5/tpool   -pie -fPIC -fstack-protector --param=ssp-buffer-size=4 -O2 -g -DNDEBUG -fno-omit-frame-pointer -D_FORTIFY_SOURCE=2 -DDBUG_OFF -Wall -Wextra -Wformat-security -Wno-format-truncation -Wno-init-self -Wno-nonnull-compare -Wno-unused-parameter -Woverloaded-virtual -Wnon-virtual-dtor -Wvla -Wwrite-strings   -std=gnu++11 -o CMakeFiles/sql.dir/sql_yacc.cc.o -c /home/diego/build-mariadb-server-10.5/sql/sql_yacc.cc

Error message:

clang-8: error: no such file or directory: '/home/diego/build-mariadb-server-10.5/sql/sql_yacc.cc'

...

Maybe it's my mistake, it was the first time I used CMake.

To scan a specific folder, I used the following commands:

diego@localhost:~/mariadbserver-10.5$ infer -- cmake -DCURSES_INCLUDE_PATH=/usr/include -DCURSES_LIBRARY=/usr/lib/x86_64-linux-gnu/libncurses.so ~/mariadbserver-10.5/CMakeLists.txt

Capturing in make/cc mode...

-- Running cmake version 3.16.3

-- The C compiler identification is Clang 8.0.0

-- The CXX compiler identification is Clang 8.0.0

-- Check for working C compiler: /opt/infer-linux64-v0.17.0/lib/infer/infer/lib/wrappers/cc

-- Check for working C compiler: /opt/infer-linux64-v0.17.0/lib/infer/infer/lib/wrappers/cc -- works

-- Detecting C compiler ABI info

-- Detecting C compiler ABI info - done

-- Detecting C compile features

-- Detecting C compile features - done

...

-- Looking for x86intrin.h - found

-- Looking for event.h

-- Looking for event.h - not found

-- Found BISON: /usr/bin/bison (found suitable version "3.5.1", minimum required is "2.0")

-- Configuring done

-- Generating done

-- Build files have been written to: /home/diego/mariadbserver-10.5

Found 139 source files to analyze in /home/diego/mariadbserver-10.5/infer-out

Analysis finished in 1min39ss

  No issues found

And then this command:

diego@localhost:~/mariadbserver-10.5$ infer --source-files-filter "mysys/*" -- make -j 4 -C ~/mariadbserver-10.5

Capturing in make/cc mode...

make: Entering directory '/home/diego/mariadbserver-10.5'

make[1]: Entering directory '/home/diego/mariadbserver-10.5'

make[2]: Entering directory '/home/diego/mariadbserver-10.5'

make[2]: Entering directory '/home/diego/mariadbserver-10.5'

make[2]: Entering directory '/home/diego/mariadbserver-10.5'

make[2]: Entering directory '/home/diego/mariadbserver-10.5'

...

[100%] Building C object extra/mariabackup/CMakeFiles/mariadb-backup.dir/__/__/libmysqld/libmysql.c.o

[100%] Linking CXX executable mariadb-backup

Creating mariabackup link

make[2]: Leaving directory '/home/diego/mariadbserver-10.5'

[100%] Built target mariadb-backup

make[1]: Leaving directory '/home/diego/mariadbserver-10.5'

make: Leaving directory '/home/diego/mariadbserver-10.5'

Found 126 source files to analyze in /home/diego/mariadbserver-10.5/infer-out

Analysis finished in 59min3ss

Found 124 issues

...

...too many issues to display (limit=10 exceeded), please see /home/diego/mariadbserver-10.5/infer-out/bugs.txt or run `infer-explore` for the remaining issues.

Summary of the reports

                 UNINITIALIZED_VALUE: 76

                          DEAD_STORE: 39

                    NULL_DEREFERENCE: 5

                         MEMORY_LEAK: 2

                       RESOURCE_LEAK: 1

  STATIC_INITIALIZATION_ORDER_FIASCO: 1

In this last one I specify the path in this way because Infer with current directory abreviation did not recognize the files with the make command.

Comment by Diego Medina [ 2020-10-12 ]

Today, a new version of Infer was released, it seems they migrated the python 2 code to a new language.

https://github.com/facebook/infer/releases/tag/v1.0.0

Comment by Daniel Black [ 2020-10-13 ]

thanks for the update. I'll try again.

[MDBF-143] buildbot - static analysis for Infer Created: 2020-07-01 Updated: 2022-02-01
Status:	Open
Project:	MariaDB Foundation Development
Component/s:	None
Affects Version/s:	None
Fix Version/s:	None

[MDBF-143] buildbot - static analysis for Infer Created: 2020-07-01 Updated: 2022-02-01