[MDEV-13382] InnoDB: Operating system error number 2 in a file operation. Created: 2017-07-26  Updated: 2017-09-27  Resolved: 2017-09-27

Status: Closed
Project: MariaDB Server
Component/s: OTHER
Affects Version/s: 10.0.26, 10.0.29
Fix Version/s: N/A

Type: Bug Priority: Major
Reporter: JEON YANG BAIK Assignee: Unassigned
Resolution: Not a Bug Votes: 0
Labels: need_feedback
Environment:

CentOS 5.8 (VMWare Guest OS)

Attachments: File my.cnf    

 Description   

Hi,

It happened DB crash with this messages.

2017-07-24 17:40:20 2b358f091940  InnoDB: Operating system error number 2 in a file operation.
 
InnoDB: The error means the system cannot find the path specified.
 
170724 17:40:20 [ERROR] InnoDB: File /tmp/#sql654b_24aa3_0.ibd: 'open' returned OS error 71. Cannot continue operation
 
pure virtual method called
 
pure virtual method called
 
pure virtual method called
 
terminate called without an active exception
 
terminate called without an active exception
 
terminate called recursively
 
170724 17:40:20 [ERROR] mysqld got signal 6 ;
 
This could be because you hit a bug. It is also possible that this binary
 
or one of the libraries it was linked against is corrupt, improperly built,
 
or misconfigured. This error can also be caused by malfunctioning hardware.
 
 
 
To report this bug, see https://mariadb.com/kb/en/reporting-bugs
 
 
 
We will try our best to scrape up some info that will hopefully help
 
diagnose the problem, but since we have already crashed,
 
something is definitely wrong and this may fail.
 
 
 
Server version: 10.0.29-MariaDB
 
key_buffer_size=402653184
 
read_buffer_size=2097152
 
max_used_connections=211
 
max_threads=8194
 
thread_count=91
 
It is possible that mysqld could use up to
 
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 84462163 K  bytes of memory
 
Hope that's ok; if not, decrease some variables in the equation.
 
 
 
Thread pointer: 0x2b353bd7e008
 
Attempting backtrace. You can use the following information to find out
 
where mysqld died. If you see no messages after this, something went
 
terribly wrong...
 
170724 17:40:23 mysqld_safe Number of processes running now: 0
 
170724 17:40:23 mysqld_safe mysqld restarted
 
170724 17:40:24 [Note] /MARIA/mariadb/bin/mysqld (mysqld 10.0.29-MariaDB) starting as process 30193 ...
 
2017-07-24 17:40:24 2b3670579600 InnoDB: Warning: Using innodb_additional_mem_pool_size is DEPRECATED. This option may be removed in future releases, together with the option innodb_use_sys_malloc a
 
nd with the InnoDB's internal memory allocator.

It happened two times for 3 month.
I don't know why this happened.
Please help me.

 Comments   
Comment by Alice Sherepa [ 2017-07-26 ]

try to configure tmpdir to custom location, i.e. not /tmp. Error 71 means that file not found, maybe it was deleted somehow.

Comment by Elena Stepanova [ 2017-08-02 ]

Unless there were other errors before the quoted fragment, the error most likely means that

  • creation of a temporary table was triggered by a user connection;
  • the table was created seemingly successfully;
  • upon opening the table right after creation, the data file couldn't be found.

The files are created and opened by OS means. There is a tiny window between creation and opening. As alice suggested above, it's hypothetically possible that something (e.g. some cleanup process) manages to delete the newly created file at this very moment. It's also possible, and I guess more likely, that some sort of a file system or OS or VM problem occurs, and the file doesn't really get created (yet) by the time the server attempts to open it.

At this point, InnoDB decides that the safer way is to abort, and shuts down with "Cannot continue operation". It's reproducible rather easily by injecting an error at a certain stage.

It does not normally cause complaints about virtual method calls and eventual SIGABRT, but their source is unclear – whether they come from the server or from the system.

Please note that CentOS 5 is extremely old and EOL-ed both by CentOS and MariaDB, it is not supported anymore. I would recommend to upgrade to something more recent, especially since you are using a VM, it shouldn't be too difficult.

Comment by JEON YANG BAIK [ 2017-08-30 ]

Once I changed tmp directory path from /tmp to /data a month ago, it has still working normally.

Could I not set the tmpdir of MariaDB 10.0 to /tmp on Centos 5?

Comment by Elena Stepanova [ 2017-09-27 ]

Generally there is no limitation on setting tmpdir to /tmp.
Possibly /tmp has a different file system, and possibly also file system type. Maybe the file system is corrupt, maybe file system type is buggy. Or maybe something else writes to /tmp and cleans it up. CentOS 5 is not supported anymore neither by CentOS itself nor by MariaDB, so there is no point trying to figure out which exactly kind of system problem causes the failure. If you keep encountering it on higher versions of OS, please let us know.

Generated at Thu Feb 08 08:05:10 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.