一、问题描述

某次一线反应，某主库表全部丢失，查看为空，登陆主机查看mysqld.log后报错：Operating system error number 2 in a file operation数据目录OS重装后修改过，但只是指向方式不同，目录还是同一目录，另怀疑被入侵遭删除，但也没发现相关异常；

关联资源：superuser、mariadb-kb、mysql5.7

二、分析如下

1、手动执行mysql_safe启动，查看日志报错如下：

2024-07-08T06:33:43.608829Z 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details).
2024-07-08T06:33:43.608961Z 0 [Note] --secure-file-priv is set to NULL. Operations related to importing and exporting data are disabled
2024-07-08T06:33:43.609008Z 0 [Note] ./bin/mysqld-debug (mysqld 5.7.44-debug-log) starting as process 2575072 ...
2024-07-08T06:33:43.616778Z 0 [Note] InnoDB: PUNCH HOLE support available
2024-07-08T06:33:43.616839Z 0 [Note] InnoDB: !!!!!!!! UNIV_DEBUG switched on !!!!!!!!!
2024-07-08T06:33:43.616850Z 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2024-07-08T06:33:43.616857Z 0 [Note] InnoDB: Uses event mutexes
2024-07-08T06:33:43.616865Z 0 [Note] InnoDB: GCC builtin __sync_synchronize() is used for memory barrier
2024-07-08T06:33:43.616873Z 0 [Note] InnoDB: Compressed tables use zlib 1.2.13
2024-07-08T06:33:43.616880Z 0 [Note] InnoDB: Using Linux native AIO
2024-07-08T06:33:43.619317Z 0 [Note] InnoDB: Number of pools: 1
2024-07-08T06:33:43.619521Z 0 [Note] InnoDB: Using CPU crc32 instructions
2024-07-08T06:33:43.621668Z 0 [Note] InnoDB: Initializing buffer pool, total size = 128M, instances = 1, chunk size = 128M
2024-07-08T06:33:43.683417Z 0 [Note] InnoDB: Completed initialization of buffer pool
2024-07-08T06:33:43.686321Z 0 [Note] InnoDB: page_cleaner coordinator priority: -20
2024-07-08T06:33:43.700000Z 0 [Note] InnoDB: Highest supported file format is Barracuda.
2024-07-08T06:33:43.766525Z 0 [ERROR] InnoDB: Operating system error number 2 in a file operation.
2024-07-08T06:33:43.766588Z 0 [ERROR] InnoDB: The error means the system cannot find the path specified.
2024-07-08T06:33:43.766599Z 0 [ERROR] InnoDB: If you are installing InnoDB, remember that you must create directories yourself, InnoDB does not creat
e them.
2024-07-08T06:33:43.766608Z 0 [ERROR] InnoDB: Cannot open datafile for read-only: './mysql/engine_cost.ibd' OS error: 71
2024-07-08T06:33:43.766639Z 0 [ERROR] InnoDB: Operating system error number 2 in a file operation.
2024-07-08T06:33:43.766648Z 0 [ERROR] InnoDB: The error means the system cannot find the path specified.
2024-07-08T06:33:43.766655Z 0 [ERROR] InnoDB: If you are installing InnoDB, remember that you must create directories yourself, InnoDB does not creat
e them.
2024-07-08T06:33:43.766665Z 0 [ERROR] InnoDB: Could not find a valid tablespace file for `mysql/engine_cost`. Please refer to http://dev.mysql.com/do
c/refman/5.7/en/innodb-troubleshooting-datadict.html for how to resolve the issue.
2024-07-08T06:33:43.766675Z 0 [Warning] InnoDB: Ignoring tablespace `mysql/engine_cost` because it could not be opened.
2024-07-08T06:33:43.766785Z 0 [ERROR] InnoDB: Operating system error number 2 in a file operation.
2024-07-08T06:33:43.766803Z 0 [ERROR] InnoDB: The error means the system cannot find the path specified.
2024-07-08T06:33:43.766811Z 0 [ERROR] InnoDB: If you are installing InnoDB, remember that you must create directories yourself, InnoDB does not creat
e them.
2024-07-08T06:33:43.766819Z 0 [ERROR] InnoDB: Cannot open datafile for read-only: './mysql/gtid_executed.ibd' OS error: 71
2024-07-08T06:33:43.766827Z 0 [ERROR] InnoDB: Operating system error number 2 in a file operation.
…………
mysqld: Table 'mysql.plugin' doesn't exist
2024-07-08T06:34:15.781770Z 0 [ERROR] Can't open the mysql.plugin table. Please run mysql_upgrade to create it.
2024-07-08T06:34:15.781880Z 0 [ERROR] Too many arguments (first extra is 'start').
2024-07-08T06:34:15.781889Z 0 [Note] Use --verbose --help to get a list of available options!
2024-07-08T06:34:15.781893Z 0 [ERROR] Aborting

如上所示，找不到innodb 数据字典文件*.ibd的，关于此类报错，官网解释如下：

With innodb_file_per_table enabled (the default), the following messages may appear at startup if a file-per-table tablespace file (.ibd file) is missing:

[ERROR] InnoDB: Operating system error number 2 in a file operation.
[ERROR] InnoDB: The error means the system cannot find the path specified.
[ERROR] InnoDB: Cannot open datafile for read-only: './test/t1.ibd' OS error: 71
[Warning] InnoDB: Ignoring tablespace `test/t1` because it could not be opened.

另外，ibdata1文件（660）作为记录InnoDB的共有表空间；数据库突然崩溃导致的该文件损坏也会导致如上报错；对应的独享表空间[mysqld]增加innodb_file_per_table；

2、强制innodb恢复，用以备份表

[mysqld]
innodb_force_recovery = 1  #值从0-6，默认0，安全值建议为1即可，不大于3，

3、mysql普通用户启动，参见Run MySQL as a Normal User

4、恢复

因现场环境被别人重装，损坏的数据删除，未能进一步排查恢复；但可参考如下：

1、备份原有的数据文件data为data_old
2、重启数据库，重新初始化重建，mysql表空间，
3、恢复源数据库数据目录到新的data下面，重启数据库验证

5、附录：
另外如果是多主环境，可配置如下，避免多主写入，带有自增列的记录导致自增列冲突的问题

auto_increment_offset = 2  #自增列偏移量，默认为1，即起始值
auto_increment_increment = 2  #自增量
#另Innodb参数
innodb_data_home_dir = /usr/local/mysql/data/
innodb_data_file_path = ibdata1:10M:autoextend
innodb_log_group_home_dir = /usr/local/mysql/data/
innodb_log_arch_dir = /usr/local/mysql/data/
innodb_additional_mem_pool_size = 16M
innodb_buffer_pool_size = 512M