mysql errorcode 28: no space left on the device

事件背景

今天一很久没联系的朋友突然QQ 上联系,说他自己服务器的 mysql 不知道因为什么原因,而无法启动。简单的询问了下,以为是他没按正常流程重启服务器导致. 让其提供服务器信息,查看mysql 错误日志,发现:

1
2
3
4
5
6
7
8
9
10
11
12
13
2019-04-20T02:22:36.885083Z 0 [Warning] 'NO_ZERO_DATE', 'NO_ZERO_IN_DATE' and 'ERROR_FOR_DIVISION_BY_ZERO' sql modes should be used with strict mode. They will be merged with strict mode in a future release.
2019-04-20T02:22:36.885096Z 0 [Warning] 'NO_AUTO_CREATE_USER' sql mode was not set.
2019-04-20T02:22:36.885127Z 0 [Note] --secure-file-priv is set to NULL. Operations related to importing and exporting data are disabled
2019-04-20T02:22:36.885182Z 0 [Note] /www/server/mysql/bin/mysqld (mysqld 5.7.25-log) starting as process 7038 ...
2019-04-20T02:22:36.890224Z 0 [Warning] Can't create test file /www/server/data/iZj6ce3zah39uzm0ea9hafZ.lower-test
2019-04-20T02:22:36.890254Z 0 [Warning] Can't create test file /www/server/data/iZj6ce3zah39uzm0ea9hafZ.lower-test
mysqld: File './mysql-bin.~rec~' not found (Errcode: 28 - No space left on device)
2019-04-20T02:22:36.892667Z 0 [ERROR] MYSQL_BIN_LOG::open_purge_index_file failed to open register file.
2019-04-20T02:22:36.892673Z 0 [ERROR] MYSQL_BIN_LOG::open_index_file failed to sync the index file.
2019-04-20T02:22:36.892677Z 0 [ERROR] Aborting

2019-04-20T02:22:36.892687Z 0 [Note] Binlog end
2019-04-20T02:22:36.892832Z 0 [Note] /www/server/mysql/bin/mysqld: Shutdown complete

解决方法:

查看错误日志:

根据错误日志提供的信息,使用df -lh 查看磁盘空间,发现空间才使用 40% ,进一步排除因磁盘空间不足而致 mysql 无法启动。接着使用 df -i 查看 inode 发现:

1
2
3
4
5
6
7
8
[root@iZj6ce3zah39uzm0ea9hafZ ~]# df -i
Filesystem Inodes IUsed Ifree IUse% Mounted on
/dev/vda1 2621440 2621440 1 100% /
devtmpfs 232675 327 232348 1% /dev
tmpfs 235324 2 235322 1% /dev/shm
tmpfs 235324 448 234876 1% /run
tmpfs 235324 16 235308 1% /sys/fs/cgroup
tmpfs 235324 1 235323 1% /run/user/0

查看inode 占用最高的目录

知道了是因为 inodes 引起的故障,那么我们就进入 / 目录,执行下面的命令,查看哪个目录占用 inodes 最高:

1
for i in /*; do echo $i; find $i | wc -l; done

删除无用的文件

判断了占用 inodes 最高的目录后,直接进入该目录,删除无用的文件。

问题解决

再次尝试启动 mysql 服务,成功!

参考文章:http://www.dahouduan.com/2014/12/19/linux-inode-full/

坚持原创技术分享,您的支持将鼓励我继续创作!