failed to create runc console socket: mkdir /tmp/pty064803763: no space left on device: unknown

Got this error today when trying to log into a docker container

$ docker exec -it mysql /bin/bash
failed to create runc console socket: mkdir /tmp/pty064803763: no space left on device: unknown

Then I tried

$ df -h
Filesystem                 Size  Used Avail Use% Mounted on
devtmpfs                   3.9G     0  3.9G   0% /dev
tmpfs                      3.9G  4.0K  3.9G   1% /dev/shm
tmpfs                      3.9G  378M  3.6G  10% /run
tmpfs                      3.9G     0  3.9G   0% /sys/fs/cgroup
/dev/mapper/rootvg-rootlv  7.8G  128M  7.3G   2% /
/dev/mapper/rootvg-usrlv   9.8G  2.3G  7.0G  25% /usr
/dev/sda1                  976M  113M  797M  13% /boot
/dev/mapper/rootvg-optlv   2.0G  1.5G  303M  84% /opt
/dev/mapper/rootvg-homelv  976M  3.9M  905M   1% /home
/dev/mapper/rootvg-tmplv   2.0G  432M  1.4G  24% /tmp
/dev/mapper/rootvg-varlv   7.8G  4.2G  3.3G  56% /var
/dev/sdc                   246G  1.1G  233G   1% /app
/dev/sdb1                   16G  2.1G   13G  14% /mnt/resource
tmpfs                      797M     0  797M   0% /run/user/48081

which did not indicate any problem. /tmp is only 24% used.

$ df -i
Filesystem                  Inodes  IUsed    IFree IUse% Mounted on
devtmpfs                   1016516    431  1016085    1% /dev
tmpfs                      1019390      2  1019388    1% /dev/shm
tmpfs                      1019390    856  1018534    1% /run
tmpfs                      1019390     16  1019374    1% /sys/fs/cgroup
/dev/mapper/rootvg-rootlv   524288   6703   517585    2% /
/dev/mapper/rootvg-usrlv    655360  86575   568785   14% /usr
/dev/sda1                    65536    343    65193    1% /boot
/dev/mapper/rootvg-optlv    131072  53934    77138   42% /opt
/dev/mapper/rootvg-homelv    65536    166    65370    1% /home
/dev/mapper/rootvg-tmplv    131072 131072        0  100% /tmp
/dev/mapper/rootvg-varlv    524288  26116   498172    5% /var
/dev/sdc                  16384000   3649 16380351    1% /app
/dev/sdb1                  1048576     13  1048563    1% /mnt/resource
tmpfs                      1019390      1  1019389    1% /run/user/48081

Above is showing all the inodes have been exhausted under /tmp. On listing /tmp I saw lot of files like this

#-> cat /tmp/tmp.01amFL2hH4
read_kt /etc/krb5.keytab
write_kt /tmp/tmp.kt

For now I manually deleted the files. And after that it works

$ docker exec -it mysql /bin/bash
root@48b2dea3706b:/home#
This entry was posted in Software. Bookmark the permalink.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s