摘要:問題描述今天在啟動容器的時候發現一段時間后宿主機上所有的容器的根目錄全部變成了只讀,并且宿主機日志報磁盤相關的錯容器內結果如下宿主機報錯如下
問題描述
今天在啟動docker容器的時候發現一段時間后宿主機上所有的容器的根目錄全部變成了只讀,并且宿主機message日志報磁盤相關的錯
容器內mount結果如下
[root@zk-1 ~]# mount /dev/mapper/docker-253:0-4298664622-7830c39693a73c13e80cf2a22a46558b22adcc5adf94ca1c893f44ece878601e on / type ext4 (ro,relatime,stripe=16,data=ordered) proc on /proc type proc (rw,nosuid,nodev,noexec,relatime) tmpfs on /dev type tmpfs (rw,nosuid,mode=755) devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=666) shm on /dev/shm type tmpfs (rw,nosuid,nodev,noexec,relatime,size=65536k) mqueue on /dev/mqueue type mqueue (rw,nosuid,nodev,noexec,relatime) sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime) tmpfs on /sys/fs/cgroup type tmpfs (rw,nosuid,nodev,noexec,relatime) cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd) cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset) cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpu,cpuacct) cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio) cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory) cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices) cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer) cgroup on /sys/fs/cgroup/net_cls,net_prio type cgroup (rw,nosuid,nodev,noexec,relatime,net_cls,net_prio) cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,perf_event) cgroup on /sys/fs/cgroup/hugetlb type cgroup (rw,nosuid,nodev,noexec,relatime,hugetlb) /dev/mapper/centos-root on /nfsc type xfs (rw,relatime,attr2,inode64,noquota) tmpfs on /run/secrets type tmpfs (rw,nosuid,nodev,noexec,relatime) /dev/mapper/centos-root on /etc/resolv.conf type xfs (rw,relatime,attr2,inode64,noquota) /dev/mapper/centos-root on /etc/hostname type xfs (rw,relatime,attr2,inode64,noquota) /dev/mapper/centos-root on /etc/hosts type xfs (rw,relatime,attr2,inode64,noquota)
宿主機報錯如下
Jan 13 15:06:01 docker2 systemd: Starting Session 6 of user root. Jan 13 15:06:01 docker2 systemd: Started Session 6 of user root. Jan 13 15:06:31 docker2 systemd: Starting Session 7 of user root. Jan 13 15:06:31 docker2 systemd-logind: New session 7 of user root. Jan 13 15:06:31 docker2 systemd: Started Session 7 of user root. Jan 13 15:07:01 docker2 systemd: Starting Session 8 of user root. Jan 13 15:07:01 docker2 systemd: Started Session 8 of user root. Jan 13 15:07:38 docker2 kernel: device-mapper: thin: 253:3: reached low water mark for data device: sending event. Jan 13 15:07:44 docker2 kernel: device-mapper: thin: 253:3: switching pool to out-of-data-space (queue IO) mode Jan 13 15:08:01 docker2 systemd: Starting Session 9 of user root. Jan 13 15:08:01 docker2 systemd: Started Session 9 of user root. Jan 13 15:08:44 docker2 kernel: device-mapper: thin: 253:3: switching pool to out-of-data-space (error IO) mode Jan 13 15:08:44 docker2 kernel: EXT4-fs warning (device dm-5): ext4_end_bio:332: I/O error -28 writing to inode 1320256 (offset 13077839872 size 8388608 starting block 9269408) Jan 13 15:08:44 docker2 kernel: Buffer I/O error on device dm-5, logical block 9269408 Jan 13 15:08:44 docker2 kernel: Aborting journal on device dm-4-8. Jan 13 15:08:44 docker2 kernel: Buffer I/O error on device dm-5, logical block 9269409 Jan 13 15:08:44 docker2 kernel: Buffer I/O error on device dm-5, logical block 9269410 Jan 13 15:08:44 docker2 kernel: EXT4-fs error (device dm-4): ext4_journal_check_start:56: Detected aborted journal Jan 13 15:08:44 docker2 kernel: EXT4-fs (dm-4): Remounting filesystem read-only Jan 13 15:08:44 docker2 kernel: Buffer I/O error on device dm-5, logical block 9269411 Jan 13 15:08:44 docker2 kernel: Buffer I/O error on device dm-5, logical block 9269412 Jan 13 15:08:44 docker2 kernel: Buffer I/O error on device dm-5, logical block 9269413 Jan 13 15:08:44 docker2 kernel: Buffer I/O error on device dm-5, logical block 9269414 Jan 13 15:08:44 docker2 kernel: Buffer I/O error on device dm-5, logical block 9269415 Jan 13 15:08:44 docker2 kernel: Buffer I/O error on device dm-5, logical block 9269416 Jan 13 15:08:44 docker2 kernel: Buffer I/O error on device dm-5, logical block 9269417 Jan 13 15:08:44 docker2 kernel: EXT4-fs warning (device dm-5): ext4_end_bio:332: I/O error -28 writing to inode 1320256 (offset 13077839872 size 8388608 starting block 9269424) Jan 13 15:08:44 docker2 kernel: EXT4-fs warning (device dm-5): ext4_end_bio:332: I/O error -28 writing to inode 1320256 (offset 13077839872 size 8388608 starting block 9269440) Jan 13 15:08:44 docker2 kernel: EXT4-fs warning (device dm-5): ext4_end_bio:332: I/O error -28 writing to inode 1320256 (offset 13077839872 size 8388608 starting block 9269456) Jan 13 15:08:44 docker2 kernel: EXT4-fs warning (device dm-5): ext4_end_bio:332: I/O error -28 writing to inode 1320256 (offset 13077839872 size 8388608 starting block 9269472) Jan 13 15:08:44 docker2 kernel: EXT4-fs warning (device dm-5): ext4_end_bio:332: I/O error -28 writing to inode 1320256 (offset 13077839872 size 8388608 starting block 9269488) Jan 13 15:08:44 docker2 kernel: EXT4-fs warning (device dm-5): ext4_end_bio:332: I/O error -28 writing to inode 1320256 (offset 13077839872 size 8388608 starting block 9269504) Jan 13 15:08:44 docker2 kernel: EXT4-fs warning (device dm-5): ext4_end_bio:332: I/O error -28 writing to inode 1320256 (offset 13077839872 size 8388608 starting block 9269520) Jan 13 15:08:44 docker2 kernel: EXT4-fs warning (device dm-5): ext4_end_bio:332: I/O error -28 writing to inode 1320256 (offset 13077839872 size 8388608 starting block 9269536) Jan 13 15:08:44 docker2 kernel: EXT4-fs warning (device dm-5): ext4_end_bio:332: I/O error -28 writing to inode 1320256 (offset 13077839872 size 8388608 starting block 9269552) Jan 13 15:08:44 docker2 kernel: Aborting journal on device dm-5-8. Jan 13 15:08:44 docker2 kernel: EXT4-fs error (device dm-5) in ext4_da_write_end:2782: IO failure Jan 13 15:08:44 docker2 kernel: EXT4-fs error (device dm-5): ext4_journal_check_start:56: Detected aborted journal Jan 13 15:08:44 docker2 kernel: EXT4-fs (dm-5): Remounting filesystem read-only Jan 13 15:08:44 docker2 kernel: EXT4-fs error (device dm-5) in ext4_do_update_inode:4504: Journal has aborted Jan 13 15:08:44 docker2 kernel: EXT4-fs error (device dm-5): mpage_map_and_submit_extent:2229: comm kworker/u98:3: Failed to mark inode 1320256 dirty Jan 13 15:08:44 docker2 kernel: EXT4-fs error (device dm-5) in ext4_writepages:2520: IO failure
第一反映是查看磁盤空間使用情況
[root@zk-1 ~]# df -Th Filesystem Type Size Used Avail Use% Mounted on /dev/mapper/docker-253:0-4298664622-7830c39693a73c13e80cf2a22a46558b22adcc5adf94ca1c893f44ece878601e ext4 99G 49G 46G 52% / tmpfs tmpfs 126G 0 126G 0% /dev shm tmpfs 64M 0 64M 0% /dev/shm tmpfs tmpfs 126G 0 126G 0% /sys/fs/cgroup /dev/mapper/centos-root xfs 1.7T 113G 1.6T 7% /nfsc tmpfs tmpfs 126G 0 126G 0% /run/secrets /dev/mapper/centos-root xfs 1.7T 113G 1.6T 7% /etc/resolv.conf /dev/mapper/centos-root xfs 1.7T 113G 1.6T 7% /etc/hostname /dev/mapper/centos-root xfs 1.7T 113G 1.6T 7% /etc/hosts
根目錄下還剩余46G未使用;感覺甚是怪異,于是網上搜索很多資料終于有了相關解釋
參考網絡上的一片文章:http://jpetazzo.github.io/201...
docker服務啟動的時候默認會創建一個107.4G的data文件,而后啟動的容器的所有更改內容全部存儲至這個data文件中;也就是說當容器內產生的相關data數據超過107.4G后容器就再也沒有多余的空間可用,從而導致所有容器的根目錄變為只讀!
宿主機的docker info信息如下
[root@docker2 ~]# docker info Containers: 169 Images: 1672 Storage Driver: devicemapper Pool Name: docker-253:0-4298664622-pool Pool Blocksize: 65.54 kB Backing Filesystem: xfs Data file: /dev/loop0 Metadata file: /dev/loop1 Data Space Used: 107.4 GB Data Space Total: 107.4 GB Data Space Available: 0 B Metadata Space Used: 137.4 MB Metadata Space Total: 2.147 GB Metadata Space Available: 2.01 GB Udev Sync Supported: true Deferred Removal Enabled: false Data loop file: /var/lib/docker/devicemapper/devicemapper/data Metadata loop file: /var/lib/docker/devicemapper/devicemapper/metadata Library Version: 1.02.107-RHEL7 (2015-10-14) Execution Driver: native-0.2 Logging Driver: json-file Kernel Version: 4.2.1-1.el7.elrepo.x86_64 Operating System: CentOS Linux 7 (Core) CPUs: 40 Total Memory: 251.9 GiB Name: docker2.stg.1qianbao.com ID: JMZF:IQ6H:RDBK:XNSN:W3IO:ZAQH:RRFB:XRIT:4I72:KOKD:R34K:FD5L WARNING: bridge-nf-call-iptables is disabled WARNING: bridge-nf-call-ip6tables is disabled
由于我的容器比較多(169個jboss應用),故直接導致整個環境不可用。
解決方案停止docker服務
service docker stop
刪除/var/lib/docker下面的所有文件(刪除后你的鏡像和容器都沒有了,建議將有用的鏡像先備份或者上傳至倉儲里面)
rm -rf /var/lib/docker/*
使用更大的文件或磁盤或邏輯卷創建/var/lib/docker/devicemapper/devicemapper/data文件
使用文件:dd if=/dev/zero of=/var/lib/docker/devicemapper/devicemapper/data bs=1G count=0 seek=1000這樣將會創建一個虛擬的1000G大小的data文件,如果不加seek參數count直接為1000的話則是創建了一個結結實實的1000G的文件
使用磁盤:ln -s /dev/sdb /var/lib/docker/devicemapper/devicemapper/data
使用邏輯卷:ln -s /dev/mapper/centos-dockerdata /var/lib/docker/devicemapper/devicemapper/data
我用的是第一種使用文件的方法創建了一個1.6T的虛擬文件
mkdir -p /var/lib/docker/devicemapper/devicemapper/ dd if=/dev/zero of=/var/lib/docker/devicemapper/devicemapper/data bs=1G count=0 seek=1600
創建完成后啟動docker服務
service docker start
這時再看下docker info中的data池
[root@docker2 ~]# docker info Containers: 169 Images: 1701 Storage Driver: devicemapper Pool Name: docker-253:0-2355438-pool Pool Blocksize: 65.54 kB Backing Filesystem: xfs Data file: /dev/loop0 Metadata file: /dev/loop1 Data Space Used: 90.9 GB Data Space Total: 1.611 TB Data Space Available: 1.52 TB Metadata Space Used: 147.5 MB Metadata Space Total: 2.147 GB Metadata Space Available: 2 GB Udev Sync Supported: true Deferred Removal Enabled: false Data loop file: /var/lib/docker/devicemapper/devicemapper/data Metadata loop file: /var/lib/docker/devicemapper/devicemapper/metadata Library Version: 1.02.107-RHEL7 (2015-10-14) Execution Driver: native-0.2 Logging Driver: json-file Kernel Version: 4.2.1-1.el7.elrepo.x86_64 Operating System: CentOS Linux 7 (Core) CPUs: 40 Total Memory: 251.9 GiB Name: docker2.stg.1qianbao.com ID: JMZF:IQ6H:RDBK:XNSN:W3IO:ZAQH:RRFB:XRIT:4I72:KOKD:R34K:FD5L WARNING: bridge-nf-call-iptables is disabled WARNING: bridge-nf-call-ip6tables is disabled
此后,你的data文件有多大就決定了你的宿主機上所有容器可用的空間的大小!
另一方面:也可以通過docker啟動參數的--storage-opt選項來限制每個容器初始化的磁盤大小,如-storage-opt dm.basesize=80G 這樣每個容器啟動后起根目錄的總空間就是80G
[root@zk-1 ~]# df -Th Filesystem Type Size Used Avail Use% Mounted on /dev/mapper/docker-253:0-27661746-8b7f953fb4759982ad82235c27e39dfe7190b55180d63cbcf3aa2fdc6569d43a ext4 79G 785M 74G 2% / tmpfs tmpfs 126G 0 126G 0% /dev shm tmpfs 64M 0 64M 0% /dev/shm tmpfs tmpfs 126G 0 126G 0% /sys/fs/cgroup tmpfs tmpfs 126G 0 126G 0% /run/secrets /dev/mapper/centos-root xfs 1.5T 338G 1.2T 23% /wls/wls81/zookeeper.out /dev/mapper/centos-root xfs 1.5T 338G 1.2T 23% /etc/resolv.conf /dev/mapper/centos-root xfs 1.5T 338G 1.2T 23% /etc/hostname /dev/mapper/centos-root xfs 1.5T 338G 1.2T 23% /etc/hosts
文章版權歸作者所有,未經允許請勿轉載,若此文章存在違規行為,您可以聯系管理員刪除。
轉載請注明本文地址:http://specialneedsforspecialkids.com/yun/26519.html
摘要:而實際上在宿主機中也會同步啟動一個進程,其在宿主機中是。如果其中的某一個容器正在執行密集型的任務,那么它就會影響其他容器的任務執行效率,導致多個容器相互影響并且搶占資源。 作者:榮幸 為什么是容器 如果問你現在最熱門的服務器端技術什么?想必很多人會不假思索的說是容器! 容器技術實際上并不是一個新鮮的名詞,現在大家一提到容器馬上想到的就是Docker,但是容器這個詞并不是Docker公司...
摘要:本文立足于新手,從容器和虛擬機兩個大的概念入手,由淺入深,由宏轉微,為我們解析了的方方面面。出于各種考量與需求,容器在外觀上與虛擬機非常相似。大家可以在數秒鐘內完成容器的創建與運行,而虛擬機則由于需要引導完整的虛擬操作系統而耗費更多時間。 當小數看到這篇文章時內心是激動的,因為或許介紹Docker容器的文章有無數,但是如此清晰易懂、對小白如此友好的卻不多見。本文立足于新手,從容器和虛擬...
摘要:解決了服務器應用快速部署的問題。解決方案是自己裝一個第二點,確保鏡像服務器跑起來。你需要將真機的端口和虛擬機端口進行綁定,這樣可以通過訪問鏡像服務器在真機上的端口,訪問到數據庫端口,進入鏡像之中。 關于Docker的基本原理 原理上, 深入淺出Docker 一文講的語言會更科學,我只撿不一定準確但直觀而符合表面理解的一些講一下。 Docker提供了基于操作系統級和應用級虛擬化的應用部署...
摘要:解決了服務器應用快速部署的問題。解決方案是自己裝一個第二點,確保鏡像服務器跑起來。你需要將真機的端口和虛擬機端口進行綁定,這樣可以通過訪問鏡像服務器在真機上的端口,訪問到數據庫端口,進入鏡像之中。 關于Docker的基本原理 原理上, 深入淺出Docker 一文講的語言會更科學,我只撿不一定準確但直觀而符合表面理解的一些講一下。 Docker提供了基于操作系統級和應用級虛擬化的應用部署...
摘要:使用推出的語言開發實現,基于內核的,,以及等技術。初步了解容器與虛擬機的區別原理不同上圖是關于和傳統虛擬機區別的截圖。從而實現模擬更改鏡像的作用。下面的實踐以為例,。指令讓外界能通過容器的端口進行網絡通信。使用可以啟動關閉容器。 Docker 是一個能讓程序跑在一個它無法感知的、用于隔絕外界環境里的容器的工具。 Docker 簡介 最初是 dotCloud 公司創始人 Solomon ...
閱讀 2574·2021-10-08 10:04
閱讀 2735·2021-09-06 15:02
閱讀 792·2019-08-30 13:50
閱讀 1547·2019-08-30 13:21
閱讀 2586·2019-08-30 11:15
閱讀 2113·2019-08-29 17:19
閱讀 1574·2019-08-26 13:55
閱讀 1261·2019-08-26 10:15