打开gitlab发现报错500,一开始以为是网络问题,但是试了好几次发现不是,进入后台开始排查问题
gitlab-ctl restart 重启没有问题
[root@localhost ~]# gitlab-ctl status
run: alertmanager: (pid 3364) 5006471s; run: log: (pid 3348) 5006471s
run: gitaly: (pid 3355) 5006471s; run: log: (pid 3352) 5006471s
run: gitlab-monitor: (pid 82813) 6875s; run: log: (pid 3353) 5006471s
run: gitlab-workhorse: (pid 3360) 5006471s; run: log: (pid 3357) 5006471s
run: grafana: (pid 3373) 5006471s; run: log: (pid 3372) 5006471s
run: logrotate: (pid 110986) 2429s; run: log: (pid 3351) 5006471s
run: nginx: (pid 3359) 5006471s; run: log: (pid 3356) 5006471s
run: node-exporter: (pid 3362) 5006471s; run: log: (pid 3349) 5006471s
run: postgres-exporter: (pid 3363) 5006471s; run: log: (pid 3350) 5006471s
run: postgresql: (pid 3344) 5006471s; run: log: (pid 3341) 5006471s
run: prometheus: (pid 3378) 5006471s; run: log: (pid 3375) 5006471s
run: redis: (pid 3368) 5006471s; run: log: (pid 3366) 5006471s
run: redis-exporter: (pid 3347) 5006471s; run: log: (pid 3346) 5006471s
run: sidekiq: (pid 3370) 5006471s; run: log: (pid 3367) 5006471s
run: unicorn: (pid 3343) 5006471s; run: log: (pid 3342) 5006471s
gitlab-ctl status 当前状态没有问题
netstat -tnlp | grep 8898 没有问题
[root@localhost ~]# netstat -tnlp | grep 8898
tcp 0 0 0.0.0.0:8898 0.0.0.0:* LISTEN 3359/nginx: master
gitlab-ctl tail 发现问题
==> /var/log/gitlab/prometheus/current <==
2019-12-13_05:47:20.83161 level=warn ts=2019-12-13T04:12:25.157365383Z caller=scrape.go:867 component="scrape manager" scrape_pool=gitlab-workhorse target=http://localhost:9229/metrics msg="append failed" err="write to WAL: log samples: create new segment file: open /var/opt/gitlab/prometheus/data/wal/00000997: no space left on device"
2019-12-13_05:47:20.83162 level=warn ts=2019-12-13T04:12:50.082097379Z caller=manager.go:532 component="rule manager" group=Node msg="rule sample appending failed" err="write to WAL: log samples: create new segment file: open /var/opt/gitlab/prometheus/data/wal/00000997: no space left on device"
....
==> /var/log/gitlab/alertmanager/state <==
问题出现
no space left on device
,没有空间了!!!!
查看当前磁盘空间
通过命令df -h,发现系统各个文件夹的空寂,我们清理磁盘即可
Filesystem Size Used Avail Use% Mounted on
/dev/vda1 50G 12G 35G 26% /
devtmpfs 911M 0 911M 0% /dev
tmpfs 920M 24K 920M 1% /dev/shm
tmpfs 920M 476K 920M 1% /run
tmpfs 920M 0 920M 0% /sys/fs/cgroup
tmpfs 184M 0 184M 0% /run/user/0
overlay 50G 12G 35G 26% /var/lib/docker/overlay2/509b42fa7d1ebe1054a897459cfeea3423324f203258aabf26e5471e33b62ceb/merged
shm 64M 0 64M 0% /var/lib/docker/containers/5691ecf7f4fab7a142e6ca4043e59830fda0073397fc80534e790c69de62b380/mounts/shm
使用 du -sh查看linux硬盘各个文件占用空间统计
du -sh /var/lib/*|grep "G"
disk usage,是通过搜索文件来计算每个文件的大小然后累加,du能看到的文件只是一些当前存在的,没有被删除的。他计算的大小就是当前他认为存在的所有文件大小的累加和。
disk free,通过文件系统来快速获取空间大小的信息,当我们删除一个文件的时候,这个文件不是马上就在文件系统当中消失了,而是暂时消失了,当所有程序都不用时,才会根据OS的规则释放掉已经删除的文件,df记录的是通过文件系统获取到的文件的大小,他比du强的地方就是能够看到已经删除的文件,而且计算大小的时候,把这一部分的空间也加上了,更精确了。