隨著國產數據庫Polardb的推廣及應用，數據庫故障開始增多，今天給大家分享一篇Polardb故障應急處理思路。10月下旬，應用反饋某模塊登陸異常，查看數據庫有大量進程active狀態，持續時間超過30分鐘，通過kill進程并重啟應用無法恢復，通過與業務協商、嘗試通過重啟數據庫實例恢復。

查看日志文件postgresql-10-26_024511.log，日志中存在以下日志：

ERROR: dsa_area could not attach to segment

涉及進程pid= 51789 53315 53316，其中53316和53315進程都退出了，日志如下：

  53315              2020-10-26 03:07:11 UTC XX000                      ERROR:  dsa_area could not attach to segment

 53316            2020-10-26 03:07:11 UTC XX000                      ERROR:  dsa_area could not attach to segment

......

 53314            2020-10-26 03:07:11 UTC XX000                      FATAL:  cannot unpin a segment that is not pinned

   54             2020-10-26 03:07:11 UTC 00000                       LOG: background worker "parallel worker" (PID53316) exited withexit code 1

   54             2020-10-26 03:07:11 UTC 00000                       LOG: background worker "parallel worker" (PID53315) exitedwith exit code 1

而pid=51789的進程沒有正常退出，很有可能是hang，并且影響到其他的進程。

……

51789 192.168.7.36(41423)  ywxt  ywxt  2020-10-26 03:07:07 UTC  00000  13 0  0  0    LOG:  execute S_1: COMMIT

 51789 192.168.7.36(41423)  ywxt  ywxt  2020-10-26 03:07:09 UTC  00000  43 0  0  0    LOG:  statement: BEGIN

 51789 192.168.7.36(41423)  ywxt  ywxt  2020-10-26 03:07:09 UTC  00000  59 1  0  40    LOG:  execute :  SELECTordersum,onestatus,twostatus,threestatus,fourstatus,archive FROM(……………… )

 51789 192.168.7.36(41423)  ywxt  ywxt  2020-10-26 03:07:09 UTC  00000  22 0  0  0    LOG:  execute S_1: COMMIT

 51789 192.168.7.36(41423)  ywxt  ywxt  2020-10-26 03:07:11 UTC  00000  35 0  0  0    LOG:  statement: BEGIN

 51789 192.168.7.36(41423)  ywxt  ywxt  2020-10-26 03:07:11 UTC  00000 69701  1  0  170881    LOG:  execute : SELECT count(1)FROM (SELECT ………………) XTABLE

 51789 192.168.7.36(41423)  ywxt  ywxt  2020-10-26 03:07:11 UTC  XX000           ERROR:  dsa_area could not attach to segment

 51789 192.168.7.36(41423)  ywxt  ywxt  2020-10-26 03:07:11 UTC  XX000           STATEMENT:  SELECT * FROM (SELECT RET.*, ROWNUM AS FSDPRN FROM(………………)WHERE FSDPRN BETWEEN 1 AND 10

 51789 192.168.7.36(41423)  ywxt  ywxt  2020-10-26 03:07:11 UTC  XX000           LOG:  statement: SELECT * FROM (SELECT RET.*, ROWNUM AS FSDPRNFROM (………………) WHERE FSDPRN BETWEEN 1 AND 10

 params:$1 = d6d9c702-8aa9-4e9b-8cc2-4d756b1adaa2, $2 = 202009, $3 =HLW, $4 = 15255864202, $5 =1a251789-9cc9-4e77-84cd-6b651da3be5a, $6 = 2020-09-14 11:06:18

 params:$1 = 40f12349-97cc-4945-94b1-e7fda75cc984, $2 = 202009, $3 =HLW, $4 = 18326858272, $5 =eaceb00f-9f78-4c6f-8cee-517898f0b200, $6 = 2020-09-15 15:16:40

查過文檔發現是屬于11.3之前已知的問題：

• Fix race conditions in management of dynamic shared memory (Thomas Munro)

These could lead to “dsa_area couldnot attach to segment” or “cannot unpin a segment thatis not pinned” errors.

if you just happen to hit PostgreSQL bugsuch as this one that was fixed in v11.3:

Once this problem occurs, your databasewill appear to be running, and can possibly be used to service somequeries... but most queries will hang, and eventually yourapplication servers will timeout causing application downtime.

通過重啟實例恢復，由于重啟過程中，實例無法正常停止，導致管控重啟流程超時失敗，通過手工修復拉起實例。手動流程如下：

• 將長時間運行的流程直接設置為中斷

cat/root/.kube/config |grep client-certificate-data|awk {print$2}|base64 -d >/root/.kube/admin.crt

cat/root/.kube/config |grep client-key-data|awk {print $2}|base64 -d>/root/.kube/admin.key

KUBEAPISERVER=127.0.0.1

PPASCLUSTERNAME=polar-xxxxx

curl-k --cert /root/.kube/admin.crt --key /root/.kube/admin.keyhttps://$KUBEAPISERVER:6443/apis/ppas.polardb.aliyun.com/v1beta1/namespaces/default/ppasclusters/$PPASCLUSTERNAME| python -m json.tool  > polar-single.yaml

vimpolar-single.yaml

找到clusterStatus:xxxxxx，改為clusterStatus:Interrupt，保存

curl-X PUT -H "Content-Type: application/json" -k --cert/root/.kube/admin.crt --key /root/.kube/admin.keyhttps://$KUBEAPISERVER:6443/apis/ppas.polardb.aliyun.com/v1beta1/namespaces/default/ppasclusters/$PPASCLUSTERNAME/status--data @polar-single.yaml

• 控制臺查看集群列表，看是否進入到中斷狀態。

• 清理ins_lock文件

kubectlget ppascluster polar-xxxx -o yaml

找到status- masterCluster - rwHostIns - insId，這個值是實例ID。

在三個計算節點找以下路徑：/data/polardb_ppas/{insId}/data，和直接進入容器查找文件效果一樣。

找到ins_lock文件，刪除。

• 將中斷的集群進行重建

kubectledit ppascluster polar-xxxx

在annotations下面增加下面兩項：

polarbox.interrupt.recover:"T"

polarbox.interrupt.previous.status:"RebuildingRW"

• 刷新控制臺，查看集群狀態進入重建中，等待重建完成

1、此問題是并行查詢過程中觸發，短期內通過關閉并行查詢來規避，目前生產已經關閉并行。

2、Polardb應急手工拉起流程的熟悉。

3、相關工具配套的場景沉淀。