最近由于公司業務發展到了瓶頸，原有的技術架構已經逐漸無法滿足業務開發和測試的需求，出現了應用測試環境搭建復雜，有許多套（真的很多很多）應用環境，應用在持續集成/持續交付也遇到了很大的困難，經過討論研究決定對應用和微服務進行容器化，這就是我首次直面docker和k8s的契機（好吧，我是菜鳥）

Kubernetes 是 Google 團隊發起的開源項目，它的目標是管理跨多個主機的容器，提供基本的部署，維護以及運用伸縮，主要實現語言為
Go 語言。
Kubernetes的特點：

易學：輕量級，簡單，容易理解
便攜：支持公有云，私有云，混合云，以及多種云平臺
可拓展：模塊化，可插拔，支持鉤子，可任意組合
自修復：自動重調度，自動重啟，自動復制

好吧，這是從別人ppt上拷下來的=。=
下面就正式開始部署我們自己的k8s吧

注：以下操作都是在root權限下執行的

安裝docker-ce，這里使用docker-ce-17.09.0.c版本，安裝方法見之前的教程

安裝Kubeadm
安裝 Kubeadm 首先我們要配置好阿里云的國內源，執行如下命令：

cat < /etc/yum.repos.d/kubernetes.repo 
[kubernetes] 
name=Kubernetes 
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64 
enabled=1 
gpgcheck=0 
EOF

之后，執行以下命令來重建yum緩存：

yum -y install epel-releaseyum 
clean all 
yum makecache

接下來需要安裝指定版本的Kubeadm（這里要安裝指定版本，因為后續依賴的鏡像由于有墻無法拉取，這里我們只有指定版本的鏡像），注意：這里是安裝指定版本的Kubeadm，k8s的版本更新之快完全超出你的想象！

yum -y install kubelet-1.11.0-0
yum -y install kubeadm-1.11.0-0
yum -y install kubectl-1.11.0-0
yum -y install kubernetes-cni

執行命令啟動Kubeadm服務：

systemctl enable kubelet && systemctl start kubelet

配置 Kubeadm 所用到的鏡像
這里是重中之重，因為在國內的原因，無法訪問到 Google 的鏡像庫，所以我們需要執行以下腳本來從 Docker Hub 倉庫中獲取相同的鏡像，并且更改 TAG 讓其變成與 Google 拉去鏡像一致。
新建一個 Shell 腳本，填入以下代碼之后保存。

#docker.sh
#!/bin/bash
images=(kube-proxy-amd64:v1.11.0 kube-scheduler-amd64:v1.11.0 kube-controller-manager-amd64:v1.11.0 kube-apiserver-amd64:v1.11.0 etcd-amd64:3.2.18 coredns:1.1.3 pause-amd64:3.1 kubernetes-dashboard-amd64:v1.8.3 k8s-dns-sidecar-amd64:1.14.9 k8s-dns-kube-dns-amd64:1.14.9 k8s-dns-dnsmasq-nanny-amd64:1.14.9 ) 
for imageName in ${images[@]} ; do
docker pull keveon/$imageName
docker tag keveon/$imageName k8s.gcr.io/$imageName
docker rmi keveon/$imageName
done
# 個人新加的一句，V 1.11.0 必加
docker tag da86e6ba6ca1 k8s.gcr.io/pause:3.1

保存后使用chmod命令賦予腳本執行權限

chmod -R 777 ./docker.sh

執行腳本拉取鏡像

sh docker.sh
#這里就開始了漫長的拉取鏡像之路 

關閉swap

sudo swapoff -a
#要永久禁掉swap分區，打開如下文件注釋掉swap那一行 
# sudo vi /etc/stab

關閉selinux

# 臨時禁用selinux
# 永久關閉 修改/etc/sysconfig/selinux文件設置
sed -i "s/SELINUX=permissive/SELINUX=disabled/" /etc/sysconfig/selinux 
# 這里按回車，下面是第二條命令
setenforce 0

關閉防火墻

systemctl disable firewalld.service && systemctl stop firewalld.service

配置轉發參數

# 配置轉發相關參數，否則可能會出錯 
cat < /etc/sysctl.d/k8s.conf 
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
vm.swappiness=0
EOF 
# 這里按回車，下面是第二條命令 
sysctl --system

這里就完成了k8s集群搭建的準備工作，集群搭建的話以上操作結束后將操作完的系統制作成系統鏡像，方便集群搭建

以下的操作都只在主節點上進行：

正式開始安裝k8s
初始化鏡像，執行以下命令：

kubeadm init --kubernetes-version=v1.11.0 #這里是之前所安裝K8S的版本號 --pod-network-cidr=10.10.0.0/16  #這里填寫集群所在網段

之后的輸出會是這樣：

I0712 10:46:30.938979   13461 feature_gate.go:230] feature gates: &{map[]}
[init] using Kubernetes version: v1.11.0
[preflight] running pre-flight checks
I0712 10:46:30.961005   13461 kernel_validator.go:81] Validating kernel version
I0712 10:46:30.961061   13461 kernel_validator.go:96] Validating kernel config
    [WARNING SystemVerification]: docker version is greater than the most recently validated version. Docker version: 18.03.1-ce. Max validated version: 17.03
    [WARNING Hostname]: hostname "g2-apigateway" could not be reached
    [WARNING Hostname]: hostname "g2-apigateway" lookup g2-apigateway on 100.100.2.138:53: no such host
[preflight/images] Pulling images required for setting up a Kubernetes cluster
[preflight/images] This might take a minute or two, depending on the speed of your internet connection
[preflight/images] You can also perform this action in beforehand using "kubeadm config images pull"
[kubelet] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[preflight] Activating the kubelet service
[certificates] Generated ca certificate and key.
[certificates] Generated apiserver certificate and key.
[certificates] apiserver serving cert is signed for DNS names [g2-apigateway kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 172.16.8.62]
[certificates] Generated apiserver-kubelet-client certificate and key.
[certificates] Generated sa key and public key.
[certificates] Generated front-proxy-ca certificate and key.
[certificates] Generated front-proxy-client certificate and key.
[certificates] Generated etcd/ca certificate and key.
[certificates] Generated etcd/server certificate and key.
[certificates] etcd/server serving cert is signed for DNS names [g2-apigateway localhost] and IPs [127.0.0.1 ::1]
[certificates] Generated etcd/peer certificate and key.
[certificates] etcd/peer serving cert is signed for DNS names [g2-apigateway localhost] and IPs [172.16.8.62 127.0.0.1 ::1]
[certificates] Generated etcd/healthcheck-client certificate and key.
[certificates] Generated apiserver-etcd-client certificate and key.
[certificates] valid certificates and keys now exist in "/etc/kubernetes/pki"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/admin.conf"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/kubelet.conf"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/controller-manager.conf"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/scheduler.conf"
[controlplane] wrote Static Pod manifest for component kube-apiserver to "/etc/kubernetes/manifests/kube-apiserver.yaml"
[controlplane] wrote Static Pod manifest for component kube-controller-manager to "/etc/kubernetes/manifests/kube-controller-manager.yaml"
[controlplane] wrote Static Pod manifest for component kube-scheduler to "/etc/kubernetes/manifests/kube-scheduler.yaml"
[etcd] Wrote Static Pod manifest for a local etcd instance to "/etc/kubernetes/manifests/etcd.yaml"
[init] waiting for the kubelet to boot up the control plane as Static Pods from directory "/etc/kubernetes/manifests" 
[init] this might take a minute or longer if the control plane images have to be pulled
[apiclient] All control plane components are healthy after 41.001672 seconds
[uploadconfig] storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[kubelet] Creating a ConfigMap "kubelet-config-1.11" in namespace kube-system with the configuration for the kubelets in the cluster
[markmaster] Marking the node g2-apigateway as master by adding the label "node-role.kubernetes.io/master="""
[markmaster] Marking the node g2-apigateway as master by adding the taints [node-role.kubernetes.io/master:NoSchedule]
[patchnode] Uploading the CRI Socket information "/var/run/dockershim.sock" to the Node API object "g2-apigateway" as an annotation
[bootstraptoken] using token: o337m9.ceq32wg9g2gro7gx
[bootstraptoken] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstraptoken] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstraptoken] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[bootstraptoken] creating the "cluster-info" ConfigMap in the "kube-public" namespace
[addons] Applied essential addon: CoreDNS
[addons] Applied essential addon: kube-proxy

Your Kubernetes master has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/

You can now join any number of machines by running the following on each node
as root:

kubeadm join 10.10.207.253:6443 --token t69z6h.lr2etdbg9mfx5r15 --discovery-token-ca-cert-hash sha256:90e3a748c0eb4cb7058f3d0ee8870ee5d746214ab0589b5e841fd5d68fec8f00

這里注意最后一行：

kubeadm join 10.10.207.253:6443 --token t69z6h.lr2etdbg9mfx5r15 --discovery-token-ca-cert-hash sha256:90e3a748c0eb4cb7058f3d0ee8870ee5d746214ab0589b5e841fd5d68fec8f00

證明集群主節點安裝成功，這里要記得保存這條命令，以便之后各個節點加入集群

配置kubetl認證信息

export KUBECONFIG=/etc/kubernetes/admin.conf 
# 如果你想持久化的話，直接執行以下命令【推薦】
echo "export KUBECONFIG=/etc/kubernetes/admin.conf" >> ~/.bash_profile

安裝flanel網絡
依次執行以下命令：

mkdir -p /etc/cni/net.d/

cat < /etc/cni/net.d/10-flannel.conf 
{ 
“name”: “cbr0”,
“type”: “flannel”,
“delegate”: { 
“isDefaultGateway”: true
} 
} 
EOF

mkdir /usr/share/oci-umount/oci-umount.d -p

mkdir /run/flannel/

cat < /run/flannel/subnet.env 
FLANNEL_NETWORK=10.244.0.0/16 
FLANNEL_SUBNET=10.244.1.0/24 
FLANNEL_MTU=1450 
FLANNEL_IPMASQ=true 
EOF

最后需要新建一個flannel.yml文件：

---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: flannel
rules:
  - apiGroups:
      - ""
    resources:
      - pods
    verbs:
      - get
  - apiGroups:
      - ""
    resources:
      - nodes
    verbs:
      - list
      - watch
  - apiGroups:
      - ""
    resources:
      - nodes/status
    verbs:
      - patch
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: flannel
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: flannel
subjects:
- kind: ServiceAccount
  name: flannel
  namespace: kube-system
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: flannel
  namespace: kube-system
---
kind: ConfigMap
apiVersion: v1
metadata:
  name: kube-flannel-cfg
  namespace: kube-system
  labels:
    tier: node
    app: flannel
data:
  cni-conf.json: |
    {
      "name": "cbr0",
      "type": "flannel",
      "delegate": {
        "isDefaultGateway": true
      }
    }
  net-conf.json: |
    {
      "Network": "10.10.0.0/16",    #這里換成集群所在的網段
      "Backend": {
        "Type": "vxlan"
      }
    }
---
apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
  name: kube-flannel-ds
  namespace: kube-system
  labels:
    tier: node
    app: flannel
spec:
  template:
    metadata:
      labels:
        tier: node
        app: flannel
    spec:
      hostNetwork: true
      nodeSelector:
        beta.kubernetes.io/arch: amd64
      tolerations:
      - key: node-role.kubernetes.io/master
        operator: Exists
        effect: NoSchedule
      serviceAccountName: flannel
      initContainers:
      - name: install-cni
        image: quay.io/coreos/flannel:v0.9.1-amd64
        command:
        - cp
        args:
        - -f
        - /etc/kube-flannel/cni-conf.json
        - /etc/cni/net.d/10-flannel.conf
        volumeMounts:
        - name: cni
          mountPath: /etc/cni/net.d
        - name: flannel-cfg
          mountPath: /etc/kube-flannel/
      containers:
      - name: kube-flannel
        image: quay.io/coreos/flannel:v0.9.1-amd64
        command: [ "/opt/bin/flanneld", "--ip-masq", "--kube-subnet-mgr" ]
        securityContext:
          privileged: true
        env:
        - name: POD_NAME
          valueFrom:
            fieldRef:
              fieldPath: metadata.name
        - name: POD_NAMESPACE
          valueFrom:
            fieldRef:
              fieldPath: metadata.namespace
        volumeMounts:
        - name: run
          mountPath: /run
        - name: flannel-cfg
          mountPath: /etc/kube-flannel/
      volumes:
        - name: run
          hostPath:
            path: /run
        - name: cni
          hostPath:
            path: /etc/cni/net.d
        - name: flannel-cfg
          configMap:
            name: kube-flannel-cfg

執行：

kubectl create -f ./flannel.yml

默認情況下，master節點不參與工作負載，但如果希望安裝出一個all-in-one的k8s環境，則可以執行以下命令，讓master節點成為一個node節點：

kubectl taint nodes --all node-role.kubernetes.io/master-

執行之后，運行以下命令，查看節點信息：

kubectl get nodes

會看到如下的輸出：

NAME            STATUS     ROLES     AGE       VERSION
k8s-master      Ready      master    18h       v1.11.0

以下是節點配置

在配置好主節點之后，就可以配置集群的其他節點了，這里建議直接安裝之前做好準備工作的系統鏡像
進入節點機器之后，直接執行之前保存好的命令

kubeadm join 10.10.207.253:6443 --token t69z6h.lr2etdbg9mfx5r15 --discovery-token-ca-cert-hash sha256:90e3a748c0eb4cb7058f3d0ee8870ee5d746214ab0589b5e841fd5d68fec8f00

執行完后會看到：

[preflight] running pre-flight checks
        [WARNING RequiredIPVSKernelModulesAvailable]: the IPVS proxier will not be used, because the following required kernel modules are not loaded: [ip_vs_wrr ip_vs_sh ip_vs ip_vs_rr] or no builtin kernel ipvs support: map[ip_vs_rr:{} ip_vs_wrr:{} ip_vs_sh:{} nf_conntrack_ipv4:{} ip_vs:{}]
you can solve this problem with following methods:
 1. Run "modprobe -- " to load missing kernel modules;
2. Provide the missing builtin kernel ipvs support

I0725 09:59:27.929247   10196 kernel_validator.go:81] Validating kernel version
I0725 09:59:27.929356   10196 kernel_validator.go:96] Validating kernel config
[discovery] Trying to connect to API Server "10.10.207.253:6443"
[discovery] Created cluster-info discovery client, requesting info from "https://10.10.207.253:6443"
[discovery] Requesting info from "https://10.10.207.253:6443" again to validate TLS against the pinned public key
[discovery] Cluster info signature and contents are valid and TLS certificate validates against pinned roots, will use API Server "10.10.207.253:6443"
[discovery] Successfully established connection with API Server "10.10.207.253:6443"
[kubelet] Downloading configuration for the kubelet from the "kubelet-config-1.11" ConfigMap in the kube-system namespace
[kubelet] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[preflight] Activating the kubelet service
[tlsbootstrap] Waiting for the kubelet to perform the TLS Bootstrap...
[patchnode] Uploading the CRI Socket information "/var/run/dockershim.sock" to the Node API object "k8s-node1" as an annotation

This node has joined the cluster:
* Certificate signing request was sent to master and a response
  was received.
* The Kubelet was informed of the new secure connection details.

Run "kubectl get nodes" on the master to see this node join the cluster.

這里就表示執行完畢了，可以去主節點執行命令：

kubectl get nodes

可以看到節點已加入集群：

NAME        STATUS    ROLES     AGE       VERSION
k8s-master  Ready     master    20h       v1.11.0
k8s-node1   Ready         20h       v1.11.0
k8s-node2   Ready         20h       v1.11.0

這期間可能需要等待一段時間，狀態才會全部變為ready

詳見：kubernetes安裝dashboard

有時會出現master節點一直處于notready的狀態，這里可能是沒有啟動flannel，只需要按照上面的教程配置好flannel，然后執行：

kubectl create -f ./flannel.yml

參考資料：https://blog.myzony.com/cento...