摘要:離線安裝包三步安裝,簡單到難以置信源碼分析說句實在話,的代碼寫的真心一般,質量不是很高。然后給該租戶綁定角色。
k8s離線安裝包 三步安裝,簡單到難以置信
kubeadm源碼分析說句實在話,kubeadm的代碼寫的真心一般,質量不是很高。
幾個關鍵點來先說一下kubeadm干的幾個核心的事:
kubeadm 生成證書在/etc/kubernetes/pki目錄下
kubeadm 生成static pod yaml配置,全部在/etc/kubernetes/manifasts下
kubeadm 生成kubelet配置,kubectl配置等 在/etc/kubernetes下
kubeadm 通過client go去啟動dns
kubeadm init代碼入口 cmd/kubeadm/app/cmd/init.go 建議大家去看看cobra
找到Run函數來分析下主要流程:
如果證書不存在,就創建證書,所以如果我們有自己的證書可以把它放在/etc/kubernetes/pki下即可, 下文細看如果生成證書
if res, _ := certsphase.UsingExternalCA(i.cfg); !res { if err := certsphase.CreatePKIAssets(i.cfg); err != nil { return err }
創建kubeconfig文件
if err := kubeconfigphase.CreateInitKubeConfigFiles(kubeConfigDir, i.cfg); err != nil { return err }
創建manifest文件,etcd apiserver manager scheduler都在這里創建, 可以看到如果你的配置文件里已經寫了etcd的地址了,就不創建了,這我們就可以自己裝etcd集群,而不用默認單點的etcd,很有用
controlplanephase.CreateInitStaticPodManifestFiles(manifestDir, i.cfg); if len(i.cfg.Etcd.Endpoints) == 0 { if err := etcdphase.CreateLocalEtcdStaticPodManifestFile(manifestDir, i.cfg); err != nil { return fmt.Errorf("error creating local etcd static pod manifest file: %v", err) } }
等待APIserver和kubelet啟動成功,這里就會遇到我們經常遇到的鏡像拉不下來的錯誤,其實有時kubelet因為別的原因也會報這個錯,讓人誤以為是鏡像弄不下來
if err := waitForAPIAndKubelet(waiter); err != nil { ctx := map[string]string{ "Error": fmt.Sprintf("%v", err), "APIServerImage": images.GetCoreImage(kubeadmconstants.KubeAPIServer, i.cfg.GetControlPlaneImageRepository(), i.cfg.KubernetesVersion, i.cfg.UnifiedControlPlaneImage), "ControllerManagerImage": images.GetCoreImage(kubeadmconstants.KubeControllerManager, i.cfg.GetControlPlaneImageRepository(), i.cfg.KubernetesVersion, i.cfg.UnifiedControlPlaneImage), "SchedulerImage": images.GetCoreImage(kubeadmconstants.KubeScheduler, i.cfg.GetControlPlaneImageRepository(), i.cfg.KubernetesVersion, i.cfg.UnifiedControlPlaneImage), } kubeletFailTempl.Execute(out, ctx) return fmt.Errorf("couldn"t initialize a Kubernetes cluster") }
給master加標簽,加污點, 所以想要pod調度到master上可以把污點清除了
if err := markmasterphase.MarkMaster(client, i.cfg.NodeName); err != nil { return fmt.Errorf("error marking master: %v", err) }
生成tocken
if err := nodebootstraptokenphase.UpdateOrCreateToken(client, i.cfg.Token, false, i.cfg.TokenTTL.Duration, kubeadmconstants.DefaultTokenUsages, []string{kubeadmconstants.NodeBootstrapTokenAuthGroup}, tokenDescription); err != nil { return fmt.Errorf("error updating or creating token: %v", err) }
調用clientgo創建dns和kube-proxy
if err := dnsaddonphase.EnsureDNSAddon(i.cfg, client); err != nil { return fmt.Errorf("error ensuring dns addon: %v", err) } if err := proxyaddonphase.EnsureProxyAddon(i.cfg, client); err != nil { return fmt.Errorf("error ensuring proxy addon: %v", err) }
筆者批判代碼無腦式的一個流程到底,要是筆者操刀定抽象成接口 RenderConf Save Run Clean等,DNS kube-porxy以及其它組件去實現,然后問題就是沒把dns和kubeproxy的配置渲染出來,可能是它們不是static pod的原因, 然后就是join時的bug下文提到
證書生成循環的調用了這一坨函數,我們只需要看其中一兩個即可,其它的都差不多
certActions := []func(cfg *kubeadmapi.MasterConfiguration) error{ CreateCACertAndKeyfiles, CreateAPIServerCertAndKeyFiles, CreateAPIServerKubeletClientCertAndKeyFiles, CreateServiceAccountKeyAndPublicKeyFiles, CreateFrontProxyCACertAndKeyFiles, CreateFrontProxyClientCertAndKeyFiles, }
根證書生成:
//返回了根證書的公鑰和私鑰 func NewCACertAndKey() (*x509.Certificate, *rsa.PrivateKey, error) { caCert, caKey, err := pkiutil.NewCertificateAuthority() if err != nil { return nil, nil, fmt.Errorf("failure while generating CA certificate and key: %v", err) } return caCert, caKey, nil }
k8s.io/client-go/util/cert 這個庫里面有兩個函數,一個生成key的一個生成cert的:
key, err := certutil.NewPrivateKey() config := certutil.Config{ CommonName: "kubernetes", } cert, err := certutil.NewSelfSignedCACert(config, key)
config里面我們也可以填充一些別的證書信息:
type Config struct { CommonName string Organization []string AltNames AltNames Usages []x509.ExtKeyUsage }
私鑰就是封裝了rsa庫里面的函數:
"crypto/rsa" "crypto/x509" func NewPrivateKey() (*rsa.PrivateKey, error) { return rsa.GenerateKey(cryptorand.Reader, rsaKeySize) }
自簽證書,所以根證書里只有CommonName信息,Organization相當于沒設置:
func NewSelfSignedCACert(cfg Config, key *rsa.PrivateKey) (*x509.Certificate, error) { now := time.Now() tmpl := x509.Certificate{ SerialNumber: new(big.Int).SetInt64(0), Subject: pkix.Name{ CommonName: cfg.CommonName, Organization: cfg.Organization, }, NotBefore: now.UTC(), NotAfter: now.Add(duration365d * 10).UTC(), KeyUsage: x509.KeyUsageKeyEncipherment | x509.KeyUsageDigitalSignature | x509.KeyUsageCertSign, BasicConstraintsValid: true, IsCA: true, } certDERBytes, err := x509.CreateCertificate(cryptorand.Reader, &tmpl, &tmpl, key.Public(), key) if err != nil { return nil, err } return x509.ParseCertificate(certDERBytes) }
生成好之后把之寫入文件:
pkiutil.WriteCertAndKey(pkiDir, baseName, cert, key); certutil.WriteCert(certificatePath, certutil.EncodeCertPEM(cert))
這里調用了pem庫進行了編碼
encoding/pem func EncodeCertPEM(cert *x509.Certificate) []byte { block := pem.Block{ Type: CertificateBlockType, Bytes: cert.Raw, } return pem.EncodeToMemory(&block) }
然后我們看apiserver的證書生成:
caCert, caKey, err := loadCertificateAuthorithy(cfg.CertificatesDir, kubeadmconstants.CACertAndKeyBaseName) //從根證書生成apiserver證書 apiCert, apiKey, err := NewAPIServerCertAndKey(cfg, caCert, caKey)
這時需要關注AltNames了比較重要,所有需要訪問master的地址域名都得加進去,對應配置文件中apiServerCertSANs字段,其它東西與根證書無差別
config := certutil.Config{ CommonName: kubeadmconstants.APIServerCertCommonName, AltNames: *altNames, Usages: []x509.ExtKeyUsage{x509.ExtKeyUsageServerAuth}, }創建k8s配置文件
可以看到創建了這些文件
return createKubeConfigFiles( outDir, cfg, kubeadmconstants.AdminKubeConfigFileName, kubeadmconstants.KubeletKubeConfigFileName, kubeadmconstants.ControllerManagerKubeConfigFileName, kubeadmconstants.SchedulerKubeConfigFileName, )
k8s封裝了兩個渲染配置的函數:
區別是你的kubeconfig文件里會不會產生token,比如你進入dashboard需要一個token,或者你調用api需要一個token那么請生成帶token的配置
生成的conf文件基本一直只是比如ClientName這些東西不同,所以加密后的證書也不同,ClientName會被加密到證書里,然后k8s取出來當用戶使用
所以重點來了,我們做多租戶時也要這樣去生成。然后給該租戶綁定角色。
return kubeconfigutil.CreateWithToken( spec.APIServer, "kubernetes", spec.ClientName, certutil.EncodeCertPEM(spec.CACert), spec.TokenAuth.Token, ), nil return kubeconfigutil.CreateWithCerts( spec.APIServer, "kubernetes", spec.ClientName, certutil.EncodeCertPEM(spec.CACert), certutil.EncodePrivateKeyPEM(clientKey), certutil.EncodeCertPEM(clientCert), ), nil
然后就是填充Config結構體嘍, 最后寫到文件里,略
"k8s.io/client-go/tools/clientcmd/api return &clientcmdapi.Config{ Clusters: map[string]*clientcmdapi.Cluster{ clusterName: { Server: serverURL, CertificateAuthorityData: caCert, }, }, Contexts: map[string]*clientcmdapi.Context{ contextName: { Cluster: clusterName, AuthInfo: userName, }, }, AuthInfos: map[string]*clientcmdapi.AuthInfo{}, CurrentContext: contextName, }創建static pod yaml文件
這里返回了apiserver manager scheduler的pod結構體,
specs := GetStaticPodSpecs(cfg, k8sVersion) staticPodSpecs := map[string]v1.Pod{ kubeadmconstants.KubeAPIServer: staticpodutil.ComponentPod(v1.Container{ Name: kubeadmconstants.KubeAPIServer, Image: images.GetCoreImage(kubeadmconstants.KubeAPIServer, cfg.GetControlPlaneImageRepository(), cfg.KubernetesVersion, cfg.UnifiedControlPlaneImage), Command: getAPIServerCommand(cfg, k8sVersion), VolumeMounts: staticpodutil.VolumeMountMapToSlice(mounts.GetVolumeMounts(kubeadmconstants.KubeAPIServer)), LivenessProbe: staticpodutil.ComponentProbe(cfg, kubeadmconstants.KubeAPIServer, int(cfg.API.BindPort), "/healthz", v1.URISchemeHTTPS), Resources: staticpodutil.ComponentResources("250m"), Env: getProxyEnvVars(), }, mounts.GetVolumes(kubeadmconstants.KubeAPIServer)), kubeadmconstants.KubeControllerManager: staticpodutil.ComponentPod(v1.Container{ Name: kubeadmconstants.KubeControllerManager, Image: images.GetCoreImage(kubeadmconstants.KubeControllerManager, cfg.GetControlPlaneImageRepository(), cfg.KubernetesVersion, cfg.UnifiedControlPlaneImage), Command: getControllerManagerCommand(cfg, k8sVersion), VolumeMounts: staticpodutil.VolumeMountMapToSlice(mounts.GetVolumeMounts(kubeadmconstants.KubeControllerManager)), LivenessProbe: staticpodutil.ComponentProbe(cfg, kubeadmconstants.KubeControllerManager, 10252, "/healthz", v1.URISchemeHTTP), Resources: staticpodutil.ComponentResources("200m"), Env: getProxyEnvVars(), }, mounts.GetVolumes(kubeadmconstants.KubeControllerManager)), kubeadmconstants.KubeScheduler: staticpodutil.ComponentPod(v1.Container{ Name: kubeadmconstants.KubeScheduler, Image: images.GetCoreImage(kubeadmconstants.KubeScheduler, cfg.GetControlPlaneImageRepository(), cfg.KubernetesVersion, cfg.UnifiedControlPlaneImage), Command: getSchedulerCommand(cfg), VolumeMounts: staticpodutil.VolumeMountMapToSlice(mounts.GetVolumeMounts(kubeadmconstants.KubeScheduler)), LivenessProbe: staticpodutil.ComponentProbe(cfg, kubeadmconstants.KubeScheduler, 10251, "/healthz", v1.URISchemeHTTP), Resources: staticpodutil.ComponentResources("100m"), Env: getProxyEnvVars(), }, mounts.GetVolumes(kubeadmconstants.KubeScheduler)), } //獲取特定版本的鏡像 func GetCoreImage(image, repoPrefix, k8sVersion, overrideImage string) string { if overrideImage != "" { return overrideImage } kubernetesImageTag := kubeadmutil.KubernetesVersionToImageTag(k8sVersion) etcdImageTag := constants.DefaultEtcdVersion etcdImageVersion, err := constants.EtcdSupportedVersion(k8sVersion) if err == nil { etcdImageTag = etcdImageVersion.String() } return map[string]string{ constants.Etcd: fmt.Sprintf("%s/%s-%s:%s", repoPrefix, "etcd", runtime.GOARCH, etcdImageTag), constants.KubeAPIServer: fmt.Sprintf("%s/%s-%s:%s", repoPrefix, "kube-apiserver", runtime.GOARCH, kubernetesImageTag), constants.KubeControllerManager: fmt.Sprintf("%s/%s-%s:%s", repoPrefix, "kube-controller-manager", runtime.GOARCH, kubernetesImageTag), constants.KubeScheduler: fmt.Sprintf("%s/%s-%s:%s", repoPrefix, "kube-scheduler", runtime.GOARCH, kubernetesImageTag), }[image] } //然后就把這個pod寫到文件里了,比較簡單 staticpodutil.WriteStaticPodToDisk(componentName, manifestDir, spec);
創建etcd的一樣,不多廢話
等待kubelet啟動成功這個錯誤非常容易遇到,看到這個基本就是kubelet沒起來,我們需要檢查:selinux swap 和Cgroup driver是不是一致
setenforce 0 && swapoff -a && systemctl restart kubelet如果不行請保證 kubelet的Cgroup driver與docker一致,docker info|grep Cg
go func(errC chan error, waiter apiclient.Waiter) { // This goroutine can only make kubeadm init fail. If this check succeeds, it won"t do anything special if err := waiter.WaitForHealthyKubelet(40*time.Second, "http://localhost:10255/healthz"); err != nil { errC <- err } }(errorChan, waiter) go func(errC chan error, waiter apiclient.Waiter) { // This goroutine can only make kubeadm init fail. If this check succeeds, it won"t do anything special if err := waiter.WaitForHealthyKubelet(60*time.Second, "http://localhost:10255/healthz/syncloop"); err != nil { errC <- err } }(errorChan, waiter)創建DNS和kubeproxy
我就是在此發現coreDNS的
if features.Enabled(cfg.FeatureGates, features.CoreDNS) { return coreDNSAddon(cfg, client, k8sVersion) } return kubeDNSAddon(cfg, client, k8sVersion)
然后coreDNS的yaml配置模板直接是寫在代碼里的:
/app/phases/addons/dns/manifests.go
CoreDNSDeployment = ` apiVersion: apps/v1beta2 kind: Deployment metadata: name: coredns namespace: kube-system labels: k8s-app: kube-dns spec: replicas: 1 selector: matchLabels: k8s-app: kube-dns template: metadata: labels: k8s-app: kube-dns spec: serviceAccountName: coredns tolerations: - key: CriticalAddonsOnly operator: Exists - key: {{ .MasterTaintKey }} ...
然后渲染模板,最后調用k8sapi創建,這種創建方式可以學習一下,雖然有點拙劣,這地方寫的遠不如kubectl好
coreDNSConfigMap := &v1.ConfigMap{} if err := kuberuntime.DecodeInto(legacyscheme.Codecs.UniversalDecoder(), configBytes, coreDNSConfigMap); err != nil { return fmt.Errorf("unable to decode CoreDNS configmap %v", err) } // Create the ConfigMap for CoreDNS or update it in case it already exists if err := apiclient.CreateOrUpdateConfigMap(client, coreDNSConfigMap); err != nil { return err } coreDNSClusterRoles := &rbac.ClusterRole{} if err := kuberuntime.DecodeInto(legacyscheme.Codecs.UniversalDecoder(), []byte(CoreDNSClusterRole), coreDNSClusterRoles); err != nil { return fmt.Errorf("unable to decode CoreDNS clusterroles %v", err) } ...
這里值得一提的是kubeproxy的configmap真應該把apiserver地址傳入進來,允許自定義,因為做高可用時需要指定虛擬ip,得修改,很麻煩
kubeproxy大差不差,不說了,想改的話改: app/phases/addons/proxy/manifests.go
kubeadm join比較簡單,一句話就可以說清楚,獲取cluster info, 創建kubeconfig,怎么創建的kubeinit里面已經說了。帶上token讓kubeadm有權限
可以拉取
return https.RetrieveValidatedClusterInfo(cfg.DiscoveryFile) cluster info內容 type Cluster struct { // LocationOfOrigin indicates where this object came from. It is used for round tripping config post-merge, but never serialized. LocationOfOrigin string // Server is the address of the kubernetes cluster (https://hostname:port). Server string `json:"server"` // InsecureSkipTLSVerify skips the validity check for the server"s certificate. This will make your HTTPS connections insecure. // +optional InsecureSkipTLSVerify bool `json:"insecure-skip-tls-verify,omitempty"` // CertificateAuthority is the path to a cert file for the certificate authority. // +optional CertificateAuthority string `json:"certificate-authority,omitempty"` // CertificateAuthorityData contains PEM-encoded certificate authority certificates. Overrides CertificateAuthority // +optional CertificateAuthorityData []byte `json:"certificate-authority-data,omitempty"` // Extensions holds additional information. This is useful for extenders so that reads and writes don"t clobber unknown fields // +optional Extensions map[string]runtime.Object `json:"extensions,omitempty"` } return kubeconfigutil.CreateWithToken( clusterinfo.Server, "kubernetes", TokenUser, clusterinfo.CertificateAuthorityData, cfg.TLSBootstrapToken, ), nil
CreateWithToken上文提到了不再贅述,這樣就能去生成kubelet配置文件了,然后把kubelet啟動起來即可
kubeadm join的問題就是渲染配置時沒有使用命令行傳入的apiserver地址,而用clusterinfo里的地址,這不利于我們做高可用,可能我們傳入一個虛擬ip,但是配置里還是apiser的地址
文章版權歸作者所有,未經允許請勿轉載,若此文章存在違規行為,您可以聯系管理員刪除。
轉載請注明本文地址:http://specialneedsforspecialkids.com/yun/27326.html
摘要:離線安裝包三步安裝,簡單到難以置信源碼分析說句實在話,的代碼寫的真心一般,質量不是很高。然后給該租戶綁定角色。 k8s離線安裝包 三步安裝,簡單到難以置信 kubeadm源碼分析 說句實在話,kubeadm的代碼寫的真心一般,質量不是很高。 幾個關鍵點來先說一下kubeadm干的幾個核心的事: kubeadm 生成證書在/etc/kubernetes/pki目錄下 kubeadm 生...
摘要:集群三步安裝概述本文教你如何用一條命令構建高可用集群且不依賴和,也無需。通過內核對進行負載均衡,并且帶健康檢測。當然你也可以把用于一些其它場景,比如代理自己的服務等 kubernetes集群三步安裝 概述 本文教你如何用一條命令構建k8s高可用集群且不依賴haproxy和keepalived,也無需ansible。通過內核ipvs對apiserver進行負載均衡,并且帶apiserve...
摘要:使用安裝安全高可用集群安裝包地址如非高可用安裝請忽略此教程,直接看產品頁的三步安裝。 使用kubeadm安裝安全高可用kubernetes集群 安裝包地址 如非高可用安裝請忽略此教程,直接看產品頁的三步安裝。 單個master流程: 單master視頻教程 解壓后在master 上 cd shell && sh init.sh ,然后sh master.sh(注意因為腳本用的相...
摘要:使用安裝安全高可用集群安裝包地址如非高可用安裝請忽略此教程,直接看產品頁的三步安裝。 使用kubeadm安裝安全高可用kubernetes集群 安裝包地址 如非高可用安裝請忽略此教程,直接看產品頁的三步安裝。 單個master流程: 單master視頻教程 解壓后在master 上 cd shell && sh init.sh ,然后sh master.sh(注意因為腳本用的相...
摘要:使用安裝安全高可用集群安裝包地址如非高可用安裝請忽略此教程,直接看產品頁的三步安裝。 使用kubeadm安裝安全高可用kubernetes集群 安裝包地址 如非高可用安裝請忽略此教程,直接看產品頁的三步安裝。 單個master流程: 單master視頻教程 解壓后在master 上 cd shell && sh init.sh ,然后sh master.sh(注意因為腳本用的相...
閱讀 3216·2021-11-23 09:51
閱讀 3558·2021-11-09 09:46
閱讀 3655·2021-11-09 09:45
閱讀 2938·2019-08-29 17:31
閱讀 1860·2019-08-26 13:39
閱讀 2715·2019-08-26 12:12
閱讀 3614·2019-08-26 12:08
閱讀 2235·2019-08-26 11:31