Kubernetes多集群管理策略统一管理多个K8s集群一、多集群管理概述Kubernetes多集群管理是指在企业环境中管理多个独立的Kubernetes集群实现统一的部署、监控和运维。1.1 多集群场景场景说明示例地域隔离不同区域部署独立集群北京、上海、广州各一个集群环境隔离开发、测试、生产分离dev、staging、prod集群租户隔离多租户共享基础设施每个租户独立集群混合云公有云私有云混合部署AWS本地IDC集群1.2 多集群架构┌─────────────────────────┐ │ 统一管理平面 │ │ (Cluster Management) │ └───────────┬─────────────┘ │ ┌───────────────────────┼───────────────────────┐ │ │ │ ▼ ▼ ▼ ┌───────────────┐ ┌───────────────┐ ┌───────────────┐ │ 集群A │ │ 集群B │ │ 集群C │ │ (Production) │ │ (Staging) │ │ (Development)│ └───────────────┘ └───────────────┘ └───────────────┘二、多集群管理工具2.1 Rancher配置apiVersion: rancher.cattle.io/v3 kind: Cluster metadata: name: production spec: rkeConfig: machinePools: - name: worker count: 3 machineConfigRef: apiVersion: rke-machine-config.cattle.io/v1 kind: DigitalOceanConfig name: do-worker2.2 Fleet配置apiVersion: fleet.cattle.io/v1alpha1 kind: GitRepo metadata: name: my-apps namespace: fleet-default spec: repo: https://github.com/example/fleet-repo branch: main targets: - name: production clusterSelector: matchLabels: env: prod - name: staging clusterSelector: matchLabels: env: staging2.3 Cluster API配置apiVersion: cluster.x-k8s.io/v1beta1 kind: Cluster metadata: name: my-cluster spec: topology: class: quick-start version: v1.27.3 workers: machineDeployments: - class: default-worker replicas: 3三、多集群网络策略3.1 集群间通信apiVersion: v1 kind: Service metadata: name: cross-cluster-service spec: type: ExternalName externalName: service.other-cluster.svc.cluster.local3.2 统一入口管理apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: global-ingress annotations: nginx.ingress.kubernetes.io/rewrite-target: / spec: rules: - host: app.example.com http: paths: - path: /api pathType: Prefix backend: service: name: api-service port: number: 80 - host: app-staging.example.com http: paths: - path: /api pathType: Prefix backend: service: name: api-service-staging port: number: 80四、多集群资源同步4.1 配置同步apiVersion: configsync.gke.io/v1beta1 kind: RootSync metadata: name: cluster-config spec: sourceFormat: unstructured git: repo: https://github.com/example/cluster-config branch: main policyDir: configs/ auth: token secretRef: name: git-creds4.2 资源分发策略apiVersion: distribution.k8s.io/v1alpha1 kind: ClusterResourceSet metadata: name: common-config spec: clusterSelector: matchLabels: environment: shared resources: - name: common-configmap kind: ConfigMap - name: common-secret kind: Secret五、多集群监控5.1 Prometheus联邦apiVersion: monitoring.coreos.com/v1 kind: ServiceMonitor metadata: name: remote-cluster namespace: monitoring spec: endpoints: - honorLabels: true interval: 30s path: /federate params: match[]: - {__name__~job:.*} port: http selector: matchLabels: app: prometheus5.2 统一告警规则apiVersion: monitoring.coreos.com/v1 kind: PrometheusRule metadata: name: cluster-alerts namespace: monitoring spec: groups: - name: cluster.rules rules: - alert: HighCPUUsage expr: avg(rate(node_cpu_seconds_total{modeidle}[5m])) 0.2 for: 10m labels: severity: critical annotations: summary: High CPU usage detected六、多集群日志管理6.1 Loki分布式日志apiVersion: loki.grafana.com/v1 kind: LokiStack metadata: name: loki namespace: monitoring spec: size: 1x.extra-small storage: schemas: - version: v13 effectiveDate: 2024-01-01 secret: name: loki-storage6.2 日志收集配置apiVersion: v1 kind: ConfigMap metadata: name: fluentd-config namespace: logging data: fluent.conf: | source type tail path /var/log/containers/*.log pos_file /var/log/fluentd-containers.log.pos tag kubernetes.* read_from_head true /source match kubernetes.** type loki url https://loki.example.com auth_user admin auth_password secret /match七、多集群安全策略7.1 统一RBAC管理apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: name: cluster-admin rules: - apiGroups: [*] resources: [*] verbs: [*] --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: admin-user subjects: - kind: User name: adminexample.com apiGroup: rbac.authorization.k8s.io roleRef: kind: ClusterRole name: cluster-admin apiGroup: rbac.authorization.k8s.io7.2 证书管理apiVersion: cert-manager.io/v1 kind: ClusterIssuer metadata: name: letsencrypt-prod spec: acme: server: https://acme-v02.api.letsencrypt.org/directory email: adminexample.com privateKeySecretRef: name: letsencrypt-prod solvers: - http01: ingress: class: nginx八、多集群成本管理8.1 资源使用监控apiVersion: v1 kind: ConfigMap metadata: name: cost-exporter-config namespace: monitoring data: config.yaml: | exporters: - name: cloud-cost type: prometheus params: endpoint: http://prometheus:9090 query: | sum(node_cpu_hours_total) * 0.05 sum(node_memory_hours_total) * 0.028.2 资源配额管理apiVersion: v1 kind: ResourceQuota metadata: name: cluster-quota spec: hard: pods: 1000 requests.cpu: 100 requests.memory: 200Gi limits.cpu: 200 limits.memory: 400Gi九、多集群故障恢复9.1 灾难恢复策略apiVersion: velero.io/v1 kind: Schedule metadata: name: daily-backup spec: schedule: 0 2 * * * template: includedNamespaces: - default - kube-system storageLocation: name: s3-backup volumeSnapshotLocations: - name: aws-ebs9.2 跨集群迁移apiVersion: apps/v1 kind: Deployment metadata: name: migration-app spec: replicas: 0 selector: matchLabels: app: migration-app template: metadata: labels: app: migration-app spec: containers: - name: app image: migration-tool:latest env: - name: SOURCE_CLUSTER value: https://source-cluster:6443 - name: TARGET_CLUSTER value: https://target-cluster:6443十、总结Kubernetes多集群管理需要考虑统一管理平面使用Rancher、Fleet等工具进行集中管理网络互联配置集群间通信和统一入口资源同步实现配置和应用的跨集群分发监控告警建立统一的监控和告警体系安全策略统一RBAC和证书管理成本优化监控和控制多集群资源使用灾难恢复制定备份和恢复策略建议根据业务需求选择合适的多集群管理方案实现高效、安全的集群运维。参考资料Rancher官方文档Cluster API文档Fleet文档
