Karmada 多调度组Multiple Scheduling Group深度解析从多组亲和到分级弹性调度的完整实现指南【免费下载链接】karmadaOpen, Multi-Cloud, Multi-Cluster Kubernetes Orchestration项目地址: https://gitcode.com/GitHub_Trending/ka/karmada导读Karmada 是面向多云、多集群场景的 Kubernetes 编排系统其PropagationPolicy通过.spec.placement.clusterAffinity声明候选集群集合。但单一亲和组在主集群优先、备集群兜底的实际诉求面前力不从心。本文以仓库内 multi-scheduling-group 设计提案 为骨架完整讲解clusterAffinities多调度组 API 的设计动机、调度语义、状态追踪字段与组件改造方案并结合 propagation_types.go 与 binding_types.go 等源码给出实现级佐证同时延伸介绍其后续演进——分级弹性溢出的 Overflow Cluster Affinities 能力。读完本文你将掌握如何用多调度组实现首选组 灾备组的调度策略以及多调度组在调度器中的真实执行链路。一、背景与动机为什么需要多个调度组1.1 现有clusterAffinity的局限在引入多调度组之前PropagationPolicy只能声明一组候选集群即.spec.placement.clusterAffinity。如下示例声明了member1、member2两个候选集群apiVersion: policy.karmada.io/v1alpha1 kind: PropagationPolicy metadata: name: foo spec: resourceSelectors: - apiVersion: apps/v1 kind: Deployment name: foo placement: clusterAffinity: clusterNames: - member1 - member2clusterAffinity将一组候选集群交给karmada-scheduler调度器再依据spreadConstraint、过滤插件filter plugins等限制条件在候选集群中做出调度决策结果只有两种成功为引用资源选出一组集群通常为候选集的子集失败无法选出满足所有限制条件的集群组。1.2 现实诉求主备分组与成本优先集群管理员通常会按分类维度如云厂商、用途将集群划分为不同分组期望把工作负载优先部署到首选分组当首选分组不满足调度限制如资源不足时再回退到备用分组。Karmada 社区从最终用户处收到了大量类似反馈例如 issue #780 与 #2085 所描述的诉求。提案中给出了两个典型用户故事成本优先用户在本地数据中心拥有私有集群同时也购买了云厂商如 AWS、Google Cloud的托管集群。由于托管集群成本更高用户希望优先部署在私有集群上把托管集群作为备份灾备优先用户拥有主集群与备份集群希望应用默认部署在主集群当主集群因数据中心断电、维护等原因不可用时自动迁移到备份集群。1.3 目标与非目标Goals扩展PropagationPolicyAPI使其能够持有多个亲和组声明扩展ResourceBindingAPI用于标注调度器当前正在评估的亲和组为karmada-controller-manager、karmada-webhook、karmada-scheduler等组件提出实现思路。Non-Goals明确不在范围内同一组内集群的相对优先级本提案聚焦于引入多个亲和组同一组内集群的优先级问题由weightPreference.spec.placement.replicaScheduling.weightPreference或为调度器定义集群打分策略来解决调度再平衡re-balance用户期望调度器出于其他目的执行额外调度的场景如讨论 #3069应由另一份独立提案跟踪。1.4 兼容性与风险本提案保持向后兼容旧版本 Karmada 构建的系统可无缝迁移到新版本旧配置YAML可直接应用于新版 Karmada 且行为不变。二、API 设计ClusterAffinities与ClusterAffinityTerm2.1Placement新增ClusterAffinities字段提案在.spec.placement中新增字段ClusterAffinities用于声明多个亲和项affinity term。该字段在源码 pkg/apis/policy/v1alpha1/propagation_types.go 中已落地核心注释完整继承了提案语义// Placement represents the rule for select clusters. type Placement struct { // ClusterAffinity represents scheduling restrictions to a certain set of clusters. // If not set, any cluster can be scheduling candidate. // optional ClusterAffinity *ClusterAffinity json:clusterAffinity,omitempty // ClusterAffinities represents scheduling restrictions to multiple cluster // groups that indicated by ClusterAffinityTerm. // // The scheduler will evaluate these groups one by one in the order they // appear in the spec, the group that does not satisfy scheduling restrictions // will be ignored which means all clusters in this group will not be selected // unless it also belongs to the next group(a cluster could belong to multiple // groups). // // If none of the groups satisfy the scheduling restrictions, then scheduling // fails, which means no cluster will be selected. // // Note: // 1. ClusterAffinities can not co-exist with ClusterAffinity. // 2. If both ClusterAffinity and ClusterAffinities are not set, any cluster // can be scheduling candidates. // // optional ClusterAffinities []ClusterAffinityTerm json:clusterAffinities,omitempty // ClusterTolerations represents the tolerations. // optional ClusterTolerations []corev1.Toleration json:clusterTolerations,omitempty // SpreadConstraints represents a list of the scheduling constraints. // optional SpreadConstraints []SpreadConstraint json:spreadConstraints,omitempty // ReplicaScheduling represents the scheduling policy on dealing with the number of replicas // when propagating resources that have replicas in spec (e.g. deployments, statefulsets) to member clusters. // optional ReplicaScheduling *ReplicaSchedulingStrategy json:replicaScheduling,omitempty }关键约束有两点ClusterAffinities与ClusterAffinity不能共存若两者都未设置则任意集群都可作为调度候选。2.2ClusterAffinityTerm命名亲和项每个亲和项本质上是一个命名的ClusterAffinity// ClusterAffinityTerm selects a set of cluster. type ClusterAffinityTerm struct { // AffinityName is the name of the cluster group. // required AffinityName string json:affinityName ClusterAffinity json:,inline }ClusterAffinity以内联方式嵌入因此每个亲和项天然支持clusterNames、excludedClusters、labelSelector、fieldSelector等所有ClusterAffinity既有的选择子。调度阶段调度器按亲和项在 spec 中出现的先后顺序逐一评估若某项不满足限制则继续评估下一项。2.3 完整配置示例三个亲和项提案给出声明 3 个亲和项的配置示例apiVersion: policy.karmada.io/v1alpha1 kind: PropagationPolicy metadata: name: nginx spec: resourceSelectors: - apiVersion: apps/v1 kind: Deployment name: nginx placement: clusterAffinities: - affinityName: dc-shanghai clusterNames: - unavailable - affinityName: dc-beijing clusterNames: - member1 - affinityName: dc-hongkong clusterNames: - member2调度过程如下调度器首先评估名为dc-shanghai的亲和项尝试从中选出可行集群集合若该亲和项中找不到可行集群则转向下一项dc-beijing一旦成功选出可行集群集合不再继续评估后续项当member1不可用时借助 Karmada 的 Failover 故障转移能力调度器先在当前亲和项内寻找替代集群若失败再转向dc-hongkong亲和项。注意每个亲和项完全独立调度器在每次调度中只选取一个亲和项但允许同一个集群出现在多个亲和项中。三、状态追踪ResourceBinding新增SchedulerObservedAffinityName3.1 为什么需要记录正在评估的亲和项多调度组按序评估时一旦发生重新调度re-schedule调度器必须知道上一轮调度基于哪个亲和项才能从该亲和项继续而不是每次从头评估。为此提案在ResourceBinding的.status中新增字段SchedulerObservedAffinityName。3.2 字段定义与示例该字段已在 pkg/apis/work/v1alpha2/binding_types.go 中实现// ResourceBindingStatus represents the overall status of the strategy as well as the referenced resources. type ResourceBindingStatus struct { // SchedulerObservedGeneration is the generation(.metadata.generation) observed by the scheduler. // If SchedulerObservedGeneration is less than the generation in metadata means the scheduler hasnt confirmed // the scheduling result or hasnt done the schedule yet. // optional SchedulerObservedGeneration int64 json:schedulerObservedGeneration,omitempty // SchedulerObservedAffinityName is the affinity terms that // scheduler looking at. // optional SchedulerObservedAffinityName string json:schedulerObservingAffinityName,omitempty // Conditions contain the different condition statuses. // optional Conditions []metav1.Condition json:conditions,omitempty // AggregatedStatus represents status list of the resource running in each member cluster. // optional AggregatedStatus []AggregatedStatusItem json:aggregatedStatus,omitempty }一个真实的ResourceBinding状态示例如下status: aggregatedStatus: - applied: true clusterName: member1 health: Healthy status: availableReplicas: 2 readyReplicas: 2 replicas: 2 updatedReplicas: 2 conditions: - lastTransitionTime: 2023-02-04T09:38:20Z message: All works have been successfully applied reason: FullyAppliedSuccess status: True type: FullyApplied - lastTransitionTime: 2023-02-04T09:38:20Z message: Binding has been scheduled reason: BindingScheduled status: True type: Scheduled schedulerObservedGeneration: 2 SchedulerObservedAffinityName: ds-hongkong其中SchedulerObservedAffinityName: ds-hongkong表示当前调度结果基于名为ds-hongkong的亲和项重新调度时调度器应从该亲和项继续评估。3.3 调度器如何消费该字段从源码 pkg/scheduler/core/common.go 可以看到调度器在调度前会检查该字段并定位到对应的亲和项从而实现在失败/重调度场景下从上次停下的位置继续if spec.Placement.ClusterAffinity ! nil || spec.Placement.ClusterAffinities nil || len(status.SchedulerObservedAffinityName) 0 { // ... } for _, affinity : range spec.Placement.ClusterAffinities { if affinity.AffinityName status.SchedulerObservedAffinityName { // 从该亲和项继续调度 } }值得注意的是ClusterAffinities相关的判断逻辑也出现在 event_handler.go、cluster_affinity.go 及 group_clusters.go 中——多调度组并非只在调度入口生效而是贯穿了事件处理、亲和过滤与集群分组等多个调度环节。四、组件改造方案4.1 karmada-controller-manager在创建或更新ResourceBinding/ClusterResourceBinding时需要将PropagationPolicy/ClusterPropagationPolicy中新增的ClusterAffinities即提案所说的OrderedClusterAffinities同步到 Binding 中确保调度器拿到的是有序的亲和项列表。4.2 karmada-scheduler当前karmada-scheduler只运行单一循环每次调度只接受一个亲和项对应 ScheduleAlgorithm 接口 的实现。引入多调度组后ScheduleAlgorithm接口将被多次调用每次喂入不同的亲和项直到调度成功第 1 次调用传入第 1 个亲和项若调度成功则结束失败则传入第 2 个亲和项继续以此类推所有亲和项均失败则本次调度失败。4.3 karmada-webhookwebhook 需要承担两类额外校验防止误导性配置互斥校验ClusterAffinities与旧字段ClusterAffinity同时存在没有意义必须拒绝二者共存的配置唯一性校验所有亲和项的affinityName必须互不相同。4.4 适用范围说明提案主体聚焦于PropagationPolicy与ResourceBinding的改动但同样适用于ClusterPropagationPolicy与ClusterResourceBinding。4.5 测试计划所有现有测试必须通过本特性不引入破坏性变更新增 E2E 测试覆盖以下场景Duplicated调度类型下的工作负载传播Divided调度类型下的工作负载传播Failover 故障转移场景。五、备选方案对比为什么不用propagatePriority提案记录了一个最早被提出的替代思路由 PR #842 跟踪在clusterAffinity中引入新字段propagatePriority来声明集群优先级并复用 Kubernetes Pod affinity 的术语。示例配置如下apiVersion: policy.karmada.io/v1alpha1 kind: PropagationPolicy metadata: name: foo spec: resourceSelectors: - apiVersion: apps/v1 kind: Deployment name: foo placement: clusterAffinity: clusterNames: - member1 - member2 propagatePriority: - weight: 30 preference: matchExpressions: - key: topology operator: In values: - us - weight: 20 preference: matchExpressions: - key: topology operator: In values: - cn该方案通过复用 Kubernetes 的PreferredSchedulingTerm来表达集群偏好。最终被否决的原因是PreferredSchedulingTerm依赖 label selector 与 field selector 对集群分组matchExpressions嵌套层级过深很容易让配置难以维护。而最终选定的ClusterAffinities方案语义直观每个亲和项就是一个命名的ClusterAffinity配置扁平、可读性强也便于调度器按序执行。六、能力演进Overflow Cluster Affinities分级弹性溢出多调度组方案落地后社区又在其基础上演进出了Overflow Cluster Affinities能力详见 overflow-affinities 子提案。二者形成了互补ClusterAffinities多调度组多个候选集群组互斥用于多集群组隔离区域/厂商与故障转移每次调度最终只选择其中一个候选组或其子集Overflow Cluster Affinities溢出亲和只有一个候选集群组组内集群按层级依次使用用于成本优化与弹性伸缩。6.1 典型场景用户同时在自建 IDC GPU 集群与云上 GPU 集群运行推理负载希望降低 GPU 成本——只有流量峰值时才使用云 GPU而非常态负载。当流量变化触发 FHPAFederatedHPA伸缩时Karmada 应始终按IDC 优先扩容、云上优先缩容的顺序执行保证 GPU 使用成本效益最大化。6.2 API 扩展ClusterAffinityTerm新增可选字段OverflowAffinities它是一个有序数组层级越靠后调度优先级越低// ClusterAffinityTerm selects a set of cluster. type ClusterAffinityTerm struct { // AffinityName is the name of the cluster group. // required AffinityName string json:affinityName ClusterAffinity json:,inline // OverflowAffinities defines additional cluster groups that the scheduler // can progressively include when the primary group (defined by the inline // ClusterAffinity) has insufficient resources. Groups are expanded in order // and contracted in reverse during scale-down. // Can only be used together with the inline ClusterAffinity (the inline // ClusterAffinity serves as the primary/preferred group). // If a cluster appears in multiple OverflowClusterAffinity entries, it is // scheduled according to the first entry in which it appears; subsequent // occurrences of the same cluster are ignored. // optional OverflowAffinities []OverflowClusterAffinity json:overflowAffinities,omitempty } // OverflowClusterAffinity represents an overflow tier of candidate clusters. type OverflowClusterAffinity struct { // AffinityName is the name of the cluster group. // required AffinityName string json:affinityName ClusterAffinity json:,inline }6.3 配置示例本地集群优先、公有云溢出apiVersion: policy.karmada.io/v1alpha1 kind: PropagationPolicy metadata: name: nginx spec: resourceSelectors: - apiVersion: apps/v1 kind: Deployment name: nginx placement: clusterAffinities: - affinityName: on-prem clusterNames: - cluster1 overflowAffinities: - affinityName: public-cloud clusterNames: - cluster2 - cluster3 replicaScheduling: replicaDivisionPreference: Weighted replicaSchedulingType: Divided weightPreference: dynamicWeight: AvailableReplicas调度语义调度器先尝试把 Deployment 的全部副本放入最高优先级层级on-premcluster1若该组集群无法容纳全部副本剩余副本按声明顺序逐级溢出到overflowAffinities定义的集群组第一个溢出亲和为第二优先级层级第二个为第三优先级层级依此类推若cluster1不可用则直接使用下一个可用溢出层级cluster2、cluster3调度当cluster1恢复可用时不会自动触发工作负载迁移避免副本抖动用户需显式使用WorkloadRebalancer资源触发重新调度或通过调整工作负载副本数间接触发。上图完整展示了溢出调度的流程副本优先填满高优先级层级层级资源耗尽后依次溢出到下一层级所有层级仍无法容纳全部副本时调度失败。6.4 调度器五阶段改造要点溢出能力对karmada-scheduler的改造集中在schedule(Cluster)ResourceBindingWithClusterAffinities流程的 Filter / Select / Assign 三个阶段入口调度入口schedule(Cluster)ResourceBindingWithClusterAffinities无需改动Filter 阶段ClusterAffinity插件需调整为支持溢出亲和——只要集群满足ClusterAffinityTerm内任意亲和组的调度条件即可通过过滤Score 阶段无需改动Select 阶段原逻辑优先选择得分最高的集群溢出场景下需调整为优先选择属于主组及更早溢出层级的集群保证高优先级层级被完全利用后副本才溢出到低优先级层级Assign 阶段按亲和组层级对集群分组按层级顺序迭代分配剩余副本每层尽量分配直至资源耗尽或副本分配完毕层内分配逻辑与现有实现保持一致全部副本分配完成即成功遍历完所有层级仍有未分配副本则调度失败。6.5 webhook 新增校验限制overflowAffinities必须与内联ClusterAffinity同时使用不能独立设置限制overflowAffinities只能配合Divided副本调度类型与dynamicWeight如AvailableReplicas使用——因为只有该模式在分配副本时会考虑成员集群的可用资源这是分级溢出行为的前提。6.6 设计取舍为何不用其他三种方案子提案还比较了三种被否决的备选方案Approach 1在ClusterAffinity上加Supplements该字段会隐式传导到ClusterAffinities增加配置与决策复杂度且溢出调度本质是多集群组场景嵌入单个ClusterAffinity在语义上错位Approach 2引入AffinityStrategy.Mode切换互斥/溢出语义会破坏现有ClusterAffinities固有的互斥契约模式开关可能让既有用户困惑Approach 3新增与ClusterAffinities平级的PreferredClusterAffinities溢出亲和只是多调度组能力的补充与子集新增顶层 API 会过度扩大功能范围、碎片化 API 表面。最终方案选择在ClusterAffinityTerm内部扩展OverflowAffinities既保持了多调度组互斥语义不变又以最小 API 增量补齐了分级弹性伸缩能力。七、实践要点总结语义区分clusterAffinities各亲和项互斥按声明顺序选中即止overflowAffinities各层级共存按顺序用完即溢配置约束clusterAffinity与clusterAffinities不可共存affinityName全局唯一溢出亲和必须与DivideddynamicWeight搭配状态恢复ResourceBinding.status.schedulerObservingAffinityName记录调度器当前评估的亲和项是重调度继续执行的关键指针故障转移多调度组与 Failover 能力 协同——当前亲和项内找不到替代集群时调度器会转向下一个亲和项版本兼容本特性向后兼容旧 YAML 配置无需改动即可在新版本上以原有行为运行源码参照API 定义见 propagation_types.go 与 binding_types.go调度器消费逻辑见 common.go、event_handler.go 及 cluster_affinity.go设计文档见 multi-scheduling-group/README.md 与 overflow-affinities/README.md。【免费下载链接】karmadaOpen, Multi-Cloud, Multi-Cluster Kubernetes Orchestration项目地址: https://gitcode.com/GitHub_Trending/ka/karmada创作声明：本文部分内容由AI辅助生成（AIGC），仅供参考
Karmada 多调度组（Multiple Scheduling Group）深度解析：从多组亲和到分级弹性调度的完整实现指南
Karmada 多调度组Multiple Scheduling Group深度解析从多组亲和到分级弹性调度的完整实现指南【免费下载链接】karmadaOpen, Multi-Cloud, Multi-Cluster Kubernetes Orchestration项目地址: https://gitcode.com/GitHub_Trending/ka/karmada导读Karmada 是面向多云、多集群场景的 Kubernetes 编排系统其PropagationPolicy通过.spec.placement.clusterAffinity声明候选集群集合。但单一亲和组在主集群优先、备集群兜底的实际诉求面前力不从心。本文以仓库内 multi-scheduling-group 设计提案 为骨架完整讲解clusterAffinities多调度组 API 的设计动机、调度语义、状态追踪字段与组件改造方案并结合 propagation_types.go 与 binding_types.go 等源码给出实现级佐证同时延伸介绍其后续演进——分级弹性溢出的 Overflow Cluster Affinities 能力。读完本文你将掌握如何用多调度组实现首选组 灾备组的调度策略以及多调度组在调度器中的真实执行链路。一、背景与动机为什么需要多个调度组1.1 现有clusterAffinity的局限在引入多调度组之前PropagationPolicy只能声明一组候选集群即.spec.placement.clusterAffinity。如下示例声明了member1、member2两个候选集群apiVersion: policy.karmada.io/v1alpha1 kind: PropagationPolicy metadata: name: foo spec: resourceSelectors: - apiVersion: apps/v1 kind: Deployment name: foo placement: clusterAffinity: clusterNames: - member1 - member2clusterAffinity将一组候选集群交给karmada-scheduler调度器再依据spreadConstraint、过滤插件filter plugins等限制条件在候选集群中做出调度决策结果只有两种成功为引用资源选出一组集群通常为候选集的子集失败无法选出满足所有限制条件的集群组。1.2 现实诉求主备分组与成本优先集群管理员通常会按分类维度如云厂商、用途将集群划分为不同分组期望把工作负载优先部署到首选分组当首选分组不满足调度限制如资源不足时再回退到备用分组。Karmada 社区从最终用户处收到了大量类似反馈例如 issue #780 与 #2085 所描述的诉求。提案中给出了两个典型用户故事成本优先用户在本地数据中心拥有私有集群同时也购买了云厂商如 AWS、Google Cloud的托管集群。由于托管集群成本更高用户希望优先部署在私有集群上把托管集群作为备份灾备优先用户拥有主集群与备份集群希望应用默认部署在主集群当主集群因数据中心断电、维护等原因不可用时自动迁移到备份集群。1.3 目标与非目标Goals扩展PropagationPolicyAPI使其能够持有多个亲和组声明扩展ResourceBindingAPI用于标注调度器当前正在评估的亲和组为karmada-controller-manager、karmada-webhook、karmada-scheduler等组件提出实现思路。Non-Goals明确不在范围内同一组内集群的相对优先级本提案聚焦于引入多个亲和组同一组内集群的优先级问题由weightPreference.spec.placement.replicaScheduling.weightPreference或为调度器定义集群打分策略来解决调度再平衡re-balance用户期望调度器出于其他目的执行额外调度的场景如讨论 #3069应由另一份独立提案跟踪。1.4 兼容性与风险本提案保持向后兼容旧版本 Karmada 构建的系统可无缝迁移到新版本旧配置YAML可直接应用于新版 Karmada 且行为不变。二、API 设计ClusterAffinities与ClusterAffinityTerm2.1Placement新增ClusterAffinities字段提案在.spec.placement中新增字段ClusterAffinities用于声明多个亲和项affinity term。该字段在源码 pkg/apis/policy/v1alpha1/propagation_types.go 中已落地核心注释完整继承了提案语义// Placement represents the rule for select clusters. type Placement struct { // ClusterAffinity represents scheduling restrictions to a certain set of clusters. // If not set, any cluster can be scheduling candidate. // optional ClusterAffinity *ClusterAffinity json:clusterAffinity,omitempty // ClusterAffinities represents scheduling restrictions to multiple cluster // groups that indicated by ClusterAffinityTerm. // // The scheduler will evaluate these groups one by one in the order they // appear in the spec, the group that does not satisfy scheduling restrictions // will be ignored which means all clusters in this group will not be selected // unless it also belongs to the next group(a cluster could belong to multiple // groups). // // If none of the groups satisfy the scheduling restrictions, then scheduling // fails, which means no cluster will be selected. // // Note: // 1. ClusterAffinities can not co-exist with ClusterAffinity. // 2. If both ClusterAffinity and ClusterAffinities are not set, any cluster // can be scheduling candidates. // // optional ClusterAffinities []ClusterAffinityTerm json:clusterAffinities,omitempty // ClusterTolerations represents the tolerations. // optional ClusterTolerations []corev1.Toleration json:clusterTolerations,omitempty // SpreadConstraints represents a list of the scheduling constraints. // optional SpreadConstraints []SpreadConstraint json:spreadConstraints,omitempty // ReplicaScheduling represents the scheduling policy on dealing with the number of replicas // when propagating resources that have replicas in spec (e.g. deployments, statefulsets) to member clusters. // optional ReplicaScheduling *ReplicaSchedulingStrategy json:replicaScheduling,omitempty }关键约束有两点ClusterAffinities与ClusterAffinity不能共存若两者都未设置则任意集群都可作为调度候选。2.2ClusterAffinityTerm命名亲和项每个亲和项本质上是一个命名的ClusterAffinity// ClusterAffinityTerm selects a set of cluster. type ClusterAffinityTerm struct { // AffinityName is the name of the cluster group. // required AffinityName string json:affinityName ClusterAffinity json:,inline }ClusterAffinity以内联方式嵌入因此每个亲和项天然支持clusterNames、excludedClusters、labelSelector、fieldSelector等所有ClusterAffinity既有的选择子。调度阶段调度器按亲和项在 spec 中出现的先后顺序逐一评估若某项不满足限制则继续评估下一项。2.3 完整配置示例三个亲和项提案给出声明 3 个亲和项的配置示例apiVersion: policy.karmada.io/v1alpha1 kind: PropagationPolicy metadata: name: nginx spec: resourceSelectors: - apiVersion: apps/v1 kind: Deployment name: nginx placement: clusterAffinities: - affinityName: dc-shanghai clusterNames: - unavailable - affinityName: dc-beijing clusterNames: - member1 - affinityName: dc-hongkong clusterNames: - member2调度过程如下调度器首先评估名为dc-shanghai的亲和项尝试从中选出可行集群集合若该亲和项中找不到可行集群则转向下一项dc-beijing一旦成功选出可行集群集合不再继续评估后续项当member1不可用时借助 Karmada 的 Failover 故障转移能力调度器先在当前亲和项内寻找替代集群若失败再转向dc-hongkong亲和项。注意每个亲和项完全独立调度器在每次调度中只选取一个亲和项但允许同一个集群出现在多个亲和项中。三、状态追踪ResourceBinding新增SchedulerObservedAffinityName3.1 为什么需要记录正在评估的亲和项多调度组按序评估时一旦发生重新调度re-schedule调度器必须知道上一轮调度基于哪个亲和项才能从该亲和项继续而不是每次从头评估。为此提案在ResourceBinding的.status中新增字段SchedulerObservedAffinityName。3.2 字段定义与示例该字段已在 pkg/apis/work/v1alpha2/binding_types.go 中实现// ResourceBindingStatus represents the overall status of the strategy as well as the referenced resources. type ResourceBindingStatus struct { // SchedulerObservedGeneration is the generation(.metadata.generation) observed by the scheduler. // If SchedulerObservedGeneration is less than the generation in metadata means the scheduler hasnt confirmed // the scheduling result or hasnt done the schedule yet. // optional SchedulerObservedGeneration int64 json:schedulerObservedGeneration,omitempty // SchedulerObservedAffinityName is the affinity terms that // scheduler looking at. // optional SchedulerObservedAffinityName string json:schedulerObservingAffinityName,omitempty // Conditions contain the different condition statuses. // optional Conditions []metav1.Condition json:conditions,omitempty // AggregatedStatus represents status list of the resource running in each member cluster. // optional AggregatedStatus []AggregatedStatusItem json:aggregatedStatus,omitempty }一个真实的ResourceBinding状态示例如下status: aggregatedStatus: - applied: true clusterName: member1 health: Healthy status: availableReplicas: 2 readyReplicas: 2 replicas: 2 updatedReplicas: 2 conditions: - lastTransitionTime: 2023-02-04T09:38:20Z message: All works have been successfully applied reason: FullyAppliedSuccess status: True type: FullyApplied - lastTransitionTime: 2023-02-04T09:38:20Z message: Binding has been scheduled reason: BindingScheduled status: True type: Scheduled schedulerObservedGeneration: 2 SchedulerObservedAffinityName: ds-hongkong其中SchedulerObservedAffinityName: ds-hongkong表示当前调度结果基于名为ds-hongkong的亲和项重新调度时调度器应从该亲和项继续评估。3.3 调度器如何消费该字段从源码 pkg/scheduler/core/common.go 可以看到调度器在调度前会检查该字段并定位到对应的亲和项从而实现在失败/重调度场景下从上次停下的位置继续if spec.Placement.ClusterAffinity ! nil || spec.Placement.ClusterAffinities nil || len(status.SchedulerObservedAffinityName) 0 { // ... } for _, affinity : range spec.Placement.ClusterAffinities { if affinity.AffinityName status.SchedulerObservedAffinityName { // 从该亲和项继续调度 } }值得注意的是ClusterAffinities相关的判断逻辑也出现在 event_handler.go、cluster_affinity.go 及 group_clusters.go 中——多调度组并非只在调度入口生效而是贯穿了事件处理、亲和过滤与集群分组等多个调度环节。四、组件改造方案4.1 karmada-controller-manager在创建或更新ResourceBinding/ClusterResourceBinding时需要将PropagationPolicy/ClusterPropagationPolicy中新增的ClusterAffinities即提案所说的OrderedClusterAffinities同步到 Binding 中确保调度器拿到的是有序的亲和项列表。4.2 karmada-scheduler当前karmada-scheduler只运行单一循环每次调度只接受一个亲和项对应 ScheduleAlgorithm 接口 的实现。引入多调度组后ScheduleAlgorithm接口将被多次调用每次喂入不同的亲和项直到调度成功第 1 次调用传入第 1 个亲和项若调度成功则结束失败则传入第 2 个亲和项继续以此类推所有亲和项均失败则本次调度失败。4.3 karmada-webhookwebhook 需要承担两类额外校验防止误导性配置互斥校验ClusterAffinities与旧字段ClusterAffinity同时存在没有意义必须拒绝二者共存的配置唯一性校验所有亲和项的affinityName必须互不相同。4.4 适用范围说明提案主体聚焦于PropagationPolicy与ResourceBinding的改动但同样适用于ClusterPropagationPolicy与ClusterResourceBinding。4.5 测试计划所有现有测试必须通过本特性不引入破坏性变更新增 E2E 测试覆盖以下场景Duplicated调度类型下的工作负载传播Divided调度类型下的工作负载传播Failover 故障转移场景。五、备选方案对比为什么不用propagatePriority提案记录了一个最早被提出的替代思路由 PR #842 跟踪在clusterAffinity中引入新字段propagatePriority来声明集群优先级并复用 Kubernetes Pod affinity 的术语。示例配置如下apiVersion: policy.karmada.io/v1alpha1 kind: PropagationPolicy metadata: name: foo spec: resourceSelectors: - apiVersion: apps/v1 kind: Deployment name: foo placement: clusterAffinity: clusterNames: - member1 - member2 propagatePriority: - weight: 30 preference: matchExpressions: - key: topology operator: In values: - us - weight: 20 preference: matchExpressions: - key: topology operator: In values: - cn该方案通过复用 Kubernetes 的PreferredSchedulingTerm来表达集群偏好。最终被否决的原因是PreferredSchedulingTerm依赖 label selector 与 field selector 对集群分组matchExpressions嵌套层级过深很容易让配置难以维护。而最终选定的ClusterAffinities方案语义直观每个亲和项就是一个命名的ClusterAffinity配置扁平、可读性强也便于调度器按序执行。六、能力演进Overflow Cluster Affinities分级弹性溢出多调度组方案落地后社区又在其基础上演进出了Overflow Cluster Affinities能力详见 overflow-affinities 子提案。二者形成了互补ClusterAffinities多调度组多个候选集群组互斥用于多集群组隔离区域/厂商与故障转移每次调度最终只选择其中一个候选组或其子集Overflow Cluster Affinities溢出亲和只有一个候选集群组组内集群按层级依次使用用于成本优化与弹性伸缩。6.1 典型场景用户同时在自建 IDC GPU 集群与云上 GPU 集群运行推理负载希望降低 GPU 成本——只有流量峰值时才使用云 GPU而非常态负载。当流量变化触发 FHPAFederatedHPA伸缩时Karmada 应始终按IDC 优先扩容、云上优先缩容的顺序执行保证 GPU 使用成本效益最大化。6.2 API 扩展ClusterAffinityTerm新增可选字段OverflowAffinities它是一个有序数组层级越靠后调度优先级越低// ClusterAffinityTerm selects a set of cluster. type ClusterAffinityTerm struct { // AffinityName is the name of the cluster group. // required AffinityName string json:affinityName ClusterAffinity json:,inline // OverflowAffinities defines additional cluster groups that the scheduler // can progressively include when the primary group (defined by the inline // ClusterAffinity) has insufficient resources. Groups are expanded in order // and contracted in reverse during scale-down. // Can only be used together with the inline ClusterAffinity (the inline // ClusterAffinity serves as the primary/preferred group). // If a cluster appears in multiple OverflowClusterAffinity entries, it is // scheduled according to the first entry in which it appears; subsequent // occurrences of the same cluster are ignored. // optional OverflowAffinities []OverflowClusterAffinity json:overflowAffinities,omitempty } // OverflowClusterAffinity represents an overflow tier of candidate clusters. type OverflowClusterAffinity struct { // AffinityName is the name of the cluster group. // required AffinityName string json:affinityName ClusterAffinity json:,inline }6.3 配置示例本地集群优先、公有云溢出apiVersion: policy.karmada.io/v1alpha1 kind: PropagationPolicy metadata: name: nginx spec: resourceSelectors: - apiVersion: apps/v1 kind: Deployment name: nginx placement: clusterAffinities: - affinityName: on-prem clusterNames: - cluster1 overflowAffinities: - affinityName: public-cloud clusterNames: - cluster2 - cluster3 replicaScheduling: replicaDivisionPreference: Weighted replicaSchedulingType: Divided weightPreference: dynamicWeight: AvailableReplicas调度语义调度器先尝试把 Deployment 的全部副本放入最高优先级层级on-premcluster1若该组集群无法容纳全部副本剩余副本按声明顺序逐级溢出到overflowAffinities定义的集群组第一个溢出亲和为第二优先级层级第二个为第三优先级层级依此类推若cluster1不可用则直接使用下一个可用溢出层级cluster2、cluster3调度当cluster1恢复可用时不会自动触发工作负载迁移避免副本抖动用户需显式使用WorkloadRebalancer资源触发重新调度或通过调整工作负载副本数间接触发。上图完整展示了溢出调度的流程副本优先填满高优先级层级层级资源耗尽后依次溢出到下一层级所有层级仍无法容纳全部副本时调度失败。6.4 调度器五阶段改造要点溢出能力对karmada-scheduler的改造集中在schedule(Cluster)ResourceBindingWithClusterAffinities流程的 Filter / Select / Assign 三个阶段入口调度入口schedule(Cluster)ResourceBindingWithClusterAffinities无需改动Filter 阶段ClusterAffinity插件需调整为支持溢出亲和——只要集群满足ClusterAffinityTerm内任意亲和组的调度条件即可通过过滤Score 阶段无需改动Select 阶段原逻辑优先选择得分最高的集群溢出场景下需调整为优先选择属于主组及更早溢出层级的集群保证高优先级层级被完全利用后副本才溢出到低优先级层级Assign 阶段按亲和组层级对集群分组按层级顺序迭代分配剩余副本每层尽量分配直至资源耗尽或副本分配完毕层内分配逻辑与现有实现保持一致全部副本分配完成即成功遍历完所有层级仍有未分配副本则调度失败。6.5 webhook 新增校验限制overflowAffinities必须与内联ClusterAffinity同时使用不能独立设置限制overflowAffinities只能配合Divided副本调度类型与dynamicWeight如AvailableReplicas使用——因为只有该模式在分配副本时会考虑成员集群的可用资源这是分级溢出行为的前提。6.6 设计取舍为何不用其他三种方案子提案还比较了三种被否决的备选方案Approach 1在ClusterAffinity上加Supplements该字段会隐式传导到ClusterAffinities增加配置与决策复杂度且溢出调度本质是多集群组场景嵌入单个ClusterAffinity在语义上错位Approach 2引入AffinityStrategy.Mode切换互斥/溢出语义会破坏现有ClusterAffinities固有的互斥契约模式开关可能让既有用户困惑Approach 3新增与ClusterAffinities平级的PreferredClusterAffinities溢出亲和只是多调度组能力的补充与子集新增顶层 API 会过度扩大功能范围、碎片化 API 表面。最终方案选择在ClusterAffinityTerm内部扩展OverflowAffinities既保持了多调度组互斥语义不变又以最小 API 增量补齐了分级弹性伸缩能力。七、实践要点总结语义区分clusterAffinities各亲和项互斥按声明顺序选中即止overflowAffinities各层级共存按顺序用完即溢配置约束clusterAffinity与clusterAffinities不可共存affinityName全局唯一溢出亲和必须与DivideddynamicWeight搭配状态恢复ResourceBinding.status.schedulerObservingAffinityName记录调度器当前评估的亲和项是重调度继续执行的关键指针故障转移多调度组与 Failover 能力 协同——当前亲和项内找不到替代集群时调度器会转向下一个亲和项版本兼容本特性向后兼容旧 YAML 配置无需改动即可在新版本上以原有行为运行源码参照API 定义见 propagation_types.go 与 binding_types.go调度器消费逻辑见 common.go、event_handler.go 及 cluster_affinity.go设计文档见 multi-scheduling-group/README.md 与 overflow-affinities/README.md。【免费下载链接】karmadaOpen, Multi-Cloud, Multi-Cluster Kubernetes Orchestration项目地址: https://gitcode.com/GitHub_Trending/ka/karmada创作声明：本文部分内容由AI辅助生成（AIGC），仅供参考
