SkyWalking 集成 Elasticsearch 监控基于 elasticsearch-exporter 与 OpenTelemetry Collector 的完整实践指南【免费下载链接】skywalkingAPM, Application Performance Monitoring System项目地址: https://gitcode.com/gh_mirrors/sky/skywalking本文基于 Apache SkyWalking 官方文档 backend-elasticsearch-monitoring.md 编写并结合当前仓库中的 MAL 规则、OpenTelemetry Collector 配置与 e2e 测试代码进行深度补充。核心主题是如何在 SkyWalking OAP 中接入 Elasticsearch 集群的指标监控。通过本指南读者可以掌握从 elasticsearch-exporter 采集指标、经 OpenTelemetry Collector 转发、最终在 SkyWalking 中以Layer: ELASTICSEARCH服务形态呈现的完整链路并理解集群 / 节点 / 索引三个维度的监控面板与底层指标表达式。监控方案总览SkyWalking 本身并不直接向 Elasticsearch 发起指标拉取而是采用「Prometheus 生态采集 OpenTelemetry 协议转发 MAL 指标计算」的分层架构elasticsearch-exporterprometheus-community/elasticsearch_exporter负责从 Elasticsearch 集群采集原始指标数据并以 Prometheus 格式暴露在:9114/metrics。OpenTelemetry Collector通过Prometheus Receiver抓取 elasticsearch-exporter 暴露的指标再由OpenTelemetry gRPC exporterotlp推送到 SkyWalking OAP Server 的 OpenTelemetry receiver。SkyWalking OAP Server使用MALMetric Analysis Language指标分析语言表达式对指标进行过滤、计算、聚合最终存储到后端并暴露为Layer: ELASTICSEARCH服务。其数据流可用下图概括Elasticsearch Cluster │ (1) 内部状态 / 统计 API ▼ elasticsearch-exporter (:9114/metrics) │ (2) Prometheus 协议抓取 ▼ OpenTelemetry Collector (Prometheus Receiver → batch → otlp exporter) │ (3) OTLP gRPC 推送 ▼ SkyWalking OAP Server (OpenTelemetry receiver) │ (4) MAL 表达式过滤 / 计算 / 聚合 ▼ Meter System → 存储 → UI 监控面板 (Layer: ELASTICSEARCH)这个方案的价值在于SkyWalking 不重复造轮子直接复用 Prometheus 社区成熟的 exporter 与 OpenTelemetry 标准协议同时通过 Meter System 将第三方指标与 SkyWalking 自身的服务/实例/端点模型统一建模。快速开始三步骤接入步骤 1部署并配置 elasticsearch-exporter官方推荐使用 elasticsearch_exporter 中的部署方式es-exporter: image: quay.io/prometheuscommunity/elasticsearch-exporter:v1.5.0 expose: - 9114 depends_on: es: condition: service_healthy command: - --log.formatlogfmt - --log.levelinfo - --es.urihttp://es:9200 # 指向 Elasticsearch 的 HTTP 地址 - --es.all # 采集集群中所有节点 - --es.indices # 采集索引级指标 - --es.indices_settings # 采集索引设置 - --es.indices_mappings # 采集索引映射 - --es.shards # 采集分片级指标 - --es.snapshots # 采集快照指标 - --es.timeout30s - --web.telemetry-path/metrics说明--es.indices等开关决定了 exporter 是否暴露索引级指标而 SkyWalking 的 Index 监控面板正是依赖这些指标因此若希望获得索引维度的监控数据务必开启--es.indices与--es.indices_settings、--es.shards等配套参数。步骤 2部署 OpenTelemetry Collector 并配置管道OpenTelemetry Collector 的配置示例位于 otel-collector-config.yaml核心管道如下receivers: prometheus: config: scrape_configs: - job_name: elasticsearch-monitoring # 该 job_name 与 MAL 规则中的 filter 严格对应 scrape_interval: 10s static_configs: - targets: [es-exporter:9114] processors: batch: exporters: # 若使用 otlp exporter请确保 OAP 版本 9.2.0 otlp: endpoint: oap:11800 tls: insecure: true service: pipelines: metrics: receivers: - prometheus processors: - batch exporters: - otlp这里有两个必须严格一致的关键点job_name采集任务的job_name必须设置为elasticsearch-monitoring。这是因为三个 MAL 规则文件的头部都带有过滤器filter: { tags - tags.job_name elasticsearch-monitoring } # The OpenTelemetry job name如果 job_name 不一致OAP 会直接过滤掉这批指标导致监控面板无数据。OTLP endpoint指向 OAP 的 gRPC 端口默认11800。receiver-otel仅支持otlp这一种 handler对应opentelemetry-receiver.md中的说明。步骤 3配置 SkyWalking OpenTelemetry receiverSkyWalking 侧需要激活receiver-otel模块。在 application.yml 中可以看到默认配置receiver-otel: selector: ${SW_OTEL_RECEIVER:default} default: enabledHandlers: ${SW_OTEL_RECEIVER_ENABLED_HANDLERS:otlp-metrics,otlp-logs} enabledOtelMetricsRules: ${SW_OTEL_RECEIVER_ENABLED_OTEL_METRICS_RULES:...,elasticsearch/*,...}通过环境变量SW_OTEL_RECEIVERdefault激活 OpenTelemetry receiverenabledOtelMetricsRules中已默认包含elasticsearch/*通配即加载otel-rules/elasticsearch/目录下全部三个规则文件若你的部署环境没有启用默认规则可以显式设置为SW_OTEL_RECEIVER_ENABLED_OTEL_METRICS_RULESelasticsearch/*。关于 OpenTelemetry receiver 的详细说明可参考 opentelemetry-receiver.md。监控模型Cluster / Node / Index 三维建模接入完成后Elasticsearch 监控会以Layer: ELASTICSEARCH的服务形态出现在 OAP 中其建模方式为每个Elasticsearch 集群对应一个Service集群内的每个节点node对应一个Instance集群内的每个索引index对应一个Endpoint。这一映射关系直接体现在三个 MAL 规则文件的expSuffix中规则文件expSuffix映射维度elasticsearch-cluster.yaml.service([cluster], Layer.ELASTICSEARCH)集群 → Serviceelasticsearch-node.yaml.instance([cluster], [name], Layer.ELASTICSEARCH)节点 → Instanceelasticsearch-index.yaml.endpoint([cluster], [index], Layer.ELASTICSEARCH)索引 → Endpoint同时三个文件都以如下表达式对集群名打标签并加上命名空间前缀expSuffix: tag({tags - tags.cluster elasticsearch:: tags.cluster}).service([cluster], Layer.ELASTICSEARCH)这意味着服务名会呈现为elasticsearch::cluster_name的形式例如 e2e 测试中的elasticsearch::cluster-e2e见 elasticsearch-cases.yaml 中的查询语句。集群Cluster级指标集群级监控面板覆盖了健康状态、分片分布、节点规模、资源使用等维度指标均以meter_elasticsearch_cluster_为前缀。完整清单如下监控面板指标名说明数据来源Cluster Healthmeter_elasticsearch_cluster_health_status所有主分片与副本分片是否分配成功颜色状态elasticsearch-exporterTripped Of Breakersmeter_elasticsearch_cluster_breakers_tripped熔断器被触发次数elasticsearch-exporterNodesmeter_elasticsearch_cluster_nodes集群中的节点数elasticsearch-exporterData Nodesmeter_elasticsearch_cluster_data_nodes集群中的数据节点数elasticsearch-exporterPending Tasksmeter_elasticsearch_cluster_pending_tasks_total尚未执行的集群级变更任务数elasticsearch-exporterCPU Usage Avg. (%)meter_elasticsearch_cluster_cpu_usage_avg集群级进程 CPU 使用百分比elasticsearch-exporterJVM Memory Used Avg. (%)meter_elasticsearch_cluster_jvm_memory_used_avg集群级 JVM 内存使用百分比elasticsearch-exporterOpen Filesmeter_elasticsearch_cluster_open_file_count打开的文件描述符数elasticsearch-exporterActive Primary Shardsmeter_elasticsearch_cluster_primary_shards_total集群中主分片总数跨所有索引的聚合elasticsearch-exporterActive Shardsmeter_elasticsearch_cluster_shards_total所有索引的全部分片聚合总数含副本分片elasticsearch-exporterInitializing Shardsmeter_elasticsearch_cluster_initializing_shards_total正在新建的分片数elasticsearch-exporterDelayed Unassigned Shardsmeter_elasticsearch_cluster_delayed_unassigned_shards_total为降低重分配开销而延迟的分片数elasticsearch-exporterRelocating Shardsmeter_elasticsearch_cluster_relocating_shards_total正在节点间迁移的分片数elasticsearch-exporterUnassigned Shardsmeter_elasticsearch_cluster_unassigned_shards_total存在于集群状态但无法在集群中找到的分片数elasticsearch-exporter以 elasticsearch-cluster.yaml 中的若干规则为例可以看到底层 MAL 表达式的计算逻辑metricPrefix: meter_elasticsearch_cluster metricsRules: # cluster health按 cluster 与 color 标签求和health 为 1 即健康 - name: health_status exp: elasticsearch_cluster_health_status.tagNotEqual(cluster,unknown_cluster).valueEqual(1).sum([cluster , color]) # 熔断次数每分钟增量 - name: breakers_tripped exp: elasticsearch_breakers_tripped.tagNotEqual(cluster,unknown_cluster).sum([cluster]).increase(PT1M) # 节点数 / 数据节点数 - name: nodes exp: elasticsearch_cluster_health_number_of_nodes.tagNotEqual(cluster,unknown_cluster).sum([cluster]) - name: data_nodes exp: elasticsearch_cluster_health_number_of_data_nodes.tagNotEqual(cluster,unknown_cluster).sum([cluster]) # JVM 内存使用率 used / max * 100 - name: jvm_memory_used_avg exp: elasticsearch_jvm_memory_used_bytes.tagNotEqual(cluster,unknown_cluster).sum([cluster]) / elasticsearch_jvm_memory_max_bytes.tagNotEqual(cluster,unknown_cluster).sum([cluster]) * 100值得注意的实现细节几乎每个表达式都带.tagNotEqual(cluster,unknown_cluster)用于排除unknown_cluster标签的数据避免脏数据污染聚合结果速率型指标如熔断次数使用.increase(PT1M)计算每分钟增量时间窗口采用 ISO-8601 格式PT1M即 1 分钟。节点Node级指标节点级监控以meter_elasticsearch_node_为前缀覆盖 JVM、CPU、文档、段segment、磁盘、网络、GC、熔断器、索引操作速率等维度。完整清单如下监控面板单位指标名说明数据来源Node Rulesmeter_elasticsearch_node_rules节点角色elasticsearch-exporterJVM Memory UsedMBmeter_elasticsearch_node_jvm_memory_used节点级 JVM 内存使用量elasticsearch-exporterCPU Percent%meter_elasticsearch_node_process_cpu_percent节点级进程 CPU 使用百分比elasticsearch-exporterDocumentsmeter_elasticsearch_node_indices_docs该节点上的索引文档数elasticsearch-exporterSegmentsmeter_elasticsearch_node_segment_count该节点上的索引段数量elasticsearch-exporterDisk Free SpaceGBmeter_elasticsearch_node_all_disk_free_space所有块设备上的可用空间elasticsearch-exporterOpen Filesmeter_elasticsearch_node_open_file_count打开的文件描述符数elasticsearch-exporterProcess CPU Usage Percent%meter_elasticsearch_node_process_cpu_percent进程 CPU 使用百分比elasticsearch-exporterOS CPU usage percent%meter_elasticsearch_node_os_cpu_percent操作系统 CPU 使用百分比elasticsearch-exporterLoad Averagemeter_elasticsearch_node_os_load1 / meter_elasticsearch_node_os_load5 / meter_elasticsearch_node_os_load15短/中/长期负载均值elasticsearch-exporterJVM Memory UsageMBmeter_elasticsearch_node_jvm_memory_nonheap_used / meter_elasticsearch_node_jvm_memory_heap_used / meter_elasticsearch_node_jvm_memory_heap_max各区域 JVM 内存使用情况elasticsearch-exporterJVM Pool Peak UsedMBmeter_elasticsearch_node_jvm_memory_pool_peak_usedJVM 各池当前使用量elasticsearch-exporterGC Countmeter_elasticsearch_node_jvm_gc_countJVM GC 运行次数elasticsearch-exporterGC Timems/minmeter_elasticsearch_node_jvm_gc_timeGC 运行时间elasticsearch-exporterAll Operations ReqRatemeter_elasticsearch_node_indices_*_req_rate节点上的所有操作请求速率elasticsearch-exporterIndexing Ratereqpsmeter_elasticsearch_node_indices_indexing_index_total_req_rate / meter_elasticsearch_node_indices_indexing_index_total_proc_rate节点上的索引写入速率elasticsearch-exporterSearching Ratereqpsmeter_elasticsearch_node_indices_search_fetch_total_req_rate / meter_elasticsearch_node_indices_search_query_time_seconds_proc_rate节点上的搜索速率elasticsearch-exporterTotal Translog Operationsmeter_elasticsearch_node_indices_translog_operationstranslog 总操作数elasticsearch-exporterTotal Translog SizeMBmeter_elasticsearch_node_indices_translog_sizetranslog 总大小elasticsearch-exporterTripped For Breakersmeter_elasticsearch_node_breakers_tripped熔断器触发次数elasticsearch-exporterEstimated Size Of BreakerMBmeter_elasticsearch_node_breakers_estimated_size熔断器估算大小elasticsearch-exporterDocuments CountKB/smeter_elasticsearch_node_indices_docs该节点文档数量elasticsearch-exporterMerged Documents Countcount/smeter_elasticsearch_node_indices_merges_docs_total累计合并文档数elasticsearch-exporterDeleted Documents Countmeter_elasticsearch_node_indices_docs_deleted_total该节点被删除文档数elasticsearch-exporterDocuments Index Ratecalls/smeter_elasticsearch_node_indices_indexing_index_total_req_rate每秒索引写入调用数elasticsearch-exporterMerged Documents RateMB/smeter_elasticsearch_node_indices_merges_total_size_bytes_total每秒合并数据量elasticsearch-exporterDocuments Deleted Ratedocs/smeter_elasticsearch_node_indices_docs_deleted该节点每秒删除文档数elasticsearch-exporterCount Of Index Segmentsmeter_elasticsearch_node_segment_count该节点索引段数量elasticsearch-exporterCurrent Memory Size Of SegmentsMBmeter_elasticsearch_node_segment_memory段当前内存大小elasticsearch-exporterNetworkbytes/secmeter_elasticsearch_node_network_send_bytes / meter_elasticsearch_node_network_receive_bytes发送与接收字节总数elasticsearch-exporterDisk Usage Percent%meter_elasticsearch_node_disk_usage_percent块设备已用空间百分比elasticsearch-exporterDisk UsageGBmeter_elasticsearch_node_disk_usage块设备已用空间大小elasticsearch-exporterDisk ReadKBsmeter_elasticsearch_node_disk_io_read_bytes从磁盘读取的千字节总数elasticsearch-exporterDisk WriteKBsmeter_elasticsearch_node_disk_io_write_bytes写入磁盘的千字节总数elasticsearch-exporter在 elasticsearch-node.yaml 中这些指标按 JVM / CPU / translog / 熔断器 / 磁盘 / 网络 / 操作速率等分组定义。几个有代表性的表达式metricPrefix: meter_elasticsearch_node metricsRules: # JVM 堆/非堆内存通过 tagEqual 区分 area - name: jvm_memory_heap_used exp: elasticsearch_jvm_memory_used_bytes.tagNotEqual(cluster,unknown_cluster).tagEqual(area , heap).sum([cluster , name]) # GC 次数每分钟增量 - name: jvm_gc_count exp: elasticsearch_jvm_gc_collection_seconds_count.tagNotEqual(cluster,unknown_cluster).sum([cluster , name , gc]).increase(PT1M) # GC 时间sum 转换为毫秒后再求增量 - name: jvm_gc_time exp: (elasticsearch_jvm_gc_collection_seconds_sum * 1000).tagNotEqual(cluster,unknown_cluster).sum([cluster , name , gc]).increase(PT1M) # 磁盘使用率100 - available * 100 / total - name: disk_usage_percent exp: 100 - (elasticsearch_filesystem_data_available_bytes * 100).tagNotEqual(cluster,unknown_cluster).sum([cluster , name , mount]) / elasticsearch_filesystem_data_size_bytes.tagNotEqual(cluster,unknown_cluster).sum([cluster , name , mount]) # 磁盘 IO通过 irate 计算瞬时速率 - name: disk_io_read_bytes exp: elasticsearch_filesystem_io_stats_device_read_size_kilobytes_sum.tagNotEqual(cluster,unknown_cluster).sum([cluster , name , mount]).irate()这里体现了 MAL 的两种速率函数选择.increase(PT1M)计算窗口内的增量适用于计数器类如 GC 次数、熔断次数.irate()计算瞬时变化率适用于磁盘 IO、网络收发等.rate(PT1M)计算窗口内的平均速率适用于请求速率类。节点规则文件还通过sum([cluster , name , es_client_node , es_data_node , es_ingest_node , es_master_node])将 exporter 暴露的节点角色标签聚合成rules指标供 UI 展示节点角色信息。索引Index级指标索引级监控以meter_elasticsearch_index_为前缀覆盖文档数、存储大小、段、索引/搜索速率、各类操作耗时等维度。完整清单如下监控面板单位指标名说明数据来源Documents Primarymeter_elasticsearch_index_indices_docs_primary所有节点上仅主分片的文档数elasticsearch-exporterDeleted Documents Primarymeter_elasticsearch_index_indices_deleted_docs_primary仅主分片的已删除文档数elasticsearch-exporterData PrimaryGBmeter_elasticsearch_index_indices_store_size_bytes_primary所有节点上仅主分片存储的索引数据总大小elasticsearch-exporterDataGBmeter_elasticsearch_index_indices_store_size_bytes_total所有节点上全部分片存储的索引数据总大小elasticsearch-exporterSegments Primarymeter_elasticsearch_index_indices_segment_count_primary所有节点上仅主分片的段数量elasticsearch-exporterSegments Memory PrimaryMBmeter_elasticsearch_index_indices_segment_memory_bytes_primary所有节点上仅主分片的段内存大小elasticsearch-exporterSegmentsmeter_elasticsearch_index_indices_segment_count_total所有节点上全部分片的段数量elasticsearch-exporterSegments MemoryMBmeter_elasticsearch_index_indices_segment_memory_bytes_total所有节点上全部分片的段内存大小elasticsearch-exporterIndexing Ratemeter_elasticsearch_index_stats_indexing_index_total_req_rate / meter_elasticsearch_index_stats_indexing_index_total_proc_rate索引写入速率elasticsearch-exporterSearching Ratemeter_elasticsearch_index_stats_search_query_total_req_rate / meter_elasticsearch_index_stats_search_query_total_proc_rate索引搜索速率elasticsearch-exporterAll Operations ReqRatemeter_elasticsearch_index_stats_*_req_rate索引上所有操作请求速率elasticsearch-exporterAll Operations Runtimemeter_elasticsearch_index_stats_*_time_seconds_total索引上所有操作耗时elasticsearch-exporterAvg. Search Time Execute / Requestsmeter_elasticsearch_index_search_fetch_avg_time / meter_elasticsearch_index_search_query_avg_time / meter_elasticsearch_index_search_scroll_avg_time / meter_elasticsearch_index_search_suggest_avg_time索引上搜索操作平均耗时elasticsearch-exporterSearch Operations Ratereq/smeter_elasticsearch_index_stats_search_query_total_req_rate / meter_elasticsearch_index_stats_search_fetch_total_req_rate / meter_elasticsearch_index_stats_search_scroll_total_req_rate / meter_elasticsearch_index_stats_search_suggest_total_req_rate索引上搜索操作请求速率elasticsearch-exporterShards Documentsmeter_elasticsearch_index_indices_shards_docs索引上每个分片的文档数elasticsearch-exporterDocuments (Primary Shards)meter_elasticsearch_index_indices_docs_primary仅主分片的文档数elasticsearch-exporterDocuments Created Per Min (Primary Shards)meter_elasticsearch_index_indices_docs_primary_rate仅主分片的文档创建速率elasticsearch-exporterTotal Size Of Index (Primary Shards)MBmeter_elasticsearch_index_indices_store_size_bytes_primary所有节点上仅主分片存储的索引数据总大小elasticsearch-exporterDocuments (All Shards)meter_elasticsearch_index_indices_docs_total全部分片的文档数elasticsearch-exporterDocuments Created Per Min (All Shards)meter_elasticsearch_index_indices_docs_total_rate全部分片的文档创建速率elasticsearch-exporterTotal Size Of Index (All Shards)MBmeter_elasticsearch_index_indices_store_size_bytes_total所有节点上全部分片存储的索引数据总大小elasticsearch-exporter在 elasticsearch-index.yaml 中除常规的速率与耗时指标外还有两个值得关注的复杂表达式metricPrefix: meter_elasticsearch_index metricsRules: # 搜索平均耗时 搜索耗时速率 / 搜索次数速率 - name: search_query_avg_time exp: elasticsearch_index_stats_search_query_time_seconds_total.tagNotEqual(cluster,unknown_cluster).sum([cluster , index]).rate(PT1M) / elasticsearch_index_stats_search_query_total.tagNotEqual(cluster,unknown_cluster).sum([cluster , index]).rate(PT1M) # 分片文档数区分 primary / replica 标签 - name: indices_shards_docs exp: elasticsearch_indices_shards_docs.tagNotEqual(cluster,unknown_cluster).sum([cluster , index , primary , shard]).tag({tags - if (tags[primary] true) {tags.primary primary} else {tags.primary replica} })最后一条表达式展示了 MAL 的标签改写能力通过tag({...})将 exporter 中的布尔型primary标签转换为可读的primary/replica枚举值供 UI 按主/副本分片维度拆分展示。UI 监控面板与 e2e 验证面板定义位置集群、节点、索引三个维度的 UI 面板配置分别位于ui-initialized-templates/elasticsearch/elasticsearch-cluster.jsonui-initialized-templates/elasticsearch/elasticsearch-node.jsonui-initialized-templates/elasticsearch/elasticsearch-index.json以集群面板 elasticsearch-cluster.json 为例面板使用 Tab 组织 Overview 等分页通过 Table / Card 等 Widget 类型与查询表达式绑定指标{ id: Elasticsearch-Cluster, configuration: { children: [ { type: Tab, children: [ { name: Overview, children: [ { type: Widget, graph: { type: Table, tableHeaderCol1: status, tableHeaderCol2: }, widget: { title: Cluster Health }, expressions: [meter_elasticsearch_cluster_health_status] }, { type: Widget, graph: { type: Card }, widget: { title: Data Nodes }, expressions: [latest(meter_elasticsearch_cluster_nodes)] } ] } ] } ] } }这些模板在 OAP 启动时初始化用户可在 SkyWalking UI 的「仪表盘」中选择ELASTICSEARCH相关服务查看。e2e 测试如何验证整条链路仓库中的 e2e 测试完整模拟了本方案的数据链路可作为接入排障的参考。测试编排见 e2e.yaml其 compose 栈docker-compose.yml包含esElasticsearch 8.3.2、es-exporterv1.5.0、otel-collector、oap与es-mock用于写入模拟数据。其中 es-e2e-data-mock.sh 会创建 10 个索引e2e-index-0~e2e-index-9每个 3 主分片 2 副本并持续执行 search / suggest 等请求以产生真实的指标流量。随后 elasticsearch-cases.yaml 通过swctl对 OAP GraphQL 接口发起指标查询并校验结果例如swctl --display yaml --base-urlhttp://${oap_host}:${oap_12800}/graphql service ls swctl --display yaml --base-urlhttp://${oap_host}:${oap_12800}/graphql metrics exec \ --expressionmeter_elasticsearch_cluster_health_status --service-nameelasticsearch::cluster-e2e swctl --display yaml --base-urlhttp://${oap_host}:${oap_12800}/graphql metrics exec \ --expressionmeter_elasticsearch_cluster_breakers_tripped --service-nameelasticsearch::cluster-e2e测试同时验证了服务service.yml、实例instance.yml、端点endpoint.yml的建模结果以及各类带标签指标的取值如metrics-has-value-label-color.yml验证 cluster health 的 color 标签、metrics-has-value-label-node.yml验证节点标签、metrics-has-value-label-mount.yml验证磁盘挂载点标签、metrics-has-value-label-pool.yml验证 JVM pool 标签、metrics-has-value-label-gc.yml验证 GC 标签、metrics-has-value-label-primary.yml验证主/副本分片标签。如果你在接入后 UI 无数据可以参照这套测试栈进行端到端排查。自定义监控指标与面板上述监控面板只是开箱即用的默认集SkyWalking 允许完全自定义指标、表达式与面板指标定义与表达式规则位于 OAP 的/config/otel-rules/elasticsearch/目录仓库中的实际路径为 oap-server/server-starter/src/main/resources/otel-rules/elasticsearch/对应elasticsearch-cluster.yaml、elasticsearch-node.yaml、elasticsearch-index.yaml三个文件面板配置位于/config/ui-initialized-templates/elasticsearch仓库中的实际路径为 oap-server/server-starter/src/main/resources/ui-initialized-templates/elasticsearch/。自定义时建议遵循以下要点保留filter中的job_name elasticsearch-monitoring过滤条件或按需修改并同步调整 Collector 的 scrape 配置新增指标时在metricsRules下追加name与exp条目可组合使用sum、avg、rate、irate、increase、tagNotEqual、tagEqual、tag等 MAL 算子修改规则后需重启 OAP 使其重新加载若 YAML 格式不合法OAP 可能启动失败需谨慎校验新增面板时可在ui-initialized-templates/elasticsearch/下新增 JSON 模板或在既有模板中追加 Widget注意expressions中引用的指标名需与 MAL 规则中的metricPrefix name一致。关于 MAL 表达式的完整语法可参考 MAL指标分析语言文档 与 Meter 系统文档。常见问题与排查建议面板无数据优先检查三点——① OpenTelemetry Collector 是否成功抓取 exporter 指标curl http://exporter-host:9114/metrics② Collector 的job_name是否与 MAL 规则的filter一致③ OAP 是否启用了receiver-otel且enabledOtelMetricsRules包含elasticsearch/*。缺少索引级指标确认 elasticsearch-exporter 启动参数是否包含--es.indices以及--es.indices_settings、--es.indices_mappings、--es.shards否则 exporter 不会暴露索引级原始指标。服务名带elasticsearch::前缀这是expSuffix中tag({tags - tags.cluster elasticsearch:: tags.cluster})的预期行为查询时需使用完整服务名如elasticsearch::cluster-e2e。OTLP 版本要求使用otlpexporter 推送指标需要 OAP 版本不低于 9.2.0详见 otel-collector-config.yaml 中的注释说明。小结SkyWalking 的 Elasticsearch 监控能力基于「elasticsearch-exporter OpenTelemetry Collector MAL」的标准技术栈构建exporter 负责原始指标采集OpenTelemetry 协议负责标准化传输MAL 规则负责把 Prometheus 风格指标转化为 SkyWalking 的 Service / Instance / Endpoint 三级模型。通过本文提供的配置样例、指标清单与源码级表达式分析开发者可以快速将 Elasticsearch 集群接入 SkyWalking并在此基础上按需扩展自定义指标与面板。【免费下载链接】skywalkingAPM, Application Performance Monitoring System项目地址: https://gitcode.com/gh_mirrors/sky/skywalking创作声明：本文部分内容由AI辅助生成（AIGC），仅供参考
SkyWalking 集成 Elasticsearch 监控：基于 elasticsearch-exporter 与 OpenTelemetry Collector 的完整实践指南
SkyWalking 集成 Elasticsearch 监控基于 elasticsearch-exporter 与 OpenTelemetry Collector 的完整实践指南【免费下载链接】skywalkingAPM, Application Performance Monitoring System项目地址: https://gitcode.com/gh_mirrors/sky/skywalking本文基于 Apache SkyWalking 官方文档 backend-elasticsearch-monitoring.md 编写并结合当前仓库中的 MAL 规则、OpenTelemetry Collector 配置与 e2e 测试代码进行深度补充。核心主题是如何在 SkyWalking OAP 中接入 Elasticsearch 集群的指标监控。通过本指南读者可以掌握从 elasticsearch-exporter 采集指标、经 OpenTelemetry Collector 转发、最终在 SkyWalking 中以Layer: ELASTICSEARCH服务形态呈现的完整链路并理解集群 / 节点 / 索引三个维度的监控面板与底层指标表达式。监控方案总览SkyWalking 本身并不直接向 Elasticsearch 发起指标拉取而是采用「Prometheus 生态采集 OpenTelemetry 协议转发 MAL 指标计算」的分层架构elasticsearch-exporterprometheus-community/elasticsearch_exporter负责从 Elasticsearch 集群采集原始指标数据并以 Prometheus 格式暴露在:9114/metrics。OpenTelemetry Collector通过Prometheus Receiver抓取 elasticsearch-exporter 暴露的指标再由OpenTelemetry gRPC exporterotlp推送到 SkyWalking OAP Server 的 OpenTelemetry receiver。SkyWalking OAP Server使用MALMetric Analysis Language指标分析语言表达式对指标进行过滤、计算、聚合最终存储到后端并暴露为Layer: ELASTICSEARCH服务。其数据流可用下图概括Elasticsearch Cluster │ (1) 内部状态 / 统计 API ▼ elasticsearch-exporter (:9114/metrics) │ (2) Prometheus 协议抓取 ▼ OpenTelemetry Collector (Prometheus Receiver → batch → otlp exporter) │ (3) OTLP gRPC 推送 ▼ SkyWalking OAP Server (OpenTelemetry receiver) │ (4) MAL 表达式过滤 / 计算 / 聚合 ▼ Meter System → 存储 → UI 监控面板 (Layer: ELASTICSEARCH)这个方案的价值在于SkyWalking 不重复造轮子直接复用 Prometheus 社区成熟的 exporter 与 OpenTelemetry 标准协议同时通过 Meter System 将第三方指标与 SkyWalking 自身的服务/实例/端点模型统一建模。快速开始三步骤接入步骤 1部署并配置 elasticsearch-exporter官方推荐使用 elasticsearch_exporter 中的部署方式es-exporter: image: quay.io/prometheuscommunity/elasticsearch-exporter:v1.5.0 expose: - 9114 depends_on: es: condition: service_healthy command: - --log.formatlogfmt - --log.levelinfo - --es.urihttp://es:9200 # 指向 Elasticsearch 的 HTTP 地址 - --es.all # 采集集群中所有节点 - --es.indices # 采集索引级指标 - --es.indices_settings # 采集索引设置 - --es.indices_mappings # 采集索引映射 - --es.shards # 采集分片级指标 - --es.snapshots # 采集快照指标 - --es.timeout30s - --web.telemetry-path/metrics说明--es.indices等开关决定了 exporter 是否暴露索引级指标而 SkyWalking 的 Index 监控面板正是依赖这些指标因此若希望获得索引维度的监控数据务必开启--es.indices与--es.indices_settings、--es.shards等配套参数。步骤 2部署 OpenTelemetry Collector 并配置管道OpenTelemetry Collector 的配置示例位于 otel-collector-config.yaml核心管道如下receivers: prometheus: config: scrape_configs: - job_name: elasticsearch-monitoring # 该 job_name 与 MAL 规则中的 filter 严格对应 scrape_interval: 10s static_configs: - targets: [es-exporter:9114] processors: batch: exporters: # 若使用 otlp exporter请确保 OAP 版本 9.2.0 otlp: endpoint: oap:11800 tls: insecure: true service: pipelines: metrics: receivers: - prometheus processors: - batch exporters: - otlp这里有两个必须严格一致的关键点job_name采集任务的job_name必须设置为elasticsearch-monitoring。这是因为三个 MAL 规则文件的头部都带有过滤器filter: { tags - tags.job_name elasticsearch-monitoring } # The OpenTelemetry job name如果 job_name 不一致OAP 会直接过滤掉这批指标导致监控面板无数据。OTLP endpoint指向 OAP 的 gRPC 端口默认11800。receiver-otel仅支持otlp这一种 handler对应opentelemetry-receiver.md中的说明。步骤 3配置 SkyWalking OpenTelemetry receiverSkyWalking 侧需要激活receiver-otel模块。在 application.yml 中可以看到默认配置receiver-otel: selector: ${SW_OTEL_RECEIVER:default} default: enabledHandlers: ${SW_OTEL_RECEIVER_ENABLED_HANDLERS:otlp-metrics,otlp-logs} enabledOtelMetricsRules: ${SW_OTEL_RECEIVER_ENABLED_OTEL_METRICS_RULES:...,elasticsearch/*,...}通过环境变量SW_OTEL_RECEIVERdefault激活 OpenTelemetry receiverenabledOtelMetricsRules中已默认包含elasticsearch/*通配即加载otel-rules/elasticsearch/目录下全部三个规则文件若你的部署环境没有启用默认规则可以显式设置为SW_OTEL_RECEIVER_ENABLED_OTEL_METRICS_RULESelasticsearch/*。关于 OpenTelemetry receiver 的详细说明可参考 opentelemetry-receiver.md。监控模型Cluster / Node / Index 三维建模接入完成后Elasticsearch 监控会以Layer: ELASTICSEARCH的服务形态出现在 OAP 中其建模方式为每个Elasticsearch 集群对应一个Service集群内的每个节点node对应一个Instance集群内的每个索引index对应一个Endpoint。这一映射关系直接体现在三个 MAL 规则文件的expSuffix中规则文件expSuffix映射维度elasticsearch-cluster.yaml.service([cluster], Layer.ELASTICSEARCH)集群 → Serviceelasticsearch-node.yaml.instance([cluster], [name], Layer.ELASTICSEARCH)节点 → Instanceelasticsearch-index.yaml.endpoint([cluster], [index], Layer.ELASTICSEARCH)索引 → Endpoint同时三个文件都以如下表达式对集群名打标签并加上命名空间前缀expSuffix: tag({tags - tags.cluster elasticsearch:: tags.cluster}).service([cluster], Layer.ELASTICSEARCH)这意味着服务名会呈现为elasticsearch::cluster_name的形式例如 e2e 测试中的elasticsearch::cluster-e2e见 elasticsearch-cases.yaml 中的查询语句。集群Cluster级指标集群级监控面板覆盖了健康状态、分片分布、节点规模、资源使用等维度指标均以meter_elasticsearch_cluster_为前缀。完整清单如下监控面板指标名说明数据来源Cluster Healthmeter_elasticsearch_cluster_health_status所有主分片与副本分片是否分配成功颜色状态elasticsearch-exporterTripped Of Breakersmeter_elasticsearch_cluster_breakers_tripped熔断器被触发次数elasticsearch-exporterNodesmeter_elasticsearch_cluster_nodes集群中的节点数elasticsearch-exporterData Nodesmeter_elasticsearch_cluster_data_nodes集群中的数据节点数elasticsearch-exporterPending Tasksmeter_elasticsearch_cluster_pending_tasks_total尚未执行的集群级变更任务数elasticsearch-exporterCPU Usage Avg. (%)meter_elasticsearch_cluster_cpu_usage_avg集群级进程 CPU 使用百分比elasticsearch-exporterJVM Memory Used Avg. (%)meter_elasticsearch_cluster_jvm_memory_used_avg集群级 JVM 内存使用百分比elasticsearch-exporterOpen Filesmeter_elasticsearch_cluster_open_file_count打开的文件描述符数elasticsearch-exporterActive Primary Shardsmeter_elasticsearch_cluster_primary_shards_total集群中主分片总数跨所有索引的聚合elasticsearch-exporterActive Shardsmeter_elasticsearch_cluster_shards_total所有索引的全部分片聚合总数含副本分片elasticsearch-exporterInitializing Shardsmeter_elasticsearch_cluster_initializing_shards_total正在新建的分片数elasticsearch-exporterDelayed Unassigned Shardsmeter_elasticsearch_cluster_delayed_unassigned_shards_total为降低重分配开销而延迟的分片数elasticsearch-exporterRelocating Shardsmeter_elasticsearch_cluster_relocating_shards_total正在节点间迁移的分片数elasticsearch-exporterUnassigned Shardsmeter_elasticsearch_cluster_unassigned_shards_total存在于集群状态但无法在集群中找到的分片数elasticsearch-exporter以 elasticsearch-cluster.yaml 中的若干规则为例可以看到底层 MAL 表达式的计算逻辑metricPrefix: meter_elasticsearch_cluster metricsRules: # cluster health按 cluster 与 color 标签求和health 为 1 即健康 - name: health_status exp: elasticsearch_cluster_health_status.tagNotEqual(cluster,unknown_cluster).valueEqual(1).sum([cluster , color]) # 熔断次数每分钟增量 - name: breakers_tripped exp: elasticsearch_breakers_tripped.tagNotEqual(cluster,unknown_cluster).sum([cluster]).increase(PT1M) # 节点数 / 数据节点数 - name: nodes exp: elasticsearch_cluster_health_number_of_nodes.tagNotEqual(cluster,unknown_cluster).sum([cluster]) - name: data_nodes exp: elasticsearch_cluster_health_number_of_data_nodes.tagNotEqual(cluster,unknown_cluster).sum([cluster]) # JVM 内存使用率 used / max * 100 - name: jvm_memory_used_avg exp: elasticsearch_jvm_memory_used_bytes.tagNotEqual(cluster,unknown_cluster).sum([cluster]) / elasticsearch_jvm_memory_max_bytes.tagNotEqual(cluster,unknown_cluster).sum([cluster]) * 100值得注意的实现细节几乎每个表达式都带.tagNotEqual(cluster,unknown_cluster)用于排除unknown_cluster标签的数据避免脏数据污染聚合结果速率型指标如熔断次数使用.increase(PT1M)计算每分钟增量时间窗口采用 ISO-8601 格式PT1M即 1 分钟。节点Node级指标节点级监控以meter_elasticsearch_node_为前缀覆盖 JVM、CPU、文档、段segment、磁盘、网络、GC、熔断器、索引操作速率等维度。完整清单如下监控面板单位指标名说明数据来源Node Rulesmeter_elasticsearch_node_rules节点角色elasticsearch-exporterJVM Memory UsedMBmeter_elasticsearch_node_jvm_memory_used节点级 JVM 内存使用量elasticsearch-exporterCPU Percent%meter_elasticsearch_node_process_cpu_percent节点级进程 CPU 使用百分比elasticsearch-exporterDocumentsmeter_elasticsearch_node_indices_docs该节点上的索引文档数elasticsearch-exporterSegmentsmeter_elasticsearch_node_segment_count该节点上的索引段数量elasticsearch-exporterDisk Free SpaceGBmeter_elasticsearch_node_all_disk_free_space所有块设备上的可用空间elasticsearch-exporterOpen Filesmeter_elasticsearch_node_open_file_count打开的文件描述符数elasticsearch-exporterProcess CPU Usage Percent%meter_elasticsearch_node_process_cpu_percent进程 CPU 使用百分比elasticsearch-exporterOS CPU usage percent%meter_elasticsearch_node_os_cpu_percent操作系统 CPU 使用百分比elasticsearch-exporterLoad Averagemeter_elasticsearch_node_os_load1 / meter_elasticsearch_node_os_load5 / meter_elasticsearch_node_os_load15短/中/长期负载均值elasticsearch-exporterJVM Memory UsageMBmeter_elasticsearch_node_jvm_memory_nonheap_used / meter_elasticsearch_node_jvm_memory_heap_used / meter_elasticsearch_node_jvm_memory_heap_max各区域 JVM 内存使用情况elasticsearch-exporterJVM Pool Peak UsedMBmeter_elasticsearch_node_jvm_memory_pool_peak_usedJVM 各池当前使用量elasticsearch-exporterGC Countmeter_elasticsearch_node_jvm_gc_countJVM GC 运行次数elasticsearch-exporterGC Timems/minmeter_elasticsearch_node_jvm_gc_timeGC 运行时间elasticsearch-exporterAll Operations ReqRatemeter_elasticsearch_node_indices_*_req_rate节点上的所有操作请求速率elasticsearch-exporterIndexing Ratereqpsmeter_elasticsearch_node_indices_indexing_index_total_req_rate / meter_elasticsearch_node_indices_indexing_index_total_proc_rate节点上的索引写入速率elasticsearch-exporterSearching Ratereqpsmeter_elasticsearch_node_indices_search_fetch_total_req_rate / meter_elasticsearch_node_indices_search_query_time_seconds_proc_rate节点上的搜索速率elasticsearch-exporterTotal Translog Operationsmeter_elasticsearch_node_indices_translog_operationstranslog 总操作数elasticsearch-exporterTotal Translog SizeMBmeter_elasticsearch_node_indices_translog_sizetranslog 总大小elasticsearch-exporterTripped For Breakersmeter_elasticsearch_node_breakers_tripped熔断器触发次数elasticsearch-exporterEstimated Size Of BreakerMBmeter_elasticsearch_node_breakers_estimated_size熔断器估算大小elasticsearch-exporterDocuments CountKB/smeter_elasticsearch_node_indices_docs该节点文档数量elasticsearch-exporterMerged Documents Countcount/smeter_elasticsearch_node_indices_merges_docs_total累计合并文档数elasticsearch-exporterDeleted Documents Countmeter_elasticsearch_node_indices_docs_deleted_total该节点被删除文档数elasticsearch-exporterDocuments Index Ratecalls/smeter_elasticsearch_node_indices_indexing_index_total_req_rate每秒索引写入调用数elasticsearch-exporterMerged Documents RateMB/smeter_elasticsearch_node_indices_merges_total_size_bytes_total每秒合并数据量elasticsearch-exporterDocuments Deleted Ratedocs/smeter_elasticsearch_node_indices_docs_deleted该节点每秒删除文档数elasticsearch-exporterCount Of Index Segmentsmeter_elasticsearch_node_segment_count该节点索引段数量elasticsearch-exporterCurrent Memory Size Of SegmentsMBmeter_elasticsearch_node_segment_memory段当前内存大小elasticsearch-exporterNetworkbytes/secmeter_elasticsearch_node_network_send_bytes / meter_elasticsearch_node_network_receive_bytes发送与接收字节总数elasticsearch-exporterDisk Usage Percent%meter_elasticsearch_node_disk_usage_percent块设备已用空间百分比elasticsearch-exporterDisk UsageGBmeter_elasticsearch_node_disk_usage块设备已用空间大小elasticsearch-exporterDisk ReadKBsmeter_elasticsearch_node_disk_io_read_bytes从磁盘读取的千字节总数elasticsearch-exporterDisk WriteKBsmeter_elasticsearch_node_disk_io_write_bytes写入磁盘的千字节总数elasticsearch-exporter在 elasticsearch-node.yaml 中这些指标按 JVM / CPU / translog / 熔断器 / 磁盘 / 网络 / 操作速率等分组定义。几个有代表性的表达式metricPrefix: meter_elasticsearch_node metricsRules: # JVM 堆/非堆内存通过 tagEqual 区分 area - name: jvm_memory_heap_used exp: elasticsearch_jvm_memory_used_bytes.tagNotEqual(cluster,unknown_cluster).tagEqual(area , heap).sum([cluster , name]) # GC 次数每分钟增量 - name: jvm_gc_count exp: elasticsearch_jvm_gc_collection_seconds_count.tagNotEqual(cluster,unknown_cluster).sum([cluster , name , gc]).increase(PT1M) # GC 时间sum 转换为毫秒后再求增量 - name: jvm_gc_time exp: (elasticsearch_jvm_gc_collection_seconds_sum * 1000).tagNotEqual(cluster,unknown_cluster).sum([cluster , name , gc]).increase(PT1M) # 磁盘使用率100 - available * 100 / total - name: disk_usage_percent exp: 100 - (elasticsearch_filesystem_data_available_bytes * 100).tagNotEqual(cluster,unknown_cluster).sum([cluster , name , mount]) / elasticsearch_filesystem_data_size_bytes.tagNotEqual(cluster,unknown_cluster).sum([cluster , name , mount]) # 磁盘 IO通过 irate 计算瞬时速率 - name: disk_io_read_bytes exp: elasticsearch_filesystem_io_stats_device_read_size_kilobytes_sum.tagNotEqual(cluster,unknown_cluster).sum([cluster , name , mount]).irate()这里体现了 MAL 的两种速率函数选择.increase(PT1M)计算窗口内的增量适用于计数器类如 GC 次数、熔断次数.irate()计算瞬时变化率适用于磁盘 IO、网络收发等.rate(PT1M)计算窗口内的平均速率适用于请求速率类。节点规则文件还通过sum([cluster , name , es_client_node , es_data_node , es_ingest_node , es_master_node])将 exporter 暴露的节点角色标签聚合成rules指标供 UI 展示节点角色信息。索引Index级指标索引级监控以meter_elasticsearch_index_为前缀覆盖文档数、存储大小、段、索引/搜索速率、各类操作耗时等维度。完整清单如下监控面板单位指标名说明数据来源Documents Primarymeter_elasticsearch_index_indices_docs_primary所有节点上仅主分片的文档数elasticsearch-exporterDeleted Documents Primarymeter_elasticsearch_index_indices_deleted_docs_primary仅主分片的已删除文档数elasticsearch-exporterData PrimaryGBmeter_elasticsearch_index_indices_store_size_bytes_primary所有节点上仅主分片存储的索引数据总大小elasticsearch-exporterDataGBmeter_elasticsearch_index_indices_store_size_bytes_total所有节点上全部分片存储的索引数据总大小elasticsearch-exporterSegments Primarymeter_elasticsearch_index_indices_segment_count_primary所有节点上仅主分片的段数量elasticsearch-exporterSegments Memory PrimaryMBmeter_elasticsearch_index_indices_segment_memory_bytes_primary所有节点上仅主分片的段内存大小elasticsearch-exporterSegmentsmeter_elasticsearch_index_indices_segment_count_total所有节点上全部分片的段数量elasticsearch-exporterSegments MemoryMBmeter_elasticsearch_index_indices_segment_memory_bytes_total所有节点上全部分片的段内存大小elasticsearch-exporterIndexing Ratemeter_elasticsearch_index_stats_indexing_index_total_req_rate / meter_elasticsearch_index_stats_indexing_index_total_proc_rate索引写入速率elasticsearch-exporterSearching Ratemeter_elasticsearch_index_stats_search_query_total_req_rate / meter_elasticsearch_index_stats_search_query_total_proc_rate索引搜索速率elasticsearch-exporterAll Operations ReqRatemeter_elasticsearch_index_stats_*_req_rate索引上所有操作请求速率elasticsearch-exporterAll Operations Runtimemeter_elasticsearch_index_stats_*_time_seconds_total索引上所有操作耗时elasticsearch-exporterAvg. Search Time Execute / Requestsmeter_elasticsearch_index_search_fetch_avg_time / meter_elasticsearch_index_search_query_avg_time / meter_elasticsearch_index_search_scroll_avg_time / meter_elasticsearch_index_search_suggest_avg_time索引上搜索操作平均耗时elasticsearch-exporterSearch Operations Ratereq/smeter_elasticsearch_index_stats_search_query_total_req_rate / meter_elasticsearch_index_stats_search_fetch_total_req_rate / meter_elasticsearch_index_stats_search_scroll_total_req_rate / meter_elasticsearch_index_stats_search_suggest_total_req_rate索引上搜索操作请求速率elasticsearch-exporterShards Documentsmeter_elasticsearch_index_indices_shards_docs索引上每个分片的文档数elasticsearch-exporterDocuments (Primary Shards)meter_elasticsearch_index_indices_docs_primary仅主分片的文档数elasticsearch-exporterDocuments Created Per Min (Primary Shards)meter_elasticsearch_index_indices_docs_primary_rate仅主分片的文档创建速率elasticsearch-exporterTotal Size Of Index (Primary Shards)MBmeter_elasticsearch_index_indices_store_size_bytes_primary所有节点上仅主分片存储的索引数据总大小elasticsearch-exporterDocuments (All Shards)meter_elasticsearch_index_indices_docs_total全部分片的文档数elasticsearch-exporterDocuments Created Per Min (All Shards)meter_elasticsearch_index_indices_docs_total_rate全部分片的文档创建速率elasticsearch-exporterTotal Size Of Index (All Shards)MBmeter_elasticsearch_index_indices_store_size_bytes_total所有节点上全部分片存储的索引数据总大小elasticsearch-exporter在 elasticsearch-index.yaml 中除常规的速率与耗时指标外还有两个值得关注的复杂表达式metricPrefix: meter_elasticsearch_index metricsRules: # 搜索平均耗时 搜索耗时速率 / 搜索次数速率 - name: search_query_avg_time exp: elasticsearch_index_stats_search_query_time_seconds_total.tagNotEqual(cluster,unknown_cluster).sum([cluster , index]).rate(PT1M) / elasticsearch_index_stats_search_query_total.tagNotEqual(cluster,unknown_cluster).sum([cluster , index]).rate(PT1M) # 分片文档数区分 primary / replica 标签 - name: indices_shards_docs exp: elasticsearch_indices_shards_docs.tagNotEqual(cluster,unknown_cluster).sum([cluster , index , primary , shard]).tag({tags - if (tags[primary] true) {tags.primary primary} else {tags.primary replica} })最后一条表达式展示了 MAL 的标签改写能力通过tag({...})将 exporter 中的布尔型primary标签转换为可读的primary/replica枚举值供 UI 按主/副本分片维度拆分展示。UI 监控面板与 e2e 验证面板定义位置集群、节点、索引三个维度的 UI 面板配置分别位于ui-initialized-templates/elasticsearch/elasticsearch-cluster.jsonui-initialized-templates/elasticsearch/elasticsearch-node.jsonui-initialized-templates/elasticsearch/elasticsearch-index.json以集群面板 elasticsearch-cluster.json 为例面板使用 Tab 组织 Overview 等分页通过 Table / Card 等 Widget 类型与查询表达式绑定指标{ id: Elasticsearch-Cluster, configuration: { children: [ { type: Tab, children: [ { name: Overview, children: [ { type: Widget, graph: { type: Table, tableHeaderCol1: status, tableHeaderCol2: }, widget: { title: Cluster Health }, expressions: [meter_elasticsearch_cluster_health_status] }, { type: Widget, graph: { type: Card }, widget: { title: Data Nodes }, expressions: [latest(meter_elasticsearch_cluster_nodes)] } ] } ] } ] } }这些模板在 OAP 启动时初始化用户可在 SkyWalking UI 的「仪表盘」中选择ELASTICSEARCH相关服务查看。e2e 测试如何验证整条链路仓库中的 e2e 测试完整模拟了本方案的数据链路可作为接入排障的参考。测试编排见 e2e.yaml其 compose 栈docker-compose.yml包含esElasticsearch 8.3.2、es-exporterv1.5.0、otel-collector、oap与es-mock用于写入模拟数据。其中 es-e2e-data-mock.sh 会创建 10 个索引e2e-index-0~e2e-index-9每个 3 主分片 2 副本并持续执行 search / suggest 等请求以产生真实的指标流量。随后 elasticsearch-cases.yaml 通过swctl对 OAP GraphQL 接口发起指标查询并校验结果例如swctl --display yaml --base-urlhttp://${oap_host}:${oap_12800}/graphql service ls swctl --display yaml --base-urlhttp://${oap_host}:${oap_12800}/graphql metrics exec \ --expressionmeter_elasticsearch_cluster_health_status --service-nameelasticsearch::cluster-e2e swctl --display yaml --base-urlhttp://${oap_host}:${oap_12800}/graphql metrics exec \ --expressionmeter_elasticsearch_cluster_breakers_tripped --service-nameelasticsearch::cluster-e2e测试同时验证了服务service.yml、实例instance.yml、端点endpoint.yml的建模结果以及各类带标签指标的取值如metrics-has-value-label-color.yml验证 cluster health 的 color 标签、metrics-has-value-label-node.yml验证节点标签、metrics-has-value-label-mount.yml验证磁盘挂载点标签、metrics-has-value-label-pool.yml验证 JVM pool 标签、metrics-has-value-label-gc.yml验证 GC 标签、metrics-has-value-label-primary.yml验证主/副本分片标签。如果你在接入后 UI 无数据可以参照这套测试栈进行端到端排查。自定义监控指标与面板上述监控面板只是开箱即用的默认集SkyWalking 允许完全自定义指标、表达式与面板指标定义与表达式规则位于 OAP 的/config/otel-rules/elasticsearch/目录仓库中的实际路径为 oap-server/server-starter/src/main/resources/otel-rules/elasticsearch/对应elasticsearch-cluster.yaml、elasticsearch-node.yaml、elasticsearch-index.yaml三个文件面板配置位于/config/ui-initialized-templates/elasticsearch仓库中的实际路径为 oap-server/server-starter/src/main/resources/ui-initialized-templates/elasticsearch/。自定义时建议遵循以下要点保留filter中的job_name elasticsearch-monitoring过滤条件或按需修改并同步调整 Collector 的 scrape 配置新增指标时在metricsRules下追加name与exp条目可组合使用sum、avg、rate、irate、increase、tagNotEqual、tagEqual、tag等 MAL 算子修改规则后需重启 OAP 使其重新加载若 YAML 格式不合法OAP 可能启动失败需谨慎校验新增面板时可在ui-initialized-templates/elasticsearch/下新增 JSON 模板或在既有模板中追加 Widget注意expressions中引用的指标名需与 MAL 规则中的metricPrefix name一致。关于 MAL 表达式的完整语法可参考 MAL指标分析语言文档 与 Meter 系统文档。常见问题与排查建议面板无数据优先检查三点——① OpenTelemetry Collector 是否成功抓取 exporter 指标curl http://exporter-host:9114/metrics② Collector 的job_name是否与 MAL 规则的filter一致③ OAP 是否启用了receiver-otel且enabledOtelMetricsRules包含elasticsearch/*。缺少索引级指标确认 elasticsearch-exporter 启动参数是否包含--es.indices以及--es.indices_settings、--es.indices_mappings、--es.shards否则 exporter 不会暴露索引级原始指标。服务名带elasticsearch::前缀这是expSuffix中tag({tags - tags.cluster elasticsearch:: tags.cluster})的预期行为查询时需使用完整服务名如elasticsearch::cluster-e2e。OTLP 版本要求使用otlpexporter 推送指标需要 OAP 版本不低于 9.2.0详见 otel-collector-config.yaml 中的注释说明。小结SkyWalking 的 Elasticsearch 监控能力基于「elasticsearch-exporter OpenTelemetry Collector MAL」的标准技术栈构建exporter 负责原始指标采集OpenTelemetry 协议负责标准化传输MAL 规则负责把 Prometheus 风格指标转化为 SkyWalking 的 Service / Instance / Endpoint 三级模型。通过本文提供的配置样例、指标清单与源码级表达式分析开发者可以快速将 Elasticsearch 集群接入 SkyWalking并在此基础上按需扩展自定义指标与面板。【免费下载链接】skywalkingAPM, Application Performance Monitoring System项目地址: https://gitcode.com/gh_mirrors/sky/skywalking创作声明：本文部分内容由AI辅助生成（AIGC），仅供参考
