Kubernetes日志管理与分析构建全面的日志体系一、日志管理概述Kubernetes日志管理涉及收集、存储、查询和分析容器日志是运维监控的重要组成部分。1.1 日志架构flowchart TD subgraph 日志收集层 A[Pod 1] B[Pod 2] C[Pod 3] D[Pod N] end subgraph 日志传输层 E[Fluentd/Fluent Bit] F[Filter Parse] end subgraph 日志存储层 G[Loki / Elasticsearch] H[Index Store] end subgraph 日志展示层 I[Grafana / Kibana] J[Visualization] end A -- E B -- E C -- E D -- E E -- F F -- G G -- H H -- I I -- J1.2 日志组件组件功能Fluentd日志收集与转发Fluent Bit轻量级日志收集Loki日志存储与查询Elasticsearch全文搜索与分析Grafana日志可视化二、Fluentd配置2.1 DaemonSet部署apiVersion: apps/v1 kind: DaemonSet metadata: name: fluentd namespace: kube-system spec: selector: matchLabels: name: fluentd template: spec: containers: - name: fluentd image: fluent/fluentd-kubernetes-daemonset:v1.15-debian-loki env: - name: FLUENT_LOKI_URL value: http://loki:3100 volumeMounts: - name: varlog mountPath: /var/log - name: varlibdockercontainers mountPath: /var/lib/docker/containers readOnly: true - name: config mountPath: /fluentd/etc volumes: - name: varlog hostPath: path: /var/log - name: varlibdockercontainers hostPath: path: /var/lib/docker/containers - name: config configMap: name: fluentd-config2.2 Fluentd配置文件apiVersion: v1 kind: ConfigMap metadata: name: fluentd-config data: fluent.conf: | source type tail path /var/log/containers/*.log pos_file /var/log/fluentd-containers.log.pos tag kubernetes.* read_from_head true parse type json time_key time time_format %Y-%m-%dT%H:%M:%S.%NZ /parse /source filter kubernetes.** type kubernetes_metadata /filter match ** type loki url #{ENV[FLUENT_LOKI_URL]} flush_interval 10s /match三、Loki配置3.1 Loki部署apiVersion: loki.grafana.com/v1 kind: LokiStack metadata: name: loki namespace: monitoring spec: size: 1x.small storage: schemas: - version: v13 effectiveDate: 2024-01-01 secret: name: loki-storage limits: global: maxStreamBytes: 1GB maxTotalBytes: 10GB3.2 Loki Service配置apiVersion: v1 kind: Service metadata: name: loki namespace: monitoring spec: selector: app: loki ports: - port: 3100 name: http四、Grafana日志查询4.1 Loki数据源配置apiVersion: grafana.integreatly.org/v1beta1 kind: Grafana metadata: name: grafana namespace: monitoring spec: config: log: mode: console datasources: - name: Loki type: loki access: proxy url: http://loki:31004.2 日志查询语法{appmy-app} | error {namespacedefault} |~ WARN|ERROR sum(count_over_time({appmy-app}[5m])) topk(5, count by(app) (count_over_time({namespacedefault}[1h])))五、Elasticsearch配置5.1 Elasticsearch部署apiVersion: elasticsearch.k8s.elastic.co/v1 kind: Elasticsearch metadata: name: quickstart spec: version: 8.6.0 nodeSets: - name: default count: 3 config: node.store.allow_mmap: false5.2 Kibana部署apiVersion: kibana.k8s.elastic.co/v1 kind: Kibana metadata: name: quickstart spec: version: 8.6.0 count: 1 elasticsearchRef: name: quickstart六、日志最佳实践6.1 结构化日志import json import logging class JsonFormatter(logging.Formatter): def format(self, record): log_record { timestamp: self.formatTime(record), level: record.levelname, logger: record.name, message: record.getMessage(), module: record.module, line: record.lineno } return json.dumps(log_record) logger logging.getLogger(my-app) logger.setLevel(logging.INFO) handler logging.StreamHandler() handler.setFormatter(JsonFormatter()) logger.addHandler(handler) logger.info(User logged in, extra{user_id: 123})6.2 日志轮转配置apiVersion: v1 kind: ConfigMap metadata: name: logrotate-config data: logrotate.conf: | /var/log/containers/*.log { daily rotate 7 compress delaycompress missingok notifempty copytruncate }6.3 日志级别管理apiVersion: v1 kind: ConfigMap metadata: name: app-log-config data: LOG_LEVEL: INFO LOG_FORMAT: json七、总结日志管理实践包括日志收集使用Fluentd/Fluent Bit收集容器日志日志存储使用Loki或Elasticsearch存储日志日志查询使用Grafana或Kibana查询分析结构化日志输出JSON格式日志便于分析日志轮转定期清理旧日志建议建立完善的日志体系实现日志的集中管理和分析。参考资料Loki文档Fluentd文档Elasticsearch文档
