Kubernetes部署Envoy详解：Sidecar与API网关配置指南

开源项目 3 小时前 0 1

概述

Envoy是云原生架构中的核心组件，在Kubernetes中部署Envoy可以实现统一的流量管理、安全认证和可观测性。本文将详细介绍如何在Kubernetes集群中部署和配置Envoy，包括Deployment、Service、ConfigMap等资源的使用，以及常见的部署场景。

前置条件

Kubernetes集群（1.19+版本）
kubectl工具配置正确
Helm 3.x（可选，用于Helm安装）
对Kubernetes基本概念的了解（Pod、Service、ConfigMap等）

部署方式选择

在Kubernetes中部署Envoy有两种主要方式：

原生部署：直接使用Deployment和ConfigMap部署
Helm部署：使用Bitnami或官方Helm Chart部署

本文将详细介绍这两种方式。

方式一：原生部署Envoy

创建ConfigMap存储配置文件

apiVersion: v1
kind: ConfigMap
metadata:
  name: envoy-config
  namespace: envoy
data:
  envoy.yaml: |
    admin:
      address:
        socket_address:
          protocol: TCP
          address: 0.0.0.0
          port_value: 9901

    static_resources:
      listeners:
      - name: http listener
        address:
          socket_address:
            protocol: TCP
            address: 0.0.0.0
            port_value: 80
        filter_chains:
        - filters:
          - name: envoy.filters.network.http_connection_manager
            typed_config:
              "@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
              codec_type: AUTO
              stat_prefix: http connection_manager
              http_filters:
              - name: envoy.filters.http.router
              route_config:
                name: local_route
                virtual_hosts:
                - name: backend
                  domains: ["*"]
                  routes:
                  - match:
                      prefix: "/"
                    route:
                      cluster: backend-service

      clusters:
      - name: backend-service
        connect_timeout: 5s
        type: STRICT_DNS
        lb_policy: ROUND_ROBIN
        load_assignment:
          cluster_name: backend-service
          endpoints:
          - lb_endpoints:
            - endpoint:
                address:
                  socket_address:
                    address: backend-service.envoy.svc.cluster.local
                    port_value: 8080

创建Namespace

apiVersion: v1
kind: Namespace
metadata:
  name: envoy
  labels:
    app: envoy

创建Deployment

apiVersion: apps/v1
kind: Deployment
metadata:
  name: envoy
  namespace: envoy
  labels:
    app: envoy
spec:
  replicas: 2
  selector:
    matchLabels:
      app: envoy
  template:
    metadata:
      labels:
        app: envoy
    spec:
      containers:
      - name: envoy
        image: envoyproxy/envoy:v1.35.0
        ports:
        - name: http
          containerPort: 80
        - name: admin
          containerPort: 9901
        volumeMounts:
        - name: envoy-config
          mountPath: /etc/envoy
          readOnly: true
        command: ["envoy"]
        args: ["-c", "/etc/envoy/envoy.yaml"]
        resources:
          requests:
            cpu: 100m
            memory: 256Mi
          limits:
            cpu: 500m
            memory: 512Mi
        livenessProbe:
          httpGet:
            path: /server_info
            port: 9901
          initialDelaySeconds: 10
          periodSeconds: 10
        readinessProbe:
          httpGet:
            path: /server_info
            port: 9901
          initialDelaySeconds: 5
          periodSeconds: 5
      volumes:
      - name: envoy-config
        configMap:
          name: envoy-config

创建Service

apiVersion: v1
kind: Service
metadata:
  name: envoy
  namespace: envoy
  labels:
    app: envoy
spec:
  type: ClusterIP
  ports:
  - name: http
    port: 80
    targetPort: 80
    protocol: TCP
  - name: admin
    port: 9901
    targetPort: 9901
    protocol: TCP
  selector:
    app: envoy

部署到集群

# 创建namespace
kubectl apply -f namespace.yaml

# 创建configmap
kubectl apply -f configmap.yaml

# 创建deployment
kubectl apply -f deployment.yaml

# 创建service
kubectl apply -f service.yaml

# 查看部署状态
kubectl get pods -n envoy
kubectl get svc -n envoy

方式二：使用Helm部署Envoy

添加Helm仓库

# 添加Bitnami仓库
helm repo add bitnami https://charts.bitnami.com/bitnami

# 更新仓库
helm repo update

# 搜索Envoy Chart
helm search repo envoy

# 查看Chart信息
helm show chart bitnami/envoy

自定义Values配置

cat > envoy-values.yaml

replicaCount: 2

image:
  repository: envoyproxy/envoy
  tag: v1.35.0
  pullPolicy: IfNotPresent

service:
  type: ClusterIP
  ports:
    http: 80
    admin: 9901

resources:
  requests:
    cpu: 100m
    memory: 256Mi
  limits:
    cpu: 500m
    memory: 512Mi

livenessProbe:
  httpGet:
    path: /server_info
    port: 9901
  initialDelaySeconds: 10
  periodSeconds: 10

readinessProbe:
  httpGet:
    path: /server_info
    port: 9901
  initialDelaySeconds: 5
  periodSeconds: 5

configmap:
  existingConfigmap: envoy-config

autoscaling:
  enabled: true
  minReplicas: 2
  maxReplicas: 10
  targetCPUUtilizationPercentage: 70

部署Envoy

# 创建namespace
kubectl create namespace envoy

# 使用Helm部署
helm install envoy bitnami/envoy   --namespace envoy   --values envoy-values.yaml

# 或者从Chart目录安装
helm install envoy ./envoy-chart   --namespace envoy   --values envoy-values.yaml

# 查看部署状态
kubectl get pods -n envoy
kubectl get svc -n envoy

# 查看Helm release
helm list -n envoy

升级和回滚

# 升级版本
helm upgrade envoy bitnami/envoy   --namespace envoy   --set image.tag=v1.36.0

# 回滚到上一版本
helm rollback envoy 1 -n envoy

# 查看发布历史
helm history envoy -n envoy

# 卸载
helm uninstall envoy -n envoy

高级配置

1. 配置Envoy为Sidecar代理

在每个Pod中注入Envoy Sidecar，实现服务网格功能。

apiVersion: v1
kind: ConfigMap
metadata:
  name: envoy-sidecar-config
  namespace: default
data:
  envoy.yaml: |
    admin:
      address:
        socket_address:
          protocol: TCP
          address: 127.0.0.1
          port_value: 9901

    static_resources:
      listeners:
      - name: envoy redir
        address:
          socket_address:
            protocol: TCP
            address: 127.0.0.1
            port_value: 15001
        filter_chains:
        - filters:
          - name: envoy.filters.network.tcp_proxy
            typed_config:
              "@type": type.googleapis.com/envoy.extensions.filters.network.tcp_proxy.v3.TcpProxy
              stat_prefix: tcp proxy

      clusters:
      - name: outbound|443||example.com
        type: STRICT_DNS
        connect_timeout: 1s
        load_assignment:
          cluster_name: outbound|443||example.com
          endpoints:
          - lb_endpoints:
            - endpoint:
                address:
                  socket_address:
                    address: example.com
                    port_value: 443

2. 配置Envoy Gateway

使用Envoy作为API网关，统一入口。

apiVersion: v1
kind: ConfigMap
metadata:
  name: envoy-gateway-config
  namespace: envoy
data:
  envoy.yaml: |
    admin:
      address:
        socket_address:
          protocol: TCP
          address: 0.0.0.0
          port_value: 9901

    static_resources:
      listeners:
      - name: http gateway
        address:
          socket_address:
            protocol: TCP
            address: 0.0.0.0
            port_value: 80
        filter_chains:
        - filters:
          - name: envoy.filters.network.http_connection_manager
            typed_config:
              "@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
              codec_type: AUTO
              stat_prefix: gateway
              http_filters:
              - name: envoy.filters.http.router
              route_config:
                name: gateway routes
                virtual_hosts:
                - name: api vhost
                  domains: ["api.example.com"]
                  routes:
                  # 用户服务路由
                  - match:
                      prefix: /users
                    route:
                      cluster: users-service
                      prefix_rewrite: /
                      timeout: 5s
                  # 订单服务路由
                  - match:
                      prefix: /orders
                    route:
                      cluster: orders-service
                      prefix_rewrite: /
                      timeout: 10s
                  # 产品服务路由
                  - match:
                      prefix: /products
                    route:
                      cluster: products-service
                      prefix_rewrite: /
                      timeout: 5s

      clusters:
      - name: users-service
        connect_timeout: 5s
        type: STRICT_DNS
        lb_policy: ROUND_ROBIN
        load_assignment:
          cluster_name: users-service
          endpoints:
          - lb_endpoints:
            - endpoint:
                address:
                  socket_address:
                    address: users-service.default.svc.cluster.local
                    port_value: 8080

      - name: orders-service
        connect_timeout: 5s
        type: STRICT_DNS
        lb_policy: ROUND_ROBIN
        load_assignment:
          cluster_name: orders-service
          endpoints:
          - lb_endpoints:
            - endpoint:
                address:
                  socket_address:
                    address: orders-service.default.svc.cluster.local
                    port_value: 8080

      - name: products-service
        connect_timeout: 5s
        type: STRICT_DNS
        lb_policy: ROUND_ROBIN
        load_assignment:
          cluster_name: products-service
          endpoints:
          - lb_endpoints:
            - endpoint:
                address:
                  socket_address:
                    address: products-service.default.svc.cluster.local
                    port_value: 8080

3. 配置TLS终止

# 创建TLS Secret
kubectl create secret tls envoy-tls-secret   --cert=server.crt   --key=server.key   -n envoy

# ConfigMap中配置TLS
apiVersion: v1
kind: ConfigMap
metadata:
  name: envoy-tls-config
  namespace: envoy
data:
  envoy.yaml: |
    admin:
      address:
        socket_address:
          protocol: TCP
          address: 0.0.0.0
          port_value: 9901

    static_resources:
      listeners:
      - name: https listener
        address:
          socket_address:
            protocol: TCP
            address: 0.0.0.0
            port_value: 443
        filter_chains:
        - filters:
          - name: envoy.filters.network.http_connection_manager
            typed_config:
              "@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
              codec_type: AUTO
              stat_prefix: https gateway
              route_config:
                name: local route
                virtual_hosts:
                - name: backend
                  domains: ["*"]
                  routes:
                  - match:
                      prefix: "/"
                    route:
                      cluster: backend
          transport_socket:
            name: envoy.transport_sockets.tls
            typed_config:
              "@type": type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.DownstreamTlsContext
              common_tls_context:
                tls_certificates:
                - certificate_chain:
                    filename: /etc/envoy/tls/server.crt
                  private_key:
                    filename: /etc/envoy/tls/server.key

      clusters:
      - name: backend
        connect_timeout: 5s
        type: STRICT_DNS
        load_assignment:
          cluster_name: backend
          endpoints:
          - lb_endpoints:
            - endpoint:
                address:
                  socket_address:
                    address: backend-service.envoy.svc.cluster.local
                    port_value: 8080

4. Deployment挂载TLS Secret

apiVersion: apps/v1
kind: Deployment
metadata:
  name: envoy-tls
  namespace: envoy
spec:
  replicas: 2
  selector:
    matchLabels:
      app: envoy-tls
  template:
    metadata:
      labels:
        app: envoy-tls
    spec:
      containers:
      - name: envoy
        image: envoyproxy/envoy:v1.35.0
        ports:
        - containerPort: 443
        - containerPort: 9901
        volumeMounts:
        - name: envoy-config
          mountPath: /etc/envoy
          readOnly: true
        - name: tls-cert
          mountPath: /etc/envoy/tls
          readOnly: true
        command: ["envoy"]
        args: ["-c", "/etc/envoy/envoy.yaml"]
      volumes:
      - name: envoy-config
        configMap:
          name: envoy-tls-config
      - name: tls-cert
        secret:
          secretName: envoy-tls-secret

5. 配置流量分割

route_config:
  name: traffic splitting
  virtual_hosts:
  - name: api vhost
    domains: ["*"]
    routes:
    # A/B测试：50%流量到版本A，50%到版本B
    - match:
        prefix: /api/v1
      route:
        weighted_clusters:
          clusters:
          - name: service-v1
            weight: 50
          - name: service-v2
            weight: 50
    
    # 金丝雀发布：10%流量到新版本
    - match:
        prefix: /api/canary
      route:
        weighted_clusters:
          clusters:
          - name: service-stable
            weight: 90
          - name: service-canary
            weight: 10

Ingress配置

使用Kubernetes Ingress

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: envoy-ingress
  namespace: envoy
  annotations:
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/proxy-body-size: "50m"
    nginx.ingress.kubernetes.io/proxy-connect-timeout: "30"
    nginx.ingress.kubernetes.io/proxy-read-timeout: "300"
    nginx.ingress.kubernetes.io/proxy-send-timeout: "300"
spec:
  rules:
  - host: api.example.com
    http:
      paths:
      - path: /users
        pathType: Prefix
        backend:
          service:
            name: users-service
            port:
              number: 8080
      - path: /orders
        pathType: Prefix
        backend:
          service:
            name: orders-service
            port:
              number: 8080

使用Envoy Gateway（推荐）

# 安装Envoy Gateway
helm repo add envoy-gateway https://envoygateway.io/charts
helm install envoy-gateway envoy-gateway/envoy-gateway -n envoy-gateway --create-namespace

# 创建Gateway
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: envoy-gateway
  namespace: envoy-gateway
spec:
  gatewayClassName: envoy-gateway
  listeners:
  - name: http
    port: 80
    protocol: HTTP
    allowedRoutes:
      namespaces:
        from: Selector
        selector:
          matchLabels:
            shared-gateway: "true"

# 创建HTTPRoute
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: backend-route
  namespace: default
spec:
  parentRefs:
  - name: envoy-gateway
    namespace: envoy-gateway
  rules:
  - matches:
    - path:
        value: /api/users
    backendRefs:
    - name: users-service
      port: 8080
  - matches:
    - path:
        value: /api/orders
    backendRefs:
    - name: orders-service
      port: 8080

监控和可观测性

1. 配置Prometheus监控

# ServiceMonitor配置
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: envoy-monitor
  namespace: monitoring
spec:
  selector:
    matchLabels:
      app: envoy
  namespaceSelector:
    matchNames:
    - envoy
  endpoints:
  - port: admin
    path: /stats/prometheus
    interval: 15s

# 查看Envoy指标
kubectl exec -n envoy -it $(kubectl get pod -n envoy -l app=envoy -o jsonpath='{.items[0].metadata.name}')   -- curl localhost:9901/stats/prometheus | head -20

2. 配置日志收集

# ConfigMap中配置访问日志
envoy.yaml: |
  admin:
    address:
      socket_address:
        protocol: TCP
        address: 0.0.0.0
        port_value: 9901

  static_resources:
    listeners:
    - name: http listener
      address:
        socket_address:
          protocol: TCP
          address: 0.0.0.0
          port_value: 80
      filter_chains:
      - filters:
        - name: envoy.filters.network.http_connection_manager
          typed_config:
            "@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
            codec_type: AUTO
            stat_prefix: http
            access_log:
            - name: envoy.access_loggers.file
              typed_config:
                "@type": type.googleapis.com/envoy.extensions.access_loggers.file.v3.FileAccessLog
                path: /var/log/envoy/access.log
                log_format:
                  text_format: "[%Y-%m-%dT%T.%fZ] %RESPONSE_CODE% %RESPONSE_FLAGS% %BYTES_RECEIVED% %BYTES_SENT% %UPSTREAM_REQU%
"

3. 配置分布式追踪

envoy.yaml: |
  admin:
    address:
      socket_address:
        protocol: TCP
        address: 0.0.0.0
        port_value: 9901

  static_resources:
    listeners:
    - name: http listener
      address:
        socket_address:
          protocol: TCP
          address: 0.0.0.0
          port_value: 80
      filter_chains:
      - filters:
        - name: envoy.filters.network.http_connection_manager
          typed_config:
            "@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
            codec_type: AUTO
            stat_prefix: http
            http_filters:
            - name: envoy.filters.http.router
              typed_config:
                "@type": type.googleapis.com/envoy.extensions.filters.http.router.v3.Router
                dynamic_stats: true
            route_config:
              name: local route
              virtual_hosts:
              - name: backend
                domains: ["*"]
                routes:
                - match:
                    prefix: "/"
                  route:
                    cluster: backend

  clusters:
  - name: backend
    connect_timeout: 5s
    type: STRICT_DNS
    load_assignment:
      cluster_name: backend
      endpoints:
      - lb_endpoints:
        - endpoint:
            address:
              socket_address:
                address: backend-service.default.svc.cluster.local
                port_value: 8080

高可用配置

1. Pod反亲和性

apiVersion: apps/v1
kind: Deployment
metadata:
  name: envoy-ha
  namespace: envoy
spec:
  replicas: 3
  selector:
    matchLabels:
      app: envoy-ha
  template:
    metadata:
      labels:
        app: envoy-ha
    spec:
      affinity:
        podAntiAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
          - weight: 100
            podAffinityTerm:
              labelSelector:
                matchExpressions:
                - key: app
                  operator: In
                  values:
                  - envoy-ha
              topologyKey: kubernetes.io/hostname
      containers:
      - name: envoy
        image: envoyproxy/envoy:v1.35.0
        ports:
        - containerPort: 80
        - containerPort: 9901
        volumeMounts:
        - name: envoy-config
          mountPath: /etc/envoy
      volumes:
      - name: envoy-config
        configMap:
          name: envoy-config

2. Pod中断预算

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: envoy-pdb
  namespace: envoy
spec:
  minAvailable: 2
  selector:
    matchLabels:
      app: envoy

3. Horizontal Pod Autoscaler

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: envoy-hpa
  namespace: envoy
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: envoy
  minReplicas: 2
  maxReplicas: 10
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 70
  - type: Resource
    resource:
      name: memory
      target:
        type: Utilization
        averageUtilization: 80

故障排查

常见问题

Q1: Envoy Pod无法启动

问题描述：Pod状态为CrashLoopBackOff或Error

解决方法：

检查Pod日志
验证配置文件语法
确认ConfigMap正确挂载

# 查看Pod日志
kubectl logs -n envoy envoy-7d8f9c4d5f-x8k2m

# 查看详细事件
kubectl describe pod -n envoy envoy-7d8f9c4d5f-x8k2m

# 验证ConfigMap
kubectl get configmap envoy-config -n envoy -o yaml

# 进入Pod调试
kubectl exec -n envoy -it envoy-7d8f9c4d5f-x8k2m -- /bin/sh

Q2: 路由不生效

问题描述：请求返回404或无法到达后端服务

解决方法：

检查路由配置是否正确
验证后端Service是否存在
确认DNS解析正常

# 检查Service是否存在
kubectl get svc -n default

# 测试DNS解析
kubectl exec -n envoy envoy-7d8f9c4d5f-x8k2m -- nslookup backend-service.default.svc.cluster.local

# 检查Envoy管理界面
kubectl port-forward -n envoy envoy-7d8f9c4d5f-x8k2m 9901:9901 &
curl http://localhost:9901/clusters
curl http://localhost:9901/config_dump

Q3: 负载均衡不均衡

问题描述：流量分配不均匀

解决方法：

检查负载均衡策略配置
验证健康检查状态
查看详细的集群统计信息

# 查看集群统计
curl http://localhost:9901/clusters | grep backend-service

# 查看负载均衡决策
curl http://localhost:9901/runtime

# 检查健康状态
kubectl get endpoints -n default backend-service

Q4: TLS配置问题

问题描述：HTTPS连接失败或证书错误

解决方法：

确认Secret已正确创建
验证证书有效期
检查证书链完整性

# 验证Secret
kubectl get secret envoy-tls-secret -n envoy -o yaml

# 检查证书
kubectl exec -n envoy envoy-tls-xxx -- cat /etc/envoy/tls/server.crt | openssl x509 -noout -dates

# 测试HTTPS连接
curl -v https://api.example.com --resolve api.example.com:443:<envoy-ip>

总结

在Kubernetes中部署Envoy可以为企业级应用提供高性能的流量管理能力。通过本文的介绍，你应该已经掌握了以下内容：

使用原生方式部署Envoy（ConfigMap + Deployment + Service）
使用Helm Chart快速部署Envoy
配置Envoy Gateway实现API网关功能
配置TLS终止和流量分割
配置监控和可观测性
实现高可用部署
故障排查方法

建议根据实际业务需求选择合适的部署方式，并结合监控工具持续观察系统表现。

参考资源

Envoy官方文档：https://www.envoyproxy.io/docs/envoy/latest
Envoy GitHub：https://github.com/envoyproxy/envoy
Bitnami Envoy Chart：https://github.com/bitnami/charts/tree/main/bitnami/envoy
Envoy Gateway：https://gateway.envoyproxy.io/

声明：本站所有文章，如无特殊说明或标注，均为本站原创发布。任何个人或组织，在未征得本站同意时，禁止复制、盗用、采集、发布本站内容到任何网站、书籍等各类媒体平台。如若本站内容侵犯了原著者的合法权益，可联系我们进行处理。

概述

前置条件

部署方式选择

方式一：原生部署Envoy

创建ConfigMap存储配置文件

创建Namespace

创建Deployment

创建Service

部署到集群

方式二：使用Helm部署Envoy

添加Helm仓库

自定义Values配置

部署Envoy

升级和回滚

高级配置

1. 配置Envoy为Sidecar代理

2. 配置Envoy Gateway

3. 配置TLS终止

4. Deployment挂载TLS Secret

5. 配置流量分割

Ingress配置

使用Kubernetes Ingress

使用Envoy Gateway（推荐）

监控和可观测性

1. 配置Prometheus监控

2. 配置日志收集

3. 配置分布式追踪

高可用配置

1. Pod反亲和性

2. Pod中断预算

3. Horizontal Pod Autoscaler

故障排查

常见问题

Q1: Envoy Pod无法启动

Q2: 路由不生效

Q3: 负载均衡不均衡

Q4: TLS配置问题

总结

参考资源

相关文章

发表回复 取消回复

发表回复取消回复