技术学习笔记

Appearance

负载均衡 ​

概述 ​

负载均衡是Kubernetes中分发网络流量到多个Pod实例的关键机制。它确保应用的高可用性、可扩展性和性能优化。Kubernetes在多个层面提供负载均衡功能，包括Service级别的四层负载均衡和Ingress级别的七层负载均衡。

负载均衡核心功能 ​

1. 服务负载均衡 ​

  • 自动流量分发
  • 健康检查集成
  • 动态端点更新
  • 多种负载均衡算法

2. 会话保持 ​

  • 基于客户端IP的会话保持
  • 可配置超时时间
  • 支持有状态应用
  • 提高用户体验

3. 外部负载均衡器 ​

  • 云服务商集成
  • 自动配置负载均衡器
  • 支持多种云平台
  • 灵活的流量管理

负载均衡架构 ​

┌─────────────────────────────────────────────────────────┐
│                     外部流量                              │
└────────────────────┬────────────────────────────────────┘
                     │
                     ▼
┌─────────────────────────────────────────────────────────┐
│              外部负载均衡器 (云服务商)                     │
│  ┌──────────────────────────────────────────────────┐  │
│  │  AWS ELB / GCP Load Balancer / Azure LB         │  │
│  └──────────────────────────────────────────────────┘  │
└────────────────────┬────────────────────────────────────┘
                     │
                     ▼
┌─────────────────────────────────────────────────────────┐
│              Service (LoadBalancer/NodePort)            │
│  ┌──────────────────────────────────────────────────┐  │
│  │  ClusterIP: 10.96.0.1                            │  │
│  │  NodePort: 30080                                 │  │
│  └──────────────────────────────────────────────────┘  │
└────────────────────┬────────────────────────────────────┘
                     │
                     ▼
┌─────────────────────────────────────────────────────────┐
│              kube-proxy (iptables/IPVS)                 │
│  ┌──────────────────────────────────────────────────┐  │
│  │  负载均衡规则                                     │  │
│  │  - Round Robin                                   │  │
│  │  - Session Affinity                              │  │
│  └──────────────────────────────────────────────────┘  │
└────────────────────┬────────────────────────────────────┘
                     │
        ┌────────────┼────────────┐
        │            │            │
        ▼            ▼            ▼
    ┌──────┐     ┌──────┐     ┌──────┐
    │ Pod1 │     │ Pod2 │     │ Pod3 │
    │ :8080│     │ :8080│     │ :8080│
    └──────┘     └──────┘     └──────┘

Service负载均衡 ​

1. ClusterIP负载均衡 ​

yaml
apiVersion: v1
kind: Service
metadata:
  name: web-service
  namespace: default
spec:
  type: ClusterIP
  selector:
    app: web
  ports:
  - port: 80
    targetPort: 8080

2. NodePort负载均衡 ​

yaml
apiVersion: v1
kind: Service
metadata:
  name: nodeport-service
  namespace: default
spec:
  type: NodePort
  selector:
    app: web
  ports:
  - port: 80
    targetPort: 8080
    nodePort: 30080
  externalTrafficPolicy: Local

3. LoadBalancer负载均衡 ​

yaml
apiVersion: v1
kind: Service
metadata:
  name: loadbalancer-service
  namespace: default
  annotations:
    service.beta.kubernetes.io/aws-load-balancer-type: nlb
    service.beta.kubernetes.io/aws-load-balancer-cross-zone-load-balancing-enabled: "true"
spec:
  type: LoadBalancer
  selector:
    app: web
  ports:
  - port: 80
    targetPort: 8080
  loadBalancerSourceRanges:
  - 192.168.0.0/16
  - 10.0.0.0/8

4. ExternalName负载均衡 ​

yaml
apiVersion: v1
kind: Service
metadata:
  name: external-service
  namespace: default
spec:
  type: ExternalName
  externalName: external.example.com

会话保持配置 ​

1. 基于客户端IP的会话保持 ​

yaml
apiVersion: v1
kind: Service
metadata:
  name: session-affinity-service
  namespace: default
spec:
  type: ClusterIP
  selector:
    app: web
  ports:
  - port: 80
    targetPort: 8080
  sessionAffinity: ClientIP
  sessionAffinityConfig:
    clientIP:
      timeoutSeconds: 10800

2. 会话保持超时配置 ​

yaml
apiVersion: v1
kind: Service
metadata:
  name: sticky-session-service
  namespace: default
spec:
  type: LoadBalancer
  selector:
    app: web
  ports:
  - port: 80
    targetPort: 8080
  sessionAffinity: ClientIP
  sessionAffinityConfig:
    clientIP:
      timeoutSeconds: 3600

3. 多端口会话保持 ​

yaml
apiVersion: v1
kind: Service
metadata:
  name: multi-port-session-service
  namespace: default
spec:
  type: ClusterIP
  selector:
    app: web
  ports:
  - name: http
    port: 80
    targetPort: 8080
  - name: https
    port: 443
    targetPort: 8443
  sessionAffinity: ClientIP
  sessionAffinityConfig:
    clientIP:
      timeoutSeconds: 7200

kube-proxy负载均衡模式 ​

1. iptables模式 ​

yaml
apiVersion: kubeproxy.config.k8s.io/v1alpha1
kind: KubeProxyConfiguration
mode: iptables
iptables:
  masqueradeAll: false
  masqueradeBit: 14
  minSyncPeriod: 0s
  syncPeriod: 30s

特点：

  • 基于iptables规则
  • 性能较好
  • 不支持会话保持的权重调整
  • 随机负载均衡算法

2. IPVS模式 ​

yaml
apiVersion: kubeproxy.config.k8s.io/v1alpha1
kind: KubeProxyConfiguration
mode: ipvs
ipvs:
  scheduler: rr
  syncPeriod: 30s
  minSyncPeriod: 5s
  excludeCIDRs: []
  strictARP: false
  tcpFinTimeout: 0s
  tcpTimeout: 0s
  udpTimeout: 0s

特点：

  • 支持多种负载均衡算法
  • 性能更好
  • 支持会话保持
  • 支持权重调整

3. IPVS负载均衡算法 ​

Round Robin (rr) ​

yaml
apiVersion: kubeproxy.config.k8s.io/v1alpha1
kind: KubeProxyConfiguration
mode: ipvs
ipvs:
  scheduler: rr

Least Connections (lc) ​

yaml
apiVersion: kubeproxy.config.k8s.io/v1alpha1
kind: KubeProxyConfiguration
mode: ipvs
ipvs:
  scheduler: lc

Destination Hashing (dh) ​

yaml
apiVersion: kubeproxy.config.k8s.io/v1alpha1
kind: KubeProxyConfiguration
mode: ipvs
ipvs:
  scheduler: dh

Source Hashing (sh) ​

yaml
apiVersion: kubeproxy.config.k8s.io/v1alpha1
kind: KubeProxyConfiguration
mode: ipvs
ipvs:
  scheduler: sh

外部负载均衡器配置 ​

1. AWS负载均衡器 ​

yaml
apiVersion: v1
kind: Service
metadata:
  name: aws-loadbalancer
  namespace: default
  annotations:
    service.beta.kubernetes.io/aws-load-balancer-type: nlb
    service.beta.kubernetes.io/aws-load-balancer-internal: "false"
    service.beta.kubernetes.io/aws-load-balancer-cross-zone-load-balancing-enabled: "true"
    service.beta.kubernetes.io/aws-load-balancer-additional-resource-tags: "Environment=prod,Team=backend"
    service.beta.kubernetes.io/aws-load-balancer-healthcheck-healthy-threshold: "3"
    service.beta.kubernetes.io/aws-load-balancer-healthcheck-unhealthy-threshold: "3"
    service.beta.kubernetes.io/aws-load-balancer-healthcheck-interval: "30"
    service.beta.kubernetes.io/aws-load-balancer-healthcheck-timeout: "5"
spec:
  type: LoadBalancer
  selector:
    app: web
  ports:
  - port: 80
    targetPort: 8080
    protocol: TCP

2. GCP负载均衡器 ​

yaml
apiVersion: v1
kind: Service
metadata:
  name: gcp-loadbalancer
  namespace: default
  annotations:
    cloud.google.com/load-balancer-type: Internal
    networking.gke.io/internal-load-balancer-allow-global-access: "true"
    networking.gke.io/internal-load-balancer-subnet: "default"
spec:
  type: LoadBalancer
  selector:
    app: web
  ports:
  - port: 80
    targetPort: 8080
    protocol: TCP
  loadBalancerIP: 10.0.0.100
  loadBalancerSourceRanges:
  - 10.0.0.0/8

3. Azure负载均衡器 ​

yaml
apiVersion: v1
kind: Service
metadata:
  name: azure-loadbalancer
  namespace: default
  annotations:
    service.beta.kubernetes.io/azure-load-balancer-internal: "true"
    service.beta.kubernetes.io/azure-load-balancer-internal-subnet: "app-subnet"
    service.beta.kubernetes.io/azure-load-balancer-tcp-idle-timeout: "4"
spec:
  type: LoadBalancer
  selector:
    app: web
  ports:
  - port: 80
    targetPort: 8080
    protocol: TCP
  loadBalancerIP: 10.0.0.100

4. MetalLB (裸机负载均衡器) ​

yaml
apiVersion: metallb.io/v1beta1
kind: IPAddressPool
metadata:
  name: production-pool
  namespace: metallb-system
spec:
  addresses:
  - 192.168.1.240-192.168.1.250
  - 192.168.1.252
---
apiVersion: metallb.io/v1beta1
kind: L2Advertisement
metadata:
  name: layer2
  namespace: metallb-system
spec:
  ipAddressPools:
  - production-pool
---
apiVersion: v1
kind: Service
metadata:
  name: metallb-service
  namespace: default
  annotations:
    metallb.universe.tf/address-pool: production-pool
    metallb.universe.tf/loadBalancerIPs: 192.168.1.240
spec:
  type: LoadBalancer
  selector:
    app: web
  ports:
  - port: 80
    targetPort: 8080

外部流量策略 ​

1. Cluster策略 ​

yaml
apiVersion: v1
kind: Service
metadata:
  name: cluster-traffic-service
  namespace: default
spec:
  type: LoadBalancer
  externalTrafficPolicy: Cluster
  selector:
    app: web
  ports:
  - port: 80
    targetPort: 8080

特点：

  • 流量可能转发到其他节点
  • 保留源IP：否
  • 性能：较好
  • 适用场景：无状态应用

2. Local策略 ​

yaml
apiVersion: v1
kind: Service
metadata:
  name: local-traffic-service
  namespace: default
spec:
  type: LoadBalancer
  externalTrafficPolicy: Local
  selector:
    app: web
  ports:
  - port: 80
    targetPort: 8080
  healthCheckNodePort: 30000

特点：

  • 只转发到本地节点
  • 保留源IP：是
  • 性能：取决于本地Pod数量
  • 适用场景：需要源IP的应用

操作命令 ​

查看负载均衡状态 ​

bash
# 查看Service
kubectl get svc
kubectl describe svc <service-name>

# 查看Endpoint
kubectl get endpoints <service-name>
kubectl describe endpoints <service-name>

# 查看负载均衡器状态
kubectl get svc <service-name> -o jsonpath='{.status.loadBalancer.ingress}'

# 查看kube-proxy模式
kubectl get configmap -n kube-system kube-proxy -o yaml | grep mode

# 查看IPVS规则
kubectl exec -n kube-system <kube-proxy-pod> -- ipvsadm -Ln

测试负载均衡 ​

bash
# 测试Service负载均衡
for i in {1..10}; do
  kubectl run test-$i --image=busybox --rm -it --restart=Never -- wget -O- http://<service-name>:<port>
done

# 测试会话保持
kubectl run test --image=busybox --rm -it --restart=Never -- sh
for i in {1..10}; do wget -O- http://<service-name>:<port> 2>/dev/null | grep "Server"; done

# 测试负载均衡器
curl http://<loadbalancer-ip>:<port>

# 查看连接分布
kubectl exec -n kube-system <kube-proxy-pod> -- ipvsadm -Ln --stats

配置kube-proxy ​

bash
# 查看kube-proxy配置
kubectl get configmap -n kube-system kube-proxy -o yaml

# 编辑kube-proxy配置
kubectl edit configmap -n kube-system kube-proxy

# 重启kube-proxy
kubectl rollout restart daemonset -n kube-system kube-proxy

# 查看kube-proxy日志
kubectl logs -n kube-system -l k8s-app=kube-proxy

负载均衡器管理 ​

bash
# 创建LoadBalancer Service
kubectl expose deployment <deployment-name> --port=80 --target-port=8080 --type=LoadBalancer

# 查看负载均衡器IP
kubectl get svc <service-name> -o jsonpath='{.status.loadBalancer.ingress[0].ip}'

# 查看负载均衡器域名
kubectl get svc <service-name> -o jsonpath='{.status.loadBalancer.ingress[0].hostname}'

# 删除LoadBalancer Service
kubectl delete svc <service-name>

实践示例 ​

示例1：高可用Web应用负载均衡 ​

yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: web-app
  namespace: production
spec:
  replicas: 5
  selector:
    matchLabels:
      app: web-app
  template:
    metadata:
      labels:
        app: web-app
        version: v1.0.0
    spec:
      containers:
      - name: web-app
        image: nginx:latest
        ports:
        - containerPort: 80
        resources:
          requests:
            cpu: 100m
            memory: 128Mi
          limits:
            cpu: 500m
            memory: 512Mi
        livenessProbe:
          httpGet:
            path: /health
            port: 80
          initialDelaySeconds: 10
          periodSeconds: 10
        readinessProbe:
          httpGet:
            path: /ready
            port: 80
          initialDelaySeconds: 5
          periodSeconds: 5
---
apiVersion: v1
kind: Service
metadata:
  name: web-app-service
  namespace: production
  annotations:
    service.beta.kubernetes.io/aws-load-balancer-type: nlb
    service.beta.kubernetes.io/aws-load-balancer-cross-zone-load-balancing-enabled: "true"
spec:
  type: LoadBalancer
  selector:
    app: web-app
  ports:
  - port: 80
    targetPort: 80
  externalTrafficPolicy: Local

示例2：有状态应用会话保持 ​

yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: shopping-cart
  namespace: production
spec:
  replicas: 3
  selector:
    matchLabels:
      app: shopping-cart
  template:
    metadata:
      labels:
        app: shopping-cart
    spec:
      containers:
      - name: shopping-cart
        image: shopping-cart:v1.0.0
        ports:
        - containerPort: 8080
        env:
        - name: SESSION_STORE
          value: "redis"
        volumeMounts:
        - name: session-data
          mountPath: /data/sessions
      volumes:
      - name: session-data
        emptyDir: {}
---
apiVersion: v1
kind: Service
metadata:
  name: shopping-cart-service
  namespace: production
spec:
  type: LoadBalancer
  selector:
    app: shopping-cart
  ports:
  - port: 80
    targetPort: 8080
  sessionAffinity: ClientIP
  sessionAffinityConfig:
    clientIP:
      timeoutSeconds: 3600

示例3：多区域负载均衡 ​

yaml
apiVersion: v1
kind: Service
metadata:
  name: global-service
  namespace: production
  annotations:
    service.beta.kubernetes.io/aws-load-balancer-type: nlb
    service.beta.kubernetes.io/aws-load-balancer-cross-zone-load-balancing-enabled: "true"
    external-dns.alpha.kubernetes.io/hostname: app.example.com
spec:
  type: LoadBalancer
  selector:
    app: global-app
  ports:
  - port: 80
    targetPort: 8080
  externalTrafficPolicy: Local
  loadBalancerSourceRanges:
  - 0.0.0.0/0
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: global-app
  namespace: production
spec:
  replicas: 3
  selector:
    matchLabels:
      app: global-app
  template:
    metadata:
      labels:
        app: global-app
    spec:
      affinity:
        podAntiAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
          - weight: 100
            podAffinityTerm:
              labelSelector:
                matchExpressions:
                - key: app
                  operator: In
                  values:
                  - global-app
              topologyKey: topology.kubernetes.io/zone
      containers:
      - name: app
        image: app:v1.0.0
        ports:
        - containerPort: 8080

示例4：蓝绿部署负载均衡 ​

yaml
apiVersion: v1
kind: Service
metadata:
  name: blue-green-service
  namespace: production
spec:
  type: LoadBalancer
  selector:
    app: web-app
    version: blue
  ports:
  - port: 80
    targetPort: 8080
  sessionAffinity: ClientIP
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: web-app-blue
  namespace: production
spec:
  replicas: 3
  selector:
    matchLabels:
      app: web-app
      version: blue
  template:
    metadata:
      labels:
        app: web-app
        version: blue
    spec:
      containers:
      - name: web-app
        image: web-app:v1.0.0
        ports:
        - containerPort: 8080
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: web-app-green
  namespace: production
spec:
  replicas: 3
  selector:
    matchLabels:
      app: web-app
      version: green
  template:
    metadata:
      labels:
        app: web-app
        version: green
    spec:
      containers:
      - name: web-app
        image: web-app:v2.0.0
        ports:
        - containerPort: 8080

示例5：金丝雀发布流量控制 ​

yaml
apiVersion: v1
kind: Service
metadata:
  name: canary-service
  namespace: production
spec:
  type: LoadBalancer
  selector:
    app: web-app
  ports:
  - port: 80
    targetPort: 8080
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: web-app-stable
  namespace: production
spec:
  replicas: 9
  selector:
    matchLabels:
      app: web-app
      track: stable
  template:
    metadata:
      labels:
        app: web-app
        track: stable
    spec:
      containers:
      - name: web-app
        image: web-app:v1.0.0
        ports:
        - containerPort: 8080
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: web-app-canary
  namespace: production
spec:
  replicas: 1
  selector:
    matchLabels:
      app: web-app
      track: canary
  template:
    metadata:
      labels:
        app: web-app
        track: canary
    spec:
      containers:
      - name: web-app
        image: web-app:v2.0.0
        ports:
        - containerPort: 8080

故障排查 ​

常见问题 ​

1. 负载均衡不生效 ​

bash
# 检查Service配置
kubectl describe svc <service-name>

# 检查Endpoint
kubectl get endpoints <service-name>
kubectl describe endpoints <service-name>

# 检查Pod标签
kubectl get pods --show-labels

# 检查kube-proxy状态
kubectl get pods -n kube-system -l k8s-app=kube-proxy
kubectl logs -n kube-system -l k8s-app=kube-proxy

# 检查iptables规则
kubectl exec -n kube-system <kube-proxy-pod> -- iptables -t nat -L -n | grep <service-name>

# 检查IPVS规则
kubectl exec -n kube-system <kube-proxy-pod> -- ipvsadm -Ln

2. 会话保持失效 ​

bash
# 检查会话保持配置
kubectl get svc <service-name> -o yaml | grep -A 5 sessionAffinity

# 测试会话保持
kubectl run test --image=busybox --rm -it --restart=Never -- sh
for i in {1..10}; do
  wget -O- http://<service-name>:<port> 2>/dev/null | grep "Server"
done

# 检查kube-proxy模式
kubectl get configmap -n kube-system kube-proxy -o yaml | grep mode

# 检查IPVS持久化服务
kubectl exec -n kube-system <kube-proxy-pod> -- ipvsadm -Ln --persistent-conn

3. 外部负载均衡器问题 ​

bash
# 检查LoadBalancer状态
kubectl describe svc <service-name>

# 查看LoadBalancer事件
kubectl get events --field-selector involvedObject.name=<service-name>

# 检查云服务商配置
kubectl get svc <service-name> -o yaml | grep annotations -A 20

# 检查节点状态
kubectl get nodes
kubectl describe nodes

# 检查安全组/防火墙规则
# AWS: aws ec2 describe-security-groups
# GCP: gcloud compute firewall-rules list
# Azure: az network nsg list

4. 流量分配不均 ​

bash
# 检查Pod分布
kubectl get pods -o wide

# 检查节点资源
kubectl top nodes
kubectl describe nodes

# 检查Endpoint分布
kubectl get endpoints <service-name> -o yaml

# 查看连接统计
kubectl exec -n kube-system <kube-proxy-pod> -- ipvsadm -Ln --stats

# 检查Pod资源限制
kubectl describe pod <pod-name>

5. 源IP丢失 ​

bash
# 检查外部流量策略
kubectl get svc <service-name> -o yaml | grep externalTrafficPolicy

# 测试源IP
kubectl run test --image=nicolaka/netshoot --rm -it --restart=Never -- curl -s http://<service-name>:<port>/api/client-ip

# 修改为Local策略
kubectl patch svc <service-name> -p '{"spec":{"externalTrafficPolicy":"Local"}}'

# 检查节点网络配置
kubectl exec -n kube-system <kube-proxy-pod> -- iptables -t nat -L -n -v

6. 性能问题 ​

bash
# 检查kube-proxy资源使用
kubectl top pods -n kube-system -l k8s-app=kube-proxy

# 检查连接数
kubectl exec -n kube-system <kube-proxy-pod> -- netstat -an | grep ESTABLISHED | wc -l

# 检查iptables规则数量
kubectl exec -n kube-system <kube-proxy-pod> -- iptables -t nat -L -n | wc -l

# 切换到IPVS模式
kubectl get configmap -n kube-system kube-proxy -o yaml | sed 's/mode: ""/mode: "ipvs"/' | kubectl apply -f -
kubectl rollout restart daemonset -n kube-system kube-proxy

# 监控负载均衡性能
kubectl port-forward -n kube-system svc/kube-proxy-metrics 10249:10249
curl http://localhost:10249/metrics

最佳实践 ​

1. 架构设计 ​

  • 使用多副本部署
  • 跨可用区分布
  • 配置Pod反亲和性
  • 合理设置资源限制

2. 负载均衡策略 ​

  • 无状态应用使用默认轮询
  • 有状态应用使用会话保持
  • 需要源IP使用Local策略
  • 大规模集群使用IPVS模式

3. 会话保持 ​

  • 合理设置超时时间
  • 考虑使用外部会话存储
  • 监控会话分布
  • 测试故障转移

4. 外部负载均衡器 ​

  • 启用跨区域负载均衡
  • 配置健康检查
  • 设置合理的超时
  • 使用内网负载均衡器

5. 性能优化 ​

  • 使用IPVS模式
  • 调整连接超时
  • 启用连接复用
  • 监控性能指标

6. 监控和告警 ​

  • 监控连接数
  • 监控流量分布
  • 监控响应时间
  • 设置异常告警

7. 安全配置 ​

  • 限制访问源IP
  • 启用网络策略
  • 使用TLS加密
  • 定期安全审计

总结 ​

负载均衡是Kubernetes实现高可用和可扩展性的关键机制。通过合理配置Service、会话保持和外部负载均衡器，可以构建稳定、高效的应用架构。

下一步学习 ​