Websocket 在客户端返回 500，在服务器返回 101

我们尝试使用 nginx-ingress 控制器在 Kubernetes 集群上实现 WebSocket。

入口.yaml：

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    ingress.kubernetes.io/proxy-read-timeout: "7200"
    ingress.kubernetes.io/proxy-send-timeout: "7200"
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/backend-protocol: HTTPS
  generation: 2
  labels:
    app: websocket
    app.kubernetes.io/managed-by: Helm
    chart: websocket-0.2.2693
    release: websocket
  name: websocket
  namespace: %NAME_SPACE%
spec:
  rules:
  - host: %HOST_NAME%
    http:
      paths:
      - backend:
          serviceName: websocket
          servicePort: 443
        path: /
      - backend:
          serviceName: websocket
          servicePort: 443
        path: /socket.io
status:
  loadBalancer:
    ingress:
    - ip: X.X.X.X
    - ip: Y.Y.Y.Y

服务.yaml

apiVersion: v1
kind: Service
metadata:
  annotations:
    meta.helm.sh/release-name: websocket
    meta.helm.sh/release-namespace: %NAME_SPACE%
  creationTimestamp: "2020-04-27T20:58:28Z"
  labels:
    app: websocket
    app.kubernetes.io/managed-by: Helm
    chart: websocket-0.2.2723
    release: websocket
  name: websocket
  namespace: %NAME_SPACE%
  resourceVersion: "2916073"
  selfLink: /api/v1/namespaces/%NAME_SPACE%/services/websocket
  uid: e4c08a00-6824-4e16-a3fa-cace0c9be519
spec:
  clusterIP: 10.0.3.45
  ports:
  - name: websocket
    port: 443
    protocol: TCP
    targetPort: 443
  selector:
    app: websocket
    release: websocket
  sessionAffinity: None
  type: ClusterIP
status:
  loadBalancer: {}

部署.yaml：

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  annotations:
    deployment.kubernetes.io/revision: "16"
    meta.helm.sh/release-name: websocket
  labels:
    app: websocket
    release: websocket
  name: websocket
spec:
  progressDeadlineSeconds: 600
  replicas: 1
  revisionHistoryLimit: 5
  selector:
    matchLabels:
      app: websocket
      release: websocket
  strategy:
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 0
    type: RollingUpdate
  template:
    spec:
      containers:
        image: %NAME_SPACE%.azurecr.io/websocket:2723
        imagePullPolicy: Always
        name: websocket
        ports:
        - containerPort: 443
          name: websocket
          protocol: TCP
        resources:
          limits:
            cpu: 1500m
            memory: 1Gi
          requests:
            cpu: 250m
            memory: 64Mi
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
      dnsPolicy: ClusterFirst
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      terminationGracePeriodSeconds: 30

从 nginx-ingress pod 获取日志时，我们注意到状态代码是101这意味着它正在工作。

kubectl logs %POD_NAME% -n nginx-ingress --since 1m | grep websocket

Output:

[11/May/2020:12:47:29 +0000] "GET /socket.io/?EIO=3&transport=websocket HTTP/1.1" 101 91

然而，客户返回500 :

Error during WebSocket handshake: Unexpected response code: 500

并且连接会自动关闭。

我熟悉以下线程，但没有一个对我有用：

https://gist.github.com/jsdevtom/7045c03c021ce46b08cb3f41db0d76da#file-ingress-service-yaml https://gist.github.com/jsdevtom/7045c03c021ce46b08cb3f41db0d76da#file-ingress-service-yaml

https://github.com/kubernetes/ingress-nginx/issues/3746 https://github.com/kubernetes/ingress-nginx/issues/3746

任何帮助都感激不尽。

看起来问题出在 Cloudflare 方面，我们针对不同的域测试了当前的解决方案，并且它运行良好。

一些可能有帮助的文章和答案：

cloudflare-和-socket-io https://stackoverflow.com/a/38704819/9931092

Socket.io + nginx + cloudflare 问题 https://community.cloudflare.com/t/socket-io-nginx-cloudflare-problems/107450/9

使用 cloudflare 通过 https 进行 Socket.io https://community.cloudflare.com/t/socket-io-over-https-with-cloudflare/76612/25