Apisix基于SNI代理TCP、UDP

30, 2023 k8s

背景: 公司项目中，需要暴露对内网的Websocket服务。

Websocket 它是一种基于 TCP 的一种独立实现，APISIX 可以对 TCP/UDP 协议进行代理并实现动态负载均衡，在 nginx 世界，称 TCP/UDP 代理为 stream 代理，在 APISIX 这里我们也遵循了这个声明，下文统称Stream。

SNI（Server Name Indication）是用来改善 SSL 和 TLS 的一项特性，它允许客户端在服务器端向其发送证书之前向服务器端发送请求的域名，服务器端根据客户端请求的域名选择合适的 SSL 证书发送给客户端。

本文主要介绍ApiSix如何基于SNI代理TCP/UDP。

启用Stream 代理

在 conf/config.yaml 配置文件设置 stream_proxy 选项，指定一组需要进行动态代理的 IP 地址。默认情况不开启 stream 代理。

apisix:
  stream_proxy: # TCP/UDP proxy
    tcp: # TCP proxy address list
      - 9100
      - "127.0.0.1:9101"
    udp: # UDP proxy address list
      - 9200
      - "127.0.0.1:9211"

如果 apisix.enable_admin 为 true，上面的配置会同时启用 HTTP 和 stream 代理。如果你设置 enable_admin 为 false，且需要同时启用 HTTP 和 stream 代理，设置 only 为 false：

apisix:
  enable_admin: false
  stream_proxy: # TCP/UDP proxy
    only: false
    tcp: # TCP proxy address list
      - 9100

启用 TLS：

首先，我们需要给对应的 TCP 地址启用 TLS

  stream_proxy: # TCP/UDP proxy
    only: false
    tcp: # TCP proxy address list
      - 9100
      - addr: 9333
        tls: true

配置路由

当连接为 TLS over TCP 时，我们可以通过 SNI 来匹配路由，比如：

curl http://127.0.0.1:9180/apisix/admin/stream_routes/1 -H 'X-API-KEY: Your-Api-Key' -X PUT -d '
  {
      "sni": "mysite.com",
      "server_port": 9333,
      "upstream": {
          "scheme": "tls",
          "nodes": {
              "192.168.1.33:8080": 1
          },
          "type": "roundrobin"
      }
  }'

查看是否创建成功:

curl http://127.0.0.1:9180/apisix/admin/stream_routes/1 -H 'X-API-KEY: Your-Api-Key'

如创建错误，可以删除

curl http://127.0.0.1:9180/apisix/admin/stream_routes/1 -H 'X-API-KEY: Your-Api-Key' -X DELETE

查看更多stream路由：

curl http://127.0.0.1:9180//apisix/admin/stream_routes -H 'X-API-KEY: Your-Api-Key'

上传SNI证书

上传站点(mysite.com)的证书到ApiSix

测试

curl -v https://mysite.com:9333/
*   Trying x.x.y.y...
* TCP_NODELAY set
* Connected to mysite.com:19333 (x.x.y.y) port 9333 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/cert.pem
  CApath: none
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-CHACHA20-POLY1305
* ALPN, server did not agree to a protocol
* Server certificate:
*  subject: CN=*.mysite.com
*  start date: Mar 29 00:00:00 2023 GMT
*  expire date: Jun 27 23:59:59 2023 GMT
*  subjectAltName: host "mysite.com" matched cert's "*.mysite.com"
*  issuer: C=AT; O=ZeroSSL; CN=ZeroSSL RSA Domain Secure Site CA
*  SSL certificate verify ok.
> GET / HTTP/1.1
> Host: mysite.com:9333
> User-Agent: curl/7.64.1
> Accept: */*
>
< HTTP/1.1 400 Bad Request
< Content-Type: text/plain; charset=utf-8
< Sec-Websocket-Version: 13
< X-Content-Type-Options: nosniff
< Date: Sun, 23 Apr 2023 05:50:07 GMT
< Content-Length: 12
<
Bad Request
* Connection #0 to host mysite.com left intact
* Closing connection 0

Success。

过程中遇见的问题

连接不上

 curl -v http://mysite.com
*   Trying x.x.y.y...
* TCP_NODELAY set
* Connected to mysite.com (x.x.y.y.) port 19333
> GET / HTTP/1.1
> Host: mysite.com:19333
> User-Agent: curl/7.64.1
> Accept: */*
>
* Recv failure: Connection reset by peer
* Closing connection 0
curl: (56) Recv failure: Connection reset by peer

需要排查路由器端口是否正常开放，ApiSix的端口是否正常，经排查原因是：配置路由时"server_port"缺失

{
      "sni": "mysite.com",
      "upstream": {
          "scheme": "tls",
          "nodes": {
              "192.168.1.33:8080": 1
          },
          "type": "roundrobin"
      }
  }

handshark失败，证书错误

* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* LibreSSL SSL_connect: SSL_ERROR_SYSCALL in connection to mysite.com:19333
* Closing connection 0
curl: (35) LibreSSL SSL_connect: SSL_ERROR_SYSCALL in connection to mysite.com:19333

排位原因是：这里作者使用的ApiSix版本是2.12，缺失前缀【addr】

  stream_proxy: # TCP/UDP proxy
    only: false
    tcp: # TCP proxy address list
      - 9100
      - 9333
        tls: true