业务Deployment滚动更新时gateway异常修复方案
1. 问题场景
smt工作负载"重新创建"滚动更新时, gateway出现出错日志,:
2026-01-30 15:15:41.751 WARN 24348 --- [ctor-http-nio-6] [traceId-] r.netty.http.client.HttpClientConnect : [a0701cc5-532, L:/10.10.3.40:11917 ! R:/192.168.174.187:30876] The connection observed an error
reactor.netty.http.client.PrematureCloseException: Connection prematurely closed BEFORE response
2026-01-30 15:15:41.751 WARN 24348 --- [ctor-http-nio-3] [traceId-] r.netty.http.client.HttpClientConnect : [b220be87-556, L:/10.10.3.40:22372 ! R:/192.168.174.187:30876] The connection observed an error
reactor.netty.http.client.PrematureCloseException: Connection prematurely closed BEFORE response
2026-01-30 15:15:41.751 WARN 24348 --- [ctor-http-nio-1] [traceId-] r.netty.http.client.HttpClientConnect : [c9f1229a-545, L:/10.10.3.40:22367 ! R:/192.168.174.187:30876] The connection observed an error
reactor.netty.http.client.PrematureCloseException: Connection prematurely closed BEFORE response
2026-01-30 15:15:41.751 WARN 24348 --- [tor-http-nio-15] [traceId-] r.netty.http.client.HttpClientConnect : [6e0ea3ad-574, L:/10.10.3.40:21015 ! R:/192.168.174.187:30876] The connection observed an error
reactor.netty.http.client.PrematureCloseException: Connection prematurely closed BEFORE response
2026-01-30 15:15:41.751 WARN 24348 --- [ctor-http-nio-4] [traceId-] r.netty.http.client.HttpClientConnect : [becab235-567, L:/10.10.3.40:28535 ! R:/192.168.174.187:30876] The connection observed an error
reactor.netty.http.client.PrematureCloseException: Connection prematurely closed BEFORE response
2026-01-30 15:15:41.751 WARN 24348 --- [tor-http-nio-11] [traceId-] r.netty.http.client.HttpClientConnect : [1617bac1-539, L:/10.10.3.40:44471 ! R:/192.168.174.187:30876] The connection observed an error
reactor.netty.http.client.PrematureCloseException: Connection prematurely closed BEFORE response
2026-01-30 15:15:41.751 WARN 24348 --- [tor-http-nio-16] [traceId-] r.netty.http.client.HttpClientConnect : [73998e1b-523, L:/10.10.3.40:3426 ! R:/192.168.174.187:30876] The connection observed an error
reactor.netty.http.client.PrematureCloseException: Connection prematurely closed BEFORE response
2026-01-30 15:15:41.751 WARN 24348 --- [ctor-http-nio-9] [traceId-] r.netty.http.client.HttpClientConnect : [5da514d8-591, L:/10.10.3.40:27869 ! R:/192.168.174.187:30876] The connection observed an error
reactor.netty.http.client.PrematureCloseException: Connection prematurely closed BEFORE response
2026-01-30 15:15:41.751 WARN 24348 --- [ctor-http-nio-5] [traceId-] r.netty.http.client.HttpClientConnect : [336e1b10-602, L:/10.10.3.40:27861 ! R:/192.168.174.187:30876] The connection observed an error
reactor.netty.http.client.PrematureCloseException: Connection prematurely closed BEFORE response
2026-01-30 15:15:41.751 WARN 24348 --- [ctor-http-nio-7] [traceId-] r.netty.http.client.HttpClientConnect : [91bbd005-593, L:/10.10.3.40:27865 ! R:/192.168.174.187:30876] The connection observed an error
2. 修复方案
问题根因: Pod进行切换时, springcloud-gateway的连接池仍使用旧的TCP连接，旧的Pod发送的RST异常会被上报给springcloud-gateway 应用。
解决方案:
- k8s层面: 使用两个副本; 加入 preStop sleep 机制; 设置terminationGracePeriodSeconds 时间, 减少流量切换的时间窗口;
- 绕开springcloud-gateway, nginx 直接转发到smt;
- springcloud-gateway 设置不使用连接池, 每次HTTP连接都使用一个新连接。
k8s层面:
kind: Deployment
apiVersion: apps/v1
metadata:
name: imom-smt
namespace: imom
labels:
app: imom-smt
component: imom-smt
tier: backend
annotations:
deployment.kubernetes.io/revision: '68'
kubesphere.io/alias-name: imom-smt
kubesphere.io/creator: admin
kubesphere.io/description: imom-smt
spec:
replicas: 2
selector:
matchLabels:
app: imom-smt
component: imom-smt
tier: backend
template:
metadata:
creationTimestamp: null
labels:
app: imom-smt
component: imom-smt
tier: backend
annotations:
kubesphere.io/creator: admin
kubesphere.io/imagepullsecrets: '{}'
kubesphere.io/restartedAt: '2026-01-30T07:11:59.853Z'
spec:
volumes:
- name: sys
hostPath:
path: /sys
type: ''
- name: volume-biz-conf
configMap:
name: biz-conf
items:
- key: common.yml
path: common.yml
- key: smt-mes-biz.yml
path: smt-mes-biz.yml
defaultMode: 420
containers:
- name: imom-smt
image: '192.168.168.207:30002/private/imom-smt:dev-20260112.487'
ports:
- name: http-0
containerPort: 80
protocol: TCP
env:
- name: PROFILES_ACTIVE
value: dev
- name: JAVA_OPTS
value: >-
-Xmx4g -XX:+UseG1GC -XX:MaxMetaspaceSize=512m
-XX:MaxGCPauseMillis=200 -XX:G1ReservePercent=15
resources: {}
volumeMounts:
- name: sys
readOnly: true
mountPath: /host/sys
- name: volume-biz-conf
readOnly: true
mountPath: /springboot_config
livenessProbe:
httpGet:
path: /actuator/health/liveness
port: 80
scheme: HTTP
initialDelaySeconds: 121
timeoutSeconds: 1
periodSeconds: 10
successThreshold: 1
failureThreshold: 3
readinessProbe:
httpGet:
path: /actuator/health/readiness
port: 80
scheme: HTTP
initialDelaySeconds: 120
timeoutSeconds: 5
periodSeconds: 10
successThreshold: 1
failureThreshold: 3
lifecycle:
preStop:
exec:
command:
- sh
- '-c'
- sleep 20
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
imagePullPolicy: IfNotPresent
securityContext:
privileged: true
allowPrivilegeEscalation: true
restartPolicy: Always
terminationGracePeriodSeconds: 61
dnsPolicy: ClusterFirst
nodeSelector:
kubernetes.io/hostname: node2
securityContext: {}
schedulerName: default-scheduler
strategy:
type: RollingUpdate
rollingUpdate:
maxUnavailable: 0
maxSurge: 1
minReadySeconds: 10
revisionHistoryLimit: 10
progressDeadlineSeconds: 600
方式一: 绕开springcloud-gateway, orche-web nginx直连smt
location /sieapi/smt/ {
proxy_pass http://imom-smt.imom/;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
}
方式二: 禁用springcloud-gateway的连接池
spring:
cloud:
gateway:
server:
webflux:
httpclient:
response-timeout: 60000
pool:
type: disabled