• 灰度发布
• 微服务全链路灰度
• 全链路灰度设计思路
  • 标签路由
  • 节点打标
  • 流量染色
  • 分布式链路追踪 ThreadLocal
  • 流量治理平台 Nacos 配置中心
• 全链路灰度实现
• Discovery
• 使用
  • 一、父pom引入Discovery
  • 二、Gateway 引入 — 网关 discovery-plugin-strategy-starter-gateway
  • 三、微服务 引入 — 网关 discovery-plugin-strategy-starter-service
  • 四、Gateway 配置 权重、流量百分比、Header参数
• 原理分析
  • Openfeign 通过 RequestInterceptor
  • Gateway 通过 GlobalFilter
  • 通过IRule
  • 通过 pluginAdapter 可以拿到需要的数据
• MSE 微服务治理全链路灰度 太贵

灰度发布

不停机旧版本，部署新版 本，高比例流量（例如：95%）走旧版本，低比例流量（例如：5%）切换到新版本，通过 监控观察无问题，逐步扩大范围，最终把所有流量都迁移到新版本上。属无损发布。

• 优点
  灵活简单，不需要用户标记驱动。
  安全性高，新版本如果出现问题，只会发生在低比例的流 量上
• 缺点
  成本较高，需要部署稳定/灰度两套环境

微服务全链路灰度

在发布过程中，我们只需部署服务的灰度版本，流量在调用链路上流转 时，由流经的网关、各个中间件以及各个微服务来识别灰度流量，并动态转发至对应服务的 灰度版本

无论是微服务网关还是微服务本身都需要识别流量，根据治理规则做出动态决策。当服务 版本发生变化时，这个调用链路的转发也会实时改变

全链路灰度设计思路

1. 标签路由

通过对服务下所有节点按照标签名和标签值不同进行分组，使得订阅该服务节点信 息的服务消费端可以按需访问该服务的某个分组

2. 节点打标

• k8s – labels.version=gray
  在使用Kubernetes Service作为服务发现的业务系统中，服务提供者通过向ApiServer提交 Service资源完成服务暴露，服务消费端监听与该Service资源下关联的Endpoint资源，从 Endpoint资源中获取关联的业务Pod 资源，读取上面的Labels数据并作为该节点的元数据 信息。所以，我们只要在业务应用描述资源Deployment中的Pod模板中为节点添加标签即可。
  

• nacos – `spring.cloud.nacos.discovery.metadata.version=gray

spring:cloud:discovery:server-addr: nacos.localhost.com:8848metadata:version: 1.1

3. 流量染色

• 可以在请求的源头上对 流量进行染色
• 可以在微服务网关上对匹配特定路由规则的请求动态添加流量标识 (Gateway GlobalFilter)
• 流量在链路中流经灰度节点时，如果请求信息中不含有灰度标识，需要自动为其染色 (Openfeign RequestInterceptor)

4. 分布式链路追踪 ThreadLocal

• 借助于分布式链路追踪思想，我们也可以传递一些自定义信息，比如灰度标识

5. 流量治理平台 Nacos 配置中心

• 需要引入一个中心化的流量治理平台， 方便各个业务线的开发者定义自己的全链路灰度规则

全链路灰度实现

Discovery

https://github.com/Nepxion/Discovery
https://github.com/Nepxion/Discovery/wiki
http://polaris-paas.gitee.io/polaris-sdk/#/
http://nepxion.gitee.io/discovery/#/?id=入门主页

使用

一、父pom引入Discovery

Hoxton.SR12
2.2.8.RELEASE
6.20.0-SNAPSHOT
...com.nepxiondiscovery${discovery.version}pomimport

二、Gateway 引入 — 网关 discovery-plugin-strategy-starter-gateway

pom.xml

com.nepxiondiscovery-plugin-register-center-starter-nacos

com.nepxiondiscovery-plugin-config-center-starter-nacos

com.nepxiondiscovery-plugin-admin-center-starter

com.nepxiondiscovery-plugin-strategy-starter-gateway

Gateway 配置

spring:application:name: demomall-gatewaystrategy:gateway:dynamic:route:enabled: true #开启网关订阅配置中心的动态路由策略,默认为falsecloud:discovery:metadata:group: discovery-group #组名必须配置 

根据实际的灰度发布维度和场景，配置染色方式的元数据

#组名必须要配置，版本、区域、环境和可用区根据具体场景选择其中一种或者几种进行配置
spring.cloud.discovery.metadata.group=discovery-guide-group 
spring.cloud.discovery.metadata.version=1.0
spring.cloud.discovery.metadata.region=dev 
spring.cloud.discovery.metadata.env=env1 
spring.cloud.discovery.metadata.zone=zone1 
spring.cloud.discovery.metadata.active=true

三、微服务 引入 — 网关 discovery-plugin-strategy-starter-service

pom.xml

com.nepxiondiscovery-plugin-register-center-starter-nacos

com.nepxiondiscovery-plugin-config-center-starter-nacos

com.nepxiondiscovery-plugin-admin-center-starter

com.nepxiondiscovery-plugin-strategy-starter-service

配置

spring:application:name: demomall-membercloud:discovery: #discovery配置，设置流量染色的元数据metadata:group: discovery-group #组名必须配置version: 1.0 #指定版本号

再新启一个配置VM

-Dserver.port=8878
-Dspring.cloud.discovery.metadata.version=1.1
-javaagent:/root/skywalking/skywalking-agent/skywalking-agent.jar
-DSW_AGENT_NAME=demomall-member
-DSW_AGENT_COLLECTOR_BACKEND_SERVICES=127.0.0.1:11800

四、Gateway 配置

1. 全链路版本权重灰度发布

在nacos配置中心中增加网关的版本权重灰度发布策略

• Group为discovery-group
• Data Id为demomall-gateway
• 策略内容如下，实现从网关发起的调用全链路1.0版本流量权重 为90%，1.1版本流量权重为10%

>1.0=90;1.1=10

2. 全链路版本条件权重灰度发布 – 指定百分比流量分配

• 灰度路由，即服务a和b 1.1版本被调用到的概率为5%
• 稳定路由，即服务a和b 1.0版本被调用到的概率为95%

{"demomall‐member":"1.1", "demomall‐promotion":"1.1"}{"demomall‐member":"1.0", "demomall‐promotion":"1.0"}

3. 根据前端传递的Header参数动态选择百分比流量分配

{"demomall‐member":"1.1", "tuli ngmall‐promotion":"1.1"}{"demomall‐member":"1.0", "tu lingmall‐promotion":"1.0"}

原理分析

Openfeign 通过 RequestInterceptor

Openfeign RequestInterceptor

public interface RequestInterceptor {void apply(RequestTemplate template);
}

com.nepxion.discovery.plugin.strategy.aop.FeignStrategyInterceptor

public class FeignStrategyInterceptor extends AbstractStrategyInterceptor implements RequestInterceptor {@Autowiredprotected StrategyContextHolder strategyContextHolder;@Value("${spring.application.strategy.feign.core.header.transmission.enabled:true}")protected Boolean feignCoreHeaderTransmissionEnabled;public FeignStrategyInterceptor(String contextRequestHeaders, String businessRequestHeaders) {super(contextRequestHeaders, businessRequestHeaders);}public void apply(RequestTemplate requestTemplate) {this.interceptInputHeader();this.applyInnerHeader(requestTemplate);this.applyOuterHeader(requestTemplate);this.interceptOutputHeader(requestTemplate);}private void applyInnerHeader(RequestTemplate requestTemplate) {requestTemplate.header("n-d-service-group", new String[]{this.pluginAdapter.getGroup()});requestTemplate.header("n-d-service-type", new String[]{this.pluginAdapter.getServiceType()});String serviceAppId = this.pluginAdapter.getServiceAppId();if (StringUtils.isNotEmpty(serviceAppId)) {requestTemplate.header("n-d-service-app-id", new String[]{serviceAppId});}requestTemplate.header("n-d-service-id", new String[]{this.pluginAdapter.getServiceId()});requestTemplate.header("n-d-service-address", new String[]{this.pluginAdapter.getHost() + ":" + this.pluginAdapter.getPort()});String version = this.pluginAdapter.getVersion();if (StringUtils.isNotEmpty(version) && !StringUtils.equals(version, "default")) {requestTemplate.header("n-d-service-version", new String[]{version});}String region = this.pluginAdapter.getRegion();if (StringUtils.isNotEmpty(region) && !StringUtils.equals(region, "default")) {requestTemplate.header("n-d-service-region", new String[]{region});}String environment = this.pluginAdapter.getEnvironment();if (StringUtils.isNotEmpty(environment) && !StringUtils.equals(environment, "default")) {requestTemplate.header("n-d-service-env", new String[]{environment});}String zone = this.pluginAdapter.getZone();if (StringUtils.isNotEmpty(zone) && !StringUtils.equals(zone, "default")) {requestTemplate.header("n-d-service-zone", new String[]{zone});}}private void applyOuterHeader(RequestTemplate requestTemplate) {Enumeration headerNames = this.strategyContextHolder.getHeaderNames();String routeAddressBlacklist;if (headerNames != null) {while(headerNames.hasMoreElements()) {String headerName = (String)headerNames.nextElement();routeAddressBlacklist = this.strategyContextHolder.getHeader(headerName);boolean isHeaderContains = this.isHeaderContainsExcludeInner(headerName.toLowerCase());if (isHeaderContains) {if (this.feignCoreHeaderTransmissionEnabled) {requestTemplate.header(headerName, new String[]{routeAddressBlacklist});} else {boolean isCoreHeaderContains = StrategyUtil.isCoreHeaderContains(headerName);if (!isCoreHeaderContains) {requestTemplate.header(headerName, new String[]{routeAddressBlacklist});}}}}}if (this.feignCoreHeaderTransmissionEnabled) {Map> headers = requestTemplate.headers();if (CollectionUtils.isEmpty((Collection)headers.get("n-d-version"))) {routeAddressBlacklist = this.strategyContextHolder.getRouteVersion();if (StringUtils.isNotEmpty(routeAddressBlacklist)) {requestTemplate.header("n-d-version", new String[]{routeAddressBlacklist});}}if (CollectionUtils.isEmpty((Collection)headers.get("n-d-region"))) {routeAddressBlacklist = this.strategyContextHolder.getRouteRegion();if (StringUtils.isNotEmpty(routeAddressBlacklist)) {requestTemplate.header("n-d-region", new String[]{routeAddressBlacklist});}}if (CollectionUtils.isEmpty((Collection)headers.get("n-d-env"))) {routeAddressBlacklist = this.strategyContextHolder.getRouteEnvironment();if (StringUtils.isNotEmpty(routeAddressBlacklist)) {requestTemplate.header("n-d-env", new String[]{routeAddressBlacklist});}}if (CollectionUtils.isEmpty((Collection)headers.get("n-d-address"))) {routeAddressBlacklist = this.strategyContextHolder.getRouteAddress();if (StringUtils.isNotEmpty(routeAddressBlacklist)) {requestTemplate.header("n-d-address", new String[]{routeAddressBlacklist});}}if (CollectionUtils.isEmpty((Collection)headers.get("n-d-version-weight"))) {routeAddressBlacklist = this.strategyContextHolder.getRouteVersionWeight();if (StringUtils.isNotEmpty(routeAddressBlacklist)) {requestTemplate.header("n-d-version-weight", new String[]{routeAddressBlacklist});}}if (CollectionUtils.isEmpty((Collection)headers.get("n-d-region-weight"))) {routeAddressBlacklist = this.strategyContextHolder.getRouteRegionWeight();if (StringUtils.isNotEmpty(routeAddressBlacklist)) {requestTemplate.header("n-d-region-weight", new String[]{routeAddressBlacklist});}}if (CollectionUtils.isEmpty((Collection)headers.get("n-d-version-prefer"))) {routeAddressBlacklist = this.strategyContextHolder.getRouteVersionPrefer();if (StringUtils.isNotEmpty(routeAddressBlacklist)) {requestTemplate.header("n-d-version-prefer", new String[]{routeAddressBlacklist});}}if (CollectionUtils.isEmpty((Collection)headers.get("n-d-version-failover"))) {routeAddressBlacklist = this.strategyContextHolder.getRouteVersionFailover();if (StringUtils.isNotEmpty(routeAddressBlacklist)) {requestTemplate.header("n-d-version-failover", new String[]{routeAddressBlacklist});}}if (CollectionUtils.isEmpty((Collection)headers.get("n-d-region-transfer"))) {routeAddressBlacklist = this.strategyContextHolder.getRouteRegionTransfer();if (StringUtils.isNotEmpty(routeAddressBlacklist)) {requestTemplate.header("n-d-region-transfer", new String[]{routeAddressBlacklist});}}if (CollectionUtils.isEmpty((Collection)headers.get("n-d-region-failover"))) {routeAddressBlacklist = this.strategyContextHolder.getRouteRegionFailover();if (StringUtils.isNotEmpty(routeAddressBlacklist)) {requestTemplate.header("n-d-region-failover", new String[]{routeAddressBlacklist});}}if (CollectionUtils.isEmpty((Collection)headers.get("n-d-env-failover"))) {routeAddressBlacklist = this.strategyContextHolder.getRouteEnvironmentFailover();if (StringUtils.isNotEmpty(routeAddressBlacklist)) {requestTemplate.header("n-d-env-failover", new String[]{routeAddressBlacklist});}}if (CollectionUtils.isEmpty((Collection)headers.get("n-d-zone-failover"))) {routeAddressBlacklist = this.strategyContextHolder.getRouteZoneFailover();if (StringUtils.isNotEmpty(routeAddressBlacklist)) {requestTemplate.header("n-d-zone-failover", new String[]{routeAddressBlacklist});}}if (CollectionUtils.isEmpty((Collection)headers.get("n-d-address-failover"))) {routeAddressBlacklist = this.strategyContextHolder.getRouteAddressFailover();if (StringUtils.isNotEmpty(routeAddressBlacklist)) {requestTemplate.header("n-d-address-failover", new String[]{routeAddressBlacklist});}}if (CollectionUtils.isEmpty((Collection)headers.get("n-d-id-blacklist"))) {routeAddressBlacklist = this.strategyContextHolder.getRouteIdBlacklist();if (StringUtils.isNotEmpty(routeAddressBlacklist)) {requestTemplate.header("n-d-id-blacklist", new String[]{routeAddressBlacklist});}}if (CollectionUtils.isEmpty((Collection)headers.get("n-d-address-blacklist"))) {routeAddressBlacklist = this.strategyContextHolder.getRouteAddressBlacklist();if (StringUtils.isNotEmpty(routeAddressBlacklist)) {requestTemplate.header("n-d-address-blacklist", new String[]{routeAddressBlacklist});}}}}private void interceptOutputHeader(RequestTemplate requestTemplate) {if (this.interceptDebugEnabled) {System.out.println("-------- Feign Intercept Output Header Information ---------");Map> headers = requestTemplate.headers();Iterator var3 = headers.entrySet().iterator();while(var3.hasNext()) {Entry> entry = (Entry)var3.next();String headerName = (String)entry.getKey();boolean isHeaderContains = this.isHeaderContains(headerName.toLowerCase());if (isHeaderContains) {Collection headerValue = (Collection)entry.getValue();System.out.println(headerName + "=" + headerValue);}}System.out.println("------------------------------------------------------------");}}protected InterceptorType getInterceptorType() {return InterceptorType.FEIGN;}
}

Gateway 通过 GlobalFilter

gateway

public interface GlobalFilter {Mono filter(ServerWebExchange exchange, GatewayFilterChain chain);
}

com.nepxion.discovery.plugin.strategy.gateway.filter.AbstractGatewayStrategyRouteFilter

public abstract class AbstractGatewayStrategyRouteFilter implements GatewayStrategyRouteFilter {@Autowiredprotected PluginAdapter pluginAdapter;@Autowiredprotected StrategyWrapper strategyWrapper;@Autowired(required = false)protected GatewayStrategyMonitor gatewayStrategyMonitor;@Value("${spring.application.strategy.gateway.header.priority:true}")protected Boolean gatewayHeaderPriority;@Value("${spring.application.strategy.gateway.original.header.ignored:true}")protected Boolean gatewayOriginalHeaderIgnored;@Value("${spring.application.strategy.gateway.core.header.transmission.enabled:true}")protected Boolean gatewayCoreHeaderTransmissionEnabled;@Value("${spring.application.strategy.gateway.route.filter.order:9000}")protected Integer filterOrder;public AbstractGatewayStrategyRouteFilter() {}public int getOrder() {return this.filterOrder;}public Mono filter(ServerWebExchange exchange, GatewayFilterChain chain) {GatewayStrategyContext.getCurrentContext().setExchange(exchange);ServerHttpRequest request = exchange.getRequest();Builder requestBuilder = request.mutate();this.applyInnerHeader(request, requestBuilder);this.applyOuterHeader(request, requestBuilder);if (this.gatewayStrategyMonitor != null) {this.gatewayStrategyMonitor.monitor(exchange);}String path = request.getPath().toString();if (path.contains("inspector/inspect")) {GatewayStrategyFilterResolver.setHeader(request, requestBuilder, "endpoint-inspector-inspect", this.pluginAdapter.getPluginInfo((String)null), true);}ServerHttpRequest newRequest = requestBuilder.build();ServerWebExchange newExchange = exchange.mutate().request(newRequest).build();GatewayStrategyContext.getCurrentContext().setExchange(newExchange);return chain.filter(newExchange);}private void applyInnerHeader(ServerHttpRequest request, Builder requestBuilder) {GatewayStrategyFilterResolver.setHeader(request, requestBuilder, "n-d-service-group", this.pluginAdapter.getGroup(), this.gatewayHeaderPriority);GatewayStrategyFilterResolver.setHeader(request, requestBuilder, "n-d-service-type", this.pluginAdapter.getServiceType(), false);String serviceAppId = this.pluginAdapter.getServiceAppId();if (StringUtils.isNotEmpty(serviceAppId)) {GatewayStrategyFilterResolver.setHeader(request, requestBuilder, "n-d-service-app-id", serviceAppId, false);}GatewayStrategyFilterResolver.setHeader(request, requestBuilder, "n-d-service-id", this.pluginAdapter.getServiceId(), false);GatewayStrategyFilterResolver.setHeader(request, requestBuilder, "n-d-service-address", this.pluginAdapter.getHost() + ":" + this.pluginAdapter.getPort(), false);GatewayStrategyFilterResolver.setHeader(request, requestBuilder, "n-d-service-version", this.pluginAdapter.getVersion(), false);GatewayStrategyFilterResolver.setHeader(request, requestBuilder, "n-d-service-region", this.pluginAdapter.getRegion(), false);GatewayStrategyFilterResolver.setHeader(request, requestBuilder, "n-d-service-env", this.pluginAdapter.getEnvironment(), false);GatewayStrategyFilterResolver.setHeader(request, requestBuilder, "n-d-service-zone", this.pluginAdapter.getZone(), false);}private void applyOuterHeader(ServerHttpRequest request, Builder requestBuilder) {String routeEnvironment = this.getRouteEnvironment();if (StringUtils.isNotEmpty(routeEnvironment)) {GatewayStrategyFilterResolver.setHeader(request, requestBuilder, "n-d-env", routeEnvironment, false);}if (this.gatewayCoreHeaderTransmissionEnabled) {Map headerMap = this.strategyWrapper.getHeaderMap();String routeAddress;String routeVersionWeight;if (MapUtils.isNotEmpty(headerMap)) {Iterator var5 = headerMap.entrySet().iterator();while(var5.hasNext()) {Entry entry = (Entry)var5.next();routeAddress = (String)entry.getKey();routeVersionWeight = (String)entry.getValue();GatewayStrategyFilterResolver.setHeader(request, requestBuilder, routeAddress, routeVersionWeight, this.gatewayHeaderPriority);}}String routeVersion = this.getRouteVersion();String routeRegion = this.getRouteRegion();routeAddress = this.getRouteAddress();routeVersionWeight = this.getRouteVersionWeight();String routeRegionWeight = this.getRouteRegionWeight();String routeIdBlacklist = this.getRouteIdBlacklist();String routeAddressBlacklist = this.getRouteAddressBlacklist();if (StringUtils.isNotEmpty(routeVersion)) {GatewayStrategyFilterResolver.setHeader(request, requestBuilder, "n-d-version", routeVersion, this.gatewayHeaderPriority);} else {GatewayStrategyFilterResolver.ignoreHeader(requestBuilder, "n-d-version", this.gatewayHeaderPriority, this.gatewayOriginalHeaderIgnored);}if (StringUtils.isNotEmpty(routeRegion)) {GatewayStrategyFilterResolver.setHeader(request, requestBuilder, "n-d-region", routeRegion, this.gatewayHeaderPriority);} else {GatewayStrategyFilterResolver.ignoreHeader(requestBuilder, "n-d-region", this.gatewayHeaderPriority, this.gatewayOriginalHeaderIgnored);}if (StringUtils.isNotEmpty(routeAddress)) {GatewayStrategyFilterResolver.setHeader(request, requestBuilder, "n-d-address", routeAddress, this.gatewayHeaderPriority);} else {GatewayStrategyFilterResolver.ignoreHeader(requestBuilder, "n-d-address", this.gatewayHeaderPriority, this.gatewayOriginalHeaderIgnored);}if (StringUtils.isNotEmpty(routeVersionWeight)) {GatewayStrategyFilterResolver.setHeader(request, requestBuilder, "n-d-version-weight", routeVersionWeight, this.gatewayHeaderPriority);} else {GatewayStrategyFilterResolver.ignoreHeader(requestBuilder, "n-d-version-weight", this.gatewayHeaderPriority, this.gatewayOriginalHeaderIgnored);}if (StringUtils.isNotEmpty(routeRegionWeight)) {GatewayStrategyFilterResolver.setHeader(request, requestBuilder, "n-d-region-weight", routeRegionWeight, this.gatewayHeaderPriority);} else {GatewayStrategyFilterResolver.ignoreHeader(requestBuilder, "n-d-region-weight", this.gatewayHeaderPriority, this.gatewayOriginalHeaderIgnored);}if (StringUtils.isNotEmpty(routeIdBlacklist)) {GatewayStrategyFilterResolver.setHeader(request, requestBuilder, "n-d-id-blacklist", routeIdBlacklist, this.gatewayHeaderPriority);} else {GatewayStrategyFilterResolver.ignoreHeader(requestBuilder, "n-d-id-blacklist", this.gatewayHeaderPriority, this.gatewayOriginalHeaderIgnored);}if (StringUtils.isNotEmpty(routeAddressBlacklist)) {GatewayStrategyFilterResolver.setHeader(request, requestBuilder, "n-d-address-blacklist", routeAddressBlacklist, this.gatewayHeaderPriority);} else {GatewayStrategyFilterResolver.ignoreHeader(requestBuilder, "n-d-address-blacklist", this.gatewayHeaderPriority, this.gatewayOriginalHeaderIgnored);}} else {GatewayStrategyFilterResolver.ignoreHeader(requestBuilder, "n-d-version");GatewayStrategyFilterResolver.ignoreHeader(requestBuilder, "n-d-region");GatewayStrategyFilterResolver.ignoreHeader(requestBuilder, "n-d-address");GatewayStrategyFilterResolver.ignoreHeader(requestBuilder, "n-d-version-weight");GatewayStrategyFilterResolver.ignoreHeader(requestBuilder, "n-d-region-weight");GatewayStrategyFilterResolver.ignoreHeader(requestBuilder, "n-d-id-blacklist");GatewayStrategyFilterResolver.ignoreHeader(requestBuilder, "n-d-address-blacklist");}}public PluginAdapter getPluginAdapter() {return this.pluginAdapter;}
}

通过IRule

com.netflix.loadbalancer.IRule

public interface IRule{public Server choose(Object key);public void setLoadBalancer(ILoadBalancer lb);public ILoadBalancer getLoadBalancer();    
}

通过 pluginAdapter 可以拿到需要的数据

 //TODO 灰度测试@Autowiredprivate PluginAdapter pluginAdapter;@RequestMapping(value = "/gray/{value}", method = RequestMethod.GET)public String gray(@PathVariable(value = "value") String value) {value = pluginAdapter.getPluginInfo(value);value = couponsFeignService.gray(value);log.info("调用路径：{}", value);return value;}

MSE 微服务治理全链路灰度 太贵

阿里云MSE服务治理产品就是一款基于Java Agent实现的无侵入式企业生产级服务治理产品，您不需要修改任何一行业务代码，即可拥有不限于全链路灰度的治理能力，并且支持近 5年内所有的 Spring Boot、Spring Cloud和Dubbo

https://help.aliyun.com/document_detail/359851.html