kubernetes - Accessing service using istio ingress gives 503 error when mTLS is enabled

Question

I have a mutual TLS enabled Istio mesh. My setup is as follows

1. A service running inside a pod (Service container + envoy)
2. An envoy gateway which stays in front of the above service. An Istio Gateway and Virtual Service attached to this. It routes /info/ route to the above service.
3. Another Istio Gateway configured for ingress using the default istio ingress pod. This also has Gateway+Virtual Service combination. The virtual service directs /info/ path to the service described in 2

I'm attempting to access the service from the ingress gateway using a curl command such as:

$ curl -X GET http://istio-ingressgateway.istio-system:80/info/ -H "Authorization: Bearer $token" -v

But I'm getting a 503 not found error as below:

$ curl -X GET http://istio-ingressgateway.istio-system:80/info/ -H "Authorization: Bearer $token" -v
Note: Unnecessary use of -X or --request, GET is already inferred.
*   Trying 10.105.138.94...
* Connected to istio-ingressgateway.istio-system (10.105.138.94) port 80 (#0)
> GET /info/ HTTP/1.1
> Host: istio-ingressgateway.istio-system
> User-Agent: curl/7.47.0
> Accept: */*
> Authorization: Bearer ...
>
< HTTP/1.1 503 Service Unavailable
< content-length: 57
< content-type: text/plain
< date: Sat, 12 Jan 2019 13:30:13 GMT
< server: envoy
<
* Connection #0 to host istio-ingressgateway.istio-system left intact

I checked the logs of istio-ingressgateway pod and the following line was logged there

[2019-01-13T05:40:16.517Z] "GET /info/ HTTP/1.1" 503 UH 0 19 6 - "10.244.0.5" "curl/7.47.0" "da02fdce-8bb5-90fe-b422-5c74fe28759b" "istio-ingressgateway.istio-system" "-"

If I logged into istio ingress pod and attempt to send the request with curl, I get a successful 200 OK.

# curl hr--gateway-service.default/info/ -H "Authorization: Bearer $token" -v

Also, I managed to get a successful response for the same curl command when the mesh was created in mTLS disabled mode. There are no conflicts shown in mTLS setup.

Here are the config details for my service mesh in case you need additional info.

Pods

$ kubectl get pods --all-namespaces
NAMESPACE       NAME                                                       READY     STATUS      RESTARTS   AGE
default         hr--gateway-deployment-688986c87c-z9nkh                    1/1       Running     0          37m
default         hr--hr-deployment-596946948d-c89bn                         2/2       Running     0          37m
default         hr--sts-deployment-694d7cff97-gjwdk                        1/1       Running     0          37m
ingress-nginx   default-http-backend-6586bc58b6-8qss6                      1/1       Running     0          42m
ingress-nginx   nginx-ingress-controller-6bd7c597cb-t4rwq                  1/1       Running     0          42m
istio-system    grafana-85dbf49c94-lfpbr                                   1/1       Running     0          42m
istio-system    istio-citadel-545f49c58b-dq5lq                             1/1       Running     0          42m
istio-system    istio-cleanup-secrets-bh5ws                                0/1       Completed   0          42m
istio-system    istio-egressgateway-7d59954f4-qcnxm                        1/1       Running     0          42m
istio-system    istio-galley-5b6449c48f-72vkb                              1/1       Running     0          42m
istio-system    istio-grafana-post-install-lwmsf                           0/1       Completed   0          42m
istio-system    istio-ingressgateway-8455c8c6f7-5khtk                      1/1       Running     0          42m
istio-system    istio-pilot-58ff4d6647-bct4b                               2/2       Running     0          42m
istio-system    istio-policy-59685fd869-h7v94                              2/2       Running     0          42m
istio-system    istio-security-post-install-cqj6k                          0/1       Completed   0          42m
istio-system    istio-sidecar-injector-75b9866679-qg88s                    1/1       Running     0          42m
istio-system    istio-statsd-prom-bridge-549d687fd9-bspj2                  1/1       Running     0          42m
istio-system    istio-telemetry-6ccf9ddb96-hxnwv                           2/2       Running     0          42m
istio-system    istio-tracing-7596597bd7-m5pk8                             1/1       Running     0          42m
istio-system    prometheus-6ffc56584f-4cm5v                                1/1       Running     0          42m
istio-system    servicegraph-5d64b457b4-jttl9                              1/1       Running     0          42m
kube-system     coredns-78fcdf6894-rxw57                                   1/1       Running     0          50m
kube-system     coredns-78fcdf6894-s4bg2                                   1/1       Running     0          50m
kube-system     etcd-ubuntu                                                1/1       Running     0          49m
kube-system     kube-apiserver-ubuntu                                      1/1       Running     0          49m
kube-system     kube-controller-manager-ubuntu                             1/1       Running     0          49m
kube-system     kube-flannel-ds-9nvf9                                      1/1       Running     0          49m
kube-system     kube-proxy-r868m                                           1/1       Running     0          50m
kube-system     kube-scheduler-ubuntu                                      1/1       Running     0          49m

Services

$ kubectl get svc --all-namespaces
NAMESPACE       NAME                                    TYPE           CLUSTER-IP       EXTERNAL-IP    PORT(S)                                                                                                                   AGE

default         hr--gateway-service                     ClusterIP      10.100.238.144   <none>         80/TCP,443/TCP                                                                                                            39m
default         hr--hr-service                          ClusterIP      10.96.193.43     <none>         80/TCP                                                                                                                    39m
default         hr--sts-service                         ClusterIP      10.99.54.137     <none>         8080/TCP,8081/TCP,8090/TCP                                                                                                39m
default         kubernetes                              ClusterIP      10.96.0.1        <none>         443/TCP                                                                                                                   52m
ingress-nginx   default-http-backend                    ClusterIP      10.109.166.229   <none>         80/TCP                                                                                                                    44m
ingress-nginx   ingress-nginx                           NodePort       10.108.9.180     192.168.60.3   80:31001/TCP,443:32315/TCP                                                                                                44m
istio-system    grafana                                 ClusterIP      10.102.141.231   <none>         3000/TCP                                                                                                                  44m
istio-system    istio-citadel                           ClusterIP      10.101.128.187   <none>         8060/TCP,9093/TCP                                                                                                         44m
istio-system    istio-egressgateway                     ClusterIP      10.102.157.204   <none>         80/TCP,443/TCP                                                                                                            44m
istio-system    istio-galley                            ClusterIP      10.96.31.251     <none>         443/TCP,9093/TCP                                                                                                          44m
istio-system    istio-ingressgateway                    LoadBalancer   10.105.138.94    <pending>      80:31380/TCP,443:31390/TCP,31400:31400/TCP,15011:31219/TCP,8060:31482/TCP,853:30034/TCP,15030:31544/TCP,15031:32652/TCP   44m
istio-system    istio-pilot                             ClusterIP      10.100.170.73    <none>         15010/TCP,15011/TCP,8080/TCP,9093/TCP                                                                                     44m
istio-system    istio-policy                            ClusterIP      10.104.77.184    <none>         9091/TCP,15004/TCP,9093/TCP                                                                                               44m
istio-system    istio-sidecar-injector                  ClusterIP      10.100.180.152   <none>         443/TCP                                                                                                                   44m
istio-system    istio-statsd-prom-bridge                ClusterIP      10.107.39.50     <none>         9102/TCP,9125/UDP                                                                                                         44m
istio-system    istio-telemetry                         ClusterIP      10.110.55.232    <none>         9091/TCP,15004/TCP,9093/TCP,42422/TCP                                                                                     44m
istio-system    jaeger-agent                            ClusterIP      None             <none>         5775/UDP,6831/UDP,6832/UDP                                                                                                44m
istio-system    jaeger-collector                        ClusterIP      10.102.43.21     <none>         14267/TCP,14268/TCP                                                                                                       44m
istio-system    jaeger-query                            ClusterIP      10.104.182.189   <none>         16686/TCP                                                                                                                 44m
istio-system    prometheus                              ClusterIP      10.100.0.70      <none>         9090/TCP                                                                                                                  44m
istio-system    servicegraph                            ClusterIP      10.97.65.37      <none>         8088/TCP                                                                                                                  44m
istio-system    tracing                                 ClusterIP      10.109.87.118    <none>         80/TCP                                                                                                                    44m
kube-system     kube-dns                                ClusterIP      10.96.0.10       <none>         53/UDP,53/TCP                                                                                                             52m

Gateway and virtual service described in point 2

$ kubectl describe gateways.networking.istio.io hr--gateway
Name:         hr--gateway
Namespace:    default
API Version:  networking.isti

Answer 1

The problem is probably as follows: istio-ingressgateway initiates mTLS to hr--gateway-service on port 80, but hr--gateway-service expects plain HTTP connections.

There are multiple solutions:

1. Define a DestinationRule to instruct clients to disable mTLS on calls to hr--gateway-service

   apiVersion: networking.istio.io/v1alpha3
   kind: DestinationRule
   metadata:
     name: hr--gateway-service-disable-mtls
   spec:
     host: hr--gateway-service.default.svc.cluster.local
     trafficPolicy:
       tls:
         mode: DISABLE

2. Instruct hr-gateway-service to accept mTLS connections. For that, configure the server TLS options on port 80 to be MUTUAL and to use Istio certificates and the private key. Specify serverCertificate, caCertificates and privateKey to be /etc/certs/cert-chain.pem, /etc/certs/root-cert.pem, /etc/certs/key.pem, respectively.