active directory - On-prem SF cluster Kerberos security (gMSA) and ServerCertificate

Question

We have created clusters that are un-secure and certificate based with success. We are trialing a Domain secured cluster by making the node-to-node communication use a gMSA. The below cluster configuration snippet shows the problematic portion:

"security": {
    "ClusterCredentialType": "Windows",
    "ServerCredentialType": "Windows",
    "WindowsIdentities": {
        "ClustergMSAIdentity": "{{ env_domain }}\{{ cluster_gmsa_identity }}",
        "ClusterSPN": "{{ cluster_gmsa_spn }}",
        "ClientIdentities": [
            {
                "Identity": "{{ env_domain_short }}\ServiceFabricAdmins",
                "IsAdmin": true
            },
            {
                "Identity": "{{ env_domain_short }}\ServiceFabricReadOnly",
                "IsAdmin": false
            }
        ]
    },
    "CertificateInformation": {
        "ServerCertificate": {
          "Thumbprint": "{{ primary_server_certificate_thumbprint }}",
          "X509StoreName": "My"
        },
        "ReverseProxyCertificate": {
            "Thumbprint": "{{ primary_server_certificate_thumbprint }}",
            "X509StoreName": "My"
        }
    }            
}

If we supply the ServerCertificate property as shown above the cluster creation process throws many exceptions (non of which seem point to certificate config issues), if I remove the ServerCertificate section (but keep the reverse proxy supplied cert) the cluster creation process is a success.

I want the ServerCertificate there to secure the http channel of communication for the management endpoints. A few points to consider:

1. The certificate referenced in the ServerCertificate property was used with success for our Certificate secured Cluster.
2. The gMSA has ACL read permissions for the private key in the certificate store.
3. The OS the Nodes are running on is Windows 2016 1709 (build 16299.334)

In order to get the cluster up and running correctly though I had to place the gMSA account in the local Administrators group (which seems wrong!!) as mentioned here.

Any thoughts would be greatly appreciated?

Answer 1

gmsa and Administrators group

you can try to give some permissions such as

1. Computer Configuration Windows Settings Security Settings Local Policies User Rights Assignment Log on as a service

2. Computer Configuration Windows Settings Security Settings Local Policies User Rights Assignment Log on as a batch job

through ther local policy editor. If you look at these groups you'll see the Administrators group already included to these local groups, and Users group(as gmsa is a usual account) is not. So if you include gmsa into these local groups, that account will get permissions to run as a service (service fabric looks like a service itself)

config for certificate (login to portal) + gmsa (server)

{
    "name": "yosfcl",
    "clusterConfigurationVersion": "1.0.1",
    "apiVersion": "10-2017",
    "nodes": [
        {
            "NodeName": "yv1-sf",
            "NodeTypeRef": "NodeType0",
            "IPAddress": "yv1-sf",
            "FaultDomain": "fd:/dc1/r1",
            "UpgradeDomain": "UD1"
        },
        {
            "NodeName": "yv2-sf",
            "NodeTypeRef": "NodeType0",
            "IPAddress": "yv2-sf",
            "FaultDomain": "fd:/dc1/r2",
            "UpgradeDomain": "UD2"
        },
        {
            "NodeName": "yv3-sf",
            "NodeTypeRef": "NodeType0",
            "IPAddress": "yv3-sf",
            "FaultDomain": "fd:/dc1/r3",
            "UpgradeDomain": "UD3"
        }
    ],
    "properties": {
        "diagnosticsStore": 
        {
            "metadata":  "Please replace the diagnostics file share with an actual file share accessible from all cluster machines. For example, \\machine1\DiagnosticsStore.",
            "dataDeletionAgeInDays": "21",
            "storeType": "FileShare",
            "connectionstring": "c:\ProgramData\SF\DiagnosticsStore"
        },      "reverseProxyCertificate": {
            "thumbprint": "[parameters('76************************8A2')]",
            "x509StoreName": "[parameters('My')]"
        },
        "security": {
            "ClusterCredentialType": "Windows",
            "ServerCredentialType": "X509",
            "WindowsIdentities": {
                "ClustergMSAIdentity": "gmsaSF@domain.lan",
                "ClusterSPN": "http/yosfcl.domain.lan",
                "ClientIdentities": [
                    {
                        "Identity": "domain\my.name",
                        "IsAdmin": true
                    }
                ]
            },
            "CertificateInformation": {
                "ServerCertificate": {
                    "Thumbprint": "76***********************************8A2",
                    "X509StoreName": "My"
                },
                "ReverseProxyCertificate": {
                    "Thumbprint": "76*************************************48A2",
                    "X509StoreName": "My"
                },
                "ClientCertificateThumbprints": [
                    {
                        "CertificateThumbprint": "94***********************************2D",
                        "IsAdmin": true
                    }
                ]
            }
        },
        "nodeTypes": [
            {
                "name": "NodeType0",
                "clientConnectionEndpointPort": "19000",
                "clusterConnectionEndpointPort": "19001",
                "leaseDriverEndpointPort": "19002",
                "serviceConnectionEndpointPort": "19003",
                "httpGatewayEndpointPort": "19080",
                "reverseProxyEndpointPort": "19081",
                "applicationPorts": {
                    "startPort": "20001",
                    "endPort": "20500"
                },
                "ephemeralPorts": {
                    "startPort": "20501",
                    "endPort": "20700"
                },
                "isPrimary": true
            }
        ],

        "fabricSettings": [
            {
                "name": "Setup",
                "parameters": [
                    {
                        "name": "FabricDataRoot",
                        "value": "D:\SF"
                    },
                    {
                        "name": "FabricLogRoot",
                        "value": "D:\SF\Logs"
                    }
                ]
            },          {
                "name": "ApplicationGateway/Http",
                "parameters": [
                    {
                        "name": "SecureOnlyMode",
                        "value": true
                    },
                    {
                        "name": "ApplicationCertificateValidationPolicy",
                        "value": "None"
                    }
                ]
            }
        ]
    } }