php - Royal Mail Shipping API - SOAP connection & pem/certificates query

Question

I am trying to setup the Royal Mail Shipping API (if anyone has any experience of this i'd be grateful if you could assist).

In the documentation they provide I need to download a certificate (a .p12 file) & import this onto my Windows machine - this is pretty straightforward using the 'Certificate Import Wizard'. Once it gets to the "Set Security Level' I must select High & this will request permission with a password each time this is used.

In Internet Explorer in 'Internet Options' within the Content tab I can view the Certificates and can clearly see that this certificate has been imported and hasn't expired.

The next step is to extract the certificate components, here I have to run the three following commands using OpenSSL to generate the .pem files.

$ openssl pkcs12 -in mycert.p12 -cacerts -nokeys -out cacert.pem
$ openssl pkcs12 -in mycert.p12 -clcerts -nokeys -out mycert.pem
$ openssl pkcs12 -in mycert.p12 -nocerts -nodes -out mykey.pem 

The documentation states the cacert.pem file can be directly referenced by an application using the file itself, which I believe I have done within my PHP script however it isn't clear where I should put the other mycert & mykey pem files.

The documentation states the following in regards to this :-

How an application passes the issued client SSL certificate when establishing an SSL network connection to is application and environment dependent but it would essentially need to access both the "mycert.pem" and "mykey.pem" file, or in some cases, a single combined file containing both cert and key.

So nowhere does it say how these two files are used by the application, at the moment I've just left them in the same directory as the cacert.pem file.

If I try to access the url https://api.royalmail.com/shipping/onboarding directly from the browser, it asks for me to select a certificate, I select this & then enter the correct password when it asks to 'Grant or deny permission to use this key'. Once I enter the correct password the following page appears - can anyone confirm if this would mean the issue is at my end OR something that Royal Mail hasn't configured correctly at their end.

In addition to this, the actual PHP script that I have that is used to send SOAP requests to the Shipping API isn't working (probably related to everything above).

Within my PHP script the soapclient options are set up as follows:

$soapclient_options['cache_wsdl'] = 'WSDL_CACHE_NONE'; 
$soapclient_options['local_cert'] = 'certs/cacert.pem';
$soapclient_options['passphrase'] = $api_certificate_passphrase;
$soapclient_options['trace'] = true;
$soapclient_options['ssl_method'] = 'SOAP_SSL_METHOD_SSLv3';
$soapclient_options['location'] = 'https://api.royalmail.com/shipping/onboarding';

$client = new SoapClient('SAPI/ShippingAPI_V2_0_8.wsdl', $soapclient_options);
$client->__setLocation($soapclient_options['location']);

When I run the PHP script (this is basically the same code that Royal Mail provide themselves with my own personal API login details) I get the following message in the browser:

Could not connect to host 
REQUEST: email@yoursite.co.ukAPI rngfJ+4dt4Gt855a5pr6u38i3B4= ODcwMTE5Nzc3 2015-10-13T11:02:20Z 2015-10-13T11:02:201.00526348001DeliveryDSD12015-10-13bobSS23, Some AvenueLondonE10g1000000

Obviously this cannot connect to the host for some unknown reason, the latter is simply the request that was sent. The rest of the PHP script is exactly the same as the Royal Mail's script they sent to me & have confirmed is used by others and working fine.

I am working in a WAMP environment although the eventual code will be in a Linux environment. Can anyone help I am really getting baffled & Royal Mail themselves haven't been able to provide any solid technical support yet.

UPDATE

This is the full error message displayed in the browser (i've changed the email address for security purposes)

Invalid Request REQUEST: myemail@company.co.ukAPI dgCW98Vqw3ladYgPPpNialODhvI= MTMzMjE1NjM4 2015-10-13T13:25:30Z 2015-10-13T13:25:302.00526348001DeliveryDSD12015-10-13Jon DoeSS23, Some RoadLondonE10g1000000

I've merged the two pem files into a single file called 'bundle.pem' & referenced this in the 'local_cert' variable for the SoapClient & BINGO this is now connecting. This now longer shows the Could not connect but states an 'Invalid Request' instead, so at least now this is connecting and giving me a different error.

My entire PHP script is below:

<?php

ini_set('default_socket_timeout', 120);
ini_set('soap.wsdl_cache_enabled',1);
ini_set('soap.wsdl_cache_ttl',1);

$api_password = "xxxxxxxxxxxxxx!";
$api_username = "xxxxxxxxx@xxxxxxxxx.co.ukAPI";
$api_application_id = "xxxxxxxxxxxx";
$api_service_type = "D";
$api_service_code = "SD1";
$api_service_format = "";
$api_certificate_passphrase = 'xxxxxxxxxx';
$api_service_enhancements = "";

$data = new ArrayObject();
$data->order_tracking_id = "";
$data->shipping_name = "Jon Doe";
$data->shipping_company = "SS";
$data->shipping_address1 = "23, Some Road"; 
$data->shipping_address2 = "";
$data->shipping_town = "London";
$data->shipping_postcode = "E1";
$data->order_tracking_boxes = "0";
$data->order_tracking_weight = "1000";    

$time = gmdate('Y-m-dTH:i:s');
$created = gmdate('Y-m-dTH:i:s');
$nonce = mt_rand();
$nonce_date_pwd = pack("A*",$nonce) . pack("A*",$created) . pack("H*", sha1($api_password));
$passwordDigest = base64_encode(pack('H*',sha1($nonce_date_pwd)));
$ENCODEDNONCE = base64_encode($nonce);


$soapclient_options = array(); 
$soapclient_options['cache_wsdl'] = 'WSDL_CACHE_NONE'; 
$soapclient_options['local_cert'] = 'royalmail/cert/bundle.pem';
$soapclient_options['passphrase'] = $api_certificate_passphrase;
$soapclient_options['trace'] = true;
$soapclient_options['ssl_method'] = 'SOAP_SSL_METHOD_SSLv3';
$soapclient_options['exceptions'] = true;
$soapclient_options['location'] = 'https://api.royalmail.com/shipping/onboarding';

//launch soap client
$client = new SoapClient('royalmail/ShippingAPI_V2_0_8.wsdl', $soapclient_options);
$client->__setLocation($soapclient_options['location']);

//headers needed for royal mail
$HeaderObjectXML  = '<wsse:Security xmlns:wsse="http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-secext-1.0.xsd"
                      xmlns:wsu="http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-utility-1.0.xsd">
           <wsse:UsernameToken wsu:Id="UsernameToken-000">
              <wsse:Username>'.$api_username.'</wsse:Username>
              <wsse:Password Type="http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-username-token-profile-1.0#PasswordDigest">'.$passwordDigest.'</wsse:Password>
              <wsse:Nonce EncodingType="http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-soap-message-security-1.0#Base64Binary">'.$ENCODEDNONCE.'</wsse:Nonce>
              <wsu:Created>'.$created.'</wsu:Created>
           </wsse:UsernameToken>
       </wsse:Security>';

//push the header into soap
$HeaderObject = new SoapVar( $HeaderObjectXML, XSD_ANYXML );

//push soap header
$header = new SoapHeader( 'http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-utility-1.0.xsd', 'Security', $HeaderObject );
$client->__setSoapHeaders($header);

//build the request
$request = array(
    'integrationHeader' => array(
        'dateTime' => $time,
        'version' => '1.0',
        'identification' => array(
            'applicationId' => $api_application_id,
            'transactionId' => $data->order_tracking_id
        )
    ),
    'requestedShipment' => array(
                                'shipmentType' => array('code' => 'Delivery'),
                                'serviceOccurence' => '1',
                                'serviceType' => array('code' => $api_service_type),
                                'serviceOffering' => array('serviceOfferingCode' => array('code' => $api_service_code)),
                                'serviceFormat' => array('serviceFormatCode' => array('code' => $api_service_format)),
                                'shippingDate' => date('Y-m-d'),
                                'recipientContact' => array('name' => $data->shipping_name, 'complementaryName' => $data->shipping_company),
                                'recipientAddress' => array('addressLine1' => $data->shipping_address1,  'addressLine2' => $data->shipping_address2, 'postTown' => $data->shipping_town, 'postcode' => $data->shipping_postcode),
                                'items' => array('item' => array(
                                            'numberOfItems' => $data->order_tracking_boxes,
                                            'weight' => array( 'unitOfMeasure' => array('unitOfMeasureCode' => array('code' => 'g')), 'value' => ($data->order_tracking_weight*1000) //weight of each individual item
                                                             )
                                                                )
                                                )
                                )               
);


//if any enhancements, add it into the array
if($api_service_enhancements != "") {
    $request['requestedShipment']['serviceEnhancements'] = array('enhancementType' => array('serviceEnhancementCode' => array('code' => $api_service_enhancements)));
}

//try make the call
try { 
    $response = $client->__soapCall( 'createShipment', array($request), array('soapaction' => 'https://api.royalmail.com/shipping/onboarding') );
} catch (Exception $e) {
    //catch the error message and echo the last request for debug
    echo $e->getMessage(); 
    echo " REQUEST:
" . $client->__getLastRequest() . "
";
    die;
}

//check for any errors
if(isset($response->integrationFooter->errors)) { 
    $build = "";

    //check it wasn't a single error message
    if(isset($response->integrationFooter->errors->error->errorCode)) { 
        $build .= $output_error->errorCode.": ".$output_error->errorDescription."<br/>"; 
    } else {
        //loop out each error message, throw exception will be added ehre
        foreach($response->integrationFooter->errors->error as $output_error) { 
            $build .= $output_error->errorCode.": ".$output_error->errorDescription."<br/>";
        }
    }

    echo $build; die;

}

print_r($response);

echo "REQUEST:
" . $client->__getLastRequest() . "
";
die;            
?>

Just for additional clarity I've added a dump of the $request variable just before it reaches the try/catch block (note this is kinda long).

Array
(
    [integrationHeader] => Array
        (
        [dateTime] => 2015-10-13T13:34:44
        [version] => 1.0
        [identification] => Array
            (
                [applicationId] => 05263

Answer 1

^{! Although you've mentioned you've done most of these steps, I'll detail them anyway to complete the instructions}

Installing the CA to the server

• Upload the p12 file to the server (in /root/Desktop for example)
• Run the following 3 commands to install

openssl pkcs12 -in mycert.p12 -cacerts -nokeys -out cacert.pem
openssl pkcs12 -in mycert.p12 -clcerts -nokeys -out mycert.pem
openssl pkcs12 -in mycert.p12 -nocerts -nodes -out mykey.pem 

• Now copy the *.pem files into /etc/ssl/certs
  • I like to create a subdirectory in here (mkdir royalmail)
  • And move the *.pem files (mv *.pem /etc/ssl/certs/certificates)
• Now create a new file and copy the contents of mycert.pem and mykey.pem into it (only from -----BEGIN .... ------- to EOF)

You should have the following files;

Using SOAPClient

Now that the certificates are installed, we can now test the connection (assuming your certificates are in /etc/pki/tls/certs/certificates/royalmail/shippingv2)

If you run

wget https://api.royalmail.com/shipping/onboarding --private-key=/etc/ssl/certs/certificates/royalmail/shippingv2/rm_bundle.pem --private-key-type=PEM

You should be able to connect to port 443 (there may be a handshake failure in OpenSSL though - at least with my test just now on our staging environment).

We can now instantiate the SoapClient using a local copy of the WSDL and specifying the local certificate in the $options parameter.

$objSoapClient = new SoapClient('lib/wsdl/royalmail/shipping/ShippingAPI_V2_0_8.wsdl', array(
    'soap_version' => SOAP_1_1,
    'trace' => 1,
    'uri' => 'http://www.royalmailgroup.com/api/ship/V2',
    'location' => 'https://api.royalmail.com/shipping/onboarding',
    'local_cert' => '/etc/ssl/certs/certificates/royalmail/shippingv2/rm_bundle.pem',
    'passphrase' => '', //Your passphrase when doing step 1
    'ssl_method' => 'SOAP_SSL_METHOD_TLS',
    'exceptions' => 1,
    'trace' => 1
));

Authentication

In your documentation, you should find a file called rm_password_digest.php or something similar, which details how to create the authentication headers.

/* The value below should be changed to your password.  If you store the password  */  
/* as hashed in your database, you will need to change the code below to remove hashing */

$password = 'just_my_royalmail_api_password';

/* CREATIONDATE - The timestamp. The computer must be on correct time or the server you are
* connecting may reject the password digest for security.
*/
$CREATIONDATE = gmdate('Y-m-dTH:i:s');

/* NONCE - A random word. The use of rand() may repeat the word if the server is
* very loaded.
*/
$nonce = mt_rand();

/* PASSWORDDIGEST This is the way to create the password digest. As per OASIS standard
*  digest = base64_encode(Sha1(nonce + creationdate + password)
*  however note that we use a SHA1(password) instead of the password above
*/
$nonce_date_pwd = pack("A*",$nonce) . pack("A*",$CREATIONDATE) . pack("H*", sha1($password));
$PASSWORDDIGEST = base64_encode(
pack('H*', sha1($nonce_date_pwd)));

/* ENCODEDNONCE - Now encode the nonce for security header */

$ENCODEDNONCE = base64_encode($nonce);

/* Now Print all the values - so we can use it for testing with tools like soapui */

print "WS Security Header elements 
";
print "--------------------------- 
";
print 'Nonce = ' . $nonce;
print "
";
print 'PASSWORDDIGEST= ' . $PASSWORDDIGEST;
print "
";
print 'ENCODEDNONCE= ' . $ENCODEDNONCE;
print "
";
print "CREATIONDATE= " . $CREATIONDATE;

This will help you build the following in the SOAPHeader

<wsse:Security xmlns:wsse="http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-secext-1.0.xsd" xmlns:wsu="http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-utility-1.0.xsd">
    <wsse:UsernameToken wsu:Id="UsernameToken-0000">
        <wsse:Username>[...]</wsse:Username>
        <wsse:Password Type="http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-username-token-profile-1.0#PasswordDigest">[...]</wsse:Password>
        <wsse:Nonce EncodingType="http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-soap-message-security-1.0#Base64Binary">[...]</wsse:Nonce>
        <wsu:Created>[...]</wsu:Created>
    </wsse:UsernameToken>
</wsse:Security>