linux - how to make mkpasswd and python hashlib.pbkdf2 produce the same result (i.e. derived key)

Question

I've discovered that mkpasswd and openssl passwd (on the one hand) do not produce the same "derived key" as python hashlib.pbkdf2_hmac() and mbedtls.pbkdf2 (on the other hand), even though both claim to implement the same standard (PBKDF2).

I am trying to compute Linux password hashes on an embedded system. I use the mbedtls library, which has the function: mbedtls_pkcs5_pbkdf2_hmac(). With the same parameters: SHA256, 5000 rounds, 'TestPasswort' and 'SALTsaltSALTsalt' I can reproduce the result with pythons hashlib:

$ python3
>>> import hashlib
>>> dk = hashlib.pbkdf2_hmac('sha256', b'TestPasswort' , b'SALTsaltSALTsalt', 5000)
>>> dk.hex()
'c27f613f2c4515136f5741a6b80f2fd05c4e3070ab03cb588be842afff6263fe'

As a second step, I run the result (32 bytes) through a modified base64 encoder (see below, for ref). The final result is: kbxVDml33FBjJo4ai.wjo3lCA50f.whMWyV0fzxWMzs

On the other hand, mkpasswd and openssl produce a different result (but consistent among these):

mkpasswd -m sha-256 -R 5000 --salt=SALTsaltSALTsalt TestPasswort
$5$rounds=5000$SALTsaltSALTsalt$NEAgjfoV3kysMtvGP0sS8D0LU9hZlXRnx/g4zn7QPb/

openssl passwd -5  --salt=SALTsaltSALTsalt TestPasswort
$5$SALTsaltSALTsalt$NEAgjfoV3kysMtvGP0sS8D0LU9hZlXRnx/g4zn7QPb/

However:

kbxVDml33FBjJo4ai.wjo3lCA50f.whMWyV0fzxWMzs         

and:

NEAgjfoV3kysMtvGP0sS8D0LU9hZlXRnx/g4zn7QPb/

are clearly different.

Am I making a simple mistake here? How can I make python hashlib.pbkdf2 and openssl passwd produce the same result, or at least map between the two?

------------for ref, my modified base64 encoder:-------------------

#!/bin/python3
ascii_lowercase = 'abcdefghijklmnopqrstuvwxyz'
ascii_uppercase = 'ABCDEFGHIJKLMNOPQRSTUVWXYZ'
digits = '0123456789'
##note the difference to base64 here:
base64_alphabet = './' + digits + ascii_uppercase + ascii_lowercase     

to_encode = input("base64 encoder ->  ")

chunks_8bit = ''
for bits in to_encode   :
    y = int(bits,16)
    chunks_8bit = chunks_8bit+format(y,'04b')

chunks_6bit = [chunks_8bit[bits:bits+6] for bits in range(0,len(chunks_8bit),6)]
padding_amount = ( 6 - len(chunks_6bit[len(chunks_6bit)-1]) )
chunks_6bit[len(chunks_6bit)-1] += padding_amount * '0'

encoded = ''.join([base64_alphabet[int(bits,2)] for bits in chunks_6bit])
print('Base64 encoded version of {to_encode} is: {result}'.format(to_encode = to_encode,    result = encoded))`

question from:https://stackoverflow.com/questions/65598224/how-to-make-mkpasswd-and-python-hashlib-pbkdf2-produce-the-same-result-i-e-der

Answer 1

I found the answer myself: openssl passwd and mkpasswd do not implement PBKDF2, but: sha-crypt; see e.g. here: https://code.woboq.org/userspace/glibc/crypt/sha256-crypt.c.html and here: https://www.akkadia.org/drepper/SHA-crypt.txt

The basic algo is the same, i.e. repeated application of a hash function as e.g. sha-256. Therefore I was mislead to believe that sha256-crypt would implement PBKDF2 with sha256 as hash function.