Skip to content

Missing mbedtls_rsa_init in _libssh2_mbedtls_rsa_new_private_frommemory causes MBEDTLS_ERR_THREADING_BAD_INPUT_DATA #1821

@P81000

Description

@P81000

Describe the bug

When using the mbedTLS backend, attempting public key authentication with a private key loaded from memory (libssh2_userauth_publickey_frommemory) fails during the signature phase.

The bug originates in src/mbedtls.c inside the _libssh2_mbedtls_rsa_new_private_frommemory function. The function allocates the RSA context using mbedtls_calloc, but it completely misses the mbedtls_rsa_init(*rsa) call.
Because the context is only zero-initialized, the internal threading Mutex is never properly created by the OS. Later, when mbedtls_rsa_pkcs1_sign is called to generate the signature (e.g., via _libssh2_mbedtls_rsa_sha2_sign), mbedTLS attempts to lock the uninitialized Mutex and immediately aborts, returning -0x001C (MBEDTLS_ERR_THREADING_BAD_INPUT_DATA).

To Reproduce

Compile libssh2 with the mbedTLS crypto backend.

Connect to an SSH server using libssh2_userauth_publickey_frommemory.

The key parsing succeeds, but the authentication callback signv returns -1.

Tracing the mbedTLS return code inside _libssh2_mbedtls_rsa_sha2_sign yields -0x001C.

Expected behavior

The RSA context should be properly initialized with its Mutex before the key data is copied into it, mirroring the exact same behavior that is already correctly implemented in the _libssh2_mbedtls_rsa_new_private (from file) function. The signature should be generated without threading panics.

Version (please complete the following information):

OS and version: Linux (Debian/Ubuntu Docker container)

libssh2 version: 1.11.2_DEV / master

crypto backend and version: mbedTLS

Additional context

The fix is trivial. In src/mbedtls.c, inside _libssh2_mbedtls_rsa_new_private_frommemory, the initialization just needs to be added right after the mbedtls_calloc call:

    *rsa = (libssh2_rsa_ctx *) mbedtls_calloc(1, sizeof(libssh2_rsa_ctx));
    if(!*rsa)
        return -1;

    /* MISSING INITIALIZATION: */
   #if MBEDTLS_VERSION_NUMBER >= 0x03000000
        mbedtls_rsa_init(*rsa);
    #else
        mbedtls_rsa_init(*rsa, MBEDTLS_RSA_PKCS_V15, 0);
    #endif

Once this is added, the mbedtls_rsa_copy works safely, the Mutex locks successfully during the signature phase, and the authentication succeeds.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions