TLS Signature Algorithms: How Certificate Signatures Are Negotiated

TLS signature algorithms are negotiated from the client’s advertised capabilities. The server must use a signature scheme the client supports and that is compatible with the server’s certificate key; the certificate chain’s issuer signatures must also be acceptable. In TLS 1.3, these are distinct concerns: signature_algorithms applies to handshake signatures such as CertificateVerify, while the optional signature_algorithms_cert describes acceptable signatures on certificates in the chain.

This distinction explains a common surprise: an RSA certificate can be usable for TLS 1.3 even though the RSA signature rules differ from older TLS. In TLS 1.3, RSA CertificateVerify signatures must use RSA-PSS, and SHA-1 is prohibited for that message. An RSA public key alone does not tell you which algorithm signed the certificate or which scheme the server uses in the handshake.

1. What is being negotiated?

A TLS connection involves multiple cryptographic choices. Cipher suites govern negotiated symmetric protection and, depending on protocol version, key-exchange details. Signature schemes govern authentication signatures. Do not read a signature-algorithm list as a cipher-suite list: they serve different purposes and appear in different parts of the handshake.

The client advertises signature schemes it can verify. The server looks for a compatible authentication path: a certificate chain it can present and a handshake signature it can create with its private key. Compatibility depends on the TLS version, the advertised schemes, certificate key type, certificate-chain signatures, server configuration, and implementation support.

There are two signatures to keep mentally separate:

• Certificate signatures: each certificate in a chain is signed by its issuer. These signatures establish the chain’s cryptographic links.
• Handshake signature: in TLS 1.3, the server proves possession of its private key with CertificateVerify, signing the handshake transcript.

A certificate can contain an RSA public key while having been signed by an issuer using ECDSA. Conversely, a chain may contain RSA-signed certificates while the endpoint uses an ECDSA key. The key type and the certificate signature algorithm are separate properties.

2. TLS 1.2 negotiation

In TLS 1.2, the client’s signature_algorithms extension is a preference-ordered list of hash/signature pairs it is willing to verify. RFC 5246 says the client uses this extension to indicate which pairs may be used in digital signatures. The server must find choices that work with the authentication and key-exchange path it intends to use. See [RFC 5246, section 7.4.1.4.1](https://www.rfc-editor.org/rfc/rfc5246#section-7.4.1.4.1).

If the extension is absent, TLS 1.2 specifies legacy defaults tied to the negotiated key-exchange family, generally SHA-1 with RSA, DSA, or ECDSA as applicable. Modern clients usually send the extension, but old peers and unusual configurations make the fallback relevant when diagnosing interoperability.

TLS 1.2’s list is a compatibility filter, not a guarantee that every listed scheme will be selected. The server’s certificate, key-exchange choice, signature implementation, and local policy constrain what it can actually use. A client that offers only schemes incompatible with the server’s credentials can prevent a handshake even if both sides otherwise support TLS 1.2.

3. TLS 1.3: two preference extensions

TLS 1.3 uses named SignatureScheme values, including ECDSA with SHA-256, RSA-PSS with SHA-256, and Ed25519. The key extension for handshake signatures remains signature_algorithms. The optional signature_algorithms_cert lets the client separately express which algorithms it accepts for signatures on certificates in the chain. RFC 8446 defines the distinction in [section 4.2.3](https://www.rfc-editor.org/rfc/rfc8446#section-4.2.3).

Question	Extension or message	What it controls
Which scheme can sign the handshake proof?	signature_algorithms	Permitted schemes for messages such as TLS 1.3 CertificateVerify.
Which issuer signatures are acceptable?	signature_algorithms_cert, when sent	Acceptability of signatures on certificates in the chain.
Which scheme did the server actually use?	CertificateVerify	The message carries a SignatureScheme and the signature bytes.

The server chooses a scheme from the client’s handshake-signature list when it can produce a valid chain. The CertificateVerify scheme must be compatible with the end-entity certificate key. The receiver verifies the signature using the public key in that certificate; failed verification terminates the handshake with decrypt_error. See [RFC 8446, section 4.4.3](https://www.rfc-editor.org/rfc/rfc8446#section-4.4.3).

When signature_algorithms_cert is absent, do not assume the client has no certificate-chain constraints. Apply the protocol’s specified behavior and the implementation’s policy; inspect the actual ClientHello and library documentation when a chain-signature mismatch is suspected. Exact support and diagnostics vary by TLS implementation and version.

4. Why an RSA certificate can fail in TLS 1.3

“RSA certificate” describes the public-key family, but it does not specify the handshake signature scheme. TLS 1.3 requires RSA signatures in CertificateVerify to use RSASSA-PSS. RSA-PKCS#1 v1.5 schemes may appear in the client’s signature_algorithms list for other protocol uses, but their presence does not make them valid for TLS 1.3 CertificateVerify. SHA-1 must not be used in any CertificateVerify signature. These rules are in [RFC 8446, section 4.4.3](https://www.rfc-editor.org/rfc/rfc8446#section-4.4.3).

So an RSA endpoint key is not automatically incompatible with TLS 1.3. It can work when the client advertises a compatible RSA-PSS scheme, the server implementation can produce that signature, and the certificate and chain satisfy the peer’s requirements. Failure can result when any link is missing: an old client lacks the needed scheme, the server is configured to use an incompatible signature, or a certificate-chain signature is rejected despite the endpoint key being acceptable.

Do not “fix” this by assuming RSA-PKCS#1 v1.5 is an acceptable TLS 1.3 handshake fallback. Check protocol version and the actual selected scheme. If a constrained legacy client requires older behavior, assess whether TLS 1.2 is appropriate under your security policy rather than silently weakening algorithms.

5. GREASE and unknown schemes

GREASE signature-scheme values are reserved to test whether implementations tolerate unknown values and remain extensible. They are deliberately not real choices. Endpoints must not negotiate GREASE values; a server must reject a client-selected GREASE value. Clients should treat unknown schemes as unsupported and continue considering known choices. See [RFC 8701](https://www.rfc-editor.org/rfc/rfc8701).

Seeing an unfamiliar value in a ClientHello is therefore not, by itself, evidence of a broken client. Robust parsers must accept the list’s extensibility without selecting a value they do not understand. When troubleshooting, distinguish an unknown advertised value from the scheme actually selected in the handshake.

6. A practical diagnostic workflow

1. Record the negotiated protocol version. TLS 1.2 and TLS 1.3 have different signature representations and RSA rules.
2. Capture the ClientHello. Identify signature_algorithms and, if present, signature_algorithms_cert. Note the full list, not just the first entry.
3. Inspect the server’s certificate chain. Record the endpoint public-key type and the signature algorithm on each certificate. These answer different questions.
4. Inspect the selected handshake scheme. In TLS 1.3, examine CertificateVerify. Confirm it was offered and is compatible with the endpoint key; for RSA, confirm it is RSA-PSS.
5. Compare client and server policy. Check library version, security level, configured signature preferences, and whether the server has another certificate chain it could present.
6. Reproduce with a known client and a packet trace. A successful connection from one client proves only that one set of capabilities and policies overlaps.
7. Change one constraint at a time. Temporarily test a suitable certificate or a narrowly adjusted algorithm policy in a controlled environment, then restore the intended production policy.

For a quick OpenSSL inspection, use the following commands. They are diagnostic examples; option availability and output differ across OpenSSL versions.

openssl s_client -connect example.com:443 -servername example.com -tls1_3 -brief
openssl s_client -connect example.com:443 -servername example.com -tls1_2 -brief
openssl s_client -connect example.com:443 -servername example.com -showcerts < /dev/null

To inspect a saved certificate, first save the relevant PEM certificate from the chain as leaf.pem, then run:

openssl x509 -in leaf.pem -noout -text

Look for the public-key algorithm and the certificate’s signature algorithm separately. A certificate dump does not reveal the live handshake’s selected CertificateVerify scheme; use a TLS-aware packet analyzer or detailed client/server handshake logging for that.

7. Common failures and fixes

Symptom	Likely cause	What to check or change
“No shared cipher” or a generic handshake failure	Often reported generically even when signature capabilities, protocol versions, or other settings do not overlap.	Inspect the full handshake trace and ClientHello extensions; do not infer the root cause from the message alone.
Unsupported signature algorithm	The peer offers no scheme compatible with the server key or configured signing policy, or the implementation lacks support.	Compare advertised schemes with the key type, TLS version, and library support. Update or reconfigure a maintained implementation as appropriate.
RSA certificate works with TLS 1.2 but fails with TLS 1.3	The TLS 1.3 handshake signature path may be attempting an unsupported or disallowed RSA scheme, or a client may lack an acceptable RSA-PSS choice.	Check the actual TLS 1.3 CertificateVerify scheme and ensure a compatible RSA-PSS scheme is available.
Endpoint key looks acceptable, but chain validation fails	A signature on an issuer certificate may violate the client’s chain-signature policy.	Inspect each certificate’s signature algorithm and the client’s signature_algorithms_cert behavior and trust configuration.
Handshake ends with decrypt_error	The receiver could not verify the CertificateVerify signature.	Check transcript integrity, selected scheme, certificate public key, and implementation logs. Avoid treating this as a decryption-key problem without evidence.
New client succeeds; older client fails	The clients advertise different schemes or enforce different chain and security policies.	Compare ClientHello captures and supported protocol versions; decide which clients the service intends to support.
Unknown scheme appears in capture	It may be an intentionally advertised GREASE value or another unsupported value.	Confirm it was not selected. Ignore unsupported values while retaining known compatible choices.

8. Performance, reliability, and operational notes

Signature negotiation is usually a small part of connection setup compared with network latency, certificate-chain transfer, and application work. The practical reliability issue is compatibility: diverse clients and servers may have different protocol versions, signature support, and local security policies. A long advertised list does not guarantee success if the server has no usable key or chain.

For reliability, keep TLS libraries and server software maintained, deploy certificates whose key and chain work with the client population you support, and test both TLS 1.2 and TLS 1.3 where both are enabled. Monitor handshake failures by protocol version and client family when logs permit. Avoid broad algorithm downgrades as a first response; isolate the exact mismatch and make a documented, policy-compatible change.

There is no universal performance or cost figure for a signature scheme in this negotiation: the cost depends on implementation, hardware, connection reuse, and workload. The protocol-level decision is not a substitute for measuring your service. Reusing connections can also reduce the frequency of full handshakes, while certificate-chain size affects bytes transferred independently of the signature-selection logic.

9. Or skip the browser setup

If documenting a TLS issue requires a clean screenshot of a browser-visible diagnostic page, ScreenshotNeo can capture a URL with one API request. It is a website screenshot API and MCP server from [ScreenshotNeo](https://screenshotneo.com); the [API documentation](https://screenshotneo.com/docs/) covers its parameters.

curl -G "https://api.screenshotneo.com/v1/shot" -d access_key=YOUR_API_KEY --data-urlencode url=https://stripe.com -o shot.webp

Cookie banners, popups, and chat widgets are removed before the shot. Bot checks, blank pages, and failed loads are never billed. An MCP server lets AI agents take screenshots. The free plan includes 1,000 screenshots a month with no card; paid plans start at $5 for 3,000. [Create a free ScreenshotNeo account](https://screenshotneo.com/account/sign-up/).

10. Frequently asked questions

Does the server choose the first algorithm in the client list?

Not necessarily. The list expresses client capabilities and preference order, but the server must also account for its key, certificates, protocol version, and its own implementation and policy.

Does signature_algorithms_cert replace signature_algorithms?

No. They address different signatures in TLS 1.3: certificate-chain signatures versus handshake signatures. CertificateVerify is governed by signature_algorithms.

Can TLS 1.3 use an RSA certificate?

Yes, provided the handshake uses an acceptable compatible scheme. RSA CertificateVerify must use RSA-PSS; the certificate’s RSA key does not make PKCS#1 v1.5 valid for that message.

Is SHA-1 always prohibited in every TLS 1.2 context?

The specific rule addressed here is that SHA-1 must not be used in TLS 1.3 CertificateVerify. Do not generalize that sentence into a full account of every legacy TLS 1.2 compatibility rule; evaluate current security policy and implementation guidance for the context in question.

What changed in the current TLS 1.3 specification?

RFC 9846, published in January 2026, is a current TLS 1.3 revision. It retains the client-offered scheme requirement for CertificateVerify with the stated exception when no valid chain can be produced without unsupported algorithms, and continues to prohibit SHA-1 for that signature. See [RFC 9846](https://www.rfc-editor.org/rfc/rfc9846).

Standards references

• RFC 5246: TLS 1.2
• RFC 8446: TLS 1.3
• RFC 8701: GREASE
• RFC 9846: TLS 1.3 revision