java - SSL handshake exception: "Algorithm constraints check failed: MD5withRSA"

Question

I tried to install Oracle Entitlements Server Client. When I call

config.cmd -smConfigId Sample-SM -prpFileName C:oracleproduct11.1.2as_1oessmSMConfigToolsmconfig.java.controlled.prp 

I got this Exception:

    javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path validation failed: java.security.cert.CertPathValidatorException: Algorithm constraints check failed: MD5withRSA
        at sun.security.ssl.Alerts.getSSLException(Alerts.java:192)
        at sun.security.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1884)
        at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:276)
        at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:270)
        at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1341)
        at sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:153)
        at sun.security.ssl.Handshaker.processLoop(Handshaker.java:868)
        at sun.security.ssl.Handshaker.process_record(Handshaker.java:804)
        at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1016)
        at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1312)
        at sun.security.ssl.SSLSocketImpl.writeRecord(SSLSocketImpl.java:702)
        at sun.security.ssl.AppOutputStream.write(AppOutputStream.java:122)
        at java.io.OutputStream.write(OutputStream.java:75)
        at oracle.security.oes.enroll.EnrollmentClient.writeToSocket(EnrollmentClient.java:330)
        at oracle.security.oes.enroll.EnrollmentClient.enroll(EnrollmentClient.java:161)
        at oracle.security.oes.enroll.EnrollmentClient.main(EnrollmentClient.java:478)
        at oracle.security.oes.tools.EnrollmentTool.doEnroll(EnrollmentTool.java:103)
        at oracle.security.oes.tools.SMConfigTool.doEnrollment(SMConfigTool.java:1192)
        at oracle.security.oes.tools.SMConfigTool.run(SMConfigTool.java:617)
        at oracle.security.oes.tools.SMConfigTool.main(SMConfigTool.java:546)
    Caused by: sun.security.validator.ValidatorException: PKIX path validation failed: java.security.cert.CertPathValidatorException: Algorithm constraints check failed: MD5withRSA
        at sun.security.validator.PKIXValidator.doValidate(PKIXValidator.java:350)
        at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:260)
        at sun.security.validator.Validator.validate(Validator.java:260)
        at sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:326)
        at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:231)
        at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:126)
        at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1323)
        ... 15 more
    Caused by: java.security.cert.CertPathValidatorException: Algorithm constraints check failed: MD5withRSA
        at sun.security.provider.certpath.PKIXMasterCertPathValidator.validate(PKIXMasterCertPathValidator.java:159)
        at sun.security.provider.certpath.PKIXCertPathValidator.doValidate(PKIXCertPathValidator.java:351)
        at sun.security.provider.certpath.PKIXCertPathValidator.engineValidate(PKIXCertPathValidator.java:191)
        at java.security.cert.CertPathValidator.validate(CertPathValidator.java:279)
        at sun.security.validator.PKIXValidator.doValidate(PKIXValidator.java:345)
        ... 21 more
sun.security.validator.ValidatorException: PKIX path validation failed: java.security.cert.CertPathValidatorException: Algorithm constraints check failed: MD5withRSA

Can you help me to find a reason?

Answer 1

The problem is caused by Oracle disabling hash algorithms which are no longer considered to be secure. Take a look at

JRE_HOME/lib/security/java.security

It contains the following properties:

jdk.certpath.disabledAlgorithms
jdk.tls.disabledAlgorithms

You can adjust them appropriately. For example, remove MD5 from the former and MD5withRSA from the latter.

Hint for docker images:

there is additional config file /etc/crypto-policies/back-ends/java.config in some docker images like keycloak in my case which overrides values in java.security