Hi Patryk,
SQL Anywhere 16.0.0.1670 and later use a different implementation of TLS. There were a few minor behaviour changes made. One of these is likely the source of your problem.
Are you using the FIPS libraries or just the standard SQL Anywhere TLS? If you are using FIPS you may need to convert your identity file from triple DES encryption to AES.
If you are using a self-signed certificate, we now require the certificate signing key usage bit to be set. Could you check your certificate using the 'viewcert' utility? What is the output beside 'Key Usage'?
Thanks,
Mikel