Hi,
Are you using a self-signed certificate on your mail server? I recently found out that it’s impossible to connect with a mail server that is using a self-signed cert.
If you set 'mail_smtpdebug' => true, in your config.php you should see the following error when using a self-signed cert:
SSL operation failed with code 1. OpenSSL Error messages:\nerror:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed
I recommend to use the tool swaks on the ownCloud server command line to check whether your email server supports STARTTLS. Once you have confirmed that it works through swaks you can try to replicate the settings in your ownCloud.
Cheers,