Neither one of the clients ownCloud Desktop or android can log in to my server this giving invalid password. I already tended to create an application password but could not succeed.
Custom Cloud 10.0.2 (beta)
Desktop Client Release 2.3.3 rc2 August 18th 2017
Obviously something is wrong in your configuration. You might want to check the clients against demo.owncloud.org (admin/admin for login) and then follow the standard installation instruction or maybe use the Appliance provided https://www.univention.de/produkte/univention-app-center/app-katalog/owncloud/ which has everything pretty much preconfigured. Or docker files ...
What is Custom Cloud?