Telegram is more invasive than WhatsApp. It asks for more permissions on my phone. When an app starts doing that, it's a signal to uninstall it. Unfortunately, some of my friends are using Telegram exclusively, so I can't.
Does anyone really believe in end-to-end encryption? I mean, the asymmetric keys are only used to exchange the symmetric keys, which are then used for real-time encryption/decryption. Accessing the asymmetric keys relies on the confirmation by a 3rd party. Which means this 3rd party (or the IM service provider itself) has access to it, and anyone who controls the channel to the 3rd party verification process. Once they have your asymmetric key, they will also get the symmetric key, which means your data is not really protected from them.
Sure, they are protected from random hackers, but don't ever think, even for a moment, there exists a true end-to-end encryption.