Alright, I've now set it up just like you said. Disabled DHCP and connected the modem to the router's LAN1 port (1 of 4). The connection seems to work in any way I try it, but the main problem remains the same. My ping is almost always above 600 which makes it impossible to play any online game, or to watch video's, streams, youtube music, anything.
It just seems like there is too little of a bandwidth to work with in the first place. The problem with that is that in weekends or at night (past midnight) or in the morning (before 08:30) speedtest gives me 16Mbit/s to work with. In the afternoon it peaks at not even 2Mbit/s.
Do you have any Idea at all what could cause this? Because I'm starting to doubt it's the router/modem and starting to blame the ISP for this mess because literally every combination works but has sky-high ping.
Edit: To make things worse, in the first ~30 seconds after a complete router restart, the internet connection has 30 ping and ~16Mbit/s. After those first 30 seconds, it goes back to 600+ ping and max 2Mbit/s. How? What? I... Huh?