It isn't weird at all, but you don't list your hardware information at all or anything for that matter then no one can help you.
I can only give you an educated guess of your hardware, for an example...
Latency equates to output sound bandwidth, if your hardware used non-standardize data pathways such as some ASUS implementations of audio in nVIDIA nFORCE series chipset, the additional latency caused loss of bandwidth.
To compensate the POST data sample windows can be shifted back and additional gain applied, this will translate to too much bass and not enough treble, or the inverse when none being used with same symptoms as yours which is too quiet due to truncation of lower bandwidth spectrum samples from the latency delay. Boosting the gain will give much hissing with little gain in bass.
