Go ahead, the best thing I did was go to the iPhone. Enough with Android's garbage heap of an OS platform.
I had four Android phones, all of them had issues with them.
There are few things that would suck to loose such as the phone buttons (back buttons) on the Samsung side as well as the perfect Google eco system integration, however the phone is not very "fun" and the OS is little bit bland. Will see, still at least a year away from upgrade.
H3llion I made the switch a bit back... I'm still a big google fan, however, I'm glad made the flip. Personally for me, I find the "S" much better than intro models (example for me iphone 6 seemed laggy and bit of early release, 6S is fast, everything 6 had that gave me troubles works fine. Same happened for 5 Series with me).
Eco system, just my opinion, however I've found iOS to have much better eco system - loss of buttons, at first I hated that, almost went back, though after some time with Apple I miss them not anymore...
I really commented as you nailed my thoughts which caused me to jump ship on Android - bland, from being a bit more unified with solid hardware technology (and our phones certainly aren't cheap to buy these days) - Apple has the Marketing power, perhaps gives their products a bit more flash than they really have, though I'd not go back to Google on the phone or Tablet.
I'm not certain which carrier you're with, I'm with Verizon, if you made the jump, they give you some time to do a return if you change your mind without things being too messy - guessing more companies have a like policy.