Interesting, but certainly a valid point: price point vs progress. However, let me augment the picture you described slightly
1) 6700K being 45% faster than 2600K: that is mainly because Intel allowed that much higher clock speeds out of the box for the latter (while at the same time the 2600K was VERY conservatively clocked, one even might say severely held back...thanks God it was dead easy to overclock to 4.5

). There were a ton of clock-for-clock comparisons through 4-6 generations from Intel, and it seemed, for a long time, that there is no real innovation from them, just higher out-of-the-box clock speeds and (eventually) more cores, keeping up the illusion of progress.
2) Intel's very successful (ring bus style) cache architecture was told to be optimised for 4 cores, so understandably, they wanted to stick with it. Without any competition, they could do just that. For 6 (!) generations, they launched 4 core 8 thread CPUs as their flagship consumer models. If you wanted to have more cores, you had to go "enterprise", with astronommical prices (think $1000+ for a 6-core Xeon (yes, the same which you can buy now off Ebay for about 15 quid) plus a $3-400 motherboard)
3) Your 8700K (cracking little CPU, btw) is most probably the result of AMD's rising competition. If they didn't launch Ryzen, maybe we still would be uising 4/8 CPUs (like the 7700K, which has the same microarchitectrue as your 8700K...or the 9900K, coming to that), as "they are the best". Also, if you have a look at those above-mentioned clock for clock, core for core comparisons, I wonder what you would make of Intel's prgress over the past few years (let's skip "ancient" history, and just start with your 8700K)
4) AMD is bringing solid IPC increase with every generation so far. I wonder if they can keep it up. Either way, the good thing is that there is SOME competition at least, vs Intel monopoly for a decade with quad core processors as the pinnacle consumer technology. And just to make clear, when times were different, and AMD Athlons ruled the world while Intel's P4 was a laughing stock, it was not good either (then it was AMD's CPU prices which started to inflate disproportionally)
For one reason or another, it seems that we have turned into an era where semiconductor products are getting more expensive (instead of getting cheaper), so it may never return that for $300 you can buy a flagship model. With pandemics and the hysteria around it, with rare-metal prices going up, with manufactoring monopolies (I'm looking at you, TSMC), with China's global (manufacturer of the world) role quiestionned, and with so many other factors, it is difficult to say or see. But I don't think it is Intel's or AMD's fault.
I for one believe that innovation is there (especially in the CPU segment, more than ever before), only (sadly) it costs more, and as a result, previously established price points are being overwritten before our very eyes.