I've seen this thought from a few times. I think there is some truth to what you say as the roster needs an upgrade for sure. But on the flip side, I think you can make the case that player development under Pagano has lacked. How many players have gotten better under him and his coaching staff. I would say not enough.
I would also say that Pagano is a miserable failure at in-game management. I won't list them all but if you've followed the Colts for the last 5 years you've seen plenty of examples of this.
I think it also something that cannot be questioned in that schematically, the Colts are generally outclassed in most games. Again, won't dive deep into this but listen to any Venturi podcast to point out flaws in planning that are repeated over and over.
I think even if you are right on your cause and effect theory it will matter very little. The perception out there is that he needs to go. The damage from this season is pretty strong and the fan-base will walk away in droves if Pagano is brought back. Irsay knows this. Pagano is as good as gone.