The Colts are much larger than merely Ballard. Ballard may potentially be the best GM in history, and yet fail because the Colts as a team (from FO, staff, coaches, trainers, players and beyond) won't align as a team to accomplish what needs to be accomplished. I have thought that the player talent was always sufficient to perform. Yes, it could be (could have been) better. And yet, they continually grossly underperform. Imagine how frustrating that is. Altogether as a team, they are clueless. We could have all #1 draft picks and not win many games (seems the Redskins tried this once).
For whatever reasons, the coaching staff isn't getting the job done, and after this length of time probably never will. Worse, they (and here I refer to the coaches) are damaging the "brand" of the Colts, almost to the point of no repair, at least for Andrew Luck's career. It would take several years to correct the culture ... if ever. Who in their right mind would want to risk that possibility with their career? See, for example, the Jets, Browns and Bengals.