Again, you could make a pretty good case that the Patriots cheated. You could punish the organization pretty fairly based on the information we have. If the decision was to suspend Bill Belichick, fine Robert Kraft, or remove a pick from the Patriots franchise, based on the idea that franchise leadership had a moral/legal obligation to prevent cheating, I wouldn't have as big of a problem.
It's the fact that effort was made to single out specifically Brady that has never made sense to me. Deflated footballs has always struck me as a franchise issue, not an individual player issue. Equipment managers who won't stand up to players who want illegal things to happen is a team infrastructure problem first and foremost.
Go after the team collectively, the owner or the HC individually, fine. Singling out individual players is something you do when you have direct evidence that they were involved.
If what you have is circumstantial evidence that they might have been involved, then you have to punish the team, the owner or the coach.