Worried? I don't think so. I don't know what part of the town you live in but no, I don't think many people here are worried at all, for various reasons. 12 is in no way better than Schaub, by any stretch of the imagination. If we win the SB this year, and next year, Schaub still wouldn't get any respect, so I don't care about the whole elite label.
I'm a lifelong UVa fan, and was a season ticket holder for all of Matt Schaub's career at UVa. I remember when he was splitting carries with Bryson Spinner, and looked below average at best. I even saw him walk across the stage and get his degree (my sister was in the same graduating class as Schaub.) I've seen him develop and become a pretty good pro quarter back. I don't consider Schaub one of the elites, but I do think he is underrated. Schaub isn't flashy, and isn't a media magnet, and I think that is why people don't give him the credit he deserves. That said I do think Luck will have the better pro career, barring injuries and such.