This is very common misconception. Food as a percentage of income has significantly shrunk over the last 100years. Mechanized farming tech, chemicals, and GMO's have lead to sizeable yeild increase. The craziest thing is a bushel of wheat cost the same as it did 50 years ago.
People spend their money on housing, entertainment, cars, healthcare and various activities.
Now an argument can be made about access to quality food. Also less people garden and raise their own livestock.