Is wearing surplus considered stolen valor?

I’m wearing my Flecktarn around town and my friends are pestering me for “stolen valor”. How could this be the case when I’m not impersonating, not wearing insignia, and just wearing it because I think it’s cool?