Truth be told, when I'm talking to someone I haven't yet converted to the "button up" terminology (read: everyone), I say Oxfords, too.

Although, apparently, that's sometimes a bit of a misnomer, as technically "Oxford" refers to a particular type of cloth, and not all shirts I'd call Oxfords are made with Oxford cloth...