I can't get away from the collar button meaning of "button-down", but then what should those other shirts be called??

Anyway, at some point, I decided to call any shirt that buttons up the front a "button-up" shirt (contrast: pull-over). So button-downs are shirts with buttons holding the collar tabs down; all the rest are button-ups. (And yes, button-downs are a usually all button-ups, too, but that seems to be ok.)