The "button-down" collar distinction comes from a time when all "shirts" were assumed to have buttons down the front in the first place. Nowadays there are many varieties of pullovers that are also referred to as shirts, so button-down has taken on a second meaning.

I feel like I should have slides for this...