I mostly use iPhoto for the ease of sharing the photos with everyone else in the world, but yeah, generally its bulky and cumbersome, and doen't think like I do.

Aperture is the same way, I'm currently using it to archive old scans of photos because I can have several libraries at once.

If you're more loosy goosy you might want to consider Adobe Bridge. I use that for all my clip art and stock photography and it's basically a glorified viewer, but nicely cross platform.