We had heard that you have to take your shoes off a lot in Japan. I even got some slip-on shoes (Clarks Privo, very nice). But actually, excepting the Ryokan, there were not that many places on the stay where we did. Here’s the list:

1) A Japanese restaurant in Kyoto, on the street up to Kiyomizudera, we had to sit on cushions. Delicious food.
2) Nijo castle in Kyoto, where you get to walk on the “nightingale floor” in your socks.

That’s it! Or course in the Ryokan you have to change slippers ever five feet, but if you are not staying at a Ryokan (or someone’s house), then I’d really not worry too much about it. Brad advised us to favor comfort over ease of removing, and I totally agree. Luckily my shoes were both comfortable, and removable, just make sure you break them in. You walk a lot in Japan – even if you taxi everywhere, the temple-type sites are big, so good shoes are a must.