Odd Token Selection Problem with Map Layer

I have what I think is a very typical dungeon set up. The map of the dungeon and some items that I don't want to accidentally pick up and drag (tables, furniture, etc) are all on the map layer. However, when I am on the map layer, I cannot select anything other than the large map unless I carry out the following sequence of events: 1. Select the map. 2. Move it. 3. Undo the move. 4. Now I can select one of the other tokens. Only once mind you. After I have moved that token, I then cannot select another one until I have repeated steps 1-3. I'm sure there's something I'm doing wrong, but this is frustrating me to tears.

Another workaround: Shift the map to the token or GM layer, work on all of your small stuff, go to the token or GM layer and shift the map back to the map layer. - Gauss

Esc works for me. Thanks for the tip Riley D. :) - Gauss

The ESC thing definitely works - many thanks, Riley!