I would prefer to have one selection tool that selects all things, and then have all things be switchable between token and drawing. I never used the drawing tools before, but in a recent campaign I was fumbling with the drawing-or-token? problem, and switching back and forth. I wanted to explain it to a player, and it was...awkward.
I would have the layers structured in such a fashion that an object that is on the Token layer is always a token, and something on, say, a Drawing layer is always a drawing, and moving an object between layers automatically changes them.
Implementation-wise, I'd separate the "Object and Token" layer into two layers - Objects, and Tokens. On the object layer, any thing that exists can be selected, it has no radial menu, etc. On the Token layer, everything is a token. In the end you have 4 layers: Background, GM, Objects, Token (in order from bottom to top). The variation in how things are treated is determined by where you place the item. Now, I confess this is my preference because it is how I am used to working with a VTT (namely MapTool), but I think it is an effective model (edited to add: of course, the GM layer could be on top, too, as an overlay to match the naming convention you've established).
