An Adornment is a completely separate Part that is associated with the adorned Part. The presence or size of a “Selection” Adornment has no effect on dragging (the DraggingTool) or on any other operation. That is one of the good things about using Adornments. But changing some Shape.fill or Shape.stroke won’t affect any other operations either. (However changing the Shape.strokeWidth may do so, because changing the size of the Shape may result in other objects being positioned or sized differently.)
So the problem is that you have a Part.dragComputation function, or some other restrictions on the DraggingTool’s behavior, that is preventing the user from moving there. You “just” have to remove those restrictions.