Are you finding that when the user does a mouse down, wait, and then drag, the DraggingTool starts rather than the DragSelectingTool?
One way is to make that Group ignore mouse events by making its interior be hollow, by setting what appears to be a light blue GraphObject.background or Shape.fill null. But that means it would no longer be light blue.
The other way is to customize the DragSelectingTool.canStart method so that it returns true when the drag is started on the group. See Rubberband drag selection is not working in groups.