This is because the Diagram Div does not have focus until you click on it once, after which it is brought into focus, but this “bringing into focus” also moves the diagram, so the end of the mouse click happens elsewhere.
One way to fix this is to set Diagram.scrollsPageOnFocus to false.