Thank you very much for the answer, @walter!
That is actually the workaround we are using now; switched to an immutable array for the ports and we just create a new one and replace it.
But we still run into the problem with links not being redrawn when we reorder the ports (this one Swapping order of ports doesn't update links) :(. The workaround for this one works as well, but it is a bit cumbersome to implement; to keep the code clean we added a model changed listener to the diagram, watch for that ports array being changed, and calling invalidateRoute for all those links connected to those ports at the end of a transaction.
Thought there would be a better solution than two workarounds :(.
Thank you very much again! Really appreciate the clarification.