You can try adjusting the ForceDirectedLayout.defaultSpringLength, defaultElectricalCharge, springLength function, and electricalCharge function to change the way nodes and links repel each other.
This topic might also be useful to you in case you want to override the functions: Clustering subtrees in force directed subtrees