Is there a way to create subgraphs that are not rectangular? I want the subgraph to hug the nodes/links fairly tightly.
Yes, you can override GoSubGraph.PaintDecoration to paint whatever you want. Normally PaintDecoration paints a rounded rectangle with bounds returned by ComputeBorder.