I’m sorry, but I don’t understand your situation and what you are trying to accomplish.
Geometry objects only apply to Shapes. A Shape can have a complex Geometry, but it only has a single stroke brush, a single fill brush, and a single background brush. So if you have SVG with multiple “shapes”, you cannot represent that with only a single GoJS Shape. You have to use Panels containing multiple GraphObjects.
Why is using a Picture insufficient in your Palette?