History problem on one undoManager per layer

we have 3 layers and we have 3 undo managers(1 per layer).
When we switch between layers we set model.undoManager as well.
But we can not reach or save history of related undoManager in the scenario below.

1 : Remove node X on layer A
2: Switch layer B.
3: Then switch back to layer A.
4: Try undo to get node X back ( but can not )!

We can not undo in this scenario.
We can only undo/redo if we do not switch layer/undoManager.
How can we reach history of undoManager before layer/undoManager change?