Hello!
Our security framework was set up by our implementation consultants so unfortunately I can't point you in the direction of any documentation. I can say that yes, we use nested security groups often.
For example, we have an entity read security group (which is assigned to the entity in the dimension library) and a child of that group would be our actuals execute workflow security group. Similarly, we have a scenario read (or write) security group and a child of that group would be the forecast or actuals execute workflow security group. Hopefully that makes sense.
It could definitely become messy depending on how many layers and what you need security around but once you outline it and replicate it for all workflows/subs/entities/scenarios/etc, it's a little easier to manage.