I once solved this same problem basically the same way.
I spawned a cube room, on the outsides top/left/right/down were the items the user could select from the next position, and the camera rendered the current. Since there was a delay of .05 seconds between UI selector, we could lazy load next group by keeping them in cache for inventory close/open.
One trick I picked up was keeping a pre rendered pool of the most common items since it removes basically all render time with inventory open/close.
It was either this or figuring out a way to dynamically colour the icon images directly (which is why I have put off actually drawing most of them so far). This seemed easier, and more fun!