I'm not sure to fully understantd your method
The only probem I see with physics is "streaming". If it's not possible for you to generate a FULL simplified mesh of you world (with A, B, C ... and every other meshes) then, you're right, it will be a bit more complex. In the other case, be sure that's not a problem to feed RayODE with a huge mesh (the problem is memory, not performance).
But, hey, maybe a quadtree or an octree will resolv all this ? (it's quite simple to write, the only problem right now is that Raydium is not able to "unload" a mesh, you can only replace a mesh by another one).