Hello Mack,
The USB controller will not be damaged is USB VBUS is present while USBVDD is off. I am assuming you're using the latest silicon revision 0.2.
I'm also assuming that the 5V_USB in your circuit is not actually connected to a power source. If you're only operating in device mode, then VBUS should only be sourced from the host through the USB connector,
I'm wondering if you're not encountering a situation described in this forum entry: http://ez.analog.com/docs/DOC-7106
Regards,
Jorge