As I understand it, when Quick Removal is disabled the device uses "write caching". This allows slightly faster operation as files/items are cached in memory and this improves performance when trasferring and managing data on the device. Mind you, I've never noticed any measurable difference when write caching is disabled to allow quick removal.
I think the worst that could happen if you disconnect abruptly when write caching is enabled, you may corrupt a file or two. Never experienced this either though, but I always enable quick removal.
Hope this is a little helpful,