I think it depends on how well they are attached- if they are hanging loose, I say remove them- if they are attached securely, I say leave them.
I removed them from one dog and had them left on the other- she has never had any issues with them, we keep them trimmed (or the groomer does
anyway).