Yes and no.
Silver fume sticks or adheres the glass better than gold does. It is why we always fume with silver first and then gold.
Silver can actually grow into the glass, this is how the whispies are done in glass. If it is a good silver fume it will not come off, but a bad silver fume will start to rub off. Good silver fume will not be white looking, but rather a yellow, or blue or purple, depending on the light.