Do you have to worry about silicon dab rigs melting from a hot banger or torch?

Yes and no.

If you keep the torch away and do cold start dabs, not much heat will go back to the silicon and you should be ok.  But we still suggest sticking with glass baby!!