How hard would it be to get off if I do glue it with the silicon?
Not hard at all. Just scrape off what you can and then use some type of solvent like Goo Gone. I used a piece of inner tube RTV'd to the swing arm under the slider on my '03 WR 450 and put 5000 miles on it and never had a problem. Ran the stock chain and sprockets that whole time and they barely showed any wear when I sold it a month ago.
I did the same fix to my YZ and won't worry a bit.