No reason to. It will not have any particularly noticeable effect on the amount of power delivered, or the responsiveness of the engine at all, except to make it smoother at lower speeds. In fact, it can actually add a little power at very low RPM by overcoming velocity losses from spinning through compression.
I can take (the flywheel weight) off if I'm going to track the bike.
I run mine all the time, and I know several others who do as well. Try it with it on before you decide to switch it on and off all the time.