You may want to try the plugin on your test site before moving on to Production or Real site. In that case you may have to De activate the plugin in your test site first.To De-activate the plugin you should first go to Plugin Settings page.
On the plugin configuration page you will see the De-activate button. Click on the button to De-activate the plugin. Once the plugin is De-activated you can delete from the site.
It is always advised to De-activate the plugin from configuration page before removing it from the site or De-activate it from Plugins menu. You should not directly remove or De-activate the plugin from Plugins menu.