When you are in layoutmode, you can remove a plugin from the layout by first selecting it in the layout-manager. Its layer is now displayed with a highlighted background. Once it is selected, just click on the Remove button and the selected plugin is deleted from the layout.
