Ctrl-click (or ⌘-click) on any group to remove it.
Hold the Alt key as well to remove the entire subtree.