You enter your search string into the text field in the search dialog, and press enter or click "Search". The Settings button opens a menu showing the settings for your search. Settings include limiting which fields are searched, deciding whether your search string should be interpreted as a regular expression, and deciding whether the search should be case-sensitive.
Once the search finishes, the entry table in the main window will change to show the results. Entries are sorted according to search score (you can see the score if you have checked the "Search" field in the Preferences dialog), and any entry in which the search string was not found, will be grayed out. The table changes to its normal state once you close the search dialog.
While a search is active, it will only display hits contained in your currently shown group. Searching and displaying groups are separate actions, and the table will always display the appropriate subset of entries decided by your active search and/or group.
You can at any time export a bibtex database containing only the entries in your currently shown group and/or search, by choosing File -> Save special in the menu.