If they have an installer (.pkg), just run it. If you have a .vst "file*", it goes into ~/Library/Audio/Plug-Ins/VST (single user) or /Library/Audio/Plug-Ins/VST (system-wide). Plugins often come in container formats such as .dmg or .zip. Open/extract as needed to access the files they contain.