If neither track is selected, placing the playhead (edit cursor) over the spot where you want to make a split will split both tracks. You can also select both tracks if you prefer by shift-clicking or control-clicking them (command-click if you're on a Mac).
I know you're not doing classical music, but the great classical guitarist Uros Baric has some good introductions to editing in Reaper here:
https://urosbaric.com/reaper-classical-music-editing; most of these tips will be applicable to your situation.