Preferences > Mouse Modifers: [Track][left click]. Set it, for instance, to [Deselect all items] instead of the default Behavior [Deselect all items and move edit cursor].
There's also one for [Media item][left click]: as example, using [Select item] instead of using the default Behavior [Select item and move edit cursor].
And, as you already mentioned yourself, there's one for [Ruler] [left click], as well