I started having similar problems a few weeks ago (I'm using a MOTU ultralite mk3.) Very intermittent, perhaps once a session, but it sucks when it's during a live take!
Unfortunately, I upgraded to REAPER 6 and Windows 10 v1909 about the same time, so I don't know who to blame.
My system was fairly rock solid before that. CPU and memory are never near max, so it's a not resource problem.