Ok, I increased the buffer size to 1056 samples and all is fine! I guess I'm just hitting what's possible with my settings and CPU. Also changing the playhead position seems to increase CPU load a lot for a very short time, causing a xrun. I also loaded an older project from last year and it played fine with 528 samples, however this project used not so demanding VSTs like U-he DIVA or Repro.
There was also a discussion at linuxmusician about the average CPU load:
https://linuxmusicians.com/viewtopic...average#p99669
Seems that even when RT CPU load is ~20% it can exceed ~95% for short times and these spikes are not seen in the CPU load of Jack or Reaper as it's strongly averaged. Also realized that the CPU load of my recent project went from 9 to 20% already. So it makes sense that now I see xruns even without the pulse bridge enabled.
Next PC I buy will be an I7 and at least 16 GB RAM if not 32 GB.