'The deal with that'
is that wherever the item starts, and apparently also by how much of it is within a an area of tempo, its rate is adjusted. Audio rate does not change at the lines (there have been requests iirc to create stretch markers automatically at these locations to deal with the issue)
To stop it, set the item timebase to time from the Item Properties.
You can also place stretch markers and they will always obey time, it seems, so long as they are after the tempo change.