I don’t agree with the common method of practicing rudiments from slow to fast and back to slow again. This tends to build bad timing into your muscle memory.
You should practice your rudiments at all tempos but you’re better off stopping and re-setting your internal clock before each tempo change.