Yep, done correctly, there is no reason not to do it every time. I know a lot of guys do, but I have not had trouble with not annealing every time.