My sil is a painter and no you don't have to sand unless you have holes that need filling in with compound. The primary reason to use primer is when you have new never painted walls and when you are changing from one color to another like dark to light. I use to change my wall colors every 2 year. Now having him in the family I just might do it every year.
