Does the property have to be my primary residence? Yes. To keep this loan, the home must be your primary (main) residence for five years. If not, the loan will become immediately due.