The only way to avoid leaving by him is to leave him first. Then he never gets a chance to leave you.