I was wondering - when exactly did Lincoln fall in love with Altlivia?

What if he fell in love with her during the switch, when Olivia was living Altlivia's life? Maybe it's Olivia that he loves instead?