The Portuguese was sacked by Roman Abramovich in March following a disastrous nine months in charge of the Stamford Bridge outfit but the Blues later went on to win the Champions League for the first time in their history under Roberto Di Matteo.
Ivanovic believes that games against Spurs are always vital but acknowledges the extra edge this time around and is determined to be on the winning side.
He told The Fulham & Hammersmith Chronicle: "It's always very difficult to play the game after the international break but especially when it's big opponents like Tottenham away.
|10/3||Tottenham v Chelsea is 10/3 with Bet365 to draw with under 2.5 goals
"Our games against Tottenham were very interesting and tough for both sides even before Andre went there. Now it is especially so for both sides with our ex-manager.
"It will be difficult in one way and in the other way we know what is the idea and what we can expect of them.
"This game will be decided by things like when you are back from your international duty and how tired you are - the small details. So it's going to be tough for us."
The Serbian defender is satisfied with Chelsea's current position at the summit of the Premier League but warns that they must keep the same level of performance after the international break with tough games coming up.
"The international break is a good time to enjoy our position but there is a very important period coming up for us after the break," he added.
"We have some very tough games and they will be a great challenge for us. At the moment it's OK but the next two games will be very, very tough."