The Scot, who delivered the Millers from League Two to Championship safety after taking the reins in 2012, leaves the club having just overseen back-to-back wins.
His departure, described by the club as a "surprise", comes after a meeting on Monday morning with chairman Tony Stewart.
Evans' position had been questioned earlier on this season after a poor start to the campaign and speculation linking him to the Peterborough United job, but after taking seven points from four games, the storm had seemingly past.
There was no indication that his future was in question until this afternoon's announcement.
First-team coach Eric Black will take caretaker charge and will prepare the side for Friday night's clash with Burnley.
A club statement read: "Following a meeting between the chairman and manager this morning it was decided Rotherham United Football Club would part company with first-team manager Steve Evans and his assistant Paul Raynor.
"During the said meeting it became clear that both parties at the present time wished to head in a different direction. The meeting was at all times positive, constructive and straightforward. It was clear that the chairman and the management team only wanted what they felt was best for Rotherham United Football Club.
"The club would like to place on record our thanks to Steve and Paul for their huge contribution over the last three and a half years, it has been simply an incredible journey.
"It has all come as a surprise but Rotherham United must move forward and the search for a new first-team manager will begin immediately. Replacing Steve Evans and Paul Raynor will indeed be a strenuous task for the board.
"Rotherham United Football Club will always welcome Steve and Paul whether as an opposition management team or as supporters of our fantastic football club."
Evans was named as Andy Scott's successor in April 2012 and took charge of 172 games, winning 71 and delivering two promotions.
