Cavendish fell off his bike in a big pile-up with less than three km left and crossed the line long after Greipel had outsprinted Italian Alessandro Petacchi.
The Briton sat stunned for a while on the road, his world champion jersey in tatters. Then Cavendish, who narrowly beat Greipel in the first bunch sprint of the second stage, got back on his bike and crossed the line before rushing to his bus and angrily throwing aside his bike.
The crash, which also involved South African Robbie Hunter, Cavendish's lead-out man Bernhard Eisel and Australian Mark Renshaw, played to Greipel's advantage.
The burly Greipel, nicknamed the "Rostock Gorilla", was perfectly led out by his Lotto Belisol team mates to snatch his second Tour victory after he won the 10th stage last year.
Tom Veelers of the Netherlands took third place while Swiss Fabian Cancellara, who narrowly avoided the pile-up, retained the overall leader's yellow jersey at the end of the 214.5-km ride from Abbeville.
Cancellara has now held the yellow jersey for 26 days, a record for a non-Tour winner.
"To win stages is great but to beat records is a great honour," he said.
Cavendish's mishap, as well as a facial injury to Eisel, were more blows to Team Sky, who were hit by crashes in the previous stage, losing Belarussian Kanstantsin Siutsou.
"It's disappointing and worrying when you see lads going down. I'm sure they'll be fine and get up and fight for another day," said team manager Dave Brailsford.
Greipel, a former team mate of Cavendish in the now-defunct HTC High Road, stayed out of trouble thanks to his team, probably one of the most efficient in the world on the final stretch.
"I have the best lead-out team in the world. I'm always fast but when you have guys like this it makes things simpler," Greipel said.
Asked about Cavendish's crash, he said: "I heard something behind me but at 60 kph you cannot pay much attention. I didn't have the hard luck to be caught in the crash but once again it was thanks to the team who positioned us perfectly."
The German said he would have loved to beat Cavendish in different circumstances.
"We just deserved victory today. Monday, we saw that I was competitive against him. I hope he's okay so that we can see another sprint against him. We do a dangerous job but I love it."
Slovakian Peter Sagan, who won the first and third stages, took fifth place and retained the green jersey for the points classification.
Thursday's fifth stage is from Rouen to St Quentin over 196.5 km. DM
Photo: Team Sky rider Mark Cavendish of Britain sits on the ground after a fall during the fourth stage of the 99th Tour de France cycling race between Abbeville and Rouen, July 4, 2012. (REUTERS/Joel Saget)