Gaviria edged Sagan at the finish line to claim Stage 4 on Tuesday and take his second stage win in his first participation at cycling's greatest race.
The 23-year-old had already bettered Sagan in a sprint to take the opening stage, amid rising expectations since his four stage victories at last year's Giro d'Italia.
BMC proved the class of the field though, completing the 35.5km circuit around Cholet in a time of 38 minutes 46 seconds, four seconds ahead of Sky, to put Olympic champion Greg Van Avermaet in yellow.
"It was a very hard sprint, but we knew how to pull it off", Gaviria said.
Gaviria looked as though he had gone too soon as Greipel charged down the middle but the Colombian came back strong with a second jump to win the stage, just holding off a late charge from Sagan. "Lets see tomorrow how he does where there's a bit of climbing".
"Tuesday's stage might be tricky, you have to watch out for winds and the narrow roads and not get trapped behind a fall", he said, perhaps thinking of the scenario on Sunday when the sprint was marked by a multiple pile-up.
Riders were enjoying a calm sunny afternoon in the cycling hotbed of northwest Brittany until a pileup near the front of the peloton with just over 5K left that sent several riders to the tarmac.
A crash in the middle of the bunch disrupted the chase further, with Tony Gallopin (Lotto-Soudal) among the worst affected, while Rigoberto Uran (EF-Drapac) was one of the GC contenders forced to chase back.
But there were no changes among the title hopefuls.
Paul Pogba defiant after starring role for France
It's always nice to win and to hear good things about the team, about yourself and everything". When he had to hold position and keep control of the game, he did. "He was very, very mature".
Porte is 14th overall at 51 seconds.
Movistar will be delighted that Mikel Landa and Alejandro Valverde are right on the heel of the Sky train, with Valverde having plenty of cobbled form.
At one point, the quartet were eight minutes clear of the peloton and they maintained their lead for more than 190km before finally being caught in the closing stages as the pack combined to reel them in.
Fans cheered and waved the black-and-white striped Breton flag on the roadside as the peloton rolled through the country villages.
Wednesday sees the Tour head for the hills, with an arduous 204.5-kilometer leg from Lorient to Quimper.
Froome is aiming to join Jacques Anquetil, Eddy Merckx, Bernard Hinault and Miguel Indurain as the only riders to win the Tour five times.
Guillaume Van Keirsbulck, who was involved in the breakaway, won the day's intermediate sprint.
The three-week Tour ends July 29 in Paris.