Colombia had already advanced before this match, earning a berth in the quarterfinals with a win over Paraguay on Tuesday and Costa Rica was eliminated from contention with the United States' 1-0 victory over Paraguay earlier on Saturday.
The loss makes Colombia's path in the tournament more difficult as the team will likely face Brazil in New Jersey because of the loss.
The Los Cafeteros would have drawn either Ecuador or Peru with a win. But the United States did benefit from Costa Rica's upset, with Colombia's loss giving the team first place in Group A.
Frank Fabra and Marlos Moreno scored Colombia's two goals. Fabra scored Colombia's own-goal in the first half to give Costa Rica a lead it would not relinquish. After failing to score in its two previous matches in the tournament, Costa Rica wasted no time getting on the board on Saturday.
Venegas put Los Ticos up 1-0 with a 20-yard strike to the upper left hand corner in the second minute. It wasn't just the last two matches in which Costa Rica hadn't scored. The goal by Venegas was the team's first in Copa America play since it beat Boliva 2-0 in 2011.
Colombia evened it up in the seventh minute when Fabra split a pair of defenders and regained control of a loose ball just in time to boot it in and make it 1-1. Fabra was shaken up midway through the first half when he slid and took a kick to the head by Christian Bolanos.
He stayed on the ground for a couple of minutes before being placed on a stretcher and taken off the field. He wasn't out long though, getting a Band-Aid over his right eye, a fresh jersey and returning in the 23rd minute.
Costa Rica took the lead when Venegas crossed the ball into the box from the left and Fabra accidentally ran into it and knocked it into the net as he fell to the ground for an own goal in the 34th minute.
With an assist from Bryan Oviedo, Borges scored from 15 yards out in the 58th minute to extend Costa Rica's lead to 3-1. Colombia's Marlos Moreno cut it to 3-2 with a perfect strike to the near post in the 73rd minute.
Because Colombia was already in the quarterfinals, Los Cafeteros rested Real Madrid star James Rodriguez in the first half. He entered the match early in the second half to a roaring ovation.
The robust crowd of 45,808 at NRG Stadium, where the NFL's Houston Texans play, was overwhelmingly filled with Colombia fans.
A sea of yellow filled the stadium, with hundreds of fans waving Colombian flags and many faces painted with bright colors.
One man went all out, dressing in a sparkly yellow costume that covered him head to toe and had webbing under both arms that made him look like a star when he lifted both arms skyward.
There were patches of Costa Rica fans, but their red crimson shirts were exponentially outnumbered by the throngs of fans who donned yellow for the match.