Sow was drafted by the Miami Heat in the second round, 47th selection, of the 2004 NBA Draft. The draft rights to Sow were acquired by the Toronto Raptors (along with a 2005 second round pick from the Heat) for the draft rights to center Albert Miralles, the 39th overall pick.
Sow played his NBA rookie season for the Raptors and showed considerable promise with his athleticism and aggressiveness in the paint although only netting 2.3 points, 2.1 rebounds and 0.1 assists per game.
After missing the entire 2005-06 pre-season play with a groin injury and playing one game with Toronto in the regular season, he was assigned to the Arkansas RimRockers of the D-League, where he wowed fans by posting a 40-point performance against the Fort Worth Flyers. In a game against the Austin Toros, Sow scored an outstanding double-double of 27 points and a D-League season-high 24 rebounds.
On January 18, 2006, Sow was called up to the Raptors by then-general manager Rob Babcock. Initially, head coach Sam Mitchell said in a radio interview that "I have no minutes for Pape Sow. We have no injuries. He's going to practice with us, he's going to be with the team. But I have no minutes for (Sow). However, due to a shoulder injury to center Rafael Araujo (who also suffered from poor play prior to the injury), Sow was able to get regular playing time.
On the first day of the 2006 Summer League season, Sow broke a vertebra during practice after driving to the hoop and falling to the ground while colliding with a teammate. Sow returned to action on January 22, 2007, playing the final 3 minutes and scoring 3 points in a game against the Charlotte Bobcats.