Lee was traded to the Yokohama Taiyo Whales in 1982, and continued his hitting prowess there however he was suddenly fired in 1985. He was re-hired by the Yakult Swallows, and played from 1986-1987 before returning to the United States, and in total he had spent 10 solid seasons in Japan.
After retiring, he worked as a minor league manager for the New York Mets, and as a scout for the Chicago Cubs, before returning to Japan as a hitting coach for the Orix BlueWave in 2003. The manager, Hiromichi Ishige, was fired in April, and Lee was promoted to manager. On May 17, 2003, the BlueWave faced the Nippon Ham Fighters, managed by Trey Hillman, for a battle between two American managers in Japan for the first time in 28 years. His team was a good offensive team, but the pitching staff marked a torrid 5.95 team ERA, and ended up in last place with a 48-88-4 record. Lee left the club at the end of the season, declining an offer to serve as hitting coach. He was the first African-American manager in Japanese baseball history. Lee also served as a consultant in the making of the 1992 film, Mr. Baseball. The film, starring Tom Selleck, detailed life in the Japanese professional baseball league from the point of view of an American player, based on older brother Leron Lee's experience in the country.