Duffy left Parnes in 1961, suffering from depression. However, he was introduced by a friend to the growing London blues club scene, and in 1963 teamed up with Graham Bond, Jack Bruce, Ginger Baker and John McLaughlin to record "I Saw Her Standing There", one of the first cover versions of a Beatles song. He continued to record for Parlophone Records through the 1960s, both as a solo artist, often backed by top session musicians, and with Alexis Korner’s Blues Incorporated, but the critical acclaim for his performances failed to be matched by sales.
In 1973 he finally released a solo album, Duffy Power on the GSF label, produced in conjunction with Andrew Loog Oldham and featuring Korner, Dana Gillespie and others. At around the same time, an album of tracks recorded in 1969 was issued on the Spark label. Although by this time he was widely recognised as an impressive singer, the albums again failed to sell.
For some years his problems were aggravated by drug use and he succumbed to mental illness. In the late 1980s he re-emerged, and some of his unreleased recordings from the 1960s were issued on CD in 2002 as Leapers and Sleepers.