Ousley began playing in the late 1940s, and in the 1950s recorded behind vocalists such as Billie Holiday and Dinah Washington.
He played as a sideman with Gene Ammons in the 1950s and with Jack McDuff and George Benson in the 1960s.
He released his first record as a leader in 1961. In the 1970s he played with Lionel Hampton and Count Basie in addition to releasing further material as a leader.
After 1977 he did not release another album under his own name until his most recent effort, 2000’s Grit-Grittin’ Feelin’.