CDs are fine, but if I can get the music I’m looking for at a thrift shop for $2, $1, 50¢, and sometimes even 20¢, then I will take advantage of the situation. The LPs at thrift shops vary widely, and those in bigger cities are often picked over. Of course, I’m partial to St. Vincent de Paul stores because they are Catholic-owned. My interest in classical music has provided the opportunity to look for LPs that others consider "trash."
Another source is Great Lakes Records, a store in Milwaukee that has literally hundreds of thousands of records, and they keep the prices fairly low. I’ve purchased some real gems there, including James Swittel’s Hymns to the Blessed Virgin Mary and John McCormack sings Sacred Music. They have every genre you can imagine.