During World War II, Austin, Texas did not have a formal magic club. Magicians living in Austin often traveled to San Antonio for magic gatherings. One day Yerger came up with the idea of a Mother's Day afternoon picnic in Austin on May 9, 1943 and he invited the San Antonio group to attend. The Sunday gathering was a tremendous success. This led Herman Yerger to plan and host  three subsequent gatherings on the Labor Day weekends of 1944, 1945, and 1946. On property he purchased near Onion Creek in Austin, Yerger erected a club house, an open-air stage, and a barbecue pit.

At the 1946 picnic, magicians in attendance proposed that Yerger should allow the annual picnics to be formalized into a state magician's organization. Yerger was opposed at first but eventually warmed up to the idea especially after it was proposed that he be the first president!

According to the first TAOM newsletter, the Texas Association of Magicians came into existence in 1946 at the third annual picnic held at Yerger’s lodge in Austin, Texas. The second business session of TAOM was called to order at the Stephen F. Austin Hotel on Sunday September 1, 1946. The 1946 gathering is officially recognized by the TAOM Board of Directors as the first convention of the Texas Association of Magicians.