I just got this back from him:
"While it's possible it's related to chlorine, it's also possible you're dealing with an immune reaction to the bacteria in that particular pool. It is very common for swimming pools to be colonized with different types of bacteria, including "atypical mycobacteria" that can set off an immune reaction in the lungs, causing pneumonia like symptoms.
The delay in symptoms (started 2 hours after starting swimming) seems more suggestive of this than a problem due to chemical irritation from chlorine.
You could try swimming in a different chlorinated pool to experiment to see if it's the chlorine or something else."
I think it kind of makes sense. And maybe also explains why I was able to swim in the pool for a few weeks before it started killing me.
I'm actually relieved to know that maybe not all chlorine pools are off limits. But, since I don't have to, I'm not interested in trying one any time soon. Also his theory says that the LA Fitness pool is fine for me, since I swam in it's bacteria daily for so many years without issue. So, whew.