Finally I read where someone with some authority has made this point for me:
Hence some people interpret symbolically the entire episode of paradise itself, where, according to the truthful account of holy Scripture, the first human beings, parents of the human race, dwelt, and they turn those trees and fruit-bearing plants into virtues and ways of life. They assume that those details were not visible and material objects but were described as such in speech or writing for purpose of illustrating symbolically spiritual realities.
How absurd to maintain that there could not have been a material paradise because it can be understood also in a spiritual sense; as if it were an argument that Abraham did not have two wives, Hagar and Sarah, and from them two sons, one by the slave and the other by a free woman, just because the Apostle says that in them the covenants were illustrated; or, again, that there was no rock from which water flowed forth when Moses struck it because it can also be interpreted as a symbol of Christ in that passage, for, in the words of the same Apostle, 'The Rock, moreover, was Christ.'
(Augustine, City of God, XIII, 21, trans. Philip Levine)
It's all of one fabric, friends, because the the whole business, the natural world, history, revelation and everything else, are spoken into existence through the one Word of God.