In the Complete Works of Yangzhai, the most respected view of planting feng shui trees in front of the door is planting osmanthus trees. Through the yin and yang meridians of earthworms, we can see: "There are osmanthus trees in front of the door, and nobles often meet them. The moral is to take the word Gui, because Gui is homophonic with noble people. Moreover, osmanthus trees are fragrant, easy to plant and have strong ornamental value. The rest of the trees are unlucky.
As for your local saying, osmanthus trees can only be planted in front of temples. I have reservations about this, because the orthodox geomantic omen is sometimes contrary to the folk saying. After all, China is a vast country, and it is impossible to unify all cultures. But in the south, many rich people have osmanthus trees at their doorsteps.