This has nothing to do with feng shui, but has a lot to do with your moral quality. So I advise you not to do this, because you have a family. Your family should be more important. You can't divorce your wife for this woman. What would you do if your wife found out? What are the consequences? You should know her importance.
So don't take chances to do what you shouldn't do, you must be down to earth. Anyway, it is not easy to find someone who understands and loves you, so you must cherish it.