According to Portmore and the description of the Malfoy family in the book, the family made the right choice at the critical moment.
Especially in the book, on the eve of Hogwarts War, Narcissa did not hesitate to be symmetrical with Voldemort in order to get news of Malfoy. Harry died. Before the decisive battle, Lucius quickly evacuated with his wife and children after confirming his son's safety, and did not help Voldemort to participate in the final decisive battle. All these show that the malfoy family is not an extreme supporter of racial movements, and the most important thing in their hearts is family and glory.
The Malfoys are indeed Voldemort's supporters, but they finally gave up supporting Voldemort, because in the eyes of the Malfoys, Voldemort's side is far stronger than the original just side, and at the same time, under the standards set by Voldemort, the Malfoys are the superior beings in the wizard family.
Later, Voldemort was defeated, and Lucius didn't want to help Voldemort make a comeback, because Lucius thought Voldemort was no longer strong, so he couldn't be regarded as a loyal follower. Later, Voldemort came to power again and chose to support Voldemort.