Training a fair classifier is essential for designing a trustworthy machine learning system. This talk will begin with an overview of the existing model fairness techniques and our new solution that overcomes their limitations. Then, I will show that the current approaches are highly vulnerable to data poisoning attacks, potentially resulting in even more unfair classifiers.