Original Method
For Generative Adversarial Networks (GANs), we aim to learn a generator function \( g : Z \to X \), where \( z \sim P_Z \) is a random latent vector, and \( g(z) \sim P_g \) represents generated data samples.
The discriminator function \( D : X \to \mathbb{R} \) maps input images to a scalar representing the probability of being real. It is trained to distinguish between:
- Real samples \( x \sim P_{\text{data}} \)
- Fake samples \( g(z) \sim P_g \)
The objective of the GAN is formulated as the following minimax problem:
\[ \min_G \max_D \; \mathbb{E}_{x \sim P_{\text{data}}} [\log D(x)] + \mathbb{E}_{z \sim P_Z} [\log (1 - D(G(z)))] \]
This can be interpreted as:
- Optimizing \( D \): Maximize \( D(x) \approx 1 \) for real data and \( D(G(z)) \approx 0 \) for generated data
- Optimizing \( G \): Make \( D(G(z)) \approx 1 \), i.e., fool the discriminator
Then, we can compute the distribution \(D(x)\) by
\[ D^*(x) = \arg\max_D \mathbb{E}_{x \sim P_{\text{data}}} [\log D(x)] + \mathbb{E}_{z \sim P_Z} [\log (1 - D(G(z)))] \]
\[ D^*(x) = \frac{P_{\text{data}} (x) }{P_{\text{data}} (x) + P_G (x)} \]
Plug in the optimial discriminator \(D^*(x)\) into the objective function, we have:\[ \mathbb{E} [\log \frac{P_{\text{data}} (x) }{P_{\text{data}} (x) + P_G (x)}] + \mathbb{E} [\log \frac{P_G (x)}{P_{\text{data}} (x) + P_G (x)}] = 2\text{JSD}(P_{\text{data} \mid \mid P_G}) - \log(4) \]
Where \( \text{JSD} \) is the Jensen-Shannon Divergence. Therefore the loss objective function become
\[ \min_G \max_D \; \mathbb{E}_{x \sim P_{\text{data}}} [\log D(x)] + \mathbb{E}_{z \sim P_Z} [\log (1 - D(G(z)))] \sim \min_G \text{JSD}(P_{\text{data}} \mid \mid P_G) \]
GAN is training a generator G to minimize the distance between \(P_{\text{data}}\) and \(P_G\)
χ2 and WGAN
For χ2 GAN, we use different divergence metric to evaluate the distance between the generator and discriminator distirbution:\[ \chi^2 (P \mid \mid Q) = \int f(\frac{p(x)}{q(x)} q(x)) dx = \max_{T:X \to \mathbb{R}} \mathbb{E}_{x \sim P} [T(x)] - \mathbb{E}_{x \sim Q} [f^*(T(x))] \]
Where \(f^*(x)\) is the Fenchel conjugate of \(f\).
For WGAN, we use Wasserstein distance to measure the distance between the generator and discriminator distirbution:
\[ W(P, Q) = \sup_{T \in Lip_1} \mathbb{E}_{x \sim P} [T(x)] - \mathbb{E}_{x \sim Q} [T(x)] \]
Where:\[ Lip_1 = \{T: X \to \mathbb{R} \mid \|\nabla T\| \le 1\} \]
Then the loss object function become:\[ \min_G W(P_{\text{data}}, P_G) = \min_G \max_T \mathbb{E}_{x \sim P_{\text{data}}} [T(x)] - \mathbb{E}_{z \sim P_Z} [T(G(z))] \]
Our Method
We try to combine the above mentioned two method. We interpret the constraint \( \|\nabla T\| \le 1 \) as a domain restriction over the function class \(T\). Then we apply this constrain to the orginal χ2 GAN. Which can be written in this explicit form:
\[ \min_{G} \max_{T \in \text{Lip}_1(\mathcal{X})} \mathbb{E}_{x \sim P_{\text{data}}} [T(x)] - \mathbb{E}_{z \sim P_Z} \left[ \frac{T(G(z))^2}{4} + T(G(z)) \right] \]
KKT Interpretation
We can view this as an optimization problem under constraint, where a Lagrange multiplier term is added via the KKT framework:
\[ \min_G \max_T \; \mathbb{E}_{x \sim P_{\text{data}}} [T(x)] - \mathbb{E}_{z \sim P_Z} \left[ \frac{T(G(z))^2}{4} + T(G(z)) \right] - \lambda_{\text{gp}} \, \mathbb{E}_{\hat{x} \sim \mathbb{P}_{\text{interp}}} \left[ \left( \left\| \nabla_{\hat{x}} T(\hat{x}) \right\|_2 - 1 \right)^2 \right] \]
Here, \( \hat{x} \sim \mathbb{P}_{\text{interp}} \) are points interpolated between real and generated samples, and \( \lambda_{\text{gp}} \) controls the strength of the Lipschitz penalty. Here is the result:
Conclusion
This formulation provides a principled way to enforce 1-Lipschitz constraints while optimizing the generator and discriminator jointly in adversarial learning frameworks.