11.4 Demonstration on particle cloud GANs

We now provide a practical demonstration of the efficacy of our proposed metrics in evaluating the high-performing generative model discussed previously in this chapter: MPGAN, GAPT, GAST, and iGAPT. The visual comparison between the iGAPT and MPGAN models has been shown in Figure 10.10, demonstrating the high performance, but also the difficulty in distinguishing visually between the two models and the real jets. This makes these models an effective test bench for our proposed metrics.

Figure 11.4 first shows correlation plots between FPD and FPND, KPD, and $W_1^M$ on 400 separate batches of 50,000 GAPT-generated jets. We observe an overall positive relationship between the metrics, as one might expect. FPD and KPD have the strongest correlation, likely because they are accessing similar information about the same set of input features. However, for low values, the correlation is weak between all metrics, indicating that these metrics are complementary in understanding different aspects of the model’s performance. As noted in Section 11.3, the correlation between FPD and FPND may improve if the former were to use a subset of lower-level particle features as well.

Next, FPD, KPD, $W_1^M$ scores, as well as the $W_1$ distance between the particle $p_T^{\mathrm{rel}}$ distributions, from the best-performing MPGAN, GAPT, GAST, and iGAPT models are shown in Table 11.3, respectively. Focusing on just the 30-particle gluon jets first, we observe that it is extremely difficult to either distinguish between the performance of the models or draw a conclusion for their viability as alternative simulators based only on visual inspection of the histograms or even the $W_1^M$ score. However, FPD provides crucial information in this regard clearly indicating that iGAPT is outperforming the other models, validating our physics-informed approach to its architecture. However, we see that the FPD scores are discrepant from the truth, indicating room for improvement.

Overall, we see from this experiment the value in employing a broad set of sensitive, interpretable metrics. Firstly, evaluators can identify specific points of failures in their models. In the case of iGAPT, we note that while its $W_1^M$, FPD and KPD scores are generally strong, it is consistently worse than MPGAN on $W_{1p}^{p_T^{\mathrm{rel}}}$, indicating difficulty in learning the particle-level features — likely because of the increased emphasis on high-level, jet, features in the IPAB architecture. Secondly, evaluators are also able to define clear, quantitative criteria for model selection for their downstream tasks: for example, if comparing different simulator options, they can simply choose the model with the lowest FPD score, or if validating a faster alternative to traditional, accurate simulations, they may wish to require all scores to be compatible (e.g., significances of $<2$) with the latter, or even with LHC data itself, before adopting the model.