Towards Explainability of the Latent Space by Disentangled Representation Learning

Ivars Namatēvs, Artūrs Ņikuļins, Anda Slaidiņa, Laura Neimane, Oskars Radziņš, Kaspars Sudars

Abstract


Deep neural networks are widely used in computer vision for image classification, segmentation and generation. They are also often criticised as “black boxes” because their decision-making process is often not interpretable by humans. However, learning explainable representations that explicitly disentangle the underlying mechanisms that structure observational data is still a challenge. To further explore the latent space and achieve generic processing, we propose a pipeline for discovering the explainable directions in the latent space of generative models. Since the latent space contains semantically meaningful directions and can be explained, we propose a pipeline to fully resolve the representation of the latent space. It consists of a Dirichlet encoder, conditional deterministic diffusion, a group-swap and a latent traversal module. We believe that this study provides an insight into the advancement of research explaining the disentanglement of neural networks in the community.

Keywords:
Diffusion modelling; disentangled representation learning; explainability; latent space

Full Text:

 PDF

References


J. Egger, A. Pepe, C. Gsaxner, Y. Jin, J. Li, and R. Kern, “Deep learning – a first meta-survey of selected reviews across scientific disciplines, their commonalities, challenges and research impact”, PeerJ Computer Science, vol. 7, 2021, Art. no. e773. https://doi.org/10.7717/peerj-cs.773

Y. Bengio, A. Courville, and P. Vincent, “Representation learning: A review and new perspectives”, TPAMI, vol. 35, no. 8, pp. 1798–1828, Aug. 2013. https://doi.org/10.1109/TPAMI.2013.50

K. Ridgeway and M.C. Mozer, “Learning deep disentangled embeddings with the F-statistic loss”, in 32nd Conference on Neural Information Processing Systems (NeurIPS2018), Montréal, Canada, 2018, pp. 1–10. https://proceedings.neurips.cc/paper_files/paper/2018/file/2b24d495052a8ce66358eb576b8912c8-Paper.pdf

R. Suter, D. Miladinovic, B. Schölkopf, and S. Bauer, “Robustly disentangled causal mechanisms: Validating deep representations for interventional robustness,” in International Conference on Machine Learning, PMLR, vol. 97, 2019, pp. 6056–6065. https://proceedings.mlr.press/v97/suter19a/suter19a.pdf

D. Friede, C. Reimers, H. Stuckenschmidt, and M. Niepert, “Learning disentangled discrete representations”, in Machine Learning and Knowledge Discovery in Databases: Research Track. ECML PKDD 2023. Lecture Notes in Computer Science, D. Koutra, C. Plant, M. Gomez Rodriguez, E. Baralis, and F. Bonchi, Eds., vol. 14172. Springer, Cham. https://doi.org/10.1007/978-3-031-43421-1_35

R. Quessard, T. D. Barrett, and W. R. Clements, “Learning group structure and disentangled representations of dynamical environments”, arXiv:2002.06991, 2020. https://doi.org/10.48550/arXiv.2002.06991

M. Cerrato, A.V. Coronel, M. Köppel, A. Segner, R. Esposito, and S. Kramer, “Fair interpretable representation learning with correction vectors”, arXiv:2202.03078v1, 2022. https://doi.org/10.48550/arXiv.2202.03078

C. Rudin, C. Chen, Z. Chen, H. Huang, L. Semenova, and C. Zhong, “Interpretable machine learning: Fundamental principles and 10 grand challenges”, arXiv:2103.11251v2, 2021. https://doi.org/10.48550/arXiv.2103.11251

F. Locatello, M. Tschannen, S. Bauer, G. Rätsch, B. Schölkopf, and O. Bachem, “Disentangling factors of variations using few labels,” in International Conference on Learning Representations (ICLR), 2020.

X. Liu, P. Sanchez, S. Thermos, A.Q. O’Neil, and S.A. Tsaftaris, “Learning disentangled representations in the imaging domain”, arXiv:2108.12043, 2021. https://doi.org/10.48550/arXiv.2108.12043

I. Higgins, D. Amos, D. Pfau, S. Racaniere, L. Matthey, D. Rezende, and A. Lerchner, “Towards a definition of disentangled representations”, arXiv:1812.02230, 2018. https://doi.org/10.48550/arXiv.1812.02230

D. P. Kingma and M. Welling, “Auto-encoding variational Bayes”, arXiv:1312.6114, 2013. https://doi.org/10.48550/arXiv.1312.6114

I. Higgins, L. Matthey, A. Pal, C. Burgess, X. Glorot, M. Botvinick, S. Mohamed, and A. Lerchner, “beta-VAE: Learning basic visual concepts with a constrained variational framework,” in International Conference on Learning Representations (ICLR), 2016. https://openreview.net/forum?id=Sy2fzU9gl

R. T. Chen, X. Li, R. B. Grosse, and D. K. Duvenaud, “Isolating sources of disentanglement in VAEs”, arXiv:1802.04942, 2018. https://doi.org/10.48550/arXiv.1802.04942

H. Kim and A. Mnih, “Disentangling by factorising”, arXiv:1802.05983, 2018. https://doi.org/10.48550/arXiv.1802.05983

X. Chen, Y. Duan, R. Houthooft, J. Schulman, I. Sutskever, and P. Abbeel, “InfoGAN: Interpretable representation learning by information maximizing generative adversarial nets,” in Proceedings of the 30th International Conference on Neural Information Processing Systems, 2016, pp. 2180–2188. https://proceedings.neurips.cc/paper_files/paper/2016/file/7c9d0b1f96aebd7b5eca8c3edaa19ebb-Paper.pdf

Z. Lin, K. Thekumparampil, G. Fanti, and S. Oh, “InfoGAN-CR and ModelCentrality: Self-supervised model training and selection for disentangling GANs”, arXiv:1906.06034, 2020. https://doi.org/10.48550/arXiv.1906.06034

T. Wang, Y. Zhang, Y. Fan, J. Wang, and Q. Chen, “High fidelity GAN inversion for image attribute editing”, in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, New Orleans, USA, Jun. 2022, pp. 11379–11388. https://doi.org/10.1109/CVPR52688.2022.01109

Z. Zhang, Z. Zhao, and Z. Lin, “Unsupervised representation learning from pre-trained diffusion probabilistic models”, in 36th Conference on Neural Information Processing Systems, 2022, pp. 1–14. https://openreview.net/pdf?id=IiCsx9KNVa0

K. Preechakul, N. Chatthee, S. Wizadwongsa, and S. Suwajanakorn, “Diffusion autoencoders: Toward a meaningful and decodable representation”, in IEEE Conference on Computer Vision and Pattern Recognition (CVPR), New Orleans, LA, USA, Jun. 2022. https://doi.org/10.1109/CVPR52688.2022.01036

X. Wang, H. Chen, S. Tang, Z. Wu, and W. Zhu, “Disentangled representation learning”, arXiv:2211.11695. 2023. https://doi.org/10.48550/arXiv.2211.11695

Y. Ge, S. Abu-El-Haija, G. Xin, and L. Itti, “Zero-shot synthesis with group-supervised learning”, in International Conference on Learning Representations (ICLR), 2021.

C. P. Burgess, I. Higgins, A. Pal, L. Matthey, N. Watters, G. Desjardins, and A. Lerchner, “Understanding disentangling in β-VAE”, arXiv:1804.03599, 2018. https://doi.org/10.48550/arXiv.1804.03599

H. Cheng, Y. Wang, H. Li, A. C. Kot, and B. Wen, “Disentangled feature representation for few-shot image classification,” arXiv:2109.12548, 2021. https://doi.org/10.48550/arXiv.2109.12548

S. Lee, S. Cho, and S. Im, “Dranet: Disentangling representation and adaptation networks for unsupervised cross-domain adaptation,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Nashville, TN, USA, Jun. 2021, pp. 15 252–15 261. https://doi.org/10.1109/CVPR46437.2021.01500

R. T. Chen, X. Li, R. Grosse, and D. Duvenaud, “Isolating sources of disentanglement in VAEs,” in Proceedings of the 32nd Int. Conference on Neural Information Processing Systems, 2019, pp. 2615–2625.

P. W. Koh, T. Nguyen, Y. S. Tang, S. Mussmann, E. Pierson, B. Kim, and P. Liang, “Concept bottleneck models”, in Proceedings of International Conference on Machine Learning (ICML), 2020, pp 5338–5348.

M. Losch, M. Fritz, and B. Schiele, “Interpretability beyond classification output: Semantic bottleneck networks”, arXiv:1907.10882, 2019. https://doi.org/10.48550/arXiv.1907.10882

L. Tran, X. Yin, and X. Liu, “Disentangled representation learning GAN for pose-invariant face recognition,” in Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR), Honolulu, HI, USA, Jul. 2017, pp. 1415–1424. https://doi.org/10.1109/CVPR.2017.141

Z. Li, J. V. Murkute, P. K. Gyawali, and L. Wang, “Progressive learning and disentanglement of hierarchical representations,” arXiv:2002.10549, 2020. https://doi.org/10.48550/arXiv.2002.10549

A. Ross and F. Doshi-Velez, “Benchmarks, algorithms, and metrics for hierarchical disentanglement,” in International Conference on Machine Learning, vol. 139, Jul. 2021, pp. 9084–9094. https://proceedings.mlr.press/v139/

L. Liu, J. Li, L. Niu, R. Xu, and L. Zhang, “Activity image-to-video retrieval by disentangling appearance and motion,” in Proc. AAAI Conf. Artif. Intell., 2021, pp. 1–9.

H. Chen, Y. Zhang, X.Wang, X. Duan, Y. Zhou, and W. Zhu, “DisenBooth: Disentangled parameter-efficient tuning for subject driven text-to-image generation,” arXiv:2305.03374, 2023. https://www.catalyzex.com/paper/arxiv:2305.03374

M. Yang, F. Liu, Z. Chen, X. Shen, J. Hao, and J. Wang, “Causal VAE: Disentangled representation learning via neural structural causal models,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Nashville, TN, USA, Jun. 2021, pp. 9593–9602. https://doi.org/10.1109/CVPR46437.2021.00947

X. Shen, F. Liu, H. Dong, Q. Lian, Z. Chen, and T. Zhang, “Disentangled generative causal representation learning,” arXiv:2010.02637, 2020.

J. Fragemann, L. Ardizzone, J. Egger, and J. Kleesiek, “Review of disentanglement approaches for medical applications”, in MICCAI MAD Workshop, 2022. https://arxiv.org/ftp/arxiv/papers/2203/2203.11132.pdf

M. Fan, C. Chen, C. Wang, J. Huang, “On the trustworthiness landscape of state-of-the-art generative models: A comprehensive survey”, arXiv:2307.16680, 2023. https://doi.org/10.48550/arXiv.2307.16680

I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio, “Generative adversarial networks”, Communications of the ACM, vol. 63, no. 11, pp. 139–144, Nov. 2020. https://doi.org/10.1145/3422622

D. J. Rezende, S. Mohamed, and D. Wierstra, “Stochastic backpropagation and approximate inference in deep generative models”, in International Conference on Machine Learning, vol. 32, no. 2, 2014, pp. 1278–1286. https://proceedings.mlr.press/v32/rezende14.html

L. Dinh, J. Sohl-Dickstein, and S. Bengio, “Density estimation using real NVP”, in International Conference on Learning Representations, 2017. [Online]. Available: https://arxiv.org/pdf/1605.08803.pdf

G. Papamakarios, E.T. Nalisnick, D.J. Rezende, S. Mohamed, and B. Lakshminarayanan, “Normalizing flows for probabilistic modeling and inference”, J. Mach. Learn. Res., vol. 22, no. 57, pp. 1–64, 2021. https://jmlr.org/papers/volume22/19-1028/19-1028.pdf

K. Preechakul, N. Chatthee, S. Wizadwongsa, and S. Suwajanakorn, “Diffusion autoencoder: Towards a meaningful and decodable representation”, arXiv:2111.12640, 2022. https://doi.org/10.48550/arXiv.2111.15640

J. Ho, A. Jain, and P. Abbeel, “Denoising diffusion probabilistic models”, in Proceedings of NeurIPS, vol. 33, Vancouver, Canada, 2020, pp. 6840–6851. https://proceedings.neurips.cc/paper/2020/file/4c5bcfec8584af0d967f1ab10179ca4b-Paper.pdf

P. Dhariwal and A. Nichol, “Diffusion models beat GANs on image synthesis”, Advances in Neural Information Processing Systems, vol. 34, 2021. https://openreview.net/pdf?id=AAWuCvzaVt

R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer, “High-resolution image synthesis with latent diffusion models,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, New Orleans, LA, USA, Jun. 2022, pp. 10 684–10 695. https://doi.org/10.1109/CVPR52688.2022.01042

J. Song, C. Meng, and S. Ermon, “Denoising diffusion implicit models”, in International Conference on Learning Representations, 2021. [Online]. Available: https://openreview.net/pdf?id=St1giarCHLP

A. Q. Nichol and P. Dhariwal, “Improved denoising diffusion probabilistic models,” in International Conference on Machine Learning, Montreal, QC, Canada, Oct. 2021, pp. 8162–8171. https://doi.org/10.1109/ICCV48922.2021.01410

Z. Chang, G.A. Koulieris, and H.P.H. Shum, “On the design of fundamentals of diffusion models: A survey”, arXiv:2306.04542v1, 2023. https://doi.org/10.48550/arXiv.2306.04542

A. Hyvärinen and E. Oja, “Independent component analysis: algorithms and applications”, Neural Networks, vol. 13, no. 4–5, pp. 411–430, May–Jun. 2000. https://doi.org/10.1016/S0893-6080(00)00026-5

I. T. Jolliffe and J. Cadima, “Principal component analysis: A review and recent developments”, Philos. Trans. A Math. Phys. Eng. Sci., vol. 374, no. 2065, Apr. 2016, Art. no. 20150202. https://doi.org/10.1098/rsta.2015.0202

H. Caselles-Dupré, M. Garcia Ortiz, and D. Filliat, “Symmetry-based disentangled representation learning requires interaction with environments”, Advances in Neural Information Processing Systems, pp. 4608–4617, Jan. 2019.

O. Oreshkov and C. Giarmatzi, “Causal and causally separable processes”, arXiv:1506.05449, 2016. https://doi.org/10.48550/arXiv.1506.05449

V. Thomas, J. Pondard, E. Bengio, M. Sarfati, P. Beaudoin, M.-J. Meurs, J. Pineau, D. Precup, and Y. Bengio, “Independently controllable factors”, arXiv:1708.01289, 2017. https://doi.org/10.48550/arXiv.1708.01289

A. Malinin and M. Gales, “Predictive uncertainty estimation via prior networks”, arXiv:1802.10501, 2018. https://doi.org/10.48550/arXiv.1802.10501

J. Gawlikowski, S. Saha, A. Kruspe, and X. X. Zhu, “An advanced Dirichlet prior network for out-of-distribution detection in remote sensing”, IEEE Transactions on Geoscience and Remote Sensing, vol. 60, Jan. 2022, Art. no. 5616819. https://doi.org/10.1109/TGRS.2022.3140324

W. Joo, W. Lee, S. Park, and I.-C. Moon, “Dirichlet variational autoencoder”, Pattern Recognition, vol. 107, 2020, Art. no. 107514. https://doi.org/10.1016/j.patcog.2020.107514

Y. Song, T. Anderson Keller, N. Sebe, and M. Welling, “Latent traversals in generative models as potential flows”, arXiv:2304.12944, 2023. https://doi.org/10.48550/arXiv.2304.12944

X. Ren, T. Yang, Y. Wang, and W. Zeng, “Learning disentangled representation by exploiting pretrained generative models: A contrastive learning view”, in ICLR, 2022.

A. Voynov and A. Babenko, “Unsupervised discovery of interpretable directions in the GAN latent space”, in Proceedings of the 37th International Conference on Machine Learning, vol. 119, pp. 9786–9796. https://proceedings.mlr.press/v119/voynov20a.html

A. Plumerault, H. L. Borgne, and C. Hudelot, “Controlling generative models with continuous factors of variations”, arXiv:2001.10238, 2020. https://doi.org/10.48550/arXiv.2001.10238

Y. Shi, X. Yang, Y. Wan, and X. Shen, “SemanticStyleGAN: Learning compositional generative priors for controllable image synthesis and editing”, in 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), New Orleans, LA, USA, 2022, pp. 11244–11254. https://doi.org/10.1109/CVPR52688.2022.01097

R. Harkness, A. F. Frangi, K. Zucker, and N. Ravikumar, “Learning disentangled representations for explainable chest X-ray classification using Dirichlet VAEs”, arXiv:2302.02979, 2023. https://doi.org/10.48550/arXiv.2303.02979




DOI: 10.7250/itms-2023-0006

Refbacks

  • There are currently no refbacks.


Copyright (c) 2023 Ivars Namatēvs, Kaspars Sudars, Artūrs Ņikuļins, Anda Slaidiņa, Laura Neimane, Oskars Radziņš, Edgars Edelmers

Creative Commons License
This work is licensed under a Creative Commons Attribution 4.0 International License.