Shang-Fu Chen

I am a machine learning researcher working on generative models and their real-world applications. I received my Ph.D. from the Graduate Institute of Communication Engineering at National Taiwan University in 2025, advised by Prof. Shao-Hua Sun.

Most recently, I was a Member of Technical Staff at Spuree, an AI startup focused on 3D animation, where I built generative models and video understanding pipelines for character animation and delivered them as cloud APIs. During my Ph.D., I interned with the Deep Generative Model Team at Sony AI and with Inventec AI Center, and earlier worked in the Vision and Learning Lab at NTU in collaboration with Chunghwa Telecom Laboratories.

sam.sfchen@gmail.com

CV Google Scholar

profile photo

News

Research

My research focuses on generative models and their applications in real-world scenarios, spanning image generation, robot learning, and character animation. Recent topics include fine-tuning diffusion models with human feedback, imitation learning from imperfect demonstrations, and generative models and video understanding for 3D animation. Earlier in my Ph.D., I worked on computer vision problems such as anomaly detection, face anti-spoofing, multi-label classification, and representation disentanglement.

Publications

Restoring Noisy Demonstration for Imitation Learning With Diffusion Models

Shang-Fu Chen*, Co Yong*, Shao-Hua Sun

IEEE Transactions on Neural Networks and Learning Systems (TNNLS) 2025

Expert demonstrations often contain imperfections caused by human errors or sensor/control inaccuracies, which most imitation learning methods cannot handle. We propose a filter-and-restore framework that first filters clean samples from noisy demonstrations and then learns conditional diffusion models to recover the noisy ones. Our framework consistently outperforms existing methods across robot arm manipulation, dexterous manipulation, and locomotion tasks, and remains robust to different noise types and levels.

[ Paper ]

HERO: Human-Feedback Efficient Reinforcement Learning for Online Diffusion Model Finetuning

Ayano Hiranaka*, Shang-Fu Chen*, Chieh-Hsin Lai*, Dongjun Kim, Naoki Murata, Takashi Shibuya, Wei-Hsiang Liao, Shao-Hua Sun†, Yuki Mitsufuji†

International Conference on Learning Representations (ICLR) 2025

To effectively and efficiently utilize human feedback, we develop a framework, HERO, which leverages online human feedback collected on the fly during model learning. Specifically, HERO features two key mechanisms: (1) an online training method that captures human feedback and provides informative learning signals for fine-tuning, and (2) generating images from SD's refined initialization samples, enabling faster convergence towards the evaluator's intent.

[ Paper ] [ Project Page ]

Diffusion Model-Augmented Behavioral Cloning

Shang-Fu Chen*, Hsiang-Chun Wang*, Ming-Hao Hsu, Chun-Mao Lai, Shao-Hua Sun

International Conference on Machine Learning (ICML) 2024

This work aims to augment BC by employing diffusion models for modeling expert behaviors and designing a learning objective that leverages learned diffusion models to guide policy learning. To this end, we propose an imitation learning framework that benefits from modeling both the conditional and joint probability of the expert distribution. Our proposed diffusion model-augmented behavioral cloning (DBC) employs a diffusion model trained to model expert behaviors and learns a policy to optimize both the BC loss (conditional) and our proposed diffusion model loss (joint). Our proposed method outperforms baselines or achieves competitive performance in various continuous control domains, including navigation, robot arm manipulation, dexterous manipulation, and locomotion.

[ Paper ] [ Project Page ] [ Poster ]

Domain-Generalized Textured Surface Anomaly Detection

Shang-Fu Chen, Yu-Min Liu, Chia-Ching Lin, Trista Pei-Chun Chen, Yu-Chiang Frank Wang

IEEE International Conference on Multimedia and Expo (ICME) 2022

In this paper, we address the task of domain-generalized textured surface anomaly detection. We propose a patch-based meta-learning model that exhibits promising generalization ability. By observing normal and abnormal surface data across multiple source domains, our model can generalize to an unseen textured surface of interest and localize abnormal regions in the query images. Our experiments verify that our model performs favorably against state-of-the-art anomaly detection and domain generalization approaches in various settings.

[ Paper ]

Learning Facial Liveness Representation for Domain Generalized Face Anti-spoofing

Zih-Ching Chen*, Lin-Hsi Tsao*, Chin-Lun Fu*, Shang-Fu Chen, Yu-Chiang Frank Wang

IEEE International Conference on Multimedia and Expo (ICME) 2022

This work aims to apply domain generalization and feature disentanglement for the face anti-spoofing (FAS) problem, which aims at distinguishing face spoof attacks from authentic ones. We propose a learning frame to disentangle facial liveness representation from the irrelevant ones (i.e., facial content and image domain features). The resulting liveness representation exhibits sufficient domain invariant properties and thus can be applied for performing domain-generalized FAS. Our experiments verify that our model performs favorably against state-of-the-art approaches on five benchmark datasets with various settings.

[ Paper ]

Representation Decomposition For Image Manipulation And Beyond

Shang-Fu Chen, Jia-Wei Yan, Ya-Fan Su, Yu-Chiang Frank Wang

IEEE International Conference on Image Processing (ICIP) 2021

This work aims to apply feature disentanglement on existing/trained generative models. To this end, we propose a decomposition-GAN (dec-GAN), which can decompose an existing latent representation into content and attribute features. Guided by the classifier pre-trained on the attributes of interest, our dec-GAN decomposes the attributes of interest from the latent representation, while data recovery and feature consistency objectives enforce the learning of our proposed method. Our experiments on multiple image datasets confirm the effectiveness and robustness of our dec-GAN over recent representation disentanglement models.

[ Paper ]

Learning Hierarchical Self-Attention for Video Summarization

Yen-Ting Liu, Yu-Jhe Li, Fu-En Yang, Shang-Fu Chen, Yu-Chiang Frank Wang

IEEE International Conference on Image Processing (ICIP) 2019

We propose a novel framework named Hierarchical Multi-Attention Network (H-MAN), which comprises a shot-level reconstruction model and a multi-head attention model. Our attention model adopts a two-stage hierarchical structure to produce various attention maps, and we are among the first to utilize the multi-attention mechanism in the video summarization task, which brings improved performance.

[ Paper ]

Order-Free RNN with Visual Attention for Multi-Label Classification

Shang-Fu Chen*, Yi-Chen Chen*, Chih-Kuan Yeh, Yu-Chiang Frank Wang

The AAAI Conference on Artificial Intelligence (AAAI) 2018

We propose a recurrent neural network (RNN) based model for image multi-label classification. Our model integrates the learning of visual attention and Long Short-Term Memory (LSTM) layers. The LSTM module learns the labels of interest and their co-occurrences, while the attention module captures the associated image regions. Unlike existing approaches, training our model does not require pre-defined label orders. We introduce a robust inference process to address the prediction error propagation problem. Our experiments on NUS-WISE and MS-COCO datasets confirm the design of our network and its effectiveness in solving multi-label classification problems.

[ Paper ]

* Equal contribution. † Equal advising.