# Hiiiiiii, I am Ruyi (Maggie) Zhang

I am a PhD student in Computer Science and Technology at the Harbin Institute of Technology (HIT), supervised by Prof. Wangmeng Zuo. Before that, I spent six happy years at the University of Manchester, where I received a BSc in Artificial Intelligence (First Class Honours) and an MSc in Robotics (Distinction).

My research focuses on building multimodal models that understand, generate and generalize in open-world scenarios:

• Multimodal Large Language Models – fine-grained understanding of academic figures, tables and documents
• Semantic Alignment – aligning vision, audio and language across heterogeneous modalities
• Open-Scene Generalization – zero-shot transfer of LMMs to unseen visual modalities (thermal, depth, X-ray, …)
• 3D Scene Generation – controllable text-to-3D scene generation with structured layout priors

Multimodal LLM Semantic Alignment Open-World Generalization 3D Scene Generation Robotics

# News

• 2026 – Two papers accepted to ECCV 2026: VVM-Tuning (generalizing LMMs to versatile visual modalities) and ScrollScape (32K image generation with video diffusion priors).
• 2026 – First-author paper ControlScene accepted to CAAI Transactions.
• 2026 – Paper on task-uncertainty-aware video restoration accepted to IEEE TMM.
• 2025.03 – Started my PhD at the Harbin Institute of Technology.
• 2024.12 – Graduated from the University of Manchester with an MSc in Robotics (Distinction).

# Education

# Research Experience

Multimodal Intelligence Lab, Harbin Institute of Technology · PhD Researcher · 2025.03 – Present

• Leading research on multimodal understanding of academic literature. Designed the Count-Curriculum framework, which tackles weak gradients on hard samples and raises figure-matching accuracy by 23.3%.
• Proposed ControlScene, a structured-layout-guided 3D scene generation method, and built the LayoutVerse benchmark with 20,000 manually annotated samples.
• Exploring semantic alignment mechanisms of multimodal large models in open scenarios, forming an “understanding – generation – generalization” research line.

Robotics Lab, University of Manchester · MSc Researcher · 2022.09 – 2024.12

• Studied visual perception for underwater robots; developed diffusion-model-based image prior modeling that notably improves detection robustness in low-light and turbid water.
• Designed and optimized a Region-of-Interest Vision (RoIV) model with external attention for real-time object detection under limited compute.

Biological Machine Learning Project, MIT · Research Assistant (Remote) · 2021.06 – 2021.09

• The youngest member of the team, responsible for data preprocessing and feature engineering for deep learning on biological sequence data.
• Helped design a multi-task autoencoder for translating between scRNA and scATAC data; the project ranked first in the group.

Heilongjiang HIT AI Industry Co., Ltd. · Algorithm Engineer Intern · 2022.06 – 2022.09

• Optimized CNN architectures for image recognition via hyper-parameter tuning and model distillation.
• Fine-tuned NLP models for textual semantic understanding in industrial data analysis.

# Selected Projects

• Autonomous Robot Design & Assembly (MSc core project, 2023.09 – 2024.02) – Designed and built an autonomous robot integrating path planning and computer vision for navigation and precise robotic-arm manipulation in complex environments; reduced control latency to improve responsiveness in dynamic scenes.
• Comparative Analysis of Dimensionality Reduction (BSc dissertation) – Systematically evaluated PCA, LDA, t-SNE and UMAP on noisy high-dimensional data and showed UMAP’s advantage in preserving local structure. Awarded Best Undergraduate Dissertation.
• Hex AI Bot (2022.11) – Co-developed an evaluation strategy based on FastVC (virtual connection) search with heuristic rules and a swap-opening strategy, reaching about 75% win rate against other AI agents.
• More course projects are on the home page – chess, maze solver, snake, fuzzy-logic washing machine and more.

# Honours & Awards

• MSc with Distinction; BSc with First Class Honours; Best Undergraduate Dissertation
• Huawei Software Elite Challenge 2022 – National rank 71 (top 3%)
• 2020 Public Welfare Website Design Competition – Best Creativity Award
• PASS Leader, University of Manchester; Outstanding Undergraduate Student Representative

# Skills

–|--
Programming | Python, PyTorch, TensorFlow, C++, Bash, SQL, LaTeX
Multimodal / LLM | Qwen-VL, LLaVA, ImageBind, CLIP, BLIP-2, LoRA / QLoRA
3D / Graphics | 3D Gaussian Splatting, NeRF, Open3D, PyTorch3D
Tools | Linux, Git, Docker, Slurm, Weights & Biases, OpenCV
Languages | Chinese (native), English (fluent, IELTS 7.0, BSc & MSc taught in English)