Optimizing Query Execution in Table QA
A CSE 576 (NLP) team project: a multimodal Table QA pipeline that splits each question by modality, filters images with CLIP, and prompts Gemini to generate SQL over a pruned table. It reported 73.94% accuracy in our course-project evaluation on MultiModalQA.
Neil Mahajan · Gokul Ramasamy · Mann Vora · Yi Xiao
Overview
Optimizing Query Execution in Table QA was a four-person team project for CSE 576 (Natural Language Processing) at Arizona State University. It addresses answering complex questions over semi-structured tables that mix text and images.
The pipeline decomposes each question into modality-specific sub-questions (ImageQ, TableQ, TextQ), filters images with CLIP embeddings, prunes irrelevant table rows, replaces images with captions, and prompts Gemini to generate SQL over the pruned table.
It reported 73.94% accuracy in our course-project evaluation on MultiModalQA.
Features
Pipeline Stages
- Preprocessing: Identifies column datatypes, image columns, and extracts contextual text
- Question Decomposition: LLM decomposes questions into ImageQ, TableQ, and TextQ sub-questions
- Relevant Image Identification: CLIP joint embedding space filters top-k relevant images
- Caption Generation & Table Pruning: Images replaced with captions conditioned on ImageQ; irrelevant rows removed
- SQL Query & Answer Extraction: Structured table passed to LLM for SQL generation and final answer
Screenshots
Proposed Architecture
Question Decomposition Tree Diagram