Seeing Before Synthesizing: VLM-Guided Transition Event Discovery for Weakly-Supervised Dense Video Captioning
This paper introduces 'Seeing Before Synthesizing' (SBS), a novel framework for weakly-supervised dense video captioning. It improves event localization and description by using Vision-Language Models (VLMs) to visually ground transition events between main actions, addressing limitations of previous LLM-only approaches.
Reading in Plain English mode — toggle below to switch
This paper introduces 'Seeing Before Synthesizing' (SBS), a novel framework for weakly-supervised dense video captioning. It improves event localization and description by using Vision-Language Models (VLMs) to visually ground transition events between main actions, addressing limitations of previous LLM-only approaches.
Imagine you have a function representing the semantic content of a video over time. Traditional methods might just label the 'peaks' (main events). SBS is like calculating the derivative of this function to find significant 'slopes' or 'inflection points' – these are the transitions between events, which are often overlooked but crucial for a complete description.
How SBS Works
- 11. Frame-level Narrative Generation: For the gaps between known events, a VLM generates detailed textual descriptions for each frame, capturing subtle visual changes.
- 22. Transition Detection: By analyzing the semantic variation across these frame-level narratives, SBS identifies where significant transitions occur, indicating a change in activity.
- 33. Temporal Mask Refinement: Once a transition is detected, its temporal boundaries are refined. This involves blending the initial temporal midpoint with the VLM-identified semantic change point and optimizing the duration to maximize vision-language alignment.
- 44. Dense Captioning: With more accurately localized main events and now grounded transitions, the system can generate more precise and comprehensive captions for the entire video.
ΔS = S(t+1) - S(t)
ΔS represents the semantic difference between consecutive frames or short segments. S(t) is the semantic representation of the video content at time t, derived from VLM narratives. A large ΔS indicates a potential transition.
In a video of someone cooking, previous methods might just caption 'chopping vegetables' then 'frying food'. SBS would identify the transition: 'moving chopped vegetables to pan', visually grounding this intermediate action, leading to a richer, more accurate video summary.
Visually grounding inter-event transitions with VLMs significantly improves the accuracy of dense video captioning, moving beyond rigid, ungrounded LLM-synthesized captions.