Inferotemporal Cortex Joins the Circuit Before the Code: Non-Serial Inter-Area Synergy in the Macaque Ventral Stream
Inferotemporal Cortex Joins the Circuit Before the Code: Non-Serial Inter-Area Synergy in the Macaque Ventral Stream
Ponnambalam, A. R.; Venkiteswaran Pottore, K.
AbstractThe ventral visual stream is widely modeled as a serial feedforward hierarchy in which V1, V4, and IT population codes develop sequentially during object recognition. We ask whether a second, concurrent coding mode exists---one organized not by anatomical order but by joint population structure across areas. Using Partial Information Decomposition applied to simultaneous multielectrode spiking recordings across all three areas at millisecond resolution---the first simultaneous three-area spiking PID analysis of the primate ventral stream---in two macaque monkeys viewing 25,000+ natural images, we decompose population coding into serial (unique per area) and synergistic (joint across areas) components at 5 ms resolution across five CNN target representations spanning low-level spatial features to high-level object identity. Three findings replicate across both animals and all five representations. First, synergistic inter-area coupling emerges before IT carries any unique object-related information---a dissociation of 15--65 ms that replicates in direction without exception across both animals---such that the joint population integrates before the apex encodes; moreover, V1--IT synergy persists for over 120 ms after V1's unique information reaches zero. Second, although V1$\leftrightarrow$IT and V1$\leftrightarrow$V4 coupling emerge simultaneously and rise in parallel, V1$\leftrightarrow$IT exhibits stronger peak synergy at mid-to-high-level targets in both animals, suggesting a dominant role for non-serial joint coding. Third, when V1 and V4 are treated as an integrated feedforward block, their synergistic coupling with IT emerges last across all tested conditions---the feedforward foundation is the final component to join the synergistic mode, not the first. Together, these results show that serial and synergistic population codes co-occur in the same recordings, overlap in time, but follow different organizational principles, Providing a new level of nuance in our understanding of the primate ventral stream and introducing concrete constraints for biologically grounded models of vision.