NeurIPS 2026

GramStatTexNet Efficient, Interpretable, and Neuro-Inspired Texture Analysis-by-Synthesis

Vasha DuTell1,2 Zeyu Yun2 Anne Harrington1 Michael Y-S. Fang4 Mark Hamilton1 Christian Kovesdi1 Edward H. Adelson1 William T. Freeman1 Ruth Rosenholtz1,3
1MIT CSAIL 2UC Berkeley 3NVIDIA 4Spectacular Labs
Grid of texture syntheses. Columns are DTD categories (chequered, striped, polka-dotted, honeycombed, cracked, lacelike, fibrous, bumpy, sprinkled, pleated, swirly); rows compare the target with GSTN-zpad, GSTN-wrap, muNCA, Portilla and Simoncelli, Ustyuzhaninov, and Gatys.
Texture syntheses across DTD categories. Top row: target textures. GSTN matches the Gramian of multi-scale Gabor filters using 29K statistics, compared with 8K hand-picked statistics for Portilla & Simoncelli and 177K–852K for the deep and random-filter Gram methods (Gatys, Ustyuzhaninov). GSTN-zpad uses zero padding to avoid wrap-around artifacts; GSTN-wrap uses a wrapped Fourier transform and runs about 2× faster.

Abstract

Deep networks achieve high realism in texture synthesis but often rely on opaque, over-parameterized representations that have limited direct connection to biological vision. Neuroscience-informed pyramid models are interpretable, compact, and biologically inspired, but rely on small, hand-curated statistics sets that are difficult to generalize or extend. We introduce GramStatTexNet, a hybrid analysis-by-synthesis framework that combines the multi-scale Gabor filter structure of classical models with the flexibility of Gramian-based correlations, organized into structured filter families. This structured representation enables analyses that are challenging for deep-feature methods: each statistic carries a named filter-family identity, allowing us to decompose synthesis fidelity into per-family contributions and to learn an interpretable, low-dimensional embedding that preserves perceptual texture content. We compare our framework to classical and deep texture-synthesis baselines across diverse texture categories, and on multiple perceptual metrics, highlighting the compactness and interpretability advantages of this hybrid representation, alongside competitive synthesis quality. This same framework extends naturally to peripheral textures via spatial pooling, and to dynamic textures using a spatiotemporal filter bank. GramStatTexNet provides a unified, interpretable framework for analyzing and modeling visual information with natural extensions across space, time, and eccentricity.

Method

GSTN convolves an image with a bank of multi-scale pyramid filters at every combination of orientation, phase, scale, and color. Each filtered output is treated as a channel, and the pairwise correlations between all channels form a Gram matrix. Its upper triangle (28,929 statistics) is the texture representation. New textures are synthesized by optimizing a noise image until its statistics match the target's.

Diagram. Left: analysis-by-synthesis pipeline from target texture through pyramid filters, pyramid images and the Gram matrix upper triangle, with synthesis by backpropagation to a noise seed. Right: feature importance module in which texture crops pass through analysis and a fully connected layer trained with contrastive learning to a reduced representation.
Left: the analysis-by-synthesis pipeline. Right: to learn a reduced statistics set, a single fully connected layer compresses the Gram upper triangle, trained with a contrastive loss so that crops from the same texture land close together.

Compact, with competitive quality

GSTN improves on the classical Portilla & Simoncelli pyramid model in most texture categories across all nine evaluation metrics. Against Gatys, GSTN wins on classical perceptual and structural metrics (FSIM, SSIM, VIF, GMSD, HaarPSI, PSD-L2) in 32–46 of 47 categories with a 30× smaller representation. Gatys leads on the deep-feature metrics (DISTS, LPIPS), which are built on the same VGG features its loss optimizes.

28,929
statistics in the full GSTN representation
30×
fewer statistics than Gatys et al. (VGG-19 Gram)
83%
fewer statistics still give acceptable syntheses (5,000 top-ranked)
>50%
compression relative to the input image for foveated synthesis
Five line plots of synthesis quality versus number of statistics for LPIPS, MS-SSIM, HaarPSI, FSIM and SSIM, comparing GSTN variants and depletion curves with P&S, Gatys and Ustyuzhaninov.
Representation size vs. synthesis quality on 141 DTD validation textures. GSTN-zpad sits on the Pareto frontier between P&S and the much larger feature-space methods. Depletion variants (top-|w|, random, bottom-|w|) extend the frontier toward smaller representations.

An interpretable representation

Every GSTN statistic is a correlation between two known filters, so the representation can be read from first principles. We group the statistics into families by which filter attributes (scale, orientation, color, phase) are shared and which differ. Structured families, where only one attribute varies, roughly match the hand-curated statistics of classical pyramid models. Unstructured families correlate filters that differ in several attributes at once.

Family fingerprints of texture categories

Dendrogram of DTD categories with exemplar thumbnails above two heatmaps: per-category mean and per-category standard deviation of z-scored statistics for each family.
Hierarchical clustering of per-category family signatures recovers a structured-to-smooth ordering without supervision. Single families map onto recognizable properties: marginal for chequered, sub_magn_Xmulti for lined and banded, lowpass for frilly and veined, and highpass for crosshatched.

Which families matter for synthesis?

Grid of chequered, bumpy and pleated syntheses with each statistics family dropped (top) or kept alone (bottom).
DROP (each family removed) and KEEP ONLY (each family alone). No single family is sufficient. The structured families analogous to classical models (n=1,758) keep only coarse texture identity. Most synthesis-relevant information is in the cross-attribute families (pass_multi, sub_Xmulti), which classical pyramid models do not include.

Learning which statistics matter

A single 28,929×100 linear layer trained with InfoNCE groups crops from the same texture. Ranking statistics by their learned weights gives an importance ordering: the top 5,000 statistics are enough for acceptable syntheses, and the bottom-ranked ones are not. Randomly chosen subsets of the same size do even better for synthesis. The contrastive task picks statistics that discriminate between textures, while synthesis needs statistics that broadly constrain appearance. This is the classic distinction between discriminative and generative objectives.

Syntheses of banded, cracked, polka-dotted and bumpy textures using the top, random, or bottom N statistics for N from 10,000 down to 100.
Stat depletion. Syntheses using only N of the 28,929 statistics, chosen as the top-N by learned importance, a random N, or the bottom-N.

Beyond static, foveal textures

Peripheral (foveated) synthesis

Applying GSTN over spatial pooling regions that grow with eccentricity gives foveated syntheses like those used in peripheral-vision research. The center of gaze is rendered at high fidelity, with distortions increasing toward the edges. With the reduced 5,000-statistic set, the whole synthesis uses fewer values than the original image.

Original photo of daffodils behind a chain-link fence, a foveated synthesis using the full Gram statistics, and one using the reduced 5,000-statistic set, with zoomed insets.
Original (1.2M values) vs. foveated syntheses from the full model (28,929 statistics × 115 pooling regions) and the reduced model (5,000 × 115 = 575K values).

Dynamic textures

Classical pyramid statistics have no obvious counterparts in time, but a Gram matrix needs no hand-picked statistics. Swapping the spatial filters for a bank of 3D spatiotemporal Gabor filters is enough for GSTN to synthesize video textures, using the same pipeline as for images.

Space-time slices for grass, fire and tree videos comparing the target, GSTN and DyNCA.
Space–time (x–t) slices: one row of pixels across all frames. GSTN slices vary over time much as the targets do, including fire's flicker bands. DyNCA's grass and tree slices are dominated by vertical streaks, meaning little change over time.
Mean over 5 DynTex videosGSTNDyNCA
Deep-feature appearance
LPIPS-Alex ↓0.3230.159
LPIPS-VGG ↓0.5150.323
DISTS ↓0.3240.199
FID ↓16283
CMMD ↓2.4411.339
Pixel-level
SSIM ↑0.1140.107
MSE ↓0.0440.044
PSNR ↑14.214.1
Temporal
Temporal spectrum distance ↓0.1760.518
|ΔCurvature| ↓12.517.8
|ΔDiversity| ↓0.0650.037

DyNCA leads on deep-feature appearance metrics and diversity; GSTN leads on temporal power-spectrum match and perceptual straightness. We don't claim to beat dedicated dynamic-texture models. The point is that the same pyramid + correlation framework extends to video unchanged.

BibTeX

@inproceedings{dutell2026gramstattexnet,
  title     = {GramStatTexNet: Efficient, Interpretable, and Neuro-Inspired
               Texture Analysis-by-Synthesis},
  author    = {DuTell, Vasha and Yun, Zeyu and Harrington, Anne and
               Fang, Michael Y-S. and Hamilton, Mark and Kovesdi, Christian and
               Adelson, Edward H. and Freeman, William T. and Rosenholtz, Ruth},
  booktitle = {Advances in Neural Information Processing Systems},
  year      = {2026}
}

Acknowledgments

This work was funded by the CSAIL MEnTorEd Opportunities in Research (METEOR) Fellowship, US National Science Foundation grant 1955219, and NSF grant BCS-1826757 to PI Rosenholtz. The authors acknowledge the MIT SuperCloud and Lincoln Laboratory Supercomputing Center for providing HPC resources.