?
Relative Position Biases for Transformer PINNs
Transformer-based physics-informed neural networks (PINNs) have recently improved PDE solving by modeling spatiotemporal interactions with self-attention, yet most variants still rely on absolute coordinate embeddings. We ask whether relative positional structure - a key inductive bias in modern Transformers can be made coordinate-aware and yield an accuracy boost with minimal overhead for PDE PINNs. Building on a PINNsFormer-style decoder-only baseline (SPformer), we introduce two drop-in modifications: (i) a spatial-distance ALiBi attention bias and (ii) spatial RoPE applied to attention queries/keys using normalized physical coordinates. Across multiple PDE benchmarks and random seeds, both relative schemes consistently improve over the baseline, with spatial RoPE providing the strongest gains in our experiments. Our results suggest that injecting coordinate-relative structure into attention is a simple and effective upgrade for Transformer PINNs.