Getting 50 GB/S Back from the Apple Neural Engine

summary

A hardware bug in the Apple M3 Neural Engine's DMA engine causes DRAM throughput to crash from 60 GB/s to under 20 GB/s whenever weight transfers are exact multiples of 1 MiB. This occurs because a 14-bit prefetch ring counter overflows, causing the system to mistake a full transfer for an empty one and disable speculative prefetching. By splitting these transfers into smaller, non-1 MiB chunks, the prefetcher is bypassed, restoring full memory bandwidth and significantly increasing LLM token throughput.

key points

1. The throughput collapse is triggered specifically by transfer sizes that are integer multiples of 1 MiB per core, which aligns with a 14-bit line pointer wraparound (2^14 lines of 64 bytes each).
2. The performance drop is caused by a lack of speculative prefetching; when the distance between the read and end pointers is zero modulo 0x4000, the prefetcher stops issuing lookahead requests, forcing the system into a slow, serialized transfer mode.
3. Software-level workarounds, such as splitting a 1 MiB transfer into two 512 KiB tasks or padding the data, can bypass the RTL bug and increase token generation speeds for models like Llama 3.2 1B by over 2x.

who should read it

Machine learning engineers and compiler developers working with Apple Silicon who need to optimize LLM inference performance on the Neural Engine.

discussion mixed