The byte order of DSD_U32 was wrong from the start. The oldest bits must be in the MSB, not in the LSB, according to snd_pcm_format_descriptions in alsa-lib.