Instructions modified from here

Example shows unofficial image (with non-free firmware) but works the same with official.

This replaces 'splash quiet' or 'quiet splash' with 'persistence ' - note space at end.

$ LANG=C sed 's/splash quiet/persistence /;s/quiet splash/persistence /' \
< debian-live-11.6.0-amd64-standard+nonfree.iso \
> debian-live-11.6.0-amd64-standard+nonfree-persist.iso

Copy to USB drive

dd oflag=direct conv=fsync bs=8M status=progress if=debian-live-11.6.0-amd64-standard+nonfree-persist.iso of=/dev/<usbdrive>

Create new partition for the persistence (will occupy the rest of the drive)

fdisk --wipe never /dev/<usbdrive> <<<$'n\np\n\n\n\nw'

Create filesystem on new partition and configure persistence

mkfs.ext4 -m 0 -L persistence /dev/<usbdrive>3
mount /dev/<usbdrive>3 /mnt
echo '/ union' > /mnt/persistence.conf
umount /mnt

For automatic start add a timeout for grub

mount /dev/<usbdrive>2 /mnt
echo 'set timeout=5' >> /mnt/efi/debian/grub.cfg
umount /mnt

You may want to configure journald.conf to use Storage=volatile and possibly disable rsyslogd.
So that you limit the amount of writes to the USB drive.