I changed this setting in the UI, but was still getting YUV sent to my display.
echo 1 > /sys/class/amhdmitx/amhdmitx0/output_rgb does result in RGB output.
Unrelated to the GUI switch not working, the RGB that is output is limited range (16-235). ISTR reading here that full range was supposed to be output. Is there any setting that controls this?