How to avoid washed-out colours - Limited and Full Range settings

grahamh · 4 March 2021 17:52

If the colours of your videos look washed out, and blacks aren’t really black, you may have issues with your ‘quantisation (or quantization) range’ settings. This guide is intended to help fix some edge cases that occur with some less common display devices.

If you are using a regular HD or 4K TV, you shouldn’t need to read this guide. With a regular TV issues would only occur if the setting on your OSMC device is not the default value of ‘limited range’. This setting is in Settings->System->Display on Vero and in config.txt on RPi (accessible through My OSMC). On Vero, you need to set Settings level to Expert to see the option.

If you have confirmed this in your display settings you should also check the video settings that are accessible (on Vero only) during video playback to make sure your brightness and contrast are set to their default values of 50%. Kodi will remember previous adjustments on a per-video basis. If adjustments were made and the “set default…” option was enabled this will make these adjustments pervasive across all videos played. If that hasn’t solved the problem or you are not using a regular TV, read on.

Before we begin, it’s a good idea to know whether the basic settings on your display (at least contrast and brightness) are set correctly. Download some test patterns, such as the ones here https://www.avsforum.com/threads/avs-hd-709-blu-ray-mp4-calibration.948496/ and follow the instructions for the ‘Basic Settings’. Make sure, though, that you are not correcting some bad choices on quantisation range using contrast and brightness settings. If after reading through this page you make any changes to your set-up, run the tests again.

Background

Video values are encoded into video files/streams as numbers from 0-255 (8-bit video) or 0-1023 (10-bit video).

For historic reasons most video content and all HDR10 video content doesn’t use all of the range available. They use ‘limited range’ in which 16 represents black and 235 represents white (64-940 for 10-bit content).

TVs will always support limited range video since that is a broadcast standard. Other displays may or may not support limited range. PC monitors are typically full-range only. TVs may have settings which make them behave more like PC monitors.

There’s a page on the Kodi wiki that explains the difference and what can go wrong, but it’s written for systems where Kodi can be set for full or limited range and the GPU can be set separately. OSMC doesn’t do this - the only settings are for the GPU. So you have only two choices to make: the setting(s) in your OSMC device and the setting(s) on your display. On RPi you need to set the options in config.txt via My OSMC or the command line and reboot. On Vero, the GPU settings are part of Kodi’s menus (Settings->System->Display). Vero may make some decisions about settings which are not reflected back to Kodi’s menu, for example, Vero will not send YCC to a display that doesn’t support YCC. It can’t make similar decisions on quantisation range because it can’t know for certain what the display supports.

What’s best? Limited or Full?

99.9% of video (100% of TV broadcast) is limited range and ‘YCC’ format. The pixels on your screen are however full-range colour so the conversion has to be done somewhere. Just like other conversions (resolution scaling and YCC-RGB) a TV/projector is probably going to be better at it than most external devices. Converting to full range in your OSMC device is certainly not going to improve anything. If your display accepts limited range, that’s what you should use.

Note that full-range is typically only available when outputting RGB (see below). OSMC always outputs RGB as 8-bit so you will not see HDR (10-bit) in the best quality if you use RGB full-range.

Unfortunately, equipment manufacturers use terms that imply full-range is better than limited-range, such as ‘normal/low’ and ‘enhanced/standard’. This is misleading. Sometimes you have an option like ‘super-white’ the purpose of which is unclear.

Does my display accept limited range?

If it’s a TV with a digital input, or any display that supports YCC, the answer is yes.

If it’s a PC monitor, the answer is probably not. If it supports video resolutions like 1280 x 720 then it may support limited range. Check the manual.

If it’s a projector, you need to check the manual. Threads on this forum reveal a wide range of behaviours for projectors.

If there’s an option in the display’s OSD menus to set it for limited range, then clearly it does support that! But read on.

I can’t find it in the manual. What can I do?

Summary of the standards HDMI and CTA-861:

if the display accepts YCC pixel encoding then it will expect limited range signals when using YCC. A display may also support full-range with YCC. The range would then be set by the source signal.
all displays must accept RGB and should default to full-range when receiving an ‘IT’ resolution (eg 1280 x 768), and limited range when receiving a video resolution (eg 1280 x 720). Again these defaults may be overridden by the source signal if the display offers that.

RPis follow the rules for IT (aka DMT or VESA) modes vs video (CEA or CTA) modes. Veros currently do not automatically set the quantisation range based on mode.

Sidenote: if you mix DMT and CEA modes in your whitelist you may see unexpected colour changes between different resolutions.

How should I set my display?

Bottom line: the display setting must match what your OSMC device is sending.

If your display supports YCC then it will support limited range and use that by default (there may be an option, see below). Just set your OSMC device to use YCC with limited range and you’re done. All displays automatically match the pixel encoding (ie RGB or YCC) of the source.

Sometimes there are manual controls, and these typically affect only RGB signals.

If there’s a setting on your display OSD menu to choose limited or full range, it should also have an ‘Auto’ option which lets the source tell the display what range to use. Use that if available. If there’s no Auto setting be aware there is no way for the display to tell the source what range it is set to. You must then make sure your OSMC device is using the same setting.

If you don’t see options in the display’s OSD menus for quantisation range then assume:

YCC is treated always as limited even if the OSMC device is sending full-range video. You must set your OSMC device to limited when using YCC.
RGB is treated according to the screen mode as above or follows the source as if ‘Auto’ was set.
Displays which accept only RGB are unlikely to support limited range.

To test how quantisation range is handled, just play a video with a wide range of colours with output set to RGB (requires a reboot) and limited range turned on and off in Kodi settings (config.txt for RPi). If you see no difference, the display does support limited range and is switching correctly. With YCC (if the display supports it) you should expect to see a difference because the display will likely not be switching range to match the signal.

But I don’t even know if my display supports YCC.

Displays tell video devices their capabilities with an ‘EDID’ transmitted through HDMI or DVI (or even VGA, but the chances of limited range support through VGA are negligible).

On Vero, you can read an EDID with the log uploader in My OSMC. Select ‘All Configs and Logs’ or just ‘Display Info’, upload the logs and follow the URL in a browser. Look for something like this:

====================== EDID =================== wE0go885
Rx Brand Name: MEI
Rx Product Name: Panasonic-TV
Manufacture Week: 0
Manufacture Year: 2008
Physical size(cm): 0 x 0
EDID Version: 1.3
EDID block number: 0x1
blk0 chksum: 0xfa
Source Physical Address[a.b.c.d]: 3.0.0.0
YCC support 0x03, VIC (native 4):
ColorDeepSupport 0xb8 10/12/16/Y444 1/1/0/1
19 4 31 16 20 5 32 18 3 17 2 22 7 21 6 1 
Audio {format, channel, freq, cce}
{1, 1, 0x07, 0x01}
Speaker Allocation: 0x00
Vendor: 0x000c03
MaxTMDSClock1 190 MHz
SCDC: 0
RR_Cap: 0
LTE_340M_Scramble: 0
  DeepColor
checkvalue: 0xfad50000

---------------------- EDID END --------------- wE0go885

YCC support 0x03 means this TV supports YCC 4:4:4 and 4:2:2. If you see 0x00 here, your display supports only RGB.

On RPi, you need to access a command line, then type:

tvservice -d edid
edidparser edid | grep 'monitor support'

You should see something like this:

HDMI:EDID monitor support - underscan IT formats:no, basic audio:yes, yuv444:yes, yuv422:yes, #native DTD:2

If you don’t see YCC (yuv) support in the EDID, set the output of your OSMC device to RGB.

Note 1: the EDID may change, depending on the settings in your display

Note 2: the EDID may not come through correctly if you are using a HDMI splitter or a special HDMI cable (eg to a projector).

My display doesn’t do YCC or I really want to use RGB - what settings should I use?

Use the test above to confirm if the display, for the resolution(s) you are going to be using:

switches range to match the range of the source
sets its range according to the resolution of the source, or
sets the range to full for all resolutions

For (1) set your device to output RGB and limited. For (2) on Vero use a whitelist that includes only CEA or only DMT modes but not both and set the range accordingly. Set RPi hdmi_pixel_encoding to default. If (3) set your device output to RGB and full.

I need to set my TV to RGB and full-range so I can use it with my games console.

Do you? Your display should switch automatically to YCC or RGB depending on the incoming signal. If your games console needs RGB full-range, that will be selected automatically as well if you follow the guide above and choose ‘Auto’ if available.

I still can’t get it to work

Please start a separate thread, describing what you’ve tried and post:

debug logs
make and model of your display (preferably a link to the manual)
make and model of any devices between the OSMC device and the display

angry.sardine · 5 March 2021 19:02

I have to quibble with that. For starters, there are plenty of displays out there nowadays which do not have an RGB sub-pixel structure. Pretty much all OLED TVs, for example, have RGBW sub-pixels, so there is no point anywhere in the signal chain that the picture is in RGB format.

Second, what do you mean when you say that the pixels “need full-range RGB”? Assuming an 8-bit signal they need to correctly display 220 possible levels of brightness, not 255 levels.

And, while you do go on to say that it’s probably better to let the television do the “conversion”, I don’t think you put enough emphasis on that. Most televisions do all of their processing (e.g. scaling) in YUV space; and it’s quite often the case that an RGB input signal will actually be internally converted to YUV for processing. Supplying a television with RGB will quite likely degrade the eventual picture quality.

Making statements like “the pixels need full-range RGB” makes it sound like, if only you can convert to RGB correctly at source, this will result in optimal image quality; and that is emphatically not true (unless you’re dealing with a PC monitor type display).

grahamh · 5 March 2021 19:38

Indeed. Without overly complicating the issue, the principle is we have to get from YCC to some mix of primaries and need to use the full range of luminances available from the pixels (LEDs, quantum dots or whatever). I’m using ‘RGB’ as shorthand for primaries.

How many times do we say use YCC and limited range?

But I specifically don’t recommend that.

I suspect we are both hampered by coming at it from a theoretical standpoint. If you go through the forum posts on that Kodi wiki article you will find reports of ‘better greyscale gradients’ with combinations of settings that shouldn’t be optimal. This thread was prompted by a user with a professional projector which supports only RGB full-range. It expanded a bit but my aim is to inform people so if they see things they don’t like the look of they will have some ideas as to why and what to change. I didn’t want to be dogmatic about what they should or should not do.

ManMower · 20 March 2021 02:41

Really appreciate you taking it upon yourself to try to reduce this topic to a couple of simple FAQs.

There’s at least one projector out there that has auto/enhanced/super white settings that apply while receiving YCC signals, so I think this may be an oversimplification that could result in confusion?

If this is true, why does the software allow it to be set “wrong”? It would seem to me that removing the option and forcing correct behaviour would be a good idea. Are you certain this statement is always true, thought? My projector allows interpretation of YCC signals as “enhanced”, so I’d assume there are other displays out there that have this setting.

As @angry.sardine mentioned, this is not strictly true. I think the display’s native format, be it RGB, pentile, or DLP mirrors and a spinning color wheel, is really its own business. But I do feel the current text is fairly clear that limited range, when available, is always the right answer, so maybe this oversimplification is forgivable for the sake of brevity.

I think there may be people with weirdo displays for which this isn’t true, so perhaps “I set limited range and now the video is all washed out, what gives?” might be a follow up question, but I wouldn’t know how to generically answer it.

grahamh · 20 March 2021 09:10

Thanks for the feedback. I don’t spend a lot of time on AV[S]forums trawling through this stuff and don’t have a projector (these always seem to be different) so it’s good to hear users’ experience.

I’m talking about pixel encoding (RGB/YCC), not quantisation range. There’s no mechanism in HDMI for sinks to report their set quantisation range to sources. Added an explanation.

Good point. This bullet is under ‘if you don’t see any options’ (hence the ‘must’ here). I don’t think we should prevent people playing with enhanced and super-white if it takes their fancy and those may need full-range to work. Who knows?

Quite. That’s what the forum is for.