Suggestion: distribute a standard look-up table for mapping from NTSC DVD colour gamut to rec.709

Feature Requests
1

This is a rather complex issue to explain if you’re not familiar with the background, and it gets particularly tricky because people use the term “colour space” to mean too many different things. In an attempt to clarify, I’m going to use the following not-necessarily-standard terms: when I say “colour decoding standard” I mean the formula used (in any given colour space) to map between YUV and RGB values; when I say “colour gamut”, I mean the range of possible colours that the colour space allows one to display, and, in particular, the position of the primary colours (red, green, and blue) on the CIE diagram. I’m also going to use the terms “PAL” and “NTSC” very imprecisely: “PAL” here means the type of DVD find in the UK (720x576i video, 50Hz, Region 2) and “NTSC” means the type found in the US (720x480i, 59.94Hz, Region 1).

High-definition (non-HDR) video uses the rec. 709 colour space. Rec.709 defines both a decoding standard and a gamut. DVD video, however, uses different colour spaces. Both “PAL” and “NTSC” DVDs use a colour decoding standard called rec.601, but they use two different colour gamuts: PAL DVDs use one called EBU, and NTSC DVDs use one called SMPTE-C.

So, a video value specified in a DVD rip consists of YUV values; compared with the same YUV values in a SDR blu ray rip, that doesn’t just specify a different amount of red: it specifies a different amount of a different red (and the same applies to blue and green).

When you send an upscaled image to the display, that has to be in rec.709 format, because that’s the only kind of hi-def signal the TV understands. So, as part of the video pipeline, it’s necessary to map from the DVD colour space to rec.709.

Last time I checked, Kodi was using a matrix calculation for mapping from one colour space to another. This correctly takes account of the difference in the colour decoding standards, but it doesn’t take account of the difference in the gamuts. You more or less get away with that for PAL DVDs, because the EBU and rec.709 gamuts are quite close; but for NTSC DVDs, this means the final image always has slightly over-saturated colours.

Realistically, the only way one can reliably map between colour gamuts is to use a look-up table. So, my suggestion is to distribute a standard Kodi look-up table with OSMC, whose function is to map correctly from SMPTE-C to rec.709 gamuts.

(The calculation/construction of such a table is beyond my abilities, I’m afraid. :frowning: )

4

What makes you think kodi/vero or your TV doesn’t do this automatically, based on resolution?

5

Well, the TV wouldn’t be able to do it if 480p/576p content has already been upscaled by Kodi …

I’ve run some tests on the output and when rec709 content is calibrated as good as I can get it then NTSC DVD rips do seem still to be ‘wrong’ …

7

Indeed, but @angry.sardine likes to avoid Kodi upscaling.

8

My setup can handle that based on resolution; but as established at considerable length on my Vero 4K+ it isn’t currently possible to output SD video at native resolution from Kodi Krypton; using Leia you can get uncorrupted native-resolution SD video if the aspect ratio is 16:9, but you can’t if it’s 4:3.

If it were actually possible to get native-resolution SD video out of OSMC (or native-Res SD 4:3 video, anyway) then this would be much less of an issue.

I’m sure Kodi isn’t handling it automatically because I have eyes - I can spot the over-saturated colours. Also because my Lumagen RadiancePro video processor is separately calibrated for rec.709 and rec.601/SMPTE-C and I can manually switch between the two if necessary. But a) I’d prefer not to have to switch manually, and b) most people who play NTSC DVD rips using OSMC don’t have that option, so they need a look-up table even if I don’t.

In the past the Kodi devs have been unwilling to acknowledge the problem - see, for example, this thread on the Kodi boards.

1 Like
12

I’m not sure I understand why that’s relevant. If a sink is going to interpret the colourspace based on resolution then it’s going to use only the lines (height) value to do it. And we’ve established you can get that right with whitelists in Leia (and by setting the GUI to an SD value in Krypton). The width scaling that you highlighted as an issue is irrelevant, surely?

As a start, I can check if Vero is sending the correct infoframe for SMPTE-C. I think @wesk05 said it is on that kodi thread. Apart from that, I don’t want to spend too much time catering to displays that don’t respect the HDMI standard.

13

So you’re saying that I should be forced to choose between correct colour gamut with loss of horizontal resolution, and full horizontal resolution but with incorrect colours? You don’t see a problem with that?

That raises an interesting question: if a TV receives an input signal where the resolution is 720p or higher but the infoframe tells it to use a colour gamut that would never normally be associated with that resolution, will the TV actually pay attention to the infoframe, or will the fact that it’s not an SD signal make it switch to the rec.709 gamut anyway? My guess is that the majority of TVs will ignore the infoframe under those conditions.

Also worth checking what happens on a Raspberry Pi. I’m not sure it’s even possible to specify colour space in the infoframe with RGB output (which is the default on Pi).

14

No, I’m saying the colourspace will be determined by the vertical resolution and/or the infoframe. You don’t have to choose.

Well I look forward to proof of that.

No, it’s not - has to be sRGB. Good point.

15

Although more work than an untouched DVD rip, you can use AviSynth to convert the color space from rec.601 to rec.709 (by changing from YUV to RGB, then back), then re-encode the video using H.264. Other than the slight loss in color accuracy, a one-pass, crf=20 encode will not lose any visible quality.

16

I’m sorry, you’ve lost me.

If the correct gamut can only be triggered by the vertical resolution being set to 480, and if setting the vertical resolution to 480 always results in horizontal downscaling, how do I get the correct gamut without horizontal downscaling?

Well, A: who the heck has the time for that? :rofl: and B: how sure are we that avisynth actually handles the difference between the colour gamuts correctly?

17

We’ve established that you can’t avoid horizontal scaling at the moment, full stop. the choice right now is therefore:

  • output at 480 with horizontal loss of resolution of 25% - display has no conflict between resolution and infoframe so should understand Rec 601 (525)
  • output at 720 or more - both axes scaled - conflict between resolution and infoframe. Display may choose Rec 601 or Rec 709 but ought to respect the infoframe.

Is it me, or is your default position something is wrong until it’s proved right?

3 Likes
18

How sure are you that any playback device handles them correctly?

The only way that I can see to test would be to have a DVD player with optional upscaling, play back the original disk both upscaled and not, and then use the same DVD player to play back an untouched rip (.mpg or .mkv container, so it doesn’t make any assumptions about the source being a DVD) both ways.

Then you need to check to see what’s in the YUV output, and verify that the color spaces are what you expect (rec.601 for non-upscaled, and rec.709 for upscaled). Then, if the color on the display is the same in all 4 cases, you know the player adjusts the gamut and colorspace correctly when upscaling, and you have a “golden” source and display.

Once you have that, you can test the Vero to see what it does differently. That’s a lot of work for a fairly small color difference on a very limited set of source videos.

Also, you would have to test RGB output, to see if the scaling takes place before or after conversion from YUV to RGB.

But, if the video is flagged with the right color information (x264 options colorprim, transfer, and colormatrix), it looks to me like the Vero does the right thing. I suspect that your DVD rips don’t have the right information, and the Vero has to guess. Maybe a different way of guessing would solve your problem, but it might break other video playback. Note this post, where you can see that some NTSC DVDs have bt470m and most have smpte170m matrix, so if the information gets stripped, the color can be wrong, since there are two “right” answers.

See this thread for some more details on how to make sure the video has the right information, and how AviSynth does its conversion.

20

We may have covered this (I know we requested a sample); but does this occur with 1:1 DVD-rip (i.e. ISO)?

21

Based on the mediainfo of a sampling of my untouched (1:1) DVD ISOs, some have color primary information in the MPEG stream, and some do not (you can check by running mediainfo on VTS_01_1.VOB, as this is usually the start of the main movie).

So, a single ISO that does show the issue doesn’t help pin down the cause without the mediainfo.

22

It’s a fairly large colour difference, seen on most American DVDs; that’s hardly “a very limited set”.

I’m sure this bothers different people to different degrees - perhaps I’m unusually sensitive to inaccuracies in colour saturation, I don’t know. But certainly the impact of this is a constant irritation if I don’t have things set correctly. There’s no chance at all I could play an American DVD with the colour space mapping done the way Kodi does it and not notice the problem.

If it turned out that no playback device handled this correctly, I wouldn’t be especially surprised. I haven’t come across one that does.

My own setup at home includes a Lumagen RadiancePro processor which solves the problem in precisely the way I’m suggesting - by using a LUT. That allows me to compare corrected and uncorrected versions of a video, and the difference is not subtle.

Generally, no; but when it comes to this specific issue - upscaling SD video in a way that doesn’t allow for differences in the colour primaries - that’s a fair statement. :slight_smile:

Now I’m curious - why might that make a difference?

23

You do understand that the video being rec.601 has nothing to do with the problem, yes…? (Sorry, I don’t mean to patronise, but I’ve had trouble explaining this in the past).

24

It has everything to do with the problem, because playback devices make assumptions about the colorspace and gamut based on information like that. In addition, as I noted, there are two different color matrix/gamut standards in use on NTSC DVDs, and if the material isn’t flagged correctly, then even a perfect playback device will show the wrong color.

If you convert the video from MPEG-2 to H.264 and make sure it is marked with the right flags, you shouldn’t have this problem. That’s the way all my DVDs get ripped/converted, and with the video data marked with the correct information (depending on NTSC or PAL DVD, and what I find in the original stream, to help with the NTSC confusion), playback color on the Vero exactly matches that from a non-upscaling DVD player.

You really need to get a non-upscaling DVD player as a source to verify what the color really should look like, since otherwise you are making assumptions that something is being done incorrectly.