Half precision floating point support.

[mkimi-lightricks]

System information (version)

• OpenCV => OpenCV iOS 3.1.
• Operating System / Platform => MacOS Sierra (v 10.12.3), iMac
• Compiler => Apple LLVM version 8.0.0 (clang-800.0.42.1)

Detailed description

1. Do you plan to support half precision floating point type?
   If yes, when?
2. Would you accept a PR which add half float support?

Steps to reproduce

None.

[alalek]

You can take a look on this PR: #6447 (changes are merged)
FP16 values are stored as CV_16S (signed 16-bit integer (short)).

There are no plans to add FP16 as cv::Mat native type or to add support for some basic operations (like add/sub/mul/div). PR with such changes will be not accepted.

[mkimi-lightricks]

Thanks.

[vpisarev]

@mkimi-lightricks, @alalek,

If we get a time machine and go back by ~10 years, I would add CV_16F as another regular data type, supported by CvMat and cv::Mat.

But even now we can probably squeeze it in with a little bit of effort. We use the 3 lower bits of CvMat/cv::Mat type field to represent the data type. Those are:

0 - CV_8U
1 - CV_8S
2 - CV_16U
3 - CV_16S
4 - CV_32S
5 - CV_32F
6 - CV_64F
7 - CV_PTR, see below ...

As you can see, we occupied all the possible values. And increasing the number of bits per depth from 3 to 4 or more is not feasible - there is too much code that may use it. But there is still workaround available.
Formally we use depth=7 (CV_PTR) to represent size_t, i.e. unsigned (or signed) integer of the same size as a pointer (32-bit on 32-bit systems, 64-bit on 64-bit ones). But we use it in such way very rarely. I think, it's still possible to override depth=7 as CV_16F.

Suppose that we've overridden depth=7 as CV_16F. What's next? C++ does not even know about such a type, as far as know. So we'd need to do the following:

1. Define the scalar type (float16? float16_t? half_t? half? ....)
2. Implement scalar operations (via conversion to/from float): +,-,*,/,abs(),min(),max().
3. Implement all the basic functionality of cv::Mat: copyTo(), setTo(), convertTo() (@alalek has already mentioned that).
4. Add support for CV_16F to cv::FileStorage (I/O to/from XML/YAML/JSON)
5. Implement basic arithmetic operations on Mat's containing 16f.
6. (maybe) Implement some image processing operations - threshold, filter2D, sepFilter2D, resize, cvtColor, ...
7. implement OpenCL kernels for some of the corresponding functions.

It's a big project. I think, we could do 1-4, stop and wait for subsequent contributions. Now the question - who will do it? :)

[alalek]

There are still no effective implementations for arithmetic operations for almost all platforms (no support on x86 - conversions only, ARM has alternative floats). Current FP16 area is special accelerators (power efficient), and usage on CPUs is very limited.

Also there is alternative to FP16 - 16-bit fixed point numbers, like Q8.7. Both of approaches are interesting, but probably they are useful for specific tasks only.

[vpisarev]

I believe, the main point here is to have some standard, "official" container for such data. Then people can add their own algorithms working on such data, including obvious scenarios when Mat is converted to CV_32F, processed and converted back to CV_16F if needed. Also, Intel, NVidia and AMD GPUs already have hardware FP16 support. There are AVX intrinsics to convert between half and float, so any more or less complex operation can do conversion on-fly, and the performance will be comparable with CV_32F: a few more conversion operations vs 2x smaller memory traffic.

Half is definitely more convenient to use than Q8.7, you still have a decent representation range: +/-65504, and at the same time can represent (normalized) values as little as 6*10^-5. With Q8.7 data you'd often need to supply the scale factor that would map your data to this quite narrow range.

[StatusReport]

@vpisarev let me shed some light on why @mkimi-lightricks and our team requires half-float support, and how we're currently made OpenCV to "work" with it.

We have an engine based on OpenGL running on iOS. On these devices the GPU's and the CPU's memory is shared, so it's zero-copy to map GPU memory to CPU and work directly on GPU textures. Our CPU matrix data type is cv::Mat, and we use it for various image processing and CV operations. While most of the algorithms can be implemented on the GPU easily, there are times where an entire algorithm or just a part of it can be implemented more efficiently on the CPU. To do this, we map an OpenGL texture to the GPU, wrap it with a cv::Mat and use our own algorithms or OpenCV's to do the manipulations we require, then render the result using the GPU.

Since OpenGL ES (3.0) supports half-float out-of-the-box (and not float), we require support on the CPU side to that type as well. Additionally, since the data stored in these matrices is usually HDR images we do not require wider data types as the amount of RAM on these devices is fairly limited. Therefore, converting a 16MP image from half-float to float is a huge waste of memory and computation power we do not have.

Our current solution is a hack - we defined a type CV_16F which is aliased to CV_16S (since we do not use it at all), in the following manner:

#define CV_16F CV_16S

#define CV_16FC1 CV_MAKETYPE(CV_16F, 1)
#define CV_16FC2 CV_MAKETYPE(CV_16F, 2)
#define CV_16FC3 CV_MAKETYPE(CV_16F, 3)
#define CV_16FC4 CV_MAKETYPE(CV_16F, 4)
#define CV_16FC(n) CV_MAKETYPE(CV_16F, (n))

namespace cv {
  typedef Vec<half_float::half, 2> Vec2hf;
  typedef Vec<half_float::half, 3> Vec3hf;
  typedef Vec<half_float::half, 4> Vec4hf;

  typedef Mat_<half_float::half> Mat1hf;
  typedef Mat_<Vec2hf> Mat2hf;
  typedef Mat_<Vec3hf> Mat3hf;
  typedef Mat_<Vec4hf> Mat4hf;

  template<>
  class DataDepth<half_float::half> {
  public:
    enum {
      value = CV_16F,
      fmt = (int)'r'
    };
  };

  template<>
  class DataType<half_float::half> {
  public:
    typedef half_float::half value_type;
    typedef value_type work_type;
    typedef value_type channel_type;
    typedef value_type vec_type;
    enum {
      generic_type = 0,
      depth = DataDepth<channel_type>::value,
      channels = 1,
      fmt = DataDepth<channel_type>::fmt,
      type = CV_MAKETYPE(depth, channels)
    };
  };
}

The underlying data type is half_float::half of the half library.

Obviously, almost all OpenCV operations doesn't work, but the cv::Mat as container is working, and element access works as in any other type. We thought about overriding CV_USRTYPE1 (=7), but as you said, it's a lot of work to add all the operators and we didn't want to fork OpenCV and write our custom additions, and that's why @mkimi-lightricks opened the issue.

[alalek]

There was an idea of creation of special "fp16" module which will handle operation on half-float type (probably via cv::Mat CV_16S).
With possible dual conversions most of these functions may be inefficient, so probably this module is not for building production-ready pipelines.

But at least such functionality would be useful for testing or prototypes (for example, it may detect "overflows" in some simulation mode).

Also such approach would not explode size of OpenCV binaries with "unused" functionality.

[mkimi-lightricks]

@vpisarev implementing the steps 1, ...,4 will be a great start.

How do you suggest to proceed from here?

[borisfom]

Let me add some background here: 16-bit floats are about to appear in C++20 (and C, at some point) as new "short float" fundamental type:
http://www.open-std.org/jtc1/sc22/wg14/www/docs/n2016.pdf
@mkimi-lightricks, @alalek - do you think we should get this time machine going by now ? :)
I am currently looking for one crucial piece missing in cudacodec::decoder:
alternative functions to produce float32/float16 frames instead of 8-bits RGBA in opencv/modules/cudacodec/src/cuda/nv12_to_rgb.cu: videoDecPostProcessFrame
Did anyone come up with those? I could use them right now.

[borisfom]

@alalek : efficient FP16 GPU implementations go way back. Most professional photo and video formats use it. Once the fundamental type appears in the standard, OpenCV would have no choice but extend for it - better to start early!

[borisfom]

@vpisarev, @mkimi-lightricks: as a main driver for "short float" proposal in ANSI C++, I would definitely volunteer to do the steps 1-4. Whoever has implemented any bits for FP16 support, please send those to me. Our company would be 100% behind it, as you might guess, but I am not authorized to speak for it :)

[borisfom]

@StatusReport: I do not see any opencv forks on your profile - if you care to share any fp16 code you guys produced by now, I would appreciate that and would incorporate with all proper credits into (hopefully happening) PR for steps 1-4!

[borisfom]

@mkimi-lightricks: I do not see any public forks of opencv in your profile either - I would be happy to consider your PR into what may become step 1-4 implementation.