mrays/s back

(L) [2015/01/06] [tby szellmann] [mrays/s] Wayback!

Hi there,
first of all: great board - thanks a lot.
I wonder what you think is a fair way to evaluate the mrays/s metric you find in many publications. Basically, what I would expect is an atomic counter somehow associated with the BVH/accelerator, and then measure wall-clock time. But then, when I read e.g. Aila/Laine 09, they leave shading or ray sorting out of the equation, so they measure differently. Other authors obviously don't consider wall-clock time either. So I wonder how you guys measure and what I have to consider during my quest to beat the 500mays/s+ some of you report  [SMILEY :D]  Do you e.g. only measure the time a certain kernel (GPGPU case) takes to execute, say the one for traversal, and leave out others, such as those for ray gen, direct lighting, etc.? This would of course imply that your code is structured in such a way. What do you think is a fair approach - for the sake of comparability - to evaluate this metric?

(L) [2015/01/07] [tby Dade] [mrays/s] Wayback!

Frankly, I have seen reported the most astronomical numbers about the supposed performance of some algorithm described in papers. Astronomical numbers that you than discover to require some once-in-1000-years moons alignment to be achieved  [SMILEY ;)]
Short version: I totally ignore the reported numbers and I look into them only for some comparison between tests reported in the same paper (i.e. I consider them meaningful in a "local" context but useless out of the scope of the paper).

(L) [2015/01/07] [tby szellmann] [mrays/s] Wayback!

So you basically find the metric impractical but for the "local" case. Those slides ([LINK https://software.intel.com/sites/default/files/managed/b8/3f/Embree_pt_1_Building_Ray_Tracing_apps_woop.pdf]) e.g. compare Embree and OptiX based on mrays/s. I wonder how much faith one should put in those figures. I mean if there were an agreed upon way to perform the measurement... mrays/s sounds more fair to me than e.g. fps. But then you'd probably like to rule out bias (such as shading).
It's just.. you prepare your algorithms, possibly for papers, so your figures get actually published.. and if you'd like to compare with the state of the art, you never even know if the others outperform you by a factor of 10x or a factor of 2x  [SMILEY :D]

(L) [2015/01/07] [tby mpeterson] [mrays/s] Wayback!

papers/research are most often about a new single component. how they perform in a more
complex system/situation is open. the same here: optix is not the most optimized gpu ray tracing system
and embree for shure not the fastest cpu rt framework. you can download both and compare directly
with your own measurement-technics of choice. with that you can have a 1:1 comparison of everything.
in optix/embree there is nothing fancy so you should get similar numbers.

(L) [2015/01/07] [tby Dade] [mrays/s] Wayback!

>> szellmann wrote:So you basically find the metric impractical but for the "local" case. Those slides ([LINK https://software.intel.com/sites/default/files/managed/b8/3f/Embree_pt_1_Building_Ray_Tracing_apps_woop.pdf]) e.g. compare Embree and OptiX based on mrays/s.
I really like Embree, I have recently introduced Embree in LuxRender and it is 40-50% faster of the previous hand-written SSE QBVH code ... however they have used a scene originally designed in LuxRender as demo/test scene so I did some comparison in the past: in my tests, a AMD HD7970 with an simple OpenCL stack-less BVH outperforms the mighty Embree on a liquid cooled i7-3930K@4.5GHz by a lot. I would be surprised if Embree (on CPU) can outperform OptiX (on GPU). Xeon Phi may be a chapter on its own but I have the feeling it is not very much used in this sector at the moment.
 >> szellmann wrote:It's just.. you prepare your algorithms, possibly for papers, so your figures get actually published.. and if you'd like to compare with the state of the art, you never even know if the others outperform you by a factor of 10x or a factor of 2x  
In my opinion, you can but you have to replicate all the tests on your own and with your renderer: you can not use numbers taken by other papers.
For instance, for me: OptiX is not an option because of CUDA, Embree is faster in C++ and my code is better for OpenCL. Your results, even if using the same OptiX/Embree, may come out different because of the different context/hardware/etc.