- Am I Canadian?
- Ruthless, Spiraling Gradient Descent
- Breaking in a notebook.
- Culture smell.
- Selfish altruism and voting?
- WSL2 setup for nerfstudio
- Another draft from 2015 - language paradigms
- Politics, Variance and Techbros.
- Draft from 2015 on the future of real-time rendering and GPUs.
[Previous log file]
- The aesthetics of the intellectual elite. (permalink)
Last week was my birthday so we went to check out one of the recently minted Michelin star restaurants in Vancouver, Burdock.
The food was very well made - and narrated even better. Everything was part of a story, connected to the land and the seasons as it's "de rigueur" for a starred place. It's like touring a museum, the experience of the paintings alone is multiplied by knowing their history.
But what struck me immediately were the aesthetics. The servers were in casual, rustic attire. The whole place was evocative of a cabin, or a ship.
The dinnerware was especially interesting, with some of the dishes looking like the first project of a kid's art class, thick slabs of clay, roughly cut to an approximate disc. Throwing away literally a millennia or so of pottery making history.
And I don't doubt everything was expensive, we don't upcycle to save money, machines can nowadays produce perfectly thin, perfectly shaped porcelain, that is for the poor. We want to see the fingerprints, know there were human hands molding shapes and turning screws and so on. It's not even about saving the world, let's not kid ourselves that our global whoes are curtailed by a few rich people choosing to buy second-hand luxury to make a statement.
This aesthetic is everywhere, and often - as often happens with anything expensive, at the price of art. Rich kids snapping photos with film cameras of entirely mundane, uninspired, uninteresting subjects for their youtube channels. Even richer kids shooting Hollywood movies in ludicrously expensive 70mm film and vintage lenses, so that the audience can bask in the glory of blurry images.
Now, I don't want to stretch it - it's not that everything that is handmade is intrinsically bad. It's not even that there is any problem in being rich, or liking fine things in life. And it makes sense that we live in the culture we live in, these things are trendy as a reaction, it's a sign of times. But let's not live in the delusion that this is not storytelling for a particular elite, one that would look down at the "traditional" signifiers of luxury, and simply found new ones so we can pretend we're different. Because if you start buying into that... then you become the kind of leftist that makes Trump win. Yeah I went there, again :)
Tue, 18 Mar 2025 12:14:53 -0700
- Am I Canadian? (permalink)
In these times, the question of identity seems more relevant than usual.
I'm mostly Italian, there is no denying that. I don't feel much patriotism towards Canada - I don't follow its politics, I have never even voted here.
I came for my job, I met my wife here, it is a nice place - and after a while, you get a passport.
At the same time the only thing I can imagine "being" of a country means, besides the obvious bureaucratic implications, is a declaration of the context that shaped you culturally, shaped your way of thinking and being.
And in that case, I can't deny that Canada taught me many lessons, and changed my perspective on the world.
So yes. I guess I am also, and increasingly so, truly, Canadian.
Thu, 13 Mar 2025 11:27:41 -0700
- Ruthless, Spiraling Gradient Descent (permalink)
Some weeks ago, in one of these "watercooler" moments in the office, I was musing with a friend over how this is how superheroes look in real life.
Tony Stark, Batman... they are tools to tell a story, but if their "powers" were real, they would not manifest in people with Greek gods' physiques and a stoic moral compass; they would be grotesque, misshapen alien abominations, egotistical, childish, broken.
It's not devoid of beauty; looking at the fragility of the human condition, incapable of becoming the comic-book hero, twisting, deforming, contorting in the vain attempt of being something that should not exist. The cost of winning (their way).
Hollywood could not imagine it. Watchmen, Peacemaker, The Boys, Jessica Jones - "breaking the mold" no more; it's common to re-imagine superhero powers in a somewhat more humanistic vein, but nobody went far enough. And they all still look pretty.
Herbert got it right. The guild navigator, the god-emperor, the extreme hedonism of the Harkonnen; all his characters are physically (mis-)shapen by the extremes of their pursuits.
Then, a terrible thought.
What if all of this is not just inevitable, as it seems clear by the simple fact of not being exceptional - of being a widespread condition in the entirely "Western" democratic world today? What if it was the optimal adaptation to the current environment?
The local optima in a ruthless gradient descent.
Like the artist that sacrifices their bodies, affections, sanity, in the single-minded pursuit of their art - we see individuals with similar extreme traits, which happen in today's environment to maximize a given "fitness" function.
Politics in democracies have always required a more than healthy dose of egotism. My late father, himself having always dabbled in local politics and activism (and the proud owner of the kind of egos often found in people who achieve extraordinary feats in life, starting from poverty), decades ago postulated that as a requirement for success in politics.
Public service is like sainthood. We like to think it's for the selfless, the heroic, but that's not how humanity works. If selflessness exists, it is rare enough not to be something we can reliably construct a society over.
It's the Mother Teresa hypothesis: selfishness driven by a different kind of greed. In the saint's case, it was to buy the entry into the exalted ranks of heaven (Catholicism in particular very well understands human needs, using sticks and carrots). In the politician, it is to feed a need for power, for recognition.
Run under this assumption, and wait a few decades, sprinkle in the accelerant of immediate feedback provided by today's interactive, real-time mass media. It's evolution!
Now, don't mistake this for admiration. I value both extraordinary minds and hearts too much to respect these outcomes. And I have little empathy either. I reserve that to people with less than the most absolute power.
Neither is it acceptance. I would call it... understanding. After all, I never believed that "extraordinary" individuals shape society; they mostly emerge from a given society. "The hero we deserve".
Ironically (we are full of irony), we find more inspiring the idea of the exceptional single, rather than the fact that we all, in tiny but collective ways, shape the context for the individual. Communism is a doomed, stupid, inhuman idea. We'll never accept the power of being ants.
Believe it or not, I think all of the above, assuming there's truth there, should be viewed with optimism. Yes, I'm saying the current landscape is inevitable given the conditions, and I think when you see how repeatable it is, this has to be a given. You can look at an apple falling down only so many times before you accept gravity exists.
But conditions always change. And understanding is half the battle. One can't love fiction. Mature, grounded love lies in being able to look at our wounds, neither ignore them nor panic, but accept that they are part of being human. Acceptance is the only way to heal.
And I see the germ of understanding. Not yet widespread, but growing.
Obvious facts that we were not in the condition to accept. The failure of the global left, one of them.
My parents are a good example, I think. My father's family was poor. After the war, poverty was normal, of course, but for context. Starving, one set of clothes, handed down from older to younger, that sort of thing. A vivid picture I have is the tale of one of the older brothers, born crippled; most of his youth had no access to a wheelchair. He had to move by swinging with his body a wooden chair around the house. He became a professor, from parents that had little instruction. My mother never starved because they were farmers. Most of her (many) brothers kept tending the farm, creating different enterprises in different locations. Everyone else graduated from university and went on to be part of the "upper-middle class". Land was developed into housing; by the time she married, my mom owned multiple apartments, furnished hers with good, designer pieces.
Granted, even in the incredible times of post-war Italy, some of their stories would still be of relative rarity - I'm named after an uncle, on one side, that was the priest of the head church of our town, and one on the other side who led pioneering expeditions in North Africa and the Middle East, of which we who survived him don't even know but a fraction.
But as poverty was normal, extraordinary was too.
This is the generation that led to social revolutions. The sixties, women's rights, welfare, public healthcare, schools, workers' rights, minorities' rights. A generation of unbridled optimism, not just fuelled by economic progress, but scientific too. How can't you be in awe of technology when you go from not having antibiotics to living long enough that "old age" diseases are the top concern (incidentally, my father was a cardiologist)? When you go from not having a bathroom to the internet, from starvation to the dangers of hyper-abundance, from farmlands to coal-powered industries to atomic power and the rise of intellectual work?
Humanity doesn't change: the flesh and bones, the hearts and minds, they are pretty much constant. Remember, no heroes. Our defining characteristic we share is our adaptability: we shape ("like water") to our container.
The container today, in the "West", doesn't have the sharp slope of the explosive growth regime in the logistic curve, but the gentle curve of the "shoulder" past that.
I'll go personal again, to illustrate, trusting that in this case, the anecdotal happens to be a sample that well represents the general problem (reminds me of the mean value theorem for integrals). If I count (I should get better at that), I'm wealthy. And I probably always have been - at any point in my life, even inflation-adjusted, I've been wealthier than my parents, and somewhere in the top percentiles of the population.
Yet, I wake up by the screams of nearby party-goers, when there are not the pains of the growing homeless population. I walk to work avoiding the scents of origins I'd rather not speculate on. I never, realistically, wanted for nothing. I increased all my numbers by orders of magnitude, yet, my quality of life has been constant. I AM NOT COMPLAINING. Just a factual assessment of a life of privilege (and happiness, love, I need for nothing). This is your best-case scenario. Contentment.
Most everyone else saw a decline, in relative terms, of their conditions - generation over generation. Relative terms, of course, are the only ones that matter; remember, the fundamental theorem of humanity is that we are ultra-adaptive.
This is where you want to be, as a society. You want... not to be poor. Not to be uneducated. To go past the explosive growth that is possible only after a deep crisis.
Understanding is the first step.
I don't think this is the only power at play; never believe anyone that in a complex world, comes up with a single, universal law. But I do believe one solid fact is this: we have not grown more bigoted, more xenophobic; souls don't change. We fed lots of people less hope, and that changes a lot.
Without anger, it is what it is - we'll always be in pain; this is ours, now. It can be motivating to think that our predecessors were particularly terrible, in a Machiavellian sense; that's good if the end results are...
but it's still a tale.
Please (the global left), stop saying that you are just misunderstood, that you did everything great, just unable to explain, to tell people how great they are doing, to patronize them. Look at the wound. Find better numbers.
Numbers help only when they capture reality (and the part of reality that matters in context); otherwise, they are just "objectivity-washing". Like pseudo-science, using the language of science in the vain hope of gaining legitimacy.
Populism works only if you're intercepting the guts of most people, and evidently, in today's world, right-wing populism trumps the equivalent offering from the left. If you're not born a natural, you'll need to work for it. The work in politics, from the losing side is, if not to love, at least understand who doesn't like you.
Ideally, we figure a way to both keep growing - it's clearly still possible, and I'm not aware of ways to predict when and why the slow-down would be inevitable (yes, Earth resources are finite, but even without becoming interplanetary like certain morons peddle, we clearly figured how to add value many times over the simple economics of taking stuff out of the dirt).
And at the same time, we figure out how to make growth not result in the inequality that is a natural consequence of the mathematics of growth. And we do that not trying to work against human nature, but by showing - factually - that a less unequal society works best. It creates more economic growth, it creates more happiness, it's the better choice, not the moral imposition of a sliver of irrelevant, disconnected intellectual elites.
I'm a leftist who lives (not literally, today) in the Silicon Valley. The struggle is real, between the simplistic delusion of tech-bros that don't understand how childish the idea of applying the SV ethos to the world is, and the obnoxious "wokeism" of people who lost all abilities to talk to the "base" that dissolved under them. I should check my own egotism now. Bye.
Sat, 1 Mar 2025 16:59:16 -0800
- Breaking in a notebook. (permalink)
Like a ballerina preparing her pointe shoes, I have a very particular ritual when starting a new notebook.
1) I label the front and back.
In general, my notebooks will become all "customized" and drawn over, but at minimum, I put my name/nick and year on the front, and something like "bad ideas" on the back. I write from the back random/personal notes, and from the front work/research stuff. Especially as some notebooks do not come with a distinct back and front, this helps me opening them from the right end.
2) I glue the first and/or last page to the cover, creating a "pocket" section. This will hold stuff, usually, a small sheet of dot stickers that I use to mark certain pages I want to revisit later (typically, pages where I wrote blog ideas for articles).
3) I often spend a few pages "swatching" all my inks so I know how they look on the notebook paper, and it helps me choose them later on as I switch pens.
4) On the "bad ideas" side - I have a couple of things now I always begin with. I make a grid in the back for things I want to keep track of daily, like whether or not I exercised, or ate well, or dedicated time for creativity. I also make a page for stuff I'm considering buying, in various categories. It helps me put these things/thoughts down for a bit - call that page "lust and release".
Sat, 15 Feb 2025 13:10:01 -0800
- Culture smell. (permalink)
Engineers are often attuned or at least opinionated about "code smell".
That's great, but misses the mark, imho. Code smell is a localized problem, and per-se, could not be a problem at all - depending on if it's "infectious" or not (isolated code can "safely" be terrible) - and if it was done as an "investment" in the future (product velocity = money = more engineers, per se, might not be a bad thing) or if the debt is a burden...
But regardless of the kind of smell/debt, code is an outcome, it is not the root of the problem. Company culture is. And among the many ways a company can go wrong, there's none worse than misalignment on what "good looks like".
If a company is creating "bad code", taking debt either for good growth or not, it is at least knowingly doing something "bad". If engineers on the other hands, different teams, groups et al, do not have the same vision of what good looks like, then they will keep producing terrible things without guilt. With pride, even.
That's one of the most terrifying things. People doing their job, creating stuff, when what they create is wrong for the company. It's like a painting made by thousands of hands on a huge canvas, where nobody sees the output of anybody else, and there is no oversight and coordination.
Sat, 15 Feb 2025 12:56:27 -0800
- Selfish altruism and voting? (permalink)
Last time I was in the states, at a dinner with a young coworker and friend he confessed he almost never voted. Tried once, the process stressed him out, and his guilt of not participating in democracy was only subdued once he realized that for him - a pretty content person with not much to ask from the government - not voting was a way to give more power to others who might have stronger needs, more to say.
I didn't and don't fully buy this, it's also hard as my generation does value participative democracy, it's too near to times where it did not exist to resign to the idea it shouldn't be a right that is exercised vigorously.
But there is something there that makes me think - a line I don't know where to draw, but that should be drawn. Too often we join fights that are not ours - that we do not live. And this is dangerous for everyone involved, it's one of the causes of today's disaffection towards the global left - perceived (rightfully) as intellectual and elitist, it loses votes not just from the working class, that has long gone, but from all the minorities it wanted to protect.
In the same vein, subsidies very rarely work, reparations and international aid in the long term never does - it's almost as altruism is something that can only offer a patch in an acute crisis, in the short-term, but in the end, people have to feel empowered and fight for themselves.
I'd love to have a more developed theory. But currently, I dont.
Mon, 20 Jan 2025 10:29:57 -0800
- WSL2 setup for nerfstudio (permalink)
DOCS
- https://learn.microsoft.com/en-us/windows/ai/directml/gpu-cuda-in-wsl
- https://www.reddit.com/r/wsl2/comments/15vlnvu/do_i_still_need_to_install_cuda_toolkit/
- https://docs.nerf.studio/
-- https://gist.github.com/SharkWipf/0a3fc1be3ea88b0c9640db6ce15b44b9
# WSL2 / Hyper-V
wsl --update
wsl --install ubuntu
# Update apt and upgrade ubuntu, why not - and also let's install a decent editor
wsl
sudo apt-get update
sudo apt-get -y upgrade
# some utilities
sudo apt-get install micro
sudo apt-get install mc
# Where are my files? It seems that now wsl2 creates a virtual HD by default
# Windows hd can be accessed using /mnt/c
# e.g. C:\Users\apesce\AppData\Local\Packages\CanonicalGroupLimited.Ubuntu_79rhkp1fndgsc\LocalState\ext4.vhdx
# Windows can mount these
# Cuda for WSL
# Should make sure NVidia drivers are up to date etc...
# Lots of different Cuda versions etc - this is tricky - right now best is to stick with 11.8
# References:
# https://developer.nvidia.com/cuda-11-8-0-download-archive?target_os=Linux&target_arch=x86_64&Distribution=WSL-Ubuntu&target_version=2.0&target_type=deb_network
# https://docs.nvidia.com/cuda/wsl-user-guide/index.html
wget https://developer.download.nvidia.com/compute/cuda/repos/wsl-ubuntu/x86_64/cuda-keyring_1.0-1_all.deb
sudo dpkg -i cuda-keyring_1.0-1_all.deb
# Install only the toolkit, not full cuda
sudo apt-get -y install cuda-toolkit-11-8 build-essential
# btw, nerfstudio guide suggests using conda - conda install -c "nvidia/label/cuda-11.8.0" cuda-toolkit
# Add environment variables to .bashrc in $home (can use micro)
export CUDA_PATH="/usr/local/cuda-11.8"
export CUDA_HOME="$CUDA_PATH"
export PATH="$CUDA_PATH/bin:$PATH"
export LD_LIBRARY_PATH="/usr/local/cuda/lib64/stubs:$CUDA_PATH/lib64:/usr/lib/wsl/lib/:$LD_LIBRARY_PATH"
export LIBRARY_PATH="$LD_LIBRARY_PATH:$LIBRARY_PATH"
# Miniconda
cd $home
wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh
bash ./Miniconda3-latest-Linux-x86_64.sh
rm Miniconda3-latest-Linux-x86_64.sh
./miniconda3/bin/conda init
conda update conda
# Conda environment and all the dependencies...
# note: next time I should try to use pixi instead, which should replace all of the steps below https://docs.nerf.studio/quickstart/installation.html#using-pixi
conda create --name nerfstudio -y python=3.8
conda activate nerfstudio
python -m pip install --upgrade pip
# the official guide uses pip, but everytime I can use conda instead it's best to do so
# pip install torch==2.1.2+cu118 torchvision==0.16.2+cu118 --extra-index-url https://download.pytorch.org/whl/cu118
# note the use of the conda-forge channel:
conda install ffmpeg 'colmap<3.9' hloc 'pytorch<2.2' cudatoolkit=11.8 'opencv==4.10' -c conda-forge
# these seem to be required for splatfacto
# nope- conda install triton
# nope- pip install pytorch-triton --extra-index-url https://download.pytorch.org/whl/cu118
# BINGO:
pip install triton==2.2.0
pip install 'setuptools<70' # if you get some errors about 'packaging' in python scripts...
# Test cuda/pytorch in python:
>>> import torch
>>> torch.cuda.is_available()
# Build tinycudann
pip install ninja git+https://github.com/NVlabs/tiny-cuda-nn/#subdirectory=bindings/torch
# Nerfstudio (latest, from source)
git clone https://github.com/nerfstudio-project/nerfstudio
pip install --upgrade pip setuptools
# this step reads the pyproject.toml for dependencies, but we should already have most of them installed by now
pip install -e .
# if this errors on a package version (...can't uninstall) it's because the conda version is different than the one in the .toml, might need to tweak things...
# optional cli bindings, will show some errors, normal:
ns-install-cli
# NOTE: if it complains about libEGL.so it means we need to install freeglut:
sudo apt-get install freeglut3-dev
# Example use:
conda activate nerfstudio
# sample
ns-download-data nerfstudio --capture-name=poster
ns-train nerfacto --data data/nerfstudio/poster
# https://docs.nerf.studio/quickstart/custom_dataset.html
ns-process-data video --data some/video.mp4 --output-dir video_colmapped
ns-train splatfacto --data video_colmapped --output-dir trained
Fri, 17 Jan 2025 13:00:21 -0800
- Another draft from 2015 - language paradigms (permalink)
Again posted as I found it (raw notes, WIP...)
- Languages to learn paradigms.
De-empathize OOP. Functional and "DOP" are gaining popularity. But as CS we should know more...
Idea - List of languages good to learn a given paradigm and their learning resources + a simple project that would be fun to do with them. Most languages are multiparadigm, but certain are better suited to given ones and are great for learning a particular style.
These are all fairly generic languages, fairly current and supported. There are many more paradigms that are restricted to very peculiar applications, and languages that are very interesting for their historical contributions.
- Racket Scheme and derived (e.g. Typed Racket). Mathematica. Clojure.
- Lisp-family. Impure functional, symbolic, reflective.
- Metaprogramming. Homoiconic. Continuation.
- Design-by-contract (Clojure). Aspects (Racket). Futures/Promises (Racket). Term Rewriting (Mathematica).
- Resources: ...
- Project idea: Genetic Programming.
- OcaML, F#
- ML-family. Impure functional, type-inferring.
- Metaprogramming. OO
- Resources: ...
- Project idea: Path Tracer.
- SWI Prolog
- Prolog-family. Declarative, logic. Constraint Programming.
- Non-deterministic.
- Resources: ...
- Project idea: ...
- C
- Imperative, procedural.
- Resources: ...
- Project idea: Software Rasterizer.
- ChucK
- Livecoding. Synchronous Reactive Programming.
- DirectCompute... Cuda, Shadertoy (GLSL), ISPC
- Stream processing
- TCL
- String based. Scripting.
- ColorForth. Roll-your-own (e.g. JonesForth). Factor
- Stack-based. Minimal. Typeless. Self-Modifying.
- VVVV
- Live, visual, flow-based
- Verilog, VHDL
- Flow-based, reactive
- Erlang
- Actors (Similar to communicating sequential processes).
- Session-based, hot-code loading. Futures/Promises.
- J...
- Array
- Haskell
- Purely functional. Lazy.
- Rust
- Linear types.
- 68000. 6510.
- Assembly...
- Map-reduce
- Declarative concurrent
- NetLogo
- ...
Tue, 14 Jan 2025 17:13:14 -0800
- Politics, Variance and Techbros. (permalink)
Travis KalanickIt worries me to see techbros close to politics. A side-effect of the tech industry is that it teaches some that they are extraordinarily smart, and that all problems can be solved with some javascript. No other experience needed, move fast, break things etc.
And that is scary when it comes to the real world, as most problems are non-trivial, and require deep understanding of the subject matter.
Now, one can argue that's not true, tech managed over and over again to have real-world impact - uber, amazon, airbnb etc - all things that work with physical things. And I doubt that the founders were all experts in the various fields before entering them, i.e. I don't think Travis Kalanick spent all his time driving a taxi before deciding to make Uber...
This works in tech because the cost of trying something is next to nothing.
Failing has no cost, and if you don't immediately fail, you can learn as you go. Expertise is mostly a variance-reduction technique, and you in government you want low-variance more than you want to luck out at the risk of failing catastrophically.
A.k.a. infinite monkeys in a room can write Shakespeare... that doesn't mean that the monkey that finally happens to do so is a genius.
tldr- techbros have reasonable reasons to do what they do - unfortunately, this creates a bias for the successful ones to overestimate the applicability of that method - and trying to solve every problem with the same hammer, regardless of the context, is catastrophic.
Mon, 13 Jan 2025 13:59:56 -0800
- Draft from 2015 on the future of real-time rendering and GPUs. (permalink)
Found this in the drafts of my old blog. I think, considering it's a decade old, it had some pretty decent ideas... Posting here unedited.
- Triangles, rasterization and quads
[links: http://www.drdobbs.com/parallel/rasterization-on-larrabee/217200602], Ryg's trip down G.Pipe and https://fgiesen.wordpress.com/2013/02/10/optimizing-the-basic-rasterizer/] - have been a constant for the past twenty years (with rare exceptions [...
https://en.wikipedia.org/wiki/NV1]).
Partially this is due to the fact that once a standard exists it's tough to break free from it. Even gaming consoles, made with unique hardware and for which we optimize every last cycle, tend to converge towards similar architectures, especially when it comes to decisions that would alter not simply code paths, but the entire logic and methods employed to achieve a given effect. It's a hard sell in modern cross-platform development to have unique hardware that necessitates specific techniques and assets to be used.
But that said, there are undoubtedly very good technical reasons for this success as well, it's not just historical. Yet, periodically we have to ask the question: what would the future hold? What could we possibly imagine that breaks past current GPU architectures? Here I want to note some options and considerations, with the disclaimer that not a lot of thought has gone into their pros and cons and even more importantly, that I am far from being a hardware designer...
When it comes to predicting the future of real-time computer graphics I often like to look at the history of offline rendering as guidance.
Unfortunately in this case I think it can be done only in the loosest sense, as the actual rendering techniques employed in the offline world [link ...REYES... prman] hardly ever had much in common with the kind of rasterization found in GPUs.
Even early CG shots [youtube links...] always had a given "offline" quality that even today is discernible: no aliasing, no visible discretization.
Primitives changed based on the processing power, from solids to NURBS, then to subdivision surfaces, sculpted displacement, and voxels. Even tools and techniques evolved much more radically, but this would be (probably) the topic of another post...
- Raytracing.
Every time one discusses of the future of realtime computer graphics, raytracing is the first thing that most people think about. I wrote about it in the past [...link] - back then I believed it wasn't going to happen in the near future (which is the only timescale I'm comfortable talking about) and still today I think it's not at our doorsteps.
The reason why I have this negative assessment is that I don't think we're near the complexity limits of rasterization: we still can and should handle more geometry, more complex shading (and note that fast raytracing usually imposes some extra burden on shading, exactly what we don't want) and more complex texturing.
From the offline CG history one can see that even there, before reaching given complexity limits, raytracing was not largely employed (the transition has been happening only in the recent few years there).
Raytracing is not trivial to implement efficiently, but once done it has certain benefits in terms of being able to express many lighting algorithms in a simpler, more natural, and accurate way. It might have, on enormously (depth-)complex scenes, even a performance edge, but we are nowhere near, IMHO, to wanting any of these things.
We don't care much (or at all) about making life easier for rendering engineers, and our scenes are nowhere near the scene complexity at which rasterization starts to buckle - if that even exists (especially for non-out-of-core rendering - LODs and culling take care of getting us back to a reasonable complexity for a given viewpoint).
In fact, we are still so lacking in performance - in being able to push complexity - that usually the best graphics comes from carefully chosen and exploited constraints on art production and on the kind of scenes we can represent - more than anything else.
This is even worse if you consider that "simple" raytracing, using mostly coherent rays for shadows or perfect reflections is perfectly useless (again one can look at the history of offline CG) - the real deal comes when we can shoot incoherent rays, which are still hard at the levels of performance needed for realtime graphics.
It is true though that even today many fundamental problems are hard to solve without incoherent visibility (e.g. indirect specular occlusion) and that some of these defects "show" in the final rendered image. There might be a "soft way" to raytracing, starting to use the technique in realtime to incrementally update the kinds of representation we today trace offline (i.e. voxels, uv-space, and so on) for dynamic objects, but I'd bet that's going to be done in software if it's needed, because lots of people will still prefer to solve these issues with more contraints (e.g. more static scenes or limited to what can be streamed).
- What else then?
What could help achieve the complexity goals we seek? One thing that could be discussed is the decoupling of shading and visibility rates. In modern GPUs this is achievable only at the subpixel level by means of MSAA, or to some extent via triangle interpolation, employing hardware tessellation but that incours into several limitations.
I think it's possible to think of several improvements to the rasterizer itself that could not be too crazy: from the simplest, like supporting quads or patches, to derivative-less quad-less rasterization, to more complex schemes [...
http://graphics.stanford.edu/papers/fragmerging/shade_sig10.pdf] to full-blown REYES [...GPU reyes].
To a degree 2x2 shading quads are reasonable: they provide a simple way to obtain derivatives, which are crucial for filtered texture reads, and they impose a very simple limit on the amount of data that needs to be passed in a wave as we will pack always at least four times more samples than primitives. But I still think there is room for improvement.
It also helps that typically on a GPU most of the area is taken by the shader units themselves, caches, texturing units, and all the components that are replicated many times. The rasterizer units are usually much smaller and shared among shader units, so it might be feasible to think of improvements, even adding functionality that could be more specific and infrequently used.
Even more in general this is about the dispatch of shading work, what could be done if we had more options, more flexibility in the way waves are created. Can we rasterize in more than two dimensions? Can we rasterize efficiently points, lines, or quads? Or allow stochastic rasterization [...
http://attila.ac.upc.edu/wiki/images/0/06/Hw_rast_hpg10.pdf TODO...
http://dl.acm.org/citation.cfm?id=1921505].
Point-clouds and splatting could be used to achieve more flexible visibility [...imperfect shadow maps] without sacrificing shading performance.
The other endpoint is interesting as well, controlling what we do with the shaded outputs. Can we lift the limitation of one sample per pixel shader for example? Allowing multiple outputs (e.g. a quad) to be sent to the blending stages would allow a degree of multi-resolution computation in shaders (certain math done per quad, certain per sample).
Computer shaders are another area of incremental improvement: they hold great potential, but so far they are severely limited by the inflexibility of dispatch and inability to reconfigure waves, especially architectures with wide waves, comparatively small register file and per-threadgroup memory severely limit the range of algorithms that are possible to implement efficiently, practically making inter-wave communication useless (this also should be the topic of a separate post...).
Being able to control GPU work from compute shaders (controlling dispatch, rasterizer, blending), being able to schedule waves on the same compute unit, reusing data already loaded in local memory, I would expect some of these capabilities to come in the future. Being able to generate "pixel shader" waves from a compute shader effectively means we could (more easily) implement software GPU rasterization [...TODO...]
Lastly, I think caching schemes will become increasingly complex and dynamic. Baking is not going to go away, even offline pure path tracing is not really the best option. Voxels, brick maps, irregular grids, sparse points, uv-space caching etcetera. How can we make these more efficient? Better compression schemes will be needed. Can there be better support for hierarchical data? Texture fetching that can traverse a mip chain, going up if certain texels are marked invalid until finding a valid sample? Or the ability of conditionally jumping to code that fills an area of a texture, if invalid texels are found. What could we do if we had more memory? SSD could give very large, fast, mostly read only memory areas, could these be directly connected to the CPU/GPUs?
Unfortunately I still think that most likely we will see fast progress in terms of raw compute power and memory, but much less progress on the actual capabilities, mostly due to cross-platform constraints.
Thu, 9 Jan 2025 17:39:53 -0800