There is something quite interesting about building software. Sometimes you spend hours, days or even months developing an idea, only to discover that the solution isn't quite what you thought it would be. Sometimes an experiment fails, sometimes it produces unexpected results, and occasionally it gives you a completely new direction to investigate.
Tonight was one of those nights for Portal View.
Portal View is our first commercial product under Leopard Changing Spots. The original idea was relatively straightforward: take an ordinary video image and enhance the perception of depth, making the picture appear more dimensional and allowing details to stand out more prominently.
We've spent a considerable amount of time developing our rendering engine, experimenting with contrast, local detail, edge separation, colour and different depth cues. The objective has always been to produce something impressive without requiring expensive hardware or dedicated graphics cards.
But I don't want Portal View to become a product where we simply keep increasing sharpening, contrast and colour saturation. I want to understand how we can actually improve the perception of depth.
That question led us down a rather interesting research path.
One of the things we investigated was the early research into stereoscopic vision.
Back in 1838, Charles Wheatstone demonstrated that presenting different perspectives of the same scene to the two eyes could produce a convincing perception of three-dimensional depth. This was one of the foundations of stereoscopic technology.
The principle is fascinating. Our eyes are separated by a small distance, meaning each eye sees the world from a slightly different position. The brain uses the differences between those images to calculate depth.
I started thinking about how we could reproduce something similar in Portal View.
We have an ordinary television, a video source and our rendering engine. What if we could calculate two slightly different perspectives of the same video and alternate between them extremely rapidly?
Perhaps we could reproduce some of the depth perception associated with switching between the two eyes.
That became our next experiment.
We called the experiment AEP, or Alternating-Eye Parallax.
The idea was to capture a video frame, estimate the depth of different regions, generate two slightly different versions of the image and alternate between them.
We started with a relatively simple approach.
The foreground would receive a greater horizontal displacement, while the background would receive a smaller displacement. The two resulting images would then alternate on the television.
Initially, this was interesting. We could produce two different images, and the effect was visible.
However, something wasn't quite right.
Instead of producing the perception of a three-dimensional scene, the image appeared to move from side to side.
We tried increasing the displacement between the two perspectives. This made the movement more noticeable, but it didn't produce the depth effect we were looking for.
The problem became particularly obvious when testing a still photograph. Instead of seeing objects separated in depth, the entire picture appeared to vibrate.
This was an important result because it helped us understand a fundamental problem with our approach.
When you close one eye and open the other, each eye receives its own perspective. When we alternate images on an ordinary television, however, both eyes receive both images in succession.
We weren't reproducing genuine binocular stereoscopic viewing. We were producing an alternating visual movement.
That distinction changed how we approached the problem.
Rather than simply moving pixels horizontally, we investigated whether we could reproduce two virtual cameras looking at the same scene from slightly different positions.
This involved more complicated mathematics.
We experimented with camera separation, perspective projection, convergence planes and depth-dependent displacement.
The objective was to generate two images that behaved more like photographs taken from different camera positions.
We also introduced a depth buffer, which attempts to determine which surfaces should appear in front when projected pixels overlap.
This was an improvement in our approach to image reconstruction, but we encountered another problem.
When the virtual camera moves, certain parts of the background that were previously hidden can become visible. A normal two-dimensional image doesn't contain the information required to reconstruct all those missing areas.
Our initial attempts at filling those gaps produced unwanted lines and visual artefacts, especially when testing moving video.
We also had to consider the timing of video frames. If the left perspective comes from one moment in a movie and the right perspective comes from the next moment, the images no longer represent the same scene.
We therefore investigated generating both perspectives from the same captured frame and keeping them together for presentation.
This was a useful architectural improvement, but it didn't solve the underlying problem of delivering independent images to each eye.
Again, the experiment taught us something important.
Generating two perspectives is only half the problem. We also need a way to deliver the correct perspective to each eye.
This brought us to the research into autostereoscopic displays.
Technologies such as lenticular lenses and parallax barriers can direct different images towards different viewing positions.
In a simplified two-view display, the left eye receives one perspective and the right eye receives another. No glasses are required.
This is interesting because it addresses the exact limitation we encountered with our ordinary television.
A software application alone cannot make an unmodified conventional television direct independent images towards each eye. Some form of optical separation is needed.
We investigated whether Portal View could generate the two perspectives and interleave their pixels, allowing a specially designed optical layer to direct the images appropriately.
This is a promising research direction, although it introduces hardware requirements, alignment challenges and compromises involving resolution and brightness.
I don't want to pretend we've solved this problem. We haven't.
But understanding why something doesn't work is just as important as knowing how to make it work.
We also explored other display technologies.
One particularly interesting example was the LED hologram fan.
These devices use rapidly rotating LED strips to create images in space through persistence of vision.
I wondered whether we could reproduce some of that effect using the pixels on an ordinary television.
The answer is that we can simulate aspects of the visual appearance in software, but the television's pixels remain physically fixed on a flat surface. It cannot reproduce the actual swept-volume effect of a rotating LED display.
We also investigated genuine holographic displays, spatial light modulators and computer-generated holography.
These technologies reconstruct light in fundamentally different ways from conventional televisions.
They are considerably more complex, but they demonstrate how much further display technology can go.
This is something I would like Leopard Changing Spots to explore over the longer term.
After all these experiments, I started reconsidering what we are really trying to achieve with Portal View.
The current Microsoft Store product is strongest when we enhance the existing visual clues that make an image appear three-dimensional.
These include:
Our existing V21 rendering engine uses a custom depth-cue map to help guide some of these effects.
However, there is a limitation.
The engine estimates depth using characteristics such as texture, edges, colour and local contrast. These are useful visual clues, but they don't necessarily tell us how far away an object actually is.
For example, a detailed mountain in the distance could receive a higher depth score than a person's face in the foreground.
The algorithm doesn't understand that it is looking at a mountain or a person.
I think this is where our next major opportunity lies.
Instead of simply improving the visual characteristics of an image, what if Portal View could understand the approximate spatial structure of the scene?
We have started investigating AI-based monocular depth estimation.
Technologies such as Depth Anything V2 provide an interesting starting point. They attempt to estimate the relative depth of different regions within an ordinary two-dimensional photograph.
Imagine Portal View being able to identify the approximate depth relationship between a person, the building behind them, the trees further back and the mountains in the distance.
We could then use that information to make much more intelligent rendering decisions.
This is the direction I want to investigate for our next development cycle.
I don't want to abandon the existing product or split our development into completely separate versions that force existing customers to buy another application.
My philosophy has always been that improvements should reach the people who have already supported us.
We have a working foundation, and I want to build on it.
For the next release, I am particularly interested in investigating three areas.
Move beyond our current heuristic depth map and investigate learned depth estimation. The objective is to understand the spatial relationships within an image more accurately.
Video is constantly changing. We need the estimated depth of objects to remain stable as they move, rather than fluctuating between frames. This could be an important improvement to the overall experience.
Once we have a better understanding of the scene, we can investigate how to strengthen natural depth cues without introducing artificial halos, glowing outlines or unwanted visual artefacts.
Performance remains equally important. Portal View was developed with modest Windows computers in mind, and I don't want our improvements to make the application unusable for people without dedicated graphics hardware.
Advanced processing will need to be optional and carefully benchmarked.
There is a tendency in software development to focus on the finished product.
We see the final application, its features and its marketing. We don't always see the experiments that happened along the way.
But research is rarely a straight line.
Our alternating-perspective experiment didn't produce the stereoscopic effect we wanted. Increasing the camera separation produced more noticeable movement rather than convincing depth. Our virtual camera reconstruction introduced new problems with missing background information.
We also learned that a conventional television cannot provide genuine independent left-eye and right-eye images through software alone.
These weren't wasted experiments.
They helped us understand the distinction between making an image move, making an image look three-dimensional and actually delivering stereoscopic perspectives.
That knowledge will influence how we approach the next generation of Portal View.
Leopard Changing Spots was established around a simple idea: Explore. Experiment. Build.
I don't want us to simply follow what other software companies are doing. I want us to investigate problems, experiment with new approaches and see where we can develop something useful.
Portal View is our first commercial product, but I see considerable opportunity in display technology, visual computing and digital media.
Eventually, I would love to explore technologies that go much further than enhancing an ordinary television image.
For now, though, our priority is making Portal View genuinely impressive.
We don't need to rush a new release just to have another version number. I would rather spend time understanding the technology and produce an improvement that people can actually see.
Tonight's experiments didn't give us the breakthrough we were hoping for.
But they gave us something equally valuable: a much clearer understanding of the problem we are trying to solve.
We will not give up until we give our consumers the best viewing experience.