@@ -107,20 +107,32 @@ iPhone and Android apps for photogrammetry and now LiDAR scanning have multiplie
...
@@ -107,20 +107,32 @@ iPhone and Android apps for photogrammetry and now LiDAR scanning have multiplie
_plenoptic_: Of or relating to all the light, travelling in every direction, in a given space.
_plenoptic_: Of or relating to all the light, travelling in every direction, in a given space.
Light fields represent an advanced form of passive sensing, aiming to capture full plenoptic content: all possible light rays emanating from a scene in any given direction. This results in a four-dimensional function, as it involves selecting a ray's position and angle. If the ideal plenoptic function was known, any novel viewpoint could be synthesized by placing a virtual camera in this space, and selecting the relevant light rays.
<small>Light fields represent an advanced form of passive sensing, aiming to capture full plenoptic content: all possible light rays emanating from a scene in any given direction. This results in a four-dimensional function, as it involves selecting a ray's position and angle. If the ideal plenoptic function was known, any novel viewpoint could be synthesized by placing a virtual camera in this space, and selecting the relevant light rays.</small>
"We begin by asking what can potentially be seen" - Edward H. Adelson & James R. Bergen, Media Lab, Vision & Modeling Group:
[The Plenoptic Function and the Elements of Early Vision](https://persci.mit.edu/pub_pdfs/elements91.pdf)
Adelson et al.
Why do we want all of the light?
Why do we want all of the light?
Image-Based Rendering (IBR) for view synthesis is a long-standing problem in the field of computer vision and graphics.
Image-Based Rendering (IBR) for view synthesis is a long-standing problem in the field of computer vision and graphics.
Applications in robot navigation, film, and AR/VR.
Applications in robot navigation, film, and AR/VR.
This is such an intensive calculation, that it prompts researchers to seek simulation shortcuts to reach this result:
Basically, the bullet time effect from the matrix.
<iframewidth="560"height="315"src="https://www.youtube.com/embed/9XM5-CJzrU0?si=eWG2ipj0fqA1Ng13&start=73"title="YouTube video player"frameborder="0"allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share"referrerpolicy="strict-origin-when-cross-origin"allowfullscreen></iframe>
[Using thousands of virtual cameras] (https://openaccess.thecvf.com/content/ACCV2022/papers/Li_Neural_Plenoptic_Sampling_Learning_Light-field_from_Thousands_of_Imaginary_Eyes_ACCV_2022_paper.pdf)
Papers proposing the use of thousands of virtual cameras and neural networks to capture a complete dense plenoptic function
This is such an intensive calculation, that it prompts researchers to seek simulation shortcuts to reach this result, such as this paper [using thousands of virtual cameras](https://openaccess.thecvf.com/content/ACCV2022/papers/Li_Neural_Plenoptic_Sampling_Learning_Light-field_fro
m_Thousands_of_Imaginary_Eyes_ACCV_2022_paper.pdf) and neural networks to capture a complete dense plenoptic function.
In practice, we can only sample light rays in discrete locations. There are two popular optical architectures for this:
In practice, we can only sample light rays in discrete locations. There are two popular optical architectures for this:
...
@@ -132,7 +144,7 @@ Simply shoot the scene from several locations using an array of camera (or a sin
...
@@ -132,7 +144,7 @@ Simply shoot the scene from several locations using an array of camera (or a sin
### Lenslets
### Lenslets
Lenslets: a single CMOS sensor with an array of lenses in front.
Lenslets: a single CMOS sensor with an array of lenses in front.
In the lenslet approach, each pixel behind a lenslet provides a unique light ray direction. The collection for all lenses is called a <strong>sub aperture image</strong>, and roughly corresponds to what a shifted camera would capture. The resolution of these images is simply the total number of lenslets, and the number of sub-aperture images available is given by the number of pixels behind a lenslet. For reference, the [Lytro Illum](https://en.wikipedia.org/wiki/Lytro)> provides 15x15 sub-aperture images of 541x434 pixels each, which is a total of ~53 Megapixels.
In the lenslet approach, each pixel behind a lenslet provides a unique light ray direction. The collection for all lenses is called a <strong>sub aperture image</strong>, and roughly corresponds to what a shifted camera would capture. The resolution of these images is simply the total number of lenslets, and the number of sub-aperture images available is given by the number of pixels behind a lenslet. For reference, the [Lytro Illum](https://en.wikipedia.org/wiki/Lytro) provides 15x15 sub-aperture images of 541x434 pixels each, which is a total of ~53 Megapixels.
<imgsrc="images/viewpoint7_png+img+margin.gif"alt="LF sub aperture images">
<imgsrc="images/viewpoint7_png+img+margin.gif"alt="LF sub aperture images">
