Efficient Methods for
Computational Radiology and Imaging
Sean I. Young, PhD
Instructor, Harvard Medical School
Research Affiliate, CSAIL, MIT
Sep 10, 2024
Efficient algorithms for computational radiology and imaging Sean I. Young Slide 2
Outline
• Previous and Current Research
• Computational imaging and radiology
• Future research directions and conclusion
• Teaching and outreach
Efficient algorithms for computational radiology and imaging Sean I. Young Slide 3
About me...
Stanford, CA
—Postdoc, EE
Sydney, NSW Boston, MA
Efficient algorithms for computational radiology and imaging Sean I. Young Slide 4
Stanford, CA
—Postdoc, EE
—Compression
—Computational Imaging
About me...
Efficient Algorithms for Imaging and Vision Problems Sean I. Young Bernd Girod Slide 3 of 9
Select publications
Young et al. NLOS surface reconstruction using the Directional LCT CVPR (oral), 2020
Capture setup
Captured transients
Recovered surface
Ground-truth surface
Estimation of hidden surface normals posed as a vectorial deconvolution problem. We achieve a 1,000×
speed-up over previous methods.
Efficient algorithms for computational radiology and imaging Sean I. Young Slide 5
Stanford, CA
—Postdoc, EE
—Compression
—Computational Imaging
About me...
Efficient Algorithms for Imaging and Vision Problems Sean I. Young Bernd Girod Slide 5 of 9
Select publications
Young et al. Transform quantization for CNN compression IEEE Trans. Pattern Anal. Mach. Intell., 2020
Kernel quantization and pruning posed as a rate-distortion optimization problem. We achieve a 10×
inference speed-up over the default 8-bit quantization.
2021
Efficient algorithms for computational radiology and imaging Sean I. Young Slide 6
About me...
Boston, MA
—Instructor, Radiology
Stanford, CASydney, NSW
Efficient algorithms for computational radiology and imaging Sean I. Young Slide 7
Boston, MA
—Computational Radiology
—Computational Imaging
About me...
Young et al. Supervision by denoising. IEEE Tra n s. Pattern Anal Mach Intell, 2023
Semi-supervised medical image segmentation posed as a denoising problem. We achieve a 10x reduction in the
number of labeled examples required while significantly improving reconstruction accuracy.
FS reference nnU-Net Post-denoise Temporal ensembling SUD (ours)
Efficient algorithms for computational radiology and imaging Sean I. Young Slide 8
About me...
Young et al. Fully convolutional SVR for single-stack MRI. CVPR, 2024
Slice-to-volume reconstruction as a self-supervised image registration task. We enable 3D reconstruction from a
single stack of MR slices, which has not previously been done.
Monocular depth estimation [4–8] Single-stack slice-to-volume reconstruction (Ours)
Boston, MA
—Computational Radiology
—Computational Imaging
Efficient algorithms for computational radiology and imaging Sean I. Young Slide 9
Outline
• Previous and Current Research
• Computational radiology and imaging
• Future research directions and conclusion
• Teaching and outreach
Efficient algorithms for computational radiology and imaging Sean I. Young Slide 10
From Computational Imaging
Extreme Inverse problems
3D Object
Reflections
on a wall
Non-line-of-sight surface reconstruction
(Young, 2020)
Efficient algorithms for computational radiology and imaging Sean I. Young Slide 11
Extreme Inverse problems
From Computational Imaging
3D Volume
2D Acquisition
Slice-to-volume reconstruction (SVR)
(Young, 2024)
Efficient algorithms for computational radiology and imaging Sean I. Young Slide 12
Efficient methods
Computational Radiology
Linear algebra
and signal processing
Signal processing
and machine learning
Machine learning
and signal compression
Transform quantization for CNN compression
(Young, 2022)
Efficient algorithms for computational radiology and imaging Sean I. Young Slide 13
Computational Radiology Lab
Computational
Radiology
Slice-to-Volume Reconstruction
for Single-Stack MRI
Based on “Fully convolutional slice-to-volume reconstruction for single-stack MRI”. CVPR 2024.
Efficient algorithms for computational radiology and imaging Sean I. Young Slide 15
Introduction
Figure from https://answers.childrenshospital.org/neurodevelopment-congenital-heart-disease/
Slice-based MR acquisitions can freeze subject motion in-plane.
Efficient algorithms for computational radiology and imaging Sean I. Young Slide 16
2D MR Acquisition
Axial
Stack
Axial view Axial view Axial view Coronal view Sagittal view
Figure adapted from Gholipour et al. (2010)
Efficient algorithms for computational radiology and imaging Sean I. Young Slide 17
2D MR Acquisition
Axial
Stack
Axial view
Figure adapted from Gholipour et al. (2010)
Efficient algorithms for computational radiology and imaging Sean I. Young Slide 18
2D MR Acquisition
Axial
Stack
Axial view
Figure adapted from Gholipour et al. (2010)
Efficient algorithms for computational radiology and imaging Sean I. Young Slide 19
2D MR Acquisition
Axial
Stack
Axial view
Figure adapted from Gholipour et al. (2010)
Efficient algorithms for computational radiology and imaging Sean I. Young Slide 20
2D MR Acquisition
Axial
Stack
Coronal view
Figure adapted from Gholipour et al. (2010)
Efficient algorithms for computational radiology and imaging Sean I. Young Slide 21
Axial
Stack
2D MR Acquisition
Sagittal view
Figure adapted from Gholipour et al. (2010)
Efficient algorithms for computational radiology and imaging Sean I. Young Slide 22
Slice-to-Volume Reconstruction
Axial
Stack
Coronal
Stack
Sagittal
Stack
Reconstructed
Volume
Figure adapted from Sobotka et al. (2022)
SVR aims to align a stack of acquired slices thereby
removing the remaining motion across slices.
Efficient algorithms for computational radiology and imaging Sean I. Young Slide 23
Towards Fully Convolutional SVR
1 x 1 x 9
Anchor
points
(x1,y1,z1)
(x2,y2,z2)
(x3,y3,z3)
Efficient algorithms for computational radiology and imaging Sean I. Young Slide 24
Towards Fully Convolutional SVR
Figure adapted from Xu et al. (2022)
Bleeding-edge transformer
models!
(a)
(b)
Unable to reconstruct accurately
from a single slice stack!
Efficient algorithms for computational radiology and imaging Sean I. Young Slide 25
Towards Fully Convolutional SVR
Figure adapted from Xu et al. (2022)
Bleeding-edge transformer
models!
(a)
(b)
We can predict slice motion
instead of slice positions
Efficient algorithms for computational radiology and imaging Sean I. Young Slide 26
Stereo Disparity Estimation
Szeliski and Golland (1999)
Efficient algorithms for computational radiology and imaging Sean I. Young Slide 27
Stereo Disparity Estimation
Right view
Efficient algorithms for computational radiology and imaging Sean I. Young Slide 28
Stereo Disparity Estimation
Left view
Efficient algorithms for computational radiology and imaging Sean I. Young Slide 29
Stereo Disparity Estimation
Right view
Efficient algorithms for computational radiology and imaging Sean I. Young Slide 30
Stereo Disparity Estimation
Disparity field
Efficient algorithms for computational radiology and imaging Sean I. Young Slide 31
Monocular Disparity Estimation
U-Net
Deep learning allows disparity estimation from a SINGLE view!
✓
Efficient algorithms for computational radiology and imaging Sean I. Young Slide 32
Monocular Disparity Estimation
U-Net
Deep learning allows disparity estimation from a SINGLE view!
✓
Single-stack MRI?
Efficient algorithms for computational radiology and imaging Sean I. Young Slide 33
Towards Fully Convolutional SVR
• 3D representation: 3D scene (2D + depth).
• Reprojection: 2D view.
Efficient algorithms for computational radiology and imaging Sean I. Young Slide 34
Towards Fully Convolutional SVR
• 3D representation: 3D volume (Splatting).
• Reprojection: 2D slices (Slicing).
• Registration of an unknown moving image to a fixed image.
?
Efficient algorithms for computational radiology and imaging Sean I. Young Slide 35
Fully Convolutional SVR
• Suppose we have both a 3D volume and a slice stack.
• Reconstruct by registering the slice stack to the 3D volume.
Efficient algorithms for computational radiology and imaging Sean I. Young Slide 36
Fully Convolutional SVR
• We do not assume the availability of a reference volume.
• Pairwise image registration between fixed (stack) and moving (volume).
Efficient algorithms for computational radiology and imaging Sean I. Young Slide 37
Supervised Learning
• Train CNN on paired (slice stack, motion stack) examples
• Synthesize slice stacks with arbitrary but known motion
𝓛
Efficient algorithms for computational radiology and imaging Sean I. Young Slide 38
Supervised Learning
• Training loss: L2 loss unduly forces all reconstructions to be aligned
precisely in a particular orientation.
andBut
Splat
✓ ✗ ✗
𝓛
Efficient algorithms for computational radiology and imaging Sean I. Young Slide 39
Supervised Learning
• Training loss: L2 loss unduly forces all reconstructions to be aligned
precisely in a particular orientation.
andand
Splat
✓ ✓ ✓
Efficient algorithms for computational radiology and imaging Sean I. Young Slide 40
Supervised Learning
• Uncompensated loss penalizes
global rotations and translations.
• Translation-compensation
improves prediction.
• Rigid compensation
improves prediction further.
Motion MSE (voxels)
Epochs
Uncompensated
Efficient algorithms for computational radiology and imaging Sean I. Young Slide 41
Supervised Learning
• Uncompensated loss penalizes
global rotations and translations.
• Translation-compensation
improves prediction.
• Rigid compensation
improves prediction further.
Motion MSE (voxels)
Epochs
Uncompensated
Trans-comp
Efficient algorithms for computational radiology and imaging Sean I. Young Slide 42
Supervised Learning
• Uncompensated loss penalizes
global rotations and translations.
• Translation-compensation
improves prediction.
• Rigid compensation
improves prediction further.
Motion MSE (voxels)
Epochs
Uncompensated
Trans-comp
Rigid-comp
Efficient algorithms for computational radiology and imaging Sean I. Young Slide 43
Supervised Learning
• Splatted volume has regions of missing intensities.
• Interpolate missing data to produce 3D reconstruction.
Efficient algorithms for computational radiology and imaging Sean I. Young Slide 44
Supervised Learning
• Splatted volume has regions of missing intensities.
• Interpolate missing data to produce 3D reconstruction.
Efficient algorithms for computational radiology and imaging Sean I. Young Slide 45
Results
Efficient algorithms for computational radiology and imaging Sean I. Young Slide 46
Results
Efficient algorithms for computational radiology and imaging Sean I. Young Slide 47
Results
Efficient algorithms for computational radiology and imaging Sean I. Young Slide 48
Results
Efficient algorithms for computational radiology and imaging Sean I. Young Slide 49
Results
Efficient algorithms for computational radiology and imaging Sean I. Young Slide 50
Results
Efficient algorithms for computational radiology and imaging Sean I. Young Slide 51
Results
Efficient algorithms for computational radiology and imaging Sean I. Young Slide 52
Results
Non-line-of-sight surface reconstruction
Based on “NLOS Surface Reconstruction Using the D-LCT”. CVPR 2020
(Worked performed while at Stanford)
Efficient algorithms for computational radiology and imaging Sean I. Young Slide 54
Non-line-of-sight Imaging
http://web.media.mit.edu/~raskar/cornar/
Can we see
inside the room?
Efficient algorithms for computational radiology and imaging Sean I. Young Slide 55
Non-line-of-sight Imaging
http://web.media.mit.edu/~raskar/cornar/
NLOS Endoscopy
Efficient algorithms for computational radiology and imaging Sean I. Young Slide 56
Non-line-of-sight Imaging
Wall
Hidden
Object
Occluder
• Image an object not in the observer’s
direct line of sight?
Laser and
Detector
(Young et al. 2020) Non-line-of-sight surface reconstruction using the directional light-cone transform
Efficient algorithms for computational radiology and imaging Sean I. Young Slide 57
Non-line-of-sight Imaging
Wall
Hidden
Object
Occluder
• Image an object not in the observer’s
direct line of sight?
Laser and
Detector
Time (seconds)
Photons Detected
0
(Young et al. 2020) Non-line-of-sight surface reconstruction using the directional light-cone transform
Efficient algorithms for computational radiology and imaging Sean I. Young Slide 58
Non-line-of-sight Imaging
Wall
Hidden
Object
Occluder
• At t
1
, we have photons from points
in the scene t
1
light-seconds away
from the wall.
Laser and
Detector
Time (seconds)
Photons Detected
0
t
1
t
1
(Young et al. 2020) Non-line-of-sight surface reconstruction using the directional light-cone transform
Efficient algorithms for computational radiology and imaging Sean I. Young Slide 59
Non-line-of-sight Imaging
Wall
Hidden
Object
Occluder
• At t
2
, we have photons from points
in the scene t
2
light-seconds away
from the wall.
Laser and
Detector
Time (seconds)
Photons Detected
0
t
2
t
2
(Young et al. 2020) Non-line-of-sight surface reconstruction using the directional light-cone transform
Efficient algorithms for computational radiology and imaging Sean I. Young Slide 60
Non-line-of-sight Imaging
Wall
Hidden
Object
Occluder
• Forward model relates time-varying
photon to the hidden surface.
Laser and
Detector
t
𝑦
𝑥
𝑧
(Young et al. 2020) Non-line-of-sight surface reconstruction using the directional light-cone transform
Efficient algorithms for computational radiology and imaging Sean I. Young Slide 61
Non-line-of-sight Imaging
Wall
Hidden
Object
Occluder
• Scan multiple known locations
on the wall to collect a 3D volume
of measurements.
Laser and
Detector
Time (seconds)
Photons Detected
0
(Young et al. 2020) Non-line-of-sight surface reconstruction using the directional light-cone transform
Efficient algorithms for computational radiology and imaging Sean I. Young Slide 62
Non-line-of-sight Imaging
Wall
Hidden
Object
Occluder
• Scan multiple known locations
on the wall to collect a 3D volume
of measurements.
Laser and
Detector
Time (seconds)
Photons Detected
0
(Young et al. 2020) Non-line-of-sight surface reconstruction using the directional light-cone transform
Efficient algorithms for computational radiology and imaging Sean I. Young Slide 63
Non-line-of-sight Imaging
Wall
Hidden
Object
Occluder
• Scan multiple known locations
on the wall to collect a 3D volume
of measurements.
Laser and
Detector
Time (seconds)
Photons Detected
0
(Young et al. 2020) Non-line-of-sight surface reconstruction using the directional light-cone transform
Efficient algorithms for computational radiology and imaging Sean I. Young Slide 64
Non-line-of-sight Imaging
Photons Detected
Hidden
Object Surface
Adapted from O’Toole et al. (2018)
Efficient algorithms for computational radiology and imaging Sean I. Young Slide 65
Non-line-of-sight Imaging
Photons Detected
Hidden
Object Surface
(Young et al. 2020) Non-line-of-sight surface reconstruction using the directional light-cone transform
Efficient algorithms for computational radiology and imaging Sean I. Young Slide 66
(Young et al. 2020) Non-line-of-sight surface reconstruction using the directional light-cone transform
Non-line-of-sight Imaging
Photons Detected
Hidden
Object Surface
Unable to resolve
surface normals!
Efficient algorithms for computational radiology and imaging Sean I. Young Slide 67
Non-line-of-sight Imaging
Wall
Hidden
Object
Occluder
• Photon contributions to a wall location
proportional to cos(
θ
).
Laser and
Detector
𝛖
$
𝐬
−
𝐬
θ
(Young et al. 2020) Non-line-of-sight surface reconstruction using the directional light-cone transform
Efficient algorithms for computational radiology and imaging Sean I. Young Slide 68
Non-line-of-sight Imaging
Wall
Hidden
Object
Occluder
• Photon contributions to a wall location
proportional to cos(
θ
).
Laser and
Detector
θ
(Young et al. 2020) Non-line-of-sight surface reconstruction using the directional light-cone transform
Efficient algorithms for computational radiology and imaging Sean I. Young Slide 69
Non-line-of-sight Imaging
Photons Detected
Hidden Object
Surface Normals
Wiener–deconv
Efficient algorithms for computational radiology and imaging Sean I. Young Slide 70
Non-line-of-sight Imaging
Photons Detected
Hidden Object
Surface Normals
Efficient algorithms for computational radiology and imaging Sean I. Young Slide 71
Non-line-of-sight Imaging
Efficient algorithms for computational radiology and imaging Sean I. Young Slide 72
Non-line-of-sight Imaging
Efficient algorithms for computational radiology and imaging Sean I. Young Slide 73
Non-line-of-sight Imaging
Efficient algorithms for computational radiology and imaging Sean I. Young Slide 74
Non-line-of-sight Imaging
Model Compression for Computational Radiology
Based on
“Foundations of large language model compression” (Preprint, 2024) and
“Transform quantization for CNN compression” (Trans PAMI, 2021)
Efficient algorithms for computational radiology and imaging Sean I. Young Slide 76
Efficient Radiology
Slice-to-volume reconstruction (SVR)
(Young, 2024)
• Many inverse problems in radiology solved using neural networks.
• Deploying 100 billion-parameter models to clinical settings is hard.
Efficient algorithms for computational radiology and imaging Sean I. Young Slide 77
Efficient Radiology
• Many inverse problems in radiology solved using neural networks.
• Deploying 100 billion-parameter models to clinical settings is hard.
Efficient algorithms for computational radiology and imaging Sean I. Young Slide 78
Model Compression
• Three approaches to model compression
Network
Architecture
Search
Weight Pruning
Quantization
Efficient algorithms for computational radiology and imaging Sean I. Young Slide 79
Weight Quantization
75.28558ºF → 75.3ºF (3 digits)
1.0100101
2
→ 1.01
2
(3 bits)
• Quantization: represent weights with a smaller number of digits (bits)
Input
3-bit
quantizer
Output
Efficient algorithms for computational radiology and imaging Sean I. Young Slide 80
Inverse problem (again)
• Model compression is a kind of inverse problem!
Efficient algorithms for computational radiology and imaging Sean I. Young Slide 81
Convex Optimization
• Similar to the classic image denoising problem
• Minimize data term D subject to constraint R
–
minimize D (B
1
, B
2
, . . . , B
5
) =
subject to R = Average(B
1
, B
2
, . . . , B
5
) = B
budget
Efficient algorithms for computational radiology and imaging Sean I. Young Slide 82
Distortion and Bit Depth
B
2
B
5
B
1
Distortion
Bit depth Bit depth Bit depth
• Exponential decay in output distortion with bit depth
Efficient algorithms for computational radiology and imaging Sean I. Young Slide 83
Pareto Optimality
B
2
B
5
B
1
Distortion
• Optimal when distortion slopes equal across layers layers.
“Pareto”
optimality
2 1 3
Efficient algorithms for computational radiology and imaging Sean I. Young Slide 84
Dual Ascent
B
2
B
5
B
1
Distortion
Bit depth Bit depth Bit depth
• Sweep distortion curves until average bit depth matches target
Iters
Average bit depth
B
budget
= 4
Efficient algorithms for computational radiology and imaging Sean I. Young Slide 85
Dual Ascent
B
2
B
5
B
1
Distortion
Bit depth Bit depth Bit depth
• Sweep distortion curves until average bit depth matches target
Iters
Average bit depth
B
budget
= 4
Efficient algorithms for computational radiology and imaging Sean I. Young Slide 86
Dual Ascent
B
2
B
5
B
1
Distortion
Bit depth Bit depth Bit depth
• Sweep distortion curves until average bit depth matches target
Iters
Average bit depth
B
budget
= 4
Efficient algorithms for computational radiology and imaging Sean I. Young Slide 87
Finding Sensitivities
B
2
B
5
B
1
Distortion
Bit depth Bit depth Bit depth
• Sensitivity is the variance of for each layer of weights W
Efficient algorithms for computational radiology and imaging Sean I. Young Slide 88
• Backprop output and compute mean squared grad per layer
Variance accumulation
Efficient algorithms for computational radiology and imaging Sean I. Young Slide 89
• Benchmark on medical language data (and more!)
Medical Language Data
An
MRI
of
the
brain
shows
an
infra
tentorial
mass
.
A
photo
micro
graph
of
the
resected
MRI
of
the
brain
shows
an
infra
tentorial
mass
.
A
photo
micro
graph
of
the
resected
specimen
Efficient algorithms for computational radiology and imaging Sean I. Young Slide 90
• MedQA—Multi-choice Q&A from Medical Licensing Exams
Medical Language Data
Q: A 22-year-old man comes to the physician because
of headaches and blurry vision for the past 6 months
... An MRI of the brain shows an infratentorial
mass. The patient undergoes surgical resection of
the mass. A photomicrograph of the resected specimen
is shown. Which of the following is the most likely
diagnosis?
A: Medulloblastoma
B: Oligodendroglioma
C: Hemangioblastoma
D: Ependymoma
C
Prompt
Output
Efficient algorithms for computational radiology and imaging Sean I. Young Slide 91
Accuracy of Llama-2 Models on MedQA (5092 Questions)
MedQA
20%
30%
40%
50%
60%
Original (16 bit)
Llama-2 70B
Rounding (3 bit)
Llama-2 70B
Proposed (3 bit)
Llama-2 70B
Original (16 bit)
Llama-2 13B
Similar memory
footprint
Efficient algorithms for computational radiology and imaging Sean I. Young Slide 92
Accuracy of Llama-2 Models on MedQA (5092 Questions)
MedQA
20%
30%
40%
50%
60%
Original (16 bit)
Llama-2 70B
Rounding (3 bit)
Llama-2 70B
Proposed (3 bit)
Llama-2 70B
Original (16 bit)
Llama-2 13B
Similar memory
footprint
Efficient algorithms for computational radiology and imaging Sean I. Young Slide 93
• GSM8K—Grade School Math Problems (5-shot)
Other Language Data
Q: Four people lost a total of 103 kilograms of
weight. The first person lost 27 kilograms. The
second person lost 7 kilograms less than the first
person. The two remaining people lost the same
amount. How many kilograms did each of the last two
people lose?
Prompt
Output
Second person = 27 - 7 = <<27-7=20>>20 kg 103 - 27 -
20 = <<103-27-20=56>>56 kg 56/2 = <<56/2=28>>28 kg
The last two people each lost 28 kilograms of
weight. #### 28
Efficient algorithms for computational radiology and imaging Sean I. Young Slide 94
Accuracy of Llama-2 Models on GSM8K (1319 Questions)
GSM8K
0%
10%
20%
30%
40%
50%
60%
Original (16 bit)
Llama-2 70B
Rounding (3 bit)
Llama-2 70B
Proposed (3 bit)
Llama-2 70B
Original (16 bit)
Llama-2 13B
Similar memory
footprint
Efficient algorithms for computational radiology and imaging Sean I. Young Slide 95
WikiText2
Efficient algorithms for computational radiology and imaging Sean I. Young Slide 96
WikiText2
Efficient algorithms for computational radiology and imaging Sean I. Young Slide 97
Ongoing Work
3D brain imaging using conventional cameras and machine learning
Dissection photography is routine in nearly every brain bank. Dissected into coronal slices and photographed
before further blocking and histological slices. Leverage existing data for 3D reconstruction.
(a) Slab preparation (b) Slab photographs (c) 3D reconstruction
Efficient algorithms for computational radiology and imaging Sean I. Young Slide 98
Ongoing Work
Ultra-precision clinical imaging and detection of Alzheimer’s Disease (NIH K99AG081493)
Longitudinal image registration and change detection as a supervised learning problem. We achieve a 10-fold
improvement in registration accuracy compared to unsupervised (VoxelMorph) approaches.
Efficient algorithms for computational radiology and imaging Sean I. Young Slide 99
Going forward…
• Detect neuroanatomical change predictive of AD
• Compensate for the motion and take the difference
0m
12m
Efficient algorithms for computational radiology and imaging Sean I. Young Slide 100
Going forward…
• Detect neuroanatomical change predictive of AD
• Compensate for the motion and take the difference
0m
12m
0m – 12m
Efficient algorithms for computational radiology and imaging Sean I. Young Slide 101
Acknowledgments
Sean I. Young Yaël Balbastre Margherita Firenze David Taubman
Bernd Girod Polina Golland Bruce Fischl J Eugenio Iglesias
