!
!
Abstract
In magnetic resonance imaging (MRI), slice-to-volume
reconstruction (SVR) refers to computational reconstruction
of an unknown 3D magnetic resonance volume from stacks
of 2D slices corrupted by motion. While promising, current
SVR methods require multiple slice stacks for accurate 3D
reconstruction, leading to long scans and limiting their use
in time-sensitive applications such as fetal fMRI. Here, we
propose a SVR method that overcomes the shortcomings of
previous work and produces state-of-the-art reconstructions
in the presence of extreme inter-slice motion. Inspired by the
recent success of single-view depth estimation methods, we
formulate SVR as a single-stack motion estimation task and
train a fully convolutional network to predict a motion stack
for a given slice stack, producing a 3D reconstruction as a
byproduct of the predicted motion. Extensive experiments on
the SVR of adult and fetal brains demonstrate that our fully
convolutional method is twice as accurate as previous SVR
methods. Our code is available at github.com/seannz/svr.
1. Introduction
! "#$!%#%&'%()*'(+!',)-'%-!#.!/0+!1$)'%!20+$+!/0+!*314+5/!
,)6!7#/+%/')886!+90'1'/!*+(+$+!3%5#%/$#88)18+!,#/'#%:*350!
)*!'%!/0+!5)*+!#.!/0+!.+/)8!7#738)/'#%:/2#&;',+%*'#%)8!<=>?!
,)-%+/'5!$+*#%)%5+!',)-'%-!<@AB?!/+50%'C3+*!)$+!/67'5)886!
3*+;!/#!)5C3'$+!)!*/)5D!#.!=>!1$)'%!*8'5+*!)%;!)**+,18+!/0+,!
'%/#!)!E>!(#83,+F!)!5#,73/)/'#%)8!7$#5+;3$+!#./+%!$+.+$$+;!
/#!)*!*8'5+&/#&(#83,+!$+5#%*/$35/'#%!<GHA?I!J%8'D+!E>!@AB!
)5C3'*'/'#%*!20+$+!/0+! +%/'$+!𝑘&*7)5+!<"#3$'+$?! 5#+K5'+%/*!
5)%!1+5#,+!5#$$37/+;!+(+%!.$#,!)!*'%-8+!,#/'#%!+(+%/F!=>!
/+50%'C3+*!5)%!$+*/$'5/!/0'*!5#$$37/'#%!/#!2'/0'%!)!*8'5+!20'8+!
8+)('%-!/0+!𝑘&*7)5+!;)/)!.#$!/0+!#/0+$!*8'5+*!'%/)5/I!L+!GHA!
7$#5+;3$+!5)%!/0+%!1+!)778'+;!#%!/0+!%#%&5#$$37/+;!*8'5+*!/#!
+%)18+!)$/'.)5/&.$++!',)-'%-!#.!/0+!1$)'%!)%;!.)5'8'/)/+!#/0+$!
;#2%*/$+),!/)*D*F!*350!)*!1$)'%!,#$70#,+/$6 MNOEPF!20#8+!
1$)'%!*+-,+%/)/'#%!MQORP!)%;!)/8)*!5$+)/'#%!MSOTPI
! U!8)$-+!%3,1+$!#.!58)**'5)8!GHA!/+50%'C3+*!MNVONWP!)$+!
;+*5+%;+;!.$#,!/0+!2#$D!#.!A#3**+)3!+/!)8I!MNTPF!20+$+!/0+!
GHA!/)*D!'*!7#*+;!)*!)%!#7/','X)/'#%!7$#18+,!/#!$+5#(+$!)%!
3%D%#2%!E>!(#83,+!.$#,!#1*+$(+;!=>!*8'5+*I!G350!'%(+$*+!
,+/0#;*!;#!%#/!0)(+!/0+!5)7)1'8'/6!/#!8+)$%!/0+!8)/+%/!*7)5+!
#.!$+5#%*/$35/'#%*F!)%;!)*!)!$+*38/F!/67'5)886!$+C3'$+!,38/'78+!
#$/0#-#%)8!*8'5+!*/)5D*!/#!5$#**&$+-38)$'X+!/0+!+*/',)/'#%!#.!
/0+!,#/'#%!#.!+)50!*/)5DF!);;'%-!)!5#,73/)/'#%)8!13$;+%!/#!
)%!)8$+);6!'/+$)/'(+!%3,+$'5)8!,+/0#;I!@#$+!$+5+%/!,+/0#;*!
M=VO=YP!#(+$5#,+!/0+!13$;+%!#.!%3,+$'5)8!#7/','X)/'#%!16!
;'$+5/86!7$+;'5/'%-!/0+!7#*'/'#%!7)$),+/+$*!#.!=>!*8'5+*!2'/0!
%+3$)8!%+/2#$D!,#;+8*I!Z#2+(+$F!/0+*+!)77$#)50+*!7$#5+**!
*8'5+*!'%;+7+%;+%/86!#.!+)50!#/0+$!)%;!5)%%#/!5#%*/$)'%!/0+!
$+5#%*/$35/'#%!'%!E>!*7)5+!M=VO==P!#$!$+C3'$+!,38/'78+!*8'5+!
*/)5D*!.#$!)553$)/+!$+5#%*/$35/'#%!M=YPF!7$+583;'%-!/0+'$!3*+!
'%!/',+&*+%*'/'(+!)778'5)/'#%*!*350!)*!.+/)8!.@AB!M=RF!=SPI!
! B%!/0+!58#*+86!$+8)/+;!/)*D*!#.!7)'$2'*+!',)-+!$+-'*/$)/'#%!
M=WOE=PF!,#/'#%!MEEOETP!)%;!;+7/0!+*/',)/'#%!MQVOQEPF!.3886!
5#%(#83/'#%)8!%+/2#$D!<"[\?!)$50'/+5/3$+*!0)(+!1++%!3*+;!
Figure 1: Method overview. Inspired by monocular disparity estimation (a), we tackle slice-to-volume reconstruction (b) by predicting
slice
motion from a single stack of 2D slices using a fully convolutional network model. ?e intensities of the 2D slices are splatted using the slice
motion predicted by our network and the splatting is then interpolated to produce an artifact-free 3D MR reconstruction.!
(a) Monocular Depth (Disparity) Estimation [38–41] (b) Single-Stack Slice-to-Volume Reconstruction (Proposed)
Splat and
Interpolate
Standard
U-Net
Splat–Slice
U-Net
Single View Disparity Map Single Slice Stack Motion Stack 3D Reconstruction
Fully Convolutional Slice-to-Volume Reconstruction for Single-Stack MRI
Sean I. Young* Yaël Balbastre*
Harvard Medical School Harvard Medical School
siyoung@mit.edu ybalbastre@mgh.harvard.edu!
Bruce Fischl Polina Golland Juan Eugenio Iglesias
Harvard Medical School MIT Harvard Medical School
fischl@nmr.mgh.harvard.edu polina@csail.mit.edu jei@mit.edu
!
!
!
*?ese authors contributed equally to this work.
!
!
/#!+%)18+!)553$)/+!%#%&'/+$)/'(+!7$+;'5/'#%!#.!)%!3%;+$86'%-!
;+.#$,)/'#%F!,#/'#%!)%;!;'*7)$'/6!.$#,!)!7)'$!#.!',)-+*I!]6!
.#$,38)/'%-!/0+!GHA!/)*D!)*!)!*37+$('*+;!,#/'#%!+*/',)/'#%!
7$#18+,F!2+!5)%!$+;35+!/0+!/)*D!)/!0)%;!/#!)!7$#18+,!/0)/!'*!
,#$+!$+);'86!/)5D8+;!3*'%-!)!5#%(#83/'#%)8!.$),+2#$DI!L'*!
5#%(#83/'#%)8!)77$#)50!/#!GHA!1+)$*!)!5+$/)'%!$+*+,18)%5+!
/#!,#%#538)$!;+7/0!+*/',)/'#%F!20+$+!;'*7)$'/6!'*!7$+;'5/+;!
.$#,!)!*'%-8+!('+2!MQ=PI!L'*!'*!'883*/$)/+;!'%!"'-3$+!NI!
! B%!/0'*!2#$DF!2+!7$#7#*+!)!.3886!5#%(#83/'#%)8!,+/0#;!.#$!
GHA!2'/0!)778'5)/'#%!/#!*'%-8+&*/)5D!@ABI!^+!7#*+!/0+!GHA!
/)*D!)*!)!*37+$('*+;!',)-+!$+-'*/$)/'#%!7$#18+,!)%;!/$)'%!)!
5#%(#83/'#%)8!%+/2#$D!#%!7)'$+;!<*8'5+!*/)5DF!,#/'#%!*/)5D?!
;)/)!/#!7$+;'5/!*8'5+!,#/'#%!;'$+5/86F!7$#;35'%-!)!E>!(#83,+!
$+5#%*/$35/'#%!)*!)!$+*38/!<"'-3$+!=?I!_3$!)77$#)50!$+C3'$+*!
#%86!)!*'%-8+!*8'5+!*/)5D!.#$!$+5#%*/$35/'#%F!'*!,)%6!#$;+$*!#.!
,)-%'/3;+!.)*/+$!/0)%!#7/','X)/'#%!)77$#)50+*!MNVONWPF!)%;!
$+;35+*!+$$#$!'%!/0+!7$+;'5/+;!,#/'#%!16!2×!5#,7)$+;!/#!/0+!
$+5+%/!/$)%*.#$,+$&1)*+;!,#;+8!M=YPI!_3$!5#%/$'13/'#%*!)$+`!
• Splat–Slice U-Net Model. ^+!'%/+$7$+/!/0+!'/+$)/'#%*!#.!
58)**'5)8!GHA!)*!)!*+$'+*!#.!*78)/!)%;!*8'5+!#7+$)/'#%*!/#!
;+*'-%!)!E>!"[\!,#;+8!.#$!.)*/!*8'5+!,#/'#%!7$+;'5/'#%I!
• Supervised Learning. ^+! .#$,38)/+!)! 7)$),+/+$&.$++!
8#**!.3%5/'#%!/#!.)5'8'/)/+!/$)'%'%-!2'/0!/$3+!*8'5+!,#/'#%!
.#$!*8'5+!,#/'#%!7$+;'5/'#%!2'/0!*31&(#9+8!)553$)56I!
• Artifact-Free Reconstruction.!^+! #1/)'%!)$/'.)5/&.$++!
E>!$+5#%*/$35/'#%*!16!*78)//'%-!*8'5+!;)/)!)%;!',73/'%-!
,'**'%-!'%/+%*'/'+*!(')!*37+$('*+;!'%/+$7#8)/'#%I!
2. Related Work
! B%!.+/)8!',)-'%-F!*8'5+&/#&(#83,+!$+5#%*/$35/'#%!<GHA?!'*!
)%!',7#$/)%/!'%(+$*+!7$#18+,!2'/0!$+5#%*/$35/'#%!,+/0#;*!
$)%-'%-!.$#,!/$);'/'#%)8!#7/','X)/'#%&1)*+;!#%+*!MNVONTP!/#!
$+5+%/!8+)$%'%-&1)*+;!#%+*!M=VO=YPF!20'50!2+!%#2!$+('+2I!
2.1. Optimization Approaches
! _7/','X)/'#%&1)*+;!GHA!MNVONWP!/$)5+*!'/*!2)6!1)5D!/#!
/0+!*+,'%)8!2#$D!#.!A#3**+)3!+/!)8I!MNTPF!20#!7$+;'5/+;!/0+!
3%;+$86'%-!E>!@AB!(#83,+!16!)8/+$%)/'%-86!+*/',)/'%-!/0+!
*8'5+!,#/'#%!)%;!(#83,+!.$#,!/0$++!#$/0#-#%)8!*8'5+!*/)5D*!
)%;!*37+$&$+*#8('%-!/0+!a%)8!7$+;'5/+;!(#83,+I!L+!,+/0#;!
#.!b')%-!+/!)8I!MNVP!'*!58#*+86!$+8)/+;!/#!MNTP!13/!);#7/*!)!-$';!
*+)$50!/#!+*/',)/+!/0+!*8'5+!,#/'#%!7)$),+/+$*!/#!#(+$5#,+!
/0+!%#%&5#%(+9'/'+*!#.!/0+!#7/','X)/'#%!7$#18+,I!c0#8'7#3$!
+/!)8I!MNNP!)%;!d3D8'*#()&@3$-)*#()!+/!)8I!MNEP!)//$'13/+!/0+!
5)3*+!#.!7##$!$+5#%*/$35/'#%*!/#!'%)553$)/+!*8'5+!)8'-%,+%/*!
)%;!7$#7#*+!$+5#%*/$35/'#%!*/$)/+-'+*!/#!5#,1)/!,'*)8'-%+;!
*8'5+*!7#*/&0#5I!J3*!+/!)8I!MNWP!+9/+%;!MNEP!/#!)55#,,#;)/+!
;+.#$,)18+!,#/'#%!,#;+8*!)*!2#38;!1+!$+C3'$+;F!+I-IF!'%!/0+!
$+5#%*/$35/'#%!#.!/0+!78)5+%/)I!^0'8+!/0+!.#53*!#.!#3$!2#$D!
'*!#%!1$)'%!$+5#%*/$35/'#%F!#3$!*37+$('*+;!8+)$%'%-!)77$#)50!
*','8)$86!)55#,,#;)/+*!;'e+$+%/!,#/'#%!,#;+8*!<$'-';!)%;!
;+.#$,)18+?!16!+%5#;'%-!$+-38)$'/6!#.!,#/'#%!;'$+5/86!'%/#!
/0+!/$)'%'%-!,#/'#%!a+8;*!2'/0#3/!);;'/'#%)8!$+-38)$'X)/'#%I!!
! G+(+$)8!GHA!)77$#)50+*!+9/+%;!MNTP!16!$+7)$),+/$'X'%-!
*8'5+!,#/'#%!.#$!)!,#$+!.)(#$)18+!5#,73/)/'#%)8!5#,78+9'/6!
#$!f+9'1'8'/6!#.!/0+!*8'5+!,#/'#%I!d',!+/!)8I!MN=P!7$#7#*+!/#!
8#2+$!/0+!5#*/!#.!,#/'#%!+*/',)/'#%!16!)8'-%'%-!N>!8'%+*!#.!
'%/+$*+5/'#%!1+/2++%!=>!*8'5+*I!U8)%*)$6!+/!)8I!MNYP!7$#7#*+!
/#!0)%;8+!;+.#$,)18+!*8'5+!,#/'#%!16!7)$/'/'#%'%-!*8'5+*!'%/#!
*C3)$+!7)/50+*!)%;!+*/',)/'%-!)!*+7)$)/+!*+/!#.!$'-';!,#/'#%!
7)$),+/+$*!.#$!+)50!7)/50I!Z#2+(+$F!/0'*!*/$)/+-6!7$#;35+*!
)!7)/50&2'*+!8'%+)$!)77$#9',)/'#%!#.!/0+!3%;+$86'%-!,#/'#%!
)%;!8+);*!/#!18#5D!)$/'.)5/*!'%!/0+!$+5#%*/$35/+;!(#83,+I!B%!
#3$!2#$DF!2+!$+7$+*+%/!*8'5+!,#/'#%!)*!;+%*+!,#/'#%!a+8;*!
/#!+%)18+!*/$)'-0/&.#$2)$;!)$/'.)5/&.$++!E>!$+5#%*/$35/'#%I!
! B%!#7/','X)/'#%&1)*+;!GHA!,+/0#;*F!-##;!'%'/')8'X)/'#%!
#.!/0+!E>!(#83,+!)%;!/0+!*8'5+!,#/'#%!5)%!0+87!,'/'-)/+!/0+!
;+8+/+$'#3*!+e+5/*!#.!8#5)8!,'%',)!'%!/0+!#14+5/'(+!.3%5/'#%!
#%!/0+!$+5#%*/$35/'#%I!d)'%X!+/!)8I!MNQP!7$#7#*+!/+50%'C3+*!
/#!)3/#,)/'5)886!*+8+5/!)!$+.+$+%5+!*8'5+!*/)5D!/#!',7$#(+!/0+!
'%'/')8'X)/'#%!#.!/0+!E>!(#83,+I!g#3$1'+$!+/!)8I!MNRP!'%'/')8'X+!
/0+!E>!(#83,+!2'/0!)%!)-+&,)/50+;!/+,78)/+!',)-+F!20'50!
)5/*!)*!7$'#$!D%#28+;-+!'%!/0+!$+5#%*/$35/'#%!.$),+2#$DI!]6!
5#%/$)*/F!2+!/)*D!#3$!.3886!5#%(#83/'#%)8!%+/2#$D!,#;+8!2'/0!
$#13*/86!;+/+$,'%'%-!*8'5+!,#/'#%!'%!/0+!7$+*+%5+!#.!',)-+!
%#%&5#%(+9'/'+*F!)88#2'%-!'/!/#!;'*7+%*+!2'/0!/+,78)/+*!)%;!
7$#;35+!*314+5/&!)%;!7)/0#8#-6&*7+5'a5!$+5#%*/$35/'#%*I!
2.2. Learning-Based Approaches
! [3$$+%/86F!/0+! ,)4#$'/6!#.! ,)50'%+&8+)$%'%-&1)*+;! GHA!
,+/0#;*!M=VO=YP!7$+;'5/!/0+!7#*'/'#%!7)$),+/+$*!#.!*8'5+*!<T!
Figure 2: Supervised learning. We train our splat–slice U-Net (a) on paired (slice stack, motion stack) examples, which are generated by
slicing 3D volumes with randomized slice-wise motion (motion stack). We additionally train a fully convolutional interpolation network (b)
on paired (splat volume, 3D volume) examples, where splat volumes are generated by splatting slice stacks with ground truth motion.
!
!
(a) Single-Stack Motion Estimation (b) Supervised Interpolation
!
Slice Stack
Splat Volume
3D Volume
𝐲
ℒ
Interpolator
𝐱
!
𝐲
ℒ
𝐱
3D Volume
Slice Stack
Motion Stack
Splat–Slice
U-Net
Slice
Splat
!
!
%3,1+$*?!2'/0'%!/0+!3%;+$86'%-!@AB!(#83,+!16!3*'%-!)!=>!
5#%(#83/'#%)8!%+3$)8!%+/2#$D!<[\\?I!L+!7$+;'5/+;!7#*'/'#%!
7)$),+/+$*!)$+!/0+%!3*+;!/#!2)$7!/0+!*8'5+*!)%;!$+5#(+$!/0+!
3%;+$86'%-!@AB!(#83,+I!L+![\\!,#;+8!#.!Z#3!+/!)8I!M=VP!
5#%*'*/*!#.!/0+!5#%(#83/'#%)8!8)6+$*!#.!Hcc&NR!7$+&/$)'%+;!
#%!B,)-+\+/!MQQP!)%;!)!;+%*+!5#%%+5/+;!0+);!/#!7$+;'5/!/0+!
)%50#$!7#'%/*!<Uh?!#.!/0+!'%;'(';3)8!*8'5+*I!i+ 3% -!+/!)8 I!M=QP!
+9/+%;!M=VP!)%;!7$+;'5/!/0+!7#*'/'#%!7)$),+/+$*!#.!)88!*8'5+*!
4#'%/86F!'%/$#;35'%-!)%!'%/+$&*8'5+!)//+%/'#%!8)6+$!1+/2++%!/0+!
+9/$)5/+;!.+)/3$+*!)%;!/0+!#3/73/!/#!5#%;'/'#%)886!7$+;'5/!/0+!
,#/'#%!.#$!+)50!*8'5+!-'(+%!/0+!.+)/3$+*!#.!#/0+$!*8'5+*I!
! @#$+!$+5+%/86F! )!%#%&5#%(#83/'#%)8!,#;+8!.#$!GHA!2)*!
7$#7#*+;!16!j3!+/!)8I!M=YP!'%!)%!3%*37+$('*+;!*+//'%-!3*'%-!
)!('*'#%!/$)%*.#$,+$!,#;+8!MQYOQSPI!U8/0#3-0!/0'*!7$#;35+*!
7$#,'*'%-!$+*38/*F!'/!*5)8+*!C3);$)/'5)886!2'/0!/0+!%3,1+$!#.!
*8'5+*!;3+!/#!'%/+$&*8'5+!)//+%/'#%!5#,73/)/'#%*!)%;!$+C3'$+*!
067+$7)$),+/+$!/3%'%-!/#!1)8)%5+!;'e+$+%/!/+$,*!'%!/0+!8#**!
.3%5/'#%I!]6!7#*'%-!/0+!GHA!/)*D!)*!)!*37+$('*+;!$+-'*/$)/'#%!
7$#18+,!2'/0'%!)!5#%(#83/'#%)8!.$),+2#$DF!2+!*'-%'a5)%/86!
8+**+%!/0+!5#,78+9'/6!#.!,#;+8'%-!7)'$2'*+!*8'5+!'%/+$)5/'#%!
/0$#3-0!5#%(#83/'#%)8!2+'-0/!*0)$'%-!)%;!#1(')/+!/0+!%++;!
.#$!067+$7)$),+/+$!#7/','X)/'#%!'%!/0+!8#**!.3%5/'#%I!
! L+!7$#18+,!#.!$+-$+**'%-!*8'5+!7#*'/'#%!<#$!+(+%!,#/'#%?!
)-)'%*/!'%/+%*'/6!;)/)!'*!0'-086!%#%&5#%(+9!;3+!/#!/0+!',)-+!
8)%;*5)7+!MNTPI!^0'8+!/0+$+!0)(+!1++%!+e#$/*!/#!#(+$5#,+!
',)-+!%#%&5#%(+9'/'+*!'%!8+)$%'%-&1)*+;!',)-+!$+-'*/$)/'#%!
MENF!E=PF!/0+*+!5#%5+$%*!)$+!%#/!);;$+**+;!'%!8+)$%'%-&1)*+;!
7#*+!7$+;'5/'#%!2#$D*!M=QF!=YP!)%;!)$+!+9)5+$1)/+;!.3$/0+$!
16!*+7)$)/+!7$#5+**'%-!#.!/0+!'%;'(';3)8!*8'5+*!M=VPI!h#*'%-!
GHA!)*!',)-+!$+-'*/$)/'#%!)88#2*!3*!/#!)77+)8!/#!+I-I!MENF!E=P!
/#!);;$+**!/0+!.3%;),+%/)886!%#%&5#%(+9!%)/3$+!#.!GHAI!
2.3. Pre- and Post-Processing
! GHA!/+50%'C3+*!/67'5)886!'%(#8(+!*8'5+!7$+&7$#5+**'%-!)*!
)!a$*/!*/+7!/#!$+4+5/!*8'5+*!+90'1'/'%-!5#%*';+$)18+!'%/$)&*8'5+!
,#/'#%!)%;!*+-,+%/!$+8+()%/F!+I-IF!/0+!1$)'%F!*/$35/3$+*!.$#,!
/0+!*5)%I!U%C3+X!+/!)8I!MQWP!7$#7#*+!)!,+/0#;!/#!*D388&*/$'7!
.+/)8!(#83,+*F!20'50!5)%!1+!3*+;!)8*#!/#!,)*D!#3/!%#%&1$)'%!
*/$35/3$+*!'%!@A!*8'5+*!.#$!*,)88!'%/+$&*8'5+!,#/'#%I!g#3$1'+$!
+/!)8I!MNRP!7$#7#*+!/#!8#5)8'X+!1$)'%!$+-'#%*!'%!@A!*8'5+*!16!
18#5D&,)/50'%-!)%!)-+&,)/50+;!/+,78)/+!/#!/0+!*8'5+*!/0+%!
$+-'*/+$'%-!/0+!,)/50+;!$+-'#%*!1)5D!/#!/0+!/+,78)/+I!d)'%X!
+/!)8I!MQTP!8#5)8'X+!1$)'%!$+-'#%*!1)*+;!#%!$#/)/'#%&'%()$')%/!
;+*5$'7/#$*!.#$!8)1+8'%-!1$)'%I!d+$)3;$+%!+/!)8I!MYVF!YNP!3*+!)!
;+%*+!GB"g!)%;!$)%;#,!.#$+*/!58)**'a5)/'#%!.#$!(#9+8&8+(+8!
*+-,+%/)/'#%I!G#,+!*0#$/5#,'%-*!#.!/0+*+!,+/0#;*!'%583;+!
0'-0!5#,73/)/'#%)8!5#,78+9'/6!MNRF!QWP!)*!2+88!)*!/0+!%++;!
/#!0)%;5$)./!.+)/3$+*!MYVF!YNP!.#$!*+-,+%/)/'#%I!!
! k+)$%'%-&1)*+;!)77$#)50+*!/#!.+/)8!*+-,+%/)/'#%!'%583;+!
/0+!,+/0#;*!#.!A)4508!+/!)8I!MY=P!)%;!G)8+0'!+/!)8I!MYEPF!1#/0!
#.!20'50!3*+!"[\!,#;+8*!MYQP!/#!7$+;'5/!.#$+-$#3%;!,)*D*!
;'$+5/86!#%!*8'5+*I!"#$!7#*/&7$#5+**'%-!#.!/0+!$+5#%*/$35/+;!
E>!(#83,+F!j3!+/!)8I!MYYP!7$#7#*+!%+3$)8!(#83,+!$+%;+$'%-!
*','8)$!/#!\+A"!MYRP!/#!*37+$&$+*#8(+!$+5#%*/$35/'#%*!)/!)%!
)$1'/$)$6! $+*#83/'#%I! B%! /0'*! 2#$DF! 2+! );;'/'#%)886! /$)'%! )%!
'%/+$7#8)/'#%!%+/2#$D!/#!7#*/7$#5+**!/0+!$)2!$+5#%*/$35/'#%*!
)%;!'%/+$7#8)/+!#(+$!$+-'#%*!#.!,'**'%-!'%/+%*'/'+*!/0)/!,)6!
)77+)$!'%!/0+!*'%-8+&*/)5D!$+5#%*/$35/'#%!5)*+F!7$#;35'%-!)%!
)$/'.)5/&.$++!$+5#%*/$35/'#%!.#$!#/0+$!;#2%*/$+),!/)*D*I!
3. Mathematical Preliminaries
3.1. Slice Acquisition Model
! B%!0'-0&$+*#83/'#%!E>!@AB!#.!*314+5/*!+90'1'/'%-!*+(+$+!
3%5#%/$#88)18+!,#/'#%F!.)*/!*8'5+!)5C3'*'/'#%!*+C3+%5+*!*350!
)*!*'%-8+!*0#/!.)*/!*7'%!+50#!)$+!3*+;!/#!l.$++X+m!/0+!,#/'#%!
'%&78)%+F!*31*/)%/')886!,'/'-)/'%-!,#/'#%!)$/'.)5/*!5#,7)$+;!
/#!,38/'&*0#/!,+/0#;*!MYSPI!GHA!7$#5+;3$+*!)',!/#!)8'-%!)%!
)5C3'$+;!*/)5D!#.!*8'5+*!/0+$+16!$+,#('%-!/0+!$+,)'%'%-!<'%&
78)%+!)%;!/0$#3-0&78)%+?!,#/'#%!)5$#**!*8'5+*I!
! k+/!3*!;+%#/+!/0+!5##$;'%)/+*!'%*';+!/0+!@AB!*5)%%+$!16!
(𝑥, 𝑦, 𝑧)!)%;!)**3,+!)!*/)5D!𝑓!#.!=>!*8'5+*!'*!)5C3'$+;!)8#%-!
/0+!𝑧!)9'*!2'/0!*7)5'%-!𝑠!.$#,!)%!3%;+$86'%-!(#83,+!𝑣I!L+!
,#;+8!.#$!/0+!)5C3'*'/'#%!#.!/0+!𝑘 /0!@A!*8'5+!'*!MNNF!NTP`!
!
𝑓
𝑘
(𝑥, 𝑦) = 𝜀 + ∫ (𝑣 ∘ 𝑢
𝑘
)(𝑥, 𝑦, 𝑧)ℎ(𝑠𝑘 − 𝑧) d𝑧
∞
−∞
!!
<N?!
.#$!𝑘 = 1, . . . , 𝐾F! 20+$+!𝑢
𝑘
: (𝑥, 𝑦, 𝑧) ↦ (𝑥, 𝑦, 𝑧)
′
!;+%#/+*!
)%!3%#1*+$(+;!*8'5+&()$6'%-!$'-';!*7)/')8!*314+5/!,#/'#%F!ℎ!
;+%#/+*!/0+!7#'%/!*7$+);!.3%5/'#%!#.!*8'5+!)5C3'*'/'#%F!)%;!𝜀!
5)7/3$+*!)88!@A&'%;35+;!+$$#$*!*350!)*!%#'*+F!1')*F!+/5I!
! g#!;'*5$+/'X+!,#;+8!<N?F!*377#*+!𝑣!'*!*),78+;!#%!)!531'5!
8)//'5+!#.!𝑁
3
!7#'%/*!)%;!𝑓
𝑘
!#%!)!=>!8)//'5+!#.!𝑁
2
!7#'%/*I!^+!
5)%!2$'/+!/0+!;'*5$+/'X+;!,#;+8!<37!/#!$)%;#,!%#'*+?!)*!
!
𝐟 =
[
𝐟
1
; . . . ; 𝐟
𝐾
]
∈ ℝ
𝑁
2
𝐾
, 𝐟
𝑘
= 𝐇
𝑘
𝐔
𝑘
𝐯 ∈ ℝ
𝑁
2
,!
<=?!
'%!20'50!𝐔
𝑘
!)%;!𝐇
𝑘
!$+7$+*+%/!;'*5$+/'X)/'#%*!#.!(∘ 𝑢
𝑘
)!)%;!
ℎ(𝑠𝑘 − ⋅)F!$+*7+5/'(+86I!B%!/0+!*,)88!,#/'#%!$+-',+F!2+!5)%!
.3$/0+$!)77+)8!/#!g)68#$n*!)77$#9',)/'#%!)%;!;+a%+!
!
𝐔
𝑘
𝐯 = 𝐯 +
[
𝐝𝐢𝐚𝐠(𝐯
𝑥
) , 𝐝𝐢𝐚𝐠(𝐯
𝑦
) , 𝐝𝐢𝐚𝐠(𝐯
𝑧
)
]
𝐒𝐮
𝑘
,!
<E?!
'%!20'50!𝐒𝐮
𝑘
$+7$+*+%/*!/0+!+97)%*'#%!#.!/0+!5#+K5'+%/*!#.!
,#/'#%!𝐮
𝑘
'%!)!1)*'*!𝐒 )%;!𝐯
{𝑥,𝑦,𝑧}
!)$+!/0+!$+*7+5/'(+!*7)/')8!
;+$'()/'(+*!#.!/0+!E>!(#83,+I!^+!)55#,,#;)/+!;+.#$,)18+!
,#/'#%!,#;+8*!16!*+//'%-!𝐒!/#!/0+!';+%/'/6!,)/$'9F!'%!20'50!
5)*+F!𝐮
𝑘
;+%#/+*!/0+!(+5/#$'X)/'#%!#.!)!E>!,#/'#%!a+8;I!
3.2. Classical SVR Methods
! "#$! 1$+('/6F! 8+/! 3*! 2$'/+!𝐔 =
[
𝐇
1
𝐔
1
; . . . ; 𝐇
𝐾
𝐔
𝐾
]
!)%;!
/0+!)**#5')/+;!,#/'#%!7)$),+/+$*!)*!𝐮 =
[
𝐮
1
; . . . ; 𝐮
𝐾
]
I!^+!
5)%!2$'/+!/0+!GHA!7$#18+,!2'/0!/0$++!#$/0#-#%)8!*/)5D*!)*!
!
minimize 𝐷(𝐮
{1,2,3}
, 𝐯
{1,2,3}
) =
‖
𝐔
1
𝐯
1
− 𝐟
1
‖
2
2
!
+
‖
𝐔
2
𝐯
2
− 𝐟
2
‖
2
2
+
‖
𝐔
3
𝐯
3
− 𝐟
3
‖
2
2
!
subject to 𝐯
1
− 𝐯
2
= 𝐯
2
− 𝐯
3
= 𝐯
3
− 𝐯
1
= 𝟎!
<Q?!
'%!20'50!𝐮
{1,2,3}
;+%#/+!/0+!7)$),+/+$*!#.!,#/'#%!'%!+)50!#.!!
/0+!*/)5D*I
!
!
'%.#$,)/'#%!8#**!MYWP!)$+!)8*#!7#738)$86!3*+;!'%!78)5+!#.!/0+!
C3);$)/'5!#%+!*0#2%!0+$+!.#$!*',78'5'/6I!Z+$+F!2+!'%/$#;35+!
#7/','X)/'#%! ()$')18+*!𝐯
{1,2,3}
!2'/0! +C3)8'/6! 5#%*/$)'%/*! )*!
#77#*+;!/#!)!*'%-8+!()$')18+!𝐯!/#!.)5'8'/)/+!*#8('%-!<Q?!3*'%-!
)8/+$%)/'%-!#7/','X)/'#%!*/$)/+-'+*!*','8)$86!/#!MNSF!NTPI!
! U!*',78+!)8/+$%)/'%-!#7/','X)/'#%!*/$)/+-6!20'50!;#+*!%#/!
'%/$#;35+!;3)8!()$')18+*!'*!/#!$+8)9!/0+!+C3)8'/6!5#%*/$)'%/*!/#!
/0+!C3);$)/'5!7+%)8/6!
!
𝑅
(
𝐯
{
1,2,3
}
)
=
‖[
𝐯
1
, 𝐯
2
, 𝐯
3
]
−
[
𝐯
2
, 𝐯
3
, 𝐯
1
]‖
2
2
!
<Y?!
)%;!#7/','X+!/0+!$+8)9)/'#%!
!
𝐽(⋅) = 𝐷(𝐮
{1,2,3}
, 𝐯
{1,2,3}
) + 𝜆𝑅(𝐯
{1,2,3}
),!
<R?!
'%!20'50!𝜆!'*!)!7)$),+/+$!5#%/$#88'%-!/0+!$+8)/'(+!2+'-0/*!#.!
/0+!;)/)!a;+8'/6!/+$,!𝐷!)%;!/0+!5#378'%-!/+$,!𝑅I!_14+5/'(+!
<R?!'%!.)5/!5#$$+*7#%;*!/#!/0+!#14+5/'(+!',78'5'/86!#7/','X+;!
16!A#3**+)3!+/!)8I!MNTP!20+%!/0+!,3/3)8!'%.#$,)/'#%!8#**!'*!
3*+;!'%!/0+!;)/)!a;+8'/6!/+$,!𝐷I!_7/','X'%-!<R?!3*'%-!18#5D!
5##$;'%)/+!;+*5+%/!6'+8;*!37;)/+*!.#$!𝐯
1
, 𝐮
1
, 𝐯
2
, 𝐮
2
, 𝐯
3
!)%;!
𝐮
3
F!20+$+!/0+!E>!$+5#%*/$35/'#%*!<𝑣&37;)/+?!)%;!/0+!,#/'#%!
7)$),+/+$*!<𝑢&37;)/+?!#.!+)50!*/)5D!)$+!37;)/+;!'%!/3$%!2'/0!
/0+!#/0+$!/2#!$+5#%*/$35/'#%*!a9+;!)/!+)50!/3$%I!
! B%!/0+!𝑛/0!'/+$)/'#%!#.!/0+!18#5D!5##$;'%)/+!;+*5+%/F!/0+!𝑣&
37;)/+!*/+7!.#$!)!-'(+%!*/)5D!),#3%/*!/#!5#,73/'%-!
!
𝐯
𝑛
← (𝐔
𝑛∗
𝐔
𝑛
+ 2𝜆𝐈)
−1
(𝐔
𝑛∗
𝐟
𝑛
+ 2𝜆𝐯
𝑛
),!
<S
?
!
'%!20'50!𝐯
𝑛
!)%;!𝐟
𝑛
;+%#/+!/0+!$+5#%*/$35/'#%!)%;!=>!*8'5+*!
#.!/0+!𝑛%3$;!*/)5DF!$+*7+5/'(+86F!)%;!𝐯
𝑛
;+%#/+*!/0+!)(+$)-+!
$+5#%*/$35/'#%!.$#,!/0+!/2#!*/)5D*!#$/0#-#%)8!/#!𝐯
𝑛
I!^+!5)%!
'%/+$7$+/!/0'*!37;)/+!)*!)!$+5#%*/$35/'#%!#.!)!%+2!E>!(#83,+!
.$#,!)!2+'-0/+;!*3,!#.!/0+!8)*/!)(+$)-+!E>!$+5#%*/$35/'#%!
)%;!2)$7+;!*8'5+*F!.#88#2+;!16!^'+%+$&;+5#%(#83/'#%I!L'*!
'%/+$7$+/)/'#%!#.!/0+!𝑣&37;)/+!2'88!-3';+!#3$!5#%*/$35/'#%!#.!
)!.3886!5#%(#83/'#%)8!%+/2#$D!8)/+$!'%!G+5!EIEI!!
! U8-+1$)'5)886F!(𝐔, 𝐔
∗
) ;+a%+!)%!);4#'%/!7)'$!#.!2)$7'%-!
#7+$)/#$*F!20'50!2+!2'88!$+.+$!/#!)*!l*8'5'%-m!)%;!l*78)//'%-m!
$+*7+5/'(+86!MYTPI!B%!/0+!5)*+!20+$+!/0+$+!'*!'%4+5/'(+!,#/'#%!
$+8)/'%-!)!*8'5+!*/)5D!𝐟
𝑛
!/#!)!E>!(#83,+!𝐯F!2+!5)%!#1/)'%!𝐟
𝑛
16!l*8'5'%-m!𝐯 2'/0!𝐔
𝑛
!)%;!#1/)'%!𝐯 16!l*78)//'%-m!𝐟
𝑛
!2'/0!
𝐔
𝑛∗
I!"'-3$+!E!'883*/$)/+*!/0+!/2#!);4#'%/!2)$7'%-!#7+$)/#$*!
'%!/0+!;'*5$+/+!5)*+!20+%!,38/'&8'%+)$!'%/+$7#8)/'#%!2+'-0/*!
)$+!3*+;!.#$!.$)5/'#%)8!*8'5'%-!)%;!*78)//'%-I!
! L+!𝑢&37;)/+!.#$!/0+!,#/'#%!#.!/0+!𝑘/0!*8'5+!'%!)!*/)5D!'*!
!
𝐮
𝑘
𝑛
← (𝐒
∗
𝐕
𝑘
𝑛∗
𝐕
𝑘
𝑛
𝐒)
−1
𝐒
∗
𝐕
𝑘
𝑛∗
(𝐇
𝑘
𝐯
𝑛
− 𝐟
𝑘
𝑛
),!
<W?!
'%!20'50!𝐕
𝑘
𝑛
= 𝐇
𝑘
[
𝐝𝐢𝐚𝐠(𝐯
𝑥
𝑛
) , 𝐝𝐢𝐚𝐠(𝐯
𝑦
𝑛
) , 𝐝𝐢𝐚𝐠(𝐯
𝑧
𝑛
)
]
F 2'/0!
/0+!𝐯
{𝑥,𝑦,𝑧}
𝑛
;+%#/'%-!/0+!$+*7+5/'(+!*7)/')8!;+$'()/'(+*!#.!/0+!
-'(+%!E>!(#83,+!$+5#%*/$35/+;!'%!/0+!(𝑛 − 1)/0!'/+$)/'#%I!
! B%!7$)5/'5+F!*314+5/!,#/'#%!'%!)%!@A!*8'5+!8'D+86!+95++;*!
#%+!(#9+8!'%!,)-%'/3;+!13/!/0+!8'%+)$!,#;+8!#.!2)$7'%-!<E?!
*/)6*!()8';!#%86!'.!/0+!*8'5+!'*!a$*/!183$$+;!)/!/0+!*5)8+!#.!/0+!
,#/'#%I!>3+!/#!/0'*F!,#/'#%!'*!#./+%!+*/',)/+;!#%!)!76$),';!
'%!)!5#)$*+&/#&a%+!,)%%+$F!3*'%-!/0+!,#/'#%!a+8;!+*/',)/+;!
)/!5#)$*+$!8+(+8*!/#!'%'/')8'X+!,#/'#%!)/!a%+$!#%+*!MRVPI!@AB!
)5C3'*'/'#%*!)8*#!*3e+$!.$#,!)!1')*!a+8;!;3+!/#!%#%&3%'.#$,!
,)'%!,)-%+/'5!a+8;!)%;!$);'#.$+C3+%56!5#'8*F!'%/$#;35'%-!)!
-8#1)8!'%/+%*'/6!*0'./!1+/2++%!/0+!3%D%#2%!(#83,+!)%;!/0+!
*8'5+*I!U!D+6!50)88+%-+!'%!#7/','X)/'#%&1)*+;!GHA!,+/0#;*!
'*!;+/+$,'%'%-F!)/!+)50!*5)8+F!,#/'#%!'%!/0+!7$+*+%5+!#.!1')*!
)%;!+9/$+,+!*8'5+!)$/'.)5/*I!@'*/)D'%-!+'/0+$!.#$!,#/'#%!5)%!
C3'5D86!8+);!/#!7'/.)88*!#.!8#5)8!,'%',)!'%!/0+!#7/','X)/'#%!
#14+5/'(+!+*7+5')886!)/!5#)$*+!*5)8+*I!Z#2+(+$F!'/!'*!;'K538/!
/#!0)%;&5$)./!)!$+5#%*/$35/'#%!7'7+8'%+!20'50!5)%!#(+$5#,+!
',)-+!%#%&5#%(+9'/'+*!;3+!/#!*0++$!%3,1+$*!#.!)5C3'*'/'#%&
*7+5'a5!;+*'-%!50#'5+*!/#!1+!,);+!<+I-IF!%#%&8'%+)$!a8/+$*?I!
4. Fully Convolutional SVR
! J*'%-!)!"[\!,#;+8!.#$!GHA!+%)18+*!3*!/#!167)**!0)%;&
5$)./+;!)5C3'*'/'#%&*7+5'a5!$+5#%*/$35/'#%!7'7+8'%+*!)*!2+88!
)*!%3,+$'5)8!#7/','X)/'#%I!A)/0+$!/0)%!.#$,38)/'%-!GHA!)*!
/0+!7$+;'5/'#%!#.!)1*#83/+!*8'5+!5##$;'%)/+*!'%!E>!*7)5+!)*!'*!
/67'5)886!;#%+!'%!#/0+$!8+)$%'%-&1)*+;!,+/0#;*!M=WOE=PF!2+!
5)*/!GHA!)*!/0+!$+-'*/$)/'#%!#.!*#,+!3%#1*+$(+;!E>!(#83,+!
/#!)%!#1*+$(+;!*/)5D!#.!=>!*8'5+*I!^+!/$)'%!)!"[\!,#;+8!/#!
7$+;'5/!,#/'#%!$+8)/'%-!/0+!*8'5+*!/#!)!E>!(#83,+!-'(+%!#%86!
/0+!*8'5+*!)*!'%73/F!7$#;35'%-!)!E>!(#83,+!)*!)!16&7$#;35/!
#.!$+-'*/$)/'#%I
!
!
<*7)5+&()$6'%-!/$)%*.#$,*?!'*!'%0+$+%/86!'%+K5'+%/F!1#/0!'%!
/+$,*!#.!7)$),+/$'X)/'#%!)%;!5#,73/)/'#%o!*++F!+I-IF!MEEF!ETPI!
! _3$!%+/2#$D!,#;+8!<"'-3$+!Q?!1+)$*!*','8)$'/'+*!/#!/0#*+!
3*+;!'%!7)'$2'*+!',)-+!$+-'*/$)/'#%!M=WOE=PF!20'50!7$+;'5/!
',)-+!,#/'#%!16!7$#5+**'%-!/0+!',)-+!7)'$!)5$#**!,38/'78+!
8+(+8*!#.!5#%(#83/'#%*I!L+*+!%+/2#$D!,#;+8*!)$+!/67'5)886!
+C3'77+;!2'/0!2)$7'%-!8)6+$*!1+/2++%!);4)5+%/!8+(+8*!*350!
/0)/!#%86!/0+!$+*';3)8!,#/'#%!%++;*!/#!1+!+*/',)/+;!)/!+(+$6!
8+(+8I!\#/'#%)886F!/0+!*8'5+!*/)5D!)**3,+*!/0+!$#8+!#.!)!a9+;!
',)-+!20'8+!/0+!E>!(#83,+F!20'50!2+!*++D!/#!+*/',)/+F!5)%!
1+!*++%!)*!)!,#('%-!',)-+I!U*!;'*53**+;!'%!G+5I!EI=F!2+!5)%!
#1/)'%!/0+!.+)/3$+*!#.!/0+!=>!*8'5+*!<a9+;!',)-+?!16!*8'5'%-!
/0+!.+)/3$+*!#.!/0+!E>!(#83,+!2'/0!/0+!+*/',)/+;!,#/'#%I!g#!
#1/)'%!/0+!.+)/3$+*!#.!/0+!E>!(#83,+F!2+!*78)/!/0+!.+)/3$+*!
#.!/0+!*8'5+!*/)5D!3*'%-!/0+!*),+!,#/'#%!+*/',)/+I!
! \#/+!/0)/!'%!#3$!5)*+F!#%86!/0+!a9+;!',)-+!<*8'5+!*/)5D?!'*!
-'(+%F!*#!/0+!;#2%2)$;!/$)%*.#$,!%++;*!/#!7$#5+**!#%86!/0+!
a9+;!',)-+I!L+!372)$;!/$)%*.#$,!*6%/0+*'X+*!a9+;!',)-+!
.+)/3$+*!16!3%7##8'%-!8#2+$&$+*#83/'#%!#%+*F!5#%5)/+%)/'%-!
/0+,!2'/0!/0+!*D'7!.+)/3$+*!)%;!7$#5+**'%-!/0+,!(')!)!*+$'+*!
#.!5#%(#83/'#%*I!L+!,#('%-!',)-+!<#$!E>!(#83,+?!.+)/3$+*!
)$+!*6%/0+*'X+;!'%!7)$)88+8!16!3%7##8'%-!/0+'$!8#2&$+*#83/'#%!
5#3%/+$7)$/*F!5#%5)/+%)/'%-!/0+,!2'/0!/0+!*78)//'%-*!#.!*D'7!
.+)/3$+*!)%;!7$#5+**'%-!/0+,!(')!5#%(#83/'#%*I!L+!,#('%-!
',)-+!.+)/3$+*!)$+!a%)886!*8'5+;F!5#%5)/+%)/+;!2'/0!/0+!a9+;!
',)-+!.+)/3$+*!)%;!7$#5+**+;!3*'%-!)!*+$'+*!#.!5#%(#83/'#%*!
/#!#3/73/!$+*';3)8!*8'5+!,#/'#%F!20'50!'*!);;+;!/#!/0+!8#2+$&
$+*#83/'#%!,#/'#%!/0)/!2)*!3*+;!;3$'%-!*78)//'%-!)%;!*8'5'%-I!
! B%!/+$,*!#.!',78+,+%/)/'#%F!1#/0!/0+!*8'5+!*/)5D!)%;!/0+!
E>!(#83,+!.+)/3$+*!)$+!*++%!)*!<]F[F>FZF^?!/+%*#$*F!20+$+!
/0+!>!;',+%*'#%!)5/*!)*!/0+!*8'5'%-!;',+%*'#%!'%!/0+!5)*+!#.!
*8'5+!*/)5D!.+)/3$+*I!E>!(#83,+!.+)/3$+*!)$+!7$#5+**+;!3*'%-!
5#%(#83/'#%!)%;!7##8'%-!D+$%+8*!#.!*0)7+!<EFEFE?!)%;!<=F=F=?!
$+*7+5/'(+86F!20'8+!*8'5+!.+)/3$+*!)$+!7$#5+**+;!3*'%-!D+$%+8*!
#.!*0)7+*!<NFEFE?!)%;!<NF=F=?F!$+*7+5/'(+86F!*#!/0)/!+)50!*8'5+!
5)%!1+!7$#5+**+;!'%;+7+%;+%/86I!^'/0'%!/0+!$+*';3)8!,#/'#%!
+9/$)5/'#%!*/)-+*F!/0+!a$*/!5#%(#83/'#%*!)$+!7+$.#$,+;!2'/0!
D+$%+8*!#.!*0)7+!<QFEFE?!)%;!*/$';+*!#.!<QFNFN?!*'%5+!*8'5+*!)$+!
'%/+$%)886!$+7$+*+%/+;!)*!E>!*8)1*!2'/0!/0'5D%+**!#.!Q!(#9+8*!
)%;!/0+!,#/'#%!%++;*!/#!1+!$+-38)$'X+;!)5$#**!+)50!*8)1I!^+!
0)%;8+!;'e+$+%/!*8'5+!*7)5'%-*!'%!$+)8!;)/)!16!*5)8'%-!/0+!=>!
'%73/!*8'5+*!/#!)!Q`N!*8)1!/0'5D%+**!/#!(#9+8!*7)5'%-!$)/'#I!!
4.2. Rigid Motion-Compensating Loss
! B%+9)5/!7#*'/'#%'%-!#.!*314+5/*!'%!@AB!*5)%%+$*!/67'5)886!
'%/$#;35+*!)!-8#1)8!*314+5/!#e*+/!<$#/)/'#%*!)%;!/$)%*8)/'#%*?!
'%!/0+!)5C3'$+;!*8'5+*I!"#$!$+5#%*/$35/'#%F!0#2+(+$F!2+!%++;!
#%86!$+5#(+$!/0+!,#/'#%!#.!/0+!*8'5+*!$+8)/'(+86!/#!+)50!#/0+$!
'-%#$'%-!-8#1)8!*0'./*I!g#!+%*3$+!/0)/!2+!7+%)8'X+!7$+;'5/'#%!
+$$#$*!#%86!'%!/0+!$+8)/'(+!*8'5+!,#/'#%*F!2+!5#,7+%*)/+!.#$!
)%6!-8#1)8!$'-';!,#/'#%!*0'./!/0)/!,)6!+9'*/!'%!#3$!7$+;'5/+;!
,#/'#%!*/)5D!1+.#$+!5#,73/'%-!/0+!/$)'%'%-!8#**!)-)'%*/!/0+!
7$+;'5/'#%!/)$-+/I!U**3,'%-!)%!@Gp!/$)'%'%-!#14+5/'(+F!2+!
5)%!+97$+**!#3$!,#/'#%&5#,7+%*)/'%-!/$)'%'%-!8#**!1+/2++%!
/0+!7$+;'5/+;!𝐮!)%;!-$#3%;!/$3/0!𝐲 ∈ ℝ
𝑁×3
,#/'#%!a+8;*!)*!
!
𝓛
(
𝐮, 𝐲
)
= min
[
𝐑,𝐭
]
‖
𝐮 + 𝐩 −
(
𝐲 + 𝐩
)
𝐑 − 𝐭
‖
𝐹
2
,
!
<T
?
!
'%!20'50!𝐑!$+7$+*+%/*!)!3 × 3!$#/)/'#%!,)/$'9F!𝐭!$+7$+*+%/*!)!
1 × 3!/$)%*8)/'#%!(+5/#$F!)%;!𝐩 ∈ ℝ
𝑁×3
'*!)!,)/$'9!#.!(#9+8!
5##$;'%)/+*I!B%!2#$;*F!/0'*!8#**!a$*/!7+$.#$,*!)!-8#1)8!$'-';!
)8'-%,+%/!1+/2++%!/0+!7#'%/!58#3;*!𝐮 + 𝐩 )%;!𝐲 + 𝐩F!*350!
/0)/!#%86!/0+!%#%&$'-';!,#/'#%!5#,7#%+%/!1+/2++%!/0+!/2#!
7#'%/!58#3;*!'*!7+%)8'X+;I!k#**!𝓛!$+;35+*!/#!/0+!3*3)8!@Gp!
8#**!'%!/0+!)1*+%5+!#.!-8#1)8!$#/)/'#%!𝐑!)%;!/$)%*8)/'#%!𝐭I!
! g#!*#8(+!/0+!#7/','X)/'#%!7$#18+,!<T?F!2+!a$*/!,'%','X+!
/0+!3%5#%*/$)'%+;!(+$*'#%!#.!7$#18+,!<T?!#(+$!/0+!*7)5+!#.!
)88!
[
𝐑, 𝐭
]
∈ ℝ
3×4
/#!#1/)'%!)!,'%','X+$!
[
𝐑
′
, 𝐭
⋆
]
F!/0+%!.)5/#$!
𝐑
′
'%/#!'/*!$#/)/'#%!)%;!*0+)$!,)/$'5+*!)*!𝐑
′
= 𝐑
⋆
𝐒
⋆
I!"$#,!
/0+!*'%-38)$!()83+!;+5#,7#*'/'#%!𝐑
′
= 𝐔𝚺𝐕
∗
F!2+!*++!/0)/!
!
𝐑
⋆
= 𝐔sign
(
𝚺
)
𝐕
∗
, 𝐒
⋆
= 𝐕abs
(
𝚺
)
𝐕
∗
,!
<NV?!
)!.)5/#$'X)/'#%!D%#2%!)*!/0+!7#8)$!;+5#,7#*'/'#%!MRNPI!L+!
$+*38/'%-!$#/)/'#%!,)/$'9!𝐑
⋆
!'*!/0+!,'%',3,!,+)%!*C3)$+;&!
+$$#$!)77$#9',)/'#%!#.!𝐑
′
F!)%;!5#%*+C3+%/86F!/0+!*#83/'#%!#.!
/0+!#$'-'%)8!$'-';!,#/'#%&5#%*/$)'%+;!7$#18+,!<T?!'*!-'(+%!16!
/0+! 7)'$!
[
𝐑
⋆
, 𝐭
⋆
]
I!L+! 8#**!𝓛(𝐮, 𝐲) '*! /0+%! 1)5D7$#7)-)/+;!
)%;!/0+!2+'-0/*!#.!#3$!GHA!%+/2#$D!,#;+8!37;)/+;I!"'-3$+!
Figure 4: Splat–Slice U-Net. ?e stack of 2D slices goes through
2D feature extraction (downward path) and reconstruction (upward
path) with 2D skip features. 3D volume features are reconstructed
by splatting 2D skip features with previous motion to form 3D skip
features. At each level, the sliced 3D feature volume and the slice
feature stack are convolved jointly to extract residual slice motion.!
Splat
-
Concat
Concat
1
24
32
32
48
48
64
96
48
64
32
48
24
3
3
64
3
Splat
-
Concat
Concat
3
Å
Å
Å
2D Feat
3D Feat
Motion
Conv
Add
Å
Slice-Conv
Figure 5: Rigid motion-compensated loss. Validation MSE and
EPE (end-point error) [33] of the predicted motion shown. Our loss
(blue), which compensates for offsets in global rigid motion, trains
faster and attains higher final prediction accuracy than the MSE loss
(orange) and the loss that compensates for translations only (gray). !
Rigid-Comp
No Comp
Trans-Comp
Motion MSE (voxels)
Motion EPE (voxels)
Rigid-Comp
No Comp
Trans-Comp
Epochs
!
!
Y!78#/*!/0+!()8';)/'#%!53$(+*!.#$!#3$!%+/2#$D!,#;+8!/$)'%+;!
3*'%-!,#/'#%&5#,7+%*)/+;!)%;!$+-38)$!8#**+*F!*0#2'%-!/0)/!
#3$!$'-';!5#,7+%*)/'%-!8#**!'*!D+6!/#!',7$#('%-!7$+;'5/'#%I!
4.3. Interpolating Reconstructions
! B%!-+%+$)8F!)!E>!(#83,+!$+5#%*/$35/+;!.$#,!)!*'%-8+!*8'5+!
*/)5D!5)%!5#%/)'%!0#8+*!'.!$+-'#%*!#.!/0+!3%;+$86'%-!(#83,+!
)$+!,'**+;!'%!)88!*8'5+!)5C3'*'/'#%*!;3+!/#!*314+5/!,#/'#%I!B%!
5#,7)$'*#%!2'/0!%)/3$)8!',)-+*F!@A!*5)%*!/67'5)886!+90'1'/!
)!8','/+;!;'(+$*'/6!'%!',)-+$6!)%;!0)(+!)!*/$#%-!7$'#$!#%!/0+!
;'*/$'13/'#%!#.!3%;+$86'%-!E>!(#83,+*I!L'*!43*/'a+*!/0+!3*+!
#.!'%/+$7#8)/'#%!/+50%'C3+*!/#!a88!0#8+*!'%!$+5#%*/$35/'#%*!/#!
*)/'*.6!$+-38)$'/6!5#%*/$)'%/*!<+I-IF!#%!7)/0#8#-6?I!^+!/3$%!/#!
)!.3886!5#%(#83/'#%)8!'%/+$7#8)/'#%!,#;+8!/$)'%+;!'%!);()%5+!
#%!</$3+!$+5#%*/$35/'#%F!(#83,+?!7)'$*I!g$3+!$+5#%*/$35/'#%*!
<2'/0!0#8+*?!5)%!1+!#1/)'%+;!$+);'86!)/!/$)'%!/',+!16!2)$7'%-!
*8'5+!*/)5D*!2'/0!-$#3%;!/$3/0!*8'5+!,#/'#%I!^0'8+!/0+!*+)$50!
.#$!/0+!1+*/!'%/+$7#8)/'#%!%+/2#$D!,#;+8!'*!%#/!/0+!.#53*!#.!
/0'*!2#$DF!2+!a%;!/0)/!)!*/)%;)$;!J&\+/!,#;+8!/$)'%+;!2'/0!
/$3+!$+5#%*/$35/'#%*!)*!/0+!/)$-+/!)%;!3*'%-!/0+!,+)%!*C3)$+;!
+$$#$!8#**!7$#;35+*!-##;!$+*38/*I!"'-3$+!=!<$'-0/?!*0#2*!/0+!
/$)'%'%-!7$#5+;3$+!.#$!#3$!.3886!5#%(#83/'#%)8!'%/+$7#8)/#$I!!
5. Experimental Results
! ^+! $3%!+9/+%*'(+!+97+$',+%/*!#%!1#/0!);38/!@AB!*/)5D*!
<.#$!20'50!-$#3%;!/$3/0!@A!(#83,+*!)$+!)()'8)18+?!)%;!.+/)8!
#%+*F!.#$!20'50!#%86!$+.+$+%5+!(#83,+*!)$+!-'(+%I!"#$!+)50!
+97+$',+%/F!2+!/$)'%!)!GHA!%+/2#$D!'%!)!*37+$('*+;!.)*0'#%!
#%!7)'$+;!<*8'5+!*/)5DF!,#/'#%!*/)5D?!;)/)!)%;!)%!'%/+$7#8)/#$!
%+/2#$D!#%!7)'$+;!<*8'5+O*78)/!(#83,+F!3%;+$86'%-!(#83,+?!
;)/)!<G+5I!QIE?I!L+!;)/)!)3-,+%/)/'#%!067+$7)$),+/+$*!)%;!
/$)'%'%-!;+/)'8*!)$+!-'(+%!'%!U77+%;'9!UI!^+! 3*+!/0+!,+)%!
*C3)$+;!+$$#$!<@Gp?!)%;!/0+!)(+$)-+!+%;&7#'%/!+$$#$!<php?!
MEEP!/#!,+)*3$+!7$+;'5/'#%!+$$#$o!*++!U77+%;'9!UI!
5.1. Single-Stack SVR of Adult Brains
! GHA!+97+$',+%/*!#%!*6%/0+/'5!*8'5+;!@AB!(#83,+*!5)%!
$+(+)8!C3)8'/)/'(+!)%;!C3)%/'/)/'(+!50)$)5/+$'*/'5*!#.!)!GHA!
,+/0#;!16!.)5'8'/)/'%-!5#,7)$'*#%!)5$#**!/0+!$+5#%*/$35/+;!
)%;!/0+!-$#3%;!/$3/0!E>!@A!(#83,+*I!^+!53$)/+!NNVV!);38/!
1$)'%!@A!*5)%*!.$#,!U]B>pF!&=!MR=PF!U>Z>!MREPF![_]Ap!
MRQPF!cGh!MRYPF!@[B[!MRRPF!_UGBG!MRSPF!hh@B!MRWPF!Jd]!
MRTP!)%;!]ET!MSVPF!)%;!*78'/!/0+!*5)%*!'%/#!NVVV!/$)'%'%-!)%;!
NVV!()8';)/'#%!+9),78+*I!U88!*5)%*!0)(+!)!=YR
3
!(#83,+!2'/0!
N,,
3
!$+*#83/'#%I!L+!*5)%*!)$+!)8'-%+;!/#!/0+!g)8)'$)50!)/8)*!
MSNP!/#!.)5'8'/)/+!*8'5'%-!)8#%-!/0+!*)-'//)8F!)9')8!)%;!5#$#%)8!
;'$+5/'#%*o!*++!U77+%;'9!U!.#$!;+/)'8*!#%!/$)'%'%-!)%;!*),78+!
-+%+$)/'#%I!!
Slicing Directions and Reconstruction. G'%-8+&*/)5D!GHA!
5)%!)50'+(+!;'e+$+%/!$+5#%*/$35/'#%!)553$)56!;+7+%;'%-!#%!
/0+!;'$+5/'#%!#.!*8'5'%-!<*)-'//)8F!)9')8!)%;!5#$#%)8?I!g#!*/3;6!
/0+!+e+5/!#.!*8'5+!;'$+5/'#%!#%!/0+!)553$)56!#.!/0+!7$+;'5/+;!
,#/'#%F!2+!/$)'%!)!*+7)$)/+!%+/2#$D!.#$!+)50!*8'5+!;'$+5/'#%!
)%;!78#/!'%!"'-3$+!RF!/0+!()8';)/'#%!@Gp!)%;!p%;7#'%/!p$$#$!
#.!/0+!,#/'#%!7$+;'5/+;!16!+)50!%+/2#$DI!L+!a%)8!()8';)/'#%!
+$$#$!)//)'%+;!'*!/0+!8#2+*/!.#$!*)-'//)8!*/)5D*!)/!VIWT!<@Gp?!
)%;!/0+!0'-0+*/!.#$!)9')8!*/)5D*!)/!NIEE!<@Gp?I!G350!)!*8'-0/!
'%5$+)*+!'%!/0+!7$+;'5/'#%!+$$#$!.#$!)9')8!*/)5D*!,#*/!8'D+86!
*/+,*!.$#,!.+2+$!l5#$%+$m!.+)/3$+*!/#!/$)5D!)%;!)8'-%!'%!)9')8!
*8'5+*F!8+);'%-!/#!)!8+**!7$+5'*+!)8'-%,+%/I!
Motion Prediction Error.!^+! 78#/!;'*/$'13/'#%*!#.!/0+!7+$&!
*314+5/!+$$#$*!#.!/0+!7$+;'5/+;!,#/'#%!*/)5D*!)5$#**!YVV!0+8;&
#3/!*5)%*!'%!"'-3$+!SI!U*!+97+5/+;!.$#,!/0+!()8';)/'#%!+$$#$!
53$(+*!*++%!'%!"'-3$+!RF!*)-'//)8!7$+;'5/'#%*!0)(+!/0+!8#2+*/!
,+;')%!,#/'#%!@Gp!)%;!/0+!/'-0/+*/!'%/+$C3)$/'8+!$)%-+I!_%!
)(+$)-+F!#3$!7$+;'5/+;!,#/'#%!'*!)553$)/+!/#!8+**!/0)%!)!(#9+8!
)8/0#3-0!/0+!hh@B!;)/)*+/!+90'1'/*!*8'-0/86!0'-0+$!+$$#$*I!
Reconstruction and Interpolation.!"'-3$+!W!('*3)8'X+*!/0+!
E>!$+5#%*/$35/'#%*!#1/)'%+;!3*'%-!/0+!7$+;'5/+;!,#/'#%I!^+!
)8'-%!)88!$+5#%*/$35/'#%*!1)5D!/#!/0+!g)8)'$)50!)/8)*!MSNP!.#$!
5#,7)$'*#%!)5$#**!;'e+$+%/!$+5#%*/$35/'#%*I!^+!)8*#!'%583;+!
$+5#%*/$35/'#%*!2'/0!/0+!,'**'%-!'%/+%*'/6!;)/)!'%/+$7#8)/+;!
3*'%-!#3$!'%/+$7#8)/#$!;+*5$'1+;!'%!G+5I!QIEI!L+!*78)/!$+*38/*!
*0#2!/0)/!#3$!%+/2#$D!7$+;'5/*!8'%+)$!,#/'#%!a+8;*!2'/0#3/!
/0+!%++;!.#$!);;'/'#%)8!8'%+)$!7$#4+5/'#%*I!L+!'%/+$7#8)/'#%!
$+,#(+*!0#8+*!)*!2+88!)*!*8'5'%-!)$/'.)5/*!/0)/!'%/+$.+$+!2'/0!
;#2%*/$+),!/)*D*!*350!)*!1$)'%!,#$70#,+/$6I!k)$-+$!0#8+*!
)$+!*++%!20+%!*78)//'%-*!)$+!('+2+;!)8#%-!/0+!*8'5'%-!)9'*I!
Figure 6: Accuracy of predicted motion. Slicing direction has an
impact on the accuracy of predicted motion. We plot the validation
MSE (left), and EPE (end-point error, right) of the predicted motion
on adult brain slices for sagittal, axial, and coronal acquisitions.
!
Sagittal
Axial
Coronal
Motion MSE (voxels)
Motion EPE (voxels)
Sagittal
Axial
Coronal
Epochs
Motion MSE (voxels)
All AB2 AB ADH B39 CO GSP MC OAS PP UKB
Figure 7: Motion accuracy across datasets. We plot the MSE of
the motion stacks predicted on 500 held-out slice stacks (1mm
3
, see
text). Motion prediction is accurate to < 1mm on average. Sagittal
and coronal motion predictions tend to be more accurate than axial.!
Sagittal
Axial
Coronal
!
!
5.2. Single-Stack SVR of Fetal Brains
! Z+$+F!2+!53$)/+!TW!g=2!.+/)8!1$)'%!)/8)*+*!)%;!$+.+$+%5+!
$+5#%*/$35/'#%*!)5$#**!/0+![Ak!"+/)8!U/8)*+*!MS=P!)%;!"+gU!
MSEP!<VIW,,
3
F!)8'-%+;!/#!)-+&,)/50+;![Ak!)/8)*+*?!)%;!*78'/!
/0+,!'%/#!WR!/$)'%'%-!)%;!N=!()8';)/'#%!"+gU!+9),78+*I!^+!
3*+!/2#!$+)8!g=2!*8'5+!*/)5D*!.$#,!@BUk!MNRP!.#$!/+*/I!B%!/0+!
5)*+!#.!.+/)8!GHAF!2+!/$)'%!)!*'%-8+!*78)/O*8'5+!%+/2#$D!#%!
)88!7#**'18+!*8'5'%-!;'$+5/'#%*!16!)7786'%-!)!$)%;#,!$#/)/'#%!
<p38+$!)%-8+*!1+/2++%!qNWVr?!/#!#3$!/$)'%'%-!+9),78+*I!^+!
5#,7)$+!#3$!,+/0#;!2'/0!*/)/+&#.&/0+&)$/!GHA%+/!M=VP!)%;!
GH#Ag!M=YPo!*++!U77+%;'9!UI!_7/','X)/'#%&1)*+;!,+/0#;*!
;#!%#/!2#$D!#%!*'%-8+!*/)5D*!)%;!)$+!%#/!5#,7)$+;!)-)'%*/I!
3D Reconstruction and Interpolation.!"'-3$+!T!('*3)8'X+*!
/0+!$+5#%*/$35/'#%*!#1/)'%+;!3*'%-!#3$!7$+;'5/+;!,#/'#%I!^+!
)8'-%!"+gU!$+5#%*/$35/'#%*!/#!/0+'$!-+*/)/'#%)8!)-+&,)/50+;!
)/8)*!MS=P!.#$!5#,7)$'*#%!)5$#**!;'e+$+%/!,+/0#;*I!GHA%+/!
M=VP!5#,78+/+86!.)'8*!/#!)8'-%!*8'5+*!'%!,#*/!#.!/0+!5)*+*!)%;!
GH#Ag!<(=?!'%/$#;35+*!8)$-+!*8'5+!,'*)8'-%,+%/*F!8+);'%-!/#!
*31#7/',)8!E>!$+5#%*/$35/'#%!2'/0!)77)$+%/!;'*/#$/'#%I!<L+!
GH#Ag!)%;!GHA%+/!$+5#%*/$35/'#%*!)$+!7$#5+**+;!3*'%-!#3$!
'%/+$7#8)/'#%I?!_3$!$+5#%*/$35/'#%*!)$+!,#$+!.)'/0.38!/#!/0+!
3%;+$86'%-!E>!(#83,+*!37!/#!),1'-3'/'+*!.$#,!*8'5'%-I!L+!
8)*/!/2#!$#2*!('*3)8'X+!$+5#%*/$35/'#%!#.!/0+!/2#!$+)8!@BUk!
)5C3'*'/'#%*!.#$!20'50!%#!-$#3%;!/$3/0!+9'*/*!)%;!,#/'#%!'*!
*,)88+$I!J%8'D+!GHA%+/F!/0+!7$#7#*+;!,+/0#;!-+%+$)8'X+*!/#!
$+)8!)5C3'*'/'#%*!+(+%!20+%!/0+!%+/2#$D!,#;+8!'*!/$)'%+;!#%!
*6%/0+/'5!*8'5+!*/)5D*!2'/0!;'e+$+%/!,#/'#%!)%;!)5C3'*'/'#%!
7)$),+/+$*F!+I-IF!hG"I!G++!U77+%;'9!]!.#$!,#$+!+9),78+*I!
Quantitative Results.!g)18+! N!8'*/*!/0+!7$+;'5/'#%!)553$)56!
#.!;'e+$+%/!,+/0#;*!#%!()8';)/'#%!*314+5/*!2'/0!Q!.#8;*I!B%!
);;'/'#%!/#!,#/'#%!phpF!2+!$+7#$/!/0+!*8'5+!hG\AF!#1/)'%+;!
16!*8'5'%-!/0+!/$3+!(#83,+!2'/0!/0+!7$+;'5/+;!,#/'#%F!)%;!/0+!
hG\A!#.!/0+!a%)8!$+5#%*/$35/'#%I!L+!/0$++&*/)5D!$+*38/*!)$+!
.$#,!M=YPF!20'50!8'*/*!/0+!)%50#$!7#'%/!+$$#$*!<Uhp?I!B%!/0+!
*'%-8+&*/)5D!5)*+F!#3$!,+/0#;!$+;35+*!+$$#$!'%!/0+!7$+;'5/+;!
,#/'#%!16!SSIRs!)%;!QQIRs!$+8)/'(+!/#!/0+!GHA%+/!M=VP!)%;!
GH#Ag!<(=?!7$+;'5/'#%*F!$+*7+5/'(+86I!L+!*8'5+!)%;!(#83,+!
hG\A!0)(+!)8*#!',7$#(+;!13/!3%8'D+!/0+!phpF!/0+*+!,+/$'5*!
;+7+%;!#%!/0+!hG"!#.!)5C3'*'/'#%!)%;!$+5#%*/$35/'#%!)%;!;#!
%#/!;+a%'/'(+86!,+)*3$+!)8'-%,+%/!)553$)56I!
ADHD200 8628223
ABIDE 50975
MCIC A00036476
Buckner39 990921
(a) Slice Stack
(b) Splat (Ours, True Motion)
(c) Interpolated (Ours, True Motion)
(d) True Volume
Figure 8: SVR of adult brain scans. We visualize our SVR results on sagittal (rows 1–2), axial (row 3), and coronal (last row) slice stacks
synthesized using random slice motion (a). Using the motion stack predicted by our network, we splat slice data to reconstruct the underlying
3D volume (b). Using our pre-trained interpolator, we then interpolate the missing intensities (holes) in our reconstruction (c). ?e result is
similar to the true 3D volume (d). We additionally visualize in (b) and (c) the splat and interpolated results when the true motion is used.
Sagittal Acquisition EPE: 1.66mm EPE: 0.00mm Sagittal View
Axial Acquisition EPE: 2.20mm EPE: 0.00mm Coronal View
Sagittal Acquisition EPE: 2.04mm EPE: 0.00mm Axial View
Coronal Acquisition EPE: 3.36mm EPE: 0.00mm Sagittal View
!
!
6. Discussion
Limitations.!_3$!.+/)8!GHA!%+/2#$D!'*!53$$+%/86!/$)'%+;!#%!
)3/#,)/'5)886!*+-,+%/+;!1$)'%!*8'5+*I!g$)'%'%-!2'/0!#$'-'%)8!
3%*+-,+%/+;!*8'5+!*/)5D*!5)%!',7$#(+!/0+!$#13*/%+**!#.!#3$!
)77$#)50!)-)'%*/!',7+$.+5/86!-+%+$)/+;!*+-,+%/)/'#%*I!
Future Work.!c'(+%!/0)/!#3$!)77$#)50!7$+;'5/*!;+%*+!*8'5+!
,#/'#%!.#$!$+5#%*/$35/'#%F!2+!78)%!/#!+9/+%;!#3$!)77$#)50!/#!
7$#18+,*!20+$+!/0+!)5C3'$+;!*8'5+*!5)%!3%;+$-#!;+.#$,)18+!
,#/'#%:+I-I!'%!.+/)8!/#$*#!#$!78)5+%/)8!$+5#%*/$35/'#%!MNWP:
)%;!5)%%#/!1+!/)5D8+;!3*'%-!$'-';!,#/'#%!GHAI!U8*#F!1+//+$!
'%/+$7#8)/'#%!%+/2#$D*F!*350!)*!/0#*+!/0)/!)$+!+C3'()$')%/!/#!
$'-';!,#/'#%F!5)%!.3$/0+$!',7$#(+!$+5#%*/$35/'#%o!*++!MSQPI!!
7. Conclusion
B%!1$)'%!',)-'%-F!*8'5+&/#&(#83,+!$+5#%*/$35/'#%! <GHA?!
'*!)%!',7#$/)%/!5#,73/)/'#%)8!/+50%'C3+!.#$!/0+!',)-'%-!#.!
*314+5/*!20#*+!,#/'#%!5)%%#/!+)*'86!1+!5#%/$#88+;I!i+/F!/0+!
,)4#$'/6!#.!GHA!/+50%'C3+*!7$+('#3*86!7$#7#*+;!;#!%#/!$+)7!
/0+!1+%+a/*!#.!"[\!,#;+8*!/0)/!0)(+!-$);3)886!1+5#,+!/0+!
,)'%*/)6!'%!58#*+86!$+8)/+;!/)*D*F!*350!)*!',)-+!$+-'*/$)/'#%!
)%;!*+-,+%/)/'#%I!L+!.+2!8+)$%'%-&1)*+;!GHA!/+50%'C3+*!
$+*#$/!/#!7$+/$)'%+;!B,)-+\+/!58)**'a5)/'#%!)%;!+(+%!('*'#%!
/$)%*.#$,+$!,#;+8*I!\#/!#%86!;#+*!#3$!2#$D!a88!/0+!-)7!'%!
/0+!GHA!,+/0#;!;+(+8#7,+%/!/',+8'%+!13/!'/!)8*#!*0#2*!/0)/!
)!"[\!,#;+8!5)%!7$#;35+!*/)/+&#.&/0+&)$/!GHA!#3/5#,+*I!
Acknowledgments. GBi!/0)%D*!@)$-0+$'/)!"'$+%X+!.#$!0+$!
/0#3-0/&7$#(#D'%-!C3+*/'#%*I!h$',)$6!*377#$/!-'(+%!16!/0+!
\BZ!-$)%/*!dTTUcVWNQTE!)%;!A"N@ZN=ENTYI!G377#$/+;!
16!\BZ!AVNUcVRQV=SF!A=NUcVW=VW=!)%;!hQNp]VEVVVRI!
FeTA Sub 073
FeTA Sub 053
MIAL001
Run
2
MIAL001
Run 1
(a) Slice Stack
(b) SVRnet
†
[20]
(c) SVoRTv2 [25]
(c) Splat (Ours)
(d) Interp (Ours)
(f) True Volume
Figure 9: Single-stack fetal SVR. We visualize the SVR results on validation subjects from the FeTA dataset [73] and two real acquisitions
from MIAL [16]. Zoomed in 4x for better visibility. Our results closely resemble the ground truth volumes while SVoRTv2 and SVRnet
reconstructions (with our interpolation) exhibit spatial distortion from inaccurate slice alignment.
†
Our implementation; see Appendix A.
Random Acquisition EPE: 8.69mm EPE: 3.70mm EPE: 1.49mm Pathological
Sagittal Acquisition EPE: Unknown EPE: Unknown EPE: Unknown
Random Acquisition EPE: 5.72mm EPE: 2.61mm EPE: 1.45mm Neurotypical
Sagittal Acquisition EPE: Unknown EPE: Unknown EPE: Unknown
Unknown
Unknown
Method
APE/Motion
EPE (mm)
Slice
PSNR (dB)
Vo l u m e
PSNR (dB)
Time
(sec)
3 Stacks
SVRnet* [20]
12.82±5.69
20.53±1.62
19.54±1.52
–
PlaneInVol* [24]
12.49±6.73
19.96±1.73
18.98±1.62
–
SVoRT* [25]
4.35±0.90
25.26±1.86
23.32±1.42
–
1 Stack
SVRnet
†
[20]
8.08±2.35
13.46±1.56
16.03±1.28
0.011s
SVoRT (v2) [25]
3.27±0.71
20.77±1.28
18.49±1.63
0.142s
Proposed (Ours)
1.81
±
0.40
23.69±1.39
23.43±1.40
0.224s
Table 1: Validati o n a ccuracy. We list the motion and 3D volume
reconstruction accuracy of different methods. *Results taken from
[25]. Timed on RTX8000.
†
Our implementation; see Appendix A.
!
!
References
[1] Mitsuhiro Nishida et al. Detailed semiautomated MRI based
morphometry of the neonatal brain: Preliminary results.
NeuroImage, 32(3):1041–1049, 2006.
[2] Chao J. Liu et al. Quantification of volumetric morphometry
and optical property in the cortex of human cerebellum at
micrometer resolution. NeuroImage, 244:118627, 2021.
[3] Maxwell L. Elliott et al. Brain morphometry in older adults
with and without dementia using extremely rapid structural
scans. NeuroImage, 276:120173, 2023.
[4] Bruce Fischl. FreeSurfer. NeuroImage, 62(2):774–781, 2012.
[5] Amy Zhao, Guha Balakrishnan, Fredo Durand, John V.
Guttag, and Adrian V. Dalca. Data augmentation using
learned transformations for one-shot medical image
segmentation. In Proc. CVPR, 2019.
[6] Lilla Zöllei, Juan Eugenio Iglesias, Yangming Ou, P. Ellen
Grant, and Bruce Fischl. Infant FreeSurfer: An automated
segmentation and surface extraction pipeline for T1-weighted
neuroimaging data of infants 0–2 years. NeuroImage,
218:116946, 2020.
[7] Alan C. Evans, Andrew L. Janke, D. Louis Collins, and
Sylvain Baillet. Brain templates and atlases. NeuroImage,
62(2):911–922, 2012.
[8] Shadab Khan et al. Fetal brain growth portrayed by a
spatiotemporal diffusion tensor MRI atlas computed from in
utero images. NeuroImage, 185:593–608, 2019.
[9] Adrià Casamitjana, and Juan Eugenio Iglesias. High-
resolution atlasing and segmentation of the subcortex:
Review and perspective on challenges and opportunities
created by machine learning. NeuroImage, 263:119616,
2022.
[10] Shuzhou Jiang, Hui Xue, Alan Glover, Mary Rutherford,
Daniel Rueckert, and Joseph V. Hajnal. MRI of moving
subjects using multislice snapshot images with volume
reconstruction (SVR): Application to fetal, neonatal, and
adult brain studies. IEEE Trans. Med. Imaging, 26(7):967–
980, 2007.
[11] Ali Gholipour, Judy A. Estroff, and Simon K. Warfield.
Robust super-resolution volume reconstruction from slice
acquisitions: Application to fetal brain MRI. IEEE Trans.
Med. Imaging, 29(10):1739–1758, 2010.
[12] Kio Kim, Piotr A. Habas, Francois Rousseau, Orit A. Glenn,
Anthony J. Barkovich, and Colin Studholme. Intersection
based motion correction of multislice MRI for 3-D in utero
fetal brain image formation. IEEE Trans. Med. Imaging,
29(1):146–158, 2010.
[13] Maria Kuklisova-Murgasova, Gerardine Quaghebeur, Mary
A. Rutherford, Joseph V. Hajnal, and Julia A. Schnabel.
Reconstruction of fetal brain MRI with intensity matching
and complete outlier removal. Med. Image Anal.,
16(8):1550–1564, 2012.
[14] Bernhard Kainz et al. Fast volume reconstruction from
motion corrupted stacks of 2D slices. IEEE Trans. Med.
Imaging, 34(9):1901–1913, 2015.
[15] Amir Alansary et al. PVR: Patch-to-Vo l u m e R e c o n s t r u c t i o n
for large area motion correction of fetal MRI. IEEE Trans.
Med. Imaging, 36(10):2031–2044, 2017.
[16] Sébastien Tourbier et al. Automated template-based brain
localization and extraction for fetal brain MRI reconstruction.
NeuroImage, 155:460–472, 2017.
[17] Michael Ebner et al. An automated framework for
localization, segmentation and super-resolution
reconstruction of fetal brain MRI. NeuroImage, 206:116324,
2020.
[18] Alena Uus, Tong Zhang, Laurence H. Jackson, ?omas A.
Roberts, Mary A. Rutherford, Joseph V. Hajnal, and Maria
Deprez. Deformable slice-to-volume registration for motion
correction of fetal body and placenta MRI. IEEE Trans. Med.
Imaging, 39(9):2750–2759, 2020.
[19] Francois Rousseau, Orit A. Glenn, Bistra Iordanova, Claudia
Rodriguez-Carranza, Daniel B. Vigneron, James A.
Barkovich, and Colin Studholme. Registration-based
approach for reconstruction of high-resolution in utero fetal
MR brain images. Acad. Radiol., 13(9):1072–1081, 2006.
[20] Benjamin Hou et al. 3-D Reconstruction in canonical co-
ordinate space from arbitrarily oriented 2-D Images. IEEE
Trans. Med. Imaging, 37(8):1737–1750, 2018.
[21] Benjamin Hou et al. Computing CNN loss and gradients for
pose estimation with Riemannian geometry. In Proc.
MICCAI, 2018.
[22] Seyed Sadegh Mohseni Salehi, Shadab Khan, Deniz
Erdogmus, and Ali Gholipour. Real-time deep pose
estimation with geodesic loss for image-to-template rigid
registration. IEEE Trans. Med. Imaging, 38(2):470–481,
2019.
[23] Yu ch en P e i, L i sh en g Wa ng , F en qi a ng Z ha o, Ta o Z ho ng ,
Lufan Liao, Dinggang Shen, and Gang Li. Anatomy-guided
convolutional neural network for motion correction in fetal
brain MRI. In Proc. MLMI, 2020.
[24] Pak-Hei Yeung, Moska Aliasi, Aris T. Papageorghiou,
Monique Haak, Weidi Xie, and Ana I. L. Namburete.
Learning to map 2D ultrasound images into 3D space with
minimal human annotation. Med. Image Anal., 70:101998,
2021.
[25] Junshen Xu, Daniel Moyer, P. Ellen Grant, Polina Golland,
Juan Eugenio Iglesias, and Elfar Adalsteinsson. SVoRT:
Iterative transformer for slice-to-volume registration in fetal
brain MRI. In Proc. MICCAI, 2022.
[26] Elise Turk et al. Functional connectome of the fetal brain. J.
Neurosci., 39(49):9716–9724, 2019.
[27] Daniel Sobotka et al. Motion correction and volumetric
reconstruction for fetal functional magnetic resonance
imaging data. NeuroImage, 255:119213, 2022.
[28] Xiaohuan Cao, Jianhua Yang, Jun Zhang, Dong Nie,
Minjeong Kim, Qian Wang, and Dinggang Shen. Deformable
image registration based on similarity-steered CNN
regression. In Proc. MICCAI, 2017.
[29] Guha Balakrishnan, Amy Zhao, Mert R. Sabuncu, John
Guttag, and Adrian V. Dalca. An unsupervised learning model
for deformable medical image registration. In Proc. CVPR,
2018.
[30] Tony C. W. Mok, and Albert C. S. Chung. Fast symmetric
diffeomorphic image registration with convolutional neural
networks. In Proc. CVPR, 2020.
[31] Sean I. Young, Yaël Balbastre, Adrian V. Dalca, William M.
!
!
Wells, Juan Eugenio Iglesias, and Bruce Fischl. SuperWarp:
Supervised learning and warping on U-Net for invariant
subvoxel-precise registration. In Proc. WBIR, 2022.
[32] Miao Kang, Xiaojun Hu, Weilin Huang, Matthew R. Scott,
and Mauricio Reyes. Dual-stream pyramid registration
network. Med. Image Anal., 78:102379, 2022.
[33] Alexey Dosovitskiy et al. FlowNet: Learning Optical Flow
With Convolutional Networks. In Proc. ICCV, 2015.
[34] Jason J. Yu, Adam W. Harley, and Konstantinos G. Derpanis.
Back to basics: Unsupervised learning of optical flow via
brightness constancy and motion smoothness. In Proc.
ECCV, 2016.
[35] Eddy Ilg, Nikolaus Mayer, Tonmoy Saikia, Margret Keuper,
Alexey Dosovitskiy, and ?omas Brox. FlowNet 2.0:
Evolution of optical flow estimation with deep networks. In
Proc. CVPR, 2017.
[36] Deqing Sun, Xiaodong Yang, Ming-Yu L i u, a n d J a n K a ut z.
PWC-Net: CNNs for optical flow using pyramid, warping,
and cost volume. In Proc. CVPR, 2018.
[37] Anurag Ranjan, and Michael J. Black. Optical flow
estimation using a spatial pyramid network. In Proc. CVPR,
2017.
[38] Pengpeng Liu, Michael Lyu, Irwin King, and Jia Xu.
SelFlow: Self-supervised learning of optical flow. In Proc.
CVPR, 2019.
[39] Junhwa Hur, and Stefan Roth. Iterative residual refinement
for joint optical flow and occlusion estimation. In Proc.
CVPR, 2019.
[40] Fayao Liu, Chunhua Shen, and Guosheng Lin. Deep
convolutional neural fields for depth estimation from a single
image. In Proc. CVPR, 2015.
[41] Jae-Han Lee, and Chang-Su Kim. Monocular depth
estimation using relative depth maps. In Proc. CVPR, 2019.
[42] Clement Godard, Oisin Mac Aodha, Michael Firman, and
Gabriel J. Brostow. Digging into self-supervised monocular
depth estimation. In Proc. ICCV, 2019.
[43] Shariq Farooq Bhat, Ibraheem Alhashim, and Peter Wonka.
AdaBins: Depth estimation using adaptive bins. In Proc.
CVPR, 2021.
[44] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and
Li Fei-Fei. ImageNet: a large-scale hierarchical image
database. In Proc. CVPR, 2009.
[45] Alexey Dosovitskiy et al. An image is worth 16x16 words:
Transformers for image recognition at scale. In Proc. ICLR,
2020.
[46] Anurag Arnab, Mostafa Dehghani, Georg Heigold, Chen Sun,
Mario Lučić, and Cordelia Schmid. ViViT: A video vision
transformer. In Proc. ICCV, 2021.
[47] Ze Liu et al. Swin Transformer: Hierarchical vision
transformer using shifted windows. In Proc. ICCV, 2021.
[48] Jeremie Anquez, Elsa D. Angelini, and Isabelle Bloch.
Automatic segmentation of head structures on fetal MRI. In
Proc. ISBI, 2009.
[49] Bernhard Kainz, Kevin Keraudren, Vanessa Kyriakopoulou,
Mary Rutherford, Joseph V. Hajnal, and Daniel Rueckert.
Fast fully automatic brain detection in fetal MRI using dense
rotation invariant image descriptors. In Proc. ISBI, 2014.
[50] Kevin Keraudren, Vanessa Kyriakopoulou, Mary Rutherford,
Joseph V. Hajnal, and Daniel Rueckert. Localisation of the
brain in fetal MRI using bundled SIFT features. In Proc.
MICCAI, 2013.
[51] Kevin Keraudren et al. Automated fetal brain segmentation
from 2D MRI slices for motion correction. NeuroImage,
101:633–643, 2014.
[52] Martin Rajchl et al. Learning under distributed weak
supervision, http://arxiv.org/abs/1606.01100, 2016.
[53] Seyed Sadegh Mohseni Salehi et al. Real-time automatic fetal
brain extraction in fetal MRI by deep learning. In Proc. ISBI,
2018.
[54] Olaf Ronneberger, Philipp Fischer, and ?omas Brox. U-Net:
convolutional networks for biomedical image segmentation.
In Proc. MICCAI, 2015.
[55] Junshen Xu, Daniel Moyer, Borjan Gagoski, Juan Eugenio
Iglesias, P. Ellen Grant, Polina Golland, and Elfar
Adalsteinsson. NeSVoR: Implicit neural representation for
slice-to-volume reconstruction in MRI. IEEE Trans. Med.
Imaging, 42(6):1707–1719, 2023.
[56] Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik,
Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng. NeRF:
Representing scenes as neural radiance fields for view
synthesis, In Proc. ECCV, 2020.
[57] Sahar N. Saleem. Fetal MRI: An approach to practice: A
review. J. Adv. Res., 5(5):507–523, 2014.
[58] Hua-mei Chen, and P.K. Varshney. Mutual information-based
CT-MR brain image registration using generalized partial
volume joint histogram estimation. IEEE Trans. Med.
Imaging, 22(9):1111–1119, 2003.
[59] Andrew Adams, Jongmin Baek, and Myers Abraham Davis.
Fast high-dimensional filtering using the permutohedral
lattice. Comput. Graph. Forum, 29(2):753–762, 2010.
[60] Nils Papenberg, Andrés Bruhn, ?omas Brox, Stephan Didas,
and Joachim Weickert. Highly accurate optic flow
computation with theoretically justified warping. Int. J.
Comput. Vis., 67(2):141–158, 2006.
[61] Nicholas J. Higham. Computing the polar decomposition—
with applications. SIAM J. Sci. Stat. Comput., 7(4):1160–
1174, 1986.
[62] A. Di Martino et al. ?e autism brain imaging data exchange:
towards a large-scale evaluation of the intrinsic brain
architecture in autism. Mol. Psychiatry, 19(6):659–667,
2014.
[63] Michael Milham, Damien Fair, Maarten Mennes, and Stewart
Mostofsky. ?e ADHD-200 consortium: a model to advance
the translational potential of neuroimaging in clinical
neuroscience. Front. Syst. Neurosci., 6, 2012.
[64] Vince Calhoun, Jing Sui, Kent Kiehl, Jessica Turner, Elena
Allen, and Godfrey Pearlson. Exploring the psychosis
functional connectome: aberrant intrinsic networks in
schizophrenia and bipolar disorder. Front. Psychiatry, 2,
2012.
[65] Avram J. Holmes et al. Brain Genomics Superstruct Project
initial data release with structural, functional, and behavioral
measures. Sci. Data, 2(1):150031, 2015.
[66] Randy L. Gollub et al. ?e MCIC Collection: a shared
repository of multi-modal, multi-site brain image data from a
clinical investigation of schizophrenia. Neuroinformatics,
!
!
11(3):367–388, 2013.
[67] Daniel S. Marcus, Tracy H. Wang, Jamie Parker, John G.
Csernansky, John C. Morris, and Randy L. Buckner. Open
Access Series of Imaging Studies (OASIS): Cross-sectional
MRI data in young, middle aged, nondemented, and
demented older adults. J. Cogn. Neurosci., 19(9):1498–1507,
2007.
[68] Kenneth Marek et al. ?e Parkinson Progression Marker
Initiative (PPMI). Prog. Neurobiol., 95(4):629–635, 2011.
[69] Cathie Sudlow et al. UK Biobank: an open access resource
for identifying the causes of a wide range of complex diseases
of middle and old age. PLOS Med., 12(3):e1001779, 2015.
[70] Bruce Fischl et al. Whole brain segmentation: Automated
labeling of neuroanatomical structures in the human brain.
Neuron, 33(3):341–355, 2002.
[71] J. Talairach, and Pierre Tournoux. Co-planar stereotaxic atlas
of the human brain: 3-dimensional proportional system : an
approach to cerebral imaging. G. ?ieme ; ?ieme Medical
Publishers, Stuttgart, New York 1988.
[72] Ali Gholipour et al. A normative spatiotemporal MRI atlas of
the fetal brain for automatic segmentation and analysis of
early brain growth. Sci. Rep., 7(1):476, 2017.
[73] Kelly Payette et al. An automatic multi-tissue human fetal
brain segmentation benchmark using the Fetal Tissue
Annotation Dataset. Sci. Data, 8(1):167, 2021.
[74] Dongdong Chen, Mike Davies, Matthias J. Ehrhardt, Carola-
Bibiane Schönlieb, Ferdia Sherry, and Julián Tachella.
Imaging with equivarient deep learning: from unrolled
network design to fully unsupervised learning. IEEE Signal
Process. Mag., 40(1):134–147, 2023.
! !
!
!
A. Training and Validation
A.1 Training Details
! "#$!/$)'%'%-F!2+!7$+7)$+!7)'$+;!<*8'5+!*/)5DF!,#/'#%!*/)5D?!
)%;!<*78)//+;!(#83,+F!-$#3%;!/$3/0!(#83,+?!/$)'%'%-!;)/)!16!
)7786'%-!)!$)%;#,!'*#/$#7'5!X##,!<q=R!(#9+8*?F!0#$'X#%/)8!
f'7F!$#/)/'#%!<p38+$!)%-8+*!q=Vr!.#$!);38/!1$)'%*!)%;!qNWVr!
.#$!.+/)8?!)%;!/$)%*8)/'#%!<qNE!(#9+8*?!/#!/0+!$+-'*/+$+;!=YR
3
!
$+.+$+%5+!(#83,+!<N,,
3
!)5C3'*'/'#%F!g)8)'$)50!.#$!);38/!)%;!
VIW,,
3
!"+gU!$+5#%*/$35/'#%F![Ak!.#$!.+/)8?!/#!a$*/!*',38)/+!
/0+!;'(+$*'/6!'%!*5)%%+;!*314+5/*!)%;!()$'#3*!7#*+*!/0)/!/0+6!
,)6!1+!*5)%%+;!3%;+$I!g#!*',38)/+!*8'5'%-F!2+!)7786!$)%;#,!
$#/)/'#%*!<p38+$!)%-8+*!q=Vr?!)%;!/$)%*8)/'#%*!<q=R!(#9+8*?!
/0)/!()$6!*,##/086!)5$#**!*8'5+*!16!$)%;#,86!*),78'%-!E=!/#!
RQ!$#/)/'#%*!)%;!/$)%*8)/'#%*!.$#,!/0+!$)%-+*!*0#2%!)1#(+!
)%;!*,##/086!'%/+$7#8)/'%-!/0+,!3*'%-!531'5!]!*78'%+*I!^+!
'%/+$8+)(+!/0+!/2#!0)8(+*!#.!,#/'#%!/$)4+5/#$'+*!/#!*',38)/+!
/2#&*0#/!*+C3+%5+*!/67'5)886!3*+;!'%!=>!@AB!)5C3'*'/'#%*I!
! G8'5+!)5C3'*'/'#%!'*!*',38)/+;!16!183$$'%-!/0+!*8'5+*!3*'%-!
)!1#95)$!hG"!<.#3$!(#9+8*!2';+?!)8#%-!/0+!*8'5'%-!;'$+5/'#%!
<*)-'//)8F!)9')8!#$!5#$#%)8!.#$!);38/!1$)'%*F!)%;!*)-'//)8!'%!/0+!
5)*+!#.!.+/)8F!20+$+!/0+!qNWVr!p38+$!)%-8+*!,+)%!/0)/!*8'5+*!
)$+!)5C3'$+;!)8#%-!$)%;#,!;'$+5/'#%*!)%62)6?!)%;!*),78'%-!
+(+$6!.#3$/0!*8'5+!)8#%-!/0+!*),+!)9'*I!G8'5+!'%/+%*'/'+*!)$+!
,)%'738)/+;!16!)7786'%-!c)3**')%!%#'*+!2'/0!%#'*+!*/)%;)$;!
;+(')/'#%!𝜎 = 0.01!)%;!-),,)!)3-,+%/)/'#%!2'/0!+97#%+%/!
𝛾 ∈
[
0.9,1
]
I!!"'%)886F!/0+!)5C3'$+;!*8'5+*!)$+!$+78'5)/+;!)8#%-!
/0+!*8'5'%-!;'$+5/'#%!16!)!.)5/#$!#.!.#3$!)-)'%!*#!/0)/!/0+!*8'5+!
*/)5D*!0)(+!'*#/$#7'5!'%&!)%;!/0$#3-0&78)%+!$+*#83/'#%*I!^+!
*31*),78+!/0+!*8'5+!*/)5D*!16!)!.)5/#$!#.!/2#!<'I+IF!/#!=,,
3
!
#$!NIR,,
3
?!/#!/$)'%!/0+!*8'5+!,#/'#%!7$+;'5/'#%!%+/2#$DI!\#!
*31*),78'%-!'*!7+$.#$,+;!20+%!/$)'%'%-!/0+!'%/+$7#8)/#$I!
! ^+! /$)'%! #3$!*8'5+!,#/'#%!)%;!'%/+$7#8)/'#%!%+/2#$D*!.#$!
256,000!*/+7*!3*'%-!U>U@F!2'/0!)%!'%'/')8!8+)$%'%-!$)/+!#.!
10
−4
F!20'50!'*!$+;35+;!/#!0!2'/0!7#86!*50+;38'%-!<+97#%+%/!
#.!0.9?F!2+'-0/!;+5)6!#.!0!)%;!,#,+%/3,!#.!0.90I!^+!,)*D!
#3/!1)5D-$#3%;!(#9+8*!20+%!5#,73/'%-!/0+!/$)'%'%-!8#**I!B%!
/0+!5)*+!#.!.+/)8!GHAF!2+!#(+$&*),78+!/0+!"+gU!7#$/'#%!#.!
/0+!/$)'%'%-!;)/)!a(+.#8;F!)%;!/0+![Ak!7#$/'#%!/+%.#8;I!^+!
*',786!7'5D!/0+!,#;+8!#.!/0+!8)*/!+7#50!.#$!,#;+8!*+8+5/'#%!
13/!*/'88!,#%'/#$!()8';)/'#%!,+/$'5*!.#$!7#/+%/')8!#(+$a//'%-I!
A.2 Validation Metrics
! G','8)$!/#!#3$!/$)'%'%-!8#**!<T?F!)88!#3$!()8';)/'#%!,+/$'5*!
5#,7+%*)/+!.#$!)%6!-8#1)8!$'-';!,#/'#%!#e*+/!,)6!/0)/!+9'*/!
1+/2++%!7$+;'5/+;!)%;!/$3+!*8'5+!,#/'#%o!*++!<T?I!B%!);;'/'#%!
/#!/0+!@Gp!𝓛
MSE
(𝐮, 𝐲) = (1 𝑁
⁄
)
‖
𝐮 − 𝐲
‖
𝐹
2
!#.!/0+!7$+;'5/+;!
*8'5+!,#/'#%!𝐮 ∈ ℝ
𝑁×3
!2I$I/I!/$3+!,#/'#%!𝐲 ∈ ℝ
𝑁×3
F!2+!3*+!
/0+!)(+$)-+!+%;&7#'%/!+$$#$!<php?!,+/$'5!MEEPF!;+a%+;!)*!
!
𝓛
EPE
(𝐮, 𝐲) = (1 𝑁
⁄
)
‖
𝐮 − 𝐲
‖
2,1
,!
<UN?!
/0)/!'*F!/0+!,+)%!p358';+)%!;'*/)%5+!1+/2++%!/0+!+%;&7#'%/*!
#.!/2#!*8'5+!,#/'#%!a+8;*!<1#/0!,+/$'5*!*0#2%!0+$+!2'/0#3/!
$'-';!5#,7+%*)/'#%!.#$!58)$'/6?I!B%!7$+('#3*!2#$D!M=VF!=YPF!)!
*','8)$!,+/$'5!'*!7$#7#*+;!/#!,+)*3$+!/0+!)(+$)-+!p358';+)%!
;'*/)%5+!1+/2++%!/0+!7$+;'5/+;!)%;!/$3+!*8'5+!7#*'/'#%*!)/!/0+!
)%50#$!7#'%/*!#.!/0+!*8'5+*!<)%50#$!7#'%/!+$$#$F!Uhp?`!
!
𝓛
APE
(𝐮, 𝐲) = (1 3
⁄
)
‖
𝐮
{0,1,2}
− 𝐲
{0,1,2}
‖
2,1
,!
<U=?!
'%!20'50!𝐮
{0,1,2}
!)%;!𝐲
{0,1,2}
!;+%#/+!/0+!7#*'/'#%!(+5/#$*!)/!
/0+!)%50#$!7#'%/*!#.!/0+!-$';*!/0)/!;+a%+!/0+!(#9+8!8#5)/'#%*!
#.!/0+!$+*7+5/'(+!*8'5+*!'%!E>!*7)5+I!g67'5)886F!)%50#$!7#'%/*!
)$+!)**3,+;!/#!1+!)/!/0+!5+%/+$F!1#//#,!8+./!)%;!1#//#,!$'-0/!
5#$%+$*!#.!)!-'(+%!*8'5+I!U**3,'%-!/0)/!*8'5+*!)$+!3%;+$-#'%-!
$'-';!,#/'#%F!/0+!Uhp!'*!+C3'()8+%/!/#!/0+!php!)(+$)-+;!#%!
/0+!$'-0/&/$')%-38)$!$+-'#%!.#$,+;!16!/0+!/0$++!)%50#$!7#'%/*!
#%!/0+!$+.+$+%5+!*8'5+I!
A.3 SVRnet Model Implementation
! "#$!$+7$#;35'1'8'/6F!2+!7#$/!/0+!#$'-'%)8!g+%*#$"8#2!NINE!
',78+,+%/)/'#%!#.!GHA%+/!M=VP!2'/0!)%!B%5+7/'#%!1)5D1#%+!
/#!h6g#$50!NINEF!20+$+!2+!3*+!)!A+*\+/&EQ!1)5D1#%+!)%;!)!
7$+;'5/'#%!0+);! 5#%*'*/'%-!#.! )!512 × 9!;+%*+!8)6+$F!20'50!
7$+;'5/*!/0+!*8'5+!7#*'/'#%!(+5/#$*!)/!/0$++!)%50#$!7#'%/*I!^+!
a%;!/0)/!=>!1)/50!%#$,)8'X)/'#%!1)*+;!#%!5#88+5/+;!*/)/'*/'5*!
;#+*!%#/!7+$.#$,!2+88!)/!/+*/!/',+!)%;!#7/!/#!%#$,)8'X+!+)50!
+9),78+!1)*+;!#%!/0+!*/)/'*/'5*!#.!+)50!*8'5+!*/)5DI!^+!3*+!
*31*),78+;!*8'5+*!#.! *'X+!128 × 128!7'9+8*! )%;! '%/+$7#8)/+!
/0+!7$+;'5/+;!)%50#$!7#'%/!7#*'/'#%!(+5/#$*!/#!)!8'%+)$!,#/'#%!
a+8;!)%;!*31/$)5/!/0+!*8'5+!(#9+8!5##$;'%)/+*!/#!#3/73/!*8'5+!
,#/'#%I!^+!/$)'%!/0'*!',78+,+%/)/'#%!#.!GHA%+/!3*'%-!/0+!
$+-38)$!@Gp!8#**!#%!#3/73/!*8'5+!,#/'#%!a+8;I!!^+!'%'/')8'X+!
/0+!,#;+8!2'/0!/0+!/#$50('*'#%!B,)-+\+/Ndt(N!2+'-0/*I!
A.4 SVoRT Model Configuration
! "#$!5#,7)$'*#%!2'/0!GH#Ag!<(=?F!2+!3*+!,#;+8!2+'-0/*!
7$#(';+;!16!j3!+/!)8I!M=YP!#%!/0+'$!$+7#*'/#$6F!5#%a-3$+!/0+!
,#;+8!/#!3*+!#%+!*8'5+!*/)5D!2'/0!)!*8'5+!-)7!#.!EI=,,F!)%;!
#7/','X+!/0+!$+5#%*/$35/'#%!hG"!<*8'5+!/0'5D%+**!#.!NIR,,?!
.#$!()8';)/'#%!)553$)56!<'I+IF!)(+$)-+!,#/'#%!+%;&7#'%/!+$$#$?!
#%!/0+!N=!"+gU!()8';)/'#%!*314+5/*!3*'%-!)%!+97#%+%/')8!-$';!
*+)$50I!^+! a9+;! /0+! */)5D! 7#*'/'#%)8! +%5#;'%-! #.! GH#Ag!/#!
V!<$)/0+$!/0)%!)!$)%;#,!'%/+-+$?!.#$!$+7$#;35'18+!$+*38/*I!^+!
5#%(+$/!GH#Agn*!/$)%*.#$,!#3/73/!/#!;+%*+!,#/'#%!a+8;*!/#!
5#,73/+!/0+!,#/'#%!@Gp!)%;!php!.#$!()8';)/'#%!)553$)56I!
B. Additional SVR Results
! Z+$+F!2+!7$#(';+!);;'/'#%)8!GHA!$+*38/*!#%!#3$!);38/!)%;!
.+/)8!;)/)*+/*I!"'-3$+!]N!('*3)8'X+*!$+5#%*/$35/'#%*!#.!);38/!
1$)'%!@A!(#83,+*!.#$!/0$++!#.!#3$!()8';)/'#%!*314+5/*F!2'/0!
)88!/0$++!#$/0#-#%)8!('+2*!*0#2%!.#$!5#,78+/+%+**I!"'-3$+!
]=!*','8)$86!('*3)8'X+*!.+/)8!$+5#%*/$35/'#%*I!^+!'%583;+!/0+!
5#$$+*7#%;'%-!GH#Ag(=!$+*38/*!.#$!5#,7)$'*#%F!%#/'%-!/0)/!
GHA%+/!$+5#%*/$35/'#%*!)$+!-)$18+;!'%!,)%6!5)*+*!<*++!a$*/!
$#2!#.!"'-3$+!T?!)%;!)$+!8+**!,+)%'%-.38!/#!5#,7)$+!)-)'%*/I!
!
!
!
ABIDE 50975
(a) Slice Stack
(b) Splat (Ours, True Motion)
(c) Interpolated (Ours, True Motion)
(d) True Volume
MCIC A00036476
(a) Slice Stack
(b) Splat (Ours, True Motion)
(c) Interpolated (Ours, True Motion)
(d) True Volume
Coronal Acquisition EPE: 2.00mm EPE: 0.00mm
Sagittal Acquisition EPE: 1.70mm EPE: 0.00mm
Axial Acquisition EPE: 2.35mm EPE: 0.00mm
!
!
!
Buckner39 990921
(a) Slice Stack
(b) Splat (Ours, True Motion)
(c) Interpolated (Ours, True Motion)
(d) True Volume
Figure B1: SVR of adult brain scans.
We visualize our SVR results on sli ce stacks synthesized usi ng random slice motion (a). Using the
predicted motion stack, we splat slice data to reconstruct the underlying 3D volume (b). W
e interpolate the missing intensities (holes) in our
reconstruction (c). We additionally visualize in (b) and (c) splat and interpolated results obtained when the true motion stack is used.
FeTA Sub 0
01
(a) Slice Stack
(b) Splat (SVoRTv2, Ours)
(c) SVoRTv2
(c) Interp (SVoRTv2, Ours)
(d) Interp (Ours)
(d) True Volume
FeTA Sub 010
Random Acquisition EPE: 3.51mm EPE: 1.97mm
Random Acquisition EPE: 4.78mm EPE: 1.64mm
!
!
!
(a) Slice Stack
(b) Splat (SVoRTv2, Ours)
(c) Interp (SVoRTv2, Ours)
(d) True Volume
FeTA Sub 036
(a) Slice Stack
(b) Splat (SVoRTv2, Ours)
(c) SVoRTv2
(c) Interp (SVoRTv2, Ours)
(d) Interp (Ours)
(d) True Volume
FeTA Sub 0
44
(a) Slice Stack
(b) Splat (SVoRTv2, Ours)
(c) SVoRTv2
(c) Interp (SVoRTv2, Ours)
(d) Interp (Ours)
(d) True Volume
Figure B2: Single-stack fetal SVR. We visualize the SVR results on validation subjects from the FeTA dataset [73]. Our
results closely
resemble the ground truth volumes while SVoRTv2 reconstructions (with our interpolation applied) exhibit spatial distortion f
rom inaccurate
slice alignment.
Random Acquisition EPE: 2.40mm EPE: 1.25mm
Random Acquisition EPE: 3.40mm EPE: 1.91mm
