Sean I Young, PhD — Non-line-of-sight Surface Reconstruction Using the Directional Light-cone Transform

Abstract!

We propose a joint albedo–normal approach to non-line-

of-sight (NLOS) surface reconstruction using the directional

light-cone transform (D-LCT). While current NLOS imaging

methods reconstruct either the albedo or surface normals of

the hidden scene, the two quantities provide complementary

information of the scene, so an eﬃcient method to estimate

both simultaneously is desirable. We formulate the recovery

of the two quantities as a vector deconvolution problem, and

solve it using the Cholesky–Wiener decomposition. We show

that surfaces ﬁtted non-parametrically using our recovered

normals are more accurate than those produced with NLOS

surface reconstruction methods recently proposed, and are

1,000× faster to compute than using inverse rendering.

1. Introduction

! #$%&'(%)&$*&+(,-.!/#0123!(45,(%,!6)5'+!7(.-!.-)!859.:;)!

5%6!;)%6);(%,!$*!5!+8)%)!.-5.!(+!%$.!(%!.-)!6(;)8.!'(%)!$*!+(,-.!

*;$4!.-)!+)%+$;<!=%!;)8)%.!>)5;+?!#012!(45,(%,!-5+!)4);,)6!

5+!5%!(49$;.5%.!@(+($%!9;$A')4?!7(.-!599'(85.($%+!(%!;)4$.)!

+)%+(%,?!6)*)%+)?!;$A$.(8!@(+($%!5%6!5:.$%$4$:+!6;(@(%,<!B!

8$44$%!(45,(%,!+).:9!(+!.$!C'$$D!5;$:%6!.-)!8$;%);E!:+(%,!

.-)!8$%*$85'!8$%F,:;5.($%!6)9(8.)6!(%!G(,:;)!"<!H>9(85''>?!5!

'(,-.!+$:;8)?!+:8-!5+!5!'5+);!A)54?!(%6(;)8.'>!('':4(%5.)+!.-)!

+8)%)!A>!;)I)8.(%,!$%.$!5!+:;*58)!.-5.!85%!A)!+))%!*;$4!A$.-!

.-)!+)%+$;!5%6!.-)!+8)%)<!J)!+)%+$;!.-)%!859.:;)+!.-)!+8)%)!

;)I)8.($%+!*;$4!.-)!+54)!+:;*58)!'$85.($%?!5%6!;)8$;6+!.-)4!

5+!5!.(4)&;)+$'@)6!+)K:)%8)!$*!.7$&6(4)%+($%5'!(45,)+!/$;!

.;5%+()%.+3?!*;$4!7-(8-!$%)!85%!8$49:.5.($%5''>!;)8$%+.;:8.!

.-)!+8)%)<!B95;.!*;$4!.;5%+()%.&A5+)6!(45,(%,?!$.-);!#012!

(45,(%,!4$65'(.()+!(%8':6)!.-$+)!A5+)6!$%!+9)8D')!L"MNO?!$;!!

(%8$-);)%.!(%.)%+(.>!4)5+:;)4)%.+!LP?!QO?!5+!7)''!5+!95++(@)!

+)%+(%,!LRMSO!5%6!58$:+.(8!(45,(%,!L"TO!.)8-%(K:)+<!

! U);)?!7)!7(''!8$%+(6);!)V8':+(@)'>!#012!(45,(%,!A5+)6!

$%!.;5%+()%.+!L""M"SO?!*;$4!7-(8-!.-)!-(66)%!#012!+8)%)!(+!

.>9(85''>!;)%6);)6!5+!5!+95.(5'?!.-;))&6(4)%+($%5'!@$':4)!$*!

5'A)6$!/@$':4).;(8!5'A)6$3?!$;!5+!5!+).!$*!$AW)8.!+:;*58)+<!=%!

.-)!@$':4).;(8!5'A)6$!95;56(,4?!.-)!$AW)8.(@)!(+!.$!)+.(45.)!

5'A)6$!@5':)+!*$;!+8)%)!@$V)'+!L""M"RO?!7-(')!(%!.-)!+:;*58)!

;)8$%+.;:8.($%!95;56(,4?!$%)!+))D+!4$;)!6(;)8.'>!.$!;)8$@);!

$AW)8.!+:;*58)+!(%!.-)!.-;))&6(4)%+($%5'!+8)%)!A>!)+.(45.(%,!

.-)(;!+:;*58)!%$;45'+!L"XM"SO<!Y-(')!+:;*58)&A5+)6!4).-$6+!

-5@)!.-)!9$.)%.(5'!.$!;)8$%+.;:8.!$AW)8.!,)$4).;>!7(.-!F%);!

6).5('!.-5%!.-)!5'A)6$!$%)+?!8:;;)%.!599;$58-)+!.$!)+.(45.(%,!

+:;*58)!%$;45'+!5;)!+)%+(.(@)!.$!%$(+)?!'(4(.)6!.$!+8)%)+!7(.-!

+(49');!$AW)8.!,)$4).;>!L"ZO?!+)%+(.(@)!.$!(%(.(5'([5.($%!L"SO!

$;!)%.5('!5!-(,-!8$49:.5.($%5'!8$49')V(.>!L"NO?!5''!$*!7-(8-!

4$.(@5.)!$:;!9;)+)%.!7$;D<!\4A)66)6!(%!.;5%+()%.+!(+!5!4(V!

$*!+:;*58)!%$;45'!5%6!5'A)6$!(%*$;45.($%?!+$!.-5.!)V9'(8(.'>!

588$:%.(%,!*$;!.-)!9;)+)%8)!$*!A$.-!(%!.;5%+()%.+!85%!9;)+)%.!

%)7!$99$;.:%(.()+!*$;!;)8$@);(%,!A$.-!K:5%.(.()+!;$A:+.'><!

Non-line-of-sight Surface Reconstruction

Using the Directional Light-cone Transform

Sean I. Young David B. Lindell

Stanford University Stanford University

sean0@stanford.edu lindell@stanford.edu

Bernd Girod David Taubman Gordon Wetzstein

Stanford University UNSW Sydney Stanford University

bgirod@stanford.edu d.taubman@unsw.edu.au gordon.wetzstein@stanford.edu

! "#$!%&'!#()*+,! ")$!%&'!-,'.#(/! "0$!1233*+!/&'4#0*!

125&'*!67!NLOS surface reconstruction via the D-LCT: 892/32-5!

:;%<!2.#52-5! .*3=,+/!3>?20#((>!'*0,@*'!,-(>! 3=*! #()*+,!,4! 3=*!

=2++*-!/0*-*7!A*!B2'*032,-#(!;CD!'*0,@*'/!),3=!3=*!#()*+,!"#$!#-+!

3=*!/&'4#0*!-,'.#(/!")$!,4!3=*!/0*-*E!#((,F2-5!&/!3,!'*0,-/3'&03!3=*!

=2++*-!,)G*03!/&'4#0*!F23=!H-*'!+*3#2(!"0$7!!



′



′

I2++*-!,)G*03!

C,((,0#3*+!(#/*'!

#-+!+*3*03,'!

!

J2/2)(*!/&'4#0*!

%00(&+*'

! !=%!.-(+!7$;D?!7)!9;$9$+)!5!W$(%.!5'A)6$M%$;45'!599;$58-!

.$!#012!+8)%)!;)8$%+.;:8.($%!A5+)6!$%!.-)!6(;)8.($%5'!'(,-.&

8$%)!.;5%+*$;4!/^&0_H3?!7-(8-!(+!5!@)8.$;(5'!,)%);5'([5.($%!

$*!.-)!/+85'5;3!'(,-.&8$%)!.;5%+*$;4!/0_H3!;)8)%.'>!9;$9$+)6!

A>!1`H$$')!et al.!L"]O<!Y)!*$;4:'5.)!.-)!;)8$@);>!$*!%$;45'+!

5%6!5'A)6$+!5+!5!@)8.$;!6)8$%@$':.($%!9;$A')4?!5%6!-5@(%,!

*$:%6!.-)!.7$!K:5%.(.()+?!7)!F.!5!+:;*58)!$%.$!.-)!;)8$@);)6!

%$;45'+<!a>!+$'@(%,!.-(+!@)8.$;(5'!6)8$%@$':.($%!9;$A')4!(%!

.-)!W$(%.!5'A)6$M%$;45'!+958)?!7)!;)8$@);!.-)!.7$!K:5%.(.()+!

;$A:+.'>!5%6!)b8()%.'>?!5''$7(%,!:+!.$!$A.5(%!5!A)..);!+:;*58)!

;)8$%+.;:8.($%<!=%!95;.(8:'5;?!$:;!599;$58-!(+!45%>!$;6);+!$*!

45,%(.:6)!*5+.);!.-5%!;)8)%.!7$;D!L"NO?!7-(8-!85%!+(4('5;'>!

)+.(45.)!5'A)6$+!5%6!+:;*58)!%$;45'+!(%!5!W$(%.!45%%);<!1:;!

9;$9$+)6!599;$58-!5'+$!7$;D+!7(.-!)V(+.(%,!8$%*$85'!#012!

(45,(%,!-5;675;)<!Y)!('':+.;5.)!$:;!$@);5''!4).-$6!9(9)'(%)!

(%!G(,:;)!]<!H$!+:445;([)?!$:;!45(%!8$%.;(A:.($%+!5;)c!

• Directional LCT:!Y)!)V9;)++!%$%&'(%)&$*&+(,-.!+:;*58)!

%$;45'!;)8$@);>!5+!5!@)8.$;!6)8$%@$':.($%!9;$A')4!$%!

.(4)&;)+$'@)6!4)5+:;)4)%.+?!@(5!.-)!^(;)8.($%5'!0(,-.&

8$%)!H;5%+*$;4!/^&0_H3<!

• Cholesky–Wiener Solver:!Y)! +$'@)! .-)! 5A$@)! @)8.$;&!

6)8$%@$':.($%!9;$A')4!)b8()%.'>!(%!.-)!G$:;();!6$45(%!

.$!;)8$@);!.-)!+:;*58)!%$;45'+<!

• Surface Fitting:!Y)!;)8$%+.;:8.!-(,-'>&588:;5.)!$AW)8.!

+:;*58)!6)+8;(9.($%+!A>!F..(%,!+:;*58)!95;54).);+!$%!.-)!

;)8$@);)6!%$;45'+<!!

2. Related Work

! H;5%+()%.!(45,(%,!75+!F;+.!8$%8)9.:5'([)6!A>!d(;45%(!et

al.!LPO?!7-$!A)'()@)6!.-5.!7)!85%!'$$D!5;$:%6!.-)!8$;%);!A>!

9;$A(%,!5!75''!7(.-!5%!:'.;5*5+.!'5+);!5%6!6).)8.$;<!05.);?!.-)!

(6)5!75+!6)4$%+.;5.)6!(%!9;58.(8)!A>!e)'.)%!et al.!L""O!:+(%,!

5!*)4.$+)8$%6!'5+);!5%6!5!+.;)5D!854);5<!17(%,!.$!.-)!8$+.!

$*!+:8-!(45,(%,!-5;675;)?!;)+)5;8-);+!-5@)!5'+$!:+)6!$.-);!

+)%+(%,!.)8-%$'$,()+!+:8-!5+!.(4)&$*&I(,-.!854);5+!L]T?!]"O!

$;!)@)%!;),:'5;!8$%+:4);!854);5+!LS?!]]O<!U$7)@);?!+(%,')&

9-$.$%!5@5'5%8-)!6($6)+!/2fB^+3!L]NM]XO!-5@)!A))%!+-$7%!

.$!A)!95;.(8:'5;'>!@);+5.(')!*$;!+)%+(%,?!5''$7(%,!:+!.$!(45,)!

:%6);!54A()%.!'(,-.(%,?!5.!*5+.!;5.)+!L]ZO?!$;!5.!'$%,!6(+.5%8)+!

L]SO<!B'.45%%!et al.!LNTO!9;$@(6)!5!8$49;)-)%+(@)!;)@()7!$*!

2fB^+!5+!7)''!5+!.-)(;!599'(85.($%+<!

! U5@(%,!859.:;)6!.-)!.;5%+()%.!65.5?!.-)!#012!+8)%)!85%!

A)!;)8$%+.;:8.)6!5+!5!@$':4)!$*!5'A)6$!L""M"RO?!$;!+:;*58)+!

$*!$AW)8.+!L"XM"SO<!g)%);5''>!+9)5D(%,?!)+.(45.($%!$*!5'A)6$!

85%!A)!9$+)6!5+!5%!(%@);+)&F'.);(%,!9;$A')4!7-);)5+!.-5.!$*!

+:;*58)+!(+!$*.)%!9$+)6!5+!5%!(%@);+)&;)%6);(%,!9;$A')4<!Y)!

%$7!9;$@(6)!5!A;()*!;)@()7!$*!.-)!.7$!(%@);+)!599;$58-)+<!

2.1. Inverse Filtering Approaches

! e) '.) %! et al.!L""O!5;)!.-)!F;+.!.$!9$+)!.-)!;)8$@);>!$*!.-)!

5'A)6$!@$':4)!5+!5%!(%@);+)!9;$A')4<!h)'5.(%,!.-)!.;5%+()%.!

4)5+:;)4)%.+!.$!+$4)!#012!+8)%)!:+(%,!-(,-);&$;6);!'(,-.!

.;5%+9$;.?!.-)>!*$;4:'5.)6!+8)%)!;)8$%+.;:8.($%!5+!5!/'(%)5;3!

')5+.&+K:5;)+!9;$A')4!5%6!+$'@)6!.-)!;)+:'.(%,!6)%+)!+>+.)4!

$*!)K:5.($%+!7(.-!F'.);)6!A58D&9;$W)8.($%!/Gaf3<!Y-(')!Gaf!

9;$6:8)+!9;$4(+(%,!;)+:'.+?!(.!$%'>!599;$V(45.)'>!+$'@)+!.-)!

$;(,(%5'!')5+.&+K:5;)+!9;$A')4?!5%6!.-)!;)8$%+.;:8.)6!+8)%)+!

85%!'58D!F%)!6).5('+<!2(%8)!.-)!8$49:.5.($%5'!8$49')V(.>!$*!

Gaf!(+!+.(''!-(,-!5.!(

3!(%!.-)!%:4A);!

!$*!@$V)'+?!'5.);!

5:.-$;+!+$:,-.!.$!;)F%)!Gaf!A>!(49;$@(%,!.-)!K:5'(.>!$*!.-)!

(.);5.(@)!+$'@);+!L"R?!N"O!$;!.-)(;!+9))6!$%!gfi+!L]RO<!

! =%!.-)!8$%*$85'!85+)?!1`H$$')!et al<!L"]O!%$.)!.-5.!-(,-);&!

$;6);!'(,-.!.;5%+9$;.!85%!A)!)V9;)++)6!5+!5!8$%@$':.($%!7(.-!

5!8-5%,)!$*!@5;(5A')+<!J)(;!$@);5''!.;5%+*$;45.($%?!;)*);;)6!

.$!5+!.-)!'(,-.&8$%)!.;5%+*$;4!/0_H3?!)V9;)++)+!.-)!9;$A')4!

$*!e)'.)%!et al.!L""O!5+!5!.-;))&6(4)%+($%5'!+(,%5'!6)A':;;(%,!

9;$A')4<!=%!8$%.;5+.!.$!.-)!Gaf?!.-)!0_H!+$'@)+!.-)!(%@);+)!

9;$A')4!)V58.'>?!5%6!-5+!5!'$7!8$49:.5.($%5'!8$49')V(.>!$*!

(

log )!(%!.-)!%:4A);!

!$*!@$V)'+?!.-5%D+!.$!.-)!:+)!

$*!.-)!G$:;();!.;5%+*$;4<!h)8)%.'>?!B-%!et al.!LN]O!9;$9$+)6!

5%!599;$V(45.)!8$%@$':.($%5'!(45,(%,!.;5%+*$;4!+(4('5;!.$!

.-)!0_H!*$;!.-)!%$%&8$%*$85'!+)..(%,<!

! =%!8$%.;5+.!7(.-!6)8$%@$':.($%?!7-(8-!7)!85%!:'.(45.)'>!

;)'5.)!.$!.-)!6(j:+($%!)K:5.($%?!0(%6)''!et al.!L"PO!+:,,)+.!.$!



! "#$!D'#-/2*-3!@,(&.*!𝜏! ")$!:,'.#(/!𝜐!4',.!𝜏!! "0$!<&'4#0*!4233*+!,-3,!𝜐! "+$!K',&-+L3'&3=!/&'4#0*!

125&'*!M7!Method overview:!N!1m × 1m × 2ns!@,(&.*!𝜏!,4!3'#-/2*-3/!"#$!2/!H(3*'*+!&/2-5!3=*!+2'*032,-#(!(25=3L0,-*!3'#-/4,'.!3,!,)3#2-!3=*!

/&'4#0*!-,'.#(/!")$7!O*!2-3*5'#3*!3=*!/&'4#0*!-,'.#(/!3,!,)3#2-!3=*!H-#(!'*0,-/3'&03*+!/&'4#0*!"0$E!F=20=!2/!/2.2(#'!3,!3=*!5',&-+L3'&3=!"+$7!

!



′



′

+$'@)!.-)!.;5%+()%.!(45,(%,!9;$A')4!A>!4$6)'(%,!-(,-&$;6);!

'(,-.!.;5%+9$;.!5+!75@)!9;$95,5.($%!(%!.-)!.-;))&6(4)%+($%5'!

+958)?!5%6!+$'@)!.-)!;)+:'.(%,!(%@);+)!9;$A')4!)b8()%.'>!(%!

.-)!G$:;();!6$45(%!:+(%,!CME!4(,;5.($%<!J(+!4).-$6!5'+$!

-5+!5!(

log )!8$49:.5.($%5'!8$49')V(.>!6:)!.$!.-)!:+)!

$*!.-)!G$:;();!.;5%+*$;4<!k).-$6+!A5+)6!$%!6(j;58.(@)!75@)!

9;$95,5.($%?!)<,<!9-5+$;!F)'6+!LNNO?!5(4!*:;.-);!.$!$@);8$4)!

.-)!'(4(.5.($%+!(%!.-)!(45,(%,!4$6)'!6:)!.$!5++:49.($%+!+:8-!

5+!+(%,')!+85..);(%,?!5%6!.-)!'58D!$*!$88':+($%+!(%!.-)!-(66)%!

+8)%)<!=*!.-)!8$%*$85'!+).:9!(+!:+)6?!.-)!9-5+$;!F)'6!4).-$6!

85%!A)!(49')4)%.)6!(%!.);4+!$*!.-)!0_H!*$;!5!(

log )!

8$49:.5.($%5'!8$49')V(.><!

! Y-);)5+!A$.-!.-)!0_H!5%6!.-)!M!4(,;5.($%!599;$58-)+!

5;)!)V.;)4)'>!)b8()%.?!.-)>!6$!%$.!(%%5.)'>!-5@)!.-)!5A('(.>!

.$!)+.(45.)!.-)!+:;*58)!%$;45'+!$*!+8)%)!$AW)8.+<!H$!F%6!.-)!

+:;*58)!%$;45'+!5'$%,!7(.-!.-)!5'A)6$!@$':4)?!U)(6)!et al.!

L"NO!9$+)!.-)!.;5%+()%.!(45,(%,!9;$A')4!5+!5%!$9.(4([5.($%!

(%!.-)!5'A)6$!5%6!.-)!+:;*58)!%$;45'!@5;(5A')+<!B'.-$:,-!.-)!

;)9$;.)6!;)+:'.+!5;)!9;$4(+(%,?!$%)!45W$;!'(4(.5.($%!$*!+:8-!

5!4).-$6!(+!.-)!(

)!8$49')V(.>!(%!A$.-!8$49:.5.($%!5%6!

4)4$;>?!5+!7)''!5+!.-)!%$%8$%@)V(.>!$*!.-)!$@);5''!9;$A')4!

*$;4:'5.($%<!J)!^&0_H!-)'9+!:+!.$!+$'@)!5!+(4('5;!5'A)6$M

%$;45'!)+.(45.($%!9;$A')4!7(.-!.-)!+54)!'$7!8$49')V(.>!$*!

.-)!0_H!5%6!M!4(,;5.($%<!g(@)%!.-)!+-5;)6!G$:;();!;$$.+!

58;$++!.-)!0_H?!M!5%6!9-5+$;!F)'6+?!(.!45>!A)!9$++(A')!.$!

599'>!$:;!6(;)8.($%5'!599;$58-!.$!M!4(,;5.($%!5%6!9-5+$;!

F)'6+!5+!7)''?!5'.-$:,-!7)!6$!%$.!5..)49.!.-(+!(%!$:;!7$;D<!

2.2. Inverse Rendering Approaches

! =%!8$%.;5+.!7(.-!.-)!(%@);+)!F'.);(%,!599;$58-)+?!(%@);+)!

;)%6);(%,!/5%5'>+(+&A>&+>%.-)+(+3!4).-$6+?!)<,<!L"SO?!+)5;8-!

*$;!@5':)+!$*!.-)!#012!+:;*58)!95;54).);+!/)<,<!ah^G+!5%6!

+:;*58)!%$;45'+3!.-5.!7$:'6!9;$6:8)!.-)!$A+);@)6!.;5%+()%.+!

(*!.-)!#012!+:;*58)!7);)!.$!A)!;)%6);)6<!2(%8)!5!*:''!+)5;8-!

(%!.-)!+:;*58)!95;54).);!+958)!7$:'6!A)!(%.;58.5A')?!(%@);+)!

;)%6);(%,!(+!.>9(85''>!9);*$;4)6!@(5!6(j);)%.(5A')!;)%6);(%,!

/(<)<?!5%!)%);,>!4(%(4([5.($%!(%!+:;*58)!95;54).);+3<!2:;*58)!

95;54).);+!;)8$@);5A')!(%!.-(+!75>!(%8':6)!+:;*58)!'$85.($%+!

5%6!%$;45'+!LNPO?!5%6!('':4(%5.($%!)j)8.+!+:8-!5+!+85..);(%,!

LNQMNZO!5%6!(%.);;)I)8.($%+!LNS?!PTO<!

2.3. Surface Fitting Methods

! 1AW)8.!+:;*58)!85%!A)!F.!$%!.-)!$A.5(%)6!+:;*58)!%$;45'+!

)(.-);!95;54).;(85''>!$;!%$%&95;54).;(85''><!#$%&95;54).;(8!

F..(%,!(+!8$44$%'>!:+)6!*$;!+.);)$&A5+)6!N^!;)8$%+.;:8.($%!

LP"MPNO!7-);)5+!95;54).;(8!599;$58-)+!5;)!4$;)!+:(.)6!*$;!

4)+-!;)F%)4)%.!LPPMPXO<!2:;*58)!F..(%,!45>!A)!+))%!5+!5%!

(%@);+)!9;$A')4! 7-);)! .-)!,$5'! (+!.$! (%.);9$'5.)!5! +4$$.-!

45%(*$'6!7(.-$:.!@($'5.(%,!.-)!,(@)%!%$;45'!8$%6(.($%+<!Y)!

85%!%5.:;5''>!*$;4:'5.)!+:;*58)!F..(%,!5+!6(j:+($%!9;$8)++)+!

$;!)%);,>!4(%(4([5.($%!4).-$6+?!A$.-!$*!7-(8-!+))D!.$!F%6!

.-)!;(,-.!.;56)&$j!A).7))%!;),:'5;(.>!/+4$$.-%)++3!5%6!65.5!

F6)'(.>!/+:;*58)!$;.-$,$%5'(.>!.$!.-)!,(@)%!%$;45'+3<!Y)!:+)!

.-)!)%);,>&4(%(4([5.($%!599;$58-!$*!LPZO!.$!F.!5!+:;*58)!$%!

$:;!;)8$@);)6!%$;45'+<!

3. Mathematical Framework

! B*.);!A;()I>!;)@()7(%,!.-)!@$':4).;(8!5'A)6$!4$6)'!5%6!

6(+8:++(%,!(.+!'(4(.5.($%+?!7)!6)@)'$9!$:;!6(;)8.($%5'!5'A)6$!

4$6)'?!5%6!9;$9$+)!)b8()%.!75>+!*$;!+$'@(%,!.-)!5++$8(5.)6!

(%@);+)!9;$A')4!$*!)+.(45.(%,!.-)!+:;*58)!%$;45'+<!!

3.1. =e Volumet ric Albedo Model

! =%!.;5%+()%.!(45,(%,!599;$58-)+?!5!.(4)&;)+$'@)6!6).)8.$;!

(+!:+)6!.$!4)5+:;)!.-)!(%8(6)%.!I:V!$*!9-$.$%+!5+!5!*:%8.($%!

$*!)4(..)6!'(,-.!(49:'+)+<!\58-!$*!.-)+)!.(4)!4)5+:;)4)%.+!

;)8$;6+!.-)!(49:'+)!;)+9$%+)!$*!.-)!#012!+8)%)!5.!9$+(.($%+!

$%!5!@(+(A')!+:;*58)!.$!9;$6:8)!5!@$':4)!$*!.;5%+()%.+<!

! 0).!:+!6)%$.)!.-)!.-;))&6(4)%+($%5'!+8)%)!8$$;6(%5.)+!A>!

(, , )?!5%6!5++:4)!.-)!@(+(A')!+:;*58)!(+!9$+(.($%)6!5'$%,!

 = 0<!Y)!6)%$.)!A>!(

′

, 

′

,  = 0)!9$+(.($%+!$%!.-(+!@(+(A')!

+:;*58)l!+))!G(,:;)!"<!B!8$44$%!.;5%+()%.!(45,(%,!4$6)'!(+!

.-)!8$%*$85'!@$':4).;(8!5'A)6$!4$6)'!

(

′

, 

′

, ) = ddd

(, , )



Ω

/"3!







(

′

−)

+ (

′

−)

+ 

−



(%!7-(8-!!6)%$.)+!5!.-;))&6(4)%+($%5'!5'A)6$!@$':4)!7(.-!

F%(.)!+:99$;.!Ω?!5%6!()!;)'5.)+!.-)!;$:%6&.;(9!.(4)!$*!I(,-.!

$*!'(,-.!7(.-!.7(8)!.-)!6(+.5%8)!!A).7))%!.-)!+8)%)!(, , )!

5%6!.-)!+)%+(%,!(

′

, 

′

,  = 0)!'$85.($%+<!U);)?!≈3 ×10

6)%$.)+!.-)!+9))6!$*!'(,-.!7-(')!1 

⁄

= (2 

⁄

)

!4$6)'+!.-)!

;56($4).;(8!*5''&$j!6:)!.$!6(+.5%8)<!J)!+85'(%,!1 

⁄

!85%!A)!

;)4$@)6!*;$4!/"3!(*!7)!9;)+85')!!A>!(2 

⁄

)

!(%!56@5%8)<!=%!

.-)!85+)!$*!;).;$&;)I)8.(@)!+:;*58)+?!5!*5''&$j!*58.$;!$*!1/

(+!4$;)!8$44$%'>!5++:4)6<

! H$!6(+8;).([)!4$6)'!/"3?!7)!+549')!Ω!:+(%,!?!!5%6!!

9$(%.+!$%!.-)!&?!&!5%6!&5V)+?!;)+9)8.(@)'><!B++:4(%,!.-5.!

.-)!.;5%+()%.!!-5+!A))%!9;)&+85')6!A>!(2 

⁄

)

?!7)!85%!7;(.)!

.-)!6(+8;).([)6!4$6)'!8$4958.'>!:+(%,!45.;(V!%$.5.($%!5+!

= ,!

/]3!

(%!7-(8-!, ∈

𝑁

𝑀

!5%6!!(+!5!A(%5;>!45.;(V!7(.-!@5':)+!

$A.5(%)6!A>!+549'(%,!()<!2(%8)!!(+!5!'$7&95++!$9);5.$;!$*!

-(,-!8$%6(.($%!%:4A);?!.-)!.5+D!$*!F%6(%,!!*;$4!,(@)%!!(+!

5%!(''&9$+)6!9;$A')4!LPSO<!h5.-);!.-5%!8$49:.)!.-)!+$':.($%!

6(;)8.'>!5+!

opt

= 

−1

?!7)!+-$:'6!F%6!(.!5+!.-)!+$':.($%!$*!

.-)!;),:'5;([)6!')5+.&+K:5;)+!9;$A')4!

minimize () =

‖

−

‖

+ 

‖



‖

(%!7-(8-!!;)9;)+)%.+!.-)!.;56)&$j!A).7))%!65.5!F6)'(.>!5%6!

;),:'5;(.>!/+4$$.-%)++3!$*!.-)!+$':.($%<!

! 1`H$$')!et al.!L"]O!%$.)!.-5.!9;$A')4!/N3!85%!A)!+$'@)6!(%!

5%!)b8()%.!45%%);!7(.-!.-)(;!+$&85'')6!'(,-.&8$%)!.;5%+*$;4!

/0_H3<!=*!7)!6)%$.)!.-)(;!;)+549'(%,!$9);5.$;!A>!?!7)!85%!

)V9;)++!9;$A')4!/N3!)K:(@5')%.'>!(%!.);4+!$*!.-)!;)+549')6!

5'A)6$! = 

∗

!5%6!.;5%+()%.+! = 

∗

!5+!

minimize () =

‖

−

‖

+ 

‖



‖

(%!7-(8-!= 

∗

!(+!5!.-;))&6(4)%+($%5'!F'.);!/.-5.!(+?!5!

'(%)5;!+958)&(%@5;(5%.!$9);5.$;3?!7-$+)!(49:'+)!;)+9$%+)!(+!

+-$7%!(%!G(,:;)!N!/53<!2)8.($%!"!$*!.-)!+:99')4)%.!9;$@(6)+!

.-)!6).5('+!$*!.-)!;)+549'(%,!$9);5.$;!

∗

! 2(%8)!!(+!5!.-;))&6(4)%+($%5'!F'.);?!7)!85%!8$49:.)!.-)!

+$':.($%!

opt

= (

∗

+ )

−1



∗

!$*!/P3!)b8()%.'>!(%!.-)!

G$:;();!6$45(%!:+(%,!Y()%);!6)8$%@$':.($%<!J)!;),:'5;(.>!

95;54).);!!85%!A)!(%.);9;).)6!5+!.-)!%$(+)&.$&+(,%5'!;5.($!(%!

.-(+!F'.);(%,!8$%.)V.<!J)!+$':.($%!$*!$:;!$;(,(%5'!9;$A')4!/N3!

(+!$A.5(%)6!A>!;)+549'(%,!.-)!6)8$%@$'@)6!+$':.($%!:+(%,!.-)!

56W$(%.!;)+549'(%,!$9);5.$;?!.-5.!(+?!

opt

= 

opt

3.2. =e Directional Albedo Model

! =%!.-)!85+)!$*!(+$.;$9(8!9$(%.!)4(..);+?!/"3!(+!5%!56)K:5.)!

4$6)'!*$;!-(,-);&$;6);!'(,-.!.;5%+9$;.<!U$7)@);?!*$;!.>9(85'!

6(j:+)!$;!054A);.(5%!$AW)8.!+:;*58)+?!+:8-!5!4$6)'!(,%$;)+!

.-)!;56($4).;(8!*5''&$j!6:)!.$!054A);.`+!8$+(%)!'57?!(<)<?!.-)!

*5''&$j!6:)!.$!.-)!5%,')!A).7))%!.-)!(%8(6)%.!'(,-.!;5>+!5%6!

.-)!+:;*58)!%$;45'+l!+))!LQTO<!=%8$;9$;5.(%,!8$+(%)!.);4+!(%!

/"3!%$.!$%'>!>()'6+!5!4$;)!588:;5.)!*$;75;6!4$6)'?!A:.!4$;)!

(49$;.5%.'>?!(.!)%5A')+!;)8$@);>!$*!+:;*58)!%$;45'+!*;$4!.-)!

.;5%+()%.+!@(5!.-)!(%@);+)!4$6)'<!

! ^)%$.(%,!.-)!.7$!+95.(5'!8$$;6(%5.)+!A>!= (, , )!5%6!



′

= (

′

, 

′

,  = 0)!*$;!A;)@(.>?!7)!:965.)!4$6)'!/"3!5+!!

(

′

, 

′

, ) = d



()()





′

−

‖

′

−‖



Ω

/Q3!

2



(

′

−)

+ (

′

−)

+ 

−,!

(%!7-(8-!() = (

𝑥

, 

𝑦

, 

𝑧

)() ∈

!(+!.-)!+:;*58)!%$;45'!

5.!<!1:;!4$6)'!85%!A)!+))%!5+!5!F;+.&$;6);!599;$V(45.($%!$*!

.-)!9->+(85''>!A5+)6!$%)!(%!L"SO?!5''$7(%,!:+!.$!;)*$;4:'5.)!

%$;45'!)+.(45.($%!5+!5!'(%)5;!')5+.&+K:5;)+!9;$A')4<!k$6)'!

/Q3!(+!5'+$!(6)%.(85'!.$!.-)!$%)!(%!L"NO!+5@)!*$;!.-)!5A+)%8)!$*!

$88':+($%!.);4+<!2)8.($%!N!$*!$:;!959);!+:99')4)%.!6);(@)+!

.-)!;)'5.($%+-(9!A).7))%!/Q3!5%6!.-)!9->+(85'!$%)<!B++:4(%,!

*:;.-);!.-5.!.-)!9;$W)8.($%+!



(),



′

−

‖

′

−‖



= cos = 1, ∀ 

′

, ,!

/R3!

4$6)'!/Q3!;)6:8)+!.$!.-)!@$':4).;(8!5'A)6$!4$6)'!/"3<!

! #$.)!(%!/Q3!.-5.!5'A)6$!() ∈!(+!5!+85'5;!K:5%.(.>?!5%6!

+:;*58)!%$;45'!() ∈

!(+!5!:%(.&%$;4!@)8.$;<!h5.-);!.-5%!

;)9;)+)%.!.-)!.7$!K:5%.(.()+!:+(%,!+)95;5.)!@5;(5A')+?!7)!85%!

8$4A(%)!.-)4!(%.$!5!+(%,')!6(;)8.($%5'&5'A)6$!@)8.$;!

() = (

𝑥

, 

𝑦

, 

𝑧

)() = ()() ∈

/X3!

+:8-!.-5.!.-)!6(;)8.($%!5%6!.-)!45,%(.:6)!$*!()!)%8$6)!(.+!

+:;*58)!%$;45'!5%6!5'A)6$?!;)+9)8.(@)'><!2:A+.(.:.(%,!/X3!(%!

4$6)'!/Q3!5%6!:+(%,! =

‖



′

−

‖

?!7)!$A.5(%!.-)!6(;)8.($%5'!

5'A)6$!4$6)'!



(



′

, 

′

, 

)



d

〈



(



)

, 

′

−

〉



Ω

/Z3!







(

′

−)

+ (

′

−)

+ 

−



;)'5.(%,!.-)!6(;)8.($%5'!5'A)6$!!.$!.-)!.;5%+()%.+!<!G(,:;)!P!

('':+.;5.)+!5%6!+:445;([)+!.-)!6(;)8.($%5'!5'A)6$!4$6)'<!

! /53!0_H!F'.);!D);%)'! ! /A3!^(;)8.($%5'!0_H!F'.);!D);%)'+!/&?!&!5%6!&6(;)8.($%+3!

Figure 3. Constructing the D-LCT ﬁlter kernels: *e light-cone transform produces a three-dimensional, shift-invariant kernel (a). *e D-

LCT (b) consists of three shift-invariant kernels that relate directional albedo (albedo + normal) to the transients. *e 𝑧-directional D-LCT

kernel (b, far right) is identical to the LCT kernel (a).

!

!

!

!

!

!

!

!

!

Figure 4. Directional albedo model: At location 𝐬 = (𝑥, 𝑦 , 𝑧) on

the object, directional albedo 𝛖(𝐬) has direction and magnitude of

the normal 𝐧(𝐬) and the albedo 𝜌(𝐬), respectively. Contribution of

albedo 𝜌(𝐬) to the surface at 𝐬

′

= (𝑥

′

, 𝑦

′

, 𝑧 = 0) decreases, in the

ﬁrst order, as the cosine of the angle 𝜃 between 𝛖(𝐬) and 𝐬

′

− 𝐬.

Visible surface





(



)

Hidden object





(



)



(



)





′

−





′



′



′





(



)

! H$!6(+8;).([)!$:;! 6(;)8.($%5'!5'A)6$!4$6)'?!7)!+549')!Ω!

+(4('5;'>!.$!/]3<!J(+!9;$6:8)+!.-)!+>+.)4!$*!'(%)5;!)K:5.($%+!

(%!$:;!6(;)8.($%5'!5'A)6$+!= 

𝑥

∗

, 

𝑦

∗

, 

𝑧

∗



∗

= !

/S3!

(%!7-(8-!!(+!.-)!45.;(V!*;$4!/]3?!5%6!7)!$A.5(%!.-)!)%.;()+!

$*!= 

𝑥

, 

𝑦

, 

𝑧

!A>!+549'(%,!(

′

−)!$%!Ω<!h)8$@);(%,!

.-)!6(;)8.($%5'!5'A)6$!!,(@)%!.-)!.;5%+()%.+!!(+!(''&9$+)6!(%!

.-5.!(.!;)K:(;)+!:+!.$!F%6!.-)!@5':)+!$*!3

!@5;(5A')+!7(.-!

$%'>!

!)K:5.($%+<!2:8-!5!;5%D&6)F8()%.!9;$A')4!85%!A)!

+$'@)6!A>!*$;4:'5.(%,!/S3!5+!.-)!')5+.&+K:5;)+!9;$A')4!

minimize () =

‖

−

‖

+ 

‖



‖

/"T3!

+(4('5;'>!.$!.-)!;),:'5;([)6!599;$58-!(%!/N3<!Y-(')!7)!8$:'6!

5'.);%5.(@)'>!$9.(4([)!.-)!He&0"!@);+($%!$*!9;$A')4!/"T3!.$!

$A.5(%!5!A)..);!+$':.($%?!7)!*$8:+!$%!$:;!0]!@5;(5%.!*$;!%$7!

.$!+$'@)!/"T3!)b8()%.'>!5+!5!@)8.$;!6)8$%@$':.($%!9;$A')4<!

3.3. Directional Light-cone Transform

! Y-);)5+!$:;!')5+.&+K:5;)+!$9.(4([5.($%!9;$A')4!/"T3!-5+!

.-)!+(49')?!8'$+)6&*$;4!+$':.($%!



opt

= (

∗



∗

+ )

−1



∗



∗

,!

/""3!

.-(+!+$':.($%!(+!.$$!)V9)%+(@)!.$!8$49:.)!%5(@)'>!*$;!.>9(85'!

9;$A')4+! 7(.-!

= 

 ≈10

!@$V)'+<! _$49:.(%,!

opt

6(;)8.'>!7(.-!%:4);(85'!4).-$6+!+:8-!5+!_-$')+D>!5%6!0^0!

6)8$49$+(.($%+!7$:'6!(%8:;!5!(

)!8$+.!7-);)5+!(.);5.(@)!

$%)+!/)<,<!8$%W:,5.)!,;56()%.+3?!5!(

)!8$+.<!J)+)!,)%);5'!

4).-$6+!5;)!.-);)*$;)!:%+:(.)6!.$!9;58.(85'!9;$A')4!+([)+<!

!! H$!+$'@)!9;$A')4!/"T3!)b8()%.'>?!7)!,)%);5'([)!.-)!0_H!

.)8-%(K:)!:+)6!(%!/P3!.$!.-)!@)8.$;(5'!9;$A')4<!Y)!85%!7;(.)!

9;$A')4!/"T3!)K:(@5')%.'>!5+!

minimize (







) =





𝑥

, 

𝑦

, 







−



+ 

‖









‖

/"]3!

(%!7-(8-!















𝑥









𝑦









𝑧









𝛖















𝑥

∗



𝑦

∗



𝑧

∗

















𝐓

𝑑

∗









𝑥



𝑦



𝑧









𝛖

/"N3!

(+!.-)!;)+549')6!@5;(5A')?!5%6! = 

∗

!5+!A)*$;)<!G;$4!.-)!

+$':.($%!





opt

!$*!/"]3?!7)!;)8$@);!.-)!+$':.($%!$*!.-)!$;(,(%5'!

9;$A')4!/"T3!5+!

opt

= 

𝑑







opt

<!2)8.($%!]!$*!.-)!+:99')4)%.!

6);(@)+!.-)!;)+549');!

𝑑

∗

!5%6!(.+!;)'5.($%+-(9!.$!

∗

!(%!/P3<!

! #$.)!(%!/"]3!.-5.!

𝑥

?!

𝑦

!5%6!!5;)!+-(*.&(%@5;(5%.?!+$!.-)>!

85%!A)!8$49$+)6!7(.-!.-)!'(,-.&8$%)!F'.);!!.$!9;$6:8)!.-)!

6(;)8.($%5'!'(,-.&8$%)!F'.);+!

𝑥

?!

𝑦

!5%6!<!J)(;!F'.);!

D);%)'+!5;)!+-$7%!(%!G(,:;)!N!/A3<!2(%8)!5''!.-)!$9);5.$;+!(%!

9;$A')4!/"]3!5;)!F'.);+?!$%)!85%!(%.);9;).!/"]3!5+!5!@)8.$;(5'!

6)8$%@$':.($%!9;$A')4?!7(.-!5!FV)6!%$(+)&.$&+(,%5'!;5.($!!

58;$++!.-)!@)8.$;!*;)K:)%8()+<!1%)!85%!+$'@)!/"]3!)b8()%.'>!

:+(%,!$:;!@)8.$;!)V.)%+($%!$*!Y()%);!6)8$%@$':.($%<!

3.4. Cholesky–Wiener Deconvolution

! ^)%$.(%,!.-)!45.;(8)+!(%!/"]3!5+!

𝑥

= 

𝑥

, 

𝑦

= 

𝑦

5%6!

𝑧

= !*$;!+(49'(8(.>?!7)!7;(.)!.-)!%$;45'!)K:5.($%+!

29-);)+

a:%%>!

2);59(+

/53!U(66)%!$AW)8.!

/A3!0_H!@$':4)!

/83!^&0_H!%$;45'!@$':4)+!/&?!&!5%6!&8$49$%)%.+3!

/63!^&0_H!+:;*58)!

Figure 5. Transient imaging using the D-LCT: D-LCT (c) captures ﬁne details of object surfaces (a) not captured by the LCT (b). Volumes

(b)–(c) are rendered using maximum intensity projection. D-LCT surfaces (d) are ﬁt directly onto the D-LCT normals (c). For (d), we used

known background masks to ﬁrst remove the background points. All hidden objects have diﬀuse surfaces.

5++$8(5.)6!7(.-!.-)!')5+.&+K:5;)+!9;$A')4!/"]3!5+!









𝑥

+  

𝑥

∗



𝑦



𝑥

∗



𝑧



𝑦

∗



𝑥



𝑦

+  

𝑦

∗



𝑧



𝑧

∗



𝑥



𝑧

∗



𝑦



𝑧

+ 

















𝐇

∗

𝐇+𝜆𝐈















𝑥









𝑦









𝑧









𝛖















𝑥

∗





𝑦

∗





𝑧

∗











𝐇

∗

𝛕







/"P3!

.-5.!(+?!5!3 ×3!A'$8D!+>+.)4!$*!)K:5.($%+?!7-);)!)58-!A'$8D!

)')4)%.!(+!5!F'.);!$;!5!F'.);!+(,%5'<!J)!+.;:8.:;)!$*!/"P3!.-:+!

+:,,)+.+!.-5.!7)!+$'@)!.-)!3 ×3!+>+.)4!:+(%,!.-)!_-$')+D>!

6)8$49$+(.($%?!9);*$;4(%,!.-)!.7$!5++$8(5.)6!*$;75;6&!5%6!

A58D&+:A+.(.:.($%+!:+(%,!F'.);(%,!$9);5.($%+<!J)!;(,-.&-5%6!

+(6)!@)8.$;!

∗

!85%!A)!8$49:.)6!(%!.-)!G$:;();!6$45(%?!5+!

)58-!

𝑥

∗

?!

𝑦

∗

!5%6!

𝑧

∗

!(+!5!F'.);<!

! i+(%,!.-)!0^0!@5;(5%.!$*!.-)!_-$')+D>!*58.$;([5.($%?!7)!

*58.$;!.-)!45.;(V!= 

∗

+ !5+!= 

∗

?!7-);)!

=











𝑦𝑥





𝑧𝑥



𝑧𝑦









, =









𝑥𝑥



𝑦𝑦



𝑧𝑧







/"Q3!

5%6!.-)!)')4)%.+!$*!!5%6!!5;)!,(@)%!A>!



𝑗𝑗

= 

𝑗

∗



𝑗

+ −

∑



𝑗𝑘



𝑘𝑘



𝑗𝑘

∗

𝑗−1

𝑘=1

/"R3!



𝑖𝑗

= 

𝑗𝑗

−1





𝑖

∗



𝑗

−

∑



𝑗𝑘



𝑘𝑘



𝑗𝑘

∗

𝑗−1

𝑘=1



,!!

:+(%,!.-)!8$%@)%.($%!1 = , 2 = , 3 = !(%!A$.-!+:4+<!Y)!

85%!;)56('>!@);(*>!.-)!6>%54(8!9;$,;544(%,!9;$8)6:;)!/"R3!

A>!599'>(%,!.-)!)'(4(%5.($%!+.)9+!$*!.-)!_-$')+D>!5',$;(.-4!

LQ"O!.$!.-)!A'$8D!)')4)%.+!$*!45.;(V!<!

! G(%5''>?!.-)!.;(5%,:'5;([)6!+>+.)4!

∗

= 

∗

!85%!A)!

+$'@)6!:+(%,!*$;75;6&!5%6!A58D&+:A+.(.:.($%+!

= 

−1



∗

, = 

−∗



−1

,!

/"X3!

A$.-!$*!7-(8-!85%!A)!9);*$;4)6!5+!5!+);()+!$*!F'.);(%,!+.)9+!

(%!.-)!G$:;();!6$45(%<!G$;!)V549')?!.-)!A'$8D&)')4)%.+!$*!!

85%!A)!$A.5(%)6!:+(%,!!

= 

𝑥

∗

?!

= 

𝑦

∗

 −

𝑦𝑥



!5%6!



= 

𝑦

∗

 −

𝑧𝑥



−

𝑧𝑦



<!1A+);@)!-);)!

𝑦𝑥



?!

𝑧𝑥



5%6!

𝑧𝑦



!45>!A)!8$49:.)6!(%!.-)!G$:;();!6$45(%?!5+!)58-!

45.;(V!

𝑦𝑥

?!

𝑧𝑥

!5%6!

𝑧𝑦

!;)9;)+)%.+!5!N^!F'.);<!Y)!8$49:.)!

.-)!)')4)%.+!$*!!(%!5!+(4('5;!45%%);<!

! G$;!'5;,)!9;$A')4+!7-);)!+.$;(%,!.-)!G$:;();!8$)b8()%.+!

$*!.-)!A'$8D!)')4)%.+!$*!!5%6!!(+!%$.!*)5+(A')?!7)!85%!:+)!

5%!(.);5.(@)!+$'@);!'(D)!8$%W:,5.)!,;56()%.+?!5%6!8$49:.)!.-)!

*$;75;6!4599(%,!(

∗

+ )!:+(%,?!5,5(%?!F'.);(%,!

$9);5.($%+!(%!.-)!G$:;();!6$45(%<!

3.5. Surface Reconstruction

! U5@(%,!$A.5(%)6!.-)!F)'6!!$*!6(;)8.($%5'!5'A)6$?!7)!:+)!

.-)!4).-$6!$*!LQ]O!.$!F.!5!%$%95;54).;(8!+:;*58)<!G(..(%,!.-)!

+:;*58)!54$:%.+!.$!;)8$@);(%,!5%!(%6(85.$;!*:%8.($%!!$*!.-)!

+8)%)!$AW)8.!+$!.-5.!.-)!,;56()%.!$*!!)K:5'+!<!\V9;)++)6!5+!

5%!$9.(4([5.($%!9;$A')4?!7)!-5@)!

minimize () =

‖



∗

−

∗



‖

+ 

‖



‖

/"Z3!

(%!7-(8-!!6)%$.)+!5!6(+8;).([5.($%!$*!.-)!.-;))&6(4)%+($%5'!

,;56()%.!$9);5.$;<!=%!9;58.(8)?!-$7)@);?!.-)![);$&')@)'&+).!$*!

.-)!+$':.($%!

opt

!$*!/"Z3!45>!6)@(5.)!*;$4!.-)!+:;*58)!$*!.-)!

.;:)!+8)%)!$AW)8.!6:)!.$!.-)!%$(+)!(%!!5%6!.-)!6(+8;).([5.($%!

Figure

7. Impact of regularity parameter on depth and surface normals: *e D-

LCT produces accurate depth (left plots) and normals

(right plots) over a wide range of values of 𝜆, demonstrating its usefulness for cases where the SNR is not known exactly. *e LCT normal

errors are for the normal vectors obtained using [53] with the LCT depth as the input. Points of minimum error marked with do

ts.

Depth

error

Normal

error

Figure

6. Accuracy of D-LCT and LCT: *e LCT and D-LCT depths (left plots) have the RMSE of 5.97 and 4.96cm,

and the MAE of 1.87

and 1.59cm, respectively. *e LCT and D

-LCT surface normals (right plots) have end-

point RMSE 0.91 and 0.52cm, and MAE of 0.61 and

0.38cm, respectively. *e LCT does not natively produce surface normals, so we obtain them using [5

4] with the LCT depth as the input.

RMSE!

0_H!

^&0_H!

!

MAE!

0_H!

^&0_H!

Depth error

Normal error

RMSE!

0_H!

^&0_H!

MAE!

0_H!

^&0_H!

0_H!

^&0_H!

0_H!

^&0_H!

$*!.-)!9;$A')4<!H$!4(.(,5.)!+:8-!(++:)+?!7)!*$''$7!5,5(%!.-)!

599;$58-!$*!LQ]O!.$!)V.;58.!5%!(+$+:;*58)!$*!

opt

!(%+.)56<!

4. Experimental Results

! 2(%8)!$:;!6(;)8.($%5'!0_H!599;$58-!;)8$@);+!A$.-!5'A)6$!

5%6!+:;*58)!%$;45'+?!(.!85%!A)!:+)6!(%!.-)!.;56(.($%5'!8$%.)V.!

$*!.7$&6(4)%+($%5'!#012!(45,(%,!5+!7)''!5+!.$!;)8$%+.;:8.!

+:;*58)+!(%!.-;))!6(4)%+($%+<!Y)!6)4$%+.;5.)!$:;!599;$58-!

*$;!A$.-!:+)&85+)+?!5'+$!8$495;(%,!(.!.$!4).-$6+!+9)8(5'([)6!

.$!)58-!$%)<!Y)!:+)!m#012!LQNO!5%6!2.5%*$;6!L"PO!65.5+).+!

*$;!)V9);(4)%.5'!@5'(65.($%<!J)!m#012!65.5+).!8$%+(+.+!$*!

4:'.(9')&A$:%8)!.;5%+()%.+!$*!+>%.-).(8!$AW)8.+!0.50m!575>!

*;$4!5!1m ×1m!@(+(A')!+:;*58)<!J)!65.5+).!-5+!5!.)49$;5'!

;)+$':.($%!$*!512!9(V)'+!7(.-!A(%+!$*!7(6.-!10ps?!5%6!+95.(5'!

;)+$':.($%+!$*!256 ×256!9(V)'+<!J)!2.5%*$;6!+).!8$%+(+.+!$*!

.;5%+()%.+!4)5+:;)6!$%!5!2m ×2m!+:;*58)!$*!%5.:;5'!-(66)%!

$AW)8.+!1m!575>?!7(.-!54A()%.!'(,-.!5%6!%$(+)<!J(+!65.5+).!

-5+!5!+95.(5'!;)+$':.($%!$*!512 ×512!$;!64 ×64!9(V)'+?!5%6!

5!.)49$;5'!;)+$':.($%!$*!512!7(.-!A(%+!$*!7(6.-!32ps<!

4.1. NLOS Imaging Experiments

Directional Transient Imaging. G(,:;)!Q!+-$7+!.-)!%$;45'!

(45,)+!$A.5(%)6!:+(%,!.-)!^&0_H<!J)+)!(45,)+!8$%.5(%!.-)!

F%)!@5;(5.($%+!(%!$AW)8.!+:;*58)+!+:8-!.-)!+4$$.-!+:;*58)!$*!

.-)!+9-);)+!5%6!.-)!*:;!$*!.-)!A:%%><!J)+)!6).5('+!7$:'6!A)!

6(b8:'.!.$!;)8$@);!9$+.&-$8!*;$4!5'A)6$&$%'>!(45,)+!:+(%,!

6).5('&)%-5%8)4)%.!.)8-%(K:)+?!*$;!)V549')<!J)!;)+:'.+!*$;!

.-)!0_H!;)+)4A')!.-)!&8$49$%)%.!$*!.-)!^&0_H!$%)+!7(.-!

+:A.')!6(j);)%8)+!.-5.!85%!A)!)V9)8.)6!*;$4!.-)!+(49'(*>(%,!

5++:49.($%!/R3<!Y)!;)%6);!6(;)8.($%5'!5'A)6$!@$':4)+!:+(%,!

45V(4:4!(%.)%+(.>!9;$W)8.($%c!*$;!)58-!9$(%.!(

′

, 

′

,  = 0)!

$%!.-)!(45,)!9'5%)?!7)!F%6!5'$%,!.-)!&5V(+!.-)!6(;)8.($%5'&

5'A)6$!7(.-!45V(4:4!&8$49$%)%.!@5':)+<!Y)! ;)8$%+.;:8.!

.-)!+:;*58)+!A>!F;+.!45+D(%,!$:.!.-)!A58D,;$:%6!9(V)'+!7(.-!

,;$:%6&.;:.-!45+D+?!5%6!9);*$;4(%,!f$(++$%!;)8$%+.;:8.($%!

$%!.-)!*$;),;$:%6!9$(%.+!5%6!.-)!6(;)8.($%5'!5'A)6$<!Y)!:+)!

= 2

!*$;!5''!+8)%)+<!!

Accuracy of Depth and Surface Normals.!G(,:;)!R!+-$7+!

.-)!);;$;!459+!*$;!.-)!;)8$@);)6!6)9.-!5%6!+:;*58)!%$;45'+!

$*!.-)!Ca:%%>E<!J)!;)8$@);)6!0_H!5%6!^&0_H!6)9.-!459+!

-5@)!;$$.&4)5%&+K:5;)6!);;$;+!/hk2\3!Q<SX84!5%6!P<SR84!

!!!!!!!!!!!2i!

!!!!!!!!!!!!!

^(+8$A$':+!

!!!!!!!!!!!!^;5,$%

/53!G);45.!*'$7!

/A3!H+5(!et al.!L"SO!

/83!^&0_H!%$;45'!@$':4)+!/&?!&!5%6!&8$49$%)%.+3!

/63!^&0_H!+:;*58)!

Figure

8. Surface reconstruction using captured data: SU has a spatial resolution of 64 × 64

pixels (1min exposure), and the remaining

scenes, 512 × 512

(180min exposure). We use

𝜆 = 2

and

for SU, Discobolus and Dragon, respectively. Insets in the left-

most column

show the scene objects. Fermat ﬂow [18] and the method of Tsai

et al. [19] only partially reconstruct the surfaces.

Methods

64 × 64

128 × 128

256 × 256

512 × 512

Albedo

FBP

0.6s

2.0s

8.3s

32.4s

Phasor Fields

0.9s

3.1s

12.6s

49.3s

𝑓–𝑘 migration

1.5s

6.6s

20.9s

72.6s

LCT

0.5s

2.4s

8.5s

30.4s

Normals

Fermat ﬂow

1.6s

5.6s

21.3s

86.5s

Heide et al.

>10h

N/A

Tsai et al.

N/A

D-LCT (Ours)

5.2s

21.9s

89.5s

370.0s

Table 1. Running times of various methods: Measured using an

8-core, 2.70GHz CPU for the Bunny, all at temporal resolutions of

512 pixels. *e methods of Heide et al. [13] and Tsai et al. [19] do

not scale to resolutions higher than 64 × 64.

5%6!.-)!4)5%!5A+$':.)!);;$;+!/kB\3!$*!"<ZX84!5%6!"<QS84!

;)+9)8.(@)'>?!58;$++!*$;),;$:%6!9(V)'+<!J)!+:;*58)!%$;45'+!

)+.(45.)6!:+(%,!.-)!0_H!5%6!.-)!^&0_H!-5@)!.-)!)%6&9$(%.!

hk2\!$*!T<S"84!5%6!T<Q]84?!kB\!$*!T<R"84!5%6!T<NZ84!

;)+9)8.(@)'><!#$.)?!.-)!0_H!6$)+!%$.?!A>!(.+)'*?!9;$6:8)!5%>!

+:;*58)!%$;45'+?!+$!7)!$A.5(%!.-)!%$;45'+!:+(%,!.-)!4).-$6!

LQPO!7(.-!.-)!0_H!6)9.-!5+!.-)!(%9:.!/7)!:+)!R!%)(,-A$;(%,!

9$(%.+!.$!9;$6:8)!.-)!$9.(4:4!;)+:'.+3<!=%!G(,:;)!X?!7)!9'$.!

.-)!(%I:)%8)!$*!;),:'5;([5.($%!95;54).);!!$%!.-)!6)9.-!5%6!

+:;*58)!%$;45'!);;$;+?!('':+.;5.(%,!.-5.!.-)!^&0_H!9);*$;4+!

+.5A')!$@);!5!7(6)!;5%,)!$*!.!J(+!85%!A)!:+)*:'!(%!9;58.(85'!

(45,(%,!+8)%5;($+!(%!7-(8-!.-)!+(,%5'&.$&%$(+)!;5.($!!$*!.-)!

859.:;)6!.;5%+()%.!65.5!(+!%$.!D%$7%!)V58.'><!

Surface Reconstruction with Captured Data.!H$!+-$7!.-)!

;$A:+.%)++!$*!.-)!^&0_H!5,5(%+.!6(j);)%.!.>9)+!$*!%$(+)!.-5.!

5;)!9;)+)%.!(%!;)5'!859.:;)!)%@(;$%4)%.+?!7)!9);*$;4!+:;*58)!

;)8$%+.;:8.($%!7(.-!.-)!2.5%*$;6!65.5+).<!G(,:;)!Z!+-$7+!.-)!

6(;)8.($%5'!5'A)6$!5%6!+:;*58)+!$*!;)8$@);)6!2i?!^(+8$A$':+!

5%6!.-)!^;5,$%!$AW)8.+<!Y)!;)8$%+.;:8.!.-)!+:;*58)+!A>!F;+.!

.-;)+-$'6(%,!.-)!%$;4!$*!6(;)8.($%5'!5'A)6$!@)8.$;+!.$!45+D!

$:.!.-)!A58D,;$:%6!9$(%.+?!.-)%!9);*$;4(%,!f$(++$%!+:;*58)!

;)8$%+.;:8.($%+!$%!.-)!;)45(%(%,!*$;),;$:%6!9$(%.+<!Y)!:+)!

= 2

?!2

!5%6!2

!*$;!.-)!.-;))!+8)%)+<!=%!.-)!2i!+8)%)?!.-)!

%$;45'!@$':4)+!;)@)5'!.-)!$;()%.5.($%!$*!.-)!')..);+!2!5%6!i!

/2!9$(%.+!.$!.-)!:99);&')*.?!i!9$(%.+!.$!.-)!:99);&;(,-.3<!J)!

')*.!95;.!$*!i!(+!95;.(5''>!$88':6)6?!+$!.-)!6(j);)%.!4).-$6+!

9;$6:8)!6(j);)%.!45V(45'!(%.)%+(.>!9;$W)8.($%+!5'$%,!.-)!&

5V(+<!G$;!.-)+)!;)8$%+.;:8.($%!.5+D+!7(.-!%$(+>!.;5%+()%.+?!.-)!

4).-$6!$*!H+5(!et al.!L"SO!5%6!G);45.!I$7!L"ZO!;)8$%+.;:8.!

$%'>!.-)!;$:,-!+-59)+!$*!.-)!$AW)8.+<!Y)!(%(.(5'([)!L"SO!:+(%,!

.-)!0_H?!A:.!$.-);!(%(.(5'([5.($%+!5;)!5'+$!9$++(A')<!Y)!:+)6!

.-)!--density!I5,!(%!.-)!f$(++$%!;)8$%+.;:8.($%!+$*.75;)!

LPZO!.$!5@$(6!.-)!*:+($%!$*!%)5;A>!+:;*58)!+),4)%.+<!

Computational Eﬃciency.!Y-(')!.-)!^&0_H!-5+!.-)!+54)!

8$49:.5.($%5'!8$49')V(.>!5+!.-)!0_H?!7)!9);*$;4!9×!4$;)!

8$49:.5.($%+!9);!@$V)'!6:)!.$!.-)!$:.);!_-$')+D>!*58.$;(%,!

;)K:(;)6<!J)!^&0_H!(+!1000×!*5+.);!8$495;)6!7(.-!+(4('5;!

4).-$6+!.-5.!5;)!8595A')!$*!;)8$@);(%,!.-)!+:;*58)!%$;45'+!

$*!$AW)8.+!7(.-!5!8$49')V!,)$4).;><!Y-(')!G);45.!I$7!L"ZO!

(+!4×!*5+.);!.-5%!$:;!599;$58-?!(.!(+!599'(85A')!4$+.'>!.$!.-)!

;)8$%+.;:8.($%!$*!+:;*58)+!$*!$AW)8.+!7(.-!+(49');!,)$4).;>!

+:8-!5+!5!A$7'!$;!5!+9-);)!/+))!2)8.($%!N!$*!.-)!+:99')4)%.!

*$;!.-)!;)8$%+.;:8.($%+3<!H5A')!"!9;$@(6)+!.-)!;:%%(%,!.(4)+!

$*!6(j);)%.!4).-$6+!$%!5%!Z&8$;)?!]<XTgU[!_fi<!

5. Discussion

! 1:;!7$;D!9;$9$+)+!5%!)b8()%.!4).-$6!.$!W$(%.'>!)+.(45.)!

.-)!5'A)6$!5%6!.-)!+:;*58)!%$;45'+!$*!#012!$AW)8.+!:+(%,!5!

6)8$%@$':.($%!599;$58-<!1:;!^(;)8.($%5'!0_H!-5+!.-)!+54)!

'$7!8$49:.5.($%5'!8$49')V(.>!5+!5'A)6$&$%'>!4).-$6+?!)<,<!

M!4(,;5.($%!5%6!.-)!0_H?!A:.!(+!8595A')!$*!;)8$%+.;:8.(%,!

-(,-&K:5'(.>!+:;*58)+<!!

Limitations.!1:;!*$;75;6!4$6)'!/Z3!5++:4)+!.-)!+8)%)!-5+!

4$+.'>!%$%&+9)8:'5;!+:;*58)+<!G$;.:%5.)'>?!$:;!')5+.&+K:5;)+!

(%@);+)!4).-$6!9;$@(6)+!+$4)!6),;))!$*!;$A:+.%)++!5,5(%+.!

+9)8:'5;(.()+!A>!.;)5.(%,!.-)4!5+!$:.'();+!/+))!)<,<!.-)!^;5,$%!

;)8$%+.;:8.($%?!G(,:;)!Z3<! 2(4('5;'>?!7)!.;)5.! $88':+($%+!(%!

.-)!+8)%)!5+!$:.'();+!.$!$:;!')5+.&+K:5;)+!*$;4:'5.($%<!i+(%,!

5%!

&A5+)6!65.5!F6)'(.>!.);4!(%+.)56!$*!$:;!

&A5+)6!$%)!

/"T3!8$:'6!*:;.-);!(49;$@)!.-)!;$A:+.%)++!$*!$:;!4).-$6?!5.!

.-)!8$+.!$*!(%8;)5+)6!8$49:.5.($%!.(4)+<!J)!

&A5+)6!65.5&

.);4!5'+$!)%*$;8)+!+95;+(.>?!7-(8-!45>!;)4$@)!.-)!%))6!*$;!

45+D(%,!$:.!A58D,;$:%6!9(V)'+<!

! 1:;!*$;75;6!4$6)'!5'+$!'(%)5;([)+!.-)!8$+(%)!*5''&$j!6:)!

.$!.-)!(%.);58.($%!A).7))%!+:;*58)!%$;45'+!5%6!.-)!.7$!'(,-.!

;5>+!/(%8(6)%.!5%6!;)I)8.)63<!1:;!'(%)5;([)6!*5''&$j!4$6)'!(+!

5%!:%6);&)+.(45.$;!$*!.-)!.;:)!*5''&$j?!5%6!+:;*58)!'$85.($%+!

.-5.!45D)!'5;,);!5%,')+!$%!5@);5,)!7(.-!.-)!@(+(A')!75''!5;)!

)+.(45.)6!.$!A)!5.!9$+(.($%+!8'$+);!.$!.-)!@(+(A')!75''?!7-);)!

.-)!*5''&$j!(+!(%6))6!')++<!J(+!85:+)+!;$:%6);!+:;*58)+!.$!A)!

)+.(45.)6!+'(,-.'>!I5..);!.-5%!.-)>!+-$:'6!A)!/+))!.-)!5;4+!$*!

.-)!^(+8$A$':+?!G(,:;)!Z3?!A:.!%$.!5+!I5.!5+!.-)!)+.(45.)+!$*!

.-)!0_H?!7-(8-!5++:4)+![);$!8$+(%)!*5''&$j<!J(+!(++:)!85%!

A)!$@);8$4)!A>!(.);5.(@)'>!;)7)(,-.(%,!.-)!F;+.!.);4!$*!/"T3!

:+(%,!.-)!;5.($!$*!.-)!.;:)!*5''&$j!.$!.-)!'(%)5;!$%)?!A5+)6!$%!!

.-)!%$;45'+!'5+.!)+.(45.)6l!+))!2)8.($%!N!$*!.-)!+:99')4)%.<!

Future Work.!H$!(49;$@)!;)8$%+.;:8.($%!.(4)+?!7)!9'5%!.$!

(49')4)%.!.-)!^&0_H!9;$8)6:;)!$%!5!gfi<!2(4('5;'>!.$!.-)!

$;(,(%5'!0_H?!.-)!^&0_H!(+!-(,-'>!95;5'')'([5A')!5%6!85%!A)!

+(,%(F85%.'>!588)');5.)6!:+(%,!5!gfi!(49')4)%.5.($%<!0(D)!

.-)!gfi!(49')4)%.5.($%!$*!.-)!0_H?!7)!)V9)8.!gfi&A5+)6!

^&0_H!.$!;)K:(;)!4(''(+)8$%6+!$*!9;$8)++(%,!.(4)!*$;!'$7);!

+95.(5'!;)+$':.($%+?!)<,<!32 ×32!$;!64 ×64!9(V)'+<!Y)! 9'5%!

5'+$!.$!8$%+(6);!

!$;!TV!;),:'5;([);+!.$!A)..);!9;)+);@)!.-)!

6(+8$%.(%:(.()+!(%!.-)!;)8$%+.;:8.)6!+:;*58)+<!

6. Conclusion

! #012!(45,(%,!599;$58-)+!-5@)!.>9(85''>!A))%!8'5++(F)6!

5+!;)8$@);(%,!)(.-);!.-)!5'A)6$!$;!.-)!+:;*58)!%$;45'+!$*!.-)!

-(66)%!$AW)8.+<!=%!.-(+!7$;D?!7)!+-$7)6!.-5.!(.!(+!9$++(A')!.$!

;)8$@);!A$.-!K:5%.(.()+!W$(%.'><!=%!8'$+(%,?!;)8$%+.;:8.($%!$*!

+:;*58)+!$*!-(66)%!N^!$AW)8.+! 85%! A)! ;),5;6)6!5+!.-)!%)V.!

*;$%.();!*$;!#012!(45,(%,!A)85:+)!(.!5''$7+!:+!.$!;)9;)+)%.!

A)..);!.-)!N^!)%@(;$%4)%.!7)!:'.(45.)'>!'(@)!(%<!Y)!A)'()@)!

.-)!^&0_H!(+!5!A(,!+.)9!.$75;6+!9:+-(%,!A)>$%6!@$':4).;(8!

5'A)6$!599;$58-)+?!9;$@(6(%,!5!9;58.(85'!75>!.$!)+.(45.)!.-)!

-(66)%!+:;*58)!%$;45'+!%))6)6!*$;!+:;*58)!;)8$%+.;:8.($%<!

Acknowledgements.!Y)! .-5%D! k<! n<! g5'(%6$! *$;! -)'9!7(.-!

65.5+).!LQNO!5%6!=<!gD($:')D5+!*$;!.-)!8$6)!$*!L"Z?!"SO<!^<0<!

75+!+:99$;.)6!A>!5!2.5%*$;6!g;56:5.)!G)''$7+-(9<!g<Y<!75+!

+:99$;.)6! A>! 5%! #2G! _Bh\\h! B75;6! /==2! "QQNNNN3?! 5!

2'$5%!G)''$7+-(9?!A>!.-)!dBi2H!1b8)!$*!29$%+$;)6!h)&

+)5;8-!.-;$:,-!.-)!e(+:5'!_$49:.(%,!_)%.);!__G!,;5%.?!.-)!

^BhfB!h\e\B0!9;$,;54?!5%6!5!f\_B2\!A>!.-)!Bh0<! !

References

[1] Jacopo Bertolotti, Elbert G. van Putten, Christian Blum, Ad

Lagendijk, Willem L. Vos, and Allard P. Mosk. Non-invasive

imaging through opaque scattering layers. Nature,

491(7423):232–234, 2012.

[2] Ori Katz, Eran Small, and Yaron Silberberg. Looking around

corners and through thin turbid layers in real time with

scattered incoherent light. Nat. Photonics, 6(8):549–553,

2012.

[3] Brandon M. Smith, Matthew O’Toole, and Mohit Gupta.

Tracking multiple objects outside the line of sight using

speckle imaging. In CVPR, 2018.

[4] Ahmed Kirmani, Tyler Hutchison, James Davis, and Ramesh

Raskar. Looking around the corner using transient imaging.

In ICCV, 2009.

[5] Christos *rampoulidis et al. Exploiting occlusion in non-

line-of-sight active imaging. IEEE Trans. Comput. Imaging,

4(3):419–431, 2018.

[6] Katherine L. Bouman et al. Turning corners into cameras:

Principles and methods. In ICCV, 2017.

[7] Manel Baradad et al. Inferring light ﬁelds from shadows. In

CVPR, 2018.

[8] Mahed Batarseh, Sergey Sukhov, Zhean Shen, Heath Gemar,

Roxana Rezvani, and Aristide Dogariu. Passive sensing

around the corner using spatial coherence. Nat. Commun.,

9(1):1–6, 2018.

[9] Charles Saunders, John Murray-Bruce, and Vivek K. Goyal.

Computational periscopy with an ordinary digital camera.

Nature, 565(7740):472–475, 2019.

[10] David B. Lindell, Gordon Wetzstein, and Vladlen Koltun.

Acoustic non-line-of-sight imaging. In CVPR, 2019.

[11] Andreas Velten, *omas Willwacher, Otkrist Gupta, Ashok

Ve er ar a gh av a n, M oun gi G . B a we n di , a n d R a me s h R a ska r.

Recovering three-dimensional shape around a corner using

ultrafast time-of-ﬂight imaging. Nat. Commun., 3:745, 2012.

[12] Matthew O’Toole, David B. Lindell, and Gordon Wetzstein.

Confocal non-line-of-sight imaging based on the light-cone

transform. Nature, 555(7696):338–341, 2018.

[13] Felix Heide, Matthew O’Toole, Kai Zang, David B. Lindell,

Steven Diamond, and Gordon Wetzstein. Non-line-of-sight

imaging with partial occluders and surface normals. ACM

Trans Graph, 38(3):22:1–22:10, 2019.

[14] David B. Lindell, Gordon Wetzstein, and Matthew O’Toole.

Wave-based non-line-of-sight imaging using fast f-k

migration. ACM Trans. Graph. TOG, 38(4):116, 2019.

[15] Christopher A. Metzler, David B. Lindell, and Gordon

Wetzstein. Keyhole imaging: non-line-of-sight imaging and

tracking of moving objects along a single optical path at long

standoﬀ distances. ArXiv191206727 Cs Eess, 2019.

[16] Marco La Manna, Fiona Kine, Eric Breitbach, Jonathan

Jackson, Talha Sultan, and Andreas Velten. Error

backprojection algorithms for non-line-of-sight imaging.

IEEE Trans. Pattern Anal. Mach. Intell., 41(7):1615–1626,

2019.

[17] Chia-Yin Tsai, Kiriakos N. Kutulakos, Srinivasa G.

Narasimhan, and Ashwin C. Sankaranarayanan. *e

geometry of ﬁrst-returning photons for non-line-of-sight

imaging. In CVPR, 2017.

[18] Shumian Xin, Sotiris Nousias, Kiriakos N. Kutulakos, Aswin

C. Sankaranarayanan, Srinivasa G. Narasimhan, and Ioannis

Gkioulekas. A theory of Fermat paths for non-line-of-sight

shape reconstruction. In CVPR, 2019.

[19] Chia-Yin Tsai, Aswin C. Sankaranarayanan, and Ioannis

Gkioulekas. Beyond volumetric albedo—a surface

optimization framework for non-line-of-sight imaging. In

CVPR, 2019.

[20] Felix Heide, Lei Xiao, Wolfgang Heidrich, and Matthias B.

Hullin. Diﬀuse mirrors: 3D reconstruction from diﬀuse

indirect illumination using inexpensive time-of-ﬂight

sensors. In CVPR, 2014.

[21] Achuta Kadambi, Hang Zhao, Boxin Shi, and Ramesh

Raskar. Occluded imaging with time-of-ﬂight sensors. ACM

Trans Graph, 35(2):15:1–15:12, 2016.

[22] Jonathan Klein, Christoph Peters, Jaime Martín, Martin

Laurenzis, and Matthias B. Hullin. Tracking objects outside

the line of sight using 2D intensity images. Sci. Rep.,

6:32491, 2016.

[23] Mauro Buttafava, Jessica Zeman, Alberto Tosi, Kevin

Eliceiri, and Andreas Velten. Non-line-of-sight imaging using

a time-gated single photon avalanche diode. Opt. Express,

23(16):20997–21011, 2015.

[24] Genevieve Gariepy et al. Single-photon sensitive light-in-

ﬁght imaging. Nat. Commun., 6:6021, 2015.

[25] Feihu Xu et al. Revealing hidden scenes by photon-eﬃcient

occlusion-based opportunistic active imaging. Opt. Express,

26(8):9945–9962, 2018.

[26] Victor Arellano, Diego Gutierrez, and Adrian Jarabo. Fast

back-projection for non-line of sight reconstruction. Opt.

Express, 25(10):11574–11583, 2017.

[27] Matthew O’Toole, Felix Heide, David B. Lindell, Kai Zang,

Steven Diamond, and Gordon Wetzstein. Reconstructing

Transient Images From Single-Photon Sensors. In CVPR,

2017.

[28] Matthew O’Toole, David B. Lindell, and Gordon Wetzstein.

Real-time Non-line-of-sight Imaging. In ACM SIGGRAPH

2018 Emerging Technologies, 2018.

[29] Susan Chan, Ryan E. Warburton, Genevieve Gariepy,

Jonathan Leach, and Daniele Faccio. Non-line-of-sight

tracking of people at long range. Opt. Express, 25(9):10109–

10117, 2017.

[30] Yoann A lt ma nn , St ep hen M cL aughlin, Miles J . Pa dg et t,

Vivek K. Goyal, Alfred O. Hero, and Daniele Faccio.

Quantum-inspired computational imaging. Science,

361(6403):eaat2298, 2018.

[31] Otkrist Gupta, *omas Willwacher, Andreas Velten, Ashok

Ve er ar a gh av a n, a n d R ame sh R as kar. R e co n st ruc ti on o f

hidden 3D shapes using diﬀuse reﬂections. Opt. Express,

20(17):19096–19108, 2012.

[32] Byeongjoo Ahn, Akshat Dave, Ashok Veeraraghavan, Ioannis

Gkioulekas, and Aswin C. Sankaranarayanan. Convolutional

Approximations to the General Non-Line-of-Sight Imaging

Operator. In ICCV, 2019.

[33] Xiaochun Liu et al. Non-line-of-sight imaging using phasor-

ﬁeld virtual wave optics. Nature, :1–4, 2019.

[34] Matthew M. Loper and Michael J. Black. OpenDR: An

"T!

Approximate Diﬀerentiable Renderer. In ECCV, 2014.

[35] Pramook Khungurn, Daniel Schroeder, Shuang Zhao, Kavita

Bala, and Steve Marschner. Matching real fabrics with micro-

appearance models. ACM Trans Graph, 35(1):1:1–1:26,

2015.

[36] Ioannis Gkioulekas, Anat Levin, and Todd Zickler. An

evaluation of computational imaging techniques for

heterogeneous inverse scattering. In ECCV, 2016.

[37] Shuang Zhao, Lifan Wu, Frédo Durand, and Ravi

Ramamoorthi. Downsampling scattering parameters for

rendering anisotropic media. ACM Trans Graph,

35(6):166:1–166:11, 2016.

[38] Adam Geva, Yoav Y. Schechner, Yonatan Chernyak, and

Rajiv Gupta. X-ray computed tomography through scatter. In

ECCV, 2018.

[39] Stephen Robert Marschner. Inverse Rendering for Computer

Graphics, PhD *esis, Cornell University, 1998.

[40] Gustavo Patow and Xavier Pueyo. A survey of inverse

surface design from light transport behavior speciﬁcation.

Comput. Graph. Forum, 22:663–687, 2003.

[41] Ben Appleton and Hugues Talbot. Globally minimal surfaces

by continuous maximal ﬂows. IEEE Trans. Pattern Anal.

Mach. Intell., 28(1):106–118, 2006.

[42] Olivier Faugeras and Renaud Keriven. Complete dense

stereovision using level set methods. In ECCV, 1998.

[43] Amaël Delaunoy and Emmanuel Prados. Gradient ﬂows for

optimizing triangular mesh-based surfaces: applications to 3d

reconstruction problems dealing with visibility. Int. J.

Comput. Vis., 95(2):100–123, 2011.

[44] Mathieu Desbrun, Mark Meyer, Peter Schröder, and Alan H.

Barr. Implicit fairing of irregular meshes using diﬀusion and

curvature ﬂow. In SIGGRAPH, 1999.

[45] Mark Meyer, Mathieu Desbrun, Peter Schröder, and Alan H.

Barr. Discrete Diﬀerential-Geometry Operators for

Triangulated 2-Manifolds. In Visualization and Mathematics

III, 2003.

[46] O. Sorkine, D. Cohen-Or, Y. Lipman, M. Alexa, C. Rössl, and

H.-P. S ei de l . L a pl a cia n S u rf a ce E di t ing . I n Proceedings of the

2004 Eurographics/ACM SIGGRAPH Symposium on

Geometry Processing, 2004.

[47] Ilya Eckstein, Jean-Philippe Pons, Yiying Tong, C.-C. Jay

Kuo, and Mathieu Desbrun. Generalized surface ﬂows for

mesh processing. In Proceedings of the Fifth Eurographics

Symposium on Geometry Processing, 2007.

[48] Michael Kazhdan, Matthew Bolitho, and Hugues Hoppe.

Poisson surface reconstruction. In Proceedings of the Fourth

Eurographics Symposium on Geometry Processing, 2006.

[49] Per Christian Hansen. Discrete Inverse Problems: Insight and

Algorithms. SIAM, 2010.

[50] Di Wu et al. Frequency Analysis of Transient Light Transport

with Applications in Bare Sensor Imaging. In Computer

Vision – ECCV 2012, 2012.

[51] Lloyd N. Trefethen and David Bau, III. Numerical Linear

Algebra. SIAM, 1997.

[52] Michael Kazhdan and Hugues Hoppe. Screened Poisson

Surface Reconstruction. ACM Trans Graph, 32(3):29:1–

29:13, 2013.

[53] Graphics and Imaging Lab. Z-NLOS Dataset, Graphics and

Imaging Lab. [Online]. Available:

http://graphics.unizar.es/nlos_dataset. [Accessed: 13-Sep-

2019].

[54] Hugues Hoppe, Tony DeRose, Tom Duchamp, John

McDonald, and Werner Stuetzle. Surface Reconstruction

from Unorganized Points. In SIGGRAPH, 1992.