IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. 0, NO. 0, DECEMBER 2023 1
!
!
• S I Young, A V Dalca, and J E Iglesias are with the Martinos Center for
Biomedical Imaging, Harvard Medical School, Boston, MA, USA, and the
Computer Science and Artificial Intelligence Lab, MIT, Cambridge, MA,
USA. E-mail: siyoung@mit.edu, adalca@mit.edu, jei@mit.edu.
• E Ferrante was with the Martinos Center for Biomedical Imaging, Harvard
Medical School, Boston, MA, USA. He is now with CONICET, Santa Fe,
Argentina. E-mail: eferrante@sinc.unl.edu.ar.
• P Golland is with the Computer Science and Artificial Intelligence Lab
(CSAIL), MIT, Cambridge, MA, USA. E-mail: polina@csail.mit.edu.
• C A Metzler is with the Department of Computer Science, University of
Maryland, College Park, MD, USA. E-mail: metzler@umd.edu.
• B Fischl is with the Martinos Center for Biomedical Imaging, Harvard
Medical School, Boston, MA, USA. E-mail: bfischl@mgh.harvard.edu.
Manuscript received 14 April 2023. (Corresponding author: Sean I. Young.)
For information on obtaining reprints of this article, please send e-mail to:
reprints@ieee.org, and reference the Digital Object Identifier below.
Digital Object Identifier no. 10.1109/TPAMI.2023.
Supervision by Denoising
Sean I. Young , Adrian V. Dalca , Enzo Ferrante , Polina Golland ,
Christopher A. Metzler , Bruce Fischl , and Juan Eugenio Iglesias
Abstract—Learning-based image reconstruction models, such as those based on the U-Net, require a large set of labeled images if
good generalization is to be guaranteed. In some imaging domains, however, labeled data with pixel- or voxel-level label accuracy are
scarce due to the cost of acquiring them. This problem is exacerbated further in domains like medical imaging, where there is no
single ground truth label, resulting in large amounts of repeat variability in the labels. Therefore, training reconstruction networks to
generalize better by learning from both labeled and unlabeled examples (called semi-supervised learning) is problem of practical and
theoretical interest. However, traditional semi-supervised learning methods for image reconstruction often necessitate handcrafting a
differentiable regularizer specific to some given imaging problem, which can be extremely time-consuming. In this work, we propose
“supervision by denoising” (SUD), a framework to supervise reconstruction models using their own denoised output as labels. SUD
unifies stochastic averaging and spatial denoising techniques under a spatio-temporal denoising framework and alternates denoising
and model weight update steps in an optimization framework for semi-supervision. As example applications, we apply SUD to two
problems from biomedical imaging—anatomical brain reconstruction (3D) and cortical parcellation (2D)—to demonstrate a significant
improvement in reconstruction over supervised-only and ensembling baselines. Our code available at https://github.com/seannz/sud.
Index Terms—semi-supervised learning, visual reconstruction, denoising, fully convolutional networks, proximal methods.
✦
1 INTRODUCTION
EARNING"BASED IMAGE RECONSTRUCTION MODELS!#$%&!
'&()*&!+,-+./&,.$'0&!1)2!.)0%+,3!$!%$.4!$22$5!)1!+,%&2.&!
/2)'0&*.!+,!+*$3+,3!$,-!%+.+),6!12)*!0)7"0&%&0!),&.!7#+(#!
+,%)0%&!+*$3&!-&,)+.+,3!$,-!2&(),.428(4+),!9:;6!9<;!4)!#+3#"
0&%&0!),&.!4#$4!,&(&..+4$4&!/+=&0"0&%&0!.&*$,4+(!.&3*&,4$4+),!
9>;6!9?;@!A#+0&!2&(),.428(4+),!*)-&0.!.8(#!$.!4#).&!'$.&-!),!
B"C&4!9D;!45/+($005!)84/&21)2*!#$,-(2$14&-!*)-&0.!+,!*$,5!
+*$3+,3!/2)'0&*.6!4#&5!($,!+,%)0%&!*+00+),.!)1!/$2$*&4&2.!
$,-6!$.!$!2&.8046!#$%&!$!4&,-&,(5!4)!)%&2E4!42$+,+,3!-$4$!$,-!
3&,&2$0+F&!/))205!4)!/2&%+)8.05!8,.&&,!+*$3&.!$4!4&.4!4+*&G
$!/2)'0&*!$0.)!&=$(&2'$4&-!'5!-+.42+'84+),!.#+14!9H;@!I8(#!$,!
+..8&!+.!&./&(+$005!,)4+(&$'0&!+,!4#&!2&(),.428(4+),!)1!*&-+($0!
+*$3&.6!7#&2&!0$'&0&-!42$+,+,3!+*$3&.!$2&!.($2(&6!/2)-8(&-!
8,-&2!),&!)1!*$,5!-+J&2&,4!+*$3+,3!K(),42$.4L!.&M+,3.!$,-!
(),4$+,!0$'&0!,)+.&!-8&!4)!4#&!%$2+$'+0+45!+,!4#&!0$'&0+,3!&%&,!
$(2)..!-)*$+,!&=/&24.!9N;@!O#&2&1)2&6!42$+,+,3!2&(),.428(4+),!
,&47)2P.!4)!3&,&2$0+F&!'&M&2!'5!0&$2,+,3!12)*!')4#!0$'&0&-!
$,-!8,0$'&0&-!&=$*/0&.!K($00&-!.&*+".8/&2%+.&-!0&$2,+,3L!+.!
$!/2)'0&*!)1!/2$(4+($0!$,-!4#&)2&4+($0!+,4&2&.4!1)2!+*/2)%+,3!
+*$3&!2&(),.428(4+),!$,-!)4#&2!-)7,.42&$*!4$.P.!9Q;6!9R;@!
!! S2)$-05!./&$P+,36!7&!($,!+*/2)%&!4#&!3&,&2$0+F$4+),!)1!$!
2&(),.428(4+),!,&47)2P!'5!$04&2+,3!$./&(4.!)1!+4.!$2(#+4&(482&!
9:T;6!9::;6!42$+,+,3!-$4$!9:<;6!9:>;6!42$+,+,3!)'U&(4+%&.!9:?;6!9:D;!
)2!)/4+*+F$4+),!.42$4&3+&.!8.&-!1)2!42$+,+,3!9:H;6!9:N;@!V84!)1!
4#&.&6!$--+,3!$!./$4+$0!2&380$2+F&2!+,!4#&!42$+,+,3!)'U&(4+%&!)1!
4#&!2&(),.428(4+),!,&47)2P!#$.!/2)%&-!&=42&*&05!8.&180!1)2!
+*/).+,3!4)/)0)3+($0!)2!./$4+$0!/2+)2.!),!4#&!2&(),.428(4+),!
9:Q;6!9:R;!$,-!.&*+".8/&2%+.&-!0&$2,+,3!KIIWL@!IIW!*&4#)-.!
'$.&-!),!2&380$2+F$4+),!.8J&2!,&+4#&2!12)*!0+*+4&-!-+%&2.+45!
)1!$83*&,4&-!-$4$!,)2!-)*$+,!3$/.!2&.804+,3!12)*!42$+,+,3!
),!.5,4#&4+(!-$4$!9<T;6!9<:;@!X)2&)%&26!4#&5!&=/0+(+405!./&(+15!
4#&!./$4+$0!2&380$2+45!,&&-&-!1)2!$!/$24+(80$2!4$.P6!+,!(),42$.4!
4)!$83*&,4$4+),!9<<;!$,-!.&01".8/&2%+.+),!9<>;!4&(#,+Y8&.@!
! Z&./+4&!4#&!'&,&E4.6!+,()2/)2$4+,3!./$4+$0!2&380$2+F&2.!4)!
1$(+0+4$4&!.&*+".8/&2%+.+),!#$.!'&&,!-+[(804!-8&!4)!4#&!,&&-!
4)!#$,-(2$14!$!-+J&2&,4+$'0&!2&380$2+F&2!1)2!$!3+%&,!/2)'0&*!
9:Q;6!9<?;6!7#+(#!($,!'&!&=42&*&05!4+*&"(),.8*+,3@!\0$..+($0!
-+J&2&,4+$'0&!2&380$2+F&2.!9<D;6!9<H;6!'5!(),42$.46!$2&!$3,).4+(!
4)!4#&!4$.P!$,-!/2)-8(&!.8'")/4+*$0!)84()*&.@!V,&!.)084+),!
4)!)%&2()*&!4#&!7&$P,&..&.!)1!(),%&,4+),$0!2&380$2+F&2.!+.!
4)!(),.428(4!4#&*!12)*!-&,)+.&2.6!7#+(#!7)80-!$00)7!8.!4)!
(),%&24!-)*$+,"./&(+E(6!0&$2,&-!)2!&%&,!,),"-+J&2&,4+$'0&!
-&,)+.&2.!4)!2&380$2+F&2.@!O#+.!/$2$-+3*!+.!$0.)!1)00)7&-!'5!
12$*&7)2P.!.8(#!$.!]2&380$2+F$4+),!'5!-&,)+.+,3^!9<N;6!9<Q;!
$,-!]/083"$,-"/0$5!/2+)2.^!9<R;6!9>T;6!7#+(#!$2&!$//0+&-!4)!
.)0%&!%$2+)8.!+,%&2.&!/2)'0&*.!+,!+*$3+,3!9>:;6!9><;@!_+%&,!
4#&!18,-$*&,4$0!2)0&!)1!-&,)+.+,3!+,!+*$3+,3!4$.P.6!+4!+.!,)4!
.)!.82/2+.+,3!4#$4!.8/&2%+.+,3!2&(),.428(4+),!,&47)2P.!($,!
'&,&E4!12)*!-&,)+.+,3!$.!7&00@!`,!1$(46!.4)(#$.4+(!$%&2$3+,3!
*&4#)-.!8.&-!+,!0&$2,&-!+*$3&!2&(),.428(4+),!9>>;6!9>?;!($,!
'&!4#)83#4!)1!$.!]2&380$2+F$4+),!'5!4&*/)2$0!-&,)+.+,3^!$,-!
4#+.!.833&.4.!2&380$2+F$4+),!'5!-&,)+.+,3!+,!')4#!./$(&!$,-!
4+*&!K$(2)..!*)-&0.L!($,!0&$-!4)!'&M&2!.&*+".8/&2%+.+),@!
! `,!4#+.!7)2P6!7&!/2)/).&!].8/&2%+.+),!'5!-&,)+.+,3^6!)2!
IBZ6!$!12$*&7)2P!4#$4!$00)7.!.8/&2%+.+),!)1!2&(),.428(4+),!
*)-&0.!8.+,3!4#&+2!-&,)+.&-!)84/84!$.!4$23&4.@!IBZ!8,+E&.!
L
2 IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. 0, NO. 0, DECEMBER 2023
./$4+$0!E04&2+,3!9<N;6!9>T;!$,-!.4)(#$.4+(!$%&2$3+,3!9<>;6!9>D;!
4&(#,+Y8&.!8,-&2!),&!./$4+)"4&*/)2$0!-&,)+.+,3!12$*&7)2P!
$,-!$04&2,$4&.!-&,)+.+,3!$,-!,&47)2P!7&+3#4!8/-$4&!+,!$,!
)/4+*+F$4+),! 12$*&7)2P! 1)2! /2+,(+/0&-! .8/&2%+.+),@! A#+0&!
IBZ!$(()**)-$4&.!$,5!0&$2,&-!$,-!&%&,!,),"-+J&2&,4+$'0&!
2&(),.428(4+),!-&,)+.&26! .+*/05! U8.4! 42$+,+,3!),&!$00&%+$4&.!
4#&!,&&-!4)!E,-!$!-&,)+.&2!.8+4&-!1)2!$!3+%&,!4$.P@!A&!0&$2,!
2&(),.428(4+),!-&,)+.&2.!'5!42$+,+,3!4#&*!.&01".8/&2%+.&-!),!
2&(),.428(4+),!&=$*/0&.!K.&3*&,4$4+),.!)2!+*$3&.L!$02&$-5!
$%$+0$'0&@!a+3@!:!.8**$2+F&.!4#&!IBZ!12$*&7)2P!7+4#!-$4$!
$83*&,4$4+),!$,-!-2)/)846!,)4+,3!4#$4!4#&!-&,)+.+,3!+.!')4#!
./$4+$0!$,-!4&*/)2$0!K$(2)..!42$+,+,3!.4&/.L!$.!.#)7,!'5!K$L!
$,-!K'L@!`,!a+3@!<6!7&!.#)7!),&!$//0+($4+),!)1!IBZ!K()24+($0!
/$2(&00$4+),L!$,-!-&*),.42$4&!4#$4!+4!.+3,+E($,405!+*/2)%&.!
.8/&2%+.+),!)1!$!7&00"/&21)2*+,3!'$.&0+,&!,&47)2P!9>H;@!A&!
,)4&!4#$4!-&,)+.+,3!2&(),.428(4+),.!/).4!#)(!/2)-8(&.!/))2!
)84()*&.!7#&2&$.!,$+%&05!$04&2,$4+,3!32$-+&,4!-&.(&,4!$,-!
-&,)+.+,3!2&,-&2.!42$+,+,3!-+%&23&,4@!
! V82!(),42+'84+),.!$2&!$.!1)00)7.@!a+2.46!7&!-&2+%&!IBZ!'5!
2&+,4&2/2&4+,3!$!.&*+".8/&2%+.&-!+*$3&!2&(),.428(4+),!4$.P!
$.!$,!)/4+*+F$4+),!/2)'0&*!4#$4!($,!'&!.)0%&-!'5!$04&2,$4+,3!
-&,)+.+,3!$,-!.4)(#$.4+(32$-+&,4!-&.(&,4!.4&/.@!!I&(),-6!7&!
$//05!IBZ!),!47)!4$.P.!12)*!*&-+($0!+*$3+,3!K<Z!()24+($0!
/$2(&00$4+),!$,-!>Z!'2$+,!2&(),.428(4+),L!$,-!-&*),.42$4&!$!
.8'.4$,4+$0!+*/2)%&*&,4!+,!4#&!2&(),.428(4+),!Y8$0+45!)%&2!
.8/&2%+.&-!$,-!.4)(#$.4+(!$%&2$3+,3!'$.&0+,&.6!&./&(+$005!+,!
4#&!'$2&05!.8/&2%+.&-!K7+4#!1&7!0$'&0&-!&=$*/0&.L!.(&,$2+)@!!
2 RELATED WORK
b&380$2+F$4+),!'5!-&,)+.+,3!KbcZL!9<N;!+.!$!(0$..+($0!+*$3&!
2&(),.428(4+),!*&4#)-!4#$4!8.&.!-&,)+.&2.!4)!2&380$2+F&!$,!
+*$3&@!S5!(),42$.46!)82!7)2P!+.!$!.&*+".8/&2%+.&-!0&$2,+,3!
*&4#)-!4#$4!8.&.!-&,)+.&2.!4)!2&380$2+F&!0&$2,+,3@!A&!'2+&d5!!
2&%+&7!bcZ!9<N;!$,-!,),"-&,)+.+,3"'$.&-!.&*+".8/&2%+.+),!
4&(#,+Y8&.!1)2!+*$3&!2&(),.428(4+),!$.!$!0&$-8/!4)!IBZ@!
2.1 Regularization by Denoising
X$,5!+,%&2.&!/2)'0&*.!+,!+*$3+,3!$,-!%+.+),!$2&!+,#&2&,405!
+00"(),-+4+),&-!)2!&%&,!+00"/).&-!$,-!2&Y8+2&!$,!$..8*/4+),!
)1!]2&380$2+45^!4)!2&()%&2!$!/0$8.+'0&!.)084+),@!b&380$2+F+,3!
+,%&2.&!/2)'0&*.!7+4#!42$-+4+),$0!4&(#,+Y8&.!)14&,!+,%)0%&.!
$!-)*$+,"$3,).4+(!2&380$2+45!4&2*!.8(#!$.!4#&!4)4$0!%$2+$4+),!
.&*+",)2*!$,-!+4.!3&,&2$0+F$4+),.!9<D;6!9<H;!)2!32$/#"'$.&-!
K,),"0)($0L!2&380$2+F&2.!9>N;6!9>Q;6!')4#!)1!7#+(#!$2&!8.8$005!
(#).&,!)84!)1!*)-&0+,3!)2!)/4+*+F$4+),!(),%&,+&,(&6!2$4#&2!
4#$,!4)!2&d&(4!$!428&!,)4+),!)1!2&380$2+45!+,!$!(#).&,!-)*$+,@!
! e*),3!4#&.&!2&380$2+F&2.6!4#&!'+0$4&2$0!2&380$2+F&2!9>R;!+.!
.#)7,!4)!'&!&J&(4+%&!1)2!.)0%+,3!()*/84&2!%+.+),!/2)'0&*.!
.8(#!$.!+*$3&!2&1)(8.!9?T;6!)/4+($0!d)7!&.4+*$4+),!9>N;6!9?:;!
$,-!+*$3&!.8/&22&.)084+),!9?<;@!S+0$4&2$0!2&380$2+F$4+),!9>R;!
&..&,4+$005!&=/2&..&.!2&380$2+45!)1!$,!+*$3&!$.!+4.!
!-+.4$,(&!
12)*!$!'+0$4&2$0"E04&2&-!%&2.+),!)1!4#&!.$*&!+*$3&!4)!-&E,&!
4#&!/2+)2!+,!$,!$,+.)42)/+(!-+J8.+),!./$(&!9?>;@!V4#&2!(0$..+(!
&-3&"/2&.&2%+,3!E04&2.!4#$4!($,!'&!/0833&-!+,4)!4#&!'+0$4&2$0!
.)0%&2!+,(08-&!,),"0)($0!*&$,.!-&,)+.+,3!9??;!$,-!42+0$4&2$0!
9?D;!E04&2.6!')4#!)1!7#+(#!($,!'&!+*/0&*&,4&-!&[(+&,405@!
! O#&!]2&380$2+F$4+),!'5!-&,)+.+,3^!KbcZL!12$*&7)2P!9<N;!
($,!'&!.&&,!$.!&=4&,-+,3!'+0$4&2$0!2&380$2+F$4+),6!&=/2&..+,3!
2&380$2+F&2.!+,!4&2*.!)1!$2'+42$25!+*$3&!-&,)+.&2.@!b&&#)2.4!
$,-!I(#,+4&2!9<Q;!(0$2+15!4#&!(),-+4+),.!),!.8(#!-&,)+.&2.!4)!
38$2$,4&&!(),%&23&,(&!7#+0&!9?H;!-&*),.42$4&!4#&!84+0+45!)1!
0&$2,&-!-&,)+.&2.!1)2!2&380$2+F$4+),@!\0).&05!2&0$4&-!4)!bcZ!
+.!4#&!)0-&2!]/083"$,-"/0$5!/2+)2.^!Kf>L!*&4#)-!9<R;6!7#+(#!
8.&.!$!-&,)+.+,3!.4&/!+,!/0$(&!)1!4#&!-&,)+.+,3!)/4+*F$4+),!
+,!bcZ@!A#+0&!IBZ!$0.)!8.&.!$,!)/4+*+F$4+),"12&&!-&,)+.+,3!
.42$4&35!4)!$((&0&2$4&!42$+,+,36!4#+.!.42$4&35!+.!-&2+%&-!12)*!
./&(42$0!E04&2!4#&)25!9?N;!4)!&,.82&!4#&!-&,)+.+,3!.4&/!*)2&!
1$+4#18005!$//2)=+*$4&.!4#+.!)/4+*+F$4+),@!A&! .42&..!,&+4#&2!
bcZ!,)2!f>!$--2&..&.!.8/&2%+.+),!$.!+,!)82!7)2P@!
! b&(&,405!/2)/).&-!+*$3&!-&,)+.+,3!4&(#,+Y8&.!45/+($005!
-+./&,.&!7+4#!*)-&0.!)1!-+J8.+),!9?>;6!$,-!0&$2,!12)*!-$4$!
$!(),%)084+),$0!,&82$0!,&47)2P!7#+(#!*$/!,)+.5!+*$3&.!4)!
,)+.&"12&&!),&.!-+2&(405@!Z&,)+.+,3!'$.&-!),!B"C&4!#$.!'&&,!
$//0+&-!4)!$!7+-&!%$2+&45!)1!+*$3&!45/&.6!+,(08-+,3!*&-+($0!
9?Q;6!9?R;6!.$4&00+4&!9DT;!$,-!,$482$0!9D:;!),&.@!I8(#!d&=+'+0+45!
)1!B"C&4!#$.!$0.)!'&&,!$//0+&-!4)!-&,)+.+,3!2&(),.428(4+),.!
9D<;6!9D>;6!9D?;6!)'%+$4+,3!4#&!,&&-!4)!-&,)+.&!8.+,3!-+.(2&4&!
E04&2.!9DD;6!9DH;6!7#+(#!$2&!()*/84$4+),$005!&=/&,.+%&@!IBZ!
0&%&2$3&.!.8(#!-&,)+.&2.!4)!.8/&2%+.&!+*$3&!2&(),.428(4+),@!
2.2 Semi-supervised Learning
`,!4#&!*$(#+,&!0&$2,+,3!0+4&2$482&6!.&*+".8/&2%+.&-!0&$2,+,3!
KIIWL!+.!$!1$*+05!)1!$//2)$(#&.!4#$4!0&$2,!8.+,3!')4#!0$'&0&-!
$,-!8,0$'&0&-!-$4$6!$00)7+,3!0&$2,+,3!$03)2+4#*.!4)!&=/0)+4!
$'8,-$,4!8,0$'&0&-!-$4$!4)!&,#$,(&!$!.*$00&2!.&4!)1!0$'&0&-!
&=$*/0&.@!\822&,4056!$!0$23&!,8*'&2!)1!7&00"/&21)2*+,3!IIW!
*&4#)-.!+,%)0%&!$--+,3!$!2&380$2+45!0)..!),!8,0$'&0&-!-$4$!
4)!4#&!42$+,+,3!)'U&(4+%&6!0&$%+,3!$00!)4#&2!$./&(4.!)1!42$+,+,3!
8,(#$,3&-!9DN;@!S2)$-05!./&$P+,36!7&!($,!(0$..+15!.8(#!IIW!
*&4#)-.!$.!&+4#&2!.4)(#$.4+(!K8.&-!+,!')4#!(0$..+E($4+),!$,-!
2&(),.428(4+),!4$.P.L!)2!./$4+$0!K8.&-!),05!+,!2&(),.428(4+),L!
.+,(&!(0$..+E($4+),!)84/84!-)!,)4!#$%&!./$4+$0!-+*&,.+),.@!
! I4)(#$.4+(!&,.&*'0+,3!4&(#,+Y8&.!9<>;6!9>>;6!9>?;6!9>D;!$--!
$!2&380$2+45!4&2*!4)!4#&!42$+,+,3!)'U&(4+%&!4)!&,.82&!4#$4!4#&!
,&47)2P!)84/84!1)2!$!3+%&,!+*$3&!-)&.!,)4!-&%+$4&!4))!*8(#!
12)*!+4.!/2&%+)8.!%$08&.@!B.&-!1)2!(0$..+E($4+),!4$.P.6!4#&!g!
*)-&0!9DQ;!+.!$!.+*/0&!4&(#,+Y8&!4#$4!/&,$0+F&.!-&%+$4+),!)1!
4#&!(822&,4!)84/84!12)*!+4.!0$.4!%$08&6!&..&,4+$005!42&$4+,3!4#&!
0$.4!)84/84!$.!$!.)14!)2!/.&8-)"4$23&4@!O&*/)2$0!&,.&*'0+,3!
Fig. 1. Supervision by Denoising. We train a reconstruction network on unlabeled data u
nolabel
. Reconstructor output z
is denoised spatially using a
spatial denoiser (a) then stochastic averaging (b) to produce pseudo-target z. The reconstruction loss between z
and z is scaled and added to that
of labeled pair (x
labeled
, y) and the combined used to update reconstructor weights. The denoising pipeline is highlighted in blue. See Algorithm 1.
Reconstructor with
channel dropout
Input image
augmentation
x
labeled
u
nolabel
y
Recon loss
Recon loss
𝜆
z
Train
Loss
Reconstruction
denoiser
1 − 𝛼
𝛼
(b)
z
(a)
𝛽
1 − 𝛽
YOUNG ET AL.: SUPERVISION BY DENOISING 3
KOcL!9<>;!+*/2)%&.!8/),!4#&!g!*)-&0!'5!$%&2$3+,3!/2&%+)8.!
,&47)2P!)84/84!4)!)'4$+,!/.&8-)"4$23&4.@!O#&!*&$,!4&$(#&2!
*)-&0!9>D;!+.!$!%$2+$,4!)1!9<>;!7#+(#!8.&.!4#&!)84/84!12)*!$!
4&*/)2$005!$%&2$3&-!,&47)2P!$.!/.&8-)"4$23&4.@!h&25!2&(&,4!
&=4&,.+),.!4)!Oc!$2&!'$.&-!),!$332&..+%&!-$4$!$83*&,4$4+),!
$,-!/).4/2)(&..+,3!)1!4#&!,&47)2P!)84/84!9DN;6!9DR;6!*804+/0&!
4&$(#&2!*)-&0.!4)!3&,&2$4&!4#&!/.&8-)"4$23&4.!9HT;6!9H:;6!$,-!
42$,.1)2*$4+),"(),.+.4&,(5!9>?;6!9>D;!/$24+(80$205!+,!4#&!($.&!
)1!2&(),.428(4+),!4$.P.@!W$4&2!7&!.#)7!Oc!($,!'&!+,4&2/2&4&-!
$.!$!1)2*!)1!].8/&2%+.+),!'5!4&*/)2$0!-&,)+.+,3^!$,-!&=4&,-!
4#&!-&,)+.+,3!4)!./$4+$0!-+*&,.+),.!4)!+*/2)%&!.8/&2%+.+),!
)1!2&(),.428(4+),!,&47)2P.@!`,!4#&!/2)(&..6!7&!$0.)!2&%&$0!4#&!
+*/0+(+4!2&380$2+F&-!42$+,+,3!)'U&(4+%&!)/4+*+F&-!'5!Oc@!
! I/$4+$0!2&380$2+F$4+),!#$.!'&&,!&*/0)5&-!&=4&,.+%&05!+,!
+*$3&! 2&(),.428(4+),! /2)'0&*.! 4)! +*/).&! 4)/)0)3+($0! $,-!
./$4+$0!(),.42$+,4.!),!2&(),.428(4+),@!O#&.&!2&380$2+F&2.!($,!
$0.)!'&!8.&-!$.!$,!8,.8/&2%+.&-!0)..!4&2*!),!8,0$'&0&-!-$4$!
4)!1$(+0+4$4&!.&*+".8/&2%+.&-!0&$2,+,3@!I4$2".#$/&!2&380$2+F&2!
9<?;!/&,$0+F&.!+.0$,-.!$,-!#)0&.!+,!4#&!2&(),.428(4+),6!7#+0&!
4#&!4)/)0)3+($0!0)..!)1!\0)83#!et al.!9:Q;!/&,$0+F&.!%+)0$4+),.!
+,!4#&!4)/)0)35!)1!4#&!2&(),.428(4+),6!.8(#!$.!4#&!,8*'&2!)1!
#)0&.6!#$,-0&.6!$,-!(),,&(4&-!()*/),&,4.@!O#&!2&380$2+F&2!
)1!9H<;!8.&.!$!_eC!4)!/&,$0+F&!2&(),.428(4&-!/+=&0.!-&&*&-!
1$P&@!i#)8!et al.!9H>;!/&,$0+F&!4#&!.+F&!)1!.428(482&.!-&%+$4+,3!
4))!*8(#!12)*!$!/2+)2!.+F&!-+.42+'84+),!7#&2&$.!4#).&!+,!9D?;!
$,-!9D<;6!-&%+$4+,3!12)*!$!/2+)2!.#$/&!-+.42+'84+),@!IBZ!($,!
$0.)!'&!4#)83#4!)1!$.!2&380$2+F+,3!.#$/&!'84!8,0+P&!9D<;6!9D?;!
$//0+&.!4#+.!2&380$2+F$4+),!4)7$2-.!IIW!$,-!)/&2$4&.!7+4#+,!
$!/2+,(+/0&-!)/4+*+F$4+),"'$.&-!12$*&7)2P@!
3 MATHEM ATICAL FRAMEWORK
`,!&..&,(&6!IBZ!0&%&2$3&.!$!2&(),.428(4+),!-&,)+.&2!4)!42$+,!
4#&!*$+,!2&(),.428(4+),!,&47)2P!),!8,0$'&0&-!-$4$j!2&1&2!4)!
a+3@!:!1)2!4#&!&,-"4)"&,-!42$+,+,3!/2)(&-82&@!_+%&,!$!0$'&0&-!
+*$3&6!)82!)'U&(4+%&!+.!.+*/05!4)!*+,+*+F&!4#&!0)..!'&47&&,!
4#&!2&(),.428(4+),!$,-!4#&!()22&./),-+,3!428&!0$'&0@!a)2!$,!
8,0$'&0&-!+*$3&6!#)7&%&26!)82!)'U&(4+%&!+.!,)7!4)!*+,+*+F&!
4#&!0)..!'&47&&,!4#&!2&(),.428(4+),!$,-!4#&!-&,)+.&-!%&2.+),!
)1!+4@!O#&!47)!0)..&.!$2&!$--&-!$,-!'$(P/2)/$3$4&-!4#2)83#!
4)!8/-$4&!4#&!,&47)2Pk.!/$2$*&4&2.@!O#+.!+.!.8**$2+F&-!+,!
e03)2+4#*!:@!A&!,)7!-&2+%&!4#&!IBZ!$03)2+4#*@!
3.1 Regularizing Reconstruction
a2)*!$!0&$2,+,3!/&2./&(4+%&6!4#&!&.4+*$4+),!)1!$,!8,P,)7,!
8,-&205+,3! +*$3&!f(x)!1)2! +,/84!x +,%)0%&.!*$=+*+F+,3! 4#&!
(),-+4+),$0!/2)'$'+0+45!)1!f(x)!3+%&,!x@!X$=+*8*!0+P&0+#))-!
+.!45/+($005!8.&-!+,!.8/&2%+.&-!0&$2,+,3!$,-!&..&,4+$005!0&$-.!
4)!.)0%+,3!$,!)/4+*+F$4+),!/2)'0&*!)1!4#&!1)2*!
!
minimize (Θ) =
∑ (y
, f(x
|Θ))
=
,
K:L!
+,!7#+(#!x
!-&,)4&.!$,!+*$3&6!$,-!f(x
|Θ)6!4#&!*$//+,3!)1!
+*$3&!x
!'5!$!,&47)2P!f!/$2$*&4&2+F&-!'5!Θ@!A&!-&,)4&!'5!
(y
, f
)6!$!*&$.82&!)1!0)..!'&47&&,!f
= f(x
|Θ)!$,-!0$'&0!
y
@!O5/+($0! 0)..!18,(4+),.!!1)2!*&-+($0!+*$3+,3!/2)'0&*.!
+,(08-&!($4&3)2+($0!(2)..!&,42)/5!$,-!4#&!Z+(&!0)..@!
! `1!4#&!42$+,+,3!-$4$.&4!(),4$+,.!),05!$!.*$00!,8*'&2!!)1!
+*$3&l0$'&0!/$+2.!(x
, y
)6!*+,+*+F+,3!)'U&(4+%&!(Θ)!()80-!
0&$-!4)!)%&2EM+,3!$,-!#$*/&2!3&,&2$0+F$4+),!)1!f!4)!+*$3&.!
)84.+-&!4#&!42$+,+,3!-+.42+'84+),@!e.!7&00!$.!-+%&2.+15+,3!4#&!
42$+,+,3!-+.42+'84+),!%+$!-$4$!$83*&,4$4+),6!),&!($,!482,!4)!
*$=+*8*!$!/).4&2+)2+!/2)'$'+0+45!KXefL!&.4+*$4+),6!7#&2&!
$!0)3"/2+)2!K)2!2&380$2+F$4+),!4&2*L!1)2!,&47)2P!/$2$*&4&2.!
+.!+,42)-8(&-!4)!4#&!42$+,+,3!)'U&(4+%&m!
!
(Θ) = (Θ) +
∑ (f(u
|Θ))
=
,
K<L!
+,!7#+(#!u
!-&,)4&.!$,!+*$3&6!.+*+0$205!4)!x
6!!2&/2&.&,4.!
$!2&380$2+F+,3!18,(4+),$06!.8(#!$.!4#&!4)/)0)3+($0!0)..!9:Q;!)2!
4#&!4)4$0!%$2+$4+),!.&*+",)2*!9<D;!$,-!!(),42)0.!4#&!7&+3#4!
)1!4#&!!4&2*.!+,!4#&!)'U&(4+%&@!I+,(&!+*$3&.!u
!,&&-!,)4!'&!
4#&!.$*&!$.!+*$3&.!x
!,)2!$,,)4$4&-!7+4#!0$'&0.6!)/4+*+F+,3!
)'U&(4+%&!K<L!($,!'&!$!8.&180!7$5!4)!+*/2)%&!3&,&2$0+F$4+),!+1!
),&!#$.!$((&..!4)!%&25!1&7!K.$56!L!0$'&0&-!+*$3&.!'84!*)2&!
KL!8,0$'&0&-!),&.6!7#+(#!+.!12&Y8&,405!4#&!($.&!+,!*&-+($0!
+*$3&!2&(),.428(4+),!/2)'0&*.@!`,!.8(#!$!].&*+".8/&2%+.&-!
0&$2,+,3^!KIIWL!/$2$-+3*6!$!.8+4$'0&!(#)+(&!)1!!$,-!42$-&"
)J!!($,!#&0/!4)!0&%&2$3&!')4#!0$'&0&-!$,-!8,0$'&0&-!-$4$!4)!
+*/2)%&!3&,&2$0+F$4+),!)1!4#&!42$+,&-!,&47)2P!9:Q;6!9H>;@!A&!
+008.42$4&!4#+.!+,!a+3@!>6!()*/$2+,3!%$0+-$4+),!$((82$(5!7+4#!
$,-!7+4#)84!2&380$2+F$4+),!1)2!4#&!()24+($0!/$2(&00$4+),!4$.P@!
! `,!*$,5!($.&.6!4#&!2&380$2+F&2!!8.&-!+,!.&*+".8/&2%+.&-!
2&(),.428(4+),!+.!*$4#&*$4+($005!(2$14&-!1)2!$!3+%&,!/2)'0&*!
9<?;6!9:Q;!$,-!)14&,!2&Y8+2&.!.+3,+E($,4!&J)24!4)!38$2$,4&&!+4!
+.!-+J&2&,4+$'0&!9:Q;@!O#+.!7)2P!/2)/).&.!4)!)%&2()*&!.8(#!
-+[(804+&.!'5!(),.428(4+,3!!$.!$!18,(4+),!)1!$!-&,)+.&2!$,-!
)/4+*+F+,3!)'U&(4+%&!K<L!'5!$04&2,$4+,3!4#&!-&,)+.+,3!$,-!4#&!
.4)(#$.4+(!32$-+&,4!-&.(&,4!.4&/.@!
HCP 810439
(a) Input image
(b) U-Net [5], [36] (0.7423)
(c) U-Net + SUD (0.8206)
(d) FS Reference [77]
Fig. 2. SUD applied to “cortical parcellation”. Validation results shown with Dice in parentheses. One labeled image used for training. Input (a) is a
three-channel image consisting of (sulcal map, white matter curvature, inflated surface curvature). Baseline U-Net produces noisy and anatomically
implausible reconstruction (b) while training it additionally under SUD produces a more plausible one (c) that is closer to the reference (d).
Algorithm 1.!I8/&2%+.+),!'5!Z&,)+.+,3!
" Input:!{x
, y
}
=
!(labeled)6!{u}
=
!(unlabeled)6!
#! !!!!
f
(
|
Θ
0
)
(reconstructor to train)6!a
()
(trained denoiser)6!
$! !!!
!(
= 0.125
,!denoiser strength)6!
%! !!!!
max
!(
= 8
,!self-supervision weight)6!
!(training steps)6!
& Output:
f|Θ
(trained reconstructor)!
'
z
=
!Initialize all soft targets to zero
( for
!
= 1, . . . , !do
)
= 1 −( ⁄ )6!!!! = ( ⁄ )
max
!!!nn!Sec 3.3.2
*
x
,
y
=
!Load!
th (
%
) image–label pair!
"+
u
,
z
=
!Load!
th (
%
) unlabeled image and soft-target
!
""
! ! z
=
a
(
f
(
u
|
Θ
)) + (1 − )
f
(
u
|
Θ
) + (1 − )
z
! !
"#! ! = (
y
,
f
(
x
|
Θ
)) + (
z
,
f
(
u
|
Θ
))
!
"$
! ! Θ
+
=
!Update!Θ
by back-propagating loss
"% end for!
!
4 IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. 0, NO. 0, DECEMBER 2023
3.2 Regularization by Denoising (RED)
e.!$!.4$24+,3!/)+,46!7&!$,$05F&!2&380$2+F$4+),!'5!-&,)+.+,3!
KbcZL!9<N;6!9<Q;6!/2)/).&-!1)2!)/4+*+F+,3!$,!)'U&(4+%&!)1!4#&!
1)2*!(f) = (x, f) + (f)!1)8,-!+,!*$,5!0)7"0&%&0!%+.+),!
$,-!+*$3+,3!/2)'0&*.@!O5/+($0056!4#&!!4&2*!*&$.82&.!4#&!
E-&0+45!)1!f!4)!.)*&!3+%&,!)'.&2%$4+),!x!7#&2&$.!4#&!!4&2*!
Y8$,4+E&.!4#&!2&380$2+45!)1!f!8.+,3!.8(#!*&$.82&.!$.!4#&!Oh!
.&*+,)2*!)2!4#&!'+0$4&2$0!2&380$2+F&2@!O#&!bcZ!12$*&7)2P!
+,.4&$-!&=/$,-.!4#&!2&380$2+45!4&2*!+,4)!4#&!'+0+,&$2!1)2*!
!
2(f) = 〈f, h(f)〉 = 〈f, f −a(f)〉,!
K>L!
+,!7#+(#!h(f)!2&/2&.&,4.!$!,),"0+,&$2!W$/0$(+$,!)1!f6!$,-!a(f)!
+.!4#&!()22&./),-+,3!.*))4#+,3!)/&2$4+),@!O#+.!$00)7.!8.!4)!
2&380$2+F&!f 8.+,3!$!,),"-+J&2&,4+$'0&!-&,)+.+,3!E04&2!a!$,-!
.4+00!&=/2&..!4#&!32$-+&,4!)1!!%+$!
(f) = f −a(f)6!/2)%+-&-!
4#$4!4#&! o$()'+$,!A = a(f)!+.!.5**&42+(6!a!+.!-+J&2&,4+$'0&!
$,-!0)($005!#)*)3&,&)8.@!b&380$2+F&2.!(),428(4&-!4#+.!7$5!
$2&!2&/)24&-!4)!7)2P!&%&,!7#&,!4#&!$')%&!(),-+4+),.!-)!,)4!
#)0-!9H?;@!`1!32$-+&,4!-&.(&,4!+.!8.&-!4)!E,-!4#&!)/4+*$0!f6!4#&!
4#!8/-$4&! .4&/! +.!&Y8+%$0&,4!4)! -&,)+.+,3!4#&!+4&2$4&!f
4)!
)'4$+,!z
= a(f
)!4#&,!-&.(&,-+,3!$0),3!!
!
d
= −
(
(
x, f
)
+
(
2
⁄ )‖
z
−f
‖
)
f
(f
),!
K?
L
!
,)4+,3!4#$4!4#&!-&2+%$4+%&!)1!(1 2⁄ )‖z
−f‖
!$4!f
+.
(f
)@!`,!
4#+.!7)2P6!K.4)(#$.4+(L!32$-+&,4!-&.(&,4!+.!)1!./&(+$0!+,4&2&.4!
$.!+4!+.!8.&-!0$4&2!4)!42$+,!2&(),.428(4+),!,&47)2P.@!
! A#+0&!bcZ!+.!,)4!-&.+3,&-!1)2!42$+,+,3!,&47)2P.6!7&!($,!
$M&*/4!4)!+,()2/)2$4&!bcZ!+,!.&*+".8/&2%+.&-!0&$2,+,3!'5!
.8'.4+484+,3!K>L!+,!)'U&(4+%&!K<L@!e//05+,3!.4)(#$.4+(!32$-+&,4!
-&.(&,4!KI_ZL!7+4#!4#&!-&.(&,4!.4&/!.+F&!!4)!4#&!.8'.4+484&-!
)'U&(4+%&!5+&0-.!4#&!.&01".8/&2%+.+),!.4&/!
!
z
=
a(f(u
|Θ
))!
(denoising)
!
KDL!
!
Θ
+
=
Θ
−
2
‖
z
−f
(
u
|
Θ
)‖
Θ
(Θ
)!
1)00)7&-!'5!4#&!8.8$0!.8/&2%+.+),!.4&/!
!
Θ
+
= Θ
+
−
(
y
, f
(
x
|
Θ
))
Θ
(Θ
),
KH
L
!
+,!7#+(#!(x
, y
)!+.!$,!+*$3&l0$'&0!/$+2!$,-!u
!$,!8,0$'&0&-!
+*$3&!K1)2!(0$2+456!7&!$..8*&!4#$4!4#&!4#!+*$3&.!$,-!0$'&0!
$2&!/+(P&-!$4!4#&!4#!.4&/L@!B/-$4&.!KDL!-&,)+.&!f(u
|Θ
)!4)!
/2)-8(&!$!.)14!4$23&4!z
!$,-!8/-$4&!Θ
4)!2&-8(&!4#&!
!0)..!
'&47&&,!4#&!2&(),.428(4+),!f(u
|Θ
)!$,-!.)14!4$23&4!z
@!O#+.!
+.!1)00)7&-!'5!$,)4#&2!8/-$4&!4)!2&-8(&!4#&!.8/&2%+.&-!0)..!
'&47&&,!y
!$,-!f(x
|Θ
)@!e04&2,$4+%&056!4#&!47)!8/-$4&.!($,!
'&!/&21)2*&-!+,!$!.+,30&!'$(P7$2-!/$..!Ke03)2+4#*!:L@!
! bcZ"+,-8(&-!8/-$4&!.4&/.!KDL!1)2(&!.)14!4$23&4!z
!4)!'&!$!
18,(4+),!)1!),05!4#&!(822&,4!2&(),.428(4+),!$,-!$--+4+),$005!
2&.42+(4.!4#&!0)..!4)!$,!
!),&@!p)7&%&26!+4!+.!P,)7,!9<>;!4#$4!
1)2!.4)(#$.4+(!)/4+*+F$4+),6!$%&2$3+,3!/2&-+(4+),.!$(2)..!4#&!
42$+,+,3!.4&/.!+.!(28(+$0!4)!2&-8(+,3!.4)(#$.4+(!,)+.&6!#+,4+,3!
4#$4!4#&!3&,&2$0!1)2*!)1!4#&!-&,)+.+,3!.4&/!+,!KDL!*$5!,)4!'&!
)/4+*$0@!`,-&&-6!7&!E,-!4#$4!+4&2$4+,3!8/-$4&!.4&/.!KDL!0&$-.!
4)!-+%&23&,4!42$+,+,3!)84()*&.!+,!/2$(4+(&@!S&.+-&.6!.&%&2$0!
0)..!18,(4+),.!#$%&!/2&%+)8.05!'&&,!/2)/).&-!./&(+E($005!4)!
.)0%&!.&3*&,4$4+),"45/&!+*$3+,3!/2)'0&*.!9:D;!'84!)2+3+,$0!
bcZ!7)80-!+*/0+(+405!8.&!4#&!XIc!0)..!),!-&,)+.&-!4$23&4.@!!
3.3 Supervision by Denoising (SUD)
I8/&2%+.+),!'5!-&,)+.+,3!KIBZL!.&&P.!4)!()*'+,&!bcZ!$,-!
.4)(#$.4+(!$%&2$3+,3!8.&-!+,!.&*+".8/&2%+.&-!0&$2,+,3!KIIWL!
4&(#,+Y8&.!9>?;6!9>D;@!A#+0&!IBZ!7)2P.!7+4#!$,5!2&(&,4!IIW!
'$(P'),&6!7&!$..8*&!4#$4!4&*/)2$0!&,.&*'0+,3!9<>;!+.!8.&-!
4)!1$(+0+4$4&!$,$05.+.@!O#+.!0&$-.!4)!$!/$24+(80$205!&0&3$,4!IBZ!
.&01".8/&2%+.+),!.4&/!)1!4#&!1)2*!
!
z
= prox
(
f(u
|Θ
)
+
(
1 −
)z
)!
KNL!
!
Θ
+
=
Θ
−
(
z
, f
(
u
|
Θ
))
Θ
(Θ
)
,
!
+,! 7#+(#!prox
(f) = argmin
t
(t) + (1/2)‖t − f‖
-&,)4&.!
4#&!]/2)=+*$0!*$/^!$..)(+$4&-!7+4#!6!!+.!$!0)..!18,(4+),!
K(2)..!&,42)/56!Z+(&6!&4(@L6!z
!K2#.L!+.!4#&!0$.4!%$08&!)1!4#&!.)14!
4$23&46!$,-!0 ≤ ≤ 1!./&(+E&.!$!()*'+,$4+),!7&+3#4!1)2!4#&!
(822&,4!2&(),.428(4+),!$,-!z
@!`,!e//&,-+=!e6!7&!-&2+%&!4#&!
8/-$4&!.4&/.!KNL!$.!$!.4)(#$.4+(!%$2+$,4!)1!/2)=+*$0!1)27$2-!
'$(P7$2-!./0+M+,3!9HD;!)1!$!42$+,+,3!)'U&(4+%&!4#$4!.&&P.!4#&!
)/4+*$0!.)14!4$23&4.!$,-!)/4+*$0!,&47)2P!7&+3#4.@!C)4&!4#$4!
42$+,+,3!)1!$!18005!(),%)084+),$0!,&47)2P!+.!+.!$!,),"(),%&=!
/2)'0&*!$,-!(),%&23&,(&!+.!),05!4)!$!0)($005!)/4+*8*@!
! e04#)83#!8/-$4&!.4&/.!KNL!($,!2&$-+05!'&!U8.4+E&-!12)*!$!
/2)=+*$0!)/4+*+F$4+),!/&2./&(4+%&6!4#&!/2&.&,(&!)1!4#&!prox!
)/&2$4)2!$0.)!,&(&..+4$4&.!.)0%+,3!$,!)/4+*+F$4+),!/2)'0&*!
$4!&%&25!42$+,+,3!.4&/6!7#+(#!($,!'&!().405@!I+,(&!a!+.!8.8$005!
$2'+42$25!K+4.!)'U&(4+%&!+.!,)4!4)!*)-&0!/#5.+($0!/#&,)*&,$!
'84!.+*/05!4)!-&,)+.&L6!7&!$//2)=+*$4&!4#&!/2)=+*$0!.4&/!+,!
KNL!$.!$!-+2&(4!-&,)+.+,3!.4&/!7+4#! = !Ke//&,-+=!eLm!!
!
z
=
(
a(f
)
+
(
1 −
)f
)
+
(
1 −
)z
!
KQL!
+,!7#+(#!/$2$*&4&2!0 ≤ ≤ 1!./&(+E&.!4#&!42$-&)J!'&47&&,!
Impulse response
Frequency response
|()|
()
Fig. 4. Spatial (left) and temporal (right) filter responses. Optimization-
based spatial filters (left, orange) add an all-pass frequency component
β while direct smoothing filters (left, blue) do not. Temporal filtering with
parameter α graduated across step n induces time-dependent temporal
impulse responses.
Spatial
Temporal
(1 − )
−
Fig. 3. Validation curves for cortical parcellation showing the effect of
regularization. U-Net (nn-UNet) is trained using one labeled image and
channel dropout with drop p of 0.05. Dice and 95HD worsen across
epochs for U-Net but improve when regularized using SUD or temporal
ensembling (TE). See Fig. 2 for visualizations of reconstructions.
Epochs
Dice Overlap
Epochs
95HD (× 100 Pixels)
U-Net + SUD
U-Net
U-Net + TE
U-Net
U-Net + TE
U-Net + SUD
YOUNG ET AL.: SUPERVISION BY DENOISING 5
4#&!.*))4#,&..!)1!z
!$,-!+4.!E-&0+45!4)!f
= f(u
|Θ
)@!A+4#!
= 06!-&,)+.+,3!.4&/!KQL!2&-8(&.!KNL!4)!$!%$2+$,4!)1!4&*/)2$0!
&,.&*'0+,3!9<>;@!O#&!bcZ!-&,)+.+,3!.4&/!)1!KDL!($,!'&!.&&,!
$.!$!0+*+4+,3!($.&!)1!KQL!7#&2&! = = 1@!
! `*/0&*&,4&-!,$+%&056!4#&!.+*/0+E&-!/2)=+*$0!.4&/!)1!KQL!
2&Y8+2&.!.4)2$3&!)1!4#&!0$.4!%$08&!)1!4#&!/.&8-)"4$23&4!z
!1)2!
&$(#!8,0$'&0&-!&=$*/0&6!7#+(#!($,!'&()*&!%&25!().405!7+4#!
$!0$23&!8,0$'&0&-!-$4$.&4@!O#+.!.4)2$3&!2&Y8+2&*&,4!+.!&$.+05!
$--2&..&-!8.+,3!$,!$04&2,$4&!$//2)$(#6!7#&2&!$,!$%&2$3&-!
,&47)2P!+.!8.&-!4)!3&,&2$4&!/.&8-)"4$23&4.@!`,!4#+.!($.&6!!
!
z
= a(f
) + (1 −)f(u
|
Θ
+ (1 −
)Θ
−
)!
KRL!
+,!7#+(#!
= ( −) (1 −)⁄ @!A+4#! = 06!-&,)+.+,3!.4&/!
KRL!2&-8(&.!4)!4#&!7&+3#4!$%&2$3+,3!)1!4#&!*&$,!4&$(#&2!9>D;!
*)-&0@!A&!E,-!4#$4!&+4#&2!-&,)+.+,3!.4&/!($,!'&!8.&-!+,!4#&!
(),4&=4!)1!IBZ!7+4#!,)!,)4+(&$'0&!-+J&2&,(&!+,!4#&!$((82$(5!
)1!4#&!42$+,&-!2&(),.428(4+),!,&47)2P@!A&!,)7!$,$05F&!4#&!
./$4+$0!$,-!4#&!4&*/)2$0!E04&2+,3!$./&(4.!)1!IBZ6!3)%&2,&-!
2&./&(4+%&05!'5!/$2$*&4&2.!!$,-!@!
3.3.1 Spatial Denoising Aspects
O#&!./$4+$0!E04&2!/$2$*&4&2!!'2+-3&.!4#&!3$/!'&47&&,!-+2&(4!
$,-!)/4+*+F$4+),"'$.&-!K/2)=+*$0L!-&,)+.+,3!*&4#)-.@!W&4!
8.!$..8*&! = 1!$,-!72+4&!z
d
= a(f
) = (a(f
))f
= Af
!4)!
&=/2&..!-+2&(4!-&,)+.+,3!)1!$,!+4&2$4&!f
6!+,%)P+,3!4#&!]0)($0!
#)*)3&,&+45^!9<Q;!)1!a K7&!$..8*&!a!+.!-+J&2&,4+$'0&!.)0&05!
1)2! $,$05.+.L@! I/&(42$0! -&()*/).+4+),!A = U(I −Λ)U
∗
!4&00.!
8.!4#$4!A!#$.!./&(42$0!E04&2!1$(4)2.!9?N;!!
!
1 ≥ 1 −
≥ 1 −
≥ ≥ 1 −
≥ 0,!
K:T
L
!
+,!7#+(#!
!-&,)4&.!4#&!4#!-+$3),$0!&0&*&,4!)1!Λ6!4#&!E2.4!
+,&Y8$0+45!()*&.!12)*!4#&!$..8*&-!.42),3!/$..+%+45!)1!a!9<Q;
$,-!4#&!0$.4!),&!()*&.!12)*!+4.!/).+4+%&!-&E,+4&,&..@!
! a)2!)/4+*+F$4+),"'$.&-!-&,)+.+,36!(),.+-&2!&%$08$4+),!)1!
4#&!*$//+,3!prox
!KNL!),!f
@!`1!7&!0+,&$2+F&!4#&!-&,)+.&2!+,!
4#&!!4&2*!$.!a(u) ≈ (a(f
))u = Au6!4#&!/2)=+*$0!*$//+,3!
($,!'&!&=/2&..&-!$.!z
= (I + (I −A))
−
f
= Hf
@!I/&(42$0!
-&()*/).+4+),!($,!'&!8.&-!.+*+0$205!4)!2&%&$0!4#$4!H!.#$2&.!
4#&!.$*&!./&(42$0!E04&2!'$.+.!U!$.!A!'84!#$.!E04&2!1$(4)2.!
!
1 ≥
1
1 +
≥ ≥
1
1 +
≥ (1 + )
−
!
K::
L
!
.)!4#&!-&,)+.&2.!A $,-!H!-+J&2!),05!+,!4#&+2!E04&2!1$(4)2.@!`,!
/$24+(80$26!K::L!.#)7.!z
!+.!$!0+,&$2!()*'+,$4+),!)1!-&,)+.&-!
+,/84!$,-!4#&!8,E04&2&-!+,/84!7+4#!7&+3#4.!(1 + )
−
!$,-!
(1 + )
−
!2&./&(4+%&05@!a+3@!?!K0&14L!.#)7.!4#+.!-&()*/).+4+),!
+,!4&2*.!)1!./&(42$0!E04&2!1$(4)2.!K::L@!!
! IBZ!&..&,4+$005!2&"/$2$*&4&2+F&.!4#&!7&+3#4.!(1 + )
−
!
$,-!(1 + )
−
!$.!!$,-!(1 − )@! A&+3#4.!, 1 − !$2&! 8.&-!4)!
()*'+,&!4#&!8,E04&2&-!f
$,-!-&,)+.&-!a(f
)!()*/),&,4.!4)!
/2)-8(&!$!.)14!4$23&4!z
d
!4#$4!$//2)=+*$4&.!4#&!/2)=+*$0!),&!
z
@!O#+.!$//2)=+*$4+),!)'%+$4&.!4#&!,&&-!4)!.)0%&!/2)=+*$0!
)/4+*+F$4+),!/2)'0&*.!$4!42$+,!4+*&!$,-!#$.!*)2&!d&=+'+0+45!
4#$,!4#$4!8.&-!'5!bcZ!9<N;6!7#+(#!+*/0+(+405!(),.42$+,.!4#&!
.)14!4$23&4!4)!z
d
= a(f
)@!a+3@!D!/0)4.!2&(),.428(4+),!$((82$(5!
8,-&2!IBZ!1)2!-+J&2&,4!%$08&.!)1!6!.#)7+,3!4#$4!)/4+*$005!
()*'+,+,3!a(f
)!$,-!f
!+.!'&,&E(+$0!1)2!.8/&2%+.+),@!
3.3.2 Temporal Denoising Aspects
e4! 42$+,! 4+*&6! 2$,-)*!,)+.&!+,!4#&!/2&-+(4&-!2&(),.428(4+),!
-8&!4)!$83*&,4$4+),!$,-!-2)/)84!*$5!($8.&!.)14!4$23&4.!4)!
(#$,3&!12)*!),&!42$+,+,3!.4&/!4)!4#&!,&=46!(2&$4+,3!$!*)%+,3!
4$23&4!/#&,)*&,),@!O#&2&1)2&6!-&,)+.+,3!.)14!4$23&4.!$(2)..!
+4&2$4+),.!#&0/.!$%&2$3&!)84!.4)(#$.4+(!/2&-+(4+),!,)+.&!$,-!
+*/2)%&.!42$+,+,3!(),%&23&,(&!9<>;@!O#&! E04&2! /$2$*&4&2!!
($,!'&!.&&,!&Y8+%$0&,405!$.!-&E,+,3!$!E2.4!)2-&2!``b!K+,E,+4&!
+*/80.&!2&./),.&L!E04&2!7+4#!4#&!4&*/)2$0!+*/80.&!2&./),.&!
!
(, (1 −)
, . . . , (1 − )
, . . . ),!
K:<
L
!
$0.)!($00&-!$,!&=/),&,4+$0!*)%+,3!$%&2$3&!E04&2!9>D;@!C)4&!
4#$4!$,!``b!E04&2!)1!)2-&2!!,&(&..+4$4&.!$--+4+),$005!.4)2+,3!
4#&!0$.4!!.)14!4$23&4.!1)2!&$(#!42$+,+,3!+*$3&@!`*/0&*&,4+,3!
$!#+3#&2!)2-&2!4&*/)2$0!``b!E04&2!+.!4#8.!*)2&!().405!$,-!5&4!
/2)%+-&!,)!(0&$2!$%&2$3+,3!'&,&E4.!)%&2!$!E2.4!)2-&2!),&@!`,!
4#&!($.&!)1!$!E2.4!)2-&2!E04&26!4&*/)2$0!E04&2+,3!($,!'&!2&($.4!
&Y8+%$0&,405!$.!$!/2)=+*$0!32$-+&,4!-&.(&,4!.4&/!7#&2&!4#&!
-&.(&,4!.4&/!.+F&!+.! = /j!.&&!e//&,-+=!e@!
! `,!/2$(4+(&6!7&!32$-8$4&!4#&!E04&2!/$2$*&4&2!!12)*!:!4)!T!
$(2)..!4#&!()82.&!)1!4#&!42$+,+,36!&,$'0+,3!4#&!.)14!4$23&4.!4)!
$-$/4!4)!4#&!2$/+-05!&%)0%+,3!,$482&!)1!f(x|Θ) $4!4#&!.4$24!)1!
42$+,+,3!$,-!(),%&23+,3!4)!E=&-!K,),"*)%+,3L!4$23&4.!$4!4#&!
&,-!)1!4#&!42$+,+,3@!a+3@!?!K2+3#4L!/0)4.!4#&!+*/80.&!2&./),.&!
)1!4#&!4+*&"%$25+,3!-&,)+.+,3!E04&26!-&/+(4+,3!4#&!+,d8&,(&!
)1!(822&,4!.)14!4$23&4.!),!18482&!+4&2$4+),.@!h$25+,3!!($,!'&!
*)4+%$4&-!$0.)!12)*!$,!)/4+*+F$4+),!/&2./&(4+%&!9HD;6!7#&2&!
-&($5+,3!.4&/!.+F&.!$2&!()**),05!8.&-!1)2!(),%&23&,(&@!A&!
E,-!4#$4!.&M+,3! = 0.9!)2!0.99!$.!+,!9>D;!.0+3#405!-&(2&$.&.!
/&21)2*$,(&G8,0+P&!(0$..+E($4+),!,&47)2P.6!7#+(#!)84/84!
$!/))0&-!/2&-+(4+),!$(2)..!4#&!&,4+2&!+*$3&6!2&(),.428(4+),!
,&47)2P.!/2)-8(&!$!/2&-+(4+),!$4!&$(#!+,-+%+-8$0!/+=&0!7+4#!
$!0$23&2!4&*/)2$0!/2&-+(4+),!%$2+$,(&!$.!$!2&.804@!Z&(2&$.+,3!
4#&!%$08&!)1!!$32&..+%&05!4)7$2-.!F&2)!&,.82&.!4#$4!4#&!.)14!
4$23&4.!(),%&23&@!A&! %$25!!+,!4#&!)//).+4&!*$,,&26!12)*!0!
4)!
max
6!'2+,3+,3!.&01".8/&2%+.+),!32$-8$005!+,4)!&J&(4@!O#+.!
+.!.+*+0$2!4)!7&+3#4!2$*/+,3!+,!9<>;6!7#+(#!8.&.!$!_$8..+$,!
Fig. 5. Effect of varying β on validation accuracy (Dice and 95HD) for
the cortical parcellation task. Numbers 1, 2, 5 and 10 on lines denote
the number of labeled training images used. Dice quickly improves with
increasing
β and gradually worsens past β
=
0.05
. 95HD improves very
quickly initially then gradually past
β
=
0.05. Plotted for λ
=
4.
Dice Overlap
95HD (Pixels)
1
2
5
1
0
10
5
2
1
!
!
32
In
64
128
256
320
320
32
32
64
64
128
128
Out
256
256
320
320
128
256
320
320
32
64
! 3
3
!,-./0.-12031435! 3
3
!,-./06-7829:!
! 3
3
!,-./!!;681<=4!#>0.-12031435! ,-?@A6B<?!
! 3
3
!,-./
t
!;681<=4!#>0.-12031435!
!
Fig. 6. Network structure used. Both our reconstruction and denoising
networks have 5–6 levels of convolutions, instance normalization and
leaky relu. We double the number of features at each new level for a
maximum of 320 features. For denoisers, we remove skip connections
and use max (un)pooling instead of strided convolutions.
6 IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. 0, NO. 0, DECEMBER 2023
K$.!)//).&-!4)!)82!0+,&$2L!.(#&-80&!1)2!%$25+,3!@!
3.4 Network Structures and Training
e!/)/80$2!(#)+(&!)1!2&(),.428(4)2!+.!B"C&4!9D;6!7#+(#!+.!$0.)!
8.&-!+,!4#+.!7)2P@!a)00)7+,3!,,B"C&4!9>H;6!7&!(),E382&!)82!
B"C&4!7+4#!E%&!0&%&0.!)1!(),%)084+),.!$,-!0&$P5!b&WB6!$,-!
-)8'0&!4#&!,8*'&2!)1!1&$482&!(#$,,&0.!$4!&$(#!,&7!0&%&0!8/!
4)!$!*$=+*8*!)1!><T!1&$482&.j!.&&!a+3@!H!1)2!+008.42$4+),@!
! O#&!*$+,!*)4+%$4+),!'&#+,-!IBZ6!$.!7&00!$.!bcZ!9<N;6!+.!
4)!0&%&2$3&!2&(&,4!-&%&0)/*&,4!+,!0&$2,&-!-&,)+.+,3!E04&2.!
4)!.)0%&!+*$3+,3!$,-!%+.+),!/2)'0&*.@!`,!4#+.!7)2P6!7&!42$+,!
$!.&(),-$25!0$'&0"-&,)+.+,3!,&47)2P!a(|Σ)!1)2!.8/&2%+.+),!
)1!4#&!*$+,!2&(),.428(4+),!,&47)2P!f(|Θ)@!a2)*!$!S$5&.+$,!
/&2./&(4+%&6!-&,)+.&2!,&47)2P!a!-&E,&.!$!0)3"/2+)2!),!Θ!%+$!
4#&!2&380$2+45!4&2*!(f(x|Θ))!7#+0&!2&(),.428(4)2!f6!4#&!0)3"
0+P&0+#))-!(y, f(x|Θ)) )1!Θ!3+%&,!(x, y)@!O#&!-&,)+.&2!/0$5.!
$,!+*/)24$,4!2)0&!7#&,!0&$2,+,3!,&47)2P!/$2$*&4&2.!Θ!'5!
.)0&05!)/4+*+F+,3!(y, f(x|Θ))!*$5!'&!(#$00&,3+,3@!`,!*$,5!
/2)'0&*.6!$!/2+)2!)%&2!4#&!2&(),.428(4+),.!K-&,)+.&2L!($,!'&!
&$.+&2!4)!0&$2,!4#$,!-+2&(405!)/4+*+F+,3!4#&!0+P&0+#))-!4&2*!
'5!42$+,+,3!4#&!2&(),.428(4+),!,&47)2P@!O#+.!+.!)14&,!4#&!($.&!
+1!4#&!2&(),.428(4+),.!0+%&!+,!4#&!./$(&!)1!,$482$0!+*$3&.6!)2!
*&-+($0!+*$3&!.&3*&,4$4+),.6!1)2!7#+(#!/2&%+)8.!2&1&2&,(&!
2&(),.428(4+),.!12)*!)4#&2!-$4$.&4.!&=+.4!4)!42$+,!$!-&,)+.&2@!!
! e.!1)2!4#&!(#)+(&!)1!-&,)+.+,3!,&47)2P!$2(#+4&(482&6!')4#!
(),%)084+),$0!K-&,)+.+,3L!$84)"&,()-&2!9D:;!$,-!B"C&4!),&.!
9HH;!#$%&!/2&%+)8.05!'&&,!8.&-6!7+4#!4#&!*$+,!$2(#+4&(482$0!
-+J&2&,(&!'&47&&,!4#&!47)!'&+,3!4#&!.#)24(84!(),,&(4+),.!+,!
4#&!0$M&2@!O#&!.#)24(84!(),,&(4+),.!$00)7!4#&!B"C&4!4)!0&$2,!
.8'42$(4+%&05!$,-!/2&-+(4!4#&!,)+.&!)2!]2&.+-8$0.^!9HN;!+,!4#&!
2&(),.428(4+),!7#+0&!$84)"&,()-&2.!*8.4!0&$2,!4)!/2&-+(4!4#&!
428&!2&(),.428(4+),!-+2&(405@!O#&2&1)2&6!$!B"C&4!-&,)+.&2!($,!
,)!0),3&2!'&!42$+,&-!),!2&(),.428(4+),.!7#+(#!$2&!()228/4&-!
'5!$24+E(+$0!_$8..+$,!,)+.&G4#&!,)+.&!*8.4!'&!.5,4#&.+F&-!
4)!&*80$4&!$24+1$(4.!+,!4#&!)84/84!)1!4#&!2&(),.428(4)2!8,-&2!
42$+,+,36!7#+(#!($,!($8.&!1824#&2!42$+,+,3!(#$00&,3&.!1)2!4#&!
-&,)+.&2@!A#+0&!$84)"&,()-&2.!0$(P!.P+/!(),,&(4+),.6!7&!-)!
,)4!E,-!4#$4!4#+.!0&$-.!4)!7)2.&!0)($0+F$4+),!)1!')8,-$2+&.!+,!
4#&!-&,)+.&-!2&(),.428(4+),@!e84)"&,()-&2.!4&,-!4)!/&21)2*!
&*/+2+($005!'&M&2!4#$,!4#&!B"C&4!),&.!+,!4#&!2&(),.428(4+),!
&=/&2+*&,4.@!O#&!)'.&2%$4+),!+,!9HQ;6!9D>;!$0.)!#+3#0+3#4.!4#&!
+,$'+0+45!)1!B"C&4!4)!+,/$+,4!+*$3&.!+1!,)4!42$+,&-!&=/0+(+405!
4)!+*/84&!*+..+,3!2&3+),.6!1824#&2!.833&.4+,3!4#$4!0&$2,+,3!
4#&!0$4&,4!./$(&!)1!2&(),.428(4+),!8.+,3!$84)"&,()-&2.!0&$-.!
4)!'&M&2!2&380$2+F$4+),!$,-!4#8.!.&*+".8/&2%+.+),@!a)2!4#+.!
2&$.),6!7&!8.&!$84)"&,()-&2!-&,)+.&2.!&=(08.+%&05!+,!$00!)82!
&=/&2+*&,4.@!V82!-&,)+.&2.!.#$2&!4#&!.$*&!.428(482&!$.!)82!
B"C&4!2&(),.428(4)2!7+4#!4#&!.P+/!(),,&(4+),.!2&*)%&-!$,-!
4#&!*$=!/))0+,3!2&/0$(&-!'5!.42+-&-!(),%)084+),.@!
! O)!3&,&2$4&!42$+,+,3!-$4$!1)2!-&,)+.+,36!7&!0)3"42$,.1)2*!
4#&!3+%&,!),&"#)4!0$'&0!K.&3*&,4$4+),L!*$/.6!(0+/!4#&*!4)!$!
[−3, 0]!2$,3&!K−3!$,-!0!()22&./),-+,3!4)!4#&!/2)'$'+0+4+&.!)1!
T!$,-!:!2&./&(4+%&05L6!()228/4!4#&*!8.+,3!$--+4+%&!_$8..+$,!
,)+.&!)1!F&2)!*&$,!$,-!2$,-)*!.4$,-$2-!-&%+$4+),!6!4#&,!
E,$005!4$P&!4#&+2!.)14*$=@!O)!3&,&2$4&!,)+.&!$(2)..!*804+/0&!
.($0&.6!7&!2$,-)*05!-)7,.$*/0&!/2)-8(&-!_$8..+$,!,)+.&!
*$/!'5!!$,-!8/.$*/0&!+4!4)!4#&!)2+3+,$0!.($0&@!a+3@!N!.#)7.!
&=$*/0&.!)1!2$,-)*05!3&,&2$4&-!,)+.5!42$+,+,3!+*$3&.!7+4#!
-+J&2&,4!%$08&.!)1!4#&!%$2+$,(&l.($0&!/$+2.!(, )@!A#&,!')4#!
!$,-!!$2&!0$23&6!.428(482&!$//&$2.!+,!4#&!,)+.&!/$M&2,!$,-!
4#&!-&,)+.+,3!4$.P!'&()*&.!.+*+0$2!4)!4#$4!)1!+,/$+,4+,3@!
! `,!*$,5!/2)'0&*!-)*$+,.6!0$'&0!*$/.!,&&-&-!4)!42$+,!$!
-&,)+.&2!($,!'&!)'4$+,&-!12)*!()*/84&2!3$*&.!9HR;6!9NT;!)2!
&%&,!(),.+.4!)1!$2'+42$25!.#$/&.!9N:;!4)!/)4&,4+$005!3&,&2$4&!
8,0+*+4&-!42$+,+,3!-$4$@!X$,5!4+*&.6!#)7&%&26!4#&!-&,)+.&2!
*8.4!+4.&01!'&!0&$2,&-!),!0+*+4&-!0$'&0!*$/!-$4$!4)!'))4.42$/!
.&*+".8/&2%+.+),@!`,!O$'0&!S?6!7&!0+.4!4#&!*&$,!Z+(&!)%&20$/!
$,-!RDpZ!$M$+,&-!'5!4#&!-&,)+.&2.!7+4#!-+J&2&,4!,8*'&2.!
)1!42$+,+,3!0$'&0!*$/.6!7+4#!$83*&,4$4+),!#5/&2/$2$*&4&2!
2$,3&.!, ∈ [0,8]@!O#&!4$'0&!.#)7.!4#$4!%&25!1&7!&=$*/0&.!
$2&!,&&-&-!4)!42$+,!$!-&,)+.&2@!!`,!a+3@!Q6!7&!.#)7!4#&!,)+.5!
2&(),.428(4+),!&=$*/0&.!8.&-!1)2!%$0+-$4+),6!$0),3!7+4#!4#&!
()22&./),-+,3!)84/84!12)*!4#&!$84)"&,()-&2!-&,)+.&2.@!A&!
/2&42$+,!)82!-&,)+.&2.!),!2&(),.428(4+),!&=$*/0&.!$,-!8.&!
4#&*!7+4#!7&+3#4.!E=&-!7#&,!42$+,+,3!4#&!2&(),.428(4)2@!
4 EXPERIMENTS
A&!%$0+-$4&!IBZ!'5!$//05+,3!+4!),!47)!'+)*&-+($0!+*$3+,3!
/2)'0&*.m!$,$4)*+($0!'2$+,!2&(),.428(4+),!K>ZL!$,-!()24+($0!
/$2(&00$4+),!K<ZL@!I&&!a+3@!R!1)2!$,!+008.42$4+),!)1!4#&!()24+($0!
/$2(&00$4+),!4$.P@!A&!8.&!4#&!,,B"C&4!9>H;!%$2+$,4!)1!B"C&4!
$,-!+*$3&!$83*&,4$4+),!.42$4&3+&.!K.&&!e//&,-+=!SL!1)2!$!
= 1!
!
!
=
2
!
!
!
!
!
= 2!
= 4!
= 8!
Fig. 7. Reconstruction example corrupted by Gaussian noise of varying
standard deviation C and scale D (argmax of classes shown). Increasing
C simply amplifies the magnitude of the noise injected. Increasing D, on
the other hand, magnifies the spatial scale of the noise. When both are
increased, the denoiser is forced to learn to inpaint rather than denoise.
eS`Zc!DTRND
!
!
!
!
eS`Zc!DTRND
!
!
!
!
!
C)+.5!2&(),!
Z&,)+.&-!
_2)8,-!4284#!
Fig. 8. Examples of noisy reconstructions (left column), denoised by our
auto-encoder denoisers (center column), together with the ground truth
(right column). Even in the absence of skip connections, auto-encoder-
based denoisers provide good localization of reconstruction boundaries
comparable to their U-Net-based counterparts.
YOUNG ET AL.: SUPERVISION BY DENOISING 7
1$+2!&%$08$4+),!$3$+,.4!4#&!.8/&2%+.&-!B"C&4!'$.&0+,&.@!V82!
]/).4"-&,)+.&^!*&4#)-!+*/0&*&,4.!4#&!*&4#)-!)1!9D>;!$,-!
-&,)+.&.!'$.&0+,&!2&(),.428(4+),.!8.+,3!IBZk.!-&,)+.&2@!O#&!
]&,.&*'0+,3^!*&4#)-!+.!.+*+0$2!4)!9>>;6!9>?;!7+4#!$!4&*/)2$0!
&,.&*'0+,3!'$(P'),&!1)2!$%&2$3+,3@!A&!8.&!#5/&2/$2$*&4&2!
.&M+,3.! = 0.2/
max
!$,-!
max
= 4!+,!$00!&=/&2+*&,4.!$,-!!
+.!%$2+&-!0+,&$205!12)*!T!4)!:!$(2)..!42$+,+,3!.4&/.@!X&4#)-.!
.8(#!$.!X+=B/!9N<;!$,-!\84X+=!9N>;!$2&!+,4&,-&-!1)2!+*$3&!
(0$..+E($4+),!4$.P.!$,-!$2&!,)4!()*/$2&-!7+4#!+,!4#+.!7)2P@!
! `,!4#&!$,$4)*+($0!'2$+,!2&(),.428(4+),!/2)'0&*6!7&!.#)7!
4#$4!IBZ!+*/2)%&.!4#&!2&(),.428(4)2k.!$'+0+45!4)!3&,&2$0+F&!
4)!+*$3&.!/2)-8(&-!8,-&2!%$.405!-+J&2&,4!+*$3&!$(Y8+.+4+),!
#$2-7$2&@!V,!4#&!)4#&2!#$,-6!+,!4#&!<Z!()24+($0!/$2(&00$4+),!
4$.P6!7&!-&*),.42$4&!4#$4!IBZ!+.!&=42&*&05!8.&180!7#&,!4#&!
0+P&0+#))-!4&2*!K:L!+.!8,2&0+$'0&!-8&!4)!0$(P!)1!+*$3&!4&=482&@!
4.1 Brain Cortical Parcellation (2D)
I&3*&,4+,3!4#&!#8*$,!(&2&'2$0!()24&=!+,4)!()24+($0!$2&$.!)1!
+,4&2&.4!KP,)7,!$.!/$2(&00$4+),6!a+3@!RL!+.!$!/2)'0&*!)1!32&$4!
+*/)24$,(&!+,!,&82).(+&,(&!9N?;@!p)7&%&26!4#&!(#$00&,3+,3!
,$482&!)1!0$'&0+,3!7#+4&!*$M&2!$,-!/+$0!.821$(&.!0&$-.!4)!4#&!
8,$%$+0$'+0+45!)1!0$'&0&-!()24+($0!*&.#&.!1)2!42$+,+,3@!`,!4#+.!
4$.P6!+*$3&!4&=482&!/2)%+-&.!0&..!8.&180!2&(),.428(4+),!(8&.!
Ka+3@!R6!'L!4#$,!+,!4#&!>Z!'2$+,!2&(),.428(4+),!4$.P!.)!$!0$23&!
$*)8,4!)1!0$'&0&-!42$+,+,3!-$4$!*$5!'&!2&Y8+2&-!4)!$(#+&%&!
3&,&2$0+F$4+),@!A&!1)(8.!),!4#&!2&(),.428(4+),!/)24+),!)1!4#&!
/$2(&00$4+),!/+/&0+,&!K#+3#0+3#4&-!+,!7#+4&6!a+3@!RL!.+,(&!4#&!
/2)U&(4+),!$,-!4#&!.$*/0+,3!.4&/.!$2&!2&0$4+%&05!.+*/0&!/2&"!
$,-!/).4"/2)(&..+,3!)/&2$4+),.!$,-!-)!,)4!2&Y8+2&!0&$2,+,3@!
! `,!4#+.!&=/&2+*&,46!7&!0&%&2$3&!IBZ!7+4#!$!0$23&!-$4$.&4!
)1!8,0$'&0&-!#8*$,!()24+($0!*)-&0.!4)!0&$2,!$!*)2&!2)'8.4!
/$2(&00$4+),!*)-&0@!a)2!0$'&0&-!-$4$6!7&!8.&!4#&!X+,-')330&!
-$4$.&4!9ND;!(),4$+,+,3!:T:!.8'U&(4.6!$,,)4$4&-!+,!>:!()24+($0!
2&3+),.@!A&!$0.)!8.&!4#&!p8*$,!\),,&(4)*&!f2)U&(4!Kp\fL!
9NH;!+*$3&!1)2!42$+,+,3!%+$!IBZ@!A&!./0+4!4#&!p\f!-$4$!+,4)!
42$+,!$,-!4&.4+,3!.8(#!4#$4!4#&2&!+.!,)!)%&20$/!+,!4#&!.8'U&(4.!
'&47&&,!42$+,+,3!+*$3&.!1)2!4#&!2&(),.428(4)2!$,-!42$+,+,3!
0$'&0.!1)2!4#&!-&,)+.&2@!A&!/+(P!4#&!,&47)2P!)1!4#&!0$.4!&/)(#!
1)2!$00!42$+,+,3!*&4#)-.!&=(&/4!4#&!18005!.8/&2%+.&-!,,B"C&4!
'$.&0+,&6!7#&2&!4#&!,&47)2P!)1!4#&!'&.4!&/)(#!+.!(#).&,@!I&&!
a+3@!>!1)2!+..8&.!7+4#!8.+,3!4#&!0$.4!&/)(#!,,B"C&4!*)-&0@!
! A&! 8.&!4#&!K.80($0!*$/6!7#+4&!*$M&2!(82%$482&6!+,d$4&-!
.821$(&!(82%$482&L!-$4$6!3&,&2$4&-!8.+,3!a2&&I821&2!9NN;6!$.!
+,/846!$//05!$!RQq!A+,.)2+,3!4)!4#&!7#+4&!*$M&2!(82%$482&!
4)!2&*)%&!)840+&2.!$,-!,)2*$0+F&!&$(#!+*$3&!(#$,,&0!4)!8,+4!
.4$,-$2-!-&%+$4+),!$,-!F&2)!*&$,@!A&!$83*&,4!+*$3&!-$4$!
8.+,3!<Z!2$,-)*!&0$.4+(!-&1)2*$4+),@!O#&!-&1)2*$4+),!E&0-!
+.!3&,&2$4&-!'5!.*))4#05!+,4&2/)0$4+,3!-+./0$(&*&,4!%&(4)2.!
-&E,&-!),!$!32+-!7+4#!16×16!(&00.@!A&!42$+,!$00!*)-&0.!1)2!$!
*$=+*8*!)1!<TT6TTT!+4&2$4+),.!$,-!$!'$4(#!.+F&!)1!:@!A&!8.&!
Fig. 9. Cortical parcellation pipeline. Given a cortical mesh model (a), we project it to a 2D image (b) and parcellate it to produce a parcellation map
(c). The parcellation map is then sampled onto the mesh model to produce the final parcellation. Since the projection and the sampling are fixed
operations, we treat parcellation as a 2D visual reconstruction problem (highlighted in white).
!
E1-F4,8!
E91,433984!
(a) Cortical Mesh (b) 2D Projection (c) 2D Delineation (d) Parcellation
G92?34!
Region
nnU-Net
Post-denoise
Ensembling
SUD (ours)
1 image 2 images 5 images 10 images
1 image 2 images 5 images 10 images
Fig. 10. Dice and 95HD statistics (test) for the cortical parcellation task. The top row plots the statistics for different training regimes (nnU-Net, with
additional denoising, temporal ensembling, and SUD), with each group of three box plots representing the statistics on the MindBoggle + HCP (90
test subjects), MindBoggle (40 test subjects) and HCP (50 test subjects). The bottom row plots the Dice overlap of the individual cortical regions
for the HCP dataset when trained using one labeled image. See Fig. 11 (top left) for a diagram of the cortical regions.
8 IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. 0, NO. 0, DECEMBER 2023
4#&!e-$*!)/4+*+F&2!7+4#!4#&!0&$2,+,3!2$4&!)1!0.00016!7&+3#4!
-&($5!)1!0!$,-!&/.+0),!)1!10
−
@!\#$,,&0!-2)/)84!7+4#!P&&/!
/2)'$'+0+45!0.95!+.!8.&-!4)!2&-8(&!)%&2EM+,3!4#$4!*$,+1&.4.!
&./&(+$005!&32&3+)8.05!+,!4#+.!()24+($0!/$2(&00$4+),!/2)'0&*@!
! a+3@!:T!/0)4.!4#&!*&$,!Z+(&!$,-!p$8.-)2J!-+.4$,(&!)1!4#&!
2&(),.428(4+),.!/2&-+(4&-!'5!B"C&4!8,-&2!-+J&2&,4!42$+,+,3!
2&3+*&.6!.8/&2%+.&-!'$.&0+,&6!.4)(#$.4+(!&,.&*'0+,36!$.!7&00!
$.!IBZ6!$(2)..!:6!<6!D!$,-!:T!0$'&0&-!+*$3&.@!A#&,!:!0$'&0&-!
+*$3&!+.!8.&-!1)2!42$+,+,36!IBZ!B"C&4!+*/2)%&.!*&$,!Z+(&!
$,-!p$8.-)2J!Z+.4$,(&.!)%&2!.4)(#$.4+(!&,.&*'0+,3!'5!T@T?!
$,-!:Q!/+=&0.!KrD?qL6!2&./&(4+%&05@!X).4!)1!4#&!+*/2)%&*&,4!
()*&.!12)*!4#&!p\f!-$4$.&46!1)2!7#+(#!,)!0$'&0&-!42$+,+,3!
+*$3&.!$2&!$%$+0$'0&@!b&3+),"7+.&6!4#&!0$23&.4!+*/2)%&*&,4.!
$2&!12)*!N!K+,1&2+)2!4&*/)2$0!0)'&L6!<N!K.8/&2+)2!/$2+&4$0L!$,-!
<R!K.8/2$"*$23+,$0L!7+4#!Z+(&!3$+,.!2$,3+,3!0.2– 0.3@!a+3@!::!
.#)7.!/$2(&00$4+),.!/2&-+(4&-!8,-&2!&$(#!)1!4#&!.8/&2%+.+),!
2&3+*&.@!O#&!'$.&0+,&!B"C&4!/2&-+(4+),.!&=#+'+4!$,!&=(&..+%&!
$*)8,4!)1!2&(),.428(4+),!,)+.&!'84!$2&!.)*&7#$4!+*/2)%&-!
'5!/).4"-&,)+.+,3@!I4)(#$.4+(!&,.&*'0+,3!*+..&.!$00!+,1&2+)2!
4&*/)2$0!0)'&.!+,!+4.!/$2(&00$4+),.!7#+0&!IBZ!/2&-+(4.!4#&*!
$((82$4&05!7+4#)84!$,5!)'%+)8.!2&(),.428(4+),!,)+.&@!
4.2 Anatomical Brain Reconstruction (3D)
e,$4)*+($0!2&(),.428(4+),!)1!'2$+,!Xb!.($,.!+.!$,)4#&2!4$.P!
)1!+*/)24$,(&!+,!,&82)+*$3+,3!$.!+4!&,$'0&.!%)08*&42+(!$,-!
*)2/#)0)3+($0!$,$05.&.!)1!4#&!'2$+,!+,!#&$04#5!$,-!-+.&$.&-!
/)/80$4+),.!9NQ;@!p)7&%&26!4#&!>Z!%)08*&42+(!,$482&!)1!Xb!
.($,.!2&,-&2.!*$,8$0!-&0+,&$4+),!)1!4#&!$,$4)*+($0!2&3+),.!
(#$00&,3+,36!,&(&..+4$4+,3!*$,5!-$5.!)1!0$'&0+,3!&J)24.!/&2!
+*$3&@!X)2&)%&26!4#&!-+J&2&,(&.!+,!Xb`!.($,,&2.!$,-!4#&+2!
.($,,+,3!/2)4)()0.!($,!#$*/&2!4#&!2&8.&!)1!$!2&(),.428(4+),!
,&47)2P!42$+,&-!),!$!/$24+(80$2!-$4$.&4!)2!+*$3&!(),42$.4@!`,!
4#+.!($.&6!7&!($,!42$+,!$!-&,)+.&2!),!/2&%+)8.!2&(),.428(4+),!
&=$*/0&.!$,-!.&*+".8/&2%+.&!4#&!2&(),.428(4)2!%+$!IBZ@!
! `,!4#+.!&=/&2+*&,46!7&!$//05!IBZ!4)!4#&!42$+,+,3!)1!$!>Z!
'2$+,!2&(),.428(4+),!*)-&0!4#$4!+.!*)2&!2)'8.4!4)!%$2+$4+),.!
+,!4#&!Xb`!.&M+,3@!A&!8.&!),05!),&!0$'&0&-!+*$3&!.+*+0$205!
4)!4#&!),&".#)4!2&(),.428(4+),!$//2)$(#!)1!i#$)!et al.!9:>;!'84!
2$4#&2!4#$,!+*/2)%&!3&,&2$0+F$4+),!'5!0&$2,+,3!4#&!)/4+*$0!
-$4$!$83*&,4$4+),!$.!+,!9:>;6!7&!$(#+&%&!4#+.!'5!42$+,+,3!),!
8,0$'&0&-!.($,.GeS`Zc":6!"<6!eZpZ<TT6!_If6!X\`\6!Bs!
S`V6!\VSbc!$,-!ffX`G$.!8.&-!+,!9NR;@!b&1&2!4)!O$'0&!S<!+,!
e//&,-+=!S!1)2!4#&!&=$(4!'2&$P-)7,@!e00!+*$3&.!#$%&!'&&,!
2)4$4&-!+,4)!$!()**),!)2+&,4$4+),@!A&!42$+,!4#&!-&,)+.&2!),!
:TTT!0$'&0!*$/.!3&,&2$4&-!8.+,3!a2&&I821&2!9NN;@!I+*+0$2!4)!
4#&!()24+($0!/$2(&00$4+),!&=/&2+*&,46!4#&!:TTT!0$'&0!*$/.!-)!
,)4!#$%&!.8'U&(4!)%&20$/.!7+4#!$,5!)1!4#&!+*$3&.!8.&-6!$,-!
4#&!%$0+-$4+),!-$4$!+.!8.&-!),05!1)2!$..&..+,3!-+%&23&,(&!)2!
/)4&,4+$0!)%&2EM+,3!)1!4#&!.8/&2%+.&-!,,B"C&4!'$.&0+,&@!A&!
)/4+*+F&!4#&!$83*&,4$4+),!K.#)7,!+,!O$'0&!S:L!$,-!42$+,!1)2!
<TT6TTT!+4&2$4+),.!8.+,3!4#&!Z+(&!0)..!8,-&2!$00!2&3+*&.@!A&!
8.&!+-&,4+($0!)/4+*+F$4+),!.&M+,3.!$.!+,!()24+($0!/$2(&00$4+),@!
! a+3@!:<!/0)4.!4#&!-+.42+'84+),!)1!*&$,!Z+(&!$,-!p$8.-)2J!
-+.4$,(&.!)1!B"C&4!2&(),.428(4+),.!/2)-8(&-!8,-&2!-+J&2&,4!
.8/&2%+.+),!2&3+*&.6!(),-+4+),&-!),!.8'U&(4!.&4.6!$.!7&00!$.!
),!'2$+,!.428(482&.@!e00!.8/&2%+.+),!2&3+*&.!/2)-8(&!.+*+0$2!
$((82$(5!2&.804.!KRDpZ!+,!/$24+(80$2L!),!S8(P,&2>R!.+,(&!4#&!
0$'&0&-!42$+,+,3!+*$3&!+.!.)82(&-!12)*!4#+.!.&4@!O#&!$((82$(5!
3$/!1)2!)4#&2!.8'U&(4!.&4.6!&./&(+$005!eS`Zc<!$,-!eZpZ6!+.!
(),.+-&2$'05!0$23&26!7#&2&!4#&!+*/2)%&*&,4!+,!*&-+$,!Z+(&!
-8&!4)!IBZ!+.!rT@TD@!O)!.&&!4#&!+*/2)%&*&,4!+,!Z+(&!$,-!RD!
p$8.-)2J!-+.4$,(&!$4!4#&!+,-+%+-8$0!.8'U&(4!0&%&06!7&!/0)4!+,!
a+3@!:>6!#+.4)32$*.!)1!Z+(&!$,-!RDpZ!(#$,3&!$(2)..!4#&!DTT!
4&.4!.8'U&(4.!'&47&&,!IBZ!$,-!&,.&*'0+,3!*&4#)-.6!$.!7&00!
$.!IBZ!$,-!/).4"-&,)+.&-@!O#&!*&$,!Z+(&!+*/2)%&*&,4!+.!
.42+(405!/).+4+%&!7+4#!/&$P!+*/2)%&*&,4!)1!T@:D!7#+0&!0)..&.!
HCP 993675
!
!
!
!
!
!
!
!
!
!
FS reference [77]
nnU-Net [36] (0.7483)
Post-denoise [53] (0.793)
Ensembling (0.805)
SUD (ours) (0.874)
MMRR-
3T7T
-
2
-
2
Manual parcellation
nnU-Net [36] (0.7317)
Post-denoise [53] (0.797)
Ensembling (0.748)
SUD (ours) (0.800)
Fig. 11. Cortical parcellations (test) produced under different training regimes, shown together with FreeSurfer reference ones (mean Dice overlap
in parentheses are with respect to the reference). All training regimes use only one labeled image and share the same underlying nn-UNet
implementation. Post-denoise and SUD are based on the same denoiser. SUD produces the most visually accurate parcellations as well as the
highest mean Dice scores. See Fig. 10 for the overall statistics. Rendered using FreeView. Figure best viewed online.
YOUNG ET AL.: SUPERVISION BY DENOISING 9
+,!4#&!+,-+%+-8$0!*&$,!RDpZ!$2&!)'.&2%&-!.)*&4+*&.@!O#+.!
+.!.)*&7#$4!&=/&(4&-!.+,(&!)82!42$+,+,3!)'U&(4+%&!&=/0+(+405!
)/4+*+F&.!Z+(&@!O#&!RDpZ!+*/2)%&*&,4!+.!.4+00!$!.4$4+.4+($005!
.+3,+E($,4!),&!K = 10
−
6!A+0()=),k.!.+3,&-"2$,P!4&.4L@!!
! a+3@!:?!%+.8$0+F&.!4#&!>Z!$,$4)*+($0!'2$+,!2&(),.428(4+),.!
/2)-8(&-!'5!B"C&4!8,-&2!4#&!-+J&2&,4!42$+,+,3!2&3+*&.@!A&!
+,(08-&!2&(),.428(4+),.!/2)-8(&-!8.+,3!a2&&I821&2!9NN;@!A&!
(#)).&!.8'U&(4.!4#$4!'&.4!-&*),.42$4&!+*/2)%&*&,4.!-8&!4)!
IBZ!7+4#!$!Z+(&!3$+,!+,!4#&!2$,3&!0.06– 0.08@!e!*)2&!45/+($0!
Z+(&!3$+,!+.!$2)8,-!0.056!$.!+.!.&&,!+,!a+3@!:>!K0&14L@!I*$00&2!
.428(482&.!K#+//)($*/8.6!$*53-$0$6!&4(@L!$2&!%+.8$005!'&M&2!
2&(),.428(4&-6!$.!+.!4#&!0$4&2$0!%&,42+(0&6!7+4#!1&7&2!#)0&.!+,!
4#&!IBZ!2&(),.428(4+),.@!Z+(&!+*/2)%&*&,4.!+,!4#&!+,1&2+)2!
0$4&2$0!%&,42+(0&G$.!+,-+($4&-!'5!a+3@!:<G$2&!,)4!.)!(0&$205!
)'.&2%&-!+,!4#&!%+.8$0+F$4+),@!A&! ,)4&! 4#$4!4#&!Z+(&!.()2&.!
$2&!()*/84&-!$3$+,.4!4#&!aI!2&1&2&,(&!2&(),.428(4+),.!$,-!
.#)80-!'&!+,4&2/2&4&-!7+4#!($84+),G),05!4#&!#8*$,!&5&!+.!
$'0&!4)!3$83&!4#&!Y8$045!$(2)..!4#&!%$2+)8.!2&(),.428(4+),.@!
5 DISCUSSION
V82!&=/&2+*&,4$0!2&.804.!($,!'&!.8**$2+F&-!$.!1)00)7.m!
• `,!.8/&2%+.+),!2&3+*&.!7+4#!%&25!1&7!0$'&0&-!+*$3&.!K),&!
+,!4#&!($.&!)1!$,$4)*+($0!'2$+,!2&(),.428(4+),!$,-!()24+($0!
/$2(&00$4+),L6!IBZ!+*/2)%&.!4#&!2&(),.428(4+),!Z+(&!.()2&!
'5!D!/)+,4.!),!$%&2$3&!)%&2!.4)(#$.4+(!&,.&*'0+,3!$,-!'5!
:T!/)+,4.!$3$+,.4!4#&!18005!.8/&2%+.&-!'$.&0+,&@!O#&!RDpZ!
+.!2&-8(&-!'5!DTlNDq!)%&2!.4)(#$.4+(!&,.&*'0+,3@!
• a)2!>Z!'2$+,!2&(),.428(4+),6!4#&!Z+(&!+*/2)%&*&,4!-8&!
4)!IBZ!)%&2!.4)(#$.4+(!$%&2$3+,3!+.!$M2+'84&-!4)!.8'.&4.!
)1!+*$3&.!1)2!7#+(#!,)!0$'&0.!$2&!$%$+0$'0&!K&@3@6!eS`Zc<6!
D!/)+,4.L@!`*/2)%&*&,4!),!S8(P,&2>R6!7#&2&!4#&!0$'&0&-!
42$+,+,3!-$4$!7$.!.)82(&-!12)*6!+.!0&..!K:!/)+,4L@!
• f).4"-&,)+.+,3!$!/2&-+(4&-!2&(),.428(4+),!-)&.!,)4!5+&0-!
4#&!.$*&!Z+(&!+*/2)%&*&,4!$.!42$+,+,3!4#&!2&(),.428(4+),!
,&47)2P! 8,-&2! IBZ@! O#+.! +.! .+*+0$2! 4)! (0$..+($0! +,%&2.&!
/2)'0&*.!K&@3@!+*$3&!-&'0822+,3L6!7#&2&!$!,$+%&!.)084+),!
($,,)4!'&!-&,)+.&-!/).4"#)(!-8&!4)!+,%&2.+),!,)+.&@!
5.1 Extending SUD
c=4&,.+),.!4)!4&*/)2$0!&,.&*'0&.!$,-!*&$,!4&$(#&2!*)-&0.!
9HT;6!9H:;!+,%)0%&!42$+,+,3!*804+/0&!4&$(#&2!,&47)2P.!7#&2&!
4#&!7&+3#4.!$2&!+,+4+$0+F&-!-+J&2&,405@!O#&!)84/84!12)*!4#&!
4&$(#&2.!+.!4#&,!8.&-!$.!/.&8-)"4$23&4.!1)2!4#&!)4#&2!-82+,3!
42$+,+,3@!O2$+,+,3!*804+/0&!,&47)2P.!0&$-.!4)!$,!+,(2&$.&!+,!
4#&!42$+,+,3!4+*&!$,-!2&.)82(&.6!'84!4#&.&!+-&$.!*$5!.4+00!'&!
8.&180!1)2!/2)%+-+,3!'&M&2!.4)(#$.4+(!$%&2$3+,3!+,!IBZ!$,-!
/)4&,4+$005!184#&2!+*/2)%+,3!.&*+".8/&2%+.+),@!b$4#&2!4#$,!
&=#$8.4+%&05!+,%&.4+3$4&!4#&!)/4+*$0!,8*'&2!)1!,&47)2P.!4)!
42$+,!1)2!.4)(#$.4+(!$%&2$3+,36!#&2&!7&!1)(8.!+,.4&$-!),!4#&!
8.&180,&..!)1!./$4+$0!-&,)+.+,3!$.!$!.8/&2%+.+),!*&(#$,+.*@!!
! e-%$,(&-!+*$3&!$83*&,4$4+),!.42$4&3+&.!.8(#!$.!X+=B/!
9N<;6!$,-!\84X+=!9N>;!()*'+,&!47)!+*$3&.!'5!$--+4+),6!$,-!
.4+4(#+,36!2&./&(4+%&05@!O#&.&!/2)%+-&!$,!&$.5!7$5!4)!1824#&2!
+*/2)%&!4#&!-+%&2.+45!)1!0$'&0&-!-$4$!1)2!%+.8$0!(0$..+E($4+),!
K+@&@!,),"2&(),.428(4+),L!/2)'0&*.@!A#&2&$.!.+*/0&!+,4&,.+45!
$,-!3&)*&425"'$.&-!+*$3&!$83*&,4$4+),!.42$4&3+&.!.8[(&!
+,!)82!47)!2&(),.428(4+),!4$.P.6!$332&..+%&!$83*&,4$4+),!)1!
-$4$.&4.!($,!'&!8.&180!&./&(+$005!7+4#!4))!1&7!0$'&0&-!+*$3&.!
Fig. 12. Dice and 95HD statistics (test) for the brain reconstruction task, conditioned on individual datasets (top) and brain structures (bottom). We
plot the two metrics for the supervised nnU-Net baseline (trained on one labeled Buckner39 image), with additional post-denoising, stochastic
ensembling, and SUD (ours). See text for details on the individual datasets.
ABD ABD2 ADH COB GSP MCIC OAS PPMI UKB B39
ABD ABD2 ADH COB GSP MCIC OAS PPMI UKB B39
All WM Cort LatV InfLatV WM Cort Thalam Caud Putam Pallid 3rdV 4thV Stem Hippo Amyg Accum VDC
Cerebral
nnU-Net
Post-denoise
Ensembling
SUD (ours)
Improvement (Dice)
Improvement (95HD)
Fig. 13. Dice and 95HD improvements (test) due to SUD, relative to the
“stochastic ensembling” (blue) and “post-denoise” (green) methods. The
same nnU-Net model is used in all three cases. Plotted across 500 test
images. The Dice improvement is strictly positive.
10 IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. 0, NO. 0, DECEMBER 2023
4)! '))4.42$/! .&*+".8/&2%+.&-!0&$2,+,3@! p)7&%&26! $-$/4+,3!
X+=B/!$,-!\84X+=!4)!$,!+*$3&!2&(),.428(4+),!4$.P!+.!,)4!.)!
.42$+3#4"1)27$2-!$,-!7&!-&1&2!4#&+2!-+.(8..+),!4)!$!.&Y8&0@!!
5.2 Possible Limitations
e!()2&!/2&*+.&!)1!IBZ!+.!4#$4!0&$2,+,3!2&(),.428(4+),!/2+)2.!
K4#$4!+.6!42$+,+,3!$!-&,)+.&2L!+.6!+,!$!.&,.&6!&$.5!()*/$2&-!4)!
)/4+*+F+,3!4#&!0+P&0+#))-!K42$+,+,3!4#&!2&(),.428(4)2L@!I8(#!
$,!$..8*/4+),!+.!2&$.),$'0&!$,-!+*/0+&-!+,!*$,5!+*$3+,3!
$,-!%+.8$0!2&(),.428(4+),!4$.P.6!7#&2&!$!-&,)+.&2!+.!2&$-+05!
42$+,&-!),!/0&,45!)1!&=+.4+,3!2&(),.428(4+),.@!`,!4$.P.!7#&2&!
1&7!2&(),.428(4+),!&=$*/0&.!&=+.4!4)!'&3+,!7+4#6!0&$2,+,3!$!
/2+)2!)%&2!2&(),.428(4+),.!($,!'&!-+[(804!)7+,3!4)!$!3&,&2$0!
0$(P!)1!P,)70&-3&!4#$4!($,!'&!]-+.4+00&-^!+,4)!$!/2+)2@!V82!
$..8*/4+),!)1!].8[(+&,405!*$,5!$,-!%$2+&-!2&(),.428(4+),!
&=$*/0&.^!+.!1$2!12)*!'&+,3!8,+Y8&!4)!)82!7)2P6!$,-!.&2%&.!
$.!4#&!'$.+.!+,!()*/84$4+),$0!/#)4)32$/#5!9:;6!9<;6!+,!7#+(#!
/2&42$+,&-!,$482$0!+*$3&!-&,)+.&2.!$2&!8.&-!$,-!'+)*&-+($0!
+*$3+,3!9NR;6!9N:;6!9D>;!7#&2&!*$,5!2&(),.428(4+),!&=$*/0&.!
1$(+0+4$4&!-&,)+.+,3!$,-!3&,&2$4+),!)1!2&(),.428(4+),.@!
6 CONCLUSION
`,!0&$2,+,3"'$.&-!+*$3&!2&(),.428(4+),6!.&*+".8/&2%+.+),!)2!
+*/2)%+,3!4#&!3&,&2$0+F$4+),!)1!$!2&(),.428(4+),!,&47)2P!'5!
$--+4+),$005!0&$2,+,3!12)*!8,0$'&0&-!+*$3&.!+.!$!/2)'0&*!)1!
4#&)2&4+($0!$,-!/2$(4+($0!+,4&2&.4@!`,!4#+.!7)2P6!7&!/2)/).&-!
.8/&2%+.+),!'5!-&,)+.+,3!KIBZL6!&,$'0+,3!.&*+".8/&2%+.+),!
8.+,3! 4#&!2&(),.428(4+),! ,&47)2Pk.!)7,! K-&,)+.&-L! )84/84!
$.!/.&8-)"4$23&4.@!IBZ!+.!(0).&05!2&0$4&-!4)!2&380$2+F$4+),!'5!
-&,)+.+,3!KbcZL!8.&-!1)2!.)0%+,3!+,%&2.&!/2)'0&*.6!7#+(#!
#+3#0+3#4.!$3$+,!4#$4!*$,5!/2)'0&*.!+,!+*$3+,3!$,-!%+.+),!
+,(08-+,3!.8/&2%+.+),!),&.6!($,!'&!&,.(),(&-!+,!$!-&,)+.+,3!
12$*&7)2P@!B,-&2!4#&!IBZ!12$*&7)2P6!.8(#!4&(#,+Y8&.!$.!
4&*/)2$0!&,.&*'0+,3!$,-!*&$,!4&$(#&2!($,!'&!2&+,4&2/2&4&-!
$.!$!1)2*!)1!IBZ!4#$4!-)&.!,)4!+,%)0%&!$!./$4+$0!-&,)+.&2@!!
! _+%&,!4#&!2$4#&2!3&,&2+(!,$482&!)1!IBZ6!7&!'&0+&%&!IBZ!
$,-!/)4&,4+$005!+4.!*&$,!4&$(#&2!%$2+$,4!($,!.&2%&!$.!$!8.&180!
4))0!1)2!.&*+".8/&2%+.&-!2&(),.428(4+),!&./&(+$005!+1!8.&-!+,!
4$,-&*!7+4#!*)2&!$-%$,(&-!-$4$!$83*&,4$4+),!4&(#,+Y8&.!
$,-!4$.P"./&(+E(!.42$4&3+&.!42$+,+,3!-&,)+.&2.@!`4!+.!)82!#)/&!
4#$4!)82!7)2P!7+00!'&!'2)$-05!$//0+($'0&!4)!$00!2&(),.428(4+),!
/2)'0&*.!1$(&-!'5!+*$3+,3!$,-!%+.+),!2&.&$2(#&2.6!,)4!U8.4!
4#).&!./&(+E($005!*&,4+),&-!+,!4#+.!7)2P@!
ACKNOWLEDGMENTS
O#+.!7)2P!+.!/2+*$2+05!.8//)24&-!4#&!C`p!Sbe`C!`,+4+$4+%&!
Kba:Xp:<>:RDL@!e--+4+),$0!.8//)24!+.!/2)%+-&-!'5!4#&!C`p!
KBT:Xp::NT<>6!bT:cST<><Q:6!bT:cSTTHNDQ6!bT:cST:RRDH6!
bT:e_TH?T<N6!ba:Xp:<:QQD6!bT:CITQ>D>?6!bT:CI:TDQ<T6!
f?:cST:DRT<L6!ebBs!K`b_<T:Re"TT>L6!cb\!KI4$24+,3!_2$,4!
HNNHRNL@!ca!7$.!18,-&-!'5!4#&!a80'2+3#4"\VC`\cO!h+.+4+,3!
b&.&$2(#&2!f2)32$*!$,-!#&!$(P,)70&-3&.!4#&!.8//)24!)1!4#&!
BCW!$,-!eCf\tO@!\eX!7$.!.8//)24&-6!+,!/$246!'5!eaVIb!
t)8,3!`,%&.4+3$4)2!f2)32$*!e7$2-!aeRDDT"<<":"T<TQ@!
ABIDEII NYU 1 29254 Session 1
FS reference [77]
nnU-Net [36] (0.761)
Post-denoise [53] (0.792)
Ensembling (0.773)
SUD (ours) (0.851)
COBRE 0040113
FS reference [77]
nnU-Net [36] (0.681)
Post-denoise [53] (0.708)
Ensembling (0.756)
SUD (ours) (0.826)
Fig. 14. Brain structure reconstructions (test) produced under different training regimes, shown together with reference FreeSurfer ones (mean Dice
overlap in parentheses are with respect to the reference). Cortex and white matter not shown for clarity. All training regimes use only one labeled
image and share the same underlying nn-UNet implementation. Post-denoise and SUD use the same auto-encoder denoiser. SUD produces the
most visually accurate reconstructions as well as the highest mean Dice scores. See Fig. 11 for the overall statistics. Rendered using FreeView.
YOUNG ET AL.: SUPERVISION BY DENOISING 11
REFERENCES
! H"I! GJ!GJ!KL9.M!NJ!G5.=91M!NJ!O--2<.98L9.M!PJ!N441919QL9/9.M!9.=!KJ!
R<819M! ST398U48V! W-X91=6! EL-8-1493<68<,! G,4.4! Y4,-.6815,8<-.!
T1-2! Z4.63466! R49651424.86M[! IEEE Trans. Pa\ern Anal. Mach.
Intell.M!/-3J!%%M!.-J!%M!??J!"*$%0"*%)M!P?1J!#+##J!
! H#I! \J!PJ!R4]341M!^J!_B-29M!`J!E4.QM!9.=!aJ!b4]684<.M!Sc44?!d?8<,6!
7-1! G<.Q34eGL-8! ^<QLec@.92<,eY9.Q4! _29Q<.QM[! <.! Proc. CVPRM!
#+#+M!??J!"$(&0"$)&J!
! H$I! OJ! O9<M! fJ! `9.QM! `J!Z<M!`J!gL9.QM! UJ! E<3391M!9.=!PJ!dh,9.M! Sc44?!
3491.<.Qe4.9i34=! /<18593! L<68-3-Q<,93! 689<.<.Q! -7! i<-3-Q<,93!
692?346M[!Light Sci. Appl.M!/-3J!"#M!.-J!"M!?J!&(M!R91J!#+#$J!
! H%I! PJ!K9319M!NJ!W9929h@9.M!GJ!KJ!Y9-M!KJ!N4.B9891929.M!YJ!Y96B91M!9.=!
PJ! K9=92i<M! Sc44?! E-391<h98<-.! \546! 7-1! W19.6?914.8! diF4,8!
G4Q24.898<-.M[!<.!Proc. CVPRM!#+#+M!??J!)'+#0)'""J!
! H&I! dJ! Y-..4i41Q41M! EJ! T<6,L41M! 9.=! WJ! O1-:M! SjeU48V! ,-./-358<-.93!
.48X-1B6!7-1! i<-24=<,93! <29Q4!64Q24.898<-.M[!<.! Proc. MICCAIM!
#+"&M!??J!#$%0#%"J!
! H'I! `J!Z5-M!ZJ!gL4.QM!WJ!a59.M!kJ!`5M!9.=!`J!`9.QM!SW9B<.Q!9!,3-641!3--B!
98!=-29<.!6L<78V!,984Q-1@e34/43!9=/41691<46!7-1!6429.8<,6!,-.6<684.8!
=-29<.!9=9?898<-.M[!<.!Proc. CVPRM!#+"*M!??J!#&+(0#&"'J!
! H(I! OJ! ^J! R4.h4! et al.M! SWL4! 2538<2-=93! i19<.! 852-1! <29Q4!
64Q24.898<-.!i4.,L291B!;OYPWG>M[!IEEE Trans. Med. ImagingM!/-3J!
$%M!.-J!"+M!??J!"**$0#+#%M!d,8J!#+"&J!
! H)I! dJ! O41.91=! et al.M! Sc44?! 3491.<.Q! 84,L.<l546! 7-1! 958-298<,! RY_!
,91=<9,! 2538<e6815,85146! 64Q24.898<-.! 9.=! =<9Q.-6<6V! _6! 8L4!
?1-i342! 6-3/4=mM[!IEEE Trans. Med. ImagingM! /-3J! $(M! .-J! ""M! ??J!
#&"%0#&#&M!U-/J!#+")J!
! H*I! GJ! Z4,341,! et al.M! Sc44?! 3491.<.Q! 7-1! 64Q24.898<-.! 56<.Q! 9.! -?4.!
391Q4e6,934! =989648! <.! #c! 4,L-,91=<-Q19?L@M[! IEEE Trans. Med.
ImagingM!/-3J!$)M!.-J!*M!??J!#"*)0##"+M!G4?J!#+"*J!
!H"+I! gJ!gL-5M!RJ!RJ!YJ!G<==<l544M!UJ!W9Fi9BL6LM!9.=!kJ!Z<9.QM!SjU48nnV!
14=46<Q.<.Q! 6B<?! ,-..4,8<-.6! 8-! 4:?3-<8! 2538<6,934! 74985146! <.!
<29Q4!64Q24.898<-.M[!IEEE Trans. Med. ImagingM!/-3J!$*M!.-J!'M!??J!
")&'0")'(M!k5.J!#+#+J!
!H""I! kJ! b9.Q! et al.M! Sc44?! L<QLe146-358<-.! 14?1464.898<-.! 3491.<.Q! 7-1!
/<6593!14,-Q.<8<-.M[!IEEE Trans. Pa\ern Anal. Mach. Intell.M!/-3J!%$M!
.-J!"+M!??J!$$%*0$$'%M!d,8J!#+#"J!
!H"#I! NJ! G9.=7-18M! KJ! `9.M! EJ! kJ! E<,BL91=8M! 9.=! YJ! RJ! G522416M! Sc989!
95Q24.898<-.!56<.Q!Q4.4198</4!9=/41691<93!.48X-1B6!;\@,34aPU>!
8-! <2?1-/4! Q4.4193<h9i<3<8@! <.! \W! 64Q24.898<-.! 896B6M[! Sci. Rep.M!
/-3J!*M!.-J!"M!?J!"'))%M!U-/J!#+"*J!
!H"$I! PJ!gL9-M!aJ!O939B1<6L.9.M!TJ!c519.=M!kJ!NJ!a5o9QM!9.=!PJ!NJ!c93,9M!
Sc989! 95Q24.898<-.! 56<.Q! 3491.4=! 819.67-1298<-.6! 7-1! -.4e6L-8!
24=<,93!<29Q4!64Q24.898<-.M[!<.!Proc. CVPRM!#+"*M!??J!)&%$0)&&$J!
!H"%I! `J!gL-5M!ZJ!f<4M!pJ!KJ!T<6L29.M!9.=!PJ!ZJ!`5<334M!Sc44?!65?41/<6<-.!
7-1!?9.,1498<,!,@68!64Q24.898<-.!<.!9i=-2<.93!\W!6,9.6M[!<.!Proc.
MICCAIM!#+"(M!??J!###0#$+J!
!H"&I! cJ!K91<2<!9.=!GJ!pJ!G93,5=49.M!SY4=5,<.Q!8L4!^956=-1q!=<689.,4!
<.! 24=<,93! <29Q4! 64Q24.898<-.! X<8L! ,-./-358<-.93! .45193!
.48X-1B6M[!IEEE Trans. Med. ImagingM!/-3J! $*M! .-J!#M!??J! %**0&"$M!
T4iJ!#+#+J!
!H"'I! kJ! \L-4M!GJ!Z44M! 9.=! ^J!GL<2M!SPo4.8<-.ei964=! =1-?-58!39@41!7-1!
X49B3@! 65?41/<64=! 6<.Q34! -iF4,8! 3-,93<h98<-.! 9.=! 6429.8<,!
64Q24.898<-.M[!IEEE Trans. Pa\ern Anal. Mach. Intell.M!/-3J!%$M!.-J!
"#M!??J!%#&'0%#("M!c4,J!#+#"J!
!H"(I! gJ!Z<5!et al.M!SG8@34!,511<,5352!3491.<.Q!7-1!1-i568!24=<,93!<29Q4!
64Q24.898<-.M[!<.!Proc. MICCAIM!#+#"M!??J!%&"0%'+J!
!H")I! kJ!\3-5QLM!UJ!O@1.4M!_J!dB65hM!NJ!PJ!g<2241M!kJ!PJ!G,L.9i43M!9.=!PJ!
K<.QM!SP!8-?-3-Q<,93!3-66!75.,8<-.!7-1!=44?e3491.<.Q!i964=!<29Q4!
64Q24.898<-.! 56<.Q! ?416<684.8! L-2-3-Q@M[! IEEE Trans. Pa\ern
Anal. Mach. Intell.M!??J!"0"M!#+#+J!
!H"*I! NJ!\L415B51<M!NJ!K5291!OJaJM!YJ!O939M!9.=!NJ!R-.Q9M!Sc44?!148<.93!
<29Q4!64Q24.898<-.!X<8L!14Q5391<h98<-.!5.=41!Q4-2481<,!?1<-16M[!
IEEE Trans. Image Process.M!/-3J!#*M!??J!#&�#&'(M!#+#+J!
!H#+I! GJ!G9.B919.919@9.9.M!`J!O939F<M!PJ!k9<.M!GJ!UJ!Z<2M!9.=!YJ!\L4339??9M!
SZ491.<.Q! 71-2! 6@.8L48<,! =989V! 9==1466<.Q! =-29<.! 6L<78! 7-1!
6429.8<,!64Q24.898<-.M[!<.!Proc. CVPRM!#+")M!??J!$(�$('"J!
!H#"I! `J! \L4.M! bJ! Z<M! fJ! \L4.M! 9.=! ZJ! NJ! a--3M! SZ491.<.Q! 6429.8<,!
64Q24.898<-.!71-2!6@.8L48<,! =989V!9!Q4-2481<,933@!Q5<=4=! <.?58e
-58?58!9=9?898<-.!9??1-9,LM[!<.!Proc. CVPRM!#+"*M!??J!")%"0")&+J!
!H##I! UJ! G1</9689/9M! aJ! ^<.8-.M! PJ! K1<hL4/6B@M! _J! G586B4/41M! 9.=! YJ!
G939BL58=<.-/M! Sc1-?-58V! 9! 6<2?34! X9@! 8-! ?14/4.8! .45193!
.48X-1B6!71-2!-/41ro<.QM[!J. Mach. Learn. Res.M!/-3J!"&M!.-J!&'M!??J!
"*#*0"*&)M!#+"%J!
!H#$I! GJ! Z9<.4! 9.=! WJ!P<39M! SW42?-193! 4.642i3<.Q! 7-1! 642<e65?41/<64=!
3491.<.QM[!<.!Proc. ICLRM!#+"(J!
!H#%I! gJ! R<1<BL919F<! 9.=! aJ! ^9291.4LM! SG891! 6L9?4! ?1<-1! <.! 7533@!
,-./-358<-.93! .48X-1B6! 7-1! 6B<.! 346<-.! 64Q24.898<-.M[! <.! Proc.
MICCAIM!#+")M!??J!($(0(%&J!
!H#&I! ZJ! _J! Y5=<.M! GJ! d6L41M! 9.=! pJ! T9842<M! SU-.3<.491! 8-893! /91<98<-.!
i964=!.-<64!142-/93!93Q-1<8L26M[!Phys. Nonlinear Phenom.M!/-3J!'+M!
.-J!"M!??J!#&*0#')M!U-/J!"**#J!
!H#'I! KJ!O14=<46M!KJ!K5.<6,LM!9.=!WJ!E-,BM!SW-893!Q4.4193<h4=!/91<98<-.M[!
SIAM J. Imaging Sci.M!/-3J!$M!.-J!$M!??J!%*#0&#'M!k9.J!#+"+J!
!H#(I! `J!Y-29.-M!RJ!p39=M!9.=!EJ!R<39.791M!SWL4!3<o34!4.Q<.4!8L98!,-53=V!
14Q5391<h98<-.!i@!=4.-<6<.Q!;Ypc>M[!SIAM J. Imaging Sci.M!/-3J!"+M!
.-J!%M!??J!")+%0")%%M!k9.J!#+"(J!
!H#)I! pJ! WJ! Y44L-168! 9.=! EJ! G,L.<841M! SY4Q5391<h98<-.! i@! =4.-<6<.QV!
,391<r,98<-.6! 9.=! .4X! <.841?14898<-.6M[! IEEE Trans. Comput.
ImagingM!/-3J!&M!.-J!"M!??J!�'(M!R91J!#+"*J!
!H#*I! GJ!^J!\L9.M!fJ!b9.QM!9.=!dJ!PJ!p3Q4.=@M!SE35Qe9.=e?39@!PcRR!
7-1!<29Q4!1468-198<-.V!r:4=e?-<.8!,-./41Q4.,4!9.=!9??3<,98<-.6M[!
IEEE Trans. Comput. ImagingM!/-3J!$M!.-J!"M!??J!)%0*)M!R91J!#+"(J!
!H$+I! GJ!NJ!N4.B989B1<6L.9.M!\J!PJ!O-529.M!9.=!OJ!b-L3i41QM!SE35Qe9.=e
?39@! ?1<-16! 7-1! 2-=43! i964=! 14,-.6815,8<-.M[! <.! Proc. GlobalSIPM!
#+"$M!??J!*%&0*%)J!
!H$"I! GJ! G144L91<! et al.M! SE35Qe9.=e?39@! ?1<-16! 7-1! i1<QL8! r43=! 434,81-.!
8-2-Q19?L@! 9.=! 6?9164! <.841?-398<-.M[! IEEE Trans. Comput.
ImagingM!/-3J!#M!.-J!%M!??J!%+)0%#$M!c4,J!#+"'J!
!H$#I! GJ! a5-M! gJ! `9.M! KJ! gL9.QM! bJ! g5-M! 9.=! ZJ! gL9.QM! SW-X91=!
,-./-358<-.93!i3<.=!=4.-<6<.Q!-7!1493!?L-8-Q19?L6M[!<.!Proc. CVPRM!
#+"*M!??J!"("#0"(##J!
!H$$I! fJ! Z<M! ZJ! `5M! ^J! \L4.M! \JebJ! T5M! ZJ! f<.QM! 9.=! EJePJ! ^4.QM!
SW19.67-1298<-.e,-.6<684.8! 6437e4.642i3<.Q! 2-=43! 7-1!
642<65?41/<64=!24=<,93!<29Q4!64Q24.898<-.M[!IEEE Trans. Neural
Netw. Learn. Syst.M!/-3J!$#M!.-J!#M!??J!&#$0&$%M!T4iJ!#+#"J!
!H$%I! aJ! O-186-/9M! TJ! c5i-68M! ZJ! ^-Q4X4QM! _J! K981929=-6M! 9.=! RJ! =4!
O15<F.4M! SG42<e65?41/<64=! 24=<,93! <29Q4! 64Q24.898<-.! /<9!
3491.<.Q! ,-.6<684.,@! 5.=41! 819.67-1298<-.6M[! <.! Proc. MICCAIM!
#+"*M!??J!)"+0)")J!
!H$&I! PJ! W91/9<.4.! 9.=! ^J! N93?-39M! SR49.! 849,L416! 914! i4o41! 1-34!
2-=436V! X4<QL8e9/419Q4=! ,-.6<684.,@! 891Q486! <2?1-/4! 642<e
65?41/<64=!=44?!3491.<.Q!1465386M[!<.!Proc. NIPSM!#+"(M!/-3J!$+J!
!H$'I! TJ!_64.644M!EJ!TJ!k94Q41M!GJ!PJ!PJ!K-L3M!kJ!E484164.M!9.=!KJ!^J!R9<41e
^4<.M! S..jeU48V! 9! 6437e,-.rQ51<.Q! 248L-=! 7-1! =44?! 3491.<.Qe
i964=!i<-24=<,93!<29Q4!64Q24.898<-.M[!Nat. MethodsM!/-3J!")M!.-J!
#M!??J!#+$0#""M!T4iJ!#+#"J!
!H$(I! EJ!K1sL4.itL3!9.=!NJ!K-385.M!Spu,<4.8!.-.3-,93!14Q5391<h98<-.!7-1!
-?8<,93!v-XM[!<.!ECCVM!#+"#M!??J!$&'0$'*J!
!H$)I! YJ! Y9.783M! KJ! O14=<46M! 9.=! WJ! E-,BM! SU-.e3-,93! 8-893! Q4.4193<h4=!
/91<98<-.!7-1!-?8<,93!v-X!468<298<-.M[!<.!Proc. ECCVM!#+"%M!??J!%$*0
%&%J!
!H$*I! kJ!WJ!O911-.!9.=!OJ!E--34M!SWL4!7968!i<3984193!6-3/41M[!<.!Proc. ECCVM!
#+"'M!/-3J!**+(M!??J!'"(0'$#J!
!H%+I! kJ!WJ!O911-.M!PJ!P=926M!`J!GL<LM!9.=!\J!^41.9.=4hM!ST968!i<3984193e
6?9,4!68414-!7-1!6@.8L48<,!=47-,56M[!<.!CVPRM!#+"&M!??J!%%''0%%(%J!
!H%"I! GJ!_J!`-5.QM!OJ!a<1-=M!9.=!cJ!W95i29.M!ST968!-?8<,93!v-X!4:819,8<-.!
71-2! ,-2?14664=! /<=4-M[! IEEE Trans. Image Process.M! /-3J! #*M! ??J!
'%+*0'%#"M!#+#+J!
!H%#I! kJ! \L4.M! PJ! P=926M! UJ! b9=LX9M! 9.=! GJ! bJ! ^96<.-qM! SO<3984193!
Q5<=4=! 5?692?3<.QM[! ACM Trans GraphM! /-3J! $&M! .-J! 'M! ?J! #+$V"e
#+$V)M!U-/J!#+"'J!
!H%$I! EJ! E41-.9! 9.=! kJ! R93<BM! SG,934e6?9,4! 9.=! 4=Q4! =484,8<-.! 56<.Q!
9.<6-81-?<,!=<q56<-.M[!IEEE Trans. Pa\ern Anal. Mach. Intell.M!/-3J!
"#M!.-J!(M!??J!'#*0'$*M!k53J!"**+J!
!H%%I! OJ! T<6,L3! 9.=! pJ! ZJ! G,LX91]M! SP=9?8</4! .-.3-,93! r3841<.QV! 9! 7968!
93841.98</4!8-!9.<6-81-?<,!=<q56<-.!7-1!<29Q4!4.L9.,424.8M[!IEEE
Trans. Pa\ern Anal. Mach. Intell.M!/-3J!#"M!.-J!"M!??J!%#0%)M!k9.J!"***J!
!H%&I! EJ!\L-5=L51@!9.=!kJ!W52i3<.M!SWL4!81<3984193!r3841!7-1!L<QL!,-.81968!
<29Q46!9.=!246L46M[!<.!Proc. Eurographics Workshop on RenderingM!
#++$M!??J!")'0"*'J!
!H%'I! KJ! gL9.QM! bJ! g5-M! GJ! a5M! 9.=! ZJ! gL9.QM! SZ491.<.Q! =44?! \UU!
=4.-<641! ?1<-1! 7-1! <29Q4! 1468-198<-.M[! <.! Proc. CVPRM! #+"(M! ??J!
$*#*0$*$)J!
!H%(I! EJ!R<39.791M!SP!8-51!-7!2-=41.!<29Q4!r3841<.QV!.4X!<.6<QL86!9.=!
248L-=6M!i-8L!?19,8<,93!9.=!8L4-148<,93M[!IEEE Signal Process. Mag.M!
/-3J!$+M!.-J!"M!??J!"+'0"#)M!k9.J!#+"$J!
!H%)I! TJ! T9.! et al.M! Sw59=198<,! 958-4.,-=41! ;wePp>! 7-1! 3-Xe=-64! \W!
=4.-<6<.QM[!IEEE Trans. Med. ImagingM!/-3J!$*M!.-J!'M!??J!#+$&0#+&+M!
k5.J!#+#+J!
!H%*I! RJ!Z<M!bJ!^65M!fJ!f<4M!kJ!\-.QM!9.=!bJ!a9-M!SGP\UUV!6437e9o4.8<-.!
,-./-358<-.93!.45193!.48X-1B!7-1!3-Xe=-64!\W!=4.-<6<.Q!X<8L!6437e
65?41/<64=! ?41,4?8593! 3-66! .48X-1BM[! IEEE Trans. Med. ImagingM!
/-3J!$*M!.-J!(M!??J!##)*0#$+"M!k53J!#+#+J!
!H&+I! bJ!c-.QM!^J!b9.QM!TJ!b5M!aJ!GL<M!9.=!fJ!Z<M!Sc44?!6?98<9306?4,8193!
14?1464.898<-.!3491.<.Q!7-1!L@?416?4,8193!<29Q4!=4.-<6<.QM[!IEEE
Trans. Comput. ImagingM!/-3J!&M!.-J!%M!??J!'$&0'%)M!c4,J!#+"*J!
12 IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. 0, NO. 0, DECEMBER 2023
!H&"I! kJ!f<4M!ZJ!f5M!9.=!pJ!\L4.M!S_29Q4!=4.-<6<.Q!9.=!<.?9<.8<.Q!X<8L!
=44?!.45193!.48X-1B6M[!<.!Proc. NIPSM!#+"#M!/-3J!#&J!
!H&#I! dJ! dB89@! et al.M! SP.98-2<,933@! ,-.6819<.4=! .45193! .48X-1B6!
;P\UU6>V! 9??3<,98<-.! 8-! ,91=<9,! <29Q4! 4.L9.,424.8! 9.=!
64Q24.898<-.M[!IEEE Trans. Med. ImagingM!/-3J!$(M! .-J!#M!??J!$)%0
$*&M!T4iJ!#+")J!
!H&$I! PJ!kJ!Z9119h9i93M!\J!R918x.4hM!OJ!a3-,B41M!9.=!pJ!T4119.84M!SE-68e
cPpV! 9.98-2<,933@! ?3956<i34! 64Q24.898<-.! /<9! ?-68e?1-,466<.Q!
X<8L!=4.-<6<.Q!958-4.,-=416M[!IEEE Trans. Med. ImagingM!/-3J!$*M!
.-J!"#M!??J!$)"$0$)#+M!c4,J!#+#+J!
!H&%I! ^J! Y9/<6L9.B91M! YJ! N4.B9891929.<M! GJ! WL<15/4.B9=92M! EJ!
G5=L9B91M! 9.=! NJ! N9<=@9M! SZ491.<.Q! 9.=! <.,-1?-198<.Q! 6L9?4!
2-=436! 7-1! 6429.8<,! 64Q24.898<-.M[! <.! Proc. MICCAIM! #+"(M! ??J!
#+$0#""J!
!H&&I! `J! O-@B-/M! dJ! N4B6341M! 9.=! YJ! g9i<LM! ST968! 9??1-:<2984! 4.41Q@!
2<.<2<h98<-.! /<9! Q19?L! ,586M[! IEEE Trans. Pa\ern Anal. Mach.
Intell.M!/-3J!#$M!.-J!""M!??J!"###0"#$*M!U-/J!#++"J!
!H&'I! NJ!K-32-Q-1-/M!S\-./41Q4.8!8144e14X4<QL84=!24669Q4!?966<.Q!7-1!
4.41Q@!2<.<2<h98<-.M[!IEEE Trans. Pa\ern Anal. Mach. Intell.M!/-3J!
#)M!.-J!"+M!??J!"&')0"&)$M!d,8J!#++'J!
!H&(I! cJ!O418L43-8M!UJ!\913<.<M!_J!a--=7433-XM!UJ!E9?41.-8M!PJ!d3</41M!9.=!
\J!PJ! Y9q43M! SR<:R98,LV!9! L-3<68<,!9??1-9,L!8-! 642<e65?41/<64=!
3491.<.QM[!<.!Proc. NIPSM!#+"*M!/-3J!$#J!
!H&)I! RJ!G9FF9=<M!RJ!k9/9.291=<M!9.=!WJ!W96=<h4.M!SY4Q5391<h98<-.!X<8L!
68-,L968<,! 819.67-1298<-.6! 9.=! ?41851i98<-.6! 7-1! =44?! 642<e
65?41/<64=!3491.<.QM[!<.!Proc. NIPSM!#+"'M!/-3J!#*J!
!H&*I! wJ! f<4M! RJeWJ! Z5-.QM! pJ! ^-/@M! 9.=! wJ! NJ! Z4M! SG437e819<.<.Q! X<8L!
.-<6@! 685=4.8! <2?1-/46!_29Q4U48! ,3966<r,98<-.M[! <.! Proc. CVPRM!
#+#+M!??J!"+')(0"+'*)J!
!H'+I! fJ! \L4.M! `J! `59.M! aJ! g4.QM! 9.=! kJ! b9.QM! SG42<eG5?41/<64=!
G429.8<,!G4Q24.898<-.!b<8L!\1-66!E645=-!G5?41/<6<-.M[!<.!Proc.
CVPRM!#+#"M!??J!#'"$0#'##J!
!H'"I! `J! Z<5M! `J! W<9.M! `J! \L4.M! TJ! Z<5M! NJ! O439Q<9..<6M! 9.=! aJ! \91.4<1-M!
SE41851i4=! 9.=! G81<,8! R49.! W49,L416! 7-1! G42<eG5?41/<64=!
G429.8<,!G4Q24.898<-.M[!<.!Proc. CVPRM!#+##M!??J!%#&)0%#'(J!
!H'#I! YJ! R4.=43M! ZJ! PJ! =4! G-5h9M! cJ! Y95i41M! kJ! EJ! E9?9M! 9.=! \J! E932M!
SG42<e65?41/<64=! 64Q24.898<-.! i964=! -.! 411-1e,-114,8<.Q!
65?41/<6<-.M[!<.!Proc. ECCVM!#+#+M!??J!"%"0"&(J!
!H'$I! `J! gL-5! et al.M! SE1<-1e9X914! .45193! .48X-1B! 7-1! ?918<933@e
65?41/<64=! 2538<e-1Q9.! 64Q24.898<-.M[! <.! Proc. CVPRM! #+"*M! ??J!
"+'(#0"+')"J!
!H'%I! WJ!R4<.L91=8M!RJ!R-3341M!\J!^9h<1i96M!9.=!cJ!\142416M!SZ491.<.Q!
E1-:<293!d?4198-16V!j6<.Q!c4.-<6<.Q!U48X-1B6!7-1!Y4Q5391<h<.Q!
_./4164!_29Q<.Q!E1-i3426M[!<.!Proc. ICCVM!#+"(M!??J!"()"0"(*+J!
!H'&I! PJ! O4,B! 9.=! RJ! W4i-5334M! ST968! Q19=<4.8ei964=! 93Q-1<8L26! 7-1!
,-.6819<.4=! 8-893! /91<98<-.! <29Q4! =4.-<6<.Q! 9.=! =4i3511<.Q!
?1-i3426M[! IEEE Trans. Image Process.M! /-3J! ")M! .-J! ""M! ??J! #%"*0
#%$%M!U-/J!#++*J!
!H''I! KJ!gL9.QM!`J!Z<M!bJ!g5-M!ZJ!gL9.QM!ZJ!N9.!a--3M!9.=!YJ!W<2-784M!
SE35Qe9.=e?39@!<29Q4!1468-198<-.!X<8L!=44?!=4.-<641!?1<-1M[!IEEE
Trans. Pa\ern Anal. Mach. Intell.M!??J!"0"M!#+#"J!
!H'(I! KJ! gL9.QM!bJ!g5-M! `J!\L4.M!cJ! R4.QM!9.=! ZJ!gL9.QM!SO4@-.=!9!
a9566<9.! =4.-<641V! 146<=593! 3491.<.Q! -7! =44?! \UU! 7-1! <29Q4!
=4.-<6<.QM[!IEEE Trans. Image Process.M!/-3J!#'M!.-J!(M!??J!$"%#0$"&&M!
k53J!#+"(J!
!H')I! cJ!j3@9.-/M!PJ!N4=93=<M!9.=!NJ!Z42?<86B@M!Sc44?!<29Q4!?1<-1M[!<.!
Proc. CVPRM!#+")M!??J!*%%'0*%&%J!
!H'*I! GJ!YJ!Y<,L841M!NJ!N<.448M!GJ!Y-8LM!9.=!NJ!K-385.M!SE39@<.Q!7-1!=989V!
Q1-5.=!8158L!71-2!,-2?5841!Q9246M[!<.!Proc. ECCVM!#+"'M!??J!"+#0
"")J!
!H(+I! aJ!Y-6M!ZJ!G433918M!kJ!R9841h@.6B9M!cJ!N9hl54hM!9.=!PJ!RJ!Z-?4hM!
SWL4!G`UW^_P!=989648V!9!391Q4!,-334,8<-.!-7!6@.8L48<,!<29Q46!7-1!
6429.8<,!64Q24.898<-.!-7!51i9.!6,4.46M[!<.!Proc. CVPRM!#+"'M!??J!
$#$%0$#%$J!
!H("I! RJ!^-q29..M!OJ!O<33-8M!cJ!UJ!a14/4M!kJ!pJ!_Q346<96M!OJ!T<6,L3M!9.=!PJ!
NJ! c93,9M! SG@.8LR-1?LV! 3491.<.Q! ,-.81968e<./91<9.8! 14Q<68198<-.!
X<8L-58!9,l5<14=!<29Q46M[!IEEE Trans. Med. ImagingM!??J!"0"M!#+#"J!
!H(#I! ^J! gL9.QM! RJ! \<664M!`J!UJ!c95?L<.M!9.=! cJ! Z-?4heE9hM! S2<:5?V!
O4@-.=!p2?<1<,93!Y<6B!R<.<2<h98<-.M[!<.!Proc. ICLRM!#+")J!
!H($I! GJ!`5.M! cJ!^9.M!GJ!kJ!dLM!GJ!\L5.M!kJ!\L-4M!9.=!`J!`--M!S\58R<:V!
Y4Q5391<h98<-.! G81984Q@! 8-! W19<.! G81-.Q! \3966<r416! b<8L!
Z-,93<h9i34!T4985146M[!<.!Proc. ICCVM!#+"*M!??J!'+#$0'+$#J!
!H(%I! RJ!TJ!a396641!et al.M!SP!2538<e2-=93!?91,43398<-.!-7!L529.!,414i193!
,-184:M[!NatureM!/-3J!&$'M!.-J!('"&M!??J!"("0"()M!P5QJ!#+"'J!
!H(&I! PJ! K34<.! 9.=! kJ! W-51/<334M! S"+"! 39i434=! i19<.! <29Q46! 9.=! 9!
,-.6<684.8!L529.!,-18<,93!39i43<.Q!?1-8-,-3M[!Front. Neurosci.M!/-3J!
'M!#+"#J!
!H('I! cJ! \J! N9.! p664.M! GJ! RJ! G2<8LM! cJ! RJ! O91,LM! WJ! pJ! kJ! O4L14.6M! pJ!
`9,-5iM! 9.=! KJ! jQ51i<3M! SWL4! bjeR<..! ^529.! \-..4,8-24!
E1-F4,8V!9.!-/41/<4XM[!NeuroImageM!/-3J!)+M!??J!'#0(*M!d,8J!#+"$J!
!H((I! OJ!T<6,L3M!ST144G51741M[!NeuroImageM!/-3J!'#M!.-J!#M!??J!((%0()"M!P5QJ!
#+"#J!
!H()I! PJ! c<! R918<.-! et al.M! SWL4! 958<62! i19<.! <29Q<.Q! =989! 4:,L9.Q4V!
8-X91=6!9!391Q4e6,934!4/93598<-.!-7!8L4!<.81<.6<,!i19<.!91,L<84,8514!
<.!958<62M[!Mol. PsychiatryM!/-3J!"*M!.-J!'M!??J!'&*0''(M!k5.J!#+"%J!
!H(*I! OJ!O<33-8!et al.M!SG@.8LG4QV!=-29<.!19.=-2<698<-.!7-1!64Q24.898<-.!
-7!i19<.!6,9.6!-7!9.@!,-.81968!9.=!146-358<-.M[!k53J!#+#"J!
!H)+I! GJ! O5i4,BM! S\-./4:!d?8<2<h98<-.V!P3Q-1<8L26!9.=! \-2?34:<8@M[!
Found. Trends® Mach. Learn.M!/-3J!)M!.-J!$0%M!??J!#$"0$&(M!U-/J!#+"&J!
!
!
!
Sean I. Young!14,4</4=!L<6! ELc!=4Q144!<.!434,81<,93!4.Q<.441<.Q!71-2!
8L4!j.</416<8@!-7!U4X!G-58L!b9346M!G@=.4@M!P568193<9J!^4!<6!,5114.83@!!
9.!_.6815,8-1!98!8L4!R918<.-6!\4.841!7-1!O<-24=<,93!_29Q<.QM!^91/91=!!
R4=<,93!G,L--3M!9.=!9!Y46491,L!Pu3<984!<.!\GP_ZM!R_WJ!E14/<-563@M!L4!
X96!9!?-68=-,8-193!146491,L41!98!G89.7-1=!j.</416<8@M!G89.7-1=M!\PJ!_.!
#+"'M!L4!X96!9!/<6<8<.Q!146491,L41!98!_.841c<Q<893!\-225.<,98<-.6M!G9.!
c<4Q-M! \PJ! ^4! 14,4</4=! 8L4!PEYGA_PEY! i468! ?9?41! 9X91=! 98! c_\WP!
#+")M!8-Q48L41!X<8L!c9/<=!W95i29.J!
Adrian V. Dalca!<6!9.!966<689.8! ?1-7466-1!98!^91/91=!R4=<,93! G,L--3M!
R9669,L564o6!a4.4193!^-6?<893M!9.=!9!Y46491,L!G,<4.8<68!<.!\GP_ZM!R_WJ!
^<6!146491,L!7-,5646!-.!29,L<.4!3491.<.Q!84,L.<l546!9.=!?1-i9i<3<68<,!
2-=436! 7-1! 24=<,93! <29Q4! 9.93@6<6J! c1</4.! i@! ,3<.<,93! l5468<-.6M! L4!
=4/43-?6!,-14!3491.<.Q!93Q-1<8L26M!96!X433!96!14Q<68198<-.M!64Q24.898<-.!
9.=!<2?5898<-.!248L-=6!9<24=!98!,3<.<,93e6-51,4=!=9896486!9.=!i1-9=3@!
9??3<,9i34!<.!<29Q4!9.93@6<6J!
Enzo Ferrante! 14,4</4=! L<6! ELc! =4Q144! <.! ,-2?5841! 6,<4.,46! 71-2!
j.</416<8y!E91<6eG9,39@!;E91<6M!T19.,4>!9.=!L4!X-1B4=!96!9!?-68=-,8-193!
146491,L41!98!8L4!O<-R4=_P!Z9iM!_2?41<93!\-334Q4!Z-.=-.!;Z-.=-.M!jK>J!
^4! <6! ,5114.83@! 9! \dU_\pW! 79,538@! 146491,L41! 9.=! ?1-7466-1! 98!
j.</416<=9=!U9,<-.93!=43!Z<8-193!<.!G9.89!T4M!P1Q4.8<.9M!XL414!L4!349=6!
8L4!146491,L!3<.4!-.!29,L<.4!3491.<.Q!7-1!i<-24=<,93!<29Q4!9.93@6<6!98!
8L4! Y46491,L! _.68<8584! 7-1! G<Q.936M! G@68426! 9.=! \-2?5898<-.93!
_.8433<Q4.,4M!6<.,;<>J!
Polina Golland <6!9!E1-7466-1!-7!p34,81<,93!p.Q<.441<.Q!9.=!\-2?5841!
G,<4.,4! 98! R_WJ! ^41! 146491,L! <.8414686! 6?9.! ,-2?5841! /<6<-.! 9.=!
29,L<.4!3491.<.QJ!^41! ,5114.8!X-1B!7-,5646!-.!=4/43-?<.Q!6898<68<,93!
9.93@6<6! 248L-=6! 7-1! ,L919,841<h98<-.! -7! i<-3-Q<,93! ?1-,46646! 56<.Q!
<29Q46! ;71-2! RY_! 8-! 2<,1-6,-?@>! 96! 9! 6-51,4! -7! <.7-1298<-.J! GL4!
14,4</4=!OG,!9.=!R968416!<.!\-2?5841!G,<4.,4!71-2!W4,L.<-.M!_61943!<.!
"**$! 9.=! "**&M! 9.=! 9! ELc! <.! p34,81<,93! p.Q<.441<.Q! 9.=! \-2?5841!
G,<4.,4!71-2!R_W!<.!#++"J!GL4!F-<.4=!8L4!79,538@!<.!#++$J!
Christopher A. Me?ler!<6!9.!P66<689.8!E1-7466-1!-7!\-2?5841!G,<4.,4!
98!8L4!j.</416<8@!-7!R91@39.=M!\-334Q4!E91BM!XL414!L4!=<14,86!8L4!jRc!
_.8433<Q4.8!G4.6<.Q!Z9iJ!^4!14,4</4=!L<6!OGM!RGM!9.=!ELc!<.!434,81<,93!
9.=!,-2?5841!4.Q<.441<.Q!71-2!Y<,4!j.</416<8@!<.!#+"$M!#+"%M!9.=!#+"*M!
146?4,8</43@J!^4!X96!9!?-68=-,!<.!8L4!G89.7-1=!\-2?5898<-.93!_29Q<.Q!
Z9i!<.!#+"*0#+#+J!^<6!X-1B!L96!14,4</4=!2538<?34!i468!?9?41!9X91=6z!L4!
14,4.83@!14,4</4=!9.!PTdGY!`-5.Q!_./468<Q98-1!E1-Q192!PX91=z!9.=!
L4! X96! 9.! _.8433<Q4.,4! \-225.<8@! E-68=-,8-193! Y46491,L! T433-XM! 9.!
UGT!a19=5984!Y46491,L!T433-XM!9!c-c! UcGpa!T433-XM!9.=!9! UPGP!
W4:96!G?9,4!a19.8!\-.6-18<52!T433-XJ!
Bruce Fischl! <6! 9! E1-7466-1! -7! Y9=<-3-Q@! 98! 8L4! R918<.-6! \4.841! 7-1!
O<-24=<,93!_29Q<.Q! 9.=! 9.! 9u3<984=! 146491,L41!98! R_WJ!^<6! 146491,L!
<./-3/46!8L4!=4/43-?24.8!-7!84,L.<l546!7-1!,-18<,93!65179,4!2-=433<.QM!
8L<,B.466! 249651424.8M! <.841e65iF4,8! 14Q<68198<-.M! XL-34ei19<.!
64Q24.898<-.!9.=!,1-66e6,934!<29Q<.QJ!WL4!8--36!8L98!L9/4!1465384=!71-2!
8L<6!146491,L!L9/4!i44.!=-X.3-9=4=!-/41!&+M+++!8<246!9.=!914!<.!564!<.!
39i6!91-5.=!8L4!X-13=J!!
Juan Eugenio Iglesias X96!i-1.!<.!G4/<334M!G?9<.J!^4!L-3=6!RG,!=4Q1446!
<.!W434,-225.<,98<-.!9.=!p34,81<,93!p.Q<.441<.Q!71-2!8L4!j.</416<8@!-7!
G4/<334!9.=!8L4!Y-@93!_.68<8584!-7!W4,L.-3-Q@!;KW^M!G8-,BL-32M!GX4=4.>J!
^4!=<=!9!ELc!<.!O<-24=<,93!p.Q<.441<.Q!98!8L4!j.</416<8@!-7!\93<7-1.<9M!
Z-6! P.Q4346! ;#++(0#+"">M! 6?-.6-14=! i@! 9! T53i1<QL8! Q19.8J! ^4! X96! 9!
?-68=-,8-193!146491,L41!98!8L4!R918<.-6!\4.841!7-1!O<-24=<,93!_29Q<.Q!
;#+""0#+"%>!9.=!8L4!O96l54!\4.841!-.!\-Q.<8<-.M!O19<.M!9.=!Z9.Q59Q4!
;#+"%0#+"'>M!6?-.6-14=!i@!9!R91<4!\51<4!T433-X6L<?J!_.!#+"'M!L4!F-<.4=!
j.</416<8@!\-334Q4!Z-.=-.!96!F5.<-1!79,538@M!75.=4=!i@!G8918<.Q!a19.8!
-7! 8L4! p51-?49.! Y46491,L! \-5.,<3J! _.! #+"*M! L4! 14F-<.4=! 8L4! R918<.-6!
\4.841M! XL414! <6! .-X! P66-,<984! E1-7466-1J! ^4! 936-! L-3=6! L-.-191@!
9??-<.824.86!98!j.</416<8@!\-334Q4!Z-.=-.!9.=!R_WJ!
YOUNG ET AL.: SUPERVISION BY DENOISING 13
APPENDIX A MATHEMATI C A L DERIVATIONS
p&2&6!7&!/2)%+-&!$!'2+&1!-&2+%$4+),!)1!4#&!-&,)+.+,3!.4&/!)1!
KNL!1)2!47)!0)..!18,(4+),.!)1!+,4&2&.4m!Y8$-2$4+(!K
L!0)..!$,-!
4#&!Z+(&!0)..@!O#&!1)2*&2!1$(+0+4$4&.!$,$05.+.!)1!4#&!-&,)+.+,3!
.4&/!$,-!()*/$2+.),!7+4#!-+2&(4!-&,)+.+,3!KQL6!7#&2&$.!4#&!
0$M&2!$!*)2&!3&,&2$0!-&.(&,4!.(#&*&!1)2!)/4+*+F$4+),"'$.&-!
-&,)+.+,3!7#&,!$!,),"Y8$-2$4+(!0)..!18,(4+),!+.!+,%)0%&-@!
A.1 Proximal Optimization and SUD
V,&!$04&2,$4+%&!4)!)'U&(4+%&!K<L!+.!4)!2&1)2*80$4&!42$+,+,3!$.!
4#&!U)+,4!)/4+*+F$4+),!)1!4#&!,&47)2P!/$2$*&4&2.!Θ!$,-!.)14!
4$23&4.!z
!1)2! = 1, . . . , @!O#&!()22&./),-+,3!)'U&(4+%&!+.!!
!
(Θ, z
{
}
) = (Θ) +
∑ (z
, f(u
|Θ)) +
(z
)
=
!!
K:>
L
!
,)4+,3!4#$4!K:>L!+.!7&00"/).&-!),05!+1!')4#!4&2*.!+,%)0%+,3!z!
$2&!/2&.&,4@!O#&!()*/).+4&!,$482&!)1!)'U&(4+%&!K:>L!2&,-&2.!+4!
$*&,$'0&!4)!)/4+*+F$4+),!%+$!$!/2)=+*$0!1)27$2-"'$(P7$2-!
./0+M+,3!*&4#)-!9HD;6!7#+(#!&..&,4+$005!$04&2,$4&.!.4)(#$.4+(!
32$-+&,4!-&.(&,4!8/-$4&!$,-!-&,)+.+,3!.4&/.@!
! `,!/$24+(80$26!4#&!.4)(#$.4+(!32$-+&,4!-&.(&,4!.4&/!)1!KNL!+.!
8.&-!4)! *+,+*+F&!Θ, z
{
}
!)%&2!Θ!7+4#!z
{
}
!E=&-@!V,!4#&!
)4#&2!#$,-6!32$-+&,4!-&.(&,4!4)!*+,+*+F&!(z
, f(u
|Θ))!+,!
4#&!.)14!4$23&4!z
!7+4#!7&+3#4.!Θ E=&-!($,!'&!72+M&,!$.!
!
z
= argmin
z∈
(z
) + 〈z,
(z
)〉+
1
2
‖z − z
‖
!
K:?
L
!
9HD;6!7#&2&!!+.!$!-+2&(4!.8*!)1!/2)'$'+0+45!.+*/0+(&.!$(2)..!
$00!/+=&0!0)($4+),.!$,-!!$!-&.(&,4!.4&/!.+F&@!!
! e//05+,3! K:?L!4)!*+,+*+F&!Θ, z
{
}
!)%&2!z
{
}
!$*)8,4.!
4)!E,-+,3!$23*+,!
(z) + 〈z,
(z
)〉+
‖z −z
‖
@!A+4#!
4#&!(#$,3&!)1!%$2+$'0&.! = !$,-! =
/6!7&!($,!&=/2&..!
4#+.!$23*+,!$.!4#&!/2)=+*$0!*$//+,3!+,!KNL!7#&,!4#&!0)..!+.!
$..8*&-!Y8$-2$4+(m!(z, f) = (1 2⁄ )‖z −f‖
@!Z+2&(4!-&,)+.+,3!
.4&/!KQL!+.!-&2+%&-!'5!0+,&$2+F+,3!(z)!$4!f
!$,-!.+*/0+15+,3!
4#&!2&.804+,3!$23*+,!&=/2&..+),!4)!!
!
z
= argmin
z
∈
1
2
‖z
−
(z
−
(z
)
−
(f
))‖
K:DL!
+,!7#+(#!
(f
) = f
−a(f
)6!1)2!.)*&!-&,)+.&2!a K>L@!`,!4#&!
($.&!7#&2&!!+.!$..8*&-!Y8$-2$4+(6!K:DL!4&00.!8.!4#&!)/4+*$0!
.)14!4$23&4!z
!+.!$!()*'+,$4+),!)1!(z
, f
, a(f
))!7+4#!7&+3#4.!
(1 − , (1 − ), )!$,-!($,!'&!72+M&,!$.!KQL@!O#&!7&+3#4.!
$2&!,),",&3$4+%&!1)2!0 ≤
≤ 6!+,!7#+(#!($.&!0 ≤ ≤ 1!-8&!
4)!4#&!8//&2!')8,-!),!4#&!.4&/!.+F&!!'&+,3!1/@!e,5!(),%&=!
()*'+,$4+),!)1!(z
, f
, a(f
)) '&0),3.!4)!!$,-!,)!/2)U&(4+),!
+.!2&Y8+2&-@!Z8&!4)!4#+.!(),%&,+&,(&6!7&!8.&!KQL!'5!$..8*+,3!
4#&!0)..!!+.!0)($005!Y8$-2$4+(!&%&,!1)2!$!,),"Y8$-2$4+(!@!
! e.!$,!&=$*/0&!)1!$!,),"Y8$-2$4+(!6!4#&!Z+(&!0)..!+.!8.&-!!
/)/80$205!+,!*&-+($0!+*$3&!2&(),.428(4+),@!A&!($,!72+4&!4#&!
32$-+&,4!)1!4#&!Z+(&!0)..!K7+4#!2&./&(4!4)!4#&!E2.4!$238*&,4L!
$.!
(z, f) = C
−
(Dz −f)6!+,!7#+(#!D!+.!$!-+$3),$0!*$42+=!)1!
(0$..!Z+(&!.()2&.6!$,-!C6!)1!$%&2$3&!(0$..!/+=&0!()8,4.@!X)2&!
./&(+E($0056!D!$,-!C!$2&!18,(4+),.!)1!(z, f)!3+%&,!'5!
!
=
2
〈
z
, f
〉
‖
z
‖
+
‖
f
‖
,
−
=
2
‖
z
‖
+
‖
f
‖
,!
K:H
L
!
2&./&(4+%&056!7#&2&!u
!-&,)4&.!4#&!4#!/2)'$'+0+45!*$/!)1!u@!
! `,!4#&!($.&!)1!4#&!Z+(&!0)..!6!),&!$04&2,$4+%&!4)!/2)U&(4&-!
32$-+&,4!-&.(&,4!K:DL!+.!&=/),&,4+$0!-&.(&,4!9QT;6!7#&2&!4#&!
Y8$-2$4+(!4&2*!)1!K:?L!+.!2&/0$(&-!7+4#!4#&!s800'$(PlW&+'0&2!
-+%&23&,(&!−
KL
(z, z
−
)@!O#+.!3+%&.!8.!4#&!-&.(&,4!.4&/!
!
z
= softmax(log(z
) −
(z
−
) −
(f
)),!
K:N
L
!
38$2$,4&&+,3!z
∈ !'5! (),.428(4+),@! A+4#!!.&3*&,4$4+),!
(0$..&.6!(),%&23&,(&!)1!4#+.!&=/),&,4+$0!-&.(&,4!.4&/!+.!'&M&2!
4#$,!4#&!/2)U&(4&-!-&.(&,4!),&!'5!$!1$(4)2!)1!
√
√
ln ⁄ !
.833&.4+,3!4#$4!K:NL!'&!8.&-!)%&2!/2)U&(4&-!32$-+&,4!-&.(&,4!
&./&(+$005!1)2!$!0$23&!@!I&&!9QT;!1)2!1824#&2!-+.(8..+),@!
A.2 Deriving Temporal Ensembling
O&*/)2$0! &,.&*'0+,3! 9<>;!($,!'&!+,4&2/2&4&-!$.!4#&!8/-$4&!
.4&/.!$..)(+$4&-!7+4#!)/4+*+F$4+),!)1!4#&!42$+,+,3!)'U&(4+%&!!
!
(Θ, z
{
}
) = (Θ) +
∑ ( 2⁄ )‖z
−f(u
|Θ)‖
=
,!!
K:QL!
4#$4!+.6!$!-&3&,&2$4&!($.&!)1!K:>L!+,!7#+(#!0)..!!+.!Y8$-2$4+(!
$,-!2&380$2+F$4+),!4&2*!
= 0@!`,!4#+.!($.&6!4#&!*+,+*+F&2!
)1!K:?L!+.!2&-8(+'0&!4)!z
= (1 −)z
+ f(u
|Θ
)6!7#+(#!+.!
4#&!"8/-$4&!.4&/!)1!9<>;@!I+,(&!4#+.!"8/-$4&!+.!1)00)7&-!'5!
$!,&47)2P!7&+3#4!8/-$4&6!4))!0$23&!$!.4&/!.+F&!!($,!0&$-!4)!
.8'")/4+*$0!42$+,+,3!)84()*&.!$,-!&%&,!-+%&23&,(&@!O#+.!+.!
&%+-&,(&-!'5!4#&!.8'")/4+*$0!/&21)2*$,(&!)1!4#&!g!*)-&0!
9DQ;6!7#+(#!+.!$!./&(+$0!($.&!)1!4&*/)2$0!&,.&*'0+,3!9<>;!4#$4!
+*/0+(+405!8.&.!.4&/!.+F&! = 1@!A&!.42&..!4#$4!7#&,!f!+.!,),"
,&3$4+%&!$,-!.42+(405!(),%&=!+,!7&+3#4.!Θ6!4#&!.&(),-!4&2*!
)1!)'U&(4+%&!!#$.!,)!2&380$2+F+,3!&J&(4!$,-!4#&!)/4+*$0!.)14!
4$23&4.!$2&!3+%&,!42+%+$005!'5!z
= f(u
|Θ)@!
! O#&!X&$,!O&$(#&2!*)-&0!)1!O$2%$+,&,!$,-!h$0/)0$!9>D;!+.!
$!%$2+$,4!)1!4&*/)2$0!&,.&*'0+,3!4#$4!3&,&2$4&.!.)14!4$23&4.!
8.+,3!$%&2$3&-!7&+3#4.!2$4#&2!4#$,!$%&2$3&!2&(),.428(4+),.!
$.!+,!9<>;@!O#&!42$+,+,3!)'U&(4+%&!$..)(+$4&-!7+4#!4#+.!*)-&0!
($,!'&!&=/2&..&-!$.!!
!
(Θ, Ω) = (Θ) +
∑ ( 2⁄ )‖f(u
|Ω) −f(u
|Θ)‖
=
,!!
K:R
L
!
4#$4!+.6!7&!2&/0$(&!4#&!!*$,5!z
{
}
!)/4+*+F$4+),!%$2+$'0&.!+,!
)'U&(4+%&!K:QL!7+4#!/$2$*&4&2.!Ω )1!$!]4&$(#&2^!,&47)2P@!e.!
'&1)2&6!),&!($,!$04&2,$4&05!)/4+*+F&!)%&2!Θ!$,-!Ω!7+4#!4#&!
)4#&2!P&/4!E=&-@!`,!/$24+(80$26!),&!($,!0&4!z
= f(u
|Ω)!$,-!
8.&!.4)(#$.4+(!32$-+&,4!-&.(&,4!.+*+0$205!4)!K:QL!4)!8/-$4&!4#&!
/$2$*&4&2.!Θ!)1!4#&!].48-&,4^!,&47)2P@!!
! B/-$4+,3!Ω
+
= (1 −)Ω
+ Θ
!1)2! 4&$(#&2! ,&47)2P!
/$2$*&4&2.!Ω!($,!'&!.&&,!$.!$//0+($4+),!)1!$!Y8$.+"C&74),!
TABLE B1
nn-UNet augmentation hyper-parameters for training
Parameter
Type
Range
Prob
Brain
Cortex
Y9.=-2!,1-?!
6L9?4!
0!
0.00!
!
!
#=A$=!6,93<.Q!
6,934!
[0.70, 1.40]!
0.20!
ü!
!
Y9.=!1-898<-.!
9.Q34!
[−30°, 30°]!
0.20!
ü!
!
Y9.=!v<??<.Q!
–!
–!
1.00!
ü!
!
p3968<,M!16×16!
6L<78
[−4.0, 4.0]!
1.00!
!
ü!
a9566<9.!.-<64!
6,934
[0.00, 0.10]!
0.10!
ü!
!
Y9.=!i3511<.Q!
6,934
[0.50, 1.00]!
0.20!
!
!
:7-12!3-Xe146!
6,934
[0.50, 1.00]!
0.25!
!
!
G,934!<.84.6<8@!
6,934
[0.75, 1.25]!
0.15!
ü!
!
:7-12!,-.81968!!
6,934
[0.75, 1.25]!
0.15!
ü!
!
_./418!,-.81968!
–!
–!
0.10!
!
!
:7-12!Q9229!
Q9229!
[0.70, 1.50]!
0.30!
ü!
!
\-./!;5.>?--3!
–!
–!
–!
!
!
E--3<.Q!39@416!
–!
–!
–!
4
4!
#=A$=!=1-?-58!
=1-?!p!
–!
–!
0.01!
0.05!
!
14 IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. 0, NO. 0, DECEMBER 2023
*&4#)-!4)!4#&!)/4+*+F$4+),!)1!K:RL!)%&2!Ω@!a+2.46!0+,&$2+F+,3!
f(u
|Ω)!$,-!f(u
|Θ)!+,!4#&!2&./&(4+%&!/$2$*&4&2.!$4!Ω
6!7&!
($,!72+4&!&$(#!0)..!4&2*!+,.+-&!4#&!.8**$4+),!)1!K:RL!$.!
!
(Ω
,
Θ)
≈
(
2
⁄ )‖
f(u
|Ω
)(Ω
−
Θ)‖
,
!
K<T
L
!
,&30&(4+,3!$00!#+3#&2!)2-&2!4&2*.6!7#&2&!f(u|Ω) -&,)4&.!4#&!
o$()'+$,!K4&,.)2L!)1!f!$4!Ω!1)2!3+%&,!u@!O$P+,3!$!C&74),!.4&/!
$4!4#&!(822&,4!+4&2$4&!Ω
!7+4#!.4&/!.+F&!!4#&,!()22&./),-.!4)!
4#&!8/-$4&!.4&/!1)2!4#&!4&$(#&2!7&+3#4.!.#)7,!$')%&@!
APPENDIX B NETWORK TRAINING DETAILS
p&2&6! 7&!-&4$+0!4#&! 42$+,+,3! -$4$!$,-!+*$3&! $83*&,4$4+),!
.42$4&3+&.!8.&-!1)2!4#&!2&(),.428(4+),!$,-!-,)+.+,3!4$.P.@!`,!
O$'0&!S:6!7&!0+.4!4#&!$83*&,4$4+),!#5/&2/$2$*&4&2.!8.&-!+,!
4#&!$,$4)*+($0!'2$+,!.&3*&,4$4+),!$,-!()24+($0!/$2(&00$4+),!
4$.P.@!O$'0&!S<!.#)7.!4#&!'2&$P-)7,!)1!4#&!:T!-$4$.&4.!8.&-!
'5!4#&!>Z!'2$+,!2&(),.428(4+),!4$.P!1)2!42$+,+,36!%$0+-$4+),!!
$,-!4&.4+,3@!O#&!0$'&0!*$/.!8.&-!4)!42$+,!4#&!-&,)+.&2!#$-!,)!
.8'U&(4!)%&20$/!7+4#!4#).&!8.&-!4)!42$+,!4#&!2&(),.28(4)2@!`,!
O$'0&!S>6!7&!/2)%+-&!$!'2&$P-)7,!)1!0$'&0&-!KX+,-S)330&L!
$,-!8,0$'&0&-!Kp\fL!-$4$!1)2!42$+,+,3!4#&!2&(),.428(4)2!$,-!
-&,)+.&2!,&47)2P.!+,!4#&!()24+($0!/$2(&00$4+),!4$.P@!O$'0&! S?!
/2)%+-&.!4#&!$((82$(5!)1!4#&!-&,)+.&2!42$+,&-!8.+,3!-+J&2&,4!
,8*'&2.!)1!42$+,+,3!0$'&0!*$/.@!!
!
TABLE B2
Datasets used for the 3D brain reconstruction task.
Split
ABD
ABD2
ADH
GSP
MCIC
OAS
UKB
COB
PPMI
B39
Total
W19<.!
–
–
–
–
–
–
–
–
–
1
1
N93<=!
10
10
10
10
10
10
10
10
10
9
99
^-3=-58!
55
55
55
55
55
55
55
50
55
10
500
U-!39i43!
158
168
103
179
24
70
179
22
97
0
1000
!
TABLE B3
Dataset split used for the 2D cortical parcellation task.
Dataset
Reconstructor Subjects
Denoiser Subjects
Train
Val
Test
Train
Val
R<.=O-QQ34!
000–039
040–059
060–099
000–039
040–059
^\E!_29Q46!
400–799
800–846
847–896
–
–
^\E!TG!Z9i436!
–
800–846
847–896
000–399
800–846
W-893!G5iF4,86!
440
!
67
!
90!
440!
67!
!
TABLE B4
Effect of training label set size on the denoiser accuracy.
Training
3D Brain Recon
2D Cortical Parcel
Average
Dice
95HD
Dice
95HD
Dice
95HD
#!39i436
+J*&'!!
+J'%'!!
+J**%!!
+J&"$!!
+J*'"!!
+J*#$!!
&!39i436!
+J*'*!
+J%$(!
+J**$!
+J&'"!
+J*)$!
+J&#'!
"+!39i436!
+J*)$!
+J#$(!
+J**&!
+J$(&!
+J**+!
+J#)%!
#+!39i436!
+J*)(!
+J"*'!
+J**&!
+J$(&!
+J**#!
+J"*%!
&+!39i436!
0.989
0.187
0.995
0.365
0.992
0.190
W19<.!=989!
+J'"%!
$#J**!
+J%((!
#'+J*!
+J&*'!
""*J"!
!
