IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. 0, NO. 0, MAY 2021 1
!
!
• S. I. Young and B. Girod are with the Information Systems Laboratory
(ISL), Department of Electrical Engineering, Stanford University, Stanford
CA 94305, USA. E-mail: sean0@stanford.edu, bgirod@stanford.edu.
• W. Zhe was with the Information Systems Laboratory (ISL), Department of
Electrical Engineering, Stanford University. He is now with the Institute
for Infocomm Research, A*STAR, Singapore.
Email: wang_zhe@i2r.a-star.edu.sg
• D. Taubman is with the School of Electrical Engineering and Telecommu-
nications, University of New South Wales, Sydney, NSW, Australia.
E-mail: d.taubman@unsw.edu.au
Manuscript received 28 August 2020; revised 12 March 2021; accepted 25
April 2021. Date of publication 25 April 2021; date of current version
3 May 2021. (Corresponding author: Sean I. Young.)
Recommended for acceptance by
For information on obtaining reprints of this article, please send email to:
reprints@ieee.org, and reference the Digital Object Identifier below.
Digital Object Identifier no. 10.1109/TPAMI.2020.
0162-8828 © 2020 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission.
See https://www.ieee.org/publications/rights/index.html for more information.
Transform Quantization for CNN Compression
Sean I. Young , Member, IEEE, Wang Zhe , Member, IEEE,
David Taubman , Fellow, IEEE, and Bernd Girod , Fellow, IEEE
Abstract—In this paper, we compress convolutional neural network (CNN) weights post-training via transform quantization. Previous
CNN quantization techniques tend to ignore the joint statistics of weights and activations, producing sub-optimal CNN performance at
a given quantization bit-rate, or consider their joint statistics during training only and do not facilitate efficient compression of already
trained CNN models. We optimally transform (decorrelate) and quantize the weights post-training using a rate–distortion framework
to improve compression at any given quantization bit-rate. Transform quantization unifies quantization and dimensionality reduction
(decorrelation) techniques in a single framework to facilitate low bit-rate compression of CNNs and efficient inference in the transform
domain. We first introduce a theory of rate and distortion for CNN quantization, and pose optimum quantization as a rate–distortion
optimization problem. We then show that this problem can be solved using optimal bit-depth allocation following decorrelation by the
optimal End-to-end Learned Transform (ELT) we derive in this paper. Experiments demonstrate that transform quantization advances
the state of the art in CNN compression in both retrained and non-retrained quantization scenarios. In particular, we find that transform
quantization with retraining is able to compress CNN models such as AlexNet, ResNet and DenseNet to very low bit-rates (1–2 bits).
Index Terms—convolutional neural networks, transform coding, compression, quantization, learned transforms.
✦
1 INTRODUCTION
ONVOLUTIONAL!"#$%&'!"#()*%+,!-.//,0!1&2#!3#4*5#!&!
$"62#%,&'!7%&5#)*%+!7*%!,*'26"8!&!"$53#%!*7!9%*3'#5,!
6"!9&:#%"!&"&';,6,!&"<!4*59$(#%!26,6*"=!7%*5!4'&,,6>4&(6*"!
?@A=!,#85#"(&(6*"!?BA!&"<!,(;'#!(%&",7#%!?C=!DA!*7!65&8#,!(*!(1#!
,;"(1#,6,!*7!65&8#,!?EAF!G16'#!.//,!*$(9#%7*%5!(%&<6(6*"&'!
"*"H'#&%"6"8H3&,#<!5#(1*<,!6"!5&";!26,6*"!9%*3'#5,=!(1#;!
4&"!6"2*'2#!(#",!*%!1$"<%#<,!*7!56''6*",!*7!9&%&5#(#%,=!&"<!
(1#6%!,1##%!3$'+!%#"<#%,!<#9'*;6"8!.//,!*"!5*36'#!<#264#,!
<6I4$'(F!.//!)#681(,!"*!'*"8#%!>(!6"!(1#!4&41#!&(!(1#!,&5#!
(65#=!&"<!(1#;!5$,(!3#!'*&<#<!7%*5!(1#!*JH4169!5#5*%;!&,!
%#K$6%#<=!)1641!4&"!3#!2#%;!#"#%8;H4*",$56"8!?LAF!M2#"!(1#!
&5*$"(!*7!*JH4169!5#5*%;!4&"!3#!'656(#<!*"!(1#,#!<#264#,!
,*!(1#!,(*%&8#!*7!)#681(,!6,!6(,#'7!&!41&''#"8#F!N*%#*2#%=!(1#!
"$53#%!*7!4*59$(&(6*",!9#%7*%5#<!*"!(1#,#!.//!)#681(,!
5&;!16"<#%!(1#!$,#!*7!.//,!7*%!4#%(&6"!6"(#%&4(62#!&"<!%#&'H!
(65#!&99'64&(6*",F!O"#!)&;!(*!%#<$4#!(1#!4*59$(&(6*"&'!&"<!
(1#!,(*%&8#!3$%<#"!*7!.//,!6,!(*!4*59%#,,!&"<!,659'67;!(1#!
%#9%#,#"(&(6*"!*7!(1#6%!"$5#%*$,!)#681(!9&%&5#(#%,F!
! P#4#"(!.//!4*59%#,,6*"!(#41"6K$#,!5&;!3#!4&(#8*%6Q#<!
&,!)#681(!9%$"6"8!?RSB@A=!)#681(!K$&"(6Q&(6*"!?BTSDLA=!&"<!
<65#",6*"&'6(;!%#<$4(6*"!*7!)#681(!5&(%64#,!?DRSELAF!O$(!*7!
(1#,#=!)#681(!K$&"(6Q&(6*"!1&,!,1*)"!9&%(64$'&%!$,#7$'"#,,!
7*%!4*59%#,,6"8!.//,!(*!36"&%;!*%!(#%(6&%;!%#9%#,#"(&(6*",!
?BUSCTA=!&(!2&%6*$,!%&(#S&44$%&4;!(%&<#H*J,!?CL=!CRA!&"<!#2#"!
9*,(H(%&6"6"8!?CUA!)6(1*$(!&";!'&3#'#<!<&(&F!V*)#2#%=!)16'#!
4$%%#"(!K$&"(6Q&(6*"!5#(1*<,!;6#'<!8**<!4*59%#,,6*"=!(1#;!
(#"<!(*!68"*%#!(1#!W*6"(!,(&(6,(64,!*7!)#681(,!&"<!&4(62&(6*",!
&"<!9%*<$4#!,$3H*9(65$5!"#()*%+!9#%7*%5&"4#!&(!&!862#"!
K$&"(6Q&(6*"!36(H%&(#=!*%!4*",6<#%!(1#6%!W*6"(!,(&(6,(64,!<$%6"8!
(%&6"6"8!*"';!?DEA=!&"<!<*!"*(!7&46'6(&(#!#I46#"(!4*59%#,,6*"!
*7!&'%#&<;H(%&6"#<!"#()*%+,F!!
! X*%!9*,(H(%&6"6"8!4*59%#,,6*"=!*9(65$5!<#4*%%#'&(6*"!*7!
(1#!)#681(,!4*$9'#<!)6(1!*9(65$5!36(H<#9(1!&''*4&(6*",!4&"!
,68"6>4&"(';!659%*2#!4*59%#,,6*"!#,9#46&'';!&(!(1#!'*)!36(H
%&(#,!,656'&%!(*!2#4(*%!K$&"(6Q&(6*"!?BBSBLA=!3$(!)6(1*$(!(1#!
6"4%#&,#!6"!(1#!#"4*<6"8!4*59'#Y6(;!7%*5!(1#!6%%#8$'&%6(;!*7!
K$&"(6Q&(6*"!4#'',F!Z"!9%&4(64#=!,4&'&%!K$&"(6Q&(6*"!7*''*)6"8!
&!<#4*%%#'&(6"8!(%&",7*%5!6,!9%#7#%%#<!(*!2#4(*%!K$&"(6Q&(6*"!
&,!,$88#,(#<!3;!(1#!,$44#,,!*7!65&8#!4*59%#,,6*"!5#(1*<,!
3&,#<!*"!#F8F!(1#![6,4%#(#!.*,6"#!\%&",7*%5!-[.\0!?ERA!&"<!
(1#![6,4%#(#!G&2#'#(!\%&",7*%5!-[G\0!?EUAF!\1#!'&(#,(!26<#*!
4*<6"8!,(&"<&%<!]].!?E^A!4&"!#2#"!,#'#4(!(1#!3#,(!(%&",7*%5!
7%*5![.\HZZ=!H]ZZZ!&"<![_\H]ZZ=!\%#''6,H4*<6"8!(1#!%#,$'(6"8!
(%&",7*%5!4*#I46#"(,!(*!7$%(1#%!659%*2#!4*59%#,,6*"F!!
! Z"!(16,!)*%+=!)#!9%*9*,#!(*!K$&"(6Q#!.//!)#681(,!9*,(H
(%&6"6"8!$,6"8!(%&",7*%5!K$&"(6Q&(6*"F!G#!&"&';Q#!(1#!.//!
4*59%#,,6*"!9%*3'#5!7%*5!&!%&(#S<6,(*%(6*"!26#)9*6"(=!&"<!
,1*)!(1&(!&!9&%(64$'&%!<#4*%%#'&(6"8!(%&",7*%5!4*$9'#<!)6(1!
*9(65$5!36(H<#9(1,!4&"!,*'2#!(16,!9%*3'#5F!\%&",7*%56"8!&!
4*"2*'$(6*"&'!*%!&!7$'';H4*""#4(6"8!'&;#%!#"&3'#,!<65#",6*"!
%#<$4(6*"!,656'&%!(*!`.a=!)16'#!K$&"(6Q6"8!(1#!6",68"6>4&"(!
(%&",7*%5!+#%"#',!(*!Q#%*!&(!'*)!%&(#,!#5$'&(#,!(1#!3#1&26*%!
*7!9%$"6"8!5#(1*<,F!\1#%#7*%#=!(%&",7*%5!K$&"(6Q&(6*"!$,#,!
&''!(1%##!6"8%#<6#"(,!*7!.//!4*59%#,,6*"b<65#",6*"&'6(;!
%#<$4(6*"!*7!)#681(,=!K$&"(6Q&(6*"=!&"<!9%$"6"8b&"<!&',*!
#"&3'#,!56Y#<H9%#46,6*"!6"7#%#"4#!*"!,9#46&'6Q#<!1&%<)&%#!
67!<6J#%#"(!36(H<#9(1,!&%#!&,,68"#<!(*!41&""#',F!Z"!X68F!@=!)#!
6''$,(%&(#!(1#!()*!4*59*"#"(,!-(%&",7*%5!&"<!K$&"(6Q&(6*"0!
C
2 IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. 0, NO. 0, MAY 2021
*7!(%&",7*%5!K$&"(6Q&(6*"F!a!ℎ × 𝑤!4*"2*'$(6*"!'&;#%!-8%&;0!
6,!4*"2#%(#<!(*!&!9&6%!*7!3&,6,!-3'$#0!&"<!(%&",7*%5!<*5&6"!
-*%&"8#0!4*"2*'$(6*"&'!'&;#%,!-&0=!(1#"!(1#!9&6%!K$&"(6Q#<!(*!
%&(#S<6,(*%(6*"! *9(65$5! 36(H<#9(1,! -30F! Z"! (1#! &3,#"4#! *7!
K$&"(6Q&(6*"=!4*59*,6"8!(1#!5&996"8,!*7!(16,!9&6%!*7!'&;#%,!
%#4*2#%,!-*%!,;"(1#,6Q#,0!(1#!*%686"&'!*"#F!_68"&'!9%*4#,,6"8!
'6(#%&($%#!$,#,!(1#!(#%5!c(%&",7*%5!4*<6"8d!5*%#!4*55*"';!
(1&"!(%&",7*%5!K$&"(6Q&(6*"F!V*)#2#%=!)#!&<*9(! (1#!'&:#%!
(#%5!6"!(16,!)*%+!(*!,(%#,,!*$%!7*4$,!*"!K$&"(6Q&(6*"!%&(1#%!
(1&"!-#"(%*9;0!4*<6"8=!)1641!)#!&<<%#,,!6"!*$%!,#K$#'F!
! O$%!5&6"!4*"(%63$(6*",!&%#!&,!7*''*),F!X6%,(=!)#!9%*9*,#!
(%&",7*%5!K$&"(6Q&(6*"!7*%!4*59%#,,6"8!.//!)#681(,b)#!
&%#!(1#!>%,(!(*!4*",6<#%!K$&"(6Q&(6*"!*7!(%&",7*%5#<!)#681(,!
&"<!3&,6,=!&"<!*9(656Q#!3*(1!9*,(H(%&6"6"8F!_#4*"<=!)#!862#!
&!(1#*%;!*7!%&(#!&"<!<6,(*%(6*"!7*%!.//!K$&"(6Q&(6*"=!3&,#<!
*"!)1641!(%&",7*%5!4*<6"8!8&6",!4&"!3#!4*59$(#<F!G#!(1#"!
<#%62#!&"!#"<H(*H#"<H'#&%"#<!(%&",7*%5!(1&(!5&Y656Q#,!(1#!
8&6",F!\16%<=!)#!&<2&"4#!(1#!,(&(#!*7!(1#!&%(!6"!4*59%#,,6*"!
*7!.//,=!6"!3*(1!%#(%&6"#<!&"<!"*"H%#(%&6"#<!,4#"&%6*,=!7*%!
65&8#!4'&,,6>4&(6*"ba'#Y/#(!?@A=!P#,/#(,!?LTA=![#",#/#(,!
?L@A=!&"<!7*%!'*)H'#2#'!26,6*"!(&,+,=![Pe/#(!-<#"*6,6"80!?LBA!
&"<!M[_P!-,$9#%H%#,*'$(6*"0!?LCAF!
2 RELATED WORK
.//!4*59%#,,6*"!5#(1*<,!3&,#<!*"!9%$"6"8!?RSBTA!"$''67;!
+#%"#',!?RS@CA=!41&""#',!?@DS@RA=!*%!#2#"!6"<626<$&'!)#681(,!
?@USBTA!(1&(!&%#!6",68"6>4&"(!&44*%<6"8!(*!,$41!4%6(#%6&!&,!&!
"*%5!?RS^A!*%!,*5#!*3W#4(62#!7$"4(6*"!?@UAF!V*)#2#%=!)16'#!
&!7#)!5#(1*<,!?BTA!4*59%#,,!(1#!9%$"#<!)#681(,!7$%(1#%!26&!
K$&"(6Q&(6*"!&"<!2&%6&3'#H'#"8(1!-V$J5&"0!4*<6"8=!*9(65&'!
K$&"(6Q&(6*"!*7!9%$"#<!"#()*%+,!8#"#%&'';!%#5&6",!&"!*9#"!
9%*3'#5F!N*%#*2#%=!(1#!9%$"6"8!4%6(#%6&!&%#!*7(#"!41*,#"!6"!
&!1#$%6,(64!5&""#%!&8"*,(64!(*!(1#!&4($&'!&44$%&4;!'*,,#,F!f;!
4*"(%&,(=! *$%! (%&",7*%5HK$&"(6Q&(6*"! 7%&5#)*%+! K$&"(6Q#,!
(*!Q#%*!-6F#F!9%$"#,0!+#%"#',!)1641!&%#!<##5#<!6",68"6>4&"(!
&44*%<6"8!(*!36(H<#9(1S&44$%&4;!*9(65&'6(;!4%6(#%6&F!
! .*"(#59*%&"#*$,!)6(1!(1#!9%$"6"8H3&,#<!(#41"6K$#,!7*%!
.//!4*59%#,,6*"=!,*5#!%#,#&%41#%,!9%*9*,#<!(*!4*59%#,,!
.//,!>%,(!3;!,9&(6&'';!<#4*%%#'&(6"8!(1#6%!)#681(,!)6(1!(1#!
[.\=!7*''*)#<!3;!2#4(*%!K$&"(6Q&(6*"!?BDA!*%!1&,16"8!?BLA!*7!
(1#!<#4*%%#'&(#<!)#681(,!(*!7$%(1#%!%#<$4#!(1#!%#<$"<&"46#,!
&4%*,,!(1#!(%&",7*%5#<!+#%"#',F!\1#,#!(%&",7*%5!&99%*&41#,!
&%#=!1*)#2#%=!&99'64&3'#!(*!(1#!4*59%#,,6*"!*7!4*"2*'$(6*"!
+#%"#',!*"';=!4*59$(&(6*"&'';!4*59'#Y!<$#!(*!(1#!4'$,(#%6"8!
6"2*'2#<!?BDA=!*%!4&41#H6"#I46#"(!<$#!(*!$,6"8!1&,1(&3'#,!&,!
(1#!5&6"!<&(&!,(%$4($%#!?BLAF!N*%#*2#%=!,*5#!.//!5*<#',!
4*"(&6"!'&%8#!"$53#%,!*7!1 × 1!4*"2*'$(6*"&'!+#%"#',=!)1#%#!
(1#!3#"#>(!*7!,9&(6&'!<#4*%%#'&(6*"!*7!+#%"#',!26&!(1#![.\!6,!
'#,,!,68"6>4&"(F!Z"!4*"(%&,(!)6(1!(1#!,9&(6&'H(%&",7*%5H3&,#<!
&99%*&41#,=!*$%!K$&"(6Q&(6*"!7%&5#)*%+!(%&",7*%5,!&4%*,,!
-%&(1#%!(1&"!)6(16"0!+#%"#',!(*!9%*26<#!(1#!4*59%#,,6*"!&"<!
#I46#"4;!8&6",!#2#"!7*%!,5&''#%!4*"2*'$(6*"!+#%"#',F!
! Z"!9&%&''#'!)6(1!(1#![.\!(#41"6K$#,!&3*2#=!*(1#%!&$(1*%,!
,*$81(!(*!%#<$4#!(1#!<65#",6*"&'6(;!*7!.//!'&;#%,!26&!`.a!
&"<!%#'&(#<!(#41"6K$#,!?DRSELAF![#"(*"!#(!&'F!?DRA!&99';!(1#!!
_][!&4%*,,!+#%"#',!6"!4*"2*'$(6*"!'&;#%,=!*%!&4%*,,!(1#!%*),!
&"<!4*'$5",!*7!)#681(!5&(%64#,!7*%!7$'';H4*""#4(6"8!'&;#%,!
(*!9%*W#4(!6"9$(!*%!*$(9$(!(*!'*)H%&"+!,$3,9&4#,=!9%*26<6"8!
4*59$(&(6*"&'!,&26"8,!<$%6"8!(1#!6"7#%#"4#!,(&8#F!g1&"8!#(!
&'F!?E@A!#Y(#"<!(16,!5#(1*<!3;!$,6"8!(1#!8#"#%&'6Q#<!_][!(*!
(&+#!6"(*!&44*$"(!(1#!9%#,#"4#!*7!P#he!&4(62&(6*",!3#()##"!
,$44#,,62#!'&;#%,F!\16,!)*%+!)&,!#Y(#"<#<!6"!($%"!3;!i65!#(!
&'F!?EBA=!)1*!&99';!(1#!_][!*"!&''!7*$%!&Y#,!*7!4*"2*'$(6*"&'!
(#",*%,F!h6!#(!&'F!?D^A!#Y(#"<!(1#!)*%+!*7!?E@A!3;!9&%(6(6*"6"8!
(1#!+#%"#',!6"(*!,$3,#(,=!(1#"!$,6"8!(1#!_][!*"!#&41!,$3,#(!
*7!+#%"#',!,#9&%&(#';F!V*)#2#%=!`.aH3&,#<!4*59%#,,6*"!6,!
"*(!4*"4#%"#<!)6(1!K$&"(6Q&(6*"=!,*!*9(65&'!K$&"(6Q&(6*"!*7!
(1#!9%*W#4(#<!"#()*%+!%#5&6",!&"!*9#"!9%*3'#5F!a',*=!`.a!
5#(1*<,! <#(#%56"#!(1#!9%*W#4(6*"!<65#",6*",!1#$%6,(64&'';!
$,6"8!<#4*%%#'&(#<!)#681(!2&%6&"4#,!?E@=!EBA=!3;!56"656Q6"8!
&!'#&,(H,K$&%#,!4%6(#%6*"!?ETA!*%!$,6"8!<65#",6*"!&,,68"5#"(!
%$'#,!?EDA=!"*"#!*7!)1641!%#j#4(!(1#!&4($&'!'*,,!*7!9%#<64(6*"!
&44$%&4;!<$#!(*!(1#!9%*W#4(6*",F!f;!4*"(%&,(=!*$%!7%&5#)*%+!
<#(#%56"#,!(1#!K$&"(6Q&(6*"!36(H<#9(1,!&"<!(1#6%!&,,*46&(#<!
9%*W#4(6*"!<65#",6*",!(*!56"656Q#!(1#!9#%7*%5&"4#!'*,,!&(!&!
862#"!K$&"(6Q&(6*"!36(H%&(#!-&2#%&8#!36(H<#9(10F!
! k$&"(6Q&(6*"!5#(1*<,!-3*(1!2#4(*%!&"<!,4&'&%0!7*%!.//,!
?BTSDLA!1&2#!<#2#'*9#<!&'*"8,6<#!9%$"6"8!5#(1*<,F!]#4(*%!
K$&"(6Q&(6*"!*7!)#681(,!?BBSBLA!6"2*'2#,!𝑘H5#&",!?BBSBDA!*%!
&!c1&,16"8!(%64+d!?BE=!BLA!&"<!5&;!&<<6(6*"&'';!6"2*'2#!(1#!
[.\!?BD=!BLA!&"<!%#,6<$&'!K$&"(6Q&(6*"!?BBAF!e"67*%5!,4&'&%!
K$&"(6Q&(6*"!*7!)#681(,!(*!*"#!36(!-36"&%;!%#9%#,#"(&(6*"0!6,!
9%*9*,#<!3;!V$3&%&!#(!&'F!?C@=!D@A=!&"<!(*!()*!36(,!-(#%"&%;0!
3;!g1$!#(!&'F!?CTAF!hkH/#(,!?CRA!'#&%"!(1#!*9(65&'!K$&"(6Q#%,!
<$%6"8!(%&6"6"8!&(!&!,9#46>#<!36(H<#9(1F!/*"H$"67*%5!,4&'&%!
K$&"(6Q&(6*"!)&,!9%*9*,#<!3;!\$"8!&"<!N*%6!?B@A=!)1#%#&,!
g1*$!#(!&'F!?CLA!<#5*",(%&(#!(1&(!)#!4&"!K$&"(6Q#!"#()*%+,!
9*,(H(%&6"6"8=!9%*26<#<!(1&(!)#681(,!4&"!3#!%#>"#<!3;!,*5#!
&<<6(6*"&'!(%&6"6"8!&7(#%!K$&"(6Q6"8!-)#!%#7#%!(*!(16,!9%*4#,,!
&,!%#H(%&6"6"80F!P#4#"(!5#(1*<,!?CU=!DCA!9%*9*,#!(*!K$&"(6Q#!
"#()*%+,!6"!&!(%&6"6"8H7%##!5&""#%=!3&,#<!*"!&!9#%H41&""#'!
36(H<#9(1!&''*4&(6*"!9&%&<685!?CUA=!*%!3;!#K$&'6Q6"8!)#681(,!
&"<!4*%%#4(6"8!36&,#,!?DCAF!G#!#Y(#"<!9#%H41&""#'!36(H<#9(1!
&''*4&(6*"!&"<!%#H(%&6"6"8!(*!(1#!(%&",7*%5!<*5&6"F!
! [#,68"6"8!'681()#681(!.//,!,$41!&,!N*36'#/#(!?LDA!&"<!
_K$##Q#/#(!?LEA!6,!&"*(1#%!7*%5!*7!5*<#'!4*59%#,,6*"F!O"#!
4&"!%#'&(#!(1#!<#9(1)6,#!4*"2*'$(6*"!'&;#%,!*7!?LDA!(*!9#%7#4(!
θ
θ
. . .
θ
t
t
. . .
t
s
s
. . .
s
S
∈ ℝ
×
×
T
∈ ℝ
×
×
→
Θ
∈ ℝ
×
×
→
(a) Transform
(b) Quantize
!
𝑅
𝑅
. . .
𝑅
𝑅
𝑅
𝑅
. . .
𝑅
= 3"#$%
𝑅
= 4"#$%
𝑅
= 5"#$%
. . .
4bits avg w/o transform
2bits avg with transform
Fig. 1. Transform quantization of CNN layers. Given weight matrices Θ
1
, Θ
2
, . . . , Θ
L
of an L-layer CNN, we represent each one as Θ
l
= S
l
T
l
(a) and
quantize both the kernel matrix
T
l
and the basis S
l
optimally (b). In (b), the bar lengths illustrate the bit-depths needed to quantize Θ
l
directly (gray
bars), or
S
l
T
l
in the transform domain (blue and orange bars) for the same performance. Elements corresponding to zero bit-depth assignments
(R
k
=
0) are indicated as white blocks in (a).
YOUNG ET AL.: TRANSFORM QUANTIZATION FOR CNN COMPRESSION 3
<6&8*"&'6Q&(6*"!*7!)#681(!5&(%64#,!$,6"8!(1#!_][=!)16'#!(1#!
%#<$4#<H6"9$(H41&""#'!3 × 3!4*"2*'$(6*"!?LEA!%#'&(#,!4'*,#';!
(*!(1#!9%*W#4(6*"!*7!6"9$(!,9&4#,!(*!(1#6%!'*)#%H<65#",6*"&'!
,$3H,9&4#,=!3*(1!<$%6"8!(%&6"6"8F!V*)#2#%=!<6,4*2#%6"8!(1#!
*9(65$5!'681()#681(!.//!5*<#',!4&"!3#!41&''#"86"8!,6"4#!
&%416(#4($%#!<#,68"!<#46,6*",!5$,(!3#!5&<#!9%6*%!(*!(%&6"6"8!
(1#!"#()*%+F!O9(65&'!K$&"(6Q&(6*"!*7!(1#!%#,$'(6"8!"#()*%+!
&',*!%#5&6",!&!9%*3'#5F!\%&",7*%5!K$&"(6Q&(6*"!6,!3&,#<!*"!
(1#!,&5#!<#4*%%#'&(6"8!-<6&8*"&'6Q6"80!9%6"469'#,=!6"<$46"8!
&!'*)H<65#",6*"&'!9%*W#4(6*"!*7!)#681(,!&(!'*)!K$&"(6Q&(6*"!
36(H%&(#,F!V*)#2#%=!*$%!7%&5#)*%+!1&,!(1#!&<2&"(&8#!(1&(!6(!
W*6"(';!&<<%#,,#,!*9(65&'!K$&"(6Q&(6*"=!9*,(H(%&6"6"8F!
! O+(&;!#(!&'F!?DEA!&"<!V$&"8!#(!&'F!?LLA!9%*9*,#!(*!659*,#!
+#%"#'!<#4*%%#'&(6*"!&(!(%&6"6"8!(65#=!%&(1#%!(1&"!<#4*%%#'&(#!
(1#5!9*,(!1*4F!a'(1*$81!,$41!,(%&(#86#,!4&"!3#!3#"#>46&'!7*%!
%#<$46"8!5*<#'!%#<$"<&"4;=!(1#;!6"4%#&,#!(%&6"6"8!(65#!67!&!
1681!<#8%##!*7!*%(1*8*"&'6(;!6,!"##<#<!?LLA=!<*!"*(!7&46'6(&(#!
4*59%#,,6*"!*7!&'%#&<;H(%&6"#<!5*<#',=!&"<!'#&2#,!*9#"!(1#!
*9(65&'!41*64#!*7!<#4*%%#'&(6"8!(%&",7*%5!?DEA!&,!)#''!&,!(1#!
9%*3'#5!*7!*9(65&'';!K$&"(6Q6"8!(1#!<#4*%%#'&(#<!+#%"#',F!a!
%#'&(#<!4'&,,!*7!5#(1*<,!+"*)"!&,!"&($%&'!8%&<6#"(!<#,4#"(!
?LRSRBA!9%*9*,#!(*!&44#'#%&(#!(1#!(%&6"6"8!*7!.//,!26&!'*4&'!
"*%5&'6Q&(6*"!*7!(1#!'*,,!'&"<,4&9#!3#7*%#!(&+6"8!<#,4#"(F!Z"!
4*"(%&,(!)6(1!"&($%&'!8%&<6#"(!<#,4#"(!)1#%#!(%&",7*%5,!&%#!
$,#<!(*!)16(#"!)#681(!8%&<6#"(=!*$%!7%&5#)*%+!<#4*%%#'&(#,!
-3$(!"*(!)16(#"0!(1#!)#681(,!(1#5,#'2#,!9%6*%!(*!K$&"(6Q6"8!
(*!659%*2#!(1#!K$&"(6Q#%l,!c9&4+6"8dF!
! m&*!#(!&'F!?RCA!,($<;!(1#!'656(,!*7!.//!4*59%#,,6*"!7%*5!
&!%&(#S<6,(*%(6*"!9#%,9#4(62#=!%#6"(#%9%#(6"8!4*59%#,,6*"!&,!
&!%&(#H&''*4&(6*"!9%*3'#5!,656'&%';!(*!(16,!)*%+F!\1#6%!,($<;!
1*)#2#%!<*#,!"*(!4*",6<#%!(1#!$,#!*7!&!(%&",7*%5!&,!9&%(!*7!
K$&"(6Q&(6*"F!_#4(6*"!D!,1*),!(1&(!&99';6"8!&!<#4*%%#'&(6"8!
(%&",7*%5!3#7*%#!K$&"(6Q6"8!6,!+#;!(*!8**<!4*59%#,,6*"F!
3 NOTATION AND REMARKS
h#(!$,!,#(!*$%!"*(&(6*"!3;!<#>"6"8!&!2#4(*%!,9&4#!*2#%!.//!
6"9$(=!*$(9$(!&"<!)#681(,=!#K$699#<!)6(1!*9#%&(6*",!3&,#<!
*"!5$'(6H6"9$(H5$'(6H*$(9$(!-NZNO0!4*"2*'$(6*",!?RDAF!G#!
<#"*(#!3;!x ∈ ℝ
×
!&"!𝑚H41&""#'!,68"&'!)1*,#!#'#5#"(,!&%#!
5&(%64#,!*7!,6Q#!𝑎 × 𝑏=!(1&(!6,=!𝑥
∈!ℝ
×
!7*%!𝑘 = 1, . . . , 𝑚F!G#!
<#"*(#!3;!Θ ∈ ℝ
×
×
!&!4*"2*'$(6*"&'!'&;#%!)6(1!𝑚!6"9$(!&"<!
𝑛!*$(9$(!41&""#',!)1*,#!4*"2*'$(6*"!+#%"#',!𝜃
∈ ℝ
×
!7*%!
𝑘 = 1, . . . , 𝑛!&"<!𝑗 = 1, . . . , 𝑚F!O"#!6,!"*)!%#&<;!(*!#K$69!(1#!
()*!,9&4#,!ℝ
×
!&"<!ℝ
×
×
!)6(1! (1#! 7*''*)6"8! 2#4(*%! ,9&4#!
*9#%&(6*",F!
Definition 1: Inner product.!\1#!6""#%!9%*<$4(!*7!()*!862#"!
𝑚H41&""#'!,68"&',!x, y ∈ ℝ
×
!4&"!3#!<#>"#<!&,!
!
〈x
,
y〉
=
〈
𝑥
, 𝑦
〉
+ ⋅ ⋅ ⋅ +
〈
𝑥
, 𝑦
〉
∈ ℝ,
!
-@
0
!
6"!)1641!〈𝑥, 𝑦〉
!<#"*(#,!(1#!X%*3#"6$,!6""#%!9%*<$4(!*7!(1#!
5&(%6YH#'#5#"(,!𝑥, 𝑦 ∈ ℝ
×
F!
Definition 2:!Euclidean norm. \1#!M$4'6<#&"!"*%5!*7!862#"!
𝑚H41&""#'!,68"&'!x ∈ ℝ
×
!4&"!3#!<#>"#<!$,6"8!-@0!&,!
!
‖x‖
=
√
〈x
,
x〉
=
√
‖
𝑥
‖
+
⋅ ⋅ ⋅ +
‖
𝑥
‖
∈ ℝ
+
,
!!
-B0!
!6"!)1641!‖𝑥‖
!<#"*(#,!(1#!X%*3#"6$,!"*%5!*7!𝑥 ∈ ℝ
×
F!
Definition 3: Layer–layer product.!\1#!'&;#%H'&;#%!9%*<$4(!
Z = XY!*7!X ∈ ℝ
×
×
!&"<!Y ∈ ℝ
×
×
!6,!<#>"#<!&,!
!
Z
=
(
𝑥
⋆ 𝑦
+
⋅ ⋅ ⋅ + 𝑥
⋆ 𝑦
)
∈ ℝ
(|
−
|
+
)
×
(|
−
|
+
)
!
-C0!
7*%!𝑘 = 1, . . . , 𝑜!&"<!𝑗 = 1, . . . , 𝑚=!)1#%#!⋆!%#9%#,#"(!2&'6<!-&,!
*99*,#<!(*!7$''!*%!,&5#0!B[!4*"2*'$(6*"
1
F!O3,#%2#!(1&(!#&41!
𝑥 ⋆ 𝑦 = 〈 𝑥, 𝑦〉
!67!𝑥!&"<!𝑦!1&2#!(1#!,&5#!<65#",6*",F!
! Z"!4*"(%&,(!)6(1
4 IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. 0, NO. 0, MAY 2021
,(6''!)#''H<#>"#<F!Z"!(16,!)*%+=!)#!#,,#"(6&'';!(%&",7*%5!&''!
4*"2*'$(6*"&'!5&996"8,!*7!(1#!7*%5!x ↦ Θx!(*!x ↦ STx!&"<!
K$&"(6Q#!S!&"<!TF!_##!X68F!@!-'#7(0=!)1#%#!Θ, T ∈ ℝ
×
×
!<#964(!
2 × 2!4*"2*'$(6*"!'&;#%,!&"<!S ∈ ℝ
×
×
=!&!3&,6,!'&;#%=!)1641!
6,!#K$62&'#"(!(*!&!1 × 1!4*"2*'$(6*"!'&;#%F!!
4 QUANTIZATION OF CNNS
\16,!)*%+!&<<%#,,#,!(1#!9%*3'#5!*7!4*59%#,,6"8!(1#!)#681(!
5&(%64#,!*7!&!.//F!G#!4&"!#Y9%#,,!(1#!#"<H(*H#"<!5&996"8!
*7!&"!𝐿H'&;#%!7##<H7*%)&%<!.//!&,!
!
y
= 𝑓
(x
|
Θ
, . . . ,
Θ
)
= 𝑓
(
⋅ ⋅ ⋅ 𝑓
(Θ
𝑓
(Θ
x)))
,
!
-D
0
!
6"!)1641!x ∈ ℝ
×
!&"<!y ∈ ℝ
×
!&%#!%#,9#4(62#';!(1#!"#()*%+!!
6"9$(!&"<!*$(9$(=!&"<!Θ
=
!&%#!(1#!𝐿!4*"2*'$(6*"&'!&"<!
7$'';H4*""#4(6"8!)#681(!5&(%64#,!(1&(!9&%&5#(#%6Q#!𝑓F!/*"H
'6"#&%6(6#,!3#()##"!4*"2*'$(6*",=!,$41!&,!9**'6"8=!&4(62&(6*"!
-P#he!*%!(&"10=!&"<!"*%5&'6Q&(6*"!1&2#!3##"!&3,*%3#<!6"(*!
7$"4(6*",!𝑓
!(*8#(1#%!)6(1!36&,#,F!a'(1*$81!36&,!9&%&5#(#%,!
&%#!&',*!'#&%"&3'#!,$41!(1&(!(1#;!(**!5$,(!3#!K$&"(6Q#<!&"<!
,(*%#<=!(1#;!&%#!,$I46#"(';!7#)!%#'&(62#!(*!)#681(,=!&"<!(1#6%!
659&4(!*"!(1#!K$&"(6Q#<!,6Q#!*7!.//,!6,!"#8'6863'#F!G1#"!&!
4'&,,6>4&(6*"!.//!6,!3#6"8!K$&"(6Q#<=!*$(9$(!y!6,!&,,$5#<!
(*!3#!(1#!'*86(,=!(1&(!6,=!%#,$'(!9%6*%!(*!,*7(5&Y!&4(62&(6*"F!
! \*!862#!&!4*"4%#(#!#Y&59'#=!a'#Y/#(!?@A!9&%&5#(#%6Q#,!𝑓!
$,6"8!𝐿 = 8!4*"2*'$(6*"&'!&"<!7$'';H4*""#4(6"8!)#681(,=!(1#!
6"9$(!x ∈ ℝ
×
!%#9%#,#"(,!&!CH4*'*%!65&8#!)6(1!224 × 224!
96Y#',=!&"<!*$(9$(!y ∈ ℝ
×
!$""*%5&'6Q#<!'*8H9%*3&36'6(6#,!
*7!5#53#%,169!*7!6"9$(!x!&4%*,,!(1#!1000!9%#<#>"#<!4'&,,#,!
-(#"41= . . . =!(*6'#(!9&9#%0F!.*"2*'$(6*"!'&;#%!Θ
∈ ℝ
×
×
!1&,!
64 × 3!4*"2*'$(6*"&'! +#%"#',! *7! ,6Q#!11 × 11=! )16'#! (1#! '&,(!
7$'';H4*""#4(#<!'&;#%!4&"!3#!<#"*(#<!Θ
∈ ℝ
×
×
F!
4.1 Layer-wise Quantization
\1#!#'#5#"(,!*7!)#681(,!Θ
!&%#!4*"(6"$*$,';H2&'$#<!,*!(1&(!
(1#;!"#4#,,6(&(#!K$&"(6Q&(6*"!7*%!#I46#"(!4*55$"64&(6*"!*%!
,(*%&8#F!Z"!(1#!,659'#,(!4&,#=!)#!4&"!$,#!$"67*%5!K$&"(6Q#%,!
*"!)#681(,!<6%#4(';F!k$&"(6Q6"8!Θ
!&(!&!36(H<#9(1!*7!𝑅
!862#,!
$,!Θ
= 𝑞(Θ
, 𝑅
, Δ
(𝑅
))=!)1#%#!𝑞(𝜃, 𝑅, Δ) = 0!67!𝑅 = 0!&"<!!
!
𝑞
(
𝜃, 𝑅, Δ
)
= Δ clip
(
round
(
Δ
−
𝜃
)
, −2
−
, 2
−
− 1
)!
-E
0
!
*(1#%)6,#=!)1#%#!𝑅
!6,!&!)1*'#!"$53#%!*7!36(,!&"<!Δ
(𝑅
)!6,!
(1#!*9(65$5!K$&"(6Q&(6*"!,(#9H,6Q#!7*%!&!862#"!𝑅
F
YOUNG ET AL.: TRANSFORM QUANTIZATION FOR CNN COMPRESSION 5
$"<#%';6"8!%&"<*5!,*$%4#!θ
=!&"<!4*'$5",!*7!T
!&,!(1*,#!*7!
%&"<*5!,*$%4#!t
!&"<!&,,$5#!#&41!*7!(1#!%&"<*5!,*$%4#,!6,!
W*6"(';!m&$,,6&"F!X*%!3%#26(;=!)#!5&;!*56(!(1#!,$3,4%69(!𝑙!*7!
5&%64#,!Θ
=!T
!&"<!,*$%4#,!θ
=!t
=!&"<!)%6(#!Θ=!T!&"<!θ=!tF!G#!
>%,(!<#%62#!(1#*%#(64&'!%#,$'(,!7*%!56"65$5!<6,(*%(6*"!)1#"!
(1#!%&"<*5!,*$%4#!θ!6,!K$&"(6Q#<!)6(1*$(!&!(%&",7*%5=!&"<!
36(H<#9(1,!&,,68"#<!(*!(1#!6"<626<$&'!#'#5#"(,!*7!θF!
Theorem 1: Minimum distortion without transform.!m62#"!
&!W*6"(';Hm&$,,6&"!,*$%4#!θ ∈ ℝ
×
=!(1#!56"65$5!<6,(*%(6*"!
𝐷 =
𝔼‖y − y‖
!<$#!(*!(1#!K$&"(6Q&(6*"!*7!θ!&(!&!,$I46#"(';!
1681!K$&"(6Q&(6*"!36(H%&(#!-&2#%&8#!36(H<#9(10!𝑅!6,!862#"!3;!
!
𝐷
pcm
=
(∏ (C
)
(C
)
=
)
⁄
𝜖
2
−
,
!!
-^0!
6"!)1641!C
!&"<!C
!%#9%#,#"(!(1#!4%*,,H41&""#'!4*2&%6&"4#!
5&(%6Y! *7!θ ∈ ℝ
×
!&"<!Γ = (𝜕y 𝜕θ⁄ ) ∈ ℝ
×
×
!%#,9#4(62#';=!𝑅!
6,!(1#!&2#%&8#!K$&"(6Q&(6*"!36(H<#9(1!7*%!(1#!𝑛!#'#5#"(,!*7!θ!
&"<!𝜖
!6,!&!4*",(&"(!)1641!<#9#"<,!*"!(1#!K$&"(6Q&(6*"!&"<!
4*<6"8!,41#5#!$,#<!?RLAF!G#!862#!(1#!9%**7!6"!a99#"<6Y!aF!
! \1#*%#5!@!#,,#"(6&'';!,(&(#,!(1#!*$(9$(!<6,(*%(6*"!<$#!(*!
K$&"(6Q6"8!(1#!#'#5#"(,!*7!θ!)6(1!*9(65&'!36(H<#9(1,!7*%!&"!
&2#%&8#!*7!𝑅!36(,!6,!6<#"(64&'!(*!<6,(*%(6*"!<$#!(*!K$&"(6Q6"8!
&!m&$,,6&"!,*$%4#!*7!2&%6&"4#!(∏ (C
)
(C
)
=
)
⁄
!&(!𝑅!
36(,F!\1#!#Y9*"#"(6&'!<#4&;!2
−
!4*5#,!7%*5!(1#!1&'26"8!*7!
(1#!K$&"(6Q&(6*"!,(#9H,6Q#!)6(1!#&41!&<<6(6*"&'!36(=!%#<$46"8!
*$(9$(!<6,(*%(6*"!3;!&!7&4(*%!*7!7*$%F!G#!"*)!659%*2#!$9*"!
(1#!%&(#H<6,(*%(6*"!7$"4(6*"!-^0!3;!(%&",7*%56"8!(1#!#'#5#"(,!
*7!θ!9%6*%!(*!K$&"(6Q&(6*"F!_$99*,#!&!(%&",7*%5!U
∈ ℝ
×
×
!6,!
>%,(!&99'6#<!*"!θ!(*!9%*<$4#!4*#I46#"(,!t = U
θ=!&"<!t!(1#"!
K$&"(6Q#<F!k$&"(6Q&(6*"!36(H<#9(1,!𝑅
!&%#!&''*4&(#<!"*)!(*!
(1#!#'#5#"(,!*7!(%&",7*%5!,*$%4#!tF!\1#*%#5!B!"*)!<#%62#,!
(1#!56"65$5!*$(9$(!<6,(*%(6*"!7*%!(%&",7*%5!K$&"(6Q&(6*"F!
Theorem 2: Minimum distortion with transform.!m62#"!(1#!
,&5#!W*6"(';!m&$,,6&"!,*$%4#!θ ∈ ℝ
×
=!(1#!56"65$5!*$(9$(!
<6,(*%(6*"!𝐷 =
𝔼‖y − y‖
!4&$,#<!3;!(1#!K$&"(6Q&(6*"!*7!(1#!
(%&",7*%5!4*#I46#"(,!t = U
θ!&(!,$I46#"(';!1681!%&(#,!𝑅!6,!
!
𝐷
tc
= (∏ (U
−
C
U
−
)
(U
C
U)
=
)
⁄
𝜖
2
−
!!
-@T
0
!
6"!)1641!𝐂
!&"<!𝐂
!&%#!<#>"#<!6<#"(64&'';!&,!3#7*%#=!&"<!
𝑅!6,!"*)!(1#!&2#%&8#!K$&"(6Q&(6*"!36(H<#9(1!&''*4&(#<!(*!(1#!
𝑛!#'#5#"(,!*7!tF!G#!9%*26<#!(1#!9%**7!6"!a99#"<6Y!aF!
! \1#!%&(6*!3#()##"!(1#!()*!<6,(*%(6*",!-^0!&"<!-@T0=!
!
𝐺 =
(∏ (
C
)
=
)
⁄
(∏ (U
−
C
U
−
)
=
)
⁄
(∏ (
C
)
=
)
⁄
(∏ (U
C
U)
=
)
⁄
!
-@@
0
!
6,!+"*)"!&,!(1#!4*<6"8!8&6"!?RRA!*7!(%&",7*%5!U
-1681#%!6,!
3#:#%0F!G#!#Y9#4(!K$&"(6Q&(6*"!*7!t = U
θ!(*!9%*<$4#!&!36(H
%&(#!,&26"8!*7!
log
𝐺!36(,!*2#%!(1&(!*7!(1#!*%686"&'!θ!,*$%4#!
7*%!&!862#"!*$(9$(!<6,(*%(6*"F!\1#!4'&,,64&'!i&%1$"#"Sh*o2#!
\%&",7*%5!-ih\0!*7!θ!-6F#F!(1#!5&(%6Y!*7!#68#"2#4(*%,!*7!C
0!
6,!1*)#2#%!,$3H*9(65&'!)6(1!%#,9#4(!(*!𝐺!,6"4#!6(!5&Y656Q#,!
*"';!(1#!,#4*"<!8&6"!(#%5!*7!-@@0F!\1#!*9(65&'!(%&",7*%5!7*%!
.//!K$&"(6Q&(6*"!6,!"*)!<#%62#<F!
Theorem 3: Optimum transforms for CNNs.!\1#!(%&",7*%5!
U
!(1&(!56"656Q#,!*$(9$(!<6,(*%(6*"!𝐷 =
𝔼‖y − y‖
!4&$,#<!
3;!(1#!K$&"(6Q&(6*"!*7!t = U
θ=!<6&8*"&'6Q#,!(1#!5&(%6Y!9&6%!
(C
, C
−
)F!\1&(!6,=!
!
U
C
U = Λ &"< U
C
−
U = I,!
-@B
0
!
6"!)1641!C
!&"<!C
!%#9%#,#"(!(1#!4%*,,H41&""#'!4*2&%6&"4#!
5&(%6Y!*7!θ!&"<!Γ!%#,9#4(62#';!&,!9%#26*$,';=!&"<!Λ!<#"*(#,!
(1#!"*""#8&(62#!<6&8*"&'!5&(%6Y!*7!8#"#%&'6Q#<!#68#"2&'$#,!
*7!5&(%6Y!9&6%!(C
, C
−
)F!Z"!8#"#%&'=!U
−
≠ U
,6"4#!U!6,!"*(!
"#4#,,&%6';!*%(1*8*"&'F!\1#!9%**7!6,!862#"!6"!a99#"<6Y!aF!
! G#!%#7#%!(*!(1#!*9(65&'!(%&",7*%5!U
!7%*5!\1#*%#5!C!&,!
(1#!M"<H(*H#"<!h#&%"#<!\%&",7*%5!-Mh\0!*7!θF!\1#!Mh\,!&%#!
,*H4&''#<!,6"4#!(1#;!,65$'(&"#*$,';!<#4*%%#'&(#!(1#!)#681(,!
θ &"<!(1#!9&%(6&'!<#%62&(62#,!𝜕y 𝜕θ⁄ =!)1641!&%#!<#%62#<!7%*5!
(1#!#"<H(*H#"<!5&996"8!*7!x!(*!*$(9$(!y = 𝑓(x|Θ
, . . . , Θ
)!
&4%*,,!&''!2&'$#,!*7!xF!Z"!9%&4(64#=!)#!4&"!4*59$(#!𝜕y 𝜕θ⁄ !3;!
3&4+H9%*9&8&(6*"!$,6"8!&!,5&''!2&'6<&(6*"!,#(F!Z"!X68F!B=!)#!
4*59&%#!(1#!4*<6"8!8&6",!*7!(1#!ih\!&"<!(1#!Mh\!&4%*,,!(1#!
'&;#%,!*7!P#,/#(H@UF!G#!,##!(1&(!(1#!Mh\!9%*<$4#,!&!7$%(1#%!
D!<f!8&6"!*2#%!(1#!ih\F!f*(1!(%&",7*%5,!)#%#!&99'6#<!*"(*!
(1#!%*),!*7!(1#!)#681(!5&(%64#,F!
! X68F!D!-&S#0!6''$,(%&(#!(1#!<6J#%#"4#!3#()##"!(1#!ih\!&"<!
(1#!Mh\F!.*",6<#%!&!,659'#!5$'(6H'&;#%!9#%4#9(%*"!
!
𝑦 = 𝑓(x|Θ
, Θ
, . . . , Θ
),!
-@C
0
!
6"!)1641!x ∈ ℝ
=!&"<!𝑦 ∈ ℝF!_$99*,#!"*)!(1&(!#'#5#"(,!𝜃!*7!
Overhead (%)
Fig. 6. ResNet-18 transform overheads as percentages of the number
of elements in
Θ
l
. Row (left) and column (right) transform overheads
shown. Layers 7, 12, 17, and 20 are fully connected.
Layer number (𝑙)
Layer number (𝑙)
Fig. 5. Log10 output distortion (left plots) and top-1 accuracy (right plots) of Resnet-18 and 50 quantized with row (solid curves) and column (dotted
curves) variants of the KLT (blue curves) and the ELT (orange curves). All transforms have better rate–accuracy trade-offs than no transform (gray
curves), providing a rate saving of 1 bit across all rates. Bit-rates of the KLT and the ELT include bits spent on the transform matrices.
Bit-rate (bits per weight) Bit-rate (bits per weight) Bit-rate (bits per weight) Bit-rate (bits per weight)
Log
10
Distortion
Top-1 accuracy (%)
ResNet-18
ResNet-18
ResNet-50
ResNet-50
6 IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. 0, NO. 0, MAY 2021
4*'$5",!θ
=
∈ ℝ
!*7!,*5#!Θ
∈ ℝ
×
=!,1*)"!&,!<*(,!
6"!-&0=!&%#!4*%%#'&(#<=!&,!&%#!(1#!#'#5#"(,!𝛾!*7!(1#!<#%62&(62#,!
γ
= 𝜕𝑦 𝜕θ
⁄ ∈ ℝ
!7*%!&''!𝑗F!G#!4&"!26,$&'6Q#!(1#!5&996"8!3;!
(1#!ih\!U
!*7!θ!&,!&"!*%(1*8*"&'!9%*W#4(6*"!*7!θ *"(*!6(,!()*!
9%6"469&'!&Y#,!𝐮
!&"<!𝐮
F!k$&"(6Q6"8!(1#!9%*W#4(#<!#'#5#"(,!
t = U
θ!5&9,!t!(*!(1#!4#"(#%,!*7!,K$&%#!4#'',!-30F!/*(6"8!(1&(!
<6,(*%(6*"!4&"!3#!)%6:#"!&,!∥C
⁄
(θ − θ
)∥
=!(1#!,K$&%#!4#'',!
3#4*5#!9&%&''#'*8%&5,!6"!(1#!<*5&6"! *7!𝜻 = C
⁄
θ!-40!&"<!
6"4$%!'&%8#%!<6,(*%(6*"!(1&"!,K$&%#!4#'',!*7!(1#!,&5#!&%#&F!O"!
(1#!*(1#%!1&"<=!(1#!Mh\!U
!*7!θ!6,!&!36*%(1*8*"&'!9%*W#4(6*"!
*7!θ!*"(*!"*"*%(1*8*"&'!2#4(*%,!𝛖
!&"<!𝛖
!-&0F!k$&"(6Q&(6*"!
*7!t = U
θ!"*)!5&9,!t (*!(1#!4#"(#%,!*7!9&%&''#'*8%&5!4#'',!
-<0F!\1#,#!9&%&''#'*8%&5!4#'',!5&9!(*!,K$&%#,!6"!(1#!<*5&6"!
*7!𝜻!-#0=!&"<!56"656Q#,!<6,(*%(6*"!∥C
⁄
(θ − θ
)∥
!&,!&!%#,$'(F!!
! X68F!E!-'#7(0!,1*),!(1&(!(1#!Mh\!-*%&"8#!4$%2#,0!9%*<$4#,!
'*)#%!*$(9$(!<6,(*%(6*"!(1&"!(1#!ih\!-3'$#!4$%2#,0!7*%!%*)H!
-,*'6<0!&"<!4*'$5"H!-<*:#<0!2&%6&"(,!*7!(1#!()*!(%&",7*%5,!
6"!(1#!4&,#!)1#%#!P#,/#(,!&%#!K$&"(6Q#<F!\1#!4'&,,6>4&(6*"!
&44$%&4;!&,,*46&(#<!)6(1!#&41!<6,(*%(6*"!6,!8%&91#<!6"!X68F!E!
-%681(0F!G#!,##!(1&(!(1#!ih\!,(6''!9%*26<#,!8**<!4'&,,6>4&(6*"!
9#%7*%5&"4#!)16'#!3#6"8!,68"6>4&"(';!#&,6#%!(*!4*",(%$4(!&,!
6(!<*#,!"*(!%#K$6%#!(1#!<#%62&(62#,!(𝜕y 𝜕θ⁄ )F!P*)!(%&",7*%5,!
9#%7*%5!3#:#%!(1&"!4*'$5"!*"#,=!,$88#,(6"8!)#681(,!1&2#!&!
'&%8#%!4*%%#'&(6*"!)6(16"!%*),!(1&"!6"!4*'$5",F!G#!%#'&(#!(1#!
ih\!&"<!(1#!Mh\!(*!_][!&"<!m_][!6"!a99#"<6Y!fF!
4.3 Quantizing Transform Bases
a'(1*$81!(1#!4*<6"8!8&6"!#Y9%#,,6*"!6"!-@@0!9%*26<#,!&!8**<!
6"<64&(6*"!*7!(1#!#Y9#4(#<!%&(#!,&26"8,!<$#!(*!(%&",7*%5=!)#!
"##<!(*!&<<6(6*"&'';!4*$"(!(1#!36(,!,9#"(!*"!K$&"(6Q6"8!&"<!
,(*%6"8!(1#!6"2#%,#!(%&",7*%5!S = U
−
=!%#K$6%#<!&(!6"7#%#"4#!
(65#=!(*8#(1#%!)6(1!'&;#%!T!(*!%#4*",(%$4(!(1#!*%686"&'!'&;#%!
ΘF!\1#!*2#%1#&<!&5*$"(!<#9#"<,!*"!)1#(1#%!)#!(%&",7*%5!
(1#!4*'$5",!-T = U
Θ0!*%!(1#!%*),!-T = ΘU0!*7!(1#!)#681(,!
Θ ∈ ℝ
×
×
F!a8&6"=!,$99*,#!(1#!7*%5#%=!&"<!𝑛 < 𝑚𝑎𝑏F!Z"!(16,!
4&,#=!*2#%1#&<,!<$#!(*!(1#!(%&",7*%5!&%#!(1#!𝑛
!#'#5#"(,!*7!
5&(%6Y!SF!Z7!𝑛 ≥ 𝑚𝑎𝑏=!1*)#2#%=!*"';!(1#!>%,(!!𝑚𝑎𝑏!-"*"HQ#%*0!
%*),!*7!T!"##<!3#!,(*%#<=!(*8#(1#%!)6(1!(1#6%!4*%%#,9*"<6"8!
𝑚𝑎𝑏!4*'$5",!*7!SF!a,!&!%#,$'(=!(1#!(*(&'!"$53#%!*7!*2#%1#&<,!
6,!min(𝑛
, (𝑚𝑎𝑏)
)!%#'&(62#!(*!K$&"(6Q6"8!Θ!<6%#4(';!)6(1*$(!
(%&",7*%5F!G#!4&"!&"&';Q#!(1#!%*)H(%&",7*%5!4&,#!,656'&%';F!
! X68F!L!9'*(,!(1#!(%&",7*%5HK$&"(6Q&(6*"!*2#%1#&<,!7*%!(1#!
'&;#%,!*7!P#,/#(H@U!&,!9#%4#"(&8#,!*7!(1#!"$53#%!*7!)#681(,!
6"!#&41!*"#F!G#!4*",6<#%!(1#!*2#%1#&<!7*%!3*(1!(1#!%*)H!&"<!
(1#!4*'$5"H(%&",7*%5#<!4&,#,F!Z"!3*(1!4&,#,=!(1#!(%&",7*%5,!
&<<!*2#%1#&<,!*7!%*$81';!@Tp!*7!(1#!"$53#%!*7!)#681(,F!\*!
&''*4&(#!36(H<#9(1,!*9(65&'';!(*!T!&"<!S!&(!,*5#!862#"!%&(#S!
<6,(*%(6*"!(%&<#H*J!𝜆=!)#!5$,(!,&(6,7;!*9(65&'6(;!4*"<6(6*",!
-U0!*"!(1#!%*),!*7!𝐓= &"<!4*'$5",!*7!SF!f6(H<#9(1!&''*4&(6*"!
9%*4#<$%#!7*%!𝐓!&"<!S!6,!862#"!6"!a'8*%6(15!@F!Z"9$(!Z!4&"!
3#!(%&",7*%5!<*5&6"!𝐓= *%!-(1#!(%&",9*,#!*70!(1#!3&,6,!SF!
! a(! '*)#%! 36(H%&(#,=!'#,,!,68"6>4&"(!%*),!*7!T ∈ ℝ
×
×
5&;!
3#!K$&"(6Q#<!,*!1#&26';!(1&(!*"';!,*5#!>%,(!𝑘 < 𝑛!%*),!*7!T!
&%#!"*"HQ#%*!-4*'$5"HMh\!&"<!4*'$5"Hih\!,*%(!(1#!%*),!*7!
T!6"!&!"*"H&,4#"<6"8!*%<#%!*7!2&%6&"4#,0F!Z"!(16,!4&,#=!)#!4&"!
#2&'$&(#!(1#!5&996"8!x ↦ Θx!#I46#"(';!6"!(1#!(%&",7*%5#<!
<*5&6"!26&!x ↦ S
(T
x)=!)1#%#!5&(%64#,!S
∈ ℝ
×
×
!&"<!!
T
∈ ℝ
×
×
!&%#!(1#!>%,(!𝑘!4*'$5",!&"<!(1#!%*),!*7!S!&"<!𝐓!
%#,9#4(62#';F!_6"4#!(1#!>%,(!5&996"8!x ↦ T
x!#"(&6',!'&%8#%!
𝑎 × 𝑏!4*"2*'$(6*",!)1#%#&,!(1#!,#4*"<!*"#!z ↦ S
z=!1 × 1!
*"#,=!)#!&416#2#!&!'&%8#!6"7#%#"4#!,9##<H$9!)1#"!𝑘!6,!,5&''F!!
! O"#!4&"!#Y9%#,,!(16,!6"7#%#"4#!,9##<H$9!&,!(1#!%&(6*!
!
𝐴 = (1
𝑛𝑘 + 𝑎𝑏𝑚𝑘) (𝑎𝑏𝑚𝑛)⁄ ,!
-@D
0
!
(1&(!6,=!(1#!%&(6*!*7!(1#!"$53#%!*7!5$'(69'64&(6*",!6"!(1#!5&9!
x ↦ Θx!(*!(1&(!6"!x ↦ S
(T
x)F!\;964&'';=!𝑎𝑏𝑚 ≫ 1
𝑛=!,*!
(1#!&44#'#%&(6*"!𝐴 ≈ 𝑘/𝑛!5&;!3#!$"<#%,(**<!&,!(1#!7%&4(6*"!
*7!"*"HQ#%*!%*),!*7!ΘF!Z"!X68F!R=!)#!8%&91!(1#!9#%4#"(&8#!*7!
"*"HQ#%*!%*),!*7!P#,/#(H@U!&"<!P#,/#(HCD!)#681(!5&(%64#,!
Θ!&(!@SB!36(,=!6''$,(%&(6"8!(1&(!,68"6>4&"(!&44#'#%&(6*"!4&"!3#!
*3(&6"#<F!a2#%&86"8!-@D0!&4%*,,!&''!𝐿!'&;#%,!)#681(#<!3;!(1#!
,6Q#,!*7!(1#!6"9$(!&4(62&(6*",!862#,!$,!(1#!*2#%&''!&44#'#%&(6*"!
6"!(1#!"$53#%!*7!j*&(6"8H9*6"(!*9#%&(6*",!-XhO`,0F!
5 OPTIMIZING QUANTIZATION
G#!"*)!<6,4$,,!*9(65&'!,4&'&%!K$&"(6Q&(6*"!*7!(1#!#'#5#"(,!
*7!t = U
θF!G16'#!2#4(*%!?RRA!&"<!<#&<HQ*"#!?RUA!K$&"(6Q#%,!
4&"!7$%(1#%!659%*2#!$9*"!(1#!%&(#S&44$%&4;!9#%7*%5&"4#!*7!
(%&",7*%5!K$&"(6Q&(6*"=!)#!*9(!(*!$,#!,659'#!$"67*%5!,4&'&%!
K$&"(6Q&(6*"!(*!&44#'#%&(#!6"7#%#"4#!$,6"8!6"(#8#%!&%6(15#(64!
<6%#4(';!*"!K$&"(6Q&(6*"!6"<64#,F!
Algorithm 1.!O9(65&'!f6(H<#9(1!a''*4&(6*"!
& Input:!Z ∈ ℝ
×
×
, 𝜆, 𝐷
, Δ
, 𝑘 = 1, . . . , 𝑚!
: Output: Δ
opt
, 𝑅
opt
, 𝑘 = 1, . . . , 𝑚!
; for!𝑘 = 1, . . . , 𝑚!do
<! ! 𝐽 = ∞, 𝑅
= 0! !
=! ! for!𝑅 = 0, . . . , 𝑀!do!
> if 𝐽 > 𝐷
(𝑅) + 𝜆𝑚𝑛𝑎𝑏𝑅!then
? 𝐽 = 𝐷
(𝑅 ) + 𝜆𝑚𝑛𝑎𝑏𝑅!
@ 𝑅
opt
= 𝑅!
A! ! ! ! !Δ
opt
= Δ
(𝑅)!
&B! ! ! end if!
&& end for!
&: end for!
!
Fig. 7. Percentages of non-zero rows of transformed weight matrices T
l
at bit-rates R = 1.1 and 2.1 bits for ResNet-34 (left), and R = 1.1 and 3.1
bits for ResNet-50 (right). Percentages of non-zero rows at lower and higher bit-rates visualized as blue and blue + orange bars, respectively. See
Fig. 9 for retrained and non-retrained classification accuracies at these bit-rates.
Non-zero rows (%)
ResNet-34 layer (𝑙)
ResNet-50 layer (𝑙)
YOUNG ET AL.: TRANSFORM QUANTIZATION FOR CNN COMPRESSION 7
5.1 Finding Optimal Step-sizes
\*!>"<!(1#!*9(65$5!K$&"(6Q&(6*"!,(#9H,6Q#!Δ!7*%!&!%&"<*5!
,*$%4#!t!&(!&!862#"!36(H<#9(1!𝑅=!)#!4&"!#2&'$&(#!*2#%!&!%&"8#!
*7!Δ=! *$(9$(!<6,(*%(6*"!𝐷
out
= 𝔼 ‖𝐲 − 𝐲‖
!<$#! (*! K$&"(6Q6"8!
,*$%4#!t
(Δ)=!&"<!>"<!&!2&'$#!*7!Δ!(1&(!56"656Q#,!𝐷
out
!-(1#!
"$53#%!*7!K$&"(6Q&(6*"!'#2#',!6,!>Y#<!&(!2
0F!_(#9H,6Q#,!(1&(!
56"656Q#!(1#!<6,(*%(6*"!𝐷
src
= 𝔼 ‖t − t
‖
!*7!(1#!,*$%4#!6(,#'7!
<#26&(#!,68"6>4&"(';!7%*5!(1#!*"#!)1641!56"656Q#,!*$(9$(!
<6,(*%(6*"!𝐷
out
F!X68F!U!-'#7(0!,1*),!(16,!7*%!(1#!ih\!,*$%4#!t!*7!
(1#!"6"(1!'&;#%!*7!P#,/#(H@UF!\16,!<6,4%#9&"4;!6"!,(#9H,6Q#,!
%#,$'(,!6"!&!@SB!36(H%&(#H'*,,!-X68F!U=!%681(0F!\1#!41&%&4(#%6,(64!
,1&9#H𝑉 !4$%2#,!6"!(1#!%681(!9'*(,!&%#!&:%63$(#<!(*!&"!6"6(6&'!
<#4%#&,#!6"!*2#%'*&<!<6,(*%(6*"!&,!,(#9H,6Q#!Δ!6"4%#&,#,=!(1#"!
&"!6"4%#&,#!6"!8%&"$'&%!<6,(*%(6*"!9&,(!*9(65$5!Δb,##!?RRA!!
7*%!7$%(1#%!<6,4$,,6*"!*"!8%&"$'&%!&"<!*2#%'*&<!<6,(*%(6*",F!
! f&""#%!#(!&'F!?CUA!<#%62#<!&"!#Y9%#,,6*"!(*!%#'&(#!*9(65&'!
K$&"(6Q&(6*"!,(#9H,6Q#,!(*!(1#!2&%6&"4#!*7!(1#!%&"<*5!,*$%4#!
6"!(1#!h&9'&4#H<6,(%63$(#<!4&,#F!V*)#2#%=!X68F!U!,$88#,(,!(1#!
K$&"(6Q&(6*"!,(#9H,6Q#!,1*$'<!3#!*9(656Q#<!*2#%!(1#!*$(9$(!
<6,(*%(6*"!#Y9'646(';!$,6"8!&!8%6<!,#&%41F!\16,!#Y9'646(!,#&%41!
&',*!*326&(#,!(1#!"##<!(*!>(!&!9&%&5#(%64!<6,(%63$(6*"!*"!(1#!
$"<#%';6"8!,*$%4#=!&"<!6,!&',*!&99'64&3'#!(*!K$&"(6Q#%,!)6(1!
<#&<HQ*"#,!?RUA=!&"<!"*"H$"67*%5!4#'',!?R^=!UTAF!a'8*%6(15!B!
'6,(,!(1#!9%*4#<$%#!7*%!*9(65&'!K$&"(6Q&(6*"!,(#9H,6Q#!,#&%41!
*"!TF _(#9H,6Q#!,#&%41#,!*"!S!6,!,656'&%=!3$(!6"2*'2#,!S
!&"<!
T!6",(#&<!*7!T
!&"<!S!-3#6"8!S
!"*)!(1#!𝑘(1!4*'$5"!*7!S0F!
5.2 Reducing Quantization Complexity
\*!5&Y656Q#!4*59%#,,6*"=!*"#!4*$'<!&''*4&(#!&"!6"<626<$&'!
36(H<#9(1!(*!#&41!%*)!*7!T!&"<!#&41!4*'$5"!*7!SF!\16,!4&"!3#!
(65#H4*",$56"8!7*%!'&;#%,!1&26"8!5&";!41&""#',!,6"4#!*"#!
"##<,!(*!>%,(!4*",(%$4(!&!36(H<#9(1S<6,(*%(6*"!4$%2#!7*%!#&41!
%*)!*7!T=!&"<!4*'$5"!*7!SF!M&41!36(H<#9(1S<6,(*%(6*"!4$%2#!
%#K$6%#,!6"!($%"!4*59$(6"8!(1#!*$(9$(!<6,(*%(6*"!7*%!&!'&%8#!
"$53#%!*7!-36(H<#9(1=!,(#9H,6Q#0!9&6%,F!\*!,&2#!(65#!%#K$6%#<!
(*!4*",(%$4(!(1#,#!4$%2#,=!)#!(1#%#7*%#!9&%(6(6*"!T -%#,9F!S0!
6"(*!𝐵!3'*4+,!*7!%*),!-%#,9F!4*'$5",0=!&"<!&''*4&(#!6",(#&<!&!
,6"8'#!36(H<#9(1!(*!#&41!3'*4+F!\16,!1&,!2#%;!'6:'#!659&4(!*"!
%&(#S<6,(*%(6*"!9#%7*%5&"4#!67!(1#!"$53#%!*7!3'*4+,!6,!'&%8#!
#"*$81b(1#! *$(9$(! 2&%6&"4#! *7! #'#5#"(,! *7!𝐓 -%#,9F!S0! 6,!
5*"*(*"64H<#4%#&,6"8!&4%*,,!(1#!%*),!-%#,9F!4*'$5",0=!&"<!
(1#!*$(9$(!2&%6&"4#,!%*$81';!(1#!,&5#!)6(16"!#&41!3'*4+F!Z7!
(1#!3'*4+,!4&"!3#!&''*4&(#<!&!5&Y65$5!36(H<#9(1!*7!𝑀=!(1#"!
(65#!,9#"(!*"!4*",(%$4(6"8!36(H<#9(1S<6,(*%(6*"!4$%2#,!*7!&''!
'&;#%,!6,!9%*9*%(6*"&'!(*!𝐵𝑆𝐼𝑀𝑃𝐿=!)1#%#!𝑆!6,!(1#!"$53#%!*7!
K$&"(6Q&(6*"!,(#9H,6Q#,!,#&%41#<=!𝐼=!"$53#%!*7!65&8#,!$,#<!
(*!8#"#%&(#!(1#!36(H<#9(1S<6,(*%(6*"!4$%2#,=!𝑃=!9#%7*%5&"4#!
*7!&!862#"!.//!6"!,#4*"<,=!&"<!𝐿!(1#!"$53#%!*7!'&;#%,F!\1#!
(;964&'!9&%&5#(#%,!&"<!&,,*46&(#<!(65#,!7*%!*9(656Q6"8!36(H!
<#9(1!&%#!,1*)"!6"!\&3'#!@!7*%!,#2#%&'!"#()*%+,F!
5.3 Fine-tuning Quantized Weights
G1#%#&,!>"#H($"6"8!*7!(1#!K$&"(6Q#<!"#()*%+,!6,!"*(!&')&;,!
9*,,63'#!<$#!(*!'&4+!*7!,$6(&3'#!(%&6"6"8!<&(&!&(!4*59%#,,6*"!
(65#=!)#!<#5*",(%&(#!(1&(!*9(6*"&'';!>"#H($"6"8!(%&",7*%5H
K$&"(6Q#<!.//,!4&"!%#,(*%#!&44$%&46#,!4'*,#!(*!(1#!*%686"&'!
*"#,F!\*!%#H(%&6"!(1#!(%&",7*%5HK$&"(6Q#<!"#()*%+,=!)#!$,#!
(1#!,(%&681(H(1%*$81!#,(65&(*%!-_\M0!&99%*&41!*7!V$3&%&!#(!
&'F!?C@=!D@A!(*!K$&"(6Q#!(1#!)#681(,!<$%6"8!(1#!7*%)&%<H9&,,!
&44*%<6"8!(*!(1#!&,,68"#<!36(H<#9(1,=!&"<!'#(!(1#6%!8%&<6#"(,!
9&,,!(1%*$81!)6(1!"*!41&"8#!<$%6"8!(1#!3&4+)&%<!9&,,F!G#!
9'*(!6"!X68F!^!(1#!36(H%&(#S&44$%&4;!4$%2#,!*7!P#,/#(,!*"!(1#!
Z5&8#/#(!?U@A!2&'6<&(6*"!,#(!)6(1!&"<!)6(1*$(!%#(%&6"6"8!*"!
(1#!(%&6"6"8!,#(F!\1#!4'&,,6>4&(6*"!&44$%&46#,!4&"!3#!%#,(*%#<!
4'*,#!(*!(1#!*%686"&'!*"#,!&(!B!36(,=!&"<!&'5*,(!'*,,'#,,';!&(!C!
36(,F!a''!5*<#',!)#%#!%#(%&6"#<!7*%!C!#9*41,!5&Y65$5!)6(1!
Algorithm 2.!O9(65&'!_(#9H,6Q#!_#&%41!
& Input:!S ∈ ℝ
×
×
, T ∈ ℝ
×
×
!
: Output: Δ
, 𝐷
, 𝑘 = 1, . . . , 𝑛!
; for!𝑘 = 1, . . . , 𝑛!do
< for 𝑅 = 1, . . . , 𝑀!do
=! ! ! 𝐷
(𝑅 ) = ∞, Δ
(𝑅) = 0!
>! ! ! for!Δ = 2
, 2
, 2
, . . . !do!
? T
= T!!qr!Z"6(6&'6Q#!)6(1!$"K$&"(6Q#<!)#681(,!rq!
@ T
= 𝑞(T
, 𝑅, Δ)!! qr!T
!6,!(1#!𝑘(1!%*)!*7!T!rq
A! ! ! ! y = 𝑓(x|Θ
, . . . ,ST
, . . . , Θ
)!
&B! ! ! ! if!𝐷
(𝑅) > 𝔼‖y − y‖
!then
&& 𝐷
(𝑅 ) = 𝔼‖y − y‖
!
&:! ! ! ! ! Δ
(𝑅) = Δ!
&;! ! ! ! end if
&<! ! ! end for! !
&= end for!
&> end for
!
Fig. 8. Weight and output distortions against quantization step-size Δ at bit-depths of R = 2 and 4 (left, shown for the first row-KLT element of layer
9 in ResNet-18). The step-size that minimizes the weight distortion
D
src
(orange lines) does not necessarily minimize the output distortion D
out
and
incurs a 1–2-bit-rate loss compared with using the optimal quantization step-size for the output (right, shown for layers 9 and 18 of ResNet-18).
Log
10
distortion
𝐷
src
𝐷
out
𝐷
(
𝑅
)
!
!
2
6
𝑅 = 8
4
𝑙 = 9
Log2 step-size (Δ) Log2 step-size (Δ) Log2 step-size (Δ) Log2 step-size (Δ)
𝑅 = 2
𝑅 = 4
𝑙 = 18
TABLE 1
Quantization times of different CNNs on Intel Xeon 6132
@ 2.60GHz + Nvidia Quadro RTX 8000
Model
Weights
Layers
Blocks
Steps
Maxbits
Perf.
Cost
C*5DE5$!
>:F;?@G!
@!
@!
@!
&>!
?'BH%!
&'>4!
I5%E5$J&@!
&&F>?AG!
:&!
@!
@!
&>!
?'BH%!
<':4!
I5%E5$J;<!
:&F?@BG!
;?!
@!
@!
&>!
?'&H%!
?'=4!
I5%E5$J=B!
:=F=B;G!
=<!
@!
@!
&>!
?':H%!
&B'A4!
K5.%5E5$J&:&!
?F@A<G!
&:&!
<!
@!
&>!
?'>H%!
&;'&4!
!
8 IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. 0, NO. 0, MAY 2021
_m[=!&"<!(1#!&4($&'!"$53#%!*7!6(#%&(6*",!*9(656Q#<!$,6"8!&!
8%6<!,#&%41!(*!&416#2#!56"65$5!2&'6<&(6*"!#%%*%F!/*(#!(1&(!
(%&",7*%5!3&,#,!S &%#!&',*!%#(%&6"#<!,*!(1#%#!6,!"*!<6J#%#"4#!
3#()##"!(1#!ih\!&"<!(1#!Mh\!4&,#,!&7(#%!%#(%&6"6"8F!
6 EXPERIMENTAL RESULTS
\*!#Y&56"#!(1#!%&(#S<6,(*%(6*"!&"<!%&(#S&44$%&4;!3#1&26*%,!
*7!(%&",7*%5HK$&"(6Q#<!"#()*%+,=!)#!4*59%#,,!&!"$53#%!*7!!
9*9$'&%!9%#(%&6"#<!65&8#!4'&,,6>4&(6*"!"#()*%+,ba'#Y/#(!
?@A=!P#,/#(,!?LTA=!&"<![#",#/#(,!?L@Ab&,!)#''!&,!9%#(%&6"#<!
65&8#!,$9#%%#,*'$(6*"!"#()*%+!M[_P!?LCA=!6"!3*(1!%#(%&6"#<!
&"<!"*"H%#(%&6"#<!,4#"&%6*,F!G#!,1*)!(%&",7*%5HK$&"(6Q#<!
.//,!&<2&"4#!(1#!,(&(#!*7!(1#!&%(!6"!3*(1!,4#"&%6*,F!X*%!7&6%!
4*59&%6,*",=!)#!4*59&%#!)6(1!*(1#%!%#,$'(,!(1&(!<*!"*(!$,#!!
#"(%*9;!4*<6"8=!#F8F!V$J5&"!?BTA!*%!&%6(15#(64!?DEAF!G#!4&"!
&')&;,!&99';!,(&"<&%<!#"(%*9;!-'*,,'#,,0!4*<6"8!(*!&";!*7!(1#!
K$&"(6Q#<!.//,!7*%!&!7$%(1#%!sCTp!%#<$4(6*"!6"!36(H%&(#,F!
Image classification accuracy.!\&3'#! B!,1*),!(1#!&44$%&46#,!
*7!(%&",7*%5HK$&"(6Q#<!a'#Y/#(=!P#,/#(,!&"<![#",#/#(,!*"!
Z5&8#/#(!?U@A!-9%#(%&6"#<!`;\*%41!2#%,6*",0!&(!@SC!36(,F!\1#!
&44$%&46#,!&%#!#2&'$&(#<!$,6"8!(1#!Z5&8#/#(BT@B!2&'6<&(6*"!
,#(!4*",6,(6"8!*7!ET+!65&8#,F!O$%!.//,!&%#!K$&"(6Q#<!$,6"8!
TABLE 2
Classification accuracies of CNN models compressed using transform quantization and other methods.
Methods
Train
epochs
Comp
ratio
Wgt/Act
Classification accuracy
bit-rate
L-MJ&!2N6!
L-MJ=!2N6!
!
I5%E5$J&@!
!
O0**JM95,#%#-.
!
P
!
P
!
;:Q;:
!
>A'?!2RB'B6
!
@A':!2RB'B6
!
!
STE!U:@V!
&>!
;:'B×!
&Q;:!
>B'@!2P@'A6!
@;'B!2P>':6!
!
LTE!U:AV!
=B!
&>'B×!
:Q;:!
>=';!2P<'<6!
@>':!2P;'B6!
!
LLW!U;BV
!
&>B
!
&>'B×
!
:Q;:
!
>>'>!2P;'&6
!
@?':!2P:'B6
!
!
XEW!U;>V!
@!
&>'B×!
:Q;:!
>>'B!2P;'?6!
@?'&!2P:'&6!
!
XEW!U;>V!
@!
&B'?×!
;Q;:!
>@'&!2P&'>6!
@@'<!2PB'@6!
!
XEW!U;>V
!
@
!
@'B×
!
<Q;:
!
>@'A!2PB'@6
!
@A'B!2PB':6
!
!
XEW!U;>V!
@!
>'<×!
=Q;:!
>A'B!2PB'?6!
@A'&!2PB'&6!
!
YWJE5$%!U;?V!
&:B!
&>'B×!
:Q;:!
>@'B!2P&'?6!
@@'B!2P&':6!
!
YWJE5$%!U;?V
!
&:B
!
&B'?×
!
;Q;:
!
>A';!2PB'<6
!
@@'@!2PB'<6
!
!
YWJE5$%!U;?V!
&:B!
@'B×!
<Q;:!
?B'B!2RB';6!
@A'&!2PB'&6!
!
KZW!U<BV!
AB!
;:'B×!
&Q;:!
>;'?!2P>'B6!
P!
!
KOW!U<;V
!
B
!
=';×
!
>Q;:
!
>>';!2P;'<6
!
P
!
!
[09%!29-3J\YL6!
B!
&B'?×!
;'BQ;:!
>?'A!2P&'@6!
@@'&!2P&'&6!
!
[09%!29-3J\YL6!
B!
@':×!
;'AQ;:!
>@'>!2P&'&6!
@@'=!2PB'?6!
!
[09%!29-3J]YL6
!
B
!
&B'?×
!
;'BQ;:
!
>?':!2P:'=6
!
@?'@!2P&'<6
!
!
[09%!29-3J]YL6!
B!
@':×!
;'AQ;:!
>@'=!2P&':6!
@@'<!2PB'@6!
!
[09%!R!95$9)#.!
3
32.0×
1.0Q;:
64.5!2–5.26
86.4!2–2.86
[09%!R!95$9)#.
!
3
22.9×
1.4Q;:
67.2!2–2.56
87.8!2–1.46
[09%!R!95$9)#.!
3
16.8×
1.9Q;:
68.2!2–1.56
88.4!2–0.86
[09%!R!95$9)#.!
;!
&B'?×!
;'BQ;:!
>A'@!2RB'&6!
@A';!2RB'&6!
I5%E5$J;<!
!
O0**JM95,#%#-.
!
P
!
P
!
;:Q;:
!
?;';!2RB'B6
!
A&';!2RB'B6
!
!
YJKEW!U;AV!
P!
&B'?×!
;Q;:!
<<'B!2P:A'6!
?:'A!2P&@'6!
!
^CW!U<>V!
&BB!
&=':×!
:'&Q;:!
?&'=!2P&'@6!
AB'&!2P&':6!
!
^CW!U<>V
!
100
10.7×
3.0/32
73.2 2–0.16
91.3 2+0.06
!
[09%!29-3J\YL6!
B!
&&'A×!
:'?Q;:!
?&'=!2P&'@6!
AB';!2P&'B6!
!
[09%!29-3J\YL6!
B!
@'<×!
;'@Q;:!
?:'>!2PB'?6!
AB'A!2PB'<6!
!
[09%!29-3J]YL6
!
B
!
&&'<×
!
:'@Q;:
!
?&'B!2P:';6
!
AB':!2P&'&6
!
!
[09%!29-3J]YL6!
B!
@'<×!
<'BQ;:!
?:'<!2PB'A6!
AB'A!2PB'<6!
!
[09%!R!95$9)#.!
3
29.1×
1.1Q;:
69.6!2–3.76
88.9!2–2.46
[09%!R!95$9)#.
!
3
22.9×
1.4Q;:
71.4!2–1.96
89.7!2–1.66
[09%!R!95$9)#.!
3
15.2×
2.1Q;:
72.8!2–0.56
90.6!2–0.76
[09%!R!95$9)#.!
3
11.9×
2.8Q;:
73.0!2–0.36
90.8!2–0.56
!
Methods
Train
epochs
Comp
ratio
Wgt/Act
Classification accuracy
bit-rate
L-MJ&!2N6!
L-MJ=!2N6
!
I5%E5$J=B
!
O0**JM95,#%#-.
!
P
!
P
!
;:Q;:
!
?>'B!2RB'B6
!
A;'B!2RB'B6
!
XEW!U;>V!
@!
>'<×!
=Q;:
!
?<'@!2P&':6!
A:'<!2PB'>6
!
YWJE5$%!U;?V!
&:B!
&>'B×!
:Q;:
!
?='&!2PB'A6!
A:';!2PB'?6
!
YWJE5$%!U;?V
!
120
8.0×
!
4Q;:
!
76.4!2+0.46
!
A;'&!2RB'&6
!
CS_K!U:;V!
A!
;:'B×!
&'BQ;:
!
>@':!2P?'@6!
P
!
CS_K!U:;V!
A!
&@'@×!
&'?Q;:
!
?;'@!2P:':6!
P
!
^CW!U<>V
!
&BB
!
&B'>×
!
;'BQ;:
!
?=';!2PB'?6
!
A:'=!2PB'=6
!
^CW!U<>V!
&BB!
@'B×!
<'BQ;:
!
?>'&!2RB'&6!
A:'A!2PB'&6
!
[09%!29-3J\YL6!
B!
&&'<×!
:'@Q;:
!
?;'>!2P:'<6!
A&'?!2P&';6
!
[09%!29-3J\YL6
!
B
!
@'<×
!
;'@Q;:
!
?<'?!2P&'&6
!
A:'<!2PB'>6
!
[09%!29-3J]YL6!
B!
&B';×!
;'&Q;:
!
?;'>!2P:'<6!
A&'?!2P&';6
!
[09%!29-3J]YL6!
B!
?'@×!
<'&Q;:
!
?<'@!2P&':6!
A:';!2PB'?6
!
[09%!R!95$9)#.!
!
3
29.1×
1.1Q;:
72.9!2–3.16
91.4!2–1.66
[09%!R!95$9)#.!
3
21.3×
1.5Q;:
74.2!2–1.86
92.2!2–0.86
[09%!R!95$9)#.!
3
16.8×
1.9Q;:
75.2!2–0.86
92.6!2–0.46
[09%!R!95$9)#.
!
3
11.4×
3.1Q;:
76.0!2+0.06
93.0!2+0.06
C*5DE5$
!
O0**JM95,#%#-.!
P!
P!
;:Q;:
!
=='?!2RB'B6!
?@'>!2RB'B6
!
STE!U:@V
!
=@
!
;:'B×
!
&Q;:
!
56.8!2+1.16
!
79.4!2+0.86
!
K-I5O)!U<:V
!
:BB
!
;:'B×
!
&Q;:
!
=;'A!2P&'@6
!
?>';!2P:';6
!
LTE!U:AV!
=B!
&>'B×!
:Q;:
!
=<'=!2P&':6!
?>'@!2P&'@6
!
LLW!U;BV
!
&>B
!
&>'B×
!
:Q;:
!
=?'=!2R&'@6
!
?A'?!2R&'&6
!
XEW!U;>V
!
@
!
>'<×
!
=Q;:
!
=?'<!2R&'?6
!
@B'=!2R&'A6
!
YWJE5$%!U;?V!
&:B!
16.0×!
2Q;:
!
60.5 2+4.86
82.7 2+4.16
[09%!29-3J\YL6
!
B
!
;:'B×
!
&'BQ;:
!
=;'<!2P:';6
!
??'&!2P&'=6
!
[09%!29-3J\YL6
!
B
!
&>'B×
!
:'BQ;:
!
=='&!2PB'>6
!
?@':!2PB'<6
!
[09%!29-3J]YL6!
B!
;:'B×!
&'BQ;:
!
=;'&!2P:'>6!
??'B!2P&'>6
!
[09%!29-3J]YL6
!
B
!
&>'B×
!
:'BQ;:
!
=='B!2PB'?6
!
?@':!2PB'<6
!
[09%!R!95$9)#.
!
3
64.0×
0.5Q;:
53.0!2–2.76
77.0!(–1.66
[09%!R!95$9)#.!
;!
;:'B×!
&'BQ;:
!
==';!2PB'<6!
?@'=!2PB'&6
!
K5.%5E5$J&:&
!
O0**JM95,#%#-.
!
P
!
P
!
;:Q;:
!
?='B!2RB'B6
!
A:';!2RB'B6
!
[09%!29-3J]YL6!
B!
8.0×!
4.0Q;:
!
72.6!2–2.46!
91.1!2–1.26
!
[09%!29-3J]YL6!
B!
6.4×!
5.0Q;:
!
73.7!2–1.36!
91.7!2–0.66
!
!
Top-1 accuracy (%)
Bit-rate (bits per weight) Bit-rate (bits per weight) Bit-rate (bits per weight) Bit-rate (bits per weight)
ResNet-18
ResNet-34
Fig. 9. Rate–accuracy curves of transform-quantized AlexNet and ResNets on the ImageNet validation set, with retraining (blue lines), and without
retraining (orange lines). Unquantized baselines shown as gray lines. All CNNs retrained on the training set for 3 epochs maximum. Shown for the
row-KLT case. Retrained row-ELT results are similar (no distinction exists between the KLT and the ELT after retraining).
ResNet-50
AlexNet
YOUNG ET AL.: TRANSFORM QUANTIZATION FOR CNN COMPRESSION 9
%*)HMh\!&"<!%*)Hih\F!X*%!%#7#%#"4#=!)#!6"4'$<#!(1#!%#,$'(,!
*7![Xk!?DCA=!hkH/#(,!?CRA=!Z/k!?CLA=!afm[!?BCA![*P#X&/#(!
?DBA!&"<!Vak!?DLAF!G#!,##!(1&(!*$%!"*"H%#(%&6"#<!P#,/#(H@U!
%#,$'(,!&%#!,68"6>4&"(';!3#:#%!(1&"!(1*,#!7*%!(1#![Xk!-<&(&H
7%##!K$&"(6Q&(6*"0!5#(1*<!?DCAF!Z"!(1#!4&,#!)1#%#!%#(%&6"6"8!
6,!9#%7*%5#<=!%*)Hih\!P#,/#(,!1&2#!&!4*",6<#%&3';!1681#%!
&44$%&4;!(1&"!Vak!&(!'*)!36(,!-P#,/#(HCD=!ET0=!&"<!,'681(';!
5*%#!&44$%&(#!*2#%&''!(1&"!hkH/#(,!-P#,/#(H@U=!ET0!3$(!)6(1!
C!#9*41,!*7!%#76"6"8F!hkH/#(,!&',*!%#K$6%#,!(1#!36(H<#9(1,!(*!
3#!<#(#%56"#<!9%6*%!(*!(%&6"6"8!(1#!5*<#'!)16'#!)#!&%#!&3'#!
(*!K$&"(6Q#!&"<!%#76"#!(1#!5*<#',!&(!&%36(%&%;!36(H%&(#,!&7(#%!
(%&6"6"8F!/*(#!(1&(!*$%!9%#(%&6"#<!a'#Y/#(!5*<#'!6,!,5&''#%!
&"<!1&,!&!'*)#%!$"K$&"(6Q#<!3&,#'6"#!(1&"!(1#!*(1#%,F!f*'<!
"$53#%,!6"<64&(#!%#,$'(,!*"!(1#!%&(#S(*9H@!7%*"(6#%F!
Pruning effect of quantization.!G#!,1*)!&44#'#%&(6*"!<$#!(*!
Q#%*!K$&"(6Q&(6*"!-9%$"6"80!*7!'#,,!,68"6>4&"(!+#%"#',!&(!'*)!
36(H%&(#,!-@SC!36(,0F!\&3'#! C! ,1*),!7*%!P#,/#(HET!(1#!*2#%&''!
&44#'#%&(6*"!&(!36(H%&(#,!𝑅 = 1.1!&"<!3.1!36(,=!&'*"8!)6(1!(1#!
%#,$'(,!7%*5!*(1#%!9%$"6"8!5#(1*<,F!G16'#!*$%!XhO`!4*$"(,!
&%#!1681#%!(1&"!(1*,#!*7!,9#46&'6Q#<!9%$"6"8!(#41"6K$#,=!*$%!
XhO`,!%#K$6%#!5$41!'*)#%!36(H<#9(1,F!\1#%#7*%#=!)#!5&;!3#!
&3'#!(*!&416#2#!7$%(1#%!&44#'#%&(6*"!67!,9#46&'6Q#<!1&%<)&%#!
4&"!3#!<#,68"#<!(*!7&46'6(&(#!'*)H36(H<#9(1!&%6(15#(64F!X68F!R!
9%*26<#,!&!9#%H'&;#%!3%#&+,H<*)"!*7!(1#!,9##<$9F!/$53#%,!
6"!3*'<!6"<64&(#!(1#!3#,(!%#,$'(!6"!#&41!4*'$5"F!
Acceleration on H/W.!G#!5#&,$%#!,9##<!$9!7%*5!(%&",7*%5!
K$&"(6Q&(6*"!*"!1681H9#%7*%5&"4#![//H(&%8#(#<!1&%<)&%#!
&44#'#%&(*%,!m**8'#!\`e!?UBA!&"<!NZ\!M;#%6,,!?UCA=!&<*9(6"8!
_.ahMH,65!?UDA!(*!,65$'&(#!(1#!(65#!4;4'#,!*7!N*36'#/#(H2B!
K$&"(6Q#<!$,6"8!*$%!&99%*&41!*%!$,6"8!Vak!?DLAF!/*(#!(1&(!
(1#!4*"2*'$(6*"&'!3'*4+,!*7!N*36'#/#(H2B!&%#!&'%#&<;!6"!(1#!
B[!(%&",7*%5!<*5&6"!-_#4(6*"!R0!,*!)#!*9(65&'';!&,,68"!36(H
<#9(1,!(*!)#681(,!&"<!K$&"(6Q#!(1#5!26&!a'8*%6(15,!@SBF!Z"!
3*(1!Vak!&"<!*$%!&99%*&41=!)#!7#(41!K$&"(6Q#<!)#681(,!(*!
*"H4169!5#5*%;!7%*5!*JH4169=!)1#%#!(1#;!&%#!<#K$&"(6Q#<!
&"<!7#<!6"(*!9%*4#,,6"8!$"6(,F!OJH4169!5#5*%;!6,!&44#,,#<!
6"!(&"<#5!)16'#!4*"2*'$(6*",!&%#!3#6"8!9#%7*%5#<F!Z"!(16,!
#Y9#%65#"(=!)#!&',*!K$&"(6Q#!&''!6"(#%5#<6&(#!&4(62&(6*",!6"!
&!'&;#%H)6,#!7&,16*"!7*%!&!7&6%#%!4*59&%6,*"!)6(1!VakF!O$%!
36(H%&(#,!&%#!&2#%&8#!K$&"(6Q&(6*"!36(H<#9(1,!&4%*,,!)#681(,!
&"<!&4(62&(6*",=!&,,$56"8!(1#!3&(41!,6Q#!*7!*"#F!G#!9%*26<#!
6"!\&3'#! D=!(1#!6"7#%#"4#!%&(#,!-6"!65&8#,!9#%!,#4*"<0!*7!*$%!
K$&"(6Q#<!N*36'#/#(H2B!&(!<6J#%#"(!36(H%&(#,=!&"<!4*59&%#!
Method
Accuracy 2N6
Bit-
rate
FLOP
ratio
Compres-
sion ratio
L-MJ&!2ΔN6!
L-MJ=!2ΔN6!
O0**JM95,#%#-.!
?>'&!2PB'B6!
A:'A!2PB'B6!
;:'B!
&BB'BN!
&'B×
!
Z-1$O`!U@V!
?<'>!2P&'=6!
A:'&!2PB'@6!
;:'B!
=@':N!
&'?×
!
EXZ`!U&AV!
?=':!2PB'A6!
P!
;:'B!
=>'BN!
&'@×
!
O`_a!UAV!
?<'@!2P&';6!
A:';!2PB'>6!
;:'B!
<>'=N!
:':×
!
Kb`!U&<V!
?='B!2P&'&6!
A:';!2PB'>6!
;:'B!
44.2%
:';×
!
_K`!U&;V!
?&'A!2P;':6!
AB'?!2P&'>6!
;:'B!
<@'?N!
:'&×
!
_SEJ=B!U&:V!
?=':!2PB'?6!
A:'<!2PB';6!
;:'B!
<<'AN!
:':×
!
_SEJ>B!U&:V!
76.2 2+0.36!
A:'@!2RB':6!
;:'B!
=A'=N!
&'?×
!
L4#E5$J=B!U&BV!
?&'B!2P&'A6!
AB'B!2P&'&6!
;:'B!
44.2%
:';×
!
L4#E5$J?B!U&BV!
?:'B!2PB'@6!
AB'?!2PB'=6!
;:'B!
>;';N!
&'>×
!
[09%!R!95$9)#.#.7!
?:'A!2P;'&6!
A&'<!2P&'>6!
1.1
>=';N!
29.4×
[09%!R!95$9)#.#.7!
?>'B!2PB'&6!
93.0 2+0.16!
:'@!
@@'>N!
&&'<×
!
!
TABLE 3
Acceleration of ResNet-50 due to zero-quantization
(row-ELT) or pruning (others) of convolution kernels
Dataset
EDSR
!
U>;V!
Factor
!
U==V!
Basis!
U<AV!
Basis!
U<AV!
1-bits
2-09%6!
2-bits
2-09%6!
3-bits
2-09%6
4-bits
2-09%6
!
2×
!
;@'&A!
;?'A=!
;@'BA!
;@'&:!
;>'@&!
;@'&B!
;@':=
!
38.27
Z5$=
!
3×
!
;<'>@!
;<';;!
;<'<?!
;<'==!
;:'&A!
;<';<!
;<'>;
!
34.74
!
4×
!
;:'<@!
;:'B=!
;:':A!
;:';A!
;B';&!
;:':@!
;:'=&
!
32.60
!
2×
!
;;'A=!
;;'=;!
;;'?=!
;;'?:!
;:'?;!
;;'?<!
;;'A;
!
34.00
Z5$&<
!
3×
!
;B'=;
!
;B';&
!
;B'<&
!
;B'<>
!
:A'BA
!
;B';A
!
;B'==
!
30.63
!
4×
!
:@'@&
!
:@'=<
!
:@'>;
!
:@'>A
!
:?'<A
!
:@'?<
!
:@'@?
!
28.93
!
2×
!
;:';=!
;:'&=!
;:':;!
;:':?!
;&'<?!
;:':&!
;:';=
!
32.37
S&BB
!
3×
!
:A':>!
:A'B@!
:A'&=!
:A'&@!
:@'&A!
:A'&;!
:A':>
!
29.32
!
4×
!
:?'?:!
:?'==!
:?'>:!
:?'><!
:>'@=!
:?'>>!
:?'?<
!
27.79
!
2×
!
;:'A?!
;&'AA!
;:';@!
;:'<>!
;B'&:!
;:'=&!
;:'A>
!
33.05
c9").&BB
!
3×
!
:@'@&
!
:@'&B
!
:@';A
!
:@'=&
!
:>'&;
!
:@';<
!
:@'?>
!
28.94
!
4×
!
:>'>=
!
:='A@
!
:>':=
!
:>';>
!
:<'>?
!
:>';@
!
:>'>=
!
26.81
!
2×
!
;<'>B!
;<'>B!
;<'??!
;<'@<!
;;'<;!
;<'?A!
;='B;
!
;5.08!
KX(:]
!
3×
!
;B'A&!
;B'A&!
;&'B>!
;&'&&!
:A':A!
;B'A?!
;&':;
!
31.34
!
4×
!
:@'A:!
:@'A:!
:A'B>!
:A'&;!
:?'>=!
:A'B?!
:A':?
!
29.35
`)9)H5$59%
!
!
&&@BG
!
!
&;>G
!
90k
!
&><G
!
!
&&@BG
!
!
&&@BG
!
!
&&@BG
!
!
&&@BG
!
b-HM95%%#-.
!
P
!
!
&&'=N
!
!
?'>N
!
!
&;'AN
!
3.1%
!
>';N
!
!
A'<N
! !
!
&:'=N
!
!
TABLE 5
PSNR (dB) of 2–4× upsampled images using
transform-quantized EDSR (row-KLT w/o retraining)
Methods
Train
epochs
Comp
ratio
Top-1
Inference rate
Accuracy
L`c!
\859#%%!
a-"#*5E5$J/:!
O0**JM95,#%#-.!
P!
P!
?&'&!2RB'B6!
&=B<!2&'BB6!
!!><!2&'BB6!
HAQ
30
0.6
71.2!(+0.16!
2067!21.376!
124!21.946!
^CW!
;B!
B'<!
>@'A!2P:':6!
:&A?!2&'<>6!
&:@!2:'BB6!
Ours + retrain
30
7.3 "#$%
71.3!(+0.26!
2197!21.466!
127!21.986!
[09%!R!95$9)#.!
;B!
?'B!"#$%!
?&'B!2PB'&6!
::B?!2&'<?6!
&:@!2:'BB6!
[09%!R!95$9)#.!
;B!
>'<!"#$%!
?B'A!2PB':6!
::=>!2&'=B6!
&;B!2:'B;6!
[09%!R!95$9)#.!
;B!
>'&!"#$%!
>?'@!2P;';6!
:;;>!2&'==6!
&;;!2:'B@6!
[09%!R!95$9)#.!
;B
='=!"#$%
><'@!2P>';6
:<B?!2&'>B6
&;>!2:'&;6
!
TABLE 4
Acceleration of transform-quantized (row-KLT)
MobileNet-v2 on Google TPU and MIT Eyeriss
Fig. 10. Average PSNR and SSIM of denoised Set12 images for noise variances 15, 25, 35, 55, 75, and 95, produced by transform-quantized
DRUNet [62] at different bit-rates. Denoising performance of DRUNet starts to drop below 2 bits per weight.
Bit-rate (bits per weight)
Denoised Similarity
Bit-rate (bits per weight)
15
Denoised PSNR (dB)
Bit-rate (bits per weight)
Bit-rate (bits per weight)
25
35
55
75
95
15
25
35
55
75
95
10 IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. 0, NO. 0, MAY 2021
(1#5!)6(1!(1*,#!*7!VakF!`#%7*%5&"4#!*7!(1#! ()*!5#(1*<,!
&%#!4*59&%&3'#!)6(1!(%&",7*%5!K$&"(6Q&(6*"!*3(&6"6"8!B@^R!
&"<!@BR!6"7#%#"4#!%&(#!*"!\`e!&"<!M;#%6,,=!%#,9#4(62#';=!)6(1!
R@FCp!(*9H@!&44$%&4;=!)16'#!7*%!Vak!(1#;!&%#!BTLR!&"<!@BD!
%#,9#4(62#';=!)6(1!R@FBp!(*9H@!&44$%&4;!-(1#,#!&%#!,1*)"!6"!
3*'<0F!k$&"(6Q#<!"#()*%+,!&%#!%#H(%&6"#<!*"!(1#!Z5&8#/#(!
(%&6"6"8!<&(&,#(!7*%!CT!#9*41,F!
Quantizing image denoising network (DRUNet).!]&%6&"(,!
*7!eH/#(!1&2#!%#4#"(';!3##"!&99'6#<!(*!65&8#!<#"*6,6"8F!G#!
&99';!%*)Hih\!K$&"(6Q&(6*"!(*!(1#!,(&(#H*7H(1#H&%(![Pe/#(!
?LBA!(*!,($<;!(1#!6"j$#"4#!*7!K$&"(6Q&(6*"!*"!(1#!<#"*6,6"8!
9#%7*%5&"4#F!G#!4*%%$9(!65&8#,!3;!&<<6(62#!m&$,,6&"!"*6,#!
*7!2&%6&"4#!7%*5!@E!(*!^E=!&"<!<#"*6,#!(1#!4*%%$9(#<!65&8#,!
$,6"8!K$&"(6Q#<![Pe/#(,F!Z"!X68F!@T=!)#!9'*(!(1#!`_/P!&"<!!
_#(@B
H
TR!
!
Ground truth
Noisy (17.24dB / 0.3449)
0.5 bits (27.85dB / 0.8297)
1 bits (28.10dB / 0.8315)
2 bits (28.23dB / 0.8345)
_#(@BHTE!
!
Ground truth
Noisy (17.28 dB / 0.4071)
0.5 bits (28.39dB / 0.8808)
1 bits (28.69dB / 0.8906)
2 bits (28.86dB / 0.8930)
_#(@B
H
@@
!
!
Ground truth
Noisy (17.23 dB / 0.3257)
0.5 bits (28.52 dB / 0.7433)
1 bits (28.71dB / 0.7452)
2 bits (28.79dB / 0.7419)
_#(@B
H
T^
!
!
Ground truth
Noisy (17.23 dB / 0.2489)
0.5 bits (27.48 dB / 0.8076)
1 bits (28.72 dB / 0.8490)
2 bits (29.40 dB / 0.8615)
_#(@B
H
@T
!
Ground truth
Noisy (17.25 dB / 0.3540)
0.5 bits (28.52 dB / 0.7605)
1 bits (28.82 dB / 0.7703)
2 bits (28.96 dB / 0.7791)
Fig. 11. Visual comparison of denoised Set12 images (noise variance is 35). Denoised images are from row-KLT quantized DRUNet [62] at 0.5, 1.0
and 2.0 bits. PSNR and SSIM in brackets are with respect to ground truth. Images denoised at 2 bits are indistinguishable to the full-precision ones
(not shown). Images denoised at 0.5 bits surfer from staircase artifacts. Best viewed online.
YOUNG ET AL.: TRANSFORM QUANTIZATION FOR CNN COMPRESSION 11
__ZN!*7!(1#!<#"*6,#<!_#(@B!65&8#,!&8&6",(!(1#!8%*$"<!(%$(1!
&4%*,,!K$&"(6Q&(6*"!36(H%&(#,!&"<!"*6,#!2&%6&"4#F!G#!,##!(1&(!
`_/P!&"<!__ZN!*7!(1#!<#"*6,#<!65&8#!,(&%(!(*!<#8%&<#!*"4#!
K$&"(6Q&(6*"!%&(#!<%*9,!3#'*)!B!36(,F!X68F!@@!26,$&'6Q#,!,#'#4(!
<#"*6,#<!*$(9$(,!&(!(1#!"#()*%+!K$&"(6Q&(6*"!%&(#,!*7!TFE=!@FT!
&"<!BFT!36(,=!(*8#(1#%!)6(1!(1#!"*6,;!6"9$(!7*%!"*6,#!2&%6&"4#!
CEF!G#!,##!(1&(!ih\HK$&"(6Q#<![Pe/#(!-)6(1*$(!%#(%&6"6"80!
1&,!8**<!<#"*6,6"8!9#%7*%5&"4#!#2#"!&(!&!2#%;!'*)!36(H%&(#!
-TFE!36(,0F!X$''!9%#46,6*"!%#,$'(,!&%#!,656'&%!(*!(1#!B!36(!*"#,F!
Quantizing super-resolution networks.!G#!&99';!%*)Hih\!
K$&"(6Q&(6*"!(*!M[_P!?LCA!(*!<#5*",(%&(#!(1#!&99'64&36'6(;!
*7!*$%!7%&5#)*%+!(*!65&8#!,$9#%H%#,*'$(6*"F!\1#!9%#(%&6"#<!
5*<#'!7%*5!?LCA!6,!K$&"(6Q#<!&"<!(#,(#<!*"![Z]Bi!?UEA=!_#(E!
?ULA=!_#(@D!?URA=!f@TT!?UUA!&"<!e%3&"@TT!?U^A!<&(&,#(,F!G#!'6,(!
(1#!`_/P!*7!(1#!$9,&59'#<!65&8#,!6"!\&3'#!E!(*8#(1#%!)6(1!
(1*,#!7%*5!3&,#'6"#!5#(1*<,=!X&4(*%!?ELA=!&"<!f&,6,!?D^A=!7*%!
4*59&%6,*"F! M2#"!)6(1*$(! %#(%&6"6"8=! (%&",7*%5HK$&"(6Q#<!
M[_P!*$(9#%7*%5,!3&,#'6"#!5#(1*<,!-%#K$6%6"8!(%&6"6"80!6"!!
f&%
3&%&!
4
×
!
Ground truth
Factor [55]: 25.86 dB
Basis-S [49]: 25.94 dB
Basis [49]: 25.96 dB
EDSR [63]: 26.45 dB
!
Nearest: 22.87 dB
Bilinear: 23.26 dB
1-bit (ours): 26.00 dB
2-bits (ours): 26.53 dB
4-bits (ours): 26.53 dB
`#'64&"
!
4
×
!
!
Ground truth
EDSR [63]: 35.27 dB
Basis [49]: 35.10 dB
2-bits (ours): 35.37 dB
4-bits (ours): 35.83 dB
h#"&!
4
×
!
Ground truth
EDSR [63]: 32.69 dB
Basis [49]: 32.68 dB
2-bits (ours): 32.67 dB
4-bits (ours): 32.77 dB
PowerPoint 2002
4
×
Ground truth
EDSR [63]: 27.17 dB
Basis [49]: 27.01 dB
2-bits (ours): 26.38 dB
4-bits (ours): 27.10 dB
Fig. 12. Visual comparison of 4× upsampled images produced by transform-quantized EDSR [63] (with retraining). We additionally show the
upsampled images from Factor [55], Basis [49], and B-spline interpolations for reference. Upsampled images produced by our row-KLT-quantized
EDSR at 2–3 bits are visually identical to those of unquantized EDSR. Results for Factor and Basis are taken from [49]. Best viewed online.
12 IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. 0, NO. 0, MAY 2021
`_/P!&"<!4*59%#,,6*"!%&(6*F!G#!"*(#!(1&(!*$%!$"K$&"(6Q#<!
3&,#'6"#!6,!'6:'#!1681#%!-TF@!<f0!(1&"!(1*,#!*7!?D^=!ELAF!X68F!@B!
9%*26<#,!&!26,$&'!4*59&%6,*"!&5*"8!65&8#,!$9,&59'#<!3;!
K$&"(6Q#<!5*<#',F!Z5&8#,!7%*5!(%&",7*%5HK$&"(6Q#<!M[_P!
&(!'*)!36(H%&(#,!&%#!2#%;!,656'&%!(*!(1*,#!*7!(1#!3&,#'6"#F!\1#!
3*'<#<!"$53#%,!6"<64&(#!(1#!3#,(!%#,$'(!6"!#&41!%*)F!
7 DISCUSSION
\%&",7*%5!K$&"(6Q&(6*"!5&;!3#!#Y(#"<#<!6"!<6%#4(6*",!(1&(!
&%#!"*(!#Y9'*%#<!6"!(16,!)*%+F!G#!"*)!<6,4$,,!#&41!*7!(1#,#!
#Y(#",6*",=!(*8#(1#%!)6(1!9*,,63'#!'656(&(6*",!*7!*$%!)*%+F!
7.1 Extension to 2D Transform
Z"!(16,!)*%+=!)#!<#4*%%#'&(#<!)#681(!5&(%64#,!Θ!*"';!&4%*,,!
(1#6%!%*),F!O"#!5&;!)*"<#%!67!&!B[!(%&",7*%5!T = U
ΘV 6"!
3*(1!%*),!&"<!4*'$5",!*7!Θ 4&"!9%*26<#!3#:#%!4*59%#,,6*"!
&(!'*)#%!36(H%&(#,F!G#!6''$,(%&(#!(16,!6"!X68F!@CF!e"7*%($"&(#';!
7*%!4*59%#,,6*"=!)#!>"<!(1&(!&<<6(6*"&'!,(*%&8#!*2#%1#&<,!
6"4$%%#<!7%*5!(1#!4*'$5"!(%&",7*%5!3&,6,!*$()#681!(1#!36(H!
,&26"8,!*3(&6"#<!7%*5!&!,9&%,#%!T!-(16,!6,!#,9#46&'';!(1#!4&,#!
,6"4#!4*'$5"!(%&",7*%5,!<*!"*(!)*%+!&,!)#''!&,!%*)!*"#,b
%#7#%!(*!X68F!EF!Z7!(1#!K$&"(6Q#<!&"<!,(*%#<!,6Q#!*7!(1#!.//,!
6,!"*(!&"!6,,$#=!B[!(%&",7*%5,!4&"!,(6''!3#!$,#7$'!7*%!9%$"6"8!
(*!%#<$4#!(1#!"$53#%!*7!4*"2*'$(6*",!9#%7*%5#<F!Z"!7&4(=!)#!
4&"!6"(#%9%#(!(1#!N*36'#/#(H2B!3'*4+,!&,!1&26"8!3##"!(%&6"#<!!
6"!(1#!B[!(%&",7*%5!<*5&6"!)6(1!<6&8*"&'!T!659*,#<F!
7.2 Extension to Intra-kernel Transform
.*%%#'&(6*"!5&;!&',*!3#!*3,#%2#<!3#()##"!)#681(,!6",6<#!&!
+#%"#'=!&"<!*"#!4&"!&'(#%"&(62#';!(%&",7*%5!)#681(!5&(%64#,!
3;! <#4*%%#'&(6"8! )#681(,! )6(16"! #&41! +#%"#'! -6"(%&H+#%"#'0!
%&(1#%!(1&"!&4%*,,!+#%"#',!-6"(#%H+#%"#'0F!O7(#"=!6"(%&H+#%"#'!
(%&",7*%5,!4&"!9%*<$4#!'&%8#%!4*<6"8!8&6",!(1&"!(1#6%!6"(#%H
+#%"#'!4*$"(#%9&%(,F!V*)#2#%=!4#%(&6"!.//!5*<#',!,$41!&,!
a'#Y/#(!&"<!]mm!4&"!4*"(&6"!,68"6>4&"(!"$53#%,!*7!1 ×1!
4*"2*'$(6*"&'!*%!7$'';!4*""#4(#<!'&;#%,=!,*!(1#!*2#%&''!$(6'6(;!
*7!6"(%&H+#%"#'!(%&",7*%5,!6,!"#8'6863'#!7*%!,$41!.//,F!
! Z"! \&3'#! L=! )#! 9%*26<#! (1#! 6"(%&H+#%"#'! 4*<6"8! 8&6",! *7!
<6J#%#"(!(%&",7*%5,!*"!(1#!4*"2*'$(6*"&'!'&;#%,!*7!a'#Y/#(!
-'&;#%,!@SE0F!a''!4*"2*'$(6*"&'!'&;#%,!4*"(&6"!+#%"#',!3 × 3!6"!
<65#",6*",=!)6(1!(1#!#Y4#9(6*"!*7!(1#!>%,(!-11 × 110!&"<!(1#!
,#4*"<!-5 ×50!'&;#%,F!X*%!6"(%&HMh\!&"<!6"(%&Hih\=!)#!$,#!&!
,#9&%&(#!(%&",7*%5!7*%!#&41!%*)!*7!(1#!)#681(!5&(%64#,!7*%!
659%*2#<!<#4*%%#'&(6*"F!\1#!4*<6"8!8&6",!*7!(1#!6"(%&HMh\!
&%#!$9!(*!E!<f!1681#%!(1&"!(1*,#!*7![.\HZZ!$,#<!3;!(1#!.//!
K$&"(6Q&(6*"!5#(1*<,!6"!?BD=!BLAF!V*)#2#%=!(1#!9#%7*%5&"4#!
*7![.\HZ!6,!&'%#&<;!2#%;!,656'&%!(*!(1&(!*7!)#681(H<#9#"<#"(!
ih\!&"<!Mh\=!)1641=!$"'6+#!(1#![.\HZ=!<*!"*(!1&2#!#I46#"(!
3$:#%j;!&'8*%6(15,!&2&6'&3'#F!!
7.3 Limitations and Future Work
_(*%&8#!*2#%1#&<,!6"4$%%#<!3;!(%&",7*%5!3&,6,!5&(%64#,!4&"!
3#!%#<$4#<!67!(1#!(%&",7*%5,!4*$'<!3#!,1&%#<!&4%*,,!5$'(69'#!
'&;#%,=!,656'&%';!(*!(1#!,1&%#<!>'(#%!3&,6,!6<#&!?DE=!D^AF!Z"!*$%!
9*,(H(%&6"6"8!K$&"(6Q&(6*"!4&,#=!1*)#2#%=!,1&%#<!(%&",7*%5,!
&%#!"*(!$,#7$'!,6"4#!(1#!%*),!*%!4*'$5",!*7!(1#!'#&%"(!)#681(!
5&(%64#,!&%#!*%<#%#<!&%36(%&%6';b(%&",7*%5,!(1&(!)*%+!)#''!
*"!*"#!)#681(!5&(%6Y!5&;!#2#"!;6#'<!"#8&(62#!4*<6"8!8&6",!
67!&99'6#<!*"!<6J#%#"(!)#681(!5&(%64#,F!/#2#%(1#'#,,=!,1&%#<!
(%&",7*%5,!5&;!9%*26<#!7$%(1#%!4*<6"8!8&6",!67!)#!4&"!(%&6"!
5*<#',!7%*5!,4%&(41!(*!#"7*%4#!(1#!,&5#!*%<#%!*7!%*),!&"<!
4*'$5",!&4%*,,!5$'(69'#!)#681(!5&(%64#,F!
! O"#!9*(#"(6&'!'656(&(6*"!*7!(16,!)*%+!6,!(1&(!K$&"(6Q&(6*"!
6"(%*<$4#,!&!,68"6>4&"(!'*,,!6"!4'&,,6>4&(6*"!&44$%&46#,!67!)#!
<*!"*(!,$3,#K$#"(';!>"#H($"#!(1#!.//F!V*)#2#%=!'*,,!<$#!
(*!K$&"(6Q&(6*"!6,!7&%!7%*5!$"6K$#!(*!(16,!)*%+b(*!(1#!3#,(!
*7!*$%!+"*)'#<8#=!.//!K$&"(6Q&(6*"!5#(1*<,!)6(1!,5&''#%!
&44$%&4;!'*,,#,!6"2*'2#!,*5#!7*%5!*7!(%&6"6"8=!)1#(1#%!6(!3#!
7%*5!,4%&(41!*%!9*,(HK$&"(6Q&(6*"F!a"!659*%(&"(!(1#*%#(64&'!
7$($%#!)*%+!5&;!3#!(*!6"2#,(68&(#!(1#!%#&,*",!7*%!,$41!&!'*,,!
6"!9#%7*%5&"4#!)1#"!)#681(,!&%#!"*(!%#(%&6"#<F!.*536"6"8!
K$&"(6Q&(6*"!)6(1!"#()*%+!&%416(#4($%#!,#&%41!,656'&%!(*!?^TA!
5&;!3#!&"*(1#%!7%$6(7$'!%#,#&%41!<6%#4(6*"F!
8 CONCLUSION
\16,!)*%+!9%*9*,#,!&!(%&",7*%5HK$&"(6Q&(6*"!7%&5#)*%+!(*!
4*59%#,,!.//!)#681(,=!9*,(H(%&6"6"8F!a,!,$88#,(#<!3;!(1#!
"&5#=!*$%!7%&5#)*%+!(%&",7*%5,!)#681(,!3#7*%#!K$&"(6Q6"8!
(1#5!(*!,68"6>4&"(';!659%*2#!4*59%#,,6*"!6"!3*(1!%#(%&6"#<!
&"<!"*"H%#(%&6"#<!K$&"(6Q&(6*"!,4#"&%6*,F!G#!*9(656Q#!3*(1!
4*59*"#"(,!*7!*$%!7%&5#)*%+!-(%&",7*%5!&"<!K$&"(6Q&(6*"0!
3;!7*%56"8!.//!4*59%#,,6*"!&,!&!%&(#S<6,(*%(6*"!9%*3'#5!
&"<!*9(656Q6"8!6(,!*3W#4(!)6(1!%#,9#4(!(*!(1#!(%&",7*%5=!&"<!
36(H<#9(1,!&,,68"#<!(*!(%&",7*%5!)#681(,!&"<!3&,#,F!_*'26"8!
(16,!*9(656Q&(6*"!9%*3'#5!&',*!%#2#&',!(1&(!(1#!4'&,,64&'!ih\!
1&,!&!,656'&%!-3$(!"*(!6<#"(64&'0!K$&"(6Q&(6*"!9#%7*%5&"4#!&,!
(1#!*9(65$5!#"<H(*H#"<!'#&%"#<!(%&",7*%5!-Mh\0!)#!<#%62#!
6"!(16,!9&9#%F!X*%!#6(1#%!(%&",7*%5=!*9(65$5!36(H<#9(1,!&%#!
*3(&6"#<!3;!56"656Q6"8!(1#!<6,(*%(6*"!6"!(1#!.//!*$(9$(!&,!
*99*,#<!(*!(1&(!*7!(1#!)#681(,!(1#5,#'2#,F!\%&",7*%5!3&,6,!
'&;#%,!&%#!K$&"(6Q#<!,656'&%';!(*!(%&",7*%5!<*5&6"!'&;#%,F!
! \1#!9*,(H(%&6"6"8!K$&"(6Q&(6*"!9&%&<685!&<*9(#<!3;!*$%!
7%&5#)*%+!&',*!9%*26<#,!(1#!j#Y636'6(;!(*!4*59%#,,!5*<#',!
&(!&%36(%&%;!36(H%&(#,=!*326&(#,!(1#!"##<!(*!(%&6"!5*<#',!&8&6"!
7%*5!,4%&(41=!&"<!6,!,(%&681(H7*%)&%<!(*!659'#5#"(F![#,96(#!
(1#!,659'646(;!*7!*$%!7%&5#)*%+=!6(!&<2&"4#,!(1#!,(&(#!*7!(1#!
&%(!6"!.//!4*59%#,,6*"!7*%!&!"$53#%!*7!.//!5*<#',F!G#!
3#'6#2#!(1&(!*$%!(%&",7*%5HK$&"(6Q&(6*"!7%&5#)*%+!)6''!&',*!
3#!$,#7$'!7*%!*(1#%!.//,!"*(!,9#46>4&'';!5#"(6*"#<!1#%#F
Fig. 13. 2D Transform of Θ. The kernels Θ are now decorrelated in both
the row and the column spaces to produce a sparser matrix
T
at lower
quantization rates. White blocks indicate zero-quantized regions.
Θ ∈ ℝ
×
×
u
1
u
2
. . .
u
θ
. . .
→
v
v
. . .
v
U ∈ ℝ
×
T ∈ ℝ
×
×
V
∈ ℝ
×
θ
θ
θ
. . .
.
.
.
t
. . .
t
t
t
. . .
.
.
.
𝑙
!
Intra-kernel transform quantization
Inter-
KbLJX
!
KbLJXX
!
]YL
!
\YL
!
]YL
!
\YL
!
C*5DE5$!2b)d5E5$6!
&!
&<'&!
='>!
19.6
&<'B!
='B!
19.0
&&'B!
='A!
16.9!
&<'B
!
='@!
19.8
4.7
6.0
:!
<';!
:'>!
6.9
<'=!
&'=!
5.9
;'&!
;'&!
6.2!
<'?
!
:'A!
7.6
6.3
7.5
;!
:'A!
:'&!
5.1
;'B!
&'>!
4.6
:'=!
:'&!
4.6!
;'&
!
:'&!
5.2
3.7
5.1
<!
:';!
&'<!
3.7
:';!
&':!
3.5
:'B!
&'=!
3.5!
:'<
!
&'=!
3.9
1.6
3.0
=!
:'B!
:'&!
4.1
:'B!
&'A!
3.9
&'A!
:':!
4.1!
:'&
!
:'&!
4.2
1.7
3.3
!
TABLE 6
Intra-kernel transform coding gains (dB) for AlexNet.
Gradient, weight and total (in bold) gains shown.
YOUNG ET AL.: TRANSFORM QUANTIZATION FOR CNN COMPRESSION 13
APPENDIX A PROOFS OF THEOREMS
A.1 Proof of Theorem 1
ProofF!Z7!)#!#Y9%#,,!Θ
= Θ + dΘ!7*%!(1#!K$&"(6Q#<!2#%,6*"!
*7!&!)#681(!5&(%6Y!Θ ∈ ℝ
×
×
=!)#!1&2#!7*%!,5&''!dΘ
!
𝐷 =
𝔼‖𝑓(x| . . . , Θ
−
, Θ
, Θ
+
, . . . ) −y‖
!!
-@E
0
!
!
≈
(a)
𝔼‖𝑓(x|Θ) + ∑ (𝜕𝑓(x|Θ) 𝜕θ
⁄ )
dθ
=
−y‖
!!
!
=
(b)
∑ 𝔼‖(𝜕y 𝜕θ
⁄ )
dθ
‖
=
!!
!
=
(c)
∑ (C
)
(C
)
𝜖
2
−
𝑘
=
!!
!
≥
(d)
(∏ (C
)
(C
)
=
)
⁄
𝜖
2
−
!!!
6"!)1641!-&0!7*''*),!7%*5!>%,(H*%<#%!\&;'*%l,!&99%*Y65&(6*"!
*7!𝑓(x| . . . ,Θ + dΘ, . . . )!&(!Θ=!-30!7*''*),!7%*5!(1#!<#>"6(6*"!
y = 𝑓(x| . . . ,Θ, . . . )!&"<!(1#!7&4(!(1&(!8%&<6#"(,!𝜕𝑓(x|Θ) 𝜕θ
⁄ !
&%#!*%(1*8*"&'F!MK$&'6(;!-40!7*''*),!7%*5!(1#!<#>"6(6*",!*7!
(1#!4*2&%6&"4#!5&(%64#,!C
= ∑ (𝜕y 𝜕θ
⁄ )(𝜕y 𝜕θ
⁄ )
=
=!&"<!
C
=
∑ θ
θ
=
=!&"<!(1#!7&4(!(1&(!𝔼(d𝜃
)
= 𝔼𝜃
2
−
𝑘
!&"<!
𝔼d𝜃
′ d𝜃
= 0, 𝑘
≠ 𝑘F!X6"&'';=!6"#K$&'6(;!-<0!6,!*3(&6"#<!)6(1!
#K$&'6(;!7*%!𝑅 =
∑ 𝑅
=
!3;!41**,6"8!𝑅
!,$41!(1&(!&''!(1#!
6"<626<$&'!<6,(*%(6*",!(C
)
(C
)
𝜖
2
−
𝑘
!&%#!#K$&'F! ∎!
A.2 Proof of Theorem 2
Proof. G%6(6"8!ST
= S(T + dT)!7*%!(1#!K$&"(6Q#<!2#%,6*"!*7!!
&!)#681(!5&(%6Y!ST ∈ ℝ
×
×
=!)#!1&2#!7*%!,5&''!dT!!
!
𝐷 =
𝔼‖𝑓(x| . . . , Θ
−
,ST
, Θ
+
, . . . ) −y‖
!!
-@L0!
!
≈
(a)
𝔼
‖
𝑓
(x
|
Θ)
+
∑ (
𝜕𝑓
(x
|
Θ)
𝜕
θ
⁄ )
U
−
dt
=
−
y‖
!
!
!
=
(b)
∑ 𝔼‖(𝜕y 𝜕θ
⁄ )
U
−
dt
‖
=
!!
!
=
∑ 𝔼‖(𝜕y 𝜕θ
⁄ )
U
−
(U
dθ
)‖
=
!!
!
=
(c)
∑ (U
−
C
U
−
)
(U
C
U)
𝜖
2
−
𝑘
=
!!
!
≥
(d)
(∏ (U
−
C
U
−
)
(U
C
U)
=
)
⁄
𝜖
2
−
!!
6"!)1641!-&0!7*''*),!7%*5!>%,(H*%<#%!\&;'*%l,!&99%*Y65&(6*"!
*7!𝑓(x| . . . , S(T + dT), . . . )!&3*$(!Θ = ST=!&"<!(1#!41&6"!%$'#!
*7!<6J#%#"(6&(6*"!
(𝜕y 𝜕t
⁄ )
= (𝜕𝑓(x|ST) 𝜕t
⁄ )
= (𝜕𝑓(Θ) 𝜕θ
⁄ )
S,!!
-@R
0
!
-30!7*''*),!7%*5!𝑓(x|Θ) = y=!&"<!-40=!7%*5!(1#!<#>"6(6*",!*7!
4*2&%6&"4#!5&(%64#,!C
=!C
=!&"<!(1&(!𝔼(d𝜃
)
= 𝔼(𝜃
)2
−
𝑘
!
&"<!𝔼d𝜃
′ d𝜃
= 0!7*%!𝑘
≠ 𝑘F!Z"#K$&'6(;!-<0!6,!*3(&6"#<!)6(1!!
#K$&'6(;!7*%!𝑅 =
∑ 𝑅
=
!3;!41**,6"8!𝑅
!,$41!(1&(!&''!(1#!
<6,(*%(6*",!(U
−
C
U
−
)
(U
C
U)
𝜖
2
−
𝑘
!&%#!#K$&'F! ∎!
A.3 Proof of Theorem 3
Proof. \1#!<#"*56"&(*%!*7!-@@0!7$%(1#%!862#,!$,!!
!
(𝜎
tc
)
= ∏ (U
−
C
U
−
)
(U
C
U)
=
!!
-@U0!
!
≥
(a)
<#((U
−
C
U
−
)<#((U
C
U)!!
!
= <#((U
−
)<#((C
)<#((C
) <#((U),!!
!
= <#((C
C
) = ∏ 𝜆
(C
C
)
=
,!!
6"!)1641!𝜆
(𝐀)!<#"*(#,!(1#!𝑘(1!#68#"2&'$#!*7!𝐀F!Z"#K$&'6(;!
-&0!7*''*),!7%*5!V&<&5&%<l,!6"#K$&'6(;=!&"<!&''!,$3,#K$#"(!
#K$&'6(6#,!7%*5!(1#!9%*9#%(6#,!*7!(1#!<#(#%56"&"(F!
! Z"#K$&'6(;!-&0!6,!*3(&6"#<!)6(1!#K$&'6(;!67!)#!,#(!U!(*!(1#!
5&(%6Y!*7!#68#"2#4(*%,!*7!C
C
F!G%6(6"8!Λ!7*%!(1#!<6&8*"&'!
5&(%6Y!*7!#68#"2&'$#,!𝜆
(C
C
)=!)#!*3(&6"!-@B0!3;!)%6(6"8!
(1#!#68#"H<#4*59*,6(6*"!U
−
(C
C
)U = Λ!&,!8#"#%&'6Q#<!
#68#"2&'$#!<#4*59*,6(6*"!*7!5&(%6Y!9&6%!(C
, C
−
)F! ∎!
APPENDIX B KLT, ELT, SVD, GSVD, HOSVD
\*!,##!1*)!(1#!ih\!U
!*7!θ ∈ ℝ
×
!%#'&(#,!(*!(1#!_][!*7!(1#!
4*%%#,9*"<6"8!7$'';H4*""#4(#<!)#681(!5&(%6Y!Θ ∈ ℝ
×
×
=!)#!
"*(64#! (1&(! 4%*,,H41&""#'! 4*2&%6&"4#,!C
= (1 𝑚⁄ )ΘΘ
=! ,*!
(1#!<6&8*"&'6Q&(6*"!U
C
U = Λ!6,!#K$62&'#"(!(*!(1#!_][!
!
Σ = U
ΘV, U
U = I, V
V = I,!
-@^0!
6"!)1641!Σ =
√
𝑛Λ=!&"<!V
= Σ
−
U
ΘF!\1#%#7*%#=!(1#!%681(!
,6"8$'&%!2#4(*%,!V!*7!Θ!&%#!(%&",7*%5!4*#I46#"(,!*7!)#681(,!
-$9!(*!,4&'6"8!3;!Σ
−
0!)1#%#&,!(1#!'#7(!*"#,!U=!(1#!(%&",7*%5!
3&,6,F!_656'&%';=!(1#!Mh\!U
!*7!θ ∈ ℝ
×
!%#'&(#,!(*!(1#!m_][!
-8#"#%&'6Q#<!_][0!%#'&(6*"!
!
Σ = U
−
ΘV
−
, U
C
−
U = I, V
V = I,!
-BT0!
"*(6"8!(1&(!U
−
≠ U
= &"<!V
−
≠ V!7*%!(1#!m_][F!G16'#!(1#!!
4*""#4(6*"!3#()##"!(1#!ih\!&"<!(1#!_][!6,!,659'#=!*9(65&'!
K$&"(6Q&(6*"!*7!Θ!6,!"*(!655#<6&(#';!*326*$,!6"!(1#!_][!*%!
m_][!<*5&6"=!&"<!3#,(!$"<#%,(**<!26&!(1#!%&(#S<6,(*%(6*"H
(1#*%#(64!7%&5#)*%+!)#!1&2#!<#2#'*9#<!6"!(16,!)*%+F!
! .//!5*<#'!4*59%#,,6*"!5#(1*<,!3&,#<!*"!1681H*%<#%H!
_][!-VO_][0!(%#&(!4*"2*'$(6*"!)#681(!5&(%64#,!&,!7*$%(1H
*%<#%!(#",*%,=!&"<!<#4*59*,#!(1#5!&4%*,,!5$'(69'#!&Y#,!-*%!
<65#",6*",0F!m62#"!(1#6%!%#'&(62#';!,5&''!,9&(6&'!<65#",6*",!
-$,$&'';!"*!'&%8#%!(1&"!3 × 30=!(1#,#!(#",*%,!&%#!<#4*59*,#<!
(;964&'';!*"';!6"!(1#6%!6"9$(!&"<!*$(9$(H41&""#'!<65#",6*",!
?EBAF!Z"!(16,!4&,#=!*"#!4&"!#Y9%#,,!(1#!%#,$'(6"8!1681#%H*%<#%!
_][!*7!(1#!)#681(!(#",*%,!#K$62&'#"(';!&,!(1#!B[!(%&",7*%5!!
!
T = U
ΘV, U
U = I, V
V = I,!
-B@0!
6"!)1641!(%&",7*%5,!U!&"<!V!&%#!5&(%64#,!*7!#68#"2#4(*%,!*7!!
C
= (1 𝑚⁄ )ΘΘ
=!&"<!C
= (1 𝑛⁄ )Θ
Θ=!%#,9#4(62#';=!&"<!T!
6,!&!"*"H<6&8*"&'!5&(%6Y!*7!(%&",7*%5#<!+#%"#',F!
REFERENCES
!!U&V! C'! ]9#e45/%G8F! X'! Z0$%G5/59F! ).+! _'! \'! ^#.$-.F! fXH)75E5$!
,*)%%#g,)$#-.!3#$4!+55M!,-./-*0$#-.)*!.509)*!.5$3-9G%Fh!#.!NIPSF!
:B&:F!MM'!&BA?P&&B='!
! U:V! ]'!^5F!_'!_G#-D)9#F!`'!K-**)9F!).+!I'!_#9%4#,GF!fa)%G!IJbEEFh!#.!
CVPRF!:B&?F!MM'!:A>&P:A>A'!
! U;V! Y'!C'!_)$8%F!C'!Z'!\,G59F!).+!a'!S5$475F!fXH)75!%$8*5!$9).%159!0%#.7!
,-./-*0$#-.)*!.509)*!.5$3-9G%Fh!#.!CVPRF!:B&>F!MM'!:<&<P:<:;'!
! U<V! i'!i-4.%-.F!C'!C*)4#F!).+!O'JO'!Y#F!f`59,5M$0)*!*-%%5%!1-9!95)*J$#H5!
%$8*5!$9).%159!).+!%0M59J95%-*0$#-.Fh!#.!ECCVF!:B&>F!MM'!>A<P?&&'!
! U=V! L'!])99)%F!L'!C#*)F!Z'!Y)#.5F!).+!i'!Y54$#.5.F!f`9-795%%#/5!79-3#.7!
-1!_CE%!1-9!#HM9-/5+!j0)*#$8F!%$)"#*#$8F!).+!/)9#)$#-.Fh!#.!ICLRF!
:B&@'!
! U>V! ('!Ze5F!k'J^'!b45.F!L'Ji'!k).7F!).+!i'!Z'!\H59F!f\l,#5.$!M9-,5%%#.7!
-1!+55M! .509)*!.5$3-9G%m!)!$0$-9#)*!).+! %09/58Fh!Proc. IEEEF!/-*'!
&B=F!.-'!&:F!MM'!::A=P:;:AF!K5,'!:B&?'!
! U?V! ^'!Y#F!C'!])+)/F!X'!K09+).-/#,F!^'!Z)H5$F!).+!^'!`'!_9)1F!f`90.#.7!
g*$59%!1-9!5l,#5.$!,-./.5$%Fh!#.!ICLRF!:B&>'!
! U@V! k'!k).7F!k'!^5F!_'!]).7F!n'!K-.7F!).+!k'!O0F!fZ-1$!g*$59!M90.#.7!1-9!
),,5*59)$#.7!+55M!,-./-*0$#-.)*!.509)*!.5$3-9G%Fh!#.!IJCAIF!:B&@F!
MM'!::;<P::<B'!
! UAV! k'! ^5F! `'! Y#0F! o'! T).7F! o'! ^0F! ).+! k'! k).7F! fO#*$59! M90.#.7! /#)!
75-H5$9#,! H5+#).! 1-9! +55M! ,-./-*0$#-.)*! .509)*! .5$3-9G%!
),,5*59)$#-.Fh!#.!CVPRF!:B&AF!MM'!<;<BP<;<A'!
14 IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. 0, NO. 0, MAY 2021
!U&BV! i'J^'!Y0-F!i'!T0F!).+!T'!Y#.F!fL4#E5$m!)!g*$59!*5/5*!M90.#.7!H5$4-+!
1-9!+55M!.509)*!.5$3-9G! ,-HM95%%#-.Fh!#.!ICCVF! :B&?F!MM'!=B=@P
=B>>'!
!U&&V! k'! o4-0F! k'! o4).7F! k'! T).7F! ).+! W'! L#).F! fC,,5*59)$5! bEE! /#)!
95,09%#/5!S)85%#).!M90.#.7Fh!#.!ICCVF!:B&AF!MM'!;;B>P;;&='!
!U&:V! o'!k-0F!]'!k).F!i'!k5F!a'!a)F!).+!`'!T).7F!f_)$5!K5,-9)$-9m!7*-")*!
g*$59!M90.#.7!H5$4-+!1-9!),,5*59)$#.7!+55M!,-./-*0$#-.)*!.509)*!
.5$3-9G%Fh!#.!NeurIPSF!:B&AF!MM'!:&;;P:&<<'!
!U&;V! O'!^0).7F!Z'!Y#.F!I'!i#F!k'!T0F!k'!Y#F!).+!S'!o4).7F!fC,,5*59)$#.7!
,-./-*0$#-.)*!.5$3-9G%! /#)! 7*-")*!p!+8.)H#,! g*$59! M90.#.7Fh!#.!
IJCAIF!:B&@F!MM'!:<:=P:<;:'!
!U&<V! o'!o40).7!et al.F!fK#%,9#H#.)$#-.J)3)95!,4)..5*!M90.#.7!1-9!+55M!
.509)*!.5$3-9G%Fh!#.!NIPSF!:B&@F!MM'!@?=P@@>'!
!U&=V! o'!Y#0F!i'!Y#F!o'!Z45.F!_'!^0).7F!Z'!k).F!).+!b'!o4).7F!fY5)9.#.7!
5l,#5.$! ,-./-*0$#-.)*! .5$3-9G%! $49-074! .5$3-9G! %*#HH#.7Fh!#.!
ICCVF!:B&?F!MM'!:?;>P:?<<'!
!U&>V! k'!^5F!n'!o4).7F!).+!i'!Z0.F!fb4)..5*!M90.#.7!1-9!),,5*59)$#.7!/598!
+55M!.509)*!.5$3-9G%Fh!#.!ICCVF!:B&?F!MM'!&;@AP&;A?'!
!U&?V! ^'! `5.7F! i'! T0F! Z'! b45.F! ).+! i'! ^0).7F! fb-**)"-9)$#/5! ,4)..5*!
M90.#.7!1-9!+55M!.5$3-9G%Fh!#.!ICMLF!:B&AF!MM'!=&&;P=&::'!
!U&@V! L'! o4).7! et al.F! fC! %8%$5H)$#,! KEE! 35#74$! M90.#.7! 19)H53-9G!
0%#.7!)*$59.)$#.7!+#95,$#-.!H5$4-+!-1!H0*$#M*#59%Fh!#.!ECCVF!:B&@F!
MM'!&@<P&AA'!
!U&AV! I'! k0! et al.F! fEXZ`m! M90.#.7! .5$3-9G%! 0%#.7! .509-.! #HM-9$).,5!
%,-95!M9-M)7)$#-.Fh!#.!CVPRF!:B&@F!MM'!A&A<PA:B;'!
!U:BV! Z'!^).F!^'!a)-F!).+!T'!i'!K)**8F!fK55M!,-HM95%%#-.m!,-HM95%%#.7!
+55M! .509)*! .5$3-9G%! 3#$4! M90.#.7F! $9)#.5+! j0).$#e)$#-.! ).+!
^0dH).!,-+#.7Fh!#.!ICLRF!:B&>'!
!U:&V! O'!L0.7!).+!_'!a-9#F!fbYX`JWm!+55M!.5$3-9G!,-HM95%%#-.!*5)9.#.7!
"8! #.JM)9)**5*! M90.#.7Jj0).$#e)$#-.Fh! #.! CVPRF! :B&@F! MM'! ?@?;P
?@@:'!
!U::V! k'! _-.7F! Y'! Y#0F! a'! k).7F! ).+! Y'! S-09+5/F! fb-HM95%%#.7! +55M!
,-./-*0$#-.)*!.5$3-9G%!0%#.7!/5,$-9!j0).$#e)$#-.Fh!#.!ICLRF!:B&='!
!U:;V! `'!Z$-,GF!C'!i-0*#.F!I'!_9#"-./)*F!S'!_9)4)HF!).+!^'!iq7-0F!fC.+!
$45!"#$!7-5%!+-3.m!95/#%#$#.7!$45!j0).$#e)$#-.!-1!.509)*!.5$3-9G%Fh!
#.!ICLRF!:B:B'!
!U:<V! k'! T).7F! b'! n0F! Z'! k-0F! K'! L)-F! ).+! b'! n0F! fbEEM),Gm! M),G#.7!
,-./-*0$#-.)*!.509)*!.5$3-9G%!#.!$45!195j05.,8!+-H)#.Fh!#.!NIPSF!
:B&>F!MM'!:=;P:>&'!
!U:=V! T'! b45.F! i'! L'! T#*%-.F! Z'! L8955F! ]'! W'! T5#."59759F! ).+! k'! b45.F!
fb-HM95%%#.7!.509)*!.5$3-9G%!3#$4!$45!4)%4#.7!$9#,GFh!#.!ICMLF!
:B&=F!MM'!::@=P::A<'!
!U:>V! T'! b45.F! i'! T#*%-.F! Z'! L8955F! ]'! W'! T5#."59759F! ).+! k'! b45.F!
fb-HM95%%#.7! ,-./-*0$#-.)*! .509)*! .5$3-9G%! #.! $45! 195j05.,8!
+-H)#.Fh!#.!SIGKDDF!:B&>F!MM'!&<?=P&<@<'!
!U:?V! i'! T0F! b'! Y5.7F! k'! T).7F! W'! ^0F! ).+! i'! b45.7F! fW0).$#e5+!
,-./-*0$#-.)*!.509)*!.5$3-9G%!1-9!H-"#*5!+5/#,5%Fh!#.!CVPRF!:B&>F!
MM'!<@:BP<@:@'!
!U:@V! a'!I)%$57)9#F!('![9+-.5eF!i'!I5+H-.F!).+!C'!O)94)+#F!fnE[IJE5$m!
XH)75E5$! ,*)%%#g,)$#-.! 0%#.7! "#.)98! ,-./-*0$#-.)*! .509)*!
.5$3-9G%Fh!#.!ECCVF!:B&>F!MM'!=:=P=<:'!
!U:AV! O'!Y#F!S'!o4).7F!).+!S'!Y#0F!fL59.)98!T5#74$!E5$3-9G%Fh!#.!NIPS
Workshop on Efficient Methods for Deep Neural NetworksF!:B&>'!
!U;BV! b'! o40F! Z'! ^).F! ^'! a)-F! ).+! T'! i'! K)**8F! fL9)#.5+! $59.)98!
j0).$#e)$#-.Fh!#.!ICLRF!:B&?'!
!U;&V! X'!^0")9)F!a'!b-09")9#)0DF!K'!Z-0+98F!I'!\*Jk).#/F! ).+! k'!S5.7#-F!
fW0).$#e5+!.509)*! .5$3-9G%m! $9)#.#.7!.509)*!.5$3-9G%! 3#$4!*-3!
M95,#%#-.!35#74$%!).+!),$#/)$#-.%Fh!J. Mach. Learn. Res.F!/-*'!&@F!.-'!
&@?F!MM'!&P;BF!:B&@'!
!U;:V! i'!O)9)-.5F!E'!O9)%59F!a'!S*-rF!).+!`'!^'!T'!Y5-.7F!fZkWm!*5)9.#.7!
%8HH5$9#,! j0).$#e)$#-.! 1-9! 5l,#5.$! +55M! .509)*! .5$3-9G%Fh! #.!
CVPRF!:B&@F!MM'!<;BBP<;BA'!
!U;;V! S'!i),-"!et al.F!fW0).$#e)$#-.!).+!$9)#.#.7!-1!.509)*!.5$3-9G%!1-9!
5l,#5.$! #.$5759J)9#$4H5$#,J-.*8! #.1595.,5Fh! #.! CVPRF! :B&@F! MM'!
:?B<P:?&;'!
!U;<V! Z'!Z-.F!Z'!E)4F!).+!]'!a0!Y55F!fb*0%$59#.7!,-./-*0$#-.)*!G59.5*%!
$-!,-HM95%%!+55M!.509)*!.5$3-9G%Fh!#.!ECCVF!:B&@F!MM'!:&>P:;:'!
!U;=V! o'! K-.7F! o'! k)-F!C'! _4-*)H#F! a'! T'! a)4-.58F! ).+! ]'! ]50s59F!
f^CTWm! ^5%%#).!CT)95! W0).$#e)$#-.! -1! .509)*! .5$3-9G%! 3#$4!
H#D5+JM95,#%#-.Fh!#.!ICCVF!:B&AF!MM'!:A;P;B:'!
!U;>V! C'!o4-0F!C'!k)-F!k'!_0-F!Y'!n0F!).+!k'!b45.F!fX.,95H5.$)*!E5$3-9G!
W0).$#e)$#-.m!$-3)9+%!*-%%*5%%!bEE%!3#$4!*-3JM95,#%#-.!35#74$%Fh!
#.!ICLRF!:B&?'!
!U;?V! K'! o4).7F! i'! k).7F! K'! k5F! ).+! _'! ^0)F! fYWJE5$%m! *5)9.5+!
j0).$#e)$#-.! 1-9! 4#74*8! ),,09)$5! ).+! ,-HM),$! +55M! .509)*!
.5$3-9G%Fh!#.!ECCVF!:B&@F!MM'!;?;P;AB'!
!U;@V! I'!S)..59F!k'!E)4%4).F!\'!^-d59F!).+!K'!Z-0+98F!f`-%$J$9)#.#.7!<J
"#$!j0).$#e)$#-.!-1!,-./-*0$#-.!.5$3-9G%!1-9!9)M#+J+5M*-8H5.$Fh!
#.!NIPSF!:B&A'!
!U;AV! Z'! b45.F! T'! T).7F! ).+! Z'! i'! `).F! fK55M! .509)*! .5$3-9G!
j0).$#e)$#-.! /#)! *)859J3#%5! -M$#H#e)$#-.! 0%#.7! *#H#$5+! $9)#.#.7!
+)$)Fh!#.!AAAIF!:B&AF!/-*'!;;F!MM'!;;:AP;;;>'!
!U<BV! I'! _-.7! et al.F! fK#d595.$#)"*5! %-1$! j0).$#e)$#-.m! "9#+7#.7! 10**J
M95,#%#-.!).+!*-3J"#$!.509)*!.5$3-9G%Fh!#.!CVPRF!:B&AF!MM'!<@=:P
<@>&'!
!U<&V! X'!^0")9)F!a'!b-09")9#)0DF!K'!Z-0+98F!I'!\*Jk).#/F! ).+! k'!S5.7#-F!
fS#.)9#e5+!.509)*!.5$3-9G%Fh!#.!NIPSF!:B&>F!MM'!<&B?P<&&='!
!U<:V! Z'!o4-0F!k'!T0F!o'!E#F!n'!o4-0F!^'!T5.F!).+!k'!o-0F!fK-I5O)JE5$m!
$9)#.#.7! *-3! "#$3#+$4! ,-./-*0$#-.)*! .509)*! .5$3-9G%! 3#$4! *-3!
"#$3#+$4!79)+#5.$%Fh!#.!arXiv:1606.06160 [cs]F!:B&@'!
!U<;V! a'!E)75*F!a'!/).!S))*5.F!L'!S*).G5/--9$F!).+!a'!T5**#.7F!fK)$)J
1955! j0).$#e)$#-.! $49-074! 35#74$! 5j0)*#e)$#-.! ).+! "#)%!
,-995,$#-.Fh!#.!CVPRF!:B&AF!MM'!&;:=P&;;<'!
!U<<V! T'!o45F!i'!Y#.F!('!b4).+9)%5G4)9F!).+!S'!_#9-+F!f[M$#H#e#.7!$45!
"#$!)**-,)$#-.!1-9!,-HM95%%#-.!-1!35#74$%!).+!),$#/)$#-.%!-1!+55M!
.509)*!.5$3-9G%Fh!#.!ICIPF!:B&AF!MM'!;@:>P;@;B'!
!U<=V! K'![G$)8F!i'!S)**qF!Z'!Z#.74F!).+!C'! Z49#/)%$)/)F! fZ,)*)"*5! H-+5*!
,-HM95%%#-.!"8!5.$9-M8!M5.)*#e5+!95M)9)H5$59#e)$#-.Fh! #.!ICLRF!
:B:B'!
!U<>V! ]'!T).7F!o'!Y#0F!k'!Y#.F!i'!Y#.F!).+!Z'!^).F!f^CWm!^)9+3)95J)3)95!
)0$-H)$5+!j0).$#e)$#-.!3#$4!H#D5+!M95,#%#-.Fh!#.!CVPRF!:B&AF!MM'!
@>&:P@>:B'!
!U<?V! \'! Y'! K5.$-.F! T'! o)95H")F! i'! S90.)F! k'! Y5b0.F! ).+! I'! O5970%F!
f\DM*-#$#.7! *#.5)9! %$90,$095! 3#$4#.! ,-./-*0$#-.)*! .5$3-9G%! 1-9!
5l,#5.$!5/)*0)$#-.Fh!#.!NIPSF!:B&<F!MM'!&:>AP&:??'!
!U<@V! S'!Y#0F!a'!T).7F!^'!O-9--%4F!a'!L)MM5.F!).+!a'!`5.%G8F!fZM)9%5!
,-./-*0$#-.)*!.509)*!.5$3-9G%Fh!#.!CVPRF!:B&=F!MM'!@B>P@&<'!
!U<AV! k'! Y#F!Z'!_0F! Y'! ('!_--*F!).+! I'! L#H-1$5F!fY5)9.#.7! g*$59!")%#%!1-9!
,-./-*0$#-.)*! .509)*! .5$3-9G! ,-HM95%%#-.Fh! #.! ICCVF! :B&AF! MM'!
=>:;P=>;:'!
!U=BV! ('!Y5"5+5/F!k'!_).#.F!a'!I)G40")F!X'![%5*5+5$%F!).+!('!Y5HM#$%G8F!
fZM55+#.7J0M!,-./-*0$#-.)*!.509)*!.5$3-9G%!0%#.7!g.5J$0.5+!b`J
+5,-HM-%#$#-.Fh!#.!ICLRF!:B&='!
!U=&V! n'!o4).7F!i'!o-0F!n'!a#.7F!]'!^5F!).+!i'!Z0.F!f\l,#5.$!).+!),,09)$5!
)MM9-D#H)$#-.%! -1! .-.*#.5)9! ,-./-*0$#-.)*! .5$3-9G%Fh! #.! CVPRF!
:B&=F!MM'!&A@<P&AA:'!
!U=:V! k'JK'! ]#HF! \'! `)9GF! Z'! k--F! L'! b4-#F! Y'! k).7F! ).+! K'! Z4#.F!
fb-HM95%%#-.!-1!+55M!,-./-*0$#-.)*!.509)*!.5$3-9G%!1-9!1)%$!).+!
*-3!M-359!H-"#*5!)MM*#,)$#-.%Fh!#.!ICLRF!:B&>'!
!U=;V! k'!b45.7F!O'!n'!k0F!I'!Z'!O59#%F!Z'!]0H)9F!C'!b4-0+4)98F!).+!Z'JO'!
b4).7F! fC.! 5DM*-9)$#-.! -1! M)9)H5$59! 95+0.+).,8! #.! +55M!
.5$3-9G%!3#$4!,#9,0*).$!M9-t5,$#-.%Fh!#.!ICCVF!:B&=F!MM'!:@=?P:@>='!
!U=<V! S'! `5.7F! T'! L).F! o'! Y#F! Z'! o4).7F! K'! n#5F! ).+! Z'! `0F! f\D$95H5!
.5$3-9G! ,-HM95%%#-.! /#)! g*$59! 79-0M! )MM9-D#H)$#-.Fh! #.! ECCVF!
:B&@F!MM'!;BBP;&>'!
!U==V! C'! E-/#G-/F! K'! `-+-M9#G4#.F! C'! [%-G#.F! ).+! K'! `'! (5$9-/F!
fL5.%-9#e#.7!.509)*!.5$3-9G%Fh!#.!NIPSF!:B&=F!MM'!<<:P<=B'!
!U=>V! a'! T).7F! S'! Y#0F! ).+! ^'! O-9--%4F! fO),$-9#e5+! b-./-*0$#-.)*!
E509)*!E5$3-9G%Fh!#.!CVPR WorkshopsF!:B&?F!MM'!=<=P==;'!
!U=?V! ]'! I'! I)-! ).+! `'! k#MF! Discrete Cosine Transform: Algorithms,
Advantages, Applications'!C,)+5H#,!`95%%F!:B&<'!
!U=@V! K'! L)0"H).! ).+! a'! a)9,5**#.F! JPEG2000: Image Compression
Fundamentals, Standards and Practice'!E-935**F!aCF!cZCm!]*0359F!
:BB&'!
!U=AV! _'!Z0**#/).F!f(59%)$#*5!(#+5-!b-+#.7!2((b6!)99#/5%Fh!#.!VCIPF!:B:BF!
MM'!&P&'!
!U>BV! ]'!^5F!n'!o4).7F!Z'!I5.F!).+!i'!Z0.F!fK55M!95%#+0)*!*5)9.#.7!1-9!
#H)75!95,-7.#$#-.Fh!#.!CVPRF!:B&>F!MM'!??BP??@'!
!U>&V! _'! ^0).7F! o'! Y#0F! Y'! /).! +59! a))$5.F! ).+! ]'! W'! T5#."59759F!
fK5.%5*8!,-..5,$5+!,-./-*0$#-.)*!.5$3-9G%Fh!#.!CVPRF!:B&?F!MM'!
<?BBP<?B@'!
!U>:V! ]'!o4).7F!k'!Y#F!T'!o0-F!Y'!o4).7F!Y'!().!_--*F!).+!I'!L#H-1$5F!
f`*07J).+JM*)8! #H)75! 95%$-9)$#-.! 3#$4! +55M! +5.-#%59! M9#-9Fh!
ArXiv200813751 Cs EessF!C07'!:B:B'!
!U>;V! S'!Y#HF!Z'!Z-.F!^'!]#HF!Z'!E)4F!).+!]'!a0!Y55F!f\.4).,5+!+55M!
95%#+0)*! .5$3-9G%! 1-9! %#.7*5! #H)75! %0M59J95%-*0$#-.Fh! #.! CVPR
WorkshopsF!:B&?F!MM'!&;>P&<<'!
!U><V! a'!Z).+*59F!C'!^-3)9+F!a'!o40F!C'!o4H-7#.-/F!).+!Y'Jb'!b45.F!
fa-"#*5E5$(:m! #./59$5+! 95%#+0)*%! ).+! *#.5)9! "-r*5.5,G%Fh! #.!
CVPRF!:B&@F!MM'!<=&BP<=:B'!
!U>=V! O'!E'!X).+-*)F!Z'!^).F!a'!T'!a-%G53#,eF!]'!C%49)1F!T'!i'!K)**8F!).+!
]'!]50s59F! fZj055e5E5$m!C*5DE5$J*5/5*!),,09),8! 3#$4!=BD! 15359!
M)9)H5$59%!).+!uB'=aS!H-+5*!%#e5Fh!#.!arXiv:1602.07360 [cs]F!:B&>'!
!U>>V! Y'! ^0).7! et al.F! fb-.$9-**)"*5! -9$4-7-.)*#e)$#-.! #.! $9)#.#.7!
YOUNG ET AL.: TRANSFORM QUANTIZATION FOR CNN COMPRESSION 15
KEE%Fh!#.!CVPRF!:B:B'!
!U>?V! _'! K5%t)9+#.%F! ]'! Z#H-.8).F! I'! `)%,).0F! ).+! ]'! ])/0G,0-7*0F!
fE)$09)*!E509)*!E5$3-9G%Fh!#.!NIPSF!:B&=F!MM'!:B?&P:B?A'!
!U>@V! i'! a)9$5.%! ).+! I'! _9-%%5F! f[M$#H#e#.7! .509)*! .5$3-9G%! 3#$4!
]9-.5,G59J1),$-95+! )MM9-D#H)$5! ,09/)$095Fh! #.! ICMLF! :B&=F! MM'!
:<B@P:<&?'!
!U>AV! I'! _9-%%5! ).+! i'! a)9$5.%F! fC! ]9-.5,G59J1),$-95+! )MM9-D#H)$5!
O#%459!H)$9#D!1-9!,-./-*0$#-.!*)859%Fh!#.!ICMLF!:B&>F!MM'!=?;P=@:'!
!U?BV! L'! _5-975F! b'! Y)095.$F! n'! S-0$4#**#59F! E'! S)**)%F! ).+! `'! (#.,5.$F!
fO)%$! )MM9-D#H)$5! .)$09)*! 79)+#5.$! +5%,5.$! #.! )! ]9-.5,G59J
1),$-95+!5#75.")%#%Fh!#.!NIPSF!:B&@F!MM'!A==BPA=>B'!
!U?&V! C'! S59.),,4#)F! a'! Y5.785*F! ).+! _'! ^5..5j0#.F! f\D),$! .)$09)*!
79)+#5.$! #.! +55M! *#.5)9! .5$3-9G%! ).+! #$%! )MM*#,)$#-.! $-! $45!
.-.*#.5)9!,)%5Fh!#.!NIPSF!:B&@F!MM'!=A<&P=A=B'!
!U?:V! Y'! ^0).7F! K'! k).7F! S'! Y).7F! ).+! i'! K5.7F! fK5,-995*)$5+! S)$,4!
E-9H)*#e)$#-.Fh!#.!CVPRF!:B&@F!MM'!?A&P@BB'!
!U?;V! T'!_)-F!k'J^'!Y#0F!b'!T).7F!).+!Z'![4F!fI)$5!+#%$-9$#-.!1-9!H-+5*!
,-HM95%%#-.m! 19-H! $45-98! $-! M9),$#,5Fh! #.! ICMLF! :B&AF! MM'! :&B:P
:&&&'!
!U?<V! i'! b'! k5F! k'! ^).F! ).+! \'! b4)F! fK55M! ,-./-*0$#-.)*! 19)H5*5$%m! )!
75.59)*!+55M!*5)9.#.7!19)H53-9G!1-9!#./59%5!M9-"*5H%Fh!SIAM J.
Imaging Sci.F!/-*'!&&F!.-'!:F!MM'!AA&P&B<@F!i).'!:B&@'!
!U?=V! a'! O*#59*! ).+! S'! _#9-+F! Video Coding with Superimposed Motion-
Compensated Signals: Applications to H.264 and Beyond'!ZM9#.759!cZF!
:BB<'!
!U?>V! ('! ]'! _-8)*F!fL45-95$#,)*! 1-0.+)$#-.%!-1! $9).%1-9H!,-+#.7Fh! IEEE
Signal Process. Mag.F!/-*'!&@F!.-'!=F!MM'!AP:&F!Z5M'!:BB&'!
!U??V! C'! _59%4-! ).+! I'! a'! _9)8F! Vector Quantization and Signal
Compression'!E-935**F!aCF!cZCm!]*0359F!&AA&'!
!U?@V! _'! i'! Z0**#/).F! f\l,#5.$! %,)*)9! j0).$#e)$#-.! -1! 5DM-.5.$#)*! ).+!
Y)M*),#).!9).+-H!/)9#)"*5%Fh!IEEE Trans. Inf. TheoryF!/-*'!<:F!.-'!=F!
MM'!&;>=P&;?<F!Z5M'!&AA>'!
!U?AV! Z'! Y*-8+F! fY5)%$! %j0)95%! j0).$#e)$#-.! #.! `baFh! IEEE Trans. Inf.
TheoryF!/-*'!:@F!.-'!:F!MM'!&:AP&;?F!a)9'!&A@:'!
!U@BV! i'! a)DF! fW0).$#e#.7! 1-9! H#.#H0H! +#%$-9$#-.Fh! IRE Trans. Inf.
TheoryF!/-*'!>F!.-'!&F!MM'!?P&:F!a)9'!&A>B'!
!U@&V! ['! I0%%)G-/%G8! et al.F! fXH)75E5$! *)975! %,)*5! /#%0)*! 95,-7.#$#-.!
,4)**5.75Fh! Int. J. Comput. Vis.F!/-*'! &&=F! .-'! ;F! MM'! :&&P:=:F! K5,'!
:B&='!
!U@:V! E'!`'!i-0MM#!et al.F!fX.JK)$),5.$59!`591-9H).,5!C.)*8%#%!-1!)!L5.%-9!
`9-,5%%#.7! c.#$Fh! #.! Proceedings of the 44th Annual International
Symposium on Computer ArchitectureF!L-9-.$-F![EF! b).)+)F! :B&?F!
MM'!&P&:'!
!U@;V! k'! b45.F! L'! ]9#%4.)F! i'! Z'! \H59F! ).+! ('! Ze5F! f\859#%%m!C.! \.5978J
\l,#5.$! I5,-.g709)"*5! C,,5*59)$-9! 1-9! K55M! b-./-*0$#-.)*!
E509)*!E5$3-9G%Fh!IEEE J. Solid-State CircuitsF!/-*'! =:F! .-'! &F! MM'!
&:?P&;@F!i).'!:B&?'!
!U@<V! C'!Z)H)t+)9F!i'!a'!i-%5M4F!k'!o40F!`'!T4)$H-074F!a'!a)r#.)F!).+!
L'! ]9#%4.)F! fC! Z8%$5H)$#,! a5$4-+-*-78! 1-9! b4)9),$59#e#.7!
Z,)*)"#*#$8!-1!KEE!C,,5*59)$-9%!0%#.7!ZbCY\JZ#HFh!#.!2020 IEEE
International Symposium on Performance Analysis of Systems and
Software (ISPASS)F!:B:BF!MM'!=@P>@'!
!U@=V! \'! C70%$%%-.! ).+! I'! L#H-1$5F! fELXI\! :B&?! ,4)**5.75! -.! %#.7*5!
#H)75! %0M59J95%-*0$#-.m! +)$)%5$! ).+! %$0+8Fh! #.! CVPR WorkshopsF!
:B&?F!MM'!&:>P&;='!
!U@>V! a'!S5/#*),j0)F!C'!I-0H8F!b'!_0#**5H-$F!).+!a'!Y'!C*"59#Ja-95*F!
fY-3J,-HM*5D#$8! %#.7*5J#H)75! %0M59J95%-*0$#-.! ")%5+! -.!
.-..57)$#/5!.5#74"-9!5H"5++#.7Fh!#.!BMVCF!:B&:F!M'!&;='&P&B'!
!U@?V! I'!o58+5F!a'!\*)+F!).+!a'!`9-r59F!f[.!%#.7*5!#H)75!%,)*5J0M!0%#.7!
%M)9%5J95M95%5.$)$#-.%Fh!#.!Curves and SurfacesF!S59*#.F!^5#+5*"597F!
:B&:F!MM'!?&&P?;B'!
!U@@V! K'!a)9$#.F!b'!O-3*G5%F!K'!L)*F!).+!i'!a)*#GF!fC!+)$)")%5!-1!40H).!
%57H5.$5+! .)$09)*! #H)75%! ).+! #$%! )MM*#,)$#-.! $-! 5/)*0)$#.7!
%57H5.$)$#-.! )*7-9#$4H%! ).+! H5)%09#.7! 5,-*-7#,)*! %$)$#%$#,%Fh! #.!
ICCVF!:BB&F!MM'!<&>P<:;'!
!U@AV! i'JS'! ^0).7F! C'! Z#.74F! ).+! E'! C40t)F! fZ#.7*5! #H)75! %0M59J
95%-*0$#-.!19-H!$9).%1-9H5+! %5*1J5D5HM*)9%Fh!#.!CVPRF!:B&=F!MM'!
=&A?P=:B>'!
!UABV! L'! T).7! et al.F! fC`Wm! i-#.$! Z5)9,4! 1-9! E5$3-9G! C9,4#$5,$095F!
`90.#.7!).+!W0).$#e)$#-.!`-*#,8Fh!#.!CVPRF!:B:BF!MM'!:B?@P:B@?'!
!
Sean I. Young!95,5#/5+!S'b-H!).+!S'\'!+57955%!#.!%-1$3)95!5.7#.559#.7!
19-H!$45!c.#/59%#$8!-1!C0,G*).+!#.!:BB@F! ).+!$45!a'\.7Z,!).+!`4'K'!
+57955%!19-H!$45!c.#/59%#$8!-1!E53!Z-0$4!T)*5%F!Z8+.58F!#.!:B&&!).+!
:B&@F!95%M5,$#/5*8'!^5!#%!,0995.$*8!)!M-%$+-,$-9)*!95%5)9,459!)$!Z$).1-9+!
c.#/59%#$8F!Z$).1-9+F!bCF!cZC'!X.!:B&>F!45!3)%!)!/#%#$#.7!95%5)9,459!)$!
X.$59K#7#$)*!b-HH0.#,)$#-.%F!Z).!K#57-F!bC'!^#%!95%5)9,4!#.$595%$%!)95!
*)975J%,)*5!-M$#H#e)$#-.!).+!#./59%5!M9-"*5H%!#.!#H)75!M9-,5%%#.7'!^5!
95,5#/5+!$45!C`IZQXC`I!"5%$!M)M59!)3)9+!)$!KXbLC!:B&@F!$-75$459!3#$4!
K)/#+!L)0"H).'!
Wang Zhe!95,5#/5+!4#%!S'\'!+57955!19-H!S5#t#.7!i#)-$-.7!c.#/59%#$8!#.!
:B&:F!).+!$45!a'Z'!+57955!#.!,-HM0$59!)MM*#5+!$5,4.-*-78!19-H!`5G#.7!
c.#/59%#$8! #.! :B&='! ^5! #%! ,0995.$*8! )! %5.#-9! 95%5)9,4! 5.7#.559! )$! $45!
X.%$#$0$5!1-9!X.1-,-HH!I5%5)9,4F!CvZLCIF!Z#.7)M-95'!X.!:B&@F!45!3)%!)!
/#%#$#.7!%,4-*)9!)$!Z$).1-9+!c.#/59%#$8F!Z$).1-9+F!bCF!cZC'!^#%!95%5)9,4!
#.$595%$%! #.,*0+5! *)975J%,)*5! #H)75! ).+! /#+5-! %5)9,4F! .509)*! .5$3-9G!
,-HM95%%#-.F!).+!+55M!*5)9.#.7!4)9+3)95'!^5!4)%!M0"*#%45+!H-95!$4).!
:B!95%5)9,4!M)M59%!#.!,-HM0$59!/#%#-.!).+!+55M!*5)9.#.7F!).+!4)%!g*5+!
=!M)$5.$%'!^#%!3-9G!-.!1)%$!#.+5D#.7!).+!7*-")*!+5%,9#M$-9!)77957)$#-.!
4)%!"55.!)+-M$5+!"8!$45!a`\_JbK(Z!2a`\_!b-HM),$!K5%,9#M$-9%!1-9!
(#%0)*!Z5)9,46!%$).+)9+'!!
David Taubman!95,5#/5+!S'Z'!).+!S'\'!+57955%!#.!5*5,$9#,)*!5.7#.559#.7!
19-H!$45!c.#/59%#$8!-1!Z8+.58F!#.!&A@>!).+!&A@@F!95%M5,$#/5*8F!).+!$45!
a'Z'!).+!`4'K'!+57955%!19-H!$45!c.#/59%#$8!-1!b)*#1-9.#)!)$!S59G5*58F!#.!
&AA:!).+!&AA<F!95%M5,$#/5*8'!O9-H!&AA<!$-!&AA@F!45!3)%!3#$4!^53*5rP
`),G)9+w%! I5%5)9,4! Y)"-9)$-9#5%F! `)*-!C*$-F! bC'! ^5! t-#.5+! cEZT! #.!
&AA@F! 34595! 45! #%! ,0995.$*8! )! `9-15%%-9! 3#$4! $45! Z,4--*! -1! \*5,$9#,)*!
\.7#.559#.7! ).+! L5*5,-HH0.#,)$#-.%'! ^5! 4)%! )0$4-95+! $45! "--G!
i`\_:BBBm!XH)75!b-HM95%%#-.!O0.+)H5.$)*%F!Z$).+)9+%!).+!`9),$#,5F!
3#$4!a'!a)9,5**#.'!^#%!95%5)9,4!#.$595%$%!#.,*0+5!4#74*8!%,)*)"*5!#H)75!
).+! /#+5-! ,-HM95%%#-.F! H-$#-.! 5%$#H)$#-.! ).+! H-+5*#.7F! #./59%5!
M9-"*5H%! #.! #H)7#.7F! M59,5M$0)*! H-+5*#.7F! ).+! H0*$#H5+#)!
+#%$9#"0$#-.! %8%$5H%'! ^5! 95,5#/5+! $45! c.#/59%#$8! a5+)*! 19-H! $45!
c.#/59%#$8!-1!Z8+.58'!^5!4)%!95,5#/5+!$3-!"5%$!M)M59!)3)9+%!19-H!$45!
X\\\! b#9,0#$%! ).+! Z8%$5H%! Z-,#5$8! 1-9! $45! &AA>! M)M59! 5.$#$*5+! C!
b-HH-.!O9)H53-9G!1-9!I)$5!).+!K#%$-9$#-.JS)%5+!Z,)*#.7!-1!^#74*8!
Z,)*)"*5! b-HM95%%5+! (#+5-F! ).+! 19-H! $45! X\\\! Z#7.)*! `9-,5%%#.7!
Z-,#5$8! 1-9! $45! :BBB! M)M59! 5.$#$*5+! ^#74! `591-9H).,5! Z,)*)"*5! XH)75!
b-HM95%%#-.!3#$4!\Sb[L'!
Bernd Girod 95,5#/5+! $45! \.7#.559#.7! K-,$-9)$5! +57955! 19-H!
c.#/59%#$8!-1!^)..-/59F!_59H).8F!).+!$45!a'Z'!+57955!19-H!_5-97#)!
X.%$#$0$5!-1!L5,4.-*-78'!c.$#*!&AAAF!45!3)%!)!`9-15%%-9!3#$4!$45!\*5,$9#,)*!
\.7#.559#.7! K5M)9$H5.$F! c.#/59%#$8! -1! \9*).75.PE095H"597'! ^5! #%!
,0995.$*8!$45!I-"59$!Y'!).+!C0+958!Z'!^).,-,G!`9-15%%-9!-1!\*5,$9#,)*!
\.7#.559#.7F!Z$).1-9+!c.#/59%#$8F!bCF!cZC'!^5!4)%!)0$4-95+!-/59!>BB!
,-.1595.,5!).+!t-09.)*!M)M59%!).+!%#D!"--G%'!^#%!95%5)9,4!#.$595%$%!)95!
#.!$45!)95)!-1!#H)75F!/#+5-!).+!H0*$#H5+#)!%8%$5H%'!C%!).!5.$95M95.509F!
45!3)%!#./-*/5+!#.!.0H59-0%!%$)9$0M!/5.$095%F!)H-.7!$45H!`-*8,-HF!
(#/-! Z-1$3)95F! @x@F! ).+! I5)*E5$3-9G%'! ^5! #%! )! \cICZX`! O5**-3F! )!
H5H"59!-1!$45!E)$#-.)*!C,)+5H8!-1!\.7#.559#.7F!).+!)!H5H"59!-1!$45!
_59H).!E)$#-.)*!C,)+5H8!-1!Z,#5.,5%!2Y5-M-*+#.)6'!^5!95,5#/5+!$45!
\cICZX`! Z#7.)*! `9-,5%%#.7! S5%$! `)M59! C3)9+! #.! :BB:F! $45! X\\\!
a0*$#H5+#)!b-HH0.#,)$#-.!S5%$!`)M59!C3)9+!#.!:BB?F!$45!\cICZX`!
XH)75!b-HH0.#,)$#-.!S5%$!`)M59!C3)9+!#.!:BB@F!$45!\cICZX`!Z#7.)*!
`9-,5%%#.7! a-%$! b#$5+! `)M59!C3)9+!#.! :BB@F! $45! \cICZX`! L5,4.#,)*!
C,4#5/5H5.$!C3)9+!#.!:BB<F!).+!$45!L5,4.#,)*!C,4#5/5H5.$!C3)9+!-1!
$45!X\\\!Z#7.)*!`9-,5%%#.7!Z-,#5$8!#.!:B&&'
