K99/R00 – Sean I Young, PhD

RE: FOA PA-20-188

June 8, 2022

Dear NIH Staff,

This is an initial submission of a K99/R00 Research Career Development application in response to the Parent

FOA PA-20-188, entitled “Ultra-precision Clinical Imaging and Detection of Alzheimer’s Disease Using Deep

Learning”. My research uses deep learning methods to improve imaging and detection of early-stage Alzheimer’s

Disease in individuals and improve aging and AD research. Therefore, I am requesting assignment of this grant

application to the National Institute of Aging (NIA).

The following individuals have agreed to write letters of reference in support of my application:

1. Dr. Bernd Girod, Ph.D. Professor of Electrical Engineering,

Department of Electrical Engineering, Stanford University;

2. Dr. Vivek Goyal, Ph.D. ECE Associate Chair of Doctoral Programs, Professor (ECE),

Department of Electrical and Computer Engineering, Boston University; and

3. Dr. Gordon Wetzstein, Ph.D. Associate Professor of Electrical Engineering,

Department of Electrical Engineering, Stanford University.

Thank you very much for handling my submission. Please do not hesitate to contact me at any time during the

application process if I can be of assistance in any way.

Sincerely,

Sean I. Young

Research fellow, Harvard Medical School

Research affiliate, CSAIL, MIT

MGH/HST Martinos Center for Biomedical Imaging

149 13th St,

Charlestown, MA, 02129

Phone: (617) 758 9783

E-mail: siyoung@mgh.harvard.edu

PROJECT SUMMARY AND ABSTRACT

In Alzheimer’s Disease (AD) studies, longitudinal within-subject imaging and analysis of the human brain gives

us valuable insight into the temporal dynamics of the early disease process in individual subjects and allows to

assess therapeutic efficacy. However, longitudinal imaging tools have not yet been optimized for clinical studies

or for use on nonharmonized scans. Challenges include reduction of noise across serial magnetic resonance

imaging (MRI) scans while weighting each time point equally to avoid biases; and appropriately accounting for

atrophy all in the presence of varying image intensity, contrasts, MR distortions and subject motion across time.

Many general tools exist for detecting longitudinal change in carefully curated research data (such as ADNI)

in which the scan protocol has been harmonized across acquisition sites so as to minimize differential distortion

and gradient nonlinearities removed prior to data release. Unfortunately, these tools do not work accurately for

unharmonized MRI scans that comprise the bulk of the research data available, and on clinical data, where the

practical need for clinicians to schedule a subject on different scanners leads to additional differences in scans

acquired across multiple scan sessions. For retrospective analysis of past scans or clinical use, it is thus critical

to develop imaging tools that are agnostic to global scanner-induced differences in images but very sensitive to

subtle neuroanatomical change, such as atrophy in AD, that is highly predictive of the early disease process.

To address the above issues, we propose to design, implement, and validate a deep learning (DL) AD image

analysis framework for detecting neuroanatomical change in the presence of large image differences due to the

acquisition process itself, including the field strength, receive coil, sequence parameters, gradient nonlinearities

and B0 distortions, scanner manufacturer, and subject motion in the images across time. We leverage the fact

that, within a subject, there is a physical deformation that relates the brain scans acquired across time unlike the

cross-subject case. Focusing exclusively on longitudinal within-subject studies allows us to craft ultra-sensitive

registration and change detection tools that drastically outperform general purpose ones used in cross-subject

studies, where registration is intended only to find approximate anatomical correspondences. Our longitudinal

imaging framework is thus able to learn to disentangle true neuroanatomical change from irrelevant distortions.

Since the applicant has a computational background, the proposed training program at Harvard, MIT and

MGH will focus on neuroscience and neurology during the K99 phase to develop the skills needed to transition

to independence in the R00 phase. The applicant aims to become an expert in clinical imaging of AD and push

the limits of what is currently possible in AD research, fundamentally enhancing the quality of healthcare. We

believe that the proposed project is a first step in this direction and the tools developed will further pave the way

for clinical imaging and analysis of AD and neurodegenerative disease processes in general.

PROJECT NARRATIVE

Certain change in the human brain is strongly predictive of early disease processes (such as atrophy in the case

of Alzheimer’s Disease—AD). However, using conventional image processing to detect this anatomical change

in clinical MR scans taken across time is difficult due to various MR effects and subject motion, which can mask

the anatomical change of interest relevant to AD diagnosis and research. The aims of the proposed project are

to leverage the recent advances in deep learning to design, develop and evaluate an AD imaging and analysis

framework that can resolve anatomical change with high accuracy, aiding in the early diagnosis and intervention

before the onset of dysfunction.

FACILITIES AND OTHER RESOURCES: MARTINOS CENTER

The main facilities and resources of the MGH/HST Martinos Center for Biomedical Imaging at Massachusetts

General Hospital (MGH) are based on the hospital’s research campus in the Charlestown Navy Yard area. The

center has close affiliations with the Harvard–MIT Division of Health Sciences and Technology (HST) and the

Harvard Center for Brain Science Imaging Facility in Cambridge, MA. Satellite research facilities are located at

the Martinos Imaging Center at MIT. The center currently occupies about 85,000 ft

of land in the Charlestown

Navy Yard and houses basic and clinical research laboratories as well as educational areas and administrative

offices.

1 Large-bore Human MRI Systems at Martinos

The Martinos Center uses a range of Siemens MRI scanners of several software versions and has strong local

support from Siemens Healthcare through on-site engineers.

Bay 1: 3T Laboratory (Skyra). This is a 3T Siemens Skyra with 128-channel receive capabilities and a 2-channel

parallel transmit. The system comes with 128 RF channels, 40 mT/m gradients and a 70 cm patient bore for the

improved subject comfort (mandatory for fetal imaging) and stimulus access. The scanner provides Siemens 32-

and 64-channel head coils, and an assortment of body arrays. Bay 1 also contains an array of audiovisual and

sensory stimuli equipment for fMRI studies, including digital high-definition rear projection, audio stimulation and

a subject response device. The stimulus equipment is set up to be run from a PC or a Mac, or the user’s laptop

computer. Stimuli can trigger or be triggered by the scanner. Bay 1 has also been equipped with a state-of-the-

art power injector. The system is configured for simultaneous TMS/MRI operation, including a video navigation

system for the TMS stimulator. The Bay 1 area contains necessary subject care environment, including waiting

and changing rooms, support areas that include a business office, a data viewing area, a physician’s office and

computer and magnet rooms.

Bay 3: 3T Laboratory (Trio). This is a 32-channel Siemens TIM Trio 3T whole-body MRI scanner which has an

insertable 36-cm (gradient coil ID) head-only gradient. The whole-body gradient system uses the same gradients

as the 1.5T Avanto (45 mT/m strength and 200 T/m/s slew rate). It has 32 independent RF receive channels for

phased-array coils and includes a Siemens 32-channel head coil and a home-built 32-channel head coil for the

gradient insert. Bay 3 also features an insertable, asymmetric head gradient coil (Siemens AC88) that is capable

of 60 mT/m and slew rates exceeding 600 T/m/s at a 70% duty cycle. This enables single-shot 3-mm resolution

EPI with an echo spacing of 300 μs at a sustained rate of 14 images per second. Bay 3 also has an assortment

of audiovisual and sensory stimulus equipment for fMRI studies, including rear projection, audio stimulation, a

subject response device and an eye-tracking setup.

Bay 4: 3T Laboratory (Prisma Fit). This is a 3T Siemens Prisma Fit, 128-channel whole-body MRI scanner with

a two-channel transmit system. The system features the Siemens XR200 gradient system with 80 mT/m gradient

strength and 200 mT/m/ms maximum slew rate. Bay 4 is equipped with a full assortment of body-imaging coils

as well as Siemens 32-channel and 64-channel head-neck coils. Bay 4 also has multinuclear capability and an

MGH-built 8-channel 31P head array is available. Additionally, it has an assortment of audiovisual and sensory

stimulus equipment for fMRI studies, such as rear projection, audio stimulation, a subject response device and

an eye-tracking setup. Bay 4 has been configured to allow simultaneous TMS stimulation, as well as recording

of simultaneous EEG.

Bay 5: 7T Laboratory. This laboratory supports an ultrahigh-field 7T whole-body MRI scanner with a 70 mT/m

(200 T/m/s max slew rate) gradient set (SC72B) and 32 RF receive channels. The 7T whole body magnet (90

cm magnet ID) was built by Magnex Scientific (Oxford, UK). Siemens provided the conventional MRI console,

the gradient, its drivers and the patient table. The system is shielded by 460 tons of steel. Integration of these

components and the design and construction of RF coils were performed jointly by MGH and Siemens. With its

high-performance gradient set, the system can provide better than 100 μm resolution and ultra-fast EPI readouts

for reduced image distortion. The system is equipped with a home-built 32-channel coil and an 8-channel head

array coil for human imaging. A selection of specialized coils is also available for ex-vivo MR microscopy and

primate imaging. The system has multinuclear imaging capability and coils for 31P and 13C are available. The

scanner has been upgraded by Siemens to contain 8 independent 1-kW transmit channels that are capable of

simultaneous parallel excitation with different RF pulse shapes for B1 shimming and/or parallel transmit methods

such as transmit SENSE. The 7T laboratory includes a visual display system and a button box for the acquisition

of subject responses in the scanner. A MedRad power injector is installed for the injection of gadolinium contrast

agents. A total of 32 GB of RAM is installed in the image reconstruction computer, facilitating higher-resolution

reconstructions on the scanner, and cabling is installed for routine streaming of data for offline processing. The

system has been upgraded with parallel transmit capability.

Bay 6: 3T Laboratory (TIM Trio with MR-PET Insert). The combined MR-PET system consists of a 3T Siemens

TIM Trio 32-channel whole-body MRI scanner (60 cm RF coil ID) with a BrainPET head-camera insert to allow

simultaneous MR-PET acquisitions. This system contains EPI, second-order shimming, CINE, MR angiography,

diffusion, perfusion and spectroscopy capabilities for neuro and body applications. It uses the same gradients

as the 1.5T Avanto (Bay 2; 45 mT/m strength, 200 T/m/s slew rate). The system is equipped with standard TIM

32-channel receivers, accommodating up to 32-element array coils. In addition, Bay 6 contains audiovisual and

sensory stimulus equipment for fMRI studies such as rear projection, audio stimulation, subject response device

and eye tracking setup. The system has one of the first PET cameras capable of simultaneous PET acquisition

during MR acquisition. The PET system is a head-only insert camera. This scanner is in close proximity to the

cyclotron and radiopharmaceutical facility, allowing for imaging studies that use radiotracers with short half-lives.

Bay 7: 3T Laboratory (Biograph mMR). The Bay 7 Biograph mMR scanner consists of a 3T whole-body super-

conductive magnet with active shielding, external interference shielding and a whole-body PET scanner. It is

equipped with a gradient system with a maximum gradient amplitude of 45 mT/m and a maximal slew rate of

200 T/m/s. Separate cooling channels that simultaneously cool primary and secondary coils allow the application

of extremely gradient-intensive sequences. This scanner is equipped with the TIM RF coils that were custom

designed to minimize the 511-keV photon attenuation. The fully integrated PET detectors use APD technology

and LSO crystals (8x8 arrays of 4x4x20 mm3 crystals). The PET scanner’s trans-axial and axial fields of view

are 59.4 cm and 25.8 cm, respectively. This scanner is housed close to the cyclotron and radiopharmaceutical

facility.

Bay 8: 3T Laboratory (Skyra with Connectom Gradients). A new Siemens Skyra 3T MRI system was installed

and upgraded to a Siemens Connectom platform with the addition of new high-power gradients. It comes with

64 RF channels and 300 mT/m gradients providing a unique system for performing in-vivo diffusion imaging. The

scanner has the capability for EPI imaging at a sustained rate of 15 images per second, CINE, MR angiography,

diffusion and perfusion studies and spectroscopy. Bay 8 contains an assortment of audio, visual, and sensory

stimulus equipment for fMRI studies including rear projection, audio stimulation, subject response device. Stimuli

can trigger or be triggered by the scanner. The stimulus equipment can be run using a PC or a Mac computer

installed in the 3T laboratory or the user’s laptop computer. This system is dedicated to connectomics imaging

in support of the multi-site Human Connectome Project consortium.

2 MRI Systems on MGH Main Campus

MGH’s main campus is located in Boston, about 15 minutes from the Martinos Center in Charlestown. Frequent

shuttle transport is provided between the two campuses for both researchers and patients. Resources on the

main campus include MRI and PET imaging, support laboratories, animal housing facilities and the MGH medical

library. These facilities are located in several buildings across the campus.

1.5T Whole-body MR Systems. Four GE and Siemens whole-body MRI systems are equipped with hardware

and software for CINE, MR angiography and spectroscopy capabilities.

3T Whole-body MR Systems. Three Siemens Trio MRI scanners are available, equipped with hardware and

software systems to perform EPI, second-order shimming, CINE, MR angiography, diffusion, perfusion, as well

as spectroscopy for both neuro and body applications.

3 Computational Resources

The following computational resources are freely available to the PI for the proposed project.

Martinos Center. The Martinos Center’s computing infrastructure consists of over 400 Linux workstations and

150 Windows and Mac desktop computers on users’ desks owned by individual research groups. There is also

a farm of over 90 Linux servers which handles central storage, e-mail, websites and other shared services. The

total storage capacity across the Martinos Center including disks in local workstations and central storage is 4

PB. The Martinos Center also hosts a compute cluster with 846 Intel Xeon E5472 3.0GHz cores and over 5900

GB of distributed memory for batch analysis jobs. These IT facilities are supported by IT staff including one full-

time PhD-level manager, who directs two full-time system administrators. In-house software tools are developed

and supported by three full-time programmers. Available commercial software is: Advanced Visual Systems

(AVS), MathWorks Matlab and Sensor Systems MEDx for general computation, simulation and image analysis.

In fall 2020, the Martinos Center installed a new HPC GPU cluster with funding from the Massachusetts Life

Science Center (MLSC). It consists of:

• four NVIDIA DGX A100 servers with eight NVIDIA A100 GPUs, two AMD EPYC 7742 64 core CPUs, 1TB

of memory, and ten 100Gbe network connections;

• five EXXACT servers having ten NVIDIA RTX8000 GPUs, two Intel Xeon Gold 6226R 16 core CPUs,

1.5TB of memory, and one 100Gbe network connection;

• thirty-two DELL R440 servers with two Intel Xeon Silver 4214R 12 core CPUs, 384GB of memory and one

25Gbe network connection; and

• a 1.35PB VAST storage server with 100% solid-state storage and eight 100Gbe network connections.

The SLURM batch scheduler is used to manage job submission by users to the cluster. The NVIDIA DGX A100

servers run Ubuntu 18.04 while all EXXACT and Dell servers run CentOS 8.

Also available are Bruker BioSpin XWIN-NMR, OriginLab Origin, Acorn NMR Nuts suite for analysis of NMR

spectra, the Siemens IDEA environment for development of pulse sequences, as well as other general-purpose

image reconstruction software. A substantial level of internal software development for image and data analysis

is ongoing, using C/C++, Java, MATLAB, Pascal, Python, Perl, and TCL/TK.

Laboratory for Computational Neuroimaging. The Laboratory for Computational Neuroimaging (LCN) at the

Martinos Center hosts high-end workstations with NVIDIA Tesla graphics processing units (GPUs) and include

four V100, four P100, two P40s, and two K40 GPUs.

For high-performance image reconstruction, LCN is equipped with a custom-designed ScaleMP vSMP system

having 128 Xeon E5472 3.0-GHz cores and 1 TB shared memory. It uses a 40-GBit/s QDR Infiniband backplane

equipped with a Rackswitch G8000 48-port aggregation switch using two 100-Gbit/s Ethernet links with fiber-

optic extenders for real time data streaming from the MRI systems. The vSMP system runs the Siemens image

reconstruction software and can be fully incorporated into any of the Martinos Center’s MRI systems to enable

online image reconstruction of the very large or high-data-rate acquisitions.

Partners HealthCare. The ERISOne Linux cluster provided by Partners HealthCare offers investigators cloud

computational resources. This is a cluster of Linux remote-desktop and compute nodes connected to high-speed

storage. ERISOne has over 7000 CPU cores, a total of 56 TB memory and 5 PB of storage as well as specialized

parallel-processing resources such as GPUs and specialized networks. Popular and on-request applications are

installed and maintained centrally and supported by the Scientific Computing support team. Also available are

high-end Microsoft Windows Analytics servers for data analysis, a large-memory server with 3 TB memory, 64

CPU cores, a GPU compute cluster with four NVIDIA Tesla P100 GPUs and 24 M2070 GPUs.

Harvard Medical School. Harvard Medical School’s O

cluster is a shared high-performance computing (HPC)

environment with dedicated hardware available for high-memory and GPU-intensive tasks. The cluster contains

11,000 state-of-the-art compute cores, 32 GPUs (8 V100, 8 M40, 16 K80) and 68 TB of memory.

Advanced Computational Image Processing and Analysis Center (ACIPAC). The ACIPAC is a satellite of the

Martinos Center on MIT campus, which was established in collaboration with the MIT Artificial Intelligence (AI)

Laboratory. This facility provides extensive resources and expertise for solving practical image-processing and

analysis problems relevant to biomedical imaging. The ACIPAC is a bridge to affiliated MIT research community

and provides MIT students a direct avenue to engage in biomedical imaging research at the Martinos Center.

4 Support Resources at Martinos Center

Electronics and Machine Workshops. Instrumentation for design, construction and repair is distributed across

the High-Field Laboratory in Bays 2–3; Bays 4–5; and the Photon Migration Laboratory. These workshops are

equipped with tools for working with electronic circuitry, fiber optics and mechanical devices. There is additional

equipment for fabrication of printed circuit boards, instrumentation for electronic testing as well as measurement

of digital, analog, and RF circuitry: power supplies, voltmeters, an R/L/C meter, RF power meters, oscilloscopes,

gaussmeters, RF sweepers, an analogue impedance meter, a digital impedance analyzer and 5 HP RF network

analyzers. Also available are machine tools including drill presses, a belt sander, a grinder, a band saw, a 13in

lathe and a small milling machine. A stock of materials, hardware and electronic components is kept. Machine

tools are available to carry out complete computer-assisted design and fabrication of probes, animal carriers,

gradient coils, etc. In addition to these resources, Martinos investigators also have access to the MGH machine

shop. Design and simulation tasks are supported by multiprocessor workstations running the Remcom BioPro

FDTD software for simulation of electromagnetic fields, Electronics Workbench Multisim 2001 for simulation of

electrical networks and IMSI TurboCad for mechanical design.

RF-coil Laboratory. The RF-coil laboratory consists of a 500 ft

area with 6 RF-compatible work benches and 5

RF network analyzers. This space includes an electronics storage room for maintaining an extensive supply of

RF parts and tools. The laboratory has a circuit-board milling machine for creating circuit boards and coil layouts

and also a Dimension SST-1200 3D printer capable of making head-shaped models and helmet designs out of

ABS plastic from CAD files generated from MRI volume scans. Additional equipment includes an RF spectrum

analyzer, oscilloscopes (includes 1 GHz bandwidth, digital), RF frequency synthesizers and common electronics

measurement and test devices. This laboratory creates custom MRI coils, such as for brain tissue imaging.

Education Area. This area consists of a conference room, an audio-visual laboratory, staff offices and general

desk space for graduate students, postdoctoral fellows, and junior faculty. The audiovisual laboratory has been

equipped with computers, TV monitors, VCRs, carousels, teaching files and tapes.

Administration Area. The Martinos Center’s administration area is located on the second floor of Building 149

in area 2301. Facilities located here include fax machines, photocopying, standard, color laser printers as well

as mailboxes for staff and faculty. The area also contains faculty and secretarial office space and a conference

room.

FACILITIES AND OTHER RESOURCES: CSAIL, MIT

The MIT Computer Science and Artificial Intelligence Laboratory (CSAIL) is located at the MIT Stata Center, a

truly unique building designed by world-famous architect Frank Gehry. The Center occupies roughly 713,000 ft

of land area and is located in the new gateway to MIT campus at 32 Vassar Street.

1 Computational Resources

Stata Center. Stata Center’s main data network was designed and implemented by the CSAIL's full-time staff of

network and computing system managers, known collectively as T!G (the Infrastructure group). It consists of a

state-of-the-art 10 gigabit, single-mode fiber backbone, Cisco Catalyst series switches, a fault-tolerant network

topology, and 10/100/1000 Ethernet service via CAT6 copper cable to the desktop. Additionally, T!G maintains

and supports various centralized computing resources for the CSAIL community, including e-mail service, web

servers, DNS, DHCP, and many other common enterprise services. An 802.11g wireless local area network is

available throughout virtually all of the occupied space in the building.

Computer Vision Group. In addition to the resources described above, the computer vision group maintains an

extensive software library. This includes a range of generic public domain and commercial software: compilers

and debuggers, and software packages (including GNU and Sun proprietary), Netscape, Matlab, Mathematica

and different security related packages such as S-key (one-time passwords for access of the lab from outside),

ssh and PGP encryption. Graphics and image processing software packages include XGL, XIL, OpenGL, Open

Inventor, Data Explorer, Analyze, XV, IslandWrite, IslandPaint, and IslandDraw, and Adobe Photoshop for Unix

systems.

High Performance Cluster. Additionally, CSAIL investigators have access to a high-performance cluster at the

Massachusetts Green High Performance Computing Center (MGHPCC), a state-of-the-art computing facility in

Holyoke, Massachusetts. This resource provides peak capacity of 9,000–18,000 cores, peak memory of 36TB

and a high-speed working storage of 1.5 PB. This system is dedicated to life sciences research and is operated

jointly by a consortium of research groups from five universities and associated partners. The system enables

individual research participants to scale to the entire system for bursts of computation but is operated so that all

participants have balanced access on average.

Medical Vision Group. In addition to the resources available at CSAIL, Polina Golland’s Medical Vision Group

maintains a 400-core, 100-GPU computing cluster for computationally demanding processing. The group owns

270TB of secure storage space that is maintained by the infrastructure group. The computing resources available

to the PI will be instrumental for the proposed research, which requires extensive computing power, adequate

memory and storage capacity for manipulating large data sets.

EQUIPMENT

Equipment available to the candidate for use at the Martinos Center, MIT and MGH during the K99 phase of the

award is listed below.

1 MGH/HST Martinos Center for Biomedical Imaging

Large-bore MRI systems:

• 3.0T Siemens Prisma Fit 128-channel whole-body MRI

• 3.0T Siemens Skyra 128-channel whole-body MRI

• 7.0T Siemens TIM ultra-high-field 32-channel head-only MRI

• 3.0T Siemens Connectom 64-channel whole-body MRI with high-amplitude gradients

• 3.0T Siemens TIM Trio 32-channel whole-body MRI

• 3.0T Siemens TIM Trio 32-channel whole-body RMI with MR-PET insert

• 3.0T Siemens Biograph mMR whole-body MR-PET

High-performance computing cluster:

• 32 NVIDIA A100 GPUs across eight AMD EPYC 7742 64-core CPUs

• 52 NVIDIA RTX8000 GPUs across twelve Intel Xeon Gold 6226R 16-core CPUs

• 18 NVIDIA RTX6000 GPUs across six Intel Xeon Gold 6226R 16-core CPUs

• 12 additional NVIDIA GPUs: four Tesla V100, four P100, two P40, and two K40

• 32 DELL R440 servers each with two Intel Xeon Silver 4214R 12-core CPUs

Other equipment and facilities:

• RF-coil laboratory

• Electronics and machine workshops

• Education area

• Administration area

2 Computer Science and Artificial Intelligence Lab, MIT

High-performance GPU computing cluster:

• 48 NVIDIA Titan XP GPUs across 48 Intel Xeon E5-2620 8-core CPUs

• 36 NVIDIA GeForce 2080ti GPUs across 36 Intel Xeon Silver 4210 10-core CPUs

• 12 NVIDIA Quadro RTX5000 GPUs across 12 Intel Xeon Silver 4110 8-core CPUs

Other equipment and facilities:

• Electronics and machine workshops

• Education area

• Administration area

3 Massachusetts General Hospital Main Campus

Large-bore MRI systems:

• 3.0T Siemens Skyra 64-channel whole-body MRI

• 3.0T Siemens TIM Trio 32-channel whole-body MRI

• 1.5T Siemens Avanto 32-channel whole-body clinical MRI

High-performance GPU computing cluster (ERISXdl):

• 40 NVIDIA Tesla V100 across 40 Intel Xeon Gold 6226R 16-core CPUs

BUDGET JUSTIFICATION: K99 PHASE

Funding for the K99 phase of the award will support the personnel, activities and equipment listed below.

1 Key Personnel

Sean Young, PhD (PI). 10.8 calendar months (90.4%), years 1–2. Dr. Young is research fellow at the Martinos

Center, Harvard Medical School and research affiliate in the Computer Science and Artificial Intelligence Lab

(CSAIL), MIT. His research training in electrical engineering focused on computational imaging and displacement

modeling and estimation techniques for video compression and medical image registration. He assumes overall

responsibility for conducting the proposed research, which includes the design, execution, and interpretation of

experiments. The requested support is consistent with the established salary structure for MGH investigators of

equivalent qualification, rank and responsibilities.

Bruce Fischl, PhD (Primary Mentor). Dr. Fischl is Professor of Radiology at Harvard Medical School, and the

director of the Laboratory for Computational Neuroimaging (LCN) at the Martinos Center. He is well known for

his work on automated segmentation and labeling of brain morphometry data, implemented in the open-source

FreeSurfer software. His current research focuses on deep learning methods to increase the speed, flexibility

and accuracy of FreeSurfer. Dr. Fischl has coauthored over 290 journal publications and has over 122,000 total

citations. He is currently a principal investigator on multiple NIH R01, R25, RF1, and U01 grants. Dr. Fischl will

supervise the PI in all aspects of his research training, hold weekly meetings with the PI to ensure access to all

resources, facilities, and mentoring. No salary support or fringe benefits requested.

Polina Golland, PhD (Co-mentor). Dr. Golland is Professor of Electrical Engineering and Computer Science at

MIT and the director of the Medical Vision Group (MVG) in the Computer Science and Artificial Intelligence Lab

(CSAIL) at MIT. Her research interests span computer vision and machine learning. Her current research is

focused on developing statistical analysis methods for the characterization of biological processes from images

from MRI to microscopy. Dr. Golland has coauthored over 300 refereed publications that have been cited over

17,000 times in total. She has been a primary advisor of 14 PhD students and 9 postdoctoral researchers, many

of whom are now independent investigators at leading research institutions. Dr. Golland will supervise the PI in

the deep learning aspects of the research, meet with the PI weekly to ensure access to resources, facilities, and

mentoring. No salary support or fringe benefits requested.

Bradley Hyman, MD, PhD (Co-mentor). Dr. Hyman is Professor of Neurology at Harvard Medical School and

directs the Alzheimer’s Unit at the MGH Institute for Neurodegenerative Disease and Massachusetts Alzheimer’s

Disease Research Center. His current research is focused on the anatomical and molecular basis of dementia

in Alzheimer’s disease, and dementia with Lewy bodies. Dr. Hyman has coauthored over 750 journal publications

cited over 167,000 times in total. He is currently a principal investigator across multiple NIH R01, RF1, R56 and

P30 grants. Dr. Hyman will meet with the PI every four weeks to train him in clinical aspects of the Alzheimer’s

Disease research. No salary support or fringe benefits requested.

Randy L. Buckner, PhD (Co-mentor). Dr. Buckner is Professor of Psychology and Neuroscience, Department

of Psychology, Harvard University, and directs the Buckner Lab, Harvard University as well as the Psychiatric

Neuroimaging Research Division at MGH. His current research seeks to determine whether dysfunction can be

detected prior to clinical symptoms in individuals at genetic risk for illness. Dr. Buckner has coauthored over 260

journal publications with over 132,000 total citations. He is currently a principal investigator across a number of

NIH R01, U01 and T32 grants. Dr. Buckner will meet with the PI every two weeks to train him in translational

Alzheimer’s Disease research. No salary support or fringe benefits requested.

André JW van der Kouwe, PhD (Collaborator). Dr. van der Kouwe is Associate Professor of Radiology at HMS

and Director of 7T Imaging at the Martinos Center and a member of the LCN. He is a renowned expert in MR

physics and pulse-sequence design. In particular, Dr. van der Kouwe is widely recognized for his expertise in

prospective motion correction and has developed and published on several different navigator sequences for

motion tracking. During the mentored phase, he will meet with the PI as needed to provide consultation with the

modeling and estimation of MR distortions. Dr. van der Kouwe will be available to discuss career development

and continue to collaborate during the R00 phase. No salary support or fringe benefits are requested.

2 Non-key Personnel

Research technician (TBD). 0.60 calendar months (5%) in year 1, and 1.20 calendar months (10%) in year 2 is

requested. The research technician will help the PI with test-data acquisition at Martinos. The budget will partly

support the salary of a current RA who also has other admin duties for the PI’s primary mentor, Dr. Fischl.

3 Travel

International Conferences. $4,500 for one international conference per year is requested in years 1 and 2. This

covers conference registration ($750), travel ($2,500), lodging ($1,000) and meals ($250). A typical conference

duration is seven days including travel. The actual conference to attend will depend on factors including travel

restrictions by the US government and the covid19 situation in the destination country. One conference will be

chosen from:

• Intl. Conf. on Medical Image Computing and Computer Associated Intervention (MICCAI), held every year

(Vancouver, Canada in 2023, and Marrakesh, Morocco in 2024);

• Information Processing in Medical Imaging (IPMI), held biennially (location TBD in 2023);

• International Society for Magnetic Resonance (ISMRM, annual; Toronto in 2023, Singapore in 2024);

• Intl. Workshop on Biomed. Image Registration (WBIR), held biennially (location TBD for 2024);

• Intl. Conf. on Computer Vision (ICCV), held biennially (Istanbul, Turkey in 2023);

• European Conf. on Computer Vision (ECCV), held biennially (location TBD for 2024);

• Asian Conf. on Computer Vision (ACCV), held biennially (location TBD for 2024); and

• Image and Vision Computing New Zealand held biennially (location TBD for 2024).

The PI will write a conference paper for presentation at the chosen conference (preferably MICCAI or ISMRM

conferences). Attendance is mandatory for the paper to be published as part of the conference proceedings.

Domestic Conferences. $5,000 for two domestic conference per year is requested for years 1–2. This will cover

conference registration ($1,500), travel ($1,000), lodging ($2,000) and meals ($500). A typical duration for local

domestic conferences is five days including travel. Dr. Young will attend both:

• The BRAIN Initiative Investigators Meeting, held annually (location TBD for 2023); and

• Conference on Computer Vision and Pattern Recognition (CVPR) held annually (Indianapolis, IN in 2023

and Seattle, WA, 2024).

The PI will write conference papers for presentation at the two conferences. Attendance is mandatory for these

papers to be published in the conference proceedings. The PI will still attend even if his paper is rejected, since

these conferences attract leading medical imaging researchers and provide the PI with an opportunity to discuss

and receive feedback on his work.

4 Materials and Supplies

Laptop Computer. $4,660 is requested in year 1 for a 16” performance laptop computer. The specifications are:

• Apple M1 Max chip with 10-core CPU, 32-core GPU, and 16-core Neural Engine—for deep learning;

• 64GB unified memory—to allow programming, manuscript preparation, and video calls all in parallel;

• 4TB SSD storage—to hold and instantly access tens of thousands of high-resolution MR volumes;

• 16-inch Liquid Retina XDR display—for crisp visualization of MR images and segmentations; and

• Pro Apps Bundle for Education—to create movie-quality oral presentation videos for published work.

This performance laptop will replace the PI’s older 2014 15-inch Mac computer to ensure continued productivity

especially while on the run (commutes on the train, flight layovers, etc) and provide a portable environment for

scientific writing, communication with collaborators, presentations, a platform for development and debugging of

deep learning systems as well as access to the Martinos GPU cluster over the Internet.

Desktop Monitor. $1,900 is requested in year 1 for a performance 27” desktop monitor. The specifications are:

• 27-inch 5K Retina display; and

• 12MP Ultra-wide camera, studio-quality mics, and six-speaker sound system—to allow high quality remote

presentations.

This performance monitor will greatly boost the PI’s productivity on the above 16” laptop computer whenever he

needs to work after hours or on weekends especially for presentations to other continents or time zones.

Computer Accessories. $1,000 is requested in year 1 for keyboard ($200) and trackpad ($150) for use with the

desktop monitor, noise-cancelling headphones ($550) for work in noisy environments, and various dongles and

cables for connecting the laptop to the wired network, projector, etc ($100).

Stationery. $81 is requested in years 1 and 2 for stationery items, including pens and paper, and notebooks.

5 Others

IT Support Charges. $750 is requested in K99 year 1, and $1,070 in year 2. This covers:

• Unix account fees ($180) per year in years 1 and 2;

• Unix workstation maintenance ($250) per year in years 1 and 2; and

• High-performance RAID data storage at Martinos. 1TB in year 1 ($320) and 2TB in year 2 ($640).

RAID storage includes automatic backups and will be used to house all data and models related to the proposed

project. Workstation maintenance includes weekly backup of the workstation data.

Open Access Publication Charges. $3,545 is requested in K99 year 1 for one open access publication in the

IEEE Trans. Medical Imaging, and $7,090 in year 2 for two publications in the same journal. $3,295 for each

publication covers the open access fee ($2,045), mandatory page charges ($1,500) at $250 per page in excess

of the first eight pages (14 pages is typical for a paper in this journal). The requested funds may be used for

open access publications in NeuroImage ($3,450 per publication) instead of IEEE Trans. Medical Imaging.

Postdoc Learn-and-Lunch. $131 is requested in K99 year 2 for the PI to host a learn-and-lunch event among

the eight or so postdocs in the lab. Since the event will be held within the lab, the money will be used to order

food from food delivery services (UberEats, DoorDash, etc).

6 Indirect Cost

Indirect Costs. Indirect costs are requested at an 8% rate per federal guidelines for K awards.

BUDGET JUSTIFICATION: R00 PHASE

During the R00 Independent phase, $249,000 is requested per year for years 3–5.

CANDIDATE’S BACKGROUND

Sean I. Young is research fellow in the Department of Radiology, Harvard Medical School, and research affiliate

in the Computer Science and Artificial Intelligence Lab (CSAIL), MIT, where he works on various computational

neuroimaging problems. Previously, he was a postdoctoral scholar at Stanford University, where he worked on

fast computational methods for non-line-of-sight imaging. He received his PhD in electrical engineering from the

University of New South Wales in Sydney, Australia. In 2018, he received (together with Prof. David Taubman)

the Australian Pattern Recognition Society (APRS)’s best paper award for his work “Fast optical flow extraction

from compressed video”.

His research expertise lies in computational imaging and, in particular, modeling and

estimation of displacement for video compression, scene understanding and image registration problems.

Sean’s computational imaging research began in 2012 with his PhD work on video compression (under the

supervision of Prof. Taubman), exploring the use of dense and parametric displacement models for an efficient

representation of video. Since two successive video frames usually contain almost the same content, one way

to reduce redundancy—thereby compressing video data—is to transmit only the first video frame together with

displacement and difference fields to warp the first and synthesize the second one. Interestingly, these are the

same principles that underlie anatomical change detection in images. However, as the displacement field itself

has a transmission cost, compactly parameterizing this displacement field and precisely determining its values

led to a significant compression improvement. Sean’s US patent

filed in 2018 demonstrates the utility of high-

order (generalization of affine) displacement models for video compression, accompanied by a procedure that

can recover the values of displacement parameters to a high precision. Such high-order nonlinear displacement

models also play an important role in registering MRI volumes in the presence of geometric MR distortions.

Concurrently with his work on parametric displacement estimation techniques, Sean investigated using dense

deformation fields to improve video compression systems. In particular, Sean demonstrated that the estimation

of deformations and many related inverse problems in imaging and vision can be solved with high-dimensional

Gaussian filtering to obviate the need for slow, iterative optimization procedures. For certain tasks, this filtering

technique accelerated displacement estimation by a factor of 100.

Sean’s doctoral work resulted in a PhD thesis

on non-linear optimization and regularization techniques for inverse problems in imaging, and six first-author

conference papers

3–8

and two first-author journal papers

9,10

. Sean’s PhD research provides a sound, theoretical

foundation for robustly registering images and detecting neuroanatomical change in the proposed project.

After obtaining his PhD, Sean moved to Stanford University in 2019 for postdoctoral training with Prof. Bernd

Girod and continued his work on efficient methods for imaging and vision problems. Collaborating closely with

Dr. Gordon Wetzstein, he published an efficient method for non-line-of-sight imaging

that can resolve surfaces

of hidden scene objects with an unprecedented level of detail and a 1000-fold speed-up over the state-of-the-art

methods. At the same time, Sean published compression algorithms for neural networks

that enable efficient

imaging and computer vision on embedded devices, improving the accuracy of compressed networks over the

state-of-the-art compression methods. Neural network compression is an important problem in radiology as well

due to the demand for real-time image processing on imaging devices using neural network models. Sean’s work

at Stanford led to two first-author papers

11,13

in flagship computer vision conferences and four (three first author

and one senior author) papers

1,12,14,15

in two top-ranked computer science and artificial intelligence journals.

In response to the 2019 pandemic, Sean became determined to pursue research closer to people’s lives. He

moved to the Martinos Center late 2020 for medical imaging research with Dr. Juan Eugenio Iglesias and worked

on semi-supervised learning for imaging, which allows segmentation networks to be trained on a mix of labeled

and unlabeled data. This semi-supervised learning framework is used later for longitudinal brain segmentation

(see Research Strategy), where both labeled (but synthetic) and unlabeled (but real acquired) brain images are

used for anatomical segmentation of the brain optimal for disease detection. This work led to one journal paper

(under revision, listed in the biosketch) and also to Sean’s collaboration with NIST as a consulting author. More

importantly, this work familiarized Sean with the intricacies of MRI that can adversely affect our ability to detect

anatomical change, which is crucial for the imaging and analysis of Alzheimer’s Disease (AD) for example. These

include gradient non-linearities and B0-distortions as well as other acquisition-induced differences in the image

(intensity and contrast), all of which hinder change detection using optimization-based displacement estimation

techniques. More recently, Sean proposed a brain image registration framework

capable of recovering smooth

deformations with sub-voxel accuracy in the presence of MR distortions and noise. The image registration and

semi-supervised segmentation frameworks together form the scientific axis of the proposed project.

Sean’s expertise in computational imaging and modeling, coupled with his newly discovered aptitude for deep

learning in MRI makes him an extremely suitable candidate to embark on the proposed project. Sean’s mentors

from Harvard, MIT and MGH will afford him guidance and mentoring in the fields of neurology and radiology to

ensure that appropriate methods are used to address clinically relevant questions. The NIH K99/R00 award will

not only provide Sean an opportunity to work with mentors and solve a problem of universal importance but also

support him to grow as an independent researcher and ultimately raise the next generation of researchers to

carry on our scientific legacy.

CAREER GOALS AND OBJECTIVES

Dr. Sean I. Young aims to become a scientist and academic with expertise in clinical computational imaging of

neurological disease processes and is eager to apply his computational background and the recent advances in

deep learning towards translational research. Transitioning to faculty, he will start his own research lab focused

on imaging of neurological diseases and other closely related problems. Sean will utilize his experience with a

range of computational imaging problems and a deep understanding of neurodegenerative diseases—gained in

the K99 phase—to spearhead the computational imaging and aging research of his future lab in the R00 phase.

Sean sees the increase in demand for engineering academics with medical knowledge as an opportunity for

translational research, either in an engineering department with strong ties to a clinic or a radiology department

having strong ties to engineering. Sean’s transitioning to faculty will also allow him to pursue his passion as an

educator to motivate and inspire the next generation of researchers, passing on to them very carefully distilled

knowledge in hopes that they will take our scientific discoveries and innovations further. Since Sean is trained in

engineering and computational imaging, his aim for the K99 phase is to address the knowledge gap in neurology

and neurobiology, more specifically as they relate to neurodegenerative diseases or disorders. A comprehensive

training program in neurology and radiology steered by a mentoring committee of global leaders in neuroimaging

and neurology at Harvard, MIT and MGH will ensure that Sean is well grounded in all aspects of his project.

Sean’s objective for the first two years of the R00 phase is to flesh out research findings and prepare R01s to

continue developing clinical and research tools that push the boundaries of what is currently achievable in the

imaging of neurodegenerative disease processes. A natural extension to AD imaging is to fine-tune the software

and repurpose it for e.g., Huntington’s disease and schizophrenia. The deep learning-based longitudinal image

registration tools developed as part of the project are also applicable to other aging-related analyses. Seizing

the opportunity, Sean will thus prepare an R01 entitled “Contrast and distortion-invariant longitudinal registration

of brain images”, leveraging the same deep learning-based framework to provide further insight into the brain

aging process. He will distribute his longitudinal imaging tools as part of the popular FreeSurfer

software suite

for neuroimaging to positively impact the wide AD, aging and neuroimaging research community.

CAREER DEVELOPMENT PLAN

The main objective of the proposed career development plan is to address the candidate’s knowledge gaps in

neurology and general neuroscience. Additionally, the plan aims to provide the candidate with training in peer

review, grant writing and components of the academic life that will be essential to the scientific independence of

the candidate. During the K99 phase of the award, 75% of the candidate’s full-time effort will be devoted to the

proposed project and 25% to mentoring and coursework. During the R00 phase, Sean will continue with a 75%

effort dedicated to the proposed project. In this section, different components of Sean’s career development

activities are discussed. See Table 1 for a summary of the planned activities across the K99/R00 timeframes.

1 Direct Mentoring

The following mentoring committee of world-class experts in radiology, medical imaging and neurology across

Harvard, MIT and MGH will direct and evaluate my progress during the K99 phase of the award. Fig. 1 plots the

expertise of the mentors on a neurology–physics spectrum, showing the responsibilities of each mentor.

Dr. Bruce Fischl, PhD, Professor of Radiology at Harvard Medical School, is the director of the Laboratory for

Computational Neuroimaging (LCN) at the Martinos Center and will serve as a primary mentor. He is known for

his work on automated segmentation and labeling of brain morphometry data, implemented in the open source

FreeSurfer software suite. His current research focuses on the use of learning to increase speed, flexibility, and

accuracy of FreeSurfer. Dr. Fischl has coauthored over 290 journal publications, which have over 122,000 total

citations. He is currently a principal investigator across many NIH R01, R25, RF1 and U01 grants. He has been

primary mentor of 5 PhD students and 10 postdoctoral trainees, all of whom are now independent investigators

at leading research institutions. Sean will meet with Dr. Fischl weekly in the lab to receive feedback on progress

Table 1. Career development activities (coursework, conferences, mentoring, supervision, professional) across K99/R00 timeframes.

K99 YEAR 1

K99 YEAR 2

R00 YEAR 3

R00 YEAR 4

R00 YEAR 5

Coursework

MIT9.01, MIT9.015, HST.580

MIT9.13, MIT9.24

Instruct classes

Conferences

MICCAI or ISMRM, BRAIN, CVPR

MICCAI, ISMRM

Mentoring

Fischl, Golland, Buckner (weekly),

Hyman (monthly), 2 formal updates

Fischl, Golland (weekly), Buckner,

Hyman (monthly), 2 formal updates

Consultation

Fischl, Golland

Touch base

Fischl, Golland

Touch base

Fischl, Golland

Supervision

Summer undergrad students (1)

Undergrads

Grad students

Grads/postdocs

Professional

Grant and IRB writing, and

responsible conduct of research

Faculty job search, interview and

negotiation skills, and lab initiation

Grant writing

Help Fischl, Golland with grants

Prepare R01

Apply for R01

Obtain R01

and training in grant writing and peer review. Sean will also assist Dr. Fischl in preparing grants for the lab.

Dr. Polina Golland, PhD, Professor of Electrical Engineering and Computer Science, MIT, is the director of the

Medical Vision Group (MVG) in the Computer Science and Artificial Intelligence Lab (CSAIL). She will serve as

Sean’s primary co-mentor at MIT. Her research interests span computer vision and machine learning fields. Her

current research focuses on developing statistical analysis methods for characterization of biological processes

from images, from MRI to microscopy. Dr. Golland has coauthored over 300 refereed publications with over

17,000 total citations. She has been a primary advisor of 14 PhD students and 9 postdoctoral researchers, many

of whom are now independent investigators at world-class research institutions. Sean will meet with Dr. Golland

once a week at MIT to receive feedback on progress and training in deep learning research. Sean will also assist

Dr. Golland in preparing grant applications for collaborative projects with Dr. Fischl’s lab.

Dr. Bradley Hyman, MD, PhD, Professor of Neurology at Harvard Medical School, directs the Alzheimer’s Unit

at the MGH Institute for Neurodegenerative Disease and also the Massachusetts Alzheimer’s Disease Research

Center and will serve as a co-mentor. His research is focused on the anatomical and molecular basis of dementia

in Alzheimer’s disease and dementia with Lewy bodies. Dr. Hyman has coauthored over 750 journal publications

and has been cited over 167,000 times. He is currently a principal investigator on multiple NIH R01, RF1, R56

and P30 grants. He has received the Metropolitan Life Award, the Potamkin Prize, an NIH–NIA Merit award and

an Alzheimer’s Association Pioneer Award. Sean will meet with Dr. Hyman formally every quarter, with monthly

check-ins in his lab to receive training in clinical and neurological aspects of Alzheimer’s Disease research.

Dr. Randy L. Buckner, PhD, Professor of Psychology and Neuroscience at Harvard University, is the director of

the Buckner Lab at Harvard University and the Psychiatric Neuroimaging Research Division at MGH and will

serve as a co-mentor. His current research seeks to determine whether neurological dysfunction can be detected

prior to clinical symptoms in individuals at genetic risk for illness. Dr. Buckner has coauthored more than 260

journal publications with over 132,000 total citations. He is currently a principal investigator across a number of

NIH R01, U01 and T32 grants. He has received the MetLife Award for Medical Research, the Troland Research

Award from the National Academy of Sciences. Sean will meet with Dr. Buckner every week initially (and once

a month afterwards) to receive training in translational aspects of Alzheimer’s Disease research.

Dr. André JW van der Kouwe, PhD, Associate Professor of Radiology at Harvard Medical School, is director of

7T Imaging at the Martinos Center and a member of the LCN and will serve as a collaborator. He is a renowned

expert in MR physics and pulse-sequence design. In particular, Dr. van der Kouwe is widely recognized for his

expertise in prospective motion correction and has developed and published on a number of different navigator

sequences for motion tracking. During the mentored phase, Sean will meet with Dr. van der Kouwe as needed

to provide consultation with the modeling and estimation of MR distortions. Dr. van der Kouwe will be available

to discuss career development and continue to collaborate during the R00 phase.

2 Coursework

The coursework will help the candidate familiarize himself with key concepts in (ultimately) neurodegeneration.

9.01 Introduction to Neuroscience (MIT). Introduction to the mammalian nervous system, with emphasis on the

structure and function of the human brain. Topics include: function of nerve cells, learning and memory, sensory

systems, control of movement and brain diseases. Instructor: M. Bear (fall, K99 year 1).

9.015 Molecular and Cellular Neuroscience Core I (MIT). Surveys selected areas in molecular and cellular

neurobiology. Topics include nervous system development, axonal pathfinding, synapse formation and function,

synaptic plasticity, ion channels and receptors, cellular neurophysiology, glial cells, sensory transduction, and

examples in human disease. Instructor: J. T. Littleton, M. Sheng (fall, K99 year 1).

0.580 Data Acquisition and Image Reconstruction in MRI (HST). Applies analysis of signals and noise in linear

systems, sampling, and Fourier properties to magnetic resonance (MR) imaging acquisition and reconstruction.

Provides adequate foundation for MR physics to enable study of RF excitation design, efficient Fourier sampling,

parallel encoding, reconstruction of non-uniformly sampled data, and the impact of hardware imperfections on

reconstruction performance. Surveys active areas of MR research. Assignments include Matlab-based work with

real data. Includes visit to a scan site for human MR studies. Instructor: to be announced (fall, K99 year 1).

Fig. 1. Expertise of mentors (black markers) and collaborators (white markers) positioned on a neuro–physics spectrum, showing mentor

responsibilities. Drs. Fischl and Golland jointly provide computational mentoring at the candidate’s two institutions. Dr. Hyman provides

mentoring in neurology. Dr. Buckner bridges neurology with computation, Dr. van der Kouwe bridges physics with computation.

Purely neurological Computational Purely physical

vd Kouwe

Fischl

Golland

Buckner

Hyman

9.13 The Human Brain (MIT). Surveys the core perceptual and cognitive abilities of the human mind and asks

how these are implemented in the brain. Key themes include the functional organization of the cortex as well as

the representations and computations, developmental origins, and degree of functional specificity of particular

cortical regions. Emphasizes the methods available in human cognitive neuroscience, and what inferences can

and cannot be drawn from each. Instructor: N. Kanwisher (spring, K99 year 2).

9.24 Disorders and Diseases of the Nervous System (MIT). Topics examined include regional functional

anatomy of the central nervous system; brain systems and circuits; neurodevelopmental disorders including

autism; neuropsychiatric disorders such as schizophrenia; neurodegenerative diseases such as Parkinson’s and

Alzheimer’s; autoimmune disorders such as multiple sclerosis; gliomas. Emphasis is given to diseases where a

molecular mechanism is well-understood. Diagnostic criteria, clinical and pathological findings, genetics, model

systems, pathophysiology and treatment for each disorder/disease. Instructor: M. Sur (spring, K99 year 2).

3 Self Study

The candidate will also study MR physics under the guidance of Dr. van der Kouwe. Books to be used are:

Totally Accessible MRI. Written by Michael D Lipton, MD, PhD, this practical guide offers an introduction to the

principles of MRI physics. Each chapter explains the why and how behind MRI physics. Readers will understand

how altering MRI parameters will have many different consequences for image quality and the speed in which

images are generated. Practical topics, selected for their value to clinical practice, include progressive changes

in key MRI parameters, imaging time, and signal to noise ratio. Illustrations, complemented by concise text, will

help the reader gain a thorough understanding of the subject without requiring prior in-depth knowledge. To be

read in K99 year 1 with the accompanying YouTube video lectures.

Principles of Magnetic Resonance Imaging. Written by Dwight G Nishimura PhD, this book presents the basic

principles of magnetic resonance imaging (MRI), focusing on image formation, image content, and performance

considerations. Emphasis is put on the signal processing elements of MRI, particularly on the Fourier transform

relationships. While developed as a teaching text for an electrical engineering course at Stanford University, the

material should be accessible to those coming from other technical fields. Chapters 1–7 cover the foundational

material. Latter chapters (Chapters 8–11) provide extensions and selected topics. To be read in K99 year 2.

4 Conferences and Workshops

During the five years of K99/R00, the candidate will continue to attend scientific conferences to communicate

his research and develop new collaborations to stay on top of current developments in the field. In each year of

the K99/R00 award period, he will attend one international conference chosen from (in the order of preference):

the International Conference on Medical Image Computing and Computer Associated Intervention (MICCAI); the

International Society for Magnetic Resonance (ISMRM) annual meeting; the Information Processing in Medical

Imaging (IPMI) meeting, held biennially; the International Conference on Computer Vision (ICCV), held biennially

and the European Conf. on Computer Vision (ECCV), held biennially. In addition, the candidate will attend two

domestic conferences: the BRAIN Initiative Investigators Meeting, and the IEEE Conference on Computer Vision

and Pattern Recognition (CVPR), both held annually. Sean will attend the education sessions held in the MICCAI

or ISMRM meetings, which consist of two full days of lectures. Sean will focus on sessions that cover medical

registration and MR imaging of neurodegenerative diseases. Sean will also participate in relevant challenges in

MICCAI workshops held each year, such as Machine Learning for Medical Image Reconstruction and Machine

Learning for Clinical Neuroimaging.

In addition to attending conferences and workshops, Sean will continue to attend local scientific seminars such

as the weekly BrainMap seminar series at the Martinos Center, and the two yearly FreeSurfer courses at LCN.

5 Career Development and Transition to Independence

The Office for Research Career Development at Massachusetts General Hospital (MGH) offers a number of

professional development workshops. To advance his academic skills, Sean will participate in workshops on

grant and IRB-protocol writing, interview and negotiation skills as well as lab initiation across the five years of

the award. During the K99 phase, Sean will enhance his grant-writing capacity by assisting LCN with grants. To

prepare for the launch of his own lab by the end of the R00 phase, Sean will write R01s in year 3, submit them

in year 4 and begin mentoring students and postdocs in year 5. Sean will still continue to work with his mentors

during and beyond the R00 phase while also establishing new collaborations to gain independence in his field.

6 Progress Review and Evaluation

Sean’s research progress and career development will be reviewed and evaluated by Dr. Fischl every week at

the Martinos Center. Dr. Golland will also evaluate Sean’s research progress weekly at MIT. Sean will additionally

coordinate two meetings per K99 year with all mentors to provide formal updates on progress and milestones.

TRAINING IN THE RESPONSIBLE CONDUCT OF RESEARCH

Research at the MGH/HST Martinos Center and MIT employs the highest moral standards under strict guidance

from the center’s faculty mentors. The PI pledges to act in accordance with these standards.

1 Training Program and Schedule

In the first year of the K99 award, the PI will complete a three-part program comprising on-line training, a large

group meeting as well as lectures and discussions. A total of 8 hours of face-to-face lectures are included:

Online Training. MGH Partners HealthCare (PHS) uses the CITI (Collaborative Institutional Training Initiative)

program. The PI will perform on-line CITI training in responsible scientific conduct, with a refresher course every

three years, as mandated by MGH. In addition, the PI will complete the modules devoted to research misconduct,

data acquisition and management, responsible authorship and peer review, mentoring, as well as collaborative

research.

Large Group Meeting. In the first year of K99, the PI will attend a four-hour RCR seminar run by PHS. This is a

large group meeting that includes faculty presentations, panels and group discussions of contents covering

compliance/post-award financial management, conflict of interest and an overview of human-subject research

policies. The majority of session time is devoted to faculty presentations and panels focused on topics such as

data integrity and documentation, authorship, publication, and research misconduct.

Lectures and Discussions. This comprises lectures and discussion groups hosted by MGH. The PI will attend

a minimum of four lectures selected from a list of hospital-based offerings eligible for RCR credit. These expand

on topics covered in the CITI program and include topics such as documentation and integrity, interactions with

industry, mentor/mentee responsibilities, and introduction to clinical research.

2 Lecture and Seminar Topics

The large-group RSR seminar and the lectures and discussion groups draw upon a roster of senior hospital

faculty who teach on a revolving basis. Examples of current instructors and their topics include:

• Dr. David Altshuler (mentorship)

• Dr. Barbara Bierer (research misconduct, conflict of interest)

• Dr. F. Richard Bringhurst (conflict of interest, data Integrity)

• Dr. Dennis Brown (data integrity, publication, peer review, authorship)

• Dr. Tayyaba Hasan (research misconduct, authorship)

• Dr. Henry Kronenberg (documentation and data integrity)

• Mary Mitchell (grants financial management)

• Dr. Karen Miller (documentation and data integrity)

• Allison Moriarty (research misconduct)

• Dr. P. Pearl O’Rourke (IRB, IACUC, IBS, privacy board and research misconduct)

• Dr. Marc Sabatine (conflict of interest)

• Sarah White (IRB, privacy board)

Upon finishing all PHS program requirements, the PI will receive a certificate of completion.

In accordance with the MGH regulations, the PI has already completed the following CITI on-line courses:

• Conflict of Interest in Research (Oct 24, 2021)

• Biomedical Research Investigators and Key Personnel (Oct 23, 2021)

During the K99 phase of the award, Drs. Bruce Fischl and Polina Golland will ensure that the PI has a thorough

understanding of practical RCR aspects before initiating the experiments. They will also monitor my RCR training

throughout the mentored phase, and make sure that these skills are fully developed while transitioning to the

independent phase.

DESCRIPTION OF INSTITUTIONAL ENVIRONMENT

The K99 phase of this project will be carried out at the Martinos Center for Biomedical Imaging and CSAIL MIT.

1 MGH/HST Martinos Center for Biomedical Imaging

The MGH/HST Martinos Center is a research center based in the Department of Radiology of Massachusetts

General Hospital (MGH) and is closely affiliated with Harvard Medical School and the Massachusetts Institute of

Technology (MIT). Its mission is to create, develop and apply a variety of innovative and advanced imaging and

other technologies to facilitate a comprehensive understanding and better care of the human mind and body.

The Martinos Center currently has roughly 100 faculty researchers and more than 200 affiliated and visiting

faculty and postdoctoral research fellows and graduate students, with expertise across a spectrum of disciplines

including engineering, physical sciences, computational sciences and informatics, and behavioral and cognitive

neurosciences; basic and applied biological sciences, chemistry, imaging, radiological sciences as well as

clinical sciences. Additionally, the center serves as a resource to the greater MGH community and biomedical

imaging experts from institutions across the greater Boston area and around the United States. MGH provides

training for professionals from MIT, Harvard Medical School, and Tufts University. One crucial component of the

center’s research is its partnerships with the government, industry and private foundations including the National

Center for Research Resources, the Office of National Drug Control Policy, Siemens Healthcare, as well as the

National Foundation for Functional Brain Imaging.

The center occupies roughly 85,000 ft

of land area on the MGH East Campus in the Charlestown Navy Yard

and includes clinical, research, educational, and administration areas. A large expansion of the office space and

clinical and experimental space was recently completed. The center uses low-field, high-speed, high-field, and

conventional magnetic resonance (MR) imaging, MR spectroscopy, optical imaging, magnetoencephalography

(MEG) and electroencephalography (EEG) to explore properties of biological systems and to study and develop

novel ways to treat human pathologies such as neurodegenerative disorders and mental illnesses and cancer

cardiovascular diseases. The Martinos Center operates 6 MRI and 2 MR-PET large-bore scanners for human

clinical studies and 4 small-bore NMR systems for animal, and chemistry applications. The center has additional

laboratories specializing in MEG, optical imaging, photon migration, molecular imaging, behavioral testing, RF-

coil electronics and biochemistry. The aim of the Martinos Center is to foster interdisciplinary interaction through

training and integration of students, fellows, clinicians, and researchers with diverse backgrounds. Toward this

aim, the center supports and interacts with various training and educational programs. Educational endeavors

include affiliation and active involvement with degree-granting institutions and training programs sponsored by

NIH. Imaging tools and technology developed by the Martinos Center staff are taught to students, scientists, and

clinicians via specialized on-site and international workshops.

The Laboratory for Computational Neuroimaging at the Martinos Center explores new ways to conduct basic

and clinical, as well as cognitive neuroscience research through the use of MR imaging technologies. The lab’s

research is focused on optimization and analysis of neuroimaging data, encompassing structural, functional, and

diffusion MRI. Efforts are also focused on developing MR scanner pulse sequences and image reconstruction

methods that enhance image tissue contrast, reduce motion artifacts and improve the reliability of scans across

and within individuals. The lab has a number of computing resources, including a high-performance computing

cluster and GPU workstations, and promotes collaboration and resource sharing with other leading institutions.

2 CSAIL, Massachusetts Institute of Technology

CSAIL has long been a leader in the fields of Artificial Intelligence, Cognitive Science, and Computer Science

and is home to 110 principal investigators, which includes both MIT faculty and research staff. These numbers

include seven current or former MacArthur fellows and eight Turing award winners. CSAIL consistently ranks at

or near the top of undergraduate and graduate computer science programs in the world. Students and postdocs

in Golland’s group interact closely with other groups in CSAIL, including computer vision, machine learning and

MRI acquisition. This environment is ideal for creativity and exchange of ideas. In addition to the many relevant

courses and seminars at MIT, Golland’s group hosts a Biomedical Imaging and Analysis seminar series. Talks

in the series are given by invited top researchers in the field of medical image computing. Their visits provide an

invaluable opportunity for the students and postdocs in the group to share their own research and to brainstorm

new research directions with the thought leaders in the field in a less formal atmosphere of hosting the visiting

speaker. Our close connections with the machine learning and statistical inference groups at CSAIL and at MIT

is particularly relevant for the proposed project to stay abreast theoretical developments relevant for our domain.

In addition to CSAIL, Golland’s group also interacts closely with the MIT Institute for Data Science and Society

(IDSS), a recently founded epicenter of statistical modeling and machine learning at MIT, and with the MGH/BWH

Clinical Data Science Center (CDSC) recently established to explore applications of machine learning in medical

image computing and radiology. Through these centers, we come together with other local groups who share

our focus on inference and learning in medical images.

SPECIFIC AIMS

In Alzheimer’s Disease (AD) studies, longitudinal within-subject approaches have immense potential to increase

sensitivity and specificity and to improve the efficiency of clinical trials by requiring fewer subjects and providing

potential surrogate endpoints to evaluate therapeutic efficacy. There is also great potential for these studies to

enhance modeling of the disease process and its temporal dynamics. However, longitudinal tools have not yet

been optimized for use in clinical studies or in the wild with nonharmonized scans. Challenges include optimum

denoising of serial scans while weighting each time point equally to avoid bias in morphometry; accounting for

potential atrophy and handling varying session-specific MRI contrast and distortion when registering images.

Current computational tools for registration and detection of neuroanatomical change are mostly intended for

use on carefully curated research data

18–29

such as ADNI

, where the scan protocol has been harmonized across

acquisition sites to minimize differential distortions and the bulk of the remaining ones e.g. gradient nonlinearities

are removed prior to data release. Unfortunately, these tools fail in the presence of differences in the acquisition

that are ubiquitous in nonharmonized datasets and in clinical imaging, where scheduling a subject on the same

scanner and protocol as a previous session is difficult or impossible for a clinician to ensure. For use in clinical

settings, it is therefore critical to develop registration and change tools which can ignore large-scale acquisition-

induced image differences but are highly sensitive to subtle neuroanatomical change that is predictive of early

disease processes. Handcrafting an image processing pipeline to achieve this would be extremely challenging

and time-consuming so we turn to deep learning and train a deep neural network instead to achieve this goal.

At first glance, learning to both register images and detect anatomical change between them may seem like

conflicting objectives—maximizing the structural overlap across any two images minimizes detectable changes

and vice-versa. This is not the case with supervised learning

, in which a deep neural network can be trained to

as a byproduct of registration. Supervised registration thus extends rigid or affine registration to high-order non-

linear registration and can compensate for acquisition-induced deformations and subject motion. To expose the

registration network to these physical effects, we train the network on synthetic images produced by our neuro–

physics simulation engine, which is capable of synthesizing pairs of brain images simulating neuro-anatomical

change, such as atrophy, and physics-induced distortion. The registration network is supervised using synthetic

deformations as target fields, allowing it to learn to disentangle actual neuroanatomical change of interest from

the irrelevant physical deformation that relate the images. Previous deep learning-based registration

32–35

cannot

be used for change detection since they merely maximize structural overlap while classical optimization-based

methods

26,28

can produce grossly inaccurate results in the presence of contrast change and MR distortions.

Supervised registration has a number of critical advantages: (1) the resulting framework is completely free of

biases in image contrast and modality since the registration network is trained on a vast intentionally unrealistic

range of pairs of synthetic image contrasts; (2) supervision using known deformation fields as targets allows the

registration network to learn to predict change with subvoxel accuracy across images; and (3) end-to-end training

of the registration network eliminates preprocessing steps (such affine preregistration and skull stripping) and

facilitates downstream deep learning tasks such as longitudinal segmentation, which require registration to be

end-to-end differentiable. The core of our AD imaging framework is simulation-powered supervised registration

and semi-supervised longitudinal segmentation on both simulated and acquired data. Our specific aims are:

Aim 1 Deep Acquisition-Invariant Registration and Change Detection

1.1. Neuro–physics engine for simulating neuro-anatomical change and MRI distortions in image pairs.

1.2. Subvoxel-accurate registration and change detection separating atrophy from irrelevant MR distortions. A

neural network is trained using the image pairs produced by the neuro–physics engine from 1.1.

Aim 2 Deep Longitudinal Image Segmentation for Disease Detection

2.1. Biological models of the disease process for supervised spatiotemporal learning.

2.2. Spatiotemporally consistent longitudinal segmentation for optimal disease detection. A deep neural network

segments structures with spatiotemporal consistency by aggregating MR scans across all time points.

Aim 3 FreeSurfer Integration and Retrospective Longitudinal Analysis

3.1. Integration into FreeSurfer for dissemination to the wider aging, AD and neuroimaging community.

3.2. Use the FreeSurfer-integrated software for retrospective analysis of existing longitudinal cohort data with

irregular scan intervals and session-specific protocols collected by the Massachusetts AD Research Center.

A key deliverable of this project is a DL-based longitudinal imaging framework that can detect anatomical change

from a pair of scans acquired from any modality with a high level of accuracy. It can additionally segment them

by aggregating the scans across all time points. While our key interest is in detecting AD from MR images, our

imaging framework is adaptable to an array of other neurological diseases and imaging modalities. As such, we

expect the project to lead to further proposals for NIA-sponsored R01s during the candidate’s R00 award phase.

RESEARCH STRATEGY: SIGNIFICANCE

Brain MRI has become a primary tool for both studying brain aging and clinical diagnosis of neurodegenerative

disorders, because MR images across many time points allows us to visualize degenerative processes showing

atrophy of specific areas and degeneration of structures.

36–40

Yet, the ubiquitously present geometric distortions

in MR scans continue to limit its full potential for use in longitudinal aging and AD studies, especially on clinical

data, where session-specific image differences across time hamper studies. In clinical environments, a subject

imaged at two time points will typically be scanned on different hardware platforms, with different vendors, field

strengths, receive coils and sequence parameters. This implies that differential distortion between the two scan

sessions will prevent rigid or affine registration from aligning most of the brain with a high level of accuracy, and

intensity and contrast properties of the two images will vary substantially. However, retrospectively harmonizing

clinical data for diagnosis and research would require a handcrafted image processing pipeline tailored to each

setting, which is impractical for economic and technical reasons. The computational community has fortunately

made remarkable progress with deep learning in the last seven years

41,42

, allowing an optimal image processing

pipeline for many problems to be learned from data. One successful deep learning paradigm for machine vision

is supervised learning from synthetic data

43,44

, which reaps the full benefits of deep supervised learning without

manually labeled data. We extend this paradigm pioneered by our lab to transform AD imaging research.

Modernizing Image Processing for Diagnosis and Research (Aims 1–2)

Currently, clinicians spend large amounts of time seeking to visualize the same slice in image data acquired at

different times. This is, of course, not entirely possible due to differences in subject positioning, slice orientation

and location, and differential MRI distortions. All of these factors obscure true anatomical change, reducing the

diagnostic accuracy, and resulting in reduced efficacy of care and poorer patient outcomes. In this research, we

seek to eliminate all of these barriers to accurate detection of human brain changes over time by (1) developing

simulators that enable us to synthesize temporal atrophy and session-specific MRI distortions, (2) leverage the

flexibility and power of modern deep learning to create an image processing pipeline to register the images over

time while learning to ignore distortions, noise and atrophy, and (3) build deep learning algorithms that directly

detect true anatomical change in the presence of MRI distortions, noise and other non-anatomical changes over

time and also generate segmentations of multi-timepoint images that are optimal for detecting disease effects.

For aging and AD research, deep learning not only speeds up image processing compared to e.g., FreeSurfer

but also leads to better study outcomes, since, in the synthesis paradigm, a neural network model is exposed to

an infinite number of brain images of different contrast and distortion and works like a superhuman RA. This will

allow clinical and neuroscientific researchers to benefit from the improved statistical power and vastly reduced

bias in longitudinal studies. Also, longitudinal frameworks currently work only on scans with an identical contrast

across time points. This is a major limitation because hospital scheduling systems typically make it impractical

to match scanner, sequence, field strength, manufacturer, head coil, etc., across time. This potentially excludes

tens of thousands of datasets especially clinical ones that use both T2w and FLAIR at different times or across

different sites. Deep neural networks trained on simulated data change this and unlock existing clinical data for

use in aging and AD studies.

Economies of Scale across AD and Aging Research Community (Aim 3)

It is also worth pointing out that the PI is a research fellow in the FreeSurfer Lab and a regular contributor to the

development of the FreeSurfer software, especially its latest deep learning components. The software written for

this project will be integrated into FreeSurfer and distributed to its 56,000 licensees worldwide, further boosting

the significance of the research project. Currently, the FreeSurfer software has been used in over 5,000 cases

by the Alzheimer’s Disease Neuroimaging Initiative as well as other AD

45–48

, Parkinson’s

49–52

, Huntington’s

53–55

Diseases and Schizophrenia

56–60

studies. The software tools developed for the PI’s research thus translates to

economies of scale across the entire aging, AD, and neuroimaging community. The PI will use the FreeSurfer-

integrated version of his tools on Massachusetts ADRC data for analysis, giving himself ample opportunities to

test its user-friendliness (usability), portability, and scalability, and finally prepare the tools for mass adoption.

To summarize, the overall significance of this research project is as follows:

• Longitudinal AD imaging will work on clinical data for diagnosis and not just on curated research data and

ongoing studies need not adhere to original scan protocol, allowing e.g., better scanners part way through.

• Better imaging using deep learning will lead to increased statistical power for longitudinal AD studies.

• Existing longitudinal clinical data will be unlocked for research, enabling further aging and AD studies.

• Integrating PI’s longitudinal software into FreeSurfer will lead to economies of scale in aging research.

RESEARCH STRATEGY: INNOVATION

Generally speaking, innovations in clinical imaging are attributable to better hardware (sensing), better software

(reconstruction), or both. Our innovations are entirely software-based and thus improve imaging outcomes for

all clinics and research labs engaged in longitudinal analysis of neuroimaging data at zero cost. Our innovations

come from using supervised deep learning on simulated data for an optimal learned longitudinal reconstruction

pipeline (registration, change detection, and segmentation) even in the presence of various forms of noise and

geometric distortion irrelevant to neuroanatomical change of interest. Our premise is that combining cutting-edge

deep learning networks—which have dramatically improved the state-of-the-art in an array of applications—with

sophisticated models of MR image formation that can simulate artifacts that exceed the realistic range, will yield

a set of tools that provide unparalleled accuracy, robustness and speed, enabling significant advances in basic

and clinical neuroscience and resulting in a sustained positive impact on neuroimaging research. The proposed

research will greatly advance the goals of the NIH standards of rigor and reproducibility, by providing access to

rapid, accurate automated segmentations of neuroanatomical change that take any of wide class of input images

would allow large-scale studies to investigate whether functional, structural, genetic, molecular, and connectional

changes are associated with an array of diseases. Here we expand on our technical innovations in Aims 1–2.

Registration and Change Detection Using a Neuro–Physics Engine (Aim 1)

In Aim 1, we will develop a comprehensive neuro–physics engine capable of generating images with simulated

contrast, resolution, distortion, and subject motion, as well as degree and location of atrophy. This builds on the

FreeSurfer laboratory’s work on a hippocampal atrophy simulator

and the deformation-based simulator

. We

embed the simulator into a deep learning framework so that different synthetic images will be created on the fly

in every minibatch. The advantage of this approach is that the networks learn to be invariant to a wide array of

effects including the direction of MRI contrast, image resolution, subject motion, gradient nonlinearities, B0 and

B1 distortions and inhomogeneities, field strength, structure-specific atrophy as well as image noise.

A critical advantage of the synthetic approach is that there can be no mismatch between the label maps from

which ground truth is derived, and the imaging data, as the intensity images are synthesized directly from the

label maps. For detecting atrophy this is even more important, as atrophic effects are by definition subtle in the

early stages of AD and thus difficult to detect and/or manually label. The advantage of synthesizing atrophy is

similar to the advantage of synthesis in general—the ground truth labels localizing the atrophy are exactly correct

since the images with atrophy are synthesized directly from the labels. We emphasize that the patterns of atrophy

need not match exactly what we see in practice. We then train the registration network on pairs of such images.

An overwhelming majority of recent image registration networks

32–35

are trained “unsupervised”, in the sense

that ground-truth deformation fields are not required in the supervision of these networks. Instead, a surrogate

photometric loss is used to maximize the similarity between the fixed image and the moving one—warped by the

predicted deformation field—in lieu of a loss that penalizes the differences between the predicted and ground-

truth deformations. This approach is not capable of detecting the anatomical change directly since warping one

brain image merely to maximize its structural similarity with another minimizes detectable change. Supervising

the registration network using ground-truth warps (which contain subject motion and MR distortions but not the

neuroanatomical change) overcomes this challenge by allowing the network to learn to register with a precisely

relaxed fit in the regions of change, separating neuroanatomical change from physical deformations to produce

a difference image of neuroanatomical change as a byproduct of registration. We will rigorously test the network’s

ability to separate neuroanatomical change from MRI distortion by building a separate capacity-matched change

detection network that has access to the full warp field in addition to the pair of registered images and verifying

that this improves change detection accuracy. Our preliminary results below suggests this is indeed the case.

Semi-Supervised Longitudinal Segmentation for Disease Detection (Aim 2)

For Aim 2, we leverage the registration and change detection tools developed in Aim 1 to implement an optimal

anatomical segmentation framework for AD detection. This will build on top of the PI’s recent work on 3D medical

image segmentation with semi-supervised learning

62,63

, which allows segmentation neural networks to be trained

using a mix of labeled and unlabeled images to significantly improve segmentation accuracy. Segmentations of

a subject’s brain images acquired across multiple time points exhibit regularity or “smoothness” not only in the

3D spatial domain but also along the longitudinal direction. We segment a subject’s longitudinal series of brain

images, all at the same time, with regularity in the longitudinal direction imposed to improve the accuracy of the

anatomical segmentation at each time point. The segmentations are then fed to a simple AD prediction network.

Regularization of segmentation networks typically produces an over-smooth or blurred reconstructions so

longitudinally consistent brain segmentation without introducing biases in the individual brain segmentations is

challenging. Bias-free segmentations is even more important for AD detection since smoothing them along the

longitudinal direction translates to a loss of important anatomical change information—atrophy, dilation, etc. We

overcome this by training a 4D segmentation network on a mix of labeled synthetic data, produced by a disease

simulation engine—a longitudinal version of the neuro–physics engine—and unlabeled clinically acquired data

in a semi-supervised setup. 4D neural networks have only very recently been proposed (within the last 3 years)

for robot navigation and a handful of medical image segmentation tasks

64–66

, with significant improvements over

the 3D ones with other regularization mechanisms. Application of a 4D segmentation network to AD imaging is

a timely adaptation of a just sufficiently validated technology that will bring innovation to AD and aging research.

A critical advantage of our 4D convolutional neural network (as opposed to a typical 3D one that sees, say N

images as a single image with N channels) is that it can handle longitudinal scan series of different lengths. By

contrast, a 3D one needs to be trained for a specific series length and is locked to the trained length at inference

time. This is advantageous in clinical settings since each subject has a different number of images. Finally, the

end-to-end differentiable nature of the registration and change detection network (Aim 1) and the length-agnostic

nature of the longitudinal segmentation network (from Aim 2) together open up the possibility of finetuning both

networks simultaneously on clinical data to maximize diagnostic accuracy for a given acquisition type.

The overall innovations in this project can be summarized as follows:

• Neuro–physics engine simulating neuro-anatomical change and physical deformations for deep learning.

• Supervised registration on simulated data separates anatomical change from physical deformations.

• Longitudinal segmentation using a 4D convolutional neural network trained on disease simulation data.

• Training of 3D registration and 4D segmentation jointly on clinical data to maximize diagnostic accuracy.

RESEARCH STRATEGY: APPROACH

The analysis of longitudinal data requires comparing the same subject across multiple time points. The changes

that one seeks to detect appear against a backdrop that is largely constant or smooth. One common approach

is to align images using optimization-based rigid registration.

26–28

However, a rigid (or even affine) displacement

model introduces bias as acquisitions at different time points contain different geometric distortion not modeled

accurately by affine functions of image coordinates, causing spurious temporal trends to appear. In general, we

emphasize the need for higher-order deformation models to accurately register images. Unfortunately, correctly

modeling an overall deformation (subject motion + geometric distortions) as an expansion in some higher-order

deformation basis functions

67–72

is difficult especially given its dependence on the acquisition (shimming, B0 field

strength and readout direction, etc.). We alleviate this challenge associated with finding optimal basis functions

by leveraging supervised learning, which allows us to train a registration network on a large set of brain image

pairs, synthetically displaced and deformed to simulate subject motion and MR distortions met in practice.

Aim 1 Registration and Change Detection Using a Neuro–Physics Engine

To predict change maps for an array of real image types, we will train a deep convolutional neural network on a

landscape of images synthesized with a deliberately unrealistic range of anatomies, acquisition parameters, and

artifacts. From a dataset of precomputed, whole-head segmentations with brain and non-brain tissue labels, we

sample a segmentation at each optimization step and use it to generate a pair of gray-scale head scans with

randomized acquisition characteristics. In effect, this paradigm yields an endless stream of arbitrary images that

are used to optimize a network with trainable parameters in a supervised fashion.

Image Synthesis. Building on previous work

43,44

, we use a generative model to synthesize a stream of random

images with substantial variation in neuro-anatomy and physical acquisitions. At each training step, parameters

that dictate synthesis components are randomly sampled from predetermined ranges and probability values. We

emphasize that while the generated scans can appear implausible, these training images do not need to be so

realistic for the model to accurately generalize to real images at test-time. Our neuro–physics engine takes place

in four stages, which are run at each iteration of the training process (see Fig. 2 for a visual summary):

(a) Pick a label map from the Buckner39 dataset

with manual segmentations, and apply a random deformation

Fig. 2. Information flow in proposed synthesis/training. A single label map is input to the procedure (a). Simulated atrophy is then applied

to generate time 0 and 1 label maps (b). These label maps are then used to synthesize images using independently sampled contrasts

so that the time 1 and 2 images have different bias fields, gamma contrast augmentation (c). The two images are subjected to B0 and B1

distortions and subject motion, and the relative deformation between images 1 and 2 is computed (d). For the training, the two synthetic

images are input to our U-Net and the relative deformation field (a three-channel image in the 3D case) is used as the training target.

Registration

U-Net (Fig. 5)

Training

MSE Loss

(a) Pick a label map

from Buckner39, add

random deformation

(b) Synth Atrophy 0

(b) Synth Atrophy 1

(d) Deform Image 0

(d) Deform Image 1

(d) Relative

Deformation

Weight Update

Deformation 1 (Inverted)

Deformation 0

to increase diversity. We emphasize that the accuracy of these manual segmentations—that is, how well

they match the intensity images they derive from—is mostly irrelevant for our purposes as we discard the

original intensity images and synthesize new intensity images based only on the segmentations.

(b) Using the label map sampled in step 1, create two label maps by adding a different synthetic atrophy to each

one, eroding/dilating voxels that lie on the border between two labels, following specific anatomical criteria

(e.g., thalamus can be eroded by ventricular voxels growing into the thalamus label). Atrophy can be absent

in either or both of these images since accurately detecting the absence of atrophy will be equally important.

levels, bias fields, anisotropic smoothing kernels (to simulate different resolutions) and gamma augmentation

(a global exponentiation of all voxel values with one normally sampled parameter). The means can be fixed

to be the same across time if we expect the sequence to be more-or-less constant, or allowed to vary, which

helps the network to learn to detect changes across different MRI contrasts (such as a T1 and a FLAIR).

(d) Sample a pair of smooth deformations, representing the combination of subject motion and session-specific

MR distortion such as B0 and gradient nonlinearities, and apply them to the two images to produce two label

maps of the same (synthetic) individual at the two timepoints. Currently, smooth deformations are generated

in the space of 3D Legendre polynomials, whose higher order expansions naturally generalize the affine and

linear deformation models typically used in classical approaches. These smoothly deformed images are then

input to the registration network. The relative deformation becomes the training target.

The four stages are visually represented in Fig. 2. The output of this procedure is a pair of images representing

the same anatomy imaged at different times on scanners with different geometric distortions as well as intensity

characteristics. We emphasize that symmetry is maintained throughout this procedure and the two time points

are treated identically in terms of computations. Symmetry with respect to atrophy is automatically achieved as

either or both of these images can be subjected to atrophy. This reduces systematic bias towards atrophy. We

show in Fig. 3, examples of synthesized image pairs with small geometric deformations added to both.

Supervised Registration. The synthetic image pairs are used to train our registration network

, architected by

the PI to benefit from the duality of contrast-invariant feature extraction and deformation prediction.

This duality

is captured in the optical flow equation

used in most optimization-based image registration methods.

67–72

This

is a partial differential equation that relates a pair of images to the apparent deformation between the two:

(𝜕f

𝜕x

⁄

) ⋅ u + (𝜕f

𝜕y

⁄

) ⋅ v + (𝜕f

𝜕z

⁄

) ⋅ w = f

− f

)

in which u,!v,!w denote the x,!y,!z-components of the deformation, and f

, f

denote an image pair. Eq (1) says

the deformation relating any two images can be predicted from their intensities if they are colinear everywhere

in the coordinates and do not undergo a global intensity change (see Fig. 4 for interpretation). Conversely, the

two images can be harmonized in an intensity-invariant space if given the deformation field u,!v,!w. This duality

principle provides the basis for our supervised registration, where the network is jointly tasked with extraction of

intensity-invariant image features and the estimation of the deformation relating the two images.

In general, the images are not colinear in their coordinates everywhere, so a multi-scale linearization strategy

is required. Images are first linearized at the coarser scale and the deformation estimated at this scale is used

to align the images, ensuring that they become colinear at the finer scale.

To use this strategy with supervised

registration, our network (see Fig. 5) makes one crucial modification to the regular U-Net

. At each level of our

upward path, the features of image 1 are warped using the estimated deformation from the previous level and

only the residual deformation field is estimated at that level. We sum the base deformation and the residuals

across all levels to produce the final deformation. The residual deformation field at each level is estimated using

convolution–ReLU layers, the first layer having 24, 32, 48, 64, etc. input channels and the last one, 3 outputs.

Atrophy–Image Pair 1 Atrophy–Image Pair 2 Atrophy–Image Pair 3 Atrophy–Image Pair 4

Fig. 3. Example of synthesized atrophy–image pairs. Atrophy is highlighted in yellow in the first image of each pair. Note the ventricles

can be enlarged or reduced to maintain symmetry. The image pairs further undergo geometric transformations (distortions and motion)

not shown here and are used to train the registration network.

Note that from the left and the right-hand sides of (1) that it is (f

, f

− f

), not (f

, f

), which is processed for

displacement estimation. This suggests that feeding the features of f

and pre-computed feature differences

between f

and (warped) f

into deformation estimation blocks salvages one convolution layer per block, which

is substantial given that these blocks typically have no more than three convolution layers each. In practice, we

reparametrize the input further to the features of (f

+ f

, f

− f

) to help extraction blocks average the spatial

derivatives of the features across the two images, similarly to the practice in optimization-based methods.

This

reparameterization can be seen as a cross-channel Hadamard transform of the two image feature sets.

Validation. This is to be carried out soon under the supervision of Dr. Bruce Fischl (see biosketch), an expert

computational neuroscientist with extensive experience in examining pairs of images to detect neuroanatomical

change. Dr. Fischl will supervise the research technician to use the MGH information systems and identify 100

subjects from the PACS that have had at least two MRI sessions that include MPRAGE sequences. These will

not be constrained by scanner manufacturer, field strength or receive coil, and we expect to get a wide variation

in these properties (GE/Siemens, 3T/1.5T, 32 channel/24 channel/8 channel, etc.) and the MPRAGE acquisition

parameters that define the signal properties (flip angle, inversion time, resolution, FOV, recovery time). These

subjects will be distributed in 4 groups based on their clinical status at the two timepoints: (1) cognitively intact

at baseline and at follow-up, (2) mild cognitive impairment at baseline and follow-up, (3) controls at baseline and

mild cognitive impairment at follow-up, and (4) MCI at baseline and AD at follow-up. In groups (1) and (2) we

expect less change than in groups (3) and (4), and it is precisely this differentiation that is clinically important to

us (who will progress and who will not). Dr. Fischl will train the research technician to identify the regions of true

anatomical change in image pairs, and the RT will draw ROIs that contain the change. These ROIs do not need

to precisely delineate structures but rather region in the time 1 image where the changes occurred. We will have

the RT and the PI himself label 20 of the images twice to guarantee sufficient inter- and intra-rater reliability.

Preliminary Results. We train our registration network (Fig. 5) on synthesized data using the strategy shown in

Fig. 2. We use a batch size of 4 and optimize using the Adam optimizer for 800,000 iterations, across which the

learning rate is reduced from 1e–4 down to 1e–6. See Table 2 for spatial and intensity transforms used by the

training data. We compare the predicted anatomical change with true change to emphasize the extremely high

accuracy of our registration and change detection method. We validate our method on both synthetic images—

generated with neuroanatomical changes from Buckner39 label maps—and the real Buckner39 images without

change. The registration network did not see any real Buckner39 images during training but is able to register

them perfectly at test time, demonstrating the generalization ability of our network. This is shown in Fig. 6. Note

the image pairs are related through quadratic geometric transforms, not affine ones, to approximate geometric

distortions in addition to subject motion. Typical affine or rigid registration tools will misalign these images. In the

future, we plan to have real image pairs with manually labeled change for further validation and testing.

Potential Limitations and Fallback. Using simulated data to train a registration and change detection network

makes sense only if the trained model works on real clinical data. While our results show training on simulated

data generalizes to real data, using some real data during training may improve the registration. In this case, we

can use a subset of the labeled real data from PACS (for testing) with heavy augmentation to enhance training.

Table 2. Parameter range and (probability) used for random spatial and intensity transforms, applied separately to each image in the pair.

Spatial Transformation

Intensity Augmentation

Translate

Scale

Rotate

Shear

Quadratic

Noise Std

Multiply

Contrast

Gamma

Range (Prob)

±12 (1.0)

[0.75,1.25] (1.0)

±30º (1.0)

±0.012 (1.0)

±4 (1.0)

[0,0.05] (1.0)

[0.75,1.25] (0.5)

[0.75,1.50] (0.5)

Fig. 4. Geometric interpretation of eq. (1) in 1D. Whenever images

and f

are both linear (i.e. colinear) in the image coordinate x

, we

can find displacement

(

)

at coordinate x

using similar triangles

(

)

−

(

))

(

)

. This does not hold where the images are

not colinear, e.g. at

, so in general, images must first be aligned

at a larger scale, where it holds everywhere, only then is u refined.

) − f

)

(x)

Image Intensity

Image Coordinate (x)

u(x

)

u(x

)

(x)

Skip

Warp

Img1

Warped

Img0

Deform

Conv-ReLU

Conv Block

Fig. 5. U-Net for image registration (first four levels shown). Images

0 and 1 are concatenated in the batch axis and processed through

the network. The features of two images are reconcatenated along

the channel axis at each level of the upward path to be processed

into a residual deformation, used to warp the moving features, and

scaled and summed across levels to produce the final deformation.

Aim 2 Anatomical Segmentation Using 4D Convolutional Neural Network

After the brain images have been registered to a common time point, we use a 4D convolutional neural network

(CNN) to produce anatomical segmentations of the images in a spatio-longitudinally consistent manner. The 4D

CNN can be thought of as a 4D generalization of the 3D one and has recently found applications in a handful of

medical imaging problems.

65,66

This 4D neural network will be used to produce anatomical segmentations from

multiple time points simultaneously. Unfortunately, there are very few labeled longitudinal brain images that can

be used to train this 4D segmentation network whereas training only on synthesized images can lead to subpar

segmentation performance on real clinically acquired images. To overcome this, we will build on the PI’s recent

work on the supervision by denoising (SUD) framework, which allows segmentation neural networks to be trained

using a mix of labeled and unlabeled data (called semi-supervised learning). In our case, we have at our disposal

a disease simulation engine, which can generate synthetic pairs of longitudinal (image, label map) data. We also

have an abundance of unlabeled but real clinically acquired longitudinal images, which can be used for finetuning

the segmentation neural network to real acquisitions. SUD is detailed in PI’s unpublished work (see biosketch).

Supervision by Denoising. Training with unlabeled data typically involves enforcing some type of regularity on

the predicted segmentation, in lieu of agreement with the ground-truth label map. However, this can introduce a

severe bias in the predictions by smoothing the segmentation maps across multiple time points, which reduces

the extracted anatomical change relative to the true change one would like to detect. Instead, we appeal to the

relationship between regularization and denoising

to convert the task of regularizing to one of denoising. This

is advantageous since optimal segmentation denoisers can be learned from unpaired longitudinal label maps

generated using the disease simulation engine. Incorporating the optimal denoiser into the training is the key

innovation behind our “supervision by denoising” (SUD) framework (Fig. 7). SUD takes place in four stages:

(a) Pick a 4D labeled image and its label map. Transform the pair spatially and transform image intensities to

add variability. Pass the image through the 4D segmentation network and obtain the predicted segmentation.

(b) Pick a 4D unlabeled image. As with the labeled case, transform the image spatially and its intensities to add

some variability. Pass the image through the 4D segmentation network to obtain a predicted segmentation.

average of the predicted and the denoised segmentations with weights 1 − β and β, respectively. Obtain

another weighted average of this average and the average computed in the previous iteration (for the same

unlabeled image) using the weights 1 − α and α, respectively to generate a pseudo label map.

Synthesized

Buckner39

Real Acquired Buckner39

Label maps

Input images and true change (in yellow)

Predicted change and displacement

Fig. 6. Registration and change detection results (test). Given images 0 and 1 as input (columns 3–5), our registration network is able to

predict displacement fields and change (last 3 columns) with extremely high accuracy in the presence of non-linear (quadratic) geometric

distortions and noise. The first two rows of results are on images synthesized from label maps simulating change (first 2 columns). Change

predicted is visually indistinguishable from the true change. The last two rows are on real acquired Buckner39 (which have no anatomical

change). The network was trained only on synthetic Buckner39 images and yet generalizes perfectly to real data.

Label map 1 Label map 2 Image pair Image 0 Image 1 Change Displaced x Displaced y

(d) For each prediction from (a) and (b), compute the 4D Dice loss between the prediction and its corresponding

(pseudo) label map. Combine the two losses with weights 1 and λ to produce the final loss. Back-propagate

this loss through the 4D segmentation network.

The four steps are summarized as a block diagram in Fig. 7. By training on an abundance of labeled simulated

data as well as unlabeled but clinically acquired data, the network is able to bootstrap the learning even with no

labeled images while avoiding potential overfitting on the synthetic images, which are not always representative

of images from a target acquisition type.

Validation. Validation will be carried out using the same PACS data manually labeled in Aim 1. Since the scans

have corresponding AD diagnosis information, we will tune the hyperparameters α, β, and λ to produce optimal

disease predictions. We will additionally validate if simultaneously training the registration and the segmentation

networks is beneficial for improving the predictions, but this is subject to memory and computational constraints.

Preliminary Results. To demonstrate how semi-supervised learning using SUD improves generalization to real

images for which no labels are available at training time, we provide here results from our 3D anatomical brain

segmentation experiments. Naturally, this 3D case corresponds to the longitudinal one where all images have

been acquired at only one time point. We use one manually labeled Buckner39 image with unlabeled image data

from ABIDE-1

, -2

, ADHD200

, COBRE

, GSP

, MCIC

, OASIS

, UKBIO

and PPMI

. We compare our

segmentation U-Net trained on both labeled (Buckner39) and unlabeled data against the baseline, trained only

on the labeled data. The key result of this experiment is that at test time, our and baseline U-Nets perform very

similarly on unseen Buckner39 images but ours significantly outperforms the baseline on unseen images from

the other, unlabeled datasets (reference FreeSurfer segmentations are used to compare the accuracy). Another

important result is that correcting the segmentation post hoc (post-corrected

) and self-ensembling

, which are

two approaches related to SUD, do not adequately improve the accuracy over the baseline ones. The accuracy

improvements in terms of the Dice overlap and the 95 Hausdorff Distance are shown in Fig. 8. 3D visualizations

of segmentations from different approaches are shown in Fig. 9 to show these improvements are substantial.

Potential Limitations and Fallback. Note that only a very mild dependency exists between Aims 1 and 2. The

segmentation network for this aim requires preregistered images but here the registration does not need to be

highly accurate (we want to train the segmentation network to be robust to inaccurate preregistration). If Aim 1

cannot be completed for any reason, we will use simpler affine or rigid registration tools to bootstrap this task.

Aim 3 Integration into FreeSurfer and Analysis of MADRC Data

Once sufficient evaluation has been performed on the previous two aims, we will integrate all our software tools

into FreeSurfer for rapid dissemination to the AD and aging community. In the past year, various deep learning-

Fig. 7. Supervision by Denoising. We train a segmentation network on a mixture of labeled x

labeled

unlabeled data u

nolabel

. Segmented

output

is denoised spatially using a learned denoiser then iteration averaging to produce soft target z. The Dice loss between z

and z

is scaled and added to that on labeled data (

labeled

, y

) and the combined loss backpropagated to update the network weights. The three

algorithm hyper-parameters α, β, and λ can be optimized by hyper-parameter tuning, with additional feedback from a neurologist.

4D segmentation

neural network

Shape & intensity

augmentation

labeled

nolabel

4D Dice loss



Loss

4D Segmentation

denoiser/correction

−

1 − β

Fig. 8. Dice and 95HD statistics (test) for the brain segmentation task, conditioned on individual datasets. We plot the two metrics for the

baseline U-Net (trained on one labeled Buckner39 image), with additional post-correction, self ensembling, and SUD U-Net (ours).

ABD ABD2 ADH COB GSP MCIC OAS PPMI UKB B39

Dice Overlap

95HD (Voxels)

0.9

0.8

0.7

0.6

0.5

Baseline U-Net

Post-corrected

Self-ensembling

SUD U-Net (ours)

based software for neuroimaging have been incorporated into FreeSurfer (FS): SynthMorph

, SynthSeg

, and

SynthStrip, for registering, segmenting and skull-stripping brain images of arbitrary imaging modality, contrast

and resolution, and TopoFit

, a program for topologically aware placement of cortical surface on segmented MR

images. These programs are written in PyTorch and TensorFlow, included with every release of FreeSurfer so

that FS users can run these deep learning-based tools without installing additional software. A specialized suite

of longitudinal imaging tools developed by the PI for the proposed project will greatly complement the growing

number of deep learning-based tools in FS. The FS-integrated version of the longitudinal tools will then be used

to analyze Massachusetts Alzheimer’s Disease Research Center (MADRC)’s longitudinal cohort data, giving the

PI to apply his tools to real world AD studies as they unfold, as well as a chance to assess and improve the

software’s usability, portability, and scalability. All of the tools we develop and contribute will include unit tests,

system tests and tutorials on their usage aimed at both machine learning researchers who may want to extend

the techniques, as well as clinicians and neuroscientists seeking to use them to analyze their own data.

Massachusetts Alzheimer’s Disease Research Center (MADRC). MADRC, directed by Dr. Bradley T. Hyman

(see biosketch) hosts an integrated demographics and clinical data from clinical database with neuropath and

biomarkers, and imaging data for clinic patients and the research cohort. Currently, there are 453 subjects total

in the longitudinal research cohort of various demographics: education, race, and sex. These subjects are linked

to specific studies with accompanying imaging data including MRI and PET images (Tau, PiB, and FDG). Each

subject has between 2 and 8 MR scans across the course of their development of AD, diagnoses (dementia of

undetermined etiology, possible AD, AD, cognitive impairment NOS), neuropsych tests (UDS 2.0 and 3.0), and

MGH supplemental tests. The wealth of additional information accessible through the MADRC database allows

the PI to test his imaging tools conditional upon various factors, e.g., clinical status, sex, age, race. We anticipate

the MADRC longitudinal cohort to reflect the demographics of the Greater Boston area as based on the data

gathered by the US Census Bureau: 72.1% White, 11.8% Hispanic or Latino, 8.7% Black or African American,

6.8% Asian, 0.5% American Indian or Alaska Native, and 0.1% Native Hawaiian or Other Pacific Islander. Every

attempt will be made to finetune the imaging tools to work equally optimally across sex and ethnic backgrounds.

We provide in Table 3 a schedule of the PI’s sub-aims and other tasks across the K99/R00 timeline. The PI will

make every effort adhere to the proposed schedule. The PI will begin to transition to an R01 in year 5.

ADHD200 OHSU

3812101

FS reference

Baseline U-Net (0.765)

Post-corrected (0.796)

Temp ensembling (0.796)

SUD U-Net (ours) (0.852)

COBRE0040113

0040113

FS reference

Baseline U-Net (0.681)

Post-corrected (0.708)

Temp ensembling (0.756)

SUD U-Net (ours) (0.826)

Fig. 9. Brain structure segmentations (test) produced under different training regimes, shown together with reference FreeSurfer ones

(mean Dice overlap in parentheses are with respect to the reference). Cortex and white matter not shown for clarity. All training regimes

use only one labeled image and share the same underlying U-Net implementation. Post-corrected and SUD use the same auto-encoder

denoiser. SUD produces the most visually accurate segmentations as well as the highest mean Dice scores. Rendered using FreeView.

Table 3. PI’s sub-aims and *other tasks across K99/R00, with definite (YES), potential (MAYBE), and unlikely (NO) effort in these years.

SUBAIMS AND OTHER TASKS

K99 YEAR 1

K99 YEAR 2

R00 YEAR 3

R00 YEAR 4

R00 YEAR 5

AIM 1

1.1 Develop neuro–physics simulation engine

YES

MAYBE

1.2 Robust registration and change detection

YES

MAYBE

* Label and test on 100 images from PACS

MAYBE

YES

AIM 2

2.1 Develop AD processes simulation engine

MAYBE

YES

2.2 Longitudinal 4D segmentation using SUD

YES

* Test longitudinal segmentation on PACS

MAYBE

YES

AIM 3

3.1 Integrate developed tools into FreeSurfer

YES

MAYBE

3.2 Analyze MADRC longitudinal cohort data

YES

* Use integrated tools on other aging studies

MAYBE

YES

REFERENCES

1. Young SI, Girod B, Taubman D. Fast optical flow extraction from compressed video. IEEE Trans Image

Process. 2020;29:6409–6421.

2. Young SI, Xiu X, He Y, Vanam R. Higher-order motion models and graduated motion parameter estimation

for video coding. United States Patent US201762504963P; 2018.

3. Young SI, Mathew RK, Taubman DS. Robust sum of linear-log squared differences distortion measure and

its applications. Proc PCS; 2013. p. 73–76.

4. Young SI, Mathew RK, Taubman DS. Joint estimation of motion and arc breakpoints for scalable

compression. 2013. Proc GlobalSIP; 2013. p. 479–482.

5. Young SI, Mathew RK, Taubman DS. Embedded coding of optical flow fields for scalable video compression.

Proc MMSP; 2014. p. 1–6.

6. Young SI, Taubman D. Rate-distortion optimized optical flow estimation. Proc ICIP; 2015. p. 1677–1681.

7. Young SI, Naman AT, Mathew RK, Taubman D. Optimized decoding of JPEG images based on generalized

graph Laplacians. Proc PCS; 2016. p. 1–5.

8. Young SI, Mathew RK, Taubman D. Optimizing block-coded motion parameters with block-partition graphs.

Proc ICIP; 2016. p. 2037–2041.

9. Young SI, Naman AT, Taubman D. COGL: Coefficient Graph Laplacians for Optimized JPEG Image

Decoding. IEEE Trans Image Process. 2019 Jan;28(1):343–355.

10. Young SI, Naman AT, Taubman D. Graph Laplacian regularization for robust optical flow estimation. IEEE

Trans Image Process. 2020;29:3970–3983.

11. Young SI, Lindell DB, Girod B, Taubman D, Wetzstein G. Non-line-of-sight surface reconstruction using the

directional light-cone transform. Proc CVPR; 2020. p. 1407–1416.

12. Young S, Wang Z, Taubman D, Girod B. Transform quantization for CNN compression. IEEE Trans Pattern

Anal Mach Intell. 2021;1–1.

13. Young SI, Naman AT, Girod B, Taubman D. Solving vision problems via filtering. Proc ICCV; 2019. p. 5592–

5601.

14. Young SI, Girod B, Taubman D. Gaussian lifting for fast bilateral and nonlocal means filtering. IEEE Trans

Image Process. 2020;29:6082–6095.

15. Kim S, Sharma A, Liu YY, Young SI. Rethinking satellite data merging: from averaging to SNR optimization.

IEEE Trans Geosci Remote Sens. 2022;60:1–15.

16. Young SI, Balbastre Y, Dalca AV, Wells WM, Iglesias JE, Fischl B. SuperWarp: supervised learning and

warping on U-Net for invariant subvoxel-precise registration. Proc WBIR; 2022.

17. Fischl B. FreeSurfer. NeuroImage. 2012 Aug 15;62(2):774–781.

18. Apostolova LG, Thompson PM, Rogers SA, Dinov ID, Zoumalan C, Steiner CA, Siu E, Green AE, Small

GW, Toga AW, Cummings JL, Phelps ME, Silverman DH. Surface feature-guided mapping of cerebral

metabolic changes in cognitively normal and mildly impaired elderly. Mol Imaging Biol. 2010 Apr

1;12(2):218–224.

19. Avants B, Anderson C, Grossman M, Gee JC. Spatiotemporal normalization for longitudinal analysis of gray

matter atrophy in frontotemporal dementia. Proc MICCAI; 2007. p. 303–310.

20. Avants B, Cook PA, McMillan C, Grossman M, Tustison NJ, Zheng Y, Gee JC. Sparse Unbiased Analysis

of Anatomical Variance in Longitudinal Imaging. Proc MICCAI; 2010. p. 324–331.

21. Bernal-Rusiel JL, Greve DN, Reuter M, Fischl B, Sabuncu MR. Statistical analysis of longitudinal

neuroimage data with Linear Mixed Effects models. NeuroImage. 2013 Feb 1;66:249–260.

22. Davatzikos C, Genc A, Xu D, Resnick SM. Voxel-Based Morphometry Using the RAVENS Maps: Methods

and Validation Using Simulated Longitudinal Atrophy. NeuroImage. 2001 Dec 1;14(6):1361–1369.

23. Iglesias JE, Van Leemput K, Augustinack J, Insausti R, Fischl B, Reuter M. Bayesian longitudinal

segmentation of hippocampal substructures in brain MRI using subject-specific atlases. NeuroImage. 2016

Nov 1;141:542–555.

24. Laird NM, Ware JH. Random-Effects Models for Longitudinal Data. Biometrics. 1982;38(4):963–974.

25. Li Y, Wang Y, Xue Z, Shi F, Lin W, Shen D. Consistent 4D Cortical Thickness Measurement for Longitudinal

Neuroimaging Study. Proc MICCAI; 2010. p. 133–142.

26. Reuter M, Fischl B. Avoiding asymmetry-induced bias in longitudinal image processing. NeuroImage. 2011

Jul 1;57(1):19–21.

27. Reuter M, Schmansky NJ, Rosas HD, Fischl B. Within-subject template estimation for unbiased longitudinal

image analysis. NeuroImage. 2012 Jul 16;61(4):1402–1418.

28. Reuter M, Rosas HD, Fischl B. Highly accurate inverse consistent registration: A robust approach.

NeuroImage. 2010 Dec 1;53(4):1181–1196.

29. Sabuncu MR, Bernal-Rusiel JL, Reuter M, Greve DN, Fischl B. Event time analysis of longitudinal

neuroimage data. NeuroImage. 2014 Aug 15;97:9–18.

30. Weiner MW, Veitch DP, Aisen PS, Beckett LA, Cairns NJ, Green RC, Harvey D, Jack CR, Jagust W, Liu E,

Morris JC, Petersen RC, Saykin AJ, Schmidt ME, Shaw L, Shen L, Siuciak JA, Soares H, Toga AW,

Trojanowski JQ. The Alzheimer’s Disease Neuroimaging Initiative: A review of papers published since its

inception. Alzheimers Dement. 2013 Sep 1;9(5):e111–e194.

31. Russell S, Norvig P. Artificial Intelligence: A Modern Approach. 3rd edition. Upper Saddle River: Pearson;

2009.

32. Balakrishnan G, Zhao A, Sabuncu MR, Guttag J, Dalca AV. An Unsupervised Learning Model for

Deformable Medical Image Registration. Proc CVPR; 2018. p. 9252–9260.

33. Dalca AV, Balakrishnan G, Guttag J, Sabuncu MR. Unsupervised Learning for Fast Probabilistic

Diffeomorphic Registration. Proc MICCAI; 2018. p. 729–738.

34. Mok TCW, Chung ACS. Fast Symmetric Diffeomorphic Image Registration with Convolutional Neural

Networks. Proc CVPR; 2020. p. 4644–4653.

35. Kang M, Hu X, Huang W, Scott MR, Reyes M. Dual-stream pyramid registration network. Med Image Anal.

2022 May 1;78:102379.

36. Bendlin BB, Carlsson CM, Johnson SC, Zetterberg H, Blennow K, Willette AA, Okonkwo OC, Sodhi A, Ries

ML, Birdsill AC, Alexander AL, Rowley HA, Puglielli L, Asthana S, Sager MA. CSF T-Tau/Aβ42 Predicts

White Matter Microstructure in Healthy Adults at Risk for Alzheimer’s Disease. PLOS ONE. Public Library

of Science; 2012 Jun 6;7(6):e37720.

37. Westman E, Muehlboeck JS, Simmons A. Combining MRI and CSF measures for classification of

Alzheimer’s disease and prediction of mild cognitive impairment conversion. NeuroImage. 2012 Aug

1;62(1):229–238.

38. Davatzikos C, Bhatt P, Shaw LM, Batmanghelich KN, Trojanowski JQ. Prediction of MCI to AD conversion,

via MRI, CSF biomarkers, and pattern classification. Neurobiol Aging. 2011 Dec 1;32(12):2322.e19-

2322.e27.

39. McEvoy LK, Brewer JB. Quantitative structural MRI for early detection of Alzheimer’s disease. Expert Rev

Neurother. Taylor & Francis; 2010 Nov 1;10(11):1675–1688.

40. Dickerson BC, Bakkour A, Salat DH, Feczko E, Pacheco J, Greve DN, Grodstein F, Wright CI, Blacker D,

Rosas HD, Sperling RA, Atri A, Growdon JH, Hyman BT, Morris JC, Fischl B, Buckner RL. The Cortical

Signature of Alzheimer’s Disease: Regionally Specific Cortical Thinning Relates to Symptom Severity in

Very Mild to Mild AD Dementia and is Detectable in Asymptomatic Amyloid-Positive Individuals. Cereb

Cortex. 2009 Mar 1;19(3):497–510.

41. Zhang K, Zuo W, Chen Y, Meng D, Zhang L. Beyond a Gaussian denoiser: residual learning of deep CNN

for image denoising. IEEE Trans Image Process. 2017 Jul;26(7):3142–3155.

42. Ronneberger O, Fischer P, Brox T. U-Net: convolutional networks for biomedical image segmentation. In:

Navab N, Hornegger J, Wells WM, Frangi AF, editors. Proc MICCAI; 2015. p. 234–241.

43. Hoffmann M, Billot B, Greve DN, Iglesias JE, Fischl B, Dalca AV. SynthMorph: learning contrast-invariant

registration without acquired images. IEEE Trans Med Imaging. 2021;1–1.

44. Iglesias JE, Billot B, Balbastre Y, Tabari A, Conklin J, Gilberto González R, Alexander DC, Golland P, Edlow

BL, Fischl B. Joint super-resolution and synthesis of 1 mm isotropic MP-RAGE volumes from clinical MRI

exams with scans of different orientation, resolution and contrast. NeuroImage. 2021 Aug 15;237:118206.

45. Dickerson BC, Goncharova I, Sullivan MP, Forchetti C, Wilson RS, Bennett DA, Beckett LA, deToledo-

Morrell L. MRI-derived entorhinal and hippocampal atrophy in incipient and very mild Alzheimer’s

disease☆☆This research was supported by grants P01 AG09466 and P30 AG10161 from the National

Institute on Aging, National Institutes of Health. Neurobiol Aging. 2001 Sep 1;22(5):747–754.

46. Salat DH, Greve DN, Pacheco JL, Quinn BT, Helmer KG, Buckner RL, Fischl B. Regional white matter

volume differences in nondemented aging and Alzheimer’s disease. NeuroImage. 2009 Feb

15;44(4):1247–1258.

47. Sabuncu MR, Desikan RS, Sepulcre J, Yeo BTT, Liu H, Schmansky NJ, Reuter M, Weiner MW, Buckner

RL, Sperling RA, Fischl B, Alzheimer’s Disease Neuroimaging Initiative for the. The Dynamics of Cortical

and Hippocampal Atrophy in Alzheimer Disease. Arch Neurol. 2011 Aug 1;68(8):1040–1048.

48. Jernigan TL, Salmon DP, Butters N, Hesselink JR. Cerebral structure on MRI, part II: Specific changes in

Alzheimer’s and Huntington’s diseases. Biol Psychiatry. 1991 Jan 1;29(1):68–81.

49. Lyoo CH, Ryu YH, Lee MS. Topographical distribution of cerebral cortical thinning in patients with mild

Parkinson’s disease without dementia. Mov Disord. 2010;25(4):496–499.

50. Biundo R, Weis L, Facchini S, Formento-Dojot P, Vallelunga A, Pilleri M, Weintraub D, Antonini A. Patterns

of cortical thickness associated with impulse control disorders in Parkinson’s disease. Mov Disord.

2015;30(5):688–695.

51. Ibarretxe-Bilbao N, Junque C, Segura B, Baggio HC, Marti MJ, Valldeoriola F, Bargallo N, Tolosa E.

Progression of cortical thinning in early Parkinson’s disease. Mov Disord. 2012;27(14):1746–1753.

52. Worker A, Blain C, Jarosz J, Chaudhuri KR, Barker GJ, Williams SCR, Brown R, Leigh PN, Simmons A.

Cortical Thickness, Surface Area and Volume Measures in Parkinson’s Disease, Multiple System Atrophy

and Progressive Supranuclear Palsy. PLOS ONE. Public Library of Science; 2014 Dec 2;9(12):e114167.

53. Rosas HD, Liu AK, Hersch S, Glessner M, Ferrante RJ, Salat DH, Kouwe A van der, Jenkins BG, Dale AM,

Fischl B. Regional and progressive thinning of the cortical ribbon in Huntington’s disease. Neurology.

Wolters Kluwer Health, Inc. on behalf of the American Academy of Neurology; 2002 Mar 12;58(5):695–701.

PMID: 11889230

54. Rosas HD, Hevelone ND, Zaleta AK, Greve DN, Salat DH, Fischl B. Regional cortical thinning in preclinical

Huntington disease and its relationship to cognition. Neurology. Wolters Kluwer Health, Inc. on behalf of

the American Academy of Neurology; 2005 Sep 13;65(5):745–747. PMID: 16157910

55. Rosas HD, Salat DH, Lee SY, Zaleta AK, Pappu V, Fischl B, Greve D, Hevelone N, Hersch SM. Cerebral

cortex and the clinical expression of Huntington’s disease: complexity and heterogeneity. Brain. 2008 Apr

1;131(4):1057–1068.

56. Habets P, Marcelis M, Gronenschild E, Drukker M, van Os J. Reduced Cortical Thickness as an Outcome

of Differential Sensitivity to Environmental Risks in Schizophrenia. Biol Psychiatry. 2011 Mar 1;69(5):487–

494.

57. Kong L, Herold CJ, Lässer MM, Schmid LA, Hirjak D, Thomann PA, Essig M, Schröder J. Association of

Cortical Thickness and Neurological Soft Signs in Patients with Chronic Schizophrenia and Healthy

Controls. Neuropsychobiology. Karger Publishers; 2015;71(4):225–233. PMID: 26277883

58. Murakami M, Takao H, Abe O, Yamasue H, Sasaki H, Gonoi W, Takano Y, Takei K, Kasai K, Ohtomo K.

Cortical thickness, gray matter volume, and white matter anisotropy and diffusivity in schizophrenia.

Neuroradiology. 2011 Nov 1;53(11):859–866.

59. Palaniyappan L, Mallikarjun P, Joseph V, White TP, Liddle PF. Regional contraction of brain surface area

involves three large-scale networks in schizophrenia. Schizophr Res. 2011 Jul 1;129(2):163–168.

60. van Erp TGM, Greve DN, Rasmussen J, Turner J, Calhoun VD, Young S, Mueller B, Brown GG, McCarthy

G, Glover GH, Lim KO, Bustillo JR, Belger A, McEwen S, Voyvodic J, Mathalon DH, Keator D, Preda A,

Nguyen D, Ford JM, Potkin SG, Fbirn. A multi-scanner study of subcortical brain volume abnormalities in

schizophrenia. Psychiatry Res Neuroimaging. 2014 Apr 30;222(1):10–16.

61. Karacali B, Davatzikos C. Estimating topology preserving and smooth displacement fields. IEEE Trans Med

Imaging. 2004 Jul;23(7):868–880.

62. Bortsova G, Dubost F, Hogeweg L, Katramados I, de Bruijne M. Semi-supervised medical image

segmentation via learning consistency under transformations. In: Shen D, Liu T, Peters TM, Staib LH,

Essert C, Zhou S, Yap PT, Khan A, editors. Proc MICCAI; 2019. p. 810–818.

63. Li X, Yu L, Chen H, Fu CW, Xing L, Heng PA. Transformation-consistent self-ensembling model for

semisupervised medical image segmentation. IEEE Trans Neural Netw Learn Syst. 2021 Feb;32(2):523–

534.

64. Choy C, Gwak J, Savarese S. 4D Spatio-Temporal ConvNets: Minkowski Convolutional Neural Networks.

Proc CVPR; 2019. p. 3075–3084.

65. Hashimoto F, Ohba H, Ote K, Kakimoto A, Tsukada H, Ouchi Y. 4D deep image prior: dynamic PET image

denoising using an unsupervised four-dimensional branch convolutional neural network. Phys Med Biol.

2021 Jan;66(1):015006.

66. Takenaga T, Hanaoka S, Nomura Y, Nemoto M, Murata M, Nakao T, Miki S, Yoshikawa T, Hayashi N, Abe

O. Four-dimensional fully convolutional residual network-based liver segmentation in Gd-EOB-DTPA-

enhanced MRI. Int J Comput Assist Radiol Surg. 2019 Aug 1;14(8):1259–1266.

67. Rueckert D, Sonoda LI, Hayes C, Hill DLG, Leach MO, Hawkes DJ. Nonrigid registration using free-form

deformations: application to breast MR images. IEEE Trans Med Imaging. 1999 Aug;18(8):712–721.

68. Bookstein FL. Principal warps: thin-plate splines and the decomposition of deformations. IEEE Trans

Pattern Anal Mach Intell. 1989 Jun;11(6):567–585.

69. du Bois d’Aische A, Craene MD, Geets X, Gregoire V, Macq B, Warfield SK. Efficient multi-modal dense

field non-rigid registration: alignment of histological and section images. Med Image Anal. 2005 Dec

1;9(6):538–546.

70. Arsigny V, Pennec X, Ayache N. Polyrigid and polyaffine transformations: A novel geometrical tool to deal

with non-rigid deformations – Application to the registration of histological slices. Med Image Anal. 2005

Dec 1;9(6):507–523.

71. Wu YT, Kanade T, Li CC, Cohn J. Image Registration Using Wavelet-Based Motion Model. Int J Comput

Vis. 2000 Jul 1;38(2):129–152.

72. Davis MH, Khotanzad A, Flamig DP, Harms SE. A physics-based coordinate transformation for 3-D image

matching. IEEE Trans Med Imaging. 1997 Jun;16(3):317–328.

73. Horn BKP, Schunck BG. Determining optical flow. Artif Intell. 1981 Aug 1;17(1):185–203.

74. Brox T, Bruhn A, Papenberg N, Weickert J. High Accuracy Optical Flow Estimation Based on a Theory for

Warping. Proc ECCV; 2004. p. 25–36.

75. Sun D, Roth S, Black MJ. A quantitative analysis of current practices in optical flow estimation and the

principles behind them. Int J Comput Vis. 2014 Jan 1;106(2):115–137.

76. Romano Y, Elad M, Milanfar P. The little engine that could: regularization by denoising (RED). SIAM J

Imaging Sci. Society for Industrial and Applied Mathematics; 2017 Jan 1;10(4):1804–1844.

77. Di Martino A, Yan CG, Li Q, Denio E, Castellanos FX, Alaerts K, Anderson JS, Assaf M, Bookheimer SY,

Dapretto M, Deen B, Delmonte S, Dinstein I, Ertl-Wagner B, Fair DA, Gallagher L, Kennedy DP, Keown CL,

Keysers C, Lainhart JE, Lord C, Luna B, Menon V, Minshew NJ, Monk CS, Mueller S, Müller RA, Nebel

MB, Nigg JT, O’Hearn K, Pelphrey KA, Peltier SJ, Rudie JD, Sunaert S, Thioux M, Tyszka JM, Uddin LQ,

Verhoeven JS, Wenderoth N, Wiggins JL, Mostofsky SH, Milham MP. The autism brain imaging data

exchange: towards a large-scale evaluation of the intrinsic brain architecture in autism. Mol Psychiatry.

2014 Jun;19(6):659–667.

78. Milham M, Fair D, Mennes M, Mostofsky S. The ADHD-200 consortium: a model to advance the

translational potential of neuroimaging in clinical neuroscience. Front Syst Neurosci. 2012.

79. Calhoun V, Sui J, Kiehl K, Turner J, Allen E, Pearlson G. Exploring the psychosis functional connectome:

aberrant intrinsic networks in schizophrenia and bipolar disorder. Front Psychiatry. 2012.

80. Holmes AJ, Hollinshead MO, O’Keefe TM, Petrov VI, Fariello GR, Wald LL, Fischl B, Rosen BR, Mair RW,

Roffman JL, Smoller JW, Buckner RL. Brain Genomics Superstruct Project initial data release with

structural, functional, and behavioral measures. Sci Data. Nature Publishing Group; 2015 Jul 7;2(1):150031.

81. Gollub RL, Shoemaker JM, King MD, White T, Ehrlich S, Sponheim SR, Clark VP, Turner JA, Mueller BA,

Magnotta V, O’Leary D, Ho BC, Brauns S, Manoach DS, Seidman L, Bustillo JR, Lauriello J, Bockholt J,

Lim KO, Rosen BR, Schulz SC, Calhoun VD, Andreasen NC. The MCIC Collection: a shared repository of

multi-modal, multi-site brain image data from a clinical investigation of schizophrenia. Neuroinformatics.

2013 Jul 1;11(3):367–388.

82. Marcus DS, Wang TH, Parker J, Csernansky JG, Morris JC, Buckner RL. Open Access Series of Imaging

Studies (OASIS): Cross-sectional MRI data in young, middle aged, nondemented, and demented older

adults. J Cogn Neurosci. 2007 Sep 1;19(9):1498–1507.

83. Sudlow C, Gallacher J, Allen N, Beral V, Burton P, Danesh J, Downey P, Elliott P, Green J, Landray M, Liu

B, Matthews P, Ong G, Pell J, Silman A, Young A, Sprosen T, Peakman T, Collins R. UK Biobank: an open

access resource for identifying the causes of a wide range of complex diseases of middle and old age.

PLOS Med. Public Library of Science; 2015 Mar 31;12(3):e1001779.

84. Marek K, Jennings D, Lasch S, Siderowf A, Tanner C, Simuni T, Coffey C, Kieburtz K, Flagg E, Chowdhury

S, Poewe W, Mollenhauer B, Klinik PE, Sherer T, Frasier M, Meunier C, Rudolph A, Casaceli C, Seibyl J,

Mendick S, Schuff N, Zhang Y, Toga A, Crawford K, Ansbach A, De Blasio P, Piovella M, Trojanowski J,

Shaw L, Singleton A, Hawkins K, Eberling J, Brooks D, Russell D, Leary L, Factor S, Sommerfeld B, Hogarth

P, Pighetti E, Williams K, Standaert D, Guthrie S, Hauser R, Delgado H, Jankovic J, Hunter C, Stern M,

Tran B, Leverenz J, Baca M, Frank S, Thomas CA, Richard I, Deeley C, Rees L, Sprenger F, Lang E, Shill

H, Obradov S, Fernandez H, Winters A, Berg D, Gauss K, Galasko D, Fontaine D, Mari Z, Gerstenhaber

M, Brooks D, Malloy S, Barone P, Longo K, Comery T, Ravina B, Grachev I, Gallagher K, Collins M, Widnell

KL, Ostrowizki S, Fontoura P, Ho T, Luthman J, Brug M van der, Reith AD, Taylor P. The Parkinson

Progression Marker Initiative (PPMI). Prog Neurobiol. 2011 Dec 1;95(4):629–635.

85. Larrazabal AJ, Martínez C, Glocker B, Ferrante E. Post-DAE: anatomically plausible segmentation via post-

processing with denoising autoencoders. IEEE Trans Med Imaging. 2020 Dec;39(12):3813–3820.

86. Laine S, Aila T. Temporal ensembling for semi-supervised learning. Proc ICLR; 2017.

87. Billot B, Greve DN, Puonti O, Thielscher A, Van Leemput K, Fischl B, Dalca AV, Iglesias JE. SynthSeg:

domain randomisation for segmentation of brain scans of any contrast and resolution. 2021 Jul.

88. Hoopes A, Iglesias JE, Fischl B, Greve D, Dalca AV. TopoFit: Rapid Reconstruction of Topologically-

Correct Cortical Surfaces. 2021.