Encoding-and-Decoding-of-Visual-Stimulus-from-Human-Brain-Activity-an-fMRI-study

Overview

Today, functional connectivity analyzes based on Functional Magnetic Resonance Imaging (fMRI) data and detecting neuronal activity in the brain have gained an important place in brain research. The physical basis of fMRI depends to blood-oxygen-level-dependent (BOLD) contrast which derived from a mismatch of blood flow and oxygen metabolism during local neural activation. Studies of how the brain represents complex visual objects and the perception experienced by the subject are extremely important in terms of understanding the working of the brain.

In this repository, there are studies on visual stimulus reconstruction from fMRI activation models, which are revealed by the presentation of visual stimuli in various object categories. In terms of applicability of statistical learning approaches involving neuroscience, the BOLD5000 dataset containing approximately 5,000 different slow event-related fMRI human functional MRI (fMRI) images will be used. As data preprocessing steps, it is aimed to increase the classification accuracy by removing the visual cortex, which is known to contain information about the distinctions to be made, from the images. It is aimed to test the performance of deep learning-based algorithms with 2 different data types: The first is to extract the cortical surface model for each subject, in which the stimulus image hides the neighborhood relations, using Freesurfer software. Secondly, the performance of the algorithm in creating the stimulus image using raw images will be analyzed. The generative adversarial network (GAN) architecture, which is often used in the machine learning framework, will be used to reconstruct the stimulus image from the pre-processed and raw images.

Preprocessing Steps

Dicom Mosaics to 3D NIfTI

python3 mri_convert.py

Recon-all

Recon-all provides pre-processing of the three-dimensional anatomical volume of a single subject. Check out the Freesurfer tutorial on the usage and outputs of the recon-all command.

$ mri_convert T1w_MPRAGE_CSI1.nii T1w_MPRAGE_CSI1.nii.gz
$ recon-all -i T1w_MPRAGE_CSI1.nii.gz -s CSI1 -all

Occipital Patch

TkSurfer tool in Freesurfer toolset was used to create occipital patches. TkSurfer allows for visualization and navigation through cortical surface data. TkSurfer can also display functional or curvature data on the surface. Take a look TkSurfer.

Broddmann V1 and V2 regions created during the cortical reconstruction process completed with recon-all were used in the occipital patch extraction process. Take a look Broddman Areas.

$ tksurfer CSI1 lh inflated -gray
$ tksurfer CSI1 rh inflated -gray

Occipital Flattened Surface

handenur@handenur-VirtualBox:/usr/local/freesurfer/subjects/CSI1/surf$ mris_flatten -w 0 lh.occip.patch.mgh lh.occip.flat.mgh
handenur@handenur-VirtualBox:/usr/local/freesurfer/subjects/CSI1/surf$ mris_flatten -w 0 rh.occip.patch.mgh rh.occip.flat.mgh

Save FMRI Activation Map

Flat surfaces were filled with activation values using linear interpolation method. Check out LinearNDInterpolator function in Scipy.

python3 flatmap.py

Concatenate Left Hem. and Right Hem.

python3 lh_rh_concat.py

References

1. Generative adversarial networks for reconstructing natural images from brain activity
2. BOLD5000, a public fMRI dataset while viewing 5000 visual images
3. End-to-End Image Reconstruction of Image from Human Functional Magnetic Resonance Imaging Based on the "Language" of Visual Cortex
4. Functional magnetic resonance imaging (fMRI)"brain reading": detecting and classifying distributed patterns of fMRI activity in human visual cortex
5. fMRI Brain Reading: detecting and classifying distributed patterns of fMRI activity in human visual cortex