Large-scale neural network modeling using Giraph

We use Giraph to model large-scale neural network on the Hadoop cluster deployed at the University of St. Thomas by Department of Graduate Programs in Software.

Note: The code has been fully tested on CDH4.4, but it is not working on CDH5 yet.

Software Packages needed:

• Maven 3
• CDH4
• Giraph 1.0.0
• Java 1.6
• Python 2.7 or higher
• R

Generate XML configuration files

python xml_templates_parser.py

A directory called 'xml_input4Hadoop' will be generated under the project home.

Compile & upload to Hadoop cluster

Modify upload.sh first with appropriate user name and Hadoop cluster name.

mvn install

This command will compile the project and upload necessary files to Hadoop cluster in the directory giraph under $HOME

Neural network generation with MapReduce

ssh into your Hadoop cluster:

cd giraph

chmod +x *.sh

./input.sh

The neural network will be generated in neuron_input on HDFS.

Run Giraph job for modeling

Delete _logs files generated at the previous step, otherwise Giraph will take it as input and an error would occur.

hadoop fs -rm -r neuron_input/_logs

Run Giraph job

./run.sh

Post analysis

Data analysis

• post-analysis/hive/create_table.hql: creating hive table from Giraph's output
• post-analysis/hive/firing_pattern.hql: extracting fired neurons
• post-analysis/hive/firing_rates.hql: extracting firing rates

Plotting

• post-analysis/plot/graphs.R: plotting firing patterns
• post-analysis/plot/channel_plot.R: plotting firing patterns for multiple channels