== Ten Tusscher Propagation in a 3072 Element Biventricular Mesh ==


=== Description ===
 * Solve the monodomain equation on a 3072 element biventricular mesh with extraordinary nodes using the ten Tusscher 2003 ionic model.
 * Submit a job to a cluster using SGE
 * Leverage GPU architecture to decrease ODE solve time
 * Render voltage solution with time
 * Render activation times and display isochromes

=== Submit an EP Simulation to the ROCCE Cluster ===
==== Gather Files on ROCCE ====
 * Log onto your guest account on the Rocce cluster
 * Copy files to your local continuity directory

{{{
[guest@login-0-0 ~]$ cd continuity/
[guest@login-0-0 continuity]$ cp /share/apps/si2012/cardiac/EP_TenTusscher_Panfilov_Epi_sympy_GPU.zip EP_TenTusscher_Panfilov_Epi_sympy_GPU.zip
[guest@login-0-0 continuity]$ cp /share/apps/si2012/cardiac/SubmitSerial.qsub SubmitSerial.qsub
[guest@login-0-0 continuity]$ cp /share/apps/si2012/cardiac/EP_BiV3072.py EP_BiV3072.py
}}}

 * Move the CUDA ionic model binaries to the appropriate location and unzip them

{{{
[guest@login-0-0 continuity]$ unzip EP_TenTusscher_Panfilov_Epi_sympy_GPU.zip
[guest@login-0-0 continuity]$ mv  EP_TenTusscher_Panfilov_Epi_sympy_GPU/ pcty/server/problem/Electrophysiology/
}}}

==== Inspect the Run and Submit Scripts ====
 * Open the file '''EP_Bi``V3072.py'''. This file contains the commands that ''Continuity'' will execute.

{{{
[guest@login-0-0 continuity]$ vi EP_BiV3072.py
}}}{{{#!python 
#************************************************************
# NBCR Summer Institute
# Category: EP
# Date: 8/1/2012
# Description: Solve the monodomain equation on a BiV mesh with
#              extrodinary nodes and the TenTusscher ionic model
# 
#************************************************************

import os
import sys
import numpy

#**************define parameters for the EP solve step**************

#Output file name
fName = 'EP_BiV3072'

#Use GPU acceleration for ionic model
CUDA = 1

#Set simulation time and step size
tstart = 0.0
duration = 60
stepsize = 0.05

#select to save voltage renderings at given intervals
#since rendering is very memory intensive, we will render
#the voltage every 10 steps or 0.5ms
renderfile = 1
rendercount = 10

#**************end define parameters for the EP solve step**************

#Load the model from the database, send, and calculate mesh
self.Load_File({'model_id':'1173', 'username':'guest', 'password':'guest', 'version':'1'}, log = 0)
self.Send(None, log=0)
self.CalcMesh([('Calculate', None), ('Do not Calculate', None), ('Do not Calculate', None), ('Angle change scale factors (for nodal derivs wrt angle change)', None)], log=0)

#Perform Simulations

self.SinitElectrophys(log=0)
self.Send(None, log=0)
self.SintElectrophys({'plicitType':'Implicit','parallelLinearSolver':0,'conductivityBasis':3, \
        'solutions':{'writeFile': renderfile, 'counter': rendercount, 'tableResult': 0, 'renderResult': 0},  \
        'stateVarInputSelections':[],'stateVarDoTable':0,'parallelODESolver':0,'tstart':tstart,'useCuda':CUDA,  \
        'stateVarOutputSelections':[],'serverKeyname':'electromech_exchange','stateVarList':'1','fileName':fName,  \
        'aps':{'writeFile': 1, 'counter': 1, 'tableResult': 0, 'node_list': 'all', 'renderResult': 0}, \
        'stateVarListType':'collocation points','useGalerkinAssembly':True,'stateVarFrequency':1,'stateVarSelections':[], \
        'dtout':stepsize,'tlen':duration,'reassemble_lhs':1,  \
        'ecgs':{'getHeartVector': False, 'writeFile': 0, 'counter': 1, 'tableResult': 0, 'renderResult': 0}}, log=0)
}}}

 * No changes are needed to this file
 * Typing ''':q''' will exit from the file
 * Open the submit script

{{{
[guest@login-0-0 continuity]$ vi SubmitSerial.qsub
}}}{{{#!python 
#!/bin/sh
# 
# EXAMPLE OPEN MPI SCRIPT FOR SGE
# Modified by Basement Supercomputing 1/2/2006 DJE
# Modified by cmrg 19/June/2008 FVL

# Your job name 
#$ -N EP_BiV_run1

# Use Verbos
#$ -V

# Use current working directory
#$ -cwd

# Join stdout and stderr
#$ -j y

# Use our GPU queue, which uses GPU and CPU nodes
# -q gpu@compute-1-5.local
#$ -q gpu

# To use CUDA nodes only
#$ -l cuda

# Set your number of processors here. 
# Requests mpich environment although we actually are using openmpi
#$ -pe orte 1

# Run job through bash shell
#$ -S /bin/bash

# Export Library path
export LD_LIBRARY_PATH=/opt/cuda/lib64:$LD_LIBRARY_PATH
export LD_LIBRARY_PATH=/opt/openmpi-myrinet_mx/lib:$LD_LIBRARY_PATH
export MX_RCACHE=0

# Use full pathname to make sure we are using the right mpirun
./continuity --full --no-threads --batch /home/*yourUserName*/continuity/EP_BiV3072.py

}}}

 * Change the job name (line 8) or python file path (line 39) as needed
 * Typing ''':i''' allows you to edit files in vim

==== Submit EP Job to ROCCE ====
 * Submit the job using the '''qsub''' command

{{{
[guest@login-0-0 continuity]$ qsub SubmitSerial.qsub
}}}

 * Check the job status in the queue using '''qstat''' or monitor the Continuity stdout using '''tail -f jobname.o#'''

=== Inspect the Cont6 File ===
==== Load from Repository ====
 *'''File→Library→Search'''
 *Right-click ''''BiV_3072'''' and left-click '''Load'''
 *When the dialog box pops up asking you save your current session, pick the middle radio button, '''Reset (without save) and proceed'''

==== Inspect Mesh ====
 * [:Continuity/Documentation/Help/MeshEditNodes:Mesh→Edit→Nodes...]
 * [:Continuity/Documentation/Help/MeshEditElements:Mesh→Edit→Elements...]

 * [:Continuity/Documentation/Help/MeshCalculateMesh:Mesh→Calculate Mesh...]
  * Under '''Calculate Global Faces''', select '''Don't Calculate'''. This step is unnecessary for EP. Leave the other options unchanged.
  * Click '''OK''' to close the `Calculate Mesh Form` and execute mesh calculations on the server

 * [:Continuity/Documentation/Help/MeshRenderElements:Mesh→Render→Elements...]
  * Click the '''lines''' radio button
  * Click '''Render''' to display mesh lines
 * [:Continuity/Documentation/Help/MeshRenderElements:Mesh→Render→Elements...]
  * Click the '''surfaces''' radio button
  * Enter '''1.0''' in the dialog box `Xi Location`. Leave the Xi-direction '''3''' unchanged. This will render the epicardial surface.
  * Click '''Render''' to display mesh lines

==== Inspect Electrophsyiology ====
 * Electrophysiology→Edit→Ionic Model
 * Electrophysiology→Edit→Conductivity Model

=== Render Voltage Solution ===
 * Continuity will have saved the EP solution on the cluster, which you can transfer to your computer from an SCP client
  * Find the directory. You should be able to get there by typing
{{{
[guest@login-0-0 continuity]$ cd $HOME/.continuity/working
}}}
  * If you are running Mac or Linux, you can SCP to your current directory from the terminal:
{{{
[guest@login-0-0 continuity]$ scp -r *yourUserName*@rocce.ucsd.edu:/home/*yourUserName*/.continuity/working/Vsoln_BiV_3072_50.pickle .
}}}
  * The last dot in the command above copies the file to your current directory.
  * If you are running windows, you will need an SCP client to transfer the file to your computer.
  * The '50' at the end of the file name indicates that only the first 50 frames are rendered. If you are rendering on Windows especially, you might want to render only 50 frames at a time due to memory limitations.

 * '''Electrophysiology→Solve→Initialize'''
 * '''Electrophysiology→Calculate→Boundary Conditions'''
 * '''File→Send'''
 * '''Electrophysiology→Solve→Load EP Solution'''
 * When the "Load VSoln Pickle" dialog box pops up, click "Browse", and find your pickle file '''Vsoln_BiV_3072_50.pickle'''
  * Press '''OK'''
 * '''Electrophysiology→Render→Render Solution'''
  * You should be prompted to choose a surface from your OpenMesh listing of rendered objects. You should have one object that has the name 'surfaces' in it, with a number.
  * Leave the frames to be rendered to the default (all will be selected).
  * For the ten Tusscher model, good limits for the color map are '''-90''' for '''Min Value''' and '''10''' for '''Max Value'''

=== Render Activation Map ===

 * Activation times can be calculated from the AP_out file. The activation times for this simulation have been stored in field variable 1 in the nodes form.
 * Before rendering the activation times, delete the voltage rendering using the "Open Mesh Controls"
 * '''Mesh→Render→Field...'''
 * When the "Continuity Render Fields Form" dialog box pops up, select the following options
  * '''Field Variable 3'''
  * '''Color range 0.0 to 60.0'''
  * '''At Xi3 location 1.0'''
 * Continuity can add isochromes from within the "Open Mesh Controls" to make the activation map more clear
 * In "Open Mesh Controls" select the object corresponding to the activation map (i.e. '''Textured Field2''')
 * Select the '''Colors''' tab
 * In the '''Contours''' area change the '''Min,Max,Step''' to '''0.0,60.0,5.0'''
 * Increase the '''Line Width''' to '''5'''
 * Select the square '''Draw''' button
 * Rendering the contours may take a few minutes

attachment:BiV3072_LVFW_isochrones.png