Ten Tusscher Propagation in a 3072 Element Biventricular Mesh
Description
- Solve the monodomain equation on a 3072 element biventricular mesh with extraordinary nodes using the ten Tusscher 2003 ionic model.
- Submit a job to a cluster using SGE
- Leverage GPU architecture to decrease ODE solve time
- Render voltage solution with time
- Render activation times and display isochromes
Submit an EP Simulation to the ROCCE Cluster
Gather Files on ROCCE
- Log onto your guest account on the Rocce cluster
- Copy files to your local continuity directory
[guest@login-0-0 ~]$ cd continuity/ [guest@login-0-0 continuity]$ cp /share/apps/si2012/cardiac/EP_TenTusscher_Panfilov_Epi_sympy_GPU.zip EP_TenTusscher_Panfilov_Epi_sympy_GPU.zip [guest@login-0-0 continuity]$ cp /share/apps/si2012/cardiac/SubmitSerial.qsub SubmitSerial.qsub [guest@login-0-0 continuity]$ cp /share/apps/si2012/cardiac/EP_BiV3072.py EP_BiV3072.py
- Move the CUDA ionic model binaries to the appropriate location and unzip them
[guest@login-0-0 continuity]$ unzip EP_TenTusscher_Panfilov_Epi_sympy_GPU.zip [guest@login-0-0 continuity]$ mv EP_TenTusscher_Panfilov_Epi_sympy_GPU/ pcty/server/problem/Electrophysiology/
Inspect the Run and Submit Scripts
Open the file EP_BiV3072.py. This file contains the commands that Continuity will execute.
[guest@login-0-0 continuity]$ vi EP_BiV3072.py
1 #************************************************************
2 # NBCR Summer Institute
3 # Category: EP
4 # Date: 8/1/2012
5 # Description: Solve the monodomain equation on a BiV mesh with
6 # extrodinary nodes and the TenTusscher ionic model
7 #
8 #************************************************************
9
10 import os
11 import sys
12 import numpy
13
14 #**************define parameters for the EP solve step**************
15
16 #Output file name
17 fName = 'EP_BiV3072'
18
19 #Use GPU acceleration for ionic model
20 CUDA = 1
21
22 #Set simulation time and step size
23 tstart = 0.0
24 duration = 60
25 stepsize = 0.05
26
27 #select to save voltage renderings at given intervals
28 #since rendering is very memory intensive, we will render
29 #the voltage every 10 steps or 0.5ms
30 renderfile = 1
31 rendercount = 10
32
33 #**************end define parameters for the EP solve step**************
34
35 #Load the model from the database, send, and calculate mesh
36 self.Load_File({'model_id':'1173', 'username':'guest', 'password':'guest', 'version':'1'}, log = 0)
37 self.Send(None, log=0)
38 self.CalcMesh([('Calculate', None), ('Do not Calculate', None), ('Do not Calculate', None), ('Angle change scale factors (for nodal derivs wrt angle change)', None)], log=0)
39
40 #Perform Simulations
41
42 self.SinitElectrophys(log=0)
43 self.Send(None, log=0)
44 self.SintElectrophys({'plicitType':'Implicit','parallelLinearSolver':0,'conductivityBasis':3, \
'solutions':{'writeFile': renderfile, 'counter': rendercount, 'tableResult': 0, 'renderResult': 0}, \
'stateVarInputSelections':[],'stateVarDoTable':0,'parallelODESolver':0,'tstart':tstart,'useCuda':CUDA, \
'stateVarOutputSelections':[],'serverKeyname':'electromech_exchange','stateVarList':'1','fileName':fName, \
'aps':{'writeFile': 1, 'counter': 1, 'tableResult': 0, 'node_list': 'all', 'renderResult': 0}, \
'stateVarListType':'collocation points','useGalerkinAssembly':True,'stateVarFrequency':1,'stateVarSelections':[], \
'dtout':stepsize,'tlen':duration,'reassemble_lhs':1, \
'ecgs':{'getHeartVector': False, 'writeFile': 0, 'counter': 1, 'tableResult': 0, 'renderResult': 0}}, log=0)
- No changes are needed to this file
Typing :q will exit from the file
- Open the submit script
[guest@login-0-0 continuity]$ vi SubmitSerial.qsub
1 #!/bin/sh
2 #
3 # EXAMPLE OPEN MPI SCRIPT FOR SGE
4 # Modified by Basement Supercomputing 1/2/2006 DJE
5 # Modified by cmrg 19/June/2008 FVL
6
7 # Your job name
8 #$ -N EP_BiV_run1
9
10 # Use Verbos
11 #$ -V
12
13 # Use current working directory
14 #$ -cwd
15
16 # Join stdout and stderr
17 #$ -j y
18
19 # Use our GPU queue, which uses GPU and CPU nodes
20 # -q gpu@compute-1-5.local
21 #$ -q gpu
22
23 # To use CUDA nodes only
24 #$ -l cuda
25
26 # Set your number of processors here.
27 # Requests mpich environment although we actually are using openmpi
28 #$ -pe orte 1
29
30 # Run job through bash shell
31 #$ -S /bin/bash
32
33 # Export Library path
34 export LD_LIBRARY_PATH=/opt/cuda/lib64:$LD_LIBRARY_PATH
35 export LD_LIBRARY_PATH=/opt/openmpi-myrinet_mx/lib:$LD_LIBRARY_PATH
36 export MX_RCACHE=0
37
38 # Use full pathname to make sure we are using the right mpirun
39 ./continuity --full --no-threads --batch /home/*yourUserName*/continuity/EP_BiV3072.py
- Change the job name (line 8) or python file path (line 39) as needed
Typing :i allows you to edit files in vim
Submit EP Job to ROCCE
Submit the job using the qsub command
[guest@login-0-0 continuity]$ qsub SubmitSerial.qsub
Check the job status in the queue using qstat or monitor the Continuity stdout using tail -f jobname.o#
Inspect the Cont6 File
Load from Repository
File→Library→Search
Right-click 'BiV_3072' and left-click Load
When the dialog box pops up asking you save your current session, pick the middle radio button, Reset (without save) and proceed
Inspect Mesh
Under Calculate Global Faces, select Don't Calculate. This step is unnecessary for EP. Leave the other options unchanged.
Click OK to close the Calculate Mesh Form and execute mesh calculations on the server
Click the lines radio button
Click Render to display mesh lines
Click the surfaces radio button
Enter 1.0 in the dialog box Xi Location. Leave the Xi-direction 3 unchanged. This will render the epicardial surface.
Click Render to display mesh lines
Inspect Electrophsyiology
- Electrophysiology→Edit→Ionic Model
- Electrophysiology→Edit→Conductivity Model
Render Voltage Solution
- Continuity will have saved the EP solution on the cluster, which you can transfer to your computer from an SCP client
- Find the directory. You should be able to get there by typing
[guest@login-0-0 continuity]$ cd $HOME/.continuity/working
- If you are running Mac or Linux, you can SCP to your current directory from the terminal:
[guest@login-0-0 continuity]$ scp -r *yourUserName*@rocce.ucsd.edu:/home/*yourUserName*/.continuity/working/Vsoln_BiV_3072_50.pickle .
- The last dot in the command above copies the file to your current directory.
- If you are running windows, you will need an SCP client to transfer the file to your computer.
- The '50' at the end of the file name indicates that only the first 50 frames are rendered. If you are rendering on Windows especially, you might want to render only 50 frames at a time due to memory limitations.
Electrophysiology→Solve→Initialize
Electrophysiology→Calculate→Boundary Conditions
File→Send
Electrophysiology→Solve→Load EP Solution
When the "Load VSoln Pickle" dialog box pops up, click "Browse", and find your pickle file Vsoln_BiV_3072_50.pickle
Press OK
Electrophysiology→Render→Render Solution
You should be prompted to choose a surface from your OpenMesh listing of rendered objects. You should have one object that has the name 'surfaces' in it, with a number.
- Leave the frames to be rendered to the default (all will be selected).
For the ten Tusscher model, good limits for the color map are -90 for Min Value and 10 for Max Value
Render Activation Map
- Activation times can be calculated from the AP_out file. The activation times for this simulation have been stored in field variable 1 in the nodes form.
- Before rendering the activation times, delete the voltage rendering using the "Open Mesh Controls"
Mesh→Render→Field...
- When the "Continuity Render Fields Form" dialog box pops up, select the following options
Field Variable 3
Color range 0.0 to 60.0
At Xi3 location 1.0
- Continuity can add isochromes from within the "Open Mesh Controls" to make the activation map more clear
In "Open Mesh Controls" select the object corresponding to the activation map (i.e. Textured Field2)
Select the Colors tab
In the Contours area change the Min,Max,Step to 0.0,60.0,5.0
Increase the Line Width to 5
Select the square Draw button
- Rendering the contours may take a few minutes
