www.eu-eela.eu e-science grid facility for europe and latin america grip - grid image processing for...
TRANSCRIPT
www.eu-eela.eu
E-science grid facility forEurope and Latin America
GRIP - Grid Image Processing for Biomedical Diagnosis
SECOND EELA-2 GRID SCHOOLQuerétaro, México.
Leandro Ciuffo (INFN - Italia)Tutor
Alejandro Llanos Fuentes (UFRO – Chile)
Diego Vicencio Morales (UTFSM – Chile)
www.eu-eela.eu Querétaro, Second EELA-2 Grid School, 09-10-2009 2
Aplicación
Resultados
71% de cancerígenos
Búsqueda de candidatos a núcleos
Eliminación de fondo
Lectura de la imagen
www.eu-eela.eu Querétaro, Second EELA-2 Grid School, 09-10-2009 3
Agenda
• Objetivos• Gridificación• OurGrid• Conclusión• Trabajos Futuros
www.eu-eela.eu Querétaro, Second EELA-2 Grid School, 09-10-2009 4
Objetivos
• Implementar paralelización• Explorar posibilidades de manejo y
almacenamiento grid
www.eu-eela.eu Querétaro, Second EELA-2 Grid School, 09-10-2009 5
Gridificación
• Múltiples Jobs:[ Type = "collection";
InputSandbox = {"carcinoma.o"};
nodes = {
[
JobType = "Normal";
NodeName = "node1";
Executable = "/bin/sh";
Arguments = "ejecucion1.sh";
InputSandbox = {"ejecucion1.sh",root.InputSandbox[0], "image6.png"};
StdOutput = "myoutput1.out";
StdError = "myerror1.err";
OutputSandbox = {"myoutput1.out","myerror1.err"};
ShallowRetryCount = 3;
], [
JobType = "Normal";
NodeName = "node2";
(...)
Múltiples imágenes
www.eu-eela.eu Querétaro, Second EELA-2 Grid School, 09-10-2009 6
Gridificación
• Múltiples Jobs:[ Type = "dag";
InputSandbox = {"carcinoma.o","cotejador.o","inicializador.o"};
nodes =[
father =[
description =[
JobType = "Normal";
Executable = "/bin/sh";
Arguments = "padre.sh";
InputSandbox = {"padre.sh",root.InputSandbox[2]};
StdOutput = "father_output";
StdError = "father_error";
OutputSandbox = {"father_output","father_error","1parametros.txt","2parametros.txt","3parametros.txt"};
ShallowRetryCount = 1;];];
son1 =
(...)
dependencies = { {father,{son1,son2,son3}} , {son1,final}, {son2,final} {son3,final} };
Dependencias
Nodo de trabajo
www.eu-eela.eu Querétaro, Second EELA-2 Grid School, 09-10-2009 7
Gridificación
UIUI WMSWMS
CE
CE
CE
WNWN
WNWN
WNWN
AMGA
SE
init.sh
carcinoma.sh
${date}_car_${PARAM}.jdl
www.eu-eela.eu Querétaro, Second EELA-2 Grid School, 09-10-2009 8
Gridificación
• init.sh (Múltiples Jobs)
#!/bin/sh
Dir=`pwd`/inputfile
NOW=$(date +%Y-%m-%d-%H_%M_%S)
echo $NOW > $Dir/temporalFileName
rm -rf `pwd`/JDLParameters/*
for (( i=7 ; i<17 ; i=i+1 ))
do
let entero=$i/10
let decimal=$i%10
numero="${entero}.${decimal}"
echo $numero
sed 's(VAR('$numero'(g
s(DIR('$Dir'(g' job-carcinoma.jdl > ./JDLParameters/${NOW}_PAR_${numero}.jdl
done
glite-wms-job-submit -d alejandro -o carId2 --collection ./JDLParameters
Editar Jobs
www.eu-eela.eu Querétaro, Second EELA-2 Grid School, 09-10-2009 9
Gridificación
• carcinoma.jdl
[
Type = "Job";
JobType = "Normal";
Executable = "carcinomaSH.sh";
StdOutput = "carcinoma.out";
StdError = "carcinoma.err";
Requirements = other.GlueCEInfoHostName=="kuragua.uniandes.edu.co" || other.GlueCEInfoHostName=="ce.labmc.inf.utfsm.cl" || other.GlueCEInfoHostName=="ce01.eela.if.ufrj.br" || other.GlueCEInfoHostName=="ce01.macc.unican.es";
InputSandbox = {"DIR/temporalFileName","DIR/mdclient.config","DIR/carcinomaSH.sh","DIR/carcinoma","DIR/image.png"};
OutputSandbox = {"carcinoma.err","carcinoma.out"};
Arguments = "VAR image.png";
ShallowRetryCount = 3;
]
www.eu-eela.eu Querétaro, Second EELA-2 Grid School, 09-10-2009 10
Gridificación
• carcinoma.sh (WN)
#!/bin/bash
export LFC_HOST=lfc.eela.ufrj.br
export LCG_GFAL_INFOSYS=bdii.eela.ufrj.br:2170
chmod +x carcinoma
export LD_LIBRARY_PATH=$VO_PROD_VO_EU_EELA_EU_SW_DIR/librerias/:$LD_LIBRARY_PATH
NOW=`cat temporalFileName`
url=$(hostname)
./carcinoma $1 $2
lcg-cr --vo $PROD -l lfn:/grid/$PROD/carcinoma/${NOW}_PAR_$1 -d lnx105.eela.if.ufrj.br file://$PWD/carcinoma.out
mdcli addentry /schooldir/grip/${NOW}_PAR_$1 imageFile $2 parameter $1 hostname $url
mdcli setattr /schooldir/grip/${NOW}_PAR_$1 outputlfn lfn:/grid/prod.vo.eu-eela.eu/carcinoma/${NOW}_PAR_$1
AMGA
SE
Directorio compartido: CE
www.eu-eela.eu Querétaro, Second EELA-2 Grid School, 09-10-2009 11
Gridificación
• AMGA
Entries imageFile (varchar 255)
paramenter (float)
hostname (varchar 255)
outputlfn (varchar 255)
2009-10-08-18_57_22_PAR_1.2 Image.png 1.2 ce01.eela…. lfn:/grid/pr…2009-10-08-18_57_22_PAR_1.0 Image.png 1.0 kuragua.un... lfn:/grid/pr..
www.eu-eela.eu Querétaro, Second EELA-2 Grid School, 09-10-2009 12
OurGrid
job:
label: carcinomaJob
requirements: (os=linux)
task:
init:
put carcinoma carcinoma
put image6.png image6.png
put CarcinomaSH.sh CarcinomaSH.sh
put lib.tar lib.tar
remote: sh CarcinomaSH.sh 0.11 image6.png
final: get salida.txt salida1.txt
task:
init:
put carcinoma carcinoma
put fragmento_re1.png fragmento_re1.png
put CarcinomaSH.sh CarcinomaSH.sh
put lib.tar lib.tar
remote: sh CarcinomaSH.sh 0.11 fragmento_re1.png
final: get salida.txt salida2.txt
71.79 porciento son cancerígenos
96.29 porciento son cancerígenos
www.eu-eela.eu Querétaro, Second EELA-2 Grid School, 09-10-2009 13
Conclusión
• Paralelización:– Implentenación de diferentes instancias
• Utilización de componentes grid:– SE– AMGA– LFC
• Middleware:– OurGrid
www.eu-eela.eu Querétaro, Second EELA-2 Grid School, 09-10-2009 14
Trabajos Futuros
• Aplicación WEB (GENIUS)• SecureStorage• Middleware DIRAC• Verificación de paralelización MPI
www.eu-eela.eu Querétaro, Second EELA-2 Grid School, 09-10-2009 15
¿Preguntas?