faultsight: a fault analysis tool for hpc researchers
TRANSCRIPT
FaultSight: A Fault Analysis Tool for HPC ResearchersEinar Horn
Department of LinguisticsUniversity of WashingtonSeattle, Washington, USA
Dakota FulpJon C. Calhoun
Holcombe Department of Electricaland Computer Engineering
Clemson UniversityClemson, South Carolina, USA{dakotaf,jonccal}@clemson.edu
Luke N. OlsonDepartment of Computer Science
University of Illinois atUrbana-ChampaignUrbana, Illinois, [email protected]
ABSTRACTSystem reliability is expected to be a significant challenge for futureextreme-scale systems. Poor reliability results in a higher frequencyof interruptions in high-performance computer (HPC) applicationsdue to system/application crashes or data corruption due to softerrors. In response, application level error detection and recoveryschemes are devised to mitigate the impact of these interruptions.Evaluating these schemes and the reliability of an application re-quires the analysis of thousands of fault injection trials, resultingin tedious and time-consuming process. Furthermore, there is noone data analysis tool that can work with all of the fault injectionframeworks currently in use. In this paper, we present FaultSight,a fault injection analysis tool capable of efficiently assisting in theanalysis of HPC application reliability as well as the effectivenessof resiliency schemes. FaultSight is designed to be flexible and workwith data coming from a variety of fault injection frameworks. Theeffectiveness of FaultSight is demonstrated by exploring the reliabil-ity of different versions of the Matrix-Matrix Multiplication kernelusing two different fault injection tools. In addition, the detectionand recovery schemes are highlighted for the HPCCG mini-app.
KEYWORDSsoft error analysis, fault analysis tool, fault tolerance, resiliency,fault injection, fault analysis
ACM Reference Format:Einar Horn, Dakota Fulp, Jon C. Calhoun, and Luke N. Olson. 2019. Fault-Sight: A Fault Analysis Tool for HPCResearchers. In Proceedings of FTXS’19:FaultTolerance for HPC at eXtreme Scale (FTXS) Workshop (FTXS’19). ACM, NewYork, NY, USA, 10 pages. https://doi.org/10.475/123_4
1 INTRODUCTIONHigh-performance computing (HPC) systems are crucial to scien-tific simulation and analysis. As HPC resources continue to playa role in scientific discoveries, simulations are increasing in sizeand complexity. Equally, in order to meet these workload demands,HPC systems continue to grow in size and computational power. As
Permission to make digital or hard copies of all or part of this work for personal orclassroom use is granted without fee provided that copies are not made or distributedfor profit or commercial advantage and that copies bear this notice and the full citationon the first page. Copyrights for components of this work owned by others than ACMmust be honored. Abstracting with credit is permitted. To copy otherwise, or republish,to post on servers or to redistribute to lists, requires prior specific permission and/or afee. Request permissions from [email protected]’19, November 22, 2019, Denver, CO© 2019 Association for Computing Machinery.ACM ISBN 123-4567-24-567/08/06. . . $15.00https://doi.org/10.475/123_4
HPC systems become larger and more powerful, design concernssuch as power usage and reliability may inhibit the ability of nextgeneration HPC systems [7, 35, 38].
Reliability is a significant challenge for HPC systems and appli-cations. To overcome fail-stop failures, checkpoint restart routinesare integrated into most applications. Checkpoint restart allowsapplications to save a portion of the computational state and use itto recover from this saved state when a failure occurs [5]. However,as processor feature size continues to shrink and power constraintsforce hardware to operate at near-threshold voltages, the occur-rence of soft errors increases [2, 9, 27]. Soft errors occur fromcharged particles or hardware degradation thus causing bit-flipsin data, computation, and control logic. If left undetected, soft er-rors can corrupt application state, leading to silent data corruption(SDC) that may alter the results of an application. Additionally, thetime-to-solution may also increase even in situations where data isleft uncorrupted.
Full protection of HPC systems from soft errors at the hardwarelevel is prohibitively expensive. Therefore, protection is mainlyconfined to the memory system in the form of parity bits and er-ror correcting codes (ECC) on data paths and storage. Generalprotection from soft errors at the application level relies on replica-tion [8, 20, 28], data analysis [3, 4], or leveraging algorithm modifi-cations [12, 16, 25]. Recovery schemes rely on checkpoint restart [4,12], replication [20], or forward recovery techniques [8, 26].
Measuring an application’s reliability to soft errors requires con-ducting large numbers of fault injection trials. Efficient evaluationof these trials is difficult, due to the lack of analysis tools for faultinjection campaigns. This makes it difficult to determine the vul-nerable code regions. Soft error detection and recovery schemesalso require thousands of fault injection trials, and likewise do nothave any common tools to aid in their analysis.
This paper presents FaultSight, a fault analysis tool designedto aid the development and analysis of application-level soft errordetection and recovery schemes. FaultSight provides an API thatallows data from a campaign of fault injection trials to be insertedinto an extendable database. The database is queried by an accompa-nying web-app that couples dynamic queries and visualizations toobtain a fine-grained view of the fault injection campaign. Qualita-tive and statistical analysis are then used to determine the baselinereliability of an application and the effectiveness of integrated de-tection and recovery schemes.
This paper makes the following contributions:
FTXS’19, November 22, 2019, Denver, CO E. Horn et al.
• an overview of the FaultSight package, a user extendablefault analysis tool that allows for efficient analysis of faultinjection data for HPC applications.• examples of fault injection analysis using FaultSight, includ-ing bit-flips in computation and how to use FaultSight withbit-flips in memory.• an analysis of the resiliency of three Matrix-Matrix Multipli-cation algorithms through use and comparison of FlipIt [11]and FSEFI [23]. As well as an analysis of detection and re-covery scheme for the mini-app HPCCG.
2 BACKGROUND AND MOTIVATIONDetermining the reliability of an application and evaluating theimprovement in reliability due to a detection or recovery schemerequires thousands of fault injection experiments. Complex HPCapplications result in millions of possible fault locations. To obtainstatistically significant results, thousands of fault injection trialsare conducted [29]. This process can be seen in the left-hand sideof Figure 1. Even through parallel execution, the thousands of faultinjection trials use significant computational resources and storage.
Upon completion of these trials, aggregation and analysis ofeach trial can be achieved. Some metrics of interest include: thereturn code of the application, the acceptability of the application’sresults, the impact on application runtime, the detection of injectedfaults, and the recovery from detected faults. These metrics areanalyzed generically for the whole application. However, this hasa limited view of when and where a fault is injected and how thatimpacts these metrics. Deeper analysis of when and where a fault isinjected is complicated by the lack of fault analysis tools. This paperaddresses the lack of such tools by presenting FaultSight, a tooldesigned to conduct detailed analysis of fault injection campaigns.
FaultSight is capable of exploring the importance of when andwhere faults occur inside of a fault injection campaign and the asso-ciated impact on key application metrics. FaultSight’s contributionsto the fault injection and analysis workflow are shown in Figure 1.FaultSight is structured as a general tool capable of incorporatingdata from many different fault injection frameworks and providesa web application to efficiently analyze the whole campaign or aspecific subset of the trials. Using FaultSight to analyze a fault in-jection campaign allows application developers/users to determinethe vulnerable regions of their application and quantify the efficacyof an integrated soft error detection and recovery scheme.
3 FAULTSIGHT DESIGNFaultSight is a tool that enables the analysis of HPC resiliency inapplications by quantifying an applications reliability, identifyingwhere to add protectionmechanisms, and assisting in the evaluationof application-based detection and recovery schemes. FaultSightaims to integrate seamlessly with most HPC fault injectors whileproviding a comprehensive set of functionality to assist with theanalysis of high-level and fine-grained fault injection campaignresults. An interactive web application presents the analysis andallows quick observations about the reliability of the tested ap-plication and the effectiveness of integrated detection/recoveryschemes.
FaultSight operates first by parsing fault injection campaign datainto an extendable database through its built-in API calls. The largerange of API endpoints allows FaultSight to support data comingfrom most fault injectors. Upon construction, FaultSight queries thedatabase to make high-level statistical analysis and visualizations.
Overall, the structure of FaultSight involves three main compo-nents: the database, the API for database communication, and theweb interface used for analytics.
3.1 DatabaseThe database is designed in a general manner and supports a varietyof fault injector types — e.g. instruction-based, memory-based, andfail-stop injectors. Section 4 explores FaultSight’s ability to analyzeresults from two different instruction-based fault injectors. Thedatabase consists of five extendable tables: Site, Trials, Injections,Signals, and Detections. Each table is discussed below, includingtheir utility for different types of fault injectors while an exampleof database creation can be seen in Algorithm 1.
3.1.1 Sites. The Sites table allows FaultSight to support a widevariety of fault injector types. The inclusion of threadId and rankfields allows FaultSight to support multi-threaded and parallel ap-plications. For instruction-based fault injectors, the type field allowsinformation about the site’s instruction classification — e.g. floating-point, fixed-point, address calculation — to be stored. In memory-based fault injectors, the type field is instead used to denote thedata type for the memory allocation or the memory address wherethe fault occurred. In both instruction and memory based injectors,information on a site’s location in the source code can be storedvia the file, func, and line fields for for subsequent visualizationand analysis. Details on the functionality of the site are stored viathe opcode field. For instruction-based injectors this can be used todenote the injection instruction. Not all fields will be used with allfault injectors and are prepopulated with NULL values. An exampleof an insertion into this table can be seen in Algorithm 1.
3.1.2 Trials. The Trials table stores information on each of thefault injection trials. In addition to assigning a unique identifierto each trial, this table holds properties of each trial, includingthe number of injections per trial, as well as a boolean value torepresent the return code of the application. Extending this tableallows users to store specific information about the trials such aswall-clock time, power/energy, and resource utilization statisticswhich results in a more detailed view of each trial.
3.1.3 Injections. The Injections table stores information on eachinjection that occurs during the fault injection campaign and asso-ciates it with a unique trial ID. Information such as an injection’ssite, process rank, and thread identifier assists in mapping back tothe fault injection location. Additionally, instruction and memorybased injectors can store information on any corrupted bits, injec-tion probability, and the static instruction of the application duringthe injection. Through the use of this table, FaultSight can performfine-grained analysis on this data. An insertion into this table canbe seen in Algorithm 1.
3.1.4 Signals. The Signals table stores information on knownsignals that are raised during trials. In addition to the Unix signal
FaultSight: A Fault Analysis Tool for HPC Researchers FTXS’19, November 22, 2019, Denver, CO
Application+
Fault Injector
Fault Injection Trials Fault Injection Results
Database FaultSight Front-end
Visualization
Statistical Testing
Data Aggregation
API Analysis
FaultSight
Figure 1: Overview fault injection and the FaultSight tool.
number, information on which process and thread raised the signalcan be stored. This table is useful in distinguishing how a programterminates unexpectedly such as through segmentation fault vio-lations or failed assertions. Also knowing if the signals are raisedon the same process/thread where an injection occurred can helpanalyze error propagation.
3.1.5 Detections. The Detections table enables the ability tostore information on the performance of implemented algorithm-based fault detectors when testing their effectiveness. By combin-ing the process/thread that detected the error/failure with the pro-cess/thread where the injection occurred, a spatial map of the eventsoccurring inside the trial after each injection highlights how theerror propagates. If temporal information is known, the latency —e.g., number of instructions since injection, number of iterationssince injection, elapsed time since injection — provides anotherdimension to explore when addressing a detector’s effectiveness.The spatial and temporal properties of detection combined withapplication knowledge allows for estimating the extent at whicherror due to the fault propagates. Knowledge of error propagationis useful in the design of custom local recovery schemes.
3.2 APIFaultSight’s API supports data coming from various injection stylesand frameworks. The API is a collection of Python functions thatwrap low-level SQL commands. Through the API, the user experi-ence is abstracted and simplified. Thus, allowing users to generatethe database more intuitively.
Algorithm 1 Creation of a FaultSight database.Input sites_data : a list of site informationInput inject ions_data : a list of injection information
# Create a database object and get a reference to itdb ← create_and_connect_database(“./database .db”)
# Extend injections table to hold number of incorrect elementsextend_inject ions_table(db, “incorrect_count ”, “I NT EGER”)
# Insert site information into the databasefor site_inf ormation in sites_data do
inser t_site(db, site_inf ormation)
# Insert injection information into the databasefor inject ion_inf ormation in inject ions_data do
star t_tr ial (db)inser t_inject ion(db, inject ion_inf ormation)end_tr ial (db)
The API facilitates the generation of the database online duringthe fault injection process as well as offline once the campaign hasbeen completed. For the experiments in Section 4, both FlipIt [11],an LLVM instruction-based fault injection tool, and FSEFI [23], asequential fault injection tool, are used. However, other HPC faultinjection frameworks [31, 36] can be used.
During fault injection, FlipIt and FSEFI each record a varietyof data on each fault injection trial, such as the instruction andbit-location in which the fault is injected. A Python script is de-veloped to map each of their output to the appropriate API callsupon completion of each fault injection campaign. Other fault injec-tors [31, 36] provide similar information, such as memory addressesof the injection for memory-based injectors, which can also bestored in the database through the API. An example of generatinga database using the API is shown in Algorithm 1.
To analyze the fault injection trials in more detail, the API allowsusers to make database schema changes such as extending tableswith additional fields to better support their situation. Section 4exploits this capability to generate databases for the Matrix-MatrixMultiplication and HPCCG applications.
3.3 Web ApplicationFaultSight interacts with the database through a web applicationthat runs locally and uses Flask1, a flexible Python web frame-work. The back-end uses SQLAlchemy2, a SQL toolkit and object-relational mapper, to query the database. To simplify the selectionof data for visualization and analysis, FaultSight uses a dedicatedweb application over other data analytics tools. There are threemain features of the web application: Statistical Analysis, Charts,and Code Analysis.
3.3.1 Statistical Analysis. FaultSight provides the end-user withtwo forms of statistical analysis: independent analysis and compar-ative analysis.
Independent analysis allows users to determine if there havebeen enough injections on a given type or location to justify mean-ingful results. For example, to accurately evaluate the reliabilityof function foo a sufficient number of injections into foo’s faultinjection space are required. A key question for instruction-basedfault injectors is whether the dynamic/static instruction percent-ages match the percentage of instruction types that suffer faults.Similarly for memory-based injectors, the injection locations are
1http://flask.pocoo.org/2https://www.sqlalchemy.org/
FTXS’19, November 22, 2019, Denver, CO E. Horn et al.
Table 1: Subset of chart types in FaultSight.
Injections Features
Injected types Detected trialsInjection type per function Detected injection bit locationsInjected bit locations Detection latencyInjected functions/components Trials unexpectedly terminatedInjected line numbers
analyzed in comparison to the memory locations used by foo. Fault-Sight supports this analysis by incorporating profiling informationfrom traces or by using static analysis on the code or static memoryallocation based on size.
Comparative analysis allows the user to filter the trial space anddivide the resulting set of trials into two sets based on given criteria.This partitioning allows the user to conduct A/B style testing forresilience. To partition the sets, the user requests aggregated in-formation about these two trial sets such as injection information,detection occurrences, and signals. For each of these fields, theuser requests a statistical comparison between the two trial sets,to determine the p-value in the statistical difference or equality ofthese sets with full control over the confidence interval to use inthese calculations. This functionality is most useful when compar-ing resiliency algorithms, effectivness of software-based detectionand/or recovery schemes, or most other quantitative comparisonsbetween two groups of fault injection trials. Section 4.1 uses thisfeature to compare the number of elements that are incorrect inthe results of three different Matrix-Matrix Multiplication routines.
3.3.2 Charts. FaultSight has the ability to generate charts ofquantitative data at various levels of granularity. Charts utilize datafrom various combinations of the data stored within the tables ofthe database. For example, FaultSight is able to visualize where thefault injection campaign injects faults — e.g., what functions, whatcomponents, what bit-locations, what process/thread. Additionally,FaultSight can generate charts of statistics over all the fault injectiontrials such as percentage of trials terminated with a segfault andpercentage of trials with detection of the injected fault.
Through the use of user defined constraints on the data, appliedthrough a series of drop down menus, FaultSight is able to generatecustom fine grain views into the campaign. For example, a usercould filter and visualize the flip locations on process rank 4 whereinjections occurred in function foo and generated a segfault. Theuse of custom constraints when visualizing data through FaultSightprovides users with a powerful tool to gain detailed knowledgeabout each fault and how they impact detection and correctness.Section 4 explores this in more detail, as FaulSight generates allplots and figures referenced. A subset of default charts generatedby FaultSight is shown in Table 1. These plots provide a strongfoundation before custom constraints are applied, leading to finer-grained views into the campaign results. FaultSight also supportsexportation to common image formats and the raw data to JSONfiles for use in other software and frameworks.
3.3.3 Code Analysis. Code analysis displays syntax highlightedregions of code where fault injections occur. If possible, the sourcecode lines are expanded to display a line’s assembly breakdown
Figure 2: FaultSight code analysis for injections into func-tion HPC_sparsemv from HPCCG (Section 4.2). Note: Not allinjected source lines displayed.
information. This allows for more accurate information and infer-ences on injection location and fault vulnerability, such as regionswhere faults are likely or where faults cause the most harm. Fig-ure 2 shows the location and percentage of injections into thefunction HPC_sparsemv, the most commonly injected function inthe HPCCG experiments (Section 4.2), for the function itself andthe whole application. Most injections (63.52% for the function and57.22% for the application) into this function occur in the inner-mostloop (line 74) wheremost of the time is spent. Prior work, has shownthis routine is a critical function to protect for linear solvers [13, 14].FaultSights code analysis assists users in understandingwhere injec-tions are likely and where they are most serious for programmaticsymptoms — e.g., unexpected termination, output corruption. Withthis knowledge, users are able to discover key data structures inneed of protection and where to place detectors to minimize de-tection overhead and error propagation. This type of informationcan be provided to instruction/data replication techniques to helpprovide high levels of resiliency with low overheads [28].
4 EXPERIMENTAL RESULTSIn this section, FaultSight is used to analyze fault injection campaignresults on a Matrix-Matrix Multiplication (MMM) kernel from theperspective of two different fault injectors and on the High Perfor-mance Computing Conjugate Gradients (HPCCG) mini-app fromthe Mantevo Suite3.
In our experiments, two different fault injectors are used. Thefirst fault injector is the LLVM based fault injector, FlipIt [11]. Trialswith FlipIt were run on Phase 6 of the Palmetto Cluster at ClemsonUniversity. The second fault injector is the sequential fault injec-tor, FSEFI [23]. Trials with FSEFI were run on the Gamma Clusterat the New Mexico Consortium located at Los Alamos NationalLaboratory.
When testing the MMM kernel, FlipIt injects a single bit-flipinto a random bit position in the result register of a random LLVMintermediate representation (IR) instruction during the main com-putation phase. Conversely, FSEFI injects a single bit-flip into arandom bit position during the computation of specified floating
3https://mantevo.org/packages.php
FaultSight: A Fault Analysis Tool for HPC Researchers FTXS’19, November 22, 2019, Denver, CO
A1,1
A2,1
A1,2
A2,2
B1,1 B1,2
B2,1 B2,2
C1,1
C2,1
C1,2
C2,2
Matrix A (n x n)
C1,1 = M1 + M4 - M5 + M7
C2,2 = M1 - M2 + M3 + M6
M1 =
(a) Calculating A x B = C with Strassen MMMM1 = (A1,1 + A2,1) (B1,1 + B2,2)
M7 = ...
M1 =
M1 =
0 errors
1 error
n/2 errors
(b) Injection during M1 calculation
C1,1 =(c) Propagation of M1 error to resulting C matrix
+ =+A1,1
X =
C =
C2,2 =
n errors
A2,1 B2,1 B2,2
M1 M4 M5
+-M7 C1,1
M1 M2 M3 M6 C2,2
- + + =
Matrix B (n x n) Matrix C (n x n)
+
Figure 3: Propagation of error in the Strassen algorithm.
point x86 instructions — e.g. fmul, fadd, fsub— during the main com-putation phase. When testing the HPCCG mini-app, FlipIt utilizesthe same configuration used when testing the MMM kernel.
Both injectors classify each injection based on how that instruc-tion is used in code: floating-point arithmetic (Arith-FP), fixed-point integer arithmetic (Arith-Fix), pointer and address calculation(Pointer), branching and comparisons (Ctrl-Branch), and controlflow/looping (Ctrl-Loop). The injection classification, fault injectionlocation in source code, and result of each trial is recorded andinserted off-line into a FaultSight database using a Python scriptand the API (see Section 3).
FaultSight directly generates all the plots and statistical analysisin this section without the need for external processing and verifiesthat fault injection campaign’s size allows meaningful statisticalanalysis (see Section 3.3.1).
4.1 Matrix-Matrix Multiplication (MMM)In this experiment, FaultSight’s ability to perform A/B testing aswell as the ability to compare two different fault injection frame-works is explored.
MMM computes C = A × B, where A and B are dense n × nmatrices of order n = 512. Three different MMM algorithms areinvestigated to determine which is the most resilient. The firstalgorithm is the canonical three loop algorithm Matmul, the secondis a cache-block tiled version of the standard algorithm Tiled, andthe third is the Strassen algorithm Strassen [39]. Figure 3(a) showshow the Strassen algorithm recursively reduces each matrix intofour equally sized blocks and recombines them to calculate eachentry, Ci j . This reduces the algorithmic complexity by eliminatingsome operations. The reliability of each algorithm is assessed byanalyzing the number of corrupted elements in the final output.Machine epsilon is the error threshold to determine if any twofloating-point values differ. Through the use of the extend API call,an attribute that represents the number of corrupted elements inthe final ouput of each trial was added.
The FlipIt fault injection campaign consists of running 6,000injection trials. The trials were split into three groups and hadthe following breakdown: 2,000 Matmul, 2,000 Tiled, and 2,000Strassen.
When FlipIt considers a function for injection, it considers thebase function and all functions it invokes. Conversely, FSEFI re-quires a specific instruction to be given that it will target. With this
in mind, a set of floating point x86 instructions were given to FSEFIto better mirror the results of FlipIt and therefore lead to a bettercomparison of the two tools.
For FSEFI, the fault injection campaign consists of running 8,000injection trials. The trials are split into five groups with the fol-lowing breakdown: 2,000 Matmul while targeting the main fmul,2,000 Tiled while targeting the main fmul, 2,000 Strassen whiletargeting the main fmul, and 2,000 Strassen with 1,000 targetingthe main fadd and 1,000 targeting the main fsub.
Throughout the FlipIt campaign, many different functions werehit with faults. During the Matmul trials all 2,000 injections hitwithin the matmul function. During the Tiled trials 1,999 injec-tions hit within the matmul_tiled function while 1 hit within themin function. During the Strassen trials 182 injections hit withinthe matmul_strassen function, 1,764 injections hit within the mat-mul function, and 54 injections hit within the matadd function.The matmul_strassen function utilizes both matmul and matadd tocombine blocks of matrices. These results help derive the set of x86instructions that FSEFI implemented in its experiments.
The number of corrupted elements for all FlipIt and FSEFI trialsare recorded. Fault injections in the Matmul trials result in a total of1,259 corrupted items in FlipIt and 1,700 corrupted items in FSEFI.On average, a single injection in FlipIt corrupted 0.63 elementsand 0.85 elements in FSEFI. Injections in the Tiled trials result ina total of 1,216 corrupted elements in FlipIt and 1,707 corrupteditems in FSEFI. On average, a single injection in FlipIt corrupted0.61 elements and 0.853 elements in FSEFI. Lastly, Fault injectionsin the Strassen trials result in 1,156,562 corrupted items in FlipItand 364,029 corrupted items in FSEFI. On average, a single injectionin FlipIt corrupted 578.3 elements and 182 elements in FSEFI.
The difference between FlipIt and FSEFI’s results from the Matmuland Tiled trials are due to variations in how significant the faultwas on the injected value. For instance, if the fault occurred inthe lower bits then the fault could fall off during floating pointcomputation or due to rounding of by the system.
The high number of elements corrupted in the Strassen results,seen from both fault injectors, come from corruption of the sub-matrices that are used to computeC . Corrupting these sub-matricespropagates the errors to even more elements of C . Figure 3(b)–(c)illustrates this propagation pattern when a fault is injected into amatadd operation during the calculation of M1. The fault results
FTXS’19, November 22, 2019, Denver, CO E. Horn et al.
30
25
20
15
10
5
0
Bit Location
Classification of Injections By Bit
0 5 10 15 20 25 30 35 40 45 50 55 60 63
Freq
uenc
y
(a) FlipIt Distribution
140
120
100
80
60
40
20
0
Classification of Injections By Bit
Bit Location 0 5 10 15 20 25 30 35 40 45 50 55 60 63
Freq
uenc
y
(b) FSEFI Distribution
Figure 4: Bit Flip Distribution of Arith-FP instruction types for each injector
in a single element (i,k) of the matadd being in error. The mul-tiplication operation propagates this error at (i,k) to all entriesin row i due to data reuse in the matmul function leading to n/2errors. The algorithm uses M1 in the calculation of C1,1 and C2,2which propagates the results to the final output. Depending on howmuch progress the algorithm has made before the fault occurs, thiscan result in corruption of up to n elements in the matrix C . Faultinjection results show a corruption of 578.3 elements from FlipItand 182 elements from FSEFI on average. From these results, it canbe observed that FSEFI chose a uniform distribution of locations tohit with some bias towards later instructions in the execution. Ifhit locations were completely unbiased then the average numberof corrupted elements when only hitting the Arith-FP instructions,as in the case of FSEFI, would be n/2 or 256 in our case. Therefore,FSEFI’s 182 denotes a uniform distribution with some bias towardslater instructions. FlipIt’s results went above n due to differentpropagation signatures that lead to increase corruption in C — e.g.terminating a loop early and not computing a portion of the matrix.
Fault injectors need to be unbiased in their choice of which bitsshould be flipped from the range they are given. Through Fault-Sight’s visualizations, we can visualize the distribution of injectionlocations from each fault injector and compare them. As seen inFigure 4, we can are able to compare the distributions for bothfault injectors Arith-FP operations and tell that both have uniformdistributions when choosing which bit to flip.
4.1.1 Algorithm A/B Testing. Using the custom partitioning fea-ture of FaultSight, we instruct FaultSight to partition the fault in-jection trials into the 3 groups for the two different fault injectors,one for each MMM. Performing a Student’s T-Test to compare fre-quency of corrupted elements in the matrix C pairwise betweentwo different MMM functions, it is found that the difference inerror rates between the Matmul and Tiled approaches is not signif-icant, with a p-value of 0.16 with FlipIt and a p-value of 0.76 withFSEFI. However, the difference between the Matmul or Tiled to theStrassen approach is significant, where both comparisons resultin a p-value of 0.0097 with FlipIt and a p-value of 1.78 × 10−211.
Based upon this analysis, if you are running in an environmentwhere the probability of suffering a bit-flip error during MMMoperation is likely and employ no additional resiliency techniques,it is beneficial to use a slower performing algorithm — i.e., Matmulor Tiled — as opposed to Strassen. This is due to the fact that
the high volume of data reuse in the Strassen algorithm leads to ahigh degree of corruption in the resulting matrix C .
4.1.2 Protecting Matrix-Matrix Multiplication. Prior work [25,41] explores protecting linear algebra operations with row and col-umn checksums in a technique known as algorithm-based faulttolerance (ABFT). Using checksums, ABFT is able to pinpoint thematrix entry that is incorrect and correct it. Based on FlipIt’s MMMinjection campaign, on average 0.63 and 0.61 elements are corruptedfor Matmul and Tiled, respectively; therefore, ABFT is an effec-tive scheme that detects and corrects all single entry errors. Eventhough, in FlipIt’s trials, Strassen resulted in 578.3 elements onaverage being corrupted, ABFT is an effective scheme for protectingthis algorithm. Since Strassen subdivides A and B, appending rowand column checksums to these sub-matrices allows for detectionof corrupt entries in the intermediate matadd and matmul beforethe error propagates via matmul operations. Applying checksumsto each of the sub-matrices results in a larger memory overheaddue to requiring more checksums by a factor of 2 for each levelof the recursion. Although ABFT is on average an attractive solu-tion to protect MMM, the faults that result in multiple erroneousentries such as early loop termination may be unrecoverable withABFT. However, augmenting ABFT protected MMM with code thatensures correct control flow [28, 33] alleviates this shortcoming.
4.2 HPCCGIn this example, FaultSight is used to show that a software-basedsoft error detection/recovery scheme successfully limits the numberof extra iterations until the problem converges compared to runs ofan unmodified HPCCG. Here, FaultSight conducts statistical testsof significance in several contexts — comparing the efficacy of adetection/recovery scheme and comparing the impact of injectionsinto different locations/bits/datatypes.
HPCCG4 is a conjugate gradient (CG) benchmark from the Man-tevo Suite that simulates a 3D chimney domain using a 27-pointfinite difference matrix. In these experiments, HPCCG is run with16 parallel MPI processes, a local block size of nz = ny = nz = 48,and a solver tolerance of 1e−10. For each trial, a single bit-flip isinjected using FlipIt during the solve phase of HPCCG on a ran-domly selected MPI process and instruction. Because a bit-flip can
4https://mantevo.org/packages.php
FaultSight: A Fault Analysis Tool for HPC Researchers FTXS’19, November 22, 2019, Denver, CO
90.1%: HPC_sparsemv2.9%: ddot7%: waxpby0%: exchange_externals
Figure 5: Fault injections mapped to functions for HPCCG.
lead to extra iterations, the number of iterations is capped at 500.Any trial reaching 500 iterations is marked as not converged.
The campaign uses 3,000 trials, where the first 1,500 trials havedetection but no recovery scheme, and the remaining 1,500 trialshave both a detection and recovery scheme implemented. The de-tection scheme verifies the plausibility of the residual at the end ofeach iteration by ensuring that the residual does not increase by anorder-of-magnitude [12]. At the end of every iteration, the solutionvector x , the residual vector r , and the search direction vector p arecheckpointed to local memory. When the residual check is violated,the in-memory checkpoint is read and the variables reinitialized tothe checkpointed values [40].
4.2.1 Impact on Convergence. FaultSight maps the injectionsback to the current executing function, Figure 5, and reveals thatgreater than 90% of injections occur in the function HPC_sparsemvthat computes sparsematrix-vector multiplication (SpMV), themosttime consuming portion of the code. This is consistent with priorstudies on other iterative linear solvers [12, 14]. Other functionswith injections include the ddot (inner-product), waxpby (scaledvector addition), and exchange_internals (communication of vec-tor values for parallel sparse matrix-vector multiplication)5. Ex-ploring the code view for each injected function highlights thevulnerable regions. Typically this is the most frequently executedcode segments such as the innermost loop (see Figure 2).
During the database generation phase of the workflow, we useFaultSight’s API (extend_trial_table) to extend the existing data-base by adding an attribute to represent the number of iterations forthe problem to converge. Combining this with FaultSight’s abilityto test hypotheses, we determine if the detection/recovery schemeis effective at limiting the number of iterations to converge in afaulty environment.
In the fault-free case, HPCCG requires 90 iterations to convergeto our set tolerance. In our fault injection trials, 1.8% of trials re-quire extra iterations. Of those trials requiring extra iterations, 5trials did not converge. Figure 6 refines Figure 5 to explore onlythose trials that cause extra iterations. Because the SpMV has ahigher degree of data reuse than vector only operations [13], errorpropagation patterns in the SpMV are similar to matmul in theStrassen algorithm.
Figure 7 shows the bit location of the injections broken downby the type of instruction for all trials that require extra iterationsto converge. In Figure 7, there are 3 clusters based on bit location
5Note there are 2 injections into exchange_internals.
and instruction type. The first group (bits 0–2) consists of injec-tions into Pointer type instructions, resulting in reads and writes ofunaligned data. Reading/writing unaligned data produces garbagevalues that have a high error between the correct value. The secondgroup consists of Ctrl-Loop and Arith-Fix operations. Corruptingthese operations leads to early termination of loops which result inarrays not being fully updated or correctly initialized. This leadsto computing on unknown values, resulting in large deviationsfrom the correct values. The third group consists of injections intoArith-FP instructions. These injections take place in the upper bitsof the mantissa (bits 48–51), all the exponent bits (bits 52–62), andthe sign bit (bit 63). Bit-flips in the most significant bits of the man-tissa cause noticeable deviations in the magnitude of floating-pointvalues. Bit-flips in the sign bit results in the negation of a floating-point value, and the deviation from the true value is proportionalto its magnitude. However, bit-flips in the exponent cause muchlarger deviations; some orders of magnitude more than bit-flips inthe mantissa or sign-bit. Large deviations result in added error thatpropagates to the vectors x , p, and r . Corruption in these vectorscause HPCCG to require more iterations to achieve the fixed solvertolerance. In addition, injections into Arith-FP instructions is statis-tically significant to lead to extra iterations (beyond fault-free) witha p-value of 6.8e−278. Other instruction types show non-significantresults due to the high rates of unexpected application termination.
Figures 8 and 9 highlight detection. The detection scheme forHPCCG verifies that the new residual is within an order of magni-tude of the previous one. Therefore, this detector only triggers whenlarge deviations occur — i.e., bit-flips in the exponent and readingof garbage values. If the detection scheme places tighter guaranteeson how much the residual increases, Figure 9 approaches Figure 7as all injections that lead to extra iterations are detected. Similarly,Figure 8 closely matches Figure 6 signifying that the detectionscheme is preventing trials that require extra iterations. Becausea tight bound is not used, only trials that require 3+ iterations toconverge are detected. Provided that other information such asprogram wall-clock time or energy is logged, FaultSight can also beused to help analyze and tune the tolerance bound on the residualdetector to minimize wall-clock time and/or energy consumption.
Out of the 90 trials that took more iterations to converge than thefault-free case (and successfully converged), 62 trials (70%) have adetection. The 62 trials are split into two trial sets based on whetherthe trial implements a recovery scheme or not. The average numberof iterations to converge of the set of trials with a recovery schemeis 96.44, while the average number of iterations to converge fortrials with no recovery scheme is 108.6. Using the Student’s T-testfor statistical independence, shows that the difference is significantwith a p-value of 8.9e−8. Thus, the detection and recovery schemeis effective at reducing the extra iterations needed to convergewhen transient soft errors corrupt portions of the CG linear solver.
4.2.2 Unexpected Termination. Over the course of the fault injec-tion campaign, 33.8% of fault injection trials experience unexpectedapplication termination. In these experiments, two symptoms causethe unexpected termination events: generation of a segmentationfault and generation of a bus error.
Figure 10 reports the bit locations that lead to a segmentationfault grouped by the injected instruction type. In Figure 10, there
FTXS’19, November 22, 2019, Denver, CO E. Horn et al.
75.6%: HPC_sparsemv14.4%: ddot10%: waxpby
Figure 6: Injections mapped to functions for HPCCG, wherethe number of iteration are greater than the fault-free case.
0 5 10 15 20 25 30 35 40 45 50 55 60 630
2
4
6
8
10
12
14
Frequency
Bit Location
Arith-FPPointerArith-FixCtrl-LoopCtrl-Branch
Figure 7: Classification of injections by bit, from trialswherethe number of iterations to converge is greater than thefault-free case.
are no segmentation faults due to injections into Arith-FP instruc-tions. Segmentation faults from injections into Pointer instructionsoccur in bits that lead to unaligned data access (0–1) yielding in-correct loads/stores or occur in bits that creates a pointer outsidethe memory allocation (21–63). Fixed-point arithmetic operations,Arith-Fix, are performed on 32-bit integers and do not have in-jections in bits 32–63. Injections into Arith-Fix instructions thatcause segmentation faults produce results that lead into incorrectaddress calculation. During compilation, clang promotes the 32-bitintegers used in loop iteration variables to 64-bit, allowing seg-faults generated to be concentrated in bits outside of the size ofallocated memory for calculation on loop iteration variables, Ctrl-Loop. Finally, injections into branches, Crtl-Branch results in anextra iteration, and during the extra iteration the program accessesout-of-bound memory — e.g., accessing off the end of an array.
The second cause of unexpected termination is due to bus errors.An x86 processor raises a bus error when the program accessesan undefined portion of a memory object. Figure 11 shows the bitposition and instruction types that lead to a bus error in our faultinjection campaign. These are confined to Pointer instructions andin particular, the most significant bits of the pointer. Bit-flips inthese positions cause a large deviation in the pointer that leads toaccessing undefined portions of memory.
Statistical analysis of the bit locations shows that injectionsin bits 32–63 are statistically more likely to lead to unexpectedtermination than bits 0–31 with a p-value of 1.08e−39. From thisanalysis, FaultSight shows that injections into the most significantbits are the most impactful. Adding protection to these bits eitherin hardware or software can lower an application’s likelihood of
88.9%: HPC_sparsemv6.3%: ddot4.8%: waxpby
Figure 8: Injections mapped to functions, for trials with de-tection.
0 5 10 15 20 25 30 35 40 45 50 55 60 630
2
4
6
8
10
12
14
Freq
uency
Bit Location
Arith-FPPointerArith-FixCtrl-LoopCtrl-Branch
Figure 9: Classification of injections by bit, for trials withdetection.
unexpectedly terminating when soft error corrupts logic and in-struction results. For example, [14] uses triple modular redundancyto protect key pointers and avoid segfaults. Additionally instructionreplication techniques [28] could add protection for pointers andpointer operations. Using code analysis to pinpoint the code regionsthat suffer unexpected termination allows for reduced overheadsin these resiliency schemes.
5 RELATEDWORKSeveral types of fault injectors have been used to evaluate thereliability of HPC applications. FSEFI [23] injects faults into anapplication running on a virtual machine, and has a very highlevel of control on where it inject faults. At a slightly higher level,FlipIt [11], LLFI [31], KULFI [36], and SAFIRE-[21] uses an LLVMcompiler pass to instrument code for fault injection. However, otherstudies corrupt MPI communication [20, 30]. Yet others, corruptapplication level variables [4, 6]. Each fault injector has a uniqueway of recording data about the location and time of the injection.The tool FaultTelescope built for KULFI identifies vulnerable regionsof code [15], but does not support data from other fault injectorsnor statistical testing. FaultSight’s API allows for data from manydifferent fault injectors to be recorded into the database for analysis.
Tools to analyze system reliability have been developed to ana-lyze reliability, availability, and serviceability of (RAS) logfiles aswell as other logfiles generated by HPC systems [17–19, 22, 24, 32,34]. These tools address similar problems working with large datavolumes, but are not designed to analyze and visualize data comingfrom fault injection campaigns and comparing the effectiveness ofintegrated application level detection and recovery schemes. Fault-Sight’s web app interface provides more efficient analysis of faultinjection data.
FaultSight: A Fault Analysis Tool for HPC Researchers FTXS’19, November 22, 2019, Denver, CO
0 5 10 15 20 25 30 35 40 45 50 55 60 630
10
20
30
40
50
Freq
uency
Bit Location
Arith-FPPointerArith-FixCtrl-LoopCtrl-Branch
Figure 10: Classification of injections by bit, for trials witha Segmentation Fault.
0 5 10 15 20 25 30 35 40 45 50 55 60 630.0
0.5
1.0
1.5
2.0
2.5
3.0
Freq
uency
Bit Location
Arith-FPPointerArith-FixCtrl-LoopCtrl-Branch
Figure 11: Classification of injections by bit, for trials witha Bus Error.
Debugging tools — e.g., Allinea DDT6, TotalView7, GDB8 —allowdeep introspection into a running application in order to identifysoftware bugs and promote correctness. While they can be used forfault injection, their view is limited to a single application’s execu-tion and do not easily allow analysis of thousands of applicationruns at once.
In order to improve performance or efficiency of HPC applica-tions software profilers are commonly used to visualize how timeand resources are spent during an application run [1, 10, 37]. Soft-ware profilers allow a detailed view of events such as functioninvocations, message ordering, load imbalance, and their associatedruntimes. However, current HPC profilers do not visualize the extrainformation coming from/about a simulation due to fault injection.
6 CONCLUSION AND FUTUREWORKTo obtain meaningful statistics when assessing an application’sresiliency to faults or to test the effectiveness of a software-basedfault detection/recovery scheme requires thousands of fault injec-tion trails. Analyzing the large volume of data is laborious as thereis no common tool for various fault injectors used in practice. Thispaper presents FaultSight: a general fault injection analysis tool ca-pable of visualizing and analyzing the results of thousands of faultinjection trials from various fault injectors thought its extendableinterface.
We use FaultSight to analyze the reliability of three Matrix-Matrix Multiplication algorithms from multiple perspectives andfind there is not much difference between the canonical threeloop version and the tiled version. However, we find that Strassen
6https://www.arm.com/products/development-tools/hpc-tools/cross-platform/forge/ddt7https://www.roguewave.com/products-services/totalview8https://www.gnu.org/software/gdb/
algorithm generates hundreds more corrupt entries in both per-spectives than the other algorithms due to its higher reuse of sub-computations. We also analyze the effectiveness of a detection andrecovery scheme for the mini-app HPCCG. Through FaultSight’sstatistical analysis feature, we show that the resiliency scheme iseffective at limiting the number of extra iterations when transientfaults occur.
Future work on FaultSight seeks to improve its interoperabilitywith other common data analysis frameworks such as Spark andprovide support for machine learning to detect trends in the faultinjection data.
Finally, prior research is replicated and supported through ana-lyzing fault injection campaigns on MMM and HPCCG.
FaultSight is publicly available under the MIT license on github:https://github.com/einarhorn/FaultSight.
ACKNOWLEDGMENTThis work was sponsored by the Air Force Office of Scientific Re-search under grant FA9550–12–1–0478. This research is part ofthe Blue Waters sustained-petascale computing project, which issupported by the National Science Foundation (NSF) (awards OCI–0725070 and ACI–1238993) and the state of Illinois. Blue Waters is ajoint effort of the University of Illinois at Urbana-Champaign and itsNational Center for Supercomputing Applications. This material isalso based in part on work supported by the Department of Energy,National Nuclear Security Administration, under Award NumberDE-NA0002374. This material is based upon work supported by theNSF under Grant No. SHF-1617488.
Wewould also like to thank TerryGrové andNathanDebardelebenat Los Alamos National Laboratory for their assistance and allowingus to utilize the Gamma Cluster.
REFERENCES[1] L. Adhianto, S. Banerjee, M. Fagan, M. Krentel, G. Marin, J. Mellor-Crummey, and
N. R. Tallent. 2010. HPCTOOLKIT: Tools for Performance Analysis of OptimizedParallel Programs Http://Hpctoolkit.Org. Concurr. Comput. : Pract. Exper. 22, 6(April 2010), 685–701. https://doi.org/10.1002/cpe.v22:6
[2] R. C. Baumann. 2005. Radiation-induced soft errors in advanced semiconductortechnologies. Device and Materials Reliability, IEEE Transactions on 5, 3 (Sept.2005), 305–316. https://doi.org/10.1109/TDMR.2005.853449
[3] Leonardo Bautista-Gomez and Franck Cappello. 2015. Detecting Silent Data Cor-ruption for Extreme-Scale MPI Applications. In Proceedings of the 22Nd EuropeanMPI Users’ Group Meeting (EuroMPI ’15). ACM, New York, NY, USA, Article 12,10 pages. https://doi.org/10.1145/2802658.2802665
[4] Leonardo Bautista-Gomez and Franck Cappello. 2015. Exploiting Spatial Smooth-ness in HPC Applications to Detect Silent Data Corruption. In Proceedings ofthe 2015 IEEE 17th International Conference on High Performance Computing andCommunications, 2015 IEEE 7th International Symposium on Cyberspace Safetyand Security, and 2015 IEEE 12th International Conf on Embedded Software andSystems (HPCC-CSS-ICESS ’15). IEEE Computer Society, Washington, DC, USA,128–133. https://doi.org/10.1109/HPCC-CSS-ICESS.2015.9
[5] Leonardo Bautista-Gomez, Seiji Tsuboi, Dimitri Komatitsch, Franck Cappello,Naoya Maruyama, and Satoshi Matsuoka. 2011. FTI: high performance fault toler-ance interface for hybrid systems. In Proceedings of 2011 International Conferencefor High Performance Computing, Networking, Storage and Analysis (SC ’11). ACM,NewYork, NY, USA, Article 32, 32 pages. https://doi.org/10.1145/2063384.2063427
[6] Austin R Benson, Sven Schmit, and Robert Schreiber. 2015. Silent error detectionin numerical time-stepping schemes. The International Journal of High Perfor-mance Computing Applications 29, 4 (2015), 403–421. https://doi.org/10.1177/1094342014532297 arXiv:http://dx.doi.org/10.1177/1094342014532297
[7] Keren Bergman, Shekhar Borkar, Dan Campbell, William Carlson, William Dally,Monty Denneau, Paul Franzon, William Harrod, Kerry Hill, Jon Hiller, et al. 2008.Exascale computing study: Technology challenges in achieving exascale systems.Technical Report. Defense Advanced Research Projects Agency InformationProcessing Techniques Office (DARPA IPTO).
FTXS’19, November 22, 2019, Denver, CO E. Horn et al.
[8] E. Berrocal, L. Bautista-Gomez, S. Di, Z. Lan, and F. Cappello. 2017. TowardGeneral Software Level Silent Data CorruptionDetection for Parallel Applications.IEEE Transactions on Parallel and Distributed Systems 28, 12 (Dec 2017), 3642–3655.https://doi.org/10.1109/TPDS.2017.2735971
[9] Shekhar Borkar. 2005. Designing Reliable Systems from Unreliable Components:The Challenges of Transistor Variability and Degradation. IEEE Micro 25, 6 (Nov.2005), 10–16. https://doi.org/10.1109/MM.2005.110
[10] S. Browne, J. Dongarra, N. Garner, G. Ho, and P. Mucci. 2000. A Portable Pro-gramming Interface for Performance Evaluation on Modern Processors. Int. J.High Perform. Comput. Appl. 14, 3 (Aug. 2000), 189–204. https://doi.org/10.1177/109434200001400303
[11] Jon Calhoun, Luke Olson, and Marc Snir. 2014. FlipIt: An LLVM Based FaultInjector for HPC. In Proceedings of the 20th International Euro-Par Conference onParallel Processing (Euro-Par ’14).
[12] Jon Calhoun, Luke Olson, Marc Snir, and William D. Gropp. 2015. Towards aMore Fault Resilient Multigrid Solver. In Proceedings of the Symposium on HighPerformance Computing (HPC ’15). Society for Computer Simulation International,San Diego, CA, USA, 1–8. http://dl.acm.org/citation.cfm?id=2872599.2872600
[13] Jon Calhoun, Luke N. Olson, Marc Snir, and William D. Gropp. 2017. Towards aMore Complete Understanding of SDC Propagation. In Proceedings of the 26thInternational Symposium on High-Performance Parallel and Distributed Computing(HPDC ’17). ACM, New York, NY, USA.
[14] Marc Casas, Bronis R. de Supinski, Greg Bronevetsky, and Martin Schulz. 2012.Fault resilience of the algebraic multi-grid solver. In Proceedings of the 26th ACMinternational conference on Supercomputing (ICS ’12). ACM, New York, NY, USA,91–100. https://doi.org/10.1145/2304576.2304590
[15] Sui Chen, Greg Bronevetsky, Bin Li, Marc Casas Guix, and Lu Peng. 2015. Aframework for evaluating comprehensive fault resilience mechanisms in numer-ical programs. The Journal of Supercomputing 71, 8 (01 Aug 2015), 2963–2984.https://doi.org/10.1007/s11227-015-1422-z
[16] Zizhong Chen. 2013. Online-ABFT: An Online Algorithm Based Fault ToleranceScheme for Soft Error Detection in Iterative Methods. In Proceedings of the 18thACM SIGPLAN Symposium on Principles and Practice of Parallel Programming(PPoPP ’13). ACM, New York, NY, USA, 167–176. https://doi.org/10.1145/2442516.2442533
[17] S. Di, R. Gupta, M. Snir, E. Pershey, and F. Cappello. 2017. LOGAIDER: A Toolfor Mining Potential Correlations of HPC Log Events. In 2017 17th IEEE/ACMInternational Symposium on Cluster, Cloud and Grid Computing (CCGRID). 442–451. https://doi.org/10.1109/CCGRID.2017.18
[18] E.Chuah et al. 2019. Towards comprehensive dependability-driven resource useand message log-analysis for HPC systems diagnosis. J. Parallel and Distrib.Comput. (May 2019), 95–112. https://doi.org/10.1016/j.jpdc.2019.05.013
[19] M. Hickman et al. 2018. Enhancing HPC System Log Analysis by IdentifyingMessage Origin in Source Code. 2018 IEEE International Symposium on SoftwareReliability Engineering Workshops (Aug. 2018), 100–105. https://doi.org/10.1109/ISSREW.2018.00-23
[20] David Fiala, Frank Mueller, Christian Engelmann, Rolf Riesen, Kurt Ferreira, andRon Brightwell. 2012. Detection and Correction of Silent Data Corruption forLarge-scale High-performance Computing. In Proceedings of the InternationalConference on High Performance Computing, Networking, Storage and Analysis (SC’12). IEEE Computer Society Press, Los Alamitos, CA, USA, Article 78, 12 pages.http://dl.acm.org/citation.cfm?id=2388996.2389102
[21] G Georgakoudis, I Laguna, H Vandierendonck, D S Nikolopoulos, and M Schulz.2019. SAFIRE: Scalable and Accurate Fault Injection ForParallel MultithreadedApplications. IEEE International Parallel Distributed ProcessingSymposium (Jan.2019). https://www.osti.gov/servlets/purl/1518562
[22] Alfredo Giménez, Todd Gamblin, Abhinav Bhatele, Chad Wood, Kathleen Shoga,Aniruddha Marathe, Peer-Timo Bremer, Bernd Hamann, and Martin Schulz. 2017.ScrubJay: Deriving Knowledge from the Disarray of HPC Performance Data.In Proceedings of the International Conference for High Performance Computing,Networking, Storage and Analysis (SC ’17). ACM, New York, NY, USA, Article 35,12 pages. https://doi.org/10.1145/3126908.3126935
[23] Qiang Guan, Nathan BeBardeleben, Panruo Wu, Stephan Eidenbenz, Sean Blan-chard, Laura Monroe, Elisabeth Baseman, and Li Tan. 2016. Design, Use andEvaluation of P-FSEFI: A Parallel Soft Error Fault Injection Framework for Emulat-ing Soft Errors in Parallel Applications. In Proceedings of the 9th EAI InternationalConference on Simulation Tools and Techniques (SIMUTOOLS’16). ICST (Institutefor Computer Sciences, Social-Informatics and Telecommunications Engineer-ing), ICST, Brussels, Belgium, Belgium, 9–17. http://dl.acm.org/citation.cfm?id=3021426.3021429
[24] Hanqi Guo, Sheng Di, Rinku Gupta, Tom Peterka, and Franck Cappello. 2018. LaVALSE: Scalable Log Visualization for Fault Characterization in Supercomputers.In Eurographics Symposium on Parallel Graphics and Visualization, Hank Childsand Fernando Cucchietti (Eds.). The Eurographics Association. https://doi.org/10.2312/pgv.20181099
[25] Kuang-Hua Huang and J. A. Abraham. 1984. Algorithm-Based Fault Tolerancefor Matrix Operations. IEEE Trans. Comput. 33, 6 (June 1984), 518–528. https://doi.org/10.1109/TC.1984.1676475
[26] Luc Jaulmes, Marc Casas, Miquel Moretó, Eduard Ayguadé, Jesús Labarta, andMateo Valero. 2015. Exploiting Asynchrony from Exact Forward Recovery forDUE in Iterative Solvers. In Proceedings of the International Conference for HighPerformance Computing, Networking, Storage and Analysis (SC ’15). ACM, NewYork, NY, USA, Article 53, 12 pages. https://doi.org/10.1145/2807591.2807599
[27] Himanshu Kaul, Mark Anders, Steven Hsu, Amit Agarwal, Ram Krishnamurthy,and Shekhar Borkar. 2012. Near-threshold Voltage (NTV) Design: Opportunitiesand Challenges. In Proceedings of the 49th Annual Design Automation Conference(DAC ’12). ACM, New York, NY, USA, 1153–1158. https://doi.org/10.1145/2228360.2228572
[28] Ignacio Laguna, Martin Schulz, David F. Richards, Jon Calhoun, and Luke Olson.2016. IPAS: Intelligent Protection Against Silent Output Corruption in Scien-tific Applications. In Proceedings of the 2016 International Symposium on CodeGeneration and Optimization (CGO 2016). ACM, New York, NY, USA, 227–238.https://doi.org/10.1145/2854038.2854059
[29] R. Leveugle, A. Calvez, P. Maistri, and P. Vanhauwaert. 2009. Statistical FaultInjection: Quantified Error and Confidence. In Proceedings of the Conferenceon Design, Automation and Test in Europe (DATE ’09). European Design andAutomation Association, 3001 Leuven, Belgium, Belgium, 502–506. http://dl.acm.org/citation.cfm?id=1874620.1874743
[30] Charng-da Lu and Daniel A. Reed. 2004. Assessing Fault Sensitivity in MPIApplications. In Proceedings of the 2004 ACM/IEEE Conference on Supercomputing(SC ’04). IEEE Computer Society, Washington, DC, USA, 37–. https://doi.org/10.1109/SC.2004.12
[31] Qining Lu, Mostafa Farahani, Jiesheng Wei, Anna Thomas, and Karthik Pattabira-man. 2015. LLFI: An Intermediate Code-Level Fault Injection Tool for HardwareFaults. In Proceedings of the 2015 IEEE International Conference on Software Quality,Reliability and Security (QRS ’15). IEEE Computer Society, Washington, DC, USA,11–16. https://doi.org/10.1109/QRS.2015.13
[32] Catello Di Martino, Saurabh Jha, William Kramer, Zbigniew Kalbarczyk, andRavishankar K. Iyer. 2015. LogDiver: A Tool for Measuring Resilience of Extreme-Scale Systems and Applications. In Proceedings of the 5th Workshop on FaultTolerance for HPC at eXtreme Scale (FTXS ’15). ACM, New York, NY, USA, 11–18.https://doi.org/10.1145/2751504.2751511
[33] Xiang Ni and Laxmikant V. Kale. 2016. FlipBack: Automatic Targeted ProtectionAgainst Silent Data Corruption. In Proceedings of the International Conference forHigh Performance Computing, Networking, Storage and Analysis (SC ’16). IEEEPress, Piscataway, NJ, USA, Article 29, 12 pages. http://dl.acm.org/citation.cfm?id=3014904.3014943
[34] Byung Hoon (Hoony) Park, Yawei Hui, Swen Boehm, Rizwan Ashraf, ChristianEngelmann, and Christopher Layton. 2018. A Big Data Analytics Framework forHPC Log Data: Three Case Studies Using the Titan Supercomputer Log. In Pro-ceedings of the 19th IEEE International Conference on Cluster Computing (Cluster)2018: 5th Workshop on Monitoring and Analysis for High Performance SystemsPlus Applications (HPCMASPA) 2018. IEEE Computer Society, Los Alamitos, CA,USA, Belfast, UK, 571–579. https://doi.org/10.1109/CLUSTER.2018.00073
[35] John Shalf, Sudip Dosanjh, and John Morrison. 2011. Exascale Computing Tech-nology Challenges. In Proceedings of the 9th International Conference on HighPerformance Computing for Computational Science (VECPAR’10). Springer-Verlag,Berlin, Heidelberg, 1–25. http://dl.acm.org/citation.cfm?id=1964238.1964240
[36] Vishal Chandra Sharma, Arvind Haran, Zvonimir Rakamarić, and GaneshGopalakrishnan. 2013. Towards Formal Approaches to System Resilience. InProceedings of the 19th IEEE Pacific Rim International Symposium on DependableComputing (PRDC).
[37] Sameer S. Shende and Allen D. Malony. 2006. The Tau Parallel PerformanceSystem. Int. J. High Perform. Comput. Appl. 20, 2 (May 2006), 287–311. https://doi.org/10.1177/1094342006064482
[38] Marc Snir, Robert W Wisniewski, Jacob A Abraham, Sarita V Adve, SaurabhBagchi, Pavan Balaji, Jim Belak, Pradip Bose, Franck Cappello, Bill Carlson, An-drew A Chien, Paul Coteus, Nathan A DeBardeleben, Pedro C Diniz, Christian En-gelmann, Mattan Erez, Saverio Fazzari, Al Geist, Rinku Gupta, Fred Johnson, Sri-ram Krishnamoorthy, Sven Leyffer, Dean Liberty, Subhasish Mitra, Todd Munson,Rob Schreiber, Jon Stearley, and Eric Van Hensbergen. 2014. Addressing Failuresin Exascale Computing. International Journal of High Performance Computing Ap-plications 28, 2 (May 2014), 127 – 171. https://doi.org/10.1177/1094342014522573
[39] Volker Strassen. 1969. Gaussian Elimination is Not Optimal. Numer. Math. 13, 4(Aug. 1969), 354–356. https://doi.org/10.1007/BF02165411
[40] Dingwen Tao, Sheng Di, Xin Liang, Zizhong Chen, and Franck Cappello. 2018. Im-proving performance of iterative methods by lossy checkponting. In Proceedingsof the 27th International Symposium on High-Performance Parallel and DistributedComputing. ACM, 52–65.
[41] Dingwen Tao, Shuaiwen Leon Song, Sriram Krishnamoorthy, Panruo Wu, XinLiang, Eddy Z Zhang, Darren Kerbyson, and Zizhong Chen. 2016. New-sum:A novel online abft scheme for general iterative methods. In Proceedings of the25th ACM International Symposium on High-Performance Parallel and DistributedComputing. ACM, 43–55.