An agency of the European Union

The masking effect associated with the measures of disproportionality analysis

Presented by: François MAIGNENPosition or Unit/Sector/Section/Team

Context: spontaneous reportingSpontaneous reports of adverse drug reactions contain several

suspected medicines (n) and several reactions (p)These reports are entered in a database and transmitted to

EudraVigilanceQuantitative methods of signal detection rely on the principle of

disproportionality i.e. methods compute the proportion of a given reaction for a given drug and COMPARE this proportion to the proportion observed with this reaction for all other products observed in the database

Idea: If increased = signal of disproportionate reporting (potential signal) (highlights a reported association).

2

Quantitative methods- Stem from classical methods used in epidemiology, measures of

association: OR, RR and work on exactly the same principles.- The content of the database is collapsed in a contingency table- Measure of disproportionality: PRR, ROR and RRR- Confidence measures: 95CI PRR, etc …- Two methods of computation:- Report level: One report will only be counted once (allocated to the product and

reaction of interest)- Drug-event level: ALL nxp records in the reports will be used in the computation- IMPORTANTLY: SDR DOES NOT MEAN PRESENCE OF A SIGNAL

3

What is the MASKING effect associated with these methods?Effect which is poorly understood by which the information contained in

the database for a given MASKING product will attenuate the strength of a REAL effect associated with another MASKED product for which a disproportionality analysis is conducted.

Therefore, we are facing two problems:- Quantify the extend (and magnitude) of the masking effect in the entire database (problem 1).- For a given disproportionality analysis for a reaction (E) and a product (P) identify the product which will induce the highest masking effect to remove it from the analysis and (hopefully) detect new (true) signals (problem 2).

4

Difficulties

Problem 1: Relies heavily on an (arbitrary) definition of what constitutes a masking effect (or what is a masking product).

Problem 2: Much more important in terms of signal detection.- If we were having an OBJECTIVE way to identify the highest

masking product, quantify the magnitude of its masking effect- Then: try to detect NEW SIGNALS (TRUE EFFECTS) by removing it

from the analysis.However: Little evidence so far of the benefits of removing a

masking product (or situations in which its removal is beneficial).

5

Our approach- Develop a mathematical algorithm for the quantitative methods used in

EudraVigilance (both PRR and 95CIPRR)- Test and validate this algorithm from a statistical standpoint- Test and validate this algorithm on real spontaneous reporting databases- Assess the practical implications of the implementation of this algorithm

(computational requirements, method of computation, handling and allocation of reports, prevalence in the database, terms affected by a masking effect, potential consequences associated with masking removal [true / false positive])

- Establish a practical approach to address the masking effect.

6

Masking effect of quantitative methods of signal detection• Effect first described by Larry Gould in 2003• Well known effect poorly understood• No current algorithm aimed at detecting, quantifying the

presence, direction and magnitude of a masking effect• Potential important implications in terms of Public Health (real

signals might be missed)• Decided to develop such algorithm in PROTECT WP 3.7• F Maignen, JM Dogne (EMA/PRAC), M Hauben, E Hung (Pfizer), L

Van Holle (GSK vaccines)7

Study aimed at developing, validating an algorithm for measures of disproportionality and their confidence intervalThe study was conducted in five steps:1. Development of an algorithm for the measures of disproportionality

(PRR, ROR and RRR)2. Validation of algorithm in EudraVigilance (not submitted / not

presented)3. Comparative analysis in two SRS databases (EV / Pfizer)4. Development and validation of an algorithm for the confidence

intervals of measures of DA5. Difference with subgroup analyses (not submitted / not presented)

8

9

Masking effect

The masking is a collateral effect of quantitative methods of signal detection which rely on disproportionality analysis by which SDRs (corresponding to true signals) might be suppressed (hidden) because of the presence of another product in the same database. •Danger: missing some signals or detecting some signals with delayGould has first described the masking effect of disproportionality analysis using the Relative Reporting Ratio (RRR). Masking is incompletely understood. To date, there is no algorithm to tackle its effect in an automated way.

MASKING EFFECT OF MEASURES OF DISPROPORTIONALITYPresentation title (to edit, click View > Header and Footer)10

Masking effect

New updated contingency table: masking product separated from the background of the database

11

Masking ratio

The exact masking ratio (MR) is defined as the ratio of the measures of disproportionality (DA) for A, without and with product B in the database:

12

Computation of the MR at the DEC level

In a report containing n drugs and p events, each nxp drug-event RECORD is treated independently. Each RECORD is allocated to the corresponding cell of the table (depending whether the record contains the product A, the masking product B or the event of interest). This method creates disjoint and independent sets. Masking is constant for a given masking product B and event E.The restriction applied to the database is easy to implement (with the exclusion of the records involving the masking product from the computation).13

Mathematical expressions of the MR: example of the RRRBy definition RRRA(withoutB) = RRRA *MRRRR

Therefore the masking ratio is the value by which the initial RRRA will be multiplied after the removal of the masking product B.The exact MR for the RRR (drug-event level) is equal to:

In general, in most of the large SRS databases the total number of reports involving the masking product is much lower compared to the total number of reports in the database (n2. << n..) 14

Simplification of the MR for the RRR: identify the HIGHEST masking product ONLYIn general n2. << n.. therefore MR for the RRR can be approximated by:

•n21: number of reports involving the masking product B•n.1: total number of reports involving event E.

Therefore, the masking:•is mostly influenced by the proportion of reports involving the masking product (B) for the reaction of interest to the total number of report including this reaction in the entire database.•Is reaction specific for a given (masking product) product B.

15

Masking induced by RRR computed at DEC level

16

Computation of the MR at the report level (EV)Each report is allocated to one cell of the contingency table and only counted once. Pb: allocating the reports to the correct cell of the contingency table.It must take into account the handling of reports containing both the product (A) and the masking product (B). PRIORITY RULES: A > B > other products•Reports containing product A -> Allocated to A•Reports containing product B BUT NOT product A -> Allocated to B (masking product)•Reports neither containing A nor B -> Background of the database (all other reports)17

Computation of the MR at the REPORT LEVEL: ALLOCATION RULES%n2i number of reports containing B but not A%n3i number of reports containing neither A and B

The computation of the masking ratio at the drug event level can pose some practical issues concerning the computation of both and %n3i.

CORRECT ALLOCATION? With this scenario the reports containing both A and B are (and should be) allocated to A but in theory, these reports could also be allocated to product B.

18

Issues associated with the computation of the MR at the report level• The computation of the masking ratio depends on the product

of interest A, the masking product B and the event under consideration E

• COMPUTATIONAL DIFFICULTIES: COMPUTATIONALLY DEMANDING approx. quadratic function of the number of drug-event combinations in the database

• Limiting step: building and calculating the values in the contingency table

• Need for simplification

19

Mathematical expression of the MR for the PRRLikewise the MRPRR is equal to:

1) The MRPRR is drug A-masking drug B-event E specific (i.e. %n21/%n31 and n2./n3.)

2) Therefore computationally demanding (build the contingency tables)3) Simplify the algorithm, make it masking drug B – event specific and relax

the allocation of reports containing both A and B (double allocation).

20

Simplification of the algorithm for the other measures of disproportionality (1)The idea is to make the masking ratio REACTION SPECIFIC (identical for a given masking product B for a given reaction E)•Firstly, the total number of reports involving the masking product B (n2.) will represent a very small subset of reports in the entire database (compared to n3. or n..). n2. << n3.

•Secondly, the total number of reports involving the product A for the reaction of interest (n11) would be low compared to the total number of reports containing the reaction of interest (n31 or n.1). n11<< n31 or n11 ~ n11 + n31

21

Simplification of the algorithm for the other measures of disproportionality (2)Double allocation of the reports containing both products A and B:•Finally, the proportion of reports containing both the product (A) of interest as well as the masking product (B) would also remain low•These reports could be allocated to both products to simplify the computations.

22

Mathematical expressions of the MR

Presentation title (to edit, click View > Header and Footer)23

Influence of the (double) allocation and % of reports containing both A and B: simulationEudraVigilance does not provide a standardised setting to study the influence of the allocation of reports containing both A and B either to A or the B:•Variable number of reports involving A and B and both. No “extreme” circumstances•Variable % of reports containing both A and B and uneven distribution across the database•Usually low % of reports containing both products•Difficult to assess the effect of the size of the databaseWe have performed a simulation study aimed at circumventing these methodological issues.

24

Simulation study• Range of values for product A, for product B and for the overall size of the

database• Range of values for the % of reports containing both A and B• The number of reports in common was applied to the smallest value of A or

B and rounded to the nearest integer• The number was deducted to B (reports allocated to A)• The number was deducted to A (reports allocated to B)• More than 2,000,000 contingency tables and 42 million computations.• PRRA, PRRB, L95CIPRRA, MR, unmasked PRRA, calculated PRRA (3 methods of

allocation A, B, A + B), difference between unmasked and calculated PRRA.

25

Two possible approaches for simulation (No reports for A and B held constant or not)No reports for A and B not held constant (approach chosen)

No reports for A and B held constant

26

ProductA

ProductB

ProductA A

ProductA

ProductB

ProductA

ProductB A

ProductsB

=

ProductB

ProductsB

Influence of the allocation of reports containing both A and B: simulation

Presentation title (to edit, click View > Header and Footer)27

Allocation of reports to product A

• The calculated PRRA was identical to the unmasked PRRA obtained after the removal of the masking.

• an increase in number of SDRs (No reports > 3 and a lower bound of 95CI > 1) after removal of the masking effect. The number of SDRs observed after removal of the masking effect induced by B was 1,090,656 SDRs

• net gain of 52,589 SDRs or 4.9% of SDRs• Masking increases # MR• PRRA increases with % of reports containing both A and B.

28

Effect of the allocation of reports containing both A and B

29

Evolution of PRRA: Allocation of reports containing both A and B to product A

30

Masking: Allocation of reports containing both A and B to product A

31

Masking: Allocation of reports containing both A and B to the MASKING product B• The masking ratio loses its ability to predict the presence,

direction and magnitude of the masking effect: the unmasked PRRA is only equal to the predicted one when the two products have not reports in common

• Important loss of SDRs (after removal of the masking). The number of SDRs observed after removal of the masking effect of product B was 1,017,795 (corresponding to a net loss of 45,272 SDRs or 4.4% of the SDRs). The difference in SDRs between the two methods of allocation (allocation to product A vs allocation to B) is a net loss of 72,861 SDRs.

32

Masking: Allocation of reports containing both A and B to the MASKING product B• Loss of the ability to predict the presence, direction and

magnitude of the masking effect

33

Masking: Allocation of reports containing both A and B to the MASKING product B• Loss of SDRs (SDRs associated with low case counts 3 – 5

reports mostly affected).

34

Double allocation of the reports• Reveals the same number of SDRs• When less than 50% of reports contain both products, the

approximate MR provides a satisfactory estimate of the exact MR. • The size of the database mitigates the under and over-estimation

of the exact MR. The under or overestimation is exclusively observed for small databases (i.e. number of reports lower than 100,000)

• when the database reaches a size of 100,000 to 1,000,000 reports, the approximate MR consistently overestimates the exact one (maximum twice its real value).

35

Double allocation of the reports

• When a high proportion and volume of reports contain both products A and B compared to the number of reports in the database (i.e. n31 or n32), the double allocation can lead to a dramatic overestimation of the exact MR or to rare computational issues (i.e. in 78,245 or 0.04% of the total number of computations).

• Drug-event pairs involving a low number of reports were less affected by these computational difficulties.

• In all cases, the approximate ratios consistently identified the highest masking product (identified by the exact ratio).

36

Double allocation of the reports

37

Double allocation of the reports: correlation between approx and exact MR

38

Masking function (2 variables)

• fct of n21/n.1 (x)• And n2./n.. (y)• Buffering zone • n21/n.1 = 0.3 – 0.7• MR = 2 when n21/n.1 = 0.5• f(x,y) -> ∞ as n21/n.1 -> 1

39

Estimation of the masking effect

• Effect can be estimated using

40

Key messages part 1• A masking ratio can be used to identify and quantify the masking effect

associated with the measures of disproportionality. • The method of computation (at the report or at the drug-event level) has a

dramatic effect on the masking mechanisms and on the number of computations.

• Simple approximations to the above masking ratio are demonstrated to be valid for large and diverse databases provided that underlying assumptions on the size of the database are verified: identification of highest masking product

• For any event, the strongest masking effect is associated with the drug with the highest number of records (or reports excluding the product of interest).

41

COMPARATIVE ANALYSIS IN TWO LARGE SRS DATABASESPresentation title (to edit, click View > Header and Footer)42

Analyses in EudraVigilance and Pfizer database

• Comparative analysis in two SRS databases: EudraVigilance and Pfizer (based on hypothesis by L Gould that masking could affect more Companies’ databases) Conducted in April 2011.

• Terms selected on the basis of:• Seriousness: Set of MedDRA terms important to PhV (EU-ADR)

and DMEs (commonly reported to EV).• Frequency of reporting to EV: Events rarely reported to

EudraVigilance (less than 100 reports) have also been included in the study.

43

MedDRA terms included in study

• Type of products: Both NCEs (dopaminergic agonists, antiretrovirals) and biologicals (including mAb, vaccines, clotting factors, etc …).

• Masking effect. Used the approximate MR to quantify the masking induced by the HIGHEST masking product.

44

Masking observed in EV• 30,645 drug event combinations (DECs), 29,245 DECs EU-ADR events,

1,400 DECs involved our additional set of events which have been rarely reported in EV.

• Masking: Approximate MR > 1 for 18,599 masking drug-event combinations (MECs) i.e. 61% of the DECs.

• MR > 1.1 for only 87 MECs (0.5% of MECs for which the MR is above 1), • MR > 1.5 for only 28 MECs (0.15%) • MR > 2 for only 20 MECs (0.1%). • All the drug-event combinations actually affected by an important

masking effect involved events rarely reported in EV

45

Highest masking effect• Induced by products for

which the reaction is known

• “Carry-over” effect (masking present induced by products removed from the market)

46

Removal of the masking effect

The removal of the masking effect has revealed 974 new signals of disproportionate reporting (SDRs, defined by a new PRR above or equal to 2). Number of (SDRs) before the removal of the highest masking product was 12,861 (i.e. 42% of the DECs included in our study)Number after removal increased to 13,835 (i.e. increase by approx. 3%).

47

DECs mostly affected• Mostly events rarely

reported• Contains some signals

of Public health importance (PML natalizumab – known in 2011)

• Mostly known signals, handful of unknown signals

48

DECs affected by masking

Presentation title (to edit, click View > Header and Footer)49

Nbre SDRs revealed

• Unclear relation between No (or proportion) of SDRs revealed and magnitude of masking

• Nbre of SDRs revealed # frequency of reporting of the event to EV

50

Nbre of SDRs revealed

51

Comparison with Pfizer db• Reveals structural

differences between the two databases:

• Products for which the Cy holds a license

• Influence of consumer / non-serious reports?

52

Comparison with Pfizer dbConsequential masking more prevalent in Pfizer db than in EV (confirms suspicions raised by L Gould)

53

Effect of removing the masking effect on the ranking of SDRsProvided that the % of reports that the two products have in common remains low, the ranking of SDRs is marginally affected by the removal of the masking effect induced by the HIGHEST MASKING PRODUCT.

54

Key messages 2• Our estimate of prevalence of significant masking showed that the

phenomenon may be rare.• An important masking effect was consistently associated to products

known to induce the reaction. • Masking mainly (but not only) affected events rarely reported in our

large spontaneous systems databases.• Differences affecting important medical events were observed between

EudraVigilance and Pfizer database. • The original ranking provided by the quantitative methods included in

our study was marginally affected by the removal of the masking product.

55

MASKING EFFECT ASSOCIATED WITH THE CONFIDENCE INTERVALS OF MEASURES OF DISPROPORTIONALITYPresentation title (to edit, click View > Header and Footer)56

Masking effect associated with L95CISimilarly to the MR for measures of disproportionality, we have defined a MR for the L95CI:

Which gives:

57

Masking effect associated with L95CI

58

Masking effect associated with L95CITherefore:• There is a direct mathematical /

statistical relationship between the masking effect associated with the measures of disproportionality and their corresponding confidence intervals.

• However, the “multiplication factor” adds an element of complexity in this relationship: no simplified algorithm.

59

Shape of the masking function

. The shape of the masking function differs according to the No of reports involving the reaction of interest (n.1) and proportion of reports involving A (n11/n.1) (LHS 10, 50%, RHS 50, 10%)

60

Masking: extent and comparison between the PRR and Lower95CI• Our simulation originally yielded 905,091 SDRs with the PRR

and 1,038,067 with its Lower95CI. • The removal of the masking resulted in a gain of 77,036 SDRs

with the PRR (an additional 8.5% SDRs) and 68,900 SDRs with the Lower95CI (an additional 6.6% SDRs).

• The removal of any effect (masking or revealing effect) resulted in a net gain of approximately 5% new SDRs for both methods.

• Any masking (approx. 60% DECs) and 30% affected by masking > 10% (MR[CI] > 1.1)

61

Masking: extent and comparison between the PRR and Lower95CI

62 PRR Lower95CI

High overlap between the SDRs unravelled by the masking between the two methods

63 PRR Lower95CI

Relation between MR PRR and MRCI fct of n11. (purple n11 = 1 to blue n11 = 10,000). The masking ratio of the confidence interval is influenced by the number of reports of the product (A) on which the disproportionality analysis is conducted (different colour lines). For an identical masking effect observed with the PRR, the masking ratio associated with the corresponding lower bound of the 95% confidence interval will decrease as the number of reports containing product A increases. Our simulation confirms that the masking product inducing the highest masking effect on a given drug-event pair for the PRR will also be the product inducing the highest masking effect for the confidence interval. 64

Relation between MR PRR and MRCI

65

Relation between MR PRR and MRCI

66

Proportion of SDRs revealed

67

Key messages 3

• The quantification of the extent, direction and magnitude of the masking effect associated with the confidence intervals (CI) of measures of disproportionality (MD) can be automated.

• There is a direct relation between the masking associated with the MD and their respective CI. Products inducing the highest masking effect for the MD will induce the highest masking effect for the CI.

• Removal of important masking is likely to remove a high proportion of common drug-event pairs for the MD and their CI.

68

What shall we retain from our studies?• First algorithms tested and validated aimed at detecting,

quantifying the presence, direction and magnitude of a masking effect (MD and CI). Computationally demanding.

• Confirm a lot of results obtained in the past empirically:• Masking induced by products known to induce the reaction• Higher prevalence of masking on SRS databases from Companies

• Prevalence of masking seems to be low in LARGE SRS databases.• Seems to affect (but not only) events rarely reported in the

database.• However, not tested on smaller databases with different pattern of

products.69

What shall we retain from our studies?

• Simplification of algorithm for MD might not be possible for CI +++

• We did not VOLUNTARILY characterise the SDRs unravelled by the removal of the highest masking effect.

• The real Public Health impact of removing a masking effect needs to be further quantified using PROSPECTIVE studies (methodological challenges incl. need to perform a blinded adjudication of SDRs).

• The removal of the masking must be dictated by the rate of true positive / false positive unravelled by the masking.

70

Perspectives / future work

• Three articles submitted to PDS so far• Assessment of the Public Health impact of masking• Influence of stratum specific in case of stratified analyses• Possible comparison with other methods of adjustment (logistic

regression / ROR)• Collaboration with University of Bordeaux (A Pariente) and GSK

(L Van Holle)

71

Conclusion

• Masking was so far poorly understood: major work and major results

• First algorithm with a demonstrated link between the MD and their CI

• Potential Public Health benefits (still to be demonstrated via well designed prospective studies)• Benefit in terms of new signals?• Benefit in terms of time gained to detect a signal?• Or both?

72

#?

email@francois.maignen@ema.europa.eu

73