Washington Weighted WAC
Survey Final Report
Summary of Focus Groups, Survey Methodology and Results
December 2016-April 2017
1
TABLE OF CONTENTS
FOCUS GROUP DATA COLLECTION ................................................................................................................ 2
SURVEY SAMPLING METHODOLOGY .............................................................................................................. 5
SURVEY RESULTS AND ANALYSIS ............................................................................................................... 11
2
FOCUS GROUP DATA COLLECTION
In December 2016 and January 2017, Public Consulting Group, Inc. (PCG), in partnership with Washington
Department of Early Learning (DEL), Thrive Washington, and local regional coalitions, facilitated 14 focus groups
across the state of Washington. The goal was to collect input from key stakeholders in the field of early childhood
education on which regulations within the Washington Administrative Code (WAC) should be included in a survey
to be weighted in accordance with best practices for differential monitoring.
This report provides a high-level summary of the methodology and findings from each of these efforts.
Methodology
Thrive Washington, a locally-based nonprofit early childhood advocacy group, reached out to the 10 Early Learning
Regional Coalitions (ELRC). The ELRCs are geographically organized, structured networks of early learning
stakeholders. ELRC leadership was responsible for logistics surrounding the focus group, including date and time,
location, invitation list, and follow-up with participants. The ELRCs reached out to a representative sample of 20
early childhood stakeholders from their regions to participate in WAC Survey focus groups, taking into account
occupation (provider types or community stakeholders), languages spoken, and demography. To verify that focus
group participants were representative of the larger state, PCG tracked focus group participation by geography,
stakeholder type, and demographics; additional focus groups were added to meet statewide representation goals.
Focus Group Table 1. Focus Group Distribution
Focus Group Number
of Focus
Groups
ELRC Region 11
Pierce County 1
Northeast 1
Central 1
Northwest 1
Olympic-Kitsap Peninsulas 1
North Central 1
Southwest 1
King County 2
Southeast 1
West Central 1
Parent Group (in King County) 1
Head Start/ECEAP Providers (in King County) 1
WA Department of Early Learning Licensors 1
TOTAL Number of Focus Groups 14
Through the 14 focus groups, 198 stakeholders participated. Below is a breakdown of representation and the
percentage of the total stakeholder group.
Focus Group Table 2. Focus Group Participation by Stakeholder Type and Demographics
Focus Group Participant Representation Count Percentage of
Total
Stakeholder Type
Center Providers 71 36%*
Home Providers 69 35%*
3
Focus Group Participant Representation Count Percentage of
Total
Union Members 44 22%
Tribal Providers 10 5%
Other Stakeholders 58 29%*
Demographics
American Indian or Alaskan Native 10 5%
Asian 7 4%
Black or African American 17 9%
Hispanic 49 25%
White 87 44%
Other 1 >1%
Prefer not to answer 1 >1%
Did not participate in optional survey 24 12%
Total 198 100% *The total of Center Providers, Home Providers, and Other Stakeholders is 100% or 198 participants. Union Members are home providers; this percentage is
representation of the total participants. Tribal Providers could be either center or home providers; this percentage is representation of the total participants. Focus
Group Participant Representation information was gathered through voluntarily, anonymous surveys taken by participants.
Focus groups were primarily conducted in English; however, interpretation services were offered at selected focus
groups for the following languages: Spanish, Somali, and Vietnamese. Written instructions were also translated into
Spanish and Somali for participants’ use; materials were translated into Russian, but no focus group participants
requested Russian interpretation supports.
PCG staff members and DEL representatives facilitated the focus groups. For consistency, all focus groups followed
the same format. After a short introduction to the event, participants were briefed on the methodology behind
differential licensing and weighted regulations. Participants were informed the purpose of the evening was to
determine which WAC regulations should be included in a subsequent survey for weighting. Given the length of the
WAC, participants were randomly assigned to one of three sub-groups to review a portion of the WAC. Sub-groups
were responsible for reading each WAC regulation and determining the value of including the WAC on the survey,
based on the perceived level of risk of harm to children. Then a representative of the sub-group recorded the final
decision on a master copy of the regulations by marking a check next to the specific regulation to include, or by
marking an X to exclude. At the end of each focus group, the master copy from each sub-group was collected by
PCG for analysis.
The length of each focus group was approximately three hours, with sub-group review of regulations accounting for
approximately 1½ hours of the time. At the end of the session, the three sub-groups were allowed to share their
decisions and methodology with the larger group, and answer questions from the larger group. Methodology for
decision-making on whether to include the WAC regulation on the survey included: democratic rule, erring on
including the WAC if there was a prolonged discussion or split vote, and excluding regulations appearing as
definitions or the DEL-specific provisions1.
As a final exercise, focus group participants were asked to recommend stakeholders to take the Weighted WAC
Survey. A variety of the recommended stakeholder groups were included in the pool from which survey respondents
were to be randomly selected.
Results
1 DEL specific provisions are classified as the duties and tasks DEL is responsible to complete. Examples include required paperwork DEL
must complete during the licensing process or the timeline in which DEL must complete inspections.
4
PCG reviewed all sub-group master copies and recorded the sub-groups’ recommendations in an Excel-based tool.
Recommendations for survey inclusion were tallied for each WAC regulation.
PCG determined, in accordance with focus group recommendations, that the following WAC regulations would be
excluded from the survey:
Definitions
DEL provisions and roles1
Provider provisions2
These types of regulations are unrelated to risk throughout the monitoring process.
PCG included specific regulations in the survey based on the majority of recommendations. In the event of a tie,
where the tallies of “include” and “exclude” recommendations were equal, PCG included the WAC in the survey.
The table below outlines the inclusion and exclusion trends.
Focus Group Table 3. Themes and Trends of Focus Group Results
Include in WAC Survey: Exclude from WAC Survey:
Written documentation of policy Definitions of terms
Documentation of child’s health and wellness DEL provisions1 and role
Job descriptions and qualifications for those in
direct contact with children
Provider provision2
Trainings related to health and safety;
emergency drills and preparedness
Written documentation of events, such as
trainings or absences
Illness, contagion, immunizations Professional development related to business
practices
Cleaning and sanitation Dietary requirements, such as menus and health
level of foods
Eating and food preparation Space requirements
Equipment and materials in child environment Curriculum requirements and philosophy of
education
Staffing, ratios, and supervision Play and creativity requirements
Special needs accommodations
Discipline policy, threat of harm, and bullying
Parent communication
As a result of the focus group recommendations, the Weighted WAC Survey was narrowed to 450 questions. The
following table illustrates the percentage of WAC regulations that will be included in the survey.
Focus Group Table 4. Percentages Included in Survey, by WAC Section
Section of WAC Percentage to
Include in Survey
Intent and Authority 29%
Child Outcomes 100%
Family Engagement and Partnerships 80%
Professional Development, Training, and Requirements 83%
Environment 92%
Interactions and Curriculum 54%
Program Administration and Oversight 56%
2 Provider provisions are regulations containing allowances. An example include a provider’s allowance to have pets. This permission is not a
standard one could violate and therefore risk of harm to children, if violated cannot be assessed (weighted).
5
SURVEY SAMPLING METHODOLOGY
Public Consulting Group, Inc. (PCG), a management consulting firm dedicated to working with the public sector, is
contracted to independently design, implement and analyze the Washington Weighted WAC Survey (Washington
Administrative Code for Child Care Regulations) on behalf of Department of Early Learning (DEL). DEL, in
collaboration with an Advisory Committee on Standards Alignment, has been undergoing a standards alignment
process3 since the fall of 2015. As part of the alignment, PCG gathered stakeholder input to design a survey to
assign risk “weights” to each regulation in the new draft proposed aligned standards issued in November 2016. 4
Risk weights will be used in a risk-based assessment for health and safety licensing and monitoring processes.
DEL will finalize the new risk-based assessment processes following final ruling on the aligned rules, pilot, review
and training. The risk-based assessment process will be a component of DEL’s differential monitoring model ‒ one
that is steeped in national best practices for child care licensing and monitoring.5
To engage stakeholders in all steps of the alignment and weighted standards processes, and to design a more
focused survey of key regulations and determine stakeholders for participation, PCG worked with Thrive
Washington (Thrive) and the state’s Early Learning Regional Coalitions to hold 14 statewide focus groups. Focus
group results drove the survey sampling methodology and survey design6.
Survey Design
The weighted WAC methodology adopted by DEL and PCG is founded in Dr. Richard Fiene’s national best practices
model for child care weighted risk-assessment.7
Risk Assessment is an approach that focuses on identifying and monitoring those rules that place children at greater
risk of mortality or morbidity if violations or citations occur.
The methodology indicates that state child care regulatory agencies put all child care regulations into a survey
format for risk-weighting by key stakeholders in the early childhood education and child care health and safety
community. The survey should be disseminated to at least 100 individuals, more if the state has over 3,000 licensed
facilities in the type of services being surveyed. DEL and PCG determined that not only should more than 100
individuals be included in the survey sample, but that the sample should be statistically representative of child care
health and safety experts in the stakeholder community. In other words, the results should have a high level of
reliability that the resulting weights for each regulation are representative of all stakeholders’ assessments of risk.
The survey was made available for a three-week period, from February 20 to March 13.8 PCG prepared frequently
asked questions (FAQ) and a live-recorded webinar to offer information about the process, survey methodology
and logistics. Both the FAQ and webinar recording were made available to all participants and posted on DEL’s and
Thrive’s websites. In addition, a dedicated email address and toll-free hotline were made available to participants
throughout the survey to provide participants with technical assistance. Assistance with interpretation and IT needs
was provided by the Early Learning Regional Coalitions in partnership with Thrive.
3 Washington State Department of Early Learning. Standards Alignment. https://www.del.wa.gov/government/standards-alignment 4 Washington State Department of Early Learning. Proposed Aligned Standards. https://www.del.wa.gov/government/proposed-aligned-
standards 5 Administration for Children and Families, Office of Child Care. Contemporary Issues in Licensing, Monitoring Strategies for Determining Compliance: Differential Monitoring, Risk Assessment, and Key Indicators.
https://childcareta.acf.hhs.gov/sites/default/files/public/1408_differential_monitoring_final_1.pdf 6 Washington State Department of Early Learning. Weighted WAC Survey Focus Group Summary https://www.del.wa.gov/sites/default/files/public/Weighted_WAC_Survey_Focus_Group_Summary.pdf 7 Fiene, R., Kroh, K. Human Service Licensing Measurement, Regulatory Compliance, and Program Monitoring Systems https://www.researchgate.net/publication/302876821 8 Initially, it was made available to participants for a 2-week period, from February 20-March 3, 2017. A one-week extension was provided to
allow sampled participants more time to complete.
6
Survey Questions
The WA Weighted WAC Survey was designed to ask participants to designate the weight—or level of risk—of the
proposed licensing rules of harm to children on a scale of 1 (lowest risk) to 8 (highest risk) if the rule is NOT followed.
In other words:
If the proposed WAC will not be followed by a child care provider, how likely is it on a scale from 1 (lowest
risk) to 8 (highest risk) that a child will be directly and/or indirectly harmed?
“Weighted”: means an assigned level of importance or value to each proposed licensing rule. The more “weight” is
assigned, the riskier the violation of the rule is and the more likely that children will be directly and/or indirectly
harmed if the rule is not followed. A higher weight also means that the level of harm could be higher.
In December 2016 and January 2017, PCG, in partnership with DEL, Thrive, and the Early Learning Regional
Coalitions, facilitated 14 focus groups across the state. The focus group participants provided input on which
stakeholders should be included as survey participants and which regulations should be included as questions for
weighting on the survey. Regulations that were not included in the survey were those that focus group participants
felt were “little to no risk” of harm to children if violated. The regulations not included will be calculated at a risk level
of 1. The resulting survey narrowed the regulations to be included on the survey from approximately 800 to 450
questions (when each regulation and sub-section were split into a single question).9
Sampling and Stakeholder Groups
PCG also took feedback from the statewide focus groups to determine the stakeholders that should be invited to
participate in the survey. Focus groups determined two survey populations: “experts” in child care health and safety
regulations—people who are very familiar with the impact of regulations on children, providers and licensors—and
other stakeholders. Other stakeholders are those who have a vested interested in health and safety but are not
necessarily proficient in all child care regulations and the impact on children.
Group 1: “Early Learning Experts”
o Providers: center, home-based, ECEAP, subsidy, Early Achievers, Head Start, Homeless child care
subsidy
o DEL Child Care Licensors
Group 2: Other Stakeholders
o Parents
o Professionals: medical, colleges, health and safety professionals and practitioners
Focus group participants believed “experts,” being well-informed on the regulations, should make up the vast
majority of the overall survey responses (at about 85%), while other stakeholders should make up a smaller portion
(approximately 15%).
Focus group participants stated that it would be difficult to recruit other stakeholder to participate due to the length
of the survey. Furthermore, stakeholders said some survey respondents could have difficulty completing risk
assessment ratings for all health and safety topics included in the WAC given their experience would fall primarily
outside of early childhood education.
Given the desire to have the results weighted by 85% “expert” and 15% “other stakeholder” and the difficulty of
reaching other stakeholder participants, PCG developed a two-pronged survey collection approach: one for experts
and one for other stakeholders.
9 Weighted WAC Survey Focus Group Data Collection summary
https://www.del.wa.gov/sites/default/files/public/Weighted_WAC_Survey_Focus_Group_Summary.pdf
7
“Experts” Sample
The random sampling methodology sought to reach a random representative sample of “expert” stakeholders
across the state. A random sample methodology is sometimes referred to as “probability sampling,” which means
that nearly every person in the population (included in the stakeholder groups) has an equal chance of being
chosen to participate in the survey.
As described above, DEL and PCG determined that the sample should be statistically representative of child care
health and safety experts in the stakeholder community. PCG designed the sample to meet a 95% confidence level.
The confidence level is a measure of reliability that the result represents the full population. In other words, 95%
confidence level means that if the survey were repeated with the expert stakeholder population, the results would
match the results of the actual full population 95% of the time. This means the average risk rating result for each
regulation cannot be explained by sampling error alone.
The survey was also designed with an acceptable range for the confidence interval for this survey to be 5-7. The
confidence interval provides more assurance about the certainty of the results (in this case risk score) in the
collected data. It is a range of results. In this survey, the confidence interval reflects the range the risk score may
be from the mean sample for a regulation. For example, if the average risk score for a regulation was 5, we expect
results to be within 4.55 and 5.45; in this example, both ends of the range, when averaged to the whole number,
equate to a risk score of 5.
Based on prior experience with similar surveys and the active early childhood education stakeholder community in
Washington, PCG assumed a 25%-30% response rate of completed surveys. With a total population of 5,111
providers, 358 providers would need to be sampled in order to meet the 95% confidence level and 5 confidence
interval.
To maintain a high level of reliability, PCG was seeking completed responses between a 5 and 7 confidence interval.
This level of reliability ensures that the results represent the population. The following table outlines the total
population, target sample at a 5 confidence interval, total outreach and results. The total responses of completions
and partial completions met the target with a confidence interval of 5.31.
Survey Sampling Methodology Table 1. Total Response Population and Confidence Interval
Stakeholder Group (85% of final sample)
Total Population
Target % of Group
Target Sample (95% confidence level)
Total Number of Participants invited
Results Confidence Interval
Providers 5,111 85% 304 1,217 243 6.14
Licensors 102 15% 54 102 77 5.56
Total "Experts" 5,213 100% 358 1,319 320 5.31
Tracking “Experts” Sub-groups
Participants were assigned identification numbers so that PCG could track responses and conduct targeted
outreach, as well as to maintain participants’ anonymity. Those identification numbers were only used for PCG’s
purposes and will not be shared with DEL or other stakeholders.
In addition to ensuring representation of the sample of the overall expert population, PCG used identification
numbers to targeted follow-up efforts to ensure representation by specific sub-groups and characteristics of the
state’s provider population. The sample results for all characteristics were representative of the full population. The
following tables provide results of the final sample representation by: provider type (center, family), primary
language, county, union affiliation, subsidy participation, Head Start participation, serves homeless children, and
Early Achievers participation.
8
Survey Sampling Methodology Table 2. “Expert” Sample by Subgroup
Provider Subgroup Family Child Care
Center Child Care
Total Total Population %
Sample %
Final Result %
Total 99 144 243 100% 100% 100%
ELC: Central 20 7 27 7% 7% 11%
ELC: King 41 31 72 35% 34% 30%
ELC: North Central 11 3 14 7% 8% 6%
ELC: Northeast 10 14 24 6% 6% 10%
ELC: Northwest 25 15 40 14% 14% 16%
ELC: Olympic Kitsap 3 2 5 3% 3% 2%
ELC: Pierce 13 12 25 9% 11% 10%
ELC: Southeast 10 7 17 6% 6% 7%
ELC: Southwest 6 5 11 5% 5% 5%
ELC: West Central 5 3 8 6% 6% 3%
Language: English 71 73 144 50% 51% 59%
Language: Spanish 25 0 25 13% 13% 10%
Language: Somali/Oromo 7 0 7 5% 5% 3%
Language: Other 3 1 4 3% 4% 2%
Language: Unknown 38 25 63 28% 26% 26%
Head Start Participation 0 7 7 2% 2% 3%
Homeless Subsidy 1 2 3 3% 2% 1%
Accepts Subsidy 92 63 155 69% 69% 64%
Union Affiliation 84 0 84 42% 42% 35%
Capacity: Large (21+) 0 91 91 28% 27% 37%
Capacity: Medium (11-20) 86 6 92 40% 40% 38%
Capacity: Small (1-10) 58 2 60 32% 33% 25%
Other Stakeholder Sample
Because the “other stakeholder group” sample is made up of parents and professionals, there is not a reliable
estimate of all stakeholders statewide that could be included in the survey. In other words, the size of the entire
population is unknown. As described above, focus group participants believed that not all individuals in the
population would be interested or knowledgeable to provide full responses and many are difficult to reach. For all
of the stated reasons, a “convenience sample methodology” was used for the “other stakeholders” sample.
A convenience sample is a non-probability sampling technique that survey researchers use to gather responses
from stakeholders who are readily available, and/or a self-selection of individuals willing to participate. PCG worked
with Thrive and the Early Learning Regional Coalitions to identify parents willing to take the survey. PCG also
worked with DEL to identify professionals in the early learning and health and safety fields. In total, PCG conducted
outreach via phone and email to 211 “other stakeholders.”
Because 85% of the final results would be “experts,” 15%, or 63 individuals would need to make up the “other
stakeholder” sample. In order to most efficiently allocate those 63 remaining responses to each of the different
populations, PCG split evenly among parents and early learning professionals: medical, colleges, and health and
safety professionals.
Participants for the “other stakeholders” sample were also assigned identification numbers so that PCG could track
responses and conduct targeted outreach, as well as to maintain participants’ anonymity. Those identification
numbers were only used for PCG’s purposes and will not be shared with DEL or other stakeholders. The following
9
table provides an overview of the “other stakeholders” outreach and results. The results of 66 respondents
exceeded the 63 target.
Survey Sampling Methodology Table 3. “Other Stakeholders” by Subgroup
Stakeholder Group Est. Size Target % of Group
Target Sample
Total Outreach
Results
Parents 61,401 50% 32 105 29
Professionals 200 50% 32 105 37
"Other Stakeholders" 61,601 100% 63 211 66
Total Responses
Of the 391 total survey responses PCG received, 231 were fully completed while 160 were partially completed. Of
the partial completions, a majority answered at least 100 questions or more, with most of the regulations in the
environment section receiving more than 300 responses. In later portions of the surveys, participants tended to
answer less questions, likely due to either fatigue or they felt that the content was not their specialty (other
stakeholders, in particular). The range of questions saw a high of 383 to a low of 213.
Based on a 95% confidence level, we can calculate that the margin of error for a question with 383 responses was
5%, while a question with 213 responses has a margin of error of 7%. This means that we can be 95% confident
that the whole population of providers, licensing staff, and other stakeholders agree with the risk as assigned
between 5%-7%, depending on how many people responded to the question in particular. Margins of error at this
level are fairly common and determined by PCG as acceptable for the purpose of this survey
Outreach
PCG conducted a series of outreach for both the “expert” and “other stakeholder” groups.
Email Campaign: Series of English and Spanish email notifications and reminders sent to participants
where email was available
o One (1) initial outreach email
o Three (3) reminder emails sent to participants who had not begun the survey
o Three (3) reminder emails sent partially completed their survey with a unique link to access their
responses
o One (1) extension notification reminder email sent to all participants who had not completed the
survey
o One (1) final reminder email sent to all participants who had not completed the survey
Phone Campaign: Series of pre-recorded English and Spanish call notifications and reminders to
participants where a telephone number was available
o One (1) initial outreach call
o Four (4) reminder calls to participants who had not begun the survey or only partially completed
the survey
o One (1) extension notification call sent to all participants who had not completed the survey
Thrive/Coalition Reminders
o PCG provided Thrive with a list of survey participants by ELRC; Thrive then disseminated the
participant list to ELRC leadership. The ELRC reached out personally to participants to provide
reminders to complete the survey, along with making participants aware of the technical and
language supports Thrive offered.
10
Technical assistance was provided by PCG through email support, hotline, and via partnership with Thrive and the
ELRC. Thrive and ELRC were responsible for coordinating translation and interpretive services, along with providing
access to technology needed to complete the survey (internet, computer).
In King County, Thrive and the ELRC worked to set up interpretive and technical assistance to the Voices of
Tomorrow, a group of Somali childcare providers.
Upon completion of the survey, participants could access another link where they could register for an incentive for
completing the survey. The incentive was provided by donation from the Washington State Library, funded by the
Institute of Museum and Library Services; it was a children’s book titled Words are Not for Hurting.
11
SURVEY RESULTS AND ANALYSIS
Data Analysis
In total, the proposed aligned standards of the Child Care WAC include about 700 regulations (when all subsections of each regulation are counted as one).10 254 WAC regulations were excluded from the survey through the Focus Group process, of those, 98 regulations were determined to be definitions or DEL/Provider provisions and authority.11 The 98 regulations, being definitions and statements of authority cannot be violated and therefore enforcement of an assigned weight is impracticable; thus, these regulations were given a weight of 0. The remaining 156 regulations identified by Focus Groups retained their weight of 1.
Survey results for the remaining 446 regulations were aggregated an analyzed using both the mean and median of all participant responses. Mean and median are two ways of determining where the central, or average, value falls in a data set. The mean is the sum of all data points divided by the total number of data points while the median orders all data points in sequential numerical order to determine the middle data point12.
Table 1 outlines the distribution of the mean weights as calculated from the survey responses, categorized by WAC Regulation Section. Also included are the focus group results.
Survey Results & Analysis Table 1. Mean Distribution
Focus Group
Results
Survey Results
WAC Regulation Section Total
Regulations 0 1 1 2 3 4 5 6 7 8
Intent and Authority 38 26 6 6
Child Outcomes 4 3 1
Environment 314 14 20 13 76 135 56
Family Engagement and Partnerships
6 4 2
Interactions and Curriculum 117 11 48 9 29 20
Professional Development and Training
81 9 20 6 19 25 2
Program Administration and Oversight
140 38 58 14 23 6 1
TOTAL Count 700 98 156 0 0 22 127 212 84 0 0
It can be noted the majority of the mean scores cluster between a weight of 3 and 6; with no mean scores from the survey reflecting a weight of 1, 2, or 8. Only one mean score resulted in a weight of 7.
Table 2 outlines the distribution of median weights as calculated from the survey responses, categorized by WAC Regulation Section. Also included are the focus group results.
10 The stakeholders in the 14 statewide focus groups reviewed all regulations and their subsections separately to determine whether weights
should be assigned as a whole or individually as subsections. When WAC regulations are counted as individual subsections there are over 800 WAC. The focus groups grouped and consolidated regulations into 700; some of which are grouped some are subsec tions. 11 Example of a definition rated a 0 is 170-300-0401 (4): the definition of the annual licensing fee; an example of a Provider provision is 170-300-0225 (1): stating a provider may have pets. 12 In an odd number of data points, the middle data point is easy to calculate; for example, in a set of 11 data points, 6 would be the middle
data point. In an even number of data points, the average of the 2 middle data points is taken; for example, in a set of 10 data points, the
average of data point 5 and 6 would be taken
12
Survey Results & Analysis Table 2. Median Distribution
Focus Group
Results
Survey Results
WAC Regulation Section Total Regulations 0 1 1 2 3 4 5 6 7 8
Intent and Authority 38 26 6 2 4
Child Outcomes 4 1 3
Environment 314 14 20 4 28 76 92 59 21
Family Engagement and Partnerships 6 4 1 1
Interactions and Curriculum 117 11 48 1 13 15 19 10
Professional Development and Training 81 9 20 4 10 19 13 6
Program Administration and Oversight 140 38 58 3 19 11 7 4
TOTAL Count 700 98 156 0 0 9 48 132 131 91 35
The distribution of median weights has a wider spread, with weights ranging from 3 to 8. Again, no regulations had a median weight of 1 or 2 resulting from the survey.
Weight Assignment Rational
According to Dr. Richard Fiene’s risk assessment methodology for child care differential monitoring13, the mean or
the median are both acceptable methods of calculating the representative risk weight of a regulation.
Taking the mean of a data set is most representative of data that are evenly distributed. When data are highly skewed14 or where outliers are present, the mean becomes less representative of the central concentration of the data. The mean can also result in an average that is not demonstrated in the original data set. For example, in a bi-modal5 data set where half the participants chose a weight of 8 and half chose a weight of 5, the mean would suggest the weight of the regulation should be a 6, a weight not distinctly chosen by any participant. 15
The median is less influenced by skewness in a data set. Where data is not normally distributed, the median is a better representation of the central concentration of the data sample16. It also assures that the central value is a data point clearly chosen by participants, as opposed to the example above where the mean can reflect a data point not explicitly chosen by participants.
During the analysis of the WA Weighted WAC Survey data, it became clear nearly half (47%) of data collected was skewed in nature. This means for nearly half of the regulations evaluated on the survey, participants’ responses were heavily concentrated around a particular weight or were bi-modal17. Because of this skewness, the mean and median were not in agreement, as they would if the data were normally distributed.
13 Fiene, R., Kroh, K. Human Service Licensing Measurement, Regulatory Compliance, and Program Monitoring Systems
https://www.researchgate.net/publication/302876821 14 Skewness refers to the distribution of data is asymmetrical, rather than a normal distribution that looks like a bell curve. In skewed data sets, the mean and median are not equal; in a normal distribution, the mean and median are equal. 15 Note that with the WA Weighted WAC survey, no outliers were identified. Normally, outlier data points can have a disproportional influence
over the mean. While outliers represent a small portion of the total sample population, outliers can significantly shift the mean away from the
central concentration of the data sample. 16 https://statistics.laerd.com/statistical-guides/measures-central-tendency-mean-mode-median.php 17 In bi-modal distribution, the responses are heavily concentrated around several weights; which means the data was not normally
distributed.
13
Table 3 outlines the number of regulations in the survey where data were normally distributed and where data was skewed in distribution.
Survey Results & Analysis Table 3. Normal vs. Skewed Distribution
Results of Survey Count Percent
Normal distribution (the mean is representative; equals the median)
273 53%
Skewed distribution (the mean is not representative; mean and median different)
209 47%
Total 446 100%
Given nearly half of the data received from the survey was not normally distributed where a mean score would be unrepresentative, statistical best practices indicate that the median is a more representative method for determining the result, or the risk weight, for this data set6.
The best practice of using the median to identify the average also allows for a wider distribution of risk-weights, that more accurately reflects the survey responses. The wider spread of the distribution of median scores seen in Graph 2 when compared to the tight distribution of the mean scores in Graph 1 reflects how the mean scores were highly influenced by skew. The presence of median scores of 7 or 8 in Graph 2 reflects a more accurate representation of the range of data in the survey responses.
Survey Results & Analysis Graph 1. Mean Distribution Survey Results & Analysis Graph 2. Median Distribution
The median scores will be the proposed risk-weight assigned to the proposed WAC draft regulations presented during the public comment period.
020406080
100120140160180
0
50
100
150
200
250
14
Results
Table 1, shown above, details the distribution of the total 700 proposed WAC draft regulations by median risk-weights as calculated from the Focus Group and Survey process for each section. Below, Table 4 provides a summary of the 700 regulations by proposed risk-weight.
Survey Results & Analysis Table 4. Summary of Regulations by Risk-weight Risk-weight 0 1 2 3 4 5 6 7 8
Number of Regulations
98 156 0 9 48 132 131 91 35
Percent of Total (700)
14% 22% 0% 1% 7% 19% 19% 13% 5%
Note: Due to rounding, percentages may not add up to 100%
To begin, 98 regulations, or 14% of the total proposed WAC draft regulations, are designated a proposed weight of 0; these regulations will be exempt from enforcement as they are definitions or DEL and Provider Provisions18.
From the Focus Group process, 156 regulations are proposed as a risk-weight of 1. No regulations received a risk-weight of a 1 or 2 because of the survey. This means 22% of proposed WAC draft regulations are proposed to be rated a 1, and 0% are proposed as a 2.
Proposed risk-weights of 3-8 are results of survey responses. Regulations receiving a proposed weight of 3 come from 3 sections: Child Outcomes, Environment, and Professional Development and Training. These represent 9 proposed regulations, or 1% of the total proposed draft.
Overall, 7% of regulations are proposed to be weighted as a 4, with at least 1 regulation from every WAC section receiving this weight. The heaviest concentration comes from Environment, with 28 regulations.
All regulation sections, except Child Outcomes, contain a risk-weight of 5. The most predominant categories include: Environment with 76 regulations, and Professional Development and Training, and Program Administration and Oversight, each with 19 regulations. Regulations assigned a proposed weight of 5 make up 19% of the total regulations.
Environment, Interactions and Curriculum, Professional Development and Training, and Program Administration and Oversight comprise the categories where regulations were proposed weighted a 6 and 7, representing 19% and 13% of the total respectively.
Those regulations where the median risk-weight was an 8 account for only 5% of the total proposed regulations. These regulations come from Environment, Interactions and Curriculum, and Program Administration and Oversight.
Table 5 shows a detailed look at the percent distribution of risk-weights by WAC regulations section.
18 Example of a definition rated a 0 is 170-300-0401 (4): the definition of the annual licensing fee; an example of a Provider provision is 170-
300-0225 (1): stating a provider may have pets
15
Survey Results & Analysis Table 5. Percent of Risk Level by Section
WAC Regulation Section Total Regulations 0 1 2 3 4 5 6 7 8
Intent and Authority 38 68% 16% 5% 11%
Child Outcomes 4 25% 75%
Environment 314 4% 6% 1% 9% 24% 29% 19% 7%
Family Engagement and Partnerships 6 66% 17% 17%
Interactions and Curriculum 117 9% 41% 1% 11% 13% 16% 9%
Professional Development and Training 81 11% 25% 5% 12% 23% 16% 7%
Program Administration and Oversight 140 27% 41% 2% 14% 8% 5% 3%
Note: Due to rounding, percentages may not add up to 100%
Another method of reviewing the distribution of the proposed risk-weights is by risk level: lower (1-3), medium (4-5), and high (6-8). Table 6 categorizes the number of WAC regulations by risk level. While the percent of lower and medium risk level weights are fairly even, there is a concentration of WAC regulations receiving a higher risk weight. Survey Results & Analysis Table 6. Risk Level Representation
Risk Level Count Percent of Total (700)
Weighted 0 98 14%
Weight: Lower Risk (1-3)
165 23.5%
Weight: Medium Risk (4-5)
180 25.7%
Weight: Higher Risk (6-8)
257 36.7%
Note: Due to rounding, percentages may not add up to 100%
Next Steps
Rule Making (NRM) and public comment process to solicit feedback on both the proposed regulation and risk-weights. The NRM and public comments period will begin in May 2017 and end in October 2017. PCG will partner with DEL in facilitating the NRM and public comment process. Throughout this process, DEL will use continue to develop the new compliance oversight and licensing review process.
www.publicconsultinggroup.com