new york city automated decision system task force ... · (e.g. medical hardship).10 the task force...
TRANSCRIPT
New York City Automated Decision System
Task Force Comments Compilation
This document is a compilation of comments or inquiries sent to the New York City Automated Decision System Task Force during its period of public engagement in Spring 2019. It includes documents from the following entities:
AI Now Institute Brennan Center for Justice
NAACP Legal Defense and Educational Fund, Inc. New York City Council Member Peter Koo
Office of the New York City Comptroller Surveillance Technology Oversight Project - S.T.O.P.
The Legal Aid Society
Published September 18, 2019
Written Comments to the New York City Automated Decision System Task Force 1
June 2019 Government administrative agencies are increasingly turning to technical solutions to aid or supplant decision-making procedures, allocate resources, or mitigate pressing concerns. Yet, many important questions about these technical systems remain unanswered because of the lack of meaningful transparency, accountability, and oversight mechanisms. Some of these important questions include:
● What type of systems are currently in use and what are their capabilities? ● Where are these systems currently and prospectively integrated into government
decision making or other processes? ● Which agencies are using them? ● Who is currently and prospectively affected by these systems? ● How does an individual or community know when they’ve been adversely
affected by a system and how do they challenge these outcomes? ● When are individuals or communities notified about the current or prospective
use of systems and when do agencies consult community members to assess concerns?
At the same time, these technical solutions are being developed by vendors that often lack context specific knowledge about the problem they are attempting to address and are driven by business interests and design standards that are often in conflict with community values or interest (e.g. equity, justice, transparency about decision making processes). Additionally, many of these emerging technologies are “black boxes” that are inscrutable to the government officials that will use the technology as well as the public that is affected by the outcomes. This transparency problem is then compounded by the fact that many vendors also obstruct efforts of transparency or algorithmic accountability through broad proprietary claims, even though there is often no evidence that legitimate inspection, auditing, or oversight poses any competitive risks. 2
1 These written comments supplement oral comments provided by Rashida Richardson, AI Now’s Director of Policy Research, at the May 30 New York City Automated Decision System Task Force Public Forum. 2 See,e.g., AI Now Institute. (2018, September). Litigating Algorithms: Challenging Government Use of Algorithmic Decision Systems. Retrieved from: https://ainowinstitute.org/litigatingalgorithms.pdf.
This dynamic leaves residents and consumers in a position where they must “trust” government agencies and vendors despite pervasive failures that have resulted in significant harms. Some notable and recent failures include:
● A school matching algorithm error that resulted in 144 students receiving false rejection notifications to one of New York City’s coveted high schools. 3
● A facial recognition pilot program on New York City’s Robert F. Kennedy Bridge
that failed to detect any faces within acceptable parameters. Despite this failed test and statements by the vendor that the current version of the technology will not work under the proposed conditions, the MTA continues to use the technologies on other City bridges and tunnels. 4
● A flawed algorithm improperly disqualified several New York small business
owners from receiving food subsidy payments for alleged fraud. 5
These consequences fuel growing concerns about government use of automated decision systems and warrant strong and nuanced policy interventions to create greater transparency, oversight, accountability, and equitable outcomes for New Yorkers. Thus, we offer the following recommendations to the Task Force: First, we encourage the Task Force to use existing recommendations and recent state and local policy developments. Last August, the Task Force received a letter with robust recommendations by a group of researchers and advocates, including AI Now. This 6
letter included detailed policy recommendations based on the provisions of the law and we encourage the task force adopt these recommendations in its final report. Additionally, several US cities and states are considering policy interventions and
3 Amin, R. (2019, April 9). Initially rejected, 144 students learn they were accepted to NYC’s coveted Lab School. ChalkBeat. Retrieved from: https://www.chalkbeat.org/posts/ny/2019/04/09/admissions-error-lab-school-for-collaborative-studies-new-york/. 4 Berger, P. (2019, April 8). MTA’s Facial-Recognition Foray Is a Bust --- Test of technology at high speed on RFK Bridge fails to identify the faces of any drivers. ProQuest. Retrieved from: http://proxy.library.nyu.edu/login?url=https://search.proquest.com/docview/2204499010?accountid=12768. 5 Brown, H. C. (2018, October 8). How an Algorithm Kicks Small Businesses Out of the Food Stamps Program on Dubious Fraud Claims. The Intercept. Retrieved from: https://theintercept.com/2018/10/08/food-stamps-snap-program-usda/. 6 (2018, August 17). Re: New York City’s Automated Decision Systems Task Force. [Letter to the Automated Decision Systems Task Force Chairs]. http://assets.ctfassets.net/8wprhhvnpfc0/1T0KpNv3U0EKAcQKseIsqA/52fee9a932837948e3698a658d6a8d50/NYC_ADS_Task_Force_Recs_Letter.pdf.
solutions to address the risks and challenges posed by government use of automated decision system (“ADS”), and we encourage the Task Force to review these bills and recently passed laws to consider whether these approaches can be adopted to address New York City issues and concerns. For example, Idaho recently adopted a law to 7
mitigate bias concerns related to the use of pretrial risk assessments, and this law explicitly prohibits ADS vendors from asserting trade secrecy claims. 8
Second , we encourage the Task Force Chairs to take a more liberal interpretation of the Task Force’s mandate so that the report includes concrete recommendations regarding issues that are immediately actionable. It is clear from the comments made by the Task Chairs at the April 4, 2019 City Council Technology Committee Oversight Hearing and the subsequent Task Force Public Forums that the Chairs see the expected 2019 Report as an initial step in a multi-year process. While we do not expect 9
the Task Force report to serve as a panacea to all New York City ADS concerns and we understand that some issues may require subsequent action by the City Council, Mayor or other governmental bodies outside of New York City; the Task Force can recommend specific actions agencies can take to address immediately actionable concerns. It is not unreasonable to expect that the report should be able to address these more immediate concerns, like how the public should be informed about current and prospective ADS uses that pose great risks to New Yorker’s civil rights and liberties as well as interim procedures for addressing harms created by current ADS use. Expecting New Yorkers to wait several years to learn about systems that may be affecting them or developing procedures for them to challenge ADS decisions and outcomes is unjust. For example, the Department of Education (“DOE”) appeal process for the controversial New York
7 See, e.g., N.Y. Legis. Assemb. A-6787. Reg. Sess. 2019-2020. (2019) https://nyassembly.gov/leg/?default_fld=&leg_video=&bn=A06787&term=2019&Summary=Y&Actions=Y&Floor%26nbspVotes=Y&Memo=Y&Text=Y ; N.Y. Legis. Assemb. A-7040. Reg. Sess. 2019-2020. (2019). https://nyassembly.gov/leg/?default_fld=&leg_video=&bn=A07040&term=2019&Summary=Y&Actions=Y&Memo=Y&Text=Y . 8 Idaho Legislature. House Bill 118. Reg. Sess. 2019. (2019). https://legislature.idaho.gov/wp-content/uploads/sessioninfo/2019/legislation/H0118.pdf ; see also, Whittaker, M., Crawford, K., Dobbe, R., Fried, G., Kaziunas, E., Mathur, V., West, S. M., Richardson, R., Schultz, J., Schwartz, O. (2018). AI Now Report 2018. Retrieved from: https://ainowinstitute.org/AI_Now_2018_Report.pdf (“Vendors and developers who create AI and automated decision systems for use in government should agree to waive any trade secrecy or other legal claim that inhibits full auditing and understanding of their software”). 9 See, The New York City Council. Retrieved from: https://legistar.council.nyc.gov/MeetingDetail.aspx?ID=685409&GUID=89DCCB04-63D3-4FB7-8691-1297A680B101&Options=info&Search=, https://www1.nyc.gov/assets/adstaskforce/downloads/pdf/Public-Engagement-Forum-4-30-2019.pdf, and New York Law School. (2019, May 30). Mayor’s Office - Algorithms Task Force II. Retrieved from: https://nyls.mediasite.com/mediasite/Catalog/catalogs/mayors-office-algorithms-task-force-ii-2019-05-30.
City high school matching does not allow students to challenge errors by the system, they can only appeal school assignment based on limited and predetermined reasons (e.g. medical hardship). The Task Force report should include specific 10
recommendations on how to modify current appeal procedures to ensure that New Yorkers can challenge ADS use that is currently affecting them. This type of recommendation can be immediately actionable by agencies and should not be viewed as outside of the Task Force’s current mandate and authority. Third, the Task Force should require all City agencies to proactively and publicly release data relevant to assessing bias and discrimination concerns related to current and prospective ADS use, and any agency that wishes to be exempt from this requirement should publicly post an explanation for non-disclosure. For example, the school assignment algorithm used by DOE has been the subject of controversy given the extreme racial and socioeconomic segregation in New York City schools. In 11
response to these growing concerns, the City enacted the School Diversity Accountability Act, which requires the DOE to publicly release demographic data 12
related to school enrollment by individual grade levels and programs within schools. However, enrollment data does not show whether there are disparities in who applies to specific schools and who actually gets in. In order to accurately assess whether the school assignment algorithm contributes to discriminatory outcomes, interested parties must have access to the assignment algorithm’s data (e.g. student choice inputs and matching outputs). This type of data should be proactively and public released by the DOE so researchers, advocates, and New York City families can assess whether this ADS is contributing to segregation in City schools, especially since research shows that where a student goes to school can significantly affect their life opportunities and outcomes. Fourth, the Task Force should provide a rights protective advisory guidance to all New York City agencies on how to interpret and comply with requests for information regarding agency use of ADS pursuant to the New York Freedom of Information Law (“FOIL”). City agencies have much discretion in assessing which documents are
10 New York City Department of Education. Retrieved from: https://www.schools.nyc.gov/enrollment/enroll-grade-by-grade/high-school. 11 Kucsera, J.,& Orfield, G.(2014). New York State’s extreme school segregation: Inequality, inaction, anda damaged future. Los Angeles, CA: The Civil Rights Project. Retrieved from: https://www.civilrightsproject.ucla.edu/research/k-12-education/integration-and-diversity/ny-norflet-report-placeholder/Kucsera-New-York-Extreme-Segregation-2014.pdf. 12 Requiring the department of education to report annually on student demographics in community school districts and high schools. Retrieved from: https://legistar.council.nyc.gov/LegislationDetail.aspx?ID=1946653&GUID=7329D54A-4E94-443D-9411-BCF5CC0C65D8&Options=&Search=.
responsive to FOIL requests regarding ADS. Yet, instead of providing the public with information they have a right to review, some agencies have claimed to not understand the technological capabilities or other relevant information about the technologies they are currently using; improperly claimed exemptions to FOIL; or employ other obstructionist practices that have resulted in Article 78 proceedings (i.e. administrative appeals). Challenging such decisions and practices is resource and time intensive, which can have a chilling effect. Given the gravity of the risks some ADS uses pose, the Task Force should ensure that existing laws, like FOIL, are interpreted in a manner that empowers New Yorkers rather than deter them. Finally, the Task Force should recommend agencies adopt data and decision provenance requirements regarding any data that is collected about individuals and communities and subsequently used in ADS or shared with other agencies for use in ADS. Datasets used to develop or implement ADS can include errors, omissions, and other misrepresentations that can produce biased or inaccurate results. Detailed 13
documentation of the data used by the ADS and decisions made throughout the design and implementation process can help agencies better anticipate or mitigate adverse outcomes as well as bolster due process rights of individuals or communities adversely affected. In fact, the Task Force recommendations or subsequent agency policies can draw from existing federal laws. We specifically encourage the Task Force to review 14
the Computer Matching and Privacy Protection Act of 1988, which regulates the use of computer matching involving personally identifiable records and provides due process rights for individuals seeking to prevent agencies from taking adverse actions without notice and independent verification. 15
13 See, Richardson, R., Schultz, J. M., & Crawford, K. (2019). Dirty Data, Bad Predictions; How Civil Rights Violations Impact Police Data , Predictive Policing Systems, and Justice. New York University Law Review Online. 14 See, Privacy Act of 1974, 5 U.S.C. § 552a. 15 The Computer Matching and Privacy Protection Act of 1988,5 U.S.C. § 552a(a)(8)-(13), (e)(12), (o), (p), (q), (r), (u).
Page 1 of 3
Written Testimony of Ángel Díaz
Counsel, Liberty & National Security Program
Brennan Center for Justice at NYU School of Law
Before the
New York City Automated Decision Systems Task Force
Recommendations Regarding Transparency into City Agencies’ Use of Automated Decision
Systems
May 30, 2019
Good evening, chairpersons and members of the Automated Decision Systems Task Force. My name
is Ángel Díaz, and I am Counsel for the Liberty and National Security Program at the Brennan Center
for Justice. I want to thank you for holding tonight’s public forum. We believe a robust public
engagement process should inform the Task Force’s forthcoming report and recommendations.
The Brennan Center is a nonpartisan law and policy institute that seeks to improve our systems of
democracy and justice. The Liberty and National Security Program focuses on government oversight
and accountability, and ensuring that law enforcement efforts to combat crime and terrorism are done
effectively but without religious or ethnic profiling.
As part of this work, we actively seek greater transparency and oversight of the NYPD’s surveillance
tools, including their use of automated decision systems. The Brennan Center is party to a multi-year
legal dispute with the NYPD to obtain information about the Department’s use of predictive policing
technologies. These systems rely on algorithms to analyze large data sets and generate statistical
estimates about crime. The estimates are then used to direct police resources.
But predictive policing tools have been roundly criticized by civil rights and civil liberties advocates,1
as they often rely on historic crime data that both reflects and recreates decades of biased enforcement
against communities of color.2 Here in New York, even ten years of historic crime data would be
tainted by the Department’s stop-and-frisk program that disproportionately targeted the city’s Black
and Latinx communities.3 Relying on this historic data to inform how police officers are deployed in
the future is likely to result in the same biased policing.
These concerns motivated our decision to file a public records request seeking information about the
NYPD’s testing, development, and use of predictive policing. After the NYPD refused to produce
documents in response to our initial public records request and a subsequent appeal, we sued. A little
over a year later, we received an order from the court ordering the police department to produce many
1 See, e.g., Leadership Conference on Civil and Human Rights, et al., Predictive Policing Today: A Shared Statement of Civil Rights Concerns (Aug. 31, 2016), available at http://civilrightsdocs.info/pdf/FINAL_JointStatementPredictivePolicing.pdf. 2 See, e.g., Jack Smith IV, Crime-prediction Tool PredPol Amplifies Racially Biased Policing, Study Shows, MIC (Oct. 9, 2016), https://mic.com/articles/156286/crime-predictiontool-pred-pol-only-amplifies-racially-biasedpolicing-study-shows (last visited Oct. 15, 2017); See also Laura Nahmias, NYPD Testing Crime-Forecast Software, POLITICO (July 8, 2015, 5:52 AM EDT), http://www.politico.com/states/new-york/cityhall/story/2015/07/nypd-testing-crime-forecast-software-090820 (quoting maker of predictive policing software as noting the importance of assessing “how we apply statistics and data in a way that’s going to be sensitive to civil rights and surveillance and privacy concerns”). 3 See e.g., Benjamin Mueller, “New York Police Dept. Agrees to Curb Stop-and-Frisk Tactics.” The New York Times, February 2, 2017, https://www.nytimes.com/2017/02/02/nyregion/new-york-police-dept-stop-and-frisk.html.
Page 2 of 3
of the records we had originally requested.4 Even then, it took almost a full year from the judge’s order
before the NYPD finally produced some of the information in our request. While the documents we
ultimately received helped to shed light on the NYPD’s predictive policing system, we still do not
have a full understanding of how it works. We believe this Task Force is uniquely situated to engage
in a comprehensive evaluation of the NYPD’s predictive policing system.
At the hearing for the establishment of Local Law 49, former Council Member and head of the
Technology Committee James Vacca specifically noted the use of predictive policing as a motivating
factor in the establishment of this Task Force.5 We urge you to evaluate how the NYPD can address
and correct any biases in its predictive policing system that may be condemning communities of color
to a lifetime of over-policing.
Despite its numerous problems, predictive policing is just one system currently being deployed
without accountability or oversight. To give just a few more examples, the NYPD uses each of the
following technologies:
• Facial Recognition.6 Studies of many commercially available products have found unacceptable error
rates when analyzing faces that are not white and male.7
• Social Media Monitoring.8 A New York court recently ordered the NYPD to release unredacted
documents relating to how the Department uses Dataminr software to monitor social media.9 This was
in response to a public records request filed by Black Lives Matter activists seeking records about
NYPD surveillance of their social media profiles.
• Automatic License Plate Readers.10 NYPD contracts with a company called Vigilant Solutions for
access to its massive database of license plate reads.11 If the NYPD shares information captured from
its own license plate readers and shares it with other customers of Vigilant Solutions, it may be
unwittingly sharing information about undocumented New Yorkers with ICE.12
We understand that this Task Force cannot work alone in holding the police department accountable.
This is why we recommend that this Task Force’s written report call on the City Council to pass a law
that would require the NYPD to publish a list of its surveillance tools and the policies in place to
4 See Rachel Levinson-Waldman and Erica Posey, “Court: Public Deserves to Know How NYPD Uses Predictive Policing Software,” January 28, 2018, https://www.brennancenter.org/blog/court-rejects-nypd-attempts-shield-predictive-policing-disclosure. 5 See Transcript of the Minutes of the Committee on Technology, October 16, 2017 at Page 8. 6 See Clare Garvie, “Garbage In, Garbage Out – Face Recognition on Flawed Data,” May 16, 2019, https://www.flawedfacedata.com/. 7 See e.g., Joy Buolamwini and Timnit Gerbu, “Gender Shades: Intersectional Accuracy Disparities in Commercial Gender Classification”, available at http://proceedings.mlr.press/v81/buolamwini18a/buolamwini18a.pdf. 8 See Jessie Gomez, “New York court rules NYPD can’t use Glomar to keep surveillance records secret,” MuckRock, January 15, 2019, 9 See Millions March NYC v. New York City Police Department, Index No. 100690/2017, January 14, 2019, available at https://www.documentcloud.org/documents/5684800-Millions-March-Nypd.html#document/p1. 10 See NYCLU “Automatic License Plate Readers,” https://www.nyclu.org/en/automatic-license-plate-readers. 11 See Anthony Romero, “Documents Uncover NYPD’s Vast License Plate Reader Database,” HuffPost, January 25, 2017, https://www.huffpost.com/entry/documents-uncover-nypds-v_b_9070270. 12 See Russell Brandom, “Exclusive: ICE is about to start tracking license plates across the US.” The Verge, January 26, 2018, https://www.theverge.com/2018/1/26/16932350/ice-immigration-customs-license-plate-recognition-contract-vigilant-solutions.
Page 3 of 3
protect the privacy of New Yorkers. The POST Act, Initiative 487, would require just that.13 Cities
across the country, including San Francisco,14 Seattle,15 Nashville,16 and Cambridge17 have already
passed even stronger community controls. It’s time for New York City to catch up.
Last year, we joined a coalition letter to this Task Force with a number of recommendations for how to accomplish its mandate under Local Law 49. We are resubmitting a copy of this letter with our written testimony, but I would like to highlight just a couple of items that pertain to the goals of tonight’s public forum.
• First, contracts with technology vendors must include provisions requiring them to disclose information for all datasets used to develop and implement ADS. This information should be maintained by city agencies, and shared with auditors evaluating ADS for disparate impact.
• Second, each city agency deploying an ADS should publish a simple description of how the system works, and how it makes decisions about New Yorkers. These descriptions should be publicly available, and should account for New York’s linguistic, socioeconomic, and cultural diversity.
• Third, city agencies should create and publish regular Algorithmic Impact Assessments, ideally before acquiring or building a new ADS. These assessments would review each ADS for fairness, bias, privacy, and related concerns.
In closing, our increasingly data-driven society requires that transparency and accountability do not fall by the wayside. The NYPD’s use of surveillance technology threatens to completely redefine the right to privacy, freedom of speech, and equal protection under the law. These foundational values must be jealously guarded if New York City is to remain a strong local democracy. Thank you again for the opportunity to testify. I am happy to answer any questions.
13 For more on the POST Act, short for Public Oversight of Surveillance Technology Act, see “The Public Oversight of Surveillance Technology (POST) Act: A Resource Page, available at https://www.brennancenter.org/analysis/public-oversight-surveillance-technology-post-act-resource-page. 14 See “Stop Secret Surveillance Ordinance,” available at https://sfgov.legistar.com/View.ashx?M=F&ID=7206781&GUID=38D37061-4D87-4A94-9AB3-CB113656159A. 15 See Ordinance 125376, available at https://seattle.legistar.com/LegislationDetail.aspx?ID=3380220&GUID=95404B0E-A22D-434E-A123-B3A0448BD6FA&Options=Advanced&Search=. 16 See Ordinance No. BL2017-646, available at https://www.nashville.gov/mc/ordinances/term_2015_2019/bl2017_646.htm. 17 See Surveillance Technology Ordinance, available at http://2f8dep2znrkt2udzwp1pbyxd-wpengine.netdna-ssl.com/wp-content/uploads/2018/12/surveillance-ordinance.pdf.
Exhibit A
August 17, 2018 Via E-Mail & First Class Mail Acting Director Emily W. Newman Deputy Commissioner Brittny Saunders Mayor’s Office of Operations 235 Broadway - 10th Floor New York, NY 10007
Re: New York City’s Automated Decision Systems Task Force
Dear Task Force Chairs Newman and Saunders:
The undersigned organizations and individuals write to offer recommendations to the Automated Decision Systems Task Force, which is mandated by Local Law 49 of 2018. The Task Force is required to present the Mayor and ultimately the public with recommendations on identifying automated decision systems in New York City government, developing procedures for identifying and remedying harms, developing a process for public review, and assessing the feasibility of archiving automated decision systems and relevant data. This is an important opportunity to ensure that emerging technologies, like automated decisions systems, are adopted and implemented fairly and equitably to serve all New Yorkers.
Though we hope the Task Force will engage experts, advocates, and community members over the next year, we are offering the following recommendations in hopes that they can assist the Task Force in answering the varied and complicated questions mandated by Local Law 49 of 2018. We also anticipate that the Task Force’s prospective findings and recommendations can serve as a national or international model for other jurisdictions grappling with the opportunities and challenges presented by the use of automated decision systems, so we hope this letter can assist other advocates in their local efforts. This letter includes general recommendations for the Task Force as well as specific recommendations related to the provisions of Local Law 49 of 2018.
General Recommendations for the Task Force ● The effects of an automated decision system will vary by agency, as will the intended
goals of the system and the public policy issues the agency seeks to address through use of the automated decision system. As the Task Force evaluates the myriad of issues presented by automated decision systems, we recommend Task Force members consult domain experts and advocates, including but not limited to those listed in the attached appendix, while developing recommendations that relate to or may significantly impact specific issue areas.
1
● The Task Force should recommend the creation of a permanent independent governmental body whose mission is to (1) help implement subsequent laws, policies, or procedures that are created based on Task Force recommendations, (2) handle enforcement against agencies that fail to comply with aforementioned laws, policies or procedures, and (3) assess when laws, policies or procedures need to be amended to reflect advancements in technology.
● While most of the provisions of Local Law 49 of 2018 seek recommendations regarding government use of automated decision systems within the civil law context, the Task Force should recognize that criminal suspects enjoy the protections of the Fourth, Fifth, Sixth, and Fourteenth Amendments. These protections must be satisfied in addition to any others the Task Force might recommend.
Recommendations on the criteria for identifying which agency automated decision systems should be subject to one or more of the procedures recommended by the Task Force
● The Task Force should adopt the following definition of “automated decision system” to determine which systems should be subject to its recommendations on procedures, rules, policies and actions regarding government use of automated decision systems.
○ An “automated decision system” is any software, system, or process that aims to aid or replace human decision making. Automated decision systems can include analyzing complex datasets to generate scores, predictions, classifications, or some recommended action(s), which are used by agencies to make decisions that impact human welfare.
● Agencies should maintain a public archive identifying automated decision systems that are subject to procedures, rules, policies or actions recommended by the Task Force as well as systems and categories of systems (e.g. short-lived Microsoft Excel formulae) excluded from the recommended procedures, rules, policies or actions, and explanations of their exclusion. The City should also implement a procedure for the public to challenge an agency’s exclusion of an automated decision system.
Recommendations on procedures, rules, policies or actions for how a person may request and receive an explanation of how an agency automated decision system determination was reached
● The City Council and the Mayor should provide agencies annual budgetary support to ensure accessibility of public documents and communications related to the agency’s use of automated decision systems. Agencies should ensure that public documents and communications account for language, socioeconomic, cultural, geographic, education, and digital access differences. This budgetary support should also be used to hire and consult group facilitation experts to design, lead and implement public meetings that are centered on soliciting community concerns regarding existing procedures.
2
● The City should require procurement contracts to include provisions requiring the vendor to provide information for all datasets used to develop and implement the systems; plain 1
language explanations of how the system makes determinations; any records of bias, fairness or any validation testing performed on the system; design documentation and information about the technical architecture; records of the vendor marketing materials; plans for ongoing maintenance and system updates; response plans for any system changes that result from updates; and any other relevant information that will assist agencies in developing explanations of how an automated decision system determination was reached and compliance with any other Task Force recommended procedures, rules, policies or actions.
● Agency explanations of an automated decision system determination should include general, plain-language descriptions of the automated decision systems’ overall function, the degree of human intervention in the system, and an explanation of the specific determination in question.
● Agencies should adopt procedures that guarantee an agency response to a request for an explanation of an automated decision system determination within a 20-day time period. Requests for explanations of automated decision system determinations pertaining to critical issues (e.g. public benefits eligibility or allocation) should have a limited response timeline of five business days. Explanations should include a description of the process and timeline to appeal an automated decision system determination.
● The City should require agencies using automated decision systems to maintain and publish metrics regarding how many requests for explanation it received, whether the explanation resulted in a challenge, and the outcome of that challenge. This information can be published in a privacy-preserving manner but it should allow the public and public officials to assess the efficacy and impact of procedures and practices as well as the utility of automated decision systems.
Recommendations on procedures and standards to determine whether an agency automated decision system disproportionately impacts persons based on protected status
● The City should require agencies to develop a pre-acquisition or development procedure to ensure experts and representatives from directly affected communities are consulted during the development of an automated decision system. Agencies should maintain a public record of external participation. Agencies must ensure that non-agency experts are consulted early in the acquisition or development process, since important policy determinations that can result in disproportionate outcomes occur early in system development.
● The explicit expectation is that automated decision systems should not result in a disproportionately negative effect on members of a protected status, and measures should
1 Timnit Gebru, et al., Datasheets for Datasets (March 2018), https://arxiv.org/pdf/1803.09010.pdf.
3
be affirmatively undertaken to eliminate disparities. In furtherance of this goal, the City should require all agencies using automated decision systems to adopt a standard for assessing disproportionate impact based on protected status that is tailored to the specific use of the automated decision system. The following is an example of a general standard agencies can consider and modify for its use cases: if an automated decision system selects or affects members of a protected status at a rate that varies by four-fifths or more, then that decision system should not be used unless the agency provides a public explanation of why its use of the system and the specific decision is necessary to achieve an important agency interest, and that there is no less-discriminatory alternative to achieving this interest available.
Recommendations on procedures and standards for addressing instances in which a person is harmed by an agency automated decision system if any such system is found to disproportionately impact persons based on protected status
● When an agency’s automated decision system is found to be discriminatory or produces discriminatory outcomes, the agency’s policy or system redesign process must include individuals and advocates from the communities or protected class whom the system is found to disproportionately impact. Inclusion of affected individuals and advocates should occur at the beginning of the redesign process and the agency should specifically design pre-meeting preparation sessions for affected individuals and advocates to ensure that they can comfortably and meaningfully participate in the redesign process.
● The City Council should pass a law providing a private right of action for individuals or groups of individuals that are injured by automated decision system determinations that are found to be discriminatory or produce discriminatory results.
● Agencies should define and publicly post a procedure allowing outside researchers or experts access to relevant information to assess whether an automated decision system produces disparities between similarly situated individuals based on protected status.
Recommendations on a process for making information publicly available that, for each agency automated decision system, will allow the public to meaningfully assess how such system functions and is used by the city, including making technical information about such system publicly available where appropriate
● The City should make publicly available online a list of automated decision systems used by agencies, disaggregated by agency. This list should also include:
○ A description of the purpose of the automated decision system, including any decisions that such system is used to make or assist in making and any specific types or groups of persons likely to be affected by those decisions.
○ A description of the procedure for individuals to determine whether and how an automated decision system was used to make a decision that affects them, the
4
procedure for how a person may challenge a decision where an automated decision system was involved, timelines for each procedure, and expected response time from the agency.
○ The degree of human intervention in the automated decision system (e.g. whether a decision-making process is fully automated or if the automated decision system is used for decision-support).
○ Relevant technical information of the system including but not limited to: ■ source code; models; documentation on the algorithms used; design
documentation and information about the technical architecture; training data; data provenance information; some justification for the validity of using a model trained on data from a potentially different context than the agency’s; the system’s intended use as-implemented (e.g. the automated decision system’s actual objective function); any records of bias, fairness or any validation testing performed on the system; materials relating to how a user interacts with a system (including wireframes or documentation on how determinations from the system are displayed and communicated).
○ Any marketing materials and training instructions or materials for public servants using the tool.
○ If a contract with a third party would prevent the agency from releasing such technical information, (i) the name of such third party, (ii) an electronic link to a copy of such contract, (iii) the date that the current term of such contract will expire and (iv) a statement explaining why the contract prevents the agency from releasing such technical information. If no such obstacles exist, a plan for publicly releasing such technical information, including the anticipated date of such release.
○ Policies and procedures relating to access, use of the system or input data, and any safeguards to protect system or input data from unauthorized access or use.
○ Documentation of any other agencies or third parties that have access to the automated decision system or input data.
○ Information regarding audits of such systems, including frequency, scope, and public availability of such audits.
○ A statement on who made policy decisions related to the development of the automated decision system model (e.g. score thresholds, system objectives) and a description of how policy decisions were made.
● There should be no exceptions to making the aforementioned list of automated decision systems information public. If an agency attempts to raise agency-specific concerns that would prevent releasing of all or some technical information, the City should require the agency to provide a detailed statement regarding the need for the limitations and review
5
of the information that can be released without revealing sensitive agency data or resulting in the described concern (e.g. historical input data, testing procedures, etc).
● The City should develop mechanisms to connect transparency requirements more strongly to enforcement. For example, the City can make some agency funding conditional upon meeting certain standards of algorithmic disclosure and interpretability through external, independent audits.
● Agencies that use or intend to use automated decision system should perform an Algorithmic Impact Assessment , preferably before acquiring or building a new 2
automated decision system. Each agency should perform a self-assessment of existing and proposed automated decision systems, evaluating potential impacts on fairness, justice, bias, privacy, civil rights, and other concerns. Agencies should provide a public notice and comment period of the self-assessment and, mitigate and respond to comments or concerns raised by the public before publicly posting the final assessment.
Recommendations on procedures for archiving agency automated decision systems, data used to determine predictive relationships among data for such systems and input data for such systems
● The City should allow outside experts and researchers access to archived input data and other relevant agency data to identify systemic and structural problems that may derive from agency practices and procedures. The findings can be used to identify optimal policy solutions. 3
● Agencies should document, archive and publicly post a retention schedule for changelogs of modifications made to the source code or models of an automated decision system, plain text describing changes, and agency-internal communication or communication between agency employees and vendors relating to any changes in the decision-making algorithms to understand how the changes affect decisions using an automated decision system over time.
We welcome the Task Force to use the undersigned as resources during this process and look forward to the Task Force’s prospective findings and recommendations. Sincerely,
2 AI Now Institute, Algorithmic Impact Assessment: A Practical Framework for Public Agency Accountability (April 2018), https://ainowinstitute.org/aiareport2018.pdf. 3 For example, data demonstrating that NYPD’s stop-and-frisk practice unlawfully targeted Black and Latino New Yorkers, and that an overwhelming majority of the stops did not lead to evidence of a crime, was only available to a group of legal organizations following racial profiling litigation. If such data were preemptively available for scrutiny then this unlawful practice and subsequent reform could have been identified without costly litigation.
6
Andrew Guthrie Ferguson UDC David A. Clarke School of Law Author, The Rise of Big Data Policing [email protected]
Brandon Holmes & Dylan Hayre JustLeadershipUSA [email protected] [email protected]
Brett Stoudt John Jay College of Criminal Justice [email protected]
Bryan Mercer & Hannah Sassaman Media Mobilizing Project [email protected] [email protected]
Chris Gottlieb NYU Family Defense Clinic [email protected]
Daniel Schwarz & April Rodriguez New York Civil Liberties Union (NYCLU) [email protected] [email protected]
Frank Pasquale University of Maryland [email protected]
Jordyn Rosenthal & Vivian Nixon College and Community Fellowship [email protected] [email protected]
Justine Olderman The Bronx Defenders [email protected] [email protected]
Katya Abazajian Sunlight Foundation, Open Cities [email protected]
Lisa Schreibersdorf Brooklyn Defender Services [email protected] [email protected]
Marc Canellas IEEE-USA Artificial Intelligence & Autonomous Systems Policy Committee (AI&ASPC) [email protected]
Marne Lenox NAACP Legal Defense & Educational Fund, Inc. (NAACP LDF) [email protected]
Megan Garcia New America's National Network [email protected]
Natasha Duarte Center for Democracy & Technology [email protected]
Noel Hidalgo BetaNYC [email protected]
Nora McCarthy RISE [email protected]
Rachel Levinson Waldman & Angel Diaz The Brennan Center for Justice at NYU School of Law [email protected] [email protected]
7
Rashida Richardson AI Now Institute [email protected] Sharon Bradford Franklin New America's Open Technology Institute [email protected]
Taylor McGraw The Bell [email protected]
William Gibney The Legal Aid Society [email protected] [email protected]
Yeshimabeit Milner Data for Black Lives [email protected] Zaps (Sarah Zapiler) IntegrateNYC [email protected]
8
APPENDIX
Recommended list of experts and advocates that the Task Force should consult when developing recommendations that relate to or may significantly impact specific issue areas.
● Children Welfare ○ Individuals: Khiara M. Bridges (Boston University School of Law), Lauren
Shapiro (Brooklyn Defender Services), Christine Gottlieb, Ashley Sawyer (Girls for Gender Equity), Lisa Freeman (The Legal Aid Society), Emma Ketteringham (Bronx Defenders), Michelle Burrell (Neighborhood Defender Services of Harlem), Michele Cortese (Center for Family Representation)
○ Organizations: Council of Family and Child Caring Agencies (COFFCA), Silberman School of Social Work, Youth Represent, Community Service Society of New York, CWOP (Child Welfare Organizing Project), RISE
● Disability Rights ○ Individuals: Beth Haroules (NYCLU), Chancey Fleet (Data & Society), Kathleen
Kelleher (The Legal Aid Society) ○ Organizations: Bazelon Center for Mental Health, New York Association of
Psychiatric Rehabilitation Services (NYAPRS) ● Education/School Choice
○ Individuals: Aaron Pallas (Columbia Teachers College), Genevieve Siegel- Hawley (Virginia Commonwealth University), Susan Eaton (Brandeis University), Claire Fontaine (Data & Society), Monica Bulger (Future of Privacy Forum), Cara Chambers (The Legal Aid Society)
○ Organizations: IntegrateNYC, Teens Take Charge, ASID, Alliance For Quality Education (AQE NY), NYSUT
● Employment/Workers Rights ○ Individuals: Peter Roman-Friedman (Outten & Golden), Annette Bernhardt (U.C.
Berkeley Labor Center), Karen Levy (Cornell), Ruth Milkman (CUNY, Murphy Institute), Louis Hyman (Cornell ILR Worker Institute), Ifeoma Ajunwa (Cornell IRL), Julia Ticona (Data & Society), Aiha Nguyen (Data & Society), Alex Rosenblat (Data & Society), Alexandra Mateescu (Data & Society), Karen Cacace (The Legal Aid Society)
○ Organizations: National Employment Law Project (NELP), Make the Road NY ● Healthcare
○ Individuals: Valerie J. Bogart (NYLAG), Kadija Ferryman (Data & Society) ○ Organizations:Empire Justice, Medicaid Matters NY
9
● Housing
○ Individuals: Jenny Laurie (Housing Court Answers), Magda Rosa-Rios (The Legal Aid Society)
○ Organizations: Picture the Homeless, YWCA Brooklyn, Manhattan Legal Services
● Immigration/Refugee Rights ○ Individuals: Hassan Shafiqullah (The Legal Aid Society), Sarah Deri Oshiro
(Bronx Defenders) ○ Organizations: Immigrant Defense Project, International Refugee Assistance
Project, ACLU Immigrant Rights Project, LatinoJustice ● Law Enforcement
○ Individuals: Brett Stoudt & K. Babe Howell (Public Science, CUNY), Andrew Guthrie Ferguson (University of the District of Columbia Law School), Cynthia Conti-Cook (Legal Aid); Desmond Patton (Columbia University School of Social Work), Lisa Freeman (The Legal Aid Society), Marne Lenox (NAACP LDF)
○ Organizations: Brennan Center for Justice (predictive policing), National Association of Criminal Defense Lawyers (NACDL), Center for Democracy & Technology
● Other City Operations Systems (Sanitation, Parking, 311, SBS/EDC subsidy programs, etc.)
○ Individuals: Anthony Townsend (Bits and Atoms) ● Pretrial
○ Individuals: Nicole Triplett (NYCLU), Vivian D. Nixon (Community College Fellowship), Marbre Stahly-Butts (Law for Black Lives), Dana M. Delger (Innocence Project), Blase Kearney (Public Defender Service), Molly Louise Kovel (ACLU), Joshua Norkin (Decarceration Project- Legal Aid Society), Lisa Freeman (The Legal Aid Society), Scott Levy (Bronx Defenders), Lisa Schreibersdorf (Brooklyn Defender Services)
○ Organizations: Bronx Defenders, Brooklyn Defender Services, Legal Aid Society, Data 4 Black Lives, LatinoJustice PRLDEF, New York Immigration Coalition: NYIC, Vocal New York, JustLeadershipUSA, New York Communities for Change (NYCC), Neighborhood Defender Service of Harlem, National Association of Criminal Defense Lawyers (NACDL)
● Privacy/Security/Surveillance ○ Individuals: Hannah Sassaman (MMP), Vincent Warren & Britney Wilson
(Center for Constitutional Rights), David Robinson (Upturn), Kristian Lum (HRDAG), Michael Price (National Association of Criminal Defense Lawyers); Esha Bhandari (ACLU Speech Privacy and Technology Project), Alvaro Bedoya
10
(Georgetown Law Center on Privacy & Technology), Mary Madden (Data & Society)
○ Organizations: Harvard Law’s Berkman Klein Center, CAIR-NY, Brennan Center for Justice, CLEAR Project at CUNY, National Association of Criminal Defense Lawyers, Center for Democracy & Technology
● Public Benefits ○ Individuals: Richard Alan Eppink (ACLU Idaho), Elizabeth Edwards (National
Health Law Program), Kevin De Liban (Legal Aid Arkansas), Susan Welber (The Legal Aid Society)
○ Organizations: FPWA ● Public Health
○ Individuals: Rodrick Wallace (New York State Psychiatric Institute), Elizabeth Edwards (National Health Law Program), George Annas (Boston University Law), Wendy Parmet (Northeastern University Law), Wendy Mariner (Boston University School of Public Health), Larry Gostin (Georgetown Law), Rebecca Novick (The Legal Aid Society)
○ Organizations: Community Service Society of New York, Center for Democracy & Technology
● Re-entry ○ Individuals: Wesley Caines (Bronx Defenders) ○ Organizations: EXODUS, Fortune Society, Center for Court Innovation, National
Association of Criminal Defense Lawyers (NACDL) ● Sentencing/Parole/Probation
○ Individuals: Beth Haroules (NYCLU) ○ Organizations: Legal Aid Society (Prisoners Rights), Urban Justice Center,
Correctional Association of New York, Center for Court Innovation, National Association of Criminal Defense Lawyers (NACDL)
● Transportation ○ Individuals: Noel Hidalgo (Beta NYC), Aaron Naparstek (Vision Zero), Mandu
Sen (RPA), Sarah Kaufman (NYU Rudin Center) ○ Organizations: Transportation Alternatives, Vision Zero, Regional Plan
Association ● Voting Rights/Political Participation
○ Organizations: Demos, Brennan Center for Justice, ACLU Voting Rights Project
11
New York Office
40 Rector Street, 5th Floor
New York, NY 10006-1738
T 212.965.2200
F 212.226.7592
www.naacpldf.org
Washington, D.C. Office
700 14th Street, NW, Suite 600
Washington, D.C. 20005
T 202.682.1300
F 202.682.1312
Written Testimony of the
NAACP Legal Defense and Educational Fund, Inc.
Submitted to the New York City
Automated Decision Systems Task Force
June 17, 2019
Submitted via email at [email protected].
1
Chairperson Thamkittikasem and Members of the Task Force:
On behalf of the NAACP Legal Defense and Educational Fund (LDF), we
thank the NYC Automated Decision Systems Task Force (the Task Force) for
holding important public forums on April 30 and May 30, 2019 to address the core
components of Local Law 49 of 2018 (Local Law 49) concerning accountability,
fairness, and transparency.
LDF is the nation’s first and foremost civil and human rights law
organization. Since its founding nearly eighty years ago, LDF has worked at the
national, state, and local levels to pursue racial justice and eliminate structural
barriers for African Americans in the areas of criminal justice, economic justice,
education, and political participation.1 As part of that work, LDF has also forged
longstanding partnerships with local advocates, activists, and attorneys to
challenge and reform unlawful and discriminatory policing in New York City,
including serving as co-counsel in Davis v. City of New York, a federal class-action
lawsuit that challenged the New York Police Department’s policy and practice of
unlawfully stopping and arresting New York City Housing Authority (NYCHA)
residents and their visitors for trespassing without the requisite level of suspicion.2
LDF is deeply concerned about the use of data and technology in
perpetuation of racial discrimination, including machine-learning algorithms,
biased data, and Automated Decisions Systems (ADSs) that rely on both. The
NYPD’s deployment and implementation of ADSs threaten to exacerbate racial
inequities in New York City. The potential discriminatory impact of these systems
raises concerns similar to the racially discriminatory and unconstitutional policing
practices that historically motivated—and continue to motivate—LDF’s litigation,
policy, and public education advocacy.
Given these significant concerns, the Task Force’s recommendations to
Mayor de Blasio and City Council Speaker Corey Johnson must ensure that all
ADSs are fair, transparent, and rigorously evaluated. Critically, ADSs must not
undermine the City’s commitment to public safety practices that are constitutional
and non-discriminatory. Equally important, the process for developing these
recommendations must foster robust community dialogue and engagement.
Moreover, this process must include mechanisms to ensure community members
who are directly impacted by these systems have direct input into shaping the
recommendations. While Local Law 49 affects decision-making in a wide variety of 1 LDF, About Us, https://www.naacpldf.org/about-us/ (last accessed June 10, 2019).
2 LDF, Davis v. City of New York, https://www.naacpldf.org/case-issue/davis-v-city-new-york/
(last accessed June 10, 2019).
2
contexts, including, but not limited to, child welfare, education, healthcare,
housing, and immigration, this testimony focuses on its discrete, durable, and
disproportionate racial impact in the area of policing and law enforcement.
Accordingly, LDF makes the following nine preliminary recommendations to this
Task Force:
1. The City Must Adopt a Uniform Definition of ADSs
To ensure necessary accountability and transparency, the Task Force must
create a broad, uniform definition of an ADS. Under the current definition in Local
Law 49, an agency ADS is defined as an “automated decision system used by an
agency to make or assist in making decisions concerning rules, policies or actions
implemented that impact the public.” During the May 30 public forum, the Task
Force announced that it created a “Checklist for Determining Whether a Tool or
System is an ADS/Agency ADS.”3 Under this definition, which Task Force
members asserted during the forum did not generate consensus among members,
an ADS is defined as “computerized implementation of algorithms, including those
derived from machine learning or other data processing or artificial intelligence
techniques, which are used to make or assist in making decisions.”4 Both of these
definitions are too limited and fail to capture the full range of systems that
agencies are considering implementing or have already implemented. Indeed,
during the May 30 forum, Task Force members agreed with these concerns,
explaining that these definitions do not capture the broad range of ADSs deployed
and implemented.
We therefore recommend adopting the ADS definition that advocates and
experts recommended to the Task Force more than eight months ago.5 To capture
the full range of ADSs, the group recommended defining an ADS as follows:
An automated decision system is any software, system, or process that aims
to aid or replace human decision-making. Automated decision systems can
include analyzing complex datasets to generate scores, predictions,
3 NYC AUTOMATED DECISION SYSTEMS TASK FORCE, CHECKLIST FOR DETERMINING WHETHER A
TOOL OR SYSTEM IS AN ADS/AGENCY ADS (2018),
https://www1.nyc.gov/assets/adstaskforce/downloads/pdf/ADS-TF-Checklist-for-Determining-ADS-
Agency.pdf.
4 Id.
5 Letter from Andrew G. Ferguson, Professor, UDC David A. Clarke School of Law, et al., to the
NYC Automated Decision Systems Task (Aug. 17, 2018),
http://assets.ctfassets.net/8wprhhvnpfc0/1T0KpNv3U0EKAcQKseIsqA/52fee9a932837948e3698a658
6a8d50/NYC_ADS_Task_Force_Recs_Letter.pdf.
3
classifications, or some recommended action(s), which are used by agencies
to make decisions that impact human welfare.6
This more expansive definition helps to ensure that all ADSs that affect
New Yorkers will be subject to the appropriate scrutiny and that the public will be
protected from unfair and inequitable consequences resulting from these systems.
During both public forums, community members, organizational representatives,
and Task Force members expressed concerns that agency accountability and
fairness cannot be meaningfully addressed by the current, narrow definition of
ADSs. Accordingly, LDF urges the Task Force to adopt this proposed definition
immediately.
2. The City Must Clarify that All ADSs and All Agencies
Using an ADS are Within the Task Force’s Purview and
Subject to its Recommendations
At its core, Local Law 49 seeks to ensure that policies and procedures are in
place to provide community members and elected officials with tools to analyze and
review these systems, as well as to provide structures to ensure community
members are involved in decision-making processes. Accordingly, any agency that
plans to implement, or has implemented, an ADS is within the Task Force’s
purview and subject to its recommendations, including the NYPD.
Indeed, given the far-reaching consequences of technological advances in the
NYPD, coupled with the Department’s well-documented history of discriminatory
and unconstitutional policing and enforcement practices, any decision to exclude
the NYPD from the Task Force’s purview or recommendations would be
antithetical to Local Law 49’s intent and purpose. New York City therefore cannot
achieve accountability, fairness, or transparency in the implementation of ADSs if
some systems are excluded from the Task Force’s purview.
Accordingly, the City must confirm that all ADSs and all agencies using
ADSs fall within the Task Force’s ADS review. Failing that, it must create an
independent review process before any system can be exempt from the Task
Force’s purview and recommendations, which includes an opportunity for the
public to challenge the exemption.
3. The City Must Commit to Full Transparency and Disclose
Information About the NYPD’s ADSs
According to a chart of known ADSs used by City agencies created by AI
6 Id.
4
Now at NYU University,7 the NYPD has already implemented or considered
implementing the following ADSs without meaningful community engagement or
oversight: Automated License Plate Readers,8 Facial Surveillance,9 Predictive
Policing,10 and Social Media Monitoring.11 This list, however, is most likely
underinclusive because the NYPD routinely conceals and fails to disclose the
development and use of ADSs, such as Domain Awareness System’s place-based
predictive policing, from both public and governmental oversight.12 By concealing
its use of ADSs, the NYPD prevents the public from adequately studying the
7 Automated Decision Systems: Examples of Government Use Cases, AI NOW AT NYU
UNIVERSITY 1, 1-4 (2019), https://ainowinstitute.org/nycadschart.pdf.
8 Information about the NYPD’s license plate reader is available at
https://www.nydailynews.com/new-york/nyc-crime/ny-metro-license-plate-reader-grab-20190119-
story.html. For more information on automated license plate readers generally, see Automated
License Plate Readers (ALPRs), ELECTRONIC FRONTIER FOUNDATION,
https://www.eff.org/pages/automated-license-plate-readers-alpr.
9 George Joseph and Kenneth Lipp, IBM Used NYPD Surveillance Footage To Develop
Technology That Lets Police Search By Skin Color, THE INTERCEPT, ( Sept. 6, 2018),
https://theintercept.com/2018/09/06/nypd-surveillance-camera-skin-tone-search/. For additional
information the NYPD’s use of facial recognition analysis, see Clare Garvie, Garbage in, Garbage Out:
Face Recognition on Flawed Data, GEORGETOWN LAW CENTER ON PRIVACY AND TECHNOLOGY (May 16,
2019), https://www.flawedfacedata.com/.
10 Rachel Levinson-Waldman and Erica Posey, Court: Public Deserves to Know How NYPD Uses
Predictive Policing Software, THE BRENNAN CENTER (Jan. 26, 2018),
https://www.brennancenter.org/blog/court-rejects-nypd-attempts-shield-predictive-policing-
disclosure.
11 America’s Cops Take an Interest in Social Media, THE ECONOMIST (Feb. 21, 2019),
https://www.economist.com/united-states/2019/02/21/americas-cops-take-an-interest-in-social-
media; Annie McDonough, Privacy Advocates Score Win Against NYPD Over Surveillance
Technology, CITY & STATE NEW YORK (Jan. 22, 2019),
https://www.cityandstateny.com/articles/policy/technology/privacy-advocates-score-win-against-
nypd-over-surveillance-technology; Clay Dillow, NYPD Creates Facebook-Police Task Force to Mine
Social Media for Clues, POPULAR SCIENCE (Aug. 10, 2011),
https://www.popsci.com/technology/article/2011-08/nypds-facebook-police-mine-social-media-clues-
about-crime.
12 Steven Melendez, NYPD Unveils Controversial Algorithm to Track Crime Patterns, FAST
COMPANY (Mar. 20, 2019), https://www.fastcompany.com/90321778/nypd-unveils-controversial-
algorithm-to-track-crime-patterns; Laura Nahmias, Police Foundation Remains a Blind Spot in NYPD
Contracting Process, Critics Say, POLITICO (July 13, 2017), https://www.politico.com/states/new-
york/city-hall/story/2017/07/13/police-foundation-remains-a-blind-spot-in-nypd-contracting-process-
critics-say-113361; Ali Winston, NYPD Attempts to Block Surveillance Transparency Law with
Misinformation, THE INTERCEPT (July 7, 2017), https://theintercept.com/2017/07/07/nypd-surveillance-
post-act-lies-misinformation-transparency/; Private Donors Supply Spy Gear to Cops, PROPUBLICA
(Oct. 13, 2014); https://www.propublica.org/article/private-donors-supply-spy-gear-to-cops.
5
impact of these systems and shields itself from accountability. Moreover,
concealing this information prevents necessary debate and dialogue, while further
sowing mistrust between the NYPD and community members.
Equally alarming, the NYPD plans to continue embedding ADSs in their
decision-making processes at a disturbingly aggressive pace. For example, on April
3, 2019, at NYU School of Law, the NYPD’s Deputy Chief of Policy and Programs,
Thomas Taffe, explained that the Department hired more than 100 civilian
analysts since 2017 to use ADS software to analyze the NYPD’s crime data.13
Moreover, based on information and belief, the NYPD also appears to be building
its own ADSs. The NYPD is thus poised to continue building its capacity to rapidly
scale up its use of ADSs without public accountability or oversight.
Importantly, the Task Force must also address concerns that the NYPD may
circumvent complying with recommendations because doing so would interfere
with “law enforcement investigation or operations.”14 The NYPD has relied on this
rationale to justify withholding information from the public under New York State
Freedom of Information Laws. Without addressing these concerns, which were
brought up by community members at the May 30 forum, the NYPD can continue
building and testing these technologies on residents without public accountability
and oversight—a result that is antithetical to Local Law 49’s intent and purpose.
For these reasons, at a minimum, the Task Force must recommend that the
NYPD publicly identify, categorize, and share a list of all ADSs that the
Department has implemented, plans to implement, or is developing. Once created,
this list of ADSs should be continuously updated in real time, and as discussed
below, it must be subject to public scrutiny.
4. The City Must Ban the Use of Data Derived from
Discriminatory and Biased Enforcement Policies and
Practices in ADSs
Because algorithms learn and transform through exposure to data, an
algorithm is only as good as the data that is selected to inform it. In other words,
ADSs, like all machine-learning technologies, inherit the biases of the data and
commands they are given. An ADS’s algorithm, therefore, will replicate any biases
13 See Zolan Kanno-Youngs, NYPD Number-Crunchers Fight Crime with Spreadsheets, THE
WALL STREET JOURNAL (July 23, 2018), https://www.wsj.com/articles/nypd-number-crunchers-fight-
crime-with- spreadsheets-1532381806.
14 As an example, see section 1(6) of Local Law 49, noting that compliance with the Task Force’s
recommendations is not required if such compliance would “interfere with a law enforcement
investigation or operations.”
6
within its training data—a phenomenon called “training bias.”15 In other words,
bias in, bias out. Training bias can lead to discrimination in at least two ways:
(1) reproducing the biases in the data and (2) drawing inferences from, and thus
prioritizing, the biases in the data.16 In the policing context, this means that data
derived from and reflecting any of the NYPD’s practices that are discriminatory,
illegal, and unconstitutional will infect any algorithm and ADS that is trained
with that data. The resulting algorithm or ADS will then carry out and perpetuate
that same discrimination—making all decisions either produced by the ADS or
relied on based on ADS-generated predictions flawed.
For decades, the NYPD engaged in widespread racial profiling against Black
and Latinx residents. Between 2004-2012, the NYPD conducted an astounding 4.4
million stops of City residents as they engaged in their daily lives.17 A
staggering 88 percent of these stops resulted in no further action—meaning a vast
majority of those stopped were not engaged in unlawful conduct.18 In about 83
percent of cases, the person stopped was Black or Latinx, even though the two
groups combined accounted for just over one-half the city population.19 When these
discriminatory practices were challenged in Floyd v. City of New York, a federal
court found the NYPD liable for violating the Fourth Amendment rights of New
Yorkers to be free from unreasonable searches and seizures. The court also found
that the NYPD’s policies and practices were racially discriminatory in violation of
the Equal Protection Clause of the Fourteenth Amendment.20
Similarly, in Davis v. City of New York, the NYPD unlawfully stopped and
arrested people of color who lived in or visited NYCHA apartments, without
reasonable suspicion or probable cause.21 The NYPD justified its racially
discriminatory arrests by alleging the residents and their visitors were “criminally
trespassing” despite the lack of evidence to support officers’ suspicions. Currently,
15 Solon Barocas and Andrew Selvst, Big Data’s Disparate Impact, 104 CAL. L. R. 671, 680-81
(2016). 16 Id. at 680-87.
17 Racial Discrimination in Stop-and-Frisk, N.Y. TIMES, (Aug. 12, 2013),
https://www.nytimes.com/2013/08/13/opinion/racial-discrimination-in-stop-and-frisk.html.
18 Id.
19 Id.
20 Floyd v. City of New York, 959 F.Supp.2d 540, 660-665 (S.D.N.Y. 2013).
21 Complaint, Davis v. City of New York, 2010 WL 9937605 (S.D.N.Y. 2011) (No. 1),
https://dev.naacpldf.org/wp-content/uploads/Complaint-
1.pdf?_ga=2.49083558.141431006.1559743995-2134253651.1504725451.
7
the Department’s aggressive, military style gang “takedowns” primarily target
public housing residents, the overwhelming majority of whom are people of color.22
Prior to executing these sweeping gang takedowns, the NYPD conducts criminal
investigations relying, in part, on a secret database that erroneously designates
thousands of New Yorkers as members of gangs or local street “crews,” often
without informing the individual or offering any due process protections.23 Officers
executing gang policing strategies rely on vague and troubling terms and
generalizations to justify their frequently erroneous designation of individuals as
gang members.24
Of equal concern is the Department’s manipulation of its data. For example,
it has been reported that the Department made arrests and issued summonses as
part of a monthly quota system until as recently as December 2018.25 Additionally,
the federal monitor overseeing the NYPD confirmed that the Department is also
undercounting street stops.26 Both of these examples underscore how the NYPD’s
22 NYPD’s Gang Takedown Efforts before the NYC City Council’s Comm. on Public Safety (June
13m 2018) (statement of Marne L. Lenox, Associate Counsel, LDF, and Darius Charney, Senior Staff
Attorney, the Center for Constitutional Rights), https://www.naacpldf.org/wp-content/uploads/City-
Council-Testimony-combined-6.13.18.pdf.
23 Id.
24 The NYPD provided its IDS Gang Entry Street and the criteria by which gang members are
certified in response to Professor Babe Howell’s Freedom of Information Law request, filed on
September 2, 2011. In addition to these criteria, the NYPD may certify someone as a gang member if
an individual admits membership during a debrief or if, through the course of an investigation, an
individual is reasonably believed to belong to a gang and is identified as such by two independent
sources, which could include other New York City agencies. K. Babe Howell, Gang Policing: The Post
Stop-and-Frisk Justification for Profile-Based Policing, 5 UNIV. DENVER. CRIM. L. R. 1, 16 (2015).
25 George Joseph, NYPD Commander’s Text Messages Show How The Quota System Persists, THE
APPEAL (Dec. 12, 2018), https://theappeal.org/nypd-commanders-text-messages-show-how-the-quota-
system-persists/; Jake Nevins, ‘We Didn’t Ask Permission’: Behind an Explosive NYPD Documentary,
THE GUARDIAN (Aug. 23, 2018); https://www.theguardian.com/film/2018/aug/23/nypd-documentary-
crime-and-punishment-stephen-maing; Rocco Parascandola & Thomas Tracy, NYPD Demands All
Uniform Officers Undergo ‘No Quota’ Training for Arrests, Tickets, DAILY NEWS (Feb. 15, 2018),
https://www.nydailynews.com/new-york/nypd-demands-uniformed-officers-undergo-no-quota-
training-article-1.3823160.
26 Monitor’s Seventh Report, NYPD Monitor 1, 4,8 (Dec. 13, 2017), http://nypdmonitor.org/wp-
content/uploads/2017/12/2017-12-13-FloydLigonDavis-Monitor-Seventh-
Report_EAST_80301353_1.pdf; Monitor’s First Report, NYPD Monitor 1, 48 (July 9, 2015); see also Al
Baker, City Police Officers Are Not Reporting All Street Stops, Monitor Says, N.Y. TIMES (Dec. 13,
2017), https://www.nytimes.com/2017/12/13/nyregion/nypd-stop-and-frisk-monitor.html; J. David
Goodman and Al Baker, New York Police Department is Undercounting Street Stops, Report Says, N.Y.
TIMES (July 9, 2015), https://www.nytimes.com/2015/07/10/nyregion/some-new-york-police-street-
stops-are-going-undocumented-report-says.html.
8
data, as well as crime data, may be skewed and inaccurate due to the
Department’s actions. Indeed, there is a litany of examples when NYPD officers
unlawfully arrested, charged, and jailed innocent people who are
disproportionately people of color.27
As a result of these and many other racially discriminatory practices, we
have substantial concerns that the NYPD’s datasets are infected with deeply
rooted biases and racial disparities. Consequently, we are likewise concerned that
any predictions or output from an ADS that relies on such data, in any capacity,
will reproduce and reinforce these biases and disparities.
Further, merely inputting NYPD data into an ADS does not eliminate or
remove the embedded biases. This misperception—so-called “tech washing”—
invites unwarranted agency deference and the belief that data can be cleansed by
putting it into an ADS, thereby producing “accurate,” “objective,” or “neutral”
predictions.28 However, because NYPD datasets and others are likely tainted by
illegal, discriminatory, and unconstitutional practices, they should be categorized
as “dirty data,” meaning, the underlying data is “inaccurate, skewed, or
systemically biased.”29 The term “dirty data” is “commonly used in the data mining
research community to refer to ‘missing data, wrong data, and non-standard
representation of the same data.’”30 However, LDF endorses Rashida Richardson,
Jason Schultz, and Kate Crawford’s recommendation to expand the definition to
include this new category of data that is “derived from or influenced by corrupt,
biased, and unlawful practices, including data that has been intentionally
manipulated . . . as well as data that is distorted by individual and societal
biases.”31
We are skeptical that such “dirty data” can ever be cleansed to separate the
“good” from the “bad,” the tainted from the untainted.32 Therefore, in committing
27 Rashida Richardson, Jason Schultz, & Kate Crawford, Dirty Data, Bad Predictions: How Civil
Rights Violations Impact Police Data, Predictive Policing Systems, and Justice, 94 N.Y.U. L. R. 192
(May 2019).
28 Ingrid Burrington, What Amazon Taught the Cops, THE NATION (May 27, 2015),
https://www.thenation.com/article/what-amazon-taught-cops/.
29 Richardson, supra note 27, at 193-96.
30 Id. at 195.
31 Id.
32 Vincent Southerland, With AI and Criminal Justice, The Devil is in the Data, ACLU (Apr. 9,
2018), https://www.aclu.org/issues/privacy-technology/surveillance-technologies/ai-and-criminal-
justice-devil-data.
9
to exclude any data classified as “dirty” from all ADSs,33 the Task Force should
adopt this definition of “dirty data” and make clear that ADSs stemming from or
operated by agencies with a history of biased or discriminatory practices or data,
begin with the presumption that such data and the resulting ADSs include bias
and are therefore unrepresentative. This method, as discussed at both the April 30
and May 30 forums, places the burden on the agency to show that no racial bias or
discriminatory impact is present in its underlying datasets or the ADSs’ use,
rather than placing the burden on the City’s residents, either as individuals or
community members. It also ensures that no ADS incorporates or uses any data
which are reasonably suspected to have been derived from discriminatory or biased
practices.
5. The City Must Adopt Processes for Determining if an ADS has
a Racially Disproportionate Impact on Communities of Color
To avoid discriminatory ADSs, the Task Force’s recommendations should
make clear that any data derived from discriminatory, illegal, or unconstitutional
policing enforcement or practices, and informs or is incorporated into an ADS,
should be presumed invalid due to the likely disproportionate impact on
communities of color. It will also require, at a minimum, that an independent
third-party conduct a racial equity impact assessment.34 In addition, for all data
used in any ADS—including data that is purportedly not derived from
discriminatory, illegal, or unconstitutional practices and data that comes from
sources other than the NYPD—the following oversight must occur to determine the
possible racially disparate impact of an ADS or ADS-generated predictions:
• A racial equity impact assessment;35
• A surveillance impact assessment and report;36
33 Richardson, supra note 27, at 193-98.
34 A Racial Equity Impact Assessment (REIA) is a “systematic examination of how different
racial and ethnic groups will likely be affected by a proposed action or decision. REIAs are used to
minimize unanticipated adverse consequences in a variety of contexts, including the analysis of
proposed policies, institutional practices, programs, plans and budgetary decisions. The REIA can be
a vital tool for preventing institutional racism and for identifying new options to remedy long-standing
inequities.” THE CENTER FOR RACIAL JUSTICE INNOVATION, Racial Equity Impact Assessment, (2019),
https://www.raceforward.org/sites/default/files/RacialJusticeImpactAssessment_v5.pdf.
35 Id.
36 David Wright & Charles D. Raab, Constructing a Surveillance Impact Assessment, 28
COMPUTER L. & SECURITY R. 613 (2012).
10
• A pre-acquisition, development, or implementation procedure to
ensure non-agency experts, representatives from affected
communities, and the public at large is consulted before and during
the development of an ADS; and
• Agencies’ maintenance of a public record of external participation.
As a starting point, these initial recommendations should be adopted and proposed
to community members during any public engagement.
In addition to the above recommendations, no ADS—including those ADSs
already implemented—should be used without a thorough review and analysis of
(1) all underlying data and (2) the effects the system has on vulnerable
communities. This analysis should benefit from the significant input of social
science experts, as well as communities who have fallen victim to NYPD’s biased
actions in the past. We therefore strongly urge the Task Force to recommend that
the City commit to not use any ADS that relies on “dirty data” or has a
discriminatory impact. This commitment, along with the above recommendations,
is a strong starting point for the City and Task Force to fight against the racial
bias present in many agencies today, including racial bias through data and
technology.
6. The City Must Remedy and Account for Proxy Factors that
Also Produce Discriminatory Results
An agency’s unsupported assertion that it has “scrubbed” either its data or
its algorithm of data derived from discriminatory, illegal, or unconstitutional
practices is not enough to establish that the algorithm and the ADSs are not
biased. First, many agencies will find it difficult to properly identify all the biases
in their training data. Second, if directly confronted with their own bias, some
agencies may even deny it, rather than attempting to address and scrub the biased
data.37 Third, even if all biased data is removed from an algorithm, algorithms can
learn biased behavior through proxy factors—factors that may appear neutral but
reflect societal and structural biases.38 For example, an algorithm may
purposefully exclude all input for race and ethnicity. However, if the algorithm
still considers factors that, due to societal constructs, correlate to race—such as
37 Julia Angwin et al., Machine Bias, PROPUBLICA (May 23, 2016),
https://www.propublica.org/article/machine-bias-risk-assessments-in-criminal-sentencing.
38 Even when removing “dirty data,” ADSs often reflect the very discriminatory behavior we
sought to avoid because “[w]hat we’re doing is using the idea of eliminating individual irrational bias
to allow this vast structural bias to sneak in the back door of the system.” Dave Gershgorn, Algorithms
Can’t Fix Societal Problems—and Often Amplify Them, QUARTZ (Oct. 17, 2018),
https://qz.com/1427159/algorithms-cant-fix-societal-problems-and-often-amplify-them/.
11
low-income neighborhoods or employment history—the algorithm’s outputs may
nonetheless be racially skewed.39 To protect against racial discrimination and bias
by proxy, the Task Force must also develop recommendations, after consultation
with experts and community members, so that agencies can address societal and
systemic factors that contribute to discriminatory ADSs and to determine ways to
mitigate the influence of proxy factors in ADSs.
7. The City Must Establish Procedures for Addressing Harms
When ADSs Have an Improper Disproportionate Impact on
Communities of Color or Harm Individuals Based on Biased
Data
Continuing to rely on ADSs without any pre-implementation processes or
safeguards,40 such as the recommendations suggested here, risks subjecting entire
communities to continued discriminatory and unconstitutional enforcement and
policing practices. ADSs could be used to justify disparate treatment of
communities of color in terms of how “suspicion” is defined, who is chosen as
“targets” for increased enforcement and surveillance, and where these machine-
learning tools are deployed—all raising significant constitutional concerns under
the First, Fourth, and Fourteenth Amendments to the U.S. Constitution. The use
of ADSs threatens to distort reasonable suspicion, expectation of privacy, and
freedom of speech doctrines.
The use of predictive policing software, especially when combined with
increased surveillance, may curtail people’s freedom of association and speech
under the First Amendment. If mere association with friends and family members
or “hanging out” in a “chronic crime” or “gang-prone” neighborhood are used as a
factor to justify police contact and surveillance, people will be forced to alter their
behavior to avoid being subjected to unwarranted police intrusion. This type of
surveillance will cast an undue level of suspicion on communities, especially
communities of color, that are already vulnerable to racially biased policing.
39 Angwin et al., supra note 37; see also Virginia Eubanks, A Child Abuse Prediction Model
Fails Poor Families, WIRED (Jan. 15, 2018) (noting that, though the Allegheny County Family
Screening Tool intentionally took steps to avoid racial disparities in its child welfare system, the
system was nevertheless biased and produced racially discriminatory results because the developers
ignored a major area of societal bias in the child welfare system overall—that people report Black and
biracial families to child welfare offices 350 percent more than white families, creating an influx
of Black family child welfare cases at the outset), https://www.wired.com/story/excerpt-from-
automating-inequality/.
40 The recommendations in this section should also apply retroactively to ADSs that were
implemented without the Task Force’s recommended procedural safeguards.
12
Further, even outside of the policing context, ADSs are being used to track
and surveil communities of color in nearly every space. New York landlords have
begun placing facial recognition software in buildings with predominately Black
and Latinx residents.41 Indeed, at the May 30 public forum, City residents stated
their landlords have installed invasive facial recognition software without their
consent in the buildings where they reside, which primarily comprise Black and
Latinx tenants.
Similarly, school districts throughout the country, including in New York
State, are considering implementing facial recognition technology as a solution to
school safety concerns.42 But as described above, these systems will likely
reproduce racial and gender biases.43 Moreover, subjecting students to constant
surveillance will redefine students’ experience, especially for Black and Latinx
students who are already subjected to disproportionate levels of discipline and law
enforcement contact in schools.44
Finally, the City should provide individuals who may be subject to harmful
decision-making that results from racially biased data a platform and procedures
for redressing that harm simply and swiftly.
These potential constitutional harms further underscore why the Task Force
must make bold and expansive recommendations to create procedures and
safeguards to protect the public, especially vulnerable communities of color, from
potential constitutional and other violations. These safeguards must be created
with vulnerable communities in mind, and thus include clear, easily accessible
processes for understanding the underlying data used in an ADSs, the agency
responsible, and a timely option for refuting the ADSs’ implementation.
41 Erin Durkin, New York Tenants Fight as Landlords Embrace Facial Recognition Cameras,
THE GUARDIAN (May 30, 2019), https://www.theguardian.com/cities/2019/may/29/new-york-facial-
recognition-cameras-apartment-complex.
42 Ava Kofman, Face Recognition is Now Being Used in Schools, But It Won’t Stop Mass
Shootings, THE INTERCEPT (May 30, 2018), https://theintercept.com/2018/05/30/face-recognition-
schools-school-shootings/.
43 Joy Buolamwini, When the Robot Doesn’t Seek Dark Skin, N.Y. TIMES (June 21, 2018),
https://www.nytimes.com/2018/06/21/opinion/facial-analysis-technology-bias.html.
44 Moriah Balingit, Racial Disparities in School Discipline are Growing, Federal Data Shows,
WASH. POST (Apr. 24, 2018), https://www.washingtonpost.com/local/education/racial-disparities-in-
school-discipline-are-growing-federal-data-shows/2018/04/24/67b5d2b8-47e4-11e8-827e-
190efaf1f1ee_story.html?utm_term=.37019ac1be8f; see also Ajmel Quereshi and Jason Okonofua,
Locked Out of the Classroom: How Implicit Bias Contributes to Disparities in School Discipline, LDF
(2017), https://www.naacpldf.org/files/about-us/Bias_Reportv2017_30_11_FINAL.pdf.
13
8. The City Must Create Accountability Structures that
Empower All Community Members to Participate in Pre- and
Post- Acquisition Decisions About ADS
The City is experimenting on its residents by relying on ADSs to make
predictions and decisions without fully understanding how these systems will
affect community members. Worse, the City has not required complete ADS
transparency or meaningful community engagement, thus excluding the important
perspectives and opinions of the very communities that will be affected by ADSs.
To date, the City has not provided sufficient mechanisms for non-agency experts
and community members to be educated about, and thoroughly evaluate, all ADSs
prior to implementation. The City must reaffirm its commitment to accountability
and transparency by creating structures that center community members—not
machines—in the decision-making process and provide them with meaningful
opportunities to give feedback and input about ADSs.
9. The ADS Task Force Should Move Swiftly to Issue Its
Recommendations and Request Additional Time, If
Needed
Lastly, we stress the Task Force’s duty to thoroughly research and
understand the many ways ADSs affect the City’s most vulnerable residents—
including low-income communities and communities of color. The Task Force also
has the important responsibility of ensuring that the City’s residents—including
the most vulnerable ones—are well-educated about ADSs, understand how ADSs
will affect their everyday life and, importantly, have ways to meaningfully voice
their concerns and feedback. Finally, the Task Force must issue critical
recommendations that incorporate community feedback, prevent bias and
discrimination in ADS use, and establish procedures for ADS accountability and
transparency. Equally important, these recommendations must include clear and
concrete procedures that provide City’s residents who are impacted, or likely to be
impacted, by an ADS and its underlying algorithm with the power to (1) challenge
any agency’s proposed use of an ADS prior to the ADS’s implementation, (2) have
access to the information and data necessary to determine whether an ADS is
being implemented with or without bias, and (3) hold any City agency accountable
when an ADS is not fair and equitable.45
Commissioned in May 2018, the Task Force has only until November 16,
2019 to fulfill its mandate pursuant to Local Law 49.46 To the extent that it
45 NYC AUTOMATED DECISION SYSTEMS TASK FORCE, See About ADS, (2019),
https://www1.nyc.gov/site/adstaskforce/about/about-ads.page.
46 “No later than 18 months after such task force is established, it shall electronically submit to
14
appears unlikely that the Task Force will be able to complete its charge within
this timeframe, including soliciting robust public engagement through a
transparent process in which the public has the information necessary evaluate
the impact of ADSs on their communities, it should seek to extend its deadline.
To date, the Task Force has not implemented a public education campaign
nor solicited robust public engagement. Aside from the experts who testified
during the public forums and the transcripts of these forums, the Task Force has
only created a single educational resource.47 While the “Checklist” is helpful, there
are no materials identifying ADSs and explaining how agencies are using them.
During both public forums, community members discussed how this lack of
transparency undermines any meaningful and robust community participation.
For example, how can community members meaningfully discuss and share
insight about ADSs when they do not have minimal basic information about ADSs
being used by agencies and how those ADSs operate? Indeed, during the May 30
forum, one Task Force member agreed that failing to include this foundational
information undermines soliciting community input. Instead, community members
generally remain unaware of the implications that ADSs may have on their daily
lives.
Moreover, although it has been more than a year since the creation of the
Task Force, it has yet to hold a community session events, or even release details—
for example, dates, times, and locations—about holding any community sessions.
The Task Force must do better. The Task Force cannot successfully understand
the impact of ADSs on the New Yorkers who have and will continue to be targeted
and exploited with ADSs—which are primarily communities of color—without
rigorous, targeted, and substantive outreach, education and engagement in these
communities.
Additionally, the Task Force should tailor its efforts to educate and include
community members in its deliberations to ensure New Yorkers will understand
the gravity of ADSs. The Task Force should ensure it and its experts use plain
language to educate the public about ADSs, rather than academic or technical
language, and that events are publicized well in advance and held in locations or
frequented by and easily accessible to the communities likely to be affected by
biased ADSs.
Overall, the Task Force has significant work to do in the next five months.
Because this work is critical and affects all New Yorkers, we urge the Task Force
to dedicate the time and resources to it that this task merits. By the same token,
the mayor and the speaker of the council a report . . . .” Local Law 49.
47 Checklist, supra note 3.
15
however, the longer the Task Force takes to issue its recommendations, the longer
ADSs will operate in New York without necessary guidance on fairness,
transparency, and accountability. Accordingly, we strongly urge the Task Force to
move swiftly and diligently in issuing its recommendations without compromising
the necessary information gathering and careful study that this undertaking
requires, and the City’s residents deserve. Further, given the limited transparency
about ADSs, coupled with the well-documented concerns and harms, the Task
Force should consider issuing interim recommendations that use of all ADSs be
suspended until the Task Force completes its charge.
Conclusion
In closing, the NYPD’s use of ADSs already creates an unprecedented
expansion of police surveillance. While the expansion implicates all residents’
privacy rights, the burdens and harms are not evenly shared among City
residents. Communities of color, particularly Black and Latinx residents, will
continue to be disproportionately subjected to profiling, policing, and punishment
to the extent that ADSs replicate the biases of the current criminal legal system
and law enforcement practices.
Worse, these harms are not limited to the criminal legal system alone. Other
City agencies are relying on ADSs to make decisions about education enrollment,
child welfare risk and safety assessment, public benefits enforcement, risk
assessment for pre-trial decisions, and much more. These systems impact a
majority of New York City residents’ lives.
Nevertheless, to date, the City has failed to provide necessary public
education and information about these systems. Moreover, these systems are being
deployed without effective mechanisms for public participation in pre- and post-
acquisitions decisions and without regard for the widely known societal and
structural racism that persists in nearly every area of life in the City. This current
situation is untenable. Residents cannot be subjected to experimental testing of
new technology that is being used to guide decision-making processes without
rigorous safeguards to ensure accountability and transparency. Implementing and
relying on these systems without understanding their impact, particularly their
racial justice impact, will exacerbate the current inequities throughout the City.
The rapid, unchecked deployment of ADSs without effective mechanisms for
public input, independent oversight, or the elimination of racial discrimination and
bias is unacceptable. Data and technology should not be weaponized by New York
City against its residents. Accordingly, this Task Force should make
recommendations that hold agencies accountable for ensuring that all ADSs—
including ADSs currently in use and any future ADSs—are transparent, fair, and
16
free from racial discrimination and bias.
Thank you for considering these recommendations. If you have any
questions, please contact us at 212-965-2200.
Respectfully submitted,
Janai Nelson
Associate Director Counsel
1
April 26, 2019
Jeff Thamkittikasem, Chair
New York City Automated Decision Systems Task Force
Mayor’s Office of Operations
City Hall
New York, NY 10007
Dear Chair Thamkittikasem:
Thank you for your testimony at the Committee on Technology Oversight Hearing regarding an Update
on Local Law 49 of 2018 in Relation to Automated Decision Systems (“ADS”) Used by Agencies on
April 4, 2019. At the hearing, we received testimony from yourself and your co-chairs, Kelly Jin and
Brittny Saunders,1 current and former members of the Automated Decision Systems Task Force (“Task
Force”) and advocates.
There were several questions raised about the ADS Task Force regarding the Task Force’s process and
the progress of actions for which we would like further explanation. Specifically, there are concerns
and questions regarding the Task Force’s compliance with the legislative intent of Local Law 49 of
2018, which required the Task Force to review automated decision systems used in New York City in
order to determine whether an agency ADS disproportionately impacts persons based upon a protected
category2 and, to make recommendations for the development and implementation of a procedure to
address instances in which a person may be harmed by an agency automated decision system. We
thank you in advance for your timely response to the following questions.
1. In your testimony, you stated, “since it was first convened, the Task Force has devoted a
substantial amount of time to clarifying which systems and tools might fall under the Law’s
definition of what constitutes an agency ADS. As you can imagine, this has been a challenging
but essential step in the Task Force’s work. And I’m not afraid to say that it’s taken more time
than we originally thought that it would take.”3
How long did the Task Force originally think that the identification of ADS would
take?
1 Kelly Jin, Chief Analytics Officer for Mayor’s Office of Data Analytics and Brittny Saunders, Deputy
Commissioner for Strategic Initiatives at the New York City Commission on Human Rights. 2 “Age, race, creed, color, religion, national origin, gender, disability, marital status, partnership status, caregiver
status, sexual orientation, alienage or citizenship status,” Local Law 49 of 2018. 3 See NYC Automated Decision Systems Task Force Testimony before the New York City Council Committee on
Technology, April 4, 2019, 12:38.
2
Will the Task Force meet its deadline of November 16, 20194 to examine current ADS
used by the City and draft recommendations as required by Local Law 49 of 2018?
2. In your testimony, you stated, “our Task Force has met on a regular basis—both as a full group
and in smaller groups.”5
Please provide the dates of when the Task Force has met, both as a whole and in smaller
groups, a list of which members were present, what topics were discussed, as well as
an explanation of why the smaller groups were necessary.
3. In your testimony, you stated, “the ADS Task Force is not going to produce a list of algorithms
in use by the city, but will develop and issue the recommendations.”6 However, several
advocates have already published lists of ADS used by the city. For example, AI Now produced
a list8, including ADS used in children welfare, criminal justice, education, fire, housing, public
benefits, and public health. Other examples of ADS were also mentioned by the advocates at
our public hearing on October 16, 2017.
Would you be willing to provide a list of ADS currently used by the City, even when
not expressly required to do so?
4. In your testimony, you mentioned that you have not taken any actions to obtain ADS
examples.11 However, in your recent interview to The Verge, you said that “[the Task Force]
is working on providing new, specific examples, including two from the Department of
Transportation and the Department of Education.”12
Which two examples will you be providing to the Task Force?
Since April 4, 2019, what other agencies have you contacted in order to obtain ADS
examples?
What was the result of your communications?
Which agencies do you plan to contact in the near future?
How many City-owned or -operated ADS will be described in the forthcoming report?
Considering the importance of context when assessing the disproportionate impact of
ADS, how useful and accurate recommendations will be without examining specific
examples of agency ADS?
5. Several Task Force members represent city agencies, including NYC Department of Education,
NYC Department of Transportation, NYC Police Department, NYC Department of Social
4 “No later than 18 months after such task force is established, it shall electronically submit to the mayor and the
speaker of the council a report … The mayor shall, no later than 10 days after receipt of the report … make such
report publicly available online through the city’s website,” Local Law 49 of 2018. 5 Jeff Thamkittikasem, Director of the Mayor’s Office of Operations, NYC Automated Decision Systems Task Force
Testimony of before the New York City Council Committee on Technology, April 4, 2019, p.2, .1.30. 6 Id., p.3, l.10-13. 8 AI Now, Automated Decision Systems Examples of Government Use Cases, Prepared in advance of NYC
Automation Decision Systems Task Force Public Forums, https://ainowinstitute.org/nycadschart.pdf. 11 See NYC Automated Decision Systems Task Force Testimony before the New York City Council Committee on
Technology, April 4, 2019, 25:51. 12 Colin Lecher, New York City’s Algorithm Task Force is Fracturing, THE VERGE, April 15, 2019,
https://www.theverge.com/2019/4/15/18309437/new-york-city-accountability-task-force-law-algorithm-
transparency-automation.
3
Services, Mayor’s Office of Criminal Justice and NYC Administration for Children’s
Services.13
To what extent have you considered examining ADS used by these agencies?
6. Your testimony mentioned privacy and security concerns related to the process of reviewing
ADS.14 You also mentioned that the NYC Chief Privacy Officer “plays an advisory role”15 to
the ADS Task Force.
Have you considered applying methods of de-identification or anonymization similar
to those that the Mayor’s Office of Data Analytics (“MODA”) uses in order to protect
an individual’s privacy?
Have you consulted or plan to consult with data scientists or privacy experts regarding
the modern methods or tools that are used to protect privacy, especially if personally
identifiable information could potentially be disclosed during the process of reviewing
ADS?
7. During the hearing, several questions were asked about the Task Force’s public engagement.
As of the date of the hearing, the only option for the public to communicate with the Task
Force was through a comment form on the ADS website.
How many comments did the Task Force receive via the website form? What about
outside of the form?
What is the process of addressing these comments and responding to the public?
Has the Task Force connected with individuals or groups who were directly impacted
by ADS?
To date, have individuals or groups who were impacted by ADS attended any Task
Force meetings?
Will the Task Force commit to increasing public participation and transparency by
making publicly available minutes or video of the meetings and public forums?
8. Several letters were sent to the Task Force and the Mayor by the public including letters dated
January 22, 2018,17 August 17, 2018,18 and March 1, 2019.19 In the letter dated January 22,
2018, the writers, which included current members of the Task Force, included the following
recommendations:
13 Howard Friedman, General Counsel, NYC Department of Education; Michael Replogle, Deputy Commissioner
for Policy, NYC Department of Transportation; Tanya Meisenholder, Assistant Commissioner for Strategic
Initiatives, New York City Police Department; Dan Hafetz, Special Counsel to the First Deputy Commissioner,
NYC Department of Social Services; Susan Sommer, General Counsel, Mayor’s Office of Criminal Justice; Andrew
White, Deputy Commissioner for Policy and Planning, NYC Administration for Children’s Services. See NYC
Automated Decision Systems Task Force, https://www1.nyc.gov/site/adstaskforce/members/members.page. 14 NYC Automated Decision Systems Task Force Testimony of before the New York City Council Committee on
Technology, April 4, 2019, 28:03. 15 Id. at 32:40. 17 January 22, 2017 open letter to Mayor de Blasio; available at https://ainowinstitute.org/nyc-algo-account-
letter.pdf. 18August 17, 2018 open letter to Acting Director Emily W. Newman, Deputy Commissioner Brittny Saunders,
Mayor’s Office of Operations; available at https://ainowinstitute.org/nyc-ads-task-force-letter-re-public-
engagement-030119.pdf. 19 March 1, 2019 open letter to Acting Director Emily W. Newman, Deputy Commissioner Brittny Saunders, New
York City Automated Decision Systems Task Force Members; available at https://ainowinstitute.org/nyc-ads-task-
force-letter-re-public-engagement-030119.pdf.
4
To the extent possible, all appointees should publicly disclose any engagement,
association, and present or prior grant funding from any vendors of automated
decision systems used in New York City government. Appointees should also
abide by the New York City Conflict of Interest Law.
Avoid appointment of New York City agencies that may have or appear to have
conflicts of interest because of pending litigation or public criticism regarding
the use of automated decision systems. For this reason, we recommend avoiding
appointment of the New York City Police Department, Mayor’s Office of
Criminal Justice, and the Office of the Chief Medical Examiner.20
Were any of these recommendations taken into consideration when new members were
added to the Task Force?
9. The Task Force’s membership was initially announced in May of 2018.21 Since then additional
members joined, including a new Chair. Currently, the ADS Task Force consists of eighteen
members, one chair and two co-chairs.22 While the list of members of the Task Force is publicly
available and transparent, the Task Force’s operations and processes are not.
How does the Task Force make decisions? Is quorum or majority vote required?
What role does the chair and co-chairs play in making decisions? Are they non-voting
members or voting members?
How was agency membership on the Task Force determined?
10. During the hearing, you were asked whether “anyone other than members of the Task Force
attend the meetings.” You answered that “the Mayor’s Office of Operations provides staff
support for different purposes depending on the topic of the discussion…”23 Albert Fox Cahn
in his testimony, however, noted the following:
One example of Task Force members being structurally disempowered is the
role of the Jane Family Foundation (“Foundation”). The Foundation effectively
drove the process for much of the first year, despite never being an official,
publicly recognized part of the Task Force. The Foundation’s role grew from
initially providing background briefings to providing proposed language and
policy documents for the Task Force to ratify. Increasingly, the Foundation was
writing a first draft of the Task Force’s report. The Foundation’s role drew
complaints from numerous Task Force members, so it was eventually phased
out, but it’s a telling example of how the role of Task Force members
themselves was circumscribed as part of this process.24
20 January 22, 2017 open letter to Mayor de Blasio; available at https://ainowinstitute.org/nyc-algo-account-
letter.pdf. 21 Mayor de Blasio Announces First-In-Nation Task Force To Examine Automated Decision Systems Used By The
City, May 16, 2018, The Official Website of the City of New York, https://www1.nyc.gov/office-of-the-
mayor/news/251-18/mayor-de-blasio-first-in-nation-task-force-examine-automated-decision-systems-used-by. 22 NYC Automated Decision Systems Task Force, https://www1.nyc.gov/site/adstaskforce/members/members.page. 23 NYC Automated Decision Systems Task Force Testimony before the New York City Council Committee on
Technology, April 4, 2019, 21:57. 24 Statement of Albert Fox Chan, Esq., and Liz O’Sullivan, Surveillance Technology Oversight Project, Inc., April
4, 2019, p. 3, l. 2.
5
What was the role of the Jane Family Foundation? Are they still involved in the
process?
Please provide the names, titles, and agencies of staff who have previously provided
support, along with their roles with respect to the Task Force.
Have any non-city employees, who are not members of the Task Force, attended any
Task Force meeting or function? If so, in what capacity?
11. In your testimony, you have mentioned that the Technology Committee members may join the
public forums that will be taking place on April 30, 2019 and May 30, 2019 at New York Law
School. As of today, some information is now publicly available on the website, but there does
not seem to be detailed information regarding public participation.
How do you plan to invite the general public and experts to the upcoming forums?
How many members of the public do you expect to attend?
o What is the minimum needed for the event to be “successful?”
What communities are important to have represented?
o What steps have you taken to ensure the diversity of attendees?
Will the public be able to engage at the forum?
o If so, how many members?
o How long will they have to speak?
o Will Task Force members be asked to respond to the public?
We understand that the forums will be livestreamed. Will they be recorded and publicly
accessible for future use?
o Will the public be able to submit real-time comments online for the hearing?
Are all Task Force members expected to be in attendance?
12. In your testimony, you testified that the Mayor’s Office of Operations (“MOO”) would be
responsible for drafting the report.25 The Task Force currently has one member working for
MOO, which is yourself, Jeff Thamkittikasem.
Will the other named members of the Task Force draft any portion of the report?
Will individuals or groups, other than named members of the Task Force, be drafting
portions of the report?
How many Task Force members’ approval will be needed to publish the report?
o Is quorum of the Task Force members needed?
o Can the report be published if a majority of members disapprove of the
findings?
o How will the report memorialize disagreements?
o Will any/all dissenting opinions by Task Force members be included in the
report?
o How long will Task Force members have to review and comment on the draft
report prior to publication?
25 See NYC Automated Decision Systems Task Force Testimony before the New York City Council Committee on
Technology, April 4, 2019, 35:40.
6
13. Lastly, we also would like to confirm that per your statement during the hearing,26 the
Technology Committee is welcome to participate at any upcoming Task Force meetings.
Kindly let us know when the next Task Force meeting is scheduled.
Local Law 49 was the first of its kind in the country, and it is essential that New York City continues
to serve as a model for other jurisdictions that are also pursuing transparency in ADS. We hope that
the upcoming report will be a significant step toward achieving governmental transparency.
We look forward to receiving a timely response to our questions and look forward to continuing to
work together in building a transparent and accountable process to ensure that the use of ADS in New
York City is fair and just.
Sincerely,
Peter Koo
Chair, Committee on Technology
Council Member, District 20
26 Id. at 22:44.
40 Rector Street, 9th Floor New York, New York 10006 www.StopSpying.org | (646) 602-5600
SUBMISSION OF
ALBERT FOX CAHN, ESQ. EXECUTIVE DIRECTOR
SURVEILLANCE TECHNOLOGY OVERSIGHT PROJECT, INC.*
FOR THE NEW YORK CITY AUTOMATED DECISION SYSTEMS TASK FORCE: PUBLIC FORUM ON TRANSPARENCY
May 30, 2019
* My sincerest thanks to James Blum, S.T.O.P. Civil Rights Intern, for his invaluable assistance in preparing these remarks.
Statement of Albert Fox Cahn, Esq. – ADS Task Force 5/30/2019 Page 2 of 4
Good evening, my name is Albert Fox Cahn, and I serve as the Executive Director for the Surveillance Technology Oversight Project (“STOP”). STOP advocates and litigates for New Yorkers’ privacy rights, fighting discriminatory surveillance. I thank the members of the Automated Decision System (“ADS”) Task Force for providing this public forum and the opportunity for public comments on the Task Force’s crucial work.
Students have long been taught the adage “show your work” as a reminder that process can be as important as outcome. We should demand no less from the city agencies who control those students’ educational fates, nor from the Task Force meant to safeguard those students, and all New Yorkers, from the growing impact of ADS.
I. ADS Transparency Reduces Discrimination and Legal Challenges
There is no reason for agencies to resist ADS transparency the way they have, as it serves both the interest of the public and the agencies. Nationwide, we see the consequences of hasty and covert ADS adoption. Arkansas’s disastrous 2016 transition to algorithmic Medicare benefits haphazardly rolled-back attendants’ hours and left vulnerable patients without clean clothing or even food.1 When the cuts were challenged, Arkansas failed to defend an algorithm it did not understand in court.
Idaho transitioned to an opaque ADS in 2011 that severely cut Medicaid services for Idahoans with developmental disabilities. As in Arkansas, the cuts were challenged. And, as in Arkansas, the agency lost.2 In the end, Idaho settled to scrap the ADS and develop a replacement system with the input and consent of affected Idahoans.
ADS promise to increase efficiency and cut costs, but faulty systems will do neither. New York decisionmakers learned this lesson at the expense of large swaths of the Bronx.3 Transparency and community engagement throughout the ADS development cycle mitigates these harms and promotes ADS that best serve New Yorkers.
II. Best Practices in ADS Transparency
This Task Force, by ensuring agency transparency in ADS adoption and use, can protect New Yorkers from discrimination, ensure the rights of the city’s most vulnerable residents, and limit future agency liability. In drafting its recommendations, the Task Force should look to the research community for best practices.
ADS complexity often confounds disclosure efforts. Without adequate explainability tools and proper training, decisionmakers may not know a model’s methodology or limits. And, decisionmakers may be unduly deferential to the model or unable to explain the ADS’ role in a particular decision. “Model cards” that explain a model’s methodology and limits should be
1 Ledgerwood v. Ark. Dep’t of Human Services, No. 60CV-17 (Pulaski Ct. (ark.) Cir. Ct. Jan. 26, 2017). 2 See K.W. ex rel. D.W. v. Armstrong, 789 F.3d 962 (9th Cir. 2015). 3 Joe Flood, Why the Bronx Burned, N.Y. POST (May 16, 2010) https://nypost.com/2010/05/16/why-the-bronx-burned/.
Statement of Albert Fox Cahn, Esq. – ADS Task Force 5/30/2019 Page 3 of 4
considered to properly limit human deference to ADS.4 The Task Force should consider human-training practices that teach decisionmakers how bias (conscious and unconscious) impacts ADS outputs and informs decisionmakers of the danger of “automation bias.”5 These best practices that inform and train decisionmakers can safeguard against arbitrary, unexplainable, and therefore opaque applications of ADS.
III. The Task Force Should Compile a List of Existing ADS
Opening Task Force meetings to public scrutiny is necessary but not sufficient to promote public discourse.6 Nearly every New Yorker has encountered an ADS. And, nearly every New Yorker was and is unaware of those encounters. A comprehensive list of active ADS will increase public awareness and engagement. And, since the Task Force’s effectiveness depends on public engagement, it cannot make meaningful recommendations without a list of active ADS.
The Task Force must compile a list of active ADS to make meaningful recommendations for ADS use and adoption. This Task Force may have been first in the nation but it was not first in the field. Legal scholars and data scientists have written at length about fairness, accountability, and transparency in automated systems, often with the understanding that overly generalized, academic recommendations have clear limits.7 A law review article or white paper to add to the towering stack is not the intended end product of this Task Force. Its recommendations should effectuate the academic ideas of this growing interdisciplinary field in New York City.8 To do so, the Task Force must know how ADS operate in New York City. Which agencies use them? Who developed them? Where is their impact felt? Before compiling a list and answering these questions, the Task Force’s work will be little more than academic.
4 Model cards explain the training materials, methodology, limitations, known biases, and unknown or untested capacities that the models might harbor. Understanding how narrow the focus of a model is, or whether it includes racial features or racial proxy features like zip Code can impact the decision-making of a human agent involved in reading the output of the algorithm. 5 “Automation bias” is the phenomenon that people presented with an algorithmic prediction will confirm its truth rather than deny it. As we saw in the Boeing 737 case, poor training can result in catastrophic outcomes, especially when the machines and the humans disagree.
6 Cary Coglianese & David Lehr, Transparency and Algorithmic Governance 71 ADMIN. L. REV. 1, 20-21 (2019) (discussing Fishbowl and Reasoned Transparency). 7 See, e.g., Id. at 29 (noting the limitations on the general analysis of transparency procedures for ADS because any process will “depend on how government actually uses machine learning—and even on what kind of machine-learning algorithm it uses”); Robert Brauneis & Ellen P. Goodman, Algorithmic Transparency for the Smart City 20 YALE J. L. & TECH
103, 136 (2018) (complaining that because there are “no means of knowing how many algorithms are currently in use, who has developed them, or which governments are using them” there is no way “to generalize from [the authors’] finding”). 8 Id. at 29 (noting the limitations on the general analysis of transparency procedures for ADS because any process will “depend on how government actually uses machine learning—and even on what kind of machine-learning algorithm it uses”).
Statement of Albert Fox Cahn, Esq. – ADS Task Force 5/30/2019 Page 4 of 4
IV. The Task Force Meetings Should Be Open to Aid Meaningful Public Discourse
A Task Force created, in large part, to increase transparency should be transparent and must be transparent to be effective. So, we applaud the Task Force’s recent efforts to increase public engagement. But, the two public forums held over the past month and the upcoming community-based events should be the beginning and not the end of these efforts.
For this public discourse to meaningfully inform the Task Force’s upcoming recommendations, the public needs to see the Task Force’s work. Task Force meetings have been kept private over the protestations of the public and members of the Task Force itself. This lack of transparency surprised city councilmembers and the justification for it is unpersuasive. Private meetings without video, audio, or minutes, officials maintain, create a safe space to encourage Task Force members to speak openly about their perspectives. We contest the claim that privacy is necessary to effectively make recommendations meant to promote transparency.
The lethargic pace of the Task Force largely undermines claims that privacy is necessary for Task Force effectiveness. The Task Force has no published draft of its recommendations. And, as of the last glimpse into its progress, the Task Force has no definition of ADS. It appears only marginally closer to fulfilling its legislated duty than it did a year ago at its founding. The time has come for the public to understand what the Task Force is struggling with and be given the opportunity to assist in working through those struggles.
V. Next Steps
New York City led the national movement for ADS transparency. This Task Force has lost that lead. By recommitting to the independence of the Task Force implicit in Local Law 49 and by recommitting to the transparency mandated by Local Law 49, this city can once again lead the march towards the fairer, more transparent, and more effective use of ADS.
1
COMMENTARY
The New York City Task Force on Algorithmic Accountability
Submitted via email: [email protected]
The Legal Aid Society
199 Water Street New York, NY 10038 By: Cynthia H. Conti-Cook (212) 577-3265 [email protected] May 30, 2019
2
Good afternoon. I am Cynthia Conti-Cook, a staff attorney at The Legal Aid Society
testifying on behalf of the Special Litigation Unit in the Criminal Practice, a specialized unit
dedicated to addressing systemic problems created by the criminal justice system. We thank this
Task Force for the opportunity to provide public comment on the various actuarial models that
New Yorkers encounter in the criminal justice system.
Since 1876, The Legal Aid Society has provided free legal services to New York City
residents who are unable to afford private counsel. Annually, through our criminal, civil and
juvenile offices in all five boroughs, our staff handles about 300,000 cases for low income
families and individuals. By contract with the City, the Society serves as the primary defender of
indigent people prosecuted in the State court system. In this capacity, and through our role as
counsel in several civil rights cases as well, the Society is in a unique position to testify about the
importance of a robust, transparent and function police disciplinary system in New York City.
We encourage the task force to examine the tools confronting New Yorkers at every stage
of the criminal justice system, from policing through parole supervision. First I want to identify
some of the systems I know of that fall into the task force’s scope of study:
(1) We know that NYPD has long used COMPstat to decide how to allocate its policing
resources throughout the City. The task force should study any currently operational models
available to NYPD personnel to either identify people, places or types of crime to focus their
resources on, including any modeling built on databases such as the gang or criminal group
database.
(2) After arrest, people are brought in front of a judge who must decide whether or not to set bail.
New laws passed through the budget recently limit the use of risk assessment instruments in that
any model must not result in discriminatory outcomes, either due to design or implementation.
3
Nonetheless, a long-problematic model designed by the Criminal Justice Agency is still being
used and a revision of that tool is currently being redesigned by CJA and the Mayor’s Office of
Criminal Justice. Even the revised tool raises significant questions for advocates and community
members about racial disparities and aggravating factors that are proxies for symptoms of
poverty, and we strongly encourage the task force to evaluate that tool, meet with multiple
stakeholders in the criminal justice system as well as community members to understand the
scope of our concerns.
(3) For those who will have bail set, there is another tool used to determine eligibility for the
Supervised Release program. This tool strongly penalizes subjects based on their age, which has
resulted in an alternative model being created for young people initially excluded from the first
program. With the Human Rights Data Analysis Group, the Legal Aid Society with other NYC
defenders has conducted an analysis of this tool and we would like to share this analysis with the
task force.
(4) In the juvenile systems, there is also a pre-trial detention tool used, designed by Vera, to
determine whether a young person should be released back to their family and community while
charges are pending or whether they should be institutionalized. This tool also deserves the task
force’s attention.
(5) Also in the juvenile system, the Youth Level of Services tool is used by the Department of
Probation to recommend a sentence following an adjudication of guilt. This tool was created by a
Canadian team using Canadian data, we don’t believe it has been locally validated and includes
aggravating factors that identify young people as high risk based on whether they live in single-
family households, have access to extracurricular activities as well as other symptoms of class
rather than inherent risk behavioral symptoms. While probation officers may divert their final
4
recommendations from the model’s outcome, they may only divert by one step below or above
and judges still can override the probation officer’s recommendation. This tool, and how it is
used by the probation department as well as the judges, is an important tool for the task force to
scrutinize.
(6) While there are additional tools used that impact New Yorkers, they are run by state systems,
for example, the Department of Correction and Community Services uses the private vendor
equivant to assess people’s eligibility for parole release and the level of supervision. Human
decision-making discretion has decreased over recent years regarding how people are assessed. If
it is possible for the task force to review these models, despite being contracted for by state
actors, they should because they greatly impact the lives of New Yorkers as well.
Regarding how the task force should evaluate each of these systems, in addition to other
systems in the public sector, I recommend this framework: for each model, ask (1) Who does the
model serve and why it was created? (2) What question does the model ask the dataset and is it a
limited question likely to be assisted by the model and (3) Where does the data relied on in the
model come from?
Using this framework, the task force should consider whether models are created for the
purposes of increasing bureaucratic efficiency, offering cover to decision-makers for
controversial action (such as decreasing the number of people incarcerated), or increasing
fairness? Who demanded a model and were the people who are the subject of the model
consulted regarding the creation of the model so they could weigh in, for example, about whether
the data relied upon is representative of what it purports to be.
5
The task force should ask what question does the model ask the dataset to assess? Often
there is an incongruity between the goal for the outcome or answer the model is asked to produce
and the capacity of the data to answer that question. If the dataset is being queried for a
problematic question like whether someone is dangerous, whether someone will be arrested
again, whether someone is likely to flee a jurisdiction, and if we know that the data we have
could never answer that question, we should pause. Unfortunately, often models are estimating
an adjacent answer based on proxy data, but presenting it as ”the answer” without the proper
contextualization that limits the meaningfulness of the model.
Finally, the task force should ask each model where does the data come from? What are
the data points, how are they created, who creates them, what forces result in them beyond the
control of the person who is the subject of the prediction or assessment? In the criminal justice
context, many of the factors used include arrests, convictions, open cases, warrants, and other
criminal justice data points that are the definition of “garbage in”.
Police officers are not trained as data collectors and their data collection is often tainted
by departmental incentives for productivity, bias, or are just subject to sloppiness because the
data’s accuracy is not the emphasis of their job. Many of these factors related to arrest data have
more to do with the collision frequency people have with police than any other innate
criminality. The number of innocent people pleading guilty to misdemeanors and felonies alike,
the multi-layered pressures of the criminal justice system to plea, the multi-layered pressures of
poverty that cause people to warrant (or not return to court) are all aspects of the data points
these tools often rely upon that make them less reliable, and yet they often carry the greatest
weight in determining someone’s liberty from the model’s perspective.
6
It would also be advantageous for the task force to study the impact of how these models
layer upon each other and result in people being subjected to multiple “risk scores” throughout
their encounter with the criminal justice system. It’s evident that one person, accused of a crime,
maybe subjected to a series of assessments throughout their journey through the criminal justice
system and accumulate the effects of these outputs to their detriment as they are assessed in later
tools. Of course this accumulation will, as mass incarceration in this country already has, amass
in poor communities of color disproportionately and continue the massive divide in this
country.
CONCLUSION
We thank you for hearing our testimony today. We look forward to the Task
Force’s report which we hope New Yorkers can rely upon in order to create meaningful
accountability structures, informed by the needs and values of community members, that are a
minimum to community trust in government, let alone governance by algorithm.