different strokes for different folks experimental ... · different strokes for different folks:...
TRANSCRIPT
DIFFERENT STROKES FOR DIFFERENT FOLKS: EXPERIMENTAL EVIDENCE ON
COMPLEMENTARITIES BETWEEN HUMAN CAPITAL AND MACHINE LEARNING
Prithwiraj Choudhury Harvard Business School
Evan Starr Robert H. Smith School of Business
University of Maryland [email protected]
Rajshree Agarwal
Robert H. Smith School of Business University of Maryland
January 2018 Abstract
The advent of artificial intelligence in the form of machine learning technologies ushers new questions regarding the pace at which it may substitute both older technology vintages and human capital. Rather than assuming superior productivity of one vintage over another, we examine contingency effects on productivity by heterogeneity in prior specialized knowledge bases in complementary human capital, and provision of concurrent expert advice. Within the research context of patent examination by the US Patent and Trademark Office, which has developed both machine learning and Boolean search technologies, we hypothesize that absorptive capacity from skill-technology matches will result in higher productivity, and these effects will be stronger in the presence of expert advice constituting technology and task specific information. We test and find support for our hypotheses using experimental methods that permit causal inferences and examination of underlying mechanisms. Our study contributes to literature streams on artificial intelligence, endogenous technological change, and strategic management of the pace of technological substitution by providing insights on complementarities between technologies and horizontally differentiated human capital.
1
Artificial intelligence (AI) may well transform future production and work. In its current
development, AI relates to machine learning, where programmed computers learn from existing data to
conduct mental tasks and facilitate decisions (Agrawal, Gans and Goldfarb, 2017; Brynjolfsson and McAfee
2014). Bughin et al., (2017) predict machine learning, when combined with robotics, may increase global
productivity growth by 0.8-1.4%, while automating nearly 50% of current work performed by humans. AI has
also ushered debate on the substitution of humans by intelligent machines, and on who may benefit or lose
(Autor 2015; Benzell, Kotlikoff, LaGarda, and Sachs, 2015; Bessen 2016; Brynjolffson and McAfee 2014).
Budding empirical research compares decision making and productivity of machine learning vs. humans, and
wage and employment effects on vertically differentiated (skilled vs. unskilled) labor (Acemoglu and Restrepo
2017; Aghion, Bergeaud, Blundell and Griffith, 2017; Cowgil, 2017; Hoffman, Kahn, and Li 2017). At least in
the foreseeable future though, workplaces will require humans to interface with machines, so left unexamined
are factors at play for the pace of technological substitution (Adner and Kapoor, 2016; Adner and Snow,
2010; Christensen, 1997) of machine learning to older technology vintages. In particular, we lack an
understanding of how horizontal differentiation in skilled labor (caused by specialized knowledge domains)
may be differentially productive when matched interact with different technology vintages for productivity
differentials. We seek to fill this gap by comparing the productivity of a machine learning technology being
developed by the United States Patent and Trademark Office (USPTO) to the existing technology, based on
its use by individuals with different skills and experience. Using experimental methods, we address the
following research questions: As organizations introduce machine learning in the presence of old vintages,
how does heterogeneity in prior specialized knowledge bases of complementary human capital impact
productivity? What effect does concurrent provision of expert advice constituting technology and task-specific
knowledge have on productivity?
Answers to these questions are important to strategic renewal of firms, as managers must choose
whether and when to adopt new technologies, and how to create positive rather than negative
complementarities with existing resources and capabilities (Bloom, Sadun and Van Reenan, 2012; Christensen
1997). Misalignment of potentially productive new technologies with existing firm capabilities can lead to
failure (Henderson 2006). Chief among existing capabilities is the firm’s workforce (Hatch and Dyer 2004),
2
given gains from productivity depend crucially on the interface of workers and technology (Autor, Levy, and
Murnane 2003; Bapna et al., 2013; Chari and Hopenhayn 1991; Teodoridis, 2017). The answers are also
important at social and individual levels, inasmuch as economic progress and social welfare requires
concomitant attention to effects of new technologies on individuals’ skill obsolescence and refurbishment,
and resultant unemployment and inequality (Bekman, Bound and Machin, 1998; Fernandez, 2001).
Our theory builds on the vintage-specific human capital literature (Chari and Hopenhayn, 1991) to
acknowledge its importance for unlocking productivity effects of any technology. We further integrate the
insight that highly skilled workers invest in developing expertise in knowledge domains (Schultz, 1961; Chase
and Simon, 1973) leading to horizontal differentiation in their prior knowledge base due to specialized skills
and experience. Using absorptive capacity logic, we hypothesize a moderating effect of prior knowledge base
on the productivity of a particular technology vintage. Specifically, we argue machine learning has higher
complementarities with computer science and engineering (CS&E) based human capital, while the older
technology vintage has higher complementarities with non-CS&E based human capital. We denote these
combinations of skills and technologies as skill-technology match. Further, we theorize that expert advice—
the provision of technology and task-specific tacit information—will enhance rather than attenuate the skill-
technology match effects on productivity of the technology.
Our research context and experimental design leverages two technologies developed by USPTO to
expedite and improve the quality of patent application examinations—the older vintage of Boolean search,
and the newer vintage of machine learning. Patent examiners, and innovators seeking patents at large, draw
from highly specialized knowledge domains representing biomedical, chemical, computer, mechanical and
natural sciences and engineering. Moreover, the patenting process also engages other specialized knowledge
domains (e.g. law, business). Accordingly, and with the help of USPTO personnel, we test our hypotheses by
conducting an experiment with 221 MBA subjects at a top-tier business school. Participants received training
to simulate the role of a patent examiner and adjudicate a patent application based on searches for prior art
using either Boolean or machine learning technology. We examine the efficacy of the two technologies, when
used by randomly assigned participants who differ in their prior CS&E human capital. We also randomized
3
the provision of expert advice to participants, in the form of communication from a USPTO patent examiner
with tips regarding the usage of the specific search technology for the task at hand.
Previewing our findings, there is strong evidence for a skill-technology match: prior CS&E
knowledge is essential for success using machine learning technology, and prior non CS&E knowledge
complements more strongly with the older Boolean logic technology vintage. Moreover, these effects are
only unlocked in the presence of expert advice, underscoring the importance of concurrent task and
technology specific information. Indeed, the only instance in which we see success is when expert advice is
provided to those whose skills are pre-aligned with the technology. Subsequent analysis reveals one
mechanism underlying these results is the inability of non CS&E examiners to integrate expert advice within
the user interface underlying the machine learning technology.
The paper contributes to a number of related literature streams at the intersection of innovation and
human capital. First, while debate abounds for the future replacement of humans at work in the era ushered
by artificial intelligence and machine learning (Autor 2015; Benzell et al., 2015; Bessen 2016; Brynjolffson and
McAfee 2014), there is scant attention to the immediate implications for productivity of machine learning
substitutions for older technology vintages, particularly as this pace of technology substitution may be
conditioned by complementary human capital knowledge bases. Our study highlights that even within the
high-skilled labor force, horizontal differentiation due to specialization of human capital accounts for
significant productivity differentials when interfacing with different technology vintages. These differentials
have not been incorporated in theories for either artificial intelligence/machine learning’s effect on
productivity and workplace practices (Aghion et al., 2017; Agrawal et al. 2016, Autor 2015; Benzell et al.
2015), or in theories of technology diffusion (Chari and Hopenhayn, 1991; Jovanovic, 2009). Both theoretical
streams assume away horizontal differentiation in prior knowledge bases, even as they distinguish between
either vertical human capital differences (i.e. lousy or lovely jobs), or vintage specific human capital (i.e.
technology specific learning by doing). In contrast, our study theorizes and shows within the high-skilled
labor force, and within same technology vintages, individuals exhibit heterogeneous productivity based on
whether their prior knowledge base are matched with the technology with which they interface. Given
specialized knowledge bases distant from CS&E will be critical to productivity because of judgment, even if
4
not prediction (Agrawal et al., 2017), these horizontal differences in specialized knowledge cannot be assumed
away as being rendered obsolete.
To the strategic management of innovation literature stream (Adner and Snow, 2010; Adner and
Kapoor, 2016; Agarwal and Prasad, 1999; Christensen, 1997), we contribute by noting new technologies may
fail to diffuse rapidly in spite of potential relative superiority because of their interactions with complementary
human capital. Finally, our study also contributes to the human capital and career specialization literature
(Becker and Murphy, 1992; Rosen, 1983; Zuckerman et al., 2003)—while scholars have noted the beneficial
effects of human capital investments in developing expertise (e.g. Castanias and Helfat, 1991), our study
highlights the need to develop a basic level of generalized skills for interfacing with a focal technology, to the
extent newer technology vintages shift the knowledge base requirements for productivity gains.
These finding are replete with important managerial implications, given the firm’s pre-existing
workforce is its most important capability, and also the most difficult to change. We suggest one reason firms
may fail following adoption of a seemingly profitable technology, despite training to update skill sets of their
workers, is that the new technology places differential emphasis on some specialized skill sets over others.
While a firm may invest in significant technology-specific training to help workers adapt to a new technology,
our results suggest such training will not help all firms, or all workers, equally: the prior knowledge bases of
the workforce will interact differently with new technologies, requiring firms and workers to re-think existing
complementarities between specialized knowledge domains, even as they consider the complementarities
between workers and technologies. Our study offers a cautionary note against a “one size fits all” training
regimen that is technology specific, and calls for more nuanced programs where the training is specific to the
technology-skill match. Similarly, to the extent that hiring, retention, innovation management, and
compensation are ultimately tied to productivity differentials based on available technologies, the implications
of our study extend to a careful re-thinking of all these practices.
BRIEF LITERATURE REVIEW
Artificial Intelligence and Implications for Productivity and Workplace Practices
Artificial intelligence (AI) is defined as “the capability of a machine to imitate intelligent behavior”
(Merriam-Webster, 2017), and in its current technological state takes the form of machine learning, where
5
computers improve their learning over time autonomously, through programs which utilize additional
observational data and information from real-world interactions (Faggella, 2017). AI represents the latest
technological discontinuity within the larger context of automation dating back into time, for sure to the
industrial revolution (Autor, Levy and Murnane, 2003; Hounshell, 1985; Mokyr, 1990). From a socio-
economic perspective, AI is bringing back to fore age-old issues regarding implications of technological
breakthroughs for productivity and management practices, as well as changing human capital and skills
composition for individuals at work (Brynjolffson and McAfee 2014, Autor 2015, Bessen 2016). While similar
to earlier technologies in resurrecting the debate on whether AI substitutes or complements labor, the
difference from automation which mainly affected labor employed in physical, routine or low-skilled tasks is
in AI’s potential to substitute humans in non-routine, cognitive tasks performed by high skilled workers
(Jones, Aghion, and Jones 2017; Zeira, 1998). Depending on whether scholars take the perspective where
humans share significant complementarities with machines (Agrawal et al. 2016, Autor 2015), or one where
machines are superior substitutes to humans (Benzell et al. 2015), the debate is not just about “lousy and
lovely jobs” (Autor et al., 2003; Goos and Manning, 2007), but whether AI represents a technological
singularity where people are completely replaced by machines (Jones et al., 2017), to the point where they
“immiserate humanity” (Benzell et al. 2015).
Within this debate, some scholars have provided reasons why, at least in the foreseeable future, AI’s
substitution for humans at work is overstated (Agrawal, et al. 2017; Autor, 2014; Jones et al., 2017). These
reasons rely on both theories regarding whether AI is capital or labor augmenting, and on the comparative
advantage of humans vs. machines in different types of tasks. Agrawal et al. (2017) break down the anatomy
of a task into data, prediction, judgement and action. Noting that machine learning, the current state of AI
technology, is applicable largely to prediction, they highlight the importance of humans for judgment—the
ability to make considered decisions by accounting for the impact actions have on outcomes in light of
prediction (Agarwal et al., 2017). In a similar vein, Autor (2014) builds on codified vs. tacit knowledge to
highlight complementarities between machines’ comparative advantage in routine, codifiable tasks, and
humans’ comparative advantage on tasks requiring tacit knowledge, flexibility, judgment and creativity.
6
Two limitations arise in the current discourse on AI, where theoretical models make different
assumptions about machines and humans in the underlying production function, task structure, and even
rates at which past investments by “dead high-tech workers” can replace effort by current period high-tech
workers (Aghion et al., 2017; Autor, 2015; Benzell et al., 2015). One, notwithstanding a budding set of
empirical papers (Acemoglu and Restrepo, 2017; Cowgil, 2017; Graetz & Michaels, 2015), there is a dearth of
empirical evidence on the effects of machine learning (the current state of artificial intelligence) on
productivity, relative to current alternative technologies. Two, the models ignore the literature on technology
diffusion, which notes new technologies may take prolonged periods of time for large scale adoption, and
firms/individuals continue to invest in old technologies even in the presence of newer regimes (Adner and
Kapoor, 2016; Chari and Hopenhayn, 1991; Mansfield, 1968), which we turn to next.
Vintages of Technology and Human Capital
Endogenous models of technological change highlight a new technology’s diffusion relative to
existing vintages depends not only to its “objective” superiority, but also on the technology’s interaction with
complementary resources and capabilities. Strategy scholars note both emergence challenges faced by new
technologies, and extension opportunities available for old technologies (Adner and Kapoor, 2016; Adner and
Snow, 2010). Emergence challenges to new technologies relate to bottlenecks posed in existing ecosystems in
the form of complementary products, resources and capabilities (Adner and Kapoor, 2016). Also, incumbents
may accelerate investments in old technologies to stave off competition from new technology entrants, or
exploit demand heterogeneity and consumer preferences for the old technology (Adner and Snow, 2010).
Similarly, models of endogenous technological change provide mechanisms underpinning prolonged periods
of technology diffusion, relating the time lag to its relative superiority to existing technology vintages and the
distribution of human capital familiar with each technology vintage (Chari and Hopenhayn, 1991; Jovanovic,
2009). Empirically, scholars examining technology diffusion have implicated differences in individuals and
workplace practices as critical to the technology’s acceptance and productivity (Agarwal and Prasad, 1999;
Black and Lynch, 2001; Greenwood, Agarwal, Agarwal and Gopal, 2017).
A salient difference of diffusion models, then, from models in the preceding section where AI and
machine learning may (exogenously) complement or substitute for human capital, is the focus not only the
7
vintage of the technology (new vs. old), but also the vintage of the human capital interfacing with the
technology. Given each technology ushers with it different tasks, and also changes the composition of tasks
performed by machines or humans (Autor, et al., 2003), Chari and Hopenhayn (1991) define vintage human
capital as technology-specific skills which relate to both task-specific human capital (Gibbons and Waldman,
2004) accumulated through learning by doing, and the evolution of these tasks across technology vintages.
Thus, the literature on technology diffusion acknowledges both how strategies impact the pace of
technology substitution, and the importance of developing vintage specific human capital. However, it
abstracts away from within technology vintage heterogeneity in human capital: productivity of different vintages of
technology may not only entail complementarities between higher vs. lower levels of human capital and skills
(Jovanovic, 2009), but also complementarities with heterogeneous human capital that is qualitatively distinct
(rather than vertically differentiated), given specialization and division of labor (Becker and Murphy, 1992;
Schultz, 1961). Thus, even within a high skilled labor force, differences in prior knowledge bases may create
differences in productivity as they interact with successive technology vintages.
Taken together, the above literature streams along with their limitations sets the stage for our study’s
research context and design, where we examine productivity differentials between two technology vintages, as
contingent on the heterogeneity of prior knowledge embodied in the individuals who are interacting with
both technology vintages, and the provision of expert advice which facilitates the development of vintage
specific human capital through transmission of technology and task specific information.
RESEARCH CONTEXT: USPTO PATENT EXAMINATION PROCESS
Our research design exploits the use of two technology vintages for use by the USPTO examiners
when reviewing patent applications.1 The older vintage relies on Boolean keyword searches, while the newer
vintage utilizes machine learning. We provide brief descriptions of the patent examination process and each
of these techniques below, with more detailed information in Appendix A and B.
1 Patents protect inventors’ intellectual property rights by “exclud[ing] others from making, using, offering for sale, or selling the invention throughout the United States or importing the invention into the United States.” (https://www.uspto.gov/learning-and-resources/electronic-data-products/issued-patents-patent-grants-data-products, accessed July 2017). Anyone who wishes to secure the rights to an invention in the U.S. must apply for a patent from the USPTO. Patent applications include an abstract, which describes the patent filing, and a list of specific claims, which specify the precise components of the invention that the applicant wishes to patent.
8
The Patent Examination Process
To meet the legal standards for a patent, claims must be both novel (i.e., a new, unique invention)
and non-obvious (i.e., sufficiently inventive), and the USPTO examiners review patent applications, compare
it to prior art (i.e., all existing patents, patent applications, and published materials), and determine whether an
inventor’s claims meet these two criteria. In maintaining a balance between expediency and patent
examination quality, the USPTO aims to provide at least a first office action within 10 months of receipt
(Brinkerhoff, 2014). However, increases in the number of patent applications received by the USPTO (e.g.
an 18 % increase in the five year period between 2010 and 2014)2 have created significant delays—over half a
million backlogged applications, and current processing of first office action at more than sixteen months3—
in spite of USPTO investing significant resources to increase the number of examiners by 28% from 2010 to
2014 (Crouch, 2014; Choudhury, Khanna and Mehta, 2017). The Government Accountability Office report
on intellectual property (GAO, 2016) notes these challenges are further exacerbated by recent increases in
disputes over patent validity, raising concerns “the USPTO examiners do not always identify the most
relevant prior art, which has resulted in granting some patents that may not meet the statutory requirements
for patentable inventions” (p. 2) To understand the reasons behind the above challenges and the potential
machine learning technology solution, we provide a brief description of the patent examination process (See
Appendix B, Exhibit 1, and Figure C1 for more details), and human capital and technologies utilized in the
examination process.
Upon application receipt, the USPTO classifies it based on technical subject matter to one or more
of classifications among thousands of different technologies. This classification routes the application to one
of eleven technology centers which collectively cover electrical, chemical and mechanical technology fields.
The eleven technology centers are further subdivided into smaller “art units” consisting of patent examiners
who are clustered based on expertise on related technologies within the unit. The technical competence of
2 U.S. Patent and Trademark Office, “U.S. Patent Statistics Chart Calendar Years 1963-2015,” June 15, 2016, https://www.uspto.gov/web/offices/ac/ido/oeip/taf/us_stat.htm, accessed July 2017. 3 U.S. Patent and Trademark Office, “First Office Action Pendency,” June 2017, https://www.uspto.gov/corda/dashboards/patents/main.dashxml?CTNAVID=1004, accessed July 2017 and U.S. Patent and Trademark Office, “June 2017 Patents Data, At a Glance,” https://www.uspto.gov/dashboards/patents/main.dashxml, accessed July 2017.
9
patent examiners is key for identifying potential prior art and to make judgments regarding claims. All patent
examiners represent highly skilled workers, but have great diversity in their knowledge specialties. Patent
examiners specialties include biomedical, chemical, computer, electrical, industrial and mechanical science and
engineering. Additionally, the General Schedule (GS) levels of patent examiners represent different levels of
seniority/experience, productivity, and management responsibilities. Extensive training programs provided to
the patent examiners, both as part of their onboarding experience and for continuous learning, include
Quality Enhancement Meetings (QEM), Patent Examiner Technical Training Program (PETTP), Scientific
and Technical Information Center (STIC) lunch and learn/individual sessions, technology fairs, as well as
knowledge management and collaboration systems and internal shared databases for search techniques
developed by other examiners.4
The assignment to patent examiner within an art unit is based on examiner workload rather than
complexity of patent. The patent examiner adjudicates on novelty and non-obviousness of the application by
comparing it to prior art, and also determines the appropriate scope (intellectual property boundaries) of the
patent based on novelty of claims. To review prior art, examiners search within various databases, including
the Patent Full-Text and Image Database (PatFT) and Patent Applications Full Text (AppFT) owned by the
USPTO, but open to the public (see Appendix B, Exhibit 2 for the PatFT interface). Examiners also rely on
searches in other databases to locate prior art stored elsewhere (e.g., published articles, trade journals). This
step in the process is the most important and also the most laborious (GAO, 2016). In part, this is because
of the characteristics of prior art and patent applications: patent examiners often deal with non-uniform
language, intentionally vague or broad patent claims, and missing relevant references, which increase prior art
search costs. However, some of the challenges relate to the processes and tools utilized when conducting
prior-art search, which we turn to next.
Technology Vintages Currently Used in Prior Art Search
The current mainstream technology for prior art search tools available to patent examiners (and
accessible to others in public versions) are named EAST (Examiners’ Automated Search Tool) and WEST
4 U.S. Patent and Trademark Office, “Prior Art Frequently Asked Questions,” July 28 2014, https://www.uspto.gov/sites/default/files/patents/init_events/prior_art_faq20140728.pdf accessed November 2017
10
(Web-based Examiners Search Tool), and rely on Boolean logic.5 Patent examiners use Boolean operators to
conduct structured searches using keywords and strings with the available databases to yield potential prior art
citations. As Alcacer and Gittelman (2006) document, two-thirds of prior art citations on the average patent
are inserted by examiners. When evaluating the current search tools and capabilities, GAO (2016) noted 48%
of the patent examiners interviewed/surveyed found these search tools made identification of prior art easier,
but 17% found them to make it more difficult. More than half of the patent examiners thought the tools
available to them in EAST/WEST as “less advanced and more reliant on keyword searching than other
available tools” (p 24). The report stated the desire expressed by the majority for search engines for increased
automation: both in search for concepts and synonyms related to keywords entered by the examiners, and in
identification of prior art based on information already included in the patent application (claims and
specification) without any reliance on patent examiner entered keywords.
In 2016, the USPTO invested resources in developing a machine learning tool called Sigma (Krishna
et al., 2016). Sigma increases automation in the search for existing patents (see Appendix B Exhibit 3).
Examiners manipulate the search algorithm by selecting patent application components (e.g., the abstract,
title, or description) they believe are most relevant, and also enter the most important search terms into a
“word cloud”.6 Sigma, in turn, adjusts the algorithm accordingly and retrieves a more refined set of search
results. Over time, Sigma learns the examiner’s habits to suggest searches based on the examiner’s prior
behavior. Sigma is currently being tested and refined for adoption. Importantly, while the first and critical
stakeholder group are the patent examiners whose productivity depends on the search technologies, as with
the earlier technology vintages, the USPTO intends to make these tools available to the general public, to
benefit the population of potential inventors and the personnel they employ and facilitate their patent
application process. Accordingly, the larger goal when investing in machine learning technologies relates to
their ease of use for all relevant stakeholders.
THEORETICAL PREDICTIONS
5 U.S. Patent and Trademark Office, “Prior Art Frequently Asked Questions,” July 28 2014, https://www.uspto.gov/sites/default/files/patents/init_events/prior_art_faq20140728.pdf accessed November 2017 6 Word clouds visualize text data by increasing the size of more important search terms and reducing the size of those deemed less important.
11
To begin with, we note our focus is on workers not as developers of machine learning technologies, but
as users. As would be expected, the development and integration of machine learning and artificial intelligence
into commercially viable technologies and products is performed by those who possess computer science and
engineering (CS&E) knowledge. However, once developed, the technologies are used to increase predictive
power by a heterogeneous workforce specializing in different knowledge contexts, and these other knowledge
bases retain their importance for other task components such as decision making and judgment (Agrawal et
al., 2017). For example, recent evidence in human resource management suggest managers can improve
recruitment and retention by incorporating machine learning methods into their decision making (Cowgil
2017; Hoffman, Kahn, and Li 2017). Thus the question of how the adoption of machine learning
technologies affects productivity for both CS&E and non-CS&E workers, relative to existing vintages, is of
great relevance. In our research context, the machine learning tool Sigma is intended for use not only by
patent examiners who vary in their specialized knowledge domains within STEM (Science, Technology,
Engineering and Math) fields, but ultimately by all interested stakeholders. Within the more general
population, individuals interested in intellectual property largely comprise of a high skilled workforce with
both STEM and non-STEM (e.g. patent attorneys, management personnel) backgrounds. Accordingly, our
theoretical predictions below relate to the moderating effects of heterogeneity of knowledge bases in the
users of a technology vintage on the technology’s productivity. We examine in particular the role of prior
specialized knowledge bases embodied in the user’s human capital, and the concurrent provision of expert
advice constituting technology and task-specific information.
Production Complementarities between Technology and Prior Specialized Knowledge
In Figure 1, we define the four cells of the matrix mapping prior knowledge base of individuals (non
CS&E and CS&E) against the two technology vintages (Boolean and Machine Learning). We start by
recognizing that each technology requires specific skills with complementary human capital (Chari and
Hopenhayn, 1991). Within our research context, and more generally, the skillsets required for interacting with
machine learning and Boolean logic technology are quite different. The use of the machine learning tools
such as Sigma, requires interfacing with the technology for choice of inputs into the word cloud, assignment
of weights for the different patent application components, followed by reading/interpreting/modifying the
12
visual cues to enhance prediction and facilitate judgment (Krishna et al. 2016). On the other hand, the use of
Boolean technology requires composing search strings, followed by reading/interpreting the resulting
documents and subsequently modifying search strings.
[Insert Figure 1 here]
When exposed to identical technologies, differentiation in prior knowledge bases will create differences
in absorptive capacity (Cohen and Levinthal, 1990), thus affecting the rate at which individuals learn and
develop technology-specific skills. Accordingly, productivity differentials will arise due to higher investment
costs incurred by workers whose prior knowledge base is more distant to the new technology than for those
whose prior knowledge base is more similar. As a result, the match of skills based on prior specialized
knowledge to the technology is an important conditioning factor for a technology vintage’s relative productivity.
We posit workers with CS&E knowledge bases (Cell 1) will be more productive with machine
learning technologies relative to those with non-CS&E knowledge bases (Cell 3). This is because the user-
interface of machine learning technologies will be more familiar to workers with prior CS&E knowledge
bases, than for those with non CS&E knowledge. In their seminal article on absorptive capacity, Cohen and
Levinthal (1990) note an individual’s prior related knowledge is key to their ability to absorb new knowledge.
They remark, “students who … mastered the principles of algebra find it easier to grasp advanced work in mathematics such as
calculus; …Pascal (computer programming language) students learned LISP (another programming language) much better
because they better appreciated the semantics of various programming concepts” (p. 130). Similarly, prior computer
programming knowledge will provide CS&E workers a better skill-set needed to interface with machine
learning tools. Prior knowledge gained from working on computer software tools that involve manipulating
user interfaces and interpreting visual cues should enhance their productivity when interacting with the tool,
relative to non-CS&E workers who may lack these skills and expertise.
In contrast, non-CS&E workers’ skill sets will be a better match for the Boolean technology (Cell 4),
relative to machine learning technology (Cell 3) to enhance prediction and facilitate judgment. Within highly
skilled workers specializing in non-CS&E domains, the logic of the Boolean search algorithms is largely
understood, given widespread availability of Boolean search engines. So, non-CS&E workers possess these
more generalized skills necessary for conducting and modifying Boolean string searches. Within our research
13
context specifically, the older technology vintage for identifying prior art references requires core skills not so
much in the technology’s user interface, but in reading documents generated through Boolean search.
Turning to productivity differentials of the diagonal terms with the other off-diagonal term, Cell 2,
representing CS&E workers on Boolean, we expect all high skilled workers to possess the generalized skills
for interfacing with the older vintage. Nonetheless, while perhaps not an absolute advantage, workers with
prior CS&E knowledge should have a comparative advantage in machine learning technology (Cell 1) relative
to Boolean logic technology (Cell 2), inasmuch as their computer skills are better matched to modification of
programs, relative to speedy reading and interpretation of resulting documents. The same comparative
advantage applies to non-CS&E workers on Boolean technology (Cell 4), relative to CS&E workers on
Boolean (Cell 2). Altogether, our logic implies that the diagonal Cells 1 and 4, which we denote as a skill-
technology match, should have higher productivity relative to the off-diagonal Cell 3, and at least as much
productivity as the off-diagonal Cell 2. Accordingly, we hypothesize:
H1: Those with a high skill-technology match will have higher productivity than those with low skill-technology match.
Provision of Concurrent Expert Advice and Productivity from Skill-Technology Match
While the above hypothesis focused on the prior knowledge base of workers interfacing with different
technology vintages, productivity will also be affected by whether workers receive concurrent knowledge as they
perform tasks using each technology. When working with a given technology vintage, individuals have to
learn-by-doing, and make costly, technology-specific investments (Chari and Hopenhayn, 1991). The
diffusion of a new technology vintage, relative to the older vintage, is then a function of how quickly workers
navigate the learning curve (Argote and Epple 1990). In addition to their own learning efforts when
navigating new technologies and workplace practices, workers also benefit through vicarious learning,
particularly from experts in the domain (Coleman, Katz and Menzel, 1957; Dasgupta and David, 1994;
Greenwood et al., 2017; Thornton and Thompson, 2001). For example, Dasgupta and David (1994) highlight
the importance of “context which makes focused perception possible, understandable and productive” (p.
493), and note the important role of expertise acquired through experience and transferred by demonstration,
personal instruction and by provision of expert services such as advice, consultation, etc. Empirical research
14
substantiates the increases in productivity of a new technology when workplace practices include the
provision of technology and task specific information (Bapna et al., 2013; Bartel, 1994; Black and Lynch,
2001), because the codification and transmission of tacit know-how and know-what increases the speed at
which followers learn from pioneers’ learning by doing and trial and error processes with the new technology
(Edmondson, Winslow, Bohmer and Pisano, 2003). Consistent with this large literature, we expect provision
of expert advice—which we define as transfer of information that is both task and technology specific from
experts to workers—will increase the productivity of all workers when interfacing with either technology.
H2: Workers will be more productive with a given technology when they are provided with expert advice.
While the above is a baseline expectation, what is unclear from the received literature is whether the
provision of expert advice will attenuate or exacerbate the productivity differentials we predicted as arising
from a skill-technology match. Provision of expert advice may reduce the gap created by prior knowledge
base. This will likely occur because the expert advice helps workers with more distant knowledge bases
overcome confusion regarding interfacing with the technology. However, the skill-technology match will also
impact the rate at which workers absorb and translate expert advice into productivity. We believe the latter
effect to dominate: while expert advice will benefit all workers, we expect the differential effect to be
stronger in the presence of a skill-technology match. Here too, the fundamental logic relates to absorptive
capacity; higher skill-technology matches will facilitate speedier absorption and transformation of advice into
new learning. Harking back to the old adage, “I can teach it to you, but I cannot learn it for you,” our
reasoning builds on the notion that expert advice interacts with the knowledge base for greater improvements
in productivity, because the worker’s skill-technology match enables them to incorporate the full value of the
information provided. Summarizing these arguments yields the following hypothesis:
H3: Those with a high skill-technology match will better integrate expert advice into productivity than those with low skill-technology match.
RESEARCH DESIGN AND EMPIRICAL METHODOLOGY
Description of Experiment
The ideal design to test our hypotheses is one in which (1) subjects do not self-select into
experiment, (2) technology vintage type and expert advice is randomly assigned, (3) there are ways to assess
15
potential contamination between treatment and control group and (4) it is possible to not only causally
identify the relationships of interest, but also shed light on the underlying mechanisms. We briefly describe
the experiment before describing how our research design allows us to address each of these concerns.
The participants in this experiment were 221 graduate students enrolled in three sections of a course
at a top tier business school, who were asked to “examine” a patent with five claims over a period of five
days. We chose experimental subjects, i.e. MBA students rather than patent examiners for two reasons. The
use of patent examiners would translate into an inability to control for unobservable differences across
subjects in prior task and technology specific learning by doing, which could bias results and create concerns
regarding our cross-technology comparisons. In contrast, MBA students have no prior experience in either
patent examination or the use of either technology, so they better simulate new workers (with differences in
prior knowledge bases) being assigned to different technologies. Secondly, our field interviews indicate the
USPTO intends both technologies to be available for general public use. The heterogeneity in prior
backgrounds of MBA students may be more representative of the skilled labor force at large, thus our results
would inform likely productivity differences for the larger stakeholder groups.
The research team simulated conditions encountered by patent examiners in designing the patent
examination exercise with invaluable help from USPTO officials. First, we determined the time period to be
allotted for examination of the patent claims, and also ensured task manageability given expectations of time
investments by the subjects participating in the experiment. In consultation with the USPTO officials, we
determined 0.6 hours per claim to be reasonable; the additional 33% of time allotment per claim provided a
cushion given the newness of the task to the participants.7 Second, the USPTO officials helped identify a past
patent application of average complexity, to be used for the experiment. The specific patent application had
been examined in 2010, and was rejected based on a prior art citation that invalidated its novelty claims (see
Appendix B Exhibits 4 and 5). Given the applicant had abandoned the process after the claims were
rejected, no record of the patent examination results were available to our experimental participants (e.g. as
could be identified through a Google search). To pare down the time investments to manageable levels, the
7 As Lemley and Sampat (2012) document, while the average patent prosecution duration at the USPTO is three to four years, the examiner spends on average 18 hours examining the claims. Also, Alison and Lemley (2000) document an average patent has around 14.9 claims, indicating a USPTO examiner takes around 0.4 hours to examine a single claim.
16
USPTO officials identified 5 claims from the 39 claims made in the original application which were deemed
most relevant for the outcome of the examination. Third, the information technology department at the
business school worked with USPTO officials to ensure smooth functioning of both technologies—Boolean
and Machine Learning, and capture the work-logs of participants as they interfaced with the technology.
The actual experiment proceeded over two weeks, as follows. In the first week, participants were
trained on details of the patent examination task by officials from USPTO, and provided information similar
to new patent examiner hires. Here, they were introduced to both search technologies, but at this point,
participants were unaware of which technology would be assigned to them. At the end of week 1, participants
were randomly assigned to a technology—Boolean or Machine learning (see details in Appendix A) using a
random number generator. Each participant received additional online documentation and general training
materials to ensure familiarity with the technology and task and were given five days to complete the exercise.
The participants then initiated the exercise, searching for prior patent references. Half of the participants
(randomly chosen again) on each technology received expert advice in the midst of this exercise, in the form
of an email from an experienced USPTO examiner who worked in the art-unit where the patent was
originally examined in 2010. The e-mail included useful tips for conducting prior art search for the specific
patent application, using the specific technology. Thus, the expert advice was both task and technology
specific (see Appendix B Exhibits 6a and 6b), and recommended explicit search strategies on how to
manipulate the user interface (for the machine learning group) and how to compose the search string (for the
Boolean search group). Correctly inputting these strategies resulted in a narrower list of prior art to review;
however, it was not sufficient for task completion. The subject still had to employ judgement in reading and
selecting prior art references relevant to adjudicating the claims. At the end of the exercise, we assessed
performance, measured by both search accuracy (referencing the relevant prior are citation) and speed.
As it relates to the four criteria for an “ideal experiment,” first, participants did not self-select into this
experiment. All participants were students enrolled in a semester-long, second-year elective MBA course, and
they were unaware at time of enrollment of their potential participation in the experiment. While
participation in the experiment was voluntary, two of the three sections of the course had a 100%
17
participation rate for students physically on campus that week, and the third had a participation rate of 81%.8
We ensure robustness of results for potential self-selection into the experiment for the sub-sample of sections
with 100% participation rate. In addition, participants signed consent forms prior to knowing which
technology they were being assigned, alleviating the concern that participation in the experiment might be
endogenous to being assigned to a technology. Second, while it was not possible to randomly assign CS&E
and non-CS&E backgrounds to participants as these related to their own prior experiences before enrolling in
the MBA program, we ensured random assignment conditional on possessing a prior skill to the technology,
ensuring the skill-technology match was randomly determined. Similarly, expert advice was also randomly
assigned among the participants within each technology. Third, to prevent across group contamination,
participants were asked to sign an academic declaration of honesty, stating they would work on the exercise
on their own and would not collaborate with anyone during the exercise. While such contamination would
only attenuate our results, this alleviates some concern that individuals in one technology group accessed the
search tool assigned to the other group. Moreover, access to the electronic activity logs enables us to test how
many of the individuals in the group that did not receive expert advice input the advice as suggested by the e-
mail: we find that this occurred in only 3 cases, and the results are robust to these cases being excluded from
the analysis. Finally, the activity logs for each participant also help shed light on the underlying mechanisms
driving the causal relations of interest, including whether or not participants integrated the expert advice into
their search activities.
Table 1 presents descriptive statistics for the sample. Overall, the average participant spent 22.6
minutes on the tool; the time spent ranged from a minimum of 0 minutes, to a maximum of 93 minutes.
While 23 participants spent no time working on the tool, these seem to be randomly distributed across the
groups. Further, neither technology assignment, nor CS&E background, nor expert advice are predictive of
the time spent on the time tool.
[Insert Table 1 here]
Variable Definitions
8 For these two sections, the participation rate was 97% and 94% respectively, counting all students, including students who were not physically on campus that week. The students who were not on campus were traveling out of town to attend job interviews.
18
Dependent Variables: Success in the task is based on whether the patent application being reviewed was rejected
for all five claims due to participants identifying the single relevant prior art citation. To measure an examiner's
success, we count the number of times they cited the prior art citation in their evaluation of each claim. To
measure an examiner's productivity, we compute the ratio of prior art citations to time spent on the platform.
Independent Variables: We manipulated two variables in this study: First, 𝑀𝐿𝑖 is a dummy variable set to 1 if the
examiner 𝑖 was assigned to use the machine-learning technology, Sigma (and 0 otherwise). Second,
𝐸𝑥𝑝𝑒𝑟𝑡 𝐴𝑑𝑣𝑖𝑐𝑒𝑖 , is a dummy variable set to 1 if the examiner 𝑖 received the expert advice email (and 0
otherwise). The prior knowledge base of examiners is captured by the indicator variable 𝐶𝑆&𝐸𝑖 , set to 1 if
examiner 𝑖 has a degree in computer science and engineering (and 0 otherwise). Finally, we define 𝑆𝑘𝑖𝑙𝑙 −
𝑇𝑒𝑐ℎ𝑛𝑜𝑙𝑜𝑔𝑦 𝑀𝑎𝑡𝑐ℎ𝑖 , as an indicator which takes a value of 1 (and 0 otherwise) if a) an examiner with a
computer science or engineering degree is working with the machine learning technology, or b) an examiner
without a computer science or engineering degree is working with the Boolean search technology.
Controls: Although randomization ensures unbiased estimates, we include several pre-treatment control
variables in our model to reduce residual variation and increase the precision of treatment effect estimates.
Controls include indicators for section, gender, whether the individual has a partner, and is a US citizen.
Balance Tests and Empirical Approach
As noted above, each examiner was randomly assigned to one of four treatment groups (Boolean vs.
Machine Learning; Received Expert Advice or not), stratified by examiner background (whether the examiner
had a CS&E degree) and examiner section. The sampling method comprised three steps. First, we sorted
student IDs by section and CS&E background. Second, we matched subjects in the four treatment groups to
the six strata (3 sections* CS&E background dummy). This ensured each of the six strata have an
approximately equal number of students assigned to each treatment group. In the final step, within each
stratum, we shuffled the treatment assignments using a random number generator. We verified our
randomized treatment assignment generated roughly equal numbers of each treatment within each stratum.
Table 2 presents means and standard errors of various pre-treatment variables by the four treatment groups.
The stratification and matching seems to have worked as designed, because the distribution of CS&E and
19
section are evenly distributed between the four treatment groups. Other pre-treatment variables are also
balanced across the groups.
[Insert Table 2 here]
To examine our hypotheses, we estimate the following set of equations using OLS:9
𝑌𝑖 = 𝛾0 + 𝛾1𝑆𝑘𝑖𝑙𝑙 − 𝑇𝑒𝑐ℎ𝑛𝑜𝑙𝑜𝑔𝑦 𝑀𝑎𝑡𝑐ℎ𝑖 + 𝐵𝑿 + 𝜀𝑖 (1)
𝑌𝑖 = 𝛼0 + 𝛼1𝐸𝑥𝑝𝑒𝑟𝑡 𝐴𝑑𝑣𝑖𝑐𝑒𝑖 + 𝐵𝑿 + 𝜀𝑖 (2)
𝑌𝑖 = 𝛽0 + 𝛽1𝑆𝑇𝑀𝑖 + 𝛽2𝐸𝑥𝑝𝑒𝑟𝑡 𝐴𝑑𝑣𝑖𝑐𝑒𝑖 + 𝛽3𝐸𝑥𝑝𝑒𝑟𝑡 𝐴𝑑𝑣𝑖𝑐𝑒𝑖 ∗ 𝑆𝑇𝑀𝑖 + 𝐵𝑿 + 𝜀𝑖 (3)
where 𝑿 includes the controls noted above and 𝑌𝑖 is either the number of times the relevant prior art citation
is referenced or the examiner’s productivity (relevant prior art citations per minute). Equation (1) estimates
the main effect of skill-technology match, while equation (2) examines the main effect of the provision of
expert advice, and equation (3) examines the interaction between expert advice and skill-technology match. In
subsequent analyses, we break out the Skill-Technology Match variable into its constituent subcomponents.
Table 3 reports the results from our tests of Hypothesis 1. Columns (1) and (2) of Table 4 indicate
that skill-technology match is associated with a 0.2 increase in the number of times the relevant prior art is
cited, a 333% increase relative to the mean of 0.06. Similarly, skill-technology match is associated with a 0.007
increase in productivity, a 200% increase. Both estimates are very precise, with p-values less than 0.01, thus
supporting Hypothesis 1. These average effects are displayed in a bar graph in Figure 2.
[Insert Table 3-5, and Figures 2-3 here]
Table 4 reports the results from the test of Hypothesis 2 regarding the effects of concurrent, expert
advice. In accordance with our expectations, we find the random provision of expert advice is associated with
a 0.2 increase in the number of prior art citations, and an increase of 0.007 prior art citations per minute, with
both estimates being highly statistically significant. Given the mean number of prior art citations cited is 0.06
and that average productivity is 0.0037 prior art citations per minute, the impact of expert advice increases the
number of prior art citations by 333%, while productivity increases by 200%.10
9 We use OLS as opposed to count models such as Poisson because many of the observations are perfectly predicted, as is readily apparent in Figures 1 through 4. We also perform robustness checks using linear probability models and find results that are quantitatively consistent, both in terms of magnitude and statistical significance. 10 Interestingly, these numbers are nearly identical to those in Table 3. This nuance is explained by the fact that the effects of expert advice are driven entirely by those with skill-technology match, as shown in Table 5.
20
Hypothesis 3 is tested in Table 5 using an interaction approach. In particular, we are interested in
whether those with skill-technology match are better able to translate expert advice into productivity.
Surprisingly, the results suggest the provision of expert advice is entirely ineffective in increasing productivity
for those whose skills are not matched with their technology. In addition, those whose skills match their
technology are not more effective than those with a poor skill-technology match as long as both do not receive
expert advice. Indeed, the only instance in which we see productivity increases is when expert advice is
provided to those whose skills are aligned with their technology, as shown in Figure 3. In this case, those who
receive expert advice and have a skill-technology alignment have a 685% increase in the number of relevant
citations (relative to the mean of 0.06) and a productivity increase of 405%.
In Table 6, we disaggregate the skill-match variable into its subcomponents, to examine the
interactions between prior background, technology vintage, and expert advice. Figures 4 and 5 also report
these results for the unconditional models. Both Table 6 (the odd columns) and Figure 4 indicate that the
only combinations of skills and technologies able to identify the relevant prior citation were non-CS&E
examiners working with the Boolean technology and CS&E examiners working with machine learning
technology. Figure 5 and the even columns of Table 6 further highlight relevant prior art citations are derived
from those who received the expert advice and were either non-CS&E working Boolean technology or CS&E
working with machine learning technology. Furthermore, we find that there is no statistically significant
difference, between CS&E working with machine learning and non-CS&E working with Boolean, as is
apparent in Figures 4 and 5.
[Insert Table 6, and Figures 4-5 here]
Mechanisms and Robustness Checks
Our central argument for why we observed these patterns is based on absorptive capacity. One
indicator of this mechanism is an individual’s inability to translate expert advice into productivity. By tracking
individual queries, we are able to assess precisely how well individuals integrated the expert advice into their
queries on the user interface, which would have given them a narrower list of prior art to review. In
particular, we create a dummy variable equal to one if the individual entered the expert advice into the user-
interface correctly, or zero if they did not. Thus, among those who have received such advice, we can assess
21
whether skill-technology matches are better able to integrate the information into the user interface. The
results of these analyses are shown in Table 7 in which we estimate the following equation:
𝑌𝑖 = 𝜑0 + 𝜑1𝑆𝑘𝑖𝑙𝑙 − 𝑇𝑒𝑐ℎ𝑛𝑜𝑙𝑜𝑔𝑦 𝑀𝑎𝑡𝑐ℎ𝑖 + 𝐹𝑿 + 𝜀𝑖 (4)
where 𝑌𝑖 is an indicator equal to one if the individual enters the proper query (as detailed in the expert advice
the individual received) in the user interface detailed.
Column (1) of Table 7 indicates that 27.8% of those who received the e-mail and did not have skill-
technology alignment were able to incorporate the information correctly (the constant term), while those
whose skill and technology matched were 24 percentage points more likely to properly incorporate the
information to the user interface, nearly double the baseline rate. Column (2) confirms these results hold with
controls. Columns (3) and (4) breakout the skill-technology match variable into its subcomponents, and
reveal that non-CS&E working with Machine Learning are roughly 30 percentage points less likely to
incorporate the information properly relative to if they were working on Boolean. CS&E examiners are, in
contrast, relatively better at integrating the information with Machine Learning technology, though the point
estimates are not statistically significantly different from zero.
Given the randomization of technology vintage and expert advice, there are relatively few empirical
concerns. One potential concern is with the count measure as the dependent variable. We re-ran all our
results with simply an indicator for whether an individual found the relevant prior art citation at all, and the
results (available from authors upon request) are qualitatively and quantitatively very similar. Another
potential concern is that voluntary participation in the study created a nonrandom selection into the study,
thus creating generalizability issues due to sample selection. As noted above, two of the three sections had a
100% participation rate, and Table 8 provides a replication of all our main analyses where we included
observations from these sections only. Our results are also robust to only considering the 198 observations
for individuals who had participated in the exercise. The results are quantitatively and qualitatively similar. A
third concern is that potential contamination of treatment and control are attenuating our results. As we
noted above, we found that only 3 individuals in the group that did not receive expert advice input the
relevant information, though these individuals did not go on to cite the relevant prior art. Thus, we conclude
that such contamination is minimal.
22
[Insert Tables 7 and 8 here]
DISCUSSION AND CONCLUSION
Our study is motivated by technological advances in machine learning—the current state of
development in AI—and the quest to examine production complementarities between technology vintage,
prior knowledge domains, and the concurrent provision of expert advice. These issues are theoretically and
practically important: complementarities between existing human capital and the technologies with which
they interface affect not only the pace of technological substitution, but also workplace practices and socio-
economic outcomes for individuals at work. We combined insights from models of technology diffusion and
specialization of knowledge domains to theorize about the importance of absorptive capacity in unlocking
production complementarities: both the match of prior skills and technology, and expert advice with task and
technology specific information should enhance productivity. Our experimental design simulated the context
of patent examiners interfacing with two technology vintages to determine patent claim novelty, based on
searches among the vast prior art.
The results confirm our theoretical predictions, in some ways even more strongly than we
anticipated. Specifically, a lack of technology-skill match resulted in zero success (no productivity), even
though we had expected only lower productivity gains. Moreover, even technology-skill matches resulted in
positive productivity only in the presence of expert advice. Without such technology and task specific
information, the potential productive benefits from prior familiarity with the technology remained unrealized.
The mirroring of these results for both technology vintages within our experimental setting is also illustrative
of the fact that these effects are not chiefly driven by a specific technology itself. Rather our results are more
indicative of the larger issue that specialized knowledge domains and vintage specific knowledge (both task
and technology specific) are critical for productivity differentials. Accordingly, objective superiority of any
technology aside, the realized gains in productivity depend on the absorptive capacity of individuals
interacting with the technology.
Of course, the generalizability of our study is subject to important limitations, some of which require
additional research to address. First, our focus in this study is on the window where workers first interface
with technologies, rather than after a prolonged association. This boundary condition is both a feature and a
23
limitation. Initial conditions are important to study, as they have path dependent outcomes for the future,
both in terms of adoption decisions, and the strategies which will enable better diffusion of adopted
technologies. However, it is a limitation inasmuch as we are unable to examine performance improvements
over longer periods of time. While technology-skill matches and provision of expert advice increase the initial
productivity, what is unclear is whether constant exposure and learning by doing by workers would cause the
relative differences between the groups to grow or shrink over time.
Second, our reliance on an experimental design and choice of MBA students as subjects was
motivated by the need for causal inferences in a sample of highly skilled but heterogeneously specialized labor
force. Further, as noted above, this choice of experimental subjects permitted us to control for prior
familiarity with both task and technology, so we could isolate the effects of prior knowledge domains, and the
provision of expert task and technology specific information to randomly distributed subjects. However, as is
true for all laboratory experiments, generalizability of the results are limited by the abstractions from reality,
and applications to other relevant sub-populations of the labor force. While we expect our sample
population of MBA students at a top tier business school to be similar to the target sector of highly skilled
individuals interested in intellectual property rights, the results of our experiments need to be replicated in
similar contexts for confirmation. Also, we deliberately abstracted away from vertical differentiation (high vs.
low skilled labor), but widespread use of new technologies may well require an inquiry expanding to this
dimension as well.
Finally, our research context and technology vintages are very specific—applicable to USPTO’s
development of the machine learning tool Sigma relative to Boolean search. Machine learning technologies
are themselves heterogeneous, and their development for tasks may render user interfaces more or less
complex, which itself has implications for complementarities with human capital. Accordingly, while our
results should be interpreted with caution, we urge scholars to add to the budding empirical research
examining productivity of all machine learning technologies, and their contingencies.
Limitations notwithstanding, our study contributes to several literature streams. Recent theoretical
models and empirical findings related to artificial intelligence and machine learning have stressed substitution
effects of technologies on human capital, particularly in the domain of prediction (Agrawal et al. 2016; Autor
24
2015; Benzell et al. 2015; Jones et al., 2017). Human capital in this literature stream is largely incorporated
either in the form of vertical differentiation (high vs. low skilled workers), or because of comparative
advantage in complementary tasks related to judgment, new discoveries, or tacit knowledge (Aghion et al.,
2017; Agrawal et al., 2017 Autor, 2015). We extend this literature in two ways—first, we integrate insights
from the related literatures on technology diffusion (Chari and Hopenhayn, 1991; Jovanovic, 2009) and pace
of technology substitution (Adner and Kapoor, 2016; Adner and Snow, 2010; Christensen, 1997) to explicitly
compare productivity of machine learning technologies with earlier vintages. Second, and contributing to all
three literature streams above, we highlight the effects on productivity of technologies of horizontal
differentiation caused by prior specialized knowledge domains, and the provision of expert advice containing
technology and task specific knowledge.
The strategic management literature on strategic renewal within the context of disruptive
technologies acknowledges both imperatives and challenges of change (Agarwal and Helfat, 2009;
Christensen, 1997; Tushman and Anderson, 1990). Moreover, scholars have noted the rate at which new
technologies substitute old technologies relate to complementarities with existing resources and capabilities,
in addition to endogenous strategies for extending the relevance of older vintages (Adner and Snow, 2010;
Agarwal and Prasad, 1999; Bloom et al., 2012; Bapna et al., 2013; Hatch and Dyer 2004; Henderson 2006).
Inasmuch as specialized knowledge domains embodied in heterogeneous human capital will continue to
retain value for judgment, and given that humans will be relevant at work in the foreseeable future, our study
highlights the importance of incorporating these contingencies in theories and empirical work examining
productivity of new (machine learning) technologies, rather than conceptualizing main effects alone. Doing so
enables the identification of real productivity costs incurred because of within-vintage differences in human
capital caused by differential rates of absorptive capacity (Cohen and Levinthal, 1990).
Finally, our study also has implications for related literature streams on human capital and career
specialization (Becker and Murphy, 1992; Lazear, 2004; Rosen, 1983; Teodoridis, 2017; Zuckerman et al.,
2003). While this literature has focused on life cycle effects of human capital investments, and generally found
beneficial effects of developing specialized knowledge (Becker and Murphy, 1992; Castanias and Helfat, 1991;
Greenwood et al., 2017), but also that benefits of specialization are contingent on market or institutional
25
factors (Castanias and Helfat, 1991; Merluzzi and Phillips, 2016). We extend this literature by highlighting the
differential effects of specialized domains for absorptive capacity to interface with entirely new technologies.
New technological advances such as AI exhibit skill-bias not only in the vertical realm (i.e. high vs. low skill),
but also to the extent they privilege computer science and engineering knowledge relative to other specialized
domains. Accordingly, benefits and costs of career specialization must also be evaluated in the context of
what skills and knowledge are considered foundational for complementarities with the latest technology
vintages.
The study has several implications for practice. First, and immediately applicable to USPTO, our
study unearths technology-skill matches as critical to productivity of alternative technologies. Accordingly,
one immediate insight may be a sequential rather than simultaneous roll-out of machine learning technology.
While GAO (2016) has highlighted the hurdles posed by existing search technologies for patent examiners’
productivity—given significant public concern about backlogs in the patent approval process—our study
shows that these may nonetheless retain value for examiners who have non-CS&E knowledge bases. Overall
productivity, both in the short and long run, may be enhanced through CS&E examiners’ immediate use of
machine learning, and development of vintage specific knowledge by these experts. Such learning-by-doing
insights can then be transformed into a codified library of task and technology specific knowledge for a larger
scale diffusion among examiners and stakeholders for intellectual property rights in the general population.
Using these tools, potential innovators may be better able to assess the likely viability of a patent application,
which is an important element in the choice to start a new firm (Gambardella, Ganco and Honore, 2015).11
Second, and at a broader level, our study highlights the need for technology-skill match specific training,
rather than technology-specific alone. Given that new technologies require re-composition of tasks relative
to old technologies, training programs often focus on the provision of technology specific skills. Our study
calls for a more nuanced approach—one where firms need to account for prior knowledge base alignment.
Third, the implications of our study extend beyond training of the existing workforce to other human
11 Our field interviews indicated that the USPTO has plans to release search tools based on machine learning algorithms, for use by the general population. An example of a related prior initiative is the “peer-to-patent” initiative whereby the USPTO, New York Law School, and IBM are collaborating to provide patent examiners structured information on prior art, based on third-party submission of prior art.
26
resource management practices, including recruitment, retention, and innovation management, especially for
firms that operate in dynamic industries. For example, the literature on recruitment outlines conditions for
when firms should hire externally vs. internally, including “acqui-hiring” (Chatterji and Patro, 2014) to gain
simultaneous access to technologies and complementary employees, or for positions with strong firm-specific
or industry-specific components (Bidwell and Keller, 2014; Bidwell and Mollick, 2015; Williams, Chen and
Agarwal, 2017). Our results indicate firms investing in new technologies need to incorporate skill-technology
matches for the specific knowledge domains as well, when making decisions on whether to hire externally, or
promote internal talent. Similarly, in the context of employee retention, in addition to factors such as
organizational culture, employee voice and work practices (Sheridan 1992; Spencer 1986; Guthrie 2001), our
study highlights job satisfaction and incentive alignment will also depend on the attention firms implementing
new technologies pay to whether employees have greater or lower skill-technology matches. Perceptions of
fairness and inequality may be exacerbated by underlying productivity differentials from working with
different technologies, so firms need to think through compensation mechanisms carefully.
Finally, and at the larger levels of innovation and education policy, our study highlights the
importance of computer programming training to increase familiarity and develop these as generalized skills
for the knowledge economy, even as subsequent human capital investments continue to focus on specialized
knowledge domains. Recently, Apple CEO, Tim Cook remarked to primary school students at a French
school that learning to code may be more important than learning English as a second language (Clifford,
2017). Elaborating on the comment further, Cook discussed the benefits of computer coding not just in
terms of higher salaries, but as a critical “backseat” complement to creativity in the “front seat,” because “the
blend with both of these, you can do such powerful things now.”
27
References "Artificial Intelligence." Merriam-Webster.com. Accessed January 3, 2018. https://www.merriam-
webster.com/dictionary/artificial intelligence. Acemoglu, Daron, and Pascual Restrepo. "Robots and Jobs: Evidence from US labor markets." (2017). Adner, Ron, and Daniel Snow. "Old technology responses to new technology threats: demand heterogeneity
and technology retreats." Industrial and Corporate Change 19, no. 5 (2010): 1655-1675.
Adner, Ron, and Rahul Kapoor. "Innovation ecosystems and the pace of substitution: Re‐examining
technology S‐curves." Strategic Management Journal 37, no. 4 (2016): 625-648. Agarwal, R., & Prasad, J. (1997). The role of innovation characteristics and perceived voluntariness in the
acceptance of information technologies. Decision Sciences, 28(3), 557-582 Agarwal, Rajshree, and Constance E. Helfat. "Strategic renewal of organizations." Organization Science 20, no. 2
(2009): 281-293. Agarwal, Ritu, and Jayesh Prasad. "Are individual differences germane to the acceptance of new information
technologies?." Decision Sciences 30, no. 2 (1999): 361-391. Aghion, Philippe, Antonin Bergeaud, Richard Blundell, and Rachel Griffith. "Innovation, Firms and Wage
Inequality." (2017). Aghion, Philippe, Benjamin F. Jones, and Charles I. Jones. Artificial Intelligence and Economic Growth. No.
w23928. National Bureau of Economic Research, 2017. Agrawal, A., J. Gans, and A. Goldfarb. "How AI will change the way we make decisions." Harvard Business
Review (2017). Alcacer, Juan, and Michelle Gittelman. "Patent citations as a measure of knowledge flows: The influence of
examiner citations." The Review of Economics and Statistics 88, no. 4 (2006): 774-779. Allison, John R., and Mark A. Lemley. "Who's Patenting What-An Empirical Exploration of Patent
Prosecution." Vand. L. Rev.53 (2000): 2099. Anderson, Philip, and Michael L. Tushman. "Technological discontinuities and dominant designs: A cyclical
model of technological change." Administrative Science Quarterly (1990): 604-633. Argote, Linda, and Dennis N. Epple. Learning curves in manufacturing. Carnegie Mellon University,
Graduate School of Industrial Administration, 1990. Autor, David H. “Why Are There Still So Many Jobs? The History and Future of Workplace
Automation." Journal of Economic Perspectives, 29(3): 3-30. (2015) Autor, David H., Frank Levy, and Richard J. Murnane. "The skill content of recent technological change: An
empirical exploration." The Quarterly Journal of Economics 118, no. 4 (2003): 1279-1333. Autor, David. Polanyi's paradox and the shape of employment growth. Vol. 20485. Cambridge, MA: National
Bureau of Economic Research, 2014. Bapna, Ravi, Nishtha Langer, Amit Mehra, Ram Gopal, and Alok Gupta. "Human capital investments and
employee performance: an analysis of IT services industry." Management Science 59, no. 3 (2013): 641-658. Bartel, A. P. (1994). Productivity gains from the implementation of employee training programs. Industrial
relations: a journal of economy and society, 33(4), 411-425. Becker, G. S., & Murphy, K. M. (1992). The division of labor, coordination costs, and knowledge. The
Quarterly Journal of Economics, 107(4), 1137-1160. Bekman, Eli, John Bound, and Stephen Machin. "Implications of skill-biased technological change:
international evidence." The Quarterly Journal of Economics 113, no. 4 (1998): 1245-1279. Benzell, Seth G., Laurence J. Kotlikoff, Guillermo LaGarda, and Jeffrey D. Sachs. Robots are us: Some
economics of human replacement. No. w20941. National Bureau of Economic Research, 2015. Bessen, James E. "How computer automation affects occupations: Technology, jobs, and skills." (2016). Bidwell, Matthew, and Ethan Mollick. "Shifts and ladders: Comparing the role of internal and external
mobility in managerial careers." Organization Science 26.6 (2015): 1629-1645. Bidwell, Matthew, and J. R. Keller. "Within or without? How firms combine internal and external labor
markets to fill jobs." Academy of Management Journal 57.4 (2014): 1035-1055. Black, S. E., & Lynch, L. M. (2001). How to compete: the impact of workplace practices and information
technology on productivity. The Review of Economics and Statistics, 83(3), 434-445.
28
Bloom, N., Sadun, R., & Van Reenen, J. (2012). Americans do IT better: US multinationals and the productivity miracle. The American Economic Review, 102(1), 167-201.
Brinckerhoff, Courtenay C. 2014. How Long Is Too Long (Or Too Short) For Patent Application Pendency? July 18. https://www.pharmapatentsblog.com/2014/07/18/how-long-is-too-long-or-too-short-for-patent-application-pendency/. Accessed July 2017.
Brynjolfsson, Erik, and Andrew McAfee. The second machine age: Work, progress, and prosperity in a time of brilliant technologies. WW Norton & Company, 2014.
Bughin, J., E. Hazan, S. Ramaswamy, M. Chui, T. Allas, P. Dahlström, N. Henke, and M. Trench. "Artificial intelligence–the next digital frontier." McKinsey Glob Institute URL https://www. mckinsey. de/files/170620_studie_ai. pdf (2017).
Castanias, Richard P., and Constance E. Helfat. "Managerial resources and rents." Journal of Management 17, no. 1 (1991): 155-171.
Chari, Varadarajan V., and Hugo Hopenhayn. "Vintage human capital, growth, and the diffusion of new technology." Journal of Political Economy 99, no. 6 (1991): 1142-1165.
Chase, W.G., Simon, H.A. 1973. Perception in Chess. Cognitive Psychology. 4(1) 55-81. Chase, William G., and Herbert A. Simon. "The mind's eye in chess." (1973). Chatterji, Aaron, and Arun Patro. "Dynamic capabilities and managing human capital." The Academy of
Management Perspectives 28, no. 4 (2014): 395-408. Choudhury, P., Khanna, T., and Mehta, S. “Future of Patent Examination at the USPTO”, Harvard Business
School Case Study. 2017. Source: https://hbr.org/product/The-Future-of-Patent-Exam/an/617027-PDF-ENG
Christensen, Clayton. "Patterns in the evolution of product competition." European Management Journal 15, no. 2 (1997): 117-127.
Clifford, Catherine. 2017. Source: https://www.cnbc.com/2017/10/12/apple-ceo-tim-cook-learning-to-code-is-so-important.html. Accessed December 2017.
Cohen, Wesley M., and Daniel A. Levinthal. "Absorptive capacity: A new perspective on learning and innovation." Administrative Science Quarterly (1990): 128-152.
Coleman, J., Katz, E., Menzel, H. 1957. The Diffusion of an Innovation among Physicians. Sociometry. 20(4) 253-270.
Cowgill, Bo. “The Labor Market Effects of Hiring through Machine Learning”, Working Paper. 2017 Crouch, Dennis. 2014. USPTO's Swelling Examiner Rolls. November 30.
https://patentlyo.com/patent/2014/11/usptos-swelling-examiner.html. Accessed July 2017. Edmondson, A. C., Winslow, A. B., Bohmer, R. M., & Pisano, G. P. (2003). Learning how and learning what:
Effects of tacit and codified knowledge on performance improvement following technology adoption. Decision Sciences, 34(2), 197-224
Faggella, Daniel. “What is Machine Learning”. September 2, 2017. Source: https://www.techemergence.com/what-is-machine-learning/. Accessed November 2017.
Fernandez, R. M. (2001). Skill-biased technological change and wage inequality: Evidence from a plant retooling. American Journal of Sociology, 107(2), 273-320.
Gambardella, A. & Ganco, M. & Honore, F. (2015). Using What You Know: Patented Knowledge in Incumbent Firms and Employee Entrepreneurship. Organization Science (26), 456-474.
Government Accountability Office (2016). Intellectual Property: Patent Office Should Strengthen Search Capabilities and Better Monitor Examiner’s Work. Report to the Chairman, Committee on the Judiciary, House of Representatives.
Gibbons, R., & Waldman, M. (2004). Task-specific human capital. The American Economic Review, 94(2), 203-207.
Goos, Maarten, and Alan Manning. "Lousy and lovely jobs: The rising polarization of work in Britain." The Review of Economics and Statistics 89, no. 1 (2007): 118-133
Graetz, Georg, and Guy Michaels. "Robots at work." (2015). Greenwood, B., Agarwal R., Agarwal R., and Gopal A. 2017 The Role of Individual and Organizational
Expertise in the Adoption of New Practices
29
Guthrie, James P. "High-involvement work practices, turnover, and productivity: Evidence from New Zealand." Academy of Management Journal 44.1 (2001): 180-190.
Hatch, Nile W., and Jeffrey H. Dyer. "Human capital and learning as a source of sustainable competitive advantage." Strategic Management Journal 25, no. 12 (2004): 1155-1178.
Henderson, Rebecca. "The innovator's dilemma as a problem of organizational competence." Journal of Product Innovation Management 23, no. 1 (2006): 5-11.
Hoffman, Mitchell, Lisa B. Kahn, and Danielle Li. Discretion in hiring. No. w21709. National Bureau of Economic Research, 2015.
Hounshell, David. From the American system to mass production, 1800-1932: The development of manufacturing technology in the United States. No. 4. JHU Press, 1985.
Jovanovic, Boyan. "The technology cycle and inequality." The Review of Economic Studies 76, no. 2 (2009): 707-729.
Krishna, Arthi M., Brian Feldman, Joseph Wolf, Greg Gabel, Scott Beliveau, and Thomas Beach. "User Interface for Customizing Patents Search: An Exploratory Study." In International Conference on Human-Computer Interaction, pp. 264-269. Springer International Publishing, 2016.
Lazear, Edward P. "Balanced skills and entrepreneurship." The American Economic Review 94, no. 2 (2004): 208-211.
Lemley, Mark A., and Bhaven Sampat. "Examiner characteristics and patent office outcomes." Review of Economics and Statistics 94.3 (2012): 817-827.
Mansfield, E. (1968). Industrial research and technological innovation; an econometric analysis. London: Longmans.
Merluzzi, Jennifer, and Damon J. Phillips. "The specialist discount: Negative returns for MBAs with focused profiles in investment banking." Administrative Science Quarterly 61, no. 1 (2016): 87-124.
Mokyr, Joel. Twenty five centuries of technological change: an historical survey. Vol. 35. Taylor & Francis, 1990.
Partha, Dasgupta, and Paul A. David. "Toward a new economics of science." Research Policy 23, no. 5 (1994): 487-521.
Rosen S. 1983. Specialization and human capital. Journal of Labor Economics 1(1): 43–49. Schultz, T.W. 1961. Investment in Human Capital. The American Economic Review 1-17 Sheridan, John E. "Organizational culture and employee retention." Academy of Management Journal 35.5 (1992):
1036-1056. Spencer, Daniel G. "Employee voice and employee retention." Academy of Management Journal 29.3 (1986): 488-
502. Teodoridis, Florenta. "Understanding Team Knowledge Production: The Interrelated Roles of Technology
and Expertise." Management Science (2017). Thornton, R. A., & Thompson, P. (2001). Learning from experience and learning from others: An
exploration of learning and spillovers in wartime shipbuilding. American Economic Review, 1350-1368. Williams, C., Chen, P. L., & Agarwal, R. (2017). Rookies and seasoned recruits: How experience in different
levels, firms, and industries shapes strategic renewal in top management. Strategic Management Journal, 38(7), 1391-1415.
Zeira, Joseph. "Workers, machines, and economic growth." The Quarterly Journal of Economics 113, no. 4 (1998): 1091-1117.
Zuckerman EW, Kim T, Ukanwa K, von Rittmann J. 2003. Robust identities or nonentities? Typecasting in
the feature‐film labor market. American Journal of Sociology 108(5): 1018–1074.
30
Figures
Figure 1. Skill-Technology Match
Figure 2. Prior Art Citations by Skill-Technology Match
31
Figure 3. Prior Art Citations by Skill-Technology Match and Expert Advice
Figure 4. Prior Art Citations by Technology and Prior Knowledge Domain
32
Figure 5. Prior Art Citations by the Receipt of Expert Advice
33
Tables
34
35
36
37
38
39
40
APPENDIX
Appendix A. Experiment Details
Research Context
The sample population for this study are second-year MBA students enrolled in a top tier
business school in the US, across three sections of the same course with approximately 70 students
each. The course explored strategies for maintaining and enhancing competitive advantage over time
During the final class of the spring semester, instructors taught “The Future of Patent
Examination at the U.S. Patent and Trademark Office (USPTO)” (HBS case no. 617-027). This case
introduces students to the patent examination process and describes the machine learning strategies
the USPTO is currently piloting to expedite patent examination and improve patent quality. To
enhance students’ learning experience, one of the researchers—working closely with the business
school information technology (IT) department and key USPTO stakeholders—created an online
exercise that allowed students to experiment with the USPTO’s machine learning tools (See Exhibit
7 in Appendix B for a complete tool development timeline.). The university IT department
partnered with key stakeholders at the USPTO to create an online tool—the Patent Examiner
Review Tool (PERT)—to simulate the process of examining a patent application. See Exhibit 8 in
Appendix B for a diagram of the PERT exercise workflow.
Designing the PERT Interface
The IT department decided to use a “reverse proxy” approach to build the PERT tool. In
essence, the IT department embedded the USPTO sites (either PatFT or Sigma) into the university
system and then built a “shell” site around them (see Exhibits 9 and 10 in Appendix B). This shell
site (i.e., the PERT tool) hid certain buttons from view to prevent students from navigating away
from the PatFT or Sigma sites. For instance, the PatFT site has an “advanced search” button (refer
41
to Exhibit 2 in Appendix B) that directs workers to a page with several filters and settings. To avoid
confusion, the PERT tool hid this option from students’ view.
PERT also featured a side panel with information about each of the five patent claims under
review. Students could click to review Claim 1, for example, and then collapse the side panel to
search either the embedded PatFT or Sigma sites (depending on their assigned group) for prior art
relevant to that particular claim. If they located a similar existing patent, they could save it to the
reference panel within PERT (see Exhibit 11 in Appendix B). Because the patent under review was
submitted in January 2010, students could only reference prior art that predated this point.
Once students believed that they had exhausted the prior art search for Claim 1, they then
indicated their decision to approve or reject the claim within the PERT tool (see Exhibit 12 in
Appendix B). If students decided to disallow the claim, they were required to cite at least one prior
patent reference to justify this decision. Students followed the same process to adjudicate the
remaining four patent claims. The university server tracked how long it took students to answer each
question.12
Challenges in Developing the PERT Tool
Several challenges had to be successfully overcome to develop the PERT tool under time
constraints. First, Sigma was a proprietary USPTO tool and it took two months for the University to
receive clearance to use Sigma. Second, the PERT exercise was the first time that the IT department
had worked with an external organization to create a simulation exercise. For all other projects up to
that point, the IT department had owned the tools and databases used during these exercises, which
gave it complete control over the students’ experience. The PERT simulation, by contrast, required
students to use tools and databases that were owned and managed by the USPTO.
12 If a student was inactive for longer than five minutes, a pop-up box appeared and asked whether the student was still working. If the student did not reply, the server stopped tracking time.
42
Reliance on these USPTO sites presented a significant amount of project risk and
uncertainty for the IT department. For instance, if students could not access the PatFT site during
the exercise or if existing database glitches suddenly emerged, our administrator would have to
contact the USPTO to correct the issue. Because PERT would be open to the students over the
course of a weekend, our administrator doubted the availability of a USPTO representative to
resolve problems in real-time. Moreover, because the PatFT database was a public site, a number of
external databases, such as Lexus Nexis, routinely trawled it for data. When site traffic was heavy,
load times could be exceedingly slow (e.g., it could take as long as five minutes for search results to
load). Delays of this magnitude during the student exercise would be untenable.
To mitigate risks, the IT department held a series of routine meetings with the USPTO team
in March and April to identify and resolve “bugs” in the system. As an example, in the days leading
up to the PERT exercise, the administrator realized that PatFT did not recognize curly quotation
marks (“…”) in a search string; the tool only worked with straight quotation marks ("..."). The IT
department contacted the USPTO team and resolved the problem.
To reduce the likelihood of system issues during the student exercise, the IT department
requested that the Patent Office avoid installing system updates or interfering with the PatFT and
Sigma interfaces in the days leading up to the exercise. The USPTO also agreed to distribute website
traffic across additional servers while the PERT exercise was open, thus improving load times.
Experimental Design
The students were randomly assigned to four groups, with each group receiving a slightly
different set of information and tools. Two groups were assigned Sigma while the other two were
assigned the Boolean technology. Each group was then asked to review a patent application that had
been submitted to the USPTO in 2010; students were asked to work alone to complete the exercise.
The application in question requested a patent for a specific method of delivering digital
43
promotional content (see Exhibit 4 in Appendix B). As part of this exercise, five of the 39 claims
included in this application were extracted and students were asked to use the PERT tool to search
the prior art, decide whether each claim met the criteria for patentability, and justify their rationale
for allowing or rejecting the claim. Unbeknownst to students at the time of the exercise, the patent
examiner who adjudicated this particular application in 2010 rejected all five claims because they
were not novel. The patent examiner had used a single prior patent (the “prior art citation”) to reject
all of the claims.
To acclimate students to the PERT tool, the IT department developed a PERT user guide,
created a training video, and scheduled three in-person training sessions for interested students (on
April 12, 13, and 19) prior to the exercise. Approximately 10 students attended each of the in-person
trainings. However, the training video and the user guide was posted online and could be accessed
by all students at all times. The team posted separate technology specific videos and technology
specific online guides for the treatment group and the control group.
Students accessed the PERT tool by logging into the online course platform, at any point
between 3 p.m. EST on Wednesday, April 19 and 11:59 p.m. EST on Sunday, April 23. They were
instructed to use the Google Chrome browser since PERT is incompatible with Internet Explorer.
Participation in the exercise was voluntary. When students opened the PERT tool, a release form
appeared, which explained that the researchers intended to use the results of the PERT simulation
for research purposes. Thus, if students opted out of the exercise or stopped at any point, they
would not be penalized. A total of 198 students participated in the exercise.
Once logged into PERT, students were shown different dashboards, depending on their
group assignment. Two student groups could use the PatFT database only, while the other two only
had access to the Sigma tool. Students using the PatFT database searched the prior art by compiling
44
lists of synonyms to describe each patent claim and conducting keyword searches, while those with
access to Sigma searched by weighting components of the application and creating word clouds.
To narrow the search results, the tool provided all students with the appropriate
“classification code” (i.e., the starting point for a search string) for this particular patent application.
The research team recognized that students would likely feel overwhelmed and unmotivated if their
search resulted in hundreds of thousands of hits. For instance, searching “digital communications”
within PatFT generates more than 18,000 results. Starting the search with the provided classification
code reduced students’ search hits down to the hundreds.
On the last day of the exercise, half of the students in each technology group were sent an e-
mail, depending on their randomly assigned technology, with expert advice from a USPTO
examiner. These messages are displayed in Appendix B Exhibit 6a and 6b. The expert advice in the
emails constituted both technology and task specific information. The e-mail sent to individuals
using the Boolean search tool provided helpful hints on keywords to use for the Boolean search
(advice related to task) and advice on using the filter option in the user interface to filter out certain
references (advice related to technology). For the email sent to individuals using the machine
learning search tool, the expert advice comprised keywords to consider in the search (advice related
to task), as well as advice on editing both the word cloud and the sliders that gave weights to the
patent abstract, claims, title and description during the search (advice related to technology). The
expert advice by itself was not sufficient for task completion. Individual students had to not only
integrate the expert advice into the search tool, but also subsequently use judgment in reading and
selecting prior art references. In other words, the task of the user was twofold: First, the task
required integrating the expert advice correctly on the user interface. Second, it required reading,
interpreting and selecting relevant prior art from the narrower set of prior art references that were
generated. Once the exercise was over and the results of the randomized trial were discussed in
45
class, all participants were granted access to Sigma so that interested students assigned to the PatFT
group could experiment with the Sigma tool.
46
Appendix B. Exhibits Related to the Experiment
Exhibit 1 Patent Examination Process
Source: PowerPoint Slides, PERT_Training
47
Exhibit 2 Interface of USPTO Database, PatFT
Source: U.S. Patent and Trademark Office, “USPTO Patent Full-Text and Image Database,”
http://patft.uspto.gov/netahtml/PTO/search-bool.html, accessed July 2017.
Exhibit 3 Sigma Tool
Source: Arthi M. Krishna, Brian Feldman, Joseph Wolf, Greg Gabel, Scott Beliveau, and Thomas
Beach, “User Interface for Customizing Patents Search: An Exploratory Study, United States Patent
and Trademark Office, 2011.
Note: Workers enter important search terms in the word cloud and manipulate the sliders on the
left-hand side to assign weight to certain components of the patent filing.
48
Exhibit 4 Patent Application (Front Page) Included in Classroom Simulation and Five Claims
49
Patent Claims: 1) A method for delivering promotional content for presentation in a first interactive media guidance
application on a first user equipment device, comprising: receiving data corresponding to a plurality of user actions in a second interactive media guidance application; associating at least one of a plurality of weights with each of the plurality of user actions; based upon the plurality of user actions and the at least one of the plurality of weights associated with each user action, determining the likelihood of a user watching at least one of a certain program, a certain type of program, and a certain channel; selecting promotional content for presentation in the first interactive media guidance application based on the determination; and transmitting the promotional content selected to the first user equipment device with program content.
2) The method of claim 1, wherein the data specifies at least one of user actions taken, the time at which user actions are taken, and programs associated with the user actions taken.
3) The method of claim 1, wherein the plurality of user actions include at least one of selecting a program, watching a program, configuring to record a program, and configuring a reminder for a program.
4) The method of claim 1, wherein the plurality of user actions include at least one of selecting a channel, watching a channel, configuring to record a channel, and configuring a reminder for a channel.
5) The method of claim 1, wherein the second interactive media guidance application is presented as a Web page.
Source: Class Exercise Materials.
50
Exhibit 5 The Relevant “Prior Art” Patent Reference
Source: Class Exercise Materials.
51
Exhibit 6a Search Hints, Email Sent to the Group of Students with Access to the PatFT Database Dear all, I received some useful tips for solving the patent examination simulation from a senior patent examiner at the USPTO. I have copied the email I received below. I HAVE CHOSEN TO SHARE THESE HINTS WITH ONLY A RANDOMLY CHOSEN SELECTION OF THE CLASS, SO PLEASE DO NOT SHARE THEM WITH ANYONE ELSE. IF YOU HAVE ALREADY COMPLETED THE EXERCISE, YOU MIGHT WANT TO USE THE HINTS TO TAKE ANOTHER LOOK AT EXAMINING THE CLAIMS, IT MIGHT IMPROVE THE ACCURACY OF YOUR EXAMINATION. Have a great rest of the day Best, Professor XX I am a patent examiner for the USPTO, and I was informed that you were interested in some input on developing a search strategy for patent application 12/177,788 (PG Pub: US20100017814A1). My experience with these types of cases has taught me to use US Patent Classification code 725/46 when performing the search. Additionally, I suggest using the Issue Date code to filter out any references published after the US20100017814A1 pub date. After reading this application, my recommendation is to use the following search string in your query: CCL/725/46 AND (weight$ OR ranking) AND (tuning OR recording OR watching OR reminding) and ("web page" OR internet) AND (profile OR advert$) AND ISD/1/1/1990->1/20/2010 This should yield around 53 hits. Review these results carefully. Hope this helps. Good luck! Cheers, Mackinzie Simmons United States Patent and Trademark Office - an agency of the Department of Commerce Source: Class Exercise Materials.
52
Exhibit 6b Search Hints, Email Sent to the Group of Students with Access to Sigma Dear student, I received some useful tips for solving the patent examination simulation from a senior patent examiner at the USPTO. I have copied the email I received below. I HAVE CHOSEN TO SHARE THESE HINTS WITH ONLY A RANDOMLY CHOSEN SELECTION OF THE CLASS, SO PLEASE DO NOT SHARE THEM WITH ANYONE ELSE. IF YOU HAVE ALREADY COMPLETED THE EXERCISE, YOU MIGHT WANT TO USE THE HINTS TO TAKE ANOTHER LOOK AT EXAMINING THE CLAIMS. IT MIGHT IMPROVE THE ACCURACY OF YOUR EXAMINATION. Have a great rest of the day Best, Professor XX I am a patent examiner for the USPTO, and I was informed that you were interested in some input on developing a search strategy for patent application 12/177,788 (PG Pub: US20100017814A1). My experience with these types of cases has taught me to leverage the weight boosters first and then edit the word cloud. After reading this application, here is my recommendation for how you can improve your search: Make sure PG Pub US20100017814A1 is loaded. If not, start a new search by entering “US20100017814A1” in the Search field at the top and click Search. Uncheck the “All Text” box and check the “Claims” box. Set the Claims weight to 10 and hit APPLY. Edit the word cloud by replacing the existing list with the following string of words (based on my experience, these words have yielded the best results in terms of this specific application):
advertis epg monitor schedul tune area grid pip screen tv button guid press select viewer can highlight profil televis watch channel inform program theme window data interact record tile displai mean region time
Close the Edit List window and click on the SEARCH button only. Review this page of these results carefully. Make sure any reference you cite is published before Jan 21, 2010 (the US20100017814A1 pub date). Hope this helps. Good luck! Cheers, Mackinzie Simmons United States Patent and Trademark Office - an agency of the Department of Commerce Source: Class Exercise Materials.
53
Exhibit 7 Timeline of Preparatory Activities for the PERT Simulation
Date Action
11/03/2016 Professors submitted project concept statement
11/10/2016 Initial faculty consultation with IT
11/21/2016 First meeting with USPTO Lead
12/05/2016 First meeting with USPTO Sigma team: establish technical requirements
12/06/2016 Gather classroom experience requirements from Professors
12/15/2016 Project proposal submitted to Division of Research and Faculty Development
01/04/2017 PERT is approved for funding
01/05/2017 USPTO approves Sigma access for university use
02/07/2017 First draft of PERT interface completed
02/08/2017 Reviewed PERT interface draft
02/09/2017 Team meeting (IT – USPTO)
02/10/2017 Team meeting (IT – USPTO)
02/17/2017 Team meeting (IT – USPTO); USPTO team demonstrated Sigma tool to professors. Sigma behavior necessitated a change to the PERT interface
03/03/2017 Team meeting (IT – USPTO): IT demonstrated PERT interface prototype
03/10/2017 Team meeting (IT – USPTO)
03/17/2017 IT showed the PERT admin interface to Professors to get feedback
03/22/2017 PERT group test (IT only)
03/22/2017 USPTO delivers the patent and claims to be used for PERT exercise
03/23/2017 PERT Dry Run (IT – USPTO); USPTO helped to address the slow loading times on the PatFT site
03/29/2017 Team meeting (IT – USPTO)
04/04/2017 PERT group test (IT only)
04/06/2017 PERT group test (IT only) at 2pm PERT Final Dry Run (IT – USPTO) at 5pm
04/11/2017 Meet with Professors: PERT training prep
04/12/2017 PERT Student Training at 5pm
04/13/2017 PERT Student Training at 5pm
04/14/2017 Professors looped in Research Computing Services
04/17/2017 Meet with professors and Research Computing Services; PERT User Guide finalized; PERT Cohort assignment finalized
04/18/2017 IT created “How to Use PERT” video
04/19/2017 PERT exercise opened at 3pm for students; PERT assignment posted on Canvas (includes User Guide, How-to video, Training Slide Deck) PERT Student Training at 5pm
04/20/2017 Prepare ‘Search Hints’ email for PERT SIGMA and PERT PatFT cohorts
04/23/2017 Search Hints email sent at 11.29am; PERT exercised closed at 11:59pm
04/24/2017 Classroom debrief of USPTO case and PERT results; PERT reopened at 3pm (all SIGMA access only)
04/26/2017 Survey distributed on PERT exercise PERT closed permanently at 6pm
Source: Class Exercise Materials.
54
Exhibit 8 Overall Student Workflow for PERT Exercise
Source: Researchers.
Exhibit 9 PERT Site with the US PatFT Database Embedded
Source: Class Exercise Materials.
55
Exhibit 10 Pert Site with the Sigma Tool Embedded
Source: Class Exercise Materials.
56
Exhibit 11 Saving Relevant Patent References within the PERT Tool
Source: Class Exercise Materials.
Exhibit 12 PERT Tool with Patent Claim 1 Expanded
Source: Class Exercise Materials.
57
Appendix C
Figure C1. Patent Application Process (GAO 2016)