journal of neurophysiology

Journal of NeurophysiologyCopy of e-mail Notification z9k8250

Proof of your article (# J00760-6 ) from "Journal of Neurophysiology" is available for download=====Dear Sir or Madam:

Please refer to this URL addresshttp://rapidproof.cadmus.com/RapidProof/retrieval/index.jsp Login: your e-mail addressPassword: ----

The file at the above URL address contains the following:• Proofreading marks• Reprint Order form• Copyedited page proof of your article

The site contains 1 file. You will need Adobe Acrobat® Reader to read this file. Adobe Acrobat® Reader is a free software, available for user downloading at http://www.adobe.com/products/acrobat/readstep.html.

After you print the PDF file of your paper, please read the page proof carefully and1) clearly indicate all changes or corrections on the margin;2) answer all queries (footnotes 1, 2, 3, etc.) listed on the last page of the PDF proof;3) carefully proofread all/any tables and equations;4) make sure that any special characters, such as Greek letters, especially µ (mu), have translated correctly;5) If you have questions about figure quality, note your concerns on the margin of the relevant page. Please keep in mind that the final printed version will be of higher quality than the PDF proof and that the online version of the published article will appear identical.

IMPORTANT: Do NOT make online alterations to the PDF itself. Corrections within the PDF, made with the Adobe Acrobat-provided tools will remain unseen by the Journal staff. This will result in the final article NOT reflecting your intended changes. ONLY the hand-marked PDF printout guarantees that the changes will be incorporated into the final text of the article.

We encourage you to retain a copy of the proof with your corrections, should further changes and/or clarifications be required.

IMPORTANT NOTES1. To guarantee the placement of your article in the next available issue of the "Journal of Neurophysiology", please return the corrected set of PDF page proof via an overnight courier service to this address WITHIN 48 HOURS:

Barbara A. Meckley DPS-Ephrata

Cadmus Professional Communications 300 West Chestnut Street

Ephrata, PA 17522 USA 717 445-8430

2. If you require a hard copy of your color image(s), you may request it by replying to this message. The color proof will then be provided to you at an additional fee of $75.00 per color figure, which will be added to the publication fees for your article. You must provide the appropriate figure number(s) and your mailing address in your reply if you wish to receive the hard copy of the color proof(s).

Journal of NeurophysiologyCopy of e-mail Notification z9k8250

3. The filled out Reprint Form must be returned within 48 hours to the address below:

Cadmus ReprintsP.O. Box 751903Charlotte, NC 28275-1903

For reprint inquiries, please contact Mary Leonard at 410 819-3912 (phone), 410-820-9765 (fax) or [email protected].

If you have any problems or questions with your electronic proofs, please contact me.PLEASE ALWAYS INCLUDE YOUR ARTICLE NO. ( J00760-6 ) WITH ALL CORRESPONDENCE.

Sincerely,

Barbara A. MeckleyEditorial Production ManagerJournal of Neurophysiology

300 West Chestnut St.P.O. Box 497Ephrata, PA 17522-0497E-mail: [email protected]

Proofreader’s Marks

MARK EXPLANATION EXAMPLE MARK EXPLANATION EXAMPLE

Journal of Neurophysiology 2007Published by The American Physiological Society

This is your reprint order form or pro forma invoice(Please keep a copy of this document for your records. This form is not for commercial ordering.)

Author Name _______________________________________________________________________________________________Title of Article _______________________________________________________________________________________________Issue of Journal______________________ Reprint # ___________ Manuscript #_____________ Publication Date___________Number of Pages________________________ Color in Article? Yes / No (Please Circle) Symbol JNPlease include the journal name and reprint number or manuscript number on your purchase order or other correspondence.

Order and Shipping Information

Reprint Costs (Please see page 2 of 2 for reprint costs/fees.)

________ Number of reprints ordered $_________________ Number of color reprints ordered $_________

Subtotal $_________Add appropriate sales tax/GST to subtotal $_________First address included, add $32 for each

additional shipping address $________

Publication Fees (Please see page 2 for fees and descriptions.)

Page Charges: $70 per journal page $_________Color Figures: $400 per color figure $_________Hard copy color proof: $75 per figure $_________Toll-Free Link: $150 $____________________ __________________________________Member No. Member Signature

Total Publication Fees $__________

TOTAL TO REMIT $__________

Shipping Address (cannot ship to a P.O. Box) Please Print Clearly

Name _________________________________________Institution______________________________________Street _________________________________________City ______________ State ______ Zip ___________Country________________________________________Quantity___________________ Fax _______________Phone: Day _________________ Evening ___________E-mail Address _________________________________

Additional Shipping Address* (cannot ship to a P.O. Box)

Name _________________________________________Institution______________________________________Street _________________________________________City ______________ State ______ Zip ___________Country________________________________________Quantity__________________ Fax ________________Phone: Day ________________ Evening ____________E-mail Address __________________________________* Add $32 for each additional shipping address

Payment and Credit Card Details (FEIN #:541274108)

Enclosed: Personal Check ___________ Institutional Purchase Order _________ Credit Card Payment Details _________

Checks must be paid in U.S. dollars and drawn on a U.S. Bank.

Credit Card: __ VISA __ Am. Exp. __ MasterCardCard Number __________________________________Expiration Date___________________________________Signature: _______________________________________

Please send your order form and purchase order or prepayment made payable to:

Cadmus Reprints P.O. Box 751903

Charlotte, NC 28275-1903

Note: Do not send express packages to this location, PO Box.

Invoice or Credit Card InformationInvoice Address Please Print ClearlyPlease complete Invoice address as it appears on credit card statementName ______________________________________Institution ______________________________________Department _____________________________________Street ______________________________________City _________________ State _____ Zip ______Country ______________________________________Phone __________________ Fax ______________E-mail Address __________________________________Purchase Order No._______________________________

Cadmus will process credit cards and Cadmus Journal Services will appear on the credit card statement.

If you do not mail your order form, you may fax i t to 410-820-9765 with your credit card information.

AUTHOR Signature __________________________________ Fax_______________________________________ Telephone __________________________________________ E-mail ____________________________________

IMPORTANT Order form must be returned within 48 hours of receipt to avoid late charges. Orders received after 48 hours will be charged an additional fee of 25%. Orders received after 30 days will be charged an additional 50%. Reprints containing color figures are available only if ordered before the journal is printed. It is the policy of Cadmus Reprints to issue only one invoice per order.Please print clearly. Please return form whether reprints are ordered or not.

Page 1 of 2 ML-9-29-06

SIGNATURE REQUIRED: By signing this form the author agrees to accept responsibility for the payment of the mandatory page charges of $70 per page, reprints ordered, as well as any color charges, late payments, and split shipment charges. If the charges are billed to an institution, the author must assume the responsibility for making the necessary arrangements for the issuance of a formal institut ional purchase order. Otherwise, it is understood that the author will bear the cost of these charges. Failure to pay any of these agreed-upon charges could jeopardize future submissions.

J00760-6

Journal of Neurophysiology 2007 Published by The American Physiological Society

REPRINT AND PUBLICATION CHARGES; Author rates only. Not to be used for commercial ordering

Black and White Reprint PricesDomestic (USA only)

# of Pages 100 200 300 400 500

Addl100’s

1-4 $216 $301 $386 $471 $556 $78

5-8 $293 $442 $594 $745 $894 $134

9-12 $377 $573 $774 $969 $1,167 $151

13-16 $452 $715 $978 $1,241 $1,506 $250

17-20 $526 $848 $1,167 $1,489 $1,805 $306

21-24 $612 $988 $1,364 $1,740 $2,116 $359

25-28 $686 $1,130 $1,572 $2,014 $2,456 $421

29-32 $774 $1,271 $1,780 $2,287 $2,795 $482

International (includes Canada and Mexico)

# of Pages

100 200 300 400 500Addl100’s

1-4 $243 $339 $440 $538 $637 $90

5-8 $332 $508 $687 $864 $1,042 $162

9-12 $431 $666 $911 $1,143 $1,386 $191

13-16 $519 $836 $1,153 $1,471 $1,790 $306

17-20 $607 $996 $1,386 $1,773 $2,157 $374

21-24 $704 $1,162 $1,621 $2,079 $2,537 $440

25-28 $795 $1,331 $1,868 $2,408 $2,941 $517

29-32 $895 $1,502 $2,120 $2,736 $3,354 $592

Minimum order is 100 copies. For orders larger than 500 copies, please consult Cadmus Reprints at 800-407-9190.

Page Charges

$70 per journal page for all pages in the article, whether or not you buy reprints.

Color

Reprints containing color figures are available. If your article contains color, you must pay subsidized color charges of$400/fig. (reprint charge is $1000/fig for those who do not pay promptly), whether or not you buy reprints. These color charges are waived for APS Members who are the first or last author of the paper. If you requested a hard copy color figure proofwhen you reviewed your S-proof, the charge is $75.

Shipping

Shipping costs are included in the reprint prices. Domestic orders are shipped via UPS Ground service. Foreign orders are shipped via an expedited air service. The shipping address printed on an institutional purchase order always supersedes.

Multiple Shipments

Orders can be shipped to more than one location. Please be aware that it will cost $32 for each additional location.

State Sales Tax and Canadian GST

Residents of Virginia, Maryland, Pennsylvania, and the District of Columbia are required to add the appropriate sales tax to each reprint order. For orders shipped to Canada, please add 6%Canadian GST unless exemption is claimed.

Color Reprint PricesDomestic (USA only)

# of Pages 100 200 300 400 500

Addl100’s

1-4 $315 $501 $685 $871 $1,055 $179

5-8 $393 $642 $894 $1,144 $1,394 $234

9-12 $477 $773 $1,074 $1,369 $1,668 $250

13-16 $553 $915 $1,279 $1,641 $2,006 $350

17-20 $626 $1,048 $1,468 $1,889 $2,305 $407

21-24 $712 $1,188 $1,664 $2,141 $2,616 $459

25-28 $786 $1,330 $1,872 $2,414 $2,956 $521

29-32 $874 $1,471 $2,081 $2,687 $3,295 $582

International (includes Canada and Mexico)

# of Pages

100 200 300 400 500Addl100’s

1-4 $343 $539 $740 $938 $1,137 $190

5-8 $432 $708 $987 $1,264 $1,542 $262

9-12 $531 $866 $1,211 $1,543 $1,886 $291

13-16 $619 $1,036 $1,453 $1,871 $2,290 $406

17-20 $707 $1,196 $1,686 $2,173 $2,657 $474

21-24 $804 $1,362 $1,921 $2,479 $3,037 $540

25-28 $895 $1,531 $2,168 $2,808 $3,441 $617

29-32 $995 $1,702 $2,420 $3,136 $3,854 $692

Late Order ChargesArticles more than 90 days from publication date will carry an additional charge of $5.50 per page for file retrieval.

TOLL-FREE LINK

A link can be created from a url of your choice to your article online so that readers accessing your article from your url can do so without a subscription. The cost is $150. This is especiallyuseful if your article contains electronic supplemental material. For more information, please click on this link:

http://www.the-aps.org/publications/sprooflink.pdf

Ordering

Please fax your order form and purchase order to410-820-9765. Prepayment of checks should be mailed to address below:

Cadmus ReprintsP.O. Box 751903Charlotte, NC 28275-1903

Note: Do not send express packages to this location.FEIN #:541274108

Please direct all inquiries to:

Mary Leonard800-407-9190 (toll free number)410-819-3912 (direct number)410-820-9765 (FAX number)[email protected]

Page 2 of 2

Reprint Order Forms and Purchase Orders or prepayments must be received 48 hours after receipt of form.

Please return thisform even if no

reprints are ordered.

Decoding M1 Neurons During Multiple Finger Movements

S. Ben Hamed,1 M. H. Schieber,1,2 and A. Pouget1

1Department of Brain and Cognitive Science and the Center for Visual Science, University of Rochester, Rochester, New York; and2Departments of Neurology and Neurobiology, University of Rochester, School of Medicine and Dentistry, Rochester, New York

Submitted 21 July 2006; accepted in final form 5 April 2007

Hamed SB, Schieber MH, Pouget A. Decoding M1 neurons duringmultiple finger movements. J Neurophysiol 98: 000–000, 2007. Firstpublished April 11, 2007; doi:10.1152/jn.00760.2006. We tested sev-eral techniques for decoding the activity of primary motor cortex (M1)neurons during movements of single fingers or pairs of fingers. Wereport that single finger movements can be decoded with �99%accuracy using as few as 30 neurons randomly selected from popu-lations of task-related neurons recorded from the M1 hand represen-tation. This number was reduced to 20 neurons or less when theneurons were not picked randomly but selected on the basis of theirinformation content. We extended techniques for decoding singlefinger movements to the problem of decoding the simultaneousmovement of two fingers. Movements of pairs of fingers were de-coded with 90.9% accuracy from 100 neurons. The techniques weused to obtain these results can be applied, not only to movements ofsingle fingers and pairs of fingers as reported here, but also tomovements of arbitrary combinations of fingers. The remarkablysmall number of neurons needed to decode a relatively large repertoireof movements involving either one or two effectors is encouraging forthe development of neural prosthetics that will control handmovements.

I N T R O D U C T I O N

Decoding population neuronal activities has become animportant topic in neuroscience. From a theoretical perspec-tive, the ability to read out neural patterns of activity can helpus understand the structure of the neuronal code, how infor-mation is distributed over several thousand neurons, and how itflows from one area to the next (Dayan and Abbott 2001;Pouget et al. 2003). From a more clinical perspective, efficientdecoding of neuronal population activity will be crucial fordevelopment of neuroprosthetics that are controlled on-line byneural activities.

Several decoders have been used in previous studies, such asthe population vector, the optimal linear estimator, the maxi-mum likelihood estimator, or the Bayesian estimator (Georgo-poulos et al. 1986; Pouget et al. 2003; Salinas and Abbott 1994;Seung and Sompolinsky 1993). All of these methods use apopulation pattern of activity as input and return an estimate(or a probability distribution in the case of the Bayesianapproach) of the encoded variable. In all cases, the encodedvariable is assumed to take only one value at any given time.This is indeed a reasonable assumption in most cases. Forinstance, in decoding a reaching movement, the hand can bemoving in only one direction at any given time. Likewise,when decoding the position of a rat in a maze, the rat can be inonly one place at any time.

The situation becomes more complex, however, in decodingthe concurrent motion of multiple effectors. Finger movementsprovide an important example because one finger might movein isolation or multiple fingers might move together. Onesolution to the problem of decoding these movements might beto develop a separate estimator for each finger, but this ap-proach rests on the assumption that each finger is controlledindependently of the others. In most natural behaviors, how-ever, multiple fingers move together (for review, see Schieberand Santello 2004), suggesting that the pattern of neuralactivity needed to move the index finger, for example, maydiffer depending on whether the index moves in combinationwith the middle finger, in combination with the ring finger, orin isolation. It is therefore unclear whether training multipledecoders in parallel—one per finger—can lead to a high levelof classification performance using a reasonably small numberof M1 neurons.

We therefore examined several different methods of decod-ing finger movements. Surprisingly, we found that 100 neuronsare sufficient to decode movements of pairs of fingers with anaccuracy of 90.9%. Moreover, the same estimator can be usedto decode single finger movements with near-perfect (�99%)accuracy using as few as 30 neurons.

M E T H O D S

This study describes analysis of data collected from two monkeysthat have been the subject of previous reports (Georgopoulos et al.1999; Schieber 1991; Schieber and Hibbard 1993). Methods used forbehavioral training and data collection have been described in thesereports and are summarized here as needed. All care and use of thesemonkeys complied with the U.S.P.H.S. Policy on Humane Care andUse of Laboratory Animals and was approved by the institutionalanimal care and use committee.

Behavioral task

Each monkey was trained to perform visually cued individuatedflexion and extension movements of the right hand fingers and/orwrist (Schieber 1991). As the monkey sat in a primate chair, the rightelbow was restrained in a molded cast, and the right hand was placedin a pistol-grip manipulandum that separated each finger into adifferent slot. At the end of each slot, the fingertip lay between twomicroswitches. By flexing or extending the digit a few millimeters, themonkey closed the ventral or dorsal switch, respectively. The manipu-landum, in turn, was mounted on an axis that permitted flexion andextension wrist movements. Each monkey viewed a display on whicheach digit (and the wrist) was represented by a row of five light-emitting diodes (LEDs). When the monkey flexed or extended a digit,

Address for reprint requests and other correspondence: A. Pouget, Brain andCognitive Science Dept., Meliora Hall, Univ. of Rochester, Rochester, NY14627 (E-mail: [email protected]).

The costs of publication of this article were defrayed in part by the paymentof page charges. The article must therefore be hereby marked “advertisement”in accordance with 18 U.S.C. Section 1734 solely to indicate this fact.

J Neurophysiol 98: 000–000, 2007.First published April 11, 2007; doi:10.1152/jn.00760.2006.

10022-3077/07 $8.00 Copyright © 2007 The American Physiological Societywww.jn.org

AQ: A AQ: A

tapraid4/z9k-neurop/z9k-neurop/z9k00707/z9k8250d07a moyerr S�6 5/11/07 22:29 MS: J00760-6 Ini: Input-dlw

closing a microswitch, the central yellow LED went out and a greenLED to the left or right, respectively, came on, cueing the monkey asto which switch(es) had been closed. Red LEDs to the far left or rightwere illuminated one at a time, under microprocessor control, instruct-ing the monkey to close that one switch (or move the wrist). If themonkey closed the instructed switch within the 700 ms allowed afterillumination of the red instruction LED and held it closed for a 500-msfinal hold period without closing any other switches, the monkeyreceived a water reward. After each rewarded trial, the movement tobe instructed for the next trial was rotated in a pseudorandom order.We abbreviate each instructed movement with the number of theinstructed digit (1 � thumb through 5 � little finger, W � wrist), andthe first letter of the instructed direction (f – flexion; e – extension);for example, “4f” indicates instructed flexion of the ring finger.

Monkeys K and S each performed all 12 possible movementsinvolving flexion or extension of a single digit or of the wrist (1f, 1e,2f, 2e, 3f, 3e, 4f, 4e, 5f, 5e, Wf, and We). In addition, each monkeyalso performed movements involving concurrent flexion or extensionof two digits. Concurrent flexion of the thumb and index finger, forexample, which we denote 1 � 2f, was instructed to the monkey bysimultaneous illumination of the red LEDs used to instruct thumbflexion and index finger flexion. To obtain a reward, the monkey wasrequired to close either the thumb or index finger flexion switch within700 ms, to close the remaining switch within 100 ms of closing thefirst, and to hold them both closed for 500 ms without closing anyother switches. Monkey K performed six multiple finger movements:1 � 2f, 1 � 2e, 2 � 3f, 2 � 3e, 4 � 5f, 4 � 5e. Monkey S performedtwo: 1 � 2f, 1 � 2e. For each monkey, these multiple fingermovements were presented intermingled with the 12 single fingermovements in a pseudorandomized block design.

Data collection

After training, aseptic surgery under general anesthesia was used toperform a craniotomy over the left central sulcus at the level of thehand representation to implant a rectangular Lucite recording chamberover the craniotomy and to implant head-holding posts. Once themonkey had recovered from this procedure and had become accus-tomed to performing the finger movement task with the head heldstationary, single neuron activity was recorded with conventionaltechniques as the monkey performed the finger movement task eachday. Data were collected from 115 task-related neurons in the primarymotor cortex (M1) of monkey K and from 61 M1 neurons in monkeyS. In both cases, neurons were recorded from area 4 in the anteriorbank and lip of the central sulcus.

Decoding methods

To include a neuron in this analysis, we required that the neuronhad been recorded for a minimum of nine trials of each type of fingermovement performed by the monkey. Data sufficient for these anal-yses were available from a total of 140 neurons: 100 from monkey Kand 40 from monkey S. For a few neurons, as many as 19 trials ofeach movement were available. For each trial and for each neuron,spikes were counted in the 100-ms interval directly preceding the endof the movement (switch closure), a period when most M1 neuronsreach peak firing rates. Using these data from M1 neurons in monkeysK and S, we examined the performance of the following five methodsof decoding finger movements:

REGULAR POPULATION VECTOR. The population vector (PV) istypically used to estimate a periodic variable, like the direction of armmovement, from a set of neuronal responses (Georgopoulos et al.1986). Georgopoulos and colleagues have shown how to extend thisapproach to discrete classes, and this approach has been used previ-ously to analyze movements of single fingers (Georgopoulos et al.1999). In this study, each finger movement is assigned a direction in

three-dimensional (3D) space, with all 12 directions equally spaced in3D. Then, on each trial k, the PV, pk, is computed according to

pk � �i�1

N

cirik (1)

where rik is the activity of neuron i on trial k (i.e., the spike count in

the 100-ms interval preceding the end of the movement), and ci is thepreferred direction for neuron i.

For all cells, the preferred directions {ci} were calculated using thedata from all available trials but two, and the PV was estimated usingthe data from the remaining two trials. This was repeated on allpossible pairs out of the nine minimum trials retained for all cells, andthe resulting PVs were averaged to produce a mean PV estimate.

In the Results, we report the performance of the PV in terms of theaverage angular difference between the vector assigned to the actualmovement on trial k and the decoded population vector pk. We alsoreport the percentage of correct responses taken as the percentage ofconditions in which the PV was closest to the vector assigned to theactual movement.

OPTIMAL PV. The optimal population vector (OPV) is similar to theregular PV estimator in the sense that, on every trial k, we firstestimate a vector pk by taking a linear combination of the neuralactivities

pk � �i�1

N

wirik

The main difference from the PV is that for the OPV the weights,�wi�i�1. . .N, are not set a priori to the preferred directions, �ci�i�1. . .N.Instead, we seek the weights that minimize the cost function

E � �t�1

k

�pk � p*k�2

where p*k is the vector corresponding to the actual movement on trialk. The optimal weights are given by

WT � � �1Crp*

where W is the weight matrix with column �wi�i�1. . .N, � is thecovariance matrix of the neuronal responses, and Crp* is the covari-ance matrix between the response and the vector p* corresponding tothe actual movements.

Here again, for all cells, the optimal weights �wi�i�1. . .N werecalculated on all available trials but two, and the PV was estimated onthe remaining two trials. This was repeated on all possible pairs out ofthe nine minimum trials retained for all cells, and the resulting PVswere averaged to produce a mean PV estimate.

LOGISTIC REGRESSION. Logistic regression (LR) is equivalent totraining a nonlinear two layer neural network. The input layer con-sisted of N units corresponding to N M1 neurons. N was varied asdescribed in the final section of METHODS. These N input unitsprojected onto 12 output units, 1 output unit for each possiblemovement (Fig. 1A). The activity of output unit i on trial k, yi

k, wasdetermined according to

yik �

1

1 � exp��j�1

N

wijrjk � bi�

where �wij�i�1. . .6, j�1. . .N are the weights, bi is the bias of output unit i,�rj

k�j�1. . .N are the activities of the M1 neurons on trial k, and � is aconstant (� � 0.0002). The weights were optimized through gradient

2 S. B. HAMED, M. H. SCHIEBER, AND A. POUGET

J Neurophysiol • VOL 98 • JULY 2007 • www.jn.org

F1


descent with cross-validation as described below. With this particularactivation function, the value of each output unit is constrained to liewithin the [0,1] interval. This enabled us to interpret the activity ofoutput unit i as encoding the probability that movement i wasproduced on trial k.

This approach was applied to both single finger movements andmovements of pairs of fingers, but with one major caveat: the numberof fingers moving on a given trial was known for each trial. If only onefinger moved, for that trial, we picked the one finger with the highestprobability. If two fingers moved, we picked the two fingers with thehighest probability.

SOFTMAX. The softmax (SM) estimator (Bishop 1995) was used toaddress the main limitation of the LR approach, the requirement thatthe number of fingers moving on each trial must be known a priori.For the SM network, the output layer was organized into six sub-groups corresponding to the five fingers and the wrist. Each subgroupcontained three units receiving inputs from all M1 neurons. Thesethree output units were trained to encode the probabilities of nomovement, of extension, and of flexion, respectively (Fig. 1B).(Again, weights were optimized through gradient descent with cross-validation, see METHODS). Because these three states of each finger aremutually exclusive (e.g., a given finger cannot flex and extend at thesame time), the three probabilities were constrained to sum to unitythrough an SM normalization; thus the activity of unit i (with i � 1.3),in the jth subgroup (j � 1.6), for the kth trial is given by

oijk �

exp� �i�1

N

wijlrtk�

�j�1

3

exp� �t�1

N

wijlr1k�

Note that this equation ensures that the activity of the three units

within a subgroup sums up to one, i.e., �j�1

3

oijk for any subgroup i and

any example k.

SM-BEST VOTE-SELECTED SET (USING MUTUAL INFORMATION). Herewe ranked the neurons from most informative to least informative. Todo so, we computed the mutual information between single neuronsand finger movements, using the standard definition (Dayan andAbbott 2001)

I�ri;m� � H�ri� � H�ri�m�

where ri and m are two random variables corresponding to theresponse of neuron i and the movement performed by the monkey.The probability distributions, p(ri) and p(ri�m), which are needed tocompute the entropies H(ri) and H(ri�m), were estimated from the databy counting the frequency of each response using all the data avail-able. This method is known to overestimate information, but we usedI only to rank the neurons from most informative to least informative,enabling us to reduce the number of neurons needed for optimaldecoding performance using the softmax estimator. This approach islabeled SM-best vote in Figs. 3 and 4.

Training and cross-validation

The parameters of all of the estimators described above wereestimated with cross-validation. The dataset contained N neurons forwhich neuronal activity recorded in 9–19 trials was available (thenumber of neurons, N, was varied). For each of the 18 movements (12single finger movements and 6 multiple finger movements), two trialsof data from each neuron were set aside for testing the network. Fortraining the network, single remaining trials of data from each neuronwere selected randomly to generate 10,000 patterns of populationactivity (56 for each of the 18 movements). In this way, wegenerated population patterns of activity for each movement eventhough the neurons actually were recorded one at a time on differentdays. For testing the network, we generated 200 new populationactivity patterns (1 for each of the 18 movements) using the twotrials of data from each neuron during each movement set aside forthis purpose.

In the case of LR and SM, we used stochastic gradient descent tofind the weights minimizing the squared error over all movements andall trials. While optimizing the weights on the training set, we alsomonitored classification performance on the testing set. Training wasstopped when performance started decreasing on the testing set, aprocedure known as early stopping. This procedure prevents overfit-ting on the training set and provides a better estimate of generalizationperformance (Morgan and Bourlard 1990).

Varying the number of neurons being decoded

For all five decoding methods, we systematically varied the numberof neurons, N, being decoded to determine the minimum number of

FIG. 1. Network representations of the logistic regression (LR) and softmax (SM) estimators. A: logistic regression. Input layer encodes the activity of N M1neurons before a finger movement (N varies from 5 to 140, depending on simulation). Output layer contains 1 unit per movement for a total of 12 units (5 fingersplus wrist and 2 movements per finger or wrist). Output units are trained to encode probability of movement they encode given current input activities. Traininginvolves optimizing weights W through gradient descent with cross-validation. B: SM. Input layer is as in A. Output layer is organized in 6 triplets of units (5fingers plus wrist). Each triplets encodes probability distributions over 3 mutually exclusive states of corresponding finger (or wrist): no movement, flexion, andextension. SM normalization is used within each triplet to ensure that activity of the 3 units always sum up to 1, as should be the case for a probability distribution.Weights W are trained through gradient descent with cross-validation.

3DECODING MULTIPLE FINGER MOVEMENTS



neurons needed for the best possible classification. N was set to 140,100, 60, 40, 30, 20, 15, 10, and 5. For each value of N, theperformance of the estimator was evaluated by randomly choosing Ncells with replacement from the 140 cells available and optimizingand testing the estimator with those N cells. This process was repeated10 times, and we report the average of these 10 repetitions.

R E S U L T S

Decoding of single finger movements with the regular PVand OPV estimators

We first decoded the 12 single finger and wrist movementsfrom the discharge of M1 neurons using both the regular PVand OPV estimators (see METHODS). The average error, ex-pressed in terms of the angle between the estimated directionand true direction, was 34.07° for the PV and 14.18° for theOPV, when using all 140 cells. These numbers are comparablewith those reported by Georgopoulos et al. (1999), althoughthis previous study did not use cross-validation.

Figure 2 shows the results expressed in terms of percentageof correct classification as a function of the number of cellsbeing decoded. The OPV outperformed the PV for all numbersof cells. In fact, the PV never approached 99% correct, evenwith 140 cells, whereas OPV reached 99.2% accuracy using asfew as 60 randomly selected cells.

Decoding of single finger movements with LR and SM

LR is the optimal classifier when the data follows a multi-variate Gaussian distribution with a fixed covariance matrix(Bishop 1995). Because this assumption holds to a first ap-proximation with neural activities, LR was a natural estimatorto try with our data set. Moreover, LR has the advantage that

it estimates the probability distribution over all possible move-ments given the vector of neural activity, instead of generatinga single answer as do the PV and OPV. Figure 2 shows theperformance of the LR estimator compared with PV and OPV.LR outperformed both population vector estimators, decodingsingle finger movements with 99.48% accuracy using as few as40 randomly selected cells.

To address the main limitation of the LR estimator, namelythe need to know how many fingers are moving on each trial,we developed another estimator, which we call an SM estima-tor. This estimator returns six probability distributions: one foreach finger and one for the wrist. Each probability distributionis defined over three possible states: flexion, extension, and nomovement. The SM estimator can be read out on every trialwithout prior knowledge of the number of fingers moving: wesimply look for the most probable state of each finger. Atypical answer might look like the following: no movement forfinger 1, flexion for finger 2, no movement for finger 3, flexionfor finger 4, no movement for finger 5, and no movement forthe wrist. This approach can be used to estimate single fingermovements, two finger movements, and more generally, anyarbitrary combination of flexion and extension over the fivefingers and wrist, regardless of the number of fingers moving.

Figure 3 shows the performance of the SM estimator (SMexact match) compared with LR on decoding single fingermovements. The SM estimator performed just as well as LR,even though it used no prior knowledge of the number offingers moving on each trial. Remarkably, the SM estimatornot only assigns the highest probability to the proper move-

FIG. 3. Comparative performance of LR and SM estimators on decodingsingle finger movements as a function of dataset size. Two different classifi-cation criteria were applied to SM. In a most restrictive decoding scheme, SMestimator was considered to be correct only when it predicted correct move-ment for moving finger and no movement for all other fingers. In SM best vote,SM estimator was considered to be correct when it assigned highest probabilityto correct movement for moving finger, even if it predicted no movement forall other fingers. In other words, we assume that we know that only 1 fingermoves on each trial, as was done for the LR estimator. Finally, for SM bestvote, selected set, we applied the same criteria as for SM best vote, but we usedthe N cells with highest mutual information between responses and movement.Gray vertical dashed line indicates number of cells for which SM best votereached a detection performance of 100%. Gray vertical continuous lineindicates number of cells for which SM best vote, selected set reached adetection performance of 100%. SE is not plotted on this graph to avoidunnecessary clutter, but they are in the same range as in Fig. 2.

FIG. 2. Comparative performance of the regular population vector (PV),optimal population vector (OPV), and LR estimators on decoding single fingermovements as a function of number of neurons being decoded. Gray barsrepresent SE. Long gray vertical line indicates number of cells for which LRreached a detection performance of 100%.



F2

COLOR

F3

COLOR


ment for the proper finger but also assigns the highest proba-bility to no movement for all the other fingers.

For further comparison with the LR estimator, we also testedthe SM estimator when assuming that we know that only onefinger moved on each trial. This means that we read out the SMestimator in the same way as the LR estimator, searching onlyfor the movement state (flexion or extension) with the highestprobability, while ignoring all other answers. Therefore even ifthe SM estimator predicted that more than one finger moved,for this comparison, we nevertheless considered the predictioncorrect as long as the most likely movement was indeed the onethat took place on that trial. As can be seen in Fig. 3 (SM bestvote), this SM approach improves performance still further,decoding which finger moved at 99.6% accuracy using as fewas 30 randomly selected neurons.

Finally, we asked whether the number of cells needed todecode at �99% accuracy could be reduced below 30 if weselected the neurons that provided the most information aboutwhich finger movement was performed. We therefore com-puted the mutual information between each neuron and thefinger movements, and we trained the SM estimator using theN neurons with the highest information content. The number Nwas varied systematically to obtain the performance of the SMestimator as a function of the number of neurons. Using themost informative neurons, 20 selected cells were sufficient for99.91% performance (Fig. 3, SM best vote-selected set). Infact, accuracy still was very high with as few as 15 cells(97.3% correct).

Of the 20 most informative cells, none discharged for onefinger movement only. Instead, each of these neurons wasintensely active during two or more finger movements andvirtually inactive during others. Such distributed coding indeedis to be expected, given that a neuron that fired during only onefinger movement would convey relatively little information(0.4 bits) compared, for instance, with a neuron that firedstrongly for six movements and not at all for the other six (1bit).

Our selection procedure was not necessarily optimal, how-ever. We do not know that the 20 neurons with the highestmutual information constitute the best subset for classifying allmovements. Therefore 20 neurons is an upper bound on theminimum number of neurons needed for near-perfect classifi-cation. However, better performance might be obtained withfewer neurons selected in a manner other than the one usedhere.

Decoding of multiple finger movements: training on singlefinger movements

Neither the PV nor the OPV is suitable for decoding simul-taneous movements of multiple fingers because these estima-tors each return a single answer on any given trial. Wetherefore studied the LR and SM approaches for decoding trialsin which two fingers moved at the same time. Here we used thepopulation of 100 neurons from monkey K that had beenrecorded during at least nine trials of the 12 single finger andwrist movements plus the 6 movements in which the monkeymoved two fingers at the same time.

First we explored whether an estimator trained on singlefinger movements can generalize to two-finger movements.Our results show that this is not the case. The LR estimator

performed reasonably well on some two-finger combinations(e.g., 94% correct on 1 � 2e) but poorly on others (2% on 4 �5e, which is below chance) for an average performance of 38%correct over all six two-finger movements. The SM estimatorfared worse, failing to generalize from single finger movementto two-finger movements. The inability of the LR and SMmethods to estimate two-finger movements after training onsingle finger movements indicates that 1) the population activ-ity patterns generated for different single-finger movementsoverlap (in that single cells discharge significantly duringseveral single-finger movements (Poliakov and Schieber 1999;Schieber and Hibbard 1993) and 2) the patterns of neuralactivity for two-finger movements are unlikely to be simplelinear combinations of the patterns for single-finger move-ments.

To examine this idea further, we ran the following test. Forall two-finger movements AB, where A and B refer to thesingle finger movements involved in this combination, wecomputed the angle between the population activity vector forAB and the vector obtained by taking the sum of the twopopulation activity vectors for A and B. If the activity for thetwo finger movement is simply equal, or proportional to, thesum of the activities for the single finger movement, the angleshould be close to zero. As can be seen on Fig. 4, the angle isclose to 45° on average, indicating that some nonlinearity isinvolved.

Given the inability of the LR and SM approaches to gener-alize from single finger movements to two-finger movements,we tested both methods after training on all 18 single-fingerand two-finger movements. Figure 5 shows the performance ofseveral networks based on the activity of 100 neurons duringthe different movements. The LR network achieved nearlyperfect performance on all combinations (overall average of98.98%; Fig. 4, LR). However, this performance still relied onprior knowledge of whether the movement involved one fingeror two fingers. The SM performance was nearly perfect onsingle-finger movements (99.25%) and several two-fingermovements (Fig. 5, SM exact match), for an average perfor-mance of 90.9% over all 18 movements. Interestingly, andunlike what we observed for single-finger movements, perfor-mance of the SM estimator did not improve when tested witha priori knowledge of whether one or two fingers moved oneach trial (Fig. 5, SM best vote).

D I S C U S S I O N

Decoding of single finger movements and implications forneural prosthetics

Although many neural prosthetic approaches use recordingsfrom large numbers of neurons, we were able to decode 12single finger movements with �99% accuracy using as few as30 neurons randomly selected from a population of 140 task-related neurons. If we assume that an individual can producethree finger movements per second, our performance corre-sponds to a bit rate of 10.75 bits/s [3 log2(12)] for 40neurons, which is significantly better than the current state ofthe art, which is 6 bits/s for 100 neurons (Santhanam et al.2006; Taylor et al. 2003). This performance is also particularlyremarkable given that it was obtained with relatively little data,because we had at most 19 trials per finger movement collected



AQ: 1

F4

F5


from any given neuron during recordings that lasted at most 30min. Decoding performance using the LR and SM estimatorsshould improve considerably with long-term recordings fromimplanted electrodes, when hundreds to thousands of trials permovement condition could be collected and used to train thedecoding estimators.

These conclusions, however, should be tempered by the factthat we used independent neurons, whereas these other studiesrelied on neurons recorded simultaneously. As we discussbelow, it is possible that our performance would decreasesignificantly with correlated neurons. Moreover, it is possiblethat the neurons we used were particularly well tuned com-pared with the neurons typically found with multiunit record-ings using grids. Indeed, in single cell recordings, the experi-mentalist can select neurons with high firing rates, which arelikely to be more informative. With grids recordings, this is notan option because the electrodes cannot be moved indepen-dently. Finally, the M1 neurons from which we recorded mayhave responded in part to proprioceptive feedback. In a para-lyzed individual, such feedback would not be present, and thedischarge of M1 neurons might differ significantly. This in turn

could affect decoding performance. The precise impact of thisfactor cannot be assessed with our data set but deserves to beinvestigated in future studies.

Decoding of multiple finger movements

Decoding multiple finger movements failed when the LRand SM estimators were trained exclusively on single fingermovements. This failure may arise in part from the fact that theneural and muscular activation required to move two fingers isnot simply the linear combination of the activations required tomove each finger in isolation (Schieber 1990). Because multi-ple adjacent fingers tend to move together, movement of asingle finger in isolation requires contraction of both themuscles that move that finger and additional muscles thatcheck unwanted motion in adjacent digits (Reilly and Schieber2003; Schieber 1995; Schieber and Santello 2004). Hencegeneralization of a network trained on single finger movementsto decode multiple finger movements is unlikely to succeed.When the LR and SM estimators were trained using data from

FIG. 4. Test on nonlinearity for 2 finger movements. Foreach 2-finger movement, we computed angle between pop-ulation activity vector for this 2-finger movement and sum ofpopulation activity for the 2 single finger movements in-volved in this pair. Black dots correspond to single trials;circles correspond to average angles for each 2-finger move-ment. For all 2-finger movements, angle is close to 45° onaverage, indicating that patterns of neural activity for 2-fin-ger movements are not simply sums of patterns for single-finger movements.

FIG. 5. Comparative performance of LR and SMestimators on decoding single and multiple finger move-ments. SM exact match and SM best vote are as in Fig.3. Downward pointing bars represent SE.




all 12 single-finger and 6 two-finger movements, however,both methods provided accurate decoding using 100 neurons.

Nonetheless, with 140 cells, we were able to obtain �90%accuracy in decoding combination finger movements. We donot have enough data to know whether this performance wouldgeneralize to arbitrary combinations of movements. Assumingfor purposes of discussion that a subject could produce threecombinations of six possible movements per second (1 bit perdigit or wrist), the bit rate potentially would reach 18 bits/s. Inpart, such a hypothetical bit rate, well above what has beenreported by other groups, may reflect the simultaneous controlof six effectors. These encouraging numbers are likely to beoverestimates, however, given that all the caveats we haveraised in decoding single movements would apply to combi-nation movements as well, and in particular, the issue ofcorrelations.

Impact of correlations

Because each of the present population of M1 neurons wasrecorded at a different time, the activities of different neuronsin our population estimates effectively were uncorrelated. Ourresults might have been different had we used neural dataobtained from simultaneous recordings of multiple single neu-rons, because the activity of simultaneously recorded corticalneurons often can be shown to be correlated (Bair et al. 2001;Maynard et al. 1999; Zohary et al. 1994). Such correlations caneither increase or decrease the quality of decoding dependingon their exact pattern (Abbott and Dayan 1999; Yoon andSompolinsky 1999), however, making it difficult to predictwhether performance would have been better or worse usingsimultaneously recorded neurons. Studies that have comparedtruly simultaneous recordings to “shuffled” recordings (equiv-alent to single unit recordings because shuffling data fromdifferent trials effectively removes the correlations betweenneurons) suggest that the impact of correlations is insignificant,or very small, for population of tens of neurons (Averbeck andLee 2003; Panzeri et al. 2003). Moreover, Serruya et al. (2002)have found recently that 7–30 neurons recorded simultaneouslyare sufficient to control the location of a computer cursor on ascreen. Although this task is quite different from the presentfinger movement task, their findings support the notion that�40 neurons can be sufficient for decoding, even when theneurons are recorded simultaneously. Our finding that as fewas 30–40 M1 neurons were sufficient for decoding singlefinger movements therefore may have been similar even withsimultaneously recorded neurons.

For larger neuronal sets, such as the 140 neurons we used forsingle finger movements, assessing the impact of correlatedneuronal activity is more difficult because little is known aboutcorrelations among such large neuronal populations in vivo.Extrapolation from pairwise recordings suggests that, if ourneurons were all equally positively correlated (the worse casescenario from the standpoint of decoding information), corre-lations become a major concern only for 1,000 neurons or more(Zohary et al. 1994). Therefore our results with 140 neuronsrecorded one at a time are likely to be reasonably close to whatwould have been obtained with 140 simultaneously recordedneurons. Nonetheless, there is little doubt that the impact of

correlations on decoding multiple movements should be thor-oughly studied in future experiments.

G R A N T S

S. B. Hamed and A. Pouget were supported by National Institute of MentalHealth Grant MH-57823, research grants from the Office of Naval Research,and the McDonnell-Pew, Sloan, and Schmitt foundations. M. H. Schieber wassupported by National Institute of Neurological Disorders and Stroke GrantNS-27686.

R E F E R E N C E S

Abbott LF, Dayan P. The effect of correlated variability on the accuracy ofa population code. Neural Comput 11: 91–101, 1999.

Averbeck BB, Lee D. Neural noise and movement-related codes in themacaque supplementary motor area. J Neurosci 23: 7630–7641, 2003.

Bair W, Zohary E, Newsome WT. Correlated firing in macaque visual areaMT: time scales and relationship to behavior. J Neurosci 21: 1676–1697,2001.

Bishop CM. Neural Networks for Pattern Recognition. New York: Oxford,1995.

Dayan P, Abbott LF. Theoretical Neuroscience. Cambridge, MA: MIT Press,2001.

Georgopoulos AP, Pellizzer G, Poliakov AV, Schieber MH. Neural codingof finger and wrist movements. J Comput Neurosci 6: 279–288, 1999.

Georgopoulos AP, Schwartz AB, Kettner RE. Neuronal population codingof movement direction. Science 233: 1416–1419, 1986.

Maynard EM, Hatsopoulos NG, Ojakangas CL, Acuna BD, Sanes JN,Normann RA, Donoghue JP. Neuronal interactions improve cortical pop-ulation coding of movement direction. J Neurosci 19: 8083–8093, 1999.

Morgan N, Bourlard HA. Generalization and parameter estimation in feed-forward nets: some experiments. In: Advances in Neural Information Pro-cessing Systems 2, edited by Touretzky D. San Mateo, CA: MorganKaufmann, 1990.

Panzeri S, Petroni F, Petersen RS, Diamond ME. Decoding neuronalpopulation activity in rat somatosensory cortex: role of columnar organiza-tion. Cereb Cortex 13: 45–52, 2003.

Pouget A, Dayan P, Zemel RS. Inference and computation with populationcodes. Annu Rev Neurosci 26: 381–410, 2003.

Reilly KT, Schieber MH. Incomplete functional subdivision of the humanmultitendoned finger muscle flexor digitorum profundus: an electromyo-graphic study. J Neurophysiol 90: 2560–2570, 2003.

Salinas E, Abbott LF. Vector reconstruction from firing rates. J ComputNeurosci 1: 89–107, 1994.

Santhanam G, Ryu SI, Yu BM, Afshar A, Shenoy KV. A high-performancebrain-computer interface. Nature 442: 195–198, 2006.

Schieber MH. How might the motor cortex individuate movements? TrendsNeurosci 13: 440–445, 1990.

Schieber MH. Individuated finger movements of rhesus monkeys: a means ofquantifying the independence of the digits. J Neurophysiol 65: 1381–1391,1991.

Schieber MH. Muscular production of individuated finger movements: theroles of extrinsic finger muscles. J Neurosci 15: 284–297, 1995.

Schieber MH, Hibbard LS. How somatotopic is the motor cortex hand area?Science 261: 489–492, 1993.

Schieber MH, Santello M. Hand function: peripheral and central constraintson performance. J Appl Physiol 96: 2293–2300, 2004.

Serruya MD, Hatsopoulos NG, Paninski L, Fellows MR, Donoghue JP.Instant neural control of a movement signal. Nature 416: 141–142, 2002.

Seung HS, Sompolinsky H. Simple models for reading neuronal populationcodes. Proc Natl Acad Sci USA 90: 10749–10753, 1993.

Taylor DM, Tillery SIH, Schwartz AB. Information conveyed throughbrain-control: cursor vs. robot. IEEE Trans Neural Syst Rehabil Eng 11:195–199, 2003.

Yoon H, Sompolinsky H.The effect of correlations on the Fisher informationof population codes. In: Advances in Neural Information Processing Sys-tems, edited by Kearns M, Solla S, Cohn D. Cambridge, MA: MIT Press,1999, p. 167–173.

Zohary E, Shadlen MN, Newsome WT. Correlated neuronal discharge rateand its implications for psychophysical performance. Nature 370: 140–143,1994.



AQ: 2


JOBNAME: AUTHOR QUERIES PAGE: 1 SESS: 1 OUTPUT: Thu May 10 05:57:21 2007/tapraid4/z9k�neurop/z9k�neurop/z9k00707/z9k8250d07a

AQA— Please verify the accuracy of your e-mail address or delete it if you do not wish itincluded.Au: Please check all Tables and Equations carefully for presentation and accuracy.All tables and display equations (those set off from running text) are set by hand during thecomposition process. I have edited all of these slightly. As you check the page proofs, pleaseindicate all necessary changes to tables and equations.

AQ1— Please add Poliakov and Schieber 1999 to reference list or delete citation here.

AQ2— Please provide page range.

AUTHOR QUERIES

AUTHOR PLEASE ANSWER ALL QUERIES 1

journal of neurophysiology

Documents