stakeholders, technical and financial evaluation...deliverable title stakeholders, technical and...
TRANSCRIPT
D8.2.3Stakeholders,TechnicalandFinancialEvaluation
CIP Competitiveness and Innovation Framework Programme CIP-Pilot Actions, 2007-2013 CIP-ICT-PSP-2012-6
Project CIP-Pilot 325101 / OpenScienceLink Deliverable D8.2.3 Distribution Public
http://opensciencelink.eu
Stakeholders, Technical and Financial Evaluation
IordanisMourouzis, Costas Pantos, AdomasBunevicius, InesaBirbilaite,
TodorTagarev, PetyaTagareva, VassilikiAndronikou, EfstathiosKaranastasis, Giorgio Iervasi, Alessandro Pingitore, Laurens Naudts, Michael R. Alvers,
George Tsatsaronis Status: Draft (Version 1.0)
Page 2 of 82
D8.2.3Stakeholders,TechnicalandFinancialEvaluation
February 2016
Project
Project ref.no. CIP-Pilot 325101 Project acronym OpenScienceLink Project full title Open Semantically-enabled, Social-aware Access to
Scientific Data Project site http://opensciencelink.eu Project start February 2013 Project duration 3 years EC Project Officer Martin Májek
Deliverable
Deliverable type Report Distribution level Public Deliverable Number D8.2.3 Deliverable title Stakeholders, Technical and Financial Evaluation Contractual date of delivery M36 (January 2016) Actual date of delivery February 2016 Relevant Task(s) WP8/Tasks 8.1, 8.2, 8.3 Partner Responsible NKUA Other contributors CNR, TUD, NTUA, LUHS, NKUA, KU Leuven, Procon,
TI Number of pages 87 Author(s) Costas Pantos, IordanisMourouzis, AdomasBunevicius,
InesaBirbilaite, TodorTagarev, PetyaTagareva, VassilikiAndronikou, EfstathiosKaranastasis, Giorgio Iervasi,Alessandro Pingitore,Freyja van den Boom, Laurens Naudts, Michael R. Alvers, George Tsatsaronis
Internal Reviewers Liliana Alvers, Michael Schroeder, EfthymiosChondrogiannis
Status & version Keywords Evaluation Framework, Project's Results, Methodologies,
Tools, Key Performance Indicators (KPIs), Metrics, Technical Evaluation, Financial Evaluation
Page 3 of 82
D8.2.3Stakeholders,TechnicalandFinancialEvaluation
Executive Summary
The current deliverable presents a multi-facet evaluation of the 3rd year version of the OSLplatformcoveringdifferentperspectives.First,wehavecollectedandanalyzedfeedbackfromallstakeholders ofthe OpenScienceLink ecosystem, including researchers, scholars, universities,research organizations, researchsponsors, funding agencies, funding authorities, open-accesspublishersand the media. The evaluation was based on severalmodalities includingquestionnaires,interviews,satisfactionsurveysandothertoolsthatwerespecifiedaspartoftheproject’s evaluation framework. At a second level, we present evaluation of the project’splatformand services froma technical and technological perspective in terms of performance,scalability, expandability, robustness, novelty and technological longevity.The third part isdevoted to thebusiness and financial evaluationof theOpenScienceLink services, according tothe different businessmodels for open access thatwere studied inWP9. In general this taskprovides insights for the formulationof realisticbusinessandsustainabilityplans. Asa result,the current deliverable is structured following the aforementioned rationale, presentinganalytically the results of evaluation based on the tools and methodologies (questionnaires,interviews,KeyPerformance Indicators (KPIs),measuresetc) thatweredefinedatdeliverable8.1.
Page 4 of 82
D8.2.3Stakeholders,TechnicalandFinancialEvaluation
Contents
ExecutiveSummary............................................................................................................................................................3Contents..................................................................................................................................................................................4ListofFigures........................................................................................................................................................................6ListofTables.........................................................................................................................................................................71 Introduction.................................................................................................................................................................81.1 ObjectivesofEvaluation...............................................................................................................................81.2 EvaluationMethodology...............................................................................................................................8
2 StakeholdersEvaluation.........................................................................................................................................92.1 ManagementandOverallProgressoftheProject............................................................................92.2 EvaluationoftheOverallPlatform'sPerformance........................................................................122.2.1EvaluationofthePlatformasaWhole...............................................................................................122.2.2UserSatisfaction..........................................................................................................................................132.2.3KeyPerformanceIndicators(KPI).......................................................................................................14
2.3 OpenScienceLinkPilotsEvaluation......................................................................................................182.3.1Pilot1:Open-accessDataJournalsDevelopment.........................................................................192.3.2Pilot2:Anovelopen,semantically-assistedpeerreviewprocess........................................222.3.3Pilot3:Servicesfordetectionandanalysisofresearchtrends..............................................242.3.4Pilot4:Servicesfordynamicresearchers’collaboration..........................................................262.3.5Pilot5:Researchevaluationservices.................................................................................................28
3 TechnicalEvaluation.............................................................................................................................................303.1 TechnicalEvaluation:TheTestCases..................................................................................................303.1.1TestsSubclasses...........................................................................................................................................313.1.2TestCaseIdentifiers...................................................................................................................................323.1.3TestCaseStructure.....................................................................................................................................34
3.2 BasicTests(BT).............................................................................................................................................343.2.1ValidOpenScienceLinkPlatformSetup.............................................................................................343.2.2SuccessfulConnection...............................................................................................................................34
3.3 CapabilityTests(CT)...................................................................................................................................353.3.1SecureEndUserConnection..................................................................................................................35
3.4 BehaviorResolutionTests(BER)..........................................................................................................363.4.1CommonFunctionalities..........................................................................................................................373.4.2Pilot1................................................................................................................................................................403.4.3Pilot2................................................................................................................................................................443.4.4Pilot3................................................................................................................................................................523.4.5Pilot4................................................................................................................................................................533.4.6Pilot5................................................................................................................................................................55
3.5 TechnicalEvaluationResults...................................................................................................................584 FinancialEvaluation..............................................................................................................................................604.1 OverallOSLPlatformFinancialEvaluation.......................................................................................604.1.1Theoverallopenaccesslandscape......................................................................................................604.1.2OpenAccessandBiomedicalResearch..............................................................................................61
Page 5 of 82
D8.2.3Stakeholders,TechnicalandFinancialEvaluation
4.2 Pilot1:Open-accessDatajournalsdevelopment.............................................................................624.2.1AnalysisofDifferentBusinessModelsandCompetitors..........................................................624.2.2Insightsfortheformulationofrealisticbusinessandsustainabilityplans......................67
4.3 Pilot2:Anovelopen,semantically-assistedpeerreviewprocess..........................................714.3.1Analysisofdifferentbusinessmodelsandcompetitors.............................................................714.3.2Targetedcustomersandstakeholders...............................................................................................73
4.4 Pilot3:Servicesfordetectionandanalysisofresearchtrends................................................734.4.1Analysisofdifferentbusinessmodelsandcompetitors.............................................................734.4.2Insightsfortheformulationofrealisticbusinessandsustainabilityplans......................75
4.5 Pilot4:ServicesforDynamicResearchers'Collaboration..........................................................764.5.1Analysisofdifferentbusinessmodelsandcompetitors.............................................................764.5.2Insightsfortheformulationofrealisticbusinessandsustainabilityplans......................78
4.6 Pilot5:Researchevaluationservices...................................................................................................784.6.1Analysisofdifferentbusinessmodelsandcompetitors.............................................................784.6.2Insightsfortheformulationofrealisticbusinessandsustainabilityplans......................80
5 SummaryandConclusions.................................................................................................................................816 References.................................................................................................................................................................82
Page 6 of 82
D8.2.3Stakeholders,TechnicalandFinancialEvaluation
List of Figures Figure1:Summaryofnumericquestionnaireresponsevalues..................................................................10Figure2:Summaryof themost importantOpenScienceLinkprojectachievementsaccording toquestionnaireresponses...............................................................................................................................................11Figure 3: Summary of OpenScienceLink project aspects that can be improved according toquestionnaireresponses...............................................................................................................................................11Figure4:EvaluationoftheOpenScienceLinkplatformasawhole............................................................13Figure5:Summaryofplatformuser’ssatisfaction...........................................................................................14Figure6:Openaccesspublishingbetween1993and2009..........................................................................60Figure7:DifferentPublisherTypesforOpenAccessArticles......................................................................61Figure8:OpenAccessArticlesacrossdifferentdisciplines..........................................................................62
Page 7 of 82
D8.2.3Stakeholders,TechnicalandFinancialEvaluation
List of Tables Table1:MissionPerspectiveKPIsforOverallPlatform.................................................................................15Table2:ResourcePerspectiveKPIsfortheOverallPlatform......................................................................16Table3:InternalBusinessProcessesKPIsfortheOverallPlatform.........................................................17Table4:LearningandGrowthPerspectiveKPIsfortheOverallPlatform..............................................17Table5:MissionPerspectiveKPIsforPilot1......................................................................................................21Table6:ResourcePerspectiveKPIsforPilot1...................................................................................................21Table7:InternalBusinessProcessesKPIsforPilot1......................................................................................21Table8:LearningandGrowthPerspectiveKPIsforPilot1..........................................................................22Table9:MissionPerspectiveKPIsforPilot2......................................................................................................23Table10:ResourcePerspectiveKPIsforPilot2.................................................................................................23Table11:InternalBusinessProcessesKPIsforPilot2...................................................................................23Table12:LearningandGrowthPerspectiveKPIsforPilot2........................................................................24Table13:MissionPerspectiveKPIsforPilot3....................................................................................................25Table14:ResourcePerspectiveKPIsforPilot3.................................................................................................25Table15:InternalBusinessProcessesKPIsforPilot3...................................................................................25Table16:LearningandGrowthPerspectiveKPIsforPilot3.......................................................................25Table17:MissionPerspectiveKPIsforPilot4....................................................................................................26Table18:ResourcePerspectiveKPIsforPilot4.................................................................................................27Table19:InternalBusinessProcessesKPIsforPilot4...................................................................................27Table20:LearningandGrowthPerspectiveKPIsforPilot4........................................................................28Table21:MissionPerspectiveKPIsforPilot5....................................................................................................28Table22:ResourcePerspectiveKPIsforPilot5.................................................................................................29Table23:InternalBusinessProcessesKPIsforPilot5...................................................................................29Table24:ClassesofTestsbasedontheISO9646.............................................................................................30Table25:TestsBehaviorTypes.................................................................................................................................30Table26:SubcategoriesandBehaviorTests.......................................................................................................32Table27:TheSubfieldsoftheTestCaseIdentifier...........................................................................................33Table28:TestCaseStructure.....................................................................................................................................34Table29:SummaryofTechnicalEvaluationResults.......................................................................................59Table30:EuropeanOpen-AccessFunders...........................................................................................................64Table31:Pilot1Competitors.....................................................................................................................................67Table32:SpendingsforPilot1..................................................................................................................................68Table33:PossibleRevenuesforPilot1.................................................................................................................68Table34:ExamplesofauthorfeesforpublicationinOAjournals.............................................................70Table35:Pilot2Competitorsandbusinessmodels.........................................................................................73Table36:Pilot3Competitorsandbusinessmodels.........................................................................................75Table37:Pilot4Competitorsandbusinessmodels.........................................................................................78Table38:Pilot5Competitorsandbusinessmodels.........................................................................................80
Page 8 of 82
D8.2.3Stakeholders,TechnicalandFinancialEvaluation
1 Introduction
1.1 Objectives of Evaluation This task gives emphasis on themulti-facet evaluation of the project’s results (i.e. coveringmultiple perspectives), as well as on the elicitation and documentation of best practicesassociated with open access to scientific information. The main objectives of the evaluationprocessareto:Establish an evaluation framework (including scientifically sound methodologies, KPIs andtools)forthedisciplinedevaluationoftheproject’sresults.To evaluate the project’s results with the involvement of all stakeholders of theOpenScienceLinkplatformi.e.takingintoaccountandanalyzingfeedbackfromallstakeholders.ToevaluatetheOpenScienceLinkfromatechnicalandtechnologicalperspective.Toevaluate theOpenScienceLinkresultsagainst theirbusinesspotentialand thepossibilityofrenderingfinancialreturns/benefitsforthevariousstakeholders.To elicit and document best practices and blueprints associated with the implementation,exploitationanduseofmodelsforopenaccesstoscientificinformation.OpenScienceLink will also study the business potential of open access paradigms, throughinvestigating and pursuing multiple business models including author fees, hard copy sales,advertisements,sponsorships,aswellassubscriptionbasedmodels.
1.2 Evaluation Methodology The evaluation methodology includes a set of methodological tools tools and KPIs forevaluating the project from a usability, business and technical perspective,while at the sametimeelicitingandanalyzingtheopinion/feedbackofallstakeholders.KeyPerformanceIndicatorsarearguablyanimportantinstrumentformonitoringtheproject’swork and evaluating the results. Taking into consideration four main perspectives (Missionperspective,resourceperspective,InternalBusinessprocesses,Learning&Growthperspective),we have composed an expansive list of KPIs for each of the pilots as well as the overallOpenScienceLinkplatform.InadditiontotheKPIs,thedifferentaspectsoftheOpenScienceLinkprojectareevaluatedusingquestionnaires,reviewformsandstakeholderinterviews.
Page 9 of 82
D8.2.3Stakeholders,TechnicalandFinancialEvaluation
2 Stakeholders Evaluation In order to get detailed feedback from stakeholders about their satisfaction with theperformanceoftheprojectinterviewswereconductedbyconsortiummembers.Inadditiontotheevaluationofindividualsubjectsmulti-purposemethodologiesincludedquestionnaires(forall subjects), as well as key performance indicators and relatedmeasures specifically for theOpenScienceLinkplatformanditspilots.Overall, there are three basic subjects of evaluation within OpenScienceLink. First, themanagementoftheprojectandtheoverallexecutionoftheworkplan, includingthetimelyachievementsofthepredefinedmilestones.Second,theOpenScienceLinkplatform,whichhasthemainroletoimplementthefiveprojectpilots.Third,theperformanceandtheoutcomeoftheactualOpenScienceLinkpilots.
2.1 Management and Overall Progress of the Project With regards to the evaluation of the OpenScienceLink project execution, and based on thedescription ofwork of the project (OpenScienceLink Consortium, 2013), themajor propertiesthatweremonitoredpertaintotheevaluationofthetimelycompletionandsubmissiontotheECofreportsanddeliverables,aswellastheachievementoftheproject'smilestones. Inaddition,theoverallmonitoringofthemanagementoftheproject,e.g.,financialmanagement,distributionandorganizationofwork,organizationofprojectmeetings,isalsoanimportantproperty.There are many ways to measure the performance of the OpenScienceLink services directlyusingdata that canbe collectedautomaticallywhile running theplatform (e.g., thenumberofusers).However,therearealso“softer”criteria,e.g.,relatingtothequalityoftheuserexperienceor the overall satisfactionwith the project’s progress, that can not bemeasured as easily. Inordertobeabletoalsomeasuresuchcriteria,wehavedesignedonlinequestionnairesthatcanbe submitted tousersof thedifferent functionalities inorder toprovide themwitha fast andeasyway toprovide feedbackabout theirexperiencewith theOpenScienceLinkplatform.Thisfeedback isusedby the technicalpartners forprioritizing requiredupdatesanddesigning thenextiterationoftheplatform.The firstquestionnairewasdesignedwith the intentionofmeasuring the internal satisfactionwith the project’s progress among the members of the consortium. Most questions can beansweredwithanumericalvalueonascalebetween1(“disappointing”)and5(“excellent”).Thefollowingquestionsfollowedthistemplate:
• PleaseratetheoverallprogressoftheOpenScienceLinkprojectforthethirdyear• PleaseratetheoverallmanagementoftheOpenScienceLinkprojectforthethirdyear• PleaseratethetechnicalachievementsoftheOpenScienceLinkprojectforthethirdyear• Pleaseratethequalitycontrolofthedeliverablesduringthethirdyear• Pleaseratetheupdatingprocessofthepilotspecifications,includingtheupdatesofthe
userrequirementsfromallstakeholders• Please rate the dissemination activities of the project for the third year, including the
pilotpreparationactivities.• Pleaseratetheexploitationplanandrespectiveactivitiesforthethirdyearoftheproject• PleaseratetheoperationoftheOpenScienceLinkBiomedicalDataJournal
Theresultsofthequestionnaireareshowninthefollowingchart.Thevalues(x-axis)representthe weighted average of the responses by the questionnaire participants, where a value ofdesignates the worst score, and the value of 5 the top (best) score. Overall, the internalevaluation via the questionnaire shows that within the third year the progress was verysatisfactory.
Page 10 of 82
D8.2.3Stakeholders,TechnicalandFinancialEvaluation
Figure1:Summaryofnumericquestionnaireresponsevalues.
Inaddition to thesequestions that requirednumerical responses, the following twoquestionsrequestedtextualanswersinordertoretrievein-depthfeedbackfromtheconsortiummembersaboutwhichaspectsoftheprojecttheyconsidermostandleastsuccessful:
• Please enter three aspects of the project that you thinkwere themajor achievementsduringthethirdyear
• Pleaseenterthreeaspectsoftheprojectthatyouthinkcanbedefinitelyimprovedinthefuture
Theresultsofthesetwoquestionnairesaresummarizedinthefollowingfigures.Astheresultssuggest,amongthemost importantachievements in the thirdyearof theprojecthasbeentheoperation of the Biomedical Data Journal, and the updated features of the OpenScienceLinkplatform (publicly available at: http://opensciencelink.org). In parallel, the participants to thequestionnaire suggested that the exploitation of the projects’ resultsisthe main aspect whichshould attract focus in the future, given the very important results of the project and theirimpact.
Page 11 of 82
D8.2.3Stakeholders,TechnicalandFinancialEvaluation
Figure 2: Summary of the most important OpenScienceLink project achievements according to questionnaire responses.
Figure 3: Summary of OpenScienceLink project aspects that should attract more focus in the future.
Page 12 of 82
D8.2.3Stakeholders,TechnicalandFinancialEvaluation
2.2 Evaluation of the Overall Platform's Performance AsfarastheevaluationoftheOpenScienceLinkplatformisconcerned,themainpropertiesthatweremonitoredpertaintothetechnicalaspectsoftheimplementationandperformanceoftheplatform,theevaluationoftheoveralluserexperience,thedegreeoffulfilmentofthecollecteduserandtechnicalrequirements,and,theoverallimpactoftheplatform.A questionnaire was designed with the intention of evaluating the release of theOpenScienceLinkplatform(Year3).Thecollecteddatacoveredthefollowingevaluationaspects:platformasawhole,usersatisfaction,novelty,andrecommendationsforfuturefeatures.Inthefollowingwepresenttheresultsofthisevaluation.
2.2.1 Evaluation of the Platform as a Whole For thepurposesof the evaluationof theOpenScienceLinkplatformas awhole, the followingquestionsweredistributedtothepartners:PleasegiveanoverallratingfortheOpenScienceLinkplatformasawhole. Answertype:scalebetween1(“disappointing”)and5(“excellent”)PleaseratethefunctionalityofPilot1,i.e.,theuploadofdatasetsandtheoverviewofuploadeddatasets. Answertype:scalebetween1(“disappointing”)and5(“excellent”)PleaseentertheaspectsofPilot1thatcanbeimprovedAnswertype:textPleaseratethefunctionalityofPilot2,i.e.,thecreationofreviewcalls. Answertype:scalebetween1(“disappointing”)and5(“excellent”)PleaseentertheaspectsofPilot2thatcanbeimprovedAnswertype:textPleaseratethefunctionalityofPilot3,i.e.,theanalysisoftrendsforagivenquery. Answertype:scalebetween1(“disappointing”)and5(“excellent”)PleaseentertheaspectsofPilot3thatcanbeimprovedAnswertype:textPleaseratethefunctionalityofPilot4,i.e.,thesuggestionofcollaborations. Answertype:scalebetween1(“disappointing”)and5(“excellent”)PleaseentertheaspectsofPilot4thatcanbeimprovedAnswertype:textPleaseratethefunctionalityofPilot5,i.e.,evaluationofresearchentities. Answertype:scalebetween1(“disappointing”)and5(“excellent”)PleaseentertheaspectsofPilot5thatcanbeimprovedAnswertype:textTheresultsofthisevaluationaresummarizedinthefollowingfigure.Withregardstothemostpositive aspects of the platform in its current form, the threemajor pointswere: simple andclearinterface,fastresponsetime,and,up-to-dateresults.Regardingthemostimportantaspectsof the platform that may be improved, these include: explanation or user guidance of the
Page 13 of 82
D8.2.3Stakeholders,TechnicalandFinancialEvaluation
platformfunctionalities,moredetails intheexplanationoftheresults,and,thesuggestionthattermsandconditions,especiallywithregardstotheprivacypolicy,shouldbeclarified.
Figure 4: Evaluation of the OpenScienceLink platform as a whole.
2.2.2 User Satisfaction Forthepurposesoftheevaluationofuserssatisfaction,thefollowingquestionswereincludedinquestionnaires:Howintuitiveistheuserinterfaceoftheplatforminyouropinion?Answertype:scalebetween1(“disappointing”)and5(“excellent”)Pleaseratetheplatform'sresponsetimetoyourinputAnswertype:scalebetween1(“veryslow”)and5(“veryfast”)Willyouusetheplatformagaininthefuture?Answertype:scalebetween1(“definitelynot”)and5(“definitelyyes”)Pleaseenterthethreemostpositiveaspectsoftheplatforminitscurrentform. Answertype:textPleaseenterthethreeaspectsoftheplatformthatyoufeelcanbeimproved. Answertype:textPlease enter awish list of functionalities you believe are interesting to be included in futurereleasesAnswertype:textTheresultsofthisevaluationaresummarizedinthefollowingfigure.Theevaluationtookplaceconsidering and testing again the last version of the OpenScienceLink platform. Overall, theresponsetimeoftheplatformwasevaluatedverywell,withtherestofthequestionssuggestingthattheuserinterfacewassignificantlyimprovedcomparedtothealpharelease.
Page 14 of 82
D8.2.3Stakeholders,TechnicalandFinancialEvaluation
Figure5:Summaryofplatformuser’ssatisfaction.
2.2.3Key Performance Indicators (KPI) ThissectiondescribestheKPIsthatwereusedforthispurposecomparingtheexpectedwiththeactual progress that has been achieved during the 3rdyear. Following the evaluationmethodology adopted by the OpenScienceLink project, the KPIs have been grouped based onfourperspectives:(1)Mission,(2)Resource,(3)InternalBusinessProcesses,and,(4)Learningand Growth. The KPIs for the four perspectives, the anticipated progress for year 3, and theactualprogressreported,arepresentedinthefollowingfourtablesrespectively.
KPI Measure Expected
progressYear3
ActualprogressYear3
Userparticipation Number of userregistrations in theOpenScienceLinkplatform
>=1100
73(butmorethan400unique/anonymousvisits
inthesearchtab)
Active userparticipation
Number of users usingthe system at least onceper month within a 3monthperiod
>=600 73
Stakeholderparticipation
Number of institutions(beyondtheconsortium)involved in theOpenScienceLink pilotoperations
>=30 9
Funderparticipation
Number of researchsponsorsand/or fundingauthorities (beyond theconsortium) involved inthe OpenScienceLinkpilotoperations
>=5 4
Page 15 of 82
D8.2.3Stakeholders,TechnicalandFinancialEvaluation
Table 1: Mission Perspective KPIs for Overall Platform.
KPI Measure
ExpectedprogressYear3
ActualprogressYear3
FundedprojectsbeyondOpenScienceLink
Number of projects inwhich theOpenScienceLink services,modelsandtoolsareusedandfurtherdeveloped.
>=11
Stakeholders withcommercialinterest
Numberofstakeholdershavingformallyexpressedcommercialinterest >=3
5(SpringerOpen,Elsevier,IBMResearch,Empolis,ExB)
Journalpublications NumberofOpenScienceLink-relatedscientificjournalpaperspublishedbyconsortiummembers
>=8 15(spreadintwoissues)
Internationalparticipation
Number of countriesfrom which theOpenScienceLinkPlatform has been usedatleastonce
>=30 14
Activeinternationalparticipation
Number of countrieswith activeOpenScienceLink users(i.e., using the system atleast once per monthwithina3monthperiod)
>=25 14
Research domaininvolvement
Number of Biomedicaland Clinical Researchareas (such ascardiology,pharmacology, etc.) withresearchersregisteredtothe OpenScienceLinkplatform
>=10 23
Active researchdomaininvolvement
Number of biomedicaland clinical researchareas with activeOpenScienceLinkusers
>=8 12
Openaccess Numberof indexedopenaccess journal papersanddatasets
>=2,000,000 ~3,100,000
Page 16 of 82
D8.2.3Stakeholders,TechnicalandFinancialEvaluation
Blog and websitepublications
NumberofpostsonblogsandwebsitesaboutOpenScienceLink
>=5 12
Conferencepublications andpresentations
NumberofOpenScienceLink-relatedpapers,presentationsandpresenceatconferences,workshopsandexhibitions
>=13 18
Joint workshops withother projects orrelated nationalinitiatives
Numberofworkshopsheldwithotherprojectsorrelatedinitiatives >=5 2
OpenScienceLinkwebsitevisitors
NumberofuniquevisitorstotheOpenScienceLinkwebsite >=400
41,005(Websiteandplatformtogether)
Links toOpenScienceLinkwebsite
NumberofexternalwebsitesreferringtoOpenScienceLinkwebsite
>=20 47(obtainedviaGoogle)
OpenScienceLinkpressreleases
NumberofOpenScienceLink-relatedpressreleasespublishedbyconsortiummembers
>=10 7
Project disseminationoutsideEurope
NumberofOpenScienceLink-relatedeventstowhichconsortiummembersparticipatedoutsideEurope
>=2
3(actualpresentationsgivenviatelcos)
Projectmarketing NumberofOpenScienceLink-relatedmarketingmaterialsmaterial(e.g.,leaflets,banners,posters)producedanddistributedbyconsortiummembers
>=800 250
Targetedcontactsandapproached potentialcustomers
Number of stakeholdersand potential customers(publishers, researchorganizations,universities) contacted byconsortiummembers
>=25 >40
Reached policy anddecisionmakers
Number of policy anddecisionmakerscontactedbyconsortiummembers
>=5 3
Table 2: Resource Perspective KPIs for the Overall Platform.
Page 17 of 82
D8.2.3Stakeholders,TechnicalandFinancialEvaluation
KPI Measure
ExpectedprogressYear3
ActualprogressYear3
Increase in data papercreation
Increase in number ofdatapapers createdperresearcher >=20% ~15%
Creation of BestPractices (BPs) andBlueprints
Numberof distinctBestPractices (BPs) andBlueprintsProduced
>=8 >20
Table 3: Internal Business Processes KPIs for the Overall Platform.
KPI Measure
ExpectedprogressYear3
ActualprogressYear3
Involvement of youngresearchersintheproject
Number of PhDstudents and Postdocsparticipating in theproject
>=10 12
Career advancement oftheOSLteam
NumberofreceivedPhDdegreesandrelevanthabilitations
2 3
Table 4: Learning and Growth Perspective KPIs for the Overall Platform.
Page 18 of 82
D8.2.3Stakeholders,TechnicalandFinancialEvaluation
2.3 OpenScienceLink Pilots Evaluation The evaluation of the OpenScienceLink pilots, the degree of fulfilment of the respectiverequirementsandtheoverallimpactareveryimportant,inadditiontotheconsiderationofthedegree of satisfaction of the engaged stakeholders and interested parties. Besides that, themethodology that has been used to accomplish the pilots' results has to be evaluated bycomparingittoexistingstateoftheartmethodologieswhenapplicable.
Afterinterviewswithstakeholders,theevaluationannotatesthefollowing:
1. Pilots1,2and5haveattractedthemostattention,andtherearestakeholdersinterestedatthisstagetoadoptsomeoftheirprocesses.
2. Pilot 3 has great potential. The trend detectionmechanismwas characterized as veryuseful.
3. All Pilots combined provide an ecosystem that offers open access to scientificinformation,fromtheinitialstageofdatasetandpaperpreparation,toaccess,re-usageandexploitation.
4. TheOSLplatformprocessespersonal,end-userdata,suchastheend-users’fullname,e-mailaddress,general fieldsof interestandemploymentdetails, for theperformanceofitspilotservices.Thedataareforinstanceusedtopersonalizetheend-user’sexperience.In order to facilitate research collaboration and evaluation, end-user data is alsorequired.Although theOSLplatform’sprivacypolicy is inaccordancewith thecurrentEuropeandataprotectionframework,futuredataprocessingactivitieswillhavetotakeinto account the upcoming data protection regulation (once it has been adopted andimplementedby theEuropean legislator).Theaddendum toD3.2already includesdataprotection compliance guidelinesunder the general data protection regulation.1Inaddition,anonymizationremainskey.Personaldatasetsshouldnotbeuploadedontotheplatform.Asdatasetsaremadeavailablefordownloadaftertheyhavebeenreviewedbyusers of the platform, the guidelines for reviewers should also clearly state that theuploadingofpersonaldataisforbidden.Indeed,atthemoment,theplatform’sreviewers,aswellastheresearchersthemselves,areinthebestpositiontodetectwhetherornotanuploadeddatasetcontainspersonaldata. If theywouldhaveanydoubtsconcerningtheanonymizednatureofdatasets,thepublicationoftherevieweddatasetshouldnotbeallowed.Furthermore,compromiseddatasetsshouldbepulledoftheplatform.
5. With regard to data mining activities performed by the OSL platform, future legaldevelopments should also be taken into account. In a recent communication, theEuropean Commission has recognized that the lack of a clear text- and data miningprovisionharmstheEU’sscientificleadership.Legislativeproposalsarenowconsideredthatwillallowpublicinterestresearchorganizationstocarryouttextanddataminingofcontent they have lawful access to, with full legal certainty, for scientific researchpurposes.SuchanexceptioncouldbenefitthefuturedevelopmentoftheOSLplatform.2
6. Finally,theOSLplatformshouldrespecttheintellectualpropertyrightsofthird-parties.
Therefore, and where necessary, proper authorisation should be obtained from the 1OSL,LegalandIPRManagementFrameworkSpecification(January2016).2European Commission, Communication towards a modern, more European copyright framework’COM(2015) 626 final, p. 7-8; available at:http://ec.europa.eu/newsroom/dae/document.cfm?action=display&doc_id=12524
Page 19 of 82
D8.2.3Stakeholders,TechnicalandFinancialEvaluation
relevantrightholderswhenperformingserviceswhichmightinfringeupontheirrights.This isparticularly thecasewhere third-party-contentwillbere-used.For instance, inthe caseof ‘linking, harvesting, text anddatamining’. Forpilot servicesmakinguseofsuchtechniques,properauthorizationshouldbesought.
7. It should beexaminedhowuseful couldbe the collectionof amoredetailedpersonalprofile of the user beyond name, surname, e-mail and institutional information.Howeveritshouldbetakenintoaccountthatthemorepersonalinformationiscollected,themorelaborioustheplatformwillbefortheusers.Theadditionalinformationcouldbeoptionalfortheuseraftertheregistration.Thisadditionalinformationmayinclude:
• Nameofuserasitappearsinpublications• SummaryofCV(inastandardizedformatsuchastheEuropeanformat)• asectordedicatedtothemainresearchfield• Fundingduringthelast5years(helpsdetectconflictsofinterest)• Website• ORCHIDID,anonlineidentificationservicecreatedspecificallytode-conflictother
authorIDschemes• Itwouldbeusefuliftheplatformcouldautomaticallyretrievelistofpublicationsfrom
Pubmed
InthefollowingtablestheKPIsperpilotarepresented.
2.3.1 Pilot 1: Open-access Data Journals Development Theevaluationannotatesthefollowing:
1. Inthefuture, itwillremainimportantto informtheplatformend-users,researchers inparticular, on the relevant legal framework concerning the open access to scientificinformation. In this regard, legal guidelines have been formulated as part of WP8 toinformplatformend-usersonrelevantmattersconcerningIPRanddataprotection.
2. Uploadedopen-accessdata-setsmightcontainanonymizeddata,i.e.datasetspertainingto previously identifiable individuals. License-holders should be prohibited from re-identifyinganyindividualandfromusingthedatatotakeanymeasureordecisionwithregardtothere-identifiedindividuals.Upondetectingthatadatasetiscompromised,i.e.has become re-identifiable, the license-holder (or any other party) should notify thelicensor.Uponnotification,theplatformshouldsuspendorterminatetheaccessibilitytothedata,forinstancebyremovingthefilefromtheplatform.Reasonableeffortsshouldalso be made to limit the negative effects of compromised datasets, for instance bydeletingallorpartsofthecompromiseddatasets.Thisshouldalsoincludeaprominentnoticeontheplatformandwebsiteaccessedbygroupsorindividualswhoarelikelytobere-usingthedata.
3. Thelastyearoftheprojectaveryimportantupdateofpilot1hasbeenaddedconcerningtheuploadingofnewfilesafterreviewingofdatapapersandtheabilityofeditorsandreviewers to access the new revised files but also the original ones and makecomparisons
Page 20 of 82
D8.2.3Stakeholders,TechnicalandFinancialEvaluation
4. Afuturedevelopmentshouldfocusonthecreationofmoredetailedmetadatainordertoachievebetter-structuredandsemantically linkeddatasets.Tothisend,thefollowingcharacteristicsofadatasetcouldbeimportant:• themethodologyused to collect data (e.g. echocardiographic evaluation, Cardiac
magnetic resonance, Exercise testing, Bloodmeasurements etc) and the variablesmeasured in each case (e.g. for echocardiographic evaluation, Variable 1: Leftventricularinternaldiameteratdiastole(LVIDd),Variable2:Leftventricularinternaldiameter at systole (LVIDs), Variable 3: Ejection fraction (EF%) etc) could beretrievedfromthesubmitteddatasets
• Subjectsusedtocollectthisdataset(humans,animals,cells,speciesetc.)• Ageofsubjectsused• Genderofsubjectsused
3Thusprovidingforgettinganimpactfactorinthefuture.
KPI Measure ExpectedprogressYear3
ActualprogressYear3
Usage of the dataarticles
Number of views permonth
2,000 4,093
Number of downloadspermonth
150 >400
Citations 10 6Usage of relateddatasets
Number of downloadspermonth
20 17
Usage by themedia
Number of articles inpopular media, includingspecializedblogs
3 2
Contributions Submitted articles perquarter 15 6
Poolofreviewers Number of researchersthat have expressed aninterest/consent
20 12
Interestedfunders
Number of ResearchSponsors and/ orFunding Authoritiesregistered at the OSLplatform
2 4
Geographiccoverage
Number of countriesrepresented on EditorialBoard, authors orreviewers
10 >20
Domaincoverage Number of biomedicaland clinical researchareas addressed bypublisheddataarticles
10 ~30
Coverage byaggregators
Inclusion in indexing(including ISI3) andaggregatorservices
3 4
Page 21 of 82
D8.2.3Stakeholders,TechnicalandFinancialEvaluation
Table 5: Mission Perspective KPIs for Pilot 1. KPI Measure Expected
progressYear3
ActualprogressYear3
Revenuesfromsubscriptionforthehardcopy
Number of paidsubscriptions 40 -
Sponsorship Numberof fundingagencieseither directly sponsoringthe journal or agreeing tosponsor the publication ofindividualcontributions
4 -
Advertising Valueofadvertisementsinthejournaloronthepublisher’swebsiteinEuro
800 -
Table 6: Resource Perspective KPIs for Pilot 1.
KPI Measure Expected
progressYear3
ActualprogressYear3
Speed of thepublication process(including peerreview)
Average time fromsubmissionofapapertoits online publication, ifaccepted, in calendarday
50 65
Table 7: Internal Business Processes KPIs for Pilot 1.
Libraryusage Inclusion in librarycatalogues 20 3
Page 22 of 82
D8.2.3Stakeholders,TechnicalandFinancialEvaluation
KPI Measure Expected
progressYear3
ActualprogressYear3
Coveragewithinthebiomedicalfield
Specialissuesandcallsforpapersforsuchissues,coveringfieldsrepresentedontheEditorialBoard
4 2
Adaptation to thechanging researchlandscape
Number of identifiednew trends andaddition of respectivemembers to theEditorialBoard
2 3
Expanding theapplication of the openaccess data journalparadigm
Numberof concepts fornew open access datajournalsdeveloped 1 1
Table 8: Learning and Growth Perspective KPIs for Pilot 1.
2.3.2 Pilot 2: A novel open, semantically-assisted peer review process Theevaluationannotatesthefollowing:
1. The platform contains 2 different functionalities for the selection of reviewers. Onefunctionality creates a list of suggested reviewers based on the metadata and theabstract of each submission. The 2nd functionality creates a list of potential reviewersbasedonthekeywordsthathavebeenselectedbytheeditor.Bothareveryuseful.
2. During the last year the platform has added the functionality of evaluating reviewers.This tool could be very helpful in order to avoid the high rate of decline of invitedreviewers.Evaluationisbasedatthemomentinascoreassignedbytheeditor.Itwouldbe very helpful if the platform also included some statistics about the performance ofreviewerssuchasnumberofpending reviews, rateof acceptanceof review invitations,meantimetocompletereviewetc.
3. Theideaofaopenreviewingprocesswouldbeofaddedvaluetotheplatformandshouldbeconsideredforfuturedevelopment.Forthisreason,asocialnetworkcouldbecreatedenabling reviewers to discuss and express opinions on the reviews and the reviewresults. Inside this network, a reviewerwill be able to see the comments of the otherreviewersandvoteinfavorornot.
4. Duringthelastyeartheplatformhasprovidedtheoptiontoreviewerstowriteashortcommentonthepublishedpaper.
Page 23 of 82
D8.2.3Stakeholders,TechnicalandFinancialEvaluation
Table 9: Mission Perspective KPIs for Pilot 2.
KPI Measure Expected
progressYear3
ActualprogressYear3
Speed of the reviewprocess
Controlgroup>30% -
Table 10: Resource Perspective KPIs for Pilot 2. KPI Measure
ExpectedprogressYear3
ActualprogressYear3
Attraction of morecompetentreviewers
Questionnaireresponses 100% -
Increased number ofreviewersperreview
Controlgroup100% -
Table 11: Internal Business Processes KPIs for Pilot 2.
KPI Measure Expected
progressYear3
ActualprogressYear3
Reviewrequestload Number of reviewrequestssent 400 ~80
Revieweracceptance Number of acceptedreviewrequests 300 39
Submittedreviewsload Number of submittedreviews 260 39
Success of reviewerrecommendations
Number of reviewinvitations based onplatformsuggestions
380 ~80
Usagebyexternalvenues Number of venues forwhich reviews wererequested
2 -
NumberofAuthorships Numberofauthorswhosepublications were put upforreview(cumulative)
240 36
Page 24 of 82
D8.2.3Stakeholders,TechnicalandFinancialEvaluation
KPI Measure
ExpectedprogressYear3
ActualprogressYear3
Profilegrowth Number of newplatform profilescreated by invitedreviewers
120 ~15
Table 12: Learning and Growth Perspective KPIs for Pilot 2.
2.3.2 Pilot 3: Services for detection and analysis of research trends
Theevaluationannotatesthefollowing:
1. During the lastyear, theplatformhasupdated trendsdetection thathasbecomemorefinegrained.
2. For future development of trends detection, important information about researchinterestcouldbecollectedfromsocialmediaandblogsandnotonlyfromanalysisofthenumberofpublications.
3. The provision of services for the detection and analysis of research trends maynecessitate the re-use of third party content, e.g. through linking, harvesting, text-anddataminingtechniques.TheOSLplatformshouldcontinuetoascertainthelicensesthatrestuponthedatabasestheywishtomine.AsmentionedinD3.2.OpenAccesslicenseslackuniformityandtheymayincluderestrictionsontheuseofthecontenttowhichtheypertain. If the license is unclear or ambiguous, data mining activities should not beperformedwithouthavingobtainedproperauthorisation.Hence,itisadvisabletoseekalicense for the application of text and data mining tools to third party databases.Licensingconditionsshouldalsobecarefullyexamined:whichrestrictionsareimposedby the relevant right holders on the free exchange of information. If the contents ofresearchpapersaretobeanalysedaswell,licensesshouldnotonlybeobtainedforthere-utilization of the data base, but also for the contents of the database. Licensesobtained must also be compatible with the license OSL wishes to grant its users.
Page 25 of 82
D8.2.3Stakeholders,TechnicalandFinancialEvaluation
Table 13: Mission Perspective KPIs for Pilot 3. KPI Measure
ExpectedprogressYear3
ActualprogressYear3
Accelerationoftrendidentification
Questionnaire (percentageof users that identifiedtrendsfaster)
65% -
Table 14: Resource Perspective KPIs for Pilot 3.
KPI Measure
ExpectedprogressYear3
ActualprogressYear3
Trend detection F-Measure
F-Measure of hot topicspredicted correctly bythe platform for thefollowingyear
>=30% ~45%
Table 15: Internal Business Processes KPIs for Pilot 3.
KPI Measure
ExpectedprogressYear3
ActualprogressYear3
Expansion of BiomedicalData Journal throughTrendDetection
Number of specialissues for BiomedicalData Journal based ondetected“hottopics”
2 3
Table 16: Learning and Growth Perspective KPIs for Pilot 3.
KPI Measure Expected
progressYear3
ActualprogressYear3
Trendsearches Number of executedtrendsearches 18,000 >3,000
Page 26 of 82
D8.2.3Stakeholders,TechnicalandFinancialEvaluation
2.3.4 Pilot 4: Services for dynamic researchers’ collaboration Theevaluationannotatesthefollowing:
1. Futuredevelopmentofthispilotshouldfocusonexpandingthefunctionalitytobeableto suggest potential new collaborations. A crucial factor to consider in potentialcollaborations is the complementary methodological expertise that could be derivedfrompublicationsorfromuserprofiles.
2. Theprovisionofservicesfordynamicresearchers’collaborationrequirestheprocessingofpersonaldata.Inparticular,dataoftheOSLplatform’send-userswillbeprocessedinordertofacilitatethecollaborationamongresearchers.TheOSLserviceshavebeenbuiltwith respect of the current European data protection framework. Although at themomenttheOSLplatform’sprivacypolicyisstillvalid,withregardtofutureprocessingactivities, itwill be necessary to take into account the newData Protection regulationonceithasbeenadoptedbytheEuropeanlegislator.Thelegalevaluationrequirementsprovided in D8.1 have been altered and added as an addendum to D3.2, taking intoaccountthechangestothedataprotectionframework.
Table 17: Mission Perspective KPIs for Pilot 4.
KPI Measure
ExpectedprogressYear3
ActualprogressYear3
Number of fundedprojects (beyondOpenScienceLink)
Numberofprojectsinwhichthe pilot models, servicesand tools are used andfurtherdeveloped
>=2 2
Number of InterestedStakeholders
Number of Stakeholders towhom the pilot has beenpresented and who haveexpressedinterest
>=2 3
KPI Measure Expected
progressYear3
ActualprogressYear3
UserAttraction
Number of Researchers/Scholarshavingusedthepilot services at leastonce
>=1,000 ~70
UserEngagement Number of Researchers/Scholarshavingusedthepilot services more thanonce
>=800 ~50
Page 27 of 82
D8.2.3Stakeholders,TechnicalandFinancialEvaluation
Pilot Visibility:Publications
Number of pilot-relatedscientific papers publishedat journals, conferencesandworkshops
>=5 2
Pilot Visibility: WebPresence
Number of posts at blogsand web sites about thepilot
>=5 7
Table 18: Resource Perspective KPIs for Pilot 4.
KPI Measure
ExpectedprogressYear3
ActualprogressYear3
Correctness ofimplicitly identifiedrelationships amongresearchers
Percentage ofrecommendationswhich are relevant tothe expert’stopic/domain and arenot part of his existingcollaborations
>=85% -
Correctness ofimplicitly identifiedrelationships betweenresearchers andresearchgroups
Percentage ofrecommendationswhich are relevant tothe expert’stopic/domain and arenot part of his existingcollaborations
>=80% -
Table 19: Internal Business Processes KPIs for Pilot 4.
KPI Measure
ExpectedprogressYear3
ActualprogressYear3
Coverage within thebiomedicalfield
Number of distinctivebiomedicalandclinicalresearch areas (suchas cardiology,pharmacology, etc) inwhich researchersmay be able to findcollaborations
>=20 >30
Page 28 of 82
D8.2.3Stakeholders,TechnicalandFinancialEvaluation
Table 20: Learning and Growth Perspective KPIs for Pilot 4.
2.3.5 Pilot 5: Research evaluation services
Theevaluationannotatesthefollowing:
1. TheOpenScoreevaluationmetrichasbeenrefinedandimprovedbasedonanupdatedformula.
2. TheOpenScoreiscalculatedonlyforauthorsinthecurrentversionbutitwouldniceforfuturedevelopmenttoexpandopenscoreinjournalsandinstitutions
3. Theproblemofdisambiguationofauthornameshasbeenpartlysolved.However,thereisplaceforfurtherimprovementsinthefutureinthiscrucialarea.
4. Theevaluationof researchmightnecessitate theprocessingof a researcher’spersonaldata.Asindicatedunder2.3.5allfuturedataprocessingactivitieswillhavetotakeintoaccountthefuturedataprotectionregulation
Table 21: Mission Perspective KPIs for Pilot 5. KPI Measure
ExpectedprogressYear3
ActualprogressYear3
ExpressedInterestofStakeholders
Number of Stakeholders(e.g., publishers) to whomthe pilot has beenpresented and who haveexpressedinterest(throughdiscussions, letter ofsupport,etc)
>=2 2
KPI Measure Expected
progressYear3
ActualprogressYear3
Stakeholders’Interest
Number of stakeholders(e.g., publishers)indicating interest in thedeveloped evaluationmetrics
>=3 3
MetricsUse
Numberofentitieswhichhave introduced at leastone of the evaluationmetrics to theirevaluationprocess
>=2 -
Page 29 of 82
D8.2.3Stakeholders,TechnicalandFinancialEvaluation
Pilot Visibility:Publications
Number of pilot-relatedscientific papers publishedatjournals,conferencesandworkshops
>=3 2
Pilot Visibility: WebPresence
Numberofpostsatblogsandwebsitesaboutthepilot
>=3 7
Table 22: Resource Perspective KPIs for Pilot 5.
KPI Measure
ExpectedprogressYear3
ActualprogressYear3
Acceptability ofresearch evaluationmetrics
Percentage of expertsindicating that thedeveloped metrics areof value for theresearchcommunity
>=80% >90%
Improvement ofresearch evaluationmetrics: expertsopinion
Percentage by whichthe research evaluationmetrics are consideredimprovement ofexistingones,suchasg-index,impactfactor,etc
>=20% >70%
Improvement ofresearch evaluationmetrics:comparison
Percentage of pastresearch papers forwhich existingevaluation metricsrequiredat least1yearmorethantheproposedmetricstoindicatetheirimportanceinthefield
>=10% -
Table 23: Internal Business Processes KPIs for Pilot 5.
Page 30 of 82
D8.2.3Stakeholders,TechnicalandFinancialEvaluation
3 Technical Evaluation Inthissection,thetechnicalevaluationofthedeployedOpenScienceLinkplatformisconducted.ThetechnicaltestingoftheplatformhasbeenconductedaccordingtotheISOTestingStandardISO9646. In the following sections, the three classesof tests aredefined, and the actual testswiththerespectiveresultsarepresented.
3.1 Technical Evaluation: The Test Cases TheISO9646definesthreeclassesoftestsasshowninthefollowingtable.
TestsClasses Description
BasicTests(BT)
A small number of tests (1 or 2) that are run to actuallydemonstratethattheplatformhasbeencorrectlyset-up.
ItisinfeasibletoperformothertestsiftheBasicTestsfail.
CapabilityTests(CT)
A moderate number of tests, the purpose of which is todemonstrate the ability of the platform under test to performbasicsetsoffunctions.
BehaviourResolutionTests(BER)
Alargenumberofteststhataimtoverifythattheplatformundertest behaves correctly under a wide range of situations,determinedbyvariousfactors.
Table 24: Classes of Tests based on the ISO 9646.
TheclassificationmethodologyofISO9646isuseful,especiallytowardschoosingafairnumberoftestsofeachoneoftheaboveclasses.
Alltestsfallintwosub-categories:TestsBehaviourTypes Description
Successful/ValidTests(V)
As the name suggests, these tests verify the correctbehaviour of the system under test under specificconditions.
Unsuccessful/InvalidTests(I)
These tests verify that the system under test acts “as itshould” under circumstances such as unexpected events,parameters out of range, invalid or unsupported options,negotiation failures and invalid combinations ofparameters.
Table 25: Tests Behavior Types.
Page 31 of 82
D8.2.3Stakeholders,TechnicalandFinancialEvaluation
InthissectionwewillpresentthetestscasesthatwereappliedattheOpenScienceLinkplatform.Initially we will define some broader categories for the tests cases. Also we will present thestructureofeachtestcasedefiningwhatweshouldmonitorabouteachone.
3.1.1 Tests Subclasses Inthefollowingtabletherearesomesubclassesofthe“main”testclassesseenintheprevioussection.Inthefirstcolumnthereisthetestclassinwhicheachsubclassbelongsto,whileinthelast column we can see the types of test (Behaviour Type) that will be performed. Thesesubclassesdepictthefunctionalitythatwillbetested.Foreachoneafewtestswillbeperformed.Thesetestswillbeanalyzedinthefollowingsections.
TestsClassName
ID
TestsSubclassName
IDBehaviourType
BasicTests BT
OpenScienceLinkPlatformSetup OPS V
Connectionscheck CC V
CapabilityTests
CT
SecureEnduserConnection SEC V
BehaviorResolutionTests
BER
CF:Login CF-L V-I
CF:Registration CF-R V-I
CF:Searching CF-S V-I
Pilot1:UploadDataSet P1-UDS V-I
Pilot1:CreateNewJournalIssue P1-NJI V-I
Pilot1:DatasetPublication P1-DSF V
Pilot2:CreateReviewCall P2-CRC V-I
Pilot2:UploadFile(s)ForReview P2-UFR V-I
Pilot2:GetReviewerSuggestion P2-GRS V-I
Pilot2:SelectandInviteReviewers P2-SIR V-I
Pilot2:ManagingtheReviewProcess P2-MRP V-I
Pilot2:ReviewSubmission P2-RS V-I
Pilot2:ReviewerRating P2-RR V
Page 32 of 82
D8.2.3Stakeholders,TechnicalandFinancialEvaluation
Pilot 3: Request for Trend AnalysisbasedonTopic
P3-RTA V-I
Pilot 4: Request for CoauthorshipGraphs
P4-RFC V-I
Pilot 5: Request for PublicationVolumebasedonTopic
P5-RPT V-I
Pilot5:RequestforAuthorEvaluationScore
P5-RAE V
Table 26: Subcategories and Behavior Tests. Notice:IntheabovetableweshouldnoticethatwehavespecifiedonlySuccessful/ValidTests(V)forthesubclassesoftheBasicTestsandtheCapabilityTests.
3.1.2 Test Case Identifiers Theidentifierofeachtestcasewillhavethefollowingformat:
<System_ID>_<Tests_Class_ID>_<Subclass_ID>_<Behaviour_Type_ID>_<nn>
Inthefollowingtablethereisashortdescriptionabouteachfieldoftheidentifier.
Field DescriptionSystemID OpenScienceLinkTestsClassID Basic Tests (BT) or Capability Tests (CT) or Behavior Resolution
Tests(BER)SubclassID Anyclassfromthesubclassesspecifiedin
TestsClassName
ID
TestsSubclassName
IDBehaviourType
BasicTests BT
OpenScienceLinkPlatformSetup
OPS V
Connectionscheck CC V
CapabilityTests
CT
Secure End userConnection
SEC V
BehaviorResolutionTests
BER
CF:Login CF-L V-I
CF:Registration CF-R V-I
CF:Searching CF-S V-I
Page 33 of 82
D8.2.3Stakeholders,TechnicalandFinancialEvaluation
Pilot1:UploadDataSet
P1-UDS
V-I
Pilot 1: CreateNewJournalIssue
P1-NJI
V-I
Pilot 1: DatasetPublication
P1-DSF
V
Pilot 2: CreateReviewCall
P2-CRC
V-I
Pilot 2: UploadFile(s)ForReview
P2-UFR
V-I
Pilot 2: GetReviewerSuggestion
P2-GRS
V-I
Pilot 2: Select andInviteReviewers
P2-SIR
V-I
Pilot 2: ManagingtheReviewProcess
P2-MRP
V-I
Pilot 2: ReviewSubmission
P2-RS
V-I
Pilot 2: ReviewerRating
P2-RR
V
Pilot 3: Request forTrend AnalysisbasedonTopic
P3-RTA
V-I
Pilot 4: Request forCoauthorshipGraphs
P4-RFC
V-I
Pilot 5: Request forPublication VolumebasedonTopic
P5-RPT
V-I
Pilot 5: Request forAuthor EvaluationScore
P5-RAE
V
Table26BehaviourTypeID Successful/Valid(V)orUnsuccessful/Invalid(I)Nn SequentialNumber(01to99)
Table 27: The Subfields of the Test Case Identifier.
Theidentifierdescribedisuniqueandcanprovideuswithinformationaboutthetestsclass,thesubclassandthetypeinwhicheachtestcasebelongsto.
Page 34 of 82
D8.2.3Stakeholders,TechnicalandFinancialEvaluation
3.1.3 Test Case Structure Eachtestcaseshouldcontainallthefieldsmentionedinthetablebelow:
Field DescriptionID Auniqueidentifierforthetestcase.Title Thetitleofthetestcase.Description Abriefdescriptionofthetestcase.Setup Theprerequisitesinordertohaveaccessinthistestcase.TestingProcedure The testing procedure that will be used in this test case. Here we
should answer in the question: “How we will examine that thesystemprovidesthefunctionalitydescribed”.
ExpectedResult Describetheexpectedresultofthetestingprocedure.Result (Success/Failure)
Table 28: Test Case Structure.
3.2 Basic Tests (BT) Two basic tests have been specified. Their purpose is to ensure that the platform is up andrunning and a connection can be established between the application components of theplatform.
3.2.1 Valid OpenScienceLink Platform Setup
TestCaseID OpenScienceLink_BT_OPS_V_01
Test CaseTitle
ValidOpenScienceLinkPlatformSetup
Test CaseDescription
Ensure that all OpenScienceLink components and services are up andrunning.
SetupThehardware,softwareandapplicationsmustbeidentifiedforeverypilot.Administratorsmustbeabletostartserversandruntheirapplications.
TestingProcedure
The administrators start their application servers and ensure that allnecessarycomponents (hardwareandservers)areupandrunning tostarttheirapplications.
ExpectedResult
AllServersApplicationcomponentsareshownasrunning.
TestResult (Success/Failure)
3.2.2 Successful Connection
TestCaseID OpenScienceLink_BT_CC_V_01
Page 35 of 82
D8.2.3Stakeholders,TechnicalandFinancialEvaluation
Test CaseTitle
SuccessfulConnection
Test CaseDescription
Verify the ability of browsers to establish a connection with allOpenScienceLinkentities.
SetupInternal testing of the connections to the services and resources ismadeoncearequesttotheplatformURLismade.
TestingProcedure
Users visit the OpenScienceLink platform URL, and the connection to theservicesURLs,databasesandresourcesisteted.
ExpectedResult
Allconnectionsshownaetablished.
TestResult (Success/Failure)
3.3 Capability Tests (CT) Onecapabilitytesthasbeenspecified.TheirpurposeistoensurethatasecureconnectioncanbeestablishedbetweentheenduserandtheOpenScienceLinkplatformandalsobetweentheapplicationcomponentsandservices.
3.3.1 Secure End User Connection TestCaseID OpenScienceLink_CT_SEC_V_01
Test CaseTitle
SuccessfulSecureEnduserConnection
Test CaseDescription
Verifythesecureconnectionofthebrowserswiththeplatformservers.
SetupBrowsers connecting to the platform are tested on whether they canestablishasecureconnectionwiththeplatformcomponents.
TestingProcedure
User attempt to connect (visit) the platform URL and the connection istestedonthetermsthatitissecureandthecontent(messages)exchangedareencrypted.
ExpectedResult
Avalidsecureconnectioncanbeestablishedbetweenclients(browsers)andtheplatformservers.
TestResult (Success/Failure)
Page 36 of 82
D8.2.3Stakeholders,TechnicalandFinancialEvaluation
3.4 Behavior Resolution Tests (BER) AlargenumberofBehaviourResolutionTestshavebeenspecifiedinordertoverifythecorrectbehaviouroftheplatform.Thesetestsareclassifiedbasedonthesubclassesseenin
TestsClassName
ID
TestsSubclassName
IDBehaviourType
BasicTests BT
OpenScienceLinkPlatformSetup OPS V
Connectionscheck CC V
CapabilityTests
CT
SecureEnduserConnection SEC V
BehaviorResolutionTests
BER
CF:Login CF-L V-I
CF:Registration CF-R V-I
CF:Searching CF-S V-I
Pilot1:UploadDataSet P1-UDS V-I
Pilot1:CreateNewJournalIssue P1-NJI V-I
Pilot1:DatasetPublication P1-DSF V
Pilot2:CreateReviewCall P2-CRC V-I
Pilot2:UploadFile(s)ForReview P2-UFR V-I
Pilot2:GetReviewerSuggestion P2-GRS V-I
Pilot2:SelectandInviteReviewers P2-SIR V-I
Pilot2:ManagingtheReviewProcess P2-MRP V-I
Pilot2:ReviewSubmission P2-RS V-I
Pilot2:ReviewerRating P2-RR V
Pilot 3: Request for Trend AnalysisbasedonTopic
P3-RTA V-I
Pilot 4: Request for CoauthorshipGraphs
P4-RFC V-I
Pilot 5: Request for PublicationVolumebasedonTopic
P5-RPT V-I
Pilot5:RequestforAuthorEvaluationScore
P5-RAE V
Page 37 of 82
D8.2.3Stakeholders,TechnicalandFinancialEvaluation
Table26.
3.4.1 Common Functionalities
3.4.1.1 CF:Login
3.4.1.1.1 SuccessfulLoginTestCaseID OpenScienceLink_BER_CF-L_V_01
Test CaseTitle
SuccessfulLogin
Test CaseDescription
Verifythecorrectoperationoftheauthenticationmechanismandevaluateitsusability
Setup AccesstotherespectiveGUIisrequired
TestingProcedure
Theuserentersavalidusernameandpasswordon the login form (eitherClientApplicationorWebarea)andpressesthe“Login”button
ExpectedResult
Upon successful login, the system shows the main window of the ClientApplication (for normal users), or the main page of the OpenScienceLinkwebarea(foradministratortesters).
TestResult (Success/Failure)
3.4.1.1.2 UnsuccessfulLogin–MissingDataTestCaseID OpenScienceLink_BER_CF-L_I_02
Test CaseTitle
UnsuccessfulLogin–MissingData
Test CaseDescription
Identifywhethertheattemptedlogincorrespondstoaregistereduser.
SetupAuser via abrowser attempts to login to theplatform. Theuserprovidesande-mailthatdoesnotexist,isnotregistered,ornoe-mail.
TestingProcedure
Theusertriestologinwithe-mailaddressthatisunknowntotheplatform,orwithemptycredentials.
ExpectedResult
Unsuccessfulloginwithanerrormessageinformingtheuser.
Page 38 of 82
D8.2.3Stakeholders,TechnicalandFinancialEvaluation
TestResult (Success/Failure)
3.4.1.1.3 UnsuccessfulLogin–InvalidCredentialsTestCaseID OpenScienceLink_BER_CF-L_I_03
Test CaseTitle
UnsuccessfulLogin–InvalidCredentials
Test CaseDescription
Verify the correct operation of the authentication and authorizationmechanismandevaluateitsusability.
Setup AccesstotherespectiveGUIisrequired.
TestingProcedure
The user enters an invalid user name and/or password on the login pageandpressesthe“login”button.
ExpectedResult
The system returnsanerrormessagealongwith the fields to re-enter thecredentials.
TestResult (Success/Failure)
3.4.1.2 CF:Registration
3.4.1.2.1 ProfileCreationTestCaseID OpenScienceLink_BER_CF-R_V_01
Test CaseTitle
RegistrationandCreationofNewProfile
Test CaseDescription
Theuserregistersfirsttimeattheplatform,andcreatesaprofile.
Setup Creationofanewuseraccountandofauserprofileintheplatform.
TestingProcedure
Theuser followsthe instructionsandrespective linkstocreateanewuseraccountandprofilewithintheOSLplatform.
ExpectedResult
Theuserreceivesaconfirmatione-mailthathisaccounthasbeencreated,and he should be able to use the credentials to login again, and see hissavedprofiledetails.
TestResult (Success/Failure)
Page 39 of 82
D8.2.3Stakeholders,TechnicalandFinancialEvaluation
3.4.1.2.2 ProfileEditingandSavingTestCaseID OpenScienceLink_BER_CF-R_V_02
Test CaseTitle
EditingandSavingofanExistingUserProfile
Test CaseDescription
Verify the correct operation of the editing and saving of an existing userprofile.
SetupUsing an existing account, the ability to edit and save the changes to theexistingaccountistested.
TestingProcedure
The user logs in to the platform with his existing account credentials,browsestohisprofiletab,andalterstheinformation.Oncethechangearesaved, thenew informationshouldbepartofhisexistingprofile, includingallchangesmade.
ExpectedResult
The saved changes in theprofile are storedandvisible in any future loginattempt.Thestoredprofileisnowthenewprofileoftheexistinguser.
TestResult (Success/Failure)
3.4.1.3 CF:Searching
3.4.1.3.1 SuccessfulSearchforDatasetsandCitationsTestCaseID OpenScienceLink_BER_CF-S_V_01
Test CaseTitle
SearchingforDatasetsandCitationswithaValidQuery
Test CaseDescription
Verifythecorrectoperationofthesearchingofdatasetsandcitationswithaquerythatisexpectedtoreturnaresultset(non-emptyset).
SetupUsing the searchmechanismof the platform to retrieve relevant datasetsandcitationsusingavalidquery.
TestingProcedure
TheuserbrowsestothesearchtaboftheOSLplatformandsubmitsavalidquerythatisexpectedtofetchsomedatasetand/orcitationresults.
ExpectedResult
The results matching the query are fetched from the platform databasesandindexesandarepresentedtotheuser.
TestResult (Success/Failure)
Page 40 of 82
D8.2.3Stakeholders,TechnicalandFinancialEvaluation
3.4.1.3.2 UnsuccessfulSearchforDatasetsandCitationsTestCaseID OpenScienceLink_BER_CF-S_I_02
Test CaseTitle
SearchingforDatasetsandCitationswithanInvalidQuery
Test CaseDescription
Verifythecorrectoperationofthesearchingofdatasetsandcitationswithaquerythatisexpectedtonotreturnanyresults(emptyset).
SetupUsing the searchmechanismof the platform to retrieve relevant datasetsandcitationsusinganinvalidquery.
TestingProcedure
The user browses to the search tab of the OSL platform and submits aninvalidquerythatisnotexpectedtofetchanydatasetsorcitationresults.
ExpectedResult
Theemptysetisreturnedandtheuserisinformedthattherearenoresultsforthegivenquery.
TestResult (Success/Failure)
3.4.2 Pilot 1
3.4.2.1 P1:UploadDataSet
3.4.2.1.1 UploadDataSetSuccessfullyTestCaseID OpenScienceLink_BER_P1-UDS_V_01
Test CaseTitle
UploadDataSetSuccessfully
Test CaseDescription
Verify that a non-empty dataset that has not yet been submitted to theplatformissuccessfullyuploaded.
Setup AccesstotherespectiveGUI;theusermustbeloggedinasauthor.
TestingProcedure
Theuserenters the“UploadDataset”page, fills inall thenecessary fields,uploadsazipfilewiththedatasetorprovidesanexternalURLandpresses“SaveChanges”.
ExpectedResult
Newlyuploadeddatasetisvisibletotheauthor.
TestResult (Success/Failure)
3.4.2.1.2 UploadDataSetFailed–DataSetAlreadyExists
Page 41 of 82
D8.2.3Stakeholders,TechnicalandFinancialEvaluation
TestCaseID OpenScienceLink_BER_P1-UDS_I_02
Test CaseTitle
UploadDataSetFailed–DataSetAlreadyExists
Test CaseDescription
Verify that if a user tries to submit a dataset that has already beensubmitted,thesubmissionprocessfails.
Setup AccesstotherespectiveGUI;theusermustbeloggedinasauthor.
TestingProcedure
Theuserenters the“UploadDataset”page, fills inall thenecessary fields,provided a zip filewith the dataset or an external URL and presses “SaveChanges”.Thedatasetwithsuchname/descriptionorURLhasalreadybeenuploaded.
ExpectedResult
The submission process is not performed; the user gets themessage thatthedatasetalreadyexists.
TestResult (Success/Failure)
3.4.2.1.3 UploadDataSetFailed–MissingdataTestCaseID OpenScienceLink_BER_P1-UDS_I_03
Test CaseTitle
UploadDataSetFailed–MissingData
Test CaseDescription
Verify that if a user tries to submit a datasetwithout providing the actualdata,thesubmissionprocessfails.
Setup AccesstotherespectiveGUI;theusermustbeloggedinasauthor.
TestingProcedure
Theuser enters the “UploadDataset” page, fills in all thenecessary fields,but does not provide a zip file with the dataset or an external URL andpresses“SaveChanges”.
ExpectedResult
Thesubmissionprocessisnotperformed;theusergetsthemessagethatthedataismissing.
TestResult (Success/Failure)
3.4.2.2 P1:CreateNewJournalIssue
3.4.2.2.1 NewJournalIssueCreatedSuccessfullyTestCaseID OpenScienceLink_BER_ P1-NJI
Page 42 of 82
D8.2.3Stakeholders,TechnicalandFinancialEvaluation
_V_01
Test CaseTitle
NewJournalIssueCreatedSuccessfully
Test CaseDescription
Verify that a new issue of the journal can be successfully created by aneditor.
Setup Theuserisloggedinaseditor.
TestingProcedure
Theusergoesto“EditorActivities”tabandpresses“AddIssue”.Then(s)heselectsthevolume,fillsintherequiredfieldsandpresses“Savechanges”.
ExpectedResult
A newly created issue should appear in the list of available issues in the“EditorActivities”tab.
TestResult (Success/Failure)
3.4.2.2.2 NewJournalIssueCreationFailed–IssueAlreadyExistsTestCaseID OpenScienceLink_BER_P1-NJI_I_02
Test CaseTitle
NewJournalIssueCreationFailed–IssueAlreadyExists
Test CaseDescription
Verifythatanissueofthejournalwithaspecifictitleanddeadlinescannotbecreatedtwice.
Setup Theuserisloggedinaseditor.
TestingProcedure
Theusergoesto“EditorActivities”tabandpresses“AddIssue”.Then(s)heselects the volume, fills in the title and deadline fieldswith the values ofalreadyexistingissue,andpresses“Savechanges”.
ExpectedResult
The creation of a new issue is not completed, a user gets amessage thatsuchissuealreadyexists.
TestResult (Success/Failure)
3.4.2.2.3 EditExistingJournalIssue
TestCaseIDOpenScienceLink_BER_ P1-NJI_V_03
Test CaseTitle
Operationsonajournalissuearedonesuccessfully
Page 43 of 82
D8.2.3Stakeholders,TechnicalandFinancialEvaluation
Test CaseDescription
Verify that a user with author privileges can view, edit and delete andexistingjournalissuethathehascreated
Setup Theuserisloggedinaseditorandisthecreatorofthejournalissue.
TestingProcedure
The user goes to “Editor Activities” tab and presses “Add Issue” andperforms the following operations on journal issues (by clicking onrespectivethumbnails):“deletetheissue”,“edittheissue”,“viewtheissue”,“viewdatasetssubmittedforthisissue”.
ExpectedResult
Allfouroperationscanbesuccessfullyperformed.
TestResult (Success/Failure)
3.4.2.3 P1:DatasetPublication
3.4.2.3.1 DatasetPublishedSuccessfully
TestCaseIDOpenScienceLink_BER_ P1-DSF_V_01
Test CaseTitle
DatasetPublishedSuccessfully
Test CaseDescription
VerifythatanewdatasetcanbesuccessfullyassignedaDOIandpublishedbyaneditor.
SetupThe user is logged in as editor. The dataset has been reviewed andaccepted.
TestingProcedure
Theusergoesto“EditorActivities”tab,viewsthedatasetssubmittedforaspecificissueandpresses“PublishthedatasetbygivingitaDOI”.Then(s)hetypestheDOIandpresses“Savechanges”.
ExpectedResult
ApublisheddatasetshouldbeindexedandavailabletothePlatformusers.
TestResult (Success/Failure)
Page 44 of 82
D8.2.3Stakeholders,TechnicalandFinancialEvaluation
3.4.3 Pilot 2
3.4.3.1 P2:CreateReviewCall
3.4.3.1.1 CallCreationSuccessfulTestCaseID OpenScienceLink_BER_P2-CRC_V_01
Test CaseTitle
CallCreationSuccessful
Test CaseDescription
Verifythecorrectoperationofthereviewcallcreation
Setup Theusermusthaveloggedintotheplatformwitheditorprivileges.
TestingProcedure
The user creates a review call by correctly filling the form that can beaccessedfromthe„editoractivities“taboftheplatform.
ExpectedResult
Newlycreatedreviewcallisdisplayedandopenforsubmissions.
TestResult (Success/Failure)
3.4.3.1.2 CallCreationFailed–MissingDataTestCaseID OpenScienceLink_BER_P2-CRC_I_02
Test CaseTitle
CallCreationFailed–MissingData
Test CaseDescription
Verifythecorrectresponseoftheservicewhenrequiredinformationhasnotbeenprovided.
Setup Theusermusthaveloggedintotheplatformwitheditorprivileges.
TestingProcedure
The user attempts to create a review call by filling the form that can beaccessed from the „editor activities“ tab of the platform, but leaves outrequiredinformation(e.g.,thevenueforthecall).
ExpectedResult
Presentationof anerrormessage to theuser indicating that requireddatahasnotbeenprovided.
TestResult (Success/Failure)
Page 45 of 82
D8.2.3Stakeholders,TechnicalandFinancialEvaluation
3.4.3.2 P2:UploadFilesforReview
3.4.3.2.1 FileUploadSuccessfulTestCaseID OpenScienceLink_BER_P2-UFR_V_01
Test CaseTitle
FileUploadSuccessful
Test CaseDescription
Verifythecorrectoperationofthefileuploadfunctionality.
Setup Theusermusthaveloggedintotheplatformwitheditorprivileges.
TestingProcedure
The user clicks the „upload“ button on the „editor activities“ tab of theplatform,correctlyfillstheformandselectsafileforupload.
ExpectedResult
Thefileisstoredontheserverandthereviewprocesscanproceedwiththenextstage.
TestResult (Success/Failure)
3.4.3.2.2 FileUploadFailed–MissingDataTestCaseID OpenScienceLink_BER_P2-UFR_I_02
Test CaseTitle
FileUploadFailed–MissingData
Test CaseDescription
Verifythecorrectresponseoftheservicewhenrequireddatahasnotbeenprovided.
Setup Theusermusthaveloggedintotheplatformwitheditorprivileges.
TestingProcedure
The user clicks the „upload“ button on the „editor activities“ tab of theplatform,leavesoutrequiredinformationwhilefillingtheformandselectsafileforupload.
ExpectedResult
Presentationof anerrormessage to theuser indicating that requireddatahasnotbeenprovided.
TestResult (Success/Failure)
Page 46 of 82
D8.2.3Stakeholders,TechnicalandFinancialEvaluation
3.4.3.3 P2:GetReviewerSuggestions
3.4.3.3.1 GetReviewerSuggestionsSuccessfullyTestCaseID OpenScienceLink_BER_P2-GRS_V_01
Test CaseTitle
GetReviewerSuggestionsSuccessfully
Test CaseDescription
Verify that the system correctly retrieves reviewer suggestions whenappropriatekeywordsareprovided.
Setup Theusermusthaveloggedintotheplatformwitheditorprivileges.
TestingProcedure
Theuser addsoneormore closely connected termsdescribing a researchtopicandrequestsreviewersuggestions.
ExpectedResult
Thesystempresentsalistofappropriatepotentialreviewers.
TestResult (Success/Failure)
3.4.3.3.2 GetReviewerSuggestionsFailed–NoAppropriateReviewersTestCaseID OpenScienceLink_BER_P2-GRS_I_02
Test CaseTitle
GetReviewerSuggestionsFailed–NoAppropriateReviewers
Test CaseDescription
Verify the correct response of the service when an inappropriate set ofkeywordsisselected.
Setup Theusermusthaveloggedintotheplatformwitheditorprivileges.
TestingProcedure
The user adds a relatively large set of very diverse terms describing aresearchtopicandrequestsreviewersuggestions.
ExpectedResult
The system presents an error message to the user indicating that noreviewercanbe found for theentered setof terms. It recommends to theusertoremovesometermswithlowimportance.
TestResult (Success/Failure)
Page 47 of 82
D8.2.3Stakeholders,TechnicalandFinancialEvaluation
3.4.3.4 P2:SelectandInviteReviewers
3.4.3.4.1 SelectandInviteSuggestedReviewersTestCaseID OpenScienceLink_BER_P2-SIR_V_01
Test CaseTitle
SelectandInviteSuggestedReviewers
Test CaseDescription
Verify that the selection and invitation of suggested reviewers works asplanned.
SetupTheusermusthave logged in to theplatformwitheditorprivileges,andalistofreviewersuggestionsmusthavebeenretrievedsuccessfully.
TestingProcedure
The user selects one or more reviewers from the list of suggestions andclicks„Invitereviewers“.
ExpectedResult
The system starts the process for inviting reviewers by presenting theinvitationformforeachreviewerwhohasbeenselectedbytheuser.
TestResult (Success/Failure)
3.4.3.4.2 InviteReviewersChosenbytheUserTestCaseID OpenScienceLink_BER_P2-SIR_I_02
Test CaseTitle
InviteReviewersChosenbytheUser
Test CaseDescription
Verifythattheusercaninvitereviewersnotonthelistofsuggestions.
SetupTheusermusthave logged in to theplatformwitheditorprivileges,andalistofreviewersuggestionsmusthavebeenretrieved(listmaybeempty).
TestingProcedure
Theuseraddsoneormorereviewerstothelistofsuggestionsbyenteringtheirnameande-mail,selectsthemandclicks„Invitereviewers“.
ExpectedResult
Thesystemstartstheprocessforinvitingreviewerspresentingtheinvitationformforeachreviewerwhohasbeenselectedbytheuserinsuccession.
TestResult (Success/Failure)
Page 48 of 82
D8.2.3Stakeholders,TechnicalandFinancialEvaluation
3.4.3.4.3 CompletetheInvitationTestCaseID OpenScienceLink_BER_P2-SIR_V_03
Test CaseTitle
CompletetheInvitation
Test CaseDescription
Verifythatthesystemsendse-mailstotheselectedreviewersaftertheuserhasfilledallinvitationforms.
Setup Theusermusthaveloggedintotheplatformwitheditorprivileges.
TestingProcedure
For testing purposes, the user adds him- or herself as a reviewer andcompletestheinvitationprocess.
ExpectedResult
The system sends ane-mail invitation to theuser containing the text thatwasenteredwhile fillingthe invitationformaswellasa linktothereviewform.Theeditor'soverview table for the invited reviewers shows that theuserhasbeeninvitedasareviewer.
TestResult (Success/Failure)
3.4.3.5 P2:ManagingtheReviewProcess
3.4.3.5.1 ManagingtheReviewProcess:RequestAccepted
TestCaseIDOpenScienceLink _BER_ P2-MRP_V_01
Test CaseTitle
ManagingtheReviewProcess:RequestAccepted
Test CaseDescription
Verifythatthesystemallowsinvitedreviewerstoaccepttheinvitationanddisplaysthisinformationtotheinvitingeditor.
Setup OpenScienceLink_BER_P2-SIR_V_03hasbeencompletedsuccessfully.
TestingProcedure
Theuseropensthelinksentbye-mailandacceptsthereviewinvitation.
ExpectedResult
Theuser's“ReviewerActivities”overviewnowcontainsalinktothereviewform.Theeditor'soverview table for the invited reviewers shows that theuseracceptedtheinvitation.
TestResult (Success/Failure)
Page 49 of 82
D8.2.3Stakeholders,TechnicalandFinancialEvaluation
3.4.3.5.2 ManagingtheReviewProcess:RequestDenied
TestCaseIDOpenScienceLink _BER_ P2-MRP_V_02
Test CaseTitle
ManagingtheReviewProcess:RequestDenied
Test CaseDescription
Verify that thesystemallows invitedreviewers toreject the invitationanddisplaysthisinformationtotheinvitingeditor.
Setup OpenScienceLink_BER_P2-SIR_V_03hasbeencompletedsuccessfully.
TestingProcedure
Theuseropensthelinksentbye-mailandrejectsthereviewinvitation.
ExpectedResult
The editor's overview table for the invited reviewers shows that the userrejectedtheinvitation.
TestResult (Success/Failure)
Page 50 of 82
D8.2.3Stakeholders,TechnicalandFinancialEvaluation
3.4.3.5.3 ManagingtheReviewProcess:SendE-MailReminder
TestCaseIDOpenScienceLink _BER_ P2-MPR_V_04
Test CaseTitle
ManagingtheReviewProcess:SendE-MailReminder
Test CaseDescription
Verifythatthesystemallowseditorstosendreminderstoinvitedreviewers.
Setup OpenScienceLink_BER_P2-SIR_V_03hasbeencompletedsuccessfully.
TestingProcedure
Theeditorusestheplatform's“sende-mailreminder”functionality.
ExpectedResult
Thesystemsendsane-mailremindertotheuser.
TestResult (Success/Failure)
3.4.3.6 P2:ReviewSubmission
3.4.3.6.1 ReviewSubmissionSuccessfulTestCaseID OpenScienceLink_BER_P2-RS_V_01
Test CaseTitle
ReviewSubmissionSuccessful
Test CaseDescription
Verifythatthesystemallowsinvitedreviewerstosubmitthecorrectlyfilledoutreviewformandeditorstoaccessthereview.
Setup OpenScienceLink_BER_P2-MPR_V_01hasbeencompletedsuccessfully.
TestingProcedure
The user fills out all required fields of the review form and submits thereview.
ExpectedResult
Presentation of a confirmation message to the user indicating that thereviewhasbeensubmittedsuccessfully.Theeditor'soverviewtablefortheinvited reviewers shows that the user submitted the review and nowcontainsalinktothereview.
TestResult (Success/Failure)
Page 51 of 82
D8.2.3Stakeholders,TechnicalandFinancialEvaluation
3.4.3.6.2 ReviewSubmissionFailed–RequiredInputMissingTestCaseID OpenScienceLink_BER_P2-RS_I_02
Test CaseTitle
ReviewSubmissionFailed–RequiredInputMissing
Test CaseDescription
Verifythatthesystemgivesthecorrecterrormessagetoinvitedreviewersiftheyattempttosubmitthereviewformwithoutfillingallrequiredfields.
Setup OpenScienceLink_BER_P2-MPR_V_01hasbeencompletedsuccessfully.
TestingProcedure
Theuserattemptstosubmitthereviewwithoutfillingoutallrequiredfields.
ExpectedResult
Presentationofanerrormessageto theuser indicating that requireddatahasnotbeenprovided.
TestResult (Success/Failure)
3.4.3.7 P2:ReviewerRating
3.4.3.7.1 ReviewerRatedSuccessfullyTestCaseID OpenScienceLink_BER_P2-RR_V_01
Test CaseTitle
ReviewerRatedSuccessfully
Test CaseDescription
Verifythatareviewercanbesuccessfullyassignedaratingaftercompletionofthereviewofanarticle/dataset.
SetupTheuserisloggedinaseditor.Thearticle/datasethasbeenreviewedbythereviewer.
TestingProcedure
Theusergoesto“EditorActivities” tab,andviewsthereviewer invitationsforaspecificarticle/dataset.Foracompletedreviewofthearticle/datasetbyareviewers(he)presses“RateReviewer”.Then(s)heratesthereviewerinascalefrom1to5andpresses“Submit”.
ExpectedResult
The rating for this reviewer is stored in the platform. The platform cancalculatethereviewer’soverallratingbasedonthestoredratings.
TestResult (Success/Failure)
Page 52 of 82
D8.2.3Stakeholders,TechnicalandFinancialEvaluation
3.4.3.7.2 ReviewerRatingPresentedSuccessfullyTestCaseID OpenScienceLink_BER_P2-RR_V_02
Test CaseTitle
ReviewerRatingPresentedSuccessfully
Test CaseDescription
Verifythattheratingofareviewerthathasbeenalreadyratedispresentedtotheeditorbeforeinvitingreviewers.
SetupTheuser is logged inaseditorandwishesto invitereviewersforaspecificissue. The suggested reviewer is also a user of the OpenScienceLinkPlatform.
TestingProcedure
Theusergoesto“EditorActivities”tab,andselectstoinvitereviewersforaspecific article/dataset. After adding any desired keywords s(he) presses“FindReviewers”andispresentedwithalistofrelevantreviewersfromtheOSLPlatformsorPubMed/MEDLINEAuthors.
ExpectedResult
Intherelevantreviewerslistthereviewer’soverallrating(scale1to5)mustappearforeachreviewerwhoisauseroftheOpenScienceLinkplatform.Ifareviewerhasnotbeenratedyet,theratingappearsblank(unrated).
TestResult (Success/Failure)
3.4.4 Pilot 3
3.4.4.1 P3:RequestforTrendAnalysisbasedonTopic
3.4.4.1.1 RequestforVisualizingtheTrendofaGivenTopicSuccessfullyTestCaseID OpenScienceLink_BER_P3-RTA_V_01
Test CaseTitle
RequestforVisualizingtheTrendofaGivenTopicSuccessfully
Test CaseDescription
Verify the correct operation of the request for topic trend analysisvisualization
Test CasePurpose
Toverifythatthetrendofexistingtopicsarepresentedtotheuserthroughplots
Setup Theuserneedstohaveloggedintheplatform.
TestingProcedure
Theuserentersoneormoretermsdescribingtheresearchtopicforwhichtheywouldliketobepresentedwiththetrendanalysisplots.
ExpectedResult
Presentationofthetrendanalysisplotfortheresearchtopicwhichtheuserhasdescribedthroughthequeryterms.
Page 53 of 82
D8.2.3Stakeholders,TechnicalandFinancialEvaluation
TestResult (Success/Failure)
3.4.4.1.2 RequestforVisualizingtheTrendofaGivenTopic-MissingorIncompleteDataTestCaseID OpenScienceLink_BER_P3-RTA_I_02
Test CaseTitle
Request forVisualizing theTrendofaGivenTopic -Missingor IncompleteData
Test CaseDescription
Verifythecorrectresponseoftheservicewhennoterms,ornotsupportedtermsdescribingatopichavebeenprovided.
Test CasePurpose
To verify the correct response of the service when no terms or notsupportedtermsdescribingatopichavebeenprovided.
Setup Theuserneedstohaveloggedintheplatform.
TestingProcedure
Theuserrequeststobepresentedwiththetrendanalysisplotwithoutfillinginanydata,orfillingtermsthatarenotsupportedorindexed.
ExpectedResult
Presentation of an error message to the user indicating that no data, orunsupporteddatahavebeenprovided.
TestResult (Success/Failure)
3.4.5 Pilot 4
3.4.5.1 P4:RequestforCo-authorshipGraphs
3.4.5.1.1 RequestforCoauthorshipGraphsSuccessfullyTestCaseID OpenScienceLink_BER_P4-RFC_V_01
Test CaseTitle
RequestforCoauthorshipGraphsSuccessfully
Test CaseDescription
Verifythecorrectoperationoftherequestforcoauthorshipgraphs
Test CasePurpose
To verify that existing coauthorships are presented to the user throughgraphs.
Setup Theuserneedstohaveloggedintheplatform.
TestingProcedure
The user enters one or more terms describing the research topic or thenameof the scientist forwhom theywould like to be presentedwith the
Page 54 of 82
D8.2.3Stakeholders,TechnicalandFinancialEvaluation
coauthorshipgraphs.
ExpectedResult
Presentation of the coauthorship graphs for the research topic which theuserhasdescribedthroughthetermsorforthescientistwhosenametheyhaveentered.
TestResult (Success/Failure)
3.4.5.1.2 RequestforCoauthorshipGraphsFailed–MissingdataTestCaseID OpenScienceLink_BER_P4-RFC_I_02
Test CaseTitle
RequestforCoauthorshipGraphsFailed–Missingdata
Test CaseDescription
Verifythecorrectresponseoftheservicewhennotermsdescribingatopicornoscientistnamehavebeenprovided.
Test CasePurpose
To verify the correct response of the servicewhen no terms describing atopicornoscientistnamehavebeenprovided.
Setup Theuserneedstohaveloggedintheplatform.
TestingProcedure
The user requests to be presented with the coauthorship graphs withoutfillinginanydata.
ExpectedResult
Presentation of an errormessage to the user indicating that no data hasbeenprovided.
TestResult (Success/Failure)
3.4.5.1.3 RequestforCoauthorshipGraphsFailed–InappropriatedataTestCaseID OpenScienceLink_BER_P4-RFC_I_03
Test CaseTitle
RequestforCoauthorshipGraphsFailed–Inappropriatedata
Test CaseDescription
Verify the correct response of the service when inappropriate data havebeenprovided.
Test CasePurpose
Toverifythecorrectresponseoftheservicewheninappropriatedatahavebeenprovided.
Setup Theuserneedstohaveloggedintheplatform.
Testing Theuserrequeststobepresentedwiththecoauthorshipgraphsafterfilling
Page 55 of 82
D8.2.3Stakeholders,TechnicalandFinancialEvaluation
Procedure in inappropriatedata (i.e., termswhichdonotdescribeabiomedical topicorcomprisethenameofascientist).
ExpectedResult
Presentationofanerrormessage to theuser indicating that inappropriatedatahasbeenprovided.
TestResult (Success/Failure)
3.4.6 Pilot 5
3.4.6.1 P5:RequestforPublicationVolumebasedonTopic
3.4.6.1.1 RequestforPublicationVolumebasedonTopicSuccessfullyTestCaseID OpenScienceLink_BER_P5-RPT_V_01
Test CaseTitle
RequestforPublicationVolumebasedonTopicSuccessfully
Test CaseDescription
VerifythecorrectoperationoftherequestforpublicationvolumebasedonTopic
Test CasePurpose
ToverifythecorrectoperationoftherequestforpublicationvolumebasedonTopic
Setup Theuserneedstohaveloggedintheplatform.
TestingProcedure
Theuserentersoneormoretermsdescribingtheresearchtopicforwhichtheywouldliketoreceiveevaluation.
ExpectedResult
Presentation of a series of ranked lists of Country, City, Journals, Authorsbasedonthenumberofpublicationsforthespecifictopic.
TestResult (Success/Failure)
3.4.6.1.2 RequestforPublicationVolumebasedonTopicFailed–MissingdataTestCaseID OpenScienceLink_BER_P5-RPT_I_02
Test CaseTitle
RequestforPublicationVolumebasedonTopicFailed–Missingdata
Test CaseDescription
Verifythecorrectresponseoftheservicewhennotermsdescribingatopichavebeenprovided.
Test Case To verify the correct response of the servicewhen no terms describing a
Page 56 of 82
D8.2.3Stakeholders,TechnicalandFinancialEvaluation
Purpose topichavebeenprovided.
Setup Theuserneedstohaveloggedintheplatform.
TestingProcedure
The user requests to be presented with the ranked lists of Country, City,Journals,Authorsbasedonthenumberofpublicationswithoutenteringanytermsdescribingatopic.
ExpectedResult
Presentation of an errormessage to the user indicating that no data hasbeenprovided.
TestResult (Success/Failure)
3.4.6.1.3 RequestforPublicationVolumebasedonTopicFailed–InappropriatedataTestCaseID OpenScienceLink_BER_P5-RPT_I_03
Test CaseTitle
RequestforPublicationVolumebasedonTopicFailed–Inappropriatedata
Test CaseDescription
Verify the correct response of the service when inappropriate data havebeenprovided.
Test CasePurpose
Toverifythecorrectresponseoftheservicewheninappropriatedatahavebeenprovided.
Setup Theuserneedstohaveloggedintheplatform.
TestingProcedure
The user requests to be presented with the ranked lists of Country, City,Journals, Authors based on the number of publications and has enteredinappropriatetermswhichdonotdescribeabiomedicaltopic.
ExpectedResult
Presentationofanerrormessage to theuser indicating that inappropriatedatahasbeenprovided.
TestResult (Success/Failure)
3.4.6.2 P5:RequestforAuthorEvaluationScore
3.4.6.2.1 RequestforAuthorEvaluationScoreSuccessfullyTestCaseID OpenScienceLink_BER_P5-RPT_V_01
Test CaseTitle
RequestforEvaluationScoreofanAuthor
Test Case Verify the correct operation of the request for evaluation score for an
Page 57 of 82
D8.2.3Stakeholders,TechnicalandFinancialEvaluation
Description Author
Test CasePurpose
To verify the correct operation of the request for evaluation score for anAuthor
SetupThe user needs to have logged in the platform and have performed arequestforpublicationvolumebasedonatopic
TestingProcedure
TheplatformautomaticallyrequestsforanevaluationscoreforeachoftheAuthors that are displayed in the ranked list of Authors as a result of arequestforpublicationvolumebasedonatopic.
ExpectedResult
Presentation of a numerical value ranging from 0 to 100 for each of theAuthorsintherankedlistofAuthors.
TestResult (Success/Failure)
Page 58 of 82
D8.2.3Stakeholders,TechnicalandFinancialEvaluation
3.5 Technical Evaluation Results The following table summarizes the technical evaluation results.As the table shows, all of thecontactedtestsweresuccessful,but,in5cases,theerrormessagesshouldbeaddedoredited(5I tests), Overall, this evaluation and testing shows that the first release of the platform isworkinginatechnicallysatisfactorymanner.
TestCaseTitle Result
OpenScienceLink_BT_OPS_V_01 Success
OpenScienceLink_BT_CC_V_01 Success
OpenScienceLink_CT_SEC_V_01 Success
OpenScienceLink_BER_CF-L_V_01 Success
OpenScienceLink_BER_CF-L_I_02 Success
OpenScienceLink_BER_CF-L_I_03 Success
OpenScienceLink_BER_CF-R_V_01 Success
OpenScienceLink_BER_CF-R_V_02 Success
OpenScienceLink_BER_CF-S_V_01 Success
OpenScienceLink_BER_CF-S_I_02 Success
OpenScienceLink_BER_P1-UDS_V_01 Success
OpenScienceLink_BER_P1-UDS_I_02 Success
OpenScienceLink_BER_P1-UDS_I_03 Success
OpenScienceLink_BER_P1-NJI_V_01 Success
OpenScienceLink_BER_P1-NJI_I_02 Success
OpenScienceLink_BER_P1-NJI_V_03 Success
OpenScienceLink_BER_P1-DSF_V_01 Success
OpenScienceLink_BER_P2-CRC_V_01 Success
OpenScienceLink_BER_P2-CRC_I_02 Success
OpenScienceLink_BER_P2-UFR_V_01 Success
OpenScienceLink_BER_P2-UFR_I_02 Success
OpenScienceLink_BER_P2-GRS_V_01 Success
OpenScienceLink_BER_P2-GRS_I_02 Success
OpenScienceLink_BER_P2-SIR_V_01 Success
OpenScienceLink_BER_P2-SIR_I_02 Success
OpenScienceLink_BER_P2-SIR_V_03 Success
OpenScienceLink_BER_P2-MRP_V_01 Success
OpenScienceLink_BER_P2-MRP_V_02 Success
OpenScienceLink_BER_P2-MPR_V_04 Success
Page 59 of 82
D8.2.3Stakeholders,TechnicalandFinancialEvaluation
OpenScienceLink_BER_P2-RS_V_01 Success
OpenScienceLink_BER_P2-RS_I_02 Success
OpenScienceLink_BER_P2-RR_V_01 Success
OpenScienceLink_BER_P2-RR_V_02 Success
OpenScienceLink_BER_P3-RTA_V_01 Success
OpenScienceLink_BER_P3-RTA_I_02 Success(althoughnomessageisprompted)
OpenScienceLink_BER_P4-RFC_V_01 Success
OpenScienceLink_BER_P4-RFC_I_02 Success(althoughnomessageisprompted)
OpenScienceLink_BER_P4-RFC_I_03 Success(althoughnomessageisprompted)
OpenScienceLink_BER_P5-RPT_V_01 Success
OpenScienceLink_BER_P5-RPT_I_02 Success(althoughnomessageisprompted)
OpenScienceLink_BER_P5-RPT_I_03 Success(althoughnomessageisprompted)
OpenScienceLink_BER_P5-RAE_V_01 Success
Table 29: Summary of Technical Evaluation Results.
Page 60 of 82
D8.2.3Stakeholders,TechnicalandFinancialEvaluation
4 Financial Evaluation
4.1 Overall OSL Platform Financial Evaluation
4.1.1 The overall open access landscape Agrowth trendhasbeenobserved for theopenaccess (OA)publicationmarketover thepastyears.Accordingto(Laakso,Welling,Bukvova,Nyman,&Björk,2011)openaccesspublicationsproductionhas increasedmore than10 timeswithin thepastdecadeboth in termsofarticlesandjournals.Infact,theirstudyindicatesthatOAisalreadyinitsconsolidationperiod.Numbersare quite indicative. The directory of open access journals (DOAJ)already includesmore than9800journalsand1.5millionarticlespublishedfromover120countries.OpenDOAR(Directoryof OpenAccess Repositories)hasmore than 2500 repositories listed, ROAR (Registry of OpenAccess Repositories)presents information about more than 2,500 repositories, while theROARMAP (Registry ofOpenAccessRepositoriesMandatoryArchivingPolicies)includesmorethan440openaccessmandatepolicies.TheDirectoryofOpenAccessBooks (DOAB) includesmore than 1,645Academic peer-reviewed books from55publishers. Moreover, the BASE(BielefeldAcademicSearchEngine)searchengine,whichfocusesonacademicopenaccesswebresources has already indexedmore than 56million documents and 2700 sources. HighWirePress by Stanford Universitycomprises an archive of over 2.3 million free full-text providedarticles. In the biomedical domain, in particular, PMC (PubMed Central)has already indexedmore than 734,000 OA articles with more than 1,200 journals providing their content withimmediatefreeaccessandover970journalsofferingalloftheirarticlesopenlyaccessible.
Figure 6: Open access publishing between 1993 and 2009.
Astudyin(Laakso&Björk,2012)showsthatalthoughinitiallyopenaccesspublishinghasbeenflourishing in the world of universities and scientific societies, commercial publishers havebecomekeyactorsontheOAscene,withanalmosttenfoldincreaseinthenumberofOAarticlesbeingpublishedwithina6yearsperiod(2005-2011).
Page 61 of 82
D8.2.3Stakeholders,TechnicalandFinancialEvaluation
Figure 7: Different Publisher Types for Open Access Articles.
Giventhatprofitcomprisesafundamentalconceptinthecommercialworldandpublicationofresearch includes a series of costs, reconciliation of publishing fees for OA journals has beenattempted through research and subscription funding. From this perspective, there are twomainmodelsforOAprovision;greenandgoldOA(Harnad,etal.,2004).InGreenOA,scientistspublish their findings through institutional or central repositories, or self-archiving (i.e., byplacing their peer-reviewed post-prints on OA web pages). In fact, research institutions andfunders, including the European Commission and Harvard, mandate their scientists to maketheirresearchfindingspubliclyavailableinordertomaximisetheimpactofthescientificworkconducted. In Gold OA, publishing is done through OA journals, such as BioMed Central andPLoS,whichmay either allowpublications for free or set a publishing fee to authors or theirfunders.Graduallymorethoroughclassificationsareintroduced,suchasdiamondOAreferringtopublishinginjournalswhichdonotchargeanyfeestoauthorsorreaders.Currently,onethirdofOAjournalschargepublishingfees,whileothersreceiveinstitutional,governmental,orthird-partyfunding(Kurata,Morioka,Yokoi,&Matsubayashi,2013).
4.1.2 Open Access and Biomedical Research AspresentedinFigure8,thevolumeofopenaccesspublicationshastremendouslyincreasedforthemajorscientificdisciplines,withthebiomedicinedomainshowingagrowthofmorethan16timesinnumberofOAarticlespublishedbetween2000and2011.Infact,accordingtoastudyin2013, OA has shown tremendous growth in the biomedical field with more than 50% of OAarticlescomingforit(Kurata,Morioka,Yokoi,&Matsubayashi,2013).OneofthemaintopicsfordebateoverOApublishingfocusesonitsassumednegativeeffectonthequalityofscientificpublishing.However,recentfindings(Laakso&Björk,2012)showthatOAjournals, and in the medical and health domain in particular, which are indexed in Web ofScienceand/orScopusandarepublishedinthefourlargestpublishingcountriesshowthesamescientific impact andqualitywith subscription journals. Interestingly,OA journalswhichposearticleprocessingchargesinordertofundpublishingareonaveragecitedmorethanotherOAjournals.
Page 62 of 82
D8.2.3Stakeholders,TechnicalandFinancialEvaluation
Figure 8: Open Access Articles across different disciplines.
4.2 Pilot 1:Open-access Data journals development
4.2.1 Analysis of Different Business Models and Competitors
Tomeetthecostofpublishing,authorswhopublishopen-accessareaskedtopayanopenaccessfee or article-processing charge (APC) per published paper. This cost is usually covered byresearchfundsfromthedepartmentsthatcontributetothiswork.
To partially remove this burden from the individual authors, some publishers have createddifferent Membership programs. A Membership Program enables academic and researchinstitutions, societies, groups, funders and corporations to actively support open access inscholarlypublishing,andhelpensurethemostwidespreaddisseminationoftheworkpublishedby their researchersormembers.Dependingon the typeofMembership,Member institutionscoversomeorallofthepublicationcostfortheirindividualresearcherswhentheysubmittoacertainpublisher.
ThreedifferenttypesofMembershiphavebeendescribed:
1. PrepayMembershipEnablesaninstitutiontocoverthewholecostofpublishingwithnoadditionalfeespaidbytheirauthors.
2. SharedSupportMembershipThecostofpublishingissplitbetweentheinstitutionandauthor.
3. SupporterMembershipMemberspayaflatrateannualMembershipfeebasedonthenumberofscienceandmedicalresearchersandgraduatestudentsattheirinstitution.A15%discountonthearticle-processingcharge(APC)isgivenwhenpublishinginthejournals.
Page 63 of 82
D8.2.3Stakeholders,TechnicalandFinancialEvaluation
Thenumberoffundingresearchorganizationsthatadoptpoliciestopromoteopen-accesstoscientificinformation.TheseorganizationscouldbeasourceoffundingfortheBiomedicaldataJournal.ThefollowingtableprovidesinformationonfundingorganizationsfromaroundEuropeandtheopenaccesspoliciestheyhaveinplace.
Organization Country OApolicy OAFunding
Fonds zur Foerderung der wissenschaftlichen Forschung (FWF) Austria
Yes Yes
Research Foundation Flanders (FWO) Belgium
Yes Yes
Centre national de la recherche scientifique (CNRS) France
Yes Yes
INSERM (Institut national de la sante et de la recherche medicale) France
Yes Yes
Deutsche Forschungsgemeinschaft (DFG) Germany
Yes Yes
Max Planck Society Germany Yes Yes
Hungarian Scientific Research Fund (OTKA) Hungary
Yes Yes
Science Foundation Ireland (SFI)
Ireland
Yes No
ConsiglioNazionale delle Ricerche (CNR) Italy
YesYes
Telethon
Italy Yes Yes
Nederlandse Organisatie voor Wetenschappelijk Onderzoek (NWO) Netherlands
NoYes
Consejo Superior de Invesigaciones Científicas (CSIC) Spain
No Yes
RiksbankensJubileumsfond
Sweden
Yes Yes
Swedish Foundation for Strategic Research Sweden
No Yes
Swedish Research Council Sweden No Yes
Swiss National Science Foundation (SNF)
Switzerland
No Yes
Breakthrough Breast Cancer UK Yes Yes
Page 64 of 82
D8.2.3Stakeholders,TechnicalandFinancialEvaluation
Organization Country OApolicy OAFunding
British Heart Foundation (BHF)
UK Yes Yes
Cancer Research UK UK Yes Yes
Research Council UK (RCUK) UK Yes Yes
Wellcome Trust UK Yes Yes
Table 30: European Open-Access Funders.
ThefollowingtableprovidesinformationaboutcompetitorjournalsforBMDJ.
Competitor BriefDescriptionRelationtoOpenScienceLink/PotentialBusinessModels
Scientific DataJournal (NatureGroup)
Starting May 2014, this data journal will allow for thepublishing,discoveryandreusingofresearchdata.http://www.nature.com/scientificdata/about/
Different scientificfocusforthedata
Data in Brief(Elsevier)
Thejournal“welcomessubmissionsthatdescribedatafromallresearch areas”. It is indexed by DOAJ, Science Direct, andPubMed Central. It provides indication of impact via“Altmetrics”, illustrating the engagement of social mediacommunitieswitharticlesinDatainBriefbasedontheamountof activity from Twitter, Facebook, science blogs, mainstreamnews, and other sources captured by Altmetric.com for eachpublicationinthelastsixmonths.Volume1hasbeenpublishedinDecember2014with18datapapers.BySeptember2015 ithas published four volumes (one volume per quarter: vol. 12with 12; vol. 3 with 43; and vol. 43 with 102 data papers).About¾ofalldatapapersarefromthefieldsofBiochemistry,GeneticsandMolecularBiology;Pharmacology,ToxicologyandPharmaceutical Science; Immunology and Microbiology. Thearticleprocessing fee is500USD,witha50%discountby theendof2015.http://www.journals.elsevier.com/data-in-brief/
Direct competitor,with substantialemphasis onbiomedicalandlifesciences
Linked OpenData fromUniversity ofMünster(Germany)LODUM
LODUMaimsatconnectingdifferentdatasourcesacrossthe15facultiesanddepartmentsattheUniversityofMünsterthroughthe implementation of Open Access and Linked Open Dataprinciplesacrosstheuniversity.http://data.uni-muenster.de/
Potential contentsource
OpenAire EUproject
The FP7 project OpenAIRE aimed at offering the means topromote and realize the widespread adoption of the OpenAccess Policy by aggregating research publications whichcomprise European funded research output. OpenAIREplus,being its successor, further aims at linking this data to the
Potential ContentsourcePlanning to sign aMoU
Page 65 of 82
D8.2.3Stakeholders,TechnicalandFinancialEvaluation
Competitor BriefDescriptionRelationtoOpenScienceLink/PotentialBusinessModels
accompanying research and project information, datasets andauthor information. Apart from the data, it also provides aseries of functionalities such as statistics and reporting toolswhichcanbeusefultoprojectmanagers.http://www.openaire.eu/
re3data
It is a global registry of research data repositories fromdifferent academic disciplineswhich is funded by the GermanResearchFoundation(DFG).http://www.re3data.org/
Potential Contentsource
myExperiment
Itcomprisesapublicrepositoryofscientificworkflowswhichiscurrently supported by three European Commission 7thFramework Programme (FP7) projects: BioVeL, SCAP and theWf4Ever Project and the e-Research SouthandmyGridEPSRCPlatformgrants.http://www.myexperiment.org/
Potential Contentsource
OpenPHACTS
The Open PHACTS Discovery Platform integratespharmacologicaldatafromavarietyofinformationsourcesandoffers tools and services for applying questions on this datawithanaimtofacilitatepharmacologicalresearch.http://www.openphacts.org
Potential Contentsource
BiodiversityDataJournal
The first issue of the Biodiversity Data Journal, featuring 27articles,appearedinitsfinalversioninSeptember2013.BDJispublishedbyPensoftLtd.,Bulgaria.http://biodiversitydatajournal.com Potential Content
source, if theOpenScienceLinkdomainbroadens.Potentialpublisherand editor whocould benefit fromthe data journalmanagement andreview servicesand additionallythe trend miningand scientometricsservices.
GeoscienceDataJournal
At the end of November 2013, five articles appear on thewebsite of the Geoscience Data Journal, including the leadeditorialarticle[20].ThejournalispublishedbyWiley.http://onlinelibrary.wiley.com/journal/10.1002/%28ISSN%292049-6060
JournalofOpenArchaeologyData
The JournalofOpenArchaeologyData ispublishedbyUbiquityPress.Thefirstvolumewithninearticlesappearedin2012.AsofendNovember2013,volume2featurestwodataarticles.http://openarchaeologydata.metajnl.com
JournalofOpenPsychologyData
The Journal of Open Psychology Data is also published byUbiquity Press. By the end of November 2013 the journalwebsite provides access to two articles, one of which is theeditorialarticle.http://openpsychologydata.metajnl.com
Earth SystemScienceData
Earth System Science Data (ESSD) is an international,interdisciplinary journal for the publication of articles onoriginal research data (sets), furthering the reuse of high-quality data of benefit to Earth system sciences. The editorsencourage submissions on original data or data collectionswhichareofsufficientqualityandhavepotentialtocontributetotheseaims.Thejournalmaintainssectionsforregular-lengtharticles, brief communications (e.g. on additions to data sets)andcommentaries,aswellasreviewarticlesandspecialissues.http://earth-system-science-data.net/
Journal of TheJournalofChemical&EngineeringDataisamonthlyjournal
Page 66 of 82
D8.2.3Stakeholders,TechnicalandFinancialEvaluation
Competitor BriefDescriptionRelationtoOpenScienceLink/PotentialBusinessModels
Chemical andEngineeringData
devoted to the publication of experimental data and theevaluation and prediction of property values. It is the onlyAmerican Chemical Society journal primarily concerned witharticles containing experimental data on the physical,thermodynamic, and transport properties of welldefinedmaterials including complex mixtures of known compositionsandsystemsofenvironmentalandbiochemicalinterest.http://pubs.acs.org/journal/jceaax
Journal ofPhysical andChemicalResearchData
TheJournalofPhysicalandChemicalReferenceDataispublishedby the American Institute of Physics (AIP) for the NationalInstituteofStandardsandTechnology(NIST).Theobjectiveofthe Journal is to provide critically evaluated physical andchemical property data, fully documented as to the originalsources and the criteria used for evaluation, preferably withuncertainty analysis. Critical reviews of measurementtechniques may also be included if they shed light on theaccuracyofavailabledata inatechnicalarea.Papersreportingcorrelationsofdataorestimationmethodsareacceptableonlyiftheyarebasedoncriticaldataevaluationandiftheyproduce“reference data”—the best available values for the relevantproperties.http://jpcrd.aip.org/
InternationalJournal ofRoboticsResearch
A leadingpeer-reviewed journal in its field formore than twodecades, the International Journal of Robotics Research (IJRR)was the first scholarly publication on robotics research. IJRRoffers incisiveand thought-provokingoriginal researchpapersand articles, perceptive reviews, and lively editorials onground-breaking trends issues, technical developments, andtheories in robotics by the outstanding scholars andpractitioners in the field. The Journal covers more than justnarrowtechnicaladvances-itembracesawidevarietyoftopics.IJRR also publishes high quality, peer reviewed datasets andmultimedia extensions alongside articles. This journal is amemberoftheCommitteeonPublicationEthics(COPE).http://ijr.sagepub.com/
F1000Research
F1000Research isanOpenSciencepublishingplatformofferingimmediate publication of posters, slides and articles with noeditorialbias.Allarticlesbenefitfromtransparentpeerreviewandtheinclusionofallsourcedata.http://f1000research.com
EcologicalArchives - DataPapers
EcologicalArchivespublishesmaterialsthataresupplementaltoarticles that appear in the ESA journals (Ecology, EcologicalApplications, Ecological Monographs, Ecosphere, EcosystemHealthandSustainabilityandBulletinof theEcologicalSocietyof America), as well as peer-reviewed data papers withabstractspublishedintheprintedjournals.EcologicalArchivesispublishedindigital,Internet-accessibleform.http://esapubs.org/archive/default.htm
GigaScience
Itisajournalwhichpublishes‘big-data’studiesfromtheentirespectrum of life and biomedical sciences. It is with BioMedCentral and supported by BGI – a Chinese non-profitorganisation which claims to be the largest genomics
Potential Contentsource, Publisherand Editorpotentially
Page 67 of 82
D8.2.3Stakeholders,TechnicalandFinancialEvaluation
Competitor BriefDescriptionRelationtoOpenScienceLink/PotentialBusinessModels
organisation. It is indexed in the Directory of Open AccessJournals(DOAJ),PubMedandPubMedCentral.
interestedprimarily in datajournalmanagement andreview servicesand additionally inthe trend miningand scientometricsservices.
GenomicsDataItisanopenaccessjournalpublishedbyElsevierwhichcoversall aspects of genome-scale analysis, including nucleic acidsanalysis, microarray and next-gen sequencing data and allorganisms.
JournalofOpenPublic HealthData
It is a data journal which is published by Ubiquity Press. Itsmainfocusisondatawithreusabilitypotentialorwhichcanbeusedforresearchvalidationpurposes.
ScientificData
ItisadatajournaltobepublishedbyNaturePublishingGroupwhich currently focuses on datasets from the life, biomedicaland environmental science communities, but is intended tocover a broad range of scientific disciplines. It has beenscheduled to launch inMay 2014. Datasets are not hosted bythepublisher. Insteadcommunity-recogniseddatarepositoriesare expected to store them, if available, or other repositoriessuchastheDryad.
Table 31: Pilot 1 Competitors.
4.2.2 Insights for the formulation of realistic business and sustainability plans
A.Spendingsperyear
ActivityRoughcostestimate
A0 Policies,businessprocess,legalissues OSL
A1 WebsiteandpresenceinsocialnetworksInitial development, software and functionality upgrades, expanding thepresenceinsocialnetworks
2,000
A2 Journalmanagement1/2 position, bachelor’s degree, working knowledge of English, someunderstanding of biomedical issues (2,500 Euro per month, includingsocialandotherinsurance,includingtheinsurancepaidbytheemployingcompany)=12monthsx1,250Europermonth(articleprocessing,e.g.pagesetting,isincludedhere)
15,000
A3 Printingandmailing4issuesx1,500Euro(4 colours cover, the usual issue is with black and white body; whennecessary, individualpages /galley/waybeprinted in colour;printout–between300and500copies;format–A4)
64,000
A4 Advertisingthejournal(online,inspecialisedmagazines,atrelevantconferences,etc.) 2,000
Page 68 of 82
D8.2.3Stakeholders,TechnicalandFinancialEvaluation
Sub-total 25,000
A5 Overhead,20%Officespace&equipment,consumables,communications,webhosting,etc. 5,000
Total 30,000
A6 ReturnofInvestments,20% 6,000
A7 Requiredannualrevenues 36,000
Table 32: Spendings for Pilot 1.
B.Possiblerevenues
Sources Required
revenues
B1 SaleofprintedcopiesincludingbysubscriptionEuro20persoldcopy(abovepackaging,posting,andtransactioncosts),100copiessoldofeachissue=4x100x20
8,000
B2 Advertisinginthejournal 2,000
B3 Directsupportbyanagencyorprogrammesupportingopenaccess 2,000
B4 Author’sfees(institutional membership may provide stability and increasepredictability,butisnotexpectedtoincreaserevenues,sinceitwillleadtowaivingauthors’fees)4issuesx15articlesperissueinaverage=60articlesFeeof400Europerarticle(oranequivalentamountforaninstitutionalmembership)Remark: Fees for publications by members of the Editorial Board and‘active’reviewersmaybewaived.Hence,thiscalculationisbasedonlyonthenumberofarticlesforwhichauthorsarerequestedtopaythefee.
24,000
Total 36,000
Table 33: Possible Revenues for Pilot 1.
Possibilitiestoreducecosts:
Increasetheefficiencyofjournalmanagementandjournalpreparation1. Assignapersonwithlowerqualificationtomanagethejournal(notdesirable).2. Seeksynergiesbyassigningthesamepersontomanagethreeorfourjournals.3. Request that authors use a dedicatedwriting tool (that provides the articles in print-
readyform).
Page 69 of 82
D8.2.3Stakeholders,TechnicalandFinancialEvaluation
4. Milderformoftheaboverequirement–providetemplates incommonlyusedsoftwaretoolsandrequestthatauthorsusetheonetheyareusedto.
5. Requestthatarticlesbewritteninalanguagethatdoesnotrequirecopy-editing.6. Provide paid services for formatting submissions to journal requirements (in case a
contributionisnotformattedaccordingtojournalrequirements)7. Provide paid copy-editing services for cases when a contribution is not written with
acceptablequalityoflanguage.
Page 70 of 82
D8.2.3Stakeholders,TechnicalandFinancialEvaluation
Selectedexamplesofauthor’sfeesforpublicationinopenaccessjournals:
Journal Atlaunch Regular Remark
Biodiversity Data Journal,Pensoft
0 “aminimalfee...thatanyone
canafford”
Manuscriptsmustbewrittenintheirown‘writingtool,’orfrom integrated externalplatforms, such asScratchpads or GBIFIntegratedPublishingToolkit
GeoscienceDataJournal,Wiley n/a $1,500
GigaScience(~BioMedCentral) 0 0 ThankstothesupportbyBGI
Journal of Open Public HealthData,UbiquityPress
0 0 “coveredbyfunding”
GenomicsData,Elsevier $100 $500 Additional charges ifformatting and copy editingisnecessary
ScientificData,NaturePublishingGroup
-30% $630/700 Depending on the type ofCreativeCommonsLicense
Journal of DefenseManagement,OMICSGroup
$1,300 $1,300 Do not seem to haveconsiderable number ofcontributors.MembersoftheEditorial Board publish freeofcharge
Table 34: Examples of author fees for publication in OA journals.
Increaserevenuepredictabilityanddirectsupport• Seeksupportbyfundingagencies• Seeksupportbyacademiclibraries,foundations,corporations,etc.
Page 71 of 82
D8.2.3Stakeholders,TechnicalandFinancialEvaluation
4.3 Pilot 2: A novel open, semantically-assisted peer review process
4.3.1 Analysis of different business models and competitors
Competitor BriefDescription Businessmodels
EasyChairEasyChairisdescribedinDeliverable9.4.3.(Table3).http://www.easychair.org/
EasyChair supports threedifferent kinds of licenses(free, professional andexecutive). There arerestrictions on the number offreelicensesthatcanbeissuedto a single conference.Professional conferenceorganisers and societies orcompanies organizing anumber of conferences arepotentialcustomers.Pricesforallnon-free licensesarebasedonthenumberofsubmissions
FluidReview FluidReviewisdescribedinDeliverable9.4.3.(Table3).http://fluidreview.com
FluidReview has flexiblepricing options that can meettheneedsof anyorganization,bigorsmall.Afreetrialperiodis provided for testing. Aspecial team works with thecustomer to determine whichplan and schedule is bestsuited for his requirements.FluidReview requires anannualcommitment.Afteroneyear, thecustomermaycancelhis subscription or continuewith the service. Flexiblepayment schedules areprovided.
PublicKnowledgeProject - OpenJournalSystems
OJSisdescribedinDeliverable9.4.3.(Table3).
All of the events (discussionforums, lectures, seminars,workshops, and symposia ontopical and timely issues fromevery discipline) arefreeandopen to the public, students,faculty,staffandschools.OJSisbased on the support of ourprivate and academicsponsors.
ArnetMiner ArnetMinerisdescribedinDeliverable9.4.3.(Table3).http://arnetminer.org/reviewer-home
Arnetminer follows the thecommercial open sourcebusiness model. Because theopen source product isavailable for free, potentialcustomers can download,install, and use the productwithout getting in touch withthe commercial firm behindtheproduct.At thesametime,the firm can track viadownload registration and
Page 72 of 82
D8.2.3Stakeholders,TechnicalandFinancialEvaluation
Competitor BriefDescription Businessmodels
community forum activitieswho is actually using theproduct. A lead analysis canthendeterminewhichoftheseusers might be potentialcustomers. More often thannot, however, the firm willwait until a non-paying usersteps forward and asks for asales contact to purchase anyof the services outlined in therevenue generation section.Thus, leads emerge from theexisting user community,either voluntarily or byanalysis.
OpenConf OpenConfisdescribedinDeliverable9.4.3.(Table3).http://www.openconf.com/
It is a peer-reviewmanagement system, used bythousands of events in over100 countries, that facilitatesthe management ofconferences, workshops, andsymposia, yet is flexibleenough that it also powersjournals, grants, books, andcompetitions.OpenConfisalsomultilingual, with translationsfor author and reviewerinterfaces included for over adozen languages. OpenConf isavailable in multiple editionsto meet various needs, andmayeitherbe licensed foruseonone’sownserverorfromamanagedhostingservice.
Peerage ofScience
It is described in Deliverable 9.4.3. (Table 3).https://www.peerageofscience.org
For Authors and Reviewers,the submission and peerreview process in Peerage ofScience is free of any charges.Subscribingjournals,however,have different publishingmodels. Open-access journalsusually require fees fromauthors upon acceptance;these fees must be stated toAuthorsinthepublishingoffersent via Peerage of Science,but Peerage of Science is notcollectinganyfeesitself.
AcademicKarma
Academic Karma is described in Deliverable 9.4.3.(Table3).http://academickarma.org/
Authors access faster, higherquality and more transparentpeer review outside thejournalsystem,forfree.Editors make access to peer-
Page 73 of 82
D8.2.3Stakeholders,TechnicalandFinancialEvaluation
Competitor BriefDescription Businessmodels
review freely available to all,bringing down costs of open-accesspublishing.Reviewers maintain thequality and reproducibility ofthe open access scientificliterature whiledemonstratingexpertiseintheirfield.
TheWinnower WinnowerisdescribedinDeliverable9.4.3.(Table3).https://thewinnower.com/
Winnower is an open accessonline scholarly publishingplatform that employs openpost-publication peer reviewwhich is free for all membersandworks in fourbasic steps:Submission, Review, RevisionandArchival.
Table 35: Pilot 2 Competitors and business models
4.3.2 Targeted customers and stakeholders. Targetedcustomersandstakeholdersforpilot2havebeenanalysedinDeliverable9.4.3.(Table9and10).
4.4 Pilot 3: Services for detection and analysis of research trends
4.4.1 Analysis of different business models and competitors
Competitor BriefDescription Businessmodel
TechCast TechCastisdescribedinDeliverable9.4.3.(Table4).http://www.techcast.org/
TechCast is an academic researchproject that depends on membershipsubscriptionstosupportitsresearch.Itoffers professional subscriptions for 1,2 and 3 years, as well as student andacademic group subscriptions. Trialmembership is also available forlimited access. Subscriptions include 2options
1. Professional Subscription forGeneral Managers, TechnologyOfficers, CIOs, Strategic Planners,Technology Transfer, BusinessDevelopment, and others inbusiness and governments who
Page 74 of 82
D8.2.3Stakeholders,TechnicalandFinancialEvaluation
Competitor BriefDescription Businessmodelneed to stay abreast and plan forstrategicchange
2. Student and Academic GroupSubscriptionforbusinessandotherstudentoracademicgroupsof10ormore for teaching or academicresearchpurposes
TheMillenniumProject
TheMillenniumProjectisdescribedinDeliverable9.4.3. (Table 4). http://www.millennium-project.org/index.html
The Project is sustainable with theimportantfinancialsupportofallkindsofsponsorssuchas• Army Environmental Policy
Institute,• Azerbaijan State Economic
University• Deloitte &Touche LLP, Cleveland,
Ohio• FoundationfortheFuture,Bellevue,
Washington• GeneralMotors,Warren,Michigan• TheHersheyCompany• HughesSpaceandCommunications,• KuwaitPetroleumCorporation• Ministry of Communications,
RepublicofAzerbaijan• Monsanto Company, St. Louis,
Missouri• MotorolaCorporation,Schaumburg,
Illinois• ·RockefellerFoundation• ShellInternational• UNESCO,Paris,France• United Nations Development
Programme,• United Nations University, Tokyo,
Japan• U.S. Department of Energy,
Washington,D.C.(2000–03)
ArnetMiner Arnetminer isdescribed inDeliverable9.4.3. (Table4).http://arnetminer.org/
Arnetminerfollowsthethecommercialopen source business model. In thismodel,theproductisavailableforfree,potential customers can download,install, and use the product withoutgetting in touch with the commercialfirm behind the product. At the sametime, the firm can track via downloadregistration and community forumactivities who is actually using theproduct. A lead analysis can thendetermine which of these users mightbe potential customers. More oftenthan not, however, the firm will waituntil a non-paying user steps forwardandasksforasalescontacttopurchaseany of the services outlined in therevenuegenerationsection.Thus,leads
Page 75 of 82
D8.2.3Stakeholders,TechnicalandFinancialEvaluation
Competitor BriefDescription Businessmodelemerge from the existing usercommunity, either voluntarily or byanalysis.
TrendMinerProject
It is described in Deliverable 9.4.3. (Table 4).http://www.trendminer-project.eu/index.php/obj TrendMinerisanEU-fundedproject
GoogleTrends Google Trends is described in Deliverable 9.4.3.(Table4).http://www.google.com/trends/
Google Trends follows the thecommercial open source businessmodel. Someof the functionalities areavailable for free, potential customerscan use the productwithout getting intouchwiththecommercialfirmbehindtheproduct.Registereduserscanhaveaccesstomorefunctionalities.Analysiscan determine which of the usersmightbepotentialcustomers.
Institute ForTheFuture
IFTF is described in Deliverable 9.4.3. (Table 4). http://www.iftf.org/
IFTF is an independent, non-profitresearch institute IFTF is sustainablewiththeimportantfinancialsupportofacademic institutions and privatesponsors.
EssentialScienceIndicators
ESI is described in Deliverable 9.4.3. (Table 4). http://wokinfo.com/products_tools/analytical/essentialscienceindicators/
ESI depends on membershipsubscriptions offering a singlesubscription model or anacademic/research institutionsubscription. Registered users need tosign in to have access to a web-basedplatform.It includes 10 year rolling coverage,updatedeverytwomonths.
Table 36: Pilot 3 Competitors and business models
4.4.2 Insights for the formulation of realistic business and sustainability plans Trendanalysis is according tomanydifferent sources thebase fordecisionmaking todayandespecially in the future. The availability and –more – important the accessibility of big datavolumes ask for tools to interpret available data. The big financial potential lies behind theanalysisoftrendsinordertobeabletocreateproductswhentheyareneeded.Companiesbeingabletodelivercustomtailoredproductswhentheyareneededmostwillhaveauniquesellingpoint andwill therefore be better than competitors. As a good example a US based companyGenomeQuestdotrendanalysisintheareaofBioinformatics–specializedingenesequencesinpatents. The company’s revenue has doubled every year and is now at around 10million USdollars.Suchasimplebusinessmodelnotevenprovidingadvancedstatisticaland/orsemanticalgorithmsallowsforsuchatremendousgrowth.The technologies developed in Open Science Link will go far beyond what players likeGenomeQuestdo.We foreseeaCAGRof15 to30% in theareaof trendanalysis. In2016 theglobalmarkedwill have reached a statewhere especially Europe should be stable positionedbefore global M&A deals start. Our “product” will by then be is in fully developed state andbeyondstateoftheart.
Page 76 of 82
D8.2.3Stakeholders,TechnicalandFinancialEvaluation
Toestimateexpectedrevenueswouldbetooearlyanddependsalsoonmarkedapproaches.Targetedcustomersandstakeholdersforpilot3havebeenanalysedinDeliverable9.4.3.(Table11).
4.5 Pilot 4: Services for Dynamic Researchers' Collaboration
4.5.1 Analysis of different business models and competitors
Competitor BriefDescription Businessmodel
ResearchGate
ResearchGate is described in Deliverable9.4.3. (Table 5).https://www.researchgate.net
ResearchGate is a for-profit enterprise.Theyhavehadtworoundsoffundingandhave investors thathavepreviouslybeeninvolved with Facebook, LinkedIn andPayPal Researchgate has raised anotherinvestment round of $35M with moneyfrom Microsoft and Tenaya Capital,Dragoneer Investment Group and ThriveCapital.. Over the longer haul, the firmhopes to charge companies anduniversities forusing it toadvertise jobs,and to operate a marketplace forlaboratory materials. It has no plans topost other advertising, though, nor tochargeitsusersdirectly.
ArnetMinerArnetMiner is described in Deliverable9.4.3.(Table5).http://arnetminer.org/
Arnetminer follows the the commercialopen source business model. In thismodel, the product is available for free,potentialcustomerscandownload,install,and use the product without getting intouch with the commercial firm behindthe product. At the same time, the firmcan track via download registration andcommunity forum activities who isactuallyusingtheproduct.Aleadanalysiscan thendeterminewhich of theseusersmightbepotentialcustomers.Moreoftenthannot,however,thefirmwillwaituntilanon-payinguserstepsforwardandasksforasalescontacttopurchaseanyoftheservices outlined in the revenuegeneration section. Thus, leads emergefrom theexistinguser community, eithervoluntarilyorbyanalysis.
MyScienceWork
MyScienceWorkisdescribedinDeliverable9.4.3. (Table 5).http://www.mysciencework.com/#the-research-network
The business model depends onsubscriptionstosupportitssustainability.Trial membership is also available forlimited access.Subscriptions areexamined on a case by case basisaccording to the size and needs of thecustomer
Page 77 of 82
D8.2.3Stakeholders,TechnicalandFinancialEvaluation
Competitor BriefDescription Businessmodel
Academia.edu Academia.eduis described in Deliverable9.4.3.(Table5).https://www.academia.edu
Theservice is freeofchargeto itsover2million users and does not offer anypremium services or institutionalmemberships. However, CEO RichardPriceexpressedhis belief that thebusiness will generate revenue in thefuture by providing ‘trending researchdatatoR&Dinstitutionsthatcanimprovethequalityoftheirdecisionsby10-20%.’
MethodspaceMethodspace is described in Deliverable9.4.3. (Table 5).http://www.methodspace.com/
Methodspace is a non-profit free onlinenetwork of researchers, resources anddebates.
MendeleyMendeleyisdescribedinDeliverable9.4.3.(Table5).http://www.mendeley.com/
The platform serves two interrelatedcustomer segments with distinct offers.Theindividualresearcherisservedwithafreemiummodel,userscanupgradetheircloudspace foramonthly feestartingat4,99 USD. In addition, since May2012Mendeley partners withinformation management servicesprovider SWETS for the MendeleyInstitutional Edition which offers real-timeanalyticstolibrariesforafee.
MicrosoftAcademicSearch
It is described in Deliverable 9.4.3. (Table5).http://academic.research.microsoft.com
Microsoft Academic Search services areprovidedbyMicrosofttothepublicfreeofcharge. The user must complete theregistration process by providing withcurrent, complete and accurateinformation as prompted by theapplicable registration form. Forpurposes of maintaining and improvingMicrosoft Academic Search services,Microsoft may record activities atMicrosoftAcademicSearchsiteandmakeuseofcertaininformation.
Biowebspin It is described in Deliverable 9.4.3. (Table5).
Biowebspin identifies itself as a leadingprofessional network in Life Sciences,connecting academia and industry.Biowebspin is the platform to find andconnect with the right partners, and theplace to network, work, and look upinformation thanks to smart tools andboards.Inbetatestfromlate2012toearly2013, Biowebspin is now in the top 3 ofthe most visited websites worldwide inLifeSciences.BiowebspinSAwascreatedin 2013 in Switzerland (capital: CHF550,000).
LinkedIn It is described in Deliverable 9.4.3. (Table5).
LinkedIn is publicly held and has adiversifiedbusinessmodelwithrevenuescoming from member subscriptions,advertisingsales,andtalentsolutions
Page 78 of 82
D8.2.3Stakeholders,TechnicalandFinancialEvaluation
Competitor BriefDescription Businessmodel
DIRECT2Experts It is described in Deliverable 9.4.3. (Table5).
TheDIRECT2Expertsnetworkisopentoall biomedical institutions, is a pilotproject facilitated by the ResearchNetworking Working Group of the NIH-supported Clinical & TranslationalScienceAward(CTSA)
BioMedUSA BioMedUSA is described in Deliverable9.4.3.(Table5).
Theopenaccessserviceisentirelyfreeofchargetoallscientistsandadministratorsatnon-profitresearchinstitutionsaswellas at commercial biomedicalorganizations.
Table 37: Pilot 4 Competitors and business models.
4.5.2 Insights for the formulation of realistic business and sustainability plans Theaforementionedplayersareavitalsignthatcollaborationcan’tbeestimatedhighenough.Asastrongtrendinsciences,subjectsgetmoreandmorecomplexandcannotbeinvestigatedby one genius brain alone. The average number of authors of high-end publications (Nature,Science,Celletc.)increasesconstantly.Itisofcrucialinteresttoprovidetoolsforcollaborationbeyond todays habits: basically sending unstructured emails to possible collaborators. Datamustbe integrated,automaticallyanalysedandamachineshouldbeable todrawconclusionsand suggest hypothesis to teams. InOpen Science Linkwepartly aim for suchnewways andgenerallyspeakingweseeahugemarket.OnechallengeablefactisthatResearchGateraised30million€fromTheGatesFoundation.Targetedcustomersandstakeholdersforpilot4havebeenanalysedinDeliverable9.4.3.(Table12).
4.6 Pilot 5:Research evaluation services
4.6.1 Analysis of different business models and competitors
Competitor BriefDescription Businessmodel
ThomsonReutersWeb ofScience
WebofScience®is described in Deliverable 9.4.3. (Table 6). http://thomsonreuters.com/web-of-science/
Thomson Reuters depends onmembership subscriptions offering asingle subscription model or anacademic/research institutionsubscription. Registered users need tosign in to have access to a web-basedplatform.
ThomsonReutersInCites
InCites™isdescribedinDeliverable9.4.3.(Table6).http://researchanalytics.thomsonreuters.com/incites/
Thomson Reuters depends onmembership subscriptions offering asingle subscription model or anacademic/research institutionsubscription. Registered users need to
Page 79 of 82
D8.2.3Stakeholders,TechnicalandFinancialEvaluation
Competitor BriefDescription Businessmodelsign in to have access to a web-basedplatform.
ResearchGate ItisdescribedinDeliverable9.4.3.(Table6).https://www.researchgate.net/
ResearchGate is a for-profit enterprise.Theyhavehadtworoundsoffundingandhaveinvestorsthathavepreviouslybeeninvolved with Facebook, LinkedIn andPayPal Researchgate has raised anotherinvestment round of $35M with moneyfrom Microsoft and Tenaya Capital,Dragoneer InvestmentGroupandThriveCapital.. Over the longer haul, the firmhopes to charge companies anduniversitiesforusingittoadvertisejobs,and to operate a marketplace forlaboratory materials. It has no plans topost other advertising, though, nor tochargeitsusersdirectly.
ArnetMiner ArnetMiner is described in Deliverable9.4.3.(Table6).http://arnetminer.org/
Arnetminer follows the the commercialopen source business model. In thismodel, the product is available for free,potential customers can download,install, and use the product withoutgettingintouchwiththecommercialfirmbehindtheproduct.Atthesametime,thefirmcan trackviadownload registrationand community forum activities who isactually using the product. A leadanalysis can then determine which oftheseusersmightbepotentialcustomers.More often than not, however, the firmwill wait until a non-paying user stepsforward and asks for a sales contact topurchase any of the services outlined inthe revenue generation section. Thus,leads emerge from the existing usercommunity, either voluntarily or byanalysis.
GoogleScholarMetrics
Google Scholar Metrics is described inDeliverable 9.4.3. (Table 6).http://scholar.google.lt
Itpresentsevaluationsof journalsbasedon popular evaluation metrics. TheOpenScienceLink models and evaluationservices could be feeding the platformenrichingtheprovidedevaluation.
Harzing’sPublish orPerish
Harzing’s Publish or Perish is describedin Deliverable 9.4.3. (Table 6).http://www.harzing.com/pop.htm
Itssustainabilitydependsonfundraisingfromacademiaandresearchinstitutions
MicrosoftAcademicSearch
Microsoft Academic Search is describedinDeliverable9.4.3.(Table6).
Microsoft Academic Search services areprovided byMicrosoft to the public freeof charge. The user must complete theregistration process by providing withcurrent, complete and accurateinformation as prompted by theapplicable registration form. Forpurposes of maintaining and improvingMicrosoft Academic Search services,
Page 80 of 82
D8.2.3Stakeholders,TechnicalandFinancialEvaluation
Competitor BriefDescription BusinessmodelMicrosoft may record activities atMicrosoftAcademicSearchsiteandmakeuseofcertaininformation.
Scopus
Scopus is described in Deliverable 9.4.3.(Table6).http://www.elsevier.com/online-tools/scopus
Due to high demand, scopus can onlyprovide pricing information to therepresentative of the institution whowould decide on purchasing asubscription. Institutional access isrequiredtofullybenefitfromScopus.
Altmetric ItisdescribedinDeliverable9.4.3.(Table6).http://www.altmetric.com
It is a relatively small company withfocus on engineering and domainknowledge. Started by wining intoElsevier’s Apps for Science competition.TheytookonadditionalinvestmentfromDigitalScience,andarestillapartof thegroup today.Usersnow include someoftheworld’sleadingjournals,fundersandinstitutions.
Table 38: Pilot 5 Competitors and business models. Targetedcustomersandstakeholdersforpilot5havebeenanalysedinDeliverable9.4.3.(Table13).
4.6.2 Insights for the formulation of realistic business and sustainability plans Researchevaluationisasimportantastrendanalysisandcanpartlybebasedonorprofitfromit.Asstatedabove,trendanalysisandtrenddetectionarekeywhenitcomestobefirstmover.The demand for reliable, scientific correct measures is obvious. Doing trend analysis onmediocredesignedindicatorsisrathersenseless.Sothefirstneedistocreatereliableindicators,whichgobeyondtheabovementionedcompetitors.OpenScienceLinkdoesexactlythis.Havingthem at hand the next issue which can be addressed is flexibility which should range fromresearcher,toresearchgroup,todepartment,toUniversity,tocity,countyandfinallystate.Alsoourmultidimensionalapproachtofocusforexampleonbioinformaticsandbeingabletoprofilee.g.EuropeagainstNorthAmericasetsOSLapartfromexistingtechnologies.Allinallitcanbenamedrealistic,thatourtechnologiesbringaddedvaluetocustomers.Therebythebusinessmodelcanbehighjackedbutextendedtomoreflexibility,moreaccuracyandmoresemanticcrosslinksforaddressingmeaninginsteadof“words”.Thinkableissellingsingletopicsinayearlysubscription(e.g.Bioinformatics,Europe,2015).SingleindividualsmaygetitcheapwhereInstitutionsgetahigherpricedproductandsoon.SeeingtheroughnumberfromThomsonReutersthemarketseemstobehugeandfastgrowing.WeestimatealsohereaCAGRof10-20%.
Page 81 of 82
D8.2.3Stakeholders,TechnicalandFinancialEvaluation
5 Summary and Conclusions Thisreportservesasayear3evaluationoftheOpenScienceLinkproject fromtheperspectiveofinterestedstakeholders,technicalachievementsandbusinessplans.Basedonstakeholdersviews,amongthemostimportantachievementsinthisperiodhasbeenthepublicationofthefirstdatapapersinBiomedicalDataJournal,andthedevelopmentofnewversion of the OpenScienceLink platform.With regards to the issues that should attractmoreattention in the future, the three major points were: the exploitation, the attraction of moreusers of the platform and knowledge transfer.Stakeholders have also provided a number ofsuggestions/criticisms that are reported in this document and will be taken into account forimprovement of the platform in the future. Based on previously defined Key PerformanceIndicators(KPIs)theactualprogressreportedhereshowsthatmostofthegoalswereachievedeveniftheinitialplanofexpectedprogresscouldbecharacterizedasveryambitious.Concerning the technicalevaluation,almostallof thecontacted testsweresuccessful,and thismatureversionoftheplatformwasshowntoworktechnicallyinaverysatisfactorymanner.Finally,fromafinancialpointofview,weevaluatedtheoverallmarketintowhichtheplatformand its services are positioned, the competitors and the business models that they use. Acomparison of theOpenScienceLink pilot serviceswith these products has been conducted inordertounveiladvantagesoftheOSLplatformovercompetitors.AdetailedanalysisofbusinessplanshasbeenincludedinaseparatedeliverableD9.3.
Page 82 of 82
D8.2.3Stakeholders,TechnicalandFinancialEvaluation
6 References
Andronikou, V., Tagarev, T., Tagareva, P., Pantos, C.,Mourouzis, I., Iervasi, G., . . . Alvers,M.R.(2013).D9.4.1:SustainabilityPlans.OpenScienceLinkConsortium.
Harnad,S.,Brody,T.,Vallieres,F.,Carr,L.,Hitchcock,S.,&Gingras,Y.(2004).Theaccess/impactproblemandthegreenandgoldroadstoopenaccess.SerRev30,310.
Kurata,K.,Morioka,T.,Yokoi,K.,&Matsubayashi,M.(2013).Remarkablegrowthofopenaccessin the biomedical field: analysis of PubMed articles from 2006 to 2010. PLoS ONE,8:e60925.
Laakso,M., & Björk, B.-C. (2012). Anatomy of open access publishing: a study of longitudinaldevelopmentandinternalstructure.BMCMedicine10(1).
Laakso,M.,Welling,P.,Bukvova,H.,Nyman,L.,&Björk,B.-C.(2011).TheDevelopmentofOpenAccessJournalPublishingfrom1993to2009.PLoSONE6(6):e20961.
OpenScienceLinkConsortium.(2013).OpenScienceLink:OpenSemantically-enabled,Social-awareAccesstoScientificData.EC.