d13.5: final testing report - ariadne...
TRANSCRIPT
D13.5: Final testing report
Authors: Federico Nurra, InrapBernard Pinglier, Inrap
Ariadne is funded by the European Commission’s 7th Framework Programme.
Theviewsandopinionsexpressedinthisreportarethesoleresponsibilityoftheauthor(s)anddonotnecessarilyreflecttheviewsoftheEuropeanCommission.
ARIADNED13.5:Finaltestingreport(Public)
Version:1.0(final) 24thJanuary2017
Authors: FedericoNurra,Inrap
BernardPinglier,InrapContributingPartners
RobertoScopigno,CNR-ISTI
BrunoFanini,CNR-ITABC
HollyWright,ADS
ARIADNED13.5:Finaltestingreport
3
TableofContents
DocumentHistory.......................................................................................................5
ExecutiveSummary....................................................................................................6
1 Introduction........................................................................................................8
2 RelatedWork....................................................................................................102.1 Firstreportonusers'needs(D2.1).............................................................................102.2 Secondreportonusers'needs(D2.2)........................................................................102.3 UserRequirements(D12.1)........................................................................................102.4 InfrastructureDesign(D12.2).....................................................................................102.5 InitialInfrastructureImplementationReport(D12.3)................................................112.6 InitialInfrastructureTestingReport(D12.4)..............................................................112.7 ServiceDesign(D13.1)................................................................................................122.8 InitialServicesImplementationReport(D13.2).........................................................122.9 InitialServicesTestingReport(D13.3).......................................................................122.10 Finalservicesimplementationreport(D13.4)............................................................13
3 Overviewofdesign,userequirementsandusers'specifications........................14
4 Servicesevaluated.............................................................................................184.1 AriadneCatalogue/Portal.........................................................................................184.2 ServicesprovidedontheAriadneCatalogue.............................................................18
4.2.1 Cataloguetextualsearch.....................................................................................................194.2.2 Spatialsearchanddisplay....................................................................................................194.2.3 Timelinesearchanddisplay.................................................................................................19
4.3 OtherServicescreatedwithintheARIADNEProject..................................................204.3.1 VisualMediaServices..........................................................................................................204.3.2 LandscapeServices..............................................................................................................21
5 Methodologyusedforevaluation......................................................................23
5.1 Scopeofthetesting....................................................................................................235.2 Generalaspectsonmethodology...............................................................................235.3 Testingscenarios........................................................................................................23
5.3.1 Evaluationquestionnaires...................................................................................................245.3.2 Groupsessions[workshop].................................................................................................26
5.4 Feedbacktotheimplementationteams....................................................................27
6 Evaluationresults..............................................................................................28
6.1 Generalfeedback.......................................................................................................28
ARIADNED13.5:Finaltestingreport
4
6.2 CatalogueServices(46answers)................................................................................306.3 VisualMediaServices(56answers)...........................................................................366.4 LandscapeServices(21answers)...............................................................................396.5 HelpSection...............................................................................................................416.6 FinalEvaluation..........................................................................................................42
7 Conclusions.......................................................................................................47
8 References.........................................................................................................49
9 Tableoffigures..................................................................................................50
10 AnnexI–QuestionnairefortheevaluationoftheARIADNEServices.............51
11 AnnexII–TestingscenariosResults...............................................................55
11.1 BasicCatalogueServices............................................................................................5511.2 AdvancedCatalogueServices.....................................................................................5511.3 VisualMediaServices.................................................................................................5611.4 OwnCloudServices....................................................................................................5811.5 LandscapeServices.....................................................................................................58
ARIADNED13.5:Finaltestingreport
5
DocumentHistory
Version Date Contributors Activity
0.1 November15th2016 FedericoNurra
BernardPinglier
Draftstructureofdocument
0.2 December6th2016 FedericoNurra Addedtestresults
Addedimages
0.3 December13th2016 BernardPinglier Drafttextreviewed
Modifiedlayout
0.4 December16h2016 RobertoScopigno Removedparagraph«Other
servicesprovided“asis”by
partners(notevaluated)»
0.5 January10th2017 FedericoNurra Added«Conclusions»
Addedtexton«AnnexII»
0.6 January12th2017 BrunoFanini Revised«LandscapeServices»
0.7 January20th2017 HollyWright QCReview
1.0 January24th2017 FedericoNurra Added«ExecutiveSummary»
Finaledit
ARIADNED13.5:Finaltestingreport
6
ExecutiveSummary
This report is a deliverable (D13.5 Final testing report) of the ARIADNE project (“AdvancedResearch Infrastructure forArchaeologicalDatasetNetworking in Europe”) fundedunder theEuropeanCommunity'sSeventhFrameworkProgramme.ItpresentstheresultsofworkcarriedoutinTask13.4Acceptancetestingwithinworkpackage13DevelopingIntegratedServices.
ThisdeliverablereportsTask13.4,whichisiscentredontestingtheservicesproducedinTask13.2andverifyingtheircorrespondencetodesign(Task13.1),userequirements(Task12.1)anduser specifications (Task 2.1 and Task 2.2). New or improved additions were approved andpassedtothenextTaskorreturnedtoT13.2foramendments.Testingwascarriedoutwithinaspecific testing environment prepared by the partners in charge of the task. A number ofsimulatedcontextsweresetup,relevanttoarchaeologicalresearch,andthebehaviourofthesystem compared with user expectations. Overall compliance was also verified withrequirements, and the performance of the system in real test conditions. These tests wereperformed regularly and theiroutcomes reported to the leadpartnerofTask13.2,providingnecessary feedback. The testing specifications and the routine testing mechanism will bedetailedinTask13.1.
Methodology
Theevaluationhasbeenimplementedintwocomplementarydirections:
• Usingpredefinedtestingscenariosduringspecifictrainingworkshopwithselectedtesters
• Usingopenevaluationquestionnaireswithpowertesters
Theevaluationcyclefollowedseveralsteps.
Preliminarytasks:
• Creationofthegroupofevaluators
• Preparationoftheevaluationschemeandreports
• Evaluation
• Analysisofresults
• Productionofastructuredandcommentedfeedbacktotheimplementationteamand(ifnecessary), implementationoftherequestedchangesandproductionofarevisedversionoftheservice/tool
Thetaskleaderhadperiodiccontactswithservicesdevelopers,concerningthebugsandissuespointedoutbythetesters.
The aim of the questionnaire was related to a specific service, to determine whether theservicemeetstheexpectationsoftheusers.
Thisincluded:
ARIADNED13.5:Finaltestingreport
7
• Theanswerstoprecisequestionsaboutusabilityoftheservice
• Opencommentsabouttheservice(usability,requestforimprovements)
• Areviewabouttheservice(similartowhatcanbefoundintheAppStore)
• Quantitative data about usage (e.g. number of downloads of an application,numberofclicks,numberoffilesuploaded,etc.)
Thepaneloftesters
AtINRAPapowertesterpanelwasformed,consistingof26people.Thepowertestersusedallthe services, answering the online questionnaire several times, following bug fixes and thedevelopmentofnewversions.TheonlyservicethatreceivedlessfeedbackthanexpectedwastheLandscapeService.Thisisduetohighertechnicalrequirementsneededbythetesters.
Eightexternaltesters,whoworkinthefieldof3Dmodelling,werecontactedtoevaluateVisualMediaServices.
The portal was tested by several partners. Some responded using the online questionnaire,others sent their informal feedback via email, or raised the critical issues during the«PortalUserGroup»Skypemeetings.
Results
Thequestionnaireanswersclearlyshowthattheserviceratingisveryhighfromtheevaluatorsand,overall,thattheperformanceshavemettheneedsofusers.Itshowssomedeficienciesinservices (such as the lackofmeasuringoptimized instruments, the lackof links between thedifferent services and the impossibility to download themajority of the elements present inservices).
Feedbackisextremelypositiveonthegeneralservices(average4.01/5.00)andthedevelopers’punctual interventions to fix thebugs foundby testershavecertainlyhelped toevaluate theservices as best as possible. Several testers have used the optional answers to suggestimprovementstothefacilities.
ARIADNED13.5:Finaltestingreport
8
1 Introduction
This report is a deliverable (D13.5 Final testing report) of the ARIADNE project (“AdvancedResearch Infrastructure forArchaeologicalDatasetNetworking in Europe”) fundedunder theEuropeanCommunity'sSeventhFrameworkProgramme.ItpresentstheresultsofworkcarriedoutinTask13.4Acceptancetestingwithinworkpackage13DevelopingIntegratedServices.
As stated in the Description ofWork (DoW) of the ARIADNE Project, themain objectives ofWP13are:• To analyse, select, design and deploy the service components of the integrated
infrastructureinordertoimprovetheprovisionoftheonlineservicestotheresearchers• Totakeintoaccountinthedesignandthedeploymentfurtherenhancementsoftheservice
architectureasrequiredbytheadditionalimplementationofresultsfromWP14-WP17
ThetasksoftheWP13are:
Task13.1–ServiceDesignandSpecifications
Task13.2–ServiceImplementation
Task13.3–Long-termPreservationServices
Task13.4–AcceptanceTesting
Thisdeliverable reports Task13.4,which is centredon: testing the servicesproduced in Task13.2andverifiestheircorrespondencetodesign(Task13.1),userequirements(Task12.1)andusers’ specifications (Task2.1andTask2.2).Newor improved servicesmaybeapprovedandpassed to thenextTaskor returned to13.2 foramendments.Testingwillbecarriedoutonaspecifictestingenvironmentpreparedbythepartners inchargeofthetask.Theywillsetupanumberofsimulatedresearchcontexts,relevantforarchaeologicalresearch,andwillcomparethe behaviour of the system with the expected one. They will furthermore verify overallcompliancewithrequirementsandtheperformanceofthesysteminsimulatedrealconditions.Such testswillbeperformedregularly.Theiroutcomeswillbe reported to the leadpartnerofTask 13.2 for the necessary feedback. The testing specifications and the routine testingmechanismwillbedetailedinTask13.1.
ARIADNED13.5:Finaltestingreport
9
Asshowninthefollowingfigure,takenfromD13.1,Task13.4isrelatedtothemajorityofworkpackagesandtaskswithintheARIADNEproject:
Figure1:ARIADNETasksandServiceDesign.
ARIADNED13.5:Finaltestingreport
10
2 RelatedWork
2.1 Firstreportonusers'needs(D2.1)
Themainobjectivesofthisreportweretoprovideevidenceonuserrequirementsofkeytargetgroups (“users”) of theARIADNEprojectwith regard to archaeological research data, and tocontributetobuildingaprospectiveusercommunity.ThisinformationsupportedtheARIADNEproject in taking informed decisions regarding the specification of the e-infrastructure andservices so that the integrated infrastructure is developed in a way that corresponds toperceivedandactual researchneeds. Themandatewas toprovideevidenceon these issues,notablythroughcollectingfeed-backfromtheusercommunitiesbywayofusersurveys.
2.2 Secondreportonusers'needs(D2.2)
Themainobjectiveofthisreportwastoprovideadditional,moredetailedevidenceaboutuserrequirements of key target groups (“users”) of the projectwith regard to the ARIADNE dataportal.Apanelofabout25researcherswereaskedtodescribeindetailtheiruseofdigitaldataarchives, to evaluate existing archives and other portals and to highlight useful features oftheseportalswhichcouldserveas“goodpractice”whencreatinganewresearchdataportal.This information supports the ARIADNE project in taking informed decisions regarding thespecification of the e-infrastructure and services so they are developed in a way thatcorrespondstoperceivedandactualresearchneeds.Themandatewastoprovideevidenceonthese issues,notably throughcollecting feed-back fromtheARIADNEcommunitybywayofausersurvey.
2.3 UserRequirements(D12.1)
Themain objective of this report was to understand the current landscape from which theARIADNEinfrastructureshouldbecreated,inordertoinformthedevelopmentoftheARIADNEportal and services. This landscape included thedata,metadata, ontologies and vocabulariesavailable foruse,alongwithanyassociated issuesof licensingandaccess, informedbyusers’needs.
2.4 InfrastructureDesign(D12.2)
Themaingoalofthisreportwastospecifyaresourceintegrationanddiscoverymechanismforuse in ARIADNE. The resources to be integrated were datasets and collections, GIS data,metadataschemas,ontologiesandvocabulariesavailablefromtheprojectpartners,aswellasinstitutions outside the ARIADNE consortium. The deliverable provided an overview of theARIADNEarchitecture,includingasummaryoftheconformanceofthearchitecturetothedataandstandardsrequirementssetoutinD12.1,aswellasthespecificationsoftheServicesofthe
ARIADNED13.5:Finaltestingreport
11
ARIADNE Infrastructure, presented in D13.1. Thiswas followed by a content analysis of themaincontenttypesdefinedinD12.1,andtheintegrationstrategyforthetwolevelsofcontent:the metadata integration, and the data integration. This will attempt to integrate selectedresources (datasets and/or metadata) from particular partners/data providers and providecross-search and access mechanisms to integrated resources, using a facetted search on“what”,“where”,“when”and“resourcetype”.
2.5 InitialInfrastructureImplementationReport(D12.3)
The main goal of this report was to present the initial infrastructure implementation ofARIADNE. The infrastructure is specifiedonD12.2 and included services such as the registry,vocabulary services, metadata enrichment services, preservation services, etc. with thefollowing components: a) an RDF store b) the first version of the portal c) the MOReaggregationinfrastructured)anElasticsearchcomponente)ametadataqualitymeasurementservicef)asetofenrichmentservicesTask12.3andthisreportfocusedonthedescriptionofcore services. The core services were described and implemented as individual servicecomponents in the overall architecture, and their complementary role was presented. Task12.3wasfollowedbyTask12.4Testing(whichisreportedinD12.4Initialinfrastructuretestingreport).Thiswas followedbyanotherroundof implementationandtesting inTasks12.5and12.6.TheAPIspecificationspresentedintheAnnexeswereespeciallyimportant,astheyallowdevelopers to build on the infrastructure and deploy services. The APIs presented are REST-basedandrequiredataformatssuchasJSON,XMLandRDF.
2.6 InitialInfrastructureTestingReport(D12.4)
ThetasksofWP12ImplementingInteroperabilityaimedtospecifyuserequirementsanddesignfortheARIADNEinfrastructureandtoimplementandtestthatdesign.Task12.4wascomprisedof testing the integrated infrastructurebuilt inTask12.3 Implementing Integration, basedonthe requirementsanddesignspecified inTask12.1UseRequirements and12.2 InfrastructureDesign. This document augmented the synthesis of requirements and design given in D12.2with the infrastructure implemented in D12.3, and tested the implementation against therequirements specified. Each requirementwas outlinedwith the infrastructural tools and/orservicesdesignedtomeetitandabriefoverviewoftheimplementationdocumentedin12.3.Recommendations were made based on this analysis. The general use requirements wereaddressedinthefirstinstance,followedbymorespecificrequirementsaddressingintegrationactivities.
The initial infrastructure implementation met many of the requirements specified. Someaspects not yet visible required further testing in Task 12.6 Final Testing Report. Someclarification of use requirements for the Preview Servicemay be necessary: D12.2 designs aservicethatpreviewssearchresults,asdescribedaboveandillustratedintheprototypeportal,asalist,onamapandtimeline(D12.2,24)butwhichcanalsopreviewindividualrecords.D12.3discussed the implementation of a Preview Service through which researchers can previewtheirowndatasets inanumberof formats,whichseemstobemorecloselyalignedwith the
ARIADNED13.5:Finaltestingreport
12
Registry (17). The Support Portal, which includes documentation and user guidance, wasapproachedholisticallyinordertoprovideclearandcoherentguidancetoavarietyofusersatdifferent points of interactionwith the infrastructure. Documentationwas required for eachmajor component of the infrastructure. The Registry input tool provides the necessaryfunctionality to enabledepositingofmetadata. Theuser experienceof theRegistry couldbeimprovedandmorecloselyguidedwithchangesindesign.Atableofrecommendationsfortheinfrastructureisincludedattheendofthisdocument.
2.7 ServiceDesign(D13.1)
This document specified the service design of the ARIADNE Portal, and provided a commonvision,auserperspectiveonthefunctionality,andaframeworktoidentify,discussandvalidatetherequirementsfortheunderlyingtechnicalservices.Assuch,theaudienceofthisdocumentwasbothtechnicalandnon-technical.
Thisservicedesignwascreatedusingthefollowing input:Task2.1Surveyofusers’needsandcommunity building, Task 12.1 Assessment of use requirements, an examination of existingportalservicesandTask13.4Acceptancetesting.Theservicedesignprovidedacommonvisionformulated in terms of principles, derived from themission statement in the Description ofWork.
2.8 InitialServicesImplementationReport(D13.2)
ThisdocumentgivesanoverviewoftheinitialimplementationoftheservicesoftheARIADNEinfrastructure,whicharetheobjectiveofWorkPackage13(WP13).WP13wasinformedbytheoutput of WP2 (mainly Task 2.1 User needs and community building and Task 2.2 SpecialInterestGroups,andparallelsthedataintegrationeffortinWP12.Moreover,theservicesmadeavailablewithinWP13 also incorporate those developedwithinWP14 throughWP16. Theseserviceswill includefunctionalitythat isalreadyavailable,andwillbeofferedtocommunitieswhomaynotcurrentlyhaveaccesstoit,butwillalsoincludenewfunctionalitycreatedadhocby thepartnerson thebasisof the requirements thatwerecollectedat thebeginningof theproject. Themainmodality of the implementation of this new functionality as a web-basedservice,althoughothertypesofservices(localtools,guidelines)wereintegratedaswell.
2.9 InitialServicesTestingReport(D13.3)
D13.3presentedtheinitialresultsofTask13.4(T13.4),withinWorkPackage13(WP13)
Thereportfocusedonthefollowingaspects:
• Overview of design, use requirements and user's specifications, the compliance withwhich the services have to be tested (see D13.1, D12.1 and D2.2 for amore detailedpresentation).
ARIADNED13.5:Finaltestingreport
13
• Servicestobeevaluated:functionaldescriptionandprogressreportforimplementationofthese(seeD13.2foramoredetailedpresentation).
• Methodologyusedforthetests.
• First testing results. Informal tests were conducted by Inrap among a community ofinternalpowerusers,showingagreatinterestintheserviceprovidedforvisualisationof3D images, but also the need for enhanced visualisation and analysis tools, especiallymeasurementtools.
2.10 Finalservicesimplementationreport(D13.4)
This Deliverable presented the final state of integration and implementation of servicesdevelopedinthecontextoftheARIADNEinfrastructure.Itpresentedindetailtheorganisation,thefeaturesandthedesignofalltheserviceswhichformtheARIADNEinfrastructure
ThisdocumentpresentedthefinallistofservicesdesignedandimplementedfortheARIADNEinfrastructure,whichwastheobjectiveofWorkPackage13(WP13).
WP13wasinformedbytheoutputofWP2(mainlyTask2.1Userneedsandcommunitybuildingand Task 2.2 Special Interest Groups), and parallels the data integration effort in WP12.Moreover,theservicesmadeavailablewithinWP13alsoincorporatedthosedevelopedwithinWP14throughWP16.
These services include functionalities that were already available, and are offered tocommunitieswhomaynothadaccesstothem,butdoalsoincludenewfunctionalitiescreatedadhocbythepartnersonthebasisoftherequirementsthatwerecollectedatthebeginningoftheproject.Themainmodalityoftheimplementationofthesenewfunctionalitiesistoprovideweb-basedservices,althoughothertypesofservices(localtools,guidelines)wereintegratedaswell.
ARIADNED13.5:Finaltestingreport
14
3 Overview of design, use requirements and users'specifications
Theusers'needsaresummarisedasfollows,derivedfromD2.2:
1.OverallfocusoftheARIADNEprojectisondatadiscoveryandaccessservices.
2. The primary focus of the design and interfaces of the data portal should be anoverview ofwhat data is accessible, including statistical information on quantity,types,anddistribution(e.g.country/area,period).
3. The portal should focus on the European/international dimension. Lack ofunderlyingresources(percountry,typeofdata,etc.)shouldnotbeseenasadeficit,but used to promote data mobilisation (e.g. implementation of national dataarchives).
4. Added value should also be created through linking data and publicationresources not held within the ARIADNE Registry (e.g. metadata of documentarchivesandopenaccesspublishers).
5. Linked Open Data (LOD) can play a core role for value generation, but furtheruptake of LOD principles by archaeological institutions and projects must beencouraged.
6.Inthedevelopmentofthedatasearch,accessandotherservices,membersoftheuser community must be thoroughly involved and regular feedback onimplementedsolutionssoughtbythewidercommunity.
7. User-focused development of the portal services and applications (relevance,usability,user-friendliness)shouldbeatthetopoftheproject’spriorities.
8. Services for websites for research communities in particular subjects orgeographicregions(e.g.alertsonrelevantdatasets)couldgreatlyexpandthereachofthedataportaland,inturn,promotefurtherdatamobilisation.
9. Full exploitation of the data resources (incl. metadata, conceptual knowledge)shouldbeenabledby interfaces forexternalapplications (e.g.awell-documentedAPI,OAI-PMHtarget,SPARQLendpoint).
10.Supportofe-research/scienceshould,inthefirstinstance,beprovidedthroughintegratingaccesstodataresourcesandbypointinguserstoexistingtoolsfordataextraction,processingandanalysis.
ARIADNED13.5:Finaltestingreport
15
Theuserequirements(D12.1)refertotherequirementsforthedesignandspecificationofthesubsequenttoolsandservicesnecessaryforintegration.D12.1producedrecommendationsforDatasets;MetadataStandards,SchemasandVocabularies;andAccessandSharingPolicies.
Datasets• Site andmonumentdatabases:Most European countries and/or regionshave them,
andcombiningthemmaybeusefulforcross-bordersearchingandgeo-location• Interventionactivity:Mayhavemultipleactivitiesassociatedwithageo-locatablesite,
whichmayallowlinkingofvariousactivitiestoasinglesiteormonument• Fieldworkdatabases:Usuallytoodiverse,soindividualdatabasesmaynotbeusefulfor
integration, but may be worth linking to intervention activities for bibliographicdiscovery
• Othercategoriesarequitespecific,butmaybeusefulforintegration:o ScientificDatabaseso ArtefactDatabaseso BurialDatabases
Balancedataqualityandquantity:specifyrequirementsthatdatasetshavetomeetinordertobeintegrated,preferablyusingformalcriteria.Therelationshipsbetweenthetypesofdataavailablefromthecontentprovidingpartnersandthe recommended integration activity to be designed within D12.2 are set out in the tablebelow.
MetadataStandards,SchemasandVocabularies• Theuseofinternationalstandardsforthedocumentationofexcavationsand
monumentssoastorenderittransparentandcomparableand,aboveall,makethemmoreinteroperable.
• Freeaccesstotools,particularlyfordatamapping,tomakeiteasiertoincorporatethesestandards,andofferingthemeansandguidanceforarchaeologiststodeposittheirdigitalrecordsinanappropriatearchive.
• Thesustainabilityofdigitaldatasetsmustalsobehighontheagenda
ARIADNED13.5:Finaltestingreport
16
TherelationshipsbetweenthewishesandconcernswithregardtometadataandtherecommendedtoolstobedesignatedordesignedwithinD12.2aresetoutinthetablebelow:
Tableshowingthewishesandconcernswithregardtodatastandards,categorisedbythetypeofschemaorvocabularywhichmayaddressthewishes,andthetoolswhichmayaddresstheconcerns.The+signifiesthelevelofimportance.AccessandSharingPolicies• Acommonmethodofdatacitationshouldbeestablishedforadoptionbypartners,and
promotedbyARIADNEtothearchaeologicalresearchcommunity.Academicrecognitionisanimportantmotivationforencouragingresearcherstoshareaccesstotheirdatasets
• AllocationofDOIsortheequivalenttodatasetsingestedtotheARIADNEinfrastructureshouldbeinvestigated.Thesystemusedshouldbecapableofidentifyingsub-setswithincollections.Persistentidentificationofdatasetsisimportantinunderpinningdatasharinganddatacitation
• Content itself (databases, document archives, images, 3D models, etc.) should beprovided to ARIADNE by content partners using the Creative Commons license suite(version4.0 ispreferred)under licensepermissionsagreedwith thecontentowner.CCBY is recommended for open access. CC BY SA or CC BY SA NC licenses may also beapplicable
• A Collection description (of the whole collection and subsets within the collection)should be published under a CC BY license for each dataset ingested to the ARIADNEinfrastructure
• Metadata records shouldbepublishedunderaCC0 license – toenable integrationofmultipledatasetswithinthemetadatarepository,supportresourcediscoveryandenablelinkedopendata
The ServicesDesign (D13.1) introduces the functionality of the services, derived from a userperspective,andthusgivesanimportantinputforacceptancetesting.Itpresentsasetofusecasesrepresentingvariousrealsituationsencounteredbymembersofthearchaeologicalcommunity:
ARIADNED13.5:Finaltestingreport
17
• Search andexplore theRegistry to look for archaeological datasets to bepreviewedand/ordownloaded
Theuse case is divided in several steps (navigation, entering searchparameters,displaying the results) and relates to several services e.g. portal access, timelinesearch,geographicalsearch,displayoftheresults,etc.
• PreviewdatatodeterminetherelevanceofthisdataforaspecificresearchworkThepreviewserviceandfunctionalitydependsonthenatureofthedata.
• Accessdatabydownloadingitforfurtherprocessing• Deposit data in the Ariadne infrastructure to allow it to be browsed and re-used by
fellowresearchersThe service requires a set of guidelines for depositing data and an archivecompatiblewiththedatatobedeposited.
• Search and access the service registry to discover tools and knowledge to supportresearch/datamanagementactivities
• PrepareandregisteranewcollectiontotheRegistry• Enrich VisualMedia Documents from one of the visual documents (3Dmodels, RTI
images, high resolution images, terrain models) stored in one of the cataloguesassociatedtoAriadnePortal
Provideavisualizationwindowshowingthevisualservicesthatcouldbeappliedtoavisualmedia
• Manage accounts to allow a securitymanager to handle and control authenticationandaccess
The security manager may have an overview of the users and eventually change theirattributes.
ARIADNED13.5:Finaltestingreport
18
4 Servicesevaluated
Thissectiongivesanoverviewoftheservicesevaluated.
4.1 AriadneCatalogue/Portal
ARIADNEprovides services forarchaeologists toenableaccess to the research infrastructure.TheARIADNEPortalprovidesthemainpointofaccessforsearchingandbrowsingdatasetsandnewservicesforprocessingandpublishingarchaeologicaldatasetsonline.
A beta version of the ARIADNE portal is available at http://portal.ariadne-infrastructure.eu/.TheportalbringstogetherexistingarchaeologicalresearchdatasetsfromARIADNEpartnerssothat researchers can browse and access the various distributed datasets for use in theirprojects.
Theportal also provides a point of access for the new services developedby the project forvisualmediaandlandscapedatasets.
Figure2:ARIADNEPortal-HomePage
4.2 ServicesprovidedontheAriadneCatalogue
The ARIADNE Catalogue is based on the ACDM (Ariadne Catalogue Data Model), whichdescribes the archaeological resourcesmade available by the ARIADNE infrastructure to theresearchers wishing to access and use them. The current version is ACDM 2.6, delivered onSept.26th,2015.
ARIADNED13.5:Finaltestingreport
19
4.2.1 Cataloguetextualsearch
TheUsercanbeginanewsearchbyusingthesearchbar.Alternatively,userscanbrowsetheCatalogueusingtheWhere/When/Whatfacets.
Figure3:ARIADNEPortal–Searchwindow
4.2.2 Spatialsearchanddisplay
The firstbrowse facet isadirect link to theARIADNEGeographicSearchMap.Userscan findresourcesbasedongeographiccriteriabyclickingonthespecificareaofinterest.
Figure4:ARIADNEPortal–Geosearchwindow
4.2.3 Timelinesearchanddisplay
Thesecondbrowsefacet istheARIADNETimeline.Userscanusetemporalcriteriatoretrieveresourcesconcerningaspecificperiod.
ARIADNED13.5:Finaltestingreport
20
Figure5:ARIADNEPortal–Timelinesearchwindow
4.3 OtherServicescreatedwithintheARIADNEProject
4.3.1 VisualMediaServices
TheARIADNEVisualMediaServiceprovideseasypublicationandpresentationon thewebofcomplexvisualmediaassets. It isanautomaticservicethatallowstheuploadofvisualmediafilestoanARIADNEserverandtotransformthemintoanefficientwebformat,makingthemreadyforweb-basedvisualisation.
http://visual.ariadne-infrastructure.eu
ARIADNED13.5:Finaltestingreport
21
Figure6:VisualMediaServices–HomePage
4.3.2 LandscapeServices
LandscapeServicesforARIADNEareasetofresponsivewebservicesthatincludelargeterraindataset generation, 3D landscape composing and 3Dmodel processing, leveraging powerfulopen-source frameworks and toolkits such asGDAL,OSGjs,OpenSceneGraph and ownCloud.Themain components include: the cloud service, the terrain generation service, the terraingalleryandthefront-endwebcomponentforinteractivevisualisation.
http://landscape.ariadne-infrastructure.eu
ARIADNED13.5:Finaltestingreport
22
Figure7:LandscapeServices–HomePage
ARIADNED13.5:Finaltestingreport
23
5 Methodologyusedforevaluation
5.1 Scopeofthetesting
Testingwascarriedoutontheservicesdescribedintheprevioussection.
Noevaluationwasconducted forpreviouslyexistingservicesoperatedbypartnersandmadeavailable through ARIADNE, only the newly provided access to these services through theARIADNEPortalhavebeenevaluatedasapartoftheARIADNEportalservice.
5.2 Generalaspectsonmethodology
Theevaluationhasbeenimplementedintwocomplementarydirections:
• Using predefined testing scenarios according to the following matrix, during specifictrainingworkshopswithselectedtesters
• Usingopenevaluationquestionnaires,bothwithpowertesters(selectedbyInrap,leaderofTask13.4andbyCNR,leaderofWP13)
Theevaluationcyclefollowedseveralsteps:
• Preliminarytasks:
o Creation of the group of evaluators, ~30 power testerswithin Inrap and ~10power testers outside (e.g. University of Cambridge, University of Lund,UniversityofTurin,etc.)
o Preparationoftheevaluationschemeandreports
• Conductingtheevaluation
• Analysingtheresults
• Producing a structured and commented feedback to the implementation team and (ifnecessary), and implementationof the requested changes andproductionof a revisedversion of the service/tool. The task leader has had periodic contacts with servicesdevelopers,concerningthebugsandissuespointedoutbythetesters.
5.3 Testingscenarios
ARIADNED13.5:Finaltestingreport
24
Eachtestingscenariowasappliedtooneorseveralservicesforwhichitwasrelevant,accordingtothefollowingmatrix(seeAnnexIIfortheresults):
SERVICES
CATALOGUE VISUALMEDIASERVICES LANDSCAPESERVICES
1.1 1.2 1.3 2.1 2.2 2.3 3.1 3.2 3.3
CatalogueSearch
SpatialSearch
TimelineSearch
3DModels RTI HighRes
ImagesOwnCloud 3DTerrain Terrain
Gallery
ACTIONS
Navigate
Search
View
Edit
GeoSearch
TimelineSearch
Upload
Conversion
Manage
Download
Embed
5.3.1 Evaluationquestionnaires
Contentofthequestionnaires
Theaimofthequestionnairerelatedtoaspecificservicewastodeterminewhethertheservicemettheexpectationsoftheusers.
Thisincluded:− Theanswerstoprecisequestionsaboutusabilityoftheservice
ARIADNED13.5:Finaltestingreport
25
− Opencommentsabouttheservice(usability,requestforimprovements)− Areviewabouttheservice(similartowhatcanbefoundintheAppStore)− Quantitativedataaboutusage(e.g.numberofdownloadsofanapplication,numberof
clicks,numberoffilesuploaded,etc.)
Themultilingual test formswerebeen createdusing the “Google Forms”platform, followingtheschemaattachedinAnnexII.
Figure8:Testingquestionnaire(GoogleForm)
Translation
The original questionnaireswere created in English and translated into French and Italian toprovide them to wider communities. The feedback received has been fully translated intoEnglish, including the comments that were useful for evaluation purposes, which andperiodicallytransferredtoservicedevelopers.
Operation
Thequestionnairewassenttoaselectedsetofusers,previouslyidentifiedas:
ARIADNED13.5:Finaltestingreport
26
− PersonswhoshowedaninterestforARIADNE(respondentstothesurveyconductedinWP2)
− Users of the specific service willing to answer questions online (tester panel withinInrap)
Thequestionnaire,accordingtotheschemepresentedinAnnexII,wasproducedonaGoogleFormandwillbesenttothelistoftesters.
Analysisofthedata
Theanalysiswasmadebythetaskleader.
TheresultsoftheperiodicanalysiswerecommunicatedbymailtotheWPleader,whohelpedtofixvariousbugsonthesystem(e.g.October2016:tiffbugfixonVisualMediaServices.«Tiff-especiallyLidar-imageprocessingwasplaguedbytifflibsupport,andhasbeenfixed»)
Particular attentionwasgiven to theglobal acceptanceand implementationof theproposedservices.
Attention was also focused on the proper functioning of services and the speed of theoperationscarriedoutbytesters.
5.3.2 Groupsessions[workshop]
AtestingworkshopwasorganisedinParisonMay30th2016.Itwasagroupsessionwithalmost30usersusingthedifferentservices.Itwasusefultocollectinformationabouttheusageoftheservice.Thefirstphaseconcernedafreeusageoftheservicesandafirstsetofanswers.Thesecond phase followed a web tutorial, which explained the detailed service functions.Afterward,a secondsetofanswerswascollected.Theworkshopparticipantshavealsobeenperiodicallyinvolvedinremotetestingsessions.
ARIADNED13.5:Finaltestingreport
27
Figure9:TestingWorkshop–Paris,May30th2016
5.4 Feedbacktotheimplementationteams
During the final year of project (after the services beta version release), there have beenperiodiccontactsbetweenthetask leaderandservicedevelopers.Thetask leader (Inrap)didnotadheretothequarterlyfrequencyplannedonD13.3InitialServicesTestingReport,astheflowofanswersfromthetesterswasnotsteady.Nevertheless,thecontactsweremadewhenappropriaterelatingtotheissuesandproblemsencounteredbytesters.
ARIADNED13.5:Finaltestingreport
28
6 Evaluationresults
Thischapterpresentstheresultsoftheonlinequestionnaires,withthesupportofexplanatorygraphicsbasedon123feedbackreportsfromabout50participants,structuredasfollows:
- AtINRAPapowertester’spanelwasformed,consistingof26people.Thepowertestersused all the services, answering the online questionnaire several times, following bugfixes and the development of new versions. The only service that has received lessfeedback than expected was the Landscape Service. This is due to higher technicalrequirementsneededbythetesters.
- Eightexternaltesters,whoworkinthefieldof3Dmodelling,werecontactedtoevaluateVisualMediaServices.
- The portal was tested by several partners. Some responded using the onlinequestionnaire,otherssenttheirinformalfeedbackviaemail,orraisedthecriticalissuesduringthe«PortalUserGroup»Skypemeetings.
6.1 Generalfeedback
Howwasthewebsitereached?
ARIADNED13.5:Finaltestingreport
29
Wastheservicedescriptionprovidedonthewebclear?
Ifnot,whynot?(mostrelevantanswers)
• Notabletofindanaccuratedescriptionofthefeatures
• LackofMultilingualapproach
• Itwouldbehelpfultohavemoreinfo
Wastheprovidedservicespeedsatisfactory?
ARIADNED13.5:Finaltestingreport
30
ServicesUsed:
6.2 CatalogueServices(46answers)
KindofSearch(multiplechoices):
ARIADNED13.5:Finaltestingreport
31
Howmanysearcheswerecarriedout?
Wastheresultsvisualisationsatisfactory?
Ifnot,whynot?(mostrelevantanswers)
• Lackofdata
• Difficulttounderstandwhichtermtousefortextualsearch
• Searchby“Subject”isunclear
• Lackofthesauriexplanation
ARIADNED13.5:Finaltestingreport
32
Werethepagenavigationcommandsclear?
IfNot,whynot?(mostrelevantanswers)
- Howtomoveonthemapwiththemouseisnotclearatall
- Difficulttounderstandthedifferencebetweenthetwoareasoftextualresearch,“startanewsearch”ontopand“newsearch”intheresultspage
- LackofMultilingualapproach
WereLinksusedtoreachtheoriginaldata?
ARIADNED13.5:Finaltestingreport
33
IfYES,didtheyworkproperly?
Weretheresultssatisfactory?
IfNot,whynot?(mostrelevantanswers)
- BrokenLinks
- MissingLinks
ARIADNED13.5:Finaltestingreport
34
Werefiltersusedforthesearch?
Whichfiltershavebeenused?
ARIADNED13.5:Finaltestingreport
35
Didthefilterworkproperly?
Wasiteasytofiltertheresults?
IfNot,whynot?(mostrelevantanswers)
- Selectionnotsatisfactory
- Textualsearchdoesn’tworkproperly
- Timelinesearchisn’tintuitive
ARIADNED13.5:Finaltestingreport
36
6.3 VisualMediaServices(56answers)
Whichdatawereuploaded?
Whatwasthetotalsize(inMb)oftheuploadeddata?
Resultsbetween1.6Mband122Mb(average:29.5Mb)
3DModels:
Wasthe3Dmodelvisualisationsatisfactory?
IfNot,whynot?(mostrelevantanswers)
ARIADNED13.5:Finaltestingreport
37
- Wrongvisualisationoflongtitleswithinthecatalogue
- Problemwithplyfilesover50Mb
- Colourprojectionneedssomeimprovement
Wasthenavigationspeedofthemodelsatisfactory?
Wasthemodelqualitysatisfactory?
IfNot,whynot?(mostrelevantanswers)
- Nogeoreferencing
- Nounitmeasure
- Lackofmetricprecision
ARIADNED13.5:Finaltestingreport
38
RTIImages
WastheRTIvisualisationsatisfactory?
100%“Yes”
Wastheresponsespeedofthelightsatisfactory?
100%“Yes”
Wasthereflectancemodelqualitysatisfactory?
100%“Yes”
HRImages
WastheHRIvisualisationsatisfactory?
100%“Yes”
Wasthezoomspeedsatisfactory?
100%“Yes”
Wastheimagedefinitionqualitysatisfactory?
100%“Yes”
ARIADNED13.5:Finaltestingreport
39
6.4 LandscapeServices(21answers)
Landscape
Wasthegalleryseen?
100%“Yes”
Wasthemodelvisualisationsatisfactory?
IfNot,whynot?(mostrelevantanswers)
- IssueswithSafariBrowseronMacOS
- Themovementsonthemodelaren’tintuitive
Haveanyprocessbeenrun?
100%“Yes”
Whatkindofoutputwaschosen?(Multiplechoice)
ARIADNED13.5:Finaltestingreport
40
NB:theDAEColladaexportsupportisafuturedevelopment
Whatkindofresolutionwasused?
WasanareadefinedwithanESRIshapefiles?
ARIADNED13.5:Finaltestingreport
41
IfYES,areyousatisfied?
100%“No”
IfNot,whynot?(mostrelevantanswers)
- Shapefilecutdoesn’twork
6.5 HelpSection
Wasthehelpsectionused?
IfYES,wasithelpful/satisfactory?
ARIADNED13.5:Finaltestingreport
42
IfYESbutitdidnotsolveissues,why?(mostrelevantanswers)
- Difficulttofindthe“helpsection”
- Notuseful
- Explanationsnotclear
Haveproblems/bugsbeenfoundwhenusingthesystem?
Ifproblems/bugswerefound,havetheybeenreportedtodevelopers?
100%“No”
NB:issueswerereportedtodevelopersthroughtheT13.4leaderandwerealwaysaddressed
6.6 FinalEvaluation
ARIADNED13.5:Finaltestingreport
43
Wastheserviceperformancesatisfactory?
IfNot,whynot?(mostrelevantanswers)
- Lackofmetrics
- Visualisationproblems
- NecessityofhavinganaccounttouploaddataonLandscapeServices
- Lackofconsistencybetweenservices
- Noteasilyusable
- Impossibletodownloadsearchresults
- LackofmetadatamanagementonVisualMediaService
- Lackofexporttools
- Lackofmeasurementtools
- Lackofanalysistools
Wasthesystemusedwhileconnectedto:
ARIADNED13.5:Finaltestingreport
44
GlobalJudgmentontheexperience:
Globalaverage:4.01/5.00
Catalogueaverage:3.90/5.00
VisualMediaaverage:4.16/5.00
LandscapeServicesaverage:4.02/5.00
Arethereanyfurthersuggestionsforthedevelopers?
ARIADNED13.5:Finaltestingreport
45
IfYES,what?(mostrelevantanswers)
- Optiontodownloadthesearchresultsonthecatalogue
- OptiontocreateaRSSfeedrelatedtoaspecificquery
- Optiontocreateemailalertrelatedtotheresourcesonthecatalogue
- Implementationofstatisticaltools
- Addnewkindof“sortby”inthecatalogue
- Improvetheergonomyofthesystem
- Accesstothemetadataforthe3Dmodels
- NewwidgetsandtoolsforVisualMedia(lights,sections,analysis…)
- Improvetheharmonisationoftheservices
- Addalegendonthemapsearch
- IncreasetheusableformatsonVisualMediaService
- NeedcomputationtoolsandspatialanalysistoolsfortheLandscapeServices
- AzoomfunctionforDEMdisplayafterbeingprocessedwouldbeuseful
- Improvethecolourprojection
- Possibilityofdefiningastartingview/coordinatesystem(VisualMedia)
Whichotherdatatypeshouldbemanagedbytheservices?
- SpatialdatafromtheCatalogue
- 3Dspatialdata
- Items’Metadata
ARIADNED13.5:Finaltestingreport
46
- GeoDB
- LinkbetweenVisualMediaItems,LandscapeModelsandCatalogue
- RawdataofphotogrammetryTerrainmodelcreation
- Archaeologicalimagescollection
- 3Dshapefiles
- 360°panoramasin.movformat
ARIADNED13.5:Finaltestingreport
47
7 Conclusions
The questionnaire’s answers clearly show that the service rating is quite high from theevaluators (§7.6 “Global Judgment on the experience”) and, overall, that the performanceshavemettheneedsofusers(§7.1“Wasthespeedoftheserviceprovidedsatisfactory?”),whileshowingsomedeficiencies inservices(suchasa lackofoptimisedmeasurement instruments,the lackof linksbetween thedifferent servicesand the inability todownload themajorityoftheelementspresentinservices).
It was taken into account in the results analysis, that most of the testers used broadbandconnections(>10Mbps).
Onlyasmallnumberoftestersusedthehelpsectionowingtothefactthat80%ofusersfoundnoproblemsintheuseofservices.
Ingeneral,theoriginsofthetestersweremostlyfromFrance,asINRAPwastheTaskLeaderofthe T13.4 and more than half of the responses came from testers who followed a specificworkshopinParis.
Accordingtotheusers,theservicesareingeneraleasytouse.
In detail, the largest number of answers related to the Visual Media Services; then to theCatalogue/PortalandfinallytotheLandscapeServices.
As for the Catalogue, it is evident that most testers have used the three different kinds ofsearch jointly (Text, Geo, Timeline - 64%) and only a small number used a single searchfunction.Theaveragenumberofsearcheswasnotveryhigh(~14peruser)butthiswasduetothe need to test the services in a reasonable amount of time in order to provide usefulfeedback.Thesearchservice in theCatalogue (Portal)was triedbymostof the testers (73%)whoevaluateditpositivelywithanaverageof3.90/5.00.Howtousetheservicewascleartomostofthetesters,andthenavigationandsearchtoolswereeffectiveandfunctional.
TheVisualmedia servicesweremainlyused foruploadingandviewing3Dmodels (56%)andHigh Resolution (HR) images (26%). This is probably due to a lack of knowledge about theprocesses used to create and use RTI images. The general acceptance of the Visual MediaServiceswas thehighest of the three services evaluated: 4.16/5.00. The satisfactionwith 3Dmodel users was quite high (71%) with peaks related to service speed (85%) and imagerenderingquality(81%).Theproblemsencounteredwererelatedtodifficultieswithuploadinglargesize3Dmodels(>50Mb)andthecolourrenderingofmodels.RegardingHRimages(13answers) and RTI images (10 answers) the global satisfaction was 100% approval, andemphasises,onceagain,theneedforarchaeologiststobeabletotakeadvantageoffreetoolsforimagemanagement.
TheLandscapeservices,despitehavingfewerevaluationsthantheothertwo,however,hadahighapproval rating (4.02/5.00). Thedisplayquality satisfied90%of the testers,withawideuse of terrain service for the construction ofmodels in JSONWebGL (navigable on common
ARIADNED13.5:Finaltestingreport
48
browsers,bothdesktopandmobile).ThesecondchoicewastheOBJformat,afreeformatfor3D model interchange. Most of the testers used a maximum resolution (60%) or mediumresolution (27%). One of the service tools (cutting with ESRI shape files) was still indevelopment,soitwasnotpossibletoevaluatethisfeature.
Toconclude,feedbackwasextremelypositiveonthegeneralservices(average4.01/5.00)andthedeveloperspunctualinterventionstofixbugsfoundbytestershelpedtheevaluationoftheservices to be the best possible. Several testers used the optional answers to suggestimprovements to the facilities: themost requested feature was the ability to download theresults from the Portal and to download objects from the Visual Media public galleries,followedbyrequestsrelatedtoimprovedmetadatamanagement(forallservices).Specificallyfor manipulating 3D models in visual media, which often requires the ability to use otherformats than .ply. Within the Landscape Services the lack of computation tools and spatialanalysistoolswashighlighted.Finally,fromtheperspectiveofthetesters’proposalsformoreservicestobedevelopedinthefuture,theyrequestedtoolsforthetreatmentof3DGISvectordata,processingof imagecollectionsandthemanagementofphotogrammetryterrainmodelcreationrawdata.Althoughproposedonlybyonetester,thesuggestionforthedevelopmentofapplicationsforthemanagementof360°panoramasin.movformatwasalsointeresting.
ARIADNED13.5:Finaltestingreport
49
8 References
ARIADNE,2012,Ariadne-Ariadne,http://www.ariadne-infrastructure.eu/
ARIADNE,2014,D2.1“Firstreportonusers’needs”,http://www.ariadne-infrastructure.eu/Resources/D2.1-First-report-on-users-needs
ARIADNE,2014,D12.1"UserRequirements",http://www.ariadne-infrastructure.eu/Resources/D12.1-Use-Requirements
ARIADNE,2014,D13.1"ServiceDesign",http://www.ariadne-infrastructure.eu/Resources/D13.1-Service-Design
ARIADNE,2015,D2.2"Secondreportonusers'needs",http://www.ariadne-infrastructure.eu/content/view/full/1188
ARIADNE,2015,D12.2"InfrastructureDesign",http://www.ariadne-infrastructure.eu/Resources/D12.2-Infrastructure-Design
ARIADNE,2015,D12.3"InitialInfrastructureImplementationReport",http://www.ariadne-infrastructure.eu/Resources/D12.3-Initial-Infrastructure-Implementation-Report
ARIADNE2015,D13.2"Initialservicesimplementationreport",http://www.ariadne-infrastructure.eu/Resources/D13.2-Initial-services-implementation-report
ARIADNE2016,D13.3"Initialservicestestingreport",http://www.ariadne-infrastructure.eu/Resources/D13.3-Infrastructure-services-testing-report
ARIADNE2016,D13.4"Finalservicesimplementationreport"[v.0.4-draftversion]
ARIADNED13.5:Finaltestingreport
50
9 Tableoffigures
Figure1:ARIADNETasksandServiceDesign. 9
Figure2:ARIADNEPortal-HomePage 18
Figure3:ARIADNEPortal–Searchwindow 19
Figure4:ARIADNEPortal–Geosearchwindow 19
Figure5:ARIADNEPortal–Timelinesearchwindow 20
Figure6:VisualMediaServices–HomePage 21
Figure7:LandscapeServices–HomePage 22
Figure8:Testingquestionnaire(GoogleForm) 25
Figure9:TestingWorkshop–Paris,May30th2016 27
ARIADNED13.5:Finaltestingreport
51
10 AnnexI–QuestionnairefortheevaluationoftheARIADNEServices
TESTERDATA
FirstName
[Text]
LastName
[Text]
Institution
[Text]
Country
[Text]
START
Howwasthewebsitereached?
[multiplechoice]
Google DirectLink AriadneSite Other
Wastheservicedescriptionprovidedonthewebclear?
YES NO
IfNO,why?
[Text]
Wastheprovidedservicespeedsatisfactory?
YES NO
IfNO,why?
[Text]
Usedservices
Catalogue VisualMedia Landscape
CATALOGUE VISUALMEDIA LANDSCAPE
ARIADNED13.5:Finaltestingreport
52
Kindofsearch Whichdatahavebeenuploaded? Hasthegallerybeenseen?
[multiplechoice] [multiplechoice] YES NO
Text Geo Timeline 3DModels RTIImages HRImages Wasthemodelsvisualizationsatisfactory?
Howmanysearcheshavebeendone? Whatwasthetotalsize(inMb)oftheuploadeddata? YES NO
[multiplechoice] [Integer] IfNO,why?
1-10 10-20 20-50 >50 [Text]
Wastheresultsvisualizationsatisfactory? Doyouwanttoanswertospecific
questionsabout3Dmodels? Haveanyprocessbeenrun?
YES NO YES NO YES NO
IfNO,why? Whatkindofoutputhasbeenchosen?
[Text] 3DMODELS [multiplechoice]
Werethepagenavigationcommandsclear?
Wasthemodelvisualizationsatisfactory?
1 2 3 4 5 6
YES NO IfNO,why? Whatkindofresolutionhasbeenused?
IfNO,why?
Wasthenavigationspeedofthemodelsatisfactory?
Low Medium High
[Text] IfNO,why? Hasbeenanareadefinedwithashpfile?
Werelinksusedtoreachtheoriginaldata? Wasthemodel
qualitysatisfactory?
YES NO
YES NO IfNO,why? IfYES,areyousatisfied?
IfYES,didtheyworkproperly? YES NO
YES NO DoyouwanttoanswertospecificquestionsaboutRTIImages? IfNO,why?
Weretheresultssatisfactory? YES NO [Text]
YES NO HavefilesinOWNCloudbeen
uploaded?
IfNO,why? RTIIMAGES YES NO
[Text] WastheRTIvisualizationsatisfactory?
Werefiltersusedforthesearch? IfNO,why? OWNCLOUD
YES NO Wastheresponsespeedofthelightsatisfactory?
Whichdatatypes
havebeenuploaded?
IfNO,why? [multiplechoice]
ADVANCED
Wasthereflectance
modelqualitysatisfactory?
DEM Raster Shp
Whichfiltershavebeenused? IfNO,why?
Whatisthetotalsize(inMb)oftheuploadeddata?
ARIADNED13.5:Finaltestingreport
53
[multiplechoice] [Integer]
Text Geo Timeline Doyouwanttoanswertospecific
questionsaboutHRImages? Wastheservice
globallysatisfactory?
Didthefilterworkproperly? YES NO YES NO
YES NO IfNO,why?
IfNO,why? HRIMAGES [Text]
[Text] WastheHRIvisualizationsatisfactory?
Hasitbeensimpletofiltertheresults? IfNO,why?
YES NO Wasthezoomresponsespeedsatisfactory?
IfNO,why? IfNO,why?
[Text] Wastheimagedefinitionqualitysatisfactory?
IfNO,why?
HELPSECTION
Hasthehelpsectionbeenused?Hasitbeenresolutive/helpful/satisfactory?
YES NO
IfYESbutitdidnotsolveissues,why?
[Text]
Haveproblems/bugsbeenexperimentedusingthesystem?
YES NO
Incaseproblems/bugshavebeenexperimented,havetheybeenreportedtothedevelopers?
YES NO
IfYES,hasananswerfromthedevelopersbeenreceived?
YES NO
FINALEVALUATION
Havebeentheserviceperformancessatisfactory?
YES NO
IfNO,why?
ARIADNED13.5:Finaltestingreport
54
[Text]
Havebeenthesystemusedwhileconnectedto:
[multiplechoice]
Highspeednetwork(>10MB) Ordinarynetwork(3-10MB) Lowspeedconnection(<3MB)
Globaljudgementontheexperience:
(*****)
Experiencedescription(optional)
[Text]
Areanysuggestionsforthedeveloperspossible?(Possiblesuggestionsforthedevelopers)
YES NO
IfYES,what?
[Text]
WhichotherdatatypeshouldbemanagedbytheServices?(Specifydatatype)
[Text]
WhichvisualizationorinteractionfeaturesshouldbeconsideredwhileextendingtheVisualMediaService?(Pleasespecify)
[Text]
Links:
Englishversion:https://goo.gl/forms/4HZE4J3vkcVvbWJi1
Frenchversion:https://goo.gl/forms/6G6DHUFRi9YpnPEq2
Italianversion:https://goo.gl/forms/hjYc96O55LIAzAUJ3
ARIADNED13.5:Finaltestingreport
55
11 AnnexII–TestingscenariosResults
11.1 BasicCatalogueServices
ScenarioGroup1 BasicCatalogueServices
Pre-requisites BasicICTskillsUserbrowsestheARIADNEPortal
Step Action Expectedresult Conclusion Validation1.1 Catalogue
NavigateVisualisationoftheentirecataloguewithinARIADNEPortal
Theportalallowstoaccesstotheentirecataloguebysearchingwithanemptyvalue
Yes
1.2 CatalogueSearch
Textualresearchonthecatalogue InthehomepageoftheARIADNEPortalthereisasearchwindowthatallowstextualresearchonthecatalogue
Yes
1.3 CatalogueView Visualisationofsearchresults ThecatalogueisrepresentedontheARIADNEportalasamultipagedynamictable(queryonelasticsearch).EachrecordhasitsownwebpagederivedtoARIADNERegistry
Yes
1.4 SpatialNavigate Navigatethecatalogueonawebmappingpage
Thegeolocationofthecataloguerecordsisdisplayableandnavigableonaleafletwebmappingpage(baseopenstreetmap)
Yes
1.5 SpatialSearch Searchforaspecificelementonawebmappingpage
Theresearchisavailablebyusingpanandzoomonaleafletwebmappingpageandbypushingonasearchbutton
Yes
1.6 SpatialView As1.4 As1.4 Yes1.7 Timeline
NavigateNavigatethecatalogueonatimelinegraph
Therepresentationofthetemporalattributeonagraphisavailableonajavascriptpage
Yes
1.8 TimelineSearch Searchforaspecifictimespanonatimelinegraph
Theresearchisavailablebyusingthemousedragonthejavascriptgraphpage
Yes
1.9 TimelineView As1.7 As1.7 Yes
11.2 AdvancedCatalogueServices
ARIADNED13.5:Finaltestingreport
56
ScenarioGroup2 AdvancedCatalogueServices
Pre-requisites BasicICTskillsBasicgeosearchskillsBasictimesearchskillsUserbrowsestheARIADNEPortalUseradvancedsearchontheARIADNEPortal
Step Action Expectedresult Conclusion Validation2.1 Catalogue
ParameteriseSearch
Specifictextualresearch(e.g.«roman») ThesearchwindowonthePortalhomepageallowsspecifictextualsearch,suggestingsimilartermsfromAATthesaurus
Yes
2.2 GeoParameteriseSearch
Specificgeographicresearch(e.g.bycoordinatesorbyplacename)
Itisn’tpossibletoinsertaspecificcoupleofcoordinatesortoinsertaplacenamefortheresearch.Thegeographicsearchisonlyavailablebyusingpanandzoom(mouseclickandmousescroll)
No
2.3 TimelineParameterisesearch
Specifictimespanresearch(e.g.byperiodnameorspecificdate)
Itisn’tpossibletoinsertaspecificdateortoinsertaperiodnamefortheresearch.Thetimelinesearchisonlyavailablebyusingmousedrag
No
11.3 VisualMediaServices
ScenarioGroup3 VisualMediaServices
Pre-requisites AdvancedimageprocessingskillsUserhasavisualmediadocument(3Dmodel,RTI,HRimage)
Step Action Expectedresult Conclusion Validation3.1 3DModels
NavigateVisualisationofthepublicregistryof3DmodelsuploadedontheVisualMediaService
Thenavigationoftheitemsisavailablefromthehomepageoftheservice(button«browse»).Itispossiblealsotoaddatypefilter
Yes
3.2 3DModelsView Visualisationofeach3DmodelpublishedontheVisualMediaService
Fromthebrowsepageitispossibletoclickoneachitemtovisualisethemodelona3DHOPpage
Yes
3.3 3DModelsUpload
Uploadtheirown3DmodelsontheVisualMediaServiceondifferentformats
Theuploadofthemodelsisavailablefromthehomepageoftheservice(button«upload»).Sofar,theonlyformatacceptedisthe.ply
Yes
3.4 3DModelsParameter
Changetheparameterof3Dmodelmanagement
Theserviceallowsitemmetadatatobemodifiedandtosetupthevisualisationwebpageof3Dmodel
Yes
3.5 3DModelsTransform
3Dmodeltransformationformat Theserviceallowstransformationofthe.plymodelsinadownloadablewebpagebasedon3DHOP(.zip).Nomoretransformationisavailable
Yes
3.6 3DModelsDownload
3Dmodelsdownload Thedownloadingofthemodelsisavailableonlyfortheirownuploadedmodels.Itisn’tpossibletodownload
No
ARIADNED13.5:Finaltestingreport
57
themodelsonthepublicgallery3.7 3DModels
Embed(onwebsite)
Embedding3Dmodelwebpageonitsownwebsite
The3DHOPwebpagecanbeembeddedonawebsiteasan«iframe»
Yes
3.8 RTINavigate VisualisationofthepublicregistryofRTIimagesuploadedontheVisualMediaService
Thenavigationoftheitemsisavailablefromthehomepageoftheservice(button«browse»).Itispossiblealsotoaddatypefilter
Yes
3.9 RTIView VisualisationofeachRTIimagepublishedontheVisualMediaService
Fromthebrowsepageitispossibletoclickoneachitemtovisualizetheimageonawebpage
Yes
3.10 RTIUpload UploadtheirownRTIimagesontheVisualMediaServiceondifferentformats
TheuploadoftheRTIimagesisavailablefromthehomepageoftheservice(button«upload»).Theformatsacceptedare:.ptmand.hsh
Yes
3.11 RTIParameter ChangetheparameterofRTIimagemanagement
Theserviceonlyallowstomodifyitemmetadata
Yes
3.12 RTITransform RTIimagetransformationformat Theserviceallowsthetransformationof.ptmand.hshimagesinadownloadablewebpagebasedon3DHOP(.zip).Nomoretransformationisavailable
Yes
3.13 RTIDownload RTIimagesdownload ThedownloadingoftheRTIimagesisavailableonlyfortheirownuploadedfiles.Itisn’tpossibletodownloadthemodelsonthepublicgallery
No
3.14 RTIEmbed(onwebsite)
EmbeddingRTIimagewebpageonitsownwebsite
The3DHOPwebpagecanbeembeddedonawebsiteasan«iframe»
Yes
3.15 HighResImagesNavigate
VisualisationofthepublicregistryofHRimagesuploadedontheVisualMediaService
Thenavigationoftheitemsisavailablefromthehomepageoftheservice(button«browse»).Itispossiblealsotoaddatypefilter
Yes
3.16 HighResImagesView
VisualisationofeachHRimagepublishedontheVisualMediaService
Fromthebrowsepageitispossibletoclickoneachitemtovisualizetheimageonawebpage
Yes
3.17 HighResImagesUpload
UploadtheirownHRimagesontheVisualMediaServiceondifferentformats
TheuploadoftheRTIimagesisavailablefromthehomepageoftheservice(button«upload»).Themostcommonimageformatsareaccepted
Yes
3.18 HighResImagesParameter
ChangetheparameterofHRimagemanagement
Theserviceonlyallowstomodifyitemmetadata
Yes
3.19 HighResImagesTransform
HRimagetransformationformat Theserviceallowsimagestobetransformed(indifferentformats)inadownloadablewebpagebasedon3DHOP(.zip).Nomoretransformationisavailable
Yes
3.20 HighResImagesDownload
HRimagesdownload ThedownloadingoftheHRimagesisavailableonlyfortheirownuploadedfiles.Itisn’tpossibletodownloadthemodelsonthepublicgallery
No
3.21 HighResImagesEmbed(onwebsite)
EmbeddingHRimagewebpageonitsownwebsite
The3DHOPwebpagecanbeembeddedonawebsiteasan«iframe»
Yes
ARIADNED13.5:Finaltestingreport
58
11.4 OwnCloudServices
ScenarioGroup4 OwnCloudServices
Pre-requisites AdvancedICTskillsUserhassomegeographicaldatasets(Raster+DEM+shpfile)
Step Action Expectedresult Conclusion Validation4.1 OwnCloud
NavigateFilesandFoldersvisualisationonawebcloudstorage
Theserviceallowsnavigationtofoldersandfilesunauserfriendlywebcloudstorage
Yes
4.2 OwnCloud(Accessand)View
Privateaccesstothewebcloudstorage Theserviceasksforausernameandapasswordfortheprivateaccesstoitsownwebstorage
Yes
4.3 OwnCloudUpload
Uploadfilesonawebcloudstorage Theserviceallowsuploadingofdifferentsizefilesinauserfriendlywebcloudstorage(draganddropandbutton“upload”)
Yes
4.4 OwnCloudManage
Managethefoldersandthefilesonawebcloudstorage
Theserviceallowsuserstomanage,rename,delete,copy,cut,paste,etc.bothfoldersandfilesinauserfriendlywebcloudstorage
Yes
11.5 LandscapeServices
ScenarioGroup5 LandscapeServices
Pre-requisites BasicGISskillsUserhasuploadedageographicaldatasetonOwnCloud
Step Action Expectedresult Conclusion Validation5.1 3DTerrain
Navigate3Dterrainmodelsnavigationonawebgallery
The3DterraingalleryisavailablefromtheLandscapeservicehomepage
Yes
5.2 3DTerrainView 3Dterrainmodelvisualisation Fromthepublicgallery,eachmodelisaccessiblefortheonlinevisualisation(bothfromdesktopandmobile)
Yes
5.3 3DTerrainParameter
Changetheparameterof3Dterrainmodelmanagement
The3Dterrainserviceallowsuserstomodifymodelmetadata,modelparameter(format,resolution,layers,etc.)
Yes
5.4 3DTerrainUpload
Uploadtheirown3DterrainmodelsontheLandscapeServiceondifferentformats
Theserviceallowstouploaditemsondifferentfileformatsbyusing“OwnCloud”(see4.1-4.4)
Yes
5.5 3DTerrainTransform
3Dterrainmodeltransformationformat
The3Dterrainserviceallowsuserstotransformtheirown3Dterrainmodelsondifferentformats(e.g.JSONWebGL,3DS,OBJ,FBX,etc.)
Yes
5.6 TerrainGallery As5.1 As5.1 Yes
ARIADNED13.5:Finaltestingreport
59
Navigate5.7 TerrainGallery
ViewAs5.1 As5.1 Yes
5.8 3DTerrainDownload
3Dterraindownload Thedownloadisavailableindifferentformatsforeachitempublishedonthepublicgalleryandfromitsownprivategallery(privateaccessvia“OwnCloud”)
Yes
5.9 3DTerrainEmbed(onwebsite)
Embedding3DTerrainwebpageonitsownwebsite
ThewebpagecontainingthemodelonJSONWebGLformatcanbeembeddedonawebsiteasan«iframe»
Yes