aussrc 2019 radio data user survey report

32
Australian SKA Regional Centre AUSSRC-DSP-2020-0002 AusSRC Radio Astronomy Data User Community Survey Report DESIGN STUDY PROGRAM Date: 16.07.2020 Work package: WP5 Dissemination level: Public Published at: https://aussrc.org/documents/ Abstract Australian SKA Regional Centre (AusSRC) Design Study Program undertook a survey of the radio astronomy data user community in Australia for the purposes of obtaining a feedback that would further inform the development of AusSRC. The results based on seventy-four responses, conclusions and future actions obtained from the survey are detailed in this report.

Upload: others

Post on 15-Apr-2022

2 views

Category:

Documents


0 download

TRANSCRIPT

Page 1: AusSRC 2019 Radio Data User Survey Report

Australian SKA Regional Centre

AUSSRC-DSP-2020-0002

AusSRC Radio Astronomy Data User Community Survey Report

DESIGN STUDY PROGRAM

Date: 16.07.2020

Work package: WP5

Dissemination level: Public

Published at: https://aussrc.org/documents/

Abstract

Australian SKA Regional Centre (AusSRC) Design Study Program undertook a survey of the radio astronomy data user community in Australia for the purposes of obtaining a feedback that would further inform the development of AusSRC. The results based on seventy-four responses, conclusions and future actions obtained from the survey are detailed in this report.

Page 2: AusSRC 2019 Radio Data User Survey Report

1

I. COPYRIGHT NOTICE Copyright © Partners of Australian SKA Regional Centre, 2019. See aussrc.org.au for details of the AusSRC DSP program. AusSRC DSP (Australian SKA Regional Centre Design Study Program) is a program funded by the Australian Government, CSIRO, ICRAR and Pawsey Supercomputing Centre. AENEAS began in July 2019 and will run for 3 years. This work is licensed under the Creative Commons Attribution-Noncommercial 3.0 License. To view a copy of this license, visit http://creativecommons.org/licenses/by-nc/3.0/ or send a letter to Creative Commons, 171 Second Street, Suite 300, San Francisco, California, 94105, and USA. The work must be attributed by attaching the following reference to the copied elements: “Copyright © Australian SKA Design Study Project, 2019. See aussrc.org.au for details of the AusSRC DSP project”. Using this document in a way and/or for purposes not foreseen in the license, requires the prior written permission of the copyright holders. The information contained in this document represents the views of the copyright holders as of the date such views are published.

II. DELIVERY SLIP Name Date

From S. Kitaeff

Author(s) L. Quartermaine, S. Kitaeff

Reviewed by P.Quinn. S.Pearce

Approved by P.Quinn 16.07.2020

III. DOCUMENT LOG

Issue Date Comment Author

1 22.11.2019 Initial draft Louisa Quartermaine

2 15.05.2020 Developed draft for discussion Louisa Quartermaine Slava Kitaeff

3 1.06.2020 Final draft Louisa Quartermaine Slava Kitaeff

4 16.07.2020 Release Louisa Quartermaine Slava Kitaeff

Page 3: AusSRC 2019 Radio Data User Survey Report

2

IV. GLOSSARY PI Principal Investigator ALMA Atacama Large Millimeter/submillimeter Array API Application Programming Interface ARC ALMA Regional Centre ASKAP Australian SKA Pathfinder ASVO All-Sky Virtual Observatory ATNF Australia Telescope National Facility CASDA CSIRO ASKAP Data Archive CSP Central Signal Processor ESA European Space Agency ESDC European SKA Data Centre ESRC European SKA Regional Centre EVN European VLBI Network FFT Fast Fourier Transform FOV Field of View GSM Global Sky Model GUI Graphical User Interface HPC High Power Computing HPSO High Priority Science Objective HST Hubble Space Telescope HTC High Tech Computing IVOA International Virtual Observatory Alliance JIVE Joint Institute for VLBI ERIC LBA Long Baseline Array LOFAR LOw Frequency ARray LSM Local Sky Model MWA Murchison Widefield Array NASA National Aeronautics and Space Administration QA Quality Assessment SDP Science Data Processor SED Spectral Energy Distribution SG Science Gateway SIAP Simple Image Access Protocol SKA Square Kilometre Array SRC SKA Regional Centre SSAP Simple Spectral Access Protocol TAP Table Access Protocol VLBA Very Long Baseline Array VLBI Very Long Baseline Interferometry VO Virtual Observatory

Page 4: AusSRC 2019 Radio Data User Survey Report

3

VII. EXECUTIVE SUMMARY

AusSRC Design Study Program surveyed the radio astronomy data user community in Australia to obtain feedback that would further inform the development of AusSRC. The results based on seventy-four responses, conclusions and future actions obtained from the survey are detailed in this report. These results provide further detailing for the community requirements for the development of the Australian SKA Regional Centre.

Page 5: AusSRC 2019 Radio Data User Survey Report

4

Table of Context I. COPYRIGHT NOTICE 1

II. DELIVERY SLIP 1

III. DOCUMENT LOG 1

IV. GLOSSARY 2

VII. EXECUTIVE SUMMARY 3

Table of Context 4

1. Introduction 6

2. Goals of this document 6

3. Survey Rationale 6

4. Survey Structure 6

5. Method of Conducting the Survey 7

6. Survey Results 7

6.1. User Profiling 7

6.1.1. What are your areas of scientific interest 8

6.1.2. What position do you hold? 9

6.1.3. What project role do you hold? 9

6.1.4. What is your skill level working with radio astronomy data? 10

6.1.5. Which programming languages and techniques do you use often? 10

6.1.6. What is your skill level with these languages and techniques? 11

6.1.7. Do you participate in any SKA related activities? 12

6.2. Projects Profiling 13

6.2.1. With what types of projects are you currently involved? 13

6.2.2. What is the nature of your current collaborations? 13

6.2.3. Which radio astronomy facilities do you currently use? 14

6.2.4. What is the observational nature of the data you are using? 15

6.2.5. What is the most challenging stage in observational projects? 16

6.3. Data Profiling 17

6.3.1. What is the size of the data with which you interact? 17

6.3.2. Where is your data stored? 18

6.3.3. What are the biggest issues you encounter when working with radio data? 18

6.3.4. Biggest issues working with the data – elaborated 19

6.4. Software and Tools Profiling 20

6.4.1. What data reduction software do you use? 20

Page 6: AusSRC 2019 Radio Data User Survey Report

5

6.4.2. What software do you use to work with catalogues for statistical data analysis and cross matching? 20

6.4.3. What software do you use for visualisation and image processing? 21

6.4.4. What are the biggest issues that you encounter using astronomy software and tools? 21

6.5. Data Processing Profiling 23

6.5.1. Where do you process your data? 23

6.5.2. What type of data manipulation/interaction do you perform? 24

6.5.3. What is your level of satisfaction with the data and compute facilities you use? 25

6.5.4. What are the biggest issues you encounter when processing the data? 26

6.6. Expectations of AusSRC 27

7. Discussion 27

7.1. Users and survey confidence 27

7.2. Computing skills level 28

7.3. Projects 28

7.4. Data 29

7.5. Software and interaction with the data 29

7.6. Gaps and challenges 29

7.7. Expectations from AusSRC 30

8. Conclusion 30

9. Acknowledgements 30

VIII. REFERENCES 31

Page 7: AusSRC 2019 Radio Data User Survey Report

6

1. Introduction In November 2017, the first Australian SKA Regional Centre community workshop was held in Perth. The output of the workshop was summaries in two papers published on AusSRC website [RD1, RD2]. In 2019, in preparation for the second community workshop, this time held at Mt Stromlo (Canberra, ACT) in November 2019, the survey detailed in this report was proposed and conducted with preliminary results presented at the workshop for further discussion and consideration.

2. Goals of this document This document aims to outline the results, conclusions and future actions obtained from the recent survey that will assist in informing the future Australian SKA Regional Centre (AusSRC) definition.

3. Survey Rationale The aim of the survey was to build on the information provided by the previously collected data with a focus on current gaps in the services, support and infrastructure provided, and on how this information could be mapped onto the community expectations from the AusSRC. The results of the survey inform and update the definition and requirements of the future AusSRC.

This information will be of significant value in further updating the AusSRC white paper or developing a new paper.

4. Survey Structure In designing the survey, we aimed at sampling all important aspects of doing science with radio astronomy data. The idea was to profile the user, tools, data, and to take a snapshot of the field now in order to repeat it two years to see what has changed. Thus, the survey contains the following six sections:

1. User Profiling - this survey section sought information about a user of radio astronomy data, such as areas of scientific interest, position and role(s) and skill level working with radio data as well as various computing skill levels.

2. Projects Profiling - this survey section sought information on the projects with which users are involved, such as type of projects, nature of the collaboration, radio astronomy facility used. In this section we have also tried to identify some of the common observational project related challenges.

Page 8: AusSRC 2019 Radio Data User Survey Report

7

3. Data Profiling - this survey section sought information about the data and some specific challenges related to it.

4. Software and Tools Profiling - this survey section sought information on the type of software and tools used to process and interact with radio astronomy data, such as data reduction, statistical analysis, cross matching of catalogues, visualisation and image processing, as well as identifying the issues encountered when using those software and tools.

5. Data Processing Profiling - this survey section sought information on where users process their data, what type of data manipulation and interaction they perform, their satisfaction with data and compute facilities used, as well as the issues they encountered when processing data.

6. Expectations from AusSRC - this section requested users to rate the importance of what AusSRC should be providing in the future.

5. Method of Conducting the Survey The survey was loaded into Qualtrics survey software platform kindly provided to us by the University of Western Australia. The survey contained 33 questions in total, and was conducted fully online. The link to the survey was sent out to the members of the Astronomical Society of Australia, twice, 6 weeks apart. Seventy-four survey responses were received, which represents approximately 60 to 70% of the member of Australian radio astronomy science community.

6. Survey Results This section of the report details the results and key observations for each of the six-survey sections described in 4.

6.1. User Profiling This survey section sought information about a user of radio astronomy data, such as areas of scientific interest, position and role(s) and skill level working with radio data as well as various computing skill levels.

Page 9: AusSRC 2019 Radio Data User Survey Report

8

6.1.1. What are your areas of scientific interest

Galaxies and active galactic nuclei18%

ISM14%

Transients10%

Cosmology and the high redshift universe

9%

Star formation and astrochemisty

8%

Galaxy clusters7%

Magnetism6%

Pulsars6%

Astrometry6%

Fundamental Physics

4%

Epoch of Reionisation…

Stellar evolutio…

Stellar and exoplanetary

science2%

Cosmic-rays2% Space geodesy

1%

Other4%

Page 10: AusSRC 2019 Radio Data User Survey Report

9

6.1.2. What position do you hold?

6.1.3. What project role do you hold?

Staff or Faculty member

54%Postdoc

29%

Student17%

Project Member

40%

Principal Investigator35%

Co-Investigator25%

Page 11: AusSRC 2019 Radio Data User Survey Report

10

6.1.4. What is your skill level working with radio astronomy data?

6.1.5. Which programming languages and techniques do you use often?

Expert 57%

Intermediate35%

Beginner8%

Python41%

C12%

Parallelisation11%

C++8%

IDL8%

Fortran6%

Javascript3%

Matlab/Octave3%

Perl3%

Cuda3%

R1%

Java1%

Page 12: AusSRC 2019 Radio Data User Survey Report

11

6.1.6. What is your skill level with these languages and techniques?

24%

18%

15%

8%

6%

7%

2%

2%

3%

51%

32%

27%

21%

26%

22%

16%

15%

8%

5%

25%

24%

35%

48%

34%

35%

21%

41%

23%

36%

0%

26%

24%

23%

34%

37%

60%

41%

68%

60%

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

Python

C

Fortran

IDL

C++

Parallelisation

Javascript

Matlab/Octave

Java

R

Percent of Responses (%)

Prog

ram

min

g La

ngua

ge o

r Tec

hniq

ue

Expert

Intermediate

Beginner

None

Page 13: AusSRC 2019 Radio Data User Survey Report

12

6.1.7. Do you participate in any SKA related activities?

SKA Science Working Group & Focus

Groups44%

I'm not involved in SKA activities

37%

SKA Pre-Construction/B

ridging11%

SKA SRC8%

Page 14: AusSRC 2019 Radio Data User Survey Report

13

6.2. Projects Profiling

6.2.1. With what types of projects are you currently involved?

6.2.2. What is the nature of your current collaborations?

Surveys30%

Observations of specific sources

27%

Multi-wavelength observations

22%

Archive mining11%

Reprocessing publicly available data

10%

Equally domestic and

International66%

Mostly Australian

22%

Mostly International

12%

Page 15: AusSRC 2019 Radio Data User Survey Report

14

6.2.3. Which radio astronomy facilities do you currently use?

ASKAP19%

ATCA18%

MWA12%VLA

10%

Parkes9%

ALMA6%

VLBI6%

GMRT5%

Mopra4%

MeerKAT3%

Molonglo3%

Arecibo2%

LoFAR2% WSRT

1%

Page 16: AusSRC 2019 Radio Data User Survey Report

15

6.2.4. What is the observational nature of the data you are using?

Continuum observations (survey)

20%

Continuum observations (pointed

observations)17%

Spectral line observations (survey)

16%

Spectral line observations (pointed

observations)14%

Time domain (pulsars, FRB) (pointed observations)

10%

Polarimetry (pointed

observations)9%

Time domain (pulsars, FRB) (survey)

8%

Polarimetry (survey)

6%

Page 17: AusSRC 2019 Radio Data User Survey Report

16

6.2.5. What is the most challenging stage in observational projects?

Processing data58%

Preparing publications26%

Forming the project11%

Preparing telescope proposal

5%

Page 18: AusSRC 2019 Radio Data User Survey Report

17

6.3. Data Profiling

6.3.1. What is the size of the data with which you interact?

3.47.5

11.3

3.57.1

12.1

18.9

29.0

12.3

23.2

5.2

27.6

20.8

22.6

17.5

23.2

8.6

13.8

17.0

19.4

28.1

21.4

25.9

20.7

17.0

11.3

14.0

14.3

10.3

8.6 1.9

1.6

10.5

1.8

13.8

3.8

5.3

3.6

6.9

1.9

1.6

1.8

15.5

1.7

1.6

1.813.8 12.1 11.3

1.65.3 3.6

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Largest totalvolume of

observationaldata

Intermediatedata products

Intermediatedata products

(retain)

Individual datasets

Biggestindividual data

sets

Final dataproducts

Perc

ent o

f Res

pons

es (%

)

I don't know

Larger than 1 PB

500-1000 TB

100-500 TB

10-100 TB

1-10 TB

100 GB - 1 TB

10-100 GB

1-10 GB

Up to 1 GB

Page 19: AusSRC 2019 Radio Data User Survey Report

18

6.3.2. Where is your data stored?

6.3.3. What are the biggest issues you encounter when working with radio data?

Local Storage

35%

HPC/Data Centres

30%

Institutional data centre

29%

Public cloud6%

Processing38%

Storing24%

Moving20%

Data formats

8%

Meta data6%

Other4%

Page 20: AusSRC 2019 Radio Data User Survey Report

19

6.3.4. Biggest issues working with the data – elaborated The respondents were asked to elaborate on what they considered as the biggest issues encountered when working with radio astronomy data. The individual responses can be summarised into the following most common issues:

• The large volume of data requires the use of supercomputers which have become bottlenecks in the areas of processing, queue lengths, disk space, retrieving data and are considered a significant impediment to scientific research;

• The large amounts of data require more storage which is often not available, is limited and is not long lived;

• Moving and accessing large amount of data is problematic - time consuming and challenging;

• Manipulating, interacting and visualising large data sets can be problematic; • The lack of meta data is a problem for querying archives; • There is a lack of information available on the quality of data which leads to

labor intensive and time-consuming manual inspection of data.

Page 21: AusSRC 2019 Radio Data User Survey Report

20

6.4. Software and Tools Profiling

6.4.1. What data reduction software do you use?

6.4.2. What software do you use to work with catalogues for statistical data analysis and cross matching?

Miriad29%

Casa22%ASKAPSoft

13%

AIPS8%

Wsclean6%

RTS3%

Python3%

Other16%

Topcat57%

Python22%

IDL10%

CDS4%

Other7%

Page 22: AusSRC 2019 Radio Data User Survey Report

21

6.4.3. What software do you use for visualisation and image processing?

6.4.4. What are the biggest issues that you encounter using astronomy software and tools?

The respondents were asked to elaborate on what they considered as the biggest issues encountered when using astronomy software and tools. The individual responses can be summarised into the following most common issues:

• Software is poorly written; • Software is poorly documented and there is a lack of examples; • Software is poorly supported and maintained – often issues are encountered

when upgrading to newer versions; • Concern that the current software packages will not be capable of supporting

the next generation of telescopes; • Lack of support for different platforms; • Lack of flexibility and support for certain use cases often results in bespoke

software creation; • Lack of HPC optimization support; • Lack of funding for long-term developer positions and lack of support from

management to provide support, training and time to improve coding skills.

The following graph shows the distribution of the responses recorded in the list above.

DS921%

Karma15%

Miriad10%Aladin

9%

CASA viewer8%

CASA6%

Python6%

Aladin Lite4%

AIPS3%

Other18%

Page 23: AusSRC 2019 Radio Data User Survey Report

22

Software is poorly supported and maintained

21%

Software is poorly documented and there is a

lack of examples18%

Software is poorly written11%

Lack of flexibility and support for certain use

cases which often results in bespoke s/w creation

9%

Support for large datasets9%

Lack of support for different platforms

7%

Lack of funding for long-term developer positions and lack of support from

management to provide support, training and time to improve coding skills

7%

Data format compatiblity

7%

Concern that the current software packages will not be capable of

supporting the next generation of telescopes

5%

Lack of HPC optimisation support

2% Installation issues2%

Memory limitations2%

Page 24: AusSRC 2019 Radio Data User Survey Report

23

6.5. Data Processing Profiling

6.5.1. Where do you process your data?

Laptop/Desktop29%

Pawsey21%

Local computational

cluster19%

Institutional HPC resource

16%

OzSTAR10%

NCI2%

Other3%

Page 25: AusSRC 2019 Radio Data User Survey Report

24

6.5.2. What type of data manipulation/interaction do you perform?

Calibration15%

Visualising15%

Imaging15%

Flagging14%

Statistical analysis13%

Source finding12%

Cross matching9%

Machine Learning4%

RM synthesis3%

Page 26: AusSRC 2019 Radio Data User Survey Report

25

6.5.3. What is your level of satisfaction with the data and compute facilities you use?

13.710.2 10 9.8

5.9 4.2 4.1

27.5

26.5

36

43.1

41.2

25.030.6

41.2

32.7

20

23.5

17.6

43.8

42.9

15.7

18.4 26

21.6

23.5

22.9 12.2

2.0

12.28

2.0

11.8

4.2

10.2

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

User support Reliability Availability Computespeed

Data storagecapacity

Job queuing Storage toprocessing

data staging

Perc

ent o

f Res

pons

es (%

)

Very dissatisfied

Dissatisfied

Neutral

Satisfied

Very satisfied

Page 27: AusSRC 2019 Radio Data User Survey Report

26

6.5.4. What are the biggest issues you encounter when processing the data?

When asked to provide further details on what respondents considered the biggest issues encountered when processing the data, 30 individual responses were received. These responses can be grouped into the following categories:

• Low reliability of HPC systems; • Not being awarded (enough or any) compute time; • Long time in the queue; • Issues with HPC filesystem (speed, accessibility of data); • Inability of current environment to accommodate future science requirements

– storage and processing requirements; • Long processing times (iterative and time consuming to achieve a satisfactory

standard during data processing); • Memory limitations (insufficient memory available in the system).

The graph below shows the distribution of the responses recorded in the list above.

Low reliability of HPC resources

26%

Not being awarded (enough or any) compute

time26%

Long time in the queue13%

Inability of current environment to

accommodate future science requirements

13%

Long processing times13%

Memory limitations

9%

Page 28: AusSRC 2019 Radio Data User Survey Report

27

6.6. Expectations of AusSRC Respondents were asked to rate the importance of a list of functions a future AusSRC

should provide.

7. Discussion

7.1. Users and survey confidence We estimate that the received seventy-four responses constitute approximately 60 to 70 percent of active radio telescope data users.

60 58

46

3632

148

30 32

46

50

40

46

36

6 88

10

22

18

28

2 44

18

16

2 2 2 4

12

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

ScienceArchive

Data Storage Computationalfacility

User support Specificastronomy

services

Facilitation ofcollaboration

Support forpreparing theSKA proposals

Perc

ent o

f Res

pons

es (%

)

Function

Not at allimportant

Only slightlyimportant

Neutral

Important

Very important

Page 29: AusSRC 2019 Radio Data User Survey Report

28

Responses presented in section 6.1.7 (Do you participate in any SKA related activities?) indicate that almost 63% of the respondents are one way or another involved in SKA related activities. The responses to questions in sections 6.1.2 (What position do you hold?), 6.1.3 (What project role do you hold?) and 6.1.4 (What is your skill level working with radio astronomy data?) give a high degree of confidence that this survey represents the view of the community highly experienced in working with radio astronomy data.

7.2. Computing skills level It is somewhat expected that the Python programming language is the most popular choice when processing data, being used by 41% of respondents. C/C++ ranked the second most popular language, used by 20% of respondents (see section 6.1.5 Which programming languages and techniques do you use often?). 75% of the respondents who are using Python estimate their skill level at Intermediate level or above. 50% and 32% of the respondents using C and C++, respectively, estimate their skill level as Intermediate or above. The important skill of parallelizing the code is only available to 11% of the respondents, and only less than one third of those (29%) feel confident in using it, with those who indicated an expert level, only 7% (see 6.1.6 What is your skill level with these languages and techniques?).

7.3. Projects 71% of the respondents use Australian telescope facilities in their observational projects (see 6.2.3 Which radio astronomy facilities do you currently use?). 78% of the respondents are collaborating internationally (see 6.2.2 What is the nature of your current collaborations? ), and 63% are involved in SKA related activities (see 6.1.7 Do you participate in any SKA related activities?). The range of projects is quite broad with the continuum observations being the most common type of observation (37%), followed by the spectral line observations (30%) (see 6.2.4 What is the observational nature of the data you are using?). The responses in Section 6.1.1 (What are your areas of scientific interest?) and Section 6.2.4 (What is the observational nature of the data you are using?) demonstrate the broad and uniform distribution of observational radio astronomy in Australia.

Page 30: AusSRC 2019 Radio Data User Survey Report

29

7.4. Data The responses in Section 6.3.1 (What is the size of the data with which you interact?) indicate that the users are already working with data sets of significant size – hundreds of gigabytes, many terabytes. The location of the data is more or less evenly divided between local storage (35%), HPC/Data centers (30%), and institutional data centers(29%). Only 6% of the respondents use public clouds to store their data (see 6.3.2 Where is your data stored?). Sections 6.3.3 (What are the biggest issues you encounter when working with radio data?) and 6.3.4 (Biggest issues working with the data – elaborated) detail the most common challenges encountered when working with data. They currently represent the impediments for more productive science with the SKA precursors. The scale of the SKA data will only exacerbate this situation, unless suitable infrastructure and software technologies are established.

7.5. Software and interaction with the data Section 6.4 Software and Tools Profiling of the survey provides a view on what software tools are currently important to the users. The vast majority of those tools are unable to support the SKA scale data sets, and are not providing a mechanism for client-server or remote interactions with the data. In addition, Section 6.4.4 (What are the biggest issues that you encounter using astronomy software and tools?) indicates that the most common problems with the software are related to: poor quality of code, insufficient documentation, lack of user and software support, lack of support for different platforms, lack of support for the HPC environment. Despite many users now working with large datasets, 29% of the users are still using desktop/laptop computer to process their data (see 6.5.1 Where do you process your data?). Pawsey and local computational clusters are used by 21% and 19% of the respondents respectively. 16% are using institutional HPC resources.

7.6. Gaps and challenges In answering 6.2.5 (What is the most challenging stage in observational projects?), 58% of the respondents have indicated that data processing is the most challenging part of their projects. 38% of the respondents also named “data processing” as the most challenging part of working with data in 6.3.3 (What are the biggest issues you encounter when working with radio data?). In Section 6.5.3 (What is your level of satisfaction with the data and compute facilities you use?) the respondents were asked to estimate their satisfaction with the data and computation facilities that they are using. The levels of satisfaction, vary from 29.2% for the job queuing to 52.9% for the compute speed. The levels of dissatisfaction, however, vary from 17.2% for the user support to 35.3% for the storage capacity. Amongst the largest problems 26% of the respondents are dissatisfied with the reliability of HPC and data

Page 31: AusSRC 2019 Radio Data User Survey Report

30

resources, and 26% are not receiving enough or any compute time on HPC. These are important indicators where AusSRC could position itself to improve the experience of users in processing data.

7.7. Expectations from AusSRC Section 6.6 indicates that significant expectations from AusSRC exist in the community. No proposed areas scored less than 46% (Support for preparing the SKA proposals) as being important and very important. The areas that have scored 90% and above or just slightly below includes science archive, data storage, computational facility, and user support. 72% of the respondents also believe that specific astronomy services are also an important area of development for AusSRC. 60% of the respondents believe that AusSRC should play a role in facilitating collaboration.

8. Conclusion The survey was conducted with the primary objective of taking a snapshot of the radio astronomy data user community in Australia in the context of the SKA and its precursors to inform the development of the Australian SKA Regional Centre. Designing the survey, the authors aimed to achieve the following:

• Identify the current gaps in the services, support and infrastructure that is used today to store and process radio astronomy data;

• Map existing needs and expectations in the community onto the requirements from AusSRC and

• Assist with identifying the requirements for the global network of SRCs. Seventy-four survey responses were received with good coverage of project roles (from project leaders to students), high qualification of data users, as well as the broad and reasonably uniform distribution of research interests covering many fields of observational radio astronomy. It sets confidence in the statistics obtained through the survey. It was observed that the respondents have good connections to SKA activities, which allowed linkage between the findings and the objectives and requirements for the future Australian SKA Region Centre and its current design study activities. The survey indicates the current existence of a broad range of types of observational projects, as well as a broad range of telescope engagement, both nationally and internationally. The survey indicates several gaps and challenges in making science with today's radio data:

• Processing is a significant challenge across all stages of the data path, including, moving and storing large volumes of data, as well as the computational reduction of this data.

• HPC support of existing software is minimal, and maintenance of the software is often poor.

Page 32: AusSRC 2019 Radio Data User Survey Report

31

• More than 2/3 of the users are using HPC facilities to reduce data; however, the lack of dependable/dedicated HPC is a significant issue.

The survey reveals the expectation in the community that the SRCs are to be an operational resource for the SKA Archive, Store, and processing needs primarily, but not the individual research centres. This survey is a snapshot of the current state. It would be reasonable to expect that with the ASKAP surveys now underway, the development of new software tools, and the refreshed infrastructure at Pawsey, the focus and challenges may shift over time. AusSRC Design Study Program needs to track and follow such changes; therefore, we anticipate taking another such a snapshot survey in November 2021.

9. Acknowledgements The authors would like to thank all the respondents for taking time in answering the survey questions. We would also like to thank AusSRC Management Committee for supporting this work.

VIII. REFERENCES [RD1] AusSRC White Paper, 2018, https://aussrc.org/wp-content/uploads/2019/05/AusSRC-White-paper-v1.pdf [RD2] AusSRC Community Workshop: Summary of Discussions, 2017, https://aussrc.org/wp-content/uploads/2019/05/Australian-SKA-Regional-Centre-Workshop_-Summary-of-Discussions.pdf