accelerator for apache spark interface specification ?· document accelerator for apache spark –...

Download Accelerator for Apache Spark Interface Specification ?· Document Accelerator for Apache Spark – Interface…

Post on 04-Jun-2018

216 views

Category:

Documents

0 download

Embed Size (px)

TRANSCRIPT

  • http://www.tibco.com

    Global Headquarters

    3303 Hillview Avenue

    Palo Alto, CA 94304

    Tel: +1 650-846-1000

    Toll Free: 1 800-420-8450

    Fax: +1 650-846-1005

    2006, TIBCO Software Inc. All rights

    reserved. TIBCO, the TIBCO logo, The

    Power of Now, and TIBCO Software are

    trademarks or registered trademarks of

    TIBCO Software Inc. in the United States

    and/or other countries. All other product and

    company names and marks mentioned in

    this document are the property of their

    respective owners and are mentioned for

    identification purposes only.

    Accelerator for Apache Spark

    Interface Specification

    23 August 2016

    Version 1.0.0

    This document outlines the interface specification for inbound and outbound

    messages for the Accelerator for Apache Spark

  • Document

    Accelerator for Apache Spark Interface Specification 2

    Revision History

    Version Date Author Comments

    0.1 10/04/2016 Piotr Smolinski Initial version

    0.2 18/04/2016 Piotr Smolinski

    0.3 06/06/2016 Piotr Smolinski

    1.0.0 23/08/2016 Piotr Smolinski Version for release

  • Document

    Accelerator for Apache Spark Interface Specification 3

    Copyright Notice

    COPYRIGHT 2016 TIBCO Software Inc. This document is unpublished and the foregoing notice is

    affixed to protect TIBCO Software Inc. in the event of inadvertent publication. All rights reserved. No

    part of this document may be reproduced in any form, including photocopying or transmission

    electronically to any computer, without prior written consent of TIBCO Software Inc. The information

    contained in this document is confidential and proprietary to TIBCO Software Inc. and may not be used

    or disclosed except as expressly authorized in writing by TIBCO Software Inc. Copyright protection

    includes material generated from our software programs displayed on the screen, such as icons, screen

    displays, and the like.

    Trademarks

    Technologies described herein are either covered by existing patents or patent applications are in

    progress. All brand and product names are trademarks or registered trademarks of their respective

    holders and are hereby acknowledged.

    Confidentiality

    The information in this document is subject to change without notice. This document contains

    information that is confidential and proprietary to TIBCO Software Inc. and may not be copied,

    published, or disclosed to others, or used for any purposes other than review, without written

    authorization of an officer of TIBCO Software Inc. Submission of this document does not represent a

    commitment to implement any portion of this specification in the products of the submitters.

    Content Warranty

    The information in this document is subject to change without notice. THIS DOCUMENT IS PROVIDED

    "AS IS" AND TIBCO MAKES NO WARRANTY, EXPRESS, IMPLIED, OR STATUTORY, INCLUDING

    BUT NOT LIMITED TO ALL WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A

    PARTICULAR PURPOSE. TIBCO Software Inc. shall not be liable for errors contained herein or for

    incidental or consequential damages in connection with the furnishing, performance or use of this

    material.

    For more information, please contact:

    TIBCO Software Inc.

    3303 Hillview Avenue

    Palo Alto, CA 94304

    USA

  • Document

    Accelerator for Apache Spark Interface Specification 4

    Table of Contents

    TABLE OF CONTENTS .............................................................................................................................4

    TABLE OF FIGURES .................................................................................................................................6

    TABLE OF TABLES ..................................................................................................................................7

    1 PREFACE ..........................................................................................................................................8

    1.1 PURPOSE OF DOCUMENT .............................................................................................................8

    1.2 SCOPE ........................................................................................................................................8

    1.3 REFERENCED DOCUMENTS ..........................................................................................................8

    2 EVENT CAPTURE AND EMISSION (KAFKA MESSAGES)............................................................9

    2.1 TRANSACTION .............................................................................................................................9

    2.2 NOTIFICATION ........................................................................................................................... 11

    3 RUNTIME STATE MAINTENANCE (HBASE) ............................................................................... 14

    4 RUNTIME DASHBOARD (LIVEVIEW DATAMART) ..................................................................... 17

    4.1 TRANSACTION .......................................................................................................................... 17

    4.2 TRANSACTIONITEMS ................................................................................................................. 18

    4.3 STORESUMMARY ...................................................................................................................... 18

    4.4 MODELSUMMARY ..................................................................................................................... 19

    4.5 WALLCLOCK ............................................................................................................................. 20

    5 DATA COLLECTION (HDFS STRUCTURES) ............................................................................... 21

    5.1 STREAMBASE TO FLUME ........................................................................................................... 22

    5.2 AVRO TO PARQUET ................................................................................................................... 24

    6 DATA ACCESS (REST/HTTP) ....................................................................................................... 26

    6.1 SERVICE LAYER ........................................................................................................................ 26

    6.1.1 Text format ......................................................................................................................... 27

    6.1.2 SBDF format ...................................................................................................................... 27

    6.2 /CATEGORIES ............................................................................................................................ 28

    6.3 /RANGES .................................................................................................................................. 28

    6.4 /TRANSACTIONS ........................................................................................................................ 28

    6.5 /PIVOT ...................................................................................................................................... 29

    6.6 /SQL ......................................................................................................................................... 30

    6.7 /ETL ......................................................................................................................................... 30

    6.7.1 /etl (GET) ............................................................................................................................ 30

    6.7.2 /etl (POST) ......................................................................................................................... 30

  • Document

    Accelerator for Apache Spark Interface Specification 5

    6.7.3 /etl/{jobId} (GET) ................................................................................................................ 30

    6.7.4 /etl/{jobId}/messages (GET) ............................................................................................... 30

    6.7.5 /etl/{jobId}/events (GET) ..................................................................................................... 30

    6.7.6 /etl/{jobId} (DELETE) .......................................................................................................... 31

    6.8 /MODELS (TRAINING) ................................................................................................................. 31

    6.8.1 /models/train (GET) ............................................................................................................ 31

    6.8.2 /models/train (POST) ......................................................................................................... 31

    6.8.3 /models/train/{jobId} (GET) ................................................................................................ 32

    6.8.4 /models/train/{jobId}/messages (GET) ............................................................................... 32

    6.8.5 /models/train/{jobId}/events (GET) ..................................................................................... 32

    6.8.6 /models/train/{jobId} (DELETE) .............................................................