manager+agents - signiantinfo.signiant.com › rs › 134-qhz-485 › images › signiant... · in...

MANAGER+AGENTSPOWERFUL AUTOMATED SYSTEM-TO-SYSTEM ENTERPRISE FILE MOVEMENT

PRODUCT WHITE PAPER

http://www.signiant.com/?cc=wp=manager-agents

https://www.signiant.com/products/signiant-manager/?cc=wp-manager-agents

TABLE OF CONTENTSManager+Agents — Powerful Automated System-to-System Enterprise File Movement

Introduction ………………………………………………………………………………………………1

Technology and System Architecture ………………………………………………………………..2

System Components ………………………………………………………………………………..2

Storage Flexibility ………………..……………………………………..........................................2

Acceleration Technology ……………………………………….……………………………….......3

Security ………………………………………………………………….…………………………...5

Automated File Transfer Basics ……………………………………………………………………... 6

Network Configuration …………………………………………………………………….….........6

The Manager Console ………………………………………………………………….………......7

Creating Jobs …………………………………………………………………………….………....7

Inter-Company Transfers ………………………………………………………………….….........8

Growing File Support………………………………………………………………………….….....8

Common Use Cases …………………………………………………………………….......................8

Content Distribution …………………………………………………………..……………….........8

Content Contribution/Aggregation …………………………………………………….…..............9

Creative Workflows …………………………………………………………..………….………......9

Cloud Workflows ………………………………………………………………….…………….......9

Sports ……………………………………………………………………………………………....10

Applications Outside M&E …………………………………………..………………….…...........10

Resource Management Option …………………………………………………………………….....10

Integration with Other Systems…………………………………………………………………….....11

REST & SOAP APIs …………………….………………………………………………………......11

Automation Engine Option …………………………………………..….……………………........11

3rd Party Integrations …………………………….……………………………………………......12

Signiant Media Shuttle for Person-Initiated Transfers …………………………..........................13

Redundancy Options ………………………………….………………………………………………..14

Manager ………………………………………………………………………………………….....14

Agents ……………………………………………………………………………………….….......15

Relay Agents ………………………………………………….…………………………..……......15

Packaging Summary ……………………………………………………………………………….......15

Basic Functionality ……………………………………………………………………..….……......15

Add-on Options ………………………………………………………………………………......... 15

Signiant SaaS Products ……………………………………………………………………….........15

Conclusion …………………………………………………………………………………………….......16

1 | MANAGER+AGENTS PRODUCT WHITE PAPER | WWW.SIGNIANT.COM

INTRODUCTIONSigniant Manager+Agents (M+A) is the gold standard for scheduled, lights-out, automated transfer of large files between geographically distributed locations. Forming the core transport backbone of most major media companies, this powerful, secure enterprise software helps studios, broadcasters, cable networks and sports organizations move petabytes of data every day throughout their global content supply chains. By providing central control, visibility and tracking of all content deliveries, the Manager+Agents system serves as a hub for file-based workflows that span the globe.

In addition to the central role it plays in the Media & Entertainment sector, Manager+Agents is used by many businesses in other industries where there is a requirement for moving large unstructured data sets around the world.

The Manager+Agents (M+A) solution is designed to move large files over any public or private IP network, and to connect to a range of storage types – both on-premises and cloud. M+A software endpoints (known as Agents) are distributed throughout the customer’s ecosystem, and a logically centralized Manager orchestrates file movement between the Agents. Transfers between Agents take advantage of Signiant’s patented acceleration technology, which minimizes the impact of latency and congestion to ensure fast, reliable data transmission regardless of distance.

M+A’s unique architecture is capable of supporting thousands of endpoints from a central control point, offering the lowest TCO at scale of any solution in its class. In contrast with competing systems where an administrator needs to touch each endpoint for individual configuration, M+A’s policy-based architecture allows configuration information to be automatically propagated to an unlimited number of endpoints. This distinction is a huge differentiator for large-scale global deployments.

A feature-rich product, M+A provides administrators with granular control of file transfers. Daily content flow can be monitored via a configurable dashboard with alarms and status indicators. With broad operating system support, FIMS-compatible SOAP and REST APIs for integration with other media applications, and optional modules for workflow automation and network resource management, no system is more flexible, powerful or reliable.

Powerful Automated System-to-System Enterprise File Movement



https://www.signiant.com/products/signiant-manager/?cc=wp-manager-agents


TECHNOLOGY AND SYSTEM ARCHITECTURESYSTEM COMPONENTSA basic Manager+Agents deployment comprises three distinct

software elements:

➜➜ Manager: Manager software is installed on a server behind the

customer’s firewall. The Manager performs all administration,

control and reporting functions and orchestrates transfers. Users

configure the system, schedule transfers, and monitor activity via

the Manager’s web-based interface, which can be accessed from

anywhere in the world.

➜➜ Agents: Every endpoint in a M+A deployment requires Agent

software running on a server near the target or source storage,

running either as a single node or in a load-balanced pool for

enhanced availability and throughput. Signiant Agents execute

the tasks that the Manager defines – primarily accelerated file

movement and interoperability with third-party software and

systems. File movement occurs from Agent to Agent.

➜➜ Relay Agents: Relay Agents can be configured to transfer

content between public and private networks, transversing

firewalls while maintaining network security. The two types of

Agents are licensed differently, but the only functional different

between a Relay Agent and a transfer Agent mentioned above

is that a Relay Agent cannot be used as a transfer endpoint.

However, a transfer Agent can be used as a relay point.

A sample system deployment is show above, with some Agents

operating inside the corporate network and others at partner

locations.

STORAGE FLEXIBILITY

Manager+Agents was originally designed for traditional on-premises

file storage and was used to transfer content between global data

centers. With the proliferation of storage choices, the product has

evolved to accommodate other forms of storage. There is now an

M+A option that supports on-premises object storage; compatibility

has been certified with S3-compatible offerings from EMC, NetApp,

CleverSafe and other major storage suppliers.

M+A workflows can also be extended to include transfers to and

from cloud storage in Amazon Web Services and/or Microsoft

Azure. This is achieved via a Signiant SaaS offering known as

Flight. On the cloud side of a transfer via Flight, there is no need

for the customer to procure compute resources or deploy software

– Signiant handles all of that on an auto-scaling basis. On the

on-premises side, an additional piece of Flight software is installed

alongside the Agent. All transfers can be monitored and tracked via

the M+A console.

Per the diagram on the following page, a deployment of

Manager+Agents and Flight provides a fully integrated system to

support a hybrid cloud storage architecture.

FIGURE 1: MANAGERS+AGENTS SAMPLE SYSTEM DEPLOYMENT

OBJECTSTORAGE

FIREWALL

ADMIN/IT

CORPORATE DMZ

FILE STORAGE

CORPORATE NETWORK

PARTNER(S)

MANAGER AGENT(S)

RELAY AGENT(S)

FIREWALL

AGENT(S)



https://www.signiant.com/products/flight/?cc=wp-manager-agents


SIGNIANT ACCELERATION TECHNOLOGYManager+Agents utilizes Signiant’s patented UDP-based

acceleration technology to move large files much faster than is

possible via TCP. Achieved with a proprietary transport protocol

that minimizes the impact of network latency or packet loss, M+A

enables the full utilization of all available bandwidth.Another way

of visualizing the performance improvement is to compare Signiant

transfers with TCP transfers over various distances, and to then

repeat this comparison for different bandwidth connections. As

shown below, with TCP the file transfer takes proportionally longer

as distance/latency increases and packet loss occurs. With Signiant,

not only is the transfer time much shorter, it’s also independent of

distance — and as more bandwidth is added the Signiant transfer

time gets proportionally faster.

Proprietary Transport Protocol

Signiant’s proprietary transport protocol implements both an

advanced transmission control protocol on top of UDP as a

replacement for TCP, and an advanced file transfer protocol as a

replacement for FTP.

One fundamental problem with TCP is that it uses a relatively

unsophisticated sliding window mechanism, sending only a certain

amount of data over the network before it expects that data to be

acknowledged as received. As TCP receives acknowledgements,

it advances its window and sends more data. If the data doesn’t

get through or an acknowledgement is lost, TCP will time out and

retransmit from the last acknowledged point in the data stream with

a reduced window size. There are a number of problems with this,

such as retransmitting data that may have already been received, or

long stalls in data sent while waiting on acknowledgements.

Signiant uses a mechanism similar to a sliding window, but the

mechanism incorporates two key improvements over traditional

TCP: adaptive window size and selective acknowledgment. Adaptive

window size is a mechanism that measures the capacity of the

network and the round-trip distance. It then uses a window that’s big

enough to keep data in flight on the network at all the times. Selective

acknowledgement allows the endpoint to verify which pieces of the

transmission have been received so that any section that is missing

— even one in the middle of the data set — can be retransmitted

rather than the entire data set.

Additionally, Signiant is constantly measuring effective throughput,

network latency and loss, and building a history. By maintaining a

history, it is possible to see how all of these factors are changing

over time, and network congestion can be located by analyzing the

frequency of changes. This allows the Signiant protocol to adapt to

network conditions much more effectively than TCP, which employs

pure additive-increase/multiplicative-decrease window size changes

in response to point-in-time packet loss.

Signiant’s advanced transport control protocol is deployed on top of

UDP (User Datagram Protocol), but also enhances UDP in significant

ways. UDP enables chunks of data to be sent on a best-effort basis

that removes the TCP overhead of back-and-forth handshaking

between the two ends of the IP transmission path mentioned above

— but UDP typically does so by foregoing reliability.

FIGURE 2: MANAGERS+AGENTS AND FLIGHT INTEGRATION SUPPORTING HYBRID CLOUD STORAGE ARCHITECTURE

OBJECTSTORAGE

FIREWALL

ADMIN/IT

CORPORATE DMZ

FILE STORAGE

CORPORATE NETWORK

PARTNER(S)

MANAGER AGENT(S)

RELAY AGENT(S)

FIREWALL

AGENT(S)

AGENT(S)




The Signiant protocol implements functionality on top of UDP

that restores reliability in a TCP-like way, but with the following

improvements:

➜➜ Flow control ensures that data is transmitted at the optimal rate

for the receiver.

➜➜ Congestion control detects when the network is being

overloaded and adapts accordingly.

➜➜ Reliability mechanisms make sure that data loss due to

congestion or other network factors is compensated for and that

the order of the stream of data is maintained.

Signiant’s transfer protocol also adds enhanced FTP-like functions

to the UDP foundation. With FTP, each file sent requires its own

set of command and response interactions, and its own TCP

connection. The Signiant protocol reduces this high per-file overhead

by communicating about files being transferred more efficiently and

multiplexing the transmission of files over a single channel, thereby

enabling file operations to be performed in parallel.

When combined with Flight the Signiant transport also performs

intelligent application-level routing choosing the optimal protocol for

the observed latency and loss characteristics on available network

links. Writing to storage from the closest possible point allows

interactions with storage to be optimize whether using HTTP-based

object storage access protocols, Network Attached Storage (NAS)

interactions with NFS or CIFS, or Direct Access Storage (DAS).

Checkpoint Restart

In addition to the various protocol-level reliability mechanisms

noted above, Signiant includes a feature called Checkpoint Restart.

Checkpoint Restart adds a layer of fault tolerance into any transfer

by ensuring that if a transfer is interrupted for any reason (from

network failure to application or OS crashes) the transmission will

automatically restart after recovery from the point at which it was

interrupted, with no loss of data. This adds not only reliability but also

efficiency to the overall transfer process by avoiding the need to start

at the beginning again after a failure.

Signiant Acceleration Compared to TCP

Signiant acceleration technology has the greatest impact for large

file transfers over distance and takes full advantage of available

bandwidth. As shown in the diagram below, compared to TCP, as

distance and latency increases, Signiant performs exponentially faster

especially with higher bandwidth connections.

Another way of visualizing the performance improvement of Signiant

acceleration compared to TCP transfers is to compare transfer

time over various distances and to then repeat this comparison for

different bandwidth connections. As shown below, with TCP the file

transfer takes proportionally longer as distance/latency increases or

packet loss occurs. With Signiant, not only is the transfer time much

shorter, it’s also independent of distance — and as more bandwidth

is added the Signiant transfer time gets proportionally faster.

FIGURE 3: SIGNIANT VS. TCP-BASED TRANSMISSIONS

0

50 Mb 1 GB 20 GB

6

TR

AN

SF

ER

TIM

E (

HR

S) 12

DATA PAYLOAD

UP TO 200X FASTER

GREEN = 1 GBPS LINK LA TO SINGAPOREBLUE = 100 Mbps LINK LA TO NY

No Signiant

No Signiant

SigniantSigniant

18 HRS

12 HRS

6 HRS

FIGURE 4: TIME TO TRANSFER 1 HR OF HD CONTENT (ENCODED @50 Mbps AVC-21GB)

100 Mbps

TCP Signiant

LA Metro 1:28 0:32LA to NY 5:38 0:32LA to London 10:59 0:32LA to Singapore 21:05 0:32

TCP SIGNIANT

500 Mbps

TCP Signiant


TCP SIGNIANT

1 GBPS

TCP Signiant


TCP SIGNIANT




SECURITY

Transport Layer Security

Transport Layer Security (TLS) is built in as a core component of the

Signiant transfer protocol software stack and provides the base layer

of security in all Signiant tools. TLS is the standard cryptographic

protocol designed to provide communication security, privacy and

data integrity over IP networks. Encryption mechanisms provided

by TLS are used to secure data as it is transmitted with regards

to data transfer, advanced authentication, data integrity, and data

confidentiality. All transfers conducted through the Signiant transfer

protocol are encrypted, appropriately authenticated, authorized and

tracked, including proof that files were delivered thereby ensuring

nonrepudiation (protection against denial of performing an action).

Additional layers of service and application security are added to the

individual Signiant products on the management, control and end-

user components specific to each.

Defense-in-Depth

TLS within the transport layer is just the starting point for security at

Signiant. Because enterprises use our software to move their most

valuable assets, we take great care to secure every layer involved in

file movement. Our security technology and implementation regularly

undergo extensive third-party reviews to ensure effective protection,

and our development team is well-versed in secure design principles.

Trusted across the Media & Entertainment industry and beyond,

Signiant was awarded the DPP ‘Committed to Security’ mark for

both production and broadcast.

Effective security requires much more than encryption during

transfer or storage. Even the strongest lock is only as effective as the

protection of the key or combination used to unlock it.

Signiant’s Manager+Agents software, in conjunction with appropriate

organizational policies and procedures, protects assets from threats

posed by individuals, hacker groups and criminal adversaries. Assets,

including computing and network resources, intellectual property,

business timelines, and reputation must be protected from a wide

range of threats that include destruction, corruption, disclosure and

interruption.

Manager+Agents mitigates these threats via the implementation of

multiple security services including authentication, authorization, data

integrity, data confidentiality and non-repudiation.

Authentication

Authentication services confirm that someone or something is

what they claim to be. Signiant’s Manager+Agents implements

authentication services in the form of login and password credentials

with the added certainty of certificate-based authentication.

Whenever passwords are used for client-side authentication, the

server side is authenticated using SSL certificates from a public

‘certificate authority’ (CA) to make sure that User IDs and passwords

are sent to the intended target and not some rogue man-in-the-

middle trying to steal authentication. Signiant ensures that mutual

certificate-based authentication keys are always employed for

machine-to-machine connections and that the distribution of these

keys is secure so that they can be reliably used in the authentication

process.

Authorization

Authorization services implement and enforce access policies to

data. Signiant also enforces password policy, including a configurable

expiration cycle that requires users to change passwords periodically,

and minimum password strength validation to require users to create

formidable passwords. Enterprise customers have the option to use

Active Directory or LDAP, where authentication occurs behind the

customer’s firewalls. Through Active Directory and LDAP, customers

can implement multi-factor authentication as well.

Manager+Agents is able to “sandbox” transfers to a particular

directory on a particular host server, so that access policies can

be enforced as designed. The software does this by connecting

to the host using a local or domain userID, and thereby adhering

to the access rights policy assigned to the user on that server.

Similarly, users can be enabled or restricted when running transfers,

monitoring, and reporting according to the access rights given to

their profile.

Data Integrity and Confidentiality

Data integrity services protect information assets from corruption.

Signiant employs hashing and digital signing of data during transit

to prevent malicious tampering and errors between sending and

receiving parties.

FIGURE 5: MANAGERS+AGENTS SECURITY



https://www.signiant.com/blog/significance-dpp-committed-security-program/?cc=wp-manager-agents


Data confidentiality services protect information assets from

disclosure. Manager+Agents employs 256-bit AES encryption to

ensure data confidentiality during transfer of files and when the file is

at rest. Though AES encryption is the ‘state-of-the-art’ for encryption,

it is very important that the encryption key management scheme is

‘state-of-the-art’ as well. You can have big lock on the door, but if

anyone can find the key under the mat, then the door is not secure.

Signiant encryption keys are managed with a local certificate authority

based on OpenSSL. This key management scheme ensures that the

encryption keys are secure in transit and at rest.

Non-Repudiation

Non-repudiation services provide a trustworthy record of what has

been done by whom and when. The reporting and log files provided

by Signiant products enable companies to determine the complete

history of a file’s movement. The data will allow a forensic analysis

of who, where, and when file assets were moved. Signiant user

interfaces an audit trail of both end-user and administrative activity.

The “certified delivery” function provided by Signiant software

identifies each file that has moved from a source to a target, along

with signed hashes of the file as computed by the source and target

agents. These signed hashes are compared and if they match, it is

guaranteed that the file has not been modified in transit.

AUTOMATED FILE TRANSFER BASICSNETWORK CONFIGURATIONAs described in the acceleration technology section, the Signiant

protocol runs over UDP. To allow Manager+Agents to move content

throughout the LAN/WAN network environment, it is therefore

necessary to ensure that the appropriate ports are open on your

router/firewall devices. Signiant’s seasoned Customer Support team

can assist with this process, providing specific port details, device

knowledge, and guidance on the use of Relay Agents.

The diagram below provides an example for a simple transfer

configuration:

Detailed network information for sample mediadropbox

WAN accelerated transfer between two locations

RELAY-AGENT

FIR

EW

AL

L

DMZ

AGENT 1MAIL SERVER

SIGNIANT MANAGER

AGENT 2

LOCATION A

LOCATION B 12

24

4

3

2

INT

ER

NA

L F

IRE

WA

LL

FIGURE 6: SIMPLE TRANSFER CONFIGURATION

1. Manager sends instructions to Agent 1 (TCP or UDP port 49221) via manually authenticated and encrypted channel. 2. Agent 1 (src port random) authenticates with Agent 2 (TCP port 49221). Agent 1 and Agent 2 negotiate the UDP port to to use for WAN acceleration (starting with UDP 49222 as a default). If needed, the traffic to or from the internal networks can be relayed through a single port on one or more agents in the DMZ (to minimize firewall rules).3. Agent 1 updates the manager with statistics information while the job is running. Using the active control channel (TCP port 49221). 4. When job completes, the control channel closes and optionally an email is sent out for the job status (via TCP port 25 to Mail server).




THE MANAGER CONSOLECentral management of the Manager+Agents system is provided

via a highly configurable web-based administration console. The

Manager dashboard gives you the ability to customize views for

system configuration, task automation, scheduling, and managing

current system activity. Customized reporting, combined with the

power to take action as needed, ensure your business processes

run smoothly. M+A provides complete visibility into file movement,

and you can create and customize job completion email notifications,

as well as customize and schedule reports, file movement jobs and

system activity.

CREATING JOBSIn the Manager+Agents system, individual scheduled file transfers

between Agents are referred to as Jobs. The most straightforward

way to create a job is to use one of the pre-configured solutions that

come with the base product and are collectively referred to as Media

Mover.

Media Mover comprises four different solutions for specific use

cases:

➜➜ Media Dropbox transfers data between a single source Agent

and one or more destination Agents, based on a time of day or

upon changes to the source directory.

➜➜ Media Aggregator retrieves files from one or more Agents

to a single target Agent and stores the files on its local disk or

attached storage.

➜➜ Media Distributor is used for a simple scheduled push

distribution from one source Agent to one or more destination

Agents.

➜➜ Media Replicator is used for a simple scheduled push

distribution from one source Agent to one or multiple destination

Agents. Media Replicator supports Synchronization option

and has performance advantages over Media Dropbox when

transferring large numbers of files.

For customers with the Local Object Storage option, a set of Object

Mover solutions are provided to perform similar functions to Media

Mover:

➜➜ Object Dropbox transfers files from a single hot folder on the

source Agent to local object storage, based on a configurable

schedule.

➜➜ Object Uploader transfers multiple files and/or folders from one Agent to local object storage, on-demand or based on a configurable schedule.

➜➜ Object Downloader transfers multiple files and/or folders from local object storage to an Agent, on-demand or based on a configurable schedule.

➜➜ Object Replicator transfers contents from one local object storage container to another on the same system or between

systems, on-demand or based on a configurable schedule.

Object Mover also works in conjuction with Flight for transfers to and

from cloud object storage.

AGENT GROUPSAn Agent Group is a logical collection of Agents that Jobs can use

in place of individual agents. When a Job uses an Agent Group, the

controlling Agent sends data to or receives data from each Agent in

the group. This facilitates sending files to/from multiple locations, as

well as provides optional redundancy and scability across pools of

Agents.

FIGURE 7: MANAGER+AGENTS DASHBOARD




In load balancing mode, transfers are automatically routed so as to

balance the load across all active Agents in a group. Signiant’s load

balancing connects to all of the machines in the Agent Group and

picks the first one that responds. (The one that responds first is most

likely the one with the lightest load or with the fastest network path

to it.)

INTER-COMPANY TRANSFERSMost large media enterprises have an extensive network of partners

and use M+A to move content across a complex global supply

chain. M+A is specifically designed to provide secure mechanisms

for accelerated transport between companies, in addition to

supporting transfers within a single company.

This is accomplished via mechanisms for peering into two or more

Manager deployments to create “web of trust” relationships between

Signiant installations. Companies can easily establish mutual

authentication between multiple Managers and their Agents. In

addition, job components running on an Agent can be viewed and

controlled from Managers other than the initiating Manager.

The peering process is performed by exporting information from one

Manager (e.g. the certificate of the CA) and importing it onto another

Manager. The import process allows the CA certificate to then be

distributed to Agents in your network.

GROWING FILE SUPPORTManager+ Agents provides support for growing files, which is the

ability to begin sending files as soon as the file starts to be written

rather than waiting for the file write to complete. This allows for an

overall reduction in file transfer times and is particularly effective in

enabling highly efficient workflows for sports, news, and other types

of live event production.

COMMON USE CASESThe Manager+Agents system is highly configurable and can be

used for virtually any task that involves the automated movement of

large data sets. From a big-picture perspective, however, use cases

within the Media & Entertainment sector generally fall within several

high-level categories. This section provides an overview of the most

common types of use cases. It is important to note that all of these

examples involve business-to-business media workflows; Signiant

technology is not typically involved in delivery of content directly to

the consumer.

CONTENT DISTRIBUTION

Throughout the global media supply chain, Manager+Agents

software plays a key role in the B-to-B distribution of content – both

within and between enterprises. There are many situations where

there is a mission-critical need for content to flow from a single

source location where it is created or packaged to various endpoints

around the world.

The basic one-to-many workflow of content distribution has

long been an essential part of the media business. Prior to the

advent of file-based workflows, millions of videotapes and film

reels were shipped around the world every day. While there are

still situations where it’s practical to ship disk drives or data tapes,

B-to-B distribution is now largely accomplished via terrestrial

delivery over IP networks. In parallel with this transition, the

scale and complexity of global media supply chains has grown

dramatically with the increasingly diverse ways that video content

is delivered to consumers. With its highly scalable architecture, the

Manager+Agents solution can readily accommodate these growing

and changing content delivery ecosystems.

These are some of the most common distribution workflows that

employ M+A to provide acceleration, security and central control:

➜➜ Deliver first-run television content to multiple global playout

centers;

➜➜ Transfer Digital Cinema Packages (DCPs) to cinemas for theatrical

release of feature films to digital screens;

➜➜ Deliver video ads to online and/or television outlets;

➜➜ Send syndicated longform television content to licensees;

➜➜ Send VOD content to cable system operators;

➜➜ Transfer promos and other marketing content around the world;

➜➜ Send source content to languaging houses for localization;

➜➜ Deliver packaged assets from a media services provider to online

platforms.

In each of these cases, the content creator/owner will typically have

a Manager installation at their headquarters location and perhaps

at other sites around the world. Agents will be deployed at major

partners or international hubs, and deliveries can be tuned for various

situations. In the case of close-to-air television content, for example,

it is essential to give priority to certain files and verify receipt by a

certain time. In other scenarios, it is best to schedule deliveries

during off-hours while the corporate network is underutilized. With

M+A, administrators have the flexibility and granular control they

need to implement whatever process best meets the needs of the

business.




CONTENT DISTRIBUTION / AGGREGATION / INGESTAnother common workflow in the global media industry involves

collecting content from disparate sources and bringing it to a central

location for processing, packaging, and subsequent distribution

to partners and/or internal facilities. These many-to-one use cases

involve some of the same complexities as distribution; they are

similarly global in scale and often have time-critical elements.

Contribution and distribution workflows are often interleaved or even

interchangeable, with the control point determined by the commercial

arrangements between the parties. For example, a major cable

network that commissions content from many different producers

may design a contribution workflow to ensure timely content

submission and a uniform ingest process. The content producer will

be provided with (or will need to purchase) an Agent that is controlled

by the cable network’s Manager. In this case, the content producer is

a participant in the cable net’s contribution workflow.

That same content-producing entity may also independently create

video assets that are marketed and sold to other outlets around the

world. Some of those contracts are likely to require that the producer

deliver a finished asset to the buyer’s premises. In this situation the

content producer will drive a distribution workflow from their own file

transfer system. Because of these diverse needs, Manager+Agents

deployments within large media enterprises are often correspondingly

complex, involving the entity’s own M+A system, Agents from

multiple partners, and various cross-trust arrangements. Signiant

software is configurable enough to support whatever workflow is

desired, the details of which are generally dictated by business model

imperatives within the M&E industry.

These are some of the most common aggregation workflows that

employ M+A to provide acceleration, security and central control:

➜➜ Collect commissioned programs from many global producers

(cable or broadcast network)

➜➜ Ingest ads from agencies and production companies (ad

distribution business)

➜➜ Ingest packaged assets from various content producers (online

platform)

➜➜ Gather news content from global bureaus (cable of broadcast

network)

CREATIVE WORKFLOWSManager+Agents software plays a key role in the creative process

for most of the produced media assets that consumers see on

television, in theaters, or online. Fast, secure electronic transport of

media content between various locations and commercial entities is

an essential element of modern production and post production.

The journey from raw camera footage to finished media asset

involves many processes, some of which occur in series and some

in parallel. Certain of these activities are talent-driven, undertaken by

creative professionals whose artistic sensibilities add crucial value

to the finished product. In contrast, some processes are purely

technical and can be fully automated – while others lie somewhere in

the middle of this spectrum. A Manager+Agents system can provide

the underpinnings for all of these processes.

The more premium the asset, the more likely that a large number of

different, highly specialized companies will be involved in the creative

process. For a high-end feature film, dozens of different specialty

post production firms may provide services ranging from basic

editing to digital effects, sound production, colorizing, etc. Even after

the program master is finished, the asset is likely to move around

the world for localization. Localization might include voiceover,

subtitles, and/or closed captioning in various languages, along with

compliance editing to align with varying laws about what kinds of

content can be shown to the public.

The M+A system owned by a creative entity is likely to be operated

in a more agile manner than systems owned by large content

distributors and aggregators. A film production company might, for

example, schedule regular transfers for the duration of a production

to send rough-cut footage every night from an on-location system to

a production facility. During the day, deliveries for the post production

pipeline will be scheduled on an as-needed basis when the content

is ready for the next step. Timelines are usually tight and resources

are usually at a premium, so the granular controls available in the

M+A system can be essential. To meet multiple deadlines with the

available bandwidth, the operator may need to use resource controls

to prioritize a certain file delivery, followed by two more in parallel.

For high-end productions, security during the creative process is of

paramount importance. The creative community depends on M+A

to provide secure transfer, as well as a detailed chain of custody

record.

CLOUD WORKFLOWSAs cloud adoption in M&E continues to gain traction, M+A workflows

are evolving to touch the cloud in various ways. Individual workflows

that fit into each of the prior three use case categories (distribution,

aggregation, creative) might be implemented entirely on-premises,

mostly in the cloud, or with a hybrid cloud approach.

The Manager+Agents system in a cloud or hybrid cloud workflow

continues to serve as the ‘brains’ of the operation, orchestrating file

transfers, managing network resources, maintaining records, and

perhaps automating the entire workflow. As discussed on page 2,

the addition of Signiant’s Flight offering to the system provides a

transparent means of connecting cloud storage in either AWS or

Azure into a file transfer workflow.




SPORTS

The cornerstone of sports video is, of course, live action footage

from the multiple cameras at the venue. However, a surprising

volume of file-based content is generated by broadcasters before,

during and after the live event, including highlights, segments, studio

components and more. For fast-turnaround content during the

game, broadcasters often transfer footage to central headquarters for

editing, formatting and packaging. From there it might flow to various

digital media outlets or even back to the trucks for insertion into the

live game feed.

This production model rests on the ability to transfer huge amounts

of data, whether large files or many small clips, in a fast, secure

and reliable manner. Manager+Agents software has long served as

the core acceleration backbone for automated delivery of content

from OB trucks to and from headquarters, quickly and securely

moving large volumes of data. Before the game, things like graphic

elements and produced segments from the studio are delivered to

the truck for use during the game production. After the game, the

“melt file” consisting of the most relevant content is transferred back

to headquarters via M+A for access by content producers and for

archiving.

During the game, the fast-turnaround workflows noted

above generate staggering amounts of data that can only be

accommodated with highly automated, accelerated file transfer

solution. For example, just one OB truck at a 6-hour game might

generate 2000 – 2500 clips, with larger weekend-long tournament

events generating up to 40,000 clips for 10 trucks.

With M+A, broadcasters can ensure that the content is steadily

flowing out of the truck and back to headquarters, where it can be

available for use within minutes of capture. Specialized M+A features

such as growing file support (see p. X) help make sports workflows

even more efficient.

APPLICATIONS OUTSIDE M&E

While specific details of the use cases outlined in this section are

unique to the M&E space, there are parallels in other data-intensive

industries. For example, consider a provider of high-res satellite

images for mapping applications. A business model in this space is

likely to depend on quickly providing very large data sets to many

customers around the world. Some customers may receive the same

set of standard images, while others have contracted for customized

packages. An M+A deployment can ensure secure, timely data

transfer with detailed visibility and tracking of all deliveries.

Other industries that rely on Manager+Agents include life sciences,

where clinical trial data, genome sequences, and medical images

can all involve extremely large data sets. Earth sciences applications

include gas & oil surveys, seismic images, and drone video for

mapping. Finally, companies involved in various kinds of design use

M+A to move large code bases, architectural drawings, or product

design files.

RESOURCE MANAGEMENT OPTIONSigniant’s Resource Management module is ideal for organizations

seeking more sophisticated control of file-based assets, network

bandwidth and server resources. Resource Management allows

administrators to prioritize delivery of assets between sites based on

business needs and content delivery windows. As content priorities

change, the queue can be adjusted by moving more critical jobs to

the top, while jobs with less stringent timelines remain lower in the

queue.

Controlled by a single Signiant Manager or multiple Managers

operating in an enterprise environment, Resource Management can

be used to control bandwidth usage to specific locations or servers

to ensure systems or networks are not overloaded, including setting

ceiling and floor ranges based on the time of day or traffic loads on

the network and limits for running file movement jobs. Administrators

can set automated or manual bandwidth limits for running jobs, and

can define serial or parallel transfer flows.

When a job is submitted, a priority and complete-by-time can

be specified. Jobs are then automatically ordered by priority and

complete-by time. As such, all high-priority jobs are sorted before

all standard and low-priority jobs. Within high-priority jobs, jobs with

the nearest complete-by times will be sorted ahead of jobs with later

complete-by times.




INTEGRATION WITH OTHER SYSTEMSREST / SOAP APISManager+Agents includes a comprehensive set of APIs designed

to enable other systems to initiate file transfer jobs and collect

information. Signiant provides both a native web service API as well

as a FIMS-compliant API. Both APIs are available in both SOAP and

REST. For API details, please see developer.signiant.com.

FIMS

The Framework for Interoperable Media Services (FIMS) is an open

standard from the Advanced Media Workflow Association (AMWA).

Utilizing a Service Oriented Architecture (SOA) approach, FIMS

replaces the tightly-coupled, highly customized integrations used by

traditional systems with a set of “Media Services” that are

interoperable, interchangeable and reusable. The interfaces between

these different services and the centralized system that runs them are

defined by FIMS. For more information about FIMS, refer to fims.tv.

Signiant was involved in development of the FIMS standard and

maintains FIMS-compliant versions of the Manager+Agents REST

and SOAP APIs.

AUTOMATION ENGINE OPTIONA sophisticated add-on feature, the Manager+Agents Automation

Engine facilitates the creation of flexible Job Templates that can be

used to create customized workflows for the processing of content.

A template may be as simple as moving a file from one Agent to

another, or as sophisticated as directing a package for transcoding

and watermarking before publishing.

FIGURE 9: PARALLEL TRANSFERS WITH QUEUING

FIGURE 8: SERIAL TRANSFERS WITH QUEUING

668 MB asset

9.7 GB asset

50 GB asset

668 MB asset

11.1 GB asset

72.6 GB asset

Unprioritized New York-originating transfer jobs

Using Signiant Manager, jobs bound for London are prioritized in queue with a concurrent job limit set to one; queue order is adjustable

Transfers are completed serially, each utilizing allocated bandwidth until queue is empty

transfer complete, arrives in London

next transfer begins after previous transfer is complete

allocated bandwidth

668 MB asset

9.7 GB asset

50 GB asset

668 MB asset

11.1 GB asset

72.6 GB asset

New York- originating transfer jobs

Using Signiant Manager, jobs bound for London are placed in queue with a concurrent job limit set to four

Four transfers start simultaneously with two transfers remaining in queue

The two 668 MG transfers are completed, allowing the 1.1 GB and 72.6 GB transfers to begin

Aggregate bandwidth is shared evenly by four transfers

As transfers complete, aggregate bandwidth is reallocated for remaining transfers

One transfer still in progress; using all of aggregate bandwidth

Three transfers still in progress; each using one-third of aggregate bandwidth



http://developer.signiant.com/

http://fims.tv


FIGURE 10: AUTOMATION ENGINE FUNCTIONALITY

1. A user creates or modifies a job template, laying out components in a sequence and mapping inputs to outputs. 2. The job template is saved in the job template library database.3. A job is created from a template by an operator supplying job parameters and a schedule through the administrative interface or an external trigger.4. The Scheduler monitors the system for jobs to run and alerts the Supervisor to initiate the job using a specific template and job parameters.5. The Supervisor requests and receives the job template from the template library and runs the job using the job parameters.6. The Supervisor passes the relevant job template components and job state information to Agents for execution.7. In the case of a file transfer components, the Supervisor contacts the controlling Agent which connects with one or more Target Agents to perform the file transfer.8. Agents report the execution status of each component back to the Manager.

The Media Mover functions described on page 7 are effectively pre-

defined Job Templates; with the Automation Engine you can create

your own.

The Automation Engine includes a toolbox of standard Components

(such as transcode or QC steps) that can be linked together via

a visual drag-&-drop interface (known as the Workflow Canvas)

to form new Job Templates. You can also create your own

Components, and both Components and Job Templates can be

stored for future use. Any interpreted script language may be used

within the workflow Components. These new Job Templates can be

combined via the Manager with manual tasks and business rules to

implement highly complex, customized file transfer workflows.

The diagram below illustrates Automation Engine functionality. Please

note that the Scheduler and Supervisor are sub-components of

the Manager. The ‘controlling’ and ‘target’ nomenclature refers to

roles Agents take on for any given transfer/job.

SUPERVISOR

WORKFLOW MODELING AGENT

PROCESSING TRANSCODE GENERATED FILE DELIVER FTP DELIVER TARGET

JOB TEMPLATE LIBRARY DATABASE

USER(S) SCHEDULER

1

TARGET TRANSFER AGENT CONTROLLING TRANSFER AGENT

FTP

23

5

4

6 8

7

THIRD PARTY INTEGRATIONS

Media technology systems are highly complex, typically involving

software and hardware from many different suppliers. Signiant works

closely with major suppliers in the industry to facilitate integrations

with Manager+Agents utilizing the APIs and workflow automation

capabilities described earlier in this section. Signiant’s media

technology partners include most major suppliers in the space such

as:

AUTOMATION ENGINE




Integrations with Manager+Agents typically fall into two categories:

Programmatic Initiation of Signiant File Transfers by 3rd-Party Products

Some media workflows are designed so that a master system

such as a MAM or DAM controls the transfer of files. In this type of

integration, the 3rd-party system will make a web service API call

to the Signiant Manager. Progress and status details relating the

transfer are reported back to the calling system, typically for display

in that product’s user interface.

Control of 3rd-Party Products from Within Custom Signiant Workflows

In other media workflows, Signiant Manager+Agents serves as the

master system and M+A jobs coordinate the processing of files by

3rd-party products such as Transcoders and Quality Control (QC)

utilities. These workflows typically involve the transfer of files from

one location to another, with the Signiant system managing other

processes that occur immediately before or after the transfer.

In this type of integration, the Manager will generally utilize an API

on the 3rd-party application to send a list of files to be processed

as well as the type of processing to perform. These API-level

integrations allow for progress and status information from the

3rd-party product to be reported back into the Manager GUI for

operator advisement. Once the 3rd-party process is complete, an

intelligent decision can be made on the next step in the workflow.

For example, if the QC check is successful, the Manager will transfer

the file to a playout system. If QC fails, the Manager will move the

original file to a quarantine folder and distribute a notification for

operator intervention.

SIGNIANT MEDIA SHUTTLE FOR PERSON-INITIATED TRANSFERSWithin most enterprises that move large files, there are use cases

for both automated transfers and transfers that involve people.

Manager+Agents handles the automation tasks, while the de facto

standard for end user transfers is a Signiant SaaS offering called

Media Shuttle. Media Shuttle is sold separately but is fully integrated

with Manager+Agents to support both person-to-system and

system-to-person workflows.

About Media Shuttle

Media Shuttle is Signiant’s SaaS solution for “hands-on” accelerated

transfer of large files. Combining the simplicity of online file sharing

with Signiant’s enterprise-class acceleration, security, and control,

Media Shuttle is the easiest way to send any size file, anywhere, fast.

Media Shuttle makes it easy to share files between people, between

systems and to and from the cloud, while providing centralized

tracking and reporting for every transaction.

Portals for Every Project and User Type

Sending files with Media Shuttle is as simple as dragging and

dropping them onto a web page or protal. It’s so intuitive that your

users will never need a tutorial, training session, or even an email

explanation. There are three different types of portals that can be

created to support almost any use case:

➜➜ SEND portals allow for fast person to person file transfer.

➜➜ SHARE portals make it easy to upload and download files within

designated folders.

➜➜ SUBMIT portals provide an onramp for users to submit files into

automated workflows with Manager+Agents.

Media Shuttle offers roles for three key user types in the organization

– end users, operational administrators and IT administrators. Each

user type has a tailored view that’s made explicitly for their job

function. Media Shuttle makes it easy to enable all users, whether

internal or external, to access only the files and features they require.

Brandable, Customizable Experience

Media Shuttle customers can create and customize as many

portals as they choose, each with their own branding and imagery.

Setting up a portal for every project or partner is a quick and easy

task and it’s a great opportunity to extend your brand and create

a professional, visually appealing experience for transferring files.

Media Shuttle also supports multiple languages which can be

configured on a portal by portal basis.

Like all of Signiant’s products, Media Shuttle leaves storage under

your control. Media Shuttle portals can be individually configured for

either on-premises storage or your cloud object storage tenancy in

Amazon Web Services or Microsoft Azure.



https://www.signiant.com/products/media-shuttle/?cc=wp-manager-agents


Media Shuttle System Architecture

Media Shuttle is often deployed in a hybrid SaaS configuration as

shown above. The data plane (and associated server and software)

are on-premises, while the system control plane are delivered

from the cloud via Signiant’s multi-tenant cloud-native SaaS. This

architecture lets you maintain complete control of high-value content,

while simultaneously deriving the classic SaaS benefits of automatic

updates, scalability, and global availability.

Integrated Workflows for Manager+Agents and Media Shuttle

The two products can be connected together via the M+A workflow

engine and an optional Media Shuttle feature called the System-

to-Person Automation API, a capability that enables customers to

integrate Media Shuttle with other systems such as MAMs / DAMs

to automate distribution (“click to download”) and acquisition (“click

to upload”) workflows. The API also facilitates the transition from

Media Exchange to Media Shuttle for Manager+Agents customers

who want to connect complex automated workflows with end users.

REDUNDANCY OPTIONSManager+Agents software plays a mission-critical role in moving

highly valuable, time-critical content around the world. The system

is designed to accommodate a range of redundancy options, and

we encourage customers to carefully plan for mitigation of failure

scenarios.

MANAGERThere are four primary options for Manager redundancy:

Virtual Machine Snapshot

In this scenario, the Manager is installed on a virtual machine and

snapshots are taken automatically to allow quick restoration up to

the snapshot point either locally or at a disaster recovery location.

Downtime is dependent upon the length of time for the manual

process to note the issue and recreate the instance. No additional

Signiant licenses are required.

High Availability Option (Red Hat Linux Cluster)

The Manager is installed on a Red Hat Linux Cluster, typically with

two servers and shared RAID disk, operating in an active/passive

mode. The operating system monitors the hardware, operating

system, and application health. An automatic failover is triggered

in the event of a system malfunction, with a downtime of only a

minute or so. No data is lost and any running jobs are automatically

restarted and pick up where they left off. A Signiant Manager High

Availability license is required.

Warm / Remote Standby Option

The Signiant Manager software is installed on a server in a separate

location, and cross-trusted to the production Manager. A Manager

backup is transferred to the standby Manager on a daily basis.

CLOUD

ON-PREMISES/ CUSTOMER NETWORK

• WEB INTERFACE • EMAIL NOTIFICATIONS• CLIENT UPDATES• FILE TRANSFER LOGISTICS• DATA COLLECTION

END USEREND USEROPS/ADMINIT

INTERNET (OR CUSTOMER NETWORK)

FIGURE 11: MEDIA SHUTTLE HYBRID SaaS ARCHITECTURE

CUSTOMER’S ON-PREMISES STORAGE

CUSTOMER’S SERVER

RUNNING SIGNIANT FILE

TRANSFER SOFTWARE

SIGNIANT CONTROLLAYER SaaS




When an issue is noted, the backup is restored on the standby

Manager, and DNS change to point the Manager hostname from

the original production Manager IP address to that of the standby

Manager. Data loss is restricted to that since the backup was taken

Jobs will restart based on their schedule and will pick up where

they left off. Downtime is dependent on the length of time to notice

the issue and restore the backup (that duration could be an hour or

so dependent on the database size). A Signiant Manager Remote

Standby License is required.

Multiple Managers

The Signiant Manager software is installed on a server in a separate

location, and cross-trusted to the production Manager. Jobs on the

production Manager are re-created on the second Manager and left

in the suspended state. The second Manager has full monitoring

and control capabilities overall all agents and jobs in the Signiant

ecosystem. In the event of Manager failure, the jobs are resumed on

the second Manager. Downtime can be as low as a few minutes. A

Signiant Manager license, with the same options and capacity as the

production Manager, is required.

AGENTSAgents can be protected in the following ways:

Windows Cluster - Transfer or Relay Agents

Agent support is provided for Windows active/passive clusters. This

provides hardware and software redundancy, with automatic failover

by Windows. No additional Signiant licenses required.

Agent Groups - Transfer or Relay Agents

Agents, of any operating system, can be deployed in groups,

providing both redundancy and scalability. Transfers are

automatically balanced between the available Agents. Interrupted

transfers will be automatically resumed from where they left off on

any Agent in the group.

Either Transfer Agents or Relay Agents can be deployed in load-

balanced groups; an additional Agent license is required for each

added Agent.

PACKAGING SUMMARYBASIC FUNCTIONALITY

Manager+Agents is sold as perpetual license software. Costs scale

with the number of endpoints served, but there are no charges

for bandwidth and no charges for the amount of data transferred

between on-premises locations. The standard software is compatible

with file storage; local object storage is a separate line item.

To get started, you will need:

➜➜ A Manager, which includes the control console for configuration,

monitoring and reporting; and four standard job templates for the

distribution, aggregation, replication, and hot folder transfer of

files. Capacity to control up to 15 Agents is included.

➜➜ Transfer Agents are required at each endpoint location to

execute the file transfer tasks defined by the Manager. Agents

intended for use as Relay Agents are offered at a lower per-agent

price.

You can get started with just one Manager and an Agent for each

endpoint that you want to send and receive data. As the system

grows, you can purchase additional Agents for other locations and

partners, or for redundancy and load balancing. As Agents are

added, you will also need to add Agent control capacity to your

Manager. Add-on capacity is sold in 5-node bundles.

ADD-ON OPTIONS

There are a number of optional modules for the Manager, including:

➜➜ Local Object Storage enablement,

➜➜ Automation Engine for customized workflow automation,

➜➜ Resource Management for advanced bandwidth control,

Agent resource throttling, and job queuing,

➜➜ Manager Bundle, which includes a standard Manager plus the

Automation Engine and Resource Management,

➜➜ High Availability for Manager redundancy,

➜➜ Remote Standby for Manager remote backup and standby.

SIGNIANT SAAS PRODUCTS

The Manager+Agents system is designed to interoperate with two

other Signiant products:

➜➜ Flight is a SaaS offering for accelerated transfers to and from the

public cloud;

➜➜ Media Shuttle is a SaaS offering that provides end users with a

simple means of sending and sharing large files.

Most large M&E enterprises use all three products, as illustrated on

the next page.



https://www.signiant.com/products/flight/?cc=wp-manager-agents

https://www.signiant.com/products/media-shuttle/?cc=wp-manager-agents


FIGURE 12: PORTFOLIO - ENTERPRISE DEPLOYMENT

CONCLUSIONIf you need to automate system-to-system file movement across your enterprise, Signiant Manager+Agents is the gold standard for speed, security, visibility and control. Designed to move large files over any public or private IP network and to connect to a range of on-premises and cloud storage, M+A’s unique architecture is capable of supporting thousands of endpoints from a central database, offering the lowest TCO at scale of any solution in its class. No other system is more flexible, powerful or reliable. To speak with a specialist about how Manager+Agents can support your global ecosystem, contact us here.

ABOUT SIGNIANT

Signiant is changing the way businesses move large, high-value digital assets around the world and into the cloud. Their on-premises software products were originally adopted by Media & Entertainment enterprises, pioneers in the electronic transport of large files. Over the last decade, Signiant has embraced cloud technology to create next-generation SaaS file transfer and cloud upload solutions with scalable, reliable, cost effective, and easy to deploy capabilities.

Today, Media & Entertainment are no longer alone in the need to move massive files, and Signiant’s rapidly growing customer base includes companies with digital assets ranging from satellite imagery and big data analytics to genome sequences and biotech research. Signiant’s technologies work for every size company to provide: accelerated file delivery up to 200 times faster than standard internet transfers; enterprise-class security along with full visibility and control of transfers and storage; and simple user-friendly tools. Find out more at www.signiant.com.

CUSTOMER NETWORK

PUBLIC CLOUDSaaS

SaaS

INTERNET



https://www.signiant.com/contact_us_page/?cc=wp-manager-agents


manager+agents - signiantinfo.signiant.com › rs › 134-qhz-485 › images › signiant... · in...

Documents