perforce best practices @ sap · sap ag 2006, perforce best practices @ sap / c. loff & t....

Perforce Best Practices @ SAP

Claudia Loff, SAP AGThomas Kroll, SAP AG

Walldorf, Germany, 2006/08/14

SAP AG 2006, Perforce Best Practices @ SAP / C. Loff & T. Kroll / 2

Delcho Christian Haydar Sherry

Milchev Schmitt-Plank Asan Wu

Presenters and the other team members

Thomas KrollThomas KrollThomas KrollThomas KrollThomas KrollThomas KrollThomas KrollThomas Kroll

Thomas graduated and finished his studies of computer science at the BA of Mannheim, Germany for IBM, in 2001.

He joined SAP in October 2001 where he was in charge of the C/++ build infrastructure on Windows platforms used for the SAP products like R/3, SAP GUI & Business One.

End of 2004 he took over the responsibility for the Perforce team and Perforce infrastructure at SAP.

Claudia LoffClaudia LoffClaudia LoffClaudia LoffClaudia LoffClaudia LoffClaudia LoffClaudia Loff

Claudia is project lead of the "Perforce Improvement Project" in the Perforce team within SAP NetWeaver Lifecycle Management department.

Before joining SAP in 1997, Claudia was a developer and project lead for more than 6 years at two mid-sized software companies. Prior to working with the Perforce team at SAP, she held several positions as

developer and project lead within SAP.

Claudia holds a masters degree of computer science from the University of Karlsruhe, Germany and took "business administration for engineers" class at University of Hagen, Germany.

Monitoring

Infrastructure introduction

History & key figures

Hardware configuration

PART IPresented by Thomas

Lessons Learned (Improvement Project)

Codeline Concept & Consolidation

Authorization Management, Mass Handling

PART IIPresented by Claudia Loff

Monitoring






History of Perforce within SAP

Evaluation of SCM systems

First Perforce server at SAP

Major Hardware investments & performance improvements

P4MS – PerforceManagementSystem running on SAP

NetWeaver

�

�

+2005

2006.1

Upgrade of all instances

4,600+

Registered Perforce users

80+

Perforce server instances

�

Productive move from ClearCase into Perforce�1999

1998

Tool infrastructure development & maintenance�’00-’04


Monthly transfered volume via P4 protocol

North AmericaPalo Alto and satellites

IndiaBangalore

BulgariaSofia

CanadaMontréal

ChinaShanghai

IsraelRa’anana

GermanyWalldorf and satellites

SAP's Global R&D Organization

HungaryBudapest

45.79%45.79%

17.74%17.74%

13.52%13.52%

6.55%6.55%

3.69%3.69%

4.57%4.57%

2.74 TB9.31 TBOverall

13.72 GB954.55 GBIsrael

0.18 GB11.70 GBIndia

0.68 GB31.45 GBBulgaria

InboundInboundInboundInboundInboundInboundInboundInboundOutboundOutboundOutboundOutboundOutboundOutboundOutboundOutboundLocationLocationLocationLocationLocationLocationLocationLocation


Key figures – User/License growth

Perforce users & licenses

USERS SERVERS FILES/REV CURRENT

+11.6% users

last year

LEVELS

700

1,200

1,700

2,200

2,700

3,200

3,700

4,200

4,700

1999

/12/

0220

00/0

3/02

2000

/06/

0220

00/0

9/02

2000

/12/

0220

01/0

3/02

2001

/06/

0220

01/0

9/02

2001

/12/

0220

02/0

3/02

2002

/06/

0220

02/0

9/02

2002

/12/

0220

03/0

3/02

2003

/06/

0220

03/0

9/02

2003

/12/

0220

04/0

3/02

2004

/06/

0220

04/0

9/02

2004

/12/

0220

05/0

3/02

2005

/06/

0220

05/0

9/02

2005

/12/

0220

06/0

3/02

2006

/06/

0220

06/0

9/02

2006

/12/

02


Key figures – Server growth

Perforce servers

USERS SERVERS FILES/REV CURRENT

+11,6% users

last year

LEVELS

~8 new

servers

per year

0

10

20

30

40

50

60

70

80

90

100

by 31.12.2000 by 31.12.2001 by 31.12.2002 by 31.12.2003 by 31.12.2004 by 31.12.2005* by 12.06.2006


Key figures – Project & codeline growth

Perforce projects (2nd) & codelines (3rd)

USERS SERVERS FILES/REV CURRENTLEVELS

0

1.000

2.000

3.000

4.000

5.000

6.000

7.000

by 31.12.2000 by 31.12.2001 by 31.12.2002 by 31.12.2003 by 31.12.2004 by 17.11.05 by 12.06.2006

Projects

Codelines

~2,000 new

projects

last year

+11,6% users

last year

~8 new

servers

per year


Key figures – File & revision growth

Perforce files & revisions


0

20.000.000

40.000.000

60.000.000

80.000.000

100.000.000

120.000.000

140.000.000

by 31.12.2000 by 31.12.2001 by 31.12.2002 by 31.12.2003 by 31.12.2004 by 17.11.05 by 12.06.2006

Files

Revisions/Activity

~2.000 new

projects

last year

+11,6% users

last year

~8 new

servers

per year

>10 Mio files

added

last year


Key figures – Current overview

Current overview


~2.000 new

projects

last year

+11,6% users

last year

~8 new

servers

per year

>10 Mio files

added

last year

4,399

79

333

6,143

6,498

55,787

46,972,123

116,482,290

1.80 TB

2,303,174

� Active Users / Licenses

� Server instances

� Depots (1st level)

� Projects (2nd level)

� Codelines unique

� Codelines overall (3rd level)

� Files

� Revisions

� Filesystem consumption

� Average number of p4 calls per day

~27 p4 calls

per second

Monitoring






Hardware setup

HA ensured byPRIMECLUSTER

� Cluster node 1 & 2:

� FSC PrimePower 450N

– CPU: 4 x 1320 MHz

– Memory: 32GB

– OS: SunOS 9

� Cluster node 3 & 4:

� FSC PrimePower 650

– CPU: 8 x 675 MHz

– Memory: 32GB

– OS: SunOS 9

� Shared Storage (SAN):

� 2 x EMC Clariion CX500 (mirrored)

– Size: 3TB each


HA Setup

Location I (ROT)Location I (ROT)Location I (ROT)Location I (ROT)Location I (ROT)Location I (ROT)Location I (ROT)Location I (ROT)

Cluster Node 1

Cluster Node 3

EMC I

2 Fibre Channel SAN switches

Redundant 2 Gbit/s Fibre Channel connections

Location II (WDF)Location II (WDF)Location II (WDF)Location II (WDF)Location II (WDF)Location II (WDF)Location II (WDF)Location II (WDF)

Cluster Node 2

Cluster Node 4

EMC II

2 Fibre Channel SAN switches

Mirroring filesystem per service

Redundant Gbit/s Ethernet connections


� Two times a week an incremental backup per Perforce service:

� Downtime: 1-2 minutes per service

� On the weekend full backup of each Perforce service

� Downtime: depending on size, approx. 2 hours

Backup strategy

stop p4d

filesystemsnapshot

start p4d

checkpointFull backup

snapshot

filesystemsnapshot

start p4d

Incr. backupsnapshot

stop p4d

� Time for complete restore from backup: ~3 hours

� Maximum loss of history: 2 workdays

Monitoring






Monitoring

� Low-level monitoring by PRIMECLUSTER

� Failover in case of problems (e.g. filesystem, HW failure)� restart on other node

Clu

ste

r/H

W

� Monitoring of OS parameters

� CPU & memory consumption

� Monitor filesystem usage => increase of disk space

� Warnings in case a service is not responding

Nag

ios

� Real time monitoring of p4 servers (based on debug level 3 logs)

� display all currently running processes on all instances

� graphical & tabularly representation per server (TOP10)

� notification on reaching warning and error values

� find bottlenecks (e.g. commands, clients etc.)

Perf

orm

an

ce M

on

ito

r


Monitoring via P4PM

Monitoring






Complete infrastructure at a glance

serverinstances

protections & triggers

P4MS – PerforceManagementSystem

self services & web representation

Exte

rnal to

ols

lik

e J

av

aC

orr

ecti

on

Wo

rkb

an

ch

(D

TI)

or

dev

elo

per

dash

bo

ardP4SAP backend

Configuration

Consolidation

Transport

several scripts

Performance Monitor

P4SAP


Complete infrastructure at a glance – inside SAP infrastructure

serverinstances


P4MS – PerforceManagementSystem

self services & web representation

Exte

rnal to

ols

lik

e J

av

aC

orr

ecti

on

Wo

rkb

an

ch

(D

TI)

or

dev

elo

per

dash

bo

ardP4SAP backend

Configuration

Consolidation

Transport

several scripts

Performance Monitor

P4SAP

Fix directory structure

//depot/project/codeline


Maturity

dev

t

Production process (development phase)

Dev

� Submit changes to ‚dev‘ after local test

� nightly build, central deployment to test system and test

� Release change for transport to ‚export‘

Export

� Collect changes of ‚dev‘ (gate to patch)

� Transport (copy) content to ‚patch‘ in regular intervals (e.g. weekly cycle) using a timestamp

Patch

� Small group of developers is allowed to apply fixes for stabilization

� approval by Quality Manager(s)

� Transport (copy) content to ‚cons‘

Cons (consolidated code line)

� ready to be used by dependentcomponents/products

cons

export

patch

develop

collect, synchronize

last call

delivery


Depot/Directory Structure

depot

project

codeline


☺☺☺☺ ��

Dependencies between changes

c1...

Call method A...

c2Method A {...}

...

c3...

Call method A...

Some fixes

* For developers, a „bundle“ is the answer to all questions;see also: D. Adams „The Hitchhiker's Guide to the Galaxy “

...Call method A

...

Method A {...}...

...Call method A

...Some fixes

��Bundle 42*

c1

c2

c3

dev export


Transports between the code lines

Integration

� manual transport (p4 integrate)

� Tools provided by Perforce (p4.exe, P4Win and P4V)

� Ignores dependencies between changes

� Used to create identical codelines = full copy e.g. initial creation of (remote) maintenance codelines out of development codelines (P4MS)

Consolidation

� controlled integration

� Tools developed within SAP based on „p4 integrate“

� takes dependencies between changes into account

� allows bundling of changes– bundle = set of changes that have to be transported together (e.g. c1 and c2)

� shows status, user and timestamp of a transport– N (new), P (consolidation in process), E (error),

W (waiting for dependent change), C (consolidated)

� No multi-server so far

Perforce@SAP


Perforce@SAP infrastructure – Fill Process

Runs every 5 minutes for each Perforce server

� collect changes of consolidation codelines and write them into thedatabase ‚P4SAP‘

– one change list in P4 = one consobject per branch(!) with status ‚N‘ in P4SAP

– preparation for consolidation frontend (SelfService)

� determine dependencies and write them into P4SAP

serverinstances


//depot/project/codeline

P4SAP backend

P4SAP


Perforce@SAP infrastructure – Consolidation Processes

Process Steps

� Look for the requested changes in P4SAP and determinetarget branch

� Determine dependencies- Any unconsolidated predecessors or bundle siblings?

� P4 integrate

� set status to ‚C‘

� Look for successors in status ‚W‘aiting and start consolidation for them

� send notification via eMail to requester (P4MS)

Preparation

Transport

Post

Processing

-> Set status to ‘W’ (waiting) ��

��


Authorization concept/User management

Authorization concept with central user and group management

� Central user/group management via database ‚P4SAP‘ using update of user data from SAP Corporate Portal

� Set password on all p4servers (P4MS), password-check

� Consolidation rights

� Protection table: keep it small for good performance! -> P4 authorization concept (=> more than 2000 -> 100 entries)

Perforce standard user and group management; protection table

Perforce@SAP� more than 80 Perforce Servers� more than 4000 users� additional processes via SAP infrastructure


Mass Handling

Mass transport via Perforce Management System (P4MS)

� background jobs (CONSOLIDATION, INTEGRATE, COPY, PATCH)

� scheduled and monitored via job monitor

� eMail notification is send to creator after the job is finished

� Mass consolidation may last very long: up to 9h for 236 changes in 229 projects

� Mass integration may last very long: 1 hour on a local server, up to 9.5 hours for 159,155 files in 154 projects for remote depots

Local small transports

Perforce@SAP� Synchronized milestones => synchronized transports

(250+ projects at the same time/timestamp)� Central responsibility for transports

(central Quality Management)


Perforce@SAP Improvement project 2005-2006 (lessons learned)

Challenges and Mitigation

� Performance, Performance, Performance, …

– Perforce Cluster

– Performance monitor (identify „bad“ calls; push better solutions)

– Authorization concept (keep protection tables small)

– Migrate to new version of operating system and database

– Separate Perforce servers for maintenance codelines- Better performance for development codelines due to less codelines

- Less need to integrate from dev to maintenance codeline (remote depot)

� Increase productivity and agility (more codelines, masshandling,...)

– Perforce Management System (P4MS)

� Multi-Perforce Server Environment

– Perforce Management System (P4MS)

– Submit template (track change lists via service request number)

� Complex (intransparent) dependency networks for change lists

– Graphical representation


Q&AQ&AQ&AQ&A

Questions?


For your For your For your For your

referencereferencereferencereference

Details


Real life example: Codelines for NW04s (working titles)

645_SP_TEST

645_SP_COR

645_SP_REL

645_VAL_COR

645_VAL_REL

645_COR

645_REL

Customer (SAP Service Marketplace)

Service Pack 04...08Service Pack 09Service Pack 10Service Pack 11

correction validation patch

consolidationby developer

consolidationby reponsible

consolidationby

Final Assembly

consolidationby

Final Assembly

Delivery by Final Assembly

Several p4 servers

perforce3645

NW04S_04_RELNW04S_04_COR

copy byPerforce Team

ordered byFinal Assemblyor integrate by

Responsible/dev

copy byPerforce Team

ordered byFinal Assembly






Perforce@SAP infrastructure – Modules&Configuration

Modules

� P4SAP backend

– Data base „P4SAP“ (MaxDB™)- Configuration- Consolidation objects (created from change lists of Perforce servers)

– LINUX environment for processes

� SelfServices (web server)

� Perforce Management System (P4MS)– New SelfService and main tool for Perforce Team, now

– Running on SAP NetWeaver

Configuration

� Meta data for projects– Name, creator, description...

– Perforce Server

– Codelines- P4 Trigger ensures that the first 3 level of a Perforce directory can be created by Perforce Team,

only

� Meta Data for consolidation (consolidation branches)- development branch in Perforce server- Definition of source and target codeline (per project)- naming convention: <PROJECT>_<CODELINE>

� Authorization, User and Group Data


Perforce@SAP infrastructure – Consolidation Processes

Task

� transport change list(s) in a fix sequence and along defined tracks

Triggered by …

� SelfService (web server)

� Perforce Management System (P4MS)

� Command line (used by tools like build environment)

Restrictions

� Source and target codeline have to be on the same Perforce server


Perforce Authorization concept (extract)

All codelines which are part of one Release arehandled with the same concept.

…

NW04s_XX_REL

NW04s_XX_COR

645_REL

645_COR

645_VAL_REL

645_VAL_COR

645_SP_REL

645_SP_TEST

645_SP_COR

Codeline ☺☺☺☺☺☺☺☺

645-patcher

☺☺☺☺☺☺☺☺☺☺☺☺

645-write (Q-team)

☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺

645-admin (layer/comp resp)

☺☺☺☺☺☺☺☺

☺☺☺☺

☺☺☺☺

☺☺☺☺☺☺☺☺ ☺☺☺☺☺☺☺☺

☺☺☺☺☺☺☺☺

645-dev

write(submit)

645_COR_<layer>-cons

645_TEST_<layer>-cons

Final Assembly, only



open (edit)

write (submit)

Maintaingroup

members

open group 645-dev * //*/*/645*_COR/...write group 645-write * //*/*/645*_COR/...write group 645-patcher * //*/*/645_SP_COR/...


Mass Consolidation using P4MS

Job type(s)

� CONSOLIDATION

Select criteria� User, project, codeline, collection

Input data� Changes

Prerequisite

� Change lists are filled into database ‚P4MS‘

Restriction� Source and target codeline have to be on the same Perforce server

Consolidation for collections with many projects may have a duration of…� up to 9h (236 changes in 229 projects)


Mass Integration using P4MS

Job type(s)

� INTEGRATE integrate sources

� COPY create an identical copy in the target branch (delete files, if needed)

� PATCH integrate change list(s)

Select criteria

� Codeline

� Collection

� Change list

� Project

� Timestamp

Input data

� target codeline

� P4MS knows on which Perforce server the target codeline for a project is

Prerequisite

� Existance of target codeline– For new codelines all meta data in the database „P4SAP“ is created on a fingertip by Perforce Team

Integration for collections with many projects may have a duration of …� more than 1 hour on the same server

� up to 9.5 h for transports to different servers (154 projects, 159,155 files)


Perforce@SAP Improvement project 2005-2006 (lessons learned)

Planned improvements

� Perforce Cluster

– Upgrade to Perforce 2005.2 or higher (p4 obliterate)

– Speed up technical Perforce access for colleagues in remote locations

– LDAP connection

� Perforce@SAP infrastructure

– Service Queue

- Integration of p4 processes in automatic workflows

- Migrate SelfServices from Webserver to SAP WebDynpro resp. webservices

– Change tracking between different servers


Copyright 2006 SAP AG. All Rights Reserved

No part of this publication may be reproduced or transmitted in any form or for any purpose without the express permission of SAP AG. The information contained herein may be changed without prior notice.

Some software products marketed by SAP AG and its distributors contain proprietary software components of other software vendors.

Microsoft, Windows, Outlook, and PowerPoint are registered trademarks of Microsoft Corporation.

IBM, DB2, DB2 Universal Database, OS/2, Parallel Sysplex, MVS/ESA, AIX, S/390, AS/400, OS/390, OS/400, iSeries, pSeries, xSeries, zSeries, z/OS, AFP, Intelligent Miner, WebSphere, Netfinity, Tivoli, and Informix are trademarks or registered trademarks of IBM Corporation.

Oracle is a registered trademark of Oracle Corporation.

UNIX, X/Open, OSF/1, and Motif are registered trademarks of the Open Group.

Citrix, ICA, Program Neighborhood, MetaFrame, WinFrame, VideoFrame, and MultiWin are trademarks or registered trademarks of Citrix Systems, Inc.

HTML, XML, XHTML and W3C are trademarks or registered trademarks of W3C®, World Wide Web Consortium, Massachusetts Institute of Technology.

Java is a registered trademark of Sun Microsystems, Inc.

JavaScript is a registered trademark of Sun Microsystems, Inc., used under license for technology invented and implemented by Netscape.

MaxDB is a trademark of MySQL AB, Sweden.

SAP, R/3, mySAP, mySAP.com, xApps, xApp, SAP NetWeaver, and other SAP products and services mentioned herein as well as their respective logos are trademarks or registered trademarks of SAP AG in Germany and in several other countries all over the world. All other product and service names mentioned are the trademarks of their respective companies. Data contained in this document serves informational purposes only. National product specifications may vary.

The information in this document is proprietary to SAP. No part of this document may be reproduced, copied, or transmitted in any form or for any purpose without the express prior written permission of SAP AG.

This document is a preliminary version and not subject to your license agreement or any other agreement with SAP. This document contains only intended strategies, developments, and functionalities of the SAP® product and is not intended to be binding upon SAP to any particular course of business, product strategy, and/or development. Please note that this document is subject to change and may be changed by SAP at any time without notice.

SAP assumes no responsibility for errors or omissions in this document. SAP does not warrant the accuracy or completeness of the information, text, graphics, links, or other items contained within this material. This document is provided without a warranty of any kind, either express or implied, including but not limited to the implied warranties of merchantability, fitness for a particular purpose, or non-infringement.

SAP shall have no liability for damages of any kind including without limitation direct, special, indirect, or consequential damages that may result from the use of these materials. This limitation shall not apply in cases of intent or gross negligence.

The statutory liability for personal injury and defective products is not affected. SAP has no control over the information that you may access through the use of hot links contained in these materials and does not endorse your use of third-party Web pages nor provide any warranty whatsoever relating to third-party Web pages.


Copyright 2006 SAP AG. Alle Rechte vorbehalten

Weitergabe und Vervielfältigung dieser Publikation oder von Teilen daraus sind, zu welchem Zweck und in welcher Form auch immer, ohne die ausdrückliche schriftliche Genehmigung durch SAP AG nicht gestattet. In dieser Publikation enthaltene Informationen können ohne vorherige Ankündigung geändert werden.

Die von SAP AG oder deren Vertriebsfirmen angebotenen Softwareprodukte können Softwarekomponenten auch anderer Softwarehersteller enthalten.

Microsoft®, WINDOWS®, NT®, EXCEL®, Word®, PowerPoint® und SQL Server® sind eingetragene Marken der Microsoft Corporation.

IBM®, DB2®, DB2 Universal Database, OS/2®, Parallel Sysplex®, MVS/ESA, AIX®, S/390®, AS/400®, OS/390®, OS/400®, iSeries, pSeries, xSeries, zSeries, z/OS, AFP, Intelligent Miner, WebSphere®, Netfinity®, Tivoli®, Informix und Informix® Dynamic ServerTM sind Marken der IBM Corporation.

ORACLE® ist eine eingetragene Marke der ORACLE Corporation.

UNIX®, X/Open®, OSF/1® und Motif® sind eingetragene Marken der Open Group.

Citrix®, das Citrix-Logo, ICA®, Program Neighborhood®, MetaFrame®, WinFrame®, VideoFrame®, MultiWin® und andere hier erwähnte Namen von Citrix-Produkten sind Marken von Citrix Systems, Inc.

HTML, DHTML, XML, XHTML sind Marken oder eingetragene Marken des W3C®, World Wide Web Consortium, Massachusetts Institute of Technology.

JAVA® ist eine eingetragene Marke der Sun Microsystems, Inc.

JAVASCRIPT® ist eine eingetragene Marke der Sun Microsystems, Inc., verwendet unter der Lizenz der von Netscape entwickelten und implementierten Technologie.

MaxDB ist eine Marke von MySQL AB, Schweden.

SAP, R/3, mySAP, mySAP.com, xApps, xApp, SAP NetWeaver, und weitere im Text erwähnte SAP-Produkte und -Dienstleistungen sowie die entsprechenden Logos sind Marken oder eingetragene Marken der SAP AG in Deutschland und anderen Ländern weltweit. Alle anderen Namen von Produkten und Dienstleistungen sind Marken der jeweiligen Firmen. Die Angaben im Text sind unverbindlich und dienen lediglich zu Informationszwecken. Produkte können länderspezifische Unterschiede aufweisen.

Die in dieser Publikation enthaltene Information ist Eigentum der SAP. Weitergabe und Vervielfältigung dieser Publikation oder von Teilen daraus sind, zu welchem Zweck und in welcher Form auch immer, nur mit ausdrücklicher schriftlicher Genehmigung durch SAP AG gestattet.

Bei dieser Publikation handelt es sich um eine vorläufige Version, die nicht Ihrem gültigen Lizenzvertrag oder anderen Vereinbarungen mit SAP unterliegt. Diese Publikation enthält nur vorgesehene Strategien, Entwicklungen und Funktionen des SAP®-Produkts. SAP entsteht aus dieser Publikation keine Verpflichtung zu einer bestimmten Geschäfts- oder Produktstrategie und/oder bestimmten Entwicklungen. Diese Publikation kann von SAP jederzeit ohne vorherige Ankündigung geändert werden.

SAP übernimmt keine Haftung für Fehler oder Auslassungen in dieser Publikation. Des Weiteren übernimmt SAP keine Garantie für die Exaktheit oder Vollständigkeit der Informationen, Texte, Grafiken, Links und sonstigen in dieser Publikation enthaltenen Elementen. Diese Publikation wird ohne jegliche Gewähr, weder ausdrücklich noch stillschweigend, bereitgestellt. Dies gilt u. a., aber nicht ausschließlich, hinsichtlich der Gewährleistung der Marktgängigkeit und der Eignung für einen bestimmten Zweck sowie für die Gewährleistung der Nichtverletzung geltenden Rechts.

SAP haftet nicht für entstandene Schäden. Dies gilt u. a. und uneingeschränkt für konkrete, besondere und mittelbare Schäden oder Folgeschäden, die aus der Nutzung dieser Materialien entstehen können. Diese Einschränkung gilt nicht bei Vorsatz oder grober Fahrlässigkeit.

Die gesetzliche Haftung bei Personenschäden oder Produkthaftung bleibt unberührt. Die Informationen, auf die Sie möglicherweise über die in diesem Material enthaltenen Hotlinkszugreifen, unterliegen nicht dem Einfluss von SAP, und SAP unterstützt nicht die Nutzung von Internetseiten Dritter durch Sie und gibt keinerlei Gewährleistungen oder Zusagen über Internetseiten Dritter ab.

perforce best practices @ sap · sap ag 2006, perforce best practices @ sap / c. loff & t....

Documents