Approaching Package ManagerTimur Safin

Intersystems

Where to

find

components?

GitHub?

SourceForge?Developers

site?

How easy to find Caché components?

There is no single location,

and there is no easy way to install extension

Popular

languages all

have package

managers

Perl

Python

JavaScript

Ruby

Haskell

Importance of a package manager

Package manager is a key in language ecosystem popularity

Historical prospectivePackage managers for ages and for languages

Timeline

CTAN

CPAN

CRAN

FreeBSD Ports

NetBSD pkgsrc

DarwinPorts/MacPorts

PythonPI

RubyGems

Node.js NPM

MacOS HomeBrew

Apache Maven

NuGet

Chocolatey

Debian Dpkg

Red Hat RPM

Apple App Store

Android Market/Google Play

Jan-92 Jan-94 Jan-96 Jan-98 Jan-00 Jan-02 Jan-04 Jan-06 Jan-08 Jan-10 Jan-12

CPANPerl Package Manager

Experienced Perl programmers often comment that half of Perl'spower is in the CPAN. It has been called Perl's killer app. Though theTeX typesetting language has an equivalent, the CTAN (and in fact theCPAN's name is based on the CTAN), few languages have anexhaustive central repository for libraries. The PHP language has PECLand PEAR, Python has a PyPI (Python Package Index) repository, Rubyhas RubyGems, R has CRAN, Node.js has npm, Lua has LuaRocks,Haskell has Hackage and an associated installer/make clone cabal butnone of these are as large as the CPAN. … Other major languages, such as Java and C++, have nothing similar to the CPAN (though forJava there is central Maven).

CPAN

”

The CPAN has grown so large and comprehensive over the yearsthat Perl users are known to express surprise when they start toencounter topics for which a CPAN module doesn't exist already.

CPAN

”

• 90ties were years of source-based package managers

• CTAN is the 1st language distribution which established practice to install

contributed content (Tex packages and extensions) from central repository;

• Real success achieved when Perl community started to use this model in CPAN:

– since the moment of CPAN inception in the 1995 it collected “140,712 Perl

modules in 30,670 distributions, written by 11,811 authors, mirrored on 251

servers”

• Similar repository created for R (statistical language) and “surprisingly” called

CRAN;

– regardless the fact that R is "relatively rarely used" CRAN has accumulated

6000+ packages of extensions

CPAN, CRAN, CTAN

• Simple instruction to install a package

• This will proceed following steps:

– Query repository for package availability;

– Download latest package version;

– Build it (if necessary)

– Test it (if described how)

– And install generated/provided package files

CPAN, CRAN, CTAN

>cpan install HTTP::Proxy

• At the same middle 90-ies FreeBSD has introduced their “ports collection” as a way to

distribute open-source software

• BSD-derivatives (e.g. OpenBSD or NetBSD) maintained their own ports collections

– Basic mechanism is the same for all BSD:

• Sources to be installed from the selected media (CD-ROM, DVD or internet site);

• Product built using the Makefile given and compiler(s) available;

• Build targets installed according to the rules written in the Makefile or package

definition file;

• From implementation prospective Darwin Ports/MacPorts are BSD derivatives.

BSD: FreeBSD ports, NetBSD pkgsrc, and Darwin ports

cd /port/location; make install

• Recursive install is available

– There is an option to handle all dependencies of a given port if there was request,

so full installation for bigger package could be initiated via single command and package

manager will handle all the recursive dependencies appropriately.

e.g. for Darwin ports

BSD: FreeBSD ports, NetBSD pkgsrc, and Darwin ports

$ sudo port install apache2

OS Package

Managers

Source

Packages

CPAN CTAN CRAN BSD ports

Binary

Packages

Linux RPM Linux DpkgWindows

Nuget

Binary package managersLinux, Windows, all the rest

• With source-code based packages we had few obvious problems:

– Not all software could be deployed in their source form, there should be

convenient way to deploy binaries of a proprietary software;

– Building of a big project source tree may took a huge chunk of time (hours)

• People needed binary package managers which could distribute already built

and ready for consumption executables and binaries;

Request for binary package managers

• So Debian has introduced .deb format as 1st binary package format:

– .deb format used by Debian package manager (dpkg);

– original format introduced in Debian 0.93 in the March 1993 and was just the tar.gz with magic

prefixes;

– currently .deb package is just the AR archive consisting of 3 files (debian-binary with version,

control.tar.gz with metadata and data.tar.* with the installed files)

– Usually dpkg is not used directly but rather via APT (Advanced Packaging Tool)

• The apt-get’ reach functionality has influenced all later package managers

Linux: Debian and Red Hat

• RPM (Red Hat Package Manager) introduced with Red Hat V2.0 the late 1995

• Red Hat quickly became the most popular Linux distribution and solid RPM features was

one of factors winning competition here.

• RPM started to be used by all RedHat-based distributions (e.g. Mandriva, ASPLinux,

SUSE, Fedora or CentOS), and even beyond Linux by Novell Netware, or IBM AIX

• Similar to APT/dpkg there is Yum wrapper for RPM packages, which provides high-level

services for dependency tracking or building/versioning.

Linux: Debian and Red Hat

• Apple App Store has became 1st truly popular application store for mobile OS.

• App Store and Google Play are essentially OS specific package managers:

– With some extra API for online purchases;

– And some extra support for multiple hardware architectures. E.g. for Android application it

handles equally well Arm, x64, or MIPS devices

• Hardware optimizations/adaptations could be done at the installation time (Android

Lollipop ART engine), but this part of installation process is simply considered a part of

OS services.

“Package manager” handles this cross-platform optimization transparently to the user or

programmer;

Mobile sores: iOS App Store, Android Market/Google Play

There were several, not very popular, repositories of Windows applications:

– Windows Store for Windows Metro applications

– NuGet package manager for .NET extensions working via Visual Studio;

– Cygwin repository for Unix applications ported to Windows via Cygwin API;

But there has never been any generic solution like apt-get in Debian Linux;

Chocolatey Nuget has fixed this:

Windows applications: Chocolatey Nuget

Chocolatey is a package manager for Windows (like apt-get or yum but for Windows). It was designed to be decentralizedframework for quickly installing applications and tools that you need. It is built on the NuGet infrastructure currently using PowerShell as its focus for delivering packages from the distros to your door, err computer.

”

• There are multiple factors which have led to recent success of JavaScript as a server-side

language.

• One of them is the availability of central Node.js modules repository - NPM (Node Package

Manager) . Which is bundled with Node.js distribution since version 0.6.3 (November 2011).

• NPM is modeled similarly to CPAN, i.e. script running from command-line:

– connects to central repository;

– search for requested module;

– downloads appropriate version of package;

– parses package metainfo;

– and if there are external dependencies then process them recursively ;

Node.js NPM

C:\Users\Timur\Downloads>npm install -g less

npm http GET https://registry.npmjs.org/less

npm http 304 https://registry.npmjs.org/less

npm http GET https://registry.npmjs.org/graceful-fs

npm http GET https://registry.npmjs.org/mime

npm http GET https://registry.npmjs.org/request

…

npm http 200 https://registry.npmjs.org/asn1

npm http GET https://registry.npmjs.org/asn1/-/asn1-0.1.11.tgz

npm http 200 https://registry.npmjs.org/asn1/-/asn1-0.1.11.tgz

C:\Users\Timur\AppData\Roaming\npm\lessc -> C:\Users\Timur\AppData\Roaming\npm\node_modules\less\bin\lessc

less@2.0.0 C:\Users\Timur\AppData\Roaming\npm\node_modules\less

├── mime@1.2.11

├── graceful-fs@3.0.4

├── promise@6.0.1 (asap@1.0.0)

├── source-map@0.1.40 (amdefine@0.1.0)

├── mkdirp@0.5.0 (minimist@0.0.8)

└── request@2.47.0 (caseless@0.6.0, forever-agent@0.5.2, aws-sign2@0.5.0, json-stringify-safe@5.0.0, tunnel-agent@0.4.0, stringstream@0.0.4, oauth-sign@0.4.0, node-uuid@1.4.1, mime-types@1.0.2, qs@2.3.2, form-data @0.1.4, tough-cookie@0.12.1, hawk@1.1.1, combined-stream@0.0.7, bl@0.9.3, http-signature@0.10.0)

Node.js NPM

Creating Caché Package ManagerOk, all this good, but what about Caché ObjectScript?

Ok, you buy importance of package manager in COS and

want it now, so…

• What should we put to package container?

• And how all this infrastructure will work?

• Which container format is the suitable for our packages?

– ZIP would be ideal

– But even Studio Project XML will fit the purpose

• even now it handles all interesting file types (CLS, RTN, INC, CSP, ZEN, CSS, GIF, etc.)

• Yes, XML is very inefficient, bloated, and keeps binary data as base64

(i.e. 3/2x length increase)

• But it’s not a big deal for initial step;

Container Format

Which

Container?

ZIP

XML

• What information should be in metadata?

– At least dependency on other packages (name, version, url);

– Anything from Node.js package.json could be reused by us, but;

• JSON is not yet natively supported now,

so could start from XML serialization of the same information (metainfo.xml)

Metadata File

{ "dependencies" :

{ "foo" : "1.0.0 - 2.9999.9999"

, "bar" : ">=1.0.2 <2.1.2"

, "qux" : "<1.0.0 || >=2.3.1 <2.4.5 || >=2.5.2 <3.0.0"

, "asd" : "http://asdf.com/asdf.tar.gz"

, "til" : "~1.2"

, "elf" : "~1.2.3"

, "lat" : "latest"

, "dyl" : "file:../dyl"

}

}

TOML

Tom's Obvious, Minimal Language.

By Tom Preston-Werner.

Objectives

TOML aims to be a minimal configuration

file format that's easy to read due to obvious

semantics. TOML is designed to map

unambiguously to a hash table. TOML

should be easy to parse into data structures

in a wide variety of languages.

Metadata File – Alternate Format

# This is a TOML document. Boom.

title = "TOML Example"

[owner]

name = "Lance Uppercut"

dob = 1979-05-27T07:32:00-08:00

[database]

server = "192.168.1.1"

ports = [ 8001, 8001, 8002 ]

enabled = true

[servers]

[servers.alpha]

ip = "10.0.0.1"

dc = "eqdc10"

[servers.beta]

ip = "10.0.0.2"

• TOML is easy and clean

– Kind of “old-good” INI format but semantically equivalent to XML and JSON

as serialization format;

• But there is no TOML parser in COS written yet;

– So once it’s ready (probably as a separate “package”) we will add appropriate dependency and

will consume TOML as metadata format.

– Not required now in V1.0;

Metadata File – Alternate Format

• There is tough problem to handle – dependency on a system packages which may be

presented only on some product flavors:

– iKnow,

– Ensemble

– DeepSee,

– TrakCare,

– HealthShare,

– particular version of product

• May mark this dependency in the general dependency list as “iKnow”, “Ensemble

2013.1+”, etc.

• But it’s too early to handle in the 1st iteration - we will revisit this later.

Dependency on a system classes

• CPAN is successful because we could easily deploy modules which are implemented in C

(some optimized libraries, OS calls, etc.);

– Perl XS API is for such Perl/C modules;

– Cache’ callout API is very similar in their nature, but much more complicated;

– Perl provides a lot of handy utilities which simplify XS modules development, e.g.

• H2xs - preprocessor which generates XS header file using given C header file;

• Xsubpp – preprocessor to convert XS code to pure C code, etc.

• Beyond default callout mechanism there are a couple of more advanced ways fro FFI:

– Builtin FFI mechanism used, for example, by iKnow libraries;

– Or CNA (Caché Native Access) callout wrapper which uses LibFFI mechanism for easy calling

of any C-runtime function from the COS code;

[Cross-platform] binary modules

From the practical prospective, taking into accountmultiple Caché platforms we should handle equallywell (Windows, Linux, Mac OS X, or even VMS), andthe fact that these FFI mechanisms are not yetofficially supported, we should admit that they are notready yet, and could not be recommended as a wayto handle deployment of mixed C/COS packages.

We are not implementing them in v1.0

[Cross-platform] binary modules

• Having builtin mechanism for unit-testing has impacted positively the whole stability of

CPAN ecosystem.

– Package will not be installed at the user system unless it’s passing internal testing;

• V1.0 will ignore unit-testing for a moment, but later, for binary modules deployment we

should have unit-testing working

Unit-testing

• User experience is a big success booster, if system is inconvenient then it may be

left unnoticed*

• So both ways to invoke package manager should be working

– To invoke package manager shell from any namespace in Cache’ terminal, .e.g

– To install COS packages from command-line, e.g.

• Eventually these operation might be wrapped in GUI. Not in V1.0

Command-line access

do ^%CPM>install TOML-Parser

cpm install deepsee-mobile-server

• There used to be old-dark times when every site had to handle DDoS on their own way,

and to keep geo-spread mirror network for their purposes;

• Now all is easily handled by VM host providers or special CDN providers:

– Amazon Azure, Google Cloud, etc;

– Amazon CloudFront, MaxCDN, etc;

• Once we have static files (distributions files, static html descriptions, static images, etc) it

could be easily handled by CDN;

• But… for community site we supposed to have some dynamic part which will work with

authors and new contributions

Advices are welcome how to handle this mix easily and in most effective way

Mirroring and CDN

Suggested usage cases for V1.0

Package User COS System COS Binaries Good for v1.0

iKnowSocial Yes No No Good

Atom-COS-Studio Yes Yes Yes

(Atom +

extension)

Bad

WebTerminal No Yes No Good

CNA Yes No Yes

(callout module)

Bad

Coslint Yes

(deployed

OBJ code)

No No Probably

(deployed)

CPM: Contributors Invited!

• https://github.com/intersystems-ru/CPM

– Private repository made public today;

– Only 2 contributors so far;

– More is always welcome!

CPM: Contributors Invited!

Идём?