How CPAN Testers helped meimprove my module

It’s quite hard to write cross-platform CPANmodules, especially when you use XS to interfacewith C libraries. Luckily, CPAN Testers tests yourmodules on many platforms for you. Come seehow CPAN Testers helped me to create a fully

portable module.Léon Brocard

YAPC::Europe 2011

Who went to. . .

Barbie’s talk “Smoking�e Onion — Tales of CPANTesters”

Who knows about. . .

CPAN Testers

MeLéon Brocard

French, live in London

Like food

Like the colour orange

Founded Amsterdam.pm, Bath.pm, Croydon.pm

Ex-leader of London.pm

Started YAPC::Europe

Looking for employment

Perl hacker

Released Perl

2004-02-23 Perl 5.005_04

2009-11-20 Perl 5.11.2

2010-04-20 Perl 5.13.0

2011-06-20 Perl 5.12.4

83 distributions on the CPAN

Acme-Buffy, Acme-Colour, App-Cache, Archive-Peek, Bisect-Perl-UsingGit,Catalyst-Plugin-CookiedSession, Catalyst-Plugin-SimpleAuth, Compress-LZF_PP,Compress-LZMA-External, CPAN-IndexPod, CPAN-Metadata-RDF, CPAN-Mini-Live,CPAN-Unpack, Crypt-Skip32-Base32Crockford, Crypt-Skip32-Base64URLSafe, Dackup,Data-Page, Data-UUID-Base64URLSafe, DateTime-Stringify, Devel-ebug, Devel-ebug-HTTP,Devel-Profit, Email-Send-Gandi, Email-Send-Gmail, File-Copy-Reliable, Fir,Games-GuessWord, GraphViz, Haul, HTML-Fraction, HTTP-Server-Simple-Kwiki,Image-Imlib2, Image-Imlib2-Thumbnail, Image-Imlib2-Thumbnail-S3, Image-WorldMap,Java-JVM-Classfile, JSON-XS-VersionOneAndTwo, JSON-YAJL, Kasago, Language-Functional,LWP-ConnCache-MaxKeepAliveRequests, Mac-EyeTV, MealMaster, Messaging-Courier,Module-CPANTS-Generator, Module-Packaged, MP3-ID3Lib, Net-Amazon-S3,Net-Amazon-S3-Client-GPG, Net-Amazon-SimpleQueue, Net-Cassandra, Net-DPAP-Client,Net-FleetDB, Net-FTP-Throttle, Net-LastFM, Net-MythTV, Net-MythWeb, Net-OpenDHT,Net-VNC, Number-DataRate, OpenFrame-Segment-Apache, OpenFrame-Segment-Apache2,Parse-BACKPAN-Packages, Parse-CPAN-Authors, Parse-CPAN-Ratings, Perl-Metric-Basic,PPIx-IndexOffsets, PPIx-LineToSub, Search-Mousse, String-Koremutake,Template-Plugin-Page, Template-Stash-Strict, Term-ProgressBar-Quiet, Test-Expect,Test-WWW-Mechanize-PSGI, Tie-GHash, Tree-Ternary_XS, TV-Anytime, WWW-Gazetteer,WWW-Gazetteer-FallingRain, WWW-Gazetteer-Getty, WWW-Mechanize-Timed,WWW-Search-Google

Out of date

Written and released during conference:CPAN::Mirror::Finder

CPAN::Webserver

Data structures

{double => 4,false => 0,integer => 123,map => {

array => [ 1, 2, 3 ],key => "value"

},null => undef,number => "3.141",string => "a string",string2 => "another string",true => 1

};

Data::Dumper

{double => 4,false => 0,integer => 123,map => {

array => [ 1, 2, 3 ],key => "value"

},null => undef,number => "3.141",string => "a string",string2 => "another string",true => 1

};

YAML Ain’t Markup Language---double: 4false: 0integer: 123map:array:

- 1- 2- 3

key: valuenull: ~number: 3.141string: a stringstring2: another stringtrue: 1

Extensible Markup Language<?xml version="1.0" encoding="UTF-8"?><map><integer>123</integer><double>4</double><number>3.141</number><string>a string</string><string>another string</string><null/><true/><false/><map><string>value</string><array>

<number>1</number><number>2</number><number>3</number>

</array></map></map>

JavaScript Object Notation

{"integer":123,"double":4,"number":3.141,"string":"a string","string2":"another string","null":null,"true":true,"false":false,"map":{"key":"value","array":[1,2,3]}

}

XML<?xml version="1.0" encoding="UTF-8"?><map><integer>123</integer><double>4</double><number>3.141</number><string>a string</string><string>another string</string><null/><true/><false/><map><string>value</string><array>

<number>1</number><number>2</number><number>3</number>

</array></map></map>

Simple API for XML

my $b = XML::LibXML::SAX::Builder->new();$b->start_document;$b->start_element( { Name => ’number’ } );$b->characters( { Data => ’3.141’ } );$b->end_element( { Name => ’number’ } );$b->end_document;say $b->result->toString;

# generates:<?xml version="1.0"?><number>3.141</number>

Yet Another JSON Library

“YAJL is a small event-driven (SAX-style) JSONparser written in ANSI C, and a small validatingJSON generator”http://lloyd.github.com/yajl/

YAJL exampleyajl_gen g;const unsigned char * buf;size_t len;

g = yajl_gen_alloc(NULL);yajl_gen_map_open(g);yajl_gen_string(g, "number", 6);yajl_gen_number(g, "3.141", 5);yajl_gen_map_close(g);

yajl_gen_get_buf(g, &buf, &len);# {"number":3.141}fwrite(buf, 1, len, stdout);yajl_gen_clear(g);yajl_gen_free(g);

A dream

my $generator = JSON::YAJL::Generator->new();$generator->map_open();$generator->string("number");$generator->number("3.141");$generator->map_close();say $generator->get_buf;# {"number":3.141}

Wait, C?

XS – eXternal Subroutine

h2xs -x

SWIG – Simpli�ed Wrapper and Interface Generator

FFI – Foreign Function Interface

Hack hack hack

Perl is written in C with macros

perlguts — Introduction to the Perl API

perlxs — XS language reference manual

perlxstut — Tutorial for writing XSUBs

perlport —Writing portable Perl

Works on my machine, so ship it

0.01 Mon Apr 11 11:46:04 CEST 2011

CPAN Testers

CPAN Testers is an e�ort to set up a QualityAssurance (QA) team for CPAN modules.

�e objective of the group is to test as many of thedistributions on CPAN as possible, on as manyplatforms as possible.

�e ultimate goal is to improve the portability of thedistributions on CPAN, and provide good feedbackto the authors.

Volunteers

Volunteers testing recent CPAN uploads

On a variety of operating systems

On a variety of platforms

On a variety of Perl versions

�e subject

From: Andreas J. Konig (ANDK)Subject: FAIL JSON-YAJL-0.01 v5.8.7 GNU/LinuxDate: 2011-04-11T10:25:15Z

This distribution has been tested as part ofthe CPAN Testers project, supporting thePerl programming language. Seehttp://wiki.cpantesters.org/ for moreinformation or email questions tocpan-testers-discuss@perl.org

Grades

PASS distribution built and tested correctlyFAIL distribution failed to test correctlyUNKNOWN distribution failed to build, had no

test suite or outcome was inconclu-sive

NA distribution is not applicable to thisplatform and/or version of Perl

PreambleDear Leon Brocard,

This is a computer-generated report forJSON-YAJL-0.01 on perl 5.8.7, created byCPAN-Reporter-1.1902.

Thank you for uploading your work to CPAN.However, there was a problem testing yourdistribution.

If you think this report is invalid, pleaseconsult the CPAN Testers Wiki for suggestionson how to avoid getting FAIL reports formissing library or binary dependencies,unsupported operating systems, and so on:

Tester comments

This report is from an automated smoketesting program and was not reviewed bya human for accuracy.

Program output

Output from ’./Build test’:

t/pod.t ..... okCan’t load ’/home/sand/.cpan/build/JSON-YAJL-0.01-vYmtrJ/blib/arch/auto/JSON/YAJL/Generator/Generator.so’ for module JSON::YAJL::Generator: /home/sand/.cpan/build/JSON-YAJL-0.01-vYmtrJ/blib/arch/auto/JSON/YAJL/Generator/Generator.so: undefined symbol: newSVpvn\_utf8 at /usr/local/perl-5.8.7-threaded/lib/5.8.7/x86\_64-linux-thread-multi-ld/DynaLoader.pm line 230.at /home/sand/.cpan/build/JSON-YAJL-0.01-vYmtrJ/blib/lib/JSON/YAJL.pm line 4

Compilation failed in require at /home/sand/.cpan/build/JSON-YAJL-0.01-vYmtrJ/blib/lib/JSON/YAJL.pm line 4.BEGIN failed--compilation aborted at /home/sand/.cpan/build/JSON-YAJL-0.01-vYmtrJ/blib/lib/JSON/YAJL.pm line 4.Compilation failed in require at t/simple.t line 5.BEGIN failed--compilation aborted at t/simple.t line 5.t/simple.t ..Dubious, test returned 9 (wstat 2304, 0x900)No subtests run

Test Summary Report-------------------t/simple.t (Wstat: 2304 Tests: 0 Failed: 0)Non-zero exit status: 9Parse errors: No plan found in TAP output

Files=2, Tests=2, 0 wallclock secs ( 0.03 usr 0.00 sys + 0.07 cusr 0.02 csys = 0.12 CPU)Result: FAILFailed 1/2 test programs. 0/2 subtests failed.

Also

Prerequisites (and version numbers)

Environment variables (PATH, SHELL etc.)

Perl special variables and OS-speci�c diagnostics

Perl module toolchain versions installed

Summary of perl con�gurationSummary of my perl5 (revision 5 version 8 subversion 7) configuration:

Platform:osname=linux, osvers=2.6.26-1-amd64, archname=x86_64-linux-thread-multi-lduname=’linux k81 2.6.26-1-amd64 #1 smp mon dec 15 17:25:36 utc 2008 x86_64 gnulinux ’config_args=’-Dprefix=/usr/local/perl-5.8.7-threaded -Uversiononly -Dusedevel -Ui_db -Duselongdouble -des -Dusethreads’hint=recommended, useposix=true, d_sigaction=defineusethreads=define use5005threads=undef useithreads=define usemultiplicity=defineuseperlio=define d_sfio=undef uselargefiles=define usesocks=undefuse64bitint=define use64bitall=define uselongdouble=defineusemymalloc=n, bincompat5005=undef

Compiler:cc=’cc’, ccflags =’-D_REENTRANT -D_GNU_SOURCE -DTHREADS_HAVE_PIDS -fno-strict-aliasing -pipe -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64’,optimize=’-O2’,cppflags=’-D_REENTRANT -D_GNU_SOURCE -DTHREADS_HAVE_PIDS -fno-strict-aliasing -pipe -I/usr/local/include’ccversion=’’, gccversion=’4.3.2’, gccosandvers=’’intsize=4, longsize=8, ptrsize=8, doublesize=8, byteorder=12345678d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=16ivtype=’long’, ivsize=8, nvtype=’long double’, nvsize=16, Off_t=’off_t’, lseeksize=8alignbytes=16, prototype=define

Linker and Libraries:ld=’cc’, ldflags =’ -L/usr/local/lib’libpth=/usr/local/lib /lib /usr/liblibs=-lnsl -ldb -ldl -lm -lcrypt -lutil -lpthread -lcperllibs=-lnsl -ldl -lm -lcrypt -lutil -lpthread -lclibc=/lib/libc-2.7.so, so=so, useshrplib=false, libperl=libperl.agnulibc_version=’2.7’

Dynamic Linking:dlsrc=dl_dlopen.xs, dlext=so, d_dlsymun=undef, ccdlflags=’-Wl,-E’cccdlflags=’-fpic’, lddlflags=’-shared -L/usr/local/lib’

�e error

Can’t load‘/home/sand/.cpan/build/JSON-YAJL-0.01-vYmtrJ-/blib/arch/auto/JSON/YAJL/Generator/Generator.so’for module JSON::YAJL::Generator:/home/sand/.cpan/build/JSON-YAJL-0.01-vYmtrJ-/blib/arch/auto/JSON/YAJL/Generator/Generator.so:unde�ned symbol: newSVpvn_utf8 at/usr/local/perl-5.8.7-threaded/lib/5.8.7/x86_64-linux-thread-multi-ld/DynaLoader.pm line230.

perlapi says

Creates a new SV and copies a string into it. If utf8 istrue, calls "SvUTF8_on" on the new SV.

SV* newSVpvn_utf8(NULLOK constchar* s, STRLEN len, U32 utf8)

Introduced in 5.11, as pointed out by Andreas J. König.

Devel::PPPort - Perl/Pollution/Portability

Perl’s API has changed over time, gaining newfeatures, new functions, increasing its �exibility, andreducing the impact on the C namespaceenvironment (reduced pollution).�e header �lewritten by this module, typically ppport.h, attemptsto bring some of the newer Perl API features to olderversions of Perl, so that you can worry less aboutkeeping track of old releases, but users can still reapthe bene�t.

Release 0.02

Include ppport.h

0.02 Mon Apr 11 15:39:18 CEST 2011

Release 0.03

Unde�ned symbol "DPPP_my_newSVpvn_�ags"

Release 0.03 (2)

ppport.h is runnable#define NEED_newSVpvn_flags#define NEED_sv_2pv_flags

Release 0.03 (3)

do not include stdint.h as we do not need it

0.03 Tue Apr 12 09:56:35 CEST 2011

Release 0.04

got: 1.2299999999999999822 expected: 1.23

Stop using 1.23 for testing �oats

What Every Computer Scientist Should Know AboutFloating-Point Arithmetic—David Goldberg

Release 0.04 (2)

symbol isinf: referenced symbol not found#if defined (__SVR4) && defined (__sun)#ifndef isinf#include#define isinf(x) (!finite((x)) && (x)==(x))#endif#endif

Release 0.04 (3)

Error: ’const char *’ not in typemap in Generator.xs,line 18

add const char * to the typemap to support Perl 5.6

Release 0.04 (4)

throw exceptions upon YAJL error states

0.04 Wed Apr 13 17:09:21 CEST 2011

Release 0.05

Half the tests failed!

error: too few arguments to function ’Perl_croak’

use aTHX_ when Perl_croak-ing to work on threadedPerlsPerl_croak(aTHX_ "YAJL: Keys must be strings");

Release 0.05 (2)

expecting: Regexp ((?-xism:Invalid number)) found:YAJL: Keys must be strings

Only test inf and nan on Perl 5.8.8 and later

0.05�u Apr 14 10:26:20 CEST 2011

Release 0.06

�x minor documentation typo

don’t test inf and nan under MSWin32

add an interface to the parser

take advantage of typemaps and declarations tominimize XS code

0.06 Sat Apr 16 19:22:54 CEST 2011

Release 0.07link to YAJL website, as pointed out by OlivierMengué

add homepage, repository and bugtrackerdistribution metadata

add LICENSE �le

add a SAX builder example

add a tokenising example

add a tokenising, parsing with Marpa example

improved documentation, clearer tests

0.07 Mon Apr 18 20:09:36 CEST 2011

Release 0.08

update to YAJL 2.0.3

0.08 Mon Jun 27 14:34:14 BST 2011

Release 0.09

don’t test inf and nan under MirOS BSD

work around not �nding isinf under Solaris

work around missing sprintf_s under Windows

0.09 Wed Jul 6 14:06:49 BST 2011

Release 0.10

work aroud the fact that win32 mingw/gcc doesn’thave sprintf_s (for Strawberry Perl)

move dSP to top of callback_call to compile underMicroso� Visual Studio 10.0

0.10�u Aug 4 08:37:47 BST 2011

/

Microso� Visual C++ does not support long long:yajl\yajl_parse.h(77) : error C2632: ’long’

followed by ’long’ is illegalyajl\yajl_parser.h(74) : error C2632: ’long’

followed by ’long’ is illegal

. . .

How to become a CPAN Tester

CPAN::Reporter

http://wiki.cpantesters.org/

CPAN Testers Leaderboard

1st 4,573,342 Chris Williams (BINGOS)

2nd 1,133,385 Andreas J. König (ANDK)

3rd 758,466 Dan Collins (DCOLLINS)

. . .

24th 46,502 Léon Brocard (LBROCARD)

Summary

1,564 test reports

39 perls

120 platforms

All helping to make CPAN modules more portable

On next at 14:35

Liela zale: Perl 6 Lists, Arrays, and Hashes vivi�ed:lazy, in�nite, �at, slurpy, typed, bound, and LoL’d,Patrick Michaud

Auditorija 310: Encryption on the Web for everyone,Lars Dieckow

Auditorija 301: Terms of endearment — theElasticSearch query language explained, ClintonGormley

Auditorija 215: Perlude: a taste of haskell in perl, MarcChantreux