rt4 - the whole sordid story

jesse@bestpractical.comhttp://bestpractical.com

RT 4(Request Tracker 4.0)

Hi, I'm Jesse Vincent

From Boston, MA in the US

Perl Hacker

Current Perl 5 “pumpking”

Android Hacker (K-9 Mail)

Kindle Hacker (Savory)

Former Perl 6 project manager

Finding me

@obra

jesse@bestpractical.com

facebook.com/jesse.vincent

Hi I'm Jesse

Original Author of RT

Partner at Best Practical

RT project lead

(That just I don't get to code much anymore)

Best Practical

We make RT

We sell support

We sell training

We sell consulting

We sell custom development

Our offices

Boston, MA

Moscow, Russia

Hangzhou, China

Pinglin, Taiwan

(We're 11 people)

RT: Request Tracker

General purpose ticketing system

GNU GPLv2

Continuousdevelopmentsince 1996

O(100) downloadsevery day

What is a ticketing system?

It keeps track of whatneeds to get done

It keeps track of whatgot done

...along with lots of metadata

...and business logic

...and access control

...It's just a TODO list

On some very serious drugs

not

Important properties ofticketing systems

● Everything has a unique ID● Everything has a timestamp● History can't be edited or erased

What do you use a ticketing system for?

Network operations

Bug tracking

Call center

Helpdesk

Customer service

Work orders

Accounts Payable

Accounts Receivable

Vacation rentals

Youth counselling

Workflow

What we use RT for

Bug Tracking

Bronze/Silver/Gold support

Customer Development

Resumes

Inbound Sales Inquiries

Sales Leads

Accounts Payable

Accounts Receivable

Who uses RT?

Who else uses RT?

http://requesttracker.wikia.com/wiki/RTUsers

RT Scales

It scales down

(for testing or development)

Run RT on your laptop

SQLite

Standalone web server

It scales up

Largest RT I know about

40,000-70,000 tickets

...every day

(Nearly 1 ticket/second)

Multiple front-end app servers

Big database server with hot standby

Designed to be hookableand pluggable

Plugins

● rt-action-linearescalate

● rt-action-notifygroup

● rt-ajaxyreplypage

● rt-authen-bitcard

● rt-authen-openid2

● rt-bugtracker

● rt-bugtracker-public

● rt-condition-complex

● rt-crypt-smime

● rt-extension-activityreports

● rt-extension-activityreports-billing

● rt-extension-addadminccsonqueuechange

● rt-extension-attributewalker

● rt-extension-captcha

● rt-extension-cloneticket-withdata

● rt-extension-commandbyemail

● rt-extension-commandbymail

● rt-extension-commentoncreate

● rt-extension-customfield-hideemptyvalues

● rt-extension-datediscordian

● rt-extension-extractcustomfieldvalues

● rt-extension-formtools

● rt-extension-jsgantt

● rt-extension-ldapimport

●

● rt-extension-log-memoryusage

● rt-extension-menubarsearches

● rt-extension-mergeusers

● rt-extension-mergeusershistory

● rt-extension-nagios

● rt-extension-notificationmatrix

● rt-extension-priorityasstring

● rt-extension-quickcalls

● rt-extension-quickdelete

● rt-extension-quickupdate

● rt-extension-reportspam

● rt-extension-rt_cpan_org

● rt-extension-spawnlinkedticketinqueue

● rt-extension-utils

● rtfm

● rtfm-extension-articletemplate

● rtir

● rtx-calendar

● rtx-emailcompletion

● rtx-ticketlist-transactions

● rtx-workflowbuilder

(and a bunch more created by RT users)

RT 4.0

Now available?

Not quite

Christmas 2010:

4.0.0RC1

March 24, 2011:

4.0.0 RC7

Release next week?

hcchien has been askingme to do a talk on RT4

since at least 2006.

I've been promising “next year” for 5 years.

We started RT4 inSeptember 2007

I named it 3.999-DANGEROUS

In literature, they call that foreshadowing

I do public RT trainingsa few times a year

I talk about RT's history

These are the slides I use

A Brief History of RT

RT 0.9 (1996)

● Designed for use at a single company

● 2 sysadmins● 30 users

RT 1.0 (1999)

● Same as RT 0.9

+ a bit more courage● Used at hundeds of companies● Dozens of CSRs● Thouands of requests per day● Intense guilt

RT 2.0 (2001)● Total rewrite● Just after Jesse escaped Microsoft● DBIx::SearchBuilder● Abstraction● Whole new UI● No more frames● “Keywords”

RT 3.0 (2003)

● Overhauled web interface● Extension mechanisms● Internationalization● Custom fields● Cleaner internals● Tests

RT 3.2 (2004)

● New search UI● Spreadsheet / RSS output● Outgoing mail preview and logging● UI improvements● No major structural changes● More tests

RT 3.4 (2005)

● Reimplemented Custom Fields● Custom fields on users, groups

transactions● Generalized Transaction system● Faster, Faster, Faster● Prettier● Even more tests

RT 3.6 (2006)

● All-CSS layout and styling● Customizable homepage● Built in charts and reports● Ticket "reminders"● Comprehensive test coverage● Cleaner code

RT 3.8 (2008)

● More user preferences● Timezones● Theme● Ticket history order● New configuration system● Even more tests

RT 3.8 (continued)

● “Favorite” tickets● Ticket relationship graphs● Branded queues● iCal feeds● PGP support

RT 4.0 (2008?)

Never trust a vendor who makes promises about unreleased products

That's really what it said!

It was sort of a joke

...little did I know

All Taiwanese know that4 is very unlucky

You're supposedto just skip 4

Nobody warned me

...until last night!

In my culture, 6 is the unlucky number.

Along came 2006

We started thinkingabout building RT 4.0

RT is big

http://www.flickr.com/photos/swiv/4426214075/

RT is big

http://www.flickr.com/photos/daymin/4715213393/

RT is complex

http://www.flickr.com/photos/18909153@N08/5241036226/

RT is complex

RT is old

http://www.flickr.com/photos/jpott/5326081706/

RT is old

http://www.flickr.com/photos/paulgissane/163290720

What would we change?

Modernize the API

Remove insane features

Use a framework!

Jifty

AJAX

Modern API

Lots of testing affordances

Plack

Automatic Database Schema Management

UI Helpers

So, we started refactoring

Not a from-scratch rewrite

..but pretty close

No deadline

"It'll be ready when it's ready"

No fixed deliverable

"We want it to be good"

So, we went to work.

What went wrong?

We moved files around

Git made that sort of ok

We decided to modernizeour coding style

We started renamingclasses and methods

RT's API was oldand InterCapped

The modern perl worldis prettier_looking

We built refactoring tools

We ported thefull test suite

RT 3.6/3.7 were still inactive development

We were fixing lots of bugs in 3.6

It was a constant battleto merge forward

3 years in, RT 3.999 was largely "done"

60% of the code of RT 3.8

Almost the samefeature set as RT 3.8.0

Almost the sametest suite as 3.8.0

Much cleaner

Ran on Jifty

Ran on Plack

New "Ticket Lifecycles" system

Much of the UI portedto Template::Declare

It was a lot better

It killed off lots of badold API decisions

...but mostly better forRT's developers

2627 commits

3 years of development

1484 files changed, 174558 insertions(+), 319761 deletions(-)

3.999 was different than 3.8 in some important ways

The API was recognizably the same

3.999 was better than 3.8 insome important ways

The API was 100% incompatible

There were lots and lots of reasons to make the change

There are lots and lots of RT extensions out there.

We've done at least 75

There are more on CPAN

Just about every RT instance has some local customizations

RT has been downloaded about 100 times every day

For the past 5+ years

How many of you have ever customized or extended

some software?

Guess what happens whenyou change an API?

Ever had an API changebreak your customization?

RT has many users

They rely on many, manyRT extensions

● rt-action-linearescalate

● rt-action-notifygroup

● rt-ajaxyreplypage

● rt-authen-bitcard

● rt-authen-openid2

● rt-bugtracker

● rt-bugtracker-public

● rt-condition-complex

● rt-crypt-smime

● rt-extension-activityreports

● rt-extension-activityreports-billing

● rt-extension-addadminccsonqueuechange

● rt-extension-attributewalker

● rt-extension-captcha

● rt-extension-cloneticket-withdata

● rt-extension-commandbyemail

● rt-extension-commandbymail

● rt-extension-commentoncreate

● rt-extension-customfield-hideemptyvalues

● rt-extension-datediscordian

● rt-extension-extractcustomfieldvalues

● rt-extension-formtools

● rt-extension-jsgantt

● rt-extension-ldapimport

●

● rt-extension-log-memoryusage

● rt-extension-menubarsearches

● rt-extension-mergeusers

● rt-extension-mergeusershistory

● rt-extension-nagios

● rt-extension-notificationmatrix

● rt-extension-priorityasstring

● rt-extension-quickcalls

● rt-extension-quickdelete

● rt-extension-quickupdate

● rt-extension-reportspam

● rt-extension-rt_cpan_org

● rt-extension-spawnlinkedticketinqueue

● rt-extension-utils

● rtfm

● rtfm-extension-articletemplate

● rtir

● rtx-calendar

● rtx-emailcompletion

● rtx-ticketlist-transactions

● rtx-workflowbuilder

We broke all of them

(all the extensions)

(all the users)

RT 3.999 had 40% lesscode than RT 3.8

RT 3.999 lookedjust like RT 3.8

RT 3.999 had almost thesame features as 3.8

...with one really big change

RT 3.x has a systemcalled “Scrips”

Scrips let you buildcustom business logic

They're sort of like“if...then...” statements

They can fire after any update

RT's approvals systemuses Scrips

RT's email-sending rulesuse Scrips

They're really powerful

...but not omnipotent

Scrips are unchangedsince RT 2.0

We decided to replaceScrips in RT 3.999

I wanted a simplemacro language

clkao built the backend

“I'm not building a stupid macro language. If we're doing this, it should support eval and apply”

We created lorzy

It was a lispy language

...with named, typed parameters

We ripped out Scrips and dropped in lorzy

Greenspun's Tenth Rule

Any sufficiently complicated C or Fortran program contains an ad hoc, informally-specified, bug-ridden, slow implementation of half of Common Lisp.

...at least we did it on purpose?

So, have I sold RT 3.999 to you?

I don't like hurting users.

I don't like hurting customers.

Last summer, we threw away3 years of work.

[sad panda]

What'd we learn?

Second system syndromehurts a lot

. o O { My problem is that I actually have users

I don't want to alienate }

A working test suite is not a magic bullet

Incremental updates vs

gut renovation

Got a working system?

Make the smallest changethat could possibly work

Build for your users

When we do client work, they say that they want high-quality software last week for almost

free.

"Good, fast and cheap, pick any two"

Turns out that youmust pick two.

Last summer, we startedRT4 over again

What'd we do differentthis time?

We were workingfor a customer

The customer wantedRT 3.8...

...with a lot of extensions.

They wanted to pay us to integrate those extensions.

We had a deadline

We had a fixed set of deliverables

We had a mostly fixed set of deliverables

We got to work

The client wanted frequentbeta releases

The new RT 3.9 needed tobe deployable at any time

Lots of topic branches

We started pulling in work we'd already done as core features

RTFM became Articles

Stock answers

Really useful

Lots of folks don't install it

So, now you don't get a choice

Lifecycles

Originally built for 3.999 & backported to 3.8

Mobile Interface

Date custom fields

IP address custom fields

Full-text search for MySQL, Postgres and Oracle

Gmail-style foldingof ticket history

Some new features, we built from scratch

New auditing anddebugging tools

A brand new visual theme

A brand new theme editor

A new permissions editor UI

Dropdown and radio-button custom fields

Autocompleters for usersand email addresses

Ported to Plack

Some stuff, you'd never notice

Overhauled how wedo boilerplate code

(Fewer files change.They change less)

Cut down our use of autoconf

Made the test suite much faster

Eliminated "noise"from the test suite

Successfully deliveredto customer

Since then, we've been stabilizing, fixing bugs and

improving docs

RC 1 came out just after christmas

RC 7 came out last week

We're very, very close.

Try out RT4

http://bestpractical.com/rt/download.html

http://bestpractical.com/rt/git.html

Release Candidate means we think it's ready for production

If it breaks, email rt-bugs@bestpractical.com

Thank you!

Questions?