Adventures incivic hacking

FixMyStreet & Open311hakim@mysociety.org

https://www.flickr.com/photos/theodevil/5457434597/

"easily launch a website that helps people to report street problems like potholes and broken street lights."

We make websites and tools that empower citizens

For communities.

For opening democracy.

For getting things changed.

FixMyStreet <-> Local Government

• User submits problem on web/app

• We work out who to contact, based on location/type of problem

• We send an email

FixMyStreet <-> Local Government

• User submits problem on web/app

• We work out who to contact, based on location/type of problem

• We send an email

• Council employee lovingly copies email into their actual system

https://www.flickr.com/photos/ltdemartinet/2811744365/

Integrations

• User submits problem on web/app

• We work out who to contact, based on location/type of problem

• We run custom code which calls council API to recreate the FMS problem on a council’s system

Integrations

• User submits problem on web/app

• We work out who to contact, based on location/type of problem

• We run custom code which calls council API to recreate the FMS problem on a council’s system• Exor

• Mayrise

• SAP

• MS Dynamics CRM (via SOAP, eeek. Later versions REST)

• etc….

Open311

…is a form of technology that provides open channels of communication for issues that concern public space and public services. Primarily, Open311 refers to a standardized protocol for location-based collaborative issue-tracking.—http://www.open311.org/learn/

http://raulrene.wordpress.com/renes-pixie-pixelarium/

Simples!

• FixMyStreet has builtin Open311 client

• Council backend is Open311 server

Simples!

• FixMyStreet has builtin Open311 client

• Council backend is not Open311 server

Here comes the architecture

Shim

Backend

CGI shim script

Simple

Few dependencies Easily run in hostile

environment

Subset of Open311

Non-standard URLS Won’t actually support

other Open311 clients

Hardcoded assumptions Hard to extend

Hard to test

Open311::Endpoint

• Plack

• Web::Simple

• DateTime::Parser::

• Data::Rx

• Moo, Types::Standard, MooX::HandlesVia

• DBD::Oracle

• XML

Plack::Test

my $endpoint = Test::Endpoint->new;

my $res = $endpoint->run_test_request(

GET => '/services.xml‘

);

End to end tests?

Shim

Backend

Stub out backend

Test::Shim

(Stubbed)

LWP::Protocol::PSGI

Test::Shim

(Stubbed)

my $endpoint = Test::Endpoint->new;

LWP::Protocol::PSGI->register($endpoint,host => 'open311.example.com‘

);

https://www.flickr.com/photos/cunaldo/468824934/

Web::Simplesub dispatch_request {

my $self = shift;

sub (GET + /services + ?*) {

my ($self, $args) = @_;

$self->call_api(GET_Service_List => $args);

},

sub (POST + /requests + %*) {

my ($self, $args) = @_;

$self->call_api(POST_Service_Request => $args);

},

https://www.flickr.com/photos/alanenglish/2780298470/

Types::Standard, MooX::HandlesVia

has latlong => (is => 'ro',isa => Tuple[ Num, Num ],default => sub { [0,0] },handles_via => 'Array',handles => {

lat => [ get => 0 ],long => [ get => 1 ],

});

Types::Standard, MooX::HandlesVia

has latlong => (is => 'ro',isa => Tuple[ Num, Num ],default => sub { [0,0] },handles_via => 'Array',handles => {

lat => [ get => 0 ],long => [ get => 1 ],

});

Types::Standard, MooX::HandlesVia

has latlong => (is => 'ro',isa => Tuple[ Num, Num ],default => sub { [0,0] },handles_via => 'Array',handles => {

lat => [ get => 0 ],long => [ get => 1 ],

});

Types::Standard, MooX::HandlesVia

has latlong => (is => 'ro',isa => Tuple[ Num, Num ],default => sub { [0,0] },handles_via => 'Array',handles => {

lat => [ get => 0 ],long => [ get => 1 ],

});

https://www.flickr.com/photos/dopey/9591636030/

Data::Rx

$schema->learn_type('tag:GeoReport_v2:rx/service', {

type => '//rec',

required => {

service_name => '//str',

type => '/open311/post_type',

metadata => '/open311/bool',

description => '//str',

},

optional => {

keywords => '//str',

group => '//str',

}});

Data::Rx

$schema->learn_type( 'tag:wiki.open311.org,GeoReport_v2:rx/datetime',

{

type => '/open311/regex',

pattern => qr{

^

\d{4} - \d{2} - \d{2} # yyyy-mm-dd

T

\d{2} : \d{2} : \d{2} # hh:mm:ss

(?:

Z # "Zulu" time, e.g. UTC

| [+-] \d{2} : \d{2} # +/- hh:mm offset

)

$

}ax, # use ascii semantics so /d means [0-9]

message => "found value isn't a datetime",

});

Inputs and Outputs

sub GET_Service_Definition_input_schema {

return { type => '//seq',

contents => [

$self->get_identifier_type('service_code'),

$self->get_jurisdiction_id_validation, ] } }

sub GET_Service_Definition_output_schema {

return { type => '//rec',

required => {

service_definition => {

type => '/open311/service_definition',

} } } }

XML::Simple

• Yeah, I know…

• Wrapped in Open311::Endpoint::Spark• interop with JSON schema

https://www.flickr.com/photos/rossap/7619777396/