Down the Rabbit Hole with Typed Data

http://2015.midcamp.org/session-proposal/down-rabbit-hole-typed-data

Matthew Radcliffe mradcliffe

@mattkineme

Use Case: Xero Integration• Xero accounting platform with a nice Restful API

• Post invoices from Open Atrium cases. Invoice done. Click send. Back to coding.

• Post bank transactions from Commerce / Ubercart payment transactions. Auto-matching Bank Statements to Bank Transactions is a huge cost savings for a small company.

• PHP-Xero

• It’s abandoned - upstream fixes no longer an option.

• A fork on Github will never be canonical.

• xero declares OAuth :(

Escaping Procedural Reality

if ( (count($arguments) == 0) || ( is_string($arguments[0]) ) || ( is_numeric($arguments[0]) ) || ( $arguments[0] === false ) ) { $where = false; //it's a GET request if ( !in_array($name, $valid_get_methods) ) { return false; } $filterid = ( count($arguments) > 0 ) ? strip_tags(strval($arguments[0])) : false; if(isset($arguments[1])) $modified_after = ( count($arguments) > 1 ) ? str_replace( 'X','T', date( 'Y-m-dXH:i:s', strtotime($arguments[1])) ) : false; if(isset($arguments[2])) $where = ( count($arguments) > 2 ) ? $arguments[2] : false; if ( is_array($where) && (count($where) > 0) ) { $temp_where = ''; foreach ( $where as $wf => $wv ) { if ( is_bool($wv) ) { $wv = ( $wv ) ? "%3d%3dtrue" : "%3d%3dfalse"; } else if ( is_array($wv) ) { if ( is_bool($wv[1]) ) { $wv = ($wv[1]) ? rawurlencode($wv[0]) . "true" : rawurlencode($wv[0]) . "false" ; } else { $wv = rawurlencode($wv[0]) . "%22{$wv[1]}%22" ; } } else { $wv = "%3d%3d%22$wv%22"; } $temp_where .= "%26%26$wf$wv"; } $where = strip_tags(substr($temp_where, 6));

• XeroOAuth-PHP

• Xero went off in their own direction.

• Tried it, contributed to it, no thanks.

• Suffers from similar code standard/quality issues as PHP-Xero.

Escaping Procedural Reality

• “xero make”: model data as associative arrays.

• “xero query”: pass-through for PHP-Xero library.

• “form helper”: builds form array snippets.

• “xero reference”: field type/widget/formatter.

Escaping Procedural Reality

• Essentially:

• Make HTTP OAuth--signed Requests

• of Xero API Types

• structured as XML

Escaping Procedural Reality

Through the Drupal 8 Looking Glass

• Guzzle 3*

• Extendable, Catchable exceptions, OAuth, XML/JSON encoding, etc…

• XeroBundle provided XeroClient:

• Library code is small and maintainable.

• Inherits what it does from Guzzle Client.

* At the time I stared through the Looking Glass, Guzzle 3 was the version in core. Guzzle 5 is included in Drupal 8 core at the time of writing this session.

• Symfony 2

• Serializer* transform data from/to formats.

• Serialization module

• Extends Serializer to transform Drupal typed data from/to formats.

Through the Drupal 8 Looking Glass

* The Serializer Component. http://symfony.com/doc/current/components/serializer.html

• Typed Data

• A way of describing and managing complex data types.

Through the Drupal 8 Looking Glass

Figuring out Xero Types

• Why Typed Data?

• Need to describe Xero types into something Drupal and Symfony understand.

• Data types: primitive and complex

• Are these entities or not?

Xero Types• Accounts

• Invoices

• Payments

• Credit Notes

• Contacts

• Users

• Bank Transactions

• Amount

• Line Items

• and more…

Xero Types• Accounts

• Invoices

• Payments

• Credit Notes

• Contacts

• Users

• Bank Transactions

• Amount

• Line Items

• and more…

Xero Types• Accounts

• Invoices

• Payments

• Credit Notes

• Contacts

• Users

• Bank Transactions

• Amount

• Line Items

• and more…

Xero Types• Accounts

• Invoices

• Payments

• Credit Notes

• Contacts

• Users

• Bank Transactions

• Amount

• Line Items

• and more…

Oh, Dear! I’ll be Late for Release!• Typed Data, the white rabbit; a means to let

me use Serializer and Guzzle.

• And so down I went after it*.

• In the past couple of years, Typed Data API has changed and so I chase after it.

• One of the remaining Drupal 8 APIs with little documentation.

* Alice in Wonderland. Lewis Carroll. 1865. Page 4. PDFFreeBooks.org. 2015.

Typed Data API Docs

https://www.drupal.org/node/1794140

My Personal Challenge: Request edit permissions for this documentation page Sunday. Work on PostgreSQL and

SQLite instead.

How to read a Map• Typed Data API provides most of these, yay!

• There’s one complex data type that is not an entity.

• Map: the new associative array.

• Next challenge: how to create this elusive data type class.

<?php namespace Drupal\xero\Plugin\DataType; !/** * Xero Payment type. * * @DataType( * id = "xero_payment", * label = @Translation("Xero Payment"), * definition_class = "\Drupal\xero\TypedData\Definition\PaymentDefinition" * ) */ class Payment extends XeroTypeBase { protected function getPropertyDefinitions() { return $this->definition->getPropertyDefinitions(); } }

plugin typeplugin id plugin label

Data Definitions• Complex data type needs definition classes to

describe its properties (or child data types).

• Symfony 2 Constraints

• Label

• Property data type

• When I began, definitions were associative arrays.

class PaymentDefinition extends ComplexDataDefinitionBase { public function getPropertyDefinitions() { if (!isset($this->propertyDefinitions)) { $info = &$this->propertyDefinitions; ! $info['Invoice'] = DataDefinition::create(‘xero_invoice’) ->setRequired(TRUE) ->setLabel('Invoice'); ! $info['Account'] = DataDefinition::create(‘xero_account’) ->setRequired(TRUE) ->setLabel('Account'); ! $info['Date'] = DataDefinition::create(‘string’) ->setRequired(TRUE) ->setLabel('Date'); ! $info['Amount'] = DataDefinition::create(‘float’) ->setRequired(TRUE) ->setLabel('Amount'); } return $this->propertyDefinitions; } }

List Data Definition

• Data definitions can be lists of data types i.e. multiple values or an indexed array equivalent.

• This is important for Xero because the API returns a list of items in addition to other odd types like Tracking Categories, Line Items, and Addresses.

MAD Tip

• New in 2015 that could save time: ClassiPHPy

• https://github.com/EclipseGc/Classiphpy

• Generate your definition classes via associative arrays with Pharborist.

• New in 2015 that could save time: ClassiPHPy

• https://github.com/EclipseGc/Classiphpy

• Generate your definition classes via associative arrays with Pharborist.

drupal module upgrader uses Pharborist to transform Drupal 7 PHP into Drupal 8 PHP (if possible).

Extra Bonus MAD Tip

MAD Tip

Denormalize Me• Normalizer classes transform data into Typed

Data and vice versa.

• Normalization is obscured from use because it runs as part of serialization.

• “$context” was a difficult parameter to understand.

• TypedDataManager use is recursive and confusing.

• Normalization classes must be defined as services in the service container.

• Must have a tag so that Symfony 2 knows to treat it differently than other services.

Denormalize Me

services: xero.normalizer: class: Drupal\xero\Normalizer\XeroNormalizer arguments: ['@typed_data_manager'] tags: - { name: normalizer }

xero.services.yml

class XeroNormalizer extends ComplexDataNormalizer implements DenormalizerInterface { ! protected $supportedInterfaceOrClass = 'Drupal\xero\TypedData\XeroTypeInterface'; ! public function __construct(TypedDataManager $typed_data_manager) { $this->typedDataManager = $typed_data_manager; } ! public function denormalize($data, $class, $format = NULL, array $context = array()) { if (!isset($context['plugin_id']) || empty($context['plugin_id'])) { throw new UnexpectedValueException(‘...’); } ! $name = $class::$xero_name; $plural_name = $class::$plural_name; ! // Wrap the data in an array if there is a singular object returned. if (count(array_filter(array_keys($data[$plural_name][$name]), 'is_string'))) { $data[$plural_name][$name] = array($data[$plural_name][$name]); } ! $list_definition = $this->typedDataManager->createListDataDefinition($context['plugin_id']); $items = $this->typedDataManager->create($list_definition, $data[$plural_name][$name]); ! return $items; }

Building the Mock Turtle

• Unit tests are awesome and fast, but Typed Data Manager is big and scary. :(

• Typed Data calls itself so mocking is not straight-forward. :(

• Need to mock the service container. :(

• What is necessary to have decent coverage and reduce complexity?

PHPUnit

• I like seeing pretty coverage and complexity graphs.

PHPUnit

• I like seeing pretty coverage and complexity graphs.

PHPUnit

• DataDefinitionInterface::getPropertyDefinitions seems to be easy to generate coverage for because it’s just returning a an array of more data definitions.

• Am I getting anything out of this test, or is it just cruft to make my coverage green?

class PaymentDefinition extends ComplexDataDefinitionBase { public function getPropertyDefinitions() { if (!isset($this->propertyDefinitions)) { $info = &$this->propertyDefinitions; ! $info['Invoice'] = DataDefinition::create(‘xero_invoice’) ->setRequired(TRUE) ->setLabel('Invoice'); ! $info['Account'] = DataDefinition::create(‘xero_account’) ->setRequired(TRUE) ->setLabel('Account'); ! $info['Date'] = DataDefinition::create(‘string’) ->setRequired(TRUE) ->setLabel('Date'); ! $info['Amount'] = DataDefinition::create(‘float’) ->setRequired(TRUE) ->setLabel('Amount'); } return $this->propertyDefinitions; } }

Typed Data Mocking• The create static methods for Typed Data will

call its manager:

• Mocks cannot just return a single known value for complex data types

• Mocks must carefully re-construct the order of how create will be called on all definitions in a complex data definition.

public function createInstance($data_type, array $configuration = array()) { $data_definition = $configuration['data_definition']; $type_definition = $this->getDefinition($data_type); ! if (!isset($type_definition)) { throw new \InvalidArgumentException(format_string('Invalid data type %plugin_id has been given.', array('%plugin_id' => $data_type))); } ! // Allow per-data definition overrides of the used classes, // i.e. take over classes specified in the type definition. $class = $data_definition->getClass(); if (!isset($class)) { throw new PluginException(sprintf('The plugin (%s) did not specify an instance class.', $data_type)); } return $class::createInstance($data_definition, $configuration['name'], $configuration['parent']); }

public function setUp() { // Typed Data Manager setup. $this->typedDataManager = $this->getMockBuilder('\Drupal\Core\TypedData\TypedDataManager') ->disableOriginalConstructor() ->getMock(); ! $this->typedDataManager->expects($this->any()) ->method('getDefinition') ->with(static::XERO_TYPE, TRUE) ->will($this->returnValue(['id' => static::XERO_TYPE, 'definition class' => static::XERO_DEFINITION_CLASS])); $this->typedDataManager->expects($this->any()) ->method('getDefaultConstraints') ->willReturn([]); ! // Validation constraint manager setup. $validation_constraint_manager = $this->getMockBuilder('\Drupal\Core\Validation\ConstraintManager') ->disableOriginalConstructor() ->getMock(); $validation_constraint_manager->expects($this->any()) ->method('create') ->willReturn([]); $this->typedDataManager->expects($this->any()) ->method('getValidationConstraintManager') ->willReturn($validation_constraint_manager); !

public function testGetPhoneNumber() { $string_def = DataDefinition::create('string'); $country = new String($string_def); $area = new String($string_def); $number = new String($string_def); ! $this->typedDataManager->expects($this->any()) ->method('getPropertyInstance') ->with($this->phone, $this->callback(function($subject) { return in_array($subject, array( ‘PhoneCountryCode', ‘PhoneAreaCode', ‘PhoneNumber') ); })) ->will($this->onConsecutiveCalls($country, $area, $number)); ! $this->phone->set('PhoneCountryCode', '01'); $this->phone->set('PhoneAreaCode', '805'); $this->phone->set('PhoneNumber', ‘255-8542'); ! $this->assertEquals('01-805-255-8542', $this->phone->getPhone()); } }

\Core\Validation\ConstraintManager') ->disableOriginalConstructor() ->getMock(); $validation_constraint_manager->expects($this->any()) ->method('create') ->willReturn([]); $this->typedDataManager->expects($this->any()) ->method('getValidationConstraintManager') ->willReturn($validation_constraint_manager); ! // Mock the container. $container = new ContainerBuilder(); $container ->set('typed_data_manager', $this->typedDataManager); \Drupal::setContainer($container); ! // Create data definition $definition_class = static::XERO_DEFINITION_CLASS; $this->dataDefinition = $definition_class::create(static::XERO_TYPE); } }

A poem about mocking the container

What I learn not to do

I resign myself to that fate.

Besides, core does it too.

So please do not hate

this container so small

is not actually in my app at all.

I do not load,

nor invoke the container directly

to get variables in my code.

I use dependency injection, actually.

So forgive the misuse I leverage.

All I want is pretty coverage.

Alice’s Evidence

• Typed Data helps to use Drupal 8 effectively:

• Dependency Injection

• Better code

• Code re-use on and “off the island”.

DefaultSettingsForm

public function buildForm(array $form, FormStateInterface $form_state) { ! $config = self::config('xero.settings'); ! $account_options = array(); ! try { $context = array('plugin_id' => 'xero_account'); $accounts = $this->query ->setType($context['plugin_id']) ->setMethod('get') ->setFormat('xml') ->execute(); ! foreach ($accounts as $account) { // Bank accounts do not have a code, exclude them. if ($account->get('Code')->getValue()) { $account_options[$account->get('Code')->getValue()] = $account->get('Name')->getValue(); } } } catch (RequestException $e) { } catch (\Exception $e) { } ! $form['defaults']['account'] = array( '#type' => 'select', '#title' => t('Default Account'), '#description' => t('Choose a default account.'), '#options' => $account_options, '#default_value' => $config->get('defaults.account'), );

XeroQuery

public function __construct(XeroClient $client, Serializer $serializer, TypedDataManager $typed_data, LoggerChannelFactoryInterface $logger_factory) { $this->client = $client; $this->serializer = $serializer; $this->typed_data = $typed_data; $this->logger = $logger_factory->get('xero'); }

public function execute() { try { $this->validate(); $this->explodeConditions(); ! $data_class = $this->type_definition['class']; $endpoint = $data_class::$plural_name; $context = array('plugin_id' => $this->type); ! if ($this->data !== NULL) { $this->options['body'] = $this->serializer->serialize($this->data, $this->format, $context); } ! $request = $this->client->{$this->method}($endpoint, $this->options['headers']); ! // Add query parameters via getQuery() instead :-( if (!empty($this->options['query'])) { $q = $request->getQuery(); foreach ($this->options['query'] as $key => $value) { $q->set($key, $value); } } ! $response = $request->send(); $data = $this->serializer->deserialize($response->getBody(TRUE), $data_class, $this->format, $context); ! return $data; }

Typed Data Summary

• Strong typing useful for RESTful APIs.

• It is complex, but powerful.

• And difficult to test.

• But what else..?

What else..?• The floor is open to ask, discuss, or share:

• Typed Data uses (e.g. entity adapters)

• Questions

• Other Drupal 8 APIs

• http://2015.midcamp.org/session-proposal/down-rabbit-hole-typed-data