轉移百萬筆資料 +

使用者無縫接合

2013年7月6日 Drupal Camp Taipei 2013

About me

● Joe Tsui，香港人● 5年 Drupal 開發經驗● Drupal Camp Taipei 2011 講者● Imagefield_crop maintainer● 成立了 ，接洽 Drupal 專案● 為香港最大的英文報紙(scmp) 和信報負責

Drupal 開發● 香港最活躍 Drupal blogger 之一● Hong Kong Drupal User group 成員

WarningProgramming ahead

轉移百萬筆資料

● 1,200,000+ entity in total● Data from 2008● 12G images● do not exists user table!

轉移百萬筆資料

● Source● Destination● Key Mapping● Field Mapping● Migration flow● PrepareRow● Prepare● Complete, CompleteRollback● Test run● Rollback● Tricks

轉移百萬筆資料

Legacy system Drupal

Migrate module (v2.5)

轉移百萬筆資料

Legacy system● Article● Comment● User●● Banned user● Image, File

Drupal● Node● Comment● User● User Picture● Banned user● Image, File

Source Destination

轉移百萬筆資料 - Source

● MigrateSourceSQL● MigrateSourceCSV● MigrateSourceList (XML)● MigrateSourceMSSQL● MigrateSourceOracle

轉移百萬筆資料 - Destination

● MigrateDestinationNode● MigrateDestinationComment● MigrateDestinationUser● MigrateDestinationRoles● MigrateDestinationTable

轉移百萬筆資料

● Source● Destination● Key Mapping● Field Mapping● Migration flow● PrepareRow● Prepare● Complete, CompleteRollback● Test run● Rollback● Tricks

Mappings

Key Mappings

Legacy: Article id: 15368

Node:Nid : 15368

Unique identifier as Key

● 一對一的關係● 唯一的 source entity● 對應一個 destination entity

Key Mappings

Example: {node}

Field Mappings/admin/content/migrate/ForumArticle

Migration flow

1. Migrate::__construct()2. Migrate::prepareRow()3. Migrate::prepare()4. Migrate - DB writes5. Migrate::complete()

轉移百萬筆資料

● Source● Destination● Key Mapping● Field Mapping● Migration flow● PrepareRow● Prepare● Complete, CompleteRollback● Test run● Rollback● Tricks

Extra customization

prepareRow($row)

Entity(destination) Row(source)

prepare($entity, $row)

● Connect to other migration (user, user picture)

complete($entity, $row)

Missing parts in other tables

轉移百萬筆資料

● Source● Destination● Key Mapping● Field Mapping● Migration flow● PrepareRow● Prepare● Complete, CompleteRollback● Test run● Rollback● Tricks

Run time

Test run

● drush mi ForumArticle --limit="10 items"● Tip: Add order by● Tip: Add condition to MigrateSourceSQL to

pin point specific data

Conditions

Rollback

● Rollback 比 import 慢得多● Rollback on 1M+ rows● 在 import 之前用 sql-dump 建立原始狀態

snapshot● mysql < backup.sql to mass “rollback”

tips 1

● Test 策略: 找出 data 的所有組合○ post with 1 image○ post with 2 images○ deleted post○ "spammed" post○ "comment closed" post

● 不移動 image ○ preserve_files')->defaultValue(TRUE);○ file_replace')->defaultValue(MigrateFile::

FILE_EXISTS_REUSE);● 預先將 files 放到 files/legacy

tips 2

● user picture -> fid○ 用戶頭像是 {user} 內的一個 fid○ 先建立 files, 再 import User 之後在 prepare() 插入 fid

● field_image[] for multiple values

$this->addFieldMapping('field_image', 'field_image_value');//prepareRow($row)

$row->field_image_value[] = substr(trim($image_path), 1);

使用者無縫接合

例子● auth.joe.com● forum.joe.com● news.joe.com

使用者無縫接合

● Active action○ Login API○ Provider side○ Active logout○ Provider side

● Passive action○ cookie API○ passive logout○ Drupal side

Active Action - login APIforum.joe.com/user auth.joe.com/auth/login

Auth?

New user?

Register

set cookie

user_login_submit()

submit

No

Yes

Yes

No

Active Action - login APIforum.joe.com/user auth.joe.com/auth/login

Auth?

New user?

Register

set cookie

user_login_submit()

submit

No

Yes

Yes

No

Active Action - login APIforum.joe.com/user auth.joe.com/auth/login

submit

Active Action - login APIforum.joe.com/user auth.joe.com/auth/login

Auth?

New user?

Register

set cookie

user_login_submit()

submit

No

Yes

Yes

No

Active Action - login APINew user?

Register

set cookie

Yes

No

Register

Cookie

● auth.joe.com● forum.joe.com● news.joe.com

● Set cookie

● $cookie_domain = '.joe.com'

auth.joe.com (Provider)Active Action - login API

forum.joe.com/user auth.joe.com/auth/login

Auth?

New user?

Register

set cookie

user_login_submit()

submit

No

Yes

Yes

No

Save Session

Cookie value agreement

news.joe.com

Active action - logout

● clear cookie● Other sites should check cookie on page

loads

hook_init()

Passive Action - cookie APIforum.joe.com/node/123 auth.joe.com/auth/authenticate

Auth?

New user?

Register

set cookie

user_login_submit()

found a cookie

No

Yes

Yes

No

Passive Action - cookie APIforum.joe.com/node/123 auth.joe.com/auth/authenticate

Auth?

New user?

Register

set cookie

user_login_submit()

found a cookie

No

Yes

Yes

No

Passive Action - cookie APIforum.joe.com/node/123 auth.joe.com/auth/authenticate

found a cookie

Passive action - logout

● Drupal logged in● on page loads, cookie value not found● logout Drupal as well

使用者無縫接合 tips

● Browser private mode● Chrome develop resources tab to check

cookie● login tester form

現場示範－網站改造馬拉松

● Ubuntu 正體中文站

● http://www.ubuntu-tw.org/