easy contributable internationalization process with sphinx @ pyconmy2015

Easy contributable internationalization process with Sphinx

Takayuki Shimizukawa

Sphinx co-maintainer

Sphinx-users.jp

1

こんにちはKon-Nichi-Wa

2

Who am I

@shimizukawa

1. Sphinx co-maintainer

2. Sphinx-users.jp

3. PyCon JP committee

BePROUD co, ltd.3

http://pycon.jp/

4

Agenda

1. Sphinx introduction & Setup for i18n

2. A easy way and Non easy ways to translate

3. Sphinx i18n feature

4. Automated translation process with several services

5. Tips, tricks, traps 5

Question

6

Question1

How many people do you use Open Source

Software?

7

Question2

How many people may have contributed to the

OSS?

8

Question3

Does the translation into familiar language

contribute to other developers?

9

10

Translator

Product Author

Translated Docs

Readers

English Docs

Translators

Tools & Services

Tools1. sphinx - documentation generator2. docutils - base of sphinx3. sphinx-intl - support utility for sphinx i18n4. transifex-client - file transfer tool for

Transifex

Services5. Transifex - translation support service6. Drone.io - continuous integration service

11

1. Sphinx introduction& Setup for i18n

12

What is Sphinx?

Sphinx is a documentation generator

Sphinx generates doc as several output formats from the reST text

markup

13


Sphinx

reSTreSTreStructuredText

(reST)reST

Parser

HTML Builder

ePub Builder

LaTeX Builder texlive

HTMLtheme

Favorite Editor

The history of Sphinx (short ver.)

14


The father of Sphinx

Too hard to maintenance

~2007

Easy to writeEasy to

maintenance2007~

Sphinx Before and After

Before

There was no standard ways to write documentsSometime, we need converting markups into other

formats

After

Generate multiple output format from single sourceIntegrated html themes make read docs easierAPI references can be integrated into narrative docsAutomated doc building and hosting by ReadTheDocs

15


Many docs are written by SphinxFor examples

Python libraries/tools:Python, Sphinx, Flask, Jinja2, Django, Pyramid, SQLAlchemy, Numpy, SciPy, scikit-learn, pandas, fabric, ansible, awscli, …

Non python libraries/tools:Chef, CakePHP(2.x), MathJax, Selenium, Varnish

16


Many docs are written by SphinxFor examples

Python libraries/tools:Python, Sphinx, Flask, Jinja2, Django, Pyramid, SQLAlchemy, Numpy, SciPy, scikit-learn, pandas, fabric, ansible, awscli, …

Non python libraries/tools:Chef, CakePHP(2.x), MathJax, Selenium, Varnish

17


SPHINXdocutils

HTML Builder HTML theme(Jinja2)

gettext Builder

*.pot

*.po

I18N

*.mo OmegaT

Pootle

Transifex

Translation Tools, Services

Favorite Editor

Sphinx i18n feature (built-in)

18


reST Parser(directive / role)

doctree(Intermediat

e)

reSTreSTreStructuredText(reST)

$ pip install sphinx

Support tools for translation

How to install Sphinx with i18n tools

To build a sphinx document

$ pip install sphinx-intl$ pip install transifex-client=0.8

19


$ git clone https://github.com/shimizukawa/deepthought.git$ cd deepthought/doc$ make html...Build finished. The HTML pages are in _build/html.

"make html" command generates html files into _build/html.

make html once

20


2. Easy contributable internationalization process with Sphinx

22

What is internationalization?

23


I18N

What is easy contributable?

24


Not a easy way (example 1/3)

The manual are provided only in the HTML files

Rewriting reST files

25



The manual are generated fromreST files and

docstrings in the source files

Rewriting reST filesRewriting docstrins in the source files.

26



OK, we are engineers. We can use git!

Learn gitLearn GitHubgit clone to get the codeTranslation (Yes! This is what I want to

do!)git commitgit pullSometimes resolve conflictgit push

27


Not easy vs easy

Not a easy way Easy way1. Learn git2. Learn GitHub3. git clone to get the

code4. Translation5. git commit, pull,

push6. Resolve conflict7. Build html your self

1. No git2. No Github3. No file4. Translation5. Update

Automatically6. No conflict7. You can get a HTML

output w/o hand-build.

28



29

Two i18n features of SphinxOutput pot files:

from reST

Input po files:to generate translated HTML

30


reST pot

reST

htmlpo

Translation flowGenerate pot

Translate pot into po

Generate Translated HTML

31


reST pot

reST

htmlpo

pot poTranslator

Pleasetranslate

Translate!Thanks for yourContribution!

#: ../../../deep_thought/utils.py:docstring of deep_thought.utils.dumps:1msgid "Serialize ``obj`` to a JSON formatted :class:`str`."msgstr ""

msgid "For example:"msgstr ""

32

Output pot files 3. Sphinx i18n feature

$ make gettext...Build finished. The message catalogs are in _build/gettext.

$ ls _build/gettextapi.pot examples.pot generated.pot index.pot

generated.pot

reST pot

34

Preparing po files to translate

$ sphinx-intl update -p _build/gettext -l zh_cnCreate: locale/zh_cn/LC_MESSAGES/api.poCreate: locale/zh_cn/LC_MESSAGES/examples.poCreate: locale/zh_cn/LC_MESSAGES/generated.poCreate: locale/zh_cn/LC_MESSAGES/index.po

At first, sphinx-intl copy pot files and rename them

pot po

sphinx-intl

$ make gettext$ sphinx-intl update -p _build/gettext -l zh_cnNot Changed: locale/zh_cn/LC_MESSAGES/api.poUpdated: locale/zh_cn/LC_MESSAGES/examples.po +3, -1Not Changed: locale/ma/LC_MESSAGES/generated.po

After change the document, sphinx-intl update differences


Translate po files

#: ../../../deep_thought/utils.py:docstring of deep_thought.utils.dumps:1msgid "Serialize `òbj`` to a JSON formatted :class:`str`."msgstr ""

generated.po

pot po

sphinx-intl

Translator

#: ../../../deep_thought/utils.py:docstring of deep_thought.utils.dumps:1msgid "Serialize `òbj`` to a JSON formatted :class:`str`."msgstr " 序列 `òbj`` 以 JSON 格式 :class:`str` 。 "

generated.po

Translate by using Vim, Emacs, OmegaT, ...

35


Input po files

36


reST

htmlpo

Translated

$ make html...Build finished. The HTML pages are in _build/html.

Big picture

37


reST pot

html

po

make gettext

sphinx-intl

Translator

make html

Doc Author

TranslationCatalog

Translatedcatalog


38

Entire process to translate sphinx docs

39


reST pot

html

po

make gettext

sphinx-intl

make html

Doc Author

TranslationCatalog

Translatedcatalog

Translator

Translator

Translator

Autor / Translator

upload

Translator

clone

Translator

To Be

40


reST pot

html

po

make gettext

sphinx-intl

make html

Doc Author

TranslationCatalog

Translatedcatalog

upload

Translators

To BeAutomated

To BeParallel

clone

Translation tool typesVim / Emacs (Editors)

Edit local filesTranslation support plugins are available.

OmegaT (Translation Tools)Edit local files.Translation support features.

Transifex (Services)Edit onlineTranslation support

features.

41


po

Translators

To BeParallel

Be Parallel by using Transifex

Transifex provides...As API:Upload potDownload po

As Web console:GlossaryTranslation memoryRecommendationAuto-translation

42


po

Translators

Parallel

pot

Upload

pot

Auto Update

sphinx-intltransifex-client

po transifex-clientDownload

Translation on Transifex web console

43


Original Text

Translated Text(you should keep reST syntax)

Suggestions fromTranslation Memory (TM)

Original(pot)

Translation(po)

Copy orig to translationMachine translation

SaveReview (if needed)

Translators

Parallel1

2

4

3

56

To Be Automated

44


po

Translators

Parallel

pot

Upload

pot

Auto Update

sphinx-intltransifex-client


reST

html

make gettext

make html

Doc Author

upload

To BeAutomated

clone

To Be Automated

45


pot

Upload sphinx-intltransifex-client


reST

html

make gettext

make htmlupload

clone1

2 3

456

To BeAutomated

The procedure for automation

1. clone repository2. make gettext3. Upload pot4. Download po5. make html6. Upload html

46


pot



reST

html

make gettext

make htmlupload

clone1

2 3

456

To BeAutomate

d

1. pip install sphinx sphinx-intl transifex-client==0.82. git clone https://github.com/shimizukawa/deepthought.git3. cd deepthought/doc4. sphinx-intl create-transifexrc # create ~/.transifexrc5. sphinx-intl create-txconfig # create .tx/config6. make gettext7. sphinx-intl -p _build/gettext update-txconfig-resources

# update .tx/config8. tx push -s # push pot files to

transifex9. tx pull -l zh_cn # pull po files from

transifex10. make html SPHINXOPTS="-D language=ma"

Shell commands for automationrun.sh

$ export SPHINXINTL_TRANSIFEX_USERNAME=mice$ export SPHINXINTL_TRANSIFEX_PASSWORD=42$ export SPHINXINTL_TRANSIFEX_PROJECT_NAME=deepthought-0_7$ export SPHINXINTL_POT_DIR=_build/gettext$ run.sh

47


The drone.io

48

WebHook

Deploy

Clone repositoryRun shell script

The drone.io is a continuous integration service.


GitHub + drone.io + S3

49

GitHub

Amazon S3

Transifex 1. Clone repository2. make gettext.3. Upload pot4. Download po5. make html6. Upload html


2

1

3

Be Automated

50

pot



reST

html

make gettext

make htmlupload

clone1

2 3

456

To BeAutomated

UploadpotDownload

poupload

WebHookclone

1

5

6make html

make gettext2

3

4

1WebHook

Automated


Automated by drone.io

51

UploadpotDownload

poupload

WebHookclone

1

56

make html

make gettext2

3

4

1WebHook

Automated


View from doc author

Doc Author doesn't require annoying procedure.

UpdateTranslation Source

Doc Author

Notify

See

Commit

52


View from doc translatorsNo gitNo fileNo conflictUpdate AutomaticallyThey can get a translated HTML output w/o hand-

build.

Translators

ParallelSee

TranslateTranslated Pages

53


The entire automated process

54

UpdateTranslation Source

Doc Author

Translators

Parallel

Notify

See

See

Publish

Translate

Commit

DownloadTranslations

Notify

Automated


Summary

Tools1. sphinx - documentation generator2. docutils - base of sphinx3. sphinx-intl - support utility for sphinx i18n4. transifex-client - file transfer tool for

Transifex

Services5. Transifex - translation support service6. Drone.io - continuous integration service

55


5. Tips, tricks, traps

56

TIP: Drone.io limits 15mins for a build

Drone.io limits 15mins for a build.Install from wheel package may help you

57


1. curl -L -s https://example.com/wheelhouse.tgz | tar vzxf -

2. export PIP_FIND_LINKS=./wheelhouse3. pip install sphinx sphinx-intl transifex-client==0.8run.sh

ex. https://bitbucket.org/sphinxjp/docutils-translation/

TRAP: Version of transifex-client

transifex-client 0.11b3 is not stable (for me)Especially for Windows users

If you have encountered trouble with using newer version, please try:

58


$ pip install "transifex-client=0.8"

Drone.io only list-up repositories which you have admin permission.

Prepare a empty repository to create a drone.io project.

59

TRICK: Preparing Drone.io project5. Tips, tricks, traps

Examples

Sphinx-1.3 doc for "ja" translationhttps://drone.io/bitbucket.org/shimizukawa/sphinx-doc13/admin

Sphinx-1.4 doc for "ja" translationhttps://drone.io/bitbucket.org/shimizukawa/sphinx-doc14/admin

Docutils doc for "ja" translationhttps://drone.io/bitbucket.org/sphinxjp/docutils-translation/admin

60


https://drone.io/bitbucket.org/sphinxjp/docutils-translation/admin








Questions?@shimizukawa

Grab me after the session :)

61

Thanks :)

62

easy contributable internationalization process with sphinx @ pyconmy2015

Technology