go & multi platform gui trials and errors

Go & multi platform GUITrials and Errors

Go Conference 2016 sprint @ 4/23/2016@shibukawa (Twitter)github.com/shibukawa

Yoshiki Shibukawa

Who Are You?• Programmer in Mobile Game Department

– http://www.slideshare.net/dena_study/final-fantasy-record-keeper

Yoshiki Shibukawa

Languages C++/JavaScript/Python/Golang etc Japanese/English/Español (studying with Duolingo)

Books Translations: Expert Python/Pomodoro Technique/The Art of

community Original: Mithril, つまみぐい勉強 etc

Before talking my presentation…

• Machine readable docs for web service API

Before talking my presentation…

• Machine readable docs for web service API

YES!

• https://shibukawa.github.io/curl_as_dsl/• I will add net/http, fasthttp server code generate in future• Welcome Ruby/Perl/emacs lisp pull request

Conclusion

• This presentation doesn’t show the best solution for multi platform GUI

• I am exploring possibility about Go and GUI and creating experimental codes.

• GUI programming is not easy• C++ is required

Solutions I triedPros Cons

Toolkit Wrapper The most intuitive solution Maintaining wrappers is extremely hard jobCan’t use deploy support tool

Create new Toolkit in Go Perfect matching with golang

Supporting multi platform is very tough

Hybrid (C++ for GUI, Go for logic)

Solution 1Toolkit Wrapper

Toolkit Wrapper

• I used wxPython, Qt for JavaScript etc…• The most intuitive solution

– You can use it in same methodology with original native language

I tried go-qml

http://qiita.com/shibukawa/items/47cb0ea60bf4a7259b5e

Toolkit Wrapper Cons (1)

• Qt provides deployment helper tool– macdeployqt, windeployqt etc– They collects required .dll/.framework and rewrite

dependency paths and create installer etc– This tool works only for Pre build Qt/C++ project

• Few documents/examples– We reinterpret original toolkit documents– C++ knowledge is required

• Debug is very hard– Where does this panic come from?

Toolkit Wrapper Cons (2)

• Maintaining toolkit wrapper is hard work– GUI toolkit is contains many C++ classes

• Qt has more than 1200 classes– Many wrappers were discarded

• QtJS, QtJambi, ruby qtbindings, go-qml etc…– GUI toolkits update continuously

• Basically newer version is better• Almost all toolkit gave up updating• I have to update wrapper by myself to use newer future

– Languages update too

Expect

Hours

Create My ApplicationMaintaining Toolkit

Actually

Hours

Create My ApplicationMaintaining Toolkit

Not all GUI toolkit wrappers are bad

PyQt Commercial (Qt 5.5 supports)

wxPython Nightly build still supports latest Python(2.7/3.5)

GTK# Xamarin supports?

pygobject or some libs gobject introspection is good for keeping fresh

Swing from Jython/JRuby Using toolkit directly

tkinter / Qt Quick Official wrapper provided by suppliers

• Some wrappers provide updates continuously• Maintaining is too heavy for individual developer• Community/Organizational support is required

Solution 2Create New Toolkit in Go

Implement GUI toolkit in Go• There are many small GUI toolkits in C/C++

– http://qiita.com/shimacpyon/items/e5d2a4e2019273345c37• Port one of them to Go!

– NanoGUI seems good to port

※ When start porting, I didn’t the existence of Shiny

GUI toolkit on GL building blocks

Font Loader Font Renderer

Texture Management

Font TextureManagement

Event Handling

Vector GraphicsImage Loader

Widget Drawing

OpenGLOS Window System

Widget Management

NanoGUI toolkit on GL building blocks

Font Loader Font Renderer

Texture Management

Font TextureManagement

Event Handling

Vector GraphicsImage Loader

Widget Drawing

OpenGLOS Window System

Widget Management

stb_truetype.h

stb_image.h

NanoVG

GLFW

NanoGUI

fontstash

My toolkit on GL building blocks

Font Loader Font Renderer

Texture Management

Font TextureManagement

Event Handling

Vector GraphicsImage Loader

Widget Drawing

OpenGLOS Window System

Widget Management

go’s image

NanoVGo

goxjs/glfwgoxjs/gl

NanoGUI.go

nanovgo

nanogui-go

Input Method Editor issue

• IME support is needed to input Asian chars• Now IME becomes more generic than before

– Voice / pen input uses same infrastructure– Modern OS treats IMEs as keyboards

• This is processed in GLFW (C++ layer)Japanese IME

English

Speech Input

Handwriting Input

OperatingSystem

Application

※On Windows, Handwriting features seem to be a part of MS IMEs

IME support levels

• Level 1: Root window mode

• Level 2: Over the spot

Images from http://www.mozilla-japan.org/projects/intl/input-method-spec.htmlThis is not common classify. My original.※

• Level 3: On the spot

• Level 4: Context aware conversion

Image from http://nyaruru.hatenablog.com/entry/20070309/p1

IME tasks viewed from ApplicationLevel 1 Level 2 Level 3 Level 4

Receive multi byte input ✔ ✔ ✔ ✔

Send cursor position ✔ ✔ ✔

React to preedit text ✔ ✔

Send surrounding text around cursor

✔

• To achieve higher level IME support, applications should communicate with IME more

• Each OS provides each API for IME. I should add new method to GLFW to absorb OS layers

Achievement Unlocked!

Current Status

• Windows– Level 3. Now I use IMM32(old), but Text Services

Framework (TSF) is preferred.– Old Chrome and Firefox use TSF (but MPL is not

good for GLFW)• MacOS X

– Level 3. Applications should implement NSTextInputClient protocol. I don’t implement Level 4 feature yet.

https://github.com/glfw/glfw/pull/658

Current Status

• X11– Level 3. XIM protocol doesn’t support level 4.– On Linux/BSDs, there many input method frameworks

(UIM, SCIM, ibus, fcitx etc) and Qt/GTK communicate with them directly, but don’t use them (not to introduce new dependency to GLFW)

• Wayland– Not yet

• Mir– Not yet

https://github.com/glfw/glfw/pull/658

Future Plan

• Improve GLFW more to support level 4 and Wayland, Mir

• Reconstruct NanoVGo– Cache geometry – Now it redraw from scratch on every frame

• Reconstruct NanoGUI.go– ListView support like QAbstractItemView /

QAbstractItemModel– Better image resource handling

Solution 3Hybrid Style

Hybrid Model

Thin GUI Logic

C++ is not so fear than before

• Use Qt via C++ directly– No wrapper issues– Using C++ is not tough anymore– C++11 is much better than 20 years ago– Qt5 supports C++11 (C++11 is required from 5.7)

• But building code in Golang is still faster than C++ for me

C++ support tool• Go is easy to build. C++ is boring• I made helper tool to make C++ easy

– https://github.com/qtpm/qtpm• Generate CMakeLists.txt. Support following IDEs:

– Qt Creator– CLion (thank you for Jetbrains OSS support program)

• It support the following commands like go:– qtpm get– qtpm build– qtpm vet– qtpm fmt

How to communicate?

Thin GUI Logic

???

• I am trying several solutions for communication

Qt’s thread model

• You should understand it if use Go with Qt• http://doc.qt.io/qt-5/threads-qobject.html• All Qt’s objects are belonging to thread

– Calling object’s method from non-owner thread causes panic

– To call other thread’s method, QMetaObject::invokeMethod is needed(signal/slot uses this behind it)

– Each thread has event loop– Of course, blocking main thread is not preferred

3.1: QProcess

• Create standalone CUI program in Golang and call via QProcess class– If the job is simple, it is the best (it becomes

simple code)• Bi-directional communication is tough

– Receiving message in SLOT (callback)– No fluent code

3.2: QLocalServer/QLocalSocket

• Qt provides special wrapper to provides high performance inter process communication– On Windows, it uses named pipe– On other environment, it uses UNIX domain socket

• Go implementation is ready– I tested on Windows and Mac

• How to manage process?– I need to implement some process supervisor

https://github.com/shibukawa/localsocket

3.3: c-archive

• Creating c-archive and link with Qt application• Go 1.7 will support it on Windows too

• If the API is simple, it is the best• All method is called synchronous by default• If callback from Golang to C++, it is little messy

– Polling is easy, but not beautiful– Pass function pointer from C++ and call it in Go

• https://gist.github.com/shibukawa/ab516964b0390cb234c7c42e5c9bfda1

• How to specify “-mmacosx-version-min=10.x”?

3.4: dRuby style

• dRuby is a my favorite API in Ruby• dRuby hooks method call and transfers

method name and parameters via TCP/IP– Go: https://github.com/shibukawa/tobubus– C++: https://github.com/qtpm/QtTobubus

DRbObjectClient

hello(“world”)

method_missing()

DRbServerActualInstance

hello(“world”)

tobubus

• Now it works on QLocalSocket/Server• It uses Qt

reflection(QMetaObject/QMetaMethod) and Go’s reflect

• Method call is serialized by CBOR– CBOR is almost RFC version of MessagePack

tobubusC++/Go Client

hello(“world”)

ProxyC++/Go ActualInstance

hello(“world”)

tobubus

CBOR on local socket

3.5: gRPC

• I didn’t try it– I don’t want to create extra interface definitions

for communication• It is suitable for loose coupling of micro services

– It seems difficult to use with Qt threading model• I may need create wrappers for all interfaces in Qt

Combinations of my solutionsInfrastructure

Connect Pros/Cons Simple Tested

QProcess stdin/stdout

bi-direction is difficultGo is always running in other process (should take care QThread)

S ✔

tobubus Not implemented yet A

QProcess + QLocalSocket/Server

tobubus It needs process supervisorC++ client code is not clean because of Qt thread modelGo is always running in other process (should take care QThread)

B ✔

c-archive linker Writing callback is little messyWindows doesn’t support until 1.7It works synchronously

S ✔

Which is better?

• QProcess is the most simple now• tobubus + QLocalSocket/Server was originally

designed for generic plug-in system. I already implemented but I feel too heavy now.

• c-archive is a stable way. It is not bad if you can wait by 1.7 release.

Conclusion

• This presentation doesn’t show the best solution for multi platform GUI

• I am exploring possibility about Go and GUI and creating experimental codes. – I will still improve GLFW IME support to realize pure

Go solution– If you can wait by Go 1.7, c-archive is better

• GUI programming is not easy• C++ is required

Actually

Hours

Create My ApplicationMaintaining Toolkit

In future I want to achieve

Hours

Create My ApplicationMaintaining Toolkit