Zulip DocumentationRelease 1.4.0

The Zulip Team

Sep 06, 2016

Overview

1 Zulip overview 31.1 Community . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31.2 Installing the Zulip Development environment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31.3 Running Zulip in production . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31.4 Ways to contribute . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41.5 How to get involved with contributing to Zulip . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41.6 License . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5

2 Zulip architectural overview 72.1 Key Codebases . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72.2 Usage assumptions and concepts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72.3 Components . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8

3 Directory structure 113.1 Core Python files . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 113.2 HTML Templates . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 113.3 JavaScript and other static assets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 123.4 Tests . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 123.5 Management commands . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 123.6 Scripts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 123.7 API and Bots . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 133.8 Production puppet configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 133.9 Additional Django apps . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 133.10 Jinja2 Compatibility Files . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 133.11 Translation files . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 133.12 Documentation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14

4 Zulip 2016 Roadmap 154.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 154.2 Burning problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 154.3 Core User Experience . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 164.4 Ease of setup and onboarding issues . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 164.5 Internationalization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 164.6 User Experience at scale . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 164.7 Administration and management . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 174.8 Scalability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 174.9 Performance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 174.10 Technology improvements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17

i

4.11 Technical Debt . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 184.12 Deployment and upgrade process . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 184.13 Security . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 184.14 Testing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 184.15 Development environment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 194.16 Documentation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 194.17 Integrations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 194.18 Android app . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 194.19 iOS app . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 204.20 Desktop apps . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 204.21 Community . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20

5 Version History 215.1 Unreleased . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 215.2 1.4 - 2016-08-25 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 215.3 1.3.13 - 2016-06-21 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 245.4 1.3.12 - 2016-05-10 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 255.5 1.3.11 - 2016-05-02 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 255.6 1.3.10 - 2016-01-21 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 265.7 1.3.9 - 2015-11-16 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 265.8 1.3.8 - 2015-11-15 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 265.9 1.3.7 - 2015-10-19 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27

6 Requirements 296.1 Server . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 296.2 Credentials needed . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30

7 Installation 317.1 Step 0: Subscribe . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 317.2 Step 1: Install SSL Certificates . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 317.3 Step 2: Download and install latest release . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 317.4 Step 3: Configure Zulip . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 327.5 Step 4: Initialize Zulip database . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 327.6 Step 5: Create a Zulip organization and login . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 327.7 Troubleshooting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33

8 Troubleshooting 358.1 Using supervisorctl . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 358.2 Troubleshooting services . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36

9 Customize Zulip 399.1 Integrations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 399.2 Streams and Topics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 399.3 Notification settings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 409.4 Mobile and desktop apps . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 409.5 All other features . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 409.6 Enjoy your Zulip installation! . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40

10 Secure, maintain, and upgrade 4110.1 Upgrading . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4110.2 Upgrading from a git repository . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4310.3 Backups . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4310.4 Monitoring . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4410.5 Scalability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4510.6 Security Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46

ii

10.7 Management commands . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48

11 Authentication methods 5111.1 Adding additional methods using python-social-auth . . . . . . . . . . . . . . . . . . . . . . . . . . 5111.2 Remote User SSO Authentication . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51

12 Postgres database details 5512.1 Remote Postgres database . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5512.2 Debugging postgres database issues . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5612.3 Stopping the Zulip postgres database . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5612.4 Debugging issues starting postgres . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5612.5 Postgres Vacuuming alerts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57

13 Development environment options 59

14 Vagrant environment setup tutorial 6114.1 Requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6114.2 Step 1: Install Prerequisites . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6214.3 Step 2: Get Zulip Code . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6414.4 Step 3: Start the dev environment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6414.5 Step 4: Developing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6614.6 Troubleshooting & Common Errors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68

15 Vagrant environment setup (in brief) 7315.1 Specifying a proxy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 74

16 Installing directly on Ubuntu 75

17 Installing manually on UNIX 7717.1 On Debian or Ubuntu systems: . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7717.2 On Fedora 22 (experimental): . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7817.3 On CentOS 7 Core (experimental): . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7917.4 On OpenBSD 5.8 (experimental): . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8017.5 Common to Fedora/CentOS instructions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8017.6 All Systems: . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 81

18 Using Docker (experimental) 83

19 Using the Development Environment 85

20 Writing a new integration 8720.1 Types of integrations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8720.2 General advice . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8820.3 Webhook integrations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8820.4 Python script and plugin integrations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8920.5 Documenting your integration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9020.6 Hello World webhook Walkthrough . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 90

21 Writing a new application feature 9721.1 General Process in brief . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9721.2 Example Feature . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 98

22 Writing views in Zulip 10322.1 What this covers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10322.2 What is a view? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10322.3 Modifying urls.py . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 103

iii

22.4 Writing human-readable views . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10322.5 Writing API REST endpoints . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10422.6 Legacy endpoints used by the web client . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10822.7 Webhook integration endpoints . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 108

23 Life of a Request 10923.1 A request is sent to the server, and handled by Nginx . . . . . . . . . . . . . . . . . . . . . . . . . . 10923.2 Nginx secures traffic with SSL . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10923.3 Static files are served directly by Nginx . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10923.4 Nginx routes other requests between tornado and django . . . . . . . . . . . . . . . . . . . . . . . . 10923.5 Django routes the request to a view in urls.py files . . . . . . . . . . . . . . . . . . . . . . . . . . . 11023.6 Views serving HTML are internationalized by server path . . . . . . . . . . . . . . . . . . . . . . . 11023.7 API endpoints use REST . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11023.8 Django calls rest_dispatch for REST endpoints, and authenticates . . . . . . . . . . . . . . . . . . . 11123.9 The view will authorize the user, extract request variables, and validate them . . . . . . . . . . . . . 11223.10 Results are given as JSON . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 112

24 Version control 11324.1 Commit Discipline . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11324.2 Commit Messages . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 114

25 Code style and conventions 11525.1 Be consistent! . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11525.2 Lint tools . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11525.3 Secrets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11525.4 Dangerous constructs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11625.5 JS array/object manipulation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11725.6 More arbitrary style things . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 118

26 Testing and writing tests 12126.1 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12126.2 Running tests . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12126.3 Schema and initial data changes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12226.4 Wiping the test databases . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12226.5 Manual testing (local app + web browser) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12326.6 Python 3 Compatibility . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 123

27 Python static type checker (mypy) 12527.1 type_debug.py . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12527.2 Zulip goals . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12627.3 Installing mypy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12627.4 Running mypy on Zulip’s code locally . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12627.5 Excluded files . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12727.6 Mypy is there to find bugs in Zulip before they impact users . . . . . . . . . . . . . . . . . . . . . . 12727.7 Annotating strings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 127

28 Settings system 12928.1 Server settings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12928.2 Realm settings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 130

29 Queue processors 13329.1 Adding a new queue processor . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13329.2 Publishing events into a queue . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13429.3 Clearing a RabbitMQ queue . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 134

iv

30 Unread counts and the pointer 13530.1 Pointer logic . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13530.2 Unread count logic . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 136

31 Markdown implementation 13731.1 Zulip’s Markdown philosophy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13731.2 Zulip’s Changes to Markdown . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 138

32 Static asset pipeline 14132.1 Primary build process . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14132.2 Webpack/CommonJS modules . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14132.3 Adding static files . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 141

33 Schema Migrations 143

34 HTML and CSS 14534.1 Zulip CSS organization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14534.2 Editing Zulip CSS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14534.3 CSS Style guidelines . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14634.4 Validating CSS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 146

35 Full-text search 14735.1 The default full-text search implementation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14735.2 An optional full-text search implementation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 147

36 Translating Zulip 14936.1 Setting Default Language in Zulip . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14936.2 Translation Resource Files . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14936.3 Transifex Config . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14936.4 Translation Process . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14936.5 Backend Translations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15036.6 Frontend Translations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15136.7 Testing Translations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 152

37 Logging and Performance Debugging 153

38 Documentation 155

39 Indices and tables 157

v

vi

Zulip Documentation, Release 1.4.0

Zulip is a powerful, open source group chat application. Written in Python and using the Django framework, Zulipsupports both private messaging and group chats via conversation streams.

Zulip also supports fast search, drag-and-drop file uploads, image previews, group private messages, audible notifica-tions, missed-message emails, desktop apps, and much more.

Further information on the Zulip project and its features can be found at https://www.zulip.org and in these docs. Ourcode is available at our GitHub repository.

This set of documents covers installation and contribution instructions.

Contents:

• Overview

• Zulip in production

• Installation for developers

• Developer tutorials

• Code contribution guide

• Subsystem documentation

Zulip overview | Community | Installing for dev | Installing for production | Ways to contribute | How to get involved| License

Overview 1

Zulip Documentation, Release 1.4.0

2 Overview

CHAPTER 1

Zulip overview

Zulip is a powerful, open source group chat application. Written in Python and using the Django framework, Zulipsupports both private messaging and group chats via conversation streams.

Zulip also supports fast search, drag-and-drop file uploads, image previews, group private messages, audible notifica-tions, missed-message emails, desktop apps, and much more.

Further information on the Zulip project and its features can be found at https://www.zulip.org.

1.1 Community

There are several places online where folks discuss Zulip.

One of those places is our public Zulip instance. You can go through the simple signup process at that link, and thenyou will soon be talking to core Zulip developers and other users. To get help in real time, you will have the best luckfinding core developers roughly between 16:00 UTC and 23:59 UTC. Most questions get answered within a day.

We have a Google mailing list that is currently pretty low traffic. It is where we do things like announce publicmeetings or major releases. You can also use it to ask questions about features or possible bugs.

Last but not least, we use GitHub to track Zulip-related issues (and store our code, of course). Anybody with a Githubaccount should be able to create Issues there pertaining to bugs or enhancement requests. We also use Pull Requestsas our primary mechanism to receive code contributions.

1.2 Installing the Zulip Development environment

The Zulip development environment is the recommended option for folks interested in trying out Zulip. This isdocumented in the developer installation guide.

1.3 Running Zulip in production

Zulip in production only supports Ubuntu 14.04 right now, but work is ongoing on adding support for additional plat-forms. The installation process is documented at https://zulip.org/server.html and in more detail in the documentation.

3

Zulip Documentation, Release 1.4.0

1.4 Ways to contribute

Zulip welcomes all forms of contributions! The page documents the Zulip development process.

• Pull requests. Before a pull request can be merged, you need to to sign the Dropbox Contributor LicenseAgreement. Also, please skim our commit message style guidelines.

• Testing. The Zulip automated tests all run automatically when you submit a pull request, but you can also runthem all in your development environment following the instructions in the testing docs. You can also try outour new desktop client, which is in alpha; we’d appreciate testing and feedback.

• Developer Documentation. Zulip has a growing collection of developer documentation on Read The Docs.Recommended reading for new contributors includes the directory structure and new feature tutorial. You canalso improve Zulip.org.

• Mailing lists and bug tracker. Zulip has a development discussion mailing list and uses GitHub issues . Thereare also lists for the Android and iOS apps. Feel free to send any questions or suggestions of areas whereyou’d love to see more documentation to the relevant list! Please report any security issues you discover tozulip-security@googlegroups.com.

• App codebases. This repository is for the Zulip server and web app (including most integrations); the desktop,Android, and iOS apps, are separate repositories, as are our experimental React Native iOS app and alphaElectron desktop app.

• Glue code. We maintain a Hubot adapter and several integrations (Phabricator, Jenkins, Puppet, Redmine, andTrello), plus node.js API bindings, and a full-text search PostgreSQL extension, as separate repos.

• Translations. Zulip is in the process of being translated into 10+ languages, and we love contributions to ourtranslations. See our translating documentation if you’re interested in contributing!

1.5 How to get involved with contributing to Zulip

First, subscribe to the Zulip development discussion mailing list.

The Zulip project uses a system of labels in our issue tracker to make it easy to find a project if you don’t have yourown project idea in mind or want to get some experience with working on Zulip before embarking on a larger projectyou have in mind:

• Integrations. Integrate Zulip with another piece of software and contribute it back to the community! Writingan integration can be a great first contribution. There’s detailed documentation on how to write integrations inthe Zulip integration writing guide.

• Bite Size: Smaller projects that might be a great first contribution.

• Documentation: The Zulip project loves contributions of new documentation.

• Help Wanted: A broader list of projects that nobody is currently working on.

• Platform support: These are open issues about making it possible to install Zulip on a wider range of platforms.

• Bugs: Open bugs.

• Feature requests: Browsing this list can be a great way to find feature ideas to implement that other Zulip usersare excited about.

• 2016 roadmap milestone: The projects that are priorities for the Zulip project. These are great projects if you’relooking to make an impact.

4 Chapter 1. Zulip overview

Zulip Documentation, Release 1.4.0

If you’re excited about helping with an open issue, just post on the conversation thread that you’re working on it.You’re encouraged to ask questions on how to best implement or debug your changes – the Zulip maintainers areexcited to answer questions to help you stay unblocked and working efficiently.

We also welcome suggestions of features that you feel would be valuable or changes that you feel would make Zulip abetter open source project, and are happy to support you in adding new features or other user experience improvementsto Zulip.

If you have a new feature you’d like to add, we recommend you start by opening a GitHub issue about the featureidea explaining the problem that you’re hoping to solve and that you’re excited to work on it. A Zulip maintainer willusually reply within a day with feedback on the idea, notes on any important issues or concerns, and and often tips onhow to implement or test it. Please feel free to ping the thread if you don’t hear a response from the maintainers – wetry to be very responsive so this usually means we missed your message.

For significant changes to the visual design, user experience, data model, or architecture, we highly recommendposting a mockup, screenshot, or description of what you have in mind to zulip-devel@ to get broad feedback beforeyou spend too much time on implementation details.

Finally, before implementing a larger feature, we highly recommend looking at the new feature tutorial and codingstyle guidelines on ReadTheDocs.

Feedback on how to make this development process more efficient, fun, and friendly to new contributors is verywelcome! Just send an email to the Zulip Developers list with your thoughts.

1.6 License

Copyright 2011-2015 Dropbox, Inc.

Licensed under the Apache License, Version 2.0 (the “License”); you may not use this file except in compliance withthe License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an“AS IS” BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See theLicense for the specific language governing permissions and limitations under the License.

The software includes some works released by third parties under other free and open source licenses. Those works areredistributed under the license terms under which the works were received. For more details, see the THIRDPARTYfile included with this distribution.

1.6. License 5

Zulip Documentation, Release 1.4.0

6 Chapter 1. Zulip overview

CHAPTER 2

Zulip architectural overview

2.1 Key Codebases

The core Zulip application is at https://github.com/zulip/zulip and is a web application written in Python 2.7 (soon toalso support Python 3) and using the Django framework. That codebase includes server-side code and the web client,as well as Python API bindings and most of our integrations with other services and applications (see the directorystructure guide).

We maintain several separate repositories for integrations and other glue code: a Hubot adapter; integrations withPhabricator, Jenkins, Puppet, Redmine, and Trello; node.js API bindings; and our full-text search PostgreSQL exten-sion.

Our mobile clients are separate code repositories: Android, iOS (stable), and our experimental React Native iOSapp. Our legacy desktop application (implemented in QT/WebKit) and our new, alpha cross-platform desktop app(implemented in Electron) are also separate repositories.

We use Transifex to do translations.

In this overview we’ll mainly discuss the core Zulip server and web application.

2.2 Usage assumptions and concepts

Zulip is a real-time web-based chat application meant for companies and similar groups ranging in size from a smallteam to more than a thousand users. It features real-time notifications, message persistence and search, public groupconversations (streams), invite-only streams, private one-on-one and group conversations, inline image previews, teampresence/buddy list, a rich API, Markdown message support, and numerous integrations with other services. Themaintainer team aims to support users who connect to Zulip using dedicated iOS, Android, Linux, Windows, and MacOS X clients, as well as people using modern web browsers or dedicated Zulip API clients.

A server can host multiple Zulip realms (organizations) at the same domain, each of which is a private chamber withits own users, streams, customizations, and so on. This means that one person might be a user of multiple Zuliprealms. The administrators of a realm can choose whether to allow anyone to register an account and join, or onlyallow people who have been invited, or restrict registrations to members of particular groups (using email domainnames or corporate single-sign-on login for verification). For more on scalability and security considerations, see thesecurity section of the production maintenance instructions.

The default Zulip home screen is like a chronologically ordered inbox; it displays messages, starting at the oldestmessage that the user hasn’t viewed yet (for more on that logic, see the guide to the pointer and unread counts). Thehome screen displays the most recent messages in all the streams a user has joined (except for the streams they’vemuted), as well as private messages from other users, in strict chronological order. A user can narrow to view only the

7

Zulip Documentation, Release 1.4.0

messages in a single stream, and can further narrow to focus on a topic (thread) within that stream. Each narrow hasits own URL.

Zulip’s philosophy is to provide sensible defaults but give the user fine-grained control over their incoming informationflow; a user can mute topics and streams, and can make fine-grained choices to reduce real-time notifications they findirrelevant.

2.3 Components

2.3.1 Tornado and Django

We use both the Tornado and Django Python web frameworks.

Django is the main web application server; Tornado runs the server-to-client real-time push system. The app serversare configured by the Supervisor configuration (which explains how to start the server processes; see “Supervisor”below) and the nginx configuration (which explains which HTTP requests get sent to which app server).

Tornado is an asynchronous server and is meant specifically to hold open tens of thousands of long-lived (long-pollingor websocket) connections – that is to say, routes that maintain a persistent connection from every running client. Forthis reason, it’s responsible for event (message) delivery, but not much else. We try to avoid any blocking calls inTornado because we don’t want to delay delivery to thousands of other connections (as this would make Zulip verymuch not real-time). For instance, we avoid doing cache or database queries inside the Tornado code paths, since thoseblocking requests carry a very high performance penalty for a single-threaded, asynchronous server.

The parts that are activated relatively rarely (e.g. when people type or click on something) are processed by the Djangoapplication server. One exception to this is that Zulip uses websockets through Tornado to minimize latency on thecode path for sending messages.

2.3.2 nginx

nginx is the front-end web server to all Zulip traffic; it serves static assets and proxies to Django andTornado. It handles HTTP requests according to the rules laid down in the many config files found inzulip/puppet/zulip/files/nginx/.

zulip/puppet/zulip/files/nginx/zulip-include-frontend/app is the most important of thesefiles. It explains what happens when requests come in from outside.

• In production, all requests to URLs beginning with /static/ are served from the cor-responding files in /home/zulip/prod-static/, and the production build process(tools/build-release-tarball) compiles, minifies, and installs the static assets into theprod-static/ tree form. In development, files are served directly from /static/ in the git repos-itory.

• Requests to /json/get_events, /api/v1/events, and /sockjs are sent to the Tornado server. Theseare requests to the real-time push system, because the user’s web browser sets up a long-lived TCP connectionwith Tornado to serve as a channel for push notifications. nginx gets the hostname for the Tornado server viapuppet/zulip/files/nginx/zulip-include-frontend/upstreams.

• Requests to all other paths are sent to the Django app via the UNIXsocket unix:/home/zulip/deployments/fastcgi-socket (defined inpuppet/zulip/files/nginx/zulip-include-frontend/upstreams). We usezproject/wsgi.py to implement FastCGI here (see django.core.wsgi).

8 Chapter 2. Zulip architectural overview

Zulip Documentation, Release 1.4.0

2.3.3 Supervisor

We use supervisord to start server processes, restart them automatically if they crash, and direct logging.

The config file is zulip/puppet/zulip/files/supervisor/conf.d/zulip.conf. This is where Tor-nado and Django are set up, as well as a number of background processes that process event queues. We use eventqueues for the kinds of tasks that are best run in the background because they are expensive (in terms of performance)and don’t have to be synchronous — e.g., sending emails or updating analytics. Also see the queuing guide.

2.3.4 memcached

memcached is used to cache database model objects. zerver/lib/cache.py andzerver/lib/cache_helpers.py manage putting things into memcached, and invalidating the cachewhen values change. The memcached configuration is in puppet/zulip/files/memcached.conf.

2.3.5 Redis

Redis is used for a few very short-term data stores, such as in the basis of zerver/lib/rate_limiter.py, aper-user rate limiting scheme example), and the email-to-Zulip integration.

Redis is configured in zulip/puppet/zulip/files/redis and it’s a pretty standard configuration except forthe last line, which turns off persistence:

# Zulip-specific configuration: disable saving to disk.save ""

memcached was used first and then we added Redis specifically to implement rate limiting. We’re discussing switchingeverything over to Redis.

2.3.6 RabbitMQ

RabbitMQ is a queueing system. Its config files live in zulip/puppet/zulip/files/rabbitmq. Initialconfiguration happens in zulip/scripts/setup/configure-rabbitmq.

We use RabbitMQ for queuing expensive work (e.g. sending emails triggered by a message, push notifications, someanalytics, etc.) that require reliable delivery but which we don’t want to do on the main thread. It’s also used forcommunication between the application server and the Tornado push system.

Two simple wrappers around pika (the Python RabbitMQ client) are in zulip/server/lib/queue.py.There’s an asynchronous client for use in Tornado and a more general client for use elsewhere.

zerver/lib/event_queue.py has helper functions for putting events into one queue or another. Most of theprocesses started by Supervisor are queue processors that continually pull things out of a RabbitMQ queue and handlethem.

Also see the queuing guide.

2.3.7 PostgreSQL

PostgreSQL (also known as Postgres) is the database that stores all persistent data, that is, data that’s expected to livebeyond a user’s current session.

2.3. Components 9

Zulip Documentation, Release 1.4.0

In production, Postgres is installed with a default configuration. The directory that would contain configuration files(puppet/zulip/files/postgresql) has only a utility script and a custom list of stopwords used by a Post-gresql extension.

In a development environment, configuration of that postgresql extension is handled bytools/postgres-init-dev-db (invoked by tools/provision.py). That file also manages settingup the development postgresql user.

tools/provision.py also invokes tools/do-destroy-rebuild-database to create the actualdatabase with its schema.

2.3.8 Nagios

Nagios is an optional component used for notifications to the system administrator, e.g., in case of outages.

zulip/puppet/zulip/manifests/nagios.pp installs Nagios plugins from pup-pet/zulip/files/nagios_plugins/.

This component is intended to install Nagios plugins intended to be run on a Nagios server; most of the ZulipNagios plugins are intended to be run on the Zulip servers themselves, and are included with the relevant com-ponent of the Zulip server (e.g. puppet/zulip/manifests/postgres_common.pp installs a few under/usr/lib/nagios/plugins/zulip_postgres_common).

10 Chapter 2. Zulip architectural overview

CHAPTER 3

Directory structure

This page documents the Zulip directory structure, where to find things, and how to decide where to put a file.

You may also find the new application feature tutorial helpful for understanding the flow through these files.

3.1 Core Python files

Zulip uses the Django web framework, so a lot of these paths will be familiar to Django developers.

• zproject/urls.py Main Django routes file. Defines which URLs are handled by which view functions ortemplates.

• zerver/models.py Main Django models file. Defines Zulip’s database tables.

• zerver/lib/actions.py Most code doing writes to user-facing database tables.

• zerver/views/*.py Most Django views.

• zerver/views/webhooks/ Webhook views for Zulip integrations.

• zerver/tornadoviews.py Tornado views.

• zerver/worker/queue_processors.py Queue workers.

• zerver/lib/*.py Most library code.

• zerver/lib/bugdown/ Backend Markdown processor.

• zproject/backends.py Authentication backends.

3.2 HTML Templates

See our translating docs for details on Zulip’s templating systems.

• templates/zerver/ For Jinja2 templates for the backend (for zerver app).

• static/templates/ Handlebars templates for the frontend.

11

Zulip Documentation, Release 1.4.0

3.3 JavaScript and other static assets

• static/js/ Zulip’s own JavaScript.

• static/styles/ Zulip’s own CSS.

• static/images/ Zulip’s images.

• static/third/ Third-party JavaScript and CSS that has been vendored.

• node_modules/ Third-party JavaScript installed via npm.

• assets/ For assets not to be served to the web (e.g. the system to generate our favicons).

3.4 Tests

• zerver/tests/ Backend tests.

• frontend_tests/node_tests/ Node Frontend unit tests.

• frontend_tests/casper_tests/ Casper frontend tests.

• tools/test-* Developer-facing test runner scripts.

3.5 Management commands

These are distinguished from scripts, below, by needing to run a Django context (i.e. with database access).

• zerver/management/commands/Management commands one might run at a production deployment site(e.g. scripts to change a value or deactivate a user properly).

3.6 Scripts

• scripts/ Scripts that production deployments might run manually (e.g., restart-server).

• scripts/lib/ Scripts that are needed on production deployments but humans should never run directly.

• scripts/setup/ Scripts that production deployments will only run once, during installation.

• tools/ Scripts used only in a Zulip development environment. These are not included in production releasetarballs for Zulip, so that we can include scripts here one wouldn’t want someone to run in production acciden-tally (e.g. things that delete the Zulip database without prompting).

• tools/setup/ Subdirectory of tools/ for things only used during the development environment setupprocess.

• tools/travis/ Subdirectory of tools/ for things only used to setup and run our tests in Travis CI. Actu-ally test suites should go in tools/.

12 Chapter 3. Directory structure

Zulip Documentation, Release 1.4.0

3.7 API and Bots

• api/ Zulip’s Python API bindings (released separately).

• api/examples/ API examples.

• api/integrations/ Bots distributed as part of the Zulip API bundle.

• bots/ Previously Zulip internal bots. These usually need a bit of work.

3.8 Production puppet configuration

This is used to deploy essentially all configuration in production.

• puppet/zulip/ For configuration for production deployments.

• puppet/zulip/manifests/voyager.pp Main manifest for Zulip standalone deployments.

3.9 Additional Django apps

• confirmation Email confirmation system.

• analytics Analytics for the Zulip server administrator (needs work to be useful to normal Zulip sites).

• corporate The old Zulip.com website. Not included in production distribution.

• zilencer Primarily used to hold management commands that aren’t used in production. Not included inproduction distribution.

3.10 Jinja2 Compatibility Files

• zproject/jinja2/__init__.py Jinja2 environment.

• zproject/jinja2/backends.py Jinja2 backend.

• zproject/jinja2/compressors.py Jinja2 compatible functions of Django-Pipeline.

3.11 Translation files

• locale/ Backend (Django) translations data files.

• static/locale/ Frontend translations data files.

3.7. API and Bots 13

Zulip Documentation, Release 1.4.0

3.12 Documentation

• docs/ Source for this documentation.

You can consult the repository’s .gitattributes file to see exactly which components are excluded from produc-tion releases (release tarballs are generated using tools/build-release-tarball).

14 Chapter 3. Directory structure

CHAPTER 4

Zulip 2016 Roadmap

4.1 Introduction

Zulip has received a great deal of interest and attention since it was released as free and open source software byDropbox. That attention has come with a lot of active development work from members of the Zulip community.From when Zulip was released as open source in late September 2015 through today (mid-April, 2016), over 300 pullrequests have been submitted to the various Zulip repositories (and over 250 have been merged!), the vast majority ofwhich are submitted by Zulip’s users around the world (as opposed to the small core team that reviews and merges thepull requests).

In any project, there can be a lot of value in periodically putting together a roadmap detailing the major areas where theproject is hoping to improve. This can be especially important in an open source project like Zulip where developmentis distributed across many people around the world. This roadmap is intended to organize a list of the most importantimprovements that should be made to Zulip in the relatively near future. Our aim is to complete most of theseimprovements in 2016.

This document is not meant to constrain in any way what contributions to Zulip will be accepted; instead, it will beused by the Zulip core team to prioritize our efforts, measure progress on improving the Zulip product, hold ourselvesaccountable for making Zulip improve rapidly, and celebrate members of the community who contribute to projectson the roadmap.

If you’re someone interested in making a larger contribution to Zulip and looking for somewhere to start, this roadmapis the best place to look for substantial projects that will definitely be of value to the community (if you’re looking fora starter project, see the guide to getting involved with Zulip).

We occasionally update this roadmap by adding strikethrough for issues that have been resolved.

Without further ado, below is the Zulip 2016 roadmap.

4.2 Burning problems

The top problem for the Zulip project is the state of the mobile apps. The Android app has started seeing rapid progressthanks to a series of contributions by Lisa Neigut of Recurse Center, and we believe it is on a good path. The iOS apphas fewer features than Android and has more bugs, but more importantly is in need of an experienced iOS developerwho has time to drive the project.

Update: Neeraj Wahi is leading an effort on to write a new React Native iOS app for Zulip to replace the old iOS app.

15

Zulip Documentation, Release 1.4.0

4.3 Core User Experience

This category includes important improvements to the core user experience that will benefit all users.

• Improve missed message notifications to make “reply” work nicely

• Add support for showing “user is typing” notifications

• Add pretty bubbles for recipients in the compose box

• Finish and merge support for pinning a few important streams

• Display stream descriptions more prominently

• Integration inline URL previews

• Add support for managing uploaded files

• Make Zulip onboarding experience smoother for teams not used to topics. That specific proposal might not beright but the issue is worth investing time in.

4.4 Ease of setup and onboarding issues

This category focuses on issues users experience when installing a new Zulip server or setting up a new Zulip realm.

• Create a web flow for setting up a new realm / the first realm on a new server (currently, it’s a command-lineprocess)

• Document or better script solution to rabbitmq startup issues

• Add a mechanism for deleting early test messages

• Merge a supported way to use Zulip in Docker in production implementation.

4.5 Internationalization

The core Zulip UI has been mostly translated into 5 languages; however, more work is required to make those transla-tions actually displayed in the Zulip UI for the users who would benefit from them.

• Merge support for using translations in Django templates

• Add text in handlebars templates to translatable string database

• Merge support for translating text in handlebars

• Add text in error messages to translatable strings

4.6 User Experience at scale

There are a few parts of the Zulip UI which could benefit from overhauls designed around making the user experiencenice for large teams.

• Make the buddy list work better for large teams

• Improve @-mentioning syntax based on stronger unique identifiers

• Show subscriber counts on streams

16 Chapter 4. Zulip 2016 Roadmap

Zulip Documentation, Release 1.4.0

• Make the streams page easier to navigate with 100s of streams

• Add support for filtering long lists of streams

4.7 Administration and management

Currently, Zulip has a number of administration features that can be controlled only via the command line.

• Make default streams web-configurable

• Make realm emoji web-configurable

• Make realm filters web-configurable

• Make realm aliases web-configurable

• Enhance the LDAP integration and make it web-configurable

• Add a SAML integration for Zulip

• Improve administrative controls for managing streams

4.8 Scalability

Zulip should support 10000 users in a realm and also support smaller realms in more resource-constrained environ-ments (probably a good initial goal is working well with only 2GB of RAM).

• Make the Zulip Tornado service support horizontal scaling

• Make presence system scale well to 10000 users in a realm.

• Support running queue workers multithreaded in production to decrease minimum memory footprint

4.9 Performance

Performance is essential for a communication tool. While some things are already quite good (e.g. narrowing andmessage sending is speedy), this is an area where one can always improve. There are a few known performanceopportunities:

• Migrate to faster jinja2 templating engine

• Don’t load zxcvbn when it isn’t needed

• Optimize the frontend performance of loading the Zulip webapp using profiling

4.10 Technology improvements

Zulip should be making use of the best Python/Django tools available.

• Add support for Zulip running on Python 3

• Add support for changing users’ email addresses

• Automatic thumbnailing of uploaded images

4.7. Administration and management 17

Zulip Documentation, Release 1.4.0

• Upgrade Zulip to use Django 1.10 once it is released. The patches needed to run Zulip were merged intomainline Django in Django 1.10, so this will mean we don’t need to use a fork of Django anymore.

4.11 Technical Debt

While the Zulip server has a great codebase compared to most projects of its size, it takes work to keep it that way.

• Migrate most web routes to REST API

• Finish purging global variables from the Zulip JavaScript

• Finish deprecating and remove the pre-REST Zulip /send_message API

• Split Tornado subsystem into a separate Django app

• Clean up clutter in the root of the zulip.git repository

• Refactor zulip.css to be broken into components

4.12 Deployment and upgrade process

• Support backwards-incompatible upgrades to Python libraries

• Minimize the downtime required in the Zulip upgrade process

4.13 Security

• Add support for 2-factor authentication on all platforms

• Add a retention policy feature that automatically deletes old messages

• Upgrade every Zulip dependency to a modern version

• The LOCAL_UPLOADS_DIR file uploads backend only supports world-readable uploads

• Add support for stronger security controls for uploaded files

4.14 Testing

• Extend Zulip’s automated test coverage to include all API endpoints

• Build automated tests for the client API bindings

• Add Python static type-checking to Zulip using mypy

• Improve the runtime of Zulip’s backend test suite

• Use caching to make Travis CI runtimes faster

• Add automated tests for the production upgrade process

• Improve Travis CI “production” test suite to catch more regressions

18 Chapter 4. Zulip 2016 Roadmap

Zulip Documentation, Release 1.4.0

4.15 Development environment

• Migrate from jslint to eslint

• Figure out a nice upgrade process for Zulip Vagrant VMs

• Replace closure-compiler with a faster minifier toolchain

• Add support for building frontend features in React

• Use a JavaScript bundler like webpack

4.16 Documentation

• Significantly expand documentation of the Zulip API and integrating with Zulip.

• Expand library of documentation on Zulip’s feature set. Currently most documentation is for either developersor system administrators.

• Expand developer documentation with more tutorials explaining how to do various types of projects.

• Overhaul new contributor documentation, especially on coding style, to better highlight and teach the importantpieces.

• Update all screenshots to show the current Zulip UI

4.17 Integrations

Integrations are essential to Zulip. While we currently have a reasonably good framework for writing new webhookintegrations for getting notifications into Zulip, it’d be great to streamline that process and make bots that receivemessages just as easy to build.

• Make it super easy to take screenshots for new webhook integrations

• Add an outgoing webhook integration system

• Build a framework to cut duplicated code in new webhook integrations

• Make setting up a new integration a smooth flow

• Optimize the integration writing documentation to make writing new ones really easy.

4.18 Android app

The Zulip Android app is ahead of the iOS app in terms of feature set, but there is still a lot of work to do. Most of thethings listed below will eventually apply to the iOS app as well.

• Support using a non-zulip.com server

• Support Google authentication with a non-Zulip.com server

• Add support for narrowing to @-mentions

• Support having multiple Zulip realms open simultaneously

• Build a slick development login page to simplify testing (similar to the development homepage on web)

• Improve the compose box to let you see what you’re replying to

4.15. Development environment 19

Zulip Documentation, Release 1.4.0

• Make it easy to compose messages with mentions, emoji, etc.

• Display unread counts and improve navigation

• Hide messages sent to muted topics

• Fill out documentation to make it easy to get started

4.19 iOS app

Most of the projects listed under Android apply here as well, but it’s worth highlighting some areas where iOS issubstantially behind Android. The top priority here is recruiting a lead developer for the iOS app. Once we have thatresolved, we’ll expand our ambitions for the app with more specific improvements.

• iOS app needs maintainer

• APNS notifications are broken

4.20 Desktop apps

The top goal for the desktop apps is to rebuild it in a modern toolchain so that it’s easy for a wide range of developers tocontribute to the apps. The new cross-platform app is implemented in Electron, a framework (maintained by GitHub)that uses Chromium and Node.js, so Zulip developers only need to write HTML, CSS, and JavaScript. The new Zulipapp is in alpha as of early August 2016.

• Migrate platform from QT/webkit to Electron

• Desktop app doesn’t recover well from entering the wrong Zulip server

• Support having multiple Zulip realms open simultaneously

• Build an efficient process for testing and releasing new versions of the desktop apps

4.21 Community

These don’t get GitHub issues since they’re not technical projects, but they are important goals for the project.

• Setup a Zulip server for the Zulip development community

• Expand the number of core developers able to do code reviews

• Expand the number of contributors regularly adding features to Zulip

• Have a successful summer with Zulip’s 3 GSOC students

20 Chapter 4. Zulip 2016 Roadmap

CHAPTER 5

Version History

All notable changes to the Zulip server are documented in this file.

5.1 Unreleased

5.2 1.4 - 2016-08-25

• Migrated Zulip’s python dependencies to be installed via a virtualenv, instead of the via apt. This is a majorchange to how Zulip is installed that we expect will simplify upgrades in the future.

• Fixed unnecessary loading of zxcvbn password strength checker. This saves a huge fraction of the uncachednetwork transfer for loading Zulip.

• Added support for using Ubuntu Xenial in production.

• Added a powerful and complete realm import/export tool.

• Added nice UI for selecting a default language to display settings.

• Added UI for searching streams in left sidebar with hotkeys.

• Added Semaphore, Bitbucket, and HelloWorld (example) integrations.

• Added new webhook-based integration for Trello.

• Added management command for creating realms through web UI.

• Added management command to send password reset emails.

• Added endpoint for mobile apps to query available auth backends.

• Added LetsEncrypt documentation for getting SSL certificates.

• Added nice rendering of unicode emoji.

• Added support for pinning streams to the top of the left sidebar.

• Added search box for filtering user list when creating a new stream.

• Added realm setting to disable message editing.

• Added realm setting to time-limit message editing. Default is 10m.

• Added realm setting for default language.

• Added year to timestamps in message interstitials for old messages.

21

Zulip Documentation, Release 1.4.0

• Added GitHub authentication (and integrated python-social-auth, so it’s easy to add additional social authenti-cation methods).

• Added TERMS_OF_SERVICE setting using markdown formatting to configure the terms of service for a Zulipserver.

• Added numerous hooks to puppet modules to enable more configurations.

• Moved several useful puppet components into the main puppet manifests (setting a redis password, etc.).

• Added automatic configuration of postgres/memcached settings based on the server’s available RAM.

• Added scripts/upgrade-zulip-from-git for upgrading Zulip from a Git repo.

• Added preliminary support for Python 3. All of Zulip’s test suites now pass using Python 3.4.

• Added support for Name <email@example.com> format when inviting users.

• Added numerous special-purpose settings options.

• Added a hex input field in color picker.

• Documented new Electron beta app and mobile apps in the /apps/ page.

• Enabled Android Google authentication support.

• Enhanced logic for tracking origin of user uploads.

• Improved error messages for various empty narrows.

• Improved missed message emails to better support directly replying.

• Increased backend test coverage of Python code to 85.5%.

• Increased mypy static type coverage of Python code to 95%. Also fixed many string annotations to properlyhandle unicode.

• Fixed major i18n-related frontend performance regression on /#subscriptions page. Saves several seconds ofload time with 1k streams.

• Fixed Jinja2 migration bug when trying to register an email that already has an account.

• Fixed narrowing to a stream from other pages.

• Fixed various frontend strings that weren’t marked for translation.

• Fixed several bugs around editing status (/me) messages.

• Fixed queue workers not restarting after changes in development.

• Fixed Casper tests hanging while development server is running.

• Fixed browser autocomplete issue when adding new stream members.

• Fixed broken create_stream and rename_stream management commands.

• Fixed zulip-puppet-apply exit code when puppet throws errors.

• Fixed EPMD restart being attempted on every puppet apply.

• Fixed message cache filling; should improve perf after server restart.

• Fixed caching race condition when changing user objects.

• Fixed buggy puppet configuration for supervisord restarts.

• Fixed some error handling race conditions when editing messages.

• Fixed fastcgi_params to protect against the httpoxy attack.

22 Chapter 5. Version History

Zulip Documentation, Release 1.4.0

• Fixed bug preventing users with mit.edu emails from registering accounts.

• Fixed incorrect settings docs for the email mirror.

• Fixed APNS push notification support (had been broken by Apple changing the APNS API).

• Fixed some logic bugs in how attachments are tracked.

• Fixed unnecessarily resource-intensive rabbitmq cron checks.

• Fixed old deployment directories leaking indefinitely.

• Fixed need to manually add localhost in ALLOWED_HOSTS.

• Fixed display positioning for the color picker on subscriptions page.

• Fixed escaping of Zulip extensions to markdown.

• Fixed requiring a reload to see newly uploaded avatars.

• Fixed @all warning firing even for @all.

• Restyled password reset form to look nice.

• Improved formatting in reset password links.

• Improved alert words UI to match style of other settings.

• Improved error experience when sending to nonexistent users.

• Portions of integrations documentation are now automatically generated.

• Restructured the URLs files to be more readable.

• Upgraded almost all Python dependencies to current versions.

• Substantially expanded and reorganized developer documentation.

• Reorganized production documentation and moved to ReadTheDocs.

• Reorganized .gitignore type files to be written under var/

• Refactored substantial portions of templates to support subdomains.

• Renamed local_settings.py symlink to prod_settings.py for clarity.

• Renamed email-mirror management command to email_mirror.

• Changed HTTP verb for create_user_backend to PUT.

• Eliminated all remaining settings hardcoded for zulip.com.

• Eliminated essentially all remaining hardcoding of mit.edu.

• Optimized the performance of all the test suites.

• Optimized Django memcached configuration.

• Removed old prototype data export tool.

• Disabled insecure RC4 cipher in nginx configuration.

• Enabled shared SSL session cache in nginx configuration.

• Updated header for Zulip static assets to reflect Zulip being open source.

5.2. 1.4 - 2016-08-25 23

Zulip Documentation, Release 1.4.0

5.3 1.3.13 - 2016-06-21

• Added nearly complete internationalization of the Zulip UI.

• Added warning when using @all/@everyone.

• Added button offering to subscribe at bottom of narrows to streams the user is not subscribed to.

• Added integrations with Airbrake, CircleCI, Crashlytics, IFTTT, Transifex, and Updown.io.

• Added menu option to mark all messages in a stream or topic as read.

• Added new Attachment model to keep track of uploaded files.

• Added caching of virtualenvs in development.

• Added mypy static type annotations to about 85% of the Zulip Python codebase.

• Added automated test of backend templates to test for regressions.

• Added lots of detailed documentation on the Zulip development environment.

• Added setting allowing only administrators to create new streams.

• Added button to exit the Zulip tutorial early.

• Added web UI for configuring default streams.

• Added new OPEN_REALM_CREATION setting (default off), providing a UI for creating additional realms ona Zulip server.

• Fixed email_gateway_password secret not working properly.

• Fixed missing helper scripts for RabbitMQ Nagios plugins.

• Fixed skipping forward to latest messages (“More messages below” button).

• Fixed netcat issue causing Zulip installation to hang on Scaleway machines.

• Fixed rendering of /me status messages after message editing.

• Fixed case sensitivity of right sidebar fading when compose is open.

• Fixed error messages when composing to invalid PM recipients.

• Fixed LDAP auth backend not working with Zulip mobile apps.

• Fixed erroneous WWW-Authenticate headers with expired sessions.

• Changed “coworkers” to “users” in the Zulip UI.

• Changed add_default_stream REST API to correctly use PUT rather than PATCH.

• Updated the Zulip emoji set (the Android Emoji) to a modern version.

• Made numerous small improvements to the Zulip development experience.

• Migrated backend templates to the faster Jinja2 templating system.

• Migrated development environment setup scripts to tools/setup/.

• Expanded test coverage for several areas of the product.

• Simplified the API for writing new webhook integrations.

• Removed most of the remaining JavaScript global variables.

24 Chapter 5. Version History

Zulip Documentation, Release 1.4.0

5.4 1.3.12 - 2016-05-10

• CVE-2016-4426: Bot API keys were accessible to other users in the same realm.

• CVE-2016-4427: Deactivated users could access messages if SSO was enabled.

• Fixed a RabbitMQ configuration bug that resulted in reordered messages.

• Added expansive test suite for authentication backends and decorators.

• Added an option to logout_all_users to delete only sessions for deactivated users.

5.5 1.3.11 - 2016-05-02

• Moved email digest support into the default Zulip production configuration.

• Added options for configuring Postgres, RabbitMQ, Redis, and memcached in settings.py.

• Added documentation on using Hubot to integrate with useful services not yet integrated with Zulip directly(e.g. Google Hangouts).

• Added new management command to test sending email from Zulip.

• Added Codeship, Pingdom, Taiga, Teamcity, and Yo integrations.

• Added Nagios plugins to the main distribution.

• Added ability for realm administrators to manage custom emoji.

• Added guide to writing new integrations.

• Enabled camo image proxy to fix mixed-content warnings for http images.

• Refactored the Zulip puppet modules to be more modular.

• Refactored the Tornado event system, fixing old memory leaks.

• Removed many old-style /json API endpoints

• Implemented running queue processors multithreaded in development, decreasing RAM requirements for aZulip development environment from ~1GB to ~300MB.

• Fixed rerendering the complete buddy list whenever a user came back from idle, which was a significant perfor-mance issue in larger realms.

• Fixed the disabling of desktop notifications from 1.3.7 for new users.

• Fixed the (admin) create_user API enforcing restricted_to_domain, even if that setting was disabled for therealm.

• Fixed bugs changing certain settings in administration pages.

• Fixed collapsing messages in narrowed views.

• Fixed 500 errors when uploading a non-image file as an avatar.

• Fixed Jira integration incorrectly not @-mentioning assignee.

5.4. 1.3.12 - 2016-05-10 25

Zulip Documentation, Release 1.4.0

5.6 1.3.10 - 2016-01-21

• Added new integration for Travis CI.

• Added settings option to control maximum file upload size.

• Added support for running Zulip development environment in Docker.

• Added easy configuration support for a remote postgres database.

• Added extensive documentation on scalability, backups, and security.

• Recent private message threads are now displayed expanded similar to the pre-existing recent topics feature.

• Made it possible to set LDAP and EMAIL_HOST passwords in /etc/zulip/secrets.conf.

• Improved the styling for the Administration page and added tabs.

• Substantially improved loading performance on slow networks by enabling GZIP compression on more assets.

• Changed the page title in narrowed views to include the current narrow.

• Fixed several backend performance issues affecting very large realms.

• Fixed bugs where draft compose content might be lost when reloading site.

• Fixed support for disabling the “zulip” notifications stream.

• Fixed missing step in postfix_localmail installation instructions.

• Fixed several bugs/inconveniences in the production upgrade process.

• Fixed realm restrictions for servers with a unique, open realm.

• Substantially cleaned up console logging from run-dev.py.

5.7 1.3.9 - 2015-11-16

• Fixed buggy #! lines in upgrade scripts.

5.8 1.3.8 - 2015-11-15

• Added options to the Python api for working with untrusted server certificates.

• Added a lot of documentation on the development environment and testing.

• Added partial support for translating the Zulip UI.

• Migrated installing Node dependencies to use npm.

• Fixed LDAP integration breaking autocomplete of @-mentions.

• Fixed admin panel reactivation/deactivation of bots.

• Fixed inaccurate documentation for downloading the desktop apps.

• Fixed various minor bugs in production installation process.

• Fixed security issue where recent history on private streams might be visible to new users (to the Zulip team) whowere invited with that private stream as one of their initial streams (https://github.com/zulip/zulip/issues/230).

• Major preliminary progress towards supporting Python 3.

26 Chapter 5. Version History

Zulip Documentation, Release 1.4.0

5.9 1.3.7 - 2015-10-19

• Turn off desktop and audible notifications for streams by default.

• Added support for the LDAP authentication integration creating new users.

• Added new endpoint to support Google auth on mobile.

• Fixed desktop notifications in modern Firefox.

• Fixed several installation issues for both production and development environments.

• Improved documentation for outgoing SMTP and the email mirror integration.

5.9. 1.3.7 - 2015-10-19 27

Zulip Documentation, Release 1.4.0

28 Chapter 5. Version History

CHAPTER 6

Requirements

Note that if you just want to play around with Zulip and see what it looks like, it is easier to install it in a developmentenvironment following these instructions, since then you don’t need to worry about setting up SSL certificates and anauthentication mechanism. Or, you can check out the developers’ chatroom (a public, running Zulip instance).

6.1 Server

6.1.1 Hardware Specifications

• CPU and Memory: For installations with 100+ users you’ll need a minimum of 2 CPUs and 4GB RAM.For installations with fewer users, 1 CPU and 2GB RAM might be sufficient. We strong recommend againstinstalling with less than 2GB of RAM, as you will likely experience out of memory issues.

• Disk space: You’ll need at least 10GB of free disk space. If you intend to store uploaded files locally rather thanon S3 you will likely need more.

6.1.2 Network and Security Specifications

• Outgoing HTTP(S) access to the public Internet. If you want to be able to send email from Zulip, you’ll alsoneed SMTP access.

6.1.3 Operating System

Ubuntu 14.04 Trusty and Ubuntu 16.04 Xenial are supported for running Zulip in production. 64-bit is recommended.

6.1.4 Domain name

You should already have a domain name available for your Zulip production instance. In order to generate valid SSLcertificates with Let’s Encrypt, and to enable other services such as Google Authentication, you’ll need to update thedomains A record to point to your production server.

29

Zulip Documentation, Release 1.4.0

6.2 Credentials needed

6.2.1 SSL Certificate

• SSL Certificate for the host you’re putting this on (e.g. zulip.example.com). The installation instructions containdocumentation for how to get an SSL certificate for free using LetsEncrypt.

6.2.2 Outgoing email

• Email credentials Zulip can use to send outgoing emails to users (e.g. email address confirmation emails duringthe signup process, missed message notifications, password reminders if you’re not using SSO, etc.).

Once you have met these requirements, see full instructions for installing Zulip in production.

30 Chapter 6. Requirements

CHAPTER 7

Installation

Ensure you have an Ubuntu system that satisfies the installation requirements. In short, you should have an Ubuntu14.04 Trusty or Ubuntu 16.04 Xenial 64-bit server instance, with at least 4GB RAM, 2 CPUs, and 10 GB disk space.You should also have a domain name available and have updated its DNS record to point to the server.

7.1 Step 0: Subscribe

Please subscribe to low-traffic the Zulip announcements Google Group to get announcements about new releases,security issues, etc.

7.2 Step 1: Install SSL Certificates

Zulip runs over https only and requires ssl certificates in order to work. It looks for the certificates in/etc/ssl/private/zulip.key and /etc/ssl/certs/zulip.combined-chain.crt. Note thatZulip uses nginx as its webserver and thus expects a chained certificate bundle

If you need an SSL certificate, see our SSL certificate documentation. If you already have an SSL certificate, justinstall (or symlink) them into place at the above paths, and move on to the next step.

7.3 Step 2: Download and install latest release

If you haven’t already, download and unpack the latest built server tarball with the following commands:

sudo -i # If not already rootwget https://www.zulip.com/dist/releases/zulip-server-latest.tar.gzrm -rf /root/zulip && mkdir /root/zuliptar -xf zulip-server-latest.tar.gz --directory=/root/zulip --strip-components=1

Then, run the Zulip install script:

/root/zulip/scripts/setup/install

This may take a while to run, since it will install a large number of dependencies.

The Zulip install script is designed to be idempotent, so if it fails, you can just rerun it after correcting the issue thatcaused it to fail. Also note that it automatically logs a transcript to /var/log/zulip/install.log; pleaseinclude a copy of that file in any bug reports.

31

Zulip Documentation, Release 1.4.0

7.4 Step 3: Configure Zulip

Configure the Zulip server instance by editing /etc/zulip/settings.py and providing values for the manda-tory settings, which are all found under the heading ### MANDATORY SETTINGS.

These settings include:

• EXTERNAL_HOST: the user-accessible Zulip domain name for your Zulip installation. This will be the domainfor which you have DNS A records pointing to this server and for which you configured SSL certificates.

• ZULIP_ADMINISTRATOR: the email address of the person or team maintaining this installation and who willget support emails.

• AUTHENTICATION_BACKENDS: a list of enabled authentication mechanisms. You’ll need to enable at leastone authentication mechanism by uncommenting its corresponding line, and then also do any additional config-uration required for that backend as documented in the settings.py file. See the section on Authenticationfor more detail on the available authentication backends and how to configure them.

• EMAIL_*, DEFAULT_FROM_EMAIL, and NOREPLY_EMAIL_ADDRESS: Regardless of which authentica-tion backends you enable, you must provide settings for an outgoing SMTP server so Zulip can send emailswhen needed. We highly recommend testing your configuration using manage.py send_test_email toconfirm your outgoing email configuration is working correctly.

• ALLOWED_HOSTS: Replace * with the fully qualified DNS name for your Zulip server here.

7.5 Step 4: Initialize Zulip database

At this point, you are done doing things as root. To initialize the Zulip database for your production install, run:

su zulip -c /home/zulip/deployments/current/scripts/setup/initialize-database

The initialize-database script will report an error if you did not fill in all the mandatory settings from/etc/zulip/settings.py. It is safe to rerun it after correcting the problem if that happens.

This completes the process of installing Zulip on your server. However, in order to use Zulip, you’ll need to create anorganization in your Zulip installation.

7.6 Step 5: Create a Zulip organization and login

• If you haven’t already, verify that your server can send email using ./manage.py send_test_emailusername@example.com. You’ll need working outgoing email to complete the setup process.

• Run the organization (realm) creation management command :

su zulip # If you weren't already the zulip usercd /home/zulip/deployments/current./manage.py generate_realm_creation_link

This will print out a secure 1-time use link that allows creation of a new Zulip organization onyour server. For most servers, you will only ever do this once, but you can run manage.pygenerate_realm_creation_link again if you want to host another organization on your Zulip server.

• Open the link generated with your web browser. You’ll see the create organization page (screenshot here). Enteryour email address and click Create organization.

32 Chapter 7. Installation

Zulip Documentation, Release 1.4.0

• Check your email to find the confirmation email and click the link. You’ll be prompted to finish setting up yourorganization and initial administrator user (screenshot here). Complete this form and log in!

Congratulations! You are logged in as an organization administrator for your new Zulip organization. After gettingoriented, we recommend visiting the special “Administration” tab linked to from the upper-right gear menu in theZulip app to configure important policy settings like how users can join your new organization. By default, yourorganization will be configured as follows (screenshot here):

• restricted_to_domain=True: Only people with emails with the same ending as yours can join.

• invite_required=False: An invitation is not required to join the realm.

• invite_by_admin_only=False: You don’t need to be an admin user to invite other users.

Next, you’ll likely want to do one of the following:

• Customize your Zulip organization.

• Learn about managing a production Zulip server.

7.7 Troubleshooting

If you get an error after scripts/setup/install completes, check /var/log/zulip/errors.log for atraceback, and consult the troubleshooting section for advice on how to debug. If that doesn’t help, please visit the“installation help” stream in the Zulip developers’ chat for realtime help or email zulip-help@googlegroups.com withthe traceback and we’ll try to help you out!

7.7. Troubleshooting 33

Zulip Documentation, Release 1.4.0

34 Chapter 7. Installation

CHAPTER 8

Troubleshooting

Zulip uses Supervisor to monitor and control its many Python services. Read the next section, Using supervisorctl, tolearn how to use the Supervisor client to monitor and manage services.

If you haven’t already, now might be a good time to read Zulip’s architectural overview, particularly the Componentssection. This will help you understand the many services Zulip uses.

If you encounter issues while running Zulip, take a look at Zulip’s logs, which are located in /var/log/zulip/.That directory contains one log file for each service, plus errors.log (has all errors), server.log (has logsfrom the Django and Tornado servers), and workers.log (has combined logs from the queue workers).

The section troubleshooting services on this page includes details about how to fix common issues with Zulip services.

If you run into additional problems, please report them so that we can update this page! The Zulip installation scriptslogs its full output to /var/log/zulip/install.log, so please include the context for any tracebacks fromthat log.

8.1 Using supervisorctl

To see what Zulip-related services are configured to use Supervisor, look at/etc/supervisor/conf.d/zulip.conf and /etc/supervisor/conf.d/zulip-db.conf.

Use the supervisor client supervisorctl to list the status of, stop, start, and restart various services.

8.1.1 Checking status with supervisorctl status

You can check if the zulip application is running using:

supervisorctl status

When everything is running as expected, you will see something like this:

process-fts-updates RUNNING pid 2194,→˓uptime 1:13:11zulip-django RUNNING pid 2192,→˓uptime 1:13:11zulip-senders:zulip-events-message_sender-0 RUNNING pid 2209,→˓uptime 1:13:11zulip-senders:zulip-events-message_sender-1 RUNNING pid 2210,→˓uptime 1:13:11zulip-senders:zulip-events-message_sender-2 RUNNING pid 2211,→˓uptime 1:13:11

35

Zulip Documentation, Release 1.4.0

zulip-senders:zulip-events-message_sender-3 RUNNING pid 2212,→˓uptime 1:13:11zulip-senders:zulip-events-message_sender-4 RUNNING pid 2208,→˓uptime 1:13:11zulip-tornado RUNNING pid 2193,→˓uptime 1:13:11zulip-workers:zulip-deliver-enqueued-emails STARTINGzulip-workers:zulip-events-confirmation-emails RUNNING pid 2199,→˓uptime 1:13:11zulip-workers:zulip-events-digest_emails RUNNING pid 2205,→˓uptime 1:13:11zulip-workers:zulip-events-email_mirror RUNNING pid 2203,→˓uptime 1:13:11zulip-workers:zulip-events-error_reports RUNNING pid 2200,→˓uptime 1:13:11zulip-workers:zulip-events-feedback_messages RUNNING pid 2207,→˓uptime 1:13:11zulip-workers:zulip-events-missedmessage_mobile_notifications RUNNING pid 2204,→˓uptime 1:13:11zulip-workers:zulip-events-missedmessage_reminders RUNNING pid 2206,→˓uptime 1:13:11zulip-workers:zulip-events-signups RUNNING pid 2198,→˓uptime 1:13:11zulip-workers:zulip-events-slowqueries RUNNING pid 2202,→˓uptime 1:13:11zulip-workers:zulip-events-user-activity RUNNING pid 2197,→˓uptime 1:13:11zulip-workers:zulip-events-user-activity-interval RUNNING pid 2196,→˓uptime 1:13:11zulip-workers:zulip-events-user-presence RUNNING pid 2195,→˓uptime 1:13:11

8.1.2 Restarting services with supervisorctl restart all

After you change configuration in /etc/zulip/settings.py or fix a misconfiguration, you will often want torestart the Zulip application. You can restart Zulip using:

supervisorctl restart all

8.1.3 Stopping services with supervisorctl stop all

Similarly, you can stop Zulip using:

supervisorctl stop all

8.2 Troubleshooting services

The Zulip application uses several major open source services to store and cache data, queue messages, and otherwisesupport the Zulip application:

• postgresql

• rabbitmq-server

36 Chapter 8. Troubleshooting

Zulip Documentation, Release 1.4.0

• nginx

• redis

• memcached

If one of these services is not installed or functioning correctly, Zulip will not work. Below we detail some commonconfiguration problems and how to resolve them:

• An AMQPConnectionError traceback or error running rabbitmqctl usually means that RabbitMQ is not running;to fix this, try:

service rabbitmq-server restart

If RabbitMQ fails to start, the problem is often that you are using a virtual machine with broken DNS configu-ration; you can often correct this by configuring /etc/hosts properly.

• If your browser reports no webserver is running, that is likely because nginx is not configured properly and thusfailed to start. nginx will fail to start if you configured SSL incorrectly or did not provide SSL certificates. Tofix this, configure them properly and then run:

service nginx restart

Next: Making your Zulip instance awesome.

8.2. Troubleshooting services 37

Zulip Documentation, Release 1.4.0

38 Chapter 8. Troubleshooting

CHAPTER 9

Customize Zulip

Once you’ve got Zulip setup, you’ll likely want to configure it the way you like. There are four big things to focus on:

1. Integrations

2. Streams and Topics

3. Notification settings

4. Mobile and desktop apps

Lastly, read about Zulip’s other great features, and then enjoy your Zulip installation!

9.1 Integrations

We recommend setting up integrations for the major tools that your team works with. For example, if you’re asoftware development team, you may want to start with integrations for your version control, issue tracker, CI system,and monitoring tools.

Spend time configuring these integrations to be how you like them – if an integration is spammy, you may want tochange it to not send messages that nobody cares about (E.g. for the zulip.com trac integration, some teams find theyonly want notifications when new tickets are opened, commented on, or closed, and not every time someone edits themetadata).

If Zulip doesn’t have an integration you want, you can add your own! Most integrations are very easy to write, andeven more complex integrations usually take less than a day’s work to build. We very much appreciate contributionsof new integrations; see the brief integration writing guide.

It can often be valuable to integrate your own internal processes to send notifications into Zulip; e.g. notifications ofnew customer signups, new error reports, or daily reports on the team’s key metrics; this can often spawn discussionsin response to the data.

9.2 Streams and Topics

If it feels like a stream has too much traffic about a topic only of interest to some of the subscribers, consider addingor renaming streams until you feel like your team is working productively.

Second, most users are not used to topics. It can require a bit of time for everyone to get used to topics and startbenefitting from them, but usually once a team is using them well, everyone ends up enthusiastic about how muchtopics make life easier. Some tips on using topics:

39

Zulip Documentation, Release 1.4.0

• When replying to an existing conversation thread, just click on the message, or navigate to it with the arrow keysand hit “r” or “enter” to reply on the same topic

• When you start a new conversation topic, even if it’s related to the previous conversation, type a new topic inthe compose box

• You can edit topics to fix a thread that’s already been started, which can be helpful when onboarding new batchesof users to the platform.

Third, setting default streams for new users is a great way to get new users involved in conversations before they’veaccustomed themselves with joining streams on their own. You can use the set_default_streams command toset default streams for users within a realm:

python manage.py set_default_streams --domain=example.com --streams=foo,bar,...

9.3 Notification settings

Zulip gives you a great deal of control over which messages trigger desktop notifications; you can configure theseextensively in the /#settings page (get there from the gear menu). If you find the desktop notifications annoying,consider changing the settings to only trigger desktop notifications when you receive a PM or are @-mentioned.

9.4 Mobile and desktop apps

Currently, the Zulip Desktop app only supports talking to servers with a properly signed SSL certificate, so you mayfind that you get a blank screen when you connect to a Zulip server using a self-signed certificate.

The Zulip Android app in the Google Play store doesn’t yet support talking to non-zulip.com servers (and the iOS onedoesn’t support Google auth SSO against non-zulip.com servers; there’s a design for how to fix that which wouldn’tbe a ton of work to implement). If you are interested in helping out with the Zulip mobile apps, shoot an email tozulip-devel@googlegroups.com and the maintainers can guide you on how to help.

For announcements about improvements to the apps, make sure to join the zulip-announce@googlegroups.com list sothat you can receive the announcements when these become available.

9.5 All other features

Hotkeys, emoji, search filters, @-mentions, etc. Zulip has lots of great features, make sure your team knows they existand how to use them effectively.

9.6 Enjoy your Zulip installation!

If you discover things that you wish had been documented, please contribute documentation suggestions either via aGitHub issue or pull request; we love even small contributions, and we’d love to make the Zulip documentation covereverything anyone might want to know about running Zulip in production.

Next: Maintaining and upgrading Zulip in production.

40 Chapter 9. Customize Zulip

CHAPTER 10

Secure, maintain, and upgrade

This page covers topics that will help you maintain a healthy, up-to-date, and secure Zulip installation, including:

• Upgrading

• Upgrading from a git repository

• Backups

• Monitoring

• Scalability

• Security Model

• Management commands

10.1 Upgrading

We recommend reading this entire section before doing your first upgrade.

To upgrade to a new version of the zulip server, download the appropriate release tarball from https://www.zulip.com/dist/releases/

You also have the option of creating your own release tarballs from a copy of zulip.git repository usingtools/build-release-tarball. And, starting with Zulip version 1.4, you can upgrade Zulip to a versionin a Git repository directly.

Next, run as root:

/home/zulip/deployments/current/scripts/upgrade-zulip zulip-server-VERSION.tar.gz

The upgrade process will shut down the Zulip service and then run apt-get upgrade, a puppet apply, any databasemigrations, and then bring the Zulip service back up. Upgrading will result in some brief downtime for the service,which should be under 30 seconds unless there is an expensive transition involved. Unless you have tested the upgradein advance, we recommend doing upgrades at off hours.

Note that upgrading an existing Zulip production server from Ubuntu 14.04 Trusty to Ubuntu 16.04 Xenial will requiresignificant manual intervention on your part to migrate the data in the database from Postgres 9.3 to Postgres 9.5.Contributions on testing and documenting this process are welcome!

41

Zulip Documentation, Release 1.4.0

10.1.1 Preserving local changes to configuration files

Warning: If you have modified configuration files installed by Zulip (e.g. the nginx configuration), the Zulip upgradeprocess will overwrite your configuration when it does the puppet apply.

You can test whether this will happen assuming no upstream changes to the configuration usingscripts/zulip-puppet-apply (without the -f option), which will do a test puppet run and output and changesit would make. Using this list, you can save a copy of any files that you’ve modified, do the upgrade, and then restoreyour configuration.

If you need to do this, please report the issue so that we can make the Zulip puppet configuration flexible enough tohandle your setup.

10.1.2 Troubleshooting with the upgrade log

The Zulip upgrade script automatically logs output to /var/log/zulip/upgrade.log. Please use those logs toinclude output that shows all errors in any bug reports.

After the upgrade, we recommend checking /var/log/zulip/errors.log to confirm that your users are notexperiencing errors after the upgrade.

10.1.3 Rolling back to a prior version

The Zulip upgrade process works by creating a new deployment under /home/zulip/deployments/containing a complete copy of the Zulip server code, and then moving the symlinks at/home/zulip/deployments/current and /root/zulip as part of the upgrade process.

This means that if the new version isn’t working, you can quickly downgrade to the old version by using/home/zulip/deployments/<date>/scripts/restart-server to return to a previous version thatyou’ve deployed (the version is specified via the path to the copy of restart-server you call).

10.1.4 Updating settings

If required, you can update your settings by editing /etc/zulip/settings.py and then run/home/zulip/deployments/current/scripts/restart-server to restart the server.

10.1.5 Applying Ubuntu system updates

While the Zulip upgrade script runs apt-get upgrade, you are responsible for running this on your system on aregular basis between Zulip upgrades to ensure that it is up to date with the latest security patches.

10.1.6 API and your Zulip URL

To use the Zulip API with your Zulip server, you will need to use the API endpoint of e.g.https://zulip.example.com/api. Our Python API example scripts support this via the--site=https://zulip.example.com argument. The API bindings support it via puttingsite=https://zulip.example.com in your .zuliprc.

Every Zulip integration supports this sort of argument (or e.g. a ZULIP_SITE variable in a zuliprc file or the en-vironment), but this is not yet documented for some of the integrations (the included integration documentation on/integrations will properly document how to do this for most integrations). We welcome pull requests forintegrations that don’t discuss this!

42 Chapter 10. Secure, maintain, and upgrade

Zulip Documentation, Release 1.4.0

Similarly, you will need to instruct your users to specify the URL for your Zulip server when using the Zulip desktopand mobile apps.

10.1.7 Memory leak mitigation

As a measure to mitigate the impact of potential memory leaks in one of the Zulip daemons, the service automaticallyrestarts itself every Sunday early morning. See /etc/cron.d/restart-zulip for the precise configuration.

10.2 Upgrading from a git repository

Starting with version 1.4, the Zulip server supports doing deployments from a Git repository. To configurethis, you will need to add zulip::static_asset_compiler to your /etc/zulip/zulip.conf file’spuppet_classes entry, like this:

puppet_classes = zulip::voyager, zulip::static_asset_compiler

Then, run scripts/zulip-puppet-apply to install the dependencies for building Zulip’s static assets. You canconfigure the git repository that you’d like to use by adding a section like this to /etc/zulip/zulip.conf; bydefault it uses the main zulip repository (shown below).

[deployment]git_repo_url = https://github.com/zulip/zulip.git

Once that is done (and assuming the currently installed version of Zulip is new enough that this script exists), you cando deployments by running as root:

/home/zulip/deployments/current/scripts/upgrade-zulip-from-git <branch>

and Zulip will automatically fetch the relevant branch from the specified repository, build the static assets, and deploythat version. Currently, the upgrade process is slow, but it doesn’t need to be; there is ongoing work on optimizing it.

10.3 Backups

There are several pieces of data that you might want to back up:

• The postgres database. That you can back up like any postgres database; wehave some example tooling for doing that incrementally into S3 using wal-e inpuppet/zulip_internal/manifests/postgres_common.pp (that’s what we use for zulip.com’sdatabase backups). Note that this module isn’t part of the Zulip server releases since it’s part of the zulip.comconfiguration (see https://github.com/zulip/zulip/issues/293 for a ticket about fixing this to make life easier forrunning backups).

• Any user-uploaded files. If you’re using S3 as storage for file uploads, this is backed up in S3, but if youhave instead set LOCAL_UPLOADS_DIR, any files uploaded by users (including avatars) will be stored in thatdirectory and you’ll want to back it up.

• Your Zulip configuration including secrets from /etc/zulip/. E.g. if you lose the value of secret_key, all userswill need to login again when you setup a replacement server since you won’t be able to verify their cookies; ifyou lose avatar_salt, any user-uploaded avatars will need to be re-uploaded (since avatar filenames are computedusing a hash of avatar_salt and user’s email), etc.

• The logs under /var/log/zulip can be handy to have backed up, but they do get large on a busy server, and it’sdefinitely lower-priority.

10.2. Upgrading from a git repository 43

Zulip Documentation, Release 1.4.0

10.3.1 Restore from backups

To restore from backups, the process is basically the reverse of the above:

• Install new server as normal by downloading a Zulip release tarball and then usingscripts/setup/install, you don’t need to run the initialize-database second stage whichputs default data into the database.

• Unpack to /etc/zulip the settings.py and secrets.conf files from your backups.

• Restore your database from the backup using wal-e; if you ran initialize-database anyway above,you’ll want to first scripts/setup/postgres-init-db to drop the initial database first.

• If you’re using local file uploads, restore those files to the path specified bysettings.LOCAL_UPLOADS_DIR and (if appropriate) any logs.

• Start the server using scripts/restart-server

This restoration process can also be used to migrate a Zulip installation from one server to another.

We recommend running a disaster recovery after you setup backups to confirm that your backupsare working; you may also want to monitor that they are up to date using the Nagios plugin at:puppet/zulip_internal/files/nagios_plugins/check_postgres_backup.

Contributions to more fully automate this process or make this section of the guide much more explicit and detailedare very welcome!

10.3.2 Postgres streaming replication

Zulip has database configuration for using Postgres streaming replication; you can see the configuration in these files:

• puppet/zulip_internal/manifests/postgres_slave.pp

• puppet/zulip_internal/manifests/postgres_master.pp

• puppet/zulip_internal/files/postgresql/*

Contribution of a step-by-step guide for setting this up (and moving this configuration to be available in the mainpuppet/zulip/ tree) would be very welcome!

10.4 Monitoring

The complete Nagios configuration (sans secret keys) used to monitor zulip.com is available underpuppet/zulip_internal in the Zulip Git repository (those files are not installed in the release tarballs).

The Nagios plugins used by that configuration are installed automatically by the Zulip installation process in subdirec-tories under /usr/lib/nagios/plugins/. The following is a summary of the various Nagios plugins includedwith Zulip and what they check:

Application server and queue worker monitoring:

• check_send_receive_time (sends a test message through the system between two bot users to check that end-to-end message sending works)

• check_rabbitmq_consumers and check_rabbitmq_queues (checks for rabbitmq being down or the queue workersbeing behind)

• check_queue_worker_errors (checks for errors reported by the queue workers)

• check_worker_memory (monitors for memory leaks in queue workers)

44 Chapter 10. Secure, maintain, and upgrade

Zulip Documentation, Release 1.4.0

• check_email_deliverer_backlog and check_email_deliverer_process (monitors for whether outgoing emails arebeing sent)

Database monitoring:

• check_postgres_replication_lag (checks streaming replication is up to date).

• check_postgres (checks the health of the postgres database)

• check_postgres_backup (checks backups are up to date; see above)

• check_fts_update_log (monitors for whether full-text search updates are being processed)

Standard server monitoring:

• check_website_response.sh (standard HTTP check)

• check_debian_packages (checks apt repository is up to date)

If you’re using these plugins, bug reports and pull requests to make it easier to monitor Zulip and maintain it inproduction are encouraged!

10.5 Scalability

This section attempts to address the considerations involved with running Zulip with a large team (>1000 users).

• We recommend using a remote postgres database for isolation, though it is not required. In the following,we discuss a relatively simple configuration with two types of servers: application servers (running Django,Tornado, RabbitMQ, Redis, Memcached, etc.) and database servers.

• You can scale to a pretty large installation (O(~1000) concurrently active users using it to chat all day) withjust a single reasonably large application server (e.g. AWS c3.2xlarge with 8 cores and 16GB of RAM) sittingmostly idle (<10% CPU used and only 4GB of the 16GB RAM actively in use). You can probably get awaywith half that (e.g. c3.xlarge), but ~8GB of RAM is highly recommended at scale. Beyond a 1000 active users,you will eventually want to increase the memory cap in memcached.conf from the default 512MB to avoidhigh rates of memcached misses.

• For the database server, we highly recommend SSD disks, and RAM is the primary resource limitation. Wehave not aggressively tested for the minimum resources required, but 8 cores with 30GB of RAM (e.g. AWS’sm3.2xlarge) should suffice; you may be able to get away with less especially on the CPU side. The databaseload per user is pretty optimized as long as memcached is working correctly. This has not been tested, butfrom extrapolating the load profile, it should be possible to scale a Zulip installation to 10,000s of active usersusing a single large database server without doing anything complicated like sharding the database.

• For reasonably high availability, it’s easy to run a hot spare application server and a hot spare database (usingPostgres streaming replication; see the section on configuring this). Be sure to check out the section on backupsif you’re hoping to run a spare application server; in particular you probably want to use the S3 backend forstoring user-uploaded files and avatars and will want to make sure secrets are available on the hot spare.

• Zulip does not support dividing traffic for a given Zulip realm between multiple application servers. There aretwo issues: you need to share the memcached/redis/rabbitmq instance (these should can be moved to a networkservice shared by multiple servers with a bit of configuration) and the Tornado event system for pushing tobrowsers currently has no mechanism for multiple frontend servers (or event processes) talking to each other.One can probably get a factor of 10 in a single server’s scalability by supporting multiple tornado processes ona single server, which is also likely the first part of any project to support exchanging events amongst multipleservers.

Questions, concerns, and bug reports about this area of Zulip are very welcome! This is an area we are hoping toimprove.

10.5. Scalability 45

Zulip Documentation, Release 1.4.0

10.6 Security Model

This section attempts to document the Zulip security model. Since this is new documentation, it likely does notcover every issue; if there are details you’re curious about, please feel free to ask questions on the Zulip developmentmailing list (or if you think you’ve found a security bug, please report it to security@googlegroups.com so we can doa responsible security announcement).

10.6.1 Secure your Zulip server like your email server

• It’s reasonable to think about security for a Zulip server like you do security for a team email server – onlytrusted administrators within an organization should have shell access to the server.

In particular, anyone with root access to a Zulip application server or Zulip database server, or with access tothe zulip user on a Zulip application server, has complete control over the Zulip installation and all of its data(so they can read messages, modify history, etc.). It would be difficult or impossible to avoid this, because theserver needs access to the data to support features expected of a group chat system like the ability to search theentire message history, and thus someone with control over the server has access to that data as well.

10.6.2 Encryption and Authentication

• Traffic between clients (web, desktop and mobile) and the Zulip is encrypted using HTTPS. By default, all Zulipservices talk to each other either via a localhost connection or using an encrypted SSL connection.

• The preferred way to login to Zulip is using an SSO solution like Google Auth, LDAP, or similar. Zulip storesuser passwords using the standard PBKDF2 algorithm. Password strength is checked and weak passwords arevisually discouraged using the zxcvbn library, but Zulip does not by default have strong requirements on userpassword strength. Modify static/js/common.js to adjust the password strength requirements (Patcheswelcome to make controlled by an easy setting!).

• Zulip requires CSRF tokens in all interactions with the web API to prevent CSRF attacks.

10.6.3 Messages and History

• Zulip message content is rendering using a specialized Markdown parser which escapes content to protectagainst cross-site scripting attacks.

• Zulip supports both public streams and private (“invite-only”) streams. Any Zulip user can join any publicstream in the realm (and can view the complete message of any public stream history without joining the stream).

• Users who are not members of a private stream cannot read messages on the stream, send messages to thestream, or join the stream, even if they are a Zulip administrator. However, any member of a private stream caninvite other users to the stream. When a new user joins a private stream, they can see future messages sent tothe stream, but they do not receive access to the stream’s message history.

• Zulip supports editing the content or topics of messages that have already been sent (and even updating the topicof messages sent by other users when editing the topic of the overall thread).

While edited messages are synced immediately to open browser windows, editing messages is not a safe way toredact secret content (e.g. a password) unintentionally shared via Zulip, because other users may have seen andsaved the content of the original message (for example, they could have taken a screenshot immediately afteryou sent the message, or have an API tool recording all messages they receive).

Zulip stores and sends to clients the content of every historical version of a message, so that future versions ofZulip could support displaying the diffs between previous versions.

46 Chapter 10. Secure, maintain, and upgrade

Zulip Documentation, Release 1.4.0

10.6.4 Users and Bots

• There are three types of users in a Zulip realm: Administrators, normal users, and bots. Administrators havethe ability to deactivate and reactivate other human and bot users, delete streams, add/remove administratorprivileges, as well as change configuration for the overall realm (e.g. whether an invitation is required to join therealm). Being a Zulip administrator does not provide the ability to interact with other users’ private messages orthe messages sent to private streams to which the administrator is not subscribed. However, a Zulip administratorsubscribed to a stream can toggle whether that stream is public or private. Also, Zulip realm administrators haveadministrative access to the API keys of all bots in the realm, so a Zulip administrator may be able to accessmessages sent to private streams that have bots subscribed, by using the bot’s credentials.

In the future, Zulip’s security model may change to allow realm administrators to access private messages (e.g.to support auditing functionality).

• Every Zulip user has an API key, available on the settings page. This API key can be used to do essentiallyeverything the user can do; for that reason, users should keep their API key safe. Users can rotate their own APIkey if it is accidentally compromised.

• To properly remove a user’s access to a Zulip team, it does not suffice to change their password or deactivate theiraccount in the SSO system, since neither of those prevents authenticating with the user’s API key or those ofbots the user has created. Instead, you should deactivate the user’s account in the Zulip administration interface(/#administration); this will automatically also deactivate any bots the user had created.

• The Zulip mobile apps authenticate to the server by sending the user’s password and retrieving the user’s APIkey; the apps then use the API key to authenticate all future interactions with the site. Thus, if a user’s phone islost, in addition to changing passwords, you should rotate the user’s Zulip API key.

• Zulip bots are used for integrations. A Zulip bot can do everything a normal user in the realm can do includingreading other, with a few exceptions (e.g. a bot cannot login to the web application or create other bots). Inparticular, with the API key for a Zulip bot, one can read any message sent to a public stream in that bot’s realm.A likely future feature for Zulip is limited bots that can only send messages.

• Certain Zulip bots can be marked as “API super users”; these special bots have the ability to send messages thatappear to have been sent by another user (an important feature for implementing integrations like the Jabber,IRC, and Zephyr mirrors).

10.6.5 User-uploaded content

• Zulip supports user-uploaded files; ideally they should be hosted from a separate domain from the main Zulipserver to protect against various same-domain attacks (e.g. zulip-user-content.example.com) using the S3 inte-gration.

The URLs of user-uploaded files are secret; if you are using the “local file upload” integration, anyone with theURL of an uploaded file can access the file. This means the local uploads integration is vulnerable to a subtleattack where if a user clicks on a link in a secret .PDF or .HTML file that had been uploaded to Zulip, access tothe file might be leaked to the other server via the Referrer header (see https://github.com/zulip/zulip/issues/320).

The Zulip S3 file upload integration is relatively safe against that attack, because the URLs of files presented tousers don’t host the content. Instead, the S3 integration checks the user has a valid Zulip session in the relevantrealm, and if so then redirects the browser to a one-time S3 URL that expires a short time later. Keeping theURL secret is still important to avoid other users in the Zulip realm from being able to access the file.

• Zulip supports using the Camo image proxy to proxy content like inline image previews that can be inserted intothe Zulip message feed by other users over HTTPS.

• By default, Zulip will provide image previews inline in the body of messages when a message contains a link toan image. You can control this using the INLINE_IMAGE_PREVIEW setting.

10.6. Security Model 47

Zulip Documentation, Release 1.4.0

10.6.6 Final notes and security response

If you find some aspect of Zulip that seems inconsistent with this security model, please report it to zulip-security@googlegroups.com so that we can investigate and coordinate an appropriate security release if needed.

Zulip security announcements will be sent to zulip-announce@googlegroups.com, so you should subscribe if you arerunning Zulip in production.

10.7 Management commands

Zulip has a large library of Django management commands. To use them, you will want to be logged inas the zulip user and for the purposes of this documentation, we assume the current working directory is/home/zulip/deployments/current.

Below, we should several useful examples, but there are more than 100 in total. We recommend skimming the usagedocs (or if there are none, the code) of a management command before using it, since they are generally less polishedand more designed for expert use than the rest of the Zulip system.

10.7.1 manage.py shell

You can get an iPython shell with full access to code within the Zulip project using manage.py shell, e.g. youcan do the following to change an email address:

$ /home/zulip/deployments/current/manage.py shellIn [1]: user_profile = get_user_profile_by_email("email@example.com")In [2]: do_change_user_email(user_profile, "new_email@example.com")

manage.py dbshell

This will start a postgres shell connected to the Zulip database.

10.7.2 Grant administrator access

You can make any user a realm administrator on the command line with the knight management command:

./manage.py knight username@example.com -f

Creating api super users with manage.py

If you need to manage the IRC, Jabber, or Zephyr mirrors, you will need to create api super users.To do this, use ./manage.py knight with the --permission=api_super_user argument. Seebots/irc-mirror.py and bots/jabber_mirror.py for further detail on these.

10.7.3 Other useful manage.py commands

There are a large number of useful management commands under zerver/manangement/commands/; you canalso see them listed using ./manage.py with no arguments.

48 Chapter 10. Secure, maintain, and upgrade

Zulip Documentation, Release 1.4.0

One such command worth highlighting because it’s a valuable feature with no UI in the Administration page is./manage.py realm_filters, which allows you to configure certain patterns in messages to be automati-cally linkified, e.g., whenever someone mentions “T1234”, it could be auto-linkified to ticket 1234 in your team’s Tracinstance.

Next: Remote User SSO Authentication.

10.7. Management commands 49

Zulip Documentation, Release 1.4.0

50 Chapter 10. Secure, maintain, and upgrade

CHAPTER 11

Authentication methods

Zulip supports several different authentications methods:

• EmailAuthBackend - Email/password authentication.

• ZulipLDAPAuthBackend - LDAP username/password authentication.

• GoogleMobileOauth2Backend - Google authentication.

• GitHubAuthBackend - GitHub authentication.

• ZulipRemoteUserBackend - Authentication using an existing Single-Sign-On (SSO) system that can setREMOTE_USER in Apache.

• DevAuthBackend - Only for development, passwordless login as any user.

It’s easy to add more, see the docs on python-social-auth below.

The setup documentation for most of these is simple enough that we’ve included it inline in/etc/zulip/settings.py, right above to the settings used to configure them. The remote user authenti-cation backend is more complex since it requires interfacing with a generic third-party authentication system, and sowe’ve documented it in detail below.

11.1 Adding additional methods using python-social-auth

The implementation for GitHubAuthBackend is a small wrapper around the popular python-social-auth library.So if you’d like to integrate Zulip with another authentication provider (e.g. Facebook, Twitter, etc.), youcan do this by writing a class similar to GitHubAuthBackend in zproject/backends.py and adding afew settings. Pull requests to add new backends are welcome; they should be tested using the framework intest_auth_backends.py.

11.2 Remote User SSO Authentication

Zulip supports integrating with a Single-Sign-On solution. There are a few ways to do it, but this section documentshow to configure Zulip to use an SSO solution that best supports Apache and will set the REMOTE_USER variable:

(0) Check that /etc/zulip/settings.py has zproject.backends.ZulipRemoteUserBackend asthe only enabled value in the AUTHENTICATION_BACKENDS list, and that SSO_APPEND_DOMAIN is correct setdepending on whether your SSO system uses email addresses or just usernames in REMOTE_USER.

Make sure that you’ve restarted the Zulip server since making this configuration change.

51

Zulip Documentation, Release 1.4.0

(1) Edit /etc/zulip/zulip.conf and change the puppet_classes line to read:

puppet_classes = zulip::voyager, zulip::apache_sso

(2) As root, run /home/zulip/deployments/current/scripts/zulip-puppet-apply to install ourSSO integration.

(3) To configure our SSO integration, edit /etc/apache2/sites-available/zulip-sso.example andfill in the configuration required for your SSO service to set REMOTE_USER and place your completed configurationfile at /etc/apache2/sites-available/zulip-sso.conf

zulip-sso.example is correct configuration for using an htpasswd file for REMOTE_USER authentication,which is useful for testing quickly. You can set it up by doing the following:

/home/zulip/deployments/current/scripts/restart-servercd /etc/apache2/sites-available/cp zulip-sso.example zulip-sso.confhtpasswd -c /home/zulip/zpasswd username@example.com # prompts for a password

and then continuing with the steps below.

(4) Run a2ensite zulip-sso to enable the Apache integration site.

(5) Run service apache2 reload to use your new configuration. If Apache isn’t already running, you mayneed to run service apache2 start instead.

Now you should be able to visit https://zulip.example.com/ and login via the SSO solution.

11.2.1 Troubleshooting Remote User SSO

This system is a little finicky to networking setup (e.g. common issues have to do with /etc/hosts not mapping set-tings.EXTERNAL_HOST to the Apache listening on 127.0.0.1/localhost, for example). It can often help while debug-ging to temporarily change the Apache config in /etc/apache2/sites-available/zulip-sso to listen on all interfaces ratherthan just 127.0.0.1 as you debug this. It can also be helpful to change /etc/nginx/zulip-include/app.d/external-sso.confto proxy_pass to a more explicit URL possibly not over HTTPS when debugging. The following log files can behelpful when debugging this setup:

• /var/log/zulip/{errors.log,server.log} (the usual places)

• /var/log/nginx/access.log (nginx access logs)

• /var/log/apache2/zulip_auth_access.log (you may want to change LogLevel to “debug” in the apache config fileto make this more verbose)

Here’s a summary of how the remote user SSO system works assuming you’re using HTTP basic auth; this summaryshould help with understanding what’s going on as you try to debug:

• Since you’ve configured /etc/zulip/settings.py to only define the zproject.backends.ZulipRemoteUserBackend,zproject/settings.py configures /accounts/login/sso as HOME_NOT_LOGGED_IN, which makeshttps://zulip.example.com/ aka the homepage for the main Zulip Django app running behindnginx redirect to /accounts/login/sso if you’re not logged in.

• nginx proxies requests to /accounts/login/sso/ to an Apache instance listening on localhost:8888 apache viathe config in /etc/nginx/zulip-include/app.d/external-sso.conf (using the upstream localhost:8888 defined in/etc/nginx/zulip-include/upstreams).

• The Apache zulip-sso site which you’ve enabled listens on localhost:8888 and presents the htpasswddialogue; you provide correct login information and the request reaches a second Zulip Django appinstance that is running behind Apache with with REMOTE_USER set. That request is served by

52 Chapter 11. Authentication methods

Zulip Documentation, Release 1.4.0

zerver.views.remote_user_sso, which just checks the REMOTE_USER variable and either logs in(sets a cookie) or registers the new user (depending whether they have an account).

• After succeeding, that redirects the user back to / on port 443 (hosted by nginx); the main Zulip Django app seesthe cookie and proceeds to load the site homepage with them logged in (just as if they’d logged in normally viausername/password).

Again, most issues with this setup tend to be subtle issues with the hostname/DNS side of the configuration. Sugges-tions for how to improve this SSO setup documentation are very welcome!

11.2. Remote User SSO Authentication 53

Zulip Documentation, Release 1.4.0

54 Chapter 11. Authentication methods

CHAPTER 12

Postgres database details

12.1 Remote Postgres database

This is a bit annoying to setup, but you can configure Zulip to use a dedicated postgres server by setting theREMOTE_POSTGRES_HOST variable in /etc/zulip/settings.py, and configuring Postgres certificate authentication (seehttp://www.postgresql.org/docs/9.1/static/ssl-tcp.html and http://www.postgresql.org/docs/9.1/static/libpq-ssl.html fordocumentation on how to set this up and deploy the certificates) to make the DATABASES configuration inzproject/settings.py work (or override that configuration).

If you want to use a remote Postgresql database, you should configure the information about the connection with theserver. You need a user called “zulip” in your database server. You can configure these options in /etc/zulip/settings.py:

• REMOTE_POSTGRES_HOST: Name or IP address of the remote host

• REMOTE_POSTGRES_SSLMODE: SSL Mode used to connect to the server, different options you can useare:

– disable: I don’t care about security, and I don’t want to pay the overhead of encryption.

– allow: I don’t care about security, but I will pay the overhead of encryption if the server insists on it.

– prefer: I don’t care about encryption, but I wish to pay the overhead of encryption if the server supports it.

– require: I want my data to be encrypted, and I accept the overhead. I trust that the network will make sureI always connect to the server I want.

– verify-ca: I want my data encrypted, and I accept the overhead. I want to be sure that I connect to a serverthat I trust.

– verify-full: I want my data encrypted, and I accept the overhead. I want to be sure that I connect to a serverI trust, and that it’s the one I specify.

Then you should specify the password of the user zulip for the database in /etc/zulip/zulip-secrets.conf:

postgres_password = xxxx

Finally, you can stop your database on the Zulip server via:

sudo service postgresql stopsudo update-rc.d postgresql disable

In future versions of this feature, we’d like to implement and document how to the remote postgres database serveritself automatically by using the Zulip install script with a different set of puppet manifests than the all-in-one feature;if you’re interested in working on this, post to the Zulip development mailing list and we can give you some tips.

55

Zulip Documentation, Release 1.4.0

12.2 Debugging postgres database issues

When debugging postgres issues, in addition to the standard pg_top tool, often it can be useful to use this query:

SELECT procpid,waiting,query_start,current_query FROM pg_stat_activity ORDER BY→˓procpid;

which shows the currently running backends and their activity. This is similar to the pg_top output, with the addedadvantage of showing the complete query, which can be valuable in debugging.

To stop a runaway query, you can run SELECT pg_cancel_backend(pid int) or SELECTpg_terminate_backend(pid int) as the ‘postgres’ user. The former cancels the backend’s currentquery and the latter terminates the backend process. They are implemented by sending SIGINT and SIGTERM tothe processes, respectively. We recommend against sending a Postgres process SIGKILL. Doing so will cause thedatabase to kill all current connections, roll back any pending transactions, and enter recovery mode.

12.3 Stopping the Zulip postgres database

To start or stop postgres manually, use the pg_ctlcluster command:

pg_ctlcluster 9.1 [--force] main {start|stop|restart|reload}

By default, using stop uses “smart” mode, which waits for all clients to disconnect before shutting down the database.This can take prohibitively long. If you use the –force option with stop, pg_ctlcluster will try to use the “fast” modefor shutting down. “Fast” mode is described by the manpage thusly:

With the –force option the “fast” mode is used which rolls back all active transactions, disconnects clients immediatelyand thus shuts down cleanly. If that does not work, shutdown is attempted again in “immediate” mode, which can leavethe cluster in an inconsistent state and thus will lead to a recovery run at the next start. If this still does not help, thepostmaster process is killed. Exits with 0 on success, with 2 if the server is not running, and with 1 on other failureconditions. This mode should only be used when the machine is about to be shut down.

Many database parameters can be adjusted while the database is running. Just modify/etc/postgresql/9.1/main/postgresql.conf and issue a reload. The logs will note the change.

12.4 Debugging issues starting postgres

pg_ctlcluster often doesn’t give you any information on why the database failed to start. It may tell you to check thelogs, but you won’t find any information there. pg_ctlcluster runs the following command underneath when it actuallygoes to start Postgres:

/usr/lib/postgresql/9.1/bin/pg_ctl start -D /var/lib/postgresql/9.1/main -s -o '-c→˓config_file="/etc/postgresql/9.1/main/postgresql.conf"'

Since pg_ctl doesn’t redirect stdout or stderr, running the above can give you better diagnostic information. However,you might want to stop Postgres and restart it using pg_ctlcluster after you’ve debugged with this approach, since itdoes bypass some of the work that pg_ctlcluster does.

56 Chapter 12. Postgres database details

Zulip Documentation, Release 1.4.0

12.5 Postgres Vacuuming alerts

The autovac_freeze postgres alert from check_postgres is particularly important. This alert indicatesthat the age (in terms of number of transactions) of the oldest transaction id (XID) is getting close to theautovacuum_freeze_max_age setting. When the oldest XID hits that age, Postgres will force a VACUUMoperation, which can often lead to sudden downtime until the operation finishes. If it did not do this and the age of theoldest XID reached 2 billion, transaction id wraparound would occur and there would be data loss. To clear the nagiosalert, perform a VACUUM in each indicated database as a database superuser (postgres).

See http://www.postgresql.org/docs/9.1/static/routine-vacuuming.html#VACUUM-FOR-WRAPAROUND for moredetails on postgres vacuuming.

12.5. Postgres Vacuuming alerts 57

Zulip Documentation, Release 1.4.0

58 Chapter 12. Postgres database details

CHAPTER 13

Development environment options

Zulip offers a wide range of options for how to install the development environment:

• Detailed tutorial for Vagrant development environment. Recommended for first-time contributors.

• Brief installation instructions for Vagrant development environment

• Installing on Ubuntu 14.04 Trusty or 16.04 Xenial directly (convenient but more work to maintain/uninstall).

• Installing manually on other UNIX platforms

• Using Docker (experimental)

• Using the Development Environment

• Testing

If you have a slow network connection, you should probably avoid installing Vagrant (which is large) and either installdirectly or use the manual install process instead.

59

Zulip Documentation, Release 1.4.0

60 Chapter 13. Development environment options

CHAPTER 14

Vagrant environment setup tutorial

This section guides first-time contributors through installing the Zulip dev environment on Windows 10, OS X ElCapitan, Ubuntu 14.04, and Ubuntu 16.04.

The recommended method for installing the Zulip dev environment is to use Vagrant with VirtualBox on Windowsand OS X, and Vagrant with LXC on Ubuntu. This method creates a virtual machine (for Windows and OS X) or aLinux container (for Ubuntu) inside which the Zulip server and all related services will run.

Contents:

• Requirements

• Step 1: Install Prerequisites

• Step 2: Get Zulip code

• Step 3: Start the dev environment

• Step 4: Developing

• Troubleshooting & Common Errors

If you encounter errors installing the Zulip development environment, check Troubleshooting & Common Errors. Ifthat doesn’t help, please visit the provision stream in the Zulip developers’ chat for realtime help, or send a noteto the Zulip-devel Google group or file an issue.

14.1 Requirements

Installing the Zulip dev environment requires downloading several hundred megabytes of dependencies. You will needan active internet connection throughout the entire installation processes. (See Specifying a proxy if you need a proxyto access the internet.)

• All: 2GB available RAM, Active broadband internet connection.

• OS X: OS X (El Capitan recommended, untested on previous versions), Git, VirtualBox, Vagrant.

• Ubuntu: 14.04 64-bit or 16.04 64-bit, Git, Vagrant, lxc.

• Windows: Windows 64-bit (Win 10 recommended; Win 7 untested), hardware virtualization enabled (VT-X orAMD-V), administrator access, Cygwin, VirtualBox, Vagrant.

Don’t see your system listed above? Check out:

• Brief installation instructions for Vagrant development environment

• Installing manually on UNIX-based platforms

61

Zulip Documentation, Release 1.4.0

14.2 Step 1: Install Prerequisites

Jump to:

• OS X

• Ubuntu

• Windows

14.2.1 OS X

1. Install VirtualBox

2. Install Vagrant

Now you are ready for Step 2: Get Zulip Code.

14.2.2 Ubuntu

The setup for Ubuntu 14.04 Trusty and Ubuntu 16.04 Xenial are the same.

If you’re in a hurry, you can copy and paste the following into your terminal after which you can jump to Step 2: GetZulip Code:

sudo apt-get purge vagrantwget https://releases.hashicorp.com/vagrant/1.8.4/vagrant_1.8.4_x86_64.debsudo dpkg -i vagrant*.debsudo apt-get install build-essential git ruby lxc lxc-templates cgroup-lite redirvagrant plugin install vagrant-lxcvagrant lxc sudoers

For a step-by-step explanation, read on.

1. Install Vagrant

For both 14.04 Trusty and 16.04 Xenial, you’ll need a more recent version of Vagrant than what’s available in theofficial Ubuntu repositories.

First uninstall any vagrant package you may have installed from the Ubuntu repository:

christie@ubuntu-desktop:~$ sudo apt-get purge vagrant

Now download and install the most recent .deb package from Vagrant:

christie@ubuntu-desktop:~$ wget https://releases.hashicorp.com/vagrant/1.8.4/vagrant_1.8.4_x86_64.deb

christie@ubuntu-desktop:~$ sudo dpkg -i vagrant*.deb

62 Chapter 14. Vagrant environment setup tutorial

Zulip Documentation, Release 1.4.0

2. Install remaining dependencies

Now install git and lxc-related packages:

christie@ubuntu-desktop:~$ sudo apt-get install build-essential git ruby lxc lxc-templates cgroup-lite redir

3. Install the vagrant lxc plugin:

christie@ubuntu-desktop:~$ vagrant plugin install vagrant-lxcInstalling the 'vagrant-lxc' plugin. This can take a few minutes...Installed the plugin 'vagrant-lxc (1.2.1)'!

If you encounter an error when trying to install the vagrant-lxc plugin, see this.

4. Configure sudo to be passwordless

Finally, configure sudo to be passwordless when using Vagrant LXC:

christie@ubuntu-desktop:~$ vagrant lxc sudoers[sudo] password for christie:

Now you are ready for Step 2: Get Zulip Code.

14.2.3 Windows 10

1. Install Cygwin. Make sure to install default required packages along with git, curl, openssh, and rsync binaries.

2. Install VirtualBox

3. Install Vagrant

Configure Cygwin

In order for symlinks to work within the Ubuntu virtual machine, you must tell Cygwin to create them as native Win-dows symlinks. The easiest way to do this is to add a line to ~/.bash_profile setting the CYGWIN environmentvariable.

Open a Cygwin window and do this:

christie@win10 ~$ echo 'export "CYGWIN=$CYGWIN winsymlinks:native"' >> ~/.bash_profile

Next, close that Cygwin window and open another. If you echo $CYGWIN you should see:

christie@win10 ~$ echo $CYGWINwinsymlinks:native

Now you are ready for Step 2: Get Zulip Code.

14.2. Step 1: Install Prerequisites 63

Zulip Documentation, Release 1.4.0

14.3 Step 2: Get Zulip Code

If you haven’t already created an ssh key and added it to your Github account, you should do that now by followingthese instructions.

1. In your browser, visit https://github.com/zulip/zulip and click the fork button. You will need to be logged into Github to do this.

2. Open Terminal (OS X/Ubuntu) or Cygwin (Windows; must run as an Administrator)

3. In Terminal/Cygwin, clone your fork:

git clone git@github.com:YOURUSERNAME/zulip.git

This will create a ‘zulip’ directory and download the Zulip code into it.

Don’t forget to replace YOURUSERNAME with your git username. You will see something like:

christie@win10 ~$ git clone git@github.com:YOURUSERNAME/zulip.gitCloning into 'zulip'...remote: Counting objects: 73571, done.remote: Compressing objects: 100% (2/2), done.remote: Total 73571 (delta 1), reused 0 (delta 0), pack-reused 73569Receiving objects: 100% (73571/73571), 105.30 MiB | 6.46 MiB/s, done.Resolving deltas: 100% (51448/51448), done.Checking connectivity... done.Checking out files: 100% (1912/1912), done.`

Now you are ready for Step 3: Start the dev environment.

14.4 Step 3: Start the dev environment

Change into the zulip directory and tell vagrant to start the Zulip dev environment with vagrant up.

christie@win10 ~$ cd zulip

christie@win10 ~/zulip$ vagrant up

The first time you run this command it will take some time because vagrant does the following:

• downloads the base Ubuntu 14.04 virtual machine image (for OS X and Windows) or container (for Ubuntu)

• configures this virtual machine/container for use with Zulip,

• creates a shared directory mapping your clone of the Zulip code inside the virtual machine/container at/srv/zulip

• runs the tools/provision.py script inside the virtual machine/container, which downloads all required de-pendencies, sets up the python environment for the Zulip dev environment, and initializes a default test database.

You will need an active internet connection during the entire processes. (See Specifying a proxy if you need a proxyto access the internet.) And if you’re running into any problems, please come chat with us in the provision streamof our developers’ chat.

Once vagrant up has completed, connect to the dev environment with vagrant ssh:

64 Chapter 14. Vagrant environment setup tutorial

Zulip Documentation, Release 1.4.0

christie@win10 ~/zulip$ vagrant ssh

You should see something like this on Windows and OS X:

Welcome to Ubuntu 14.04.4 LTS (GNU/Linux 3.13.0-85-generic x86_64)

* Documentation: https://help.ubuntu.com/

System information as of Wed May 4 21:45:43 UTC 2016

System load: 0.61 Processes: 88Usage of /: 3.5% of 39.34GB Users logged in: 0Memory usage: 7% IP address for eth0: 10.0.2.15Swap usage: 0%

Graph this data and manage this system at:https://landscape.canonical.com/

Get cloud support with Ubuntu Advantage Cloud Guest:http://www.ubuntu.com/business/services/cloud

0 packages can be updated.0 updates are security updates.

Or something as brief as this in the case of Ubuntu:

Welcome to Ubuntu 14.04.1 LTS (GNU/Linux 4.4.0-21-generic x86_64)

* Documentation: https://help.ubuntu.com/

Congrats, you’re now inside the Zulip dev environment!

You can confirm this by looking at the command prompt, which starts with (zulip-venv).

Next, start the Zulip server:

(zulip-venv)vagrant@vagrant-ubuntu-trusty-64:~ $/srv/zulip/tools/run-dev.py --interface=''

As you can see above the application’s root directory, where you can execute Django’s command line utilities is:

/srv/zulip/

You will see several lines of output starting with something like:

2016-05-04 22:20:33,895 INFO: process_fts_updates startingRecompiling templates2016-05-04 18:20:34,804 INFO: Not in recovery; listening for FTS updatesdoneValidating Django models.py...System check identified no issues (0 silenced).

Django version 1.8Tornado server is running at http://localhost:9993/Quit the server with CTRL-C.2016-05-04 18:20:40,716 INFO Tornado loaded 0 event queues in 0.001s2016-05-04 18:20:40,722 INFO Tornado 95.5% busy over the past 0.0 secondsPerforming system checks...

14.4. Step 3: Start the dev environment 65

Zulip Documentation, Release 1.4.0

And ending with something similar to:

http://localhost:9994/webpack-dev-server/webpack result is served from http://localhost:9991/webpack/content is served from /srv/zulip

webpack: bundle is now VALID.2016-05-06 21:43:29,553 INFO Tornado 31.6% busy over the past 10.6 seconds2016-05-06 21:43:35,007 INFO Tornado 23.9% busy over the past 16.0 seconds

Now the Zulip server should be running and accessible. Verify this by navigating to http://localhost:9991/ in yourbrowser on your main machine.

You should see something like (this screenshot of the Zulip dev environment).

The Zulip server will continue to run and send output to the terminal window. When you navigate to Zulip in yourbrowser, check your terminal and you should see something like:

2016-05-04 18:21:57,547 INFO 127.0.0.1 GET 302 582ms (+start: 417ms) /→˓(unauth via ?)[04/May/2016 18:21:57]"GET / HTTP/1.0" 302 02016-05-04 18:21:57,568 INFO 127.0.0.1 GET 301 4ms /login (unauth via→˓?)[04/May/2016 18:21:57]"GET /login HTTP/1.0" 301 02016-05-04 18:21:57,819 INFO 127.0.0.1 GET 200 209ms (db: 7ms/2q) /→˓login/ (unauth via ?)

Now you’re ready for Step 4: Developing.

14.5 Step 4: Developing

14.5.1 Where to edit files

You’ll work by editing files on your host machine, in the directory where you cloned Zulip. Use your favorite editor(Sublime, Atom, Vim, Emacs, Notepad++, etc.).

When you save changes they will be synced automatically to the Zulip dev environment on the virtual ma-chine/container.

Each component of the Zulip development server will automatically restart itself or reload data appropriately whenyou make changes. So, to see your changes, all you usually have to do is reload your browser. More details on howthis works are available below.

Don’t forget to read through the code style guidelines for details about how to configure your editor for Zulip. Forexample, indentation should be set to 4 spaces rather than tabs.

14.5.2 Understanding run-dev.py debugging output

It’s good to have the terminal running run-dev.py up as you work since error messages including tracebacks alongwith every backend request will be printed there.

See Logging for further details on the run-dev.py console output.

66 Chapter 14. Vagrant environment setup tutorial

Zulip Documentation, Release 1.4.0

14.5.3 Committing and pushing changes with git

When you’re ready to commit or push changes via git, you will do this by running git commands in Terminal (OSX/Ubuntu) or Cygwin (Windows) in the directory where you cloned Zulip on your main machine.

If you’re new to working with Git/Github, check out this guide.

14.5.4 Maintaining the dev environment

If after rebasing onto a new version of the Zulip server, you receive new errors while starting the Zulip server orrunning tests, this is probably not because Zulip’s master branch is broken. Instead, this is likely because we’verecently merged changes to the development environment provisioning process that you need to apply to your devel-opment environmnet. To update your environment, you’ll need to re-provision your vagrant machine using vagrantprovision (or just python tools/provision.py from /srv/zulip inside the Vagrant guest); this shouldbe pretty fast and we’re working to make it faster.

See also the documentation on the testing page for how to destroy and rebuild your database if you want to clear outtest data.

14.5.5 Rebuilding the dev environment

If you ever want to recreate your development environment again from scratch (e.g. to test as change you’ve made tothe provisioning process, or because you think something is broken), you can do so using vagrant destroy andthen vagrant up. This will usually be much faster than the original vagrant up since the base image is alreadycached on your machine (it takes about 5 minutes to run with a fast Internet connection).

14.5.6 Shutting down the dev environment for use later

To shut down but preserve the dev environment so you can use it again later use vagrant halt or vagrantsuspend.

You can do this from the same Terminal/Cygwin window that is running run-dev.py by pressing ^C to halt the serverand then typing exit. Or you can halt vagrant from another Terminal/Cygwin window.

From the window where run-dev.py is running:

2016-05-04 18:33:13,330 INFO 127.0.0.1 GET 200 92ms /register/ (unauth→˓via ?)^CKeyboardInterrupt(zulip-venv)vagrant@vagrant-ubuntu-trusty-64:/srv/zulip$ exitlogoutConnection to 127.0.0.1 closed.christie@win10 ~/zulip

Now you can suspend the dev environment:

christie@win10 ~/zulip$ vagrant suspend==> default: Saving VM state and suspending execution...

If vagrant suspend doesn’t work, try vagrant halt:

14.5. Step 4: Developing 67

Zulip Documentation, Release 1.4.0

christie@win10 ~/zulip$ vagrant halt==> default: Attempting graceful shutdown of VM...

Check out the Vagrant documentation to learn more about suspend and halt.

14.5.7 Resuming the dev environment

When you’re ready to work on Zulip again, run vagrant up. You will also need to connect to the virtual machinewith vagrant ssh and re-start the Zulip server:

christie@win10 ~/zulip$ vagrant up$ vagrant ssh/srv/zulip/tools/run-dev.py --interface=''

14.5.8 Next Steps

At this point you should [read about using the development environment][using-dev-environment.html].

14.6 Troubleshooting & Common Errors

Zulip’s vagrant provisioning process logs useful debugging output to /var/log/zulip_provision.log; ifyou encounter a new issue, please attach a copy of that file to your bug report.

14.6.1 The box ‘ubuntu/trusty64’ could not be found (Windows/Cygwin)

If you see the following error when you run vagrant up on Windows:

The box 'ubuntu/trusty64' could not be found orcould not be accessed in the remote catalog. If this is a privatebox on HashiCorp's Atlas, please verify you're logged in via`vagrant login`. Also, please double-check the name. The expandedURL and error message are shown below:URL: ["https://atlas.hashicorp.com/ubuntu/trusty64"]

Then the version of curl that ships with Vagrant is not working on your machine. The fix is simple: replace it with theversion from Cygwin.

First, determine the location of Cygwin’s curl with which curl:

christie@win10 ~/zulip$ which curl/usr/bin/curl

Now determine the location of Vagrant with which vagrant:

christie@win10 ~/zulip$ which vagrant/cygdrive/c/HashiCorp/Vagrant/bin/vagrant

68 Chapter 14. Vagrant environment setup tutorial

Zulip Documentation, Release 1.4.0

The path up until /bin/vagrant is what you need to know. In the example above it’s/cygdrive/c/HashiCorp/Vagrant.

Finally, copy Cygwin’s curl to Vagrant embedded/bin directory:

christie@win10 ~/zulip$ cp /usr/bin/curl.exe /cygdrive/c/HashiCorp/Vagrant/embedded/bin/

Now re-run vagrant up and vagrant should be able to fetch the required box file.

14.6.2 os.symlink error

If you receive the following error while running vagrant up:

==> default: Traceback (most recent call last):==> default: File "./emoji_dump.py", line 75, in <module>==> default:==> default: os.symlink('unicode/{}.png'.format(code_point), 'out/{}.png'.→˓format(name))==> default: OSError==> default: :==> default: [Errno 71] Protocol error

Then Vagrant was not able to create a symbolic link.

First, if you are using Windows, make sure you have run Cygwin as an administrator. By default, only adminis-trators can create symbolic links on Windows.

Second, VirtualBox does not enable symbolic links by default. Vagrant starting with version 1.6.0 enables symboliclinks for VirtualBox shared folder.

You can check to see that this is enabled for your virtual machine with vboxmanage command.

Get the name of your virtual machine by running vboxmanage list vms and then print out the custom settingsfor this virtual machine with vboxmanage getextradata YOURVMNAME enumerate:

christie@win10 ~/zulip$ vboxmanage list vms"zulip_default_1462498139595_55484" {5a65199d-8afa-4265-b2f6-6b1f162f157d}

christie@win10 ~/zulip$ vboxmanage getextradata zulip_default_1462498139595_55484 enumerateKey: VBoxInternal2/SharedFoldersEnableSymlinksCreate/srv_zulip, Value: 1Key: supported, Value: false

If you see “command not found” when you try to run VBoxManage, you need to add the VirtualBox directory to yourpath. On Windows this is mostly likely C:\Program Files\Oracle\VirtualBox\.

If vboxmanage enumerate prints nothing, or shows a value of 0 for VBoxInter-nal2/SharedFoldersEnableSymlinksCreate/srv_zulip, then enable symbolic links by running this command inTerminal/Cygwin:

vboxmanage setextradata YOURVMNAME VBoxInternal2/SharedFoldersEnableSymlinksCreate/→˓srv_zulip 1

The virtual machine needs to be shut down when you run this command.

14.6. Troubleshooting & Common Errors 69

Zulip Documentation, Release 1.4.0

14.6.3 Connection timeout on vagrant up

If you see the following error after running vagrant up:

default: SSH address: 127.0.0.1:2222default: SSH username: vagrantdefault: SSH auth method: private keydefault: Error: Connection timeout. Retrying...default: Error: Connection timeout. Retrying...default: Error: Connection timeout. Retrying...

A likely cause is that hardware virtualization is not enabled for your computer. This must be done via your computer’sBIOS settings. Look for a setting called VT-x (Intel) or (AMD-V).

If this is already enabled in your BIOS, double-check that you are running a 64-bit operating system.

For further information about troubleshooting vagrant timeout errors see this post.

14.6.4 npm install error

The tools/provision.py script may encounter an error related to npm install that looks something like:

==> default: + npm install==> default: Traceback (most recent call last):==> default: File "/srv/zulip/tools/provision.py", line 195, in <module>==> default:==> default: sys.exit(main())==> default: File "/srv/zulip/tools/provision.py", line 191, in main==> default:==> default: run(["npm", "install"])==> default: File "/srv/zulip/scripts/lib/zulip_tools.py", line 78, in run==> default:==> default: raise subprocess.CalledProcessError(rc, args)==> default: subprocess==> default: .==> default: CalledProcessError==> default: :==> default: Command '['npm', 'install']' returned non-zero exit status 34The SSH command responded with a non-zero exit status. Vagrantassumes that this means the command failed. The output for this commandshould be in the log above. Please read the output to determine whatwent wrong.

Usually this error is not fatal. Try connecting to the dev environment and re-trying the command from withing thevirtual machine:

christie@win10 ~/zulip$ vagrant ssh(zulip-venv)vagrant@vagrant-ubuntu-trusty-64:~$ cd /srv/zulip(zulip-venv)vagrant@vagrant-ubuntu-trusty-64:/srv/zulip$ npm installnpm WARN optional Skipping failed optional dependency /chokidar/fsevents:npm WARN notsup Not compatible with your operating system or architecture: fsevents@1.→˓0.12

These are just warnings so it is okay to proceed and start the Zulip server.

70 Chapter 14. Vagrant environment setup tutorial

Zulip Documentation, Release 1.4.0

14.6.5 NoMethodError when installing vagrant-lxc plugin (Ubuntu 16.04)

If you see the following error when you try to install the vagrant-lxc plugin:

/usr/lib/ruby/2.3.0/rubygems/specification.rb:946:in `all=': undefined method `group_→˓by' for nil:NilClass (NoMethodError)from /usr/lib/ruby/vendor_ruby/vagrant/bundler.rb:275:in `with_isolated_gem'from /usr/lib/ruby/vendor_ruby/vagrant/bundler.rb:231:in `internal_install'from /usr/lib/ruby/vendor_ruby/vagrant/bundler.rb:102:in `install'from /usr/lib/ruby/vendor_ruby/vagrant/plugin/manager.rb:62:in `block in install_

→˓plugin'from /usr/lib/ruby/vendor_ruby/vagrant/plugin/manager.rb:72:in `install_plugin'from /usr/share/vagrant/plugins/commands/plugin/action/install_gem.rb:37:in `call'from /usr/lib/ruby/vendor_ruby/vagrant/action/warden.rb:34:in `call'from /usr/lib/ruby/vendor_ruby/vagrant/action/builder.rb:116:in `call'from /usr/lib/ruby/vendor_ruby/vagrant/action/runner.rb:66:in `block in run'from /usr/lib/ruby/vendor_ruby/vagrant/util/busy.rb:19:in `busy'from /usr/lib/ruby/vendor_ruby/vagrant/action/runner.rb:66:in `run'from /usr/share/vagrant/plugins/commands/plugin/command/base.rb:14:in `action'from /usr/share/vagrant/plugins/commands/plugin/command/install.rb:32:in `block in

→˓execute'from /usr/share/vagrant/plugins/commands/plugin/command/install.rb:31:in `each'from /usr/share/vagrant/plugins/commands/plugin/command/install.rb:31:in `execute'from /usr/share/vagrant/plugins/commands/plugin/command/root.rb:56:in `execute'from /usr/lib/ruby/vendor_ruby/vagrant/cli.rb:42:in `execute'from /usr/lib/ruby/vendor_ruby/vagrant/environment.rb:268:in `cli'from /usr/bin/vagrant:173:in `<main>'

And you have vagrant version 1.8.1, then you need to patch vagrant manually. See this post for an explanation of theissue, which should be fixed when Vagrant 1.8.2 is released.

In the meantime, read this post for how to create and apply the patch.

It will look something like this:

christie@xenial:~$ sudo patch --directory /usr/lib/ruby/vendor_ruby/vagrant < vagrant-plugin.patchpatching file bundler.rb

14.6.6 Permissions errors when running the test suite in LXC

See “Possible testing issues”.

14.6. Troubleshooting & Common Errors 71

Zulip Documentation, Release 1.4.0

72 Chapter 14. Vagrant environment setup tutorial

CHAPTER 15

Vagrant environment setup (in brief)

Start by cloning this repository: git clone https://github.com/zulip/zulip.git

This is the recommended approach for all platforms, and will install the Zulip development environment inside a VMor container and works on any platform that supports Vagrant.

The best performing way to run the Zulip development environment is using an LXC container on a Linux host, butwe support other platforms such as Mac via Virtualbox (but everything will be 2-3x slower).

• If your host is Ubuntu 15.04 or newer, you can install and configure the LXC Vagrant provider directly usingapt:

sudo apt-get install vagrant lxc lxc-templates cgroup-lite redirvagrant plugin install vagrant-lxc

You may want to configure sudo to be passwordless when using Vagrant LXC.

• If your host is Ubuntu 14.04, you will need to download a newer version of Vagrant, and then do the following:

sudo apt-get install lxc lxc-templates cgroup-lite redirsudo dpkg -i vagrant*.deb # in directory where you downloaded vagrantvagrant plugin install vagrant-lxc

You may want to configure sudo to be passwordless when using Vagrant LXC.

• For other Linux hosts with a kernel above 3.12, follow the Vagrant LXC installation instructions to get Vagrantwith LXC for your platform.

• If your host is OS X or older Linux, download VirtualBox, download Vagrant, and install them both.

• If you’re on OS X and have VMWare, it should be possible to patch Vagrantfile to use the VMWare vagrantprovider which should perform much better than Virtualbox. Patches to do this by default if VMWare is availableare welcome!

• On Windows: You can use Vagrant and Virtualbox/VMWare on Windows with Cygwin, similar to the Macsetup. Be sure to create your git clone using git clone https://github.com/zulip/zulip.git-c core.autocrlf=false to avoid Windows line endings being added to files (this causes weird errors).

Once that’s done, simply change to your zulip directory and run vagrant up in your terminal to install the devel-opment server. This will take a long time on the first run because Vagrant needs to download the Ubuntu Trusty baseimage, but later you can run vagrant destroy and then vagrant up again to rebuild the environment and itwill be much faster.

Once that finishes, you can run the development server as follows:

73

Zulip Documentation, Release 1.4.0

vagrant ssh# Now inside the container/srv/zulip/tools/run-dev.py --interface=''

To get shell access to the virtual machine running the server to run lint, management commands, etc., use vagrantssh.

(A small note on tools/run-dev.py: the --interface='' option will make the development server listen on allnetwork interfaces. While this is correct for the Vagrant guest sitting behind a NAT, you probably don’t want to usethat option when using run-dev.py in other environments).

At this point you should read about using the development environment.

15.1 Specifying a proxy

If you need to use a proxy server to access the Internet, you will need to specify the proxy settings before runningVagrant up. First, install the Vagrant plugin vagrant-proxyconf:

vagrant plugin install vagrant-proxyconf.

Then create ~/.zulip-vagrant-config and add the following lines to it (with the appropriate values in it foryour proxy):

HTTP_PROXY http://proxy_host:portHTTPS_PROXY http://proxy_host:portNO_PROXY localhost,127.0.0.1,.example.com

Now run vagrant up in your terminal to install the development server. If you ran vagrant up before and failed,you’ll need to run vagrant destroy first to clean up the failed installation.

You can also change the port on the host machine that Vagrant uses by adding to your~/.zulip-vagrant-config file. E.g. if you set:

HOST_PORT 9971

(and halt and restart the Vagrant guest), then you would visit http://localhost:9971/ to connect to your developmentserver.

74 Chapter 15. Vagrant environment setup (in brief)

CHAPTER 16

Installing directly on Ubuntu

Start by cloning this repository: git clone https://github.com/zulip/zulip.git

If you’d like to install a Zulip development environment on a computer that’s already running Ubuntu 14.04 Trusty orUbuntu 16.04 Xenial, you can do that by just running:

# From a clone of zulip.git./tools/provision.pysource /srv/zulip-venv/bin/activate./tools/run-dev.py # starts the development server

Note that there is no supported uninstallation process without Vagrant (with Vagrant, you can just do vagrantdestroy to clean up the development environment).

Once you’ve done the above setup, you can pick up the documentation on using the Zulip development environment,ignoring the parts about vagrant (since you’re not using it).

75

Zulip Documentation, Release 1.4.0

76 Chapter 16. Installing directly on Ubuntu

CHAPTER 17

Installing manually on UNIX

• Debian or Ubuntu systems

• Fedora 22 (experimental)

• CentOS 7 Core (experimental)

• OpenBSD 5.8 (experimental)

• Fedora/CentOS common steps

• Steps for all systems

If you really want to install everything manually, the below instructions should work.

Install the following non-Python dependencies:

• libffi-dev — needed for some Python extensions

• postgresql 9.1 or later — our database (client, server, headers)

• nodejs 0.10 (and npm)

• memcached (and headers)

• rabbitmq-server

• libldap2-dev

• python-dev

• redis-server — rate limiting

• tsearch-extras — better text search

• libfreetype6-dev — needed before you pip install Pillow to properly generate emoji PNGs

17.1 On Debian or Ubuntu systems:

17.1.1 Using the official Ubuntu repositories and tsearch-extras deb package:

Start by cloning this repository: git clone https://github.com/zulip/zulip.git

sudo apt-get install closure-compiler libfreetype6-dev libffi-dev \memcached rabbitmq-server libldap2-dev redis-server \postgresql-server-dev-all libmemcached-dev python-dev \hunspell-en-us nodejs nodejs-legacy npm git yui-compressor \

77

Zulip Documentation, Release 1.4.0

puppet gettext postgresql

# Next, install Zulip's tsearch-extras postgresql extension# If on 14.04 or 16.04, you can use the Zulip PPA for tsearch-extras:cd zulipsudo apt-add-repository -yus ppa:tabbott/zulip# On 14.04sudo apt-get install postgresql-9.3-tsearch-extras# On 16.04sudo apt-get install postgresql-9.5-tsearch-extras

# Otherwise, you can download a .deb directly# If on 12.04 or wheezy:wget https://dl.dropboxusercontent.com/u/283158365/zuliposs/postgresql-9.1-tsearch-→˓extras_0.1.2_amd64.debsudo dpkg -i postgresql-9.1-tsearch-extras_0.1.2_amd64.deb

# If on 14.04:https://launchpad.net/~tabbott/+archive/ubuntu/zulip/+files/postgresql-9.3-tsearch-→˓extras_0.1.3_amd64.debsudo dpkg -i postgresql-9.3-tsearch-extras_0.1.3_amd64.deb

# If on 15.04 or jessie:wget https://dl.dropboxusercontent.com/u/283158365/zuliposs/postgresql-9.4-tsearch-→˓extras_0.1_amd64.debsudo dpkg -i postgresql-9.4-tsearch-extras_0.1_amd64.deb

# If on 16.04 or stretchwget https://launchpad.net/~tabbott/+archive/ubuntu/zulip/+files/postgresql-9.5-→˓tsearch-extras_0.2_amd64.debsudo dpkg -i postgresql-9.5-tsearch-extras_0.2_amd64.deb

Alternatively, you can always build the package from tsearch-extras git.

Now continue with the All Systems instructions below.

17.1.2 Using the official Zulip PPA (for 14.04 Trusty or 16.04 Xenial):

Start by cloning this repository: git clone https://github.com/zulip/zulip.git

sudo add-apt-repository ppa:tabbott/zulipsudo apt-get updatesudo apt-get install closure-compiler libfreetype6-dev libffi-dev \

memcached rabbitmq-server libldap2-dev redis-server \postgresql-server-dev-all libmemcached-dev python-dev \hunspell-en-us nodejs nodejs-legacy npm git yui-compressor \puppet gettext tsearch-extras

Now continue with the All Systems instructions below.

17.2 On Fedora 22 (experimental):

These instructions are experimental and may have bugs; patches welcome!

78 Chapter 17. Installing manually on UNIX

Zulip Documentation, Release 1.4.0

Start by cloning this repository: git clone https://github.com/zulip/zulip.git

sudo dnf install libffi-devel memcached rabbitmq-server \openldap-devel python-devel redis postgresql-server \postgresql-devel postgresql libmemcached-devel freetype-devel \nodejs npm yuicompressor closure-compiler gettext

Now continue with the Common to Fedora/CentOS instructions below.

17.3 On CentOS 7 Core (experimental):

These instructions are experimental and may have bugs; patches welcome!

Start by cloning this repository: git clone https://github.com/zulip/zulip.git

# Add user zulip to the system (not necessary if you configured zulip# as the administrator user during the install process of CentOS 7).useradd zulip

# Create a password for zulip userpasswd zulip

# Allow zulip to sudovisudo# Add this line after line `root ALL=(ALL) ALL`zulip ALL=(ALL) ALL

# Switch to zulip usersu zulip

# Enable EPEL 7 repo so we can install rabbitmq-server, redis and# other dependenciessudo yum install https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.→˓rpm

# Install dependenciessudo yum install libffi-devel memcached rabbitmq-server openldap-devel \

python-devel redis postgresql-server postgresql-devel postgresql \libmemcached-devel wget python-pip openssl-devel freetype-devel \libjpeg-turbo-devel zlib-devel nodejs yuicompressor \closure-compiler gettext

# We need these packages to compile tsearch-extrassudo yum groupinstall "Development Tools"

# clone Zulip's git repo and cd into itcd && git clone https://github.com/zulip/zulip && cd zulip/

## NEEDS TESTING: The next few DB setup items may not be required at all.# Initialize the postgres dbsudo postgresql-setup initdb

# Edit the postgres settings:sudo vi /var/lib/pgsql/data/pg_hba.conf

# Change these lines:host all all 127.0.0.1/32 ident

17.3. On CentOS 7 Core (experimental): 79

Zulip Documentation, Release 1.4.0

host all all ::1/128 ident# to this:host all all 127.0.0.1/32 md5host all all ::1/128 md5

Now continue with the Common to Fedora/CentOS instructions below.

17.4 On OpenBSD 5.8 (experimental):

These instructions are experimental and may have bugs; patches welcome!

Start by cloning this repository: git clone https://github.com/zulip/zulip.git

doas pkg_add sudo bash gcc postgresql-server redis rabbitmq \memcached node libmemcached py-Pillow py-cryptography py-cffi

# Get tsearch_extras and build it (using a modified version which# aliases int4 on OpenBSD):git clone https://github.com/blablacio/tsearch_extrascd tsearch_extrasgmake && sudo gmake install

# Point environment to custom include locations and use newer GCC# (needed for Node modules):export CFLAGS="-I/usr/local/include -I/usr/local/include/sasl"export CXX=eg++

# Create tsearch_data directory:sudo mkdir /usr/local/share/postgresql/tsearch_data

# Hack around missing dictionary files -- need to fix this to get the# proper dictionaries from what in debian is the hunspell-en-us# package.sudo touch /usr/local/share/postgresql/tsearch_data/english.stopsudo touch /usr/local/share/postgresql/tsearch_data/en_us.dictsudo touch /usr/local/share/postgresql/tsearch_data/en_us.affix

Finally continue with the All Systems instructions below.

17.5 Common to Fedora/CentOS instructions

Start by cloning this repository: git clone https://github.com/zulip/zulip.git

# Build and install postgres tsearch-extras modulewget https://launchpad.net/~tabbott/+archive/ubuntu/zulip/+files/tsearch-extras_0.1.3.→˓tar.gztar xvzf tsearch-extras_0.1.3.tar.gzcd ts2makesudo make install

# Hack around missing dictionary files -- need to fix this to get the# proper dictionaries from what in debian is the hunspell-en-us

80 Chapter 17. Installing manually on UNIX

Zulip Documentation, Release 1.4.0

# package.sudo touch /usr/share/pgsql/tsearch_data/english.stopsudo touch /usr/share/pgsql/tsearch_data/en_us.dictsudo touch /usr/share/pgsql/tsearch_data/en_us.affix

# Edit the postgres settings:sudo vi /var/lib/pgsql/data/pg_hba.conf

# Add this line before the first uncommented line to enable password# auth:host all all 127.0.0.1/32 md5

# Start the servicessudo systemctl start redis memcached rabbitmq-server postgresql

# Enable automatic service startup after the system startupsudo systemctl enable redis rabbitmq-server memcached postgresql

Finally continue with the All Systems instructions below.

17.6 All Systems:

Make sure you have followed the steps specific for your platform:

• Debian or Ubuntu systems

• Fedora 22 (experimental)

• CentOS 7 Core (experimental)

• OpenBSD 5.8 (experimental)

• Fedora/CentOS

For managing Zulip’s python dependencies, we recommend using virtualenvs.

You must create two virtualenvs. One for Python 2 and one for Python 3. You must also install appropriate pythonpackages in them.

You should either install the virtualenvs in /srv, or put symlinks to them in /srv. If you don’t do that, some scriptsmight not work correctly.

You can run tools/setup/setup_venvs.py to do this. This script will create two virtualenvs - /srv/zulip-venvand /srv/zulip-py3-venv.

If you want to do it manually, here are the steps:

virtualenv /srv/zulip-venv -p python2 # Create a python2 virtualenvsource /srv/zulip-venv/bin/activate # Activate python2 virtualenvpip install --upgrade pip # upgrade pip itself because older versions have known→˓issuespip install --no-deps -r requirements/py2_dev.txt # install python packages required→˓for development

virtualenv /srv/zulip-py3-venv -p python3 # Create a python3 virtualenvsource /srv/zulip-py3-venv/bin/activate # Activate python3 virtualenvpip install --upgrade pip # upgrade pip itself because older versions have known→˓issuespip install --no-deps -r requirements/py3_dev.txt # install python packages required→˓for development

17.6. All Systems: 81

Zulip Documentation, Release 1.4.0

Now run these commands:

./tools/setup/install-phantomjs

./tools/install-mypy

./tools/setup/download-zxcvbn

./tools/setup/emoji_dump/build_emoji

./scripts/setup/generate_secrets.py -dif [ $(uname) = "OpenBSD" ]; then sudo cp ./puppet/zulip/files/postgresql/zulip_→˓english.stop /var/postgresql/tsearch_data/; else sudo cp ./puppet/zulip/files/→˓postgresql/zulip_english.stop /usr/share/postgresql/9.*/tsearch_data/; fi./scripts/setup/configure-rabbitmq./tools/setup/postgres-init-dev-db./tools/do-destroy-rebuild-database./tools/setup/postgres-init-test-db./tools/do-destroy-rebuild-test-database./manage.py compilemessagesnpm install

If npm install fails, the issue may be that you need a newer version of npm. You can use npm install -gnpm to update your version of npm and try again.

To start the development server:

./tools/run-dev.py

. . . and visit http://localhost:9991/.

17.6.1 Proxy setup for by-hand installation

If you are building the development environment on a network where a proxy is required to access the Internet, youwill need to set the proxy in the environment as follows:

• On Ubuntu, set the proxy environment variables using:

export https_proxy=http://proxy_host:portexport http_proxy=http://proxy_host:port

• And set the npm proxy and https-proxy using:

npm config set proxy http://proxy_host:portnpm config set https-proxy http://proxy_host:port

82 Chapter 17. Installing manually on UNIX

CHAPTER 18

Using Docker (experimental)

Start by cloning this repository: git clone https://github.com/zulip/zulip.git

The docker instructions for development are experimental, so they may have bugs. If you try them and run into anyissues, please report them!

You can also use Docker to run a Zulip development environment. First, you need to install Docker in your develop-ment machine following the instructions. Some other interesting links for somebody new in Docker are:

• Get Started

• Understand the architecture

• Docker run reference

• Dockerfile reference

Then you should create the Docker image based on Ubuntu Linux, first go to the directory with the Zulip source code:

docker build -t user/zulipdev .

Now you’re going to install Zulip dependencies in the image:

docker run -itv $(pwd):/srv/zulip -p 9991:9991 user/zulipdev /bin/bash$ /usr/bin/python /srv/zulip/tools/provision.py --dockerdocker ps -af ancestor=user/zulipdevdocker commit -m "Zulip installed" <container id> user/zulipdev:v2

Finally you can run the docker server with:

docker run -itv $(pwd):/srv/zulip -p 9991:9991 user/zulipdev:v2 \/srv/zulip/tools/start-dockers

If you want to connect to the Docker instance to build a release tarball you can use:

docker psdocker exec -it <container id> /bin/bash$ source /home/zulip/.bash_profile$ <Your commands>$ exit

To stop the server use:

docker psdocker kill <container id>

83

Zulip Documentation, Release 1.4.0

If you want to run all the tests you need to start the servers first, you can do it with:

docker run -itv $(pwd):/srv/zulip user/zulipdev:v2 /bin/bash$ tools/test-all-docker

You can modify the source code in your development machine and review the results in your browser.

84 Chapter 18. Using Docker (experimental)

CHAPTER 19

Using the Development Environment

Once the development environment is running, you can visit http://localhost:9991/ in your browser. By default, thedevelopment server homepage just shows a list of the users that exist on the server and you can login as any of them byjust clicking on a user. This setup saves time for the common case where you want to test something other than the loginprocess; to test the login process you’ll want to change AUTHENTICATION_BACKENDS in the not-PRODUCTIONcase of zproject/settings.py from zproject.backends.DevAuthBackend to use the auth method(s) you’d liketo test.

While developing, it’s helpful to watch the run-dev.py console output, which will show any errors your Zulipdevelopment server encounters.

When you make a change, here’s a guide for what you need to do in order to see your change take effect in Develop-ment:

• If you change JavaScript, CSS, or Jinja2 backend templates (under templates/), you’lljust need to reload the browser window to see changes take effect. The Handlebarsfrontend HTML templates (static/templates) are automatically recompiled by thetools/compile-handlebars-templates job, which runs as part of tools/run-dev.py.

• If you change Python code used by the the main Django/Tornado server processes, these services are run ontop of Django’s manage.py runserver which will automatically restart the Zulip Django and Tornado serverswhenever you save changes to Python code. You can watch this happen in the run-dev.py console to makesure the backend has reloaded.

• The Python queue workers will also automatically restart when you save changes. However, you may need toctrl-C and then restart run-dev.py manually if a queue worker has crashed.

• If you change the database schema, you’ll need to use the standard Django migrations process to create and thenrun your migrations; see the new feature tutorial for an example. Additionally you should check out the detailedtesting docs for how to run the tests properly after doing a migration.

(In production, everything runs under supervisord and thus will restart if it crashes, and upgrade-zulip will takecare of running migrations and then cleanly restaring the server for you).

85

Zulip Documentation, Release 1.4.0

86 Chapter 19. Using the Development Environment

CHAPTER 20

Writing a new integration

Integrations are one of the most important parts of a group chat tool like Zulip, and we are committed to makingintegrating with Zulip and getting you integration merged upstream so everyone else can benefit from it as easy aspossible while maintaining the high quality of the Zulip integrations library.

On this page you’ll find:

• An overvew of the different types of integrations possible with Zulip.

• General advice for writing integrations.

• Details about writing webhook integrations.

• Details about writing Python script and plugin integrations.

• A guide to documenting your integration.

• A detailed walkthrough of a simple “Hello World” integration.

Contributions to this guide are very welcome, so if you run into any issues following these instructions or come upwith any tips or tools that help writing integration, please email zulip-devel@googlegroups.com, open an issue, orsubmit a pull request to share your ideas!

20.1 Types of integrations

We have several different ways that we integrate with 3rd part products, ordered here by which types we prefer towrite:

1. Webhook integrations (examples: Freshdesk, GitHub), where the third-party service supports posting contentto a particular URI on our site with data about the event. For these, you usually just need to add a new han-dler in zerver/views/webhooks.py (plus test/document/etc.). An example commit implementing a newwebhook is: https://github.com/zulip/zulip/pull/324.

2. Python script integrations (examples: SVN, Git), where we can get the service to call our integration (byshelling out or otherwise), passing in the required data. Our preferred model for these is to ship these integrationsin our API release tarballs (by writing the integration in api/integrations).

3. Plugin integrations (examples: Jenkins, Hubot, Trac) where the user needs to install a plugin into their existingsoftware. These are often more work, but for some products are the only way to integrate with the product at all.

87

Zulip Documentation, Release 1.4.0

20.2 General advice

• Consider using our Zulip markup to make the output from your integration especially attractive or useful (e.g.emoji, markdown emphasis, @-mentions, or !avatar(email)).

• Use topics effectively to ensure sequential messages about the same thing are threaded together; this makes formuch better consumption by users. E.g. for a bug tracker integration, put the bug number in the topic for allmessages; for an integration like Nagios, put the service in the topic.

• Integrations that don’t match a team’s workflow can often be uselessly spammy. Give careful thought to provid-ing options for triggering Zulip messages only for certain message types, certain projects, or sending differentmessages to different streams/topics, to make it easy for teams to configure the integration to support theirworkflow.

• Consistently capitalize the name of the integration in the documentation and the Client name the way the vendordoes. It’s OK to use all-lower-case in the implementation.

• Sometimes it can be helpful to contact the vendor if it appears they don’t have an API or webhook we can use –sometimes the right API is just not properly documented.

• A helpful tool for testing your integration is UltraHook, which allows you to receive webhook calls via yourlocal Zulip dev environment. This enables you to do end-to-end testing with live data from the service you’reintegrating and can help you spot why something isn’t working or if the service is using custom HTTP headers.

20.3 Webhook integrations

New Zulip webhook integrations can take just a few hours to write, including tests and documentation, if you use theright process. Here’s how we recommend doing it:

• First, use http://requestb.in/ or a similar site to capture an example webhook payload from the service you’reintegrating. You can use these captured payloads to create a set of test fixtures for your integration underzerver/fixtures.

• Then write a draft webhook handler under zerver/views/webhooks/; there are a lot of examples inthat directory. We recommend templating off a short one (like stash.py or zendesk.py), since thelonger ones usually just have more complex parsing which can obscure what’s common to all webhook in-tegrations. In addition to writing the integration itself, you’ll need to create Integration object and add itto WEBHOOK_INTEGRATIONS in zerver/lib/integrations.py'; search forwebhook‘ in thatfile to find the existing ones (and please add yours in the alphabetically correct place).

• Then write a test for your fixture in zerver/tests/test_hooks.py, and you can iterate on the tests andwebhooks handler until they work, all without ever needing to post directly from the server you’re integratingto your Zulip development machine. To run just the tests from the test class you wrote, you can use e.g.

test-backend zerver.tests.test_hooks.PagerDutyHookTests

See this guide for more details on the Zulip test runner.

• Once you’ve gotten your webhook working and passing a test, capture payloads for the other common typesof posts the service’s webhook will make, and add tests for them; usually this part of the process is pretty fast.Webhook integration tests should all use fixtures (as opposed to contacting the service), since otherwise the testscan’t run without Internet access and some sort of credentials for the service.

• Finally, write documentation for the integration; there’s a detailed guide below.

See the Hello World webhook Walkthrough below for a detailed look at how to write a simple webhook.

88 Chapter 20. Writing a new integration

Zulip Documentation, Release 1.4.0

20.3.1 Files that need to be created

Select a name for your webhook and use it consistently. The examples below are for a webhook named ‘MyWebHook’.

• static/images/integrations/logos/mywebhook.png: An image to represent your integrationin the user interface. Generally this Should be the logo of the platform/server/product you are integrating. SeeDocumenting your integration for details.

• static/images/integrations/mywebbook/001.png: A screen capture of your integration for usein the user interface. You can add as many images as needed to effectively document your webhook integration.See Documenting your integration for details.

• zerver/fixtures/mywebhook/mywebhook_messagetype.json: Sample json payload data usedby tests. Add one fixture file per type of message supported by your integration. See Testing and writing testsfor details.

• zerver/views/webhooks/mywebhook.py: Includes the main webhook integration function includingany needed helper functions.

20.3.2 Files that need to be updated

• templates/zerver/integrations.html: Edit to add end-user documentation. See Documentingyour integration for details.

• zerver/test_hooks.py: Edit to include tests for your webbook. See Testing and writing tests for details.

• zerver/lib/integrations.py: Add your integration to WEBHOOK_INTEGRATIONSto register it. This will automatically register a url for the webhook of the formapi/v1/external/mywebhook and associate with the function called api_mywebhook_webhook inzerver/views/webhooks/mywebhook.py.

20.4 Python script and plugin integrations

For plugin integrations, usually you will need to consult the documentation for the third party software in order tolearn how to write the integration. But we have a few notes on how to do these:

• You should always send messages by POSTing to URLs of the formhttps://zulip.example.com/v1/messages/, not the legacy /api/v1/send_messagemessage sending API.

• We usually build Python script integration with (at least) 2 files: ‘zulip_foo_config.py‘‘ containing the configu-ration for the integration including the bots’ API keys, plus a script that reads from this configuration to actuallydo the work (that way, it’s possible to update the script without breaking users’ configurations).

• Be sure to test your integration carefully and document how to install it (see notes on documentation below).

• You should specify a clear HTTP User-Agent for your integration. The user agent should at a minimum identifythe integration and version number, separated by a slash. If possible, you should collect platform informationand include that in ()s after the version number. Some examples of ideal UAs are:

ZulipDesktop/0.7.0 (Ubuntu; 14.04)ZulipJenkins/0.1.0 (Windows; 7.2)ZulipMobile/0.5.4 (Android; 4.2; maguro)

20.4. Python script and plugin integrations 89

Zulip Documentation, Release 1.4.0

20.5 Documenting your integration

Every Zulip integration must be documented in templates/zerver/integrations.html. Usually, this in-volves a few steps:

• Make sure you’ve added your integration to zerver/lib/integrations.py; this results in your integra-tion appearing on the /integrations page. You’ll need to add a logo image for your integration under thestatic/images/integrations/logos/<name>.png, where <name> is the name of the integration,all in lower case.

• Add an integration-instructions class block also in the alphabetically correct place, explaining allthe steps required to setup the integration, including what URLs to use, etc. If there are any screens in theproduct involved, take a few screenshots with the input fields filled out with sample values in order to make theinstructions really easy to follow. For the screenshots, use something like github-bot@example.com forthe email addresses and an obviously fake API key like abcdef123456790.

• Finally, generate a message sent by the integration and take a screenshot of the message to provide an examplemessage in the documentation. If your new integration is a webhook integration, you can generate such amessage from your test fixtures using send_webhook_fixture_message:

./manage.py send_webhook_fixture_message \--fixture=zerver/fixtures/pingdom/pingdom_imap_down_to_up.json \'--url=/api/v1/external/pingdom?stream=stream_name&api_key=api_key'

When generating the screenshot of a sample message, give your test bot a nice name like “GitHub Bot”, use theproject’s logo as the bot’s avatar, and take the screenshots showing the stream/topic bar for the message, not justthe message body.

When writing documentation for your integration, be sure to use the {{ external_api_uri }} templatevariable, so that your integration documentation will provide the correct URL for whatever server it is deployedon. If special configuration is required to set the SITE variable, you should document that too, inside an {% ifapi_site_required %} check.

20.6 Hello World webhook Walkthrough

Below explains each part of a simple webhook integration, called Hello World. This webhook sends a “hello” messageto the test stream and includes a link to the Wikipedia article of the day, which it formats from json data it receivesin the http request.

Use this walkthrough to learn how to write your first webhook integration.

20.6.1 Step 0: Create fixtures

The first step in creating a webhook is to examine the data that the service you want to integrate will be sending toZulip.

You can use requestb.in or a similar tool to capture webook payload(s) from the service you are integrating. Examiningthis data allows you to do two things:

1. Determine how you will need to structure your webook code, including what message types your integrationshould support and how; and,

2. Create fixtures for your webook tests.

90 Chapter 20. Writing a new integration

Zulip Documentation, Release 1.4.0

Fixtures enable the testing of webhook integration code without the need to actually contact the service being inte-grated.

Because Hello World is a very simple webhook that does one thing, it requires only one fixture,zerver/fixtures/helloworld/helloworld_hello.json:

{"featured_title":"Marilyn Monroe","featured_url":"https://en.wikipedia.org/wiki/Marilyn_Monroe",

}

When writing your own webhook integration, you’ll want to write a test function for each distinct message conditionyour webhook supports. You’ll also need a corresponding fixture for each of these tests. See Step 3: Create tests orTesting for further details.

20.6.2 Step 1: Create main webhook code

The majority of the code for your webhook integration will be in a single python file inzerver/views/webhooks/. The name of this file should be the name of your webhook, all lower-case,with file extension .py: mywebhook.py.

The Hello World integration is in zerver/views/webhooks/helloworld.py:

from __future__ import absolute_importfrom django.utils.translation import ugettext as _from zerver.lib.actions import check_send_messagefrom zerver.lib.response import json_success, json_errorfrom zerver.decorator import REQ, has_request_variables, api_key_only_webhook_viewfrom zerver.lib.validator import check_dict, check_string

from zerver.models import Client, UserProfile

from django.http import HttpRequest, HttpResponsefrom six import text_typefrom typing import Dict, Any, Iterable, Optional

@api_key_only_webhook_view('HelloWorld')@has_request_variablesdef api_helloworld_webhook(request, user_profile, client,

payload=REQ(argument_type='body'),stream=REQ(default='test'),topic=REQ(default='Hello World')):

# type: (HttpRequest, UserProfile, Client, Dict[str, Iterable[Dict[str, Any]]],→˓text_type, Optional[text_type]) -> HttpResponse

# construct the body of the messagebody = 'Hello! I am happy to be here! :smile:'

# try to add the Wikipedia article of the day# return appropriate error if not successfultry:

body_template = '\nThe Wikipedia featured article for today is **[{featured_→˓title}]({featured_url})**'

body += body_template.format(**payload)except KeyError as e:

return json_error(_("Missing key {} in JSON").format(str(e)))

20.6. Hello World webhook Walkthrough 91

Zulip Documentation, Release 1.4.0

# send the messagecheck_send_message(user_profile, client, 'stream', [stream], topic, body)

# return json resultreturn json_success()

The above code imports the required functions and defines the main webhook function api_helloworld_webook,decorating it with api_key_only_webhook_view and has_request_variables.

You must pass the name of your webhook to the api_key_only_webhook_view decorator. Here we have usedHelloWorld. To be consistent with Zulip code style, use the name of the product you are integrating in camel case,spelled as the product spells its own name (except always first letter upper-case).

You should name your webhook function as such api_webhookname_webhook where webhookname is thename of your webhook and is always lower-case.

At minimum, the webhook function must accept request (Django HttpRequest object), user_profile (Zulip’suser object), and client (Zulip’s analogue of UserAgent). You may also want to define additional parameters usingthe REQ object.

In the example above, we have defined payload which is populated from the body of the http request, stream witha default of test (available by default in Zulip dev environment), and topic with a default of Hello World.

The line that begins # type is a mypy type annotation. See this page for details about how to properly annotate yourwebhook functions.

In the body of the function we define the body of the message as Hello! I am happy to be here!:smile:. The :smile: indicates an emoji. Then we append a link to the Wikipedia article of the day as pro-vided by the json payload. If the json payload does not include data for featured_title and featured_urlwe catch a KeyError and use json_error to return the appropriate information: a 400 http status code withrelevant details.

Then we send a public (stream) message with check_send_message which will validate the message and thensend it.

Finally, we return a 200 http status with a JSON format success message via json_success().

20.6.3 Step 2: Create an api endpoint for the webhook

In order for a webhook to be externally available, it must be mapped to a url. This is done inzerver/lib/integrations.py.

Look for the lines beginning with:

WEBHOOK_INTEGRATIONS = [

And you’ll find the entry for Hello World:

WebhookIntegration('helloworld', display_name='Hello World'),

This tells the Zulip api to call the api_helloworld_webhook function inzerver/views/webhooks/helloworld.py when it receives a request at/api/v1/external/helloworld.

This line also tells Zulip to generate an entry for Hello World on the Zulip integrations page usingstatic/images/integrations/logos/helloworld.png as its icon.

At this point, if you’re following along and/or writing your own Hello World webhook, you have written enough codeto test your integration.

92 Chapter 20. Writing a new integration

Zulip Documentation, Release 1.4.0

You can do so by using Zulip itself or curl on the command line.

Using manage.py from within Zulip Dev environment:

(zulip-venv)vagrant@vagrant-ubuntu-trusty-64:/srv/zulip$./manage.py send_webhook_fixture_message \> --fixture=zerver/fixtures/helloworld/helloworld_hello.json \> '--url=http://localhost:9991/api/v1/external/helloworld?api_key=<api_key>'

After which you should see something similar to:

2016-07-07 15:06:59,187 INFO 127.0.0.1 POST 200 143ms (mem: 6ms/13) (md:→˓43ms/1) (db: 20ms/9q) (+start: 147ms) /api/v1/external/helloworld (helloworld-→˓bot@zulip.com via ZulipHelloWorldWebhook)

Using curl:

curl -X POST -H "Content-Type: application/json" -d '{ "featured_title":"Marilyn→˓Monroe", "featured_url":"https://en.wikipedia.org/wiki/Marilyn_Monroe" }' http://→˓localhost:9991/api/v1/external/helloworld\?api_key\=<api_key>

After which you should see:

{"msg":"","result":"success"}

Using either method will create a message in Zulip:

20.6.4 Step 3: Create tests

Every webhook integraton should have a corresponding test class in zerver/tests/test_hooks.py.

You should name the class <WebhookName>HookTests and this class should accept WebhookTestCase. Forour HelloWorld webhook, we name the test class HelloWorldHookTests:

class HelloWorldHookTests(WebhookTestCase):STREAM_NAME = 'test'URL_TEMPLATE = "/api/v1/external/helloworld?&api_key={api_key}"FIXTURE_DIR_NAME = 'helloworld'

# Note: Include a test function per each distinct message condition your→˓integration supports

def test_hello_message(self):# type: () -> Noneexpected_subject = u"Hello World";expected_message = u"Hello! I am happy to be here! :smile: \nThe Wikipedia

→˓featured article for today is **[Marilyn Monroe](https://en.wikipedia.org/wiki/→˓Marilyn_Monroe)**";

# use fixture named helloworld_helloself.send_and_test_stream_message('hello', expected_subject, expected_message,

content_type="application/x-www-form-→˓urlencoded")

20.6. Hello World webhook Walkthrough 93

Zulip Documentation, Release 1.4.0

def get_body(self, fixture_name):# type: (text_type) -> text_typereturn self.fixture_data("helloworld", fixture_name, file_type="json")

When writing tests for your webook, you’ll want to include one test function (and corresponding fixture) per eachdistinct message condition that your integration supports.

If, for example, we added support for sending a goodbye message to our Hello World webook, we would addanother test function to HelloWorldHookTests class called something like test_goodbye_message:

def test_goodbye_message(self):# type: () -> Noneexpected_subject = u"Hello World";expected_message = u"Hello! I am happy to be here! :smile:\nThe Wikipedia

→˓featured article for today is **[Goodbye](https://en.wikipedia.org/wiki/Goodbye)**";

# use fixture named helloworld_goodbyeself.send_and_test_stream_message('goodbye', expected_subject, expected_

→˓message,content_type="application/x-www-form-

→˓urlencoded")

As well as a new fixture helloworld_goodbye.json in zerver/fixtures/helloworld/:

{"featured_title":"Goodbye","featured_url":"https://en.wikipedia.org/wiki/Goodbye",

}

Once you have written some tests, you can run just these new tests from within the Zulip dev environment with thiscommand:

(zulip-venv)vagrant@vagrant-ubuntu-trusty-64:/srv/zulip$./tools/test-backend zerver.tests.test_hooks.HelloWorldHookTests

(Note: You must run the tests from /srv/zulip directory.)

You will see some script output and if all the tests have passed, you will see:

Running zerver.tests.test_hooks.HelloWorldHookTests.test_hello_messageDONE!

20.6.5 Step 4: Create documentation

Next, we add end-user documentation for our webhook integration to templates/zerver/integrations.html.

There are two parts to the end-user documentation on this page.

The first is a div with class integration-lozenge for each integration. This div shows the logo of yourwebhook, its name, and a link to its installation and usage instructions.

Because there is an entry for the Hello World webhook in WEBHOOK_INTEGRATIONS inzerver/lib/integratins.py, this div will be generated automatically.

The second part is a div with the webhook’s usage instructions:

94 Chapter 20. Writing a new integration

Zulip Documentation, Release 1.4.0

<div id="helloworld" class="integration-instructions">

<p>Learn how Zulip integrations work with this simple Hello World example!</p>

<p>The Hello World webhook will use the <code>test<code> stream, which iscreated by default in the Zulip dev environment. If you are runningZulip in production, you should make sure this stream exists.</p>

<p>Next, on your <a href="/#settings" target="_blank">Zulipsettings page</a>, create a Hello World bot. Construct the URL forthe Hello World bot using the API key and stream name:

<code>{{ external_api_uri }}/v1/external/helloworld?api_key=abcdefgh&amp;→˓stream=test</code>

</p>

<p>To trigger a notication using this webhook, use `send_webhook_fixture_message`→˓from the Zulip command line:</p>

<div class="codehilite"><pre>(zulip-venv)vagrant@vagrant-ubuntu-trusty-64:/srv/zulip$

./manage.py send_webhook_fixture_message \> --fixture=zerver/fixtures/helloworld/helloworld_hello.json \> '--url=http://localhost:9991/api/v1/external/helloworld?api_key=<api_key>'</pre>

</div>

<p>Or, use curl:</p><div class="codehilite">

<pre>curl -X POST -H "Content-Type: application/json" -d '{ "featured_title":→˓"Marilyn Monroe", "featured_url":"https://en.wikipedia.org/wiki/Marilyn_Monroe" }'→˓http://localhost:9991/api/v1/external/helloworld\?api_key\=<api_key></pre>

</div>

<p><b>Congratulations! You're done!</b><br /> Your messages may look like:</p>

<img class="screenshot" src="/static/images/integrations/helloworld/001.png" /></div>

These documentation blocks should fall alphabetically. For the integration-lozenge div this happens automat-ically when the html is generated. For the integration-instructions div, we have added the div between theblocks for Github and Hubot, respectively.

See Documenting your integration for further details, including how to easily create the message screenshot.

20.6.6 Step 5: Preparing a pull request to zulip/zulip

When you have finished your webhook integration and are ready for it to be available in the Zulip product, followthese steps to prepare your pull request:

1. Run tests including linters and ensure you have addressed any issues they report. See Testing for details.

2. Read through Code styles and conventions and take a look through your code to double-check that you’vefollowed Zulip’s guidelines.

3. Take a look at your git history to ensure your commits have been clear and logical (see Version Control fortips). If not, consider revising them with git rebase --interactive. For most webhooks, you’ll wantto squash your changes into a single commit and include a good, clear commit message.

4. Push code to your fork.

20.6. Hello World webhook Walkthrough 95

Zulip Documentation, Release 1.4.0

5. Submit a pull request to zulip/zulip.

If you would like feedback on your integration as you go, feel free to submit pull requests as you go, prefixing themwith [WIP].

96 Chapter 20. Writing a new integration

CHAPTER 21

Writing a new application feature

The changes needed to add a new feature will vary, of course, but this document provides a general outline of whatyou may need to do, as well as an example of the specific steps needed to add a new feature: adding a new option tothe application that is dynamically synced through the data system in real-time to all browsers the user may have open.

21.1 General Process in brief

21.1.1 Adding a field to the database

Update the model: The server accesses the underlying database in zerver/ models.py. Add a new field in theappropriate class.

Create and run the migration: To create and apply a migration, run:

./manage.py makemigrations

./manage.py migrate

Test your changes: Once you’ve run the migration, restart memcached on your development server(/etc/init.d/memcached restart) and then restart run-dev.py to avoid interacting with cached objects.

21.1.2 Backend changes

Database interaction: Add any necessary code for updating and interacting with the database inzerver/lib/actions.py. It should update the database and send an event announcing the change.

Application state: Modify the fetch_initial_state_data and apply_events functions inzerver/lib/actions.py to update the state based on the event you just created.

Backend implementation: Make any other modifications to the backend required for your change.

New views: Add any new application views to zerver/urls.py. This includes both views that serve HTML (newpages on Zulip) as well as new API endpoints that serve JSON-formatted data.

Testing: At the very least, add a test of your event data flowing through the system in test_events.py.

21.1.3 Frontend changes

JavaScript: Zulip’s JavaScript is located in the directory static/js/. The exact files you may need to changedepend on your feature. If you’ve added a new event that is sent to clients, be sure to add a handler for it tostatic/js/server_events.js.

97

Zulip Documentation, Release 1.4.0

CSS: The primary CSS file is static/styles/zulip.css. If your new feature requires UI changes, you mayneed to add additional CSS to this file.

Templates: The initial page structure is rendered via Jinja2 templates located in templates/zerver. ForJavaScript, Zulip uses Handlebars templates located in static/templates. Templates are precompiled as part ofthe build/deploy process.

Testing: There are two types of frontend tests: node-based unit tests and blackbox end-to-end tests. The blackboxtests are run in a headless browser using Casper.js and are located in frontend_tests/casper_tests/. Theunit tests use Node’s assert module are located in frontend_tests/node_tests/. For more information onwriting and running tests see the testing documentation.

21.2 Example Feature

This example describes the process of adding a new setting to Zulip: a flag that restricts inviting new users to adminsonly (the default behavior is that any user can invite other users). It is based on an actual Zulip feature, and you canreview the original commit in the Zulip git repo. (Note that Zulip has since been upgraded from Django 1.6 to 1.8, sothe migration format has changed.)

21.2.1 Update the model

First, update the database and model to store the new setting. Add a new boolean field, invite_by_admins_only,to the Realm model in zerver/models.py.

--- a/zerver/models.py+++ b/zerver/models.py@@ -139,6 +139,7 @@ class Realm(ModelReprMixin, models.Model):

restricted_to_domain = models.BooleanField(default=True) # type: boolinvite_required = models.BooleanField(default=False) # type: bool

+ invite_by_admins_only = models.BooleanField(default=False) # type: boolcreate_stream_by_admins_only = models.BooleanField(default=False) # type: boolmandatory_topics = models.BooleanField(default=False) # type: bool

21.2.2 Create the migration

Create the migration file: ./manage.py makemigrations. Make sure to commit the generated file to git:git add zerver/migrations/NNNN_realm_invite_by_admins_only.py (NNNN is a number thatis equal to the number of migrations.)

If you run into problems, the Django migration documentation is helpful.

21.2.3 Test your migration changes

Apply the migration: ./manage.py migrate

Output:

shell $ ./manage.py migrateOperations to perform:

Synchronize unmigrated apps: staticfiles, analytics, pipelineApply all migrations: zilencer, confirmation, sessions, guardian, zerver, sites,

→˓auth, contenttypesSynchronizing apps without migrations:

98 Chapter 21. Writing a new application feature

Zulip Documentation, Release 1.4.0

Creating tables...Running deferred SQL...

Installing custom SQL...Running migrations:

Rendering model states... DONEApplying zerver.0026_realm_invite_by_admins_only... OK

21.2.4 Handle database interactions

Next, we will move on to implementing the backend part of this feature. Like typical apps, we will need our backendto update the database and send some response to the client that made the request.

Beyond that, we need to orchestrate notifications to other clients (or other users, if you will) that our setting haschanged. Clients find out about settings through two closely related code paths. When a client first contacts the server,the server sends the client its initial state. Subsequently, clients subscribe to “events,” which can (among other things)indicate that settings have changed. For the backend piece, we will need our action to make a call to send_eventto send the event to clients that are active. We will also need to modify fetch_initial_state_data so thatfuture clients see the new changes.

Anyway, getting back to implementation details...

In zerver/lib/actions.py, create a new function named do_set_realm_invite_by_admins_only.This function will update the database and trigger an event to notify clients when this setting changes. In this casethere was an existing realm|update event type which was used for setting similar flags on the Realm model, soit was possible to add a new property to that event rather than creating a new one. The property name matches thedatabase field to make it easy to understand what it indicates.

The second argument to send_event is the list of users whose browser sessions should be notified. Depending onthe setting, this can be a single user (if the setting is a personal one, like time display format), only members in aparticular stream or all active users in a realm. :

# zerver/lib/actions.py

def do_set_realm_invite_by_admins_only(realm, invite_by_admins_only):realm.invite_by_admins_only = invite_by_admins_onlyrealm.save(update_fields=['invite_by_admins_only'])event = dict(type="realm",op="update",property='invite_by_admins_only',value=invite_by_admins_only,

)send_event(event, active_user_ids(realm))return {}

21.2.5 Update application state

You then need to add code that will handle the event and update the application state. Inzerver/lib/actions.py update the fetch_initial_state and apply_events functions. :

def fetch_initial_state_data(user_profile, event_types, queue_id):# ...state['realm_invite_by_admins_only'] = user_profile.realm.invite_by_admins_only`

21.2. Example Feature 99

Zulip Documentation, Release 1.4.0

In this case you don’t need to change apply_events because there is already code that will correctly handle therealm update event type: :

def apply_events(state, events, user_profile):for event in events:# ...elif event['type'] == 'realm':

field = 'realm_' + event['property']state[field] = event['value']

21.2.6 Add a new view

You then need to add a view for clients to access that will call the newly-added actions.py code to update thedatabase. This example feature adds a new parameter that should be sent to clients when the application loads and beaccessible via JavaScript, and there is already a view that does this for related flags: update_realm. So in this case,we can add out code to the existing view instead of creating a new one. :

# zerver/views/__init__.py

def home(request):# ...page_params = dict(# ...realm_invite_by_admins_only = register_ret['realm_invite_by_admins_only'],# ...

)

Since this feature also adds a checkbox to the admin page, and adds a new property the Realm model that can bemodified from there, you also need to make changes to the update_realm function in the same file: :

# zerver/views/__init__.py

def update_realm(request, user_profile,name=REQ(validator=check_string, default=None),restricted_to_domain=REQ(validator=check_bool, default=None),invite_by_admins_only=REQ(validator=check_bool,default=None)):

# ...

if invite_by_admins_only is not None andrealm.invite_by_admins_only != invite_by_admins_only:

do_set_realm_invite_by_admins_only(realm, invite_by_admins_only)data['invite_by_admins_only'] = invite_by_admins_only

Then make the required front end changes: in this case a checkbox needs to be added to the admin page (and its valueadded to the data sent back to server when a realm is updated) and the change event needs to be handled on the client.

To add the checkbox to the admin page, modify the relevant template,static/templates/admin_tab.handlebars (omitted here since it is relatively straightforward). Then addcode to handle changes to the new form control in static/js/admin.js. :

var url = "/json/realm";var new_invite_by_admins_only =

$("#id_realm_invite_by_admins_only").prop("checked");data[invite_by_admins_only] = JSON.stringify(new_invite_by_admins_only);

100 Chapter 21. Writing a new application feature

Zulip Documentation, Release 1.4.0

channel.patch({url: url,data: data,success: function (data) {# ...if (data.invite_by_admins_only) {

ui.report_success("New users must be invited by an admin!", invite_by_admins_→˓only_status);

} else {ui.report_success("Any user may now invite new users!", invite_by_admins_only_

→˓status);}# ...

}});

Finally, update server_events.js to handle related events coming from the server. :

# static/js/server_events.js

function dispatch_normal_event(event) {switch (event.type) {# ...case 'realm':

if (event.op === 'update' && event.property === 'invite_by_admins_only') {page_params.realm_invite_by_admins_only = event.value;

}}

}

Any code needed to update the UI should be placed in dispatch_normal_event callback (rather than thechannel.patch) function. This ensures the appropriate code will run even if the changes are made in anotherbrowser window. In this example most of the changes are on the backend, so no UI updates are required.

21.2. Example Feature 101

Zulip Documentation, Release 1.4.0

102 Chapter 21. Writing a new application feature

CHAPTER 22

Writing views in Zulip

22.1 What this covers

This page documents how views work in Zulip. You may want to read the new feature tutorial or the integration guide,and treat this as a reference.

If you have experience with Django, much of this will be familiar, but you may want to read about how REST requestsare dispatched, and how request authentication works.

This document supplements the new feature tutorial and the testing documentation.

22.2 What is a view?

A view in Zulip is everything that helps implement a server endpoint. Every path that the Zulip server supports (doesn’tshow a 404 page for) is a view. The obvious ones are those you can visit in your browser, for example /integrations,which shows the integration documentation. These paths show up in the address bar of the browser. There are otherviews that are only seen by software, namely the API views. They are used to build the various clients that Zulip has,namely the web client (which is also used by the desktop client) and the mobile clients.

22.3 Modifying urls.py

A view is anything with an entry in the appropriate urls.py, usually zproject/urls.py. Zulip views either serveHTML (pages for browsers) or JSON (data for Zulip clients on all platforms, custom bots, and integrations).

The format of the URL patterns in Django is documented here, and the Zulip specific details for these are discussed indetail in the life of a request doc.

We have two Zulip-specific conventions we use for internationalization and for our REST API, respectively.

22.4 Writing human-readable views

If you’re writing a new page for the website, make sure to add it to i18n_urls in zproject/urls.py

i18n_urls = [...

+ url(r'^quote-of-the-day/$', TemplateView.as_view(template_name='zerver/qotd.html→˓')),

103

Zulip Documentation, Release 1.4.0

+ url(r'^postcards/$', 'zerver.views.postcards'),]

As an example, if a request comes in for Spanish, language code es, the server path will be something like:es/features/.

22.4.1 Decorators used for webpage views

This section documents a few simple decorators that we use for webpage views, as an introduction to view decorators.

require_post:

@require_postdef accounts_register(request):

# type: (HttpRequest) -> HttpResponse

This decorator ensures that the requst was a POST–here, we’re checking that the registration submission page isrequested with a post, and inside the function, we’ll check the form data. If you request this page with GET, you’ll geta HTTP 405 METHOD NOT ALLOWED error.

zulip_login_required:

This decorator verifies that the browser is logged in (i.e. has a valid session cookie) before providing the view for thisroute, or redirects the browser to a login page. This is used in the root path (/) of the website for the web client. If arequest comes from a browser without a valid session cookie, they are redirected to a login page. It is a small fork ofDjango’s login_required, adding a few extra checks specific to Zulip.

@zulip_login_requireddef home(request):

# type: (HttpRequest) -> HttpResponse

22.4.2 Writing a template

Templates for the main website are found in templates/zerver.

22.5 Writing API REST endpoints

These are code-parseable views that take x-www-form-urlencoded or JSON request bodies, and return JSON-stringresponses. Almost all Zulip view code is in the implementations of API REST endpoints.

The REST API does authentication of the user through rest_dispatch, which is documented in detail atzerver/lib/rest.py. This method will authenticate the user either through a session token from a cookie on the browser,or from a base64 encoded email:api-key string given via HTTP Basic Auth for API clients.

>>> import requests>>> r = requests.get('https://api.github.com/user', auth=('hello@example.com',→˓'0123456789abcdeFGHIJKLmnopQRSTUV'))>>> r.status_code-> 200

104 Chapter 22. Writing views in Zulip

Zulip Documentation, Release 1.4.0

22.5.1 Request variables

Most API views will have some arguments that are passed as part of the request to control the behavior of the view. Inany well-engineered view, you need to write code to parse and validate that the arguments exist and have the correctform. For many applications, this leads to one of serveral bad outcomes:

• The code isn’t written, so arguments aren’t validated, leading to bugs and confusing error messages for users ofthe API.

• Every function starts with a long list of semi-redundant validation code, usually with highly inconsistent errormessages.

• Every view function comes with another function that does the validation that has the problems from the lastbullet point.

In Zulip, we solve this problem with a the special decorator called has_request_variables which allows adeveloper to declare the arguments a view function takes and validate their types all within the def line of the function.We like this framework because we have found it makes the validation code compact, readable, and convenientlylocated in the same place as the method it is validating arguments for.

Here’s an example:

from zerver.decorator import has_request_variables, REQ, JsonableError, \require_realm_admin

@require_realm_admin@has_request_variablesdef create_user_backend(request, user_profile, email=REQ(), password=REQ(),

full_name=REQ(), short_name=REQ()):# ... code here

You will notice the special REQ() in the keyword arguments to create_user_backend.has_request_variables parses the declared keyword arguments of the decorated function, and for eachthat has an instance of REQ as the default value, it extracts the HTTP parameter with that name from the request,parses it as JSON, and passes it to the function. It will return an nicely JSON formatted HTTP 400 error in the eventthat an argument is missing, doesn’t parse as JSON, or otherwise is invalid.

require_realm_admin is another decorator which checks the authorization of the given user_profile tomake sure it belongs to a realm administrator (and thus has permission to create a user); we show it here primarily toshow how has_request_variables should be the inner decorator.

The implementation of has_request_variables is documented in detail in zerver/lib/request.py)

REQ also helps us with request variable validation. For example:

• msg_ids = REQ(validator=check_list(check_int)) will check that the msg_ids HTTP pa-rameter is a list of integers, marshalled as JSON, and pass it into the function as the msg_ids Python keywordargument.

• streams_raw = REQ("subscriptions",validator=check_list(check_string)) willcheck that the “subscriptions” HTTP parameter is a list of strings, marshalled as JSON, and pass it into thefunction with the Python keyword argument streams_raw.

• message_id=REQ(converter=to_non_negative_int) will check that the message_id HTTPparameter is a string containing a non-negative integer (converter differs from validator in that it doesnot automatically marshall the input from JSON).

See zerver/lib/validator.py for more validators and their documentation.

22.5. Writing API REST endpoints 105

Zulip Documentation, Release 1.4.0

22.5.2 Deciding which HTTP verb to use

When writing a new API view, you should writing a view to do just one type of thing. Usually that’s either a read orwrite operation.

If you’re reading data, GET is the best option. Other read-only verbs are HEAD, which should be used for testing if aresource is available to be read with GET, without the expense of the full GET. OPTIONS is also read-only, and usedby clients to determine which HTTP verbs are available for a given path. This isn’t something you need to write, as ithappens automatically in the implementation of rest_dispatch–see zerver/lib/rest.py for more.

If you’re creating new data, try to figure out if the thing you are creating is uniquely identifiable. For example, if you’recreating a user, there’s only one user per email. If you can find a unique ID, you should use PUT for the view. If youwant to create the data multiple times for multiple requests (for example, requesting the send_message view multipletimes with the same content should send multiple messages), you should use POST.

If you’re updating existing data, use PATCH.

If you’re removing data, use DELETE.

22.5.3 Idempotency

When writing a new API endpoint, with the exception of things like sending messages, requests should be safe torepeat, without impacting the state of the server. This is idempotency.

You will often want to return an error if a request to change something would do nothing because the state is alreadyas desired, to make debugging Zulip clients easier. This means that the response for repeated requests may not be thesame, but the repeated requests won’t change the server more than once or cause unwanted side effects.

22.5.4 Making changes to the database

If the view does any modification to the database, that change is done in a helper function inzerver/lib/actions.py. Those functions are responsible for doing a complete update to the state of the server,which often entails both updating the database and sending any events to notify clients about the state change. Whenpossible, we prefer to design a clean boundary between the view function and the actions function is such that all userinput validation happens in the view code (i.e. all 400 type errors are thrown there), and the actions code is responsiblefor atomically executing the change (this is usually signalled by having the actions function have a name starting withdo_. So in most cases, errors in an actions function will be the result of an operational problem (e.g. lost connectionto the database) and lead to a 500 error. If an actions function is responsible for validation as well, it should have aname starting with check_.

For example, in zerver/views/init.py:

@require_realm_admin@has_request_variablesdef update_realm(request, user_profile, name=REQ(validator=check_string,→˓default=None), ...)):

# type: (HttpRequest, UserProfile, ...) -> HttpResponserealm = user_profile.realmdata = {} # type: Dict[str, Any]if name is not None and realm.name != name:

do_set_realm_name(realm, name)data['name'] = 'updated'

and in zerver/lib/actions.py:

106 Chapter 22. Writing views in Zulip

Zulip Documentation, Release 1.4.0

def do_set_realm_name(realm, name):# type: (Realm, text_type) -> Nonerealm.name = namerealm.save(update_fields=['name'])event = dict(

type="realm",op="update",property='name',value=name,

)send_event(event, active_user_ids(realm))

realm.save() actually saves the changes to the realm to the database, and send_event sends the event to activeclients belonging to the provided list of users (in this case, all altive users in the Zulip realm).

22.5.5 Calling from the web application

You should always use channel. to make an HTTP <method> call to the Zulip JSON API. As an example, instatic/js/admin.js

var url = "/json/realm";var data = {

name: JSON.stringify(new_name),}channel.patch({

url: url,data: data,success: function (response_data) {

if (response_data.name !== undefined) {ui.report_success(i18n.t("Name changed!"), name_status);

}...

22.5.6 Calling from an API client

Here’s how you might manually make a call from python:

payload = {'name': new_name}

# email and API keyapi_auth = ('hello@example.com', '0123456789abcdeFGHIJKLmnopQRSTUV')

r = requests.patch(SERVER_URL + 'api/v1/realm',data=json.dumps(payload),auth=api_auth,

)

This is simply an illustration; we recommend making use of the Zulip Python API bindings since they provide a niceinterface for accessing the API.

22.5. Writing API REST endpoints 107

Zulip Documentation, Release 1.4.0

22.6 Legacy endpoints used by the web client

New features should conform the REST API style. The legacy, web-only endpoints can’t effectively enforce usage ofa browser, so they aren’t preferable from a security perspective, and it is generally a good idea to make your featureavailable to other clients, especially the mobile clients.

These endpoints make use of some older authentication decorators, authenticated_json_api_view,authenticated_json_post_view, and authenticated_json_view, so you may see them in the code.

22.7 Webhook integration endpoints

Webhooks are called by other services, often to send a message as part of those services’ integrations. They are mostoften POST requests, and often there is very little you can customize about them. Usually you can expect that thewebhook for a service will allow specification for the target server for the webhook, and an API key.

If the webhook does not have an option to provide a bot email, use the api_key_only_webhook_view decorator,to fill in the user_profile and client fields of a request:

@api_key_only_webhook_view('PagerDuty')@has_request_variablesdef api_pagerduty_webhook(request, user_profile, client,

payload=REQ(argument_type='body'),stream=REQ(default='pagerduty'),topic=REQ(default=None)):

The client will be the result of get_client("ZulipPagerDutyWebhook") in this example.

108 Chapter 22. Writing views in Zulip

CHAPTER 23

Life of a Request

It can sometimes be confusing to figure out how to write a new feature, or debug an existing one. Let us try to followa request through the Zulip codebase, and dive deep into how each part works.

We will use as our example the creation of users through the API, but we will also highlight how alternative requestsare handled.

23.1 A request is sent to the server, and handled by Nginx

When Zulip is deployed in production, all requests go through nginx. For the most part we don’t need to know howthis works, except for when it isn’t working. Nginx does the first level of routing–deciding which application willserve the request (or deciding to serve the request itself for static content).

In development, tools/run-dev.py fills the role of nginx. Static files are in your git checkout under static,and are served unminified.

23.2 Nginx secures traffic with SSL

If you visit your Zulip server in your browser and discover that your traffic isn’t being properly encrypted, an nginxmisconfiguration is the likely culprit.

23.3 Static files are served directly by Nginx

Static files include JavaScript, css, static assets (like emoji, avatars), and user uploads (if stored locally and not on S3).

location /static/ {alias /home/zulip/prod-static/;error_page 404 /static/html/404.html;

}

23.4 Nginx routes other requests between tornado and django

All our connected clients hold open long-polling connections so that they can recieve events (messages, presencenotifications, and so on) in real-time. Events are served by Zulip’s tornado application.

109

Zulip Documentation, Release 1.4.0

Nearly every other kind of request is served by the zerver Django application.

Here is the relevant nginx routing configuration.

23.5 Django routes the request to a view in urls.py files

There are various urls.py files throughout the server codebase, which are covered in more detail in the directorystructure doc.

The main Zulip Django app is zerver. The routes are found in

zproject/urls.pyzproject/legacy_urls.py

There are HTML-serving, REST API, legacy, and webhook url patterns. We will look at how each of these types ofrequests are handled, and focus on how the REST API handles our user creation example.

23.6 Views serving HTML are internationalized by server path

If we look in zproject/urls.py, we can see something called i18n_urls. These urls show up in the address bar of thebrowser, and serve HTML.

For example, the /hello page (preview here) gets translated in Chinese at zh-cn/hello/ (preview here).

Note the zh-cn prefix–that url pattern gets added by i18n_patterns.

23.7 API endpoints use REST

Our example is a REST API endpoint. It’s a PUT to /users.

With the exception of Webhooks (which we do not usually control the format of), legacy endpoints, and logged-outendpoints, Zulip uses REST for its API. This means that we use:

• POST for creating something new where we don’t have a unique ID. Also used as a catch-all if no other verb isappropriate.

• PUT for creating something for which we have a unique ID.

• DELETE for deleting something

• PATCH for updating or editing attributes of something.

• GET to get something (read-only)

• HEAD to check the existence of something to GET, without getting it; useful to check a link without download-ing a potentially large link

• OPTIONS (handled automatically, see more below)

Of these, PUT, DELETE, HEAD, OPTIONS, and GET are idempotent, which means that we can send the requestmultiple times and get the same state on the server. You might get a different response after the first request, as we liketo give our clients an error so they know that no new change was made by the extra requests.

POST is not idempotent–if I send a message multiple times, Zulip will show my message multiple times. PATCH isspecial–it can be idempotent, and we like to write API endpoints in an idempotent fashion, as much as possible.

This cookbook and tutorial can be helpful if you are new to REST web applications.

110 Chapter 23. Life of a Request

Zulip Documentation, Release 1.4.0

23.7.1 PUT is only for creating new things

If you’re used to using PUT to update or modify resources, you might find our convention a little strange.

We use PUT to create resources with unique identifiers, POST to create resources without unique identifiers (likesending a message with the same content multiple times), and PATCH to modify resources.

In our example, create_user_backend uses PUT, because there’s a unique identifier, the user’s email.

23.7.2 OPTIONS

The OPTIONS method will yield the allowed methods.

This request: OPTIONS https://zulip.tabbott.net/api/v1/users yields a response with this HTTPheader: Allow: PUT,GET

We can see this reflected in zproject/urls.py:

url(r'^users$', 'zerver.lib.rest.rest_dispatch',{'GET': 'zerver.views.users.get_members_backend','PUT': 'zerver.views.users.create_user_backend'}),

In this way, the API is partially self-documenting.

23.7.3 Legacy endpoints are used by the web client

The endpoints from the legacy JSON API are written without REST in mind. They are used extensively by the webclient, and use POST.

You can see them in zproject/legacy_urls.py.

23.7.4 Webhook integrations may not be RESTful

Zulip endpoints that are called by other services for integrations have to conform to the service’s request format. Theyare likely to use only POST.

23.8 Django calls rest_dispatch for REST endpoints, and authenti-cates

For requests that correspond to a REST url pattern, Zulip configures its url patterns (see zerver/lib/rest.py) so thatthe action called is rest_dispatch. This method will authenticate the user, either through a session token from acookie, or from an email:api-key string given via HTTP Basic Auth for API clients.

It will then look up what HTTP verb was used (GET, POST, etc) to make the request, and then figure out which viewto show from that.

In our example,

{'GET': 'zerver.views.users.get_members_backend','PUT': 'zerver.views.users.create_user_backend'}

is supplied as an argument to rest_dispatch, along with the HTTPRequest. The requesthas the HTTP verb PUT, which rest_dispatch can use to find the correct view to show:zerver.views.users.create_user_backend.

23.8. Django calls rest_dispatch for REST endpoints, and authenticates 111

Zulip Documentation, Release 1.4.0

23.9 The view will authorize the user, extract request variables, andvalidate them

This is covered in good detail in the writing views doc

23.10 Results are given as JSON

Our API works on JSON requests and responses. Every API endpoint should return json_error in the case of anerror, which gives a JSON string:

{'result': 'error','msg': <some error message>}

in a HTTP Response with a content type of ‘application/json’.

To pass back data from the server to the calling client, in the event of a successfully handledrequest, we use json_success(data=<some python object which can be converted to aJSON string>.

This will result in a JSON string:

{'result': 'success','msg': '','data'='{'var_name1': 'var_value1','var_name2':'var_value2'...}

with a HTTP 200 status and a content type of ‘application/json’.

That’s it!

112 Chapter 23. Life of a Request

CHAPTER 24

Version control

24.1 Commit Discipline

We follow the Git project’s own commit discipline practice of “Each commit is a minimal coherent idea”. Thisdiscipline takes a bit of work, but it makes it much easier for code reviewers to spot bugs, and makes the commithistory a much more useful resource for developers trying to understand why the code works the way it does, whichalso helps a lot in preventing bugs.

Coherency requirements for any commit:

• It should pass tests (so test updates needed by a change should be in the same commit as the original change,not a separate “fix the tests that were broken by the last commit” commit).

• It should be safe to deploy individually, or comment in detail in the commit message as to why it isn’t (maybewith a [manual] tag). So implementing a new API endpoint in one commit and then adding the security checksin a future commit should be avoided – the security checks should be there from the beginning.

• Error handling should generally be included along with the code that might trigger the error.

• TODO comments should be in the commit that introduces the issue or functionality with further work required.

When you should be minimal:

• Significant refactorings should be done in a separate commit from functional changes.

• Moving code from one file to another should be done in a separate commits from functional changes or evenrefactoring within a file.

• 2 different refactorings should be done in different commits.

• 2 different features should be done in different commits.

• If you find yourself writing a commit message that reads like a list of somewhat dissimilar things that you did,you probably should have just done 2 commits.

When not to be overly minimal:

• For completely new features, you don’t necessarily need to split out new commits for each little subfeature ofthe new feature. E.g. if you’re writing a new tool from scratch, it’s fine to have the initial tool have plenty ofoptions/features without doing separate commits for each one. That said, reviewing a 2000-line giant blob ofnew code isn’t fun, so please be thoughtful about submitting things in reviewable units.

• Don’t bother to split back end commits from front end commits, even though the backend can often be coherenton its own.

Other considerations:

113

Zulip Documentation, Release 1.4.0

• Overly fine commits are easily squashed, but not vice versa, so err toward small commits, and the code reviewercan advise on squashing.

• If a commit you write doesn’t pass tests, you should usually fix that by amending the commit to fix the bug, notwriting a new “fix tests” commit on top of it.

Zulip expects you to structure the commits in your pull requests to form a clean history before we will merge them;it’s best to write your commits following these guidelines in the first place, but if you don’t, you can always fix yourhistory using git rebase -i.

It can take some practice to get used to writing your commits with a clean history so that you don’t spend much timedoing interactive rebases. For example, often you’ll start adding a feature, and discover you need to a refactoring part-way through writing the feature. When that happens, we recommend stashing your partial feature, do the refactoring,commit it, and then finish implementing your feature.

24.2 Commit Messages

• The first line of commit messages should be written in the imperative and be kept relatively short while conciselyexplaining what the commit does. For example:

Bad:

bugfixgather_subscriptions was brokenfix bug #234.

Good:

Fix gather_subscriptions throwing an exception when given bad input.

• Use present-tense action verbs in your commit messages.

Bad:

Fixing gather_subscriptions throwing an exception when given bad input.Fixed gather_subscriptions throwing an exception when given bad input.

Good:

Fix gather_subscriptions throwing an exception when given bad input.

• Please use a complete sentence in the summary, ending with a period.

• The rest of the commit message should be written in full prose and explain why and how the change was made. Ifthe commit makes performance improvements, you should generally include some rough benchmarks showingthat it actually improves the performance.

• When you fix a GitHub issue, mark that you’ve fixed the issue in your commit message so that the issue isautomatically closed when your code is merged. Zulip’s preferred style for this is to have the final paragraph ofthe commit message read e.g. “Fixes: #123.”

• Any paragraph content in the commit message should be line-wrapped to less than 76 characters per line, so thatyour commit message will be reasonably readable in git log in a normal terminal.

• In your commit message, you should describe any manual testing you did in addition to running the automatedtests, and any aspects of the commit that you think are questionable and you’d like special attention applied to.

114 Chapter 24. Version control

CHAPTER 25

Code style and conventions

25.1 Be consistent!

Look at the surrounding code, or a similar part of the project, and try to do the same thing. If you think the other codehas actively bad style, fix it (in a separate commit).

When in doubt, send an email to zulip-devel@googlegroups.com with your question.

25.2 Lint tools

You can run them all at once with

./tools/lint-all

You can set this up as a local Git commit hook with

``tools/setup-git-repo``

The Vagrant setup process runs this for you.

lint-all runs many lint checks in parallel, including

• JavaScript (JSLint)

tools/jslint/check-all.js contains a pretty fine-grained set of JSLint options, rule excep-tions, and allowed global variables. If you add a new global, you’ll need to add it to the list.

• Python (Pyflakes)

• templates

• Puppet configuration

• custom checks (e.g. trailing whitespace and spaces-not-tabs)

25.3 Secrets

Please don’t put any passwords, secret access keys, etc. inline in the code. Instead, use the get_secret function inzproject/settings.py to read secrets from /etc/zulip/secrets.conf.

115

Zulip Documentation, Release 1.4.0

25.4 Dangerous constructs

25.4.1 Misuse of database queries

Look out for Django code like this:

[Foo.objects.get(id=bar.x.id)for bar in Bar.objects.filter(...)if bar.baz < 7]

This will make one database query for each Bar, which is slow in production (but not in local testing!). Instead of alist comprehension, write a single query using Django’s QuerySet API.

If you can’t rewrite it as a single query, that’s a sign that something is wrong with the database schema. So don’t deferthis optimization when performing schema changes, or else you may later find that it’s impossible.

25.4.2 UserProfile.objects.get() / Client.objects.get / etc.

In our Django code, never do direct UserProfile.objects.get(email=foo) database queries. Insteadalways use get_user_profile_by_{email,id}. There are 3 reasons for this:

1. It’s guaranteed to correctly do a case-inexact lookup

2. It fetches the user object from remote cache, which is faster

3. It always fetches a UserProfile object which has been queried using .selected_related(), and thus will performwell when one later accesses related models like the Realm.

Similarly we have get_client and get_stream functions to fetch those commonly accessed objects via remotecache.

25.4.3 Using Django model objects as keys in sets/dicts

Don’t use Django model objects as keys in sets/dictionaries – you will get unexpected behavior when dealing withobjects obtained from different database queries:

For example, UserProfile.objects.only("id").get(id=17) in set([UserProfile.objects.get(id=17)])is False

You should work with the IDs instead.

25.4.4 user_profile.save()

You should always pass the update_fields keyword argument to .save() when modifying an existing Django modelobject. By default, .save() will overwrite every value in the column, which results in lots of race conditions whereunrelated changes made by one thread can be accidentally overwritten by another thread that fetched its UserProfileobject before the first thread wrote out its change.

25.4.5 Using raw saves to update important model objects

In most cases, we already have a function in zephyr/lib/actions.py with a name like do_activate_user that will correctlyhandle lookups, caching, and notifying running browsers via the event system about your change. So please checkwhether such a function exists before writing new code to modify a model object, since your new code has a goodchance of getting at least one of these things wrong.

116 Chapter 25. Code style and conventions

Zulip Documentation, Release 1.4.0

25.4.6 x.attr(’zid’) vs. rows.id(x)

Our message row DOM elements have a custom attribute zidwhich contains the numerical message ID. Don’t accessthis directly as x.attr('zid') ! The result will be a string and comparisons (e.g. with <=) will give the wrongresult, occasionally, just enough to make a bug that’s impossible to track down.

You should instead use the id function from the rows module, as in rows.id(x). This returns a number. Evenin cases where you do want a string, use the id function, as it will simplify future code changes. In most contexts inJavaScript where a string is needed, you can pass a number without any explicit conversion.

25.4.7 JavaScript var

Always declare JavaScript variables using var:

var x = ...;

In a function, var is necessary or else x will be a global variable. For variables declared at global scope, this has noeffect, but we do it for consistency.

JavaScript has function scope only, not block scope. This means that a var declaration inside a for or if acts thesame as a var declaration at the beginning of the surrounding function. To avoid confusion, declare all variablesat the top of a function.

25.4.8 JavaScript for (i in myArray)

Don’t use it: [1], [2], [3]

25.4.9 jQuery global state

Don’t mess with jQuery global state once the app has loaded. Code like this is very dangerous:

$.ajaxSetup({ async: false });$.get(...);$.ajaxSetup({ async: true });

jQuery and the browser are free to run other code while the request is pending, which could perform other Ajaxrequests with the altered settings.

Instead, switch to the more general $.ajax_ function, which can take options like async.

25.4.10 State and logs files

Do not write state and logs files inside the current working directory in the production environment. This will not howyou expect, because the current working directory for the app changes every time we do a deploy. Instead, hardcode apath in settings.py – see SERVER_LOG_PATH in settings.py for an example.

25.5 JS array/object manipulation

For generic functions that operate on arrays or JavaScript objects, you should generally use Underscore. We usedto use jQuery’s utility functions, but the Underscore equivalents are more consistent, better-behaved and offer morechoices.

25.5. JS array/object manipulation 117

Zulip Documentation, Release 1.4.0

A quick conversion table:

$.each → _.each (parameters to the callback reversed)$.inArray → _.indexOf (parameters reversed)$.grep → _.filter$.map → _.map$.extend → _.extend

There’s a subtle difference in the case of _.extend; it will replace attributes with undefined, whereas jQuery won’t:

$.extend({foo: 2}, {foo: undefined}); // yields {foo: 2}, BUT..._.extend({foo: 2}, {foo: undefined}); // yields {foo: undefined}!

Also, _.each does not let you break out of the iteration early by returning false, the way jQuery’s version does. Ifyou’re doing this, you probably want _.find, _.every, or _.any, rather than ‘each’.

Some Underscore functions have multiple names. You should always use the canonical name (given in large print inthe Underscore documentation), with the exception of _.any, which we prefer over the less clear ‘some’.

25.6 More arbitrary style things

25.6.1 General

Indentation is four space characters for Python, JS, CSS, and shell scripts. Indentation is two space characters forHTML templates.

We never use tabs anywhere in source code we write, but we have some third-party files which contain tabs.

Keep third-party static files under the directory zephyr/static/third/, with one subdirectory per third-partyproject.

We don’t have an absolute hard limit on line length, but we should avoid extremely long lines. A general guideline is:refactor stuff to get it under 85 characters, unless that makes the code a lot uglier, in which case it’s fine to go up to120 or so.

Whitespace guidelines:

• Put one space (or more for alignment) around binary arithmetic and equality operators.

• Put one space around each part of the ternary operator.

• Put one space between keywords like if and while and their associated open paren.

• Put one space between the closing paren for if and while-like constructs and the opening curly brace. Put thecurly brace on the same line unless doing otherwise improves readability.

• Put no space before or after the open paren for function calls and no space before the close paren for functioncalls.

• For the comma operator and colon operator in languages where it is used for inline dictionaries, put no spacebefore it and at least one space after. Only use more than one space for alignment.

25.6.2 JavaScript

Don’t use == and != because these operators perform type coercions, which can mask bugs. Always use === and!==.

End every statement with a semicolon.

118 Chapter 25. Code style and conventions

Zulip Documentation, Release 1.4.0

if statements with no braces are allowed, if the body is simple and its extent is abundantly clear from context andformatting.

Anonymous functions should have spaces before and after the argument list:

var x = function (foo, bar) { // ...

When calling a function with an anonymous function as an argument, use this style:

$.get('foo', function (data) {var x = ...;// ...

});

The inner function body is indented one level from the outer function call. The closing brace for the inner functionand the closing parenthesis for the outer call are together on the same line. This style isn’t necessarily appropriate forcalls with multiple anonymous functions or other arguments following them.

Use

$(function () { ...

rather than

$(document).ready(function () { ...

and combine adjacent on-ready functions, if they are logically related.

The best way to build complicated DOM elements is a Mustache template likezephyr/static/templates/message.handlebars. For simpler things you can use jQuery DOMbuilding APIs like so:

var new_tr = $('<tr />').attr('id', zephyr.id);

Passing a HTML string to jQuery is fine for simple hardcoded things:

foo.append('<p id="selected">foo</p>');

but avoid programmatically building complicated strings.

We used to favor attaching behaviors in templates like so:

<p onclick="select_zephyr({{id}})">

but there are some reasons to prefer attaching events using jQuery code:

• Potential huge performance gains by using delegated events where possible

• When calling a function from an onclick attribute, this is not bound to the element like you might think

• jQuery does event normalization

Either way, avoid complicated JavaScript code inside HTML attributes; call a helper function instead.

25.6.3 HTML / CSS

Don’t use the style= attribute. Instead, define logical classes and put your styles in external files such aszulip.css.

25.6. More arbitrary style things 119

Zulip Documentation, Release 1.4.0

Don’t use the tag name in a selector unless you have to. In other words, use .foo instead of span.foo. We shouldn’thave to care if the tag type changes in the future.

Don’t use inline event handlers (onclick=, etc. attributes). Instead, attach a jQuery event handler($('#foo').on('click',function () {...})) when the DOM is ready (inside a $(function (){...}) block).

Use this format when you have the same block applying to multiple CSS styles (separate lines for each selector):

selector1,selector2 {};

25.6.4 Python

• Scripts should start with #!/usr/bin/env python and not #/usr/bin/python (the right Python maynot be installed in /usr/bin) or #/usr/bin/env/python2.7 (bad for Python 3 compatibility). Don’tput a shebang line on a Python file unless it’s meaningful to run it as a script. (Some libraries can also be run asscripts, e.g. to run a test suite.)

• The first import in a file should be from __future__ import absolute_import, per PEP 328

• Put all imports together at the top of the file, absent a compelling reason to do otherwise.

• Unpacking sequences doesn’t require list brackets:

[x, y] = xs # unnecessaryx, y = xs # better

• For string formatting, use x % (y,) rather than x % y, to avoid ambiguity if y happens to be a tuple.

• When selecting by id, don’t use foo.pk when you mean foo.id. E.g.

recipient = Recipient(type_id=huddle.pk, type=Recipient.HUDDLE)

should be written as

recipient = Recipient(type_id=huddle.id, type=Recipient.HUDDLE)

in case we ever change the primary keys.

25.6.5 Tests

All significant new features should come with tests. See testing.

25.6.6 Third party code

When adding new third-party packages to our codebase, please include “[third]” at the beginning of the commitmessage. You don’t necessarily need to do this when patching third-party code that’s already in tree.

120 Chapter 25. Code style and conventions

CHAPTER 26

Testing and writing tests

26.1 Overview

Zulip has a full test suite that includes many components. The most important components are documented in depthin their own sections:

• Django: backend Python tests

• Casper: end-to-end UI tests

• Node: unit tests for JS front end code

• Linters

This document covers more general testing issues, such as how to run the entire test suite, how to troubleshoot databaseissues, how to manually test the front end, and how to plan for the future upgrade to Python3.

26.2 Running tests

Zulip tests must be run inside a Zulip development environment; if you’re using Vagrant, you will need to enter theVagrant environment before running the tests:

vagrant sshcd /srv/zulip

Then, to run the full Zulip test suite, do this:

./tools/test-all

This runs the linter (tools/lint-all) plus all of our test suites; they can all be run separately (just readtools/test-all to see them). You can also run individual tests which can save you a lot of time debugginga test failure, e.g.:

./tools/lint-all # Runs all the linters in parallel

./tools/test-backend zerver.tests.test_bugdown.BugdownTest.test_inline_youtube

./tools/test-js-with-casper 09-navigation.js

./tools/test-js-with-node utils.js

The above setup instructions include the first-time setup of test databases, but you may need to rebuild the test databaseoccasionally if you’re working on new database migrations. To do this, run:

121

Zulip Documentation, Release 1.4.0

./tools/do-destroy-rebuild-test-database

26.2.1 Possible testing issues

• When running the test suite, if you get an error like this:

sqlalchemy.exc.ProgrammingError: (ProgrammingError) function ts_match_locs_→˓array(unknown, text, tsquery) does not exist

LINE 2: ...ECT message_id, flags, subject, rendered_content, ts_match_l...^

. . . then you need to install tsearch-extras, described above. Afterwards, re-run the init*-db and thedo-destroy-rebuild*-database scripts.

• When building the development environment using Vagrant and the LXC provider, if you encounter permissionserrors, you may need to chown -R 1000:$(whoami) /path/to/zulip on the host before runningvagrant up in order to ensure that the synced directory has the correct owner during provision. This issuewill arise if you run id username on the host where username is the user running Vagrant and the outputis anything but 1000. This seems to be caused by Vagrant behavior; for more information, see the vagrant-lxcFAQ entry about shared folder permissions.

26.3 Schema and initial data changes

If you change the database schema or change the initial test data, you have to regenerate the pristine test database byrunning tools/do-destroy-rebuild-test-database.

26.4 Wiping the test databases

You should first try running: tools/do-destroy-rebuild-test-database

If that fails you should try to do:

sudo -u postgres psql> DROP DATABASE zulip_test;> DROP DATABASE zulip_test_template;

and then run tools/do-destroy-rebuild-test-database

26.4.1 Recreating the postgres cluster

warning

This is irreversible, so do it with care, and never do this anywhere in production.

If your postgres cluster (collection of databases) gets totally trashed permissions-wise, and you can’t otherwise repairit, you can recreate it. On Ubuntu:

sudo pg_dropcluster --stop 9.1 mainsudo pg_createcluster --locale=en_US.utf8 --start 9.1 main

122 Chapter 26. Testing and writing tests

Zulip Documentation, Release 1.4.0

26.5 Manual testing (local app + web browser)

26.5.1 Clearing the manual testing database

You can use:

./tools/do-destroy-rebuild-database

to drop the database on your development environment and repopulate your it with the Shakespeare characters andsome test messages between them. This is run automatically as part of the development environment setup process,but is occasionally useful when you want to return to a clean state for testing.

26.5.2 JavaScript manual testing

debug.js has some tools for profiling JavaScript code, including:

• ‘print_elapsed_time‘: Wrap a function with it to print the time that function takes to the JavaScript console.

• ‘IterationProfiler‘: Profile part of looping constructs (like a for loop or $.each). You mark sections of the iterationbody and the IterationProfiler will sum the costs of those sections over all iterations.

Chrome has a very good debugger and inspector in its developer tools. Firebug for Firefox is also pretty good. Theyboth have profilers, but Chrome’s is a sampling profiler while Firebug’s is an instrumenting profiler. Using them bothcan be helpful because they provide different information.

26.6 Python 3 Compatibility

Zulip is working on supporting Python 3, and all new code in Zulip should be Python 2+3 compatible. We haveconverted most of the codebase to be compatible with Python 3 using a suite of 2to3 conversion tools and somemanual work. In order to avoid regressions in that compatibility as we continue to develop new features in Zulip,we have a special tool, tools/check-py3, which checks all code for Python 3 syntactic compatibility by runninga subset of the automated migration tools and checking if they trigger any changes. tools/check-py3 is runautomatically in Zulip’s Travis CI tests (in the ‘static-analysis’ build) to avoid any regressions, but is not included intest-all since it is quite slow.

To run tools/check-py3, you need to install the modernize and future Python packages (which are includedin requirements/py3k.txt, which itself is included in requirements/dev.txt, so you probably alreadyhave these packages installed).

To run check-py3 on just the Python files in a particular directory, you can change the current working directory(e.g. cd zerver/) and run check-py3 from there.

26.5. Manual testing (local app + web browser) 123

Zulip Documentation, Release 1.4.0

124 Chapter 26. Testing and writing tests

CHAPTER 27

Python static type checker (mypy)

mypy is a compile-time static type checker for Python, allowing optional, gradual typing of Python code. Zulip isusing mypy’s Python 2 compatible syntax for type annotations, which means that type annotations are written insidecomments that start with # type:. Here’s a brief example of the mypy syntax we’re using in Zulip:

user_dict = {} # type: Dict[str, UserProfile]

def get_user_profile_by_email(email):# type: (str) -> UserProfile... # Actual code of the function here

You can learn more about it at:

• The mypy cheat sheet is the best resource for quickly understanding how to write the PEP 484 type annotationsused by mypy correctly.

• The Python 2 type annotation syntax spec in PEP 484

• Using mypy with Python 2 code

The mypy type checker is run automatically as part of Zulip’s Travis CI testing process in the ‘static-analysis’ build.

27.1 type_debug.py

zerver/lib/type_debug.py has a useful decorator print_types. It prints the types of the parameters ofthe decorated function and the return type whenever that function is called. This can help find out what parametertypes a function is supposed to accept, or if parameters with the wrong types are being passed to a function.

Here is an example using the interactive console:

>>> from zerver.lib.type_debug import print_types>>>>>> @print_types... def func(x, y):... return x + y...>>> func(1.0, 2)func(float, int) -> float3.0>>> func('a', 'b')func(str, str) -> str'ab'>>> func((1, 2), (3,))

125

Zulip Documentation, Release 1.4.0

func((int, int), (int,)) -> (int, int, int)(1, 2, 3)>>> func([1, 2, 3], [4, 5, 6, 7])func([int, ...], [int, ...]) -> [int, ...][1, 2, 3, 4, 5, 6, 7]

print_all prints the type of the first item of lists. So [int,...] represents a list whose first element’s type isint. Types of all items are not printed because a list can have many elements, which would make the output too large.

Similarly in dicts, one key’s type and the corresponding value’s type are printed. So {1: 'a',2: 'b',3:'c'} will be printed as {int: str,...}.

27.2 Zulip goals

Zulip is hoping to reach 100% of the codebase annotated with mypy static types, and then enforce that it stays thatway. Our current coverage is shown in Coveralls.

27.3 Installing mypy

If you installed Zulip’s development environment correctly, mypy should already be installed inside the Python 3virtualenv at zulip-py3-venv (mypy only supports Python 3). If it isn’t installed (e.g. because you haven’treprovisioned recently), you can run tools/install-mypy to install it.

27.4 Running mypy on Zulip’s code locally

To run mypy on Zulip’s python code, run the command:

tools/run-mypy

It will output errors in the same style as a compiler would. For example, if your code has a type error like this:

foo = 1foo = '1'

you’ll get an error like this:

test.py: note: In function "test":test.py:200: error: Incompatible types in assignment (expression has type "str",→˓variable has type "int")

If you need help interpreting or debugging mypy errors, please feel free to mention @sharmaeklavya2 or @timabbotton your pull request (or email zulip-devel@googlegroups.com) to get help; we’d love to both build a great trou-bleshooting guide in this doc and also help contribute improvements to error messages upstream.

Since mypy is a new tool under rapid development and occasionally makes breaking changes, Zulip is using a pinnedversion of mypy from its git repository rather than tracking the (older) latest mypy release on PyPI.

126 Chapter 27. Python static type checker (mypy)

Zulip Documentation, Release 1.4.0

27.5 Excluded files

Since several Python files in Zulip’s code don’t pass mypy’s checks (even for unannotated code) right now, a list offiles to be excluded from the check for CI is present in tools/run-mypy.

To run mypy on all Python files, ignoring the exclude list, you can pass the --all option to tools/run-mypy.

tools/run-mypy --all

If you type annotate some of those files so that they pass without errors, please remove them from the exclude list.

27.6 Mypy is there to find bugs in Zulip before they impact users

For the purposes of Zulip development, you can treat mypy like a much more powerful linter that can catch a widerange of bugs. If, after running tools/run-mypy on your Zulip branch, you get mypy errors, it’s important to getto the bottom of the issue, not just do something quick to silence the warnings. Possible explanations include:

• A bug in any new type annotations you added.

• A bug in the existing type annotations.

• A bug in Zulip!

• Some Zulip code is correct but confusingly reuses variables with different types.

• A bug in mypy (though this is increasingly rare as mypy is now fairly mature as a project).

Each explanation has its own solution, but in every case the result should be solving the mypy warning in a waythat makes the Zulip codebase better. If you need help understanding an issue, please feel free to mention @shar-maeklavya2 or @timabbott on the relevant pull request or issue on GitHub.

If you think you have found a bug in Zulip or mypy, inform the zulip developers by opening an issue on Zulip’s GitHubrepository or posting on zulip-devel. If it’s indeed a mypy bug, we can help with reporting it upstream.

27.7 Annotating strings

In Python 3, strings can have non-ASCII characters without any problems. Such characters are required to supportlanguages which use non-latin scripts like Japanese and Hindi. They are also needed to support special characters likemathematical symbols, musical symbols, etc. In Python 2, however, str generally doesn’t work well with non-ASCIIcharacters. That’s why unicode was introduced in Python 2.

But there are problems with the unicode and str system. Implicit conversions between str and unicode use theascii codec, which fails on strings containing non-ASCII characters. Such errors are hard to detect by people whoalways write in English. To minimize such implicit conversions, we should have a strict separation between str andunicode in Python 2. It might seem that using unicode everywhere will solve all problems, but unfortunately itdoesn’t. This is because some parts of the standard library and the Python language (like keyword argument unpacking)insist that parameters passed to them are str.

To make our code work correctly in Python 2, we have to identify strings which contain data which could come fromnon-ASCII sources like stream names, people’s names, domain names, content of messages, emails, etc. These stringsshould be unicode. We also have to identify strings which should be str like Exception names, attribute names,parameter names, etc.

Mypy can help with this. We just have to annotate each string as either str or unicode and mypy’s static typechecking will tell us if we are incorrectly mixing the two. However, unicode is not defined in Python 3. We want

27.5. Excluded files 127

Zulip Documentation, Release 1.4.0

our code to be Python 3 compatible in the future. This can be achieved using ‘six’, a Python 2 and 3 compatibilitylibrary.

six.text_type is defined as str in Python 3 and as unicode in Python 2. We’ll be using text_type (insteadof unicode) and str to annotate strings in Zulip’s code. We follow the style of doing from six importtext_type and using text_type for annotation instead of doing import six and using six.text_typefor annotation, because text_type is used so extensively for type annotations that we don’t need to be that verbose.

Sometimes you’ll find that you have to convert strings from one type to another. zerver/lib/str_utils.pyhas utility functions to help with that. It also has documentation (in docstrings) which explains the right way to usethem.

128 Chapter 27. Python static type checker (mypy)

CHAPTER 28

Settings system

The page documents the Zulip settings system, and hopefully should help you decide how to correctly implementnew settings you’re adding to Zulip. We have two types of administrative settings in Zulip: server settings (which areset via configuration files are apply to the whole Zulip installation), and realm settings (which are usually set via the/#administration page in the Zulip web application) and apply to a single Zulip realm/organization (which for mostZulip servers is the only realm on the server).

Philosophically, the goals of the settings system are to make it convenient for:

• Zulip server administrations to configure Zulip’s featureset for their server without needing to patch Zulip

• Realm administrators to configure settings for their organization independently without needing to talk with theserver administrator.

• Secrets (passwords, API keys, etc.) to be stored in a separate place from shareable configuration.

28.1 Server settings

Zulip uses the Django settings system, which means that the settings files are Python programs that set a lot of variableswith all-capital names like EMAIL_GATEWAY_PATTERN. You can access these anywhere in the Zulip Django codeusing e.g.:

from django.conf import settingsprint(settings.EMAIL_GATEWAY_PATTERN)

Additionally, if you need to access a Django setting in a shell script (or just on the command line for debugging), youcan use e.g.:

$ ./scripts/get-django-setting EMAIL_GATEWAY_PATTERN%s@localhost:9991

Zulip has separated those settings that we expect a system administrator to change (with nice documentation) from the~1000 lines of settings needed by the Zulip Django app. As a result, there are a few files involved in the Zulip settingsfor server administrations. In a production environment, we have:

• /etc/zulip/settings.py (the template is in the Zulip repo atzproject/prod_settings_template.py) is the main system administration facing settings filefor Zulip. It contains all the server-specific settings, such as how to send outgoing email, the hostname of thePostgres database, etc., but does not contain any secrets (e.g. passwords, secret API keys, cryptographic keys,etc.). The way we generally do settings that can be controlled with shell access to a Zulip server is to put adefault in zproject/settings.py, and then override it here.

129

Zulip Documentation, Release 1.4.0

• /etc/zulip/zulip-secrets.conf (generated by scripts/setup/generate-secrets.py aspart of installation) contains secrets used by the Zulip installation. These are read using the standardPython ConfigParser, and accessed in zproject/settings.py by the get_secret function. Allsecrets/API keys/etc. used by the Zulip Django application should be stored here, and read using theget_secret function in zproject/settings.py.

• zproject/settings.py is the main Django settings file for Zulip. It contains all the settingsthat are constant for all Zulip installations (e.g. configuration for logging, static assets, middleware,etc.), as well as default values for the settings the user would set in /etc/zulip/settings.py(you can look at the DEFAULT_SETTINGS dictionary to easily review the settings available).zproject/settings.py has a line from prod_settings import *, which has the effect of im-porting /etc/zulip/settings.py in a prod environment (via a symlink).

In a development environment, we have zproject/settings.py, and additionally:

• zproject/dev_settings.py has the settings for the Zulip development environment; it mostly just im-ports prod_settings_template.py.

• zproject/dev-secrets.conf replaces /etc/zulip/zulip-secrets.conf.

• zproject/test_settings.py has the (default) settings used for the Zulip tests (both backend andCasper), which are applied on top of the development environment settings.

When adding a new server setting to Zulip, you will typically add it in two or three places:

• In DEFAULT_SETTINGS in zproject/settings.py, with a default value for production environments.If the settings has a secret key, you’ll add a get_secret call in zproject/settings.py (and the userwill add the value when they configure the feature).

• In an appropriate section of zproject/prod_settings_template.py, with documentation in the com-ments explaining the settings’s purpose and effect.

• Possibly also zproject/dev_settings.py, if the desired value of the setting for Zulip development en-vironments is different from the default for production (and similarly for zproject/test_settings.py).

Most settings should be enabled in the development environment, to maximize convenience of testing all of Zulip’sfeatures; they should be enabled by default in production if we expect most Zulip sites to want those settings.

28.1.1 Testing non-default settings

You can write tests for settings using e.g. with self.settings(GOOGLE_CLIENT_ID=None). However,this only works for settings which are checked at runtime, not settings which are only accessed in initialization ofDjango (or Zulip) internals (e.g. DATABASES). See the Django docs on overriding settings in tests for more details.

28.2 Realm settings

Realm settings are preferred for any configuration that is a matter of organizational policy (as opposed to technicalcapabilities of the server). As a result, configuration options for user-facing functionality is almost always added as anew realm setting, not a server setting. The new feature tutorial documents the process for adding a new realm settingto Zulip.

So for example, the following server settings will eventually be replaced with realm settings:

• NAME_CHANGES_DISABLED

• INLINE_IMAGE_PREVIEW

• ENABLE_GRAVATAR

130 Chapter 28. Settings system

Zulip Documentation, Release 1.4.0

• Which authentication methods are allowed should probably appear in both places; in server settings indicatingthe capabilities of the server, and in the realm settings indicating which methods the realm administrator wantsto allow users to login with.

28.2. Realm settings 131

Zulip Documentation, Release 1.4.0

132 Chapter 28. Settings system

CHAPTER 29

Queue processors

Zulip uses RabbitMQ to manage a system of internal queues. These are used for a variety of purposes:

• Asynchronously doing expensive operations like sending email notifications which can take seconds per emailand thus would otherwise timeout when 100s are triggered at once (E.g. inviting a lot of new users to a realm).

• Asynchronously doing non-time-critical somewhat expensive operations like updating analytics tables (e.g.UserActivityInternal) which don’t have any immediate runtime effect.

• Communicating events to push to clients (browsers, etc.) from the main Zulip Django application process to theTornado-based events system. Example events might be that a new message was sent, a user has changed theirsubscriptions, etc.

• Processing mobile push notifications and email mirroring system messages.

• Processing various errors, frontend tracebacks, and slow database queries in a batched fashion.

• Doing markdown rendering for messages delivered to the Tornado via websockets.

Needless to say, the RabbitMQ-based queuing system is an important part of the overall Zulip architecture, since it’s incritical code paths for everything from signing up for account, to rendering messages, to delivering updates to clients.

We use the pika library to interface with RabbitMQ, using a simple custom integration defined inzerver/lib/queue.py.

29.1 Adding a new queue processor

To add a new queue processor:

• Define the processor in zerver/worker/queue_processors.py using the @assign_queue decora-tor; it’s pretty easy to get the template for an existing similar queue processor. This suffices to test your queueworker in the Zulip development environment (tools/run-dev.py will automatically restart the queue pro-cessors and start running your new queue processor code). You can also run a single queue processor manuallyusing e.g. ./manage.py process_queue --queue=user_activity.

• So that supervisord will known to run the queue processor in production, you will need to define a program entryfor it in servers/puppet/modules/zulip/files/supervisor/conf.d/zulip.conf and addit to the zulip-workers group further down in the file.

• For monitoring, you need to add a check that your worker is running to puppet/zulip/files/cron.d/rabbitmq-numconsumers if it’s a one-at-a-time consumer like user_activity_internal or a custom nagios checkif it is a bulk processor like slow_queries.

133

Zulip Documentation, Release 1.4.0

29.2 Publishing events into a queue

You can publish events to a RabbitMQ queue using the queue_json_publish function defined inzerver/lib/queue.py.

29.3 Clearing a RabbitMQ queue

If you need to clear a queue (delete all the events in it), run ./manage.py purge_queue <queue_name>, forexample:

./manage.py purge_queue user_activity

You can also use the amqp tools directly. Install amqp-tools from apt and then run:

amqp-delete-queue --username=zulip --password='...' --server=localhost \--queue=user_presence

with the RabbitMQ password from /etc/zulip/zulip-secrets.conf.

134 Chapter 29. Queue processors

CHAPTER 30

Unread counts and the pointer

When you’re using Zulip and you reload, or narrow to a stream, how does Zulip decide where to place you?

Conceptually, Zulip takes you to the place where you left off (e.g. the first unread message), not the most recentmessages, to facilitate reviewing all the discussions that happened while you were away from your computer. Thescroll position is then set to keep that message in view and away from both the top and bottom of the visible sectionof messages.

But there a lot of details around doing this right, and around counting unread messages. Here’s how Zulip currentlydecides which message to select, along with some notes on improvements we’d like to make to the model.

First a bit of terminology:

• “Narrowing” is the process of filtering to a particular subset of the messages the user has access to.

• The blue cursor box (the “pointer”) is around is called the “selected” message. Zulip ensures that the currentlyselected message is always in-view.

30.1 Pointer logic

30.1.1 Recipient bar: message you clicked

If you enter a narrow by clicking on a message group’s recipient bar (stream/topic or private message recipient list atthe top of a group of messages), Zulip will select the the message you clicked on. This provides a nice user experiencewhere you get to see the stuff near what you clicked on, and in fact the message you clicked on stays at exactly thesame scroll position in the window after the narrowing as it was at before.

30.1.2 Search or sidebar click: unread/recent matching narrow

If you instead narrow by clicking on something in the left sidebar or typing some terms into the search box, Zulipwill instead selected on the first unread message matching that narrow, or if there are none, the most recent messagesmatching that narrow. This provides the nice user experience of taking you to the start of the new stuff (with enoughmessages you’ev seen before still in view at the top to provide you with context), which is usually what you want.(When finding the “first unread message”, Zulip ignores unread messages in muted streams or in muted topics withinnon-muted streams.)

135

Zulip Documentation, Release 1.4.0

30.1.3 Unnarrow: previous sequence

When you unnarrow using e.g. the escape key, you will automatically be taken to the same message that was selectedin the home view before you narrowed, unless in the narrow you read new messages, in which case you will be jumpedforward to the first unread and non-muted message in the home view (or the bottom of the feed if there is none). Thismakes for a nice experience reading threads via the home view in sequence.

30.1.4 New home view: “high watermark”

When you open a new browser window or tab to the home view (a.k.a. the interleaved view you get if you visit /),Zulip will select the furthest down that your cursor has ever reached in the home view. Because of the logic aroundunnarrowing in the last bullet, this is usually just before the first unread message in the home view, but if you never goto the home view, or you leave messages unread on some streams in your home view, this can lag.

We plan to change this to automatically advance the pointer in a way similar to the unnarrow logic.

30.1.5 Narrow in a new tab: closest to pointer

When you load a new browser tab or window to a narrowed view, Zulip will select the message closest to your pointer,which is what you would have got had you loaded the browser window to your home view and then clicked on thenearest message matching your narrow (which might have been offscreen).

We plan to change this to match the Search/sidebar behavior.

30.1.6 Forced reload: state preservation

When the server forces a reload of a browser that’s otherwise caught up (which happens within 30 minutes when anew version of the server is deployed, usually at a type when the user isn’t looking at the browser), Zulip will preservethe state – what (if any) narrow the user was in, the selected message, and even exact scroll position!

For more on the user experience philosophy guiding these decisions, see the architectural overview.

30.2 Unread count logic

How does Zulip decide whether a message has been read by the user? The algorithm needs to correctly handle a rangeof ways people might use the product. The algorithm is as follows:

• Any message which is selected or above a message which is selected is marked as read. So messages are markedas read as you scroll down the keyboard when the pointer passes over them.

• If the whitspace at the very bottom of the feed is in view, all messages in view are marked as read.

These two simple rules, combined with the pointer logic above, end up matching user expectations well for whetherthe product should treat them as having read a set of messages (or not).

136 Chapter 30. Unread counts and the pointer

CHAPTER 31

Markdown implementation

Zulip has a special flavor of Markdown, currently called ‘bugdown’ after Zulip’s original name of “humbug”. Endusers are using Bugdown within the client, not original Markdown.

Zulip has two implementations of Bugdown. The first is based on Python-Markdown (zerver/lib/bugdown/)and is used to authoritatively render messages on the backend (and implements expensive features like querying theTwitter API to render tweets nicely). The other is in JavaScript, based on marked (static/js/echo.js), and isused to preview and locally echo messages the moment the sender hits enter, without waiting for round trip from theserver. The two implementations are tested for compatibility via zerver/tests/test_bugdown.py and thefixtures under zerver/fixtures/bugdown-data.json.

The JavaScript implementation knows which types of messages it can render correctly, and thus while there is code torerender messages based on the authoritative backend rendering (which would clause a change in the rendering visibleonly to the sender shortly after a message is sent), this should never happen, and whenever it does it is considereda bug. Instead, if the frontend doesn’t know how to correctly render a message, we simply won’t echo the messagefor the sender until it’s rendered by the backend. So for example, a message containing a link to Twitter will notbe rendered by the JavaScript implementation because it doesn’t support doing the 3rd party API queries required torender tweets nicely.

I should note that the below documentation is based on a comparison with original Markdown, not newer Markdownvariants like CommonMark.

31.1 Zulip’s Markdown philosophy

Markdown is great for group chat for the same reason it’s been successful in products ranging from blogs to wikis tobug trackers: it’s close enough to how people try to express themselves when writing plain text (e.g. emails) that ishelps more than getting in the way.

The main issue for using Markdown in instant messaging is that the Markdown standard syntax used in a lot ofwikis/blogs has nontrivial error rates, where the author needs to go back and edit the post to fix the formatting aftertyping it the first time. While that’s basically fine when writing a blog, it gets annoying very fast in a chat product; eventhough you can edit messages to fix formatting mistakes, you don’t want to be doing that often. There are basically 2types of error rates that are important for a product like Zulip:

• What fraction of the time, if you pasted a short technical email that you wrote to your team and passed it throughyour Markdown implementation, would you need to change the text of your email for it to render in a reasonableway? This is the “accidental Markdown syntax” problem, common with Markdown syntax like the italics syntaxinteracting with talking about char *s.

• What fraction of the time do users attempting to use a particular Markdown syntax actually succeed at doingso correctly? Syntax like required a blank line between text and the start of a bulleted list raise this figuresubstantially.

137

Zulip Documentation, Release 1.4.0

Both of these are minor issues for most products using Markdown, but they are major problems in the instant messagingcontext, because one can’t edit a message that has already been sent and users are generally writing quickly. Zulip’sMarkdown strategy is based on the principles of giving users the power they need to express complicated ideas in achat context while minimizing those two error rates.

31.2 Zulip’s Changes to Markdown

Below, we document the changes that Zulip has against stock Python-Markdown; some of the features we modify /disable may already be non-standard.

31.2.1 Basic syntax

• Enable ‘nl2br extension: this means one newline creates a line break (not paragraph break).

• Disable italics entirely. This resolves an issue where people were using * and _ and hitting it by mistake toooften. E.g. with stock Markdown You should use char * instead of void * there wouldtrigger italics.

• Allow only ** syntax for bold, not __ (easy to hit by mistake if discussing Python __init__ or something)

• Disable special use of \ to escape other syntax. Rendering \\ as \ was hugely controversial, but having noescape syntax is also controversial. We may revisit this. For now you can always put things in code blocks.

31.2.2 Lists

• Allow tacking a bulleted list or block quote onto the end of a paragraph, i.e. without a blank line before it

• Allow only * for bulleted lists, not + or - (previously created confusion with diff-style text sloppily not includedin a code block)

• Disable ordered list syntax: it automatically renumbers, which can be really confusing when sending a numberedlist across multiple messages.

31.2.3 Links

• Enable auto-linkification, both for http://... and guessing at things like t.co/foo.

• Force links to be absolute. [foo](google.com) will go to http://google.com, and nothttp://zulip.com/google.com which is the default behavior.

• Set target="_blank" and title=(the url) on every link tag so clicking always opens a new window

• Disable link-by-reference syntax, [foo][bar] ... [bar]: http://google.com

31.2.4 Code

• Enable fenced code block extension, with syntax highlighting

• Disable line-numbering within fenced code blocks – the <table> output confused our web client code.

138 Chapter 31. Markdown implementation

Zulip Documentation, Release 1.4.0

31.2.5 Other

• Disable headings, both # foo and == foo == syntax: they don’t make much sense for chat messages.

• Disabled images.

• Allow embedding any avatar as a tiny (list bullet size) image. This is used primarily by version control integra-tions.

• We added the ~~~ quote block quote syntax.

31.2. Zulip’s Changes to Markdown 139

Zulip Documentation, Release 1.4.0

140 Chapter 31. Markdown implementation

CHAPTER 32

Static asset pipeline

This page documents additional information that may be useful when developing new features for Zulip that requirefront-end changes. For a more general overview, see the new-feature-tutorial. The code-style documentation also hasrelevant information about how Zulip’s code is structured.

32.1 Primary build process

Most of the existing JS in Zulip is written in IIFE-wrapped modules, one per file in the static/js directory. Whenrunning Zulip in development mode, each file is loaded seperately. In production mode (and when creating a releasetarball using tools/build-release-tarball), JavaScript files are concatenated and minified.

If you add a new JavaScript file, it needs to be specified in the JS_SPECS dictionary defined in zproject/settings.py tobe included in the concatenated file.

32.2 Webpack/CommonJS modules

New JS written for Zulip can be written as CommonJS modules (bundled using webpack, though this will taken careof automatically whenever run-dev.py is running). (CommonJS is the same module format that Node uses, so seethe Node documentation for more information on the syntax.)

Benefits of using CommonJS modules over the IIFE module approach:

• namespacing/module boilerplate will be added automatically in the bundling process

• dependencies between modules are more explicit and easier to trace

• no separate list of JS files needs to be maintained for concatenation and minification

• third-party libraries can be more easily installed/versioned using npm

• running the same code in the browser and in Node for testing is simplified (as both environments use the samemodule syntax)

The entry point file for the bundle generated by webpack is static/js/src/main.js. Any modules you addwill need to be required from this file (or one of its dependencies) in order to be included in the script bundle.

32.3 Adding static files

To add a static file to the app (JavaScript, CSS, images, etc), first add it to the appropriate place under static/.

141

Zulip Documentation, Release 1.4.0

• Third-party files should all go in static/third/. Tag the commit with “[third]” when adding or modifyinga third-party package.

• Our own JS lives under static/js; CSS lives under static/styles.

• JavaScript and CSS files are combined and minified in production. In this case all you need to do is add thefilename to PIPELINE[‘STYLESHEET’] or JS_SPECS in zproject/settings.py. (If you plan to onlyuse the JS/CSS within the app proper, and not on the login page or other standalone pages, put it in the ‘app’category.)

If you want to test minified files in development, look for the PIPELINE_ENABLED = line inzproject/settings.py and set it to True – or just set DEBUG = False.

Note that static/html/{400,5xx}.html will only render properly if minification is enabled, since they hard-code the path static/min/portico.css.

142 Chapter 32. Static asset pipeline

CHAPTER 33

Schema Migrations

Zulip uses the standard Django system for doing schema migrations. There is some example usage in the new featuretutorial.

This page documents some important issues related to writing schema migrations.

• Large tables: For large tables like Message and UserMessage, you want to take precautions when addingcolumns to the table, performing data backfills, or building indexes. We have a zerver/lib/migrate.pylibrary to help with adding columns and backfilling data. For building indexes on these tables, we should do thisusing SQL with postgres’s CONCURRENTLY keyword.

• Numbering conflicts across branches: If you’ve done your schema change in a branch, and meanwhile anotherschema change has taken place, Django will now have two migrations with the same number. To fix this, youneed to renumber your migration(s), fix up the “dependencies” entries in your migration(s), and rewrite your githistory as needed. There is a tutorial here that walks you though that process.

143

Zulip Documentation, Release 1.4.0

144 Chapter 33. Schema Migrations

CHAPTER 34

HTML and CSS

34.1 Zulip CSS organization

The Zulip application’s CSS can be found in the static/styles/ directory. Zulip uses Bootstrap as its mainthird-party CSS library.

Zulip currently does not use any CSS preprocessors, and is organized into several files. For most pages, theCSS is combined into a single CSS file by the static asset pipeline, controlled by the PIPELINE_CSS code inzproject/settings.py.

The CSS files are:

• portico.css - Main CSS for logged-out pages

• pygments.css - CSS for Python syntax highlighting

• activity.css - CSS for the activity app

• fonts.css - Fonts for text in the Zulip app

• static/styles/thirdparty-fonts.css - Font Awesome (used for icons)

The CSS for the Zulip web application UI is primarily here:

• settings.css - CSS for the Zulip settings and administration pages

• zulip.css - CSS for the rest of the Zulip logged-in app

• media.css - CSS for media queries (particularly related to screen width)

We are in the process of splitting zulip.css into several more files; help with that project is very welcome!

34.2 Editing Zulip CSS

If you aren’t experienced with doing web development and want to make CSS changes, we recommend reading theexcellent Chrome web inspector guide on editing HTML/CSS, especially the section on CSS to learn about all the greattools that you can use to modify and test changes to CSS interactively in-browser (without even having the reload thepage!).

145

Zulip Documentation, Release 1.4.0

34.3 CSS Style guidelines

34.3.1 Avoid duplicated code

Without care, it’s easy for a web application to end up with thousands of lines of duplicated CSS code, which canmake it very difficult to understand the current styling or modify it. We would very much like to avoid such a fate. Soplease make an effort to reuse existing styling, clean up now-unused CSS, etc., to keep things maintainable.

34.3.2 Be consistent with existing similar UI

Ideally, do this by reusing existing CSS declarations, so that any improvements we make to the styling can improveall similar UI elements.

34.3.3 Use clear, unique names for classes and object IDs

This makes it much easier to read the code and use git grep to find where a particular class is used.

34.4 Validating CSS

When changing any part of the Zulip CSS, it’s important to check that the new CSS looks good at a wide range ofscreen widths, from very wide screen (e.g. 1920px) all the way down to narrow phone screens (e.g. 480px).

For complex changes, it’s definitely worth testing in a few different browsers to make sure things look the same.

146 Chapter 34. HTML and CSS

CHAPTER 35

Full-text search

Zulip supports full-text search, which can be combined arbitrarily with Zulip’s full suite of narrowing operators. Bydefault, it only supports English text, but there is an experimental PGroonga integration that provides full-text searchfor all languages.

The user interface and feature set for Zulip’s full-text search is documented in the “Search help” documentation sectionin the Zulip app’s gear menu.

35.1 The default full-text search implementation

Zulip’s uses PostgreSQL’s built-in full-text search feature, with a custom set of English stop words to improve thequality of the search results.

We use a small extension, tsearch_extras, for highlighting of the matching words. There is some discussion of remov-ing this extension, at least as an option, so that Zulip can be used with database-as-a-service platforms.

In order to optimize the performance of delivering messages, the full-text search index is updated for newlysent messages in the background, after the message has been delivered. This background updating is done bypuppet/zulip/files/postgresql/process_fts_updates, which is usually deployed on the databaseserver, but could be deployed on an application server instead.

35.2 An optional full-text search implementation

See the option PGroonga pull request for details on the status of the PGroonga integration.

147

Zulip Documentation, Release 1.4.0

148 Chapter 35. Full-text search

CHAPTER 36

Translating Zulip

Zulip has full support for unicode, so you can already use your preferred language everywhere in Zulip.

To make Zulip even better for users around the world, the Zulip UI is being translated into a number of major languages,including Spanish, German, French, Chinese, Russian, and Japanese, with varying levels of progress. If you speak alanguage other than English, your help with translating Zulip would be greatly appreciated!

If you’re interested in contributing translations to Zulip, join the Zulip project on Transifex and ask to join any lan-guages you’d like to contribute to (or add new ones). Transifex’s notification system sometimes fails to notify themaintainers when you ask to join a project, so please send a quick email to zulip-core@googlegroups.com when yourequest to join the project or add a language so that we can be sure to accept your request to contribute.

36.1 Setting Default Language in Zulip

Zulip allows you to set the default language through the settings page under ‘Display Settings’ section.

36.2 Translation Resource Files

All the translation magic happens through resource files which hold the translated text. Backend resource files arelocated at static/locale/<lang_code>/LC_MESSAGES/django.po, while frontend resource files are lo-cated at static/locale/<lang_code>/translations.json. These files are uploaded to Transifex usingtx push, where they can be translated. Once translated, they are downloaded back into the codebase using txpull.

36.3 Transifex Config

The config file that maps the resources from Zulip to Transifex is located at .tx/config. Django recognizes zh_CNinstead of zh-HANS for simplified Chinese language (this is fixed in Django 1.9). This idiosyncrasy is also handledin the Transifex config file.

36.4 Translation Process

The end-to-end process to get the translations working is as follows:

1. Mark the strings for translations (see sections for backend and frontend translations for details on this).

149

Zulip Documentation, Release 1.4.0

2. Create JSON formatted resource files using the python manage makemessages command. This com-mand will create a resource file called translations.json for frontend and django.po for backend forevery language under static/locale. The location for frontend resource file can be changed by passingan argument to the command (see the help for the command for further details). However, make sure that thelocation is publicly accessible since frontend files are loaded through XHR in the frontend which will only workwith publicly accessible resources.

The makemessages command is idempotent in that:

• It will only delete singular keys in the resource file when they are no longer used in Zulip code.

• It will only delete plural keys (see below for the documentation on plural translations) when the corre-sponding singular key is absent.

• It will not override the value of a singular key if that value contains a translated text.

3. Upload the resource files to Transifex using the tx push -s -a command.

4. Download the updated resource files from Transifex using the tx pull -a command. This command willdownload the resource files from Transifex and replace your local resource files with them.

36.5 Backend Translations

All user-facing text in the Zulip UI should be generated by an HTML template so that it can be translated.

Zulip uses two types of templates: backend templates (powered by the Jinja2 template engine, though the originalDjango template engine is still supported) and frontend templates (powered by Handlebars).

To mark a string for translation in the Jinja2 and Django template engines, you can use the _() function in thetemplates like this:

{{ _("English text") }}

If a string contains both a literal string component and variables, you can use a block translation, which makes useof placeholders to help translators to translate an entire sentence. To translate a block, Jinja2 uses the trans tag whileDjango uses the blocktrans tag. So rather than writing something ugly and confusing for translators like this:

# Don't do this!{{ _("This string will have") }} {{ value }} {{ _("inside") }}

You can instead use:

# Jinja2 style{% trans %}This string will have {{ value }} inside.{% endtrans %}# Django style{% blocktrans %}This string will have {{ value }} inside.{% endblocktrans %}

Zulip expects all the error messages to be translatable as well. To ensure this, the error message passed tojson_error and JsonableError should always be a literal string enclosed by _() function, e.g:

json_error(_('English Text'))JsonableError(_('English Text'))

To ensure we always internationalize our JSON errors messages, the Zulip linter (tools/lint-all) checks forcorrect usage.

150 Chapter 36. Translating Zulip

Zulip Documentation, Release 1.4.0

36.6 Frontend Translations

Zulip uses the i18next library for frontend translations. There are two types of files in Zulip frontend which can holdtranslatable strings, JavaScript code files and Handlebar templates. To mark a string translatable in JavaScript filespass it to the i18n.t function.

i18n.t('English Text', context);i18n.t('English text with a __variable__', {'variable': 'Variable value'});

Note: In the second example above, instead of enclosing the variable with handlebars, {{ }}, we enclose it with__ because we need to differentiate the variable from the Handlebar tags. The symbol which is used to enclose thevariables can be changed in /static/js/src/main.js.

i18next also supports plural translations. To support plurals make sure your resource file contatins the related keys:

{"en": {

"translation": {"key": "item","key_plural": "items","keyWithCount": "__count__ item","keyWithCount_plural": "__count__ items"

}}

}

With this resource you can show plurals like this:

i18n.t('key', {count: 0}); // output: 'items'i18n.t('key', {count: 1}); // output: 'item'i18n.t('key', {count: 5}); // output: 'items'i18n.t('key', {count: 100}); // output: 'items'i18n.t('keyWithCount', {count: 0}); // output: '0 items'i18n.t('keyWithCount', {count: 1}); // output: '1 item'i18n.t('keyWithCount', {count: 5}); // output: '5 items'i18n.t('keyWithCount', {count: 100}); // output: '100 items'

For further reading on plurals, read the official documentation.

To mark the strings as translatable in the Handlebar templates, Zulip registers two Handlebar helpers. The syntax forsimple strings is:

{{t 'English Text' }}

The syntax for block strings or strings containing variables is:

{{tr context}}Block of English text.

{{/tr}}

var context = {'variable': 'variable value'};{{tr context}}

Block of English text with a __variable__.{{/tr}}

The rules for plurals are same as for JavaScript files. You just have to declare the appropriate keys in the resource fileand then include the count in the context.

36.6. Frontend Translations 151

Zulip Documentation, Release 1.4.0

36.7 Testing Translations

First of all make sure that you have compiled the translation strings using python manage.pycompilemessages.

Django figures out the effective language by going through the following steps:

1. It looks for the language code in the url.

2. It looks for the LANGUGE_SESSION_KEY key in the current user’s session.

3. It looks for the cookie named ‘django_language’. You can set a different name through LAN-GUAGE_COOKIE_NAME setting.

4. It looks for the Accept-Language HTTP header in the HTTP request. Normally your browser will take careof this.

The easiest way to test translations is through the i18n urls e.g. if you have German translations available you canaccess the German version of a page by going to /de/path_to_page.

To test translations using other methods you will need an HTTP client library like requests, cURL or urllib.Here is a sample code to test Accept-Language header using requests:

import requestsheaders = {"Accept-Language": "de"}response = requests.get("http://localhost:9991/login/", headers=headers)print(response.content)

152 Chapter 36. Translating Zulip

CHAPTER 37

Logging and Performance Debugging

It’s good to have the terminal running run-dev.py up as you work since error messages including tracebacks alongwith every backend request will be printed there.

The messages will look similar to:

2016-05-20 14:50:22,056 INFO 127.0.0.1 GET 302 528ms (db: 1ms/1q)→˓(+start: 123ms) / (unauth via ?)[20/May/2016 14:50:22]"GET / HTTP/1.0" 302 02016-05-20 14:50:22,272 INFO 127.0.0.1 GET 200 124ms (db: 3ms/2q) /→˓login/ (unauth via ?)2016-05-20 14:50:26,333 INFO 127.0.0.1 POST 302 37ms (db: 6ms/7q) /→˓accounts/login/local/ (unauth via ?)[20/May/2016 14:50:26]"POST /accounts/login/local/ HTTP/1.0" 302 02016-05-20 14:50:26,538 INFO 127.0.0.1 GET 200 12ms (db: 1ms/2q)→˓(+start: 53ms) /api/v1/events [1463769771:0/0] (cordelia@zulip.com via internal)2016-05-20 14:50:26,657 INFO 127.0.0.1 GET 200 10ms (+start: 8ms) /api/→˓v1/events [1463769771:0/0] (cordelia@zulip.com via internal)2016-05-20 14:50:26,959 INFO 127.0.0.1 GET 200 588ms (db: 26ms/21q) /→˓[1463769771:0] (cordelia@zulip.com via website)

The format of this output is: timestamp, loglevel, IP, HTTP Method, HTTP status code, time to process, (optional perfdata details, e.g. database time/queries, memcached time/queries, Django process startup time, markdown processingtime, etc.), URL, and “email via client” showing user account involved (if logged in) and the type of client they used(“web”, “Android”, etc.).

153

Zulip Documentation, Release 1.4.0

154 Chapter 37. Logging and Performance Debugging

CHAPTER 38

Documentation

These docs are written in Commonmark Markdown with a small bit of rST. We’ve chosen Markdown because it iseasy to write. The docs are served in production at zulip.readthedocs.io.

If you want to build the documentation locally (e.g. to test your changes), the dependencies are automatically installedas part of Zulip development environment provisioning, and you can build the documentation using:

cd docs/make html

and then opening file:///path/to/zulip/docs/_build/html/index.html in your browser (you canalso use e.g. firefox docs/_build/html/index.html from the root of your Zulip checkout).

You can also usually test your changes by pushing a branch to GitHub and looking at the content on the GitHub webUI, since GitHub renders Markdown.

When editing dependencies for the Zulip documentation, you should edit requirements/docs.txt (which isused by ReadTheDocs to build the documentation quickly, without installing all of Zulip’s dependencies).

155

Zulip Documentation, Release 1.4.0

156 Chapter 38. Documentation

CHAPTER 39

Indices and tables

• genindex

• modindex

• search

157