Download - Introducing to Asynchronous Programming
Introducing to asynchronous programming
Oleksandr Fedorov• Software Engineer.• More than 5 years in Python.• 1.5 years of QA Automation.
E-mail: [email protected] ID: a.g.fedorof
Agenda1. The evolution of a loop.2. We will call you back.3. The Future is here.4. Await asynchronously.
BDD with Python
Part 1 - The evolution of a loop
The evolution of a loop
from anykey import get_key
def loop(): while True: key = get_key() if key is not None: break
if __name__ == '__main__': loop()
The evolution of a loop
Tasks: 183 total, 2 running, 181 sleeping, 0 stopped, 0 zombie
PID USER PR NI SHR S %CPU %MEM TIME+ COMMAND 1949 alex 20 0 4408 R 99,7 0,1 0:29.94 python 1602 alex 20 0 28484 S 3,7 1,2 6:20.05 Viber 29713 alex 20 0 79952 S 2,0 1,9 1:41.21 skype 640 root 20 0 23980 S 1,3 0,4 2:29.21 Xorg
The evolution of a loop
import time
from anykey import get_key
def loop(): while True: key = get_key() if key is not None: break time.sleep(0.5)
if __name__ == '__main__': loop()
The evolution of a loop – `select` example.
# twisted/internet/selectreactor.py# class SelectReactor... Kinda of.
def doSelect(self, timeout): r, w, ignored = _select(self._reads, self._writes, [], timeout)
for selectables, method in ((r, "doRead"), (w, "doWrite")): for selectable in selectables: self._doReadOrWrite(selectable, method)
The evolution of a loop. I/O loops.
# Our exampleloop() # Twisted
from twisted.internet import reactorreactor.run()
# Tornadoimport tornado.iolooptornado.ioloop.IOLoop.current().start()
# pygamewhile 1: for event in pygame.event.get(): if event.type == pygame.QUIT: sys.exit() pygame.display.flip()
# TkinterApp().mainloop()
# Asyncio import asyncio asyncio.get_event_loop().run_forever()
# JavaScript /* Nothing to do, it already runs */
Part 2 – We will call you back
We will call you back
def loop(): while True: key = get_key()
if key == 'q': break
if __name__ == '__main__': loop()
We will call you back
def loop(): while True: key = get_key()
if key == 'a': go_left() elif key == 'b': ... elif key == 'c': ... elif key == 'q': break
if __name__ == '__main__': loop()
def loop(callbacks): while True: key = get_key()
if key in callbacks: callbacks[key]() elif key == 'q': break
if __name__ == '__main__':
loop({ 'a': go_left, 'b': ..., 'c': ..., })
We will call you back - jQuery
$.ajax({ url: "page.html", success: function(){ alert("done"); }});
$.ajax({ statusCode: { 404: function() { alert( "page not found" ); } }});
We will call you back - Tkinter
import tkinter as tk
class Application(tk.Frame): def __init__(self, master=None): ... self.createWidgets()
def createWidgets(self): self.hi_there = tk.Button(self) self.hi_there["text"] = "click me" self.hi_there["command"] = self.say_hi
self.QUIT = tk.Button(self, text="QUIT", fg="red", command=root.destroy)
def say_hi(self): print("hi there, everyone!")
root = tk.Tk()app = Application(master=root)app.mainloop()
Part 3 – The Future is here
The Future is here
Future (aka Deferred, aka Promise)
• Composite of callbacks.• Callbacks can be added in both "pending" and "done" states.• The result passed to a future is propagated to its callbacks.• It's possible to propagate errors (exceptions).
The Future is here – Simple example
class Future: _nope = object()
def __init__(self): self._callbacks = [] self._result = self._nope
def add_callback(self, function, *args, **kwargs): future = Future() self._callbacks.append( (function, args, kwargs, future))
if self._result != self._nope: self.done(self._result) return future
def done(self, result=None): self._result = result
while self._callbacks: function, args, kwargs, future = \ self._callbacks.pop()
func_result = function( result, *args, **kwargs) future.done(func_result)
def callback1(result): print('Callback 1 gets:', result) return 'callback1_result'
def callback2(result): print('Callback 2 gets:', result) return 'callback2_result'
def callback_n(result, n=1): print('Callback {} gets:'.format(n), result) return 'callback_n_result'
future = Future()new_future = future.add_callback(callback1)\ .add_callback(callback2)
future.done('Initial data')new_future.add_callback(callback_n, n=3)
Callback 1 gets: Initial dataCallback 2 gets: callback1_resultCallback 3 gets: callback2_result
The Future is here – Promise in JavaScript
/* A function that returns a promise */function readFile(filename, enc){ return new Promise(function (fulfill, reject){ fs.readFile(filename, enc, function (err, res){ if (err) reject(err); else fulfill(res); }); });}
/* Adding callbacks */readFile(filename, 'utf8').then( function (res){ alert('Done: ' + res); }, function (reason){ alert('Error: ' + reason); });
readFile(filename, 'utf8').then(...).then(...).then(...);
The Future is here – Deferred in Twisted
from twisted.internet import defer
def get_data(): d = defer.Deferred() d.callback(3) return d
def print_result(res): print res raise AssertionError('Catch me!')
def handle_exception(failure): failure.trap(AssertionError) sys.stderr.write(str(failure))
d = get_data()d.addCallback(print_data)d.addErrback(handle_exception)
The Future is here – Future in asyncio(not to be confused with concurrent.futures.Future)
import asyncio
def get_data(): future = asyncio.Future() future.set_result('Future is done!') return future
def print_result(future): print(future.result()) loop.stop()
loop = asyncio.get_event_loop()future = asyncio.ensure_future(get_data())future.add_done_callback(print_result)
try: loop.run_forever()finally: loop.close()
The Future is here
Problems
• Spaghetti code.• The execution flow is not clear.• Errors handling is harder.
https://xkcd.com/338/
Part 4 – Await asynchronously.
Await asynchronously – General coroutine
def coroutine(data):
print('Passed 1:', data) res1 = yield data + 1 print('Passed 2:', res1)
res2 = yield res1 + 1 print('Passed 3:', res2)
Passed 1: 1Got 1: 2Passed 2: 20Got 2: 21
cor = coroutine(1)
val1 = next(cor)print('Got 1:', val1)
val2= cor.send(val1 * 10)
print('Got 2:', val2)
Await asynchronously – General coroutine
def coroutine(data):
print('Passed 1:', data) res1 = yield data + 1 print('Passed 2:', res1)
yield res1 + 1
class Coroutine: def __init__(self, data): self._data = data self._state = 0
def __next__(self): self.send(None)
def send(self, res):
if self._state == 0:
if res is not None:
raise TypeError(
"can't send non-None value to "
"a just-started generator")
print('Passed 1:', self._data)
return self._data + 1
elif self._state == 1:
print('Passed 2:', res)
return res + 1
else:
raise StopIteration
self._state += 1
Await asynchronously – General asyncio coroutine
@coroutinedef my_coroutine(*args, **kwargs): future1 = run_some_coroutine() res1 = yield future1 print("First result:", res1)
res2 = yield run_some_coroutine(res1) print("Second result:", res2) return res2 + 1
res_future = my_coroutine()loop = asyncio.get_event_loop()loop.run_until_complete(res_future)print('Total result', res_future.result())
First result: 10Second result: 20Total result 21
Await asynchronously – General asyncio coroutine
@coroutinedef my_coroutine(*args): res1 = yield future1 res2 = yield future2 return res2
res_future = my_coroutine()print(res_future.result())
def coroutine(func): def outer(*args, **kwargs): cor = func(*args, **kwargs) future1 = next(cor)
res_future = asyncio.Future()
future1.add_done_callback( partial(rewind_future_callback, cor=cor, res_future=res_future))
return res_future return outer
12
3
3
4
Await asynchronously – General asyncio coroutine
@coroutinedef my_coroutine(*args): res1 = yield future1 res2 = yield future2 return res2
res_future = my_coroutine()print(res_future.result())
def rewind_future_callback( future1, cor, res_future):
res1 = future1.result()
try: coroutine_result = cor.send(res1) except StopIteration as ex: res_future.set_result(ex.value) else: coroutine_result.add_done_callback( partial(rewind_future_callback, cor=cor, res_future=res_future))
1
2
3
4
Await asynchronously – General asyncio coroutine
@coroutinedef my_coroutine(*args): res1 = yield from future1 res2 = yield from future2 return res2
async def my_coroutine(*args): res1 = await future1 res2 = await future2 return res2
Await asynchronously – Some rules
async def my_coroutine(*args): res = await get_data()
1. Use async def to create a coroutine.2. Use await to get real data from a future, coroutine or a task.
async def do_stuff(): try: return await moar_stuff() except SomeException: return None
3. Handle exceptions as usual.
class ClassWithCoroutines: def __init__(self, loop): self._loop = loop
4. Keep your loop around.
async with asyncfile() as file: async for line in file: print(line)
5. Use async with and async for. They are awesome.
Thank you!