ast: threats and opportunities

ASTOpportunities and Threats

ITGM, Dec 2016

RDD

TDD

Slap this sh*t & release

ITGM, Dec 2016

Code Test

Test

Test

ITGM, Dec 2016

Code Test

Test

Test

def f(a,b): return a+b

5 == f(2,3)

0 == f(2,-2)

1 == f(1,0)

ITGM, Dec 2016

Code Test

Test

Test

def f(a,b): return a+b

5 == f(2,3)

0 == f(2,-2)

1 == f(1,0)

Test 1e15 == f(1,1e15-1)

ITGM, Dec 2016

DIS

ITGM, Dec 2016

>>> def foo(): ... a = 2 ... b = 3 ... return a + b ...

ITGM, Dec 2016

>>> import dis >>> dis.dis(foo) 2 0 LOAD_CONST 1 (2) 3 STORE_FAST 0 (a)

3 6 LOAD_CONST 2 (3) 9 STORE_FAST 1 (b)

4 12 LOAD_FAST 0 (a) 15 LOAD_FAST 1 (b) 18 BINARY_ADD 19 RETURN_VALUE

ITGM, Dec 2016

ITGM, Dec 2016

ITGM, Dec 2016

Zephyr ASDLParser/Python.asdl

ITGM, Dec 2016

Python/Python-ast.cimport _ast

ITGM, Dec 2016

Lib/ast.pyimport ast

ITGM, Dec 2016

• AST

• Add, And, Assert …

• NodeVisitor

• NodeTransformer

ITGM, Dec 2016

class DepthVisitor(ast.NodeVisitor): depth = 0

def generic_visit(self, node): parent = node.parent if parent: while parent: self.depth += 1 parent = parent.parent ast.NodeVisitor.generic_visit(self, node)

ITGM, Dec 2016

• parse

• copy_location

• fix_missing_locations

• dump

• increment_lineno

ITGM, Dec 2016

Модификация AST

ITGM, Dec 2016

class HackedImporter: def load_module(self, name): # do magic stuff return module

sys.path_hook.insert(0, HackedImporter)

ITGM, Dec 2016

https://github.com/alifanov/ast-spbpython-example

ITGM, Dec 2016

class ReturnIncrement(ast.NodeTransformer): def visit_Return(self, node): node.value = (ast.BinOp( left=node.value, op=ast.Add(), right=ast.Num(n=1) ) return node

ITGM, Dec 2016

def transform_with(transformer): def transform_decorator(func): node = func_to_node(func) node = transformer().visit(node) node = ast.fix_missing_locations(node) func = node_to_func(node, func) return func return transform_decorator

ITGM, Dec 2016

@transform_with(ReturnIncrementer) def f(a,b): return a+b

f(2,2) # 5

ITGM, Dec 2016

Threats

• Debug glitch

• Errors with multiple applies

• Strange errors

• Compiler version dependence

ITGM, Dec 2016

>>> parseprint("func(a, b=c, *d, **e)") # Python 3.4 Module(body=[ Expr(value=Call(func=Name(id='func', ctx=Load()), args=[Name(id='a', ctx=Load())], keywords=[keyword(arg='b', value=Name(id='c', ctx=Load()))], starargs=Name(id='d', ctx=Load()), # gone in 3.5 kwargs=Name(id='e', ctx=Load()))), # gone in 3.5 ])

>>> parseprint("func(a, b=c, *d, **e)") # Python 3.5 Module(body=[ Expr(value=Call(func=Name(id='func', ctx=Load()), args=[ Name(id='a', ctx=Load()), Starred(value=Name(id='d', ctx=Load()), ctx=Load()) # new in 3.5 ], keywords=[ keyword(arg='b', value=Name(id='c', ctx=Load())), keyword(arg=None, value=Name(id='e', ctx=Load())) # new in 3.5 ])) ])

ITGM, Dec 2016

Opportunities• Constant propagation

• Autologger

• Autoprofiler

• Macros (karnickel)

• Extend language

• MacroPy

• PonyORM

• Linters (auto codereview)ITGM, Dec 2016

>>> from macropy.tracing import macros, trace >>> trace[[x*2 for x in range(3)]] range(3) -> [0, 1, 2] x*2 -> 0 x*2 -> 2 x*2 -> 4 x*2 for x in range(3) -> [0, 2, 4] [0, 2, 4]

Macros

ITGM, Dec 2016

welcome = gettext("Welcome, {}!").format(user_name)

welcome = gettext("Welcome, {}!".format(user_name))

Linters

ITGM, Dec 2016

https://github.com/edx/edx-lint/blob/master/edx_lint/pylint/i18n_check.py

Linters

ITGM, Dec 2016

TRANSLATION_FUNCTIONS = set([ '_', 'gettext', 'ngettext', 'ngettext_lazy', 'npgettext', 'npgettext_lazy', 'pgettext', 'pgettext_lazy', 'ugettext', 'ugettext_lazy', 'ugettext_noop', 'ungettext', 'ungettext_lazy', ])

def visit_callfunc(self, node): if not isinstance(node.func, astroid.Name): # It isn't a simple name, can't deduce what function it is. return

if node.func.name not in self.TRANSLATION_FUNCTIONS: # Not a function we care about. return

if not self.linter.is_message_enabled(self.MESSAGE_ID): return

first = node.args[0] if isinstance(first, astroid.Const): if isinstance(first.value, basestring): # The first argument is a constant string! All is well! return

# Bad! self.add_message(self.MESSAGE_ID, args=node.func.name, node=node)

ITGM, Dec 2016

Tools• astdump

• astviewer

• ast_tool_box

• astroid

• baron

• redbaron

• astor

• meta

• astmonkey

ITGM, Dec 2016

Recommendation unit tests system

ITGM, Dec 2016

Pythoscope

ITGM, Dec 2016

Smother

ITGM, Dec 2016

app.views:add,app/tests.py::FunctionTest::test_add app.views:fact,app/tests.py::FunctionTest::test_fact app.views:hello,app/tests.py::FunctionTest::test_hello

ITGM, Dec 2016

Code1 Test1

Code1 Test2

Code1 Test3

Code2 Test1

Code2 Test2

ITGM, Dec 2016

def test_default_ordering(self): class OrderingListView(generics.ListAPIView): queryset = OrderingFilterModel.objects.all() serializer_class = OrderingFilterSerializer filter_backends = (filters.OrderingFilter,) ordering = ('title',) ordering_fields = ('text',)

view = OrderingListView.as_view() request = factory.get('') response = view(request) self.assertEqual( response.data, [ {'id': 3, 'title': 'xwv', 'text': 'cde'}, {'id': 2, 'title': 'yxw', 'text': 'bcd'}, {'id': 1, 'title': 'zyx', 'text': 'abc'}, ] ) # not in context

ITGM, Dec 2016

def fact(a): if a == 0: return 1 return a * fact(a - 1)

def test_fact(self): self.assertTrue(fact(0) == 1) self.assertTrue(fact(1) == 1) self.assertTrue(fact(2) == 2) self.assertTrue(fact(3) == 6) self.assertTrue(fact(4) == 24)

ITGM, Dec 2016

github.com

ITGM, Dec 2016

https://www.cosc.canterbury.ac.nz/research/reports/HonsReps/2015/hons_1508.pdf

ITGM, Dec 2016

ITGM, Dec 2016

ITGM, Dec 2016

ITGM, Dec 2016

ITGM, Dec 2016

ITGM, Dec 2016

ITGM, Dec 2016

Edges

ITGM, Dec 2016

BinOpLeftNum

ITGM, Dec 2016

Child Edge Parent Freq

Num left BinOp 12

Num value Assign 20

ITGM, Dec 2016

Depth

ITGM, Dec 2016

Builtin function

ITGM, Dec 2016

https://github.com/alifanov/testedio

ITGM, Dec 2016

Roadmap

ITGM, Dec 2016

Improve searchmodules names as features

ITGM, Dec 2016

Import github projectsunit-test monitoring

ITGM, Dec 2016

Fuzzy recommendations

hypothesis*pylint plugin

ITGM, Dec 2016

Performance optimizationmap -> list comprehension

*pylint plugin

ITGM, Dec 2016

Other languagesJavaScript

ITGM, Dec 2016

Real TDDtests -> code generation

ITGM, Dec 2016

Linkshttps://github.com/mkwiatkowski/pythoscope http://smother.readthedocs.io/en/latest/index.html http://testmon.org/ https://www.cosc.canterbury.ac.nz/research/reports/HonsReps/2015/hons_1508.pdf http://is.ifmo.ru/diploma-theses/2013/bachelor/rost/rost.pdf https://habrahabr.ru/post/65944/ http://pyke.sourceforge.net/

ITGM, Dec 2016

Contacts

Telegram: @jetbootsmaker

Alexander Lifanov

ITGM, Dec 2016