Programming Forums > Script Programming > Python

Arevos · Dec 21st, 2005, 3:38 PM

I've been looking into generating Python bytecode from a Python program and this is what I've come up with:

def write_pyc(code, filename):
   "Write code to a PYC file"
   file = open(filename, 'wb')
   
   # PYC header specifying version and timestamp
   file.write(imp.get_magic())
   file.write(struct.pack('<l', time.time()))
   
   marshal.dump(code, file)

   file.close()


def compile_module(parse_tree, filename):
   "Compile a raw AST module"
   parse_tree.filename = filename
   return compiler.pycodegen.ModuleCodeGenerator(parse_tree).getCode()

The compile_module function is used to compile an AST (Abstract Syntax Tree) into a code object. The write_pyc function can write a compiled code object to a standard pyc file.

How do you create an AST? A good way to get a feel of Python ASTs is to get Python to generate them for you from existing files:

(Toggle Plain Text)

print compiler.parseFile(py_filename)

You can also check out the table of AST node types that are listed here

Here's an AST I constructed myself:

(Toggle Plain Text)

module = ast.Module(
   None,
   ast.Stmt(
      [ast.Printnl(
         [ast.CallFunc(
            ast.Name('pow'), [ast.Const(2), ast.Const(2)], None, None
         )],
         None
      )]
   )
)

This AST corresponds to the Python code: "print pow(2, 2)".

To compile this AST and write it to a file:

(Toggle Plain Text)

filename = "four.pyc"
write_pyc(compile_module(module, filename), filename)

The bytecode can then be executed using:

(Toggle Plain Text)

python four.pyc

If all goes well, the number 4 should be printed to the screen.

Nebula · Dec 21st, 2005, 3:54 PM

Python is interpreted, isn't it?

Arevos · Dec 21st, 2005, 3:59 PM

Quote:

Originally Posted by Nebula

Python is interpreted, isn't it?

Not exactly. It's compiled on the fly into a custom bytecode format, and then the bytecode is executed. Bytecode from imported modules is cached in pyc files, so that each module doesn't have to be parsed each time it is run.

All this makes Python much faster than a purely interpreted language.

Dameon · Dec 21st, 2005, 4:03 PM

Pretty much like C# or Java.

Kaja Fumei · Jan 27th, 2006, 3:41 PM

Quote:

Originally Posted by Dameon

Pretty much like C# or Java.

But without the JIT.

A couple days ago, I started building the compiler for my virtual machine project (so sick of AST's right now) and when it's all set up, I'm going to make a basic python front end (probably not a full front end right away but some basic features) and so I can test python vs. mine.

hydroxide · Jan 28th, 2006, 4:24 AM

Quote:

Originally Posted by Kaja Fumei

But without the JIT.

A couple days ago, I started building the compiler for my virtual machine project (so sick of AST's right now) and when it's all set up, I'm going to make a basic python front end (probably not a full front end right away but some basic features) and so I can test python vs. mine.

psyco is a JIT for x86. Jython and IronPython can take advantage of NET/Java JITters - IronPython is currently achieving better than CPython speeds on things (but it's unfinished so it may slow down when it actually has to run the complete testsuite). If you're interested in Python+VM you may like to look into PyPy which aims to do all kinds of interesting things. There will be new AST code in Py2.5 which should make manipulating them easier.

-T.

Dec 21st, 2005, 3:38 PM	#1
Arevos Programming Guru Join Date: Aug 2005 Location: England Posts: 1,499 Rep Power: 4	Hacking the Python compiler I've been looking into generating Python bytecode from a Python program and this is what I've come up with: (Toggle Plain Text) def write_pyc(code, filename): "Write code to a PYC file" file = open(filename, 'wb') # PYC header specifying version and timestamp file.write(imp.get_magic()) file.write(struct.pack('<l', time.time())) marshal.dump(code, file) file.close() def compile_module(parse_tree, filename): "Compile a raw AST module" parse_tree.filename = filename return compiler.pycodegen.ModuleCodeGenerator(parse_tree).getCode() def write_pyc(code, filename): "Write code to a PYC file" file = open(filename, 'wb') # PYC header specifying version and timestamp file.write(imp.get_magic()) file.write(struct.pack('<l', time.time())) marshal.dump(code, file) file.close() def compile_module(parse_tree, filename): "Compile a raw AST module" parse_tree.filename = filename return compiler.pycodegen.ModuleCodeGenerator(parse_tree).getCode() The compile_module function is used to compile an AST (Abstract Syntax Tree) into a code object. The write_pyc function can write a compiled code object to a standard pyc file. How do you create an AST? A good way to get a feel of Python ASTs is to get Python to generate them for you from existing files: (Toggle Plain Text) print compiler.parseFile(py_filename) print compiler.parseFile(py_filename) You can also check out the table of AST node types that are listed here Here's an AST I constructed myself: (Toggle Plain Text) module = ast.Module( None, ast.Stmt( [ast.Printnl( [ast.CallFunc( ast.Name('pow'), [ast.Const(2), ast.Const(2)], None, None )], None )] ) ) module = ast.Module( None, ast.Stmt( [ast.Printnl( [ast.CallFunc( ast.Name('pow'), [ast.Const(2), ast.Const(2)], None, None )], None )] ) ) This AST corresponds to the Python code: "print pow(2, 2)". To compile this AST and write it to a file: (Toggle Plain Text) filename = "four.pyc" write_pyc(compile_module(module, filename), filename) filename = "four.pyc" write_pyc(compile_module(module, filename), filename) The bytecode can then be executed using: (Toggle Plain Text) python four.pyc python four.pyc If all goes well, the number 4 should be printed to the screen.

Dec 21st, 2005, 3:54 PM	#2
Nebula Hobbyist Programmer Join Date: Oct 2005 Posts: 193 Rep Power: 3	Python is interpreted, isn't it? __________________ When will Jesus bring the porkchops?

Dec 21st, 2005, 4:03 PM	#4
Dameon Troll Join Date: Apr 2005 Location: Texas Posts: 730 Rep Power: 4	Pretty much like C# or Java. __________________ MD5(sig) = bcef75433db02e9ad9bf81d6f7c5c270

Thread Tools
Show Printable Version Email this Page
Display Modes
Linear Mode Switch to Hybrid Mode Switch to Threaded Mode

Currently Active Users Viewing This Thread: 1 (0 members and 1 guests)

Programming Forums > Script Programming > Python

Hacking the Python compiler