Sunday, 14 June 2009

Python standard library in Native Client

The Python standard library now works under Native Client in the web browser, including the Sqlite extension module.

By that I mean that it is possible to import modules from the standard library, but a lot of system calls won't be available. Sqlite works for in-memory use.

Here's a screenshot of the browser-based Python REPL:

The changes needed to make this work were quite minor:

  • Implementing stat() in glibc so that it does an RPC to the Javascript running on the web page. Python uses stat() when searching for modules in sys.path.
  • Changing fstat() in NaCl so that it doesn't return a fixed inode number. Python uses st_ino to determine whether an extension module is the same as a previously-loaded module, which doesn't work if st_ino is the same for different files.
  • Some tweaks to nacl-glibc-gcc, a wrapper script for nacl-gcc.
  • Ensuring that NaCl-glibc's libpthread.so works.
I didn't have to modify Python or Sqlite at all. I didn't even have to work out what arguments to pass to Sqlite's configure script - the Debian package for Sqlite builds without modification. It's just a matter of setting PATH to override gcc to run nacl-glibc-gcc.

13 comments:

kripken said...

Very nice!

Reid Kleckner said...

It's too bad this won't work with unladen-swallow since it JIT compiles native code.

Brett said...

Great work, Mark!

Mark Seaborn said...

Unladen Swallow could work under Native Client but whether it's worthwhile to use would depend on how fine-grained the JIT compilation is.

If it's a one-off compilation when a module is loaded, that should work fine. The compiler would present the generated code to NaCl, which would make a copy and run the validator on it before mapping it as executable. The dynamic loading support in my branch of NaCl already has a validate-and-map-code operation.

JITting smaller fragments of code repeatedly might not work so well because there would be more calls to the validate-and-map-code operation, which has some overhead. To avoid wasting memory you'd probably want this operation to work at finer granularity than a page (which is 64k on Windows rather than 4k). Deallocating code could be tricky - when reallocating a page in the multithreaded case NaCl would have to make sure that no thread is currently executing instructions on that page.

Daedius said...

That's great! Python applets here we come! =)

rgz said...

The sweet!

Reuben said...

ITYM "the Ubuntu package for sqlite", not "the Debian package for sqlite", given the link...

Mark Seaborn said...

@Reuben: You could call it a small-d debian package. It uses Debian packaging and builds with "dpkg-buildpackage", even if it's not made specifically for Debian's distro.

Christopher said...

Oh nice!

Hugh Isaacs II said...

Being that Native Client has SDL I can't wait to see PyGame ported over.

Flavio Coelho said...

Has anyone tryed to run some benchmarks to see how much performance is lost by running python under NaCl?

Ben McDonald said...

Hey Mark,

Where would I find the code to compile Python under Native Client? I would like to use the code for this project https://chrome.google.com/webstore/detail/gdiimmpmdoofmahingpgabiikimjgcia hosted on Google code.

Mark Seaborn said...

@Ben: There isn't currently a port of Python to run under NaCl in Chromium, as far as I know. This blog post is quite old, and it predates Chrome's PPAPI interface, NaCl's x86-64 sandbox, the newer nacl-glibc port, etc. - and I haven't got around to refreshing the Python port to work with all of that. I'd suggest asking on native-client-discuss.