Sunday 14 June 2009

Python standard library in Native Client

The Python standard library now works under Native Client in the web browser, including the Sqlite extension module.

By that I mean that it is possible to import modules from the standard library, but a lot of system calls won't be available. Sqlite works for in-memory use.

Here's a screenshot of the browser-based Python REPL:

The changes needed to make this work were quite minor:

  • Implementing stat() in glibc so that it does an RPC to the Javascript running on the web page. Python uses stat() when searching for modules in sys.path.
  • Changing fstat() in NaCl so that it doesn't return a fixed inode number. Python uses st_ino to determine whether an extension module is the same as a previously-loaded module, which doesn't work if st_ino is the same for different files.
  • Some tweaks to nacl-glibc-gcc, a wrapper script for nacl-gcc.
  • Ensuring that NaCl-glibc's libpthread.so works.
I didn't have to modify Python or Sqlite at all. I didn't even have to work out what arguments to pass to Sqlite's configure script - the Debian package for Sqlite builds without modification. It's just a matter of setting PATH to override gcc to run nacl-glibc-gcc.
at

12 comments:

  1. kripken14 June 2009 at 18:22

    Very nice!

    ReplyDelete
  2. Reid K14 June 2009 at 23:36

    It's too bad this won't work with unladen-swallow since it JIT compiles native code.

    ReplyDelete
  3. Unknown15 June 2009 at 03:45

    Great work, Mark!

    ReplyDelete
  4. Mark Seaborn15 June 2009 at 12:55

    Unladen Swallow could work under Native Client but whether it's worthwhile to use would depend on how fine-grained the JIT compilation is.

    If it's a one-off compilation when a module is loaded, that should work fine. The compiler would present the generated code to NaCl, which would make a copy and run the validator on it before mapping it as executable. The dynamic loading support in my branch of NaCl already has a validate-and-map-code operation.

    JITting smaller fragments of code repeatedly might not work so well because there would be more calls to the validate-and-map-code operation, which has some overhead. To avoid wasting memory you'd probably want this operation to work at finer granularity than a page (which is 64k on Windows rather than 4k). Deallocating code could be tricky - when reallocating a page in the multithreaded case NaCl would have to make sure that no thread is currently executing instructions on that page.

    ReplyDelete
  5. Daedius16 June 2009 at 12:17

    That's great! Python applets here we come! =)

    ReplyDelete
  6. rgz18 June 2009 at 17:19

    The sweet!

    ReplyDelete
  7. Reuben Thomas18 June 2009 at 18:09

    ITYM "the Ubuntu package for sqlite", not "the Debian package for sqlite", given the link...

    ReplyDelete
  8. Mark Seaborn18 June 2009 at 18:45

    @Reuben: You could call it a small-d debian package. It uses Debian packaging and builds with "dpkg-buildpackage", even if it's not made specifically for Debian's distro.

    ReplyDelete
  9. Christopher3 July 2009 at 09:35

    Oh nice!

    ReplyDelete
  10. Hugh25 July 2009 at 03:46

    Being that Native Client has SDL I can't wait to see PyGame ported over.

    ReplyDelete
  11. Unknown27 July 2009 at 13:00

    Has anyone tryed to run some benchmarks to see how much performance is lost by running python under NaCl?

    ReplyDelete
  12. Mark Seaborn17 November 2011 at 08:40

    @Ben: There isn't currently a port of Python to run under NaCl in Chromium, as far as I know. This blog post is quite old, and it predates Chrome's PPAPI interface, NaCl's x86-64 sandbox, the newer nacl-glibc port, etc. - and I haven't got around to refreshing the Python port to work with all of that. I'd suggest asking on native-client-discuss.

    ReplyDelete