kernel_Notes/Zim/Programme/python/IPython.txt

Content-Type: text/x-zim-wiki
Wiki-Format: zim 0.4
Creation-Date: 2012-01-06T14:57:10+08:00

====== IPython ======
Created Friday 06 January 2012
http://ipython.org/ipython-doc/rel-0.12/overview.html

===== Introduction =====

==== Overview ====
One of Python’s most useful features is __its interactive interpreter__. This system allows very fast __testing of ideas__ without the overhead of creating test files as is typical in most programming languages. However, the interpreter supplied with the standard Python distribution is **somewhat limited** for extended interactive use.

The goal of IPython is to __create a comprehensive environment for interactive and exploratory computing__. To support this goal, IPython has two main components:

* An enhanced interactive Python shell.
* An architecture for **interactive parallel computing**.

All of IPython is open source (released under the revised BSD license).

==== Enhanced interactive Python shell ====
IPython’s interactive shell (ipython), has the following goals, amongst others:

* Provide an__ interactive shell __superior to Python’s default. IPython has many features for **object introspection**, __system shell access__, and its own__ special command system__ for adding functionality when working interactively. It tries to be a very efficient environment both for Python code development and for exploration of problems using Python objects (in situations like data analysis).
* Serve as an __embeddable, ready to use interpreter __for your own programs. IPython can be started with a single call from inside another program, providing access to the __current namespace__. This can be very useful both for __debugging__ purposes and for situations where a blend of batch-processing and interactive exploration are needed. New in the 0.9 version of IPython is a reusable wxPython based IPython widget.
* Offer a flexible framework which can be used as the base environment for other systems with Python as the underlying language. Specifically scientific environments like Mathematica, IDL and Matlab inspired its design, but similar ideas can be useful in many fields.
* Allow interactive testing of threaded graphical toolkits. IPython has support for interactive, non-blocking control of GTK, Qt and WX applications via special threading flags. The normal Python shell can only do this for Tkinter applications.

==== Main features of the interactive shell ====
* Dynamic object introspection. One can access docstrings, function definition prototypes, source code, source files and other details of any object accessible to the interpreter with a single keystroke (__?, and using ??__ provides additional detail).

* Searching through modules and namespaces with * wildcards, both when using the ? system and via the__ %psearch__ command.

* Completion in the **local namespace**, by typing __TAB__ at the prompt. This works for keywords, modules, methods, variables and **files** in the current directory. This is supported via the__ readline library__, and full access to configuring readline’s behavior is provided. Custom completers can be implemented easily for different purposes (system commands, magic arguments etc.)

* Numbered input/output prompts with __command history__ (persistent across sessions and tied to each profile), full searching in this history and caching of all input and output.

* User-extensible__ ‘magic’ commands__. A set of commands prefixed with % is available for controlling IPython itself and provides directory control, namespace information and many __aliases __to common system shell commands.

* Alias facility for defining your own system aliases.

* Complete __system shell access__. Lines starting with ! are passed directly to the system shell, and using __!! or var = !cmd__ captures shell output into python variables for further use.

* Background execution of Python commands in a separate thread. IPython has an __internal job manager__ called jobs, and a convenience backgrounding magic function called__ %bg__.

* The ability to **expand python variables when calling the system shell**. In a shell command, any python variable prefixed with __$ __is expanded. A double $$ allows passing a literal $ to the shell (for access to** shell and environment variables** like PATH).

* Filesystem navigation, via a magic__ %cd__ command, along with a persistent bookmark system (using __%bookmark__) for fast access to frequently visited directories.

* A lightweight** persistence framework** via the__ %store__ command, which allows you to save arbitrary Python variables. These get restored automatically when your session restarts.

* Automatic indentation (optional) of code as you type (through the readline library).

* __Macro system__ for quickly re-executing multiple lines of previous input with a single name. Macros can be stored persistently via __%store__ and edited via __%edit__.

* Session logging (you can then later use these logs as code in your programs). Logs can optionally __timestamp all input,__ and also store session output (marked as comments, so the log remains valid Python source code).

* Session restoring: logs can be__ replayed__ to restore a previous session to the state where you left it.

* Verbose and colored **exception traceback** printouts. Easier to parse visually, and in verbose mode they produce a lot of useful debugging information (basically a terminal version of the cgitb module).

* Auto-parentheses: callable objects can be __executed without parenthese__s: sin 3 is automatically converted to sin(3).

* Auto-quoting:__ using , or ; as the first character__ forces auto-quoting of the rest of the line: ,my_function a b becomes automatically my_function("a","b"), while ;my_function a b becomes my_function("a b").

* Extensible input syntax. You can define filters that pre-process user input to simplify input in special situations. This allows for example pasting multi-line code fragments which start with >>> or ... such as those from other python sessions or the standard Python documentation.

* Flexible configuration system. It uses a **configuration file** which allows permanent setting of all command-line options, module loading, code and file execution. The system allows__ recursive file inclusion__, so you can have a base file with defaults and layers which load other customizations for particular projects.

* Embeddable. You can call IPython as a python shell __inside your own python programs__. This can be used both for debugging code or for providing interactive abilities to your programs with knowledge about the** local namespaces** (very useful in debugging and data analysis situations).

* Easy debugger access. You can set IPython to call up an enhanced version of the Python debugger (pdb) every time there is an uncaught exception. This drops you inside the code which triggered the exception with all the data live and it is possible to navigate the stack to rapidly isolate the source of a bug. The __%run __magic command (with the -d option) can **run any script under pdb’s control,** automatically setting initial breakpoints for you. This version of pdb has IPython-specific improvements, including tab-completion and traceback coloring support. For even easier debugger access, try __%debug__ after seeing an exception. winpdb is also supported, see ipy_winpdb extension.

* Profiler support. You can run single statements (similar to profile.run()) or complete programs under the profiler’s control. While this is possible with standard cProfile or profile modules, IPython wraps this functionality with magic commands (see %prun and %run -p) convenient for rapid interactive work.

* Doctest support. The special__ %doctest_mode__ command toggles a mode that allows you to paste existing doctests (with leading >>> prompts and whitespace) and uses doctest-compatible prompts and output, so you can use IPython sessions as doctest code.

===== Interactive parallel computing =====
Increasingly, parallel computer hardware, such as multicore CPUs, clusters and supercomputers, is becoming ubiquitous. Over the last 3 years, we have developed an architecture within IPython that allows such hardware to be used quickly and easily from Python. Moreover, this architecture is designed to support interactive and collaborative parallel computing.

The main features of this system are:

    Quickly parallelize Python code from an interactive Python/IPython session.
    A flexible and dynamic process model that be deployed on anything from multicore workstations to supercomputers.
    An architecture that supports many different styles of parallelism, from message passing to task farming. And all of these styles can be handled interactively.
    Both blocking and fully asynchronous interfaces.
    High level APIs that enable many things to be parallelized in a few lines of code.
    Write parallel code that will run unchanged on everything from multicore workstations to supercomputers.
    Full integration with Message Passing libraries (MPI).
    Capabilities based security model with full encryption of network connections.
    Share live parallel jobs with other users securely. We call this collaborative parallel computing.
    Dynamically load balanced task farming system.
    Robust error handling. Python exceptions raised in parallel execution are gathered and presented to the top-level code.

For more information, see our overview of using IPython for parallel computing.

===== Portability and Python requirements =====

As of the 0.11 release, IPython works with Python 2.6 and 2.7. Versions 0.9 and 0.10 worked with Python 2.4 and above. IPython now also supports Python 3, although for now the code for this is separate, and kept up to date with the main IPython repository. In the future, these will converge to a single codebase which can be automatically translated using 2to3.

IPython is known to work on the following operating systems:

        Linux
        Most other Unix-like OSs (AIX, Solaris, BSD, etc.)
        Mac OS X
        Windows (CygWin, XP, Vista, etc.)

See here for instructions on how to install IPython.

===== What’s new in IPython =====
This section documents the changes that have been made in various versions of IPython. Users should consult these pages to learn about new features, bug fixes and backwards incompatibilities. Developers should summarize the development work they do here in a user friendly format.

==== Release 0.12 ====

IPython 0.12 contains several major new features, as well as a large amount of bug and regression fixes. The 0.11 release brought with it a lot of new functionality and major refactorings of the codebase; by and large this has proven to be a success as the number of contributions to the project has increased dramatically, proving that the code is now much more approachable. But in the refactoring inevitably some bugs were introduced, and we have also squashed many of those as well as recovered some functionality that had been temporarily disabled due to the API changes.

The following major new features appear in this version.

==== An interactive browser-based Notebook with rich media support ====
A powerful new interface puts IPython in your browser. You can start it with the command__ ipython notebook__:
The new IPython notebook showing text, mathematical expressions in__ LaTeX__, code, results and embedded figures created with __Matplotlib__.

This new interface maintains all the features of IPython you are used to, as it is **a new client that communicates with the same IPython kernels** used by the terminal and Qt console. But the web notebook provides for a different workflow where you can integrate, along with code execution, also text, mathematical expressions, graphics, video, and virtually any content that a modern browser is capable of displaying.

You can save your work sessions as documents that retain all these elements and which can be version controlled, emailed to colleagues or saved as HTML or PDF files for printing or publishing statically on the web. The internal storage format is a__ JSON__ file that can be easily manipulated for manual exporting to other formats.

This Notebook is a major milestone for IPython, as for years we have tried to build this kind of system. We were inspired originally by the excellent implementation in Mathematica, we made a number of attempts using older technologies in earlier Summer of Code projects in 2005 (both students and Robert Kern developed early prototypes), and in recent years we have seen the excellent implementation offered by the Sage <http://sagemath.org> system. But we continued to work on something that would be consistent with the rest of IPython’s design, and it is clear now that the effort was worth it: based on the ZeroMQ communications architecture introduced in version 0.11, the notebook can now__ retain 100% of the features of the real IPython__. But it can also provide the rich media support and high quality Javascript libraries that were not available in browsers even one or two years ago (such as high-quality mathematical rendering or built-in video).

The notebook has too many useful and important features to describe in these release notes; our documentation now contains a directory called examples/notebooks with several notebooks that illustrate various aspects of the system. You should start by reading those named 00_notebook_tour.ipynb and 01_notebook_introduction.ipynb first, and then can proceed to read the others in any order you want.

To start the notebook server, go to a directory containing the notebooks you want to open (or where you want to create new ones) and type:

**ipython notebook**

You can see all the relevant options with:

ipython notebook --help
ipython notebook --help-all  # even more

and just like the Qt console, you can start the notebook server with pylab support by using:

**ipython notebook --pylab**

for floating matplotlib windows or:

**ipython notebook --pylab inline**

for plotting support with automatically inlined figures. Note that it is now possible also to activate pylab support at runtime via %pylab, so you do not need to make this decision when starting the server.

See the Notebook docs for technical details.

===== Two-process terminal console =====
Based on the same architecture as the notebook and the Qt console, we also have now a terminal-based console that can connect to an external IPython kernel (the same kernels used by the Qt console or the notebook, in fact). While this client behaves almost identically to the usual IPython terminal application, this capability can be very useful to __attach an interactive console to an existing kernel__ that was started externally. It lets you use the interactive %debug facilities in a notebook, for example (the web browser can’t interact directly with the debugger) or debug a third-party code where you may have embedded an IPython kernel.

This is also something that we have wanted for a long time, and which is a culmination (as a team effort) of the work started last year during the 2010 Google Summer of Code project.

===== Tabbed QtConsole =====
The QtConsole now supports starting multiple kernels in tabs, and has a menubar, so it looks and behaves more like a real application. Keyboard enthusiasts can disable the menubar with ctrl-shift-M (PR #887).

The improved Qt console for IPython, now with tabs to control multiple kernels and full menu support.

===== Full Python 3 compatibility =====

IPython can now be installed from a single codebase on Python 2 and Python 3. The installation process for Python 3 automatically runs 2to3. The same ‘default’ profile is now used for Python 2 and 3 (the previous version had a separate ‘python3’ profile).
Standalone Kernel

The ipython kernel subcommand has been added, to allow starting a standalone kernel, that can be used with various frontends. You can then later connect a Qt console or a terminal console to this kernel by typing e.g.:

ipython qtconsole --existing

if it’s the only one running, or by passing explicitly the connection parameters (printed by the kernel at startup).

===== PyPy support =====

The terminal interface to IPython now runs under __PyPy__. We will continue to monitor PyPy’s progress, and hopefully before long at least we’ll be able to also run the notebook. The Qt console may take longer, as Qt is a very complex set of bindings to a huge C++ library, and that is currently the area where PyPy still lags most behind. But for everyday interactive use at the terminal, with this release and PyPy 1.7, things seem to work quite well from our admittedly limited testing.

===== Other important new features =====

* SSH Tunnels: In 0.11, the IPython.parallel Client could tunnel its connections to the Controller via ssh. Now, the QtConsole supports ssh tunneling, as do parallel engines.
* relaxed command-line parsing: 0.11 was released with overly-strict command-line parsing, preventing the ability to specify arguments with spaces, e.g. ipython --pylab qt or ipython -c "print 'hi'". This has been fixed, by using argparse. The new parsing is a strict superset of 0.11, so any commands in 0.11 should still work in 0.12.
* HistoryAccessor: The HistoryManager class for interacting with your IPython SQLite history database has been split, adding a parent HistoryAccessor class, so that users can write code to access and search their IPython history without being in an IPython session (PR #824).
* kernel %gui and %pylab: The %gui and %pylab magics have been restored to the IPython kernel (e.g. in the qtconsole or notebook). This allows activation of pylab-mode, or eventloop integration after starting the kernel, which was unavailable in 0.11. Unlike in the terminal, this can be set only once, and cannot be changed.
* %config: A new %config magic has been added, giving easy access to the IPython configuration system at runtime (PR #923).
* Multiline History: Multiline readline history has been restored to the Terminal frontend by default (PR #838).
* %store: The %store magic from earlier versions has been updated and re-enabled (storemagic; PR #1029). To autorestore stored variables on startup, specify c.StoreMagic.autorestore = True in ipython_config.py.