kernel/kernel_Notes
1
0
Fork 0
mirror of https://github.com/beyondx/Notes.git synced 2026-02-03 18:33:26 +08:00
Code Issues Packages Projects Releases Wiki Activity

Add New Notes

Browse Source
This commit is contained in:
geekard
2012-08-08 14:26:04 +08:00
commit 5ef7c20052
2374 changed files with 276187 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

38
Zim/Programme/python/library/Python_Subprocess_Module_Examples.txt Normal file
View File

@@ -0,0 +1,38 @@
Content-Type: text/x-zim-wiki
Wiki-Format: zim 0.4
Creation-Date: 2012-01-05T22:29:27+08:00
====== Python Subprocess Module Examples ======
Created Thursday 05 January 2012
http://www.moosechips.com/2010/07/python-subprocess-module-examples/
Some examples using the subprocess python module.
Make a system call three different ways:
#! /usr/bin/env python
import subprocess
# Use a sequence of args
return_code = subprocess.call(["echo", "hello sequence"])
# Set shell=true so we can use a simple string for the command
return_code = subprocess.call("echo hello string", shell=True)
# subprocess.call() is equivalent to using subprocess.Popen() and wait()
proc = subprocess.Popen("echo hello popen", shell=True)
return_code = proc.wait() # wait for process to finish so we can get the return code
Control stderr and stdout:
#! /usr/bin/env python
import subprocess
# Put stderr and stdout into pipes
proc = subprocess.Popen("echo hello stdout; echo hello stderr >&2", \
shell=True, stderr=subprocess.PIPE, stdout=subprocess.PIPE)
return_code = proc.wait()
# Read from pipes
for line in proc.stdout:
print("stdout: " + line.rstrip())
for line in proc.stderr:
print("stderr: " + line.rstrip())

364
Zim/Programme/python/library/pexpect.txt Normal file
View File

@@ -0,0 +1,364 @@
Content-Type: text/x-zim-wiki
Wiki-Format: zim 0.4
Creation-Date: 2012-01-05T15:16:44+08:00
====== pexpect ======
Created Thursday 05 January 2012
http://www.noah.org/wiki/Pexpect
pexpect
(Redirected from Pexpect)
Contents
1 Pexpect version 2.3
2 Description of Pexpect
3 License
4 Download and Installation
5 Project Status
6 Requirements for use of Pexpect
6.1 Python
6.2 pty module
7 Source Code
8 Examples
8.1 topip.py
8.2 hive.py
8.3 script.py
8.4 fix_cvs_files.py
8.5 ftp.py
8.6 monitor.py
8.7 passmass.py
8.8 python.py
8.9 rippy.py
8.10 sshls.py
8.11 ssh_tunnel.py
8.12 uptime.py
9 API Documentation
9.1 Core
9.2 experimental extensions
10 Overview
11 Special EOF and TIMEOUT patterns
12 Lists if patterns
13 Find the end of line -- CR/LF conventions
14 Beware of + and * at the end of input
15 Matching groups
16 Debugging
17 Exceptions
18 FAQ
18.1 Q: Isn't there already a Python Expect?
18.2 Q: The `before` and `after` properties sound weird.
18.3 Q: Why not just use Expect?
18.4 Q: Why not just use a pipe (popen())?
18.5 Q: Can I do screen scraping with this thing?
===== Pexpect version 2.3 =====
Download the current version here from the SourceForge site here: pexpect current
===== Description of Pexpect =====
Pexpect is a **pure Python module** that makes Python a better tool for__ controlling and automating other programs__. Pexpect is similar to the Don Libes `Expect` system, but Pexpect as a different interface that is easier to understand.
Pexpect is__ basically a pattern matching system__. It runs programs and watches output. When output matches a given pattern Pexpect can **respond** as if a human were typing responses. Pexpect can be used for** automation, testing, and screen scraping**.
Pexpect can be used for__ automating interactive console applications __such as ssh, ftp, passwd, telnet, etc.
It can also be used to__ control web applications__ via `lynx`, `w3m`, or some other **text-based **web browser. Pexpect is pure Python. Unlike other Expect-like modules for Python Pexpect does not require TCL or Expect nor does it require C extensions to be compiled. It should work on any platform that supports the standard Python pty module.
Send questions to: noah@noah.org Put 'pexpect' in the subject.
License
MIT style -- Free, open source, and all that good stuff.
Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
Pexpect Copyright (c) 2010 Noah Spurrier
===== Download and Installation =====
Download the current version here from the SourceForge site here: pexpect current
The Pexpect tarball is **a standard Python Distutil distribution**. Running the following commands should get you a working Pexpect module. Note that you have to have root access to install a site package.
wget http://pexpect.sourceforge.net/pexpect-2.3.tar.gz
tar xzf pexpect-2.3.tar.gz
cd pexpect-2.3
sudo python ./setup.py install
===== Project Status =====
Automated __pyunit tests__ reach over 80% code coverage on pexpect.py. I regularly test on Linux and BSD platforms. I try to test on Solaris and Irix.
===== Requirements for use of Pexpect =====
* **Python**
Pexpect was written and tested with Python 2.5. It should work on earlier versions that have the pty module. I sometimes even manually test it with Python 1.5.2, but I can't easily run the PyUnit test framework against Python 1.5.2.
* **pty module**
Any POSIX system (UNIX) with a working pty module should be able to run Pexpect. The pty module is part of the Standard Python Library, so if you are running on a POSIX system you should have it. The pty module does not run the same on all platforms. It should be solid on Linux and BSD systems. I have taken effort to try to smooth the wrinkles out of the different platforms. To learn more about the wrinkles see Bugs and Testing.
Pexpect does **not currently work on the standard Windows Python** (see the pty requirement); however, it seems to work fine using Cygwin. It is possible to build something like a pty for Windows, but it would have to use a different technique that I am still investigating. I know it's possible because Libes' Expect was ported to Windows. If you have any ideas or skills to contribute in this area then I would really appreciate some tips on how to approach this problem.
===== Source Code =====
You can browse the source code in SVN online here: SVN trunk
Note that all PyUnit tests are under the tests/ directory. To run the unit tests you will need to source the test.env environment file:
source test.env
then you will need to run the testall.py script under the tools/ directory:
tools/testall.py
===== Examples =====
Under the distribution tarball directory you should find an "examples" directory.__ This is the best way to learn to use Pexpect__. See the descriptions of Pexpect Examples.
* topip.py
This runs `netstat` on a local or remote server. It **calculates some simple statistical information** on the number of external inet connections. This can be used to detect if one IP address is taking up an excessive number of connections. It can also send an email alert if a given IP address exceeds a threshold between runs of the script. This script can be used as a drop-in Munin plugin or it can be used stand-alone from cron. I used this on a busy web server that would sometimes get hit with denial of service attacks. This made it easy to see if a script was opening many multiple connections. A typical browser would open fewer than 10 connections at once. A script might open over 100 simultaneous connections.
* hive.py
This script creates SSH connections to a list of hosts that you provide. Then you are given a command line prompt. Each shell command that you enter is** sent to all the hosts**. The response from each host is collected and printed. For example, you could connect to a dozen different machines and reboot them all at once.
* script.py
This implements a command similar to the classic BSD "script" command. This will start a subshell and log all input and output to a file. This demonstrates the interact() method of Pexpect.
* fix_cvs_files.py
This is for cleaning up binary files improperly added to CVS. This script scans the given path to find binary files; checks with CVS to see if the sticky options are set to -kb; finally if sticky options are not -kb then uses 'cvs admin' to set the -kb option.
* ftp.py
This demonstrates an FTP "bookmark". This connects to an ftp site; does a few ftp tasks; and then gives the user interactive control over the session. In this case the "bookmark" is to a directory on the OpenBSD ftp server. It puts you in the i386 packages directory. You can easily modify this for other sites. This demonstrates the interact() method of Pexpect.
* monitor.py
This runs a sequence of commands on a remote host using SSH. It runs a simple system checks such as uptime and free to monitor the state of the remote host.
* passmass.py
This will login to each given server and change the password of the given user. This demonstrates scripting logins and passwords.
* python.py
This starts the python interpreter and prints the greeting message backwards. It then gives the user iteractive control of Python. It's pretty useless!
* rippy.py
This is a wizard for mencoder. It greatly simplifies the process of ripping a DVD to Divx (mpeg4) format. It can transcode from any video file to another. It has options for resampling the audio stream; removing interlace artifacts, fitting to a target file size, etc. There are lots of options, but the process is simple and easy to use.
* sshls.py
This lists a directory on a remote machine.
* ssh_tunnel.py
This starts an SSH tunnel to a remote machine. It monitors the connection and restarts the tunnel if it goes down.
* uptime.py
This will run the uptime command and parse the output into variables. This demonstrates using a single regular expression to match the output of a command and capturing different variable in match groups. The grouping regular expression handles a wide variety of different uptime formats.
===== API Documentation =====
==== Core ====
**pexpect **- This is the main module that you want.
**pxssh** - Pexpect SSH is an extension of 'pexpect.spawn' that specializes in SSH.
==== experimental extensions ====
fdpexpect - fdpexpect extension of 'pexpect.spawn' that uses an open file descriptor. You can pass it __any file descriptor__ and use pexpect commands to match output from your fd.
screen - This represents a virtual 'screen'.
ANSI - This parses ANSI/VT-100 terminal escape codes.
FSM - This is a finite state machine used by ANSI.
===== Overview =====
Pexpect can be used for __automating interactive applications__ such as ssh, ftp, mencoder, passwd, etc. The Pexpect interface was designed to be easy to use. Here is an example of Pexpect in action:
# This connects to the openbsd ftp site and
# downloads the recursive directory listing.
import pexpect
child = pexpect.spawn ('ftp ftp.openbsd.org')
child.__expect__ ('Name .*: ')
child.__sendline__ ('anonymous')
child.expect ('Password:')
child.sendline ('noah@example.com')
child.expect ('ftp> ')
child.sendline ('cd pub')
child.expect('ftp> ')
child.sendline ('get ls-lR.gz')
child.expect('ftp> ')
child.sendline ('bye')
Obviously you could write an ftp client using Python's own ftplib module, but this is just a demonstration. You can use this technique with any application. This is especially handy if you are writing **automated test tools**.
There are two important methods in Pexpect --__ expect()__ and __send()__ (or __sendline()__ which is like send() with a linefeed). The expect() method waits for the child application to** return a given string**. The string you specify is a __regular expression__, so you can match complicated patterns. The send() method writes a string to the child application. From the child's point of view it looks just like someone **typed the text from a terminal**. After each call to expect() the __before and after properties__ will be set to the text printed by child application. The before property will contain all text up to the expected string pattern. The after string will contain the text that was **matched **by the expected pattern. The __match property__ is set to the re MatchObject.
An example of Pexpect in action may make things more clear. This example uses ftp to login to the OpenBSD site; list files in a directory; and then pass interactive control of the ftp session to the human user.
import pexpect
child = pexpect.spawn ('ftp ftp.openbsd.org')
child.expect ('Name .*: ')
child.sendline ('anonymous')
child.expect ('Password:')
child.sendline ('noah@example.com')
child.expect ('ftp> ')
child.sendline ('ls /pub/OpenBSD/')
child.expect ('ftp> ')
__print child.before__ # Print the result of the ls command.#打印上个expect匹配前输出的内容。
__child.interact() __ # Give control of the child to the user.
===== Special EOF and TIMEOUT patterns =====
There are two special patterns to match the End Of File or a Timeout condition. You you can pass these patterns to expect(). These patterns are not regular expressions. Use them like __predefined constants__.
If the child has **died** and you have read all the child's output then ordinarily expect() will raise an **EOF exception**. You can read everything up to the EOF without generating an exception by using the EOF pattern __expect(pexpect.EOF)__. In this case everything the child has output will be available in the **before property**.
===== Lists if patterns =====
The pattern given to expect() may be __a regular expression__ or it may also be __a list of regular expressions__. This allows you to__ match multiple optional responses__. The expect() method returns the** index** of the pattern that was matched. For example, say you wanted to login to a server. After entering a password you could get various responses from the server -- your password could be rejected; or you could be allowed in and asked for your terminal type; or you could be let right in and given a command prompt. The following code fragment gives an example of this:
child.expect('password:')
child.sendline (my_secret_password)
# We expect any of these three patterns...
i = child.expect** (['Permission denied', 'Terminal type', '[#\$] '])**
if i==0:
print 'Permission denied on host. Can't login'
__child.kill(0)__
elif i==2:
print 'Login OK... need to send terminal type.'
child.sendline('vt100')
child.expect ('[#\$] ')
elif i==3:
print 'Login OK.'
print 'Shell command prompt', __child.after__
If nothing matches an expected pattern then__ expect will eventually raise a TIMEOUT exception__. The default time is 30 seconds, but you can change this by passing a t**imeout argument** to expect():
# Wait no more than 2 minutes (120 seconds) for password prompt.
child.expect('password:', __timeout=120__)
===== Find the end of line -- CR/LF conventions =====
Matching at the __end of a line__ can be tricky. In general **the $ regex pattern is useless**. Pexpect matches regular expressions a little differently than what you might be used to. The $ matches the end of string, but Pexpect reads from the child **one character at a time**, so each character looks like the end of a line. Pexpect can't do a look-ahead into the child's output stream. In general you would have this situation when using regular expressions with a stream. Note, pexpect does have an internal buffer, so reads are faster than one character at a time, but from the user's perspective the regex pattern test happens one character at a time.
The best way to match the end of a line is to **look for the **__newline__**: "\r\n" (CR/LF)**. Yes, that does appear to be DOS-style. It may surprise some UNIX people to learn that **terminal TTY device drivers** (dumb, vt100, ANSI, xterm, etc.) all use the CR/LF combination to mark the end of line. **UNIX uses just linefeeds to end lines in **__files__**, but not when it comes to TTY devices**!
Pexpect uses a__ Pseudo-TTY__ device to talk to the child application, so when the child application prints a "\n" your TTY device actually sees "\r\n". TTY devices are more like the Windows world. Each line of text end with a CR/LF combination. When you intercept data from a UNIX command from a TTY device you will find that the TTY device outputs a CR/LF combination.
**A UNIX command may** only write a linefeed (\n), __but the TTY device driver converts it to CR/LF__. This means that your terminal will see lines end with CR/LF (hex 0D 0A). Since **Pexpect emulates a terminal**, to match ends of lines you have to expect the CR/LF combination.
child.expect__ ('\r\n')__
If you just need to skip past a new line then expect ('\n') by itself will work, but if you are expecting a specific pattern before the end of line then you need to explicitly look for the \r. For example the following **expects a word at the end of a line**:
__ child.expect ('\w+\r\n')__
But the following would both fail:
child.expect ('\w+\n')
And as explained before, trying to use '$' to match the end of line would not work either:
child.expect ('\w+$')
So if you need to explicitly look for the END OF LINE, you want to look for the __CR/LF combination__ -- not just the LF and not the $ pattern.
This problem is not limited to Pexpect. **This problem happens any time you try to perform a regular expression match on a stream.** Regular expressions need to look ahead. **With a stream it is hard to look ahead** because the process generating the stream may not be finished. There is no way to know if the process has paused momentarily or is finished and waiting for you.
Pexpect must implicitly always do a NON greedy match (minimal) at the end of a input {### already said this}.
Pexpect compiles all regular expressions with the __DOTALL __flag. With the DOTALL flag a "." will match a newline.
===== Beware of + and * at the end of input =====
Remember that any time you try to match a pattern that needs look-ahead that __you will always get a minimal match (non greedy)__. For example, the following will always return just one character:
child.expect ('.+')
This example will match successfully, but will always return no characters:
child.expect ('.*')
Generally any star * expression will **match as little as possible.**
One thing you can do is to try to force a non-ambiguous character at the end of your \d+ pattern. Expect that character to delimit the string. For example, you might try making thr end of your pattrn be \D+ instead of \D*. That means number digits alone would not satisfy the (\d+) pattern. You would need some number(s) and at least one \D at the end.
===== Matching groups =====
You can group regular expression using **parenthesis**. After a match, the __match parameter__ of the spawn object will contain the Python re.match object.
===== Debugging =====
If you get the __string value__ of a pexpect.spawn object you will get lots of useful debugging information. For debugging it's very useful to use the following pattern:
try:
i = child.expect ([pattern1, pattern2, pattern3, etc])
except:
print "Exception was thrown"
print "debug information:"
print__ str__(child)
It is also useful to__ log the child's input and out to a file or the screen__. The following will turn on logging and send output to stdout (the screen).
child = pexpect.spawn (foo)
child.logfile = sys.stdout
===== Exceptions =====
* EOF
Note that two flavors of EOF Exception may be thrown. They are virtually identical except for the message string. For practical purposes you should have no need to distinguish between them, but they do give a little extra information about what type of platform you are running. The two messages are:
End Of File (EOF) in read(). Exception style platform.
End Of File (EOF) in read(). Empty string style platform.
Some UNIX platforms will throw an exception when you try to read from a file descriptor in the EOF state. Other UNIX platforms instead quietly return an empty string to indicate that the EOF state has been reached.
Expecting EOF
If you wish to read up to the end of the child's output **without generating an EOF exception** then use the __expect(pexpect.EOF)__ method.
* TIMEOUT
The expect() and read() methods will also timeout if the child does not generate any output for a given amount of time. If this happens they will raise a TIMEOUT exception. You can have these method ignore a timeout and block indefinitely by passing None for the timeout parameter.
child.expect(pexpect.EOF, timeout=None)
===== FAQ =====
Q: Isn't there already a Python Expect?
A: Yes, there are several of them. They usually require you to compile C. I wanted something that was pure Python and preferably a single module that was simple to install. I also wanted something that was easy to use. This pure Python expect only recently became possible with the introduction of the** pty module** in the standard Python library. Previously C extensions were required.
Q: The `before` and `after` properties sound weird.
A:__ This is how the -B and -A options in grep works__, so that made it easier for me to remember. Whatever makes my life easier is what's best. Originally I was going to model Pexpect after Expect, but then I found that I didn't actually like the way Expect did some things. It was more confusing. The `after` property can be a little confusing at first, because** it will actually include the matched string**. __The `after` means after the point of match, not after the matched string.__
Q: Why not just use Expect?
A: I love it. It's great. I has bailed me out of some real jams, but I wanted something that would do 90% of what I need from Expect; be 10% of the size; and allow me to write my code in Python instead of TCL. Pexpect is not nearly as big as Expect, but Pexpect does everything I have ever used Expect for.
Q: Why not just use a pipe (popen())?
A: **A pipe works fine for getting the output to**__ non-interactive__** programs**. If you just want to get the output from ls, uname, or ping then this works. Pipes do not work very well for interactive programs and pipes will almost certainly fail for most applications that ask for passwords such as telnet, ftp, or ssh.
There are two reasons for this.
===== First =====
an application may **bypass stdout** and__ print directly to its controlling TTY__. Something like SSH will do this when it asks you for a password. This is why you cannot redirect the password prompt because it does not go through stdout or stderr.
===== second =====
reason is because most applications are built using the **C Standard IO Library** (anything that uses #include <stdio.h>). One of the features of the stdio library is that __it buffers all input and output.__ Normally output is** line buffered** when a program is printing to a TTY (your terminal screen). Everytime the program prints a line-feed the currently buffered data will get printed to your screen. The problem comes when you connect a __pipe__. The stdio library is smart and can tell that it is printing to a pipe instead of a TTY. In that case it switches from line buffer mode to __block buffered__. In this mode the currently buffered data is flushed when the__ buffer is full__. This causes most interactive programs to __deadlock__. Block buffering is more efficient when writing to disks and pipes. Take the situation where a program prints a message "Enter your user name:\n" and then waits for you type type something. In block buffered mode, __the stdio library will not put the message into the pipe __even though a linefeed is printed. The result is that you never receive the message, yet the child application will sit and wait for you to type a response. Don't confuse the stdio lib's buffer with the pipe's buffer. The pipe buffer is another area that can cause problems. You could flush the input side of a pipe, whereas you have __no control over the stdio library__ buffer.
===== More information =====
: the Standard IO library has three states for a FILE *. These are: _IOFBF for block buffered; _IOLBF for line buffered; and _IONBF for unbuffered. The STDIO lib will use block buffering when talking to a **block file descriptor** such as a pipe. This is usually not helpful for interactive programs. Short of recompiling your program to include __fflush() __everywhere or recompiling a custom stdio library there is not much a controlling application can do about this if talking over a pipe.
The program may have put data in its output that remains unflushed because the output buffer is not full; then the program will go and deadlock while waiting for input -- because you never send it any because you are still waiting for its output (still stuck in the STDIO's output buffer).
The answer is to use a
===== pseudo-tty =====
. A TTY device will force line buffering (as opposed to block buffering). Line buffering means that you will get each line when the child program sends a line feed. This corresponds to the way most interactive programs operate -- send a line of output then wait for a line of input.
I put "answer" in quotes because it's ugly solution and because there is no POSIX standard for pseudo-TTY devices (even though they have a TTY standard...). What would make more sense to me would be to have some way to __set a mode on a file descriptor __so that it will tell the STDIO to be line-buffered. I have investigated, and I don't think there is a way to set the buffered state of a child process. The STDIO Library does not maintain any external state in the kernel or whatnot, so I don't think there is any way for you to alter it. I'm not quite sure how this line-buffered/block-buffered state change happens internally in the STDIO library. I think the STDIO lib looks at the file descriptor and decides to change behavior based on whether it's a TTY or a block file (see isatty()).
I hope that this qualifies as helpful. __Don't use a pipe to control another application__...
Pexpect may seem similar to **os.popen()** or commands module. The main difference is that__ Pexpect (like Expect) uses a pseudo-TTY to talk to the child__ __application__.
Most applications do no work well through the system() call or through pipes. And probably all applications that ask a user to type in a password will fail. These applications bypass the stdin and read directly from the TTY device. Many applications do not explicitly flush their output buffers. This causes deadlocks if you try to control an interactive application using a pipe. What happens is that most UNIX applications use the stdio (#include <stdio.h>) for input and output. The stdio library behaves differently depending on where the output is going. There is no way to control this behavior from the client end.
Q: Can I do screen scraping with this thing?
A: That depends. If your application just does line-oriented output then this is easy. If it does screen-oriented output then it may work, but it could be hard. For example, trying to scrape data from the 'top' command would be hard. The top command repaints the text window.
I am working on an ANSI / VT100 terminal emulator that will have methods to get characters from an arbitrary X,Y coordinate of the virtual screen. It works and you can play with it, but I have no working examples at this time.

491
Zim/Programme/python/library/subprocess.txt Normal file
View File

@@ -0,0 +1,491 @@
Content-Type: text/x-zim-wiki
Wiki-Format: zim 0.4
Creation-Date: 2012-01-05T16:23:46+08:00
====== subprocess ======
Created Thursday 05 January 2012
===== 17.1. subprocess — Subprocess management =====
New in version 2.4.
The subprocess module allows you to__ spawn__ new processes, connect to their__ input/output/error pipes__, and obtain their __return codes__. This module intends to **replace** several other, older modules and functions, such as:
* os.system
* os.spawn*
* os.popen*
* popen2.*
* commands.*
subprocess是推荐的执行shell命令或子进程的方式。
Information about how the subprocess module can be used to replace these modules and functions can be found in the following sections.
See also： PEP 324 – PEP proposing the subprocess module
===== 17.1.1. Using the subprocess Module =====
The recommended approach to invoking subprocesses is to use the following convenience functions for all use cases they can handle. For more advanced use cases, the underlying __Popen__ interface can be used directly.
三个模块函数：
* subprocess.__call__(**args**, *, stdin=None, stdout=None, stderr=None, shell=False)
* subprocess.__check_call__(args, *, stdin=None, stdout=None, stderr=None, shell=False)
* subprocess.__check_output__(args, *, stdin=None, stderr=None, shell=False, **universal_newlines**=False)
两个模块变量：
__subprocess.PIPE__
__subprocess.STDOUT__
在功能能满足的情况下，尽量使用上面的三个函数，更复杂的交互式控制可以使用subprocess中提供的__Popen()类__。前两个函数返回命令的退出码，最后一个命令返回命令的输出为字符串。
[*] subprocess.__call__(**args**, *, stdin=None, stdout=None, stderr=None, shell=False)
Run the command described by __args__. __Wait __for command to complete, then return the **returncode** attribute.
适合非交互式进程，执行args指定的程序直到其结束，返回退出码，即使命令执行失败，也不产生异常。stdin,stdout,stderr一般不使用PIPE，因为调用进程一般不读这种非交互式进程的输出。stdin为None表示子进程的stdin将__继承__调用进程的描述符。
The arguments shown above are merely the most common ones, described below in Frequently Used Arguments (hence the slightly odd notation in the abbreviated signature). The full function signature is the same as that of the __Popen__ constructor - this functions passes all supplied arguments directly through to that interface.
Examples:
>>>
__#如果shell=False(默认)，则python调用os.execlp()来执行args，因此，如果args是字符串，则只能包含命令名；如果是序列，则可以包含命令参数(参数不支持shell的特性如文件名扩展等。)，而且每个元素一般包括一个参数(否则调用程序可能会产生解析错误)。__
>>> subprocess.call(__["ls", "-l"]__)
0
__#如果shell=True，则python调用system shell来执行args，调用形式为: execlp("/bin/sh", '-c', 'arg1', 'arg2', ....)因此，如果args是字符串，则其中可以包含命令参数，而且支持各种shell特性如文件名扩展、命令扩展等；如果args是序列，则其第一个元素为命令名，其它元素为被当作shell本身的参数。__
>>> subprocess.call("exit 1", shell=True)
1
__总的来说__**：**如果shell=False, python使用os.execvp(...)来执行args，因此，如果args是一个string，则该string只能是命令名称, 如果要为命令附加参数，则只能使用序列类型; 如果shell=True，则python使用shell来执行行args，因此，args最好是一个string，该string可以包含命令名及其参数，如果使用序列，则从第二个元素开始作为shell本身的参数(而非命令的参数。)。
Invoking the system shell with shell=True can be **a security hazard** if combined with untrusted input. See the warning under Frequently Used Arguments for details.
__Do not use stdout=PIPE or stderr=PIPE with this function__. As the pipes are not being read in the current process, the child process may block if it generates enough output to a pipe to fill up the OS pipe buffer.
当使用stdout=PIPE时，如果子进程填满了管道而调用进程没有读该管道时__子进程就会被阻塞__。因此使用上面三个函数时一般不使用这个参数。
The standard input and output channels for the process started by call() are__ bound to the parent’s input and output__. That means the calling programm cannot capture the output of the command. Use check_output() to capture the output for later processing.
[*] subprocess.**check_call**(args, *, stdin=None, stdout=None, stderr=None, shell=False)
和call类似，但是对返回值进行检查，如果非0则引发异常。
Run command with arguments. Wait for command to complete. **If the return code was zero then return**, otherwise raise__ CalledProcessError__. The CalledProcessError object will have the **return code **in the **returncode** attribute.
T Examples:
>>>
>>> subprocess.check_call(["ls", "-l"])
0
>>> subprocess.check_call("exit 1", shell=True)
Traceback (most recent call last):
...
subprocess.CalledProcessError: Command 'exit 1' returned non-zero exit status 1
Note
Do not use stdout=PIPE or stderr=PIPE with this function. As the pipes are not being read in the current process, the child process may block if it generates enough output to a pipe to fill up the OS pipe buffer.
[*] subprocess.__check_output__(args, *, stdin=None, stderr=None, shell=False, **universal_newlines**=False)
执行命令，返回命令的输出为一字符串。如果命令执行失败，则产生异常。
Run command with arguments and return its output as a__ byte string__.
If the return code was non-zero it raises a CalledProcessError. The CalledProcessError object will have the return code in the returncode attribute and any output in the output attribute.
Examples:
>>>
>>> subprocess.check_output(["echo", "Hello World!"])
'Hello World!\n'
>>> subprocess.check_output("exit 1", shell=True)
Traceback (most recent call last):
...
subprocess.CalledProcessError: Command 'exit 1' returned non-zero exit status 1
To also capture standard error in the result, use__ stderr=subprocess.STDOUT__:
>>>
>>> subprocess.check_output(
... "ls non_existent_file; exit 0",
... stderr=subprocess.STDOUT,
... shell=True)
'ls: non_existent_file: No such file or directory\n'
Note
Do not use stderr=PIPE with this function. As the pipe is not being read in the current process, the child process may block if it generates enough output to the pipe to fill up the OS pipe buffer.
__subprocess.PIPE__
Special value that can be used as the** stdin, stdout or stderr** argument to Popen and indicates that a pipe to the standard stream should be opened.
如果stdin=PIPE, 则Popen(...)返回的Popen型对象的stdin属性返回一__写打开的文件对象__，向此文件写入的内容将会被传给子进程。stdout, stderr类似。
__subprocess.STDOUT__
Special value that can be used as the **stderr** argument to Popen and indicates that standard error should go into the same handle as standard output.
表示子进程的stderr被重定向**其stdout对应的文件中**。
=== 17.1.1.1. Frequently Used Arguments ===
To support a wide variety of use cases, the __Popen constructor__ (and the convenience functions) accept a large number of optional arguments. For most typical use cases, many of these arguments can be safely left at their default values. The arguments that are most commonly needed are:
* args is required for all calls and should be **a **__string__**, or a **__sequence__** of program arguments**. Providing a sequence of arguments is generally preferred, as it allows the module to take care of any required escaping and quoting of arguments (e.g. to permit spaces in file names). If passing a single string, either __shell must be True__ (see below) or else the string must simply name the program to be executed __without__ specifying any arguments.
使用序列形式是首选的方式，这时一般将shell=False，python直接调用os.execlp()。这样可以避免shell代码注入风险。
* stdin, stdout and stderr specify the __executed program’s __standard input, standard output and standard error **file handles**, respectively. Valid values are__ PIPE, an existing file descriptor (a positive integer), an existing file object, and None__. PIPE indicates that a new pipe to the child should be created. With the default settings of None, no** redirection** will occur; the child’s file handles will be inherited from the parent. Additionally, stderr can be STDOUT, which indicates that the stderr data from the child process should be captured into **the same file** handle as for stdout.
When stdout or stderr are pipes and universal_newlines is True then all line endings will be converted to__ '\n'__ as described for the universal newlines ‘U’` mode argument to open().
* If shell is True, the specified command will be** executed through the shell**. This can be useful if you are using Python primarily for the enhanced control flow it offers over most system shells and still want __access to other shell features__ such as **filename wildcards, shell pipes and environment variable expansion**.
使用shell时，args的字符串形式是首选的方式。
Warning
Executing shell commands that incorporate unsanitized input from an untrusted source makes a program vulnerable to shell injection, a serious security flaw which can result in arbitrary command execution. For this reason, the use of __shell=True is strongly discouraged__ in cases where the command string is constructed from external input:
>>>
>>> from subprocess import call
>>> filename = input("What file would you like to display?\n")
What file would you like to display?
non_existent; rm -rf / #
>>> call("cat " + filename, shell=True) # Uh-oh. This will end badly...
shell=False disables all shell based features, but does not suffer from this vulnerability; see the Note in the Popen constructor documentation for helpful hints in getting shell=False to work.
These options, along with all of the other options, are described in more detail in the Popen constructor documentation.
==== 17.1.1.2. Popen Constructor ====
subprocess模块中只定义了一个类：Popen
The **underlying** process creation and management in this module is handled by the__ Popen__ class. It offers a lot of flexibility so that developers are able to handle the less common cases not covered by the convenience functions.
__class subprocess.Popen__(args, bufsize=0, **executable**=None, stdin=None, stdout=None, stderr=None, preexec_fn=None, close_fds=False, shell=False, cwd=None, env=None, universal_newlines=False, startupinfo=None, creationflags=0)
Arguments are:
* args should be a string, or a sequence of program arguments. The program to execute is normally the **first item** in the args sequence or the string if a string is given, but can be **explicitly set **by using the __executable__ argument. When executable is given, the first item in the args sequence is still treated by most programs as the** command name**, which can then be different from the actual executable name. On Unix, it becomes the** display name** for the executing program in utilities such as ps.
On Unix, with shell=False (default): In this case, the __Popen class uses os.execvp()__ to execute the child program. args should **normally be a sequence.** If a string is specified for args, it will be used as the **name or path of the program **to execute; this will only work if the program is being given __no__ arguments.
Note
__shlex.split()__ can be useful when determining the correct tokenization for args, especially in complex cases:
>>>
>>> import shlex, subprocess
>>> command_line = raw_input()
/bin/vikings -input eggs.txt -output "spam spam.txt" -cmd "echo '$MONEY'"
>>> args = shlex.split(command_line)
>>> print args
['/bin/vikings', '-input', 'eggs.txt', '-output', 'spam spam.txt', '-cmd', "echo '$MONEY'"]
>>> p = subprocess.Popen(args) # Success!
Note in particular that options (such as -input) and arguments (such as eggs.txt) that are separated by whitespace in the shell go in **separate list elements**, while arguments that need quoting or backslash escaping when used in the shell (such as filenames containing spaces or the echo command shown above) are single list elements.
On Unix, with shell=True: __If args is a string, it specifies the command string to execute through the shell.__ This means that the string must be formatted exactly as it would be when typed at the shell prompt. This includes, for example, quoting or backslash escaping filenames with spaces in them. If args is a sequence, the first item specifies the command string, and any additional items will be treated as additional arguments __to the shell itself__. That is to say, Popen does the equivalent of:
在shell=True的情况下，args最好是一个字符串。如果是序列，则从第二个元素开始是__作为shell本身的参数__，而非命令的参数。
__ Popen(['/bin/sh', '-c', args[0], args[1], ...])__
On Windows: the Popen class uses CreateProcess() to execute the child child program, which __operates on strings.__ If args is a sequence, it will be converted to a string in a manner described in Converting an argument sequence to a string on Windows.
**总的来说：如果shell=False, python使用os.execvp(...)来执行args，因此，如果args是一个string，则该string只能是命令名称, 如果要为命令执行参数，则只能使用序列类型; 如果shell=True，则python使用shell来执行行args，因此，args最好是一个string，该string可以包含命令名及其参数，如果使用序列，则从第二个元素开始作为**__shell本身__**的参数(而非命令的参数。)。**
* bufsize, if given, has the same meaning as the corresponding argument to the built-in open() function: **0 means **__unbuffered__**, 1 means**__ line buffered__**, any other positive value means use a buffer of (approximately) that size**. A negative bufsize means to use the system **default**, which usually means__ fully__ buffered. The** default value for bufsize is 0** (unbuffered).
控制打开文件的缓冲方式。
Note
If you experience performance issues, it is recommended that you try to enable buffering by setting bufsize to either -1 or a large enough positive value (such as 4096).
* The executable argument **specifies the program** to execute. It is very seldom needed: Usually, the program to execute is defined by the **args** argument. If shell=True, the executable argument specifies __which shell to use__. On Unix, the default shell is /bin/sh. On Windows, the default shell is specified by the COMSPEC environment variable. The only reason you would need to specify shell=True on Windows is where the command you wish to execute is actually built in to the shell, eg dir, copy. You don’t need shell=True to run a batch file, nor to run a console-based executable.
* stdin, stdout and stderr specify the __executed program’s __standard input, standard output and standard error file handles, respectively. Valid values are __PIPE, an existing file descriptor (a positive integer), an existing file object, and None__. PIPE indicates that a new pipe to the child should be created. With the **default settings of None**, no redirection will occur; the child’s file handles will be inherited from the parent. Additionally, stderr can be STDOUT, which indicates that the stderr data from the child process should be captured into the same file handle as for stdout.
* If__ preexec_fn__ is set to a callable object, this object will be called in the child process just **before** the child is executed. (Unix only)
* If close_fds is true,__ all file descriptors except 0, 1 and 2 will be closed__ before the child process is executed. (Unix only). Or, on Windows, if close_fds is true then no handles will be inherited by the child process. Note that on Windows, you cannot set close_fds to true and also redirect the standard handles by setting stdin, stdout or stderr.
* If shell is True, the specified command will be executed through the shell.
Warning
Enabling this option can be a security hazard if combined with untrusted input. See the warning under Frequently Used Arguments for details.
* If cwd is not None, the child’s current directory will be changed to cwd **before **it is executed. Note that this directory is not considered when searching the executable, so you can’t specify the program’s path relative to cwd.
* If env is not None, it must be __a mapping__ that defines the environment variables for the new process; these are used instead of** inheriting** the current process’ environment, which is the default behavior.
Note
If specified, __env must provide any variables__ required for the program to execute. On Windows, in order to run a side-by-side assembly the specified env must include a valid SystemRoot.
* If universal_newlines is True, the file objects stdout and stderr are __opened as text files__, but lines may be terminated by any of '\n', the Unix end-of-line convention, '\r', the old Macintosh convention or '\r\n', the Windows convention. All of these external representations are __seen as '\n' __by the Python program.
Note
This feature is only available if Python is built with universal newline support (the default). Also, the newlines attribute of the file objects stdout, stdin and stderr are not updated by the communicate() method.
* If given, startupinfo will be a STARTUPINFO object, which is passed to the underlying __CreateProcess __function. creationflags, if given, can be CREATE_NEW_CONSOLE or CREATE_NEW_PROCESS_GROUP. (Windows only)
==== 17.1.1.3. Exceptions ====
Exceptions raised in the child process, before the new program has started to execute, will be__ re-raised in the parent__. Additionally, the exception object will have one extra attribute called **child_traceback**, which is a string containing traceback information from the child’s point of view.
The most common exception raised is __OSError__. This occurs, for example, when trying to execute a non-existent file. Applications should prepare for OSError exceptions.
A __ValueError __will be raised if Popen is called with invalid arguments.
check_call() and check_output() will raise__ CalledProcessError__ if the called process returns a** non-zero** return code.
==== 17.1.1.4. Security ====
最好不要使用shell=True参数。
Unlike some other popen functions, this implementation will **never call a system shell implicitly**. This means that all characters, including shell metacharacters, can __safely__ be passed to child processes. Obviously, if the shell is invoked explicitly, then it is the application’s responsibility to ensure that all whitespace and metacharacters are quoted appropriately.
==== 17.1.2. Popen Objects ====
Popen()函数返回一个__Popen对象__。
Instances of the Popen class have the following methods:
**Popen.poll()**
Check if child process __has terminated__. Set and return returncode attribute.
**Popen.wait()**
Wait for child process to terminate. Set and return returncode attribute.
Warning
This will deadlock when using stdout=PIPE and/or stderr=PIPE and the child process generates enough output to a pipe such that it blocks waiting for the OS pipe buffer to accept more data. Use communicate() to avoid that.
__Popen.communicate__(input=None)
Interact with process: **Send data to stdin**. Read data from stdout and stderr, until __end-of-file__ is reached. Wait for process to terminate. The optional input argument should be __a string to be sent to the child process__, or None, if no data should be sent to the child.
communicate() returns __a tuple__ (stdoutdata, stderrdata).
Note that if you want to send data to the process’s stdin, you need to create the Popen object with__ stdin=PIPE__. Similarly, to get anything other than None in the result tuple, you need to give__ stdout=PIPE__ and/or stderr=PIPE too.
Note
The data read is buffered in memory, so do not use this method if the data size is large or unlimited.
**Popen.send_signal(signal)**
Sends the signal signal to the child.
Note
On Windows, **SIGTERM **is an alias for terminate(). CTRL_C_EVENT and CTRL_BREAK_EVENT can be sent to processes started with a creationflags parameter which includes CREATE_NEW_PROCESS_GROUP.
**Popen.terminate()**
Stop the child. On Posix OSs the method sends SIGTERM to the child. On Windows the Win32 API function TerminateProcess() is called to stop the child.
**Popen.kill()**
Kills the child. On Posix OSs the function sends __SIGKILL__ to the child. On Windows kill() is an alias for terminate().
===== The following attributes are also available: =====
**Popen对象**具有下列属性：
Warning
Use communicate() rather than .stdin.write, .stdout.read or .stderr.read to avoid deadlocks due to any of the other OS pipe buffers filling up and blocking the child process.
Popen.stdin #对Popen调用stdin属性，将返回一个**写打开的文件对象**。
If the stdin argument was PIPE, this attribute is **a file object** that provides input to the child process. Otherwise, it is__ None__.
Popen.stdout
If the stdout argument was PIPE, this attribute is a file object that provides output from the child process. Otherwise, it is None.
Popen.stderr
If the stderr argument was PIPE, this attribute is a file object that provides error output from the child process. Otherwise, it is None.
Popen.__pid__
The process ID of the child process. Note that if you set the shell argument to True, this is the process ID of the __spawned shell__.
Popen.__returncode__
The child return code, **set by poll() and wait()** (and indirectly by communicate()). A __None__ value indicates that the process hasn’t terminated yet. A negative value -N indicates that the child was **terminated by signal N** (Unix only).
===== 17.1.3.1. Constants =====
The subprocess module exposes the following constants.
subprocess.STD_INPUT_HANDLE
The __standard input device__. Initially, this is the console input buffer, CONIN$.
subprocess.STD_OUTPUT_HANDLE
The standard output device. Initially, this is the active console screen buffer, CONOUT$.
subprocess.STD_ERROR_HANDLE
The standard error device. Initially, this is the active console screen buffer, CONOUT$.
subprocess.SW_HIDE
Hides the window. Another window will be activated.
subprocess.STARTF_USESTDHANDLES
Specifies that the STARTUPINFO.hStdInput, STARTUPINFO.hStdOutput, and STARTUPINFO.hStdError attributes contain additional information.
subprocess.STARTF_USESHOWWINDOW
Specifies that the STARTUPINFO.wShowWindow attribute contains additional information.
subprocess.CREATE_NEW_CONSOLE
The new process ha**s a new console**, instead of inheriting its parent’s console (the default).
This flag is always set when Popen is created with shell=True.
subprocess.CREATE_NEW_PROCESS_GROUP
A Popen creationflags parameter to specify that __a new process group__ will be created. This flag is necessary for using __os.kill()__ on the subprocess.
This flag is ignored if CREATE_NEW_CONSOLE is specified.
==== 17.1.4. Replacing Older Functions with the subprocess Module ====
In this section, “a becomes b” means that b can be used as a replacement for a.
Note
All “a” functions in this section** fail (more or less) silently** if the executed program cannot be found; the “b” replacements raise__ OSError__ instead.
In addition, the replacements using check_output() will fail with a __CalledProcessError__ if the requested operation produces a **non-zero** return code. The output is still available as the output attribute of the raised exception.
如果子进程返回非0值，check_output将引发异常。
In the following examples, we assume that the relevant functions have already been imported from the subprocess module.
=== 17.1.4.1. Replacing /bin/sh shell backquote ===
output=`mycmd myarg`
# becomes
**output **= __check_output__(["mycmd", "myarg"])
=== 17.1.4.2. Replacing shell pipeline ===
output=`dmesg | grep hda`
# becomes
p1 = Popen(["dmesg"], __stdout=PIPE__)
p2 = Popen(["grep", "hda"], stdin=__p1.stdout__, stdout=PIPE) #p1是Popen对象，其stdout返回一个读打开的文件对象（因为p1的stdout为PIPE）。
__p1.stdout__.close() # Allow p1 to receive a SIGPIPE if p2 exits.
output = __p2.communicate()__[0] #communicate()返回一个__元组__，这里只取出第一个元素即子进程的标准输出。
The p1.stdout.close() call after starting the p2 is __important in order__ for p1 to receive a SIGPIPE if p2 exits before p1.
Alternatively, for trusted input, the shell’s own pipeline support may still be used directly:
output=`dmesg | grep hda` # becomes output=__check_output__(“dmesg | grep hda”,__ shell=True__)
==== 17.1.4.3. Replacing os.system() ====
sts = os.system("mycmd" + " myarg") #os.system是**调用一个shell**来执行其参数对应的命令，返回的是子进程的退出值。
# becomes
sts =__ call__("mycmd" + " myarg",__ shell=True__) #返回子进程的退出值，即使为非0值时不会引发异常。
Notes:
Calling the program through the shell is usually not required.
A more realistic example would look like this:
try:
**retcode** = call("mycmd" + " myarg", shell=True)
if retcode < 0:
print >>sys.stderr, "Child was terminated by signal", __-retcode__
else:
print >>sys.stderr, "Child returned", retcode
__except OSError, e:__
print **>>sys.stderr**, "Execution failed:", e
==== 17.1.4.4. Replacing the os.spawn family ====
P_NOWAIT example:
__pid__ = os.spawnlp(os.P_NOWAIT, "/bin/mycmd", "mycmd", "myarg")
==>
pid = Popen(["/bin/mycmd", "myarg"]).__pid__
P_WAIT example:
__retcode__ = os.spawnlp(os.P_WAIT, "/bin/mycmd", "mycmd", "myarg")
==>
retcode = call(["/bin/mycmd", "myarg"])
Vector example:
os.spawnvp(os.P_NOWAIT, path, args)
==>
Popen([path] + args[1:])
Environment example:
os.spawnlpe(os.P_NOWAIT, "/bin/mycmd", "mycmd", "myarg", env)
==>
Popen(["/bin/mycmd", "myarg"], **env**={"PATH": "/usr/bin"})
==== 17.1.4.5. Replacing os.popen(), os.popen2(), os.popen3() ====
pipe = os.popen("cmd", __'r'__, bufsize) #第二个参数指定管道打开的方式。
==>
pipe = Popen("cmd", shell=True, bufsize=bufsize, stdout=PIPE)__.stdout__
pipe = os.popen("cmd", 'w', bufsize)
==>
pipe = Popen("cmd", shell=True, bufsize=bufsize, stdin=PIPE)__.stdin__
(child_stdin, child_stdout) = os.popen2("cmd", mode, bufsize) #popen2同时返回与子进程相连的读、写__管道描述符__。
==>
p = Popen("cmd", shell=True, bufsize=bufsize,
stdin=PIPE, stdout=PIPE, close_fds=True)
(child_stdin, child_stdout) = (p.stdin, p.stdout) #返回与子进程相连的读、写__文件对象__。
(child_stdin,
child_stdout,
child_stderr) = os.popen3("cmd", mode, bufsize)
==>
p = Popen("cmd", shell=True, bufsize=bufsize,
stdin=PIPE, stdout=PIPE, **stderr=PIPE**, close_fds=True)
(child_stdin,
child_stdout,
child_stderr) = (p.stdin, p.stdout, p.stderr)
(child_stdin, child_stdout_and_stderr) = os.popen4("cmd", mode,
bufsize)
==>
p = Popen("cmd", shell=True, bufsize=bufsize,
stdin=PIPE, stdout=PIPE, __stderr=STDOUT__, close_fds=True) #STDOUT标示，子进程的标准出错和__它的__标准输出__重定向到同一个__文件。
(child_stdin, child_stdout_and_stderr) = (p.stdin, p.stdout)
On Unix, os.popen2, os.popen3 and os.popen4 also accept __a sequence__ as the command to execute, in which case arguments will be passed directly to the program without shell intervention. This usage can be replaced as follows:
(child_stdin, child_stdout) = os.popen2(["/bin/ls", "-l"], mode,
bufsize)
==>
p = Popen(["/bin/ls", "-l"], bufsize=bufsize, stdin=PIPE, stdout=PIPE)
(child_stdin, child_stdout) = (p.stdin, p.stdout)
Return code handling translates as follows:
pipe = os.popen("cmd", 'w')
...
rc = pipe.close()
if rc is not None and rc >> 8:
print "There were some errors"
==>
process = Popen("cmd", 'w', shell=True, stdin=PIPE)
...
process.stdin.close()
if **process.wait()** != 0:
print "There were some errors"
==== 17.1.4.6. Replacing functions from the popen2 module ====
(child_stdout, child_stdin) = popen2.popen2("somestring", bufsize, mode)
==>
p = Popen(["somestring"], shell=True, bufsize=bufsize,
stdin=PIPE, stdout=PIPE, close_fds=True)
(child_stdout, child_stdin) = (p.stdout, p.stdin)
On Unix, popen2 also accepts a sequence as the command to execute, in which case arguments will be passed directly to the program without shell intervention. This usage can be replaced as follows:
(child_stdout, child_stdin) = popen2.popen2(["mycmd", "myarg"], bufsize,
mode)
==>
p = Popen(["mycmd", "myarg"], bufsize=bufsize,
stdin=PIPE, stdout=PIPE, close_fds=True)
(child_stdout, child_stdin) = (p.stdout, p.stdin)
popen2.Popen3 and popen2.Popen4 basically work as subprocess.Popen, except that:
Popen raises an exception if the execution fails.
the capturestderr argument is replaced with the stderr argument.
stdin=PIPE and stdout=PIPE must be specified.
popen2 closes all file descriptors by default, but you have to specify close_fds=True with Popen.
===== 17.1.5. Notes =====
17.1.5.1. Converting an argument sequence to a string on Windows
On Windows, an args sequence is converted to a string that can be parsed using the following rules (which correspond to the rules used by the MS C runtime):
Arguments are delimited by white space, which is either a space or a tab.
A string surrounded by double quotation marks is interpreted as a single argument, regardless of white space contained within. A quoted string can be embedded in an argument.
A double quotation mark preceded by a backslash is interpreted as a literal double quotation mark.
Backslashes are interpreted literally, unless they immediately precede a double quotation mark.
If backslashes immediately precede a double quotation mark, every pair of backslashes is interpreted as a literal backslash. If the number of backslashes is odd, the last backslash escapes the next double quotation mark as described in rule 3.

566
Zim/Programme/python/library/subprocess/subprocess_-_New_process_module.txt Normal file
View File

@@ -0,0 +1,566 @@
Content-Type: text/x-zim-wiki
Wiki-Format: zim 0.4
Creation-Date: 2012-01-06T10:23:27+08:00
====== subprocess - New process module ======
Created Friday 06 January 2012
http://www.python.org/dev/peps/pep-0324/
PEP: 324
Title: subprocess - New process module
Version: 17a68e052d4f
Last-Modified: 2007-06-19 04:20:07 +0000 (Tue, 19 Jun 2007)
Author: Peter Astrand <astrand at lysator.liu.se>
Status: Final
Type: Standards Track
Content-Type: text/plain
Created: 19-Nov-2003
Python-Version: 2.4
Post-History:
Abstract
This PEP describes a new module for starting and communicating
with processes.
===== Motivation =====
Starting new processes is a common task in any programming
language, and very common in a high-level language like Python.
Good support for this task is needed, because:
- Inappropriate functions for starting processes could mean a
security risk: If the program is __started through the shell__, and
the arguments contain shell meta characters, the result can be
disastrous. [1]
因此,subprocess模块中的函数和类，默认是不使用shell的。
- It makes Python an even better replacement language for
over-complicated shell scripts.
subprocess中的Popen类，支持子进程与调用进程的PIPE连接。
而且调用进程还可以使用信号等与子进程同步。
Currently, Python has a large number of different functions for
process creation. This makes it hard for developers to choose.
The subprocess module provides the following __enhancements__ over
previous functions:
- One "unified" module provides __all__ functionality from previous
functions.
- __Cross-process exceptions__: Exceptions happening in the child
before the new process has started to execute are re-raised in
the parent. This means that it's easy to handle **exec()**
** failures**, for example. With popen2, for example, it's
impossible to detect if the execution failed.
exec() failures会在调用进程中产生OSError异常，注意和子进程返回非0值的
SubprocessError进行区分。
- __A hook__ for executing custom code between fork and exec. This
can be used for, for example, changing uid.
这个__非常有用__：如定义进程组，改变进程权限，重定向文件等。
- **No implicit **call of /bin/sh. This means that there is no need
for escaping dangerous shell meta characters.
- All combinations of file descriptor redirection is possible.
For example, the "python-dialog" [2] needs to spawn a process
and redirect stderr, but not stdout. This is not possible with
current functions, without using temporary files.
子进程的stdin, stdout, stderr可以独立、任意的重定向。
- With the subprocess module, it's possible to control if all open
file descriptors __should be closed__ before the new program is
executed.
- Support for __connecting several subprocesses__ (shell "pipe").
- Universal newline support.
- A __communicate()__ method, which makes it easy to send stdin data
and read stdout and stderr data, without risking deadlocks.
Most people are aware of the __flow control issues__ involved with
child process communication, but not all have the patience or
skills to write a fully correct and deadlock-free select loop.
This means that many Python applications contain race
conditions. A communicate() method in the standard library
solves this problem.
===== Rationale =====
The following points summarizes the design:
- subprocess was based on popen2, which is tried-and-tested.
- The factory functions in popen2 have been removed, because I
consider the class constructor equally easy to work with.
- popen2 contains several factory functions and classes for
different combinations of redirection. subprocess, however,
contains **one single class**. Since the subprocess module supports
__12 different combinations__ of redirection, providing a class or
function for each of them would be cumbersome and not very
intuitive. Even with popen2, this is a readability problem.
For example, many people cannot tell the difference between
popen2.popen2 and popen2.popen4 without using the documentation.
- One small utility function is provided: __subprocess.call()__. It
aims to be an enhancement over os.system(), while still very
easy to use:
- It does not use the Standard C function system(), which has
limitations.
- It does not call the shell implicitly.
- No need for quoting; using __an argument list__.
- The return value is easier to work with.
The call() utility function accepts an 'args' argument, just
like the__ Popen__ class constructor. It waits for the command to
complete, then returns the **returncode** attribute. The
implementation is very simple:
def call(*args, **kwargs):
return __Popen(*args, **kwargs).wait()__
The motivation behind the call() function is simple: Starting a
process and wait for it to finish is a common task.
While Popen supports a wide range of options, many users have
simple needs. Many people are using os.system() today, mainly
because it provides a simple interface. Consider this example:
os.system("stty sane -F " + device)
With subprocess.call(), this would look like:
subprocess.call(["stty", "sane", "-F", device])
or, if executing **through the shell**:
subprocess.call("stty sane -F " + device, shell=True)
- The __"preexec" __functionality makes it possible to run arbitrary
code **between fork and exec**. One might ask why there are special
arguments for setting the **environment** and **current directory**, but
not for, for example, setting the uid. The answer is:
- Changing environment and working directory is considered
fairly common.
- Old functions like spawn() has support for an
"env"-argument.
- __env__ and __cwd__ are considered quite cross-platform: They make
sense even on Windows.
- On POSIX platforms, no extension module is required: the module
uses **os.fork(), os.execvp()** etc.
- On Windows platforms, the module requires either Mark Hammond's
Windows extensions[5], or a small extension module called
_subprocess.
===== Specification =====
This module defines__ one class called Popen__:
class Popen(args, bufsize=0, executable=None,
stdin=None, stdout=None, stderr=None,
preexec_fn=None, close_fds=False, shell=False,
cwd=None, env=None, universal_newlines=False,
startupinfo=None, creationflags=0):
Arguments are:
- args should be __a string__, or__ a sequence__ of program arguments.
The program to execute is normally the first item in the args
sequence or string, but can be __explicitly set__ by using the
executable argument.
On UNIX, with shell=False (default): In this case, the Popen
class uses __os.execvp()__ to execute the child program. args
should normally be a **sequence**. A string will be treated as a
sequence with the string as the only item (the program to
execute).
On UNIX, with shell=True: If args is a string, it specifies the
__command string__ to execute through the shell. If args is a
sequence, the first item specifies the command string, and any
additional items will be treated as __additional shell arguments__.
On Windows: the Popen class uses CreateProcess() to execute the
child program, which operates on strings. If args is a
sequence, it will be converted to a string using the
list2cmdline method. Please note that not all MS Windows
applications interpret the command line the same way: The
list2cmdline is designed for applications using the same rules
as the MS C runtime.
- bufsize, if given, has the same meaning as the corresponding
argument to the built-in open() function: 0 means __unbuffered__, 1
means __line buffered__, any other positive value means use a buffer
of (approximately) that size. A negative bufsize means to use
the system default, which usually means __fully buffered__. The
default value for bufsize is 0 (unbuffered).
- stdin, stdout and stderr specify the executed programs' standard
input, standard output and standard error **file handles**,
respectively. Valid values are__ PIPE__, an existing __file__
__ descriptor__ (a positive integer), an existing __file object__, and
__None__. PIPE indicates that a new pipe to the child should be
created. With None, no redirection will occur; the child's file
handles will be __inherited__ from the parent. Additionally, __stderr__
__ can be STDOUT__, which indicates that the stderr data from the
applications should be captured into the same file handle as for
stdout.
- If preexec_fn is set to a__ callable object__, this object will be
called in the child process just before the child is executed.
- If close_fds is true, all file descriptors except 0, 1 and 2
will be closed before the child process is executed.
- If shell is true, the specified command will be executed through
the shell.
- If cwd is not None, the current directory will be changed to cwd
before the child is executed.
- If env is not None, it defines the environment variables for the
new process.
- If** universal_newlines** is true, the file objects stdout and
stderr are opened as a text file, but lines may be terminated
by any of '\n', the Unix end-of-line convention, '\r', the
Macintosh convention or '\r\n', the Windows convention. All of
these external representations are seen as '\n' by the Python
program. Note: This feature is only available if Python is
built with universal newline support (the default). Also, the
newlines attribute of the file objects stdout, stdin and stderr
are not updated by the communicate() method.
- The startupinfo and creationflags, if given, will be passed to
the underlying CreateProcess() function. They can specify
things such as appearance of the main window and priority for
the new process. (Windows only)
This module also defines two shortcut functions:
- call(*args, **kwargs):
Run command with arguments. Wait for command to complete,
then return the returncode attribute.
The arguments are the same as for the Popen constructor.
Example:
retcode = call(["ls", "-l"])
===== Exceptions =====
----------
Exceptions raised in the child process, before the new program has
started to execute, will be__ re-raised__ in the parent.
在fork之后，exec之前发生的异常(如exec执行失败)将在父进程中重新生成。
Additionally, the exception object will have one extra attribute
called '**child_traceback**', which is a string containing traceback
information from the child's point of view.
The most common exception raised is __OSError__. This occurs, for
example, when trying to execute a non-existent file. Applications
should prepare for OSErrors.
A __ValueError__ will be raised if Popen is called with invalid
arguments.
===== Security =====
--------
Unlike some other popen functions, this implementation will never
call /bin/sh implicitly. This means that all characters,
including shell meta-characters, can safely be passed to child
processes.
===== Popen objects =====
-------------
Instances of the Popen class have the following methods:
poll()
Check if child process has terminated. Returns** returncode**
attribute.
wait()
Wait for child process to terminate. Returns **returncode**
attribute.
__communicate__(input=None)
Interact with process: Send data to stdin. Read data from
stdout and stderr, until end-of-file is reached. Wait for
process to terminate. The optional stdin argument should be a
string to be sent to the child process, or None, if no data
should be sent to the child.
communicate() returns a tuple (stdout, stderr).
Note: The data read is buffered in memory, so do not use this
method if the data size is large or unlimited.
The following attributes are also available:
__ stdin__
If the stdin argument is PIPE, this attribute is a __file object__
that provides input to the child process. Otherwise, it is
None.
stdout
If the stdout argument is PIPE, this attribute is a file
object that provides output from the child process.
Otherwise, it is None.
stderr
If the stderr argument is PIPE, this attribute is file object
that provides error output from the child process. Otherwise,
it is None.
__ pid__
The process ID of the child process.
__returncode__
The child return code. A None value indicates that the
process hasn't terminated yet. A negative value -N indicates
that the child was terminated by__ signal N__ (UNIX only).
===== Replacing older functions with the subprocess module =====
In this section, "a ==> b" means that b can be used as a
replacement for a.
Note: All functions in this section fail (more or less) silently
if the executed program cannot be found; this module raises an
OSError exception.
In the following examples, we assume that the subprocess module is
imported with "from subprocess import *".
Replacing /bin/sh shell backquote
---------------------------------
output=`mycmd myarg`
==>
output = Popen(["mycmd", "myarg"], stdout=PIPE).communicate()[0]
Replacing shell pipe line
-------------------------
output=`dmesg | grep hda`
==>
p1 = Popen(["dmesg"], stdout=PIPE)
p2 = Popen(["grep", "hda"], stdin=p1.stdout, stdout=PIPE)
output = p2.communicate()[0]
Replacing os.system()
---------------------
sts = os.system("mycmd" + " myarg")
==>
p = Popen("mycmd" + " myarg", shell=True)
sts = os.waitpid(p.pid, 0)
Note:
* Calling the program through the shell is usually not required.
* It's easier to look at the returncode attribute than the
exit status.
A more real-world example would look like this:
try:
retcode = call("mycmd" + " myarg", shell=True)
if retcode < 0:
print >>sys.stderr, "Child was terminated by signal", -retcode
else:
print >>sys.stderr, "Child returned", retcode
except OSError, e:
print >>sys.stderr, "Execution failed:", e
Replacing os.spawn*
-------------------
P_NOWAIT example:
pid = os.spawnlp(os.P_NOWAIT, "/bin/mycmd", "mycmd", "myarg")
==>
pid = Popen(["/bin/mycmd", "myarg"]).pid
P_WAIT example:
retcode = os.spawnlp(os.P_WAIT, "/bin/mycmd", "mycmd", "myarg")
==>
retcode = call(["/bin/mycmd", "myarg"])
Vector example:
os.spawnvp(os.P_NOWAIT, path, args)
==>
Popen([path] + args[1:])
Environment example:
os.spawnlpe(os.P_NOWAIT, "/bin/mycmd", "mycmd", "myarg", env)
==>
Popen(["/bin/mycmd", "myarg"], env={"PATH": "/usr/bin"})
Replacing os.popen*
-------------------
pipe = os.popen(cmd, mode='r', bufsize)
==>
pipe = Popen(cmd, shell=True, bufsize=bufsize, stdout=PIPE).stdout
pipe = os.popen(cmd, mode='w', bufsize)
==>
pipe = Popen(cmd, shell=True, bufsize=bufsize, stdin=PIPE).stdin
(child_stdin, child_stdout) = os.popen2(cmd, mode, bufsize)
==>
p = Popen(cmd, shell=True, bufsize=bufsize,
stdin=PIPE, stdout=PIPE, close_fds=True)
(child_stdin, child_stdout) = (p.stdin, p.stdout)
(child_stdin,
child_stdout,
child_stderr) = os.popen3(cmd, mode, bufsize)
==>
p = Popen(cmd, shell=True, bufsize=bufsize,
stdin=PIPE, stdout=PIPE, stderr=PIPE, close_fds=True)
(child_stdin,
child_stdout,
child_stderr) = (p.stdin, p.stdout, p.stderr)
(child_stdin, child_stdout_and_stderr) = os.popen4(cmd, mode, bufsize)
==>
p = Popen(cmd, shell=True, bufsize=bufsize,
stdin=PIPE, stdout=PIPE, stderr=STDOUT, close_fds=True)
(child_stdin, child_stdout_and_stderr) = (p.stdin, p.stdout)
Replacing popen2.*
------------------
Note: If the cmd argument to popen2 functions is a string, the
command is executed through /bin/sh. If it is a list, the command
is directly executed.
(child_stdout, child_stdin) = popen2.popen2("somestring", bufsize, mode)
==>
p = Popen(["somestring"], shell=True, bufsize=bufsize
stdin=PIPE, stdout=PIPE, close_fds=True)
(child_stdout, child_stdin) = (p.stdout, p.stdin)
(child_stdout, child_stdin) = popen2.popen2(["mycmd", "myarg"], bufsize, mode)
==>
p = Popen(["mycmd", "myarg"], bufsize=bufsize,
stdin=PIPE, stdout=PIPE, close_fds=True)
(child_stdout, child_stdin) = (p.stdout, p.stdin)
The popen2.Popen3 and popen3.Popen4 basically works as
subprocess.Popen, except that:
* subprocess.Popen raises an exception if the execution fails
* the capturestderr argument is replaced with the stderr argument.
* stdin=PIPE and stdout=PIPE must be specified.
* popen2 closes all file descriptors by default, but you have to
specify close_fds=True with subprocess.Popen.
Open Issues
Some features have been requested but is not yet implemented.
This includes:
* Support for managing a whole flock of subprocesses
* Support for managing "daemon" processes
* Built-in method for killing subprocesses
While these are useful features, it's expected that these can be
added later without problems.
* expect-like functionality, including pty support.
pty support is highly platform-dependent, which is a
problem. Also, there are already other modules that provide this
kind of functionality[6].
Backwards Compatibility
Since this is a new module, no major backward compatible issues
are expected. The module name "subprocess" might collide with
other, previous modules[3] with the same name, but the name
"subprocess" seems to be the best suggested name so far. The
first name of this module was "popen5", but this name was
considered too unintuitive. For a while, the module was called
"process", but this name is already used by Trent Mick's
module[4].
The functions and modules that this new module is trying to
replace (os.system, os.spawn*, os.popen*, popen2.*, commands.*)
are expected to be available in future Python versions for a long
time, to preserve backwards compatibility.
Reference Implementation
A reference implementation is available from
http://www.lysator.liu.se/~astrand/popen5/.
References
[1] Secure Programming for Linux and Unix HOWTO, section 8.3.
http://www.dwheeler.com/secure-programs/
[2] Python Dialog
http://pythondialog.sourceforge.net/
[3] http://www.iol.ie/~padraiga/libs/subProcess.py
[4] http://starship.python.net/crew/tmick/
[5] http://starship.python.net/crew/mhammond/win32/
[6] http://www.lysator.liu.se/~ceder/pcl-expect/
Copyright
This document has been placed in the public domain.

667
Zim/Programme/python/library/subprocess_–_Work_with_additional_processes.txt Normal file
View File

@@ -0,0 +1,667 @@
Content-Type: text/x-zim-wiki
Wiki-Format: zim 0.4
Creation-Date: 2012-01-05T21:07:10+08:00
====== subprocess – Work with additional processes ======
Created Thursday 05 January 2012
http://www.doughellmann.com/PyMOTW/subprocess/#module-subprocess
Purpose: Spawn and communicate with additional processes.
Available In: 2.4 and later
The subprocess module provides __a consistent interface__ to creating and working with additional processes. It offers a higher-level interface than some of the other available modules, and is intended to replace functions such as// os.system(), os.spawn*(), os.popen*(), popen2.*() and commands.*()//. To make it easier to compare subprocess with those other modules, many of the examples here re-create the ones used for// os and popen//.
The subprocess module defines one class,__ Popen__ and a few** wrapper functions** that use that class. The constructor for Popen takes arguments to set up the new process so the __parent can communicate with it via pipes__. It provides all of the functionality of the other modules and functions it replaces, and more. The API is consistent for all uses, and many of the extra steps of overhead needed (such as closing extra file descriptors and ensuring the pipes are closed) are “built in” instead of being handled by the application code separately.
Note
The API is roughly the same, but the underlying implementation is slightly different between Unix and Windows. All of the examples shown here were tested on Mac OS X. Behavior on a non-Unix OS will vary.
===== Running External Command =====
To run an external command __without interacting with it__, such as one would do with os.system(), Use the call() function.
import subprocess
# Simple command
[x] subprocess.call(['ls', '-1'], shell=True)
上面的代码是错误的，因为在使用序列的情况下, shell应该为__False__.否则python调用一个shell来执行序列中的第一个命令，而将序列中的其它元素作为**shell本身**的参数。这也是下面的输出结果不是长格式的原因。
The command line arguments are passed as a list of strings, which avoids the need for escaping quotes or other special characters that might be interpreted by the shell.
$ python subprocess_os_system.py
__init__.py
index.rst
interaction.py
repeater.py
signal_child.py
signal_parent.py
subprocess_check_call.py
subprocess_check_output.py
subprocess_check_output_error.py
subprocess_check_output_error_trap_output.py
subprocess_os_system.py
subprocess_pipes.py
subprocess_popen2.py
subprocess_popen3.py
subprocess_popen4.py
subprocess_popen_read.py
subprocess_popen_write.py
subprocess_shell_variables.py
subprocess_signal_parent_shell.py
subprocess_signal_setsid.py
Setting the__ shell__ argument to a true value causes subprocess to **spawn an intermediate shell process**, and tell it to run the command. The default is to run the command directly.
import subprocess
# Command with shell expansion
subprocess.call('echo $HOME', shell=True) #由于是fork一个shell来执行args，因此参数字符串中可以包含__shell可以识别的各种特殊字符和语法__。
Using an intermediate shell means that variables, glob patterns, and other special shell features in the command string are processed before the command is run.
$ python subprocess_shell_variables.py
/Users/dhellmann
===== Error Handling =====
The return value from call() is the__ exit code__ of the program. The caller is responsible for interpreting it to detect errors. The** check_call()** function works like call() except that the__ exit code is checked__, and if it indicates an error happened then a **CalledProcessError** exception is raised.
import subprocess
subprocess.check_call(['false'])
The false command always exits with a non-zero status code, which check_call() interprets as an error.
$ python subprocess_check_call.py
Traceback (most recent call last):
File "subprocess_check_call.py", line 11, in <module>
subprocess.check_call(['false'])
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.
7/subprocess.py", line 504, in check_call
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['false']' returned non-zero exit status 1
===== Capturing Output =====
The standard input and output channels for the process started by call() are__ bound to the parent’s input and output__. That means the **calling programm cannot capture the output of the command**. Use check_output() to capture the output for later processing.
import subprocess
__output__ = subprocess.check_output(['ls', '-1']) #捕获子进程的__标准输出__，并将其作为字符串返回。同时对命令的退出码进行检测，__若非0则产生异常，这时子进程被捕获的标准输出将被丢弃__。
print 'Have %d bytes in output' % len(output)
print output
The ls -1 command runs successfully, so the text it prints to standard output is captured and returned.
$ python subprocess_check_output.py
Have 462 bytes in output
__init__.py
index.rst
interaction.py
repeater.py
signal_child.py
signal_parent.py
subprocess_check_call.py
subprocess_check_output.py
subprocess_check_output_error.py
subprocess_check_output_error_trap_output.py
subprocess_os_system.py
subprocess_pipes.py
subprocess_popen2.py
subprocess_popen3.py
subprocess_popen4.py
subprocess_popen_read.py
subprocess_popen_write.py
subprocess_shell_variables.py
subprocess_signal_parent_shell.py
subprocess_signal_setsid.py
This script runs a series of commands__ in a subshell__. Messages are sent to standard output and standard error__ before__ the commands exit with an error code.
import subprocess
output = subprocess.check_output(
'echo to stdout; echo to stderr 1>&2; exit 1',
**shell=True**,
)
print 'Have %d bytes in output' % len(output)
print output
The message to standard error is printed to the console, but the __message to standard output is hidden__.
这是因为如果子进程没有成果执行，被__捕获的标准输出会被丢弃__，而stderr被**输出到终端**。
$ python subprocess_check_output_error.py
__to stderr__
Traceback (most recent call last):
File "subprocess_check_output_error.py", line 14, in <module>
shell=True,
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.
7/subprocess.py", line 537, in check_output
raise CalledProcessError(retcode, cmd, output=output)
subprocess.CalledProcessError: Command 'echo to stdout; echo to stderr
1>&2; exit 1' returned non-zero exit status 1
To prevent error messages from commands run through check_output() from being written to the console, set the stderr parameter to the constant __STDOUT__.
import subprocess
output = subprocess.check_output(
'echo to stdout; echo to stderr 1>&2; exit 1',
shell=True,
stderr=subprocess.STDOUT, #子进程的标准出错被重定向到**该进程的标准输出流**中，因此子进程的__标准出错流也会被捕获__。
)
print 'Have %d bytes in output' % len(output)
print output
Now the error and standard output channels are __merged together__ so if the command prints error messages, __they are captured__ and not sent to the console.
$ python subprocess_check_output_error_trap_output.py
Traceback (most recent call last):
File "subprocess_check_output_error_trap_output.py", line 15, in <module>
stderr=subprocess.STDOUT,
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 537, in check_output
raise CalledProcessError(retcode, cmd, output=output)
subprocess.CalledProcessError: Command 'echo to stdout; echo to stderr 1>&2; exit 1' returned non-zero exit status 1
===== Working with Pipes Directly =====
By passing different arguments for **stdin, stdout, and stderr** it is possible to mimic the variations of os.popen().
==== popen ====
To run a process and **read all of its output**, set the stdout value to __PIPE__ and call __communicate()__.
PIPE是在subprocess模块中定义的__全局变量，还有STDOUT__。
import subprocess
print '\nread:'
proc = subprocess.Popen(['echo', '"to stdout"'], #如果使用的是列表，则shell应该设置为False。
**stdout**=subprocess.__PIPE__,
)
#proc为一个**Popen实例**。
stdout_value = proc.communicate()__[0] #向子进程发送空内容，然后捕获它的所有标准输出和标准出错，直到子进程退出。使用此方法时，Popen()应该使用PIPE。__
print '\tstdout:', repr(stdout_value)
This is similar to the way popen() works, except that the reading is managed internally by the **Popen instance**.
$ python subprocess_popen_read.py
read:
stdout: '"to stdout"\n'
To set up a pipe to allow the calling program to **write data to it**, set __stdin to PIPE__.
import subprocess
print '\nwrite:'
proc = subprocess.Popen(['cat', '-'],
__stdin=subprocess.PIPE__,
)
proc.communicate('\tstdin: to stdin\n')
To send data to the standard input channel of the process **one time**, pass the data to communicate(). This is similar to using popen() with mode 'w'.
$ python -u subprocess_popen_write.py
write:
stdin: to stdin
==== popen2 ====
To set up the Popen instance for__ reading and writing__, use a combination of the previous techniques.
import subprocess
print '\npopen2:'
proc = subprocess.Popen(['cat', '-'],
** stdin**=subprocess.PIPE,
**stdout**=subprocess.PIPE,
)
stdout_value = proc.communicate('through stdin to stdout')__[0]__
print '\tpass through:', repr(stdout_value)
This sets up the pipe to mimic popen2().
$ python -u subprocess_popen2.py
popen2:
pass through: 'through stdin to stdout'
==== popen3 ====
It is also possible watch both of the streams for **stdin、stdout and stderr**, as with popen3().
import subprocess
print '\npopen3:'
proc = subprocess.Popen('cat -; echo "to stderr" 1>&2',
shell=True,
stdin=subprocess.PIPE,
stdout=subprocess.PIPE,
stderr=subprocess.PIPE,
)
**stdout_value, stderr_value** = proc.communicate('through stdin to stdout')
print '\tpass through:', repr(stdout_value)
print '\tstderr :', repr(stderr_value)
Reading from stderr works the same as with stdout. Passing PIPE tells Popen to attach to the channel, and communicate() reads all of the data from it before returning.
$ python -u subprocess_popen3.py
popen3:
pass through: 'through stdin to stdout'
stderr : 'to stderr\n'
==== popen4 ====
To** direct the error output from the process to its standard output channel**, use __STDOUT __for stderr instead of PIPE.
import subprocess
print '\npopen4:'
proc = subprocess.Popen('cat -; echo "to stderr" 1>&2',
shell=True,
stdin=subprocess.PIPE,
stdout=subprocess.PIPE,
** stderr=subprocess.STDOUT**,
)
stdout_value, stderr_value = proc.communicate('through stdin to stdout\n')
print '\tcombined output:', repr(stdout_value)
print '\tstderr value :', repr(stderr_value)
Combining the output in this way is similar to how popen4() works.
$ python -u subprocess_popen4.py
popen4:
combined output: 'through stdin to stdout\nto stderr\n'
stderr value : None
===== Connecting Segments of a Pipe =====
Multiple commands can be connected into a pipeline, similar to the way the Unix shell works, by creating separate Popen instances and__ chaining their inputs and outputs together__. The stdout attribute of one Popen instance is used as the stdin argument for the next in the pipeline, instead of the constant PIPE. The output is read from the stdout handle for the** final** command in the pipeline.
import subprocess
cat = subprocess.Popen(['cat', 'index.rst'],
stdout=subprocess.PIPE,
)
grep = subprocess.Popen(['grep', '.. include::'],
stdin=cat.stdout, #本子进程的标准输入__通过pipe__和上一个进程的标准输出相连。
stdout=subprocess.__PIPE__,
)
cut = subprocess.Popen(['cut', '-f', '3', '-d:'],
stdin=grep.stdout,
stdout=subprocess.PIPE,
)
end_of_pipe = __cut.stdout #返回一个文件对象。__
print 'Included files:'
for line in end_of_pipe: #打印文件中的所有行。
print '\t', line.strip()
This example reproduces the command line cat index.rst | grep ".. include" | cut -f 3 -d:, which reads the reStructuredText source file for this section and finds all of the lines that include other files, then prints only the filenames.
$ python -u subprocess_pipes.py
Included files:
subprocess_os_system.py
subprocess_shell_variables.py
subprocess_check_call.py
subprocess_check_output.py
subprocess_check_output_error.py
subprocess_check_output_error_trap_output.py
subprocess_popen_read.py
subprocess_popen_write.py
subprocess_popen2.py
subprocess_popen3.py
subprocess_popen4.py
subprocess_pipes.py
repeater.py
interaction.py
signal_child.py
signal_parent.py
subprocess_signal_parent_shell.py
subprocess_signal_setsid.py
===== Interacting with Another Command =====
All of the above examples assume a limited amount of interaction. The **communicate() **method reads __all __of the output and__ waits __for child process to exit before returning. It is also possible to write to and read from the** individual pipe handles** used by the Popen instance. A simple echo program that reads from standard input and writes to standard output illustrates this:
import **sys**
__sys.stderr__.write('repeater.py: starting\n')
sys.stderr.flush()
while True:
next_line = **sys.stdin.readline()**
if not next_line:
break
sys.stdout.write(next_line)
sys.stdout.flush()
sys.stderr.write('repeater.py: exiting\n')
sys.stderr.flush()
The script, repeater.py, writes to stderr when it starts and stops. That information can be used to show the__ lifetime__ of the child process.
The next interaction example uses the stdin and stdout file handles __owned by the Popen instance__ in different ways. In the first example, a sequence of 10 numbers are written to stdin of the process, and after each write the next line of output is read back. In the second example, the same 10 numbers are written but the output is __read all at once __using communicate().
import subprocess
print 'One line at a time:'
proc = subprocess.Popen('python repeater.py',
shell=True,
** stdin=subprocess.PIPE,**
** stdout=subprocess.PIPE,**
)
for i in range(10):
__proc.stdin.write__('%d\n' % i)
output = __proc.stdout.readline__()
print output.rstrip()
remainder = proc.communicate__()__[0] #发送空字符，因此子进程认为是读到文件结束符__因此终止__。
print remainder
print
print 'All output at once:'
proc = subprocess.Popen('python repeater.py',
shell=True,
stdin=subprocess.PIPE,
stdout=subprocess.PIPE,
)
for i in range(10):
proc.stdin.write('%d\n' % i)
output = proc.communicate()[0]
print output
The "repeater.py: exiting" lines **come at different points** in the output for each loop style.
$ python -u interaction.py
One line at a time:
repeater.py: starting
0
1
2
3
4
5
6
7
8
9
repeater.py: exiting
All output at once:
repeater.py: starting
repeater.py: exiting
0
1
2
3
4
5
6
7
8
9
===== Signaling Between Processes =====
每个Popen实例都有多个属性如：pid, returncode, stdin, stdout, stderr。
The os examples include a demonstration of __signaling between processes__ using **os.fork() and os.kill()**. Since__ each Popen instance provides a pid attribute__ with the process id of the child process, it is possible to do something similar with subprocess. For example, using this script for the child process to be executed by the parent process
#子进程源文件(signal_child.py)：
import **os**
import **signal**
import **time**
import** sys**
pid = os.getpid()
received = False
def signal_usr1(signum, frame):
"Callback invoked when a signal is received"
global received
received = True
print 'CHILD %6s: Received USR1' % pid
__ sys.stdout.flush()__
print 'CHILD %6s: Setting up signal handler' % pid
sys.stdout.flush()
signal.signal(__signal.SIGUSR1__, signal_usr1)
print 'CHILD %6s: Pausing to wait for signal' % pid
sys.stdout.flush()
time.sleep(3)
if not received:
print 'CHILD %6s: Never received signal' % pid
combined with this parent process
import os
import signal
import subprocess
import time
import sys
proc = subprocess.Popen(['python', 'signal_child.py']) #子进程在后台执行。
print 'PARENT : Pausing before sending signal...'
sys.stdout.flush()
time.sleep(1) #这个sleep是为了与子进程同步。
print 'PARENT : Signaling child'
sys.stdout.flush()
__os.kill(proc.pid, signal.SIGUSR1)__
the output is:
$ python signal_parent.py
PARENT : Pausing before sending signal...
CHILD 11668: Setting up signal handler
CHILD 11668: Pausing to wait for signal
PARENT : Signaling child
CHILD 11668: Received USR1
===== Process Groups / Sessions =====
Because of the way the__ process tree__ works under Unix, if the process created by **Popen spawns sub-processes**, those children will not receive any signals sent to the parent. That means, for example, it will be difficult to cause them to terminate by sending SIGINT or SIGTERM.
import os
import signal
import subprocess
import **tempfile**
import time
import sys
__script__ = '''#!/bin/sh
echo "Shell script in process $$"
set -x
python signal_child.py
'''
**script_file** = tempfile.__NamedTemporaryFile__('wt')
script_file.write(script)
script_file.flush()
proc = subprocess.Popen(['sh', script_file.__name__], close_fds=True)
print 'PARENT : Pausing before sending signal to child %s...' % proc.pid
sys.stdout.flush()
time.sleep(1)
print 'PARENT : Signaling child %s' % proc.pid
sys.stdout.flush()
os.kill(proc.__pid__, signal.SIGUSR1)
time.sleep(3)
The pid used to send the signal does not match the pid of the child of the shell script waiting for the signal because in this example, there are __three separate__ processes interacting:
* subprocess_signal_parent_shell.py
* __The Unix shell__ process **running the script** created by the main python program.
* signal_child.py
$ python subprocess_signal_parent_shell.py
PARENT : Pausing before sending signal to child 11671...
Shell script in process 11671
+ python signal_child.py
CHILD 11672: Setting up signal handler
CHILD 11672: Pausing to wait for signal
PARENT : Signaling child 11671
CHILD 11672: Never received signal
The solution to this problem is to__ use a process group__ to associate the children so they can **be signaled together**. The process group is created with __os.setsid()__, setting the “session id” to the process id of the current process. All child processes** inherit the session id**, and since it should **only be set set in the shell created by Popen and its descendants,** os.setsid() __should not be called in the parent process__. Instead, the function is passed to Popen as the __preexec_fn__ argument so it is run **after the fork() inside the new process, before it uses exec() to run the shell**.
import os
import signal
import subprocess
import tempfile
import time
import sys
script = '''#!/bin/sh
echo "Shell script in process $$"
set -x
python signal_child.py
'''
script_file = tempfile.NamedTemporaryFile('wt')
script_file.write(script)
script_file.flush()
proc = subprocess.Popen(['sh', script_file.name],
close_fds=True,
__preexec_fn=os.setsid__,
) #在fork后执行(exec)sh前执行os.setsid，这样__shell和ignal_child.py进程都在一个进程组中__。注意，这里的shell为__非交互式shell__，所以它在执行脚本时不会将脚本放在另一个进程组中以实现交互式shell的前后台命令的功能。
print 'PARENT : Pausing before sending signal to child %s...' % proc.pid
sys.stdout.flush()
time.sleep(1)
print 'PARENT : Signaling process group %s' % proc.pid
sys.stdout.flush()
__os.killpg__(__proc.pid__, signal.SIGUSR1)
time.sleep(3)
The sequence of events is:
The parent program instantiates Popen.
The Popen instance forks a new process.
The new process runs os.setsid().
The new process runs exec() to start the shell.
The shell runs the shell script.
The shell script forks again and that process execs Python.
Python runs signal_child.py.
The parent program signals the process group using the pid of the shell.
The shell and Python processes receive the signal. The shell ignores it. Python invokes the signal handler.
To signal the **entire process group**, use __os.killpg()__ with the pid value from the Popen instance.
$ python subprocess_signal_setsid.py
PARENT : Pausing before sending signal to child 11676...
Shell script in process 11676
+ python signal_child.py
CHILD 11677: Setting up signal handler
CHILD 11677: Pausing to wait for signal
PARENT : Signaling process group 11676
CHILD 11677: Received USR1
===== See also =====
* **subprocess**
Standard library documentation for this module.
* os
Although many are deprecated, the functions for working with processes found in the os module are still widely used in existing code.
* UNIX SIgnals and Process Groups
A good description of UNIX signaling and how process groups work.
* Advanced Programming in the UNIX(R) Environment
Covers working with multiple processes, such as handling signals, closing duplicated file descriptors, etc.
* pipes
Unix shell command pipeline templates in the standard library.
=================================================================
Doug Hellmann
It isn't clear if you're working exactly with the example program or if you have modified it.** When stdin is set to PIPE the new process will block trying to read input.** __The parent has to write something or close the stdin handle to send an end-of-file to the new process so it will have data to read so it does not block.__
Like
Reply
3 months ago
in reply to cheenu
Vincent
Note that by default the value for the __"bufsize" parameter of Popen is 0__, which means that stream communications are** not buffered**, resulting in a drop of performance if your process yields a lot of information on stdout/stderr.
Setting bufsize to__ -1 __improves the situation a lot !
I'm trying to make a python script that will log me onto a ssh server and then pass over the controls to me in the terminal.
So far I have:
import subprocess
proc = subprocess.call('ssh -t -t USERserver ',shell=True,)
But I cannot get it so that the python script can interact with ssh session and enter password and then pass over control to terminal user.
Like
Reply
6 months ago
Doug Hellmann
You probably need to set stdin and stdout for the process. You should be able to get more help on comp.lang.python.
Like
Reply
6 months ago
in reply to Scott S
nerd
The "Python Version: 2.4 and later" indicator at the top of this article is incorrect as "check_output" was not released until python 2.7
Doug
Wow. helpful in so many ways, especially that last bit about__ setting a process group and using os.killpg()__. I'm not sure I would have ever gotten that right just reading from the docs. I ran into the same issue...trying to end the child process, which was the shell, not realizing that the script the shell was executing was a __third process__, so it would always hang. I got around it by using "sh exec mycommand" but that might not work in every case, and I'm not sure what other side effects may be lurking because of it. Thanks again.
Like
barbara
Ha, thank you for this - I was poking around online last night looking for some help when I came across this post and had a lightbulb moment. :)
Like
Reply
2 years ago
Doug Hellmann
What platform are you on @pythonnewb? It sounds like the repeater.py file needs to have its permissions changed from the way they are packaged by default.
Like
Reply
2 years ago
pythonnewb
Ok, i get it..
here is an example
import __urllib2__
import subprocess
openurl = urllib2.urlopen(''http........')
grep = subprocess.Popen('grep HEAD',
shell=True,
stdin=subprocess.PIPE,
stdout=subprocess.PIPE,
stderr=subprocess.STDOUT
)
grep_stdout=grep.communicate(openurl.read(3000))[0]
print(grep_stdout)

148
Zim/Programme/python/library/virtualenvwrapper.txt Normal file
View File

@@ -0,0 +1,148 @@
Content-Type: text/x-zim-wiki
Wiki-Format: zim 0.4
Creation-Date: 2012-01-06T13:47:03+08:00
====== virtualenvwrapper ======
Created Friday 06 January 2012
http://mathematism.com/2009/07/30/presentation-pip-and-virtualenv/
Jul 30
Presentation: pip and virtualenv
I put together a walkthrough of __pip__ and __virtualenv __for the django-district July meeting. It was a slide-less presentation, focusing mostly on talking points and command-line walkthroughs. This post provides a recap of what I covered.
===== virtualenv =====
virtualenv is a tool to create** isolated Python environments**. It was created by Ian Bicking to provide a solution to a consistent problem.
===== The Problem =====
To illustrate the problem let’s pretend you have a Web server with two Web sites on it: mysite.com and anothersite.com. Each of your Web sites are sharing the **global site-packages** directory in /usr/lib/python2.x/site-packages.
{{./1.jpg}}
The global site-packages directory has Django 1.0.2, Pinax 0.5 and PIL 1.1.6 among other things. The sites are purring along, living in complete harmony.
One day, the client for anothersite.com decides they’d like to take advantage of some of the new features in Django 1.1. Now we have a problem. How do we upgrade anothersite.com’s dependencies without affecting mysite.com?
===== The Solution =====
anothersite.com and mysite.com need the ability to have __their own set of dependencies__. The way to achieve this is to create isolated Python environments using __virtualenv__.
{{./2.jpg}}
virtualenv creates an environment that has__ its own installation directories__ and isolates itself from other virtual environments. The environment contains** a site-packages directory**, installs__ setuptools__ and a Python interpreter that is **aware of its environment**.
This allows us to install a different set of dependencies for each of our sites. Now anothersite.com can upgrade to Django 1.1 and Pinax trunk without affecting mysite.com at all.
===== Installation =====
virtualenv is available on **PyPI**. Run the following command to install:
//easy_install virtualenv//
===== Working with virtualenv =====
While I could go into detail about the commands available with virtualenv, I honestly never use the virtualenv commands directly. I use suite of conveniences for virtualenv called __virtualenvwrapper__.
===== virtualenvwrapper =====
Doug Hellmann was kind enough to create virtualenvwrapper, which is best described on PyPI:
virtualenvwrapper is **a set of extensions** to Ian Bicking’s virtualenv tool. The extensions include wrappers for creating and deleting virtual environments and otherwise managing your development workflow, making it easier to work on more than one project at a time without introducing conflicts in their dependencies.
===== Screencast =====
I put together a quick screencast to go over the wrappers and hooks that virtualenvwrapper provides as stated below. You can also view the source code on Bitbucket. Doug’s code is well commented and I found out about several of these wrappers only after taking a look at the source.
===== Wrappers =====
The wrappers provided by **virtualenvwrapper** (that I know of) are:
下面的封装函数只有当用source /path/to/virtualenvwrapper.sh后才有效。
* mkvirtualenv (create a new virtualenv)
* rmvirtualenv (remove an existing virtualenv)
* workon (change the current virtualenv)
* add2virtualenv (add __external packages__ in a .pth file to current virtualenv)
* cdsitepackages (cd into the site-packages directory of **current** virtualenv)
* cdvirtualenv (cd into the__ root__ of the current virtualenv)
* deactivate (deactivate virtualenv, which calls several hooks)
===== Hooks =====
One of the coolest things about virtualenvwrapper is the ability to provide hooks __when an event occurs__. Hook files can be placed in ENV/bin/ and are simply plain-text files with __shell commands__. virtualenvwrapper provides the following hooks:
* postmkvirtualenv
* prermvirtualenv
* postrmvirtualenv
* postactivate
* predeactivate
* postdeactivate
===== Installation =====
virtualenvwrapper is available on __PyPI__. Run the following command to install:
//easy_install virtualenvwrapper//
Add two lines to your__ .bashrc__ to set the location where the virtual environments should live and the location of the script installed with this package:
//export WORKON_HOME=$HOME/.virtualenvs//
//source /usr/local/bin/virtualenvwrapper_bashrc//
You should also take a look at Doug’s blog post on virtualenvwrapper, which provides a great overview of why he created virtualenvwrapper and provides detailed setup instructions.
===== pip(easy_install的替代者，提供自动搜索、安装、移除，以及依赖文件) =====
Another tool created by Ian Bicking, __pip stands for pip installs Python packages.__ It is a replacement for easy_install that provides some great improvements including requirements files and support for version control systems.
==== Requirements files ====
Requirements files are plain text files that contain **a list of packages to be installed**. These text files allow you to create __repeatable installations__. As illustrated in this example file, there are several ways to specify a required package.
Package name with version requirements:
Django>=1.1
Pinax==0.5
Direct URL to a tarball containing a setup.py script:
http://effbot.org/downloads/Imaging-1.1.6.tar.gz
Editable checkouts from VCS repositories:
-e svn+http://svn.myproject.org/svn/MyProject/trunk#egg=MyProject
-e git+http://git.myproject.org/MyProject/#egg=MyProject
-e hg+ssh://hg@myproject.org/MyProject/#egg=MyProject
-e bzr+https://bzr.myproject.org/MyProject/trunk#egg=MyProject
Editable checkouts from VCS repositories with revision information:
-e svn+http://svn.myproject.org/svn/MyProject/trunk@2019#egg=MyProject
-e git://git.myproject.org/MyProject.git@da39a3ee5e6b4b0d3255bfef95601890afd80709#egg=MyProject
-e hg+http://hg.myproject.org/MyProject/@2019#egg=MyProject
-e bzr+https://bzr.myproject.org/MyProject/trunk/@2019#egg=MyProject
View my sample requirements file for more examples of checking out with revision information including branches, tags and dates.
===== Usage =====
Once you have a requirements file, Installing a package using **pip** is as simple as the following command:
__pip install -r /path/to/requirements.txt__
I should note that you don’t have to create a requirements file. You can simply issue any of the commands listed in the requirements file directly from the command line like so:
__pip install Django>=1.1__
pip install http://effbot.org/downloads/Imaging-1.1.6.tar.gz
pip install -e svn+http://myrepo/svn/MyApp#egg=MyApp
===== pip freeze =====
打印出**当前环境中所有非标准package名称及其版本号**，非常方便与复制当前环境。
pip doesn’t just install packages. It also provides a few other commands, including freeze:
pip freeze > /path/to/requirements.txt
This will __inspect the current environment and generate a requirements files that contains explicit version number__ for each of the installed packages. I generally use this as a starting point for creating a stable requirements file.
There are other commands as well. I’d encourage you to take a look at the documentation for pip to learn more about the various options.
===== Installation =====
pip is available on PyPI. Run the following command to install:
**easy_install pip**
You can also check out a copy from bitbucket. Run one of the following commands:
wget http://bitbucket.org/ianb/pip/raw/0.4/pip.py
OR
hg clone http://bitbucket.org/ianb/pip/
===== Conclusion =====
Using __virtualenv, virtualenvwrapper and pip__ together can help you create__ isolated, repeatable environments __with ease. We use these tools on a daily basis at Discovery Creative and it has saved us countless hours in addition to providing documentation for requirements for our applications and sites.

BIN
Zim/Programme/python/library/virtualenvwrapper/1.jpg Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 17 KiB

BIN
Zim/Programme/python/library/virtualenvwrapper/2.jpg Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 22 KiB