kernel/kernel_Notes
1
0
Fork 0
mirror of https://github.com/beyondx/Notes.git synced 2026-02-04 10:54:00 +08:00
Code Issues Packages Projects Releases Wiki Activity

updated

Browse Source
This commit is contained in:
geekard
2012-12-30 14:55:16 +08:00
parent 411a288513
commit f8638a5844
58 changed files with 9509 additions and 81 deletions
Download Patch File Download Diff File Expand all files Collapse all files

38
Zim/Programme/python/The-Python-Standard-Library/subprocess/Python_Subprocess_Module_Examples.txt Normal file
View File

@@ -0,0 +1,38 @@
Content-Type: text/x-zim-wiki
Wiki-Format: zim 0.4
Creation-Date: 2012-01-05T22:29:27+08:00
====== Python Subprocess Module Examples ======
Created Thursday 05 January 2012
http://www.moosechips.com/2010/07/python-subprocess-module-examples/
Some examples using the subprocess python module.
Make a system call three different ways:
#! /usr/bin/env python
import subprocess
# Use a sequence of args
return_code = subprocess.call(["echo", "hello sequence"])
# Set shell=true so we can use a simple string for the command
return_code = subprocess.call("echo hello string", shell=True)
# subprocess.call() is equivalent to using subprocess.Popen() and wait()
proc = subprocess.Popen("echo hello popen", shell=True)
return_code = proc.wait() # wait for process to finish so we can get the return code
Control stderr and stdout:
#! /usr/bin/env python
import subprocess
# Put stderr and stdout into pipes
proc = subprocess.Popen("echo hello stdout; echo hello stderr >&2", \
shell=True, stderr=subprocess.PIPE, stdout=subprocess.PIPE)
return_code = proc.wait()
# Read from pipes
for line in proc.stdout:
print("stdout: " + line.rstrip())
for line in proc.stderr:
print("stderr: " + line.rstrip())

566
Zim/Programme/python/The-Python-Standard-Library/subprocess/subprocess_-_New_process_module.txt Normal file
View File

@@ -0,0 +1,566 @@
Content-Type: text/x-zim-wiki
Wiki-Format: zim 0.4
Creation-Date: 2012-01-06T10:23:27+08:00
====== subprocess - New process module ======
Created Friday 06 January 2012
http://www.python.org/dev/peps/pep-0324/
PEP: 324
Title: subprocess - New process module
Version: 17a68e052d4f
Last-Modified: 2007-06-19 04:20:07 +0000 (Tue, 19 Jun 2007)
Author: Peter Astrand <astrand at lysator.liu.se>
Status: Final
Type: Standards Track
Content-Type: text/plain
Created: 19-Nov-2003
Python-Version: 2.4
Post-History:
Abstract
This PEP describes a new module for starting and communicating
with processes.
===== Motivation =====
Starting new processes is a common task in any programming
language, and very common in a high-level language like Python.
Good support for this task is needed, because:
- Inappropriate functions for starting processes could mean a
security risk: If the program is __started through the shell__, and
the arguments contain shell meta characters, the result can be
disastrous. [1]
因此,subprocess模块中的函数和类，默认是不使用shell的。
- It makes Python an even better replacement language for
over-complicated shell scripts.
subprocess中的Popen类，支持子进程与调用进程的PIPE连接。
而且调用进程还可以使用信号等与子进程同步。
Currently, Python has a large number of different functions for
process creation. This makes it hard for developers to choose.
The subprocess module provides the following __enhancements__ over
previous functions:
- One "unified" module provides __all__ functionality from previous
functions.
- __Cross-process exceptions__: Exceptions happening in the child
before the new process has started to execute are re-raised in
the parent. This means that it's easy to handle **exec()**
** failures**, for example. With popen2, for example, it's
impossible to detect if the execution failed.
exec() failures会在调用进程中产生OSError异常，注意和子进程返回非0值的
SubprocessError进行区分。
- __A hook__ for executing custom code between fork and exec. This
can be used for, for example, changing uid.
这个__非常有用__：如定义进程组，改变进程权限，重定向文件等。
- **No implicit **call of /bin/sh. This means that there is no need
for escaping dangerous shell meta characters.
- All combinations of file descriptor redirection is possible.
For example, the "python-dialog" [2] needs to spawn a process
and redirect stderr, but not stdout. This is not possible with
current functions, without using temporary files.
子进程的stdin, stdout, stderr可以独立、任意的重定向。
- With the subprocess module, it's possible to control if all open
file descriptors __should be closed__ before the new program is
executed.
- Support for __connecting several subprocesses__ (shell "pipe").
- Universal newline support.
- A __communicate()__ method, which makes it easy to send stdin data
and read stdout and stderr data, without risking deadlocks.
Most people are aware of the __flow control issues__ involved with
child process communication, but not all have the patience or
skills to write a fully correct and deadlock-free select loop.
This means that many Python applications contain race
conditions. A communicate() method in the standard library
solves this problem.
===== Rationale =====
The following points summarizes the design:
- subprocess was based on popen2, which is tried-and-tested.
- The factory functions in popen2 have been removed, because I
consider the class constructor equally easy to work with.
- popen2 contains several factory functions and classes for
different combinations of redirection. subprocess, however,
contains **one single class**. Since the subprocess module supports
__12 different combinations__ of redirection, providing a class or
function for each of them would be cumbersome and not very
intuitive. Even with popen2, this is a readability problem.
For example, many people cannot tell the difference between
popen2.popen2 and popen2.popen4 without using the documentation.
- One small utility function is provided: __subprocess.call()__. It
aims to be an enhancement over os.system(), while still very
easy to use:
- It does not use the Standard C function system(), which has
limitations.
- It does not call the shell implicitly.
- No need for quoting; using __an argument list__.
- The return value is easier to work with.
The call() utility function accepts an 'args' argument, just
like the__ Popen__ class constructor. It waits for the command to
complete, then returns the **returncode** attribute. The
implementation is very simple:
def call(*args, **kwargs):
return __Popen(*args, **kwargs).wait()__
The motivation behind the call() function is simple: Starting a
process and wait for it to finish is a common task.
While Popen supports a wide range of options, many users have
simple needs. Many people are using os.system() today, mainly
because it provides a simple interface. Consider this example:
os.system("stty sane -F " + device)
With subprocess.call(), this would look like:
subprocess.call(["stty", "sane", "-F", device])
or, if executing **through the shell**:
subprocess.call("stty sane -F " + device, shell=True)
- The __"preexec" __functionality makes it possible to run arbitrary
code **between fork and exec**. One might ask why there are special
arguments for setting the **environment** and **current directory**, but
not for, for example, setting the uid. The answer is:
- Changing environment and working directory is considered
fairly common.
- Old functions like spawn() has support for an
"env"-argument.
- __env__ and __cwd__ are considered quite cross-platform: They make
sense even on Windows.
- On POSIX platforms, no extension module is required: the module
uses **os.fork(), os.execvp()** etc.
- On Windows platforms, the module requires either Mark Hammond's
Windows extensions[5], or a small extension module called
_subprocess.
===== Specification =====
This module defines__ one class called Popen__:
class Popen(args, bufsize=0, executable=None,
stdin=None, stdout=None, stderr=None,
preexec_fn=None, close_fds=False, shell=False,
cwd=None, env=None, universal_newlines=False,
startupinfo=None, creationflags=0):
Arguments are:
- args should be __a string__, or__ a sequence__ of program arguments.
The program to execute is normally the first item in the args
sequence or string, but can be __explicitly set__ by using the
executable argument.
On UNIX, with shell=False (default): In this case, the Popen
class uses __os.execvp()__ to execute the child program. args
should normally be a **sequence**. A string will be treated as a
sequence with the string as the only item (the program to
execute).
On UNIX, with shell=True: If args is a string, it specifies the
__command string__ to execute through the shell. If args is a
sequence, the first item specifies the command string, and any
additional items will be treated as __additional shell arguments__.
On Windows: the Popen class uses CreateProcess() to execute the
child program, which operates on strings. If args is a
sequence, it will be converted to a string using the
list2cmdline method. Please note that not all MS Windows
applications interpret the command line the same way: The
list2cmdline is designed for applications using the same rules
as the MS C runtime.
- bufsize, if given, has the same meaning as the corresponding
argument to the built-in open() function: 0 means __unbuffered__, 1
means __line buffered__, any other positive value means use a buffer
of (approximately) that size. A negative bufsize means to use
the system default, which usually means __fully buffered__. The
default value for bufsize is 0 (unbuffered).
- stdin, stdout and stderr specify the executed programs' standard
input, standard output and standard error **file handles**,
respectively. Valid values are__ PIPE__, an existing __file__
__ descriptor__ (a positive integer), an existing __file object__, and
__None__. PIPE indicates that a new pipe to the child should be
created. With None, no redirection will occur; the child's file
handles will be __inherited__ from the parent. Additionally, __stderr__
__ can be STDOUT__, which indicates that the stderr data from the
applications should be captured into the same file handle as for
stdout.
- If preexec_fn is set to a__ callable object__, this object will be
called in the child process just before the child is executed.
- If close_fds is true, all file descriptors except 0, 1 and 2
will be closed before the child process is executed.
- If shell is true, the specified command will be executed through
the shell.
- If cwd is not None, the current directory will be changed to cwd
before the child is executed.
- If env is not None, it defines the environment variables for the
new process.
- If** universal_newlines** is true, the file objects stdout and
stderr are opened as a text file, but lines may be terminated
by any of '\n', the Unix end-of-line convention, '\r', the
Macintosh convention or '\r\n', the Windows convention. All of
these external representations are seen as '\n' by the Python
program. Note: This feature is only available if Python is
built with universal newline support (the default). Also, the
newlines attribute of the file objects stdout, stdin and stderr
are not updated by the communicate() method.
- The startupinfo and creationflags, if given, will be passed to
the underlying CreateProcess() function. They can specify
things such as appearance of the main window and priority for
the new process. (Windows only)
This module also defines two shortcut functions:
- call(*args, **kwargs):
Run command with arguments. Wait for command to complete,
then return the returncode attribute.
The arguments are the same as for the Popen constructor.
Example:
retcode = call(["ls", "-l"])
===== Exceptions =====
----------
Exceptions raised in the child process, before the new program has
started to execute, will be__ re-raised__ in the parent.
在fork之后，exec之前发生的异常(如exec执行失败)将在父进程中重新生成。
Additionally, the exception object will have one extra attribute
called '**child_traceback**', which is a string containing traceback
information from the child's point of view.
The most common exception raised is __OSError__. This occurs, for
example, when trying to execute a non-existent file. Applications
should prepare for OSErrors.
A __ValueError__ will be raised if Popen is called with invalid
arguments.
===== Security =====
--------
Unlike some other popen functions, this implementation will never
call /bin/sh implicitly. This means that all characters,
including shell meta-characters, can safely be passed to child
processes.
===== Popen objects =====
-------------
Instances of the Popen class have the following methods:
poll()
Check if child process has terminated. Returns** returncode**
attribute.
wait()
Wait for child process to terminate. Returns **returncode**
attribute.
__communicate__(input=None)
Interact with process: Send data to stdin. Read data from
stdout and stderr, until end-of-file is reached. Wait for
process to terminate. The optional stdin argument should be a
string to be sent to the child process, or None, if no data
should be sent to the child.
communicate() returns a tuple (stdout, stderr).
Note: The data read is buffered in memory, so do not use this
method if the data size is large or unlimited.
The following attributes are also available:
__ stdin__
If the stdin argument is PIPE, this attribute is a __file object__
that provides input to the child process. Otherwise, it is
None.
stdout
If the stdout argument is PIPE, this attribute is a file
object that provides output from the child process.
Otherwise, it is None.
stderr
If the stderr argument is PIPE, this attribute is file object
that provides error output from the child process. Otherwise,
it is None.
__ pid__
The process ID of the child process.
__returncode__
The child return code. A None value indicates that the
process hasn't terminated yet. A negative value -N indicates
that the child was terminated by__ signal N__ (UNIX only).
===== Replacing older functions with the subprocess module =====
In this section, "a ==> b" means that b can be used as a
replacement for a.
Note: All functions in this section fail (more or less) silently
if the executed program cannot be found; this module raises an
OSError exception.
In the following examples, we assume that the subprocess module is
imported with "from subprocess import *".
Replacing /bin/sh shell backquote
---------------------------------
output=`mycmd myarg`
==>
output = Popen(["mycmd", "myarg"], stdout=PIPE).communicate()[0]
Replacing shell pipe line
-------------------------
output=`dmesg | grep hda`
==>
p1 = Popen(["dmesg"], stdout=PIPE)
p2 = Popen(["grep", "hda"], stdin=p1.stdout, stdout=PIPE)
output = p2.communicate()[0]
Replacing os.system()
---------------------
sts = os.system("mycmd" + " myarg")
==>
p = Popen("mycmd" + " myarg", shell=True)
sts = os.waitpid(p.pid, 0)
Note:
* Calling the program through the shell is usually not required.
* It's easier to look at the returncode attribute than the
exit status.
A more real-world example would look like this:
try:
retcode = call("mycmd" + " myarg", shell=True)
if retcode < 0:
print >>sys.stderr, "Child was terminated by signal", -retcode
else:
print >>sys.stderr, "Child returned", retcode
except OSError, e:
print >>sys.stderr, "Execution failed:", e
Replacing os.spawn*
-------------------
P_NOWAIT example:
pid = os.spawnlp(os.P_NOWAIT, "/bin/mycmd", "mycmd", "myarg")
==>
pid = Popen(["/bin/mycmd", "myarg"]).pid
P_WAIT example:
retcode = os.spawnlp(os.P_WAIT, "/bin/mycmd", "mycmd", "myarg")
==>
retcode = call(["/bin/mycmd", "myarg"])
Vector example:
os.spawnvp(os.P_NOWAIT, path, args)
==>
Popen([path] + args[1:])
Environment example:
os.spawnlpe(os.P_NOWAIT, "/bin/mycmd", "mycmd", "myarg", env)
==>
Popen(["/bin/mycmd", "myarg"], env={"PATH": "/usr/bin"})
Replacing os.popen*
-------------------
pipe = os.popen(cmd, mode='r', bufsize)
==>
pipe = Popen(cmd, shell=True, bufsize=bufsize, stdout=PIPE).stdout
pipe = os.popen(cmd, mode='w', bufsize)
==>
pipe = Popen(cmd, shell=True, bufsize=bufsize, stdin=PIPE).stdin
(child_stdin, child_stdout) = os.popen2(cmd, mode, bufsize)
==>
p = Popen(cmd, shell=True, bufsize=bufsize,
stdin=PIPE, stdout=PIPE, close_fds=True)
(child_stdin, child_stdout) = (p.stdin, p.stdout)
(child_stdin,
child_stdout,
child_stderr) = os.popen3(cmd, mode, bufsize)
==>
p = Popen(cmd, shell=True, bufsize=bufsize,
stdin=PIPE, stdout=PIPE, stderr=PIPE, close_fds=True)
(child_stdin,
child_stdout,
child_stderr) = (p.stdin, p.stdout, p.stderr)
(child_stdin, child_stdout_and_stderr) = os.popen4(cmd, mode, bufsize)
==>
p = Popen(cmd, shell=True, bufsize=bufsize,
stdin=PIPE, stdout=PIPE, stderr=STDOUT, close_fds=True)
(child_stdin, child_stdout_and_stderr) = (p.stdin, p.stdout)
Replacing popen2.*
------------------
Note: If the cmd argument to popen2 functions is a string, the
command is executed through /bin/sh. If it is a list, the command
is directly executed.
(child_stdout, child_stdin) = popen2.popen2("somestring", bufsize, mode)
==>
p = Popen(["somestring"], shell=True, bufsize=bufsize
stdin=PIPE, stdout=PIPE, close_fds=True)
(child_stdout, child_stdin) = (p.stdout, p.stdin)
(child_stdout, child_stdin) = popen2.popen2(["mycmd", "myarg"], bufsize, mode)
==>
p = Popen(["mycmd", "myarg"], bufsize=bufsize,
stdin=PIPE, stdout=PIPE, close_fds=True)
(child_stdout, child_stdin) = (p.stdout, p.stdin)
The popen2.Popen3 and popen3.Popen4 basically works as
subprocess.Popen, except that:
* subprocess.Popen raises an exception if the execution fails
* the capturestderr argument is replaced with the stderr argument.
* stdin=PIPE and stdout=PIPE must be specified.
* popen2 closes all file descriptors by default, but you have to
specify close_fds=True with subprocess.Popen.
Open Issues
Some features have been requested but is not yet implemented.
This includes:
* Support for managing a whole flock of subprocesses
* Support for managing "daemon" processes
* Built-in method for killing subprocesses
While these are useful features, it's expected that these can be
added later without problems.
* expect-like functionality, including pty support.
pty support is highly platform-dependent, which is a
problem. Also, there are already other modules that provide this
kind of functionality[6].
Backwards Compatibility
Since this is a new module, no major backward compatible issues
are expected. The module name "subprocess" might collide with
other, previous modules[3] with the same name, but the name
"subprocess" seems to be the best suggested name so far. The
first name of this module was "popen5", but this name was
considered too unintuitive. For a while, the module was called
"process", but this name is already used by Trent Mick's
module[4].
The functions and modules that this new module is trying to
replace (os.system, os.spawn*, os.popen*, popen2.*, commands.*)
are expected to be available in future Python versions for a long
time, to preserve backwards compatibility.
Reference Implementation
A reference implementation is available from
http://www.lysator.liu.se/~astrand/popen5/.
References
[1] Secure Programming for Linux and Unix HOWTO, section 8.3.
http://www.dwheeler.com/secure-programs/
[2] Python Dialog
http://pythondialog.sourceforge.net/
[3] http://www.iol.ie/~padraiga/libs/subProcess.py
[4] http://starship.python.net/crew/tmick/
[5] http://starship.python.net/crew/mhammond/win32/
[6] http://www.lysator.liu.se/~ceder/pcl-expect/
Copyright
This document has been placed in the public domain.

667
Zim/Programme/python/The-Python-Standard-Library/subprocess/subprocess_–_Work_with_additional_processes.txt Normal file
View File

@@ -0,0 +1,667 @@
Content-Type: text/x-zim-wiki
Wiki-Format: zim 0.4
Creation-Date: 2012-01-05T21:07:10+08:00
====== subprocess – Work with additional processes ======
Created Thursday 05 January 2012
http://www.doughellmann.com/PyMOTW/subprocess/#module-subprocess
Purpose: Spawn and communicate with additional processes.
Available In: 2.4 and later
The subprocess module provides __a consistent interface__ to creating and working with additional processes. It offers a higher-level interface than some of the other available modules, and is intended to replace functions such as// os.system(), os.spawn*(), os.popen*(), popen2.*() and commands.*()//. To make it easier to compare subprocess with those other modules, many of the examples here re-create the ones used for// os and popen//.
The subprocess module defines one class,__ Popen__ and a few** wrapper functions** that use that class. The constructor for Popen takes arguments to set up the new process so the __parent can communicate with it via pipes__. It provides all of the functionality of the other modules and functions it replaces, and more. The API is consistent for all uses, and many of the extra steps of overhead needed (such as closing extra file descriptors and ensuring the pipes are closed) are “built in” instead of being handled by the application code separately.
Note
The API is roughly the same, but the underlying implementation is slightly different between Unix and Windows. All of the examples shown here were tested on Mac OS X. Behavior on a non-Unix OS will vary.
===== Running External Command =====
To run an external command __without interacting with it__, such as one would do with os.system(), Use the call() function.
import subprocess
# Simple command
[x] subprocess.call(['ls', '-1'], shell=True)
上面的代码是错误的，因为在使用序列的情况下, shell应该为__False__.否则python调用一个shell来执行序列中的第一个命令，而将序列中的其它元素作为**shell本身**的参数。这也是下面的输出结果不是长格式的原因。
The command line arguments are passed as a list of strings, which avoids the need for escaping quotes or other special characters that might be interpreted by the shell.
$ python subprocess_os_system.py
__init__.py
index.rst
interaction.py
repeater.py
signal_child.py
signal_parent.py
subprocess_check_call.py
subprocess_check_output.py
subprocess_check_output_error.py
subprocess_check_output_error_trap_output.py
subprocess_os_system.py
subprocess_pipes.py
subprocess_popen2.py
subprocess_popen3.py
subprocess_popen4.py
subprocess_popen_read.py
subprocess_popen_write.py
subprocess_shell_variables.py
subprocess_signal_parent_shell.py
subprocess_signal_setsid.py
Setting the__ shell__ argument to a true value causes subprocess to **spawn an intermediate shell process**, and tell it to run the command. The default is to run the command directly.
import subprocess
# Command with shell expansion
subprocess.call('echo $HOME', shell=True) #由于是fork一个shell来执行args，因此参数字符串中可以包含__shell可以识别的各种特殊字符和语法__。
Using an intermediate shell means that variables, glob patterns, and other special shell features in the command string are processed before the command is run.
$ python subprocess_shell_variables.py
/Users/dhellmann
===== Error Handling =====
The return value from call() is the__ exit code__ of the program. The caller is responsible for interpreting it to detect errors. The** check_call()** function works like call() except that the__ exit code is checked__, and if it indicates an error happened then a **CalledProcessError** exception is raised.
import subprocess
subprocess.check_call(['false'])
The false command always exits with a non-zero status code, which check_call() interprets as an error.
$ python subprocess_check_call.py
Traceback (most recent call last):
File "subprocess_check_call.py", line 11, in <module>
subprocess.check_call(['false'])
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.
7/subprocess.py", line 504, in check_call
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['false']' returned non-zero exit status 1
===== Capturing Output =====
The standard input and output channels for the process started by call() are__ bound to the parent’s input and output__. That means the **calling programm cannot capture the output of the command**. Use check_output() to capture the output for later processing.
import subprocess
__output__ = subprocess.check_output(['ls', '-1']) #捕获子进程的__标准输出__，并将其作为字符串返回。同时对命令的退出码进行检测，__若非0则产生异常，这时子进程被捕获的标准输出将被丢弃__。
print 'Have %d bytes in output' % len(output)
print output
The ls -1 command runs successfully, so the text it prints to standard output is captured and returned.
$ python subprocess_check_output.py
Have 462 bytes in output
__init__.py
index.rst
interaction.py
repeater.py
signal_child.py
signal_parent.py
subprocess_check_call.py
subprocess_check_output.py
subprocess_check_output_error.py
subprocess_check_output_error_trap_output.py
subprocess_os_system.py
subprocess_pipes.py
subprocess_popen2.py
subprocess_popen3.py
subprocess_popen4.py
subprocess_popen_read.py
subprocess_popen_write.py
subprocess_shell_variables.py
subprocess_signal_parent_shell.py
subprocess_signal_setsid.py
This script runs a series of commands__ in a subshell__. Messages are sent to standard output and standard error__ before__ the commands exit with an error code.
import subprocess
output = subprocess.check_output(
'echo to stdout; echo to stderr 1>&2; exit 1',
**shell=True**,
)
print 'Have %d bytes in output' % len(output)
print output
The message to standard error is printed to the console, but the __message to standard output is hidden__.
这是因为如果子进程没有成果执行，被__捕获的标准输出会被丢弃__，而stderr被**输出到终端**。
$ python subprocess_check_output_error.py
__to stderr__
Traceback (most recent call last):
File "subprocess_check_output_error.py", line 14, in <module>
shell=True,
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.
7/subprocess.py", line 537, in check_output
raise CalledProcessError(retcode, cmd, output=output)
subprocess.CalledProcessError: Command 'echo to stdout; echo to stderr
1>&2; exit 1' returned non-zero exit status 1
To prevent error messages from commands run through check_output() from being written to the console, set the stderr parameter to the constant __STDOUT__.
import subprocess
output = subprocess.check_output(
'echo to stdout; echo to stderr 1>&2; exit 1',
shell=True,
stderr=subprocess.STDOUT, #子进程的标准出错被重定向到**该进程的标准输出流**中，因此子进程的__标准出错流也会被捕获__。
)
print 'Have %d bytes in output' % len(output)
print output
Now the error and standard output channels are __merged together__ so if the command prints error messages, __they are captured__ and not sent to the console.
$ python subprocess_check_output_error_trap_output.py
Traceback (most recent call last):
File "subprocess_check_output_error_trap_output.py", line 15, in <module>
stderr=subprocess.STDOUT,
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 537, in check_output
raise CalledProcessError(retcode, cmd, output=output)
subprocess.CalledProcessError: Command 'echo to stdout; echo to stderr 1>&2; exit 1' returned non-zero exit status 1
===== Working with Pipes Directly =====
By passing different arguments for **stdin, stdout, and stderr** it is possible to mimic the variations of os.popen().
==== popen ====
To run a process and **read all of its output**, set the stdout value to __PIPE__ and call __communicate()__.
PIPE是在subprocess模块中定义的__全局变量，还有STDOUT__。
import subprocess
print '\nread:'
proc = subprocess.Popen(['echo', '"to stdout"'], #如果使用的是列表，则shell应该设置为False。
**stdout**=subprocess.__PIPE__,
)
#proc为一个**Popen实例**。
stdout_value = proc.communicate()__[0] #向子进程发送空内容，然后捕获它的所有标准输出和标准出错，直到子进程退出。使用此方法时，Popen()应该使用PIPE。__
print '\tstdout:', repr(stdout_value)
This is similar to the way popen() works, except that the reading is managed internally by the **Popen instance**.
$ python subprocess_popen_read.py
read:
stdout: '"to stdout"\n'
To set up a pipe to allow the calling program to **write data to it**, set __stdin to PIPE__.
import subprocess
print '\nwrite:'
proc = subprocess.Popen(['cat', '-'],
__stdin=subprocess.PIPE__,
)
proc.communicate('\tstdin: to stdin\n')
To send data to the standard input channel of the process **one time**, pass the data to communicate(). This is similar to using popen() with mode 'w'.
$ python -u subprocess_popen_write.py
write:
stdin: to stdin
==== popen2 ====
To set up the Popen instance for__ reading and writing__, use a combination of the previous techniques.
import subprocess
print '\npopen2:'
proc = subprocess.Popen(['cat', '-'],
** stdin**=subprocess.PIPE,
**stdout**=subprocess.PIPE,
)
stdout_value = proc.communicate('through stdin to stdout')__[0]__
print '\tpass through:', repr(stdout_value)
This sets up the pipe to mimic popen2().
$ python -u subprocess_popen2.py
popen2:
pass through: 'through stdin to stdout'
==== popen3 ====
It is also possible watch both of the streams for **stdin、stdout and stderr**, as with popen3().
import subprocess
print '\npopen3:'
proc = subprocess.Popen('cat -; echo "to stderr" 1>&2',
shell=True,
stdin=subprocess.PIPE,
stdout=subprocess.PIPE,
stderr=subprocess.PIPE,
)
**stdout_value, stderr_value** = proc.communicate('through stdin to stdout')
print '\tpass through:', repr(stdout_value)
print '\tstderr :', repr(stderr_value)
Reading from stderr works the same as with stdout. Passing PIPE tells Popen to attach to the channel, and communicate() reads all of the data from it before returning.
$ python -u subprocess_popen3.py
popen3:
pass through: 'through stdin to stdout'
stderr : 'to stderr\n'
==== popen4 ====
To** direct the error output from the process to its standard output channel**, use __STDOUT __for stderr instead of PIPE.
import subprocess
print '\npopen4:'
proc = subprocess.Popen('cat -; echo "to stderr" 1>&2',
shell=True,
stdin=subprocess.PIPE,
stdout=subprocess.PIPE,
** stderr=subprocess.STDOUT**,
)
stdout_value, stderr_value = proc.communicate('through stdin to stdout\n')
print '\tcombined output:', repr(stdout_value)
print '\tstderr value :', repr(stderr_value)
Combining the output in this way is similar to how popen4() works.
$ python -u subprocess_popen4.py
popen4:
combined output: 'through stdin to stdout\nto stderr\n'
stderr value : None
===== Connecting Segments of a Pipe =====
Multiple commands can be connected into a pipeline, similar to the way the Unix shell works, by creating separate Popen instances and__ chaining their inputs and outputs together__. The stdout attribute of one Popen instance is used as the stdin argument for the next in the pipeline, instead of the constant PIPE. The output is read from the stdout handle for the** final** command in the pipeline.
import subprocess
cat = subprocess.Popen(['cat', 'index.rst'],
stdout=subprocess.PIPE,
)
grep = subprocess.Popen(['grep', '.. include::'],
stdin=cat.stdout, #本子进程的标准输入__通过pipe__和上一个进程的标准输出相连。
stdout=subprocess.__PIPE__,
)
cut = subprocess.Popen(['cut', '-f', '3', '-d:'],
stdin=grep.stdout,
stdout=subprocess.PIPE,
)
end_of_pipe = __cut.stdout #返回一个文件对象。__
print 'Included files:'
for line in end_of_pipe: #打印文件中的所有行。
print '\t', line.strip()
This example reproduces the command line cat index.rst | grep ".. include" | cut -f 3 -d:, which reads the reStructuredText source file for this section and finds all of the lines that include other files, then prints only the filenames.
$ python -u subprocess_pipes.py
Included files:
subprocess_os_system.py
subprocess_shell_variables.py
subprocess_check_call.py
subprocess_check_output.py
subprocess_check_output_error.py
subprocess_check_output_error_trap_output.py
subprocess_popen_read.py
subprocess_popen_write.py
subprocess_popen2.py
subprocess_popen3.py
subprocess_popen4.py
subprocess_pipes.py
repeater.py
interaction.py
signal_child.py
signal_parent.py
subprocess_signal_parent_shell.py
subprocess_signal_setsid.py
===== Interacting with Another Command =====
All of the above examples assume a limited amount of interaction. The **communicate() **method reads __all __of the output and__ waits __for child process to exit before returning. It is also possible to write to and read from the** individual pipe handles** used by the Popen instance. A simple echo program that reads from standard input and writes to standard output illustrates this:
import **sys**
__sys.stderr__.write('repeater.py: starting\n')
sys.stderr.flush()
while True:
next_line = **sys.stdin.readline()**
if not next_line:
break
sys.stdout.write(next_line)
sys.stdout.flush()
sys.stderr.write('repeater.py: exiting\n')
sys.stderr.flush()
The script, repeater.py, writes to stderr when it starts and stops. That information can be used to show the__ lifetime__ of the child process.
The next interaction example uses the stdin and stdout file handles __owned by the Popen instance__ in different ways. In the first example, a sequence of 10 numbers are written to stdin of the process, and after each write the next line of output is read back. In the second example, the same 10 numbers are written but the output is __read all at once __using communicate().
import subprocess
print 'One line at a time:'
proc = subprocess.Popen('python repeater.py',
shell=True,
** stdin=subprocess.PIPE,**
** stdout=subprocess.PIPE,**
)
for i in range(10):
__proc.stdin.write__('%d\n' % i)
output = __proc.stdout.readline__()
print output.rstrip()
remainder = proc.communicate__()__[0] #发送空字符，因此子进程认为是读到文件结束符__因此终止__。
print remainder
print
print 'All output at once:'
proc = subprocess.Popen('python repeater.py',
shell=True,
stdin=subprocess.PIPE,
stdout=subprocess.PIPE,
)
for i in range(10):
proc.stdin.write('%d\n' % i)
output = proc.communicate()[0]
print output
The "repeater.py: exiting" lines **come at different points** in the output for each loop style.
$ python -u interaction.py
One line at a time:
repeater.py: starting
0
1
2
3
4
5
6
7
8
9
repeater.py: exiting
All output at once:
repeater.py: starting
repeater.py: exiting
0
1
2
3
4
5
6
7
8
9
===== Signaling Between Processes =====
每个Popen实例都有多个属性如：pid, returncode, stdin, stdout, stderr。
The os examples include a demonstration of __signaling between processes__ using **os.fork() and os.kill()**. Since__ each Popen instance provides a pid attribute__ with the process id of the child process, it is possible to do something similar with subprocess. For example, using this script for the child process to be executed by the parent process
#子进程源文件(signal_child.py)：
import **os**
import **signal**
import **time**
import** sys**
pid = os.getpid()
received = False
def signal_usr1(signum, frame):
"Callback invoked when a signal is received"
global received
received = True
print 'CHILD %6s: Received USR1' % pid
__ sys.stdout.flush()__
print 'CHILD %6s: Setting up signal handler' % pid
sys.stdout.flush()
signal.signal(__signal.SIGUSR1__, signal_usr1)
print 'CHILD %6s: Pausing to wait for signal' % pid
sys.stdout.flush()
time.sleep(3)
if not received:
print 'CHILD %6s: Never received signal' % pid
combined with this parent process
import os
import signal
import subprocess
import time
import sys
proc = subprocess.Popen(['python', 'signal_child.py']) #子进程在后台执行。
print 'PARENT : Pausing before sending signal...'
sys.stdout.flush()
time.sleep(1) #这个sleep是为了与子进程同步。
print 'PARENT : Signaling child'
sys.stdout.flush()
__os.kill(proc.pid, signal.SIGUSR1)__
the output is:
$ python signal_parent.py
PARENT : Pausing before sending signal...
CHILD 11668: Setting up signal handler
CHILD 11668: Pausing to wait for signal
PARENT : Signaling child
CHILD 11668: Received USR1
===== Process Groups / Sessions =====
Because of the way the__ process tree__ works under Unix, if the process created by **Popen spawns sub-processes**, those children will not receive any signals sent to the parent. That means, for example, it will be difficult to cause them to terminate by sending SIGINT or SIGTERM.
import os
import signal
import subprocess
import **tempfile**
import time
import sys
__script__ = '''#!/bin/sh
echo "Shell script in process $$"
set -x
python signal_child.py
'''
**script_file** = tempfile.__NamedTemporaryFile__('wt')
script_file.write(script)
script_file.flush()
proc = subprocess.Popen(['sh', script_file.__name__], close_fds=True)
print 'PARENT : Pausing before sending signal to child %s...' % proc.pid
sys.stdout.flush()
time.sleep(1)
print 'PARENT : Signaling child %s' % proc.pid
sys.stdout.flush()
os.kill(proc.__pid__, signal.SIGUSR1)
time.sleep(3)
The pid used to send the signal does not match the pid of the child of the shell script waiting for the signal because in this example, there are __three separate__ processes interacting:
* subprocess_signal_parent_shell.py
* __The Unix shell__ process **running the script** created by the main python program.
* signal_child.py
$ python subprocess_signal_parent_shell.py
PARENT : Pausing before sending signal to child 11671...
Shell script in process 11671
+ python signal_child.py
CHILD 11672: Setting up signal handler
CHILD 11672: Pausing to wait for signal
PARENT : Signaling child 11671
CHILD 11672: Never received signal
The solution to this problem is to__ use a process group__ to associate the children so they can **be signaled together**. The process group is created with __os.setsid()__, setting the “session id” to the process id of the current process. All child processes** inherit the session id**, and since it should **only be set set in the shell created by Popen and its descendants,** os.setsid() __should not be called in the parent process__. Instead, the function is passed to Popen as the __preexec_fn__ argument so it is run **after the fork() inside the new process, before it uses exec() to run the shell**.
import os
import signal
import subprocess
import tempfile
import time
import sys
script = '''#!/bin/sh
echo "Shell script in process $$"
set -x
python signal_child.py
'''
script_file = tempfile.NamedTemporaryFile('wt')
script_file.write(script)
script_file.flush()
proc = subprocess.Popen(['sh', script_file.name],
close_fds=True,
__preexec_fn=os.setsid__,
) #在fork后执行(exec)sh前执行os.setsid，这样__shell和ignal_child.py进程都在一个进程组中__。注意，这里的shell为__非交互式shell__，所以它在执行脚本时不会将脚本放在另一个进程组中以实现交互式shell的前后台命令的功能。
print 'PARENT : Pausing before sending signal to child %s...' % proc.pid
sys.stdout.flush()
time.sleep(1)
print 'PARENT : Signaling process group %s' % proc.pid
sys.stdout.flush()
__os.killpg__(__proc.pid__, signal.SIGUSR1)
time.sleep(3)
The sequence of events is:
The parent program instantiates Popen.
The Popen instance forks a new process.
The new process runs os.setsid().
The new process runs exec() to start the shell.
The shell runs the shell script.
The shell script forks again and that process execs Python.
Python runs signal_child.py.
The parent program signals the process group using the pid of the shell.
The shell and Python processes receive the signal. The shell ignores it. Python invokes the signal handler.
To signal the **entire process group**, use __os.killpg()__ with the pid value from the Popen instance.
$ python subprocess_signal_setsid.py
PARENT : Pausing before sending signal to child 11676...
Shell script in process 11676
+ python signal_child.py
CHILD 11677: Setting up signal handler
CHILD 11677: Pausing to wait for signal
PARENT : Signaling process group 11676
CHILD 11677: Received USR1
===== See also =====
* **subprocess**
Standard library documentation for this module.
* os
Although many are deprecated, the functions for working with processes found in the os module are still widely used in existing code.
* UNIX SIgnals and Process Groups
A good description of UNIX signaling and how process groups work.
* Advanced Programming in the UNIX(R) Environment
Covers working with multiple processes, such as handling signals, closing duplicated file descriptors, etc.
* pipes
Unix shell command pipeline templates in the standard library.
=================================================================
Doug Hellmann
It isn't clear if you're working exactly with the example program or if you have modified it.** When stdin is set to PIPE the new process will block trying to read input.** __The parent has to write something or close the stdin handle to send an end-of-file to the new process so it will have data to read so it does not block.__
Like
Reply
3 months ago
in reply to cheenu
Vincent
Note that by default the value for the __"bufsize" parameter of Popen is 0__, which means that stream communications are** not buffered**, resulting in a drop of performance if your process yields a lot of information on stdout/stderr.
Setting bufsize to__ -1 __improves the situation a lot !
I'm trying to make a python script that will log me onto a ssh server and then pass over the controls to me in the terminal.
So far I have:
import subprocess
proc = subprocess.call('ssh -t -t USERserver ',shell=True,)
But I cannot get it so that the python script can interact with ssh session and enter password and then pass over control to terminal user.
Like
Reply
6 months ago
Doug Hellmann
You probably need to set stdin and stdout for the process. You should be able to get more help on comp.lang.python.
Like
Reply
6 months ago
in reply to Scott S
nerd
The "Python Version: 2.4 and later" indicator at the top of this article is incorrect as "check_output" was not released until python 2.7
Doug
Wow. helpful in so many ways, especially that last bit about__ setting a process group and using os.killpg()__. I'm not sure I would have ever gotten that right just reading from the docs. I ran into the same issue...trying to end the child process, which was the shell, not realizing that the script the shell was executing was a __third process__, so it would always hang. I got around it by using "sh exec mycommand" but that might not work in every case, and I'm not sure what other side effects may be lurking because of it. Thanks again.
Like
barbara
Ha, thank you for this - I was poking around online last night looking for some help when I came across this post and had a lightbulb moment. :)
Like
Reply
2 years ago
Doug Hellmann
What platform are you on @pythonnewb? It sounds like the repeater.py file needs to have its permissions changed from the way they are packaged by default.
Like
Reply
2 years ago
pythonnewb
Ok, i get it..
here is an example
import __urllib2__
import subprocess
openurl = urllib2.urlopen(''http........')
grep = subprocess.Popen('grep HEAD',
shell=True,
stdin=subprocess.PIPE,
stdout=subprocess.PIPE,
stderr=subprocess.STDOUT
)
grep_stdout=grep.communicate(openurl.read(3000))[0]
print(grep_stdout)