"__future__" — Future statement definitions ******************************************* **Source code:** Lib/__future__.py ====================================================================== Imports of the form "from __future__ import feature" are called future statements. These are special-cased by the Python compiler to allow the use of new Python features in modules containing the future statement before the release in which the feature becomes standard. While these future statements are given additional special meaning by the Python compiler, they are still executed like any other import statement and the "__future__" exists and is handled by the import system the same way any other Python module would be. This design serves three purposes: * To avoid confusing existing tools that analyze import statements and expect to find the modules they’re importing. * To document when incompatible changes were introduced, and when they will be — or were — made mandatory. This is a form of executable documentation, and can be inspected programmatically via importing "__future__" and examining its contents. * To ensure that future statements run under releases prior to Python 2.1 at least yield runtime exceptions (the import of "__future__" will fail, because there was no module of that name prior to 2.1). Module Contents =============== No feature description will ever be deleted from "__future__". Since its introduction in Python 2.1 the following features have found their way into the language using this mechanism: +---------------------------+---------------------------+---------------------------+---------------------------+ | feature | optional in | mandatory in | effect | |===========================|===========================|===========================|===========================| | __future__.nested_scopes | 2.1.0b1 | 2.2 | **PEP 227**: *Statically | | | | | Nested Scopes* | +---------------------------+---------------------------+---------------------------+---------------------------+ | __future__.generators | 2.2.0a1 | 2.3 | **PEP 255**: *Simple | | | | | Generators* | +---------------------------+---------------------------+---------------------------+---------------------------+ | __future__.division | 2.2.0a2 | 3.0 | **PEP 238**: *Changing | | | | | the Division Operator* | +---------------------------+---------------------------+---------------------------+---------------------------+ | __future__.absolute_impo | 2.5.0a1 | 3.0 | **PEP 328**: *Imports: | | rt | | | Multi-Line and | | | | | Absolute/Relative* | +---------------------------+---------------------------+---------------------------+---------------------------+ | __future__.with_statement | 2.5.0a1 | 2.6 | **PEP 343**: *The “with” | | | | | Statement* | +---------------------------+---------------------------+---------------------------+---------------------------+ | __future__.print_function | 2.6.0a2 | 3.0 | **PEP 3105**: *Make print | | | | | a function* | +---------------------------+---------------------------+---------------------------+---------------------------+ | __future__.unicode_liter | 2.6.0a2 | 3.0 | **PEP 3112**: *Bytes | | als | | | literals in Python 3000* | +---------------------------+---------------------------+---------------------------+---------------------------+ | __future__.generator_stop | 3.5.0b1 | 3.7 | **PEP 479**: | | | | | *StopIteration handling | | | | | inside generators* | +---------------------------+---------------------------+---------------------------+---------------------------+ | __future__.annotations | 3.7.0b1 | Never [1] | **PEP 563**: *Postponed | | | | | evaluation of | | | | | annotations* | +---------------------------+---------------------------+---------------------------+---------------------------+ class __future__._Feature Each statement in "__future__.py" is of the form: FeatureName = _Feature(OptionalRelease, MandatoryRelease, CompilerFlag) where, normally, *OptionalRelease* is less than *MandatoryRelease*, and both are 5-tuples of the same form as "sys.version_info": (PY_MAJOR_VERSION, # the 2 in 2.1.0a3; an int PY_MINOR_VERSION, # the 1; an int PY_MICRO_VERSION, # the 0; an int PY_RELEASE_LEVEL, # "alpha", "beta", "candidate" or "final"; string PY_RELEASE_SERIAL # the 3; an int ) _Feature.getOptionalRelease() *OptionalRelease* records the first release in which the feature was accepted. _Feature.getMandatoryRelease() In the case of a *MandatoryRelease* that has not yet occurred, *MandatoryRelease* predicts the release in which the feature will become part of the language. Else *MandatoryRelease* records when the feature became part of the language; in releases at or after that, modules no longer need a future statement to use the feature in question, but may continue to use such imports. *MandatoryRelease* may also be "None", meaning that a planned feature got dropped or that it is not yet decided. _Feature.compiler_flag *CompilerFlag* is the (bitfield) flag that should be passed in the fourth argument to the built-in function "compile()" to enable the feature in dynamically compiled code. This flag is stored in the "_Feature.compiler_flag" attribute on "_Feature" instances. [1] "from __future__ import annotations" was previously scheduled to become mandatory in Python 3.10, but the Python Steering Council twice decided to delay the change (announcement for Python 3.10; announcement for Python 3.11). No final decision has been made yet. See also **PEP 563** and **PEP 649**. See also: Future statements How the compiler treats future imports. **PEP 236** - Back to the __future__ The original proposal for the __future__ mechanism. "__main__" — Top-level code environment *************************************** ====================================================================== In Python, the special name "__main__" is used for two important constructs: 1. the name of the top-level environment of the program, which can be checked using the "__name__ == '__main__'" expression; and 2. the "__main__.py" file in Python packages. Both of these mechanisms are related to Python modules; how users interact with them and how they interact with each other. They are explained in detail below. If you’re new to Python modules, see the tutorial section Modules for an introduction. "__name__ == '__main__'" ======================== When a Python module or package is imported, "__name__" is set to the module’s name. Usually, this is the name of the Python file itself without the ".py" extension: >>> import configparser >>> configparser.__name__ 'configparser' If the file is part of a package, "__name__" will also include the parent package’s path: >>> from concurrent.futures import process >>> process.__name__ 'concurrent.futures.process' However, if the module is executed in the top-level code environment, its "__name__" is set to the string "'__main__'". What is the “top-level code environment”? ----------------------------------------- "__main__" is the name of the environment where top-level code is run. “Top-level code” is the first user-specified Python module that starts running. It’s “top-level” because it imports all other modules that the program needs. Sometimes “top-level code” is called an *entry point* to the application. The top-level code environment can be: * the scope of an interactive prompt: >>> __name__ '__main__' * the Python module passed to the Python interpreter as a file argument: $ python helloworld.py Hello, world! * the Python module or package passed to the Python interpreter with the "-m" argument: $ python -m tarfile usage: tarfile.py [-h] [-v] (...) * Python code read by the Python interpreter from standard input: $ echo "import this" | python The Zen of Python, by Tim Peters Beautiful is better than ugly. Explicit is better than implicit. ... * Python code passed to the Python interpreter with the "-c" argument: $ python -c "import this" The Zen of Python, by Tim Peters Beautiful is better than ugly. Explicit is better than implicit. ... In each of these situations, the top-level module’s "__name__" is set to "'__main__'". As a result, a module can discover whether or not it is running in the top-level environment by checking its own "__name__", which allows a common idiom for conditionally executing code when the module is not initialized from an import statement: if __name__ == '__main__': # Execute when the module is not initialized from an import statement. ... See also: For a more detailed look at how "__name__" is set in all situations, see the tutorial section Modules. Idiomatic Usage --------------- Some modules contain code that is intended for script use only, like parsing command-line arguments or fetching data from standard input. If a module like this was imported from a different module, for example to unit test it, the script code would unintentionally execute as well. This is where using the "if __name__ == '__main__'" code block comes in handy. Code within this block won’t run unless the module is executed in the top-level environment. Putting as few statements as possible in the block below "if __name__ == '__main__'" can improve code clarity and correctness. Most often, a function named "main" encapsulates the program’s primary behavior: # echo.py import shlex import sys def echo(phrase: str) -> None: """A dummy wrapper around print.""" # for demonstration purposes, you can imagine that there is some # valuable and reusable logic inside this function print(phrase) def main() -> int: """Echo the input arguments to standard output""" phrase = shlex.join(sys.argv) echo(phrase) return 0 if __name__ == '__main__': sys.exit(main()) # next section explains the use of sys.exit Note that if the module didn’t encapsulate code inside the "main" function but instead put it directly within the "if __name__ == '__main__'" block, the "phrase" variable would be global to the entire module. This is error-prone as other functions within the module could be unintentionally using the global variable instead of a local name. A "main" function solves this problem. Using a "main" function has the added benefit of the "echo" function itself being isolated and importable elsewhere. When "echo.py" is imported, the "echo" and "main" functions will be defined, but neither of them will be called, because "__name__ != '__main__'". Packaging Considerations ------------------------ "main" functions are often used to create command-line tools by specifying them as entry points for console scripts. When this is done, pip inserts the function call into a template script, where the return value of "main" is passed into "sys.exit()". For example: sys.exit(main()) Since the call to "main" is wrapped in "sys.exit()", the expectation is that your function will return some value acceptable as an input to "sys.exit()"; typically, an integer or "None" (which is implicitly returned if your function does not have a return statement). By proactively following this convention ourselves, our module will have the same behavior when run directly (i.e. "python echo.py") as it will have if we later package it as a console script entry-point in a pip-installable package. In particular, be careful about returning strings from your "main" function. "sys.exit()" will interpret a string argument as a failure message, so your program will have an exit code of "1", indicating failure, and the string will be written to "sys.stderr". The "echo.py" example from earlier exemplifies using the "sys.exit(main())" convention. See also: Python Packaging User Guide contains a collection of tutorials and references on how to distribute and install Python packages with modern tools. "__main__.py" in Python Packages ================================ If you are not familiar with Python packages, see section Packages of the tutorial. Most commonly, the "__main__.py" file is used to provide a command-line interface for a package. Consider the following hypothetical package, “bandclass”: bandclass ├── __init__.py ├── __main__.py └── student.py "__main__.py" will be executed when the package itself is invoked directly from the command line using the "-m" flag. For example: $ python -m bandclass This command will cause "__main__.py" to run. How you utilize this mechanism will depend on the nature of the package you are writing, but in this hypothetical case, it might make sense to allow the teacher to search for students: # bandclass/__main__.py import sys from .student import search_students student_name = sys.argv[1] if len(sys.argv) >= 2 else '' print(f'Found student: {search_students(student_name)}') Note that "from .student import search_students" is an example of a relative import. This import style can be used when referencing modules within a package. For more details, see Intra-package References in the Modules section of the tutorial. Idiomatic Usage --------------- The content of "__main__.py" typically isn’t fenced with an "if __name__ == '__main__'" block. Instead, those files are kept short and import functions to execute from other modules. Those other modules can then be easily unit-tested and are properly reusable. If used, an "if __name__ == '__main__'" block will still work as expected for a "__main__.py" file within a package, because its "__name__" attribute will include the package’s path if imported: >>> import asyncio.__main__ >>> asyncio.__main__.__name__ 'asyncio.__main__' This won’t work for "__main__.py" files in the root directory of a ".zip" file though. Hence, for consistency, a minimal "__main__.py" without a "__name__" check is preferred. See also: See "venv" for an example of a package with a minimal "__main__.py" in the standard library. It doesn’t contain a "if __name__ == '__main__'" block. You can invoke it with "python -m venv [directory]". See "runpy" for more details on the "-m" flag to the interpreter executable. See "zipapp" for how to run applications packaged as *.zip* files. In this case Python looks for a "__main__.py" file in the root directory of the archive. "import __main__" ================= Regardless of which module a Python program was started with, other modules running within that same program can import the top-level environment’s scope (*namespace*) by importing the "__main__" module. This doesn’t import a "__main__.py" file but rather whichever module that received the special name "'__main__'". Here is an example module that consumes the "__main__" namespace: # namely.py import __main__ def did_user_define_their_name(): return 'my_name' in dir(__main__) def print_user_name(): if not did_user_define_their_name(): raise ValueError('Define the variable `my_name`!') if '__file__' in dir(__main__): print(__main__.my_name, "found in file", __main__.__file__) else: print(__main__.my_name) Example usage of this module could be as follows: # start.py import sys from namely import print_user_name # my_name = "Dinsdale" def main(): try: print_user_name() except ValueError as ve: return str(ve) if __name__ == "__main__": sys.exit(main()) Now, if we started our program, the result would look like this: $ python start.py Define the variable `my_name`! The exit code of the program would be 1, indicating an error. Uncommenting the line with "my_name = "Dinsdale"" fixes the program and now it exits with status code 0, indicating success: $ python start.py Dinsdale found in file /path/to/start.py Note that importing "__main__" doesn’t cause any issues with unintentionally running top-level code meant for script use which is put in the "if __name__ == "__main__"" block of the "start" module. Why does this work? Python inserts an empty "__main__" module in "sys.modules" at interpreter startup, and populates it by running top-level code. In our example this is the "start" module which runs line by line and imports "namely". In turn, "namely" imports "__main__" (which is really "start"). That’s an import cycle! Fortunately, since the partially populated "__main__" module is present in "sys.modules", Python passes that to "namely". See Special considerations for __main__ in the import system’s reference for details on how this works. The Python REPL is another example of a “top-level environment”, so anything defined in the REPL becomes part of the "__main__" scope: >>> import namely >>> namely.did_user_define_their_name() False >>> namely.print_user_name() Traceback (most recent call last): ... ValueError: Define the variable `my_name`! >>> my_name = 'Jabberwocky' >>> namely.did_user_define_their_name() True >>> namely.print_user_name() Jabberwocky Note that in this case the "__main__" scope doesn’t contain a "__file__" attribute as it’s interactive. The "__main__" scope is used in the implementation of "pdb" and "rlcompleter". "_thread" — Low-level threading API *********************************** ====================================================================== This module provides low-level primitives for working with multiple threads (also called *light-weight processes* or *tasks*) — multiple threads of control sharing their global data space. For synchronization, simple locks (also called *mutexes* or *binary semaphores*) are provided. The "threading" module provides an easier to use and higher-level threading API built on top of this module. Changed in version 3.7: This module used to be optional, it is now always available. This module defines the following constants and functions: exception _thread.error Raised on thread-specific errors. Changed in version 3.3: This is now a synonym of the built-in "RuntimeError". _thread.LockType This is the type of lock objects. _thread.start_new_thread(function, args[, kwargs]) Start a new thread and return its identifier. The thread executes the function *function* with the argument list *args* (which must be a tuple). The optional *kwargs* argument specifies a dictionary of keyword arguments. When the function returns, the thread silently exits. When the function terminates with an unhandled exception, "sys.unraisablehook()" is called to handle the exception. The *object* attribute of the hook argument is *function*. By default, a stack trace is printed and then the thread exits (but other threads continue to run). When the function raises a "SystemExit" exception, it is silently ignored. Raises an auditing event "_thread.start_new_thread" with arguments "function", "args", "kwargs". Changed in version 3.8: "sys.unraisablehook()" is now used to handle unhandled exceptions. _thread.interrupt_main(signum=signal.SIGINT, /) Simulate the effect of a signal arriving in the main thread. A thread can use this function to interrupt the main thread, though there is no guarantee that the interruption will happen immediately. If given, *signum* is the number of the signal to simulate. If *signum* is not given, "signal.SIGINT" is simulated. If the given signal isn’t handled by Python (it was set to "signal.SIG_DFL" or "signal.SIG_IGN"), this function does nothing. Changed in version 3.10: The *signum* argument is added to customize the signal number. Note: This does not emit the corresponding signal but schedules a call to the associated handler (if it exists). If you want to truly emit the signal, use "signal.raise_signal()". _thread.exit() Raise the "SystemExit" exception. When not caught, this will cause the thread to exit silently. _thread.allocate_lock() Return a new lock object. Methods of locks are described below. The lock is initially unlocked. _thread.get_ident() Return the ‘thread identifier’ of the current thread. This is a nonzero integer. Its value has no direct meaning; it is intended as a magic cookie to be used e.g. to index a dictionary of thread- specific data. Thread identifiers may be recycled when a thread exits and another thread is created. _thread.get_native_id() Return the native integral Thread ID of the current thread assigned by the kernel. This is a non-negative integer. Its value may be used to uniquely identify this particular thread system-wide (until the thread terminates, after which the value may be recycled by the OS). Availability: Windows, FreeBSD, Linux, macOS, OpenBSD, NetBSD, AIX, DragonFlyBSD, GNU/kFreeBSD. Added in version 3.8. Changed in version 3.13: Added support for GNU/kFreeBSD. _thread.stack_size([size]) Return the thread stack size used when creating new threads. The optional *size* argument specifies the stack size to be used for subsequently created threads, and must be 0 (use platform or configured default) or a positive integer value of at least 32,768 (32 KiB). If *size* is not specified, 0 is used. If changing the thread stack size is unsupported, a "RuntimeError" is raised. If the specified stack size is invalid, a "ValueError" is raised and the stack size is unmodified. 32 KiB is currently the minimum supported stack size value to guarantee sufficient stack space for the interpreter itself. Note that some platforms may have particular restrictions on values for the stack size, such as requiring a minimum stack size > 32 KiB or requiring allocation in multiples of the system memory page size - platform documentation should be referred to for more information (4 KiB pages are common; using multiples of 4096 for the stack size is the suggested approach in the absence of more specific information). Availability: Windows, pthreads. Unix platforms with POSIX threads support. _thread.TIMEOUT_MAX The maximum value allowed for the *timeout* parameter of "Lock.acquire". Specifying a timeout greater than this value will raise an "OverflowError". Added in version 3.2. Lock objects have the following methods: lock.acquire(blocking=True, timeout=-1) Without any optional argument, this method acquires the lock unconditionally, if necessary waiting until it is released by another thread (only one thread at a time can acquire a lock — that’s their reason for existence). If the *blocking* argument is present, the action depends on its value: if it is false, the lock is only acquired if it can be acquired immediately without waiting, while if it is true, the lock is acquired unconditionally as above. If the floating-point *timeout* argument is present and positive, it specifies the maximum wait time in seconds before returning. A negative *timeout* argument specifies an unbounded wait. You cannot specify a *timeout* if *blocking* is false. The return value is "True" if the lock is acquired successfully, "False" if not. Changed in version 3.2: The *timeout* parameter is new. Changed in version 3.2: Lock acquires can now be interrupted by signals on POSIX. lock.release() Releases the lock. The lock must have been acquired earlier, but not necessarily by the same thread. lock.locked() Return the status of the lock: "True" if it has been acquired by some thread, "False" if not. In addition to these methods, lock objects can also be used via the "with" statement, e.g.: import _thread a_lock = _thread.allocate_lock() with a_lock: print("a_lock is locked while this executes") **Caveats:** * Interrupts always go to the main thread (the "KeyboardInterrupt" exception will be received by that thread.) * Calling "sys.exit()" or raising the "SystemExit" exception is equivalent to calling "_thread.exit()". * It is platform-dependent whether the "acquire()" method on a lock can be interrupted (so that the "KeyboardInterrupt" exception will happen immediately, rather than only after the lock has been acquired or the operation has timed out). It can be interrupted on POSIX, but not on Windows. * When the main thread exits, it is system defined whether the other threads survive. On most systems, they are killed without executing "try" … "finally" clauses or executing object destructors. "abc" — Abstract Base Classes ***************************** **Source code:** Lib/abc.py ====================================================================== This module provides the infrastructure for defining *abstract base classes* (ABCs) in Python, as outlined in **PEP 3119**; see the PEP for why this was added to Python. (See also **PEP 3141** and the "numbers" module regarding a type hierarchy for numbers based on ABCs.) The "collections" module has some concrete classes that derive from ABCs; these can, of course, be further derived. In addition, the "collections.abc" submodule has some ABCs that can be used to test whether a class or instance provides a particular interface, for example, if it is *hashable* or if it is a *mapping*. This module provides the metaclass "ABCMeta" for defining ABCs and a helper class "ABC" to alternatively define ABCs through inheritance: class abc.ABC A helper class that has "ABCMeta" as its metaclass. With this class, an abstract base class can be created by simply deriving from "ABC" avoiding sometimes confusing metaclass usage, for example: from abc import ABC class MyABC(ABC): pass Note that the type of "ABC" is still "ABCMeta", therefore inheriting from "ABC" requires the usual precautions regarding metaclass usage, as multiple inheritance may lead to metaclass conflicts. One may also define an abstract base class by passing the metaclass keyword and using "ABCMeta" directly, for example: from abc import ABCMeta class MyABC(metaclass=ABCMeta): pass Added in version 3.4. class abc.ABCMeta Metaclass for defining Abstract Base Classes (ABCs). Use this metaclass to create an ABC. An ABC can be subclassed directly, and then acts as a mix-in class. You can also register unrelated concrete classes (even built-in classes) and unrelated ABCs as “virtual subclasses” – these and their descendants will be considered subclasses of the registering ABC by the built-in "issubclass()" function, but the registering ABC won’t show up in their MRO (Method Resolution Order) nor will method implementations defined by the registering ABC be callable (not even via "super()"). [1] Classes created with a metaclass of "ABCMeta" have the following method: register(subclass) Register *subclass* as a “virtual subclass” of this ABC. For example: from abc import ABC class MyABC(ABC): pass MyABC.register(tuple) assert issubclass(tuple, MyABC) assert isinstance((), MyABC) Changed in version 3.3: Returns the registered subclass, to allow usage as a class decorator. Changed in version 3.4: To detect calls to "register()", you can use the "get_cache_token()" function. You can also override this method in an abstract base class: __subclasshook__(subclass) (Must be defined as a class method.) Check whether *subclass* is considered a subclass of this ABC. This means that you can customize the behavior of "issubclass()" further without the need to call "register()" on every class you want to consider a subclass of the ABC. (This class method is called from the "__subclasscheck__()" method of the ABC.) This method should return "True", "False" or "NotImplemented". If it returns "True", the *subclass* is considered a subclass of this ABC. If it returns "False", the *subclass* is not considered a subclass of this ABC, even if it would normally be one. If it returns "NotImplemented", the subclass check is continued with the usual mechanism. For a demonstration of these concepts, look at this example ABC definition: class Foo: def __getitem__(self, index): ... def __len__(self): ... def get_iterator(self): return iter(self) class MyIterable(ABC): @abstractmethod def __iter__(self): while False: yield None def get_iterator(self): return self.__iter__() @classmethod def __subclasshook__(cls, C): if cls is MyIterable: if any("__iter__" in B.__dict__ for B in C.__mro__): return True return NotImplemented MyIterable.register(Foo) The ABC "MyIterable" defines the standard iterable method, "__iter__()", as an abstract method. The implementation given here can still be called from subclasses. The "get_iterator()" method is also part of the "MyIterable" abstract base class, but it does not have to be overridden in non-abstract derived classes. The "__subclasshook__()" class method defined here says that any class that has an "__iter__()" method in its "__dict__" (or in that of one of its base classes, accessed via the "__mro__" list) is considered a "MyIterable" too. Finally, the last line makes "Foo" a virtual subclass of "MyIterable", even though it does not define an "__iter__()" method (it uses the old-style iterable protocol, defined in terms of "__len__()" and "__getitem__()"). Note that this will not make "get_iterator" available as a method of "Foo", so it is provided separately. The "abc" module also provides the following decorator: @abc.abstractmethod A decorator indicating abstract methods. Using this decorator requires that the class’s metaclass is "ABCMeta" or is derived from it. A class that has a metaclass derived from "ABCMeta" cannot be instantiated unless all of its abstract methods and properties are overridden. The abstract methods can be called using any of the normal ‘super’ call mechanisms. "abstractmethod()" may be used to declare abstract methods for properties and descriptors. Dynamically adding abstract methods to a class, or attempting to modify the abstraction status of a method or class once it is created, are only supported using the "update_abstractmethods()" function. The "abstractmethod()" only affects subclasses derived using regular inheritance; “virtual subclasses” registered with the ABC’s "register()" method are not affected. When "abstractmethod()" is applied in combination with other method descriptors, it should be applied as the innermost decorator, as shown in the following usage examples: class C(ABC): @abstractmethod def my_abstract_method(self, arg1): ... @classmethod @abstractmethod def my_abstract_classmethod(cls, arg2): ... @staticmethod @abstractmethod def my_abstract_staticmethod(arg3): ... @property @abstractmethod def my_abstract_property(self): ... @my_abstract_property.setter @abstractmethod def my_abstract_property(self, val): ... @abstractmethod def _get_x(self): ... @abstractmethod def _set_x(self, val): ... x = property(_get_x, _set_x) In order to correctly interoperate with the abstract base class machinery, the descriptor must identify itself as abstract using "__isabstractmethod__". In general, this attribute should be "True" if any of the methods used to compose the descriptor are abstract. For example, Python’s built-in "property" does the equivalent of: class Descriptor: ... @property def __isabstractmethod__(self): return any(getattr(f, '__isabstractmethod__', False) for f in (self._fget, self._fset, self._fdel)) Note: Unlike Java abstract methods, these abstract methods may have an implementation. This implementation can be called via the "super()" mechanism from the class that overrides it. This could be useful as an end-point for a super-call in a framework that uses cooperative multiple-inheritance. The "abc" module also supports the following legacy decorators: @abc.abstractclassmethod Added in version 3.2. Deprecated since version 3.3: It is now possible to use "classmethod" with "abstractmethod()", making this decorator redundant. A subclass of the built-in "classmethod()", indicating an abstract classmethod. Otherwise it is similar to "abstractmethod()". This special case is deprecated, as the "classmethod()" decorator is now correctly identified as abstract when applied to an abstract method: class C(ABC): @classmethod @abstractmethod def my_abstract_classmethod(cls, arg): ... @abc.abstractstaticmethod Added in version 3.2. Deprecated since version 3.3: It is now possible to use "staticmethod" with "abstractmethod()", making this decorator redundant. A subclass of the built-in "staticmethod()", indicating an abstract staticmethod. Otherwise it is similar to "abstractmethod()". This special case is deprecated, as the "staticmethod()" decorator is now correctly identified as abstract when applied to an abstract method: class C(ABC): @staticmethod @abstractmethod def my_abstract_staticmethod(arg): ... @abc.abstractproperty Deprecated since version 3.3: It is now possible to use "property", "property.getter()", "property.setter()" and "property.deleter()" with "abstractmethod()", making this decorator redundant. A subclass of the built-in "property()", indicating an abstract property. This special case is deprecated, as the "property()" decorator is now correctly identified as abstract when applied to an abstract method: class C(ABC): @property @abstractmethod def my_abstract_property(self): ... The above example defines a read-only property; you can also define a read-write abstract property by appropriately marking one or more of the underlying methods as abstract: class C(ABC): @property def x(self): ... @x.setter @abstractmethod def x(self, val): ... If only some components are abstract, only those components need to be updated to create a concrete property in a subclass: class D(C): @C.x.setter def x(self, val): ... The "abc" module also provides the following functions: abc.get_cache_token() Returns the current abstract base class cache token. The token is an opaque object (that supports equality testing) identifying the current version of the abstract base class cache for virtual subclasses. The token changes with every call to "ABCMeta.register()" on any ABC. Added in version 3.4. abc.update_abstractmethods(cls) A function to recalculate an abstract class’s abstraction status. This function should be called if a class’s abstract methods have been implemented or changed after it was created. Usually, this function should be called from within a class decorator. Returns *cls*, to allow usage as a class decorator. If *cls* is not an instance of "ABCMeta", does nothing. Note: This function assumes that *cls*’s superclasses are already updated. It does not update any subclasses. Added in version 3.10. -[ Footnotes ]- [1] C++ programmers should note that Python’s virtual base class concept is not the same as C++’s. "aifc" — Read and write AIFF and AIFC files ******************************************* Deprecated since version 3.11, removed in version 3.13. This module is no longer part of the Python standard library. It was removed in Python 3.13 after being deprecated in Python 3.11. The removal was decided in **PEP 594**. The last version of Python that provided the "aifc" module was Python 3.12. Generic Operating System Services ********************************* The modules described in this chapter provide interfaces to operating system features that are available on (almost) all operating systems, such as files and a clock. The interfaces are generally modeled after the Unix or C interfaces, but they are available on most other systems as well. Here’s an overview: * "os" — Miscellaneous operating system interfaces * File Names, Command Line Arguments, and Environment Variables * Python UTF-8 Mode * Process Parameters * File Object Creation * File Descriptor Operations * Querying the size of a terminal * Inheritance of File Descriptors * Files and Directories * Timer File Descriptors * Linux extended attributes * Process Management * Interface to the scheduler * Miscellaneous System Information * Random numbers * "io" — Core tools for working with streams * Overview * Text I/O * Binary I/O * Raw I/O * Text Encoding * Opt-in EncodingWarning * High-level Module Interface * Class hierarchy * I/O Base Classes * Raw File I/O * Buffered Streams * Text I/O * Performance * Binary I/O * Text I/O * Multi-threading * Reentrancy * "time" — Time access and conversions * Functions * Clock ID Constants * Timezone Constants * "logging" — Logging facility for Python * Logger Objects * Logging Levels * Handler Objects * Formatter Objects * Filter Objects * LogRecord Objects * LogRecord attributes * LoggerAdapter Objects * Thread Safety * Module-Level Functions * Module-Level Attributes * Integration with the warnings module * "logging.config" — Logging configuration * Configuration functions * Security considerations * Configuration dictionary schema * Dictionary Schema Details * Incremental Configuration * Object connections * User-defined objects * Handler configuration order * Access to external objects * Access to internal objects * Import resolution and custom importers * Configuring QueueHandler and QueueListener * Configuration file format * "logging.handlers" — Logging handlers * StreamHandler * FileHandler * NullHandler * WatchedFileHandler * BaseRotatingHandler * RotatingFileHandler * TimedRotatingFileHandler * SocketHandler * DatagramHandler * SysLogHandler * NTEventLogHandler * SMTPHandler * MemoryHandler * HTTPHandler * QueueHandler * QueueListener * "platform" — Access to underlying platform’s identifying data * Cross platform * Java platform * Windows platform * macOS platform * iOS platform * Unix platforms * Linux platforms * Android platform * Command-line usage * "errno" — Standard errno system symbols * "ctypes" — A foreign function library for Python * ctypes tutorial * Loading dynamic link libraries * Accessing functions from loaded dlls * Calling functions * Fundamental data types * Calling functions, continued * Calling variadic functions * Calling functions with your own custom data types * Specifying the required argument types (function prototypes) * Return types * Passing pointers (or: passing parameters by reference) * Structures and unions * Structure/union alignment and byte order * Bit fields in structures and unions * Arrays * Pointers * Type conversions * Incomplete Types * Callback functions * Accessing values exported from dlls * Surprises * Variable-sized data types * ctypes reference * Finding shared libraries * Loading shared libraries * Foreign functions * Function prototypes * Utility functions * Data types * Fundamental data types * Structured data types * Arrays and pointers Data Compression and Archiving ****************************** The modules described in this chapter support data compression with the zlib, gzip, bzip2 and lzma algorithms, and the creation of ZIP- and tar-format archives. See also Archiving operations provided by the "shutil" module. * "zlib" — Compression compatible with **gzip** * "gzip" — Support for **gzip** files * Examples of usage * Command Line Interface * Command line options * "bz2" — Support for **bzip2** compression * (De)compression of files * Incremental (de)compression * One-shot (de)compression * Examples of usage * "lzma" — Compression using the LZMA algorithm * Reading and writing compressed files * Compressing and decompressing data in memory * Miscellaneous * Specifying custom filter chains * Examples * "zipfile" — Work with ZIP archives * ZipFile Objects * Path Objects * PyZipFile Objects * ZipInfo Objects * Command-Line Interface * Command-line options * Decompression pitfalls * From file itself * File System limitations * Resources limitations * Interruption * Default behaviors of extraction * "tarfile" — Read and write tar archive files * TarFile Objects * TarInfo Objects * Extraction filters * Default named filters * Filter errors * Hints for further verification * Supporting older Python versions * Stateful extraction filter example * Command-Line Interface * Command-line options * Examples * Reading examples * Writing examples * Supported tar formats * Unicode issues "argparse" — Parser for command-line options, arguments and subcommands *********************************************************************** Added in version 3.2. **Source code:** Lib/argparse.py Note: While "argparse" is the default recommended standard library module for implementing basic command line applications, authors with more exacting requirements for exactly how their command line applications behave may find it doesn’t provide the necessary level of control. Refer to Choosing an argument parsing library for alternatives to consider when "argparse" doesn’t support behaviors that the application requires (such as entirely disabling support for interspersed options and positional arguments, or accepting option parameter values that start with "-" even when they correspond to another defined option). ====================================================================== Tutorial ^^^^^^^^ This page contains the API reference information. For a more gentle introduction to Python command-line parsing, have a look at the argparse tutorial. The "argparse" module makes it easy to write user-friendly command- line interfaces. The program defines what arguments it requires, and "argparse" will figure out how to parse those out of "sys.argv". The "argparse" module also automatically generates help and usage messages. The module will also issue errors when users give the program invalid arguments. The "argparse" module’s support for command-line interfaces is built around an instance of "argparse.ArgumentParser". It is a container for argument specifications and has options that apply to the parser as whole: parser = argparse.ArgumentParser( prog='ProgramName', description='What the program does', epilog='Text at the bottom of help') The "ArgumentParser.add_argument()" method attaches individual argument specifications to the parser. It supports positional arguments, options that accept values, and on/off flags: parser.add_argument('filename') # positional argument parser.add_argument('-c', '--count') # option that takes a value parser.add_argument('-v', '--verbose', action='store_true') # on/off flag The "ArgumentParser.parse_args()" method runs the parser and places the extracted data in a "argparse.Namespace" object: args = parser.parse_args() print(args.filename, args.count, args.verbose) Note: If you’re looking for a guide about how to upgrade "optparse" code to "argparse", see Upgrading Optparse Code. ArgumentParser objects ====================== class argparse.ArgumentParser(prog=None, usage=None, description=None, epilog=None, parents=[], formatter_class=argparse.HelpFormatter, prefix_chars='-', fromfile_prefix_chars=None, argument_default=None, conflict_handler='error', add_help=True, allow_abbrev=True, exit_on_error=True) Create a new "ArgumentParser" object. All parameters should be passed as keyword arguments. Each parameter has its own more detailed description below, but in short they are: * prog - The name of the program (default: "os.path.basename(sys.argv[0])") * usage - The string describing the program usage (default: generated from arguments added to parser) * description - Text to display before the argument help (by default, no text) * epilog - Text to display after the argument help (by default, no text) * parents - A list of "ArgumentParser" objects whose arguments should also be included * formatter_class - A class for customizing the help output * prefix_chars - The set of characters that prefix optional arguments (default: ‘-‘) * fromfile_prefix_chars - The set of characters that prefix files from which additional arguments should be read (default: "None") * argument_default - The global default value for arguments (default: "None") * conflict_handler - The strategy for resolving conflicting optionals (usually unnecessary) * add_help - Add a "-h/--help" option to the parser (default: "True") * allow_abbrev - Allows long options to be abbreviated if the abbreviation is unambiguous. (default: "True") * exit_on_error - Determines whether or not "ArgumentParser" exits with error info when an error occurs. (default: "True") Changed in version 3.5: *allow_abbrev* parameter was added. Changed in version 3.8: In previous versions, *allow_abbrev* also disabled grouping of short flags such as "-vv" to mean "-v -v". Changed in version 3.9: *exit_on_error* parameter was added. The following sections describe how each of these are used. prog ---- By default, "ArgumentParser" calculates the name of the program to display in help messages depending on the way the Python interpreter was run: * The "base name" of "sys.argv[0]" if a file was passed as argument. * The Python interpreter name followed by "sys.argv[0]" if a directory or a zipfile was passed as argument. * The Python interpreter name followed by "-m" followed by the module or package name if the "-m" option was used. This default is almost always desirable because it will make the help messages match the string that was used to invoke the program on the command line. However, to change this default behavior, another value can be supplied using the "prog=" argument to "ArgumentParser": >>> parser = argparse.ArgumentParser(prog='myprogram') >>> parser.print_help() usage: myprogram [-h] options: -h, --help show this help message and exit Note that the program name, whether determined from "sys.argv[0]" or from the "prog=" argument, is available to help messages using the "%(prog)s" format specifier. >>> parser = argparse.ArgumentParser(prog='myprogram') >>> parser.add_argument('--foo', help='foo of the %(prog)s program') >>> parser.print_help() usage: myprogram [-h] [--foo FOO] options: -h, --help show this help message and exit --foo FOO foo of the myprogram program usage ----- By default, "ArgumentParser" calculates the usage message from the arguments it contains. The default message can be overridden with the "usage=" keyword argument: >>> parser = argparse.ArgumentParser(prog='PROG', usage='%(prog)s [options]') >>> parser.add_argument('--foo', nargs='?', help='foo help') >>> parser.add_argument('bar', nargs='+', help='bar help') >>> parser.print_help() usage: PROG [options] positional arguments: bar bar help options: -h, --help show this help message and exit --foo [FOO] foo help The "%(prog)s" format specifier is available to fill in the program name in your usage messages. description ----------- Most calls to the "ArgumentParser" constructor will use the "description=" keyword argument. This argument gives a brief description of what the program does and how it works. In help messages, the description is displayed between the command-line usage string and the help messages for the various arguments. By default, the description will be line-wrapped so that it fits within the given space. To change this behavior, see the formatter_class argument. epilog ------ Some programs like to display additional description of the program after the description of the arguments. Such text can be specified using the "epilog=" argument to "ArgumentParser": >>> parser = argparse.ArgumentParser( ... description='A foo that bars', ... epilog="And that's how you'd foo a bar") >>> parser.print_help() usage: argparse.py [-h] A foo that bars options: -h, --help show this help message and exit And that's how you'd foo a bar As with the description argument, the "epilog=" text is by default line-wrapped, but this behavior can be adjusted with the formatter_class argument to "ArgumentParser". parents ------- Sometimes, several parsers share a common set of arguments. Rather than repeating the definitions of these arguments, a single parser with all the shared arguments and passed to "parents=" argument to "ArgumentParser" can be used. The "parents=" argument takes a list of "ArgumentParser" objects, collects all the positional and optional actions from them, and adds these actions to the "ArgumentParser" object being constructed: >>> parent_parser = argparse.ArgumentParser(add_help=False) >>> parent_parser.add_argument('--parent', type=int) >>> foo_parser = argparse.ArgumentParser(parents=[parent_parser]) >>> foo_parser.add_argument('foo') >>> foo_parser.parse_args(['--parent', '2', 'XXX']) Namespace(foo='XXX', parent=2) >>> bar_parser = argparse.ArgumentParser(parents=[parent_parser]) >>> bar_parser.add_argument('--bar') >>> bar_parser.parse_args(['--bar', 'YYY']) Namespace(bar='YYY', parent=None) Note that most parent parsers will specify "add_help=False". Otherwise, the "ArgumentParser" will see two "-h/--help" options (one in the parent and one in the child) and raise an error. Note: You must fully initialize the parsers before passing them via "parents=". If you change the parent parsers after the child parser, those changes will not be reflected in the child. formatter_class --------------- "ArgumentParser" objects allow the help formatting to be customized by specifying an alternate formatting class. Currently, there are four such classes: class argparse.RawDescriptionHelpFormatter class argparse.RawTextHelpFormatter class argparse.ArgumentDefaultsHelpFormatter class argparse.MetavarTypeHelpFormatter "RawDescriptionHelpFormatter" and "RawTextHelpFormatter" give more control over how textual descriptions are displayed. By default, "ArgumentParser" objects line-wrap the description and epilog texts in command-line help messages: >>> parser = argparse.ArgumentParser( ... prog='PROG', ... description='''this description ... was indented weird ... but that is okay''', ... epilog=''' ... likewise for this epilog whose whitespace will ... be cleaned up and whose words will be wrapped ... across a couple lines''') >>> parser.print_help() usage: PROG [-h] this description was indented weird but that is okay options: -h, --help show this help message and exit likewise for this epilog whose whitespace will be cleaned up and whose words will be wrapped across a couple lines Passing "RawDescriptionHelpFormatter" as "formatter_class=" indicates that description and epilog are already correctly formatted and should not be line-wrapped: >>> parser = argparse.ArgumentParser( ... prog='PROG', ... formatter_class=argparse.RawDescriptionHelpFormatter, ... description=textwrap.dedent('''\ ... Please do not mess up this text! ... -------------------------------- ... I have indented it ... exactly the way ... I want it ... ''')) >>> parser.print_help() usage: PROG [-h] Please do not mess up this text! -------------------------------- I have indented it exactly the way I want it options: -h, --help show this help message and exit "RawTextHelpFormatter" maintains whitespace for all sorts of help text, including argument descriptions. However, multiple newlines are replaced with one. If you wish to preserve multiple blank lines, add spaces between the newlines. "ArgumentDefaultsHelpFormatter" automatically adds information about default values to each of the argument help messages: >>> parser = argparse.ArgumentParser( ... prog='PROG', ... formatter_class=argparse.ArgumentDefaultsHelpFormatter) >>> parser.add_argument('--foo', type=int, default=42, help='FOO!') >>> parser.add_argument('bar', nargs='*', default=[1, 2, 3], help='BAR!') >>> parser.print_help() usage: PROG [-h] [--foo FOO] [bar ...] positional arguments: bar BAR! (default: [1, 2, 3]) options: -h, --help show this help message and exit --foo FOO FOO! (default: 42) "MetavarTypeHelpFormatter" uses the name of the type argument for each argument as the display name for its values (rather than using the dest as the regular formatter does): >>> parser = argparse.ArgumentParser( ... prog='PROG', ... formatter_class=argparse.MetavarTypeHelpFormatter) >>> parser.add_argument('--foo', type=int) >>> parser.add_argument('bar', type=float) >>> parser.print_help() usage: PROG [-h] [--foo int] float positional arguments: float options: -h, --help show this help message and exit --foo int prefix_chars ------------ Most command-line options will use "-" as the prefix, e.g. "-f/--foo". Parsers that need to support different or additional prefix characters, e.g. for options like "+f" or "/foo", may specify them using the "prefix_chars=" argument to the "ArgumentParser" constructor: >>> parser = argparse.ArgumentParser(prog='PROG', prefix_chars='-+') >>> parser.add_argument('+f') >>> parser.add_argument('++bar') >>> parser.parse_args('+f X ++bar Y'.split()) Namespace(bar='Y', f='X') The "prefix_chars=" argument defaults to "'-'". Supplying a set of characters that does not include "-" will cause "-f/--foo" options to be disallowed. fromfile_prefix_chars --------------------- Sometimes, when dealing with a particularly long argument list, it may make sense to keep the list of arguments in a file rather than typing it out at the command line. If the "fromfile_prefix_chars=" argument is given to the "ArgumentParser" constructor, then arguments that start with any of the specified characters will be treated as files, and will be replaced by the arguments they contain. For example: >>> with open('args.txt', 'w', encoding=sys.getfilesystemencoding()) as fp: ... fp.write('-f\nbar') ... >>> parser = argparse.ArgumentParser(fromfile_prefix_chars='@') >>> parser.add_argument('-f') >>> parser.parse_args(['-f', 'foo', '@args.txt']) Namespace(f='bar') Arguments read from a file must be one per line by default (but see also "convert_arg_line_to_args()") and are treated as if they were in the same place as the original file referencing argument on the command line. So in the example above, the expression "['-f', 'foo', '@args.txt']" is considered equivalent to the expression "['-f', 'foo', '-f', 'bar']". Note: Empty lines are treated as empty strings ("''"), which are allowed as values but not as arguments. Empty lines that are read as arguments will result in an “unrecognized arguments” error. "ArgumentParser" uses *filesystem encoding and error handler* to read the file containing arguments. The "fromfile_prefix_chars=" argument defaults to "None", meaning that arguments will never be treated as file references. Changed in version 3.12: "ArgumentParser" changed encoding and errors to read arguments files from default (e.g. "locale.getpreferredencoding(False)" and ""strict"") to the *filesystem encoding and error handler*. Arguments file should be encoded in UTF-8 instead of ANSI Codepage on Windows. argument_default ---------------- Generally, argument defaults are specified either by passing a default to "add_argument()" or by calling the "set_defaults()" methods with a specific set of name-value pairs. Sometimes however, it may be useful to specify a single parser-wide default for arguments. This can be accomplished by passing the "argument_default=" keyword argument to "ArgumentParser". For example, to globally suppress attribute creation on "parse_args()" calls, we supply "argument_default=SUPPRESS": >>> parser = argparse.ArgumentParser(argument_default=argparse.SUPPRESS) >>> parser.add_argument('--foo') >>> parser.add_argument('bar', nargs='?') >>> parser.parse_args(['--foo', '1', 'BAR']) Namespace(bar='BAR', foo='1') >>> parser.parse_args([]) Namespace() allow_abbrev ------------ Normally, when you pass an argument list to the "parse_args()" method of an "ArgumentParser", it recognizes abbreviations of long options. This feature can be disabled by setting "allow_abbrev" to "False": >>> parser = argparse.ArgumentParser(prog='PROG', allow_abbrev=False) >>> parser.add_argument('--foobar', action='store_true') >>> parser.add_argument('--foonley', action='store_false') >>> parser.parse_args(['--foon']) usage: PROG [-h] [--foobar] [--foonley] PROG: error: unrecognized arguments: --foon Added in version 3.5. conflict_handler ---------------- "ArgumentParser" objects do not allow two actions with the same option string. By default, "ArgumentParser" objects raise an exception if an attempt is made to create an argument with an option string that is already in use: >>> parser = argparse.ArgumentParser(prog='PROG') >>> parser.add_argument('-f', '--foo', help='old foo help') >>> parser.add_argument('--foo', help='new foo help') Traceback (most recent call last): .. ArgumentError: argument --foo: conflicting option string(s): --foo Sometimes (e.g. when using parents) it may be useful to simply override any older arguments with the same option string. To get this behavior, the value "'resolve'" can be supplied to the "conflict_handler=" argument of "ArgumentParser": >>> parser = argparse.ArgumentParser(prog='PROG', conflict_handler='resolve') >>> parser.add_argument('-f', '--foo', help='old foo help') >>> parser.add_argument('--foo', help='new foo help') >>> parser.print_help() usage: PROG [-h] [-f FOO] [--foo FOO] options: -h, --help show this help message and exit -f FOO old foo help --foo FOO new foo help Note that "ArgumentParser" objects only remove an action if all of its option strings are overridden. So, in the example above, the old "-f/--foo" action is retained as the "-f" action, because only the "-- foo" option string was overridden. add_help -------- By default, "ArgumentParser" objects add an option which simply displays the parser’s help message. If "-h" or "--help" is supplied at the command line, the "ArgumentParser" help will be printed. Occasionally, it may be useful to disable the addition of this help option. This can be achieved by passing "False" as the "add_help=" argument to "ArgumentParser": >>> parser = argparse.ArgumentParser(prog='PROG', add_help=False) >>> parser.add_argument('--foo', help='foo help') >>> parser.print_help() usage: PROG [--foo FOO] options: --foo FOO foo help The help option is typically "-h/--help". The exception to this is if the "prefix_chars=" is specified and does not include "-", in which case "-h" and "--help" are not valid options. In this case, the first character in "prefix_chars" is used to prefix the help options: >>> parser = argparse.ArgumentParser(prog='PROG', prefix_chars='+/') >>> parser.print_help() usage: PROG [+h] options: +h, ++help show this help message and exit exit_on_error ------------- Normally, when you pass an invalid argument list to the "parse_args()" method of an "ArgumentParser", it will print a *message* to "sys.stderr" and exit with a status code of 2. If the user would like to catch errors manually, the feature can be enabled by setting "exit_on_error" to "False": >>> parser = argparse.ArgumentParser(exit_on_error=False) >>> parser.add_argument('--integers', type=int) _StoreAction(option_strings=['--integers'], dest='integers', nargs=None, const=None, default=None, type=, choices=None, help=None, metavar=None) >>> try: ... parser.parse_args('--integers a'.split()) ... except argparse.ArgumentError: ... print('Catching an argumentError') ... Catching an argumentError Added in version 3.9. The add_argument() method ========================= ArgumentParser.add_argument(name or flags..., *[, action][, nargs][, const][, default][, type][, choices][, required][, help][, metavar][, dest][, deprecated]) Define how a single command-line argument should be parsed. Each parameter has its own more detailed description below, but in short they are: * name or flags - Either a name or a list of option strings, e.g. "'foo'" or "'-f', '--foo'". * action - The basic type of action to be taken when this argument is encountered at the command line. * nargs - The number of command-line arguments that should be consumed. * const - A constant value required by some action and nargs selections. * default - The value produced if the argument is absent from the command line and if it is absent from the namespace object. * type - The type to which the command-line argument should be converted. * choices - A sequence of the allowable values for the argument. * required - Whether or not the command-line option may be omitted (optionals only). * help - A brief description of what the argument does. * metavar - A name for the argument in usage messages. * dest - The name of the attribute to be added to the object returned by "parse_args()". * deprecated - Whether or not use of the argument is deprecated. The following sections describe how each of these are used. name or flags ------------- The "add_argument()" method must know whether an optional argument, like "-f" or "--foo", or a positional argument, like a list of filenames, is expected. The first arguments passed to "add_argument()" must therefore be either a series of flags, or a simple argument name. For example, an optional argument could be created like: >>> parser.add_argument('-f', '--foo') while a positional argument could be created like: >>> parser.add_argument('bar') When "parse_args()" is called, optional arguments will be identified by the "-" prefix, and the remaining arguments will be assumed to be positional: >>> parser = argparse.ArgumentParser(prog='PROG') >>> parser.add_argument('-f', '--foo') >>> parser.add_argument('bar') >>> parser.parse_args(['BAR']) Namespace(bar='BAR', foo=None) >>> parser.parse_args(['BAR', '--foo', 'FOO']) Namespace(bar='BAR', foo='FOO') >>> parser.parse_args(['--foo', 'FOO']) usage: PROG [-h] [-f FOO] bar PROG: error: the following arguments are required: bar action ------ "ArgumentParser" objects associate command-line arguments with actions. These actions can do just about anything with the command- line arguments associated with them, though most actions simply add an attribute to the object returned by "parse_args()". The "action" keyword argument specifies how the command-line arguments should be handled. The supplied actions are: * "'store'" - This just stores the argument’s value. This is the default action. * "'store_const'" - This stores the value specified by the const keyword argument; note that the const keyword argument defaults to "None". The "'store_const'" action is most commonly used with optional arguments that specify some sort of flag. For example: >>> parser = argparse.ArgumentParser() >>> parser.add_argument('--foo', action='store_const', const=42) >>> parser.parse_args(['--foo']) Namespace(foo=42) * "'store_true'" and "'store_false'" - These are special cases of "'store_const'" used for storing the values "True" and "False" respectively. In addition, they create default values of "False" and "True" respectively: >>> parser = argparse.ArgumentParser() >>> parser.add_argument('--foo', action='store_true') >>> parser.add_argument('--bar', action='store_false') >>> parser.add_argument('--baz', action='store_false') >>> parser.parse_args('--foo --bar'.split()) Namespace(foo=True, bar=False, baz=True) * "'append'" - This stores a list, and appends each argument value to the list. It is useful to allow an option to be specified multiple times. If the default value is non-empty, the default elements will be present in the parsed value for the option, with any values from the command line appended after those default values. Example usage: >>> parser = argparse.ArgumentParser() >>> parser.add_argument('--foo', action='append') >>> parser.parse_args('--foo 1 --foo 2'.split()) Namespace(foo=['1', '2']) * "'append_const'" - This stores a list, and appends the value specified by the const keyword argument to the list; note that the const keyword argument defaults to "None". The "'append_const'" action is typically useful when multiple arguments need to store constants to the same list. For example: >>> parser = argparse.ArgumentParser() >>> parser.add_argument('--str', dest='types', action='append_const', const=str) >>> parser.add_argument('--int', dest='types', action='append_const', const=int) >>> parser.parse_args('--str --int'.split()) Namespace(types=[, ]) * "'extend'" - This stores a list and appends each item from the multi-value argument list to it. The "'extend'" action is typically used with the nargs keyword argument value "'+'" or "'*'". Note that when nargs is "None" (the default) or "'?'", each character of the argument string will be appended to the list. Example usage: >>> parser = argparse.ArgumentParser() >>> parser.add_argument("--foo", action="extend", nargs="+", type=str) >>> parser.parse_args(["--foo", "f1", "--foo", "f2", "f3", "f4"]) Namespace(foo=['f1', 'f2', 'f3', 'f4']) Added in version 3.8. * "'count'" - This counts the number of times a keyword argument occurs. For example, this is useful for increasing verbosity levels: >>> parser = argparse.ArgumentParser() >>> parser.add_argument('--verbose', '-v', action='count', default=0) >>> parser.parse_args(['-vvv']) Namespace(verbose=3) Note, the *default* will be "None" unless explicitly set to *0*. * "'help'" - This prints a complete help message for all the options in the current parser and then exits. By default a help action is automatically added to the parser. See "ArgumentParser" for details of how the output is created. * "'version'" - This expects a "version=" keyword argument in the "add_argument()" call, and prints version information and exits when invoked: >>> import argparse >>> parser = argparse.ArgumentParser(prog='PROG') >>> parser.add_argument('--version', action='version', version='%(prog)s 2.0') >>> parser.parse_args(['--version']) PROG 2.0 You may also specify an arbitrary action by passing an "Action" subclass (e.g. "BooleanOptionalAction") or other object that implements the same interface. Only actions that consume command-line arguments (e.g. "'store'", "'append'", "'extend'", or custom actions with non-zero "nargs") can be used with positional arguments. The recommended way to create a custom action is to extend "Action", overriding the "__call__()" method and optionally the "__init__()" and "format_usage()" methods. You can also register custom actions using the "register()" method and reference them by their registered name. An example of a custom action: >>> class FooAction(argparse.Action): ... def __init__(self, option_strings, dest, nargs=None, **kwargs): ... if nargs is not None: ... raise ValueError("nargs not allowed") ... super().__init__(option_strings, dest, **kwargs) ... def __call__(self, parser, namespace, values, option_string=None): ... print('%r %r %r' % (namespace, values, option_string)) ... setattr(namespace, self.dest, values) ... >>> parser = argparse.ArgumentParser() >>> parser.add_argument('--foo', action=FooAction) >>> parser.add_argument('bar', action=FooAction) >>> args = parser.parse_args('1 --foo 2'.split()) Namespace(bar=None, foo=None) '1' None Namespace(bar='1', foo=None) '2' '--foo' >>> args Namespace(bar='1', foo='2') For more details, see "Action". nargs ----- "ArgumentParser" objects usually associate a single command-line argument with a single action to be taken. The "nargs" keyword argument associates a different number of command-line arguments with a single action. See also Specifying ambiguous arguments. The supported values are: * "N" (an integer). "N" arguments from the command line will be gathered together into a list. For example: >>> parser = argparse.ArgumentParser() >>> parser.add_argument('--foo', nargs=2) >>> parser.add_argument('bar', nargs=1) >>> parser.parse_args('c --foo a b'.split()) Namespace(bar=['c'], foo=['a', 'b']) Note that "nargs=1" produces a list of one item. This is different from the default, in which the item is produced by itself. * "'?'". One argument will be consumed from the command line if possible, and produced as a single item. If no command-line argument is present, the value from default will be produced. Note that for optional arguments, there is an additional case - the option string is present but not followed by a command-line argument. In this case the value from const will be produced. Some examples to illustrate this: >>> parser = argparse.ArgumentParser() >>> parser.add_argument('--foo', nargs='?', const='c', default='d') >>> parser.add_argument('bar', nargs='?', default='d') >>> parser.parse_args(['XX', '--foo', 'YY']) Namespace(bar='XX', foo='YY') >>> parser.parse_args(['XX', '--foo']) Namespace(bar='XX', foo='c') >>> parser.parse_args([]) Namespace(bar='d', foo='d') One of the more common uses of "nargs='?'" is to allow optional input and output files: >>> parser = argparse.ArgumentParser() >>> parser.add_argument('infile', nargs='?', type=argparse.FileType('r'), ... default=sys.stdin) >>> parser.add_argument('outfile', nargs='?', type=argparse.FileType('w'), ... default=sys.stdout) >>> parser.parse_args(['input.txt', 'output.txt']) Namespace(infile=<_io.TextIOWrapper name='input.txt' encoding='UTF-8'>, outfile=<_io.TextIOWrapper name='output.txt' encoding='UTF-8'>) >>> parser.parse_args([]) Namespace(infile=<_io.TextIOWrapper name='' encoding='UTF-8'>, outfile=<_io.TextIOWrapper name='' encoding='UTF-8'>) * "'*'". All command-line arguments present are gathered into a list. Note that it generally doesn’t make much sense to have more than one positional argument with "nargs='*'", but multiple optional arguments with "nargs='*'" is possible. For example: >>> parser = argparse.ArgumentParser() >>> parser.add_argument('--foo', nargs='*') >>> parser.add_argument('--bar', nargs='*') >>> parser.add_argument('baz', nargs='*') >>> parser.parse_args('a b --foo x y --bar 1 2'.split()) Namespace(bar=['1', '2'], baz=['a', 'b'], foo=['x', 'y']) * "'+'". Just like "'*'", all command-line arguments present are gathered into a list. Additionally, an error message will be generated if there wasn’t at least one command-line argument present. For example: >>> parser = argparse.ArgumentParser(prog='PROG') >>> parser.add_argument('foo', nargs='+') >>> parser.parse_args(['a', 'b']) Namespace(foo=['a', 'b']) >>> parser.parse_args([]) usage: PROG [-h] foo [foo ...] PROG: error: the following arguments are required: foo If the "nargs" keyword argument is not provided, the number of arguments consumed is determined by the action. Generally this means a single command-line argument will be consumed and a single item (not a list) will be produced. Actions that do not consume command-line arguments (e.g. "'store_const'") set "nargs=0". const ----- The "const" argument of "add_argument()" is used to hold constant values that are not read from the command line but are required for the various "ArgumentParser" actions. The two most common uses of it are: * When "add_argument()" is called with "action='store_const'" or "action='append_const'". These actions add the "const" value to one of the attributes of the object returned by "parse_args()". See the action description for examples. If "const" is not provided to "add_argument()", it will receive a default value of "None". * When "add_argument()" is called with option strings (like "-f" or " --foo") and "nargs='?'". This creates an optional argument that can be followed by zero or one command-line arguments. When parsing the command line, if the option string is encountered with no command- line argument following it, the value of "const" will be assumed to be "None" instead. See the nargs description for examples. Changed in version 3.11: "const=None" by default, including when "action='append_const'" or "action='store_const'". default ------- All optional arguments and some positional arguments may be omitted at the command line. The "default" keyword argument of "add_argument()", whose value defaults to "None", specifies what value should be used if the command-line argument is not present. For optional arguments, the "default" value is used when the option string was not present at the command line: >>> parser = argparse.ArgumentParser() >>> parser.add_argument('--foo', default=42) >>> parser.parse_args(['--foo', '2']) Namespace(foo='2') >>> parser.parse_args([]) Namespace(foo=42) If the target namespace already has an attribute set, the action *default* will not overwrite it: >>> parser = argparse.ArgumentParser() >>> parser.add_argument('--foo', default=42) >>> parser.parse_args([], namespace=argparse.Namespace(foo=101)) Namespace(foo=101) If the "default" value is a string, the parser parses the value as if it were a command-line argument. In particular, the parser applies any type conversion argument, if provided, before setting the attribute on the "Namespace" return value. Otherwise, the parser uses the value as is: >>> parser = argparse.ArgumentParser() >>> parser.add_argument('--length', default='10', type=int) >>> parser.add_argument('--width', default=10.5, type=int) >>> parser.parse_args() Namespace(length=10, width=10.5) For positional arguments with nargs equal to "?" or "*", the "default" value is used when no command-line argument was present: >>> parser = argparse.ArgumentParser() >>> parser.add_argument('foo', nargs='?', default=42) >>> parser.parse_args(['a']) Namespace(foo='a') >>> parser.parse_args([]) Namespace(foo=42) For required arguments, the "default" value is ignored. For example, this applies to positional arguments with nargs values other than "?" or "*", or optional arguments marked as "required=True". Providing "default=argparse.SUPPRESS" causes no attribute to be added if the command-line argument was not present: >>> parser = argparse.ArgumentParser() >>> parser.add_argument('--foo', default=argparse.SUPPRESS) >>> parser.parse_args([]) Namespace() >>> parser.parse_args(['--foo', '1']) Namespace(foo='1') type ---- By default, the parser reads command-line arguments in as simple strings. However, quite often the command-line string should instead be interpreted as another type, such as a "float" or "int". The "type" keyword for "add_argument()" allows any necessary type-checking and type conversions to be performed. If the type keyword is used with the default keyword, the type converter is only applied if the default is a string. The argument to "type" can be a callable that accepts a single string or the name of a registered type (see "register()") If the function raises "ArgumentTypeError", "TypeError", or "ValueError", the exception is caught and a nicely formatted error message is displayed. Other exception types are not handled. Common built-in types and functions can be used as type converters: import argparse import pathlib parser = argparse.ArgumentParser() parser.add_argument('count', type=int) parser.add_argument('distance', type=float) parser.add_argument('street', type=ascii) parser.add_argument('code_point', type=ord) parser.add_argument('dest_file', type=argparse.FileType('w', encoding='latin-1')) parser.add_argument('datapath', type=pathlib.Path) User defined functions can be used as well: >>> def hyphenated(string): ... return '-'.join([word[:4] for word in string.casefold().split()]) ... >>> parser = argparse.ArgumentParser() >>> _ = parser.add_argument('short_title', type=hyphenated) >>> parser.parse_args(['"The Tale of Two Cities"']) Namespace(short_title='"the-tale-of-two-citi') The "bool()" function is not recommended as a type converter. All it does is convert empty strings to "False" and non-empty strings to "True". This is usually not what is desired. In general, the "type" keyword is a convenience that should only be used for simple conversions that can only raise one of the three supported exceptions. Anything with more interesting error-handling or resource management should be done downstream after the arguments are parsed. For example, JSON or YAML conversions have complex error cases that require better reporting than can be given by the "type" keyword. A "JSONDecodeError" would not be well formatted and a "FileNotFoundError" exception would not be handled at all. Even "FileType" has its limitations for use with the "type" keyword. If one argument uses "FileType" and then a subsequent argument fails, an error is reported but the file is not automatically closed. In this case, it would be better to wait until after the parser has run and then use the "with"-statement to manage the files. For type checkers that simply check against a fixed set of values, consider using the choices keyword instead. choices ------- Some command-line arguments should be selected from a restricted set of values. These can be handled by passing a sequence object as the *choices* keyword argument to "add_argument()". When the command line is parsed, argument values will be checked, and an error message will be displayed if the argument was not one of the acceptable values: >>> parser = argparse.ArgumentParser(prog='game.py') >>> parser.add_argument('move', choices=['rock', 'paper', 'scissors']) >>> parser.parse_args(['rock']) Namespace(move='rock') >>> parser.parse_args(['fire']) usage: game.py [-h] {rock,paper,scissors} game.py: error: argument move: invalid choice: 'fire' (choose from 'rock', 'paper', 'scissors') Note that inclusion in the *choices* sequence is checked after any type conversions have been performed, so the type of the objects in the *choices* sequence should match the type specified. Any sequence can be passed as the *choices* value, so "list" objects, "tuple" objects, and custom sequences are all supported. Use of "enum.Enum" is not recommended because it is difficult to control its appearance in usage, help, and error messages. Formatted choices override the default *metavar* which is normally derived from *dest*. This is usually what you want because the user never sees the *dest* parameter. If this display isn’t desirable (perhaps because there are many choices), just specify an explicit metavar. required -------- In general, the "argparse" module assumes that flags like "-f" and "-- bar" indicate *optional* arguments, which can always be omitted at the command line. To make an option *required*, "True" can be specified for the "required=" keyword argument to "add_argument()": >>> parser = argparse.ArgumentParser() >>> parser.add_argument('--foo', required=True) >>> parser.parse_args(['--foo', 'BAR']) Namespace(foo='BAR') >>> parser.parse_args([]) usage: [-h] --foo FOO : error: the following arguments are required: --foo As the example shows, if an option is marked as "required", "parse_args()" will report an error if that option is not present at the command line. Note: Required options are generally considered bad form because users expect *options* to be *optional*, and thus they should be avoided when possible. help ---- The "help" value is a string containing a brief description of the argument. When a user requests help (usually by using "-h" or "--help" at the command line), these "help" descriptions will be displayed with each argument. The "help" strings can include various format specifiers to avoid repetition of things like the program name or the argument default. The available specifiers include the program name, "%(prog)s" and most keyword arguments to "add_argument()", e.g. "%(default)s", "%(type)s", etc.: >>> parser = argparse.ArgumentParser(prog='frobble') >>> parser.add_argument('bar', nargs='?', type=int, default=42, ... help='the bar to %(prog)s (default: %(default)s)') >>> parser.print_help() usage: frobble [-h] [bar] positional arguments: bar the bar to frobble (default: 42) options: -h, --help show this help message and exit As the help string supports %-formatting, if you want a literal "%" to appear in the help string, you must escape it as "%%". "argparse" supports silencing the help entry for certain options, by setting the "help" value to "argparse.SUPPRESS": >>> parser = argparse.ArgumentParser(prog='frobble') >>> parser.add_argument('--foo', help=argparse.SUPPRESS) >>> parser.print_help() usage: frobble [-h] options: -h, --help show this help message and exit metavar ------- When "ArgumentParser" generates help messages, it needs some way to refer to each expected argument. By default, "ArgumentParser" objects use the dest value as the “name” of each object. By default, for positional argument actions, the dest value is used directly, and for optional argument actions, the dest value is uppercased. So, a single positional argument with "dest='bar'" will be referred to as "bar". A single optional argument "--foo" that should be followed by a single command-line argument will be referred to as "FOO". An example: >>> parser = argparse.ArgumentParser() >>> parser.add_argument('--foo') >>> parser.add_argument('bar') >>> parser.parse_args('X --foo Y'.split()) Namespace(bar='X', foo='Y') >>> parser.print_help() usage: [-h] [--foo FOO] bar positional arguments: bar options: -h, --help show this help message and exit --foo FOO An alternative name can be specified with "metavar": >>> parser = argparse.ArgumentParser() >>> parser.add_argument('--foo', metavar='YYY') >>> parser.add_argument('bar', metavar='XXX') >>> parser.parse_args('X --foo Y'.split()) Namespace(bar='X', foo='Y') >>> parser.print_help() usage: [-h] [--foo YYY] XXX positional arguments: XXX options: -h, --help show this help message and exit --foo YYY Note that "metavar" only changes the *displayed* name - the name of the attribute on the "parse_args()" object is still determined by the dest value. Different values of "nargs" may cause the metavar to be used multiple times. Providing a tuple to "metavar" specifies a different display for each of the arguments: >>> parser = argparse.ArgumentParser(prog='PROG') >>> parser.add_argument('-x', nargs=2) >>> parser.add_argument('--foo', nargs=2, metavar=('bar', 'baz')) >>> parser.print_help() usage: PROG [-h] [-x X X] [--foo bar baz] options: -h, --help show this help message and exit -x X X --foo bar baz dest ---- Most "ArgumentParser" actions add some value as an attribute of the object returned by "parse_args()". The name of this attribute is determined by the "dest" keyword argument of "add_argument()". For positional argument actions, "dest" is normally supplied as the first argument to "add_argument()": >>> parser = argparse.ArgumentParser() >>> parser.add_argument('bar') >>> parser.parse_args(['XXX']) Namespace(bar='XXX') For optional argument actions, the value of "dest" is normally inferred from the option strings. "ArgumentParser" generates the value of "dest" by taking the first long option string and stripping away the initial "--" string. If no long option strings were supplied, "dest" will be derived from the first short option string by stripping the initial "-" character. Any internal "-" characters will be converted to "_" characters to make sure the string is a valid attribute name. The examples below illustrate this behavior: >>> parser = argparse.ArgumentParser() >>> parser.add_argument('-f', '--foo-bar', '--foo') >>> parser.add_argument('-x', '-y') >>> parser.parse_args('-f 1 -x 2'.split()) Namespace(foo_bar='1', x='2') >>> parser.parse_args('--foo 1 -y 2'.split()) Namespace(foo_bar='1', x='2') "dest" allows a custom attribute name to be provided: >>> parser = argparse.ArgumentParser() >>> parser.add_argument('--foo', dest='bar') >>> parser.parse_args('--foo XXX'.split()) Namespace(bar='XXX') deprecated ---------- During a project’s lifetime, some arguments may need to be removed from the command line. Before removing them, you should inform your users that the arguments are deprecated and will be removed. The "deprecated" keyword argument of "add_argument()", which defaults to "False", specifies if the argument is deprecated and will be removed in the future. For arguments, if "deprecated" is "True", then a warning will be printed to "sys.stderr" when the argument is used: >>> import argparse >>> parser = argparse.ArgumentParser(prog='snake.py') >>> parser.add_argument('--legs', default=0, type=int, deprecated=True) >>> parser.parse_args([]) Namespace(legs=0) >>> parser.parse_args(['--legs', '4']) snake.py: warning: option '--legs' is deprecated Namespace(legs=4) Added in version 3.13. Action classes -------------- "Action" classes implement the Action API, a callable which returns a callable which processes arguments from the command-line. Any object which follows this API may be passed as the "action" parameter to "add_argument()". class argparse.Action(option_strings, dest, nargs=None, const=None, default=None, type=None, choices=None, required=False, help=None, metavar=None) "Action" objects are used by an "ArgumentParser" to represent the information needed to parse a single argument from one or more strings from the command line. The "Action" class must accept the two positional arguments plus any keyword arguments passed to "ArgumentParser.add_argument()" except for the "action" itself. Instances of "Action" (or return value of any callable to the "action" parameter) should have attributes "dest", "option_strings", "default", "type", "required", "help", etc. defined. The easiest way to ensure these attributes are defined is to call "Action.__init__()". __call__(parser, namespace, values, option_string=None) "Action" instances should be callable, so subclasses must override the "__call__()" method, which should accept four parameters: * *parser* - The "ArgumentParser" object which contains this action. * *namespace* - The "Namespace" object that will be returned by "parse_args()". Most actions add an attribute to this object using "setattr()". * *values* - The associated command-line arguments, with any type conversions applied. Type conversions are specified with the type keyword argument to "add_argument()". * *option_string* - The option string that was used to invoke this action. The "option_string" argument is optional, and will be absent if the action is associated with a positional argument. The "__call__()" method may perform arbitrary actions, but will typically set attributes on the "namespace" based on "dest" and "values". format_usage() "Action" subclasses can define a "format_usage()" method that takes no argument and return a string which will be used when printing the usage of the program. If such method is not provided, a sensible default will be used. class argparse.BooleanOptionalAction A subclass of "Action" for handling boolean flags with positive and negative options. Adding a single argument such as "--foo" automatically creates both "--foo" and "--no-foo" options, storing "True" and "False" respectively: >>> import argparse >>> parser = argparse.ArgumentParser() >>> parser.add_argument('--foo', action=argparse.BooleanOptionalAction) >>> parser.parse_args(['--no-foo']) Namespace(foo=False) Added in version 3.9. The parse_args() method ======================= ArgumentParser.parse_args(args=None, namespace=None) Convert argument strings to objects and assign them as attributes of the namespace. Return the populated namespace. Previous calls to "add_argument()" determine exactly what objects are created and how they are assigned. See the documentation for "add_argument()" for details. * args - List of strings to parse. The default is taken from "sys.argv". * namespace - An object to take the attributes. The default is a new empty "Namespace" object. Option value syntax ------------------- The "parse_args()" method supports several ways of specifying the value of an option (if it takes one). In the simplest case, the option and its value are passed as two separate arguments: >>> parser = argparse.ArgumentParser(prog='PROG') >>> parser.add_argument('-x') >>> parser.add_argument('--foo') >>> parser.parse_args(['-x', 'X']) Namespace(foo=None, x='X') >>> parser.parse_args(['--foo', 'FOO']) Namespace(foo='FOO', x=None) For long options (options with names longer than a single character), the option and value can also be passed as a single command-line argument, using "=" to separate them: >>> parser.parse_args(['--foo=FOO']) Namespace(foo='FOO', x=None) For short options (options only one character long), the option and its value can be concatenated: >>> parser.parse_args(['-xX']) Namespace(foo=None, x='X') Several short options can be joined together, using only a single "-" prefix, as long as only the last option (or none of them) requires a value: >>> parser = argparse.ArgumentParser(prog='PROG') >>> parser.add_argument('-x', action='store_true') >>> parser.add_argument('-y', action='store_true') >>> parser.add_argument('-z') >>> parser.parse_args(['-xyzZ']) Namespace(x=True, y=True, z='Z') Invalid arguments ----------------- While parsing the command line, "parse_args()" checks for a variety of errors, including ambiguous options, invalid types, invalid options, wrong number of positional arguments, etc. When it encounters such an error, it exits and prints the error along with a usage message: >>> parser = argparse.ArgumentParser(prog='PROG') >>> parser.add_argument('--foo', type=int) >>> parser.add_argument('bar', nargs='?') >>> # invalid type >>> parser.parse_args(['--foo', 'spam']) usage: PROG [-h] [--foo FOO] [bar] PROG: error: argument --foo: invalid int value: 'spam' >>> # invalid option >>> parser.parse_args(['--bar']) usage: PROG [-h] [--foo FOO] [bar] PROG: error: no such option: --bar >>> # wrong number of arguments >>> parser.parse_args(['spam', 'badger']) usage: PROG [-h] [--foo FOO] [bar] PROG: error: extra arguments found: badger Arguments containing "-" ------------------------ The "parse_args()" method attempts to give errors whenever the user has clearly made a mistake, but some situations are inherently ambiguous. For example, the command-line argument "-1" could either be an attempt to specify an option or an attempt to provide a positional argument. The "parse_args()" method is cautious here: positional arguments may only begin with "-" if they look like negative numbers and there are no options in the parser that look like negative numbers: >>> parser = argparse.ArgumentParser(prog='PROG') >>> parser.add_argument('-x') >>> parser.add_argument('foo', nargs='?') >>> # no negative number options, so -1 is a positional argument >>> parser.parse_args(['-x', '-1']) Namespace(foo=None, x='-1') >>> # no negative number options, so -1 and -5 are positional arguments >>> parser.parse_args(['-x', '-1', '-5']) Namespace(foo='-5', x='-1') >>> parser = argparse.ArgumentParser(prog='PROG') >>> parser.add_argument('-1', dest='one') >>> parser.add_argument('foo', nargs='?') >>> # negative number options present, so -1 is an option >>> parser.parse_args(['-1', 'X']) Namespace(foo=None, one='X') >>> # negative number options present, so -2 is an option >>> parser.parse_args(['-2']) usage: PROG [-h] [-1 ONE] [foo] PROG: error: no such option: -2 >>> # negative number options present, so both -1s are options >>> parser.parse_args(['-1', '-1']) usage: PROG [-h] [-1 ONE] [foo] PROG: error: argument -1: expected one argument If you have positional arguments that must begin with "-" and don’t look like negative numbers, you can insert the pseudo-argument "'--'" which tells "parse_args()" that everything after that is a positional argument: >>> parser.parse_args(['--', '-f']) Namespace(foo='-f', one=None) See also the argparse howto on ambiguous arguments for more details. Argument abbreviations (prefix matching) ---------------------------------------- The "parse_args()" method by default allows long options to be abbreviated to a prefix, if the abbreviation is unambiguous (the prefix matches a unique option): >>> parser = argparse.ArgumentParser(prog='PROG') >>> parser.add_argument('-bacon') >>> parser.add_argument('-badger') >>> parser.parse_args('-bac MMM'.split()) Namespace(bacon='MMM', badger=None) >>> parser.parse_args('-bad WOOD'.split()) Namespace(bacon=None, badger='WOOD') >>> parser.parse_args('-ba BA'.split()) usage: PROG [-h] [-bacon BACON] [-badger BADGER] PROG: error: ambiguous option: -ba could match -badger, -bacon An error is produced for arguments that could produce more than one options. This feature can be disabled by setting allow_abbrev to "False". Beyond "sys.argv" ----------------- Sometimes it may be useful to have an "ArgumentParser" parse arguments other than those of "sys.argv". This can be accomplished by passing a list of strings to "parse_args()". This is useful for testing at the interactive prompt: >>> parser = argparse.ArgumentParser() >>> parser.add_argument( ... 'integers', metavar='int', type=int, choices=range(10), ... nargs='+', help='an integer in the range 0..9') >>> parser.add_argument( ... '--sum', dest='accumulate', action='store_const', const=sum, ... default=max, help='sum the integers (default: find the max)') >>> parser.parse_args(['1', '2', '3', '4']) Namespace(accumulate=, integers=[1, 2, 3, 4]) >>> parser.parse_args(['1', '2', '3', '4', '--sum']) Namespace(accumulate=, integers=[1, 2, 3, 4]) The Namespace object -------------------- class argparse.Namespace Simple class used by default by "parse_args()" to create an object holding attributes and return it. This class is deliberately simple, just an "object" subclass with a readable string representation. If you prefer to have dict-like view of the attributes, you can use the standard Python idiom, "vars()": >>> parser = argparse.ArgumentParser() >>> parser.add_argument('--foo') >>> args = parser.parse_args(['--foo', 'BAR']) >>> vars(args) {'foo': 'BAR'} It may also be useful to have an "ArgumentParser" assign attributes to an already existing object, rather than a new "Namespace" object. This can be achieved by specifying the "namespace=" keyword argument: >>> class C: ... pass ... >>> c = C() >>> parser = argparse.ArgumentParser() >>> parser.add_argument('--foo') >>> parser.parse_args(args=['--foo', 'BAR'], namespace=c) >>> c.foo 'BAR' Other utilities =============== Sub-commands ------------ ArgumentParser.add_subparsers(*[, title][, description][, prog][, parser_class][, action][, dest][, required][, help][, metavar]) Many programs split up their functionality into a number of subcommands, for example, the "svn" program can invoke subcommands like "svn checkout", "svn update", and "svn commit". Splitting up functionality this way can be a particularly good idea when a program performs several different functions which require different kinds of command-line arguments. "ArgumentParser" supports the creation of such subcommands with the "add_subparsers()" method. The "add_subparsers()" method is normally called with no arguments and returns a special action object. This object has a single method, "add_parser()", which takes a command name and any "ArgumentParser" constructor arguments, and returns an "ArgumentParser" object that can be modified as usual. Description of parameters: * *title* - title for the sub-parser group in help output; by default “subcommands” if description is provided, otherwise uses title for positional arguments * *description* - description for the sub-parser group in help output, by default "None" * *prog* - usage information that will be displayed with sub- command help, by default the name of the program and any positional arguments before the subparser argument * *parser_class* - class which will be used to create sub-parser instances, by default the class of the current parser (e.g. "ArgumentParser") * action - the basic type of action to be taken when this argument is encountered at the command line * dest - name of the attribute under which sub-command name will be stored; by default "None" and no value is stored * required - Whether or not a subcommand must be provided, by default "False" (added in 3.7) * help - help for sub-parser group in help output, by default "None" * metavar - string presenting available subcommands in help; by default it is "None" and presents subcommands in form {cmd1, cmd2, ..} Some example usage: >>> # create the top-level parser >>> parser = argparse.ArgumentParser(prog='PROG') >>> parser.add_argument('--foo', action='store_true', help='foo help') >>> subparsers = parser.add_subparsers(help='subcommand help') >>> >>> # create the parser for the "a" command >>> parser_a = subparsers.add_parser('a', help='a help') >>> parser_a.add_argument('bar', type=int, help='bar help') >>> >>> # create the parser for the "b" command >>> parser_b = subparsers.add_parser('b', help='b help') >>> parser_b.add_argument('--baz', choices=('X', 'Y', 'Z'), help='baz help') >>> >>> # parse some argument lists >>> parser.parse_args(['a', '12']) Namespace(bar=12, foo=False) >>> parser.parse_args(['--foo', 'b', '--baz', 'Z']) Namespace(baz='Z', foo=True) Note that the object returned by "parse_args()" will only contain attributes for the main parser and the subparser that was selected by the command line (and not any other subparsers). So in the example above, when the "a" command is specified, only the "foo" and "bar" attributes are present, and when the "b" command is specified, only the "foo" and "baz" attributes are present. Similarly, when a help message is requested from a subparser, only the help for that particular parser will be printed. The help message will not include parent parser or sibling parser messages. (A help message for each subparser command, however, can be given by supplying the "help=" argument to "add_parser()" as above.) >>> parser.parse_args(['--help']) usage: PROG [-h] [--foo] {a,b} ... positional arguments: {a,b} subcommand help a a help b b help options: -h, --help show this help message and exit --foo foo help >>> parser.parse_args(['a', '--help']) usage: PROG a [-h] bar positional arguments: bar bar help options: -h, --help show this help message and exit >>> parser.parse_args(['b', '--help']) usage: PROG b [-h] [--baz {X,Y,Z}] options: -h, --help show this help message and exit --baz {X,Y,Z} baz help The "add_subparsers()" method also supports "title" and "description" keyword arguments. When either is present, the subparser’s commands will appear in their own group in the help output. For example: >>> parser = argparse.ArgumentParser() >>> subparsers = parser.add_subparsers(title='subcommands', ... description='valid subcommands', ... help='additional help') >>> subparsers.add_parser('foo') >>> subparsers.add_parser('bar') >>> parser.parse_args(['-h']) usage: [-h] {foo,bar} ... options: -h, --help show this help message and exit subcommands: valid subcommands {foo,bar} additional help Furthermore, "add_parser()" supports an additional *aliases* argument, which allows multiple strings to refer to the same subparser. This example, like "svn", aliases "co" as a shorthand for "checkout": >>> parser = argparse.ArgumentParser() >>> subparsers = parser.add_subparsers() >>> checkout = subparsers.add_parser('checkout', aliases=['co']) >>> checkout.add_argument('foo') >>> parser.parse_args(['co', 'bar']) Namespace(foo='bar') "add_parser()" supports also an additional *deprecated* argument, which allows to deprecate the subparser. >>> import argparse >>> parser = argparse.ArgumentParser(prog='chicken.py') >>> subparsers = parser.add_subparsers() >>> run = subparsers.add_parser('run') >>> fly = subparsers.add_parser('fly', deprecated=True) >>> parser.parse_args(['fly']) chicken.py: warning: command 'fly' is deprecated Namespace() Added in version 3.13. One particularly effective way of handling subcommands is to combine the use of the "add_subparsers()" method with calls to "set_defaults()" so that each subparser knows which Python function it should execute. For example: >>> # subcommand functions >>> def foo(args): ... print(args.x * args.y) ... >>> def bar(args): ... print('((%s))' % args.z) ... >>> # create the top-level parser >>> parser = argparse.ArgumentParser() >>> subparsers = parser.add_subparsers(required=True) >>> >>> # create the parser for the "foo" command >>> parser_foo = subparsers.add_parser('foo') >>> parser_foo.add_argument('-x', type=int, default=1) >>> parser_foo.add_argument('y', type=float) >>> parser_foo.set_defaults(func=foo) >>> >>> # create the parser for the "bar" command >>> parser_bar = subparsers.add_parser('bar') >>> parser_bar.add_argument('z') >>> parser_bar.set_defaults(func=bar) >>> >>> # parse the args and call whatever function was selected >>> args = parser.parse_args('foo 1 -x 2'.split()) >>> args.func(args) 2.0 >>> >>> # parse the args and call whatever function was selected >>> args = parser.parse_args('bar XYZYX'.split()) >>> args.func(args) ((XYZYX)) This way, you can let "parse_args()" do the job of calling the appropriate function after argument parsing is complete. Associating functions with actions like this is typically the easiest way to handle the different actions for each of your subparsers. However, if it is necessary to check the name of the subparser that was invoked, the "dest" keyword argument to the "add_subparsers()" call will work: >>> parser = argparse.ArgumentParser() >>> subparsers = parser.add_subparsers(dest='subparser_name') >>> subparser1 = subparsers.add_parser('1') >>> subparser1.add_argument('-x') >>> subparser2 = subparsers.add_parser('2') >>> subparser2.add_argument('y') >>> parser.parse_args(['2', 'frobble']) Namespace(subparser_name='2', y='frobble') Changed in version 3.7: New *required* keyword-only parameter. FileType objects ---------------- class argparse.FileType(mode='r', bufsize=-1, encoding=None, errors=None) The "FileType" factory creates objects that can be passed to the type argument of "ArgumentParser.add_argument()". Arguments that have "FileType" objects as their type will open command-line arguments as files with the requested modes, buffer sizes, encodings and error handling (see the "open()" function for more details): >>> parser = argparse.ArgumentParser() >>> parser.add_argument('--raw', type=argparse.FileType('wb', 0)) >>> parser.add_argument('out', type=argparse.FileType('w', encoding='UTF-8')) >>> parser.parse_args(['--raw', 'raw.dat', 'file.txt']) Namespace(out=<_io.TextIOWrapper name='file.txt' mode='w' encoding='UTF-8'>, raw=<_io.FileIO name='raw.dat' mode='wb'>) FileType objects understand the pseudo-argument "'-'" and automatically convert this into "sys.stdin" for readable "FileType" objects and "sys.stdout" for writable "FileType" objects: >>> parser = argparse.ArgumentParser() >>> parser.add_argument('infile', type=argparse.FileType('r')) >>> parser.parse_args(['-']) Namespace(infile=<_io.TextIOWrapper name='' encoding='UTF-8'>) Changed in version 3.4: Added the *encodings* and *errors* parameters. Argument groups --------------- ArgumentParser.add_argument_group(title=None, description=None, *[, argument_default][, conflict_handler]) By default, "ArgumentParser" groups command-line arguments into “positional arguments” and “options” when displaying help messages. When there is a better conceptual grouping of arguments than this default one, appropriate groups can be created using the "add_argument_group()" method: >>> parser = argparse.ArgumentParser(prog='PROG', add_help=False) >>> group = parser.add_argument_group('group') >>> group.add_argument('--foo', help='foo help') >>> group.add_argument('bar', help='bar help') >>> parser.print_help() usage: PROG [--foo FOO] bar group: bar bar help --foo FOO foo help The "add_argument_group()" method returns an argument group object which has an "add_argument()" method just like a regular "ArgumentParser". When an argument is added to the group, the parser treats it just like a normal argument, but displays the argument in a separate group for help messages. The "add_argument_group()" method accepts *title* and *description* arguments which can be used to customize this display: >>> parser = argparse.ArgumentParser(prog='PROG', add_help=False) >>> group1 = parser.add_argument_group('group1', 'group1 description') >>> group1.add_argument('foo', help='foo help') >>> group2 = parser.add_argument_group('group2', 'group2 description') >>> group2.add_argument('--bar', help='bar help') >>> parser.print_help() usage: PROG [--bar BAR] foo group1: group1 description foo foo help group2: group2 description --bar BAR bar help The optional, keyword-only parameters argument_default and conflict_handler allow for finer-grained control of the behavior of the argument group. These parameters have the same meaning as in the "ArgumentParser" constructor, but apply specifically to the argument group rather than the entire parser. Note that any arguments not in your user-defined groups will end up back in the usual “positional arguments” and “optional arguments” sections. Changed in version 3.11: Calling "add_argument_group()" on an argument group is deprecated. This feature was never supported and does not always work correctly. The function exists on the API by accident through inheritance and will be removed in the future. Mutual exclusion ---------------- ArgumentParser.add_mutually_exclusive_group(required=False) Create a mutually exclusive group. "argparse" will make sure that only one of the arguments in the mutually exclusive group was present on the command line: >>> parser = argparse.ArgumentParser(prog='PROG') >>> group = parser.add_mutually_exclusive_group() >>> group.add_argument('--foo', action='store_true') >>> group.add_argument('--bar', action='store_false') >>> parser.parse_args(['--foo']) Namespace(bar=True, foo=True) >>> parser.parse_args(['--bar']) Namespace(bar=False, foo=False) >>> parser.parse_args(['--foo', '--bar']) usage: PROG [-h] [--foo | --bar] PROG: error: argument --bar: not allowed with argument --foo The "add_mutually_exclusive_group()" method also accepts a *required* argument, to indicate that at least one of the mutually exclusive arguments is required: >>> parser = argparse.ArgumentParser(prog='PROG') >>> group = parser.add_mutually_exclusive_group(required=True) >>> group.add_argument('--foo', action='store_true') >>> group.add_argument('--bar', action='store_false') >>> parser.parse_args([]) usage: PROG [-h] (--foo | --bar) PROG: error: one of the arguments --foo --bar is required Note that currently mutually exclusive argument groups do not support the *title* and *description* arguments of "add_argument_group()". However, a mutually exclusive group can be added to an argument group that has a title and description. For example: >>> parser = argparse.ArgumentParser(prog='PROG') >>> group = parser.add_argument_group('Group title', 'Group description') >>> exclusive_group = group.add_mutually_exclusive_group(required=True) >>> exclusive_group.add_argument('--foo', help='foo help') >>> exclusive_group.add_argument('--bar', help='bar help') >>> parser.print_help() usage: PROG [-h] (--foo FOO | --bar BAR) options: -h, --help show this help message and exit Group title: Group description --foo FOO foo help --bar BAR bar help Changed in version 3.11: Calling "add_argument_group()" or "add_mutually_exclusive_group()" on a mutually exclusive group is deprecated. These features were never supported and do not always work correctly. The functions exist on the API by accident through inheritance and will be removed in the future. Parser defaults --------------- ArgumentParser.set_defaults(**kwargs) Most of the time, the attributes of the object returned by "parse_args()" will be fully determined by inspecting the command- line arguments and the argument actions. "set_defaults()" allows some additional attributes that are determined without any inspection of the command line to be added: >>> parser = argparse.ArgumentParser() >>> parser.add_argument('foo', type=int) >>> parser.set_defaults(bar=42, baz='badger') >>> parser.parse_args(['736']) Namespace(bar=42, baz='badger', foo=736) Note that parser-level defaults always override argument-level defaults: >>> parser = argparse.ArgumentParser() >>> parser.add_argument('--foo', default='bar') >>> parser.set_defaults(foo='spam') >>> parser.parse_args([]) Namespace(foo='spam') Parser-level defaults can be particularly useful when working with multiple parsers. See the "add_subparsers()" method for an example of this type. ArgumentParser.get_default(dest) Get the default value for a namespace attribute, as set by either "add_argument()" or by "set_defaults()": >>> parser = argparse.ArgumentParser() >>> parser.add_argument('--foo', default='badger') >>> parser.get_default('foo') 'badger' Printing help ------------- In most typical applications, "parse_args()" will take care of formatting and printing any usage or error messages. However, several formatting methods are available: ArgumentParser.print_usage(file=None) Print a brief description of how the "ArgumentParser" should be invoked on the command line. If *file* is "None", "sys.stdout" is assumed. ArgumentParser.print_help(file=None) Print a help message, including the program usage and information about the arguments registered with the "ArgumentParser". If *file* is "None", "sys.stdout" is assumed. There are also variants of these methods that simply return a string instead of printing it: ArgumentParser.format_usage() Return a string containing a brief description of how the "ArgumentParser" should be invoked on the command line. ArgumentParser.format_help() Return a string containing a help message, including the program usage and information about the arguments registered with the "ArgumentParser". Partial parsing --------------- ArgumentParser.parse_known_args(args=None, namespace=None) Sometimes a script only needs to handle a specific set of command- line arguments, leaving any unrecognized arguments for another script or program. In these cases, the "parse_known_args()" method can be useful. This method works similarly to "parse_args()", but it does not raise an error for extra, unrecognized arguments. Instead, it parses the known arguments and returns a two item tuple that contains the populated namespace and the list of any unrecognized arguments. >>> parser = argparse.ArgumentParser() >>> parser.add_argument('--foo', action='store_true') >>> parser.add_argument('bar') >>> parser.parse_known_args(['--foo', '--badger', 'BAR', 'spam']) (Namespace(bar='BAR', foo=True), ['--badger', 'spam']) Warning: Prefix matching rules apply to "parse_known_args()". The parser may consume an option even if it’s just a prefix of one of its known options, instead of leaving it in the remaining arguments list. Customizing file parsing ------------------------ ArgumentParser.convert_arg_line_to_args(arg_line) Arguments that are read from a file (see the *fromfile_prefix_chars* keyword argument to the "ArgumentParser" constructor) are read one argument per line. "convert_arg_line_to_args()" can be overridden for fancier reading. This method takes a single argument *arg_line* which is a string read from the argument file. It returns a list of arguments parsed from this string. The method is called once per line read from the argument file, in order. A useful override of this method is one that treats each space- separated word as an argument. The following example demonstrates how to do this: class MyArgumentParser(argparse.ArgumentParser): def convert_arg_line_to_args(self, arg_line): return arg_line.split() Exiting methods --------------- ArgumentParser.exit(status=0, message=None) This method terminates the program, exiting with the specified *status* and, if given, it prints a *message* to "sys.stderr" before that. The user can override this method to handle these steps differently: class ErrorCatchingArgumentParser(argparse.ArgumentParser): def exit(self, status=0, message=None): if status: raise Exception(f'Exiting because of an error: {message}') exit(status) ArgumentParser.error(message) This method prints a usage message, including the *message*, to "sys.stderr" and terminates the program with a status code of 2. Intermixed parsing ------------------ ArgumentParser.parse_intermixed_args(args=None, namespace=None) ArgumentParser.parse_known_intermixed_args(args=None, namespace=None) A number of Unix commands allow the user to intermix optional arguments with positional arguments. The "parse_intermixed_args()" and "parse_known_intermixed_args()" methods support this parsing style. These parsers do not support all the "argparse" features, and will raise exceptions if unsupported features are used. In particular, subparsers, and mutually exclusive groups that include both optionals and positionals are not supported. The following example shows the difference between "parse_known_args()" and "parse_intermixed_args()": the former returns "['2', '3']" as unparsed arguments, while the latter collects all the positionals into "rest". >>> parser = argparse.ArgumentParser() >>> parser.add_argument('--foo') >>> parser.add_argument('cmd') >>> parser.add_argument('rest', nargs='*', type=int) >>> parser.parse_known_args('doit 1 --foo bar 2 3'.split()) (Namespace(cmd='doit', foo='bar', rest=[1]), ['2', '3']) >>> parser.parse_intermixed_args('doit 1 --foo bar 2 3'.split()) Namespace(cmd='doit', foo='bar', rest=[1, 2, 3]) "parse_known_intermixed_args()" returns a two item tuple containing the populated namespace and the list of remaining argument strings. "parse_intermixed_args()" raises an error if there are any remaining unparsed argument strings. Added in version 3.7. Registering custom types or actions ----------------------------------- ArgumentParser.register(registry_name, value, object) Sometimes it’s desirable to use a custom string in error messages to provide more user-friendly output. In these cases, "register()" can be used to register custom actions or types with a parser and allow you to reference the type by their registered name instead of their callable name. The "register()" method accepts three arguments - a *registry_name*, specifying the internal registry where the object will be stored (e.g., "action", "type"), *value*, which is the key under which the object will be registered, and object, the callable to be registered. The following example shows how to register a custom type with a parser: >>> import argparse >>> parser = argparse.ArgumentParser() >>> parser.register('type', 'hexadecimal integer', lambda s: int(s, 16)) >>> parser.add_argument('--foo', type='hexadecimal integer') _StoreAction(option_strings=['--foo'], dest='foo', nargs=None, const=None, default=None, type='hexadecimal integer', choices=None, required=False, help=None, metavar=None, deprecated=False) >>> parser.parse_args(['--foo', '0xFA']) Namespace(foo=250) >>> parser.parse_args(['--foo', '1.2']) usage: PROG [-h] [--foo FOO] PROG: error: argument --foo: invalid 'hexadecimal integer' value: '1.2' Exceptions ========== exception argparse.ArgumentError An error from creating or using an argument (optional or positional). The string value of this exception is the message, augmented with information about the argument that caused it. exception argparse.ArgumentTypeError Raised when something goes wrong converting a command line string to a type. -[ Guides and Tutorials ]- * Argparse Tutorial * Migrating "optparse" code to "argparse" "array" — Efficient arrays of numeric values ******************************************** ====================================================================== This module defines an object type which can compactly represent an array of basic values: characters, integers, floating-point numbers. Arrays are sequence types and behave very much like lists, except that the type of objects stored in them is constrained. The type is specified at object creation time by using a *type code*, which is a single character. The following type codes are defined: +-------------+----------------------+---------------------+-------------------------+---------+ | Type code | C Type | Python Type | Minimum size in bytes | Notes | |=============|======================|=====================|=========================|=========| | "'b'" | signed char | int | 1 | | +-------------+----------------------+---------------------+-------------------------+---------+ | "'B'" | unsigned char | int | 1 | | +-------------+----------------------+---------------------+-------------------------+---------+ | "'u'" | wchar_t | Unicode character | 2 | (1) | +-------------+----------------------+---------------------+-------------------------+---------+ | "'w'" | Py_UCS4 | Unicode character | 4 | (2) | +-------------+----------------------+---------------------+-------------------------+---------+ | "'h'" | signed short | int | 2 | | +-------------+----------------------+---------------------+-------------------------+---------+ | "'H'" | unsigned short | int | 2 | | +-------------+----------------------+---------------------+-------------------------+---------+ | "'i'" | signed int | int | 2 | | +-------------+----------------------+---------------------+-------------------------+---------+ | "'I'" | unsigned int | int | 2 | | +-------------+----------------------+---------------------+-------------------------+---------+ | "'l'" | signed long | int | 4 | | +-------------+----------------------+---------------------+-------------------------+---------+ | "'L'" | unsigned long | int | 4 | | +-------------+----------------------+---------------------+-------------------------+---------+ | "'q'" | signed long long | int | 8 | | +-------------+----------------------+---------------------+-------------------------+---------+ | "'Q'" | unsigned long long | int | 8 | | +-------------+----------------------+---------------------+-------------------------+---------+ | "'f'" | float | float | 4 | | +-------------+----------------------+---------------------+-------------------------+---------+ | "'d'" | double | float | 8 | | +-------------+----------------------+---------------------+-------------------------+---------+ Notes: 1. It can be 16 bits or 32 bits depending on the platform. Changed in version 3.9: "array('u')" now uses "wchar_t" as C type instead of deprecated "Py_UNICODE". This change doesn’t affect its behavior because "Py_UNICODE" is alias of "wchar_t" since Python 3.3. Deprecated since version 3.3, will be removed in version 3.16: Please migrate to "'w'" typecode. 2. Added in version 3.13. The actual representation of values is determined by the machine architecture (strictly speaking, by the C implementation). The actual size can be accessed through the "array.itemsize" attribute. The module defines the following item: array.typecodes A string with all available type codes. The module defines the following type: class array.array(typecode[, initializer]) A new array whose items are restricted by *typecode*, and initialized from the optional *initializer* value, which must be a "bytes" or "bytearray" object, a Unicode string, or iterable over elements of the appropriate type. If given a "bytes" or "bytearray" object, the initializer is passed to the new array’s "frombytes()" method; if given a Unicode string, the initializer is passed to the "fromunicode()" method; otherwise, the initializer’s iterator is passed to the "extend()" method to add initial items to the array. Array objects support the ordinary sequence operations of indexing, slicing, concatenation, and multiplication. When using slice assignment, the assigned value must be an array object with the same type code; in all other cases, "TypeError" is raised. Array objects also implement the buffer interface, and may be used wherever *bytes-like objects* are supported. Raises an auditing event "array.__new__" with arguments "typecode", "initializer". typecode The typecode character used to create the array. itemsize The length in bytes of one array item in the internal representation. append(x) Append a new item with value *x* to the end of the array. buffer_info() Return a tuple "(address, length)" giving the current memory address and the length in elements of the buffer used to hold array’s contents. The size of the memory buffer in bytes can be computed as "array.buffer_info()[1] * array.itemsize". This is occasionally useful when working with low-level (and inherently unsafe) I/O interfaces that require memory addresses, such as certain "ioctl()" operations. The returned numbers are valid as long as the array exists and no length-changing operations are applied to it. Note: When using array objects from code written in C or C++ (the only way to effectively make use of this information), it makes more sense to use the buffer interface supported by array objects. This method is maintained for backward compatibility and should be avoided in new code. The buffer interface is documented in Buffer Protocol. byteswap() “Byteswap” all items of the array. This is only supported for values which are 1, 2, 4, or 8 bytes in size; for other types of values, "RuntimeError" is raised. It is useful when reading data from a file written on a machine with a different byte order. count(x) Return the number of occurrences of *x* in the array. extend(iterable) Append items from *iterable* to the end of the array. If *iterable* is another array, it must have *exactly* the same type code; if not, "TypeError" will be raised. If *iterable* is not an array, it must be iterable and its elements must be the right type to be appended to the array. frombytes(buffer) Appends items from the *bytes-like object*, interpreting its content as an array of machine values (as if it had been read from a file using the "fromfile()" method). Added in version 3.2: "fromstring()" is renamed to "frombytes()" for clarity. fromfile(f, n) Read *n* items (as machine values) from the *file object* *f* and append them to the end of the array. If less than *n* items are available, "EOFError" is raised, but the items that were available are still inserted into the array. fromlist(list) Append items from the list. This is equivalent to "for x in list: a.append(x)" except that if there is a type error, the array is unchanged. fromunicode(s) Extends this array with data from the given Unicode string. The array must have type code "'u'" or "'w'"; otherwise a "ValueError" is raised. Use "array.frombytes(unicodestring.encode(enc))" to append Unicode data to an array of some other type. index(x[, start[, stop]]) Return the smallest *i* such that *i* is the index of the first occurrence of *x* in the array. The optional arguments *start* and *stop* can be specified to search for *x* within a subsection of the array. Raise "ValueError" if *x* is not found. Changed in version 3.10: Added optional *start* and *stop* parameters. insert(i, x) Insert a new item with value *x* in the array before position *i*. Negative values are treated as being relative to the end of the array. pop([i]) Removes the item with the index *i* from the array and returns it. The optional argument defaults to "-1", so that by default the last item is removed and returned. remove(x) Remove the first occurrence of *x* from the array. clear() Remove all elements from the array. Added in version 3.13. reverse() Reverse the order of the items in the array. tobytes() Convert the array to an array of machine values and return the bytes representation (the same sequence of bytes that would be written to a file by the "tofile()" method.) Added in version 3.2: "tostring()" is renamed to "tobytes()" for clarity. tofile(f) Write all items (as machine values) to the *file object* *f*. tolist() Convert the array to an ordinary list with the same items. tounicode() Convert the array to a Unicode string. The array must have a type "'u'" or "'w'"; otherwise a "ValueError" is raised. Use "array.tobytes().decode(enc)" to obtain a Unicode string from an array of some other type. The string representation of array objects has the form "array(typecode, initializer)". The *initializer* is omitted if the array is empty, otherwise it is a Unicode string if the *typecode* is "'u'" or "'w'", otherwise it is a list of numbers. The string representation is guaranteed to be able to be converted back to an array with the same type and value using "eval()", so long as the "array" class has been imported using "from array import array". Variables "inf" and "nan" must also be defined if it contains corresponding floating-point values. Examples: array('l') array('w', 'hello \u2641') array('l', [1, 2, 3, 4, 5]) array('d', [1.0, 2.0, 3.14, -inf, nan]) See also: Module "struct" Packing and unpacking of heterogeneous binary data. NumPy The NumPy package defines another array type. "ast" — Abstract Syntax Trees ***************************** **Source code:** Lib/ast.py ====================================================================== The "ast" module helps Python applications to process trees of the Python abstract syntax grammar. The abstract syntax itself might change with each Python release; this module helps to find out programmatically what the current grammar looks like. An abstract syntax tree can be generated by passing "ast.PyCF_ONLY_AST" as a flag to the "compile()" built-in function, or using the "parse()" helper provided in this module. The result will be a tree of objects whose classes all inherit from "ast.AST". An abstract syntax tree can be compiled into a Python code object using the built-in "compile()" function. Abstract Grammar ================ The abstract grammar is currently defined as follows: -- ASDL's 4 builtin types are: -- identifier, int, string, constant module Python { mod = Module(stmt* body, type_ignore* type_ignores) | Interactive(stmt* body) | Expression(expr body) | FunctionType(expr* argtypes, expr returns) stmt = FunctionDef(identifier name, arguments args, stmt* body, expr* decorator_list, expr? returns, string? type_comment, type_param* type_params) | AsyncFunctionDef(identifier name, arguments args, stmt* body, expr* decorator_list, expr? returns, string? type_comment, type_param* type_params) | ClassDef(identifier name, expr* bases, keyword* keywords, stmt* body, expr* decorator_list, type_param* type_params) | Return(expr? value) | Delete(expr* targets) | Assign(expr* targets, expr value, string? type_comment) | TypeAlias(expr name, type_param* type_params, expr value) | AugAssign(expr target, operator op, expr value) -- 'simple' indicates that we annotate simple name without parens | AnnAssign(expr target, expr annotation, expr? value, int simple) -- use 'orelse' because else is a keyword in target languages | For(expr target, expr iter, stmt* body, stmt* orelse, string? type_comment) | AsyncFor(expr target, expr iter, stmt* body, stmt* orelse, string? type_comment) | While(expr test, stmt* body, stmt* orelse) | If(expr test, stmt* body, stmt* orelse) | With(withitem* items, stmt* body, string? type_comment) | AsyncWith(withitem* items, stmt* body, string? type_comment) | Match(expr subject, match_case* cases) | Raise(expr? exc, expr? cause) | Try(stmt* body, excepthandler* handlers, stmt* orelse, stmt* finalbody) | TryStar(stmt* body, excepthandler* handlers, stmt* orelse, stmt* finalbody) | Assert(expr test, expr? msg) | Import(alias* names) | ImportFrom(identifier? module, alias* names, int? level) | Global(identifier* names) | Nonlocal(identifier* names) | Expr(expr value) | Pass | Break | Continue -- col_offset is the byte offset in the utf8 string the parser uses attributes (int lineno, int col_offset, int? end_lineno, int? end_col_offset) -- BoolOp() can use left & right? expr = BoolOp(boolop op, expr* values) | NamedExpr(expr target, expr value) | BinOp(expr left, operator op, expr right) | UnaryOp(unaryop op, expr operand) | Lambda(arguments args, expr body) | IfExp(expr test, expr body, expr orelse) | Dict(expr* keys, expr* values) | Set(expr* elts) | ListComp(expr elt, comprehension* generators) | SetComp(expr elt, comprehension* generators) | DictComp(expr key, expr value, comprehension* generators) | GeneratorExp(expr elt, comprehension* generators) -- the grammar constrains where yield expressions can occur | Await(expr value) | Yield(expr? value) | YieldFrom(expr value) -- need sequences for compare to distinguish between -- x < 4 < 3 and (x < 4) < 3 | Compare(expr left, cmpop* ops, expr* comparators) | Call(expr func, expr* args, keyword* keywords) | FormattedValue(expr value, int conversion, expr? format_spec) | JoinedStr(expr* values) | Constant(constant value, string? kind) -- the following expression can appear in assignment context | Attribute(expr value, identifier attr, expr_context ctx) | Subscript(expr value, expr slice, expr_context ctx) | Starred(expr value, expr_context ctx) | Name(identifier id, expr_context ctx) | List(expr* elts, expr_context ctx) | Tuple(expr* elts, expr_context ctx) -- can appear only in Subscript | Slice(expr? lower, expr? upper, expr? step) -- col_offset is the byte offset in the utf8 string the parser uses attributes (int lineno, int col_offset, int? end_lineno, int? end_col_offset) expr_context = Load | Store | Del boolop = And | Or operator = Add | Sub | Mult | MatMult | Div | Mod | Pow | LShift | RShift | BitOr | BitXor | BitAnd | FloorDiv unaryop = Invert | Not | UAdd | USub cmpop = Eq | NotEq | Lt | LtE | Gt | GtE | Is | IsNot | In | NotIn comprehension = (expr target, expr iter, expr* ifs, int is_async) excepthandler = ExceptHandler(expr? type, identifier? name, stmt* body) attributes (int lineno, int col_offset, int? end_lineno, int? end_col_offset) arguments = (arg* posonlyargs, arg* args, arg? vararg, arg* kwonlyargs, expr* kw_defaults, arg? kwarg, expr* defaults) arg = (identifier arg, expr? annotation, string? type_comment) attributes (int lineno, int col_offset, int? end_lineno, int? end_col_offset) -- keyword arguments supplied to call (NULL identifier for **kwargs) keyword = (identifier? arg, expr value) attributes (int lineno, int col_offset, int? end_lineno, int? end_col_offset) -- import name with optional 'as' alias. alias = (identifier name, identifier? asname) attributes (int lineno, int col_offset, int? end_lineno, int? end_col_offset) withitem = (expr context_expr, expr? optional_vars) match_case = (pattern pattern, expr? guard, stmt* body) pattern = MatchValue(expr value) | MatchSingleton(constant value) | MatchSequence(pattern* patterns) | MatchMapping(expr* keys, pattern* patterns, identifier? rest) | MatchClass(expr cls, pattern* patterns, identifier* kwd_attrs, pattern* kwd_patterns) | MatchStar(identifier? name) -- The optional "rest" MatchMapping parameter handles capturing extra mapping keys | MatchAs(pattern? pattern, identifier? name) | MatchOr(pattern* patterns) attributes (int lineno, int col_offset, int end_lineno, int end_col_offset) type_ignore = TypeIgnore(int lineno, string tag) type_param = TypeVar(identifier name, expr? bound, expr? default_value) | ParamSpec(identifier name, expr? default_value) | TypeVarTuple(identifier name, expr? default_value) attributes (int lineno, int col_offset, int end_lineno, int end_col_offset) } Node classes ============ class ast.AST This is the base of all AST node classes. The actual node classes are derived from the "Parser/Python.asdl" file, which is reproduced above. They are defined in the "_ast" C module and re-exported in "ast". There is one class defined for each left-hand side symbol in the abstract grammar (for example, "ast.stmt" or "ast.expr"). In addition, there is one class defined for each constructor on the right-hand side; these classes inherit from the classes for the left-hand side trees. For example, "ast.BinOp" inherits from "ast.expr". For production rules with alternatives (aka “sums”), the left-hand side class is abstract: only instances of specific constructor nodes are ever created. _fields Each concrete class has an attribute "_fields" which gives the names of all child nodes. Each instance of a concrete class has one attribute for each child node, of the type as defined in the grammar. For example, "ast.BinOp" instances have an attribute "left" of type "ast.expr". If these attributes are marked as optional in the grammar (using a question mark), the value might be "None". If the attributes can have zero-or-more values (marked with an asterisk), the values are represented as Python lists. All possible attributes must be present and have valid values when compiling an AST with "compile()". _field_types The "_field_types" attribute on each concrete class is a dictionary mapping field names (as also listed in "_fields") to their types. >>> ast.TypeVar._field_types {'name': , 'bound': ast.expr | None, 'default_value': ast.expr | None} Added in version 3.13. lineno col_offset end_lineno end_col_offset Instances of "ast.expr" and "ast.stmt" subclasses have "lineno", "col_offset", "end_lineno", and "end_col_offset" attributes. The "lineno" and "end_lineno" are the first and last line numbers of source text span (1-indexed so the first line is line 1) and the "col_offset" and "end_col_offset" are the corresponding UTF-8 byte offsets of the first and last tokens that generated the node. The UTF-8 offset is recorded because the parser uses UTF-8 internally. Note that the end positions are not required by the compiler and are therefore optional. The end offset is *after* the last symbol, for example one can get the source segment of a one-line expression node using "source_line[node.col_offset : node.end_col_offset]". The constructor of a class "ast.T" parses its arguments as follows: * If there are positional arguments, there must be as many as there are items in "T._fields"; they will be assigned as attributes of these names. * If there are keyword arguments, they will set the attributes of the same names to the given values. For example, to create and populate an "ast.UnaryOp" node, you could use node = ast.UnaryOp(ast.USub(), ast.Constant(5, lineno=0, col_offset=0), lineno=0, col_offset=0) If a field that is optional in the grammar is omitted from the constructor, it defaults to "None". If a list field is omitted, it defaults to the empty list. If a field of type "ast.expr_context" is omitted, it defaults to "Load()". If any other field is omitted, a "DeprecationWarning" is raised and the AST node will not have this field. In Python 3.15, this condition will raise an error. Changed in version 3.8: Class "ast.Constant" is now used for all constants. Changed in version 3.9: Simple indices are represented by their value, extended slices are represented as tuples. Deprecated since version 3.8: Old classes "ast.Num", "ast.Str", "ast.Bytes", "ast.NameConstant" and "ast.Ellipsis" are still available, but they will be removed in future Python releases. In the meantime, instantiating them will return an instance of a different class. Deprecated since version 3.9: Old classes "ast.Index" and "ast.ExtSlice" are still available, but they will be removed in future Python releases. In the meantime, instantiating them will return an instance of a different class. Deprecated since version 3.13, will be removed in version 3.15: Previous versions of Python allowed the creation of AST nodes that were missing required fields. Similarly, AST node constructors allowed arbitrary keyword arguments that were set as attributes of the AST node, even if they did not match any of the fields of the AST node. This behavior is deprecated and will be removed in Python 3.15. Note: The descriptions of the specific node classes displayed here were initially adapted from the fantastic Green Tree Snakes project and all its contributors. Root nodes ---------- class ast.Module(body, type_ignores) A Python module, as with file input. Node type generated by "ast.parse()" in the default ""exec"" *mode*. "body" is a "list" of the module’s Statements. "type_ignores" is a "list" of the module’s type ignore comments; see "ast.parse()" for more details. >>> print(ast.dump(ast.parse('x = 1'), indent=4)) Module( body=[ Assign( targets=[ Name(id='x', ctx=Store())], value=Constant(value=1))]) class ast.Expression(body) A single Python expression input. Node type generated by "ast.parse()" when *mode* is ""eval"". "body" is a single node, one of the expression types. >>> print(ast.dump(ast.parse('123', mode='eval'), indent=4)) Expression( body=Constant(value=123)) class ast.Interactive(body) A single interactive input, like in Interactive Mode. Node type generated by "ast.parse()" when *mode* is ""single"". "body" is a "list" of statement nodes. >>> print(ast.dump(ast.parse('x = 1; y = 2', mode='single'), indent=4)) Interactive( body=[ Assign( targets=[ Name(id='x', ctx=Store())], value=Constant(value=1)), Assign( targets=[ Name(id='y', ctx=Store())], value=Constant(value=2))]) class ast.FunctionType(argtypes, returns) A representation of an old-style type comments for functions, as Python versions prior to 3.5 didn’t support **PEP 484** annotations. Node type generated by "ast.parse()" when *mode* is ""func_type"". Such type comments would look like this: def sum_two_number(a, b): # type: (int, int) -> int return a + b "argtypes" is a "list" of expression nodes. "returns" is a single expression node. >>> print(ast.dump(ast.parse('(int, str) -> List[int]', mode='func_type'), indent=4)) FunctionType( argtypes=[ Name(id='int', ctx=Load()), Name(id='str', ctx=Load())], returns=Subscript( value=Name(id='List', ctx=Load()), slice=Name(id='int', ctx=Load()), ctx=Load())) Added in version 3.8. Literals -------- class ast.Constant(value) A constant value. The "value" attribute of the "Constant" literal contains the Python object it represents. The values represented can be instances of "str", "bytes", "int", "float", "complex", and "bool", and the constants "None" and "Ellipsis". >>> print(ast.dump(ast.parse('123', mode='eval'), indent=4)) Expression( body=Constant(value=123)) class ast.FormattedValue(value, conversion, format_spec) Node representing a single formatting field in an f-string. If the string contains a single formatting field and nothing else the node can be isolated otherwise it appears in "JoinedStr". * "value" is any expression node (such as a literal, a variable, or a function call). * "conversion" is an integer: * -1: no formatting * 115: "!s" string formatting * 114: "!r" repr formatting * 97: "!a" ascii formatting * "format_spec" is a "JoinedStr" node representing the formatting of the value, or "None" if no format was specified. Both "conversion" and "format_spec" can be set at the same time. class ast.JoinedStr(values) An f-string, comprising a series of "FormattedValue" and "Constant" nodes. >>> print(ast.dump(ast.parse('f"sin({a}) is {sin(a):.3}"', mode='eval'), indent=4)) Expression( body=JoinedStr( values=[ Constant(value='sin('), FormattedValue( value=Name(id='a', ctx=Load()), conversion=-1), Constant(value=') is '), FormattedValue( value=Call( func=Name(id='sin', ctx=Load()), args=[ Name(id='a', ctx=Load())]), conversion=-1, format_spec=JoinedStr( values=[ Constant(value='.3')]))])) class ast.List(elts, ctx) class ast.Tuple(elts, ctx) A list or tuple. "elts" holds a list of nodes representing the elements. "ctx" is "Store" if the container is an assignment target (i.e. "(x,y)=something"), and "Load" otherwise. >>> print(ast.dump(ast.parse('[1, 2, 3]', mode='eval'), indent=4)) Expression( body=List( elts=[ Constant(value=1), Constant(value=2), Constant(value=3)], ctx=Load())) >>> print(ast.dump(ast.parse('(1, 2, 3)', mode='eval'), indent=4)) Expression( body=Tuple( elts=[ Constant(value=1), Constant(value=2), Constant(value=3)], ctx=Load())) class ast.Set(elts) A set. "elts" holds a list of nodes representing the set’s elements. >>> print(ast.dump(ast.parse('{1, 2, 3}', mode='eval'), indent=4)) Expression( body=Set( elts=[ Constant(value=1), Constant(value=2), Constant(value=3)])) class ast.Dict(keys, values) A dictionary. "keys" and "values" hold lists of nodes representing the keys and the values respectively, in matching order (what would be returned when calling "dictionary.keys()" and "dictionary.values()"). When doing dictionary unpacking using dictionary literals the expression to be expanded goes in the "values" list, with a "None" at the corresponding position in "keys". >>> print(ast.dump(ast.parse('{"a":1, **d}', mode='eval'), indent=4)) Expression( body=Dict( keys=[ Constant(value='a'), None], values=[ Constant(value=1), Name(id='d', ctx=Load())])) Variables --------- class ast.Name(id, ctx) A variable name. "id" holds the name as a string, and "ctx" is one of the following types. class ast.Load class ast.Store class ast.Del Variable references can be used to load the value of a variable, to assign a new value to it, or to delete it. Variable references are given a context to distinguish these cases. >>> print(ast.dump(ast.parse('a'), indent=4)) Module( body=[ Expr( value=Name(id='a', ctx=Load()))]) >>> print(ast.dump(ast.parse('a = 1'), indent=4)) Module( body=[ Assign( targets=[ Name(id='a', ctx=Store())], value=Constant(value=1))]) >>> print(ast.dump(ast.parse('del a'), indent=4)) Module( body=[ Delete( targets=[ Name(id='a', ctx=Del())])]) class ast.Starred(value, ctx) A "*var" variable reference. "value" holds the variable, typically a "Name" node. This type must be used when building a "Call" node with "*args". >>> print(ast.dump(ast.parse('a, *b = it'), indent=4)) Module( body=[ Assign( targets=[ Tuple( elts=[ Name(id='a', ctx=Store()), Starred( value=Name(id='b', ctx=Store()), ctx=Store())], ctx=Store())], value=Name(id='it', ctx=Load()))]) Expressions ----------- class ast.Expr(value) When an expression, such as a function call, appears as a statement by itself with its return value not used or stored, it is wrapped in this container. "value" holds one of the other nodes in this section, a "Constant", a "Name", a "Lambda", a "Yield" or "YieldFrom" node. >>> print(ast.dump(ast.parse('-a'), indent=4)) Module( body=[ Expr( value=UnaryOp( op=USub(), operand=Name(id='a', ctx=Load())))]) class ast.UnaryOp(op, operand) A unary operation. "op" is the operator, and "operand" any expression node. class ast.UAdd class ast.USub class ast.Not class ast.Invert Unary operator tokens. "Not" is the "not" keyword, "Invert" is the "~" operator. >>> print(ast.dump(ast.parse('not x', mode='eval'), indent=4)) Expression( body=UnaryOp( op=Not(), operand=Name(id='x', ctx=Load()))) class ast.BinOp(left, op, right) A binary operation (like addition or division). "op" is the operator, and "left" and "right" are any expression nodes. >>> print(ast.dump(ast.parse('x + y', mode='eval'), indent=4)) Expression( body=BinOp( left=Name(id='x', ctx=Load()), op=Add(), right=Name(id='y', ctx=Load()))) class ast.Add class ast.Sub class ast.Mult class ast.Div class ast.FloorDiv class ast.Mod class ast.Pow class ast.LShift class ast.RShift class ast.BitOr class ast.BitXor class ast.BitAnd class ast.MatMult Binary operator tokens. class ast.BoolOp(op, values) A boolean operation, ‘or’ or ‘and’. "op" is "Or" or "And". "values" are the values involved. Consecutive operations with the same operator, such as "a or b or c", are collapsed into one node with several values. This doesn’t include "not", which is a "UnaryOp". >>> print(ast.dump(ast.parse('x or y', mode='eval'), indent=4)) Expression( body=BoolOp( op=Or(), values=[ Name(id='x', ctx=Load()), Name(id='y', ctx=Load())])) class ast.And class ast.Or Boolean operator tokens. class ast.Compare(left, ops, comparators) A comparison of two or more values. "left" is the first value in the comparison, "ops" the list of operators, and "comparators" the list of values after the first element in the comparison. >>> print(ast.dump(ast.parse('1 <= a < 10', mode='eval'), indent=4)) Expression( body=Compare( left=Constant(value=1), ops=[ LtE(), Lt()], comparators=[ Name(id='a', ctx=Load()), Constant(value=10)])) class ast.Eq class ast.NotEq class ast.Lt class ast.LtE class ast.Gt class ast.GtE class ast.Is class ast.IsNot class ast.In class ast.NotIn Comparison operator tokens. class ast.Call(func, args, keywords) A function call. "func" is the function, which will often be a "Name" or "Attribute" object. Of the arguments: * "args" holds a list of the arguments passed by position. * "keywords" holds a list of "keyword" objects representing arguments passed by keyword. The "args" and "keywords" arguments are optional and default to empty lists. >>> print(ast.dump(ast.parse('func(a, b=c, *d, **e)', mode='eval'), indent=4)) Expression( body=Call( func=Name(id='func', ctx=Load()), args=[ Name(id='a', ctx=Load()), Starred( value=Name(id='d', ctx=Load()), ctx=Load())], keywords=[ keyword( arg='b', value=Name(id='c', ctx=Load())), keyword( value=Name(id='e', ctx=Load()))])) class ast.keyword(arg, value) A keyword argument to a function call or class definition. "arg" is a raw string of the parameter name, "value" is a node to pass in. class ast.IfExp(test, body, orelse) An expression such as "a if b else c". Each field holds a single node, so in the following example, all three are "Name" nodes. >>> print(ast.dump(ast.parse('a if b else c', mode='eval'), indent=4)) Expression( body=IfExp( test=Name(id='b', ctx=Load()), body=Name(id='a', ctx=Load()), orelse=Name(id='c', ctx=Load()))) class ast.Attribute(value, attr, ctx) Attribute access, e.g. "d.keys". "value" is a node, typically a "Name". "attr" is a bare string giving the name of the attribute, and "ctx" is "Load", "Store" or "Del" according to how the attribute is acted on. >>> print(ast.dump(ast.parse('snake.colour', mode='eval'), indent=4)) Expression( body=Attribute( value=Name(id='snake', ctx=Load()), attr='colour', ctx=Load())) class ast.NamedExpr(target, value) A named expression. This AST node is produced by the assignment expressions operator (also known as the walrus operator). As opposed to the "Assign" node in which the first argument can be multiple nodes, in this case both "target" and "value" must be single nodes. >>> print(ast.dump(ast.parse('(x := 4)', mode='eval'), indent=4)) Expression( body=NamedExpr( target=Name(id='x', ctx=Store()), value=Constant(value=4))) Added in version 3.8. Subscripting ~~~~~~~~~~~~ class ast.Subscript(value, slice, ctx) A subscript, such as "l[1]". "value" is the subscripted object (usually sequence or mapping). "slice" is an index, slice or key. It can be a "Tuple" and contain a "Slice". "ctx" is "Load", "Store" or "Del" according to the action performed with the subscript. >>> print(ast.dump(ast.parse('l[1:2, 3]', mode='eval'), indent=4)) Expression( body=Subscript( value=Name(id='l', ctx=Load()), slice=Tuple( elts=[ Slice( lower=Constant(value=1), upper=Constant(value=2)), Constant(value=3)], ctx=Load()), ctx=Load())) class ast.Slice(lower, upper, step) Regular slicing (on the form "lower:upper" or "lower:upper:step"). Can occur only inside the *slice* field of "Subscript", either directly or as an element of "Tuple". >>> print(ast.dump(ast.parse('l[1:2]', mode='eval'), indent=4)) Expression( body=Subscript( value=Name(id='l', ctx=Load()), slice=Slice( lower=Constant(value=1), upper=Constant(value=2)), ctx=Load())) Comprehensions ~~~~~~~~~~~~~~ class ast.ListComp(elt, generators) class ast.SetComp(elt, generators) class ast.GeneratorExp(elt, generators) class ast.DictComp(key, value, generators) List and set comprehensions, generator expressions, and dictionary comprehensions. "elt" (or "key" and "value") is a single node representing the part that will be evaluated for each item. "generators" is a list of "comprehension" nodes. >>> print(ast.dump( ... ast.parse('[x for x in numbers]', mode='eval'), ... indent=4, ... )) Expression( body=ListComp( elt=Name(id='x', ctx=Load()), generators=[ comprehension( target=Name(id='x', ctx=Store()), iter=Name(id='numbers', ctx=Load()), is_async=0)])) >>> print(ast.dump( ... ast.parse('{x: x**2 for x in numbers}', mode='eval'), ... indent=4, ... )) Expression( body=DictComp( key=Name(id='x', ctx=Load()), value=BinOp( left=Name(id='x', ctx=Load()), op=Pow(), right=Constant(value=2)), generators=[ comprehension( target=Name(id='x', ctx=Store()), iter=Name(id='numbers', ctx=Load()), is_async=0)])) >>> print(ast.dump( ... ast.parse('{x for x in numbers}', mode='eval'), ... indent=4, ... )) Expression( body=SetComp( elt=Name(id='x', ctx=Load()), generators=[ comprehension( target=Name(id='x', ctx=Store()), iter=Name(id='numbers', ctx=Load()), is_async=0)])) class ast.comprehension(target, iter, ifs, is_async) One "for" clause in a comprehension. "target" is the reference to use for each element - typically a "Name" or "Tuple" node. "iter" is the object to iterate over. "ifs" is a list of test expressions: each "for" clause can have multiple "ifs". "is_async" indicates a comprehension is asynchronous (using an "async for" instead of "for"). The value is an integer (0 or 1). >>> print(ast.dump(ast.parse('[ord(c) for line in file for c in line]', mode='eval'), ... indent=4)) # Multiple comprehensions in one. Expression( body=ListComp( elt=Call( func=Name(id='ord', ctx=Load()), args=[ Name(id='c', ctx=Load())]), generators=[ comprehension( target=Name(id='line', ctx=Store()), iter=Name(id='file', ctx=Load()), is_async=0), comprehension( target=Name(id='c', ctx=Store()), iter=Name(id='line', ctx=Load()), is_async=0)])) >>> print(ast.dump(ast.parse('(n**2 for n in it if n>5 if n<10)', mode='eval'), ... indent=4)) # generator comprehension Expression( body=GeneratorExp( elt=BinOp( left=Name(id='n', ctx=Load()), op=Pow(), right=Constant(value=2)), generators=[ comprehension( target=Name(id='n', ctx=Store()), iter=Name(id='it', ctx=Load()), ifs=[ Compare( left=Name(id='n', ctx=Load()), ops=[ Gt()], comparators=[ Constant(value=5)]), Compare( left=Name(id='n', ctx=Load()), ops=[ Lt()], comparators=[ Constant(value=10)])], is_async=0)])) >>> print(ast.dump(ast.parse('[i async for i in soc]', mode='eval'), ... indent=4)) # Async comprehension Expression( body=ListComp( elt=Name(id='i', ctx=Load()), generators=[ comprehension( target=Name(id='i', ctx=Store()), iter=Name(id='soc', ctx=Load()), is_async=1)])) Statements ---------- class ast.Assign(targets, value, type_comment) An assignment. "targets" is a list of nodes, and "value" is a single node. Multiple nodes in "targets" represents assigning the same value to each. Unpacking is represented by putting a "Tuple" or "List" within "targets". type_comment "type_comment" is an optional string with the type annotation as a comment. >>> print(ast.dump(ast.parse('a = b = 1'), indent=4)) # Multiple assignment Module( body=[ Assign( targets=[ Name(id='a', ctx=Store()), Name(id='b', ctx=Store())], value=Constant(value=1))]) >>> print(ast.dump(ast.parse('a,b = c'), indent=4)) # Unpacking Module( body=[ Assign( targets=[ Tuple( elts=[ Name(id='a', ctx=Store()), Name(id='b', ctx=Store())], ctx=Store())], value=Name(id='c', ctx=Load()))]) class ast.AnnAssign(target, annotation, value, simple) An assignment with a type annotation. "target" is a single node and can be a "Name", an "Attribute" or a "Subscript". "annotation" is the annotation, such as a "Constant" or "Name" node. "value" is a single optional node. "simple" is always either 0 (indicating a “complex” target) or 1 (indicating a “simple” target). A “simple” target consists solely of a "Name" node that does not appear between parentheses; all other targets are considered complex. Only simple targets appear in the "__annotations__" dictionary of modules and classes. >>> print(ast.dump(ast.parse('c: int'), indent=4)) Module( body=[ AnnAssign( target=Name(id='c', ctx=Store()), annotation=Name(id='int', ctx=Load()), simple=1)]) >>> print(ast.dump(ast.parse('(a): int = 1'), indent=4)) # Annotation with parenthesis Module( body=[ AnnAssign( target=Name(id='a', ctx=Store()), annotation=Name(id='int', ctx=Load()), value=Constant(value=1), simple=0)]) >>> print(ast.dump(ast.parse('a.b: int'), indent=4)) # Attribute annotation Module( body=[ AnnAssign( target=Attribute( value=Name(id='a', ctx=Load()), attr='b', ctx=Store()), annotation=Name(id='int', ctx=Load()), simple=0)]) >>> print(ast.dump(ast.parse('a[1]: int'), indent=4)) # Subscript annotation Module( body=[ AnnAssign( target=Subscript( value=Name(id='a', ctx=Load()), slice=Constant(value=1), ctx=Store()), annotation=Name(id='int', ctx=Load()), simple=0)]) class ast.AugAssign(target, op, value) Augmented assignment, such as "a += 1". In the following example, "target" is a "Name" node for "x" (with the "Store" context), "op" is "Add", and "value" is a "Constant" with value for 1. The "target" attribute cannot be of class "Tuple" or "List", unlike the targets of "Assign". >>> print(ast.dump(ast.parse('x += 2'), indent=4)) Module( body=[ AugAssign( target=Name(id='x', ctx=Store()), op=Add(), value=Constant(value=2))]) class ast.Raise(exc, cause) A "raise" statement. "exc" is the exception object to be raised, normally a "Call" or "Name", or "None" for a standalone "raise". "cause" is the optional part for "y" in "raise x from y". >>> print(ast.dump(ast.parse('raise x from y'), indent=4)) Module( body=[ Raise( exc=Name(id='x', ctx=Load()), cause=Name(id='y', ctx=Load()))]) class ast.Assert(test, msg) An assertion. "test" holds the condition, such as a "Compare" node. "msg" holds the failure message. >>> print(ast.dump(ast.parse('assert x,y'), indent=4)) Module( body=[ Assert( test=Name(id='x', ctx=Load()), msg=Name(id='y', ctx=Load()))]) class ast.Delete(targets) Represents a "del" statement. "targets" is a list of nodes, such as "Name", "Attribute" or "Subscript" nodes. >>> print(ast.dump(ast.parse('del x,y,z'), indent=4)) Module( body=[ Delete( targets=[ Name(id='x', ctx=Del()), Name(id='y', ctx=Del()), Name(id='z', ctx=Del())])]) class ast.Pass A "pass" statement. >>> print(ast.dump(ast.parse('pass'), indent=4)) Module( body=[ Pass()]) class ast.TypeAlias(name, type_params, value) A type alias created through the "type" statement. "name" is the name of the alias, "type_params" is a list of type parameters, and "value" is the value of the type alias. >>> print(ast.dump(ast.parse('type Alias = int'), indent=4)) Module( body=[ TypeAlias( name=Name(id='Alias', ctx=Store()), value=Name(id='int', ctx=Load()))]) Added in version 3.12. Other statements which are only applicable inside functions or loops are described in other sections. Imports ~~~~~~~ class ast.Import(names) An import statement. "names" is a list of "alias" nodes. >>> print(ast.dump(ast.parse('import x,y,z'), indent=4)) Module( body=[ Import( names=[ alias(name='x'), alias(name='y'), alias(name='z')])]) class ast.ImportFrom(module, names, level) Represents "from x import y". "module" is a raw string of the ‘from’ name, without any leading dots, or "None" for statements such as "from . import foo". "level" is an integer holding the level of the relative import (0 means absolute import). >>> print(ast.dump(ast.parse('from y import x,y,z'), indent=4)) Module( body=[ ImportFrom( module='y', names=[ alias(name='x'), alias(name='y'), alias(name='z')], level=0)]) class ast.alias(name, asname) Both parameters are raw strings of the names. "asname" can be "None" if the regular name is to be used. >>> print(ast.dump(ast.parse('from ..foo.bar import a as b, c'), indent=4)) Module( body=[ ImportFrom( module='foo.bar', names=[ alias(name='a', asname='b'), alias(name='c')], level=2)]) Control flow ------------ Note: Optional clauses such as "else" are stored as an empty list if they’re not present. class ast.If(test, body, orelse) An "if" statement. "test" holds a single node, such as a "Compare" node. "body" and "orelse" each hold a list of nodes. "elif" clauses don’t have a special representation in the AST, but rather appear as extra "If" nodes within the "orelse" section of the previous one. >>> print(ast.dump(ast.parse(""" ... if x: ... ... ... elif y: ... ... ... else: ... ... ... """), indent=4)) Module( body=[ If( test=Name(id='x', ctx=Load()), body=[ Expr( value=Constant(value=Ellipsis))], orelse=[ If( test=Name(id='y', ctx=Load()), body=[ Expr( value=Constant(value=Ellipsis))], orelse=[ Expr( value=Constant(value=Ellipsis))])])]) class ast.For(target, iter, body, orelse, type_comment) A "for" loop. "target" holds the variable(s) the loop assigns to, as a single "Name", "Tuple", "List", "Attribute" or "Subscript" node. "iter" holds the item to be looped over, again as a single node. "body" and "orelse" contain lists of nodes to execute. Those in "orelse" are executed if the loop finishes normally, rather than via a "break" statement. type_comment "type_comment" is an optional string with the type annotation as a comment. >>> print(ast.dump(ast.parse(""" ... for x in y: ... ... ... else: ... ... ... """), indent=4)) Module( body=[ For( target=Name(id='x', ctx=Store()), iter=Name(id='y', ctx=Load()), body=[ Expr( value=Constant(value=Ellipsis))], orelse=[ Expr( value=Constant(value=Ellipsis))])]) class ast.While(test, body, orelse) A "while" loop. "test" holds the condition, such as a "Compare" node. >>> print(ast.dump(ast.parse(""" ... while x: ... ... ... else: ... ... ... """), indent=4)) Module( body=[ While( test=Name(id='x', ctx=Load()), body=[ Expr( value=Constant(value=Ellipsis))], orelse=[ Expr( value=Constant(value=Ellipsis))])]) class ast.Break class ast.Continue The "break" and "continue" statements. >>> print(ast.dump(ast.parse("""\ ... for a in b: ... if a > 5: ... break ... else: ... continue ... ... """), indent=4)) Module( body=[ For( target=Name(id='a', ctx=Store()), iter=Name(id='b', ctx=Load()), body=[ If( test=Compare( left=Name(id='a', ctx=Load()), ops=[ Gt()], comparators=[ Constant(value=5)]), body=[ Break()], orelse=[ Continue()])])]) class ast.Try(body, handlers, orelse, finalbody) "try" blocks. All attributes are list of nodes to execute, except for "handlers", which is a list of "ExceptHandler" nodes. >>> print(ast.dump(ast.parse(""" ... try: ... ... ... except Exception: ... ... ... except OtherException as e: ... ... ... else: ... ... ... finally: ... ... ... """), indent=4)) Module( body=[ Try( body=[ Expr( value=Constant(value=Ellipsis))], handlers=[ ExceptHandler( type=Name(id='Exception', ctx=Load()), body=[ Expr( value=Constant(value=Ellipsis))]), ExceptHandler( type=Name(id='OtherException', ctx=Load()), name='e', body=[ Expr( value=Constant(value=Ellipsis))])], orelse=[ Expr( value=Constant(value=Ellipsis))], finalbody=[ Expr( value=Constant(value=Ellipsis))])]) class ast.TryStar(body, handlers, orelse, finalbody) "try" blocks which are followed by "except*" clauses. The attributes are the same as for "Try" but the "ExceptHandler" nodes in "handlers" are interpreted as "except*" blocks rather then "except". >>> print(ast.dump(ast.parse(""" ... try: ... ... ... except* Exception: ... ... ... """), indent=4)) Module( body=[ TryStar( body=[ Expr( value=Constant(value=Ellipsis))], handlers=[ ExceptHandler( type=Name(id='Exception', ctx=Load()), body=[ Expr( value=Constant(value=Ellipsis))])])]) Added in version 3.11. class ast.ExceptHandler(type, name, body) A single "except" clause. "type" is the exception type it will match, typically a "Name" node (or "None" for a catch-all "except:" clause). "name" is a raw string for the name to hold the exception, or "None" if the clause doesn’t have "as foo". "body" is a list of nodes. >>> print(ast.dump(ast.parse("""\ ... try: ... a + 1 ... except TypeError: ... pass ... """), indent=4)) Module( body=[ Try( body=[ Expr( value=BinOp( left=Name(id='a', ctx=Load()), op=Add(), right=Constant(value=1)))], handlers=[ ExceptHandler( type=Name(id='TypeError', ctx=Load()), body=[ Pass()])])]) class ast.With(items, body, type_comment) A "with" block. "items" is a list of "withitem" nodes representing the context managers, and "body" is the indented block inside the context. type_comment "type_comment" is an optional string with the type annotation as a comment. class ast.withitem(context_expr, optional_vars) A single context manager in a "with" block. "context_expr" is the context manager, often a "Call" node. "optional_vars" is a "Name", "Tuple" or "List" for the "as foo" part, or "None" if that isn’t used. >>> print(ast.dump(ast.parse("""\ ... with a as b, c as d: ... something(b, d) ... """), indent=4)) Module( body=[ With( items=[ withitem( context_expr=Name(id='a', ctx=Load()), optional_vars=Name(id='b', ctx=Store())), withitem( context_expr=Name(id='c', ctx=Load()), optional_vars=Name(id='d', ctx=Store()))], body=[ Expr( value=Call( func=Name(id='something', ctx=Load()), args=[ Name(id='b', ctx=Load()), Name(id='d', ctx=Load())]))])]) Pattern matching ---------------- class ast.Match(subject, cases) A "match" statement. "subject" holds the subject of the match (the object that is being matched against the cases) and "cases" contains an iterable of "match_case" nodes with the different cases. Added in version 3.10. class ast.match_case(pattern, guard, body) A single case pattern in a "match" statement. "pattern" contains the match pattern that the subject will be matched against. Note that the "AST" nodes produced for patterns differ from those produced for expressions, even when they share the same syntax. The "guard" attribute contains an expression that will be evaluated if the pattern matches the subject. "body" contains a list of nodes to execute if the pattern matches and the result of evaluating the guard expression is true. >>> print(ast.dump(ast.parse(""" ... match x: ... case [x] if x>0: ... ... ... case tuple(): ... ... ... """), indent=4)) Module( body=[ Match( subject=Name(id='x', ctx=Load()), cases=[ match_case( pattern=MatchSequence( patterns=[ MatchAs(name='x')]), guard=Compare( left=Name(id='x', ctx=Load()), ops=[ Gt()], comparators=[ Constant(value=0)]), body=[ Expr( value=Constant(value=Ellipsis))]), match_case( pattern=MatchClass( cls=Name(id='tuple', ctx=Load())), body=[ Expr( value=Constant(value=Ellipsis))])])]) Added in version 3.10. class ast.MatchValue(value) A match literal or value pattern that compares by equality. "value" is an expression node. Permitted value nodes are restricted as described in the match statement documentation. This pattern succeeds if the match subject is equal to the evaluated value. >>> print(ast.dump(ast.parse(""" ... match x: ... case "Relevant": ... ... ... """), indent=4)) Module( body=[ Match( subject=Name(id='x', ctx=Load()), cases=[ match_case( pattern=MatchValue( value=Constant(value='Relevant')), body=[ Expr( value=Constant(value=Ellipsis))])])]) Added in version 3.10. class ast.MatchSingleton(value) A match literal pattern that compares by identity. "value" is the singleton to be compared against: "None", "True", or "False". This pattern succeeds if the match subject is the given constant. >>> print(ast.dump(ast.parse(""" ... match x: ... case None: ... ... ... """), indent=4)) Module( body=[ Match( subject=Name(id='x', ctx=Load()), cases=[ match_case( pattern=MatchSingleton(value=None), body=[ Expr( value=Constant(value=Ellipsis))])])]) Added in version 3.10. class ast.MatchSequence(patterns) A match sequence pattern. "patterns" contains the patterns to be matched against the subject elements if the subject is a sequence. Matches a variable length sequence if one of the subpatterns is a "MatchStar" node, otherwise matches a fixed length sequence. >>> print(ast.dump(ast.parse(""" ... match x: ... case [1, 2]: ... ... ... """), indent=4)) Module( body=[ Match( subject=Name(id='x', ctx=Load()), cases=[ match_case( pattern=MatchSequence( patterns=[ MatchValue( value=Constant(value=1)), MatchValue( value=Constant(value=2))]), body=[ Expr( value=Constant(value=Ellipsis))])])]) Added in version 3.10. class ast.MatchStar(name) Matches the rest of the sequence in a variable length match sequence pattern. If "name" is not "None", a list containing the remaining sequence elements is bound to that name if the overall sequence pattern is successful. >>> print(ast.dump(ast.parse(""" ... match x: ... case [1, 2, *rest]: ... ... ... case [*_]: ... ... ... """), indent=4)) Module( body=[ Match( subject=Name(id='x', ctx=Load()), cases=[ match_case( pattern=MatchSequence( patterns=[ MatchValue( value=Constant(value=1)), MatchValue( value=Constant(value=2)), MatchStar(name='rest')]), body=[ Expr( value=Constant(value=Ellipsis))]), match_case( pattern=MatchSequence( patterns=[ MatchStar()]), body=[ Expr( value=Constant(value=Ellipsis))])])]) Added in version 3.10. class ast.MatchMapping(keys, patterns, rest) A match mapping pattern. "keys" is a sequence of expression nodes. "patterns" is a corresponding sequence of pattern nodes. "rest" is an optional name that can be specified to capture the remaining mapping elements. Permitted key expressions are restricted as described in the match statement documentation. This pattern succeeds if the subject is a mapping, all evaluated key expressions are present in the mapping, and the value corresponding to each key matches the corresponding subpattern. If "rest" is not "None", a dict containing the remaining mapping elements is bound to that name if the overall mapping pattern is successful. >>> print(ast.dump(ast.parse(""" ... match x: ... case {1: _, 2: _}: ... ... ... case {**rest}: ... ... ... """), indent=4)) Module( body=[ Match( subject=Name(id='x', ctx=Load()), cases=[ match_case( pattern=MatchMapping( keys=[ Constant(value=1), Constant(value=2)], patterns=[ MatchAs(), MatchAs()]), body=[ Expr( value=Constant(value=Ellipsis))]), match_case( pattern=MatchMapping(rest='rest'), body=[ Expr( value=Constant(value=Ellipsis))])])]) Added in version 3.10. class ast.MatchClass(cls, patterns, kwd_attrs, kwd_patterns) A match class pattern. "cls" is an expression giving the nominal class to be matched. "patterns" is a sequence of pattern nodes to be matched against the class defined sequence of pattern matching attributes. "kwd_attrs" is a sequence of additional attributes to be matched (specified as keyword arguments in the class pattern), "kwd_patterns" are the corresponding patterns (specified as keyword values in the class pattern). This pattern succeeds if the subject is an instance of the nominated class, all positional patterns match the corresponding class-defined attributes, and any specified keyword attributes match their corresponding pattern. Note: classes may define a property that returns self in order to match a pattern node against the instance being matched. Several builtin types are also matched that way, as described in the match statement documentation. >>> print(ast.dump(ast.parse(""" ... match x: ... case Point2D(0, 0): ... ... ... case Point3D(x=0, y=0, z=0): ... ... ... """), indent=4)) Module( body=[ Match( subject=Name(id='x', ctx=Load()), cases=[ match_case( pattern=MatchClass( cls=Name(id='Point2D', ctx=Load()), patterns=[ MatchValue( value=Constant(value=0)), MatchValue( value=Constant(value=0))]), body=[ Expr( value=Constant(value=Ellipsis))]), match_case( pattern=MatchClass( cls=Name(id='Point3D', ctx=Load()), kwd_attrs=[ 'x', 'y', 'z'], kwd_patterns=[ MatchValue( value=Constant(value=0)), MatchValue( value=Constant(value=0)), MatchValue( value=Constant(value=0))]), body=[ Expr( value=Constant(value=Ellipsis))])])]) Added in version 3.10. class ast.MatchAs(pattern, name) A match “as-pattern”, capture pattern or wildcard pattern. "pattern" contains the match pattern that the subject will be matched against. If the pattern is "None", the node represents a capture pattern (i.e a bare name) and will always succeed. The "name" attribute contains the name that will be bound if the pattern is successful. If "name" is "None", "pattern" must also be "None" and the node represents the wildcard pattern. >>> print(ast.dump(ast.parse(""" ... match x: ... case [x] as y: ... ... ... case _: ... ... ... """), indent=4)) Module( body=[ Match( subject=Name(id='x', ctx=Load()), cases=[ match_case( pattern=MatchAs( pattern=MatchSequence( patterns=[ MatchAs(name='x')]), name='y'), body=[ Expr( value=Constant(value=Ellipsis))]), match_case( pattern=MatchAs(), body=[ Expr( value=Constant(value=Ellipsis))])])]) Added in version 3.10. class ast.MatchOr(patterns) A match “or-pattern”. An or-pattern matches each of its subpatterns in turn to the subject, until one succeeds. The or-pattern is then deemed to succeed. If none of the subpatterns succeed the or- pattern fails. The "patterns" attribute contains a list of match pattern nodes that will be matched against the subject. >>> print(ast.dump(ast.parse(""" ... match x: ... case [x] | (y): ... ... ... """), indent=4)) Module( body=[ Match( subject=Name(id='x', ctx=Load()), cases=[ match_case( pattern=MatchOr( patterns=[ MatchSequence( patterns=[ MatchAs(name='x')]), MatchAs(name='y')]), body=[ Expr( value=Constant(value=Ellipsis))])])]) Added in version 3.10. Type annotations ---------------- class ast.TypeIgnore(lineno, tag) A "# type: ignore" comment located at *lineno*. *tag* is the optional tag specified by the form "# type: ignore ". >>> print(ast.dump(ast.parse('x = 1 # type: ignore', type_comments=True), indent=4)) Module( body=[ Assign( targets=[ Name(id='x', ctx=Store())], value=Constant(value=1))], type_ignores=[ TypeIgnore(lineno=1, tag='')]) >>> print(ast.dump(ast.parse('x: bool = 1 # type: ignore[assignment]', type_comments=True), indent=4)) Module( body=[ AnnAssign( target=Name(id='x', ctx=Store()), annotation=Name(id='bool', ctx=Load()), value=Constant(value=1), simple=1)], type_ignores=[ TypeIgnore(lineno=1, tag='[assignment]')]) Note: "TypeIgnore" nodes are not generated when the *type_comments* parameter is set to "False" (default). See "ast.parse()" for more details. Added in version 3.8. Type parameters --------------- Type parameters can exist on classes, functions, and type aliases. class ast.TypeVar(name, bound, default_value) A "typing.TypeVar". "name" is the name of the type variable. "bound" is the bound or constraints, if any. If "bound" is a "Tuple", it represents constraints; otherwise it represents the bound. "default_value" is the default value; if the "TypeVar" has no default, this attribute will be set to "None". >>> print(ast.dump(ast.parse("type Alias[T: int = bool] = list[T]"), indent=4)) Module( body=[ TypeAlias( name=Name(id='Alias', ctx=Store()), type_params=[ TypeVar( name='T', bound=Name(id='int', ctx=Load()), default_value=Name(id='bool', ctx=Load()))], value=Subscript( value=Name(id='list', ctx=Load()), slice=Name(id='T', ctx=Load()), ctx=Load()))]) Added in version 3.12. Changed in version 3.13: Added the *default_value* parameter. class ast.ParamSpec(name, default_value) A "typing.ParamSpec". "name" is the name of the parameter specification. "default_value" is the default value; if the "ParamSpec" has no default, this attribute will be set to "None". >>> print(ast.dump(ast.parse("type Alias[**P = [int, str]] = Callable[P, int]"), indent=4)) Module( body=[ TypeAlias( name=Name(id='Alias', ctx=Store()), type_params=[ ParamSpec( name='P', default_value=List( elts=[ Name(id='int', ctx=Load()), Name(id='str', ctx=Load())], ctx=Load()))], value=Subscript( value=Name(id='Callable', ctx=Load()), slice=Tuple( elts=[ Name(id='P', ctx=Load()), Name(id='int', ctx=Load())], ctx=Load()), ctx=Load()))]) Added in version 3.12. Changed in version 3.13: Added the *default_value* parameter. class ast.TypeVarTuple(name, default_value) A "typing.TypeVarTuple". "name" is the name of the type variable tuple. "default_value" is the default value; if the "TypeVarTuple" has no default, this attribute will be set to "None". >>> print(ast.dump(ast.parse("type Alias[*Ts = ()] = tuple[*Ts]"), indent=4)) Module( body=[ TypeAlias( name=Name(id='Alias', ctx=Store()), type_params=[ TypeVarTuple( name='Ts', default_value=Tuple(ctx=Load()))], value=Subscript( value=Name(id='tuple', ctx=Load()), slice=Tuple( elts=[ Starred( value=Name(id='Ts', ctx=Load()), ctx=Load())], ctx=Load()), ctx=Load()))]) Added in version 3.12. Changed in version 3.13: Added the *default_value* parameter. Function and class definitions ------------------------------ class ast.FunctionDef(name, args, body, decorator_list, returns, type_comment, type_params) A function definition. * "name" is a raw string of the function name. * "args" is an "arguments" node. * "body" is the list of nodes inside the function. * "decorator_list" is the list of decorators to be applied, stored outermost first (i.e. the first in the list will be applied last). * "returns" is the return annotation. * "type_params" is a list of type parameters. type_comment "type_comment" is an optional string with the type annotation as a comment. Changed in version 3.12: Added "type_params". class ast.Lambda(args, body) "lambda" is a minimal function definition that can be used inside an expression. Unlike "FunctionDef", "body" holds a single node. >>> print(ast.dump(ast.parse('lambda x,y: ...'), indent=4)) Module( body=[ Expr( value=Lambda( args=arguments( args=[ arg(arg='x'), arg(arg='y')]), body=Constant(value=Ellipsis)))]) class ast.arguments(posonlyargs, args, vararg, kwonlyargs, kw_defaults, kwarg, defaults) The arguments for a function. * "posonlyargs", "args" and "kwonlyargs" are lists of "arg" nodes. * "vararg" and "kwarg" are single "arg" nodes, referring to the "*args, **kwargs" parameters. * "kw_defaults" is a list of default values for keyword-only arguments. If one is "None", the corresponding argument is required. * "defaults" is a list of default values for arguments that can be passed positionally. If there are fewer defaults, they correspond to the last n arguments. class ast.arg(arg, annotation, type_comment) A single argument in a list. "arg" is a raw string of the argument name; "annotation" is its annotation, such as a "Name" node. type_comment "type_comment" is an optional string with the type annotation as a comment >>> print(ast.dump(ast.parse("""\ ... @decorator1 ... @decorator2 ... def f(a: 'annotation', b=1, c=2, *d, e, f=3, **g) -> 'return annotation': ... pass ... """), indent=4)) Module( body=[ FunctionDef( name='f', args=arguments( args=[ arg( arg='a', annotation=Constant(value='annotation')), arg(arg='b'), arg(arg='c')], vararg=arg(arg='d'), kwonlyargs=[ arg(arg='e'), arg(arg='f')], kw_defaults=[ None, Constant(value=3)], kwarg=arg(arg='g'), defaults=[ Constant(value=1), Constant(value=2)]), body=[ Pass()], decorator_list=[ Name(id='decorator1', ctx=Load()), Name(id='decorator2', ctx=Load())], returns=Constant(value='return annotation'))]) class ast.Return(value) A "return" statement. >>> print(ast.dump(ast.parse('return 4'), indent=4)) Module( body=[ Return( value=Constant(value=4))]) class ast.Yield(value) class ast.YieldFrom(value) A "yield" or "yield from" expression. Because these are expressions, they must be wrapped in an "Expr" node if the value sent back is not used. >>> print(ast.dump(ast.parse('yield x'), indent=4)) Module( body=[ Expr( value=Yield( value=Name(id='x', ctx=Load())))]) >>> print(ast.dump(ast.parse('yield from x'), indent=4)) Module( body=[ Expr( value=YieldFrom( value=Name(id='x', ctx=Load())))]) class ast.Global(names) class ast.Nonlocal(names) "global" and "nonlocal" statements. "names" is a list of raw strings. >>> print(ast.dump(ast.parse('global x,y,z'), indent=4)) Module( body=[ Global( names=[ 'x', 'y', 'z'])]) >>> print(ast.dump(ast.parse('nonlocal x,y,z'), indent=4)) Module( body=[ Nonlocal( names=[ 'x', 'y', 'z'])]) class ast.ClassDef(name, bases, keywords, body, decorator_list, type_params) A class definition. * "name" is a raw string for the class name * "bases" is a list of nodes for explicitly specified base classes. * "keywords" is a list of "keyword" nodes, principally for ‘metaclass’. Other keywords will be passed to the metaclass, as per **PEP 3115**. * "body" is a list of nodes representing the code within the class definition. * "decorator_list" is a list of nodes, as in "FunctionDef". * "type_params" is a list of type parameters. >>> print(ast.dump(ast.parse("""\ ... @decorator1 ... @decorator2 ... class Foo(base1, base2, metaclass=meta): ... pass ... """), indent=4)) Module( body=[ ClassDef( name='Foo', bases=[ Name(id='base1', ctx=Load()), Name(id='base2', ctx=Load())], keywords=[ keyword( arg='metaclass', value=Name(id='meta', ctx=Load()))], body=[ Pass()], decorator_list=[ Name(id='decorator1', ctx=Load()), Name(id='decorator2', ctx=Load())])]) Changed in version 3.12: Added "type_params". Async and await --------------- class ast.AsyncFunctionDef(name, args, body, decorator_list, returns, type_comment, type_params) An "async def" function definition. Has the same fields as "FunctionDef". Changed in version 3.12: Added "type_params". class ast.Await(value) An "await" expression. "value" is what it waits for. Only valid in the body of an "AsyncFunctionDef". >>> print(ast.dump(ast.parse("""\ ... async def f(): ... await other_func() ... """), indent=4)) Module( body=[ AsyncFunctionDef( name='f', args=arguments(), body=[ Expr( value=Await( value=Call( func=Name(id='other_func', ctx=Load()))))])]) class ast.AsyncFor(target, iter, body, orelse, type_comment) class ast.AsyncWith(items, body, type_comment) "async for" loops and "async with" context managers. They have the same fields as "For" and "With", respectively. Only valid in the body of an "AsyncFunctionDef". Note: When a string is parsed by "ast.parse()", operator nodes (subclasses of "ast.operator", "ast.unaryop", "ast.cmpop", "ast.boolop" and "ast.expr_context") on the returned tree will be singletons. Changes to one will be reflected in all other occurrences of the same value (e.g. "ast.Add"). "ast" Helpers ============= Apart from the node classes, the "ast" module defines these utility functions and classes for traversing abstract syntax trees: ast.parse(source, filename='', mode='exec', *, type_comments=False, feature_version=None, optimize=-1) Parse the source into an AST node. Equivalent to "compile(source, filename, mode, flags=FLAGS_VALUE, optimize=optimize)", where "FLAGS_VALUE" is "ast.PyCF_ONLY_AST" if "optimize <= 0" and "ast.PyCF_OPTIMIZED_AST" otherwise. If "type_comments=True" is given, the parser is modified to check and return type comments as specified by **PEP 484** and **PEP 526**. This is equivalent to adding "ast.PyCF_TYPE_COMMENTS" to the flags passed to "compile()". This will report syntax errors for misplaced type comments. Without this flag, type comments will be ignored, and the "type_comment" field on selected AST nodes will always be "None". In addition, the locations of "# type: ignore" comments will be returned as the "type_ignores" attribute of "Module" (otherwise it is always an empty list). In addition, if "mode" is "'func_type'", the input syntax is modified to correspond to **PEP 484** “signature type comments”, e.g. "(str, int) -> List[str]". Setting "feature_version" to a tuple "(major, minor)" will result in a “best-effort” attempt to parse using that Python version’s grammar. For example, setting "feature_version=(3, 9)" will attempt to disallow parsing of "match" statements. Currently "major" must equal to "3". The lowest supported version is "(3, 7)" (and this may increase in future Python versions); the highest is "sys.version_info[0:2]". “Best-effort” attempt means there is no guarantee that the parse (or success of the parse) is the same as when run on the Python version corresponding to "feature_version". If source contains a null character ("\0"), "ValueError" is raised. Warning: Note that successfully parsing source code into an AST object doesn’t guarantee that the source code provided is valid Python code that can be executed as the compilation step can raise further "SyntaxError" exceptions. For instance, the source "return 42" generates a valid AST node for a return statement, but it cannot be compiled alone (it needs to be inside a function node).In particular, "ast.parse()" won’t do any scoping checks, which the compilation step does. Warning: It is possible to crash the Python interpreter with a sufficiently large/complex string due to stack depth limitations in Python’s AST compiler. Changed in version 3.8: Added "type_comments", "mode='func_type'" and "feature_version". Changed in version 3.13: The minimum supported version for "feature_version" is now "(3, 7)". The "optimize" argument was added. ast.unparse(ast_obj) Unparse an "ast.AST" object and generate a string with code that would produce an equivalent "ast.AST" object if parsed back with "ast.parse()". Warning: The produced code string will not necessarily be equal to the original code that generated the "ast.AST" object (without any compiler optimizations, such as constant tuples/frozensets). Warning: Trying to unparse a highly complex expression would result with "RecursionError". Added in version 3.9. ast.literal_eval(node_or_string) Evaluate an expression node or a string containing only a Python literal or container display. The string or node provided may only consist of the following Python literal structures: strings, bytes, numbers, tuples, lists, dicts, sets, booleans, "None" and "Ellipsis". This can be used for evaluating strings containing Python values without the need to parse the values oneself. It is not capable of evaluating arbitrarily complex expressions, for example involving operators or indexing. This function had been documented as “safe” in the past without defining what that meant. That was misleading. This is specifically designed not to execute Python code, unlike the more general "eval()". There is no namespace, no name lookups, or ability to call out. But it is not free from attack: A relatively small input can lead to memory exhaustion or to C stack exhaustion, crashing the process. There is also the possibility for excessive CPU consumption denial of service on some inputs. Calling it on untrusted data is thus not recommended. Warning: It is possible to crash the Python interpreter due to stack depth limitations in Python’s AST compiler.It can raise "ValueError", "TypeError", "SyntaxError", "MemoryError" and "RecursionError" depending on the malformed input. Changed in version 3.2: Now allows bytes and set literals. Changed in version 3.9: Now supports creating empty sets with "'set()'". Changed in version 3.10: For string inputs, leading spaces and tabs are now stripped. ast.get_docstring(node, clean=True) Return the docstring of the given *node* (which must be a "FunctionDef", "AsyncFunctionDef", "ClassDef", or "Module" node), or "None" if it has no docstring. If *clean* is true, clean up the docstring’s indentation with "inspect.cleandoc()". Changed in version 3.5: "AsyncFunctionDef" is now supported. ast.get_source_segment(source, node, *, padded=False) Get source code segment of the *source* that generated *node*. If some location information ("lineno", "end_lineno", "col_offset", or "end_col_offset") is missing, return "None". If *padded* is "True", the first line of a multi-line statement will be padded with spaces to match its original position. Added in version 3.8. ast.fix_missing_locations(node) When you compile a node tree with "compile()", the compiler expects "lineno" and "col_offset" attributes for every node that supports them. This is rather tedious to fill in for generated nodes, so this helper adds these attributes recursively where not already set, by setting them to the values of the parent node. It works recursively starting at *node*. ast.increment_lineno(node, n=1) Increment the line number and end line number of each node in the tree starting at *node* by *n*. This is useful to “move code” to a different location in a file. ast.copy_location(new_node, old_node) Copy source location ("lineno", "col_offset", "end_lineno", and "end_col_offset") from *old_node* to *new_node* if possible, and return *new_node*. ast.iter_fields(node) Yield a tuple of "(fieldname, value)" for each field in "node._fields" that is present on *node*. ast.iter_child_nodes(node) Yield all direct child nodes of *node*, that is, all fields that are nodes and all items of fields that are lists of nodes. ast.walk(node) Recursively yield all descendant nodes in the tree starting at *node* (including *node* itself), in no specified order. This is useful if you only want to modify nodes in place and don’t care about the context. class ast.NodeVisitor A node visitor base class that walks the abstract syntax tree and calls a visitor function for every node found. This function may return a value which is forwarded by the "visit()" method. This class is meant to be subclassed, with the subclass adding visitor methods. visit(node) Visit a node. The default implementation calls the method called "self.visit_*classname*" where *classname* is the name of the node class, or "generic_visit()" if that method doesn’t exist. generic_visit(node) This visitor calls "visit()" on all children of the node. Note that child nodes of nodes that have a custom visitor method won’t be visited unless the visitor calls "generic_visit()" or visits them itself. visit_Constant(node) Handles all constant nodes. Don’t use the "NodeVisitor" if you want to apply changes to nodes during traversal. For this a special visitor exists ("NodeTransformer") that allows modifications. Deprecated since version 3.8: Methods "visit_Num()", "visit_Str()", "visit_Bytes()", "visit_NameConstant()" and "visit_Ellipsis()" are deprecated now and will not be called in future Python versions. Add the "visit_Constant()" method to handle all constant nodes. class ast.NodeTransformer A "NodeVisitor" subclass that walks the abstract syntax tree and allows modification of nodes. The "NodeTransformer" will walk the AST and use the return value of the visitor methods to replace or remove the old node. If the return value of the visitor method is "None", the node will be removed from its location, otherwise it is replaced with the return value. The return value may be the original node in which case no replacement takes place. Here is an example transformer that rewrites all occurrences of name lookups ("foo") to "data['foo']": class RewriteName(NodeTransformer): def visit_Name(self, node): return Subscript( value=Name(id='data', ctx=Load()), slice=Constant(value=node.id), ctx=node.ctx ) Keep in mind that if the node you’re operating on has child nodes you must either transform the child nodes yourself or call the "generic_visit()" method for the node first. For nodes that were part of a collection of statements (that applies to all statement nodes), the visitor may also return a list of nodes rather than just a single node. If "NodeTransformer" introduces new nodes (that weren’t part of original tree) without giving them location information (such as "lineno"), "fix_missing_locations()" should be called with the new sub-tree to recalculate the location information: tree = ast.parse('foo', mode='eval') new_tree = fix_missing_locations(RewriteName().visit(tree)) Usually you use the transformer like this: node = YourTransformer().visit(node) ast.dump(node, annotate_fields=True, include_attributes=False, *, indent=None, show_empty=False) Return a formatted dump of the tree in *node*. This is mainly useful for debugging purposes. If *annotate_fields* is true (by default), the returned string will show the names and the values for fields. If *annotate_fields* is false, the result string will be more compact by omitting unambiguous field names. Attributes such as line numbers and column offsets are not dumped by default. If this is wanted, *include_attributes* can be set to true. If *indent* is a non-negative integer or string, then the tree will be pretty-printed with that indent level. An indent level of 0, negative, or """" will only insert newlines. "None" (the default) selects the single line representation. Using a positive integer indent indents that many spaces per level. If *indent* is a string (such as ""\t""), that string is used to indent each level. If *show_empty* is false (the default), optional empty lists will be omitted from the output. Optional "None" values are always omitted. Changed in version 3.9: Added the *indent* option. Changed in version 3.13: Added the *show_empty* option. >>> print(ast.dump(ast.parse("""\ ... async def f(): ... await other_func() ... """), indent=4, show_empty=True)) Module( body=[ AsyncFunctionDef( name='f', args=arguments( posonlyargs=[], args=[], kwonlyargs=[], kw_defaults=[], defaults=[]), body=[ Expr( value=Await( value=Call( func=Name(id='other_func', ctx=Load()), args=[], keywords=[])))], decorator_list=[], type_params=[])], type_ignores=[]) Compiler Flags ============== The following flags may be passed to "compile()" in order to change effects on the compilation of a program: ast.PyCF_ALLOW_TOP_LEVEL_AWAIT Enables support for top-level "await", "async for", "async with" and async comprehensions. Added in version 3.8. ast.PyCF_ONLY_AST Generates and returns an abstract syntax tree instead of returning a compiled code object. ast.PyCF_OPTIMIZED_AST The returned AST is optimized according to the *optimize* argument in "compile()" or "ast.parse()". Added in version 3.13. ast.PyCF_TYPE_COMMENTS Enables support for **PEP 484** and **PEP 526** style type comments ("# type: ", "# type: ignore "). Added in version 3.8. Command-Line Usage ================== Added in version 3.9. The "ast" module can be executed as a script from the command line. It is as simple as: python -m ast [-m ] [-a] [infile] The following options are accepted: -h, --help Show the help message and exit. -m --mode Specify what kind of code must be compiled, like the *mode* argument in "parse()". --no-type-comments Don’t parse type comments. -a, --include-attributes Include attributes such as line numbers and column offsets. -i --indent Indentation of nodes in AST (number of spaces). If "infile" is specified its contents are parsed to AST and dumped to stdout. Otherwise, the content is read from stdin. See also: Green Tree Snakes, an external documentation resource, has good details on working with Python ASTs. ASTTokens annotates Python ASTs with the positions of tokens and text in the source code that generated them. This is helpful for tools that make source code transformations. leoAst.py unifies the token-based and parse-tree-based views of python programs by inserting two-way links between tokens and ast nodes. LibCST parses code as a Concrete Syntax Tree that looks like an ast tree and keeps all formatting details. It’s useful for building automated refactoring (codemod) applications and linters. Parso is a Python parser that supports error recovery and round-trip parsing for different Python versions (in multiple Python versions). Parso is also able to list multiple syntax errors in your Python file. "asynchat" — Asynchronous socket command/response handler ********************************************************* Deprecated since version 3.6, removed in version 3.12. This module is no longer part of the Python standard library. It was removed in Python 3.12 after being deprecated in Python 3.6. The removal was decided in **PEP 594**. Applications should use the "asyncio" module instead. The last version of Python that provided the "asynchat" module was Python 3.11. High-level API Index ******************** This page lists all high-level async/await enabled asyncio APIs. Tasks ===== Utilities to run asyncio programs, create Tasks, and await on multiple things with timeouts. +----------------------------------------------------+----------------------------------------------------+ | "run()" | Create event loop, run a coroutine, close the | | | loop. | +----------------------------------------------------+----------------------------------------------------+ | "Runner" | A context manager that simplifies multiple async | | | function calls. | +----------------------------------------------------+----------------------------------------------------+ | "Task" | Task object. | +----------------------------------------------------+----------------------------------------------------+ | "TaskGroup" | A context manager that holds a group of tasks. | | | Provides a convenient and reliable way to wait for | | | all tasks in the group to finish. | +----------------------------------------------------+----------------------------------------------------+ | "create_task()" | Start an asyncio Task, then returns it. | +----------------------------------------------------+----------------------------------------------------+ | "current_task()" | Return the current Task. | +----------------------------------------------------+----------------------------------------------------+ | "all_tasks()" | Return all tasks that are not yet finished for an | | | event loop. | +----------------------------------------------------+----------------------------------------------------+ | "await" "sleep()" | Sleep for a number of seconds. | +----------------------------------------------------+----------------------------------------------------+ | "await" "gather()" | Schedule and wait for things concurrently. | +----------------------------------------------------+----------------------------------------------------+ | "await" "wait_for()" | Run with a timeout. | +----------------------------------------------------+----------------------------------------------------+ | "await" "shield()" | Shield from cancellation. | +----------------------------------------------------+----------------------------------------------------+ | "await" "wait()" | Monitor for completion. | +----------------------------------------------------+----------------------------------------------------+ | "timeout()" | Run with a timeout. Useful in cases when | | | "wait_for" is not suitable. | +----------------------------------------------------+----------------------------------------------------+ | "to_thread()" | Asynchronously run a function in a separate OS | | | thread. | +----------------------------------------------------+----------------------------------------------------+ | "run_coroutine_threadsafe()" | Schedule a coroutine from another OS thread. | +----------------------------------------------------+----------------------------------------------------+ | "for in" "as_completed()" | Monitor for completion with a "for" loop. | +----------------------------------------------------+----------------------------------------------------+ -[ Examples ]- * Using asyncio.gather() to run things in parallel. * Using asyncio.wait_for() to enforce a timeout. * Cancellation. * Using asyncio.sleep(). * See also the main Tasks documentation page. Queues ====== Queues should be used to distribute work amongst multiple asyncio Tasks, implement connection pools, and pub/sub patterns. +----------------------------------------------------+----------------------------------------------------+ | "Queue" | A FIFO queue. | +----------------------------------------------------+----------------------------------------------------+ | "PriorityQueue" | A priority queue. | +----------------------------------------------------+----------------------------------------------------+ | "LifoQueue" | A LIFO queue. | +----------------------------------------------------+----------------------------------------------------+ -[ Examples ]- * Using asyncio.Queue to distribute workload between several Tasks. * See also the Queues documentation page. Subprocesses ============ Utilities to spawn subprocesses and run shell commands. +----------------------------------------------------+----------------------------------------------------+ | "await" "create_subprocess_exec()" | Create a subprocess. | +----------------------------------------------------+----------------------------------------------------+ | "await" "create_subprocess_shell()" | Run a shell command. | +----------------------------------------------------+----------------------------------------------------+ -[ Examples ]- * Executing a shell command. * See also the subprocess APIs documentation. Streams ======= High-level APIs to work with network IO. +----------------------------------------------------+----------------------------------------------------+ | "await" "open_connection()" | Establish a TCP connection. | +----------------------------------------------------+----------------------------------------------------+ | "await" "open_unix_connection()" | Establish a Unix socket connection. | +----------------------------------------------------+----------------------------------------------------+ | "await" "start_server()" | Start a TCP server. | +----------------------------------------------------+----------------------------------------------------+ | "await" "start_unix_server()" | Start a Unix socket server. | +----------------------------------------------------+----------------------------------------------------+ | "StreamReader" | High-level async/await object to receive network | | | data. | +----------------------------------------------------+----------------------------------------------------+ | "StreamWriter" | High-level async/await object to send network | | | data. | +----------------------------------------------------+----------------------------------------------------+ -[ Examples ]- * Example TCP client. * See also the streams APIs documentation. Synchronization =============== Threading-like synchronization primitives that can be used in Tasks. +----------------------------------------------------+----------------------------------------------------+ | "Lock" | A mutex lock. | +----------------------------------------------------+----------------------------------------------------+ | "Event" | An event object. | +----------------------------------------------------+----------------------------------------------------+ | "Condition" | A condition object. | +----------------------------------------------------+----------------------------------------------------+ | "Semaphore" | A semaphore. | +----------------------------------------------------+----------------------------------------------------+ | "BoundedSemaphore" | A bounded semaphore. | +----------------------------------------------------+----------------------------------------------------+ | "Barrier" | A barrier object. | +----------------------------------------------------+----------------------------------------------------+ -[ Examples ]- * Using asyncio.Event. * Using asyncio.Barrier. * See also the documentation of asyncio synchronization primitives. Exceptions ========== +----------------------------------------------------+----------------------------------------------------+ | "asyncio.CancelledError" | Raised when a Task is cancelled. See also | | | "Task.cancel()". | +----------------------------------------------------+----------------------------------------------------+ | "asyncio.BrokenBarrierError" | Raised when a Barrier is broken. See also | | | "Barrier.wait()". | +----------------------------------------------------+----------------------------------------------------+ -[ Examples ]- * Handling CancelledError to run code on cancellation request. * See also the full list of asyncio-specific exceptions. Developing with asyncio *********************** Asynchronous programming is different from classic “sequential” programming. This page lists common mistakes and traps and explains how to avoid them. Debug Mode ========== By default asyncio runs in production mode. In order to ease the development asyncio has a *debug mode*. There are several ways to enable asyncio debug mode: * Setting the "PYTHONASYNCIODEBUG" environment variable to "1". * Using the Python Development Mode. * Passing "debug=True" to "asyncio.run()". * Calling "loop.set_debug()". In addition to enabling the debug mode, consider also: * setting the log level of the asyncio logger to "logging.DEBUG", for example the following snippet of code can be run at startup of the application: logging.basicConfig(level=logging.DEBUG) * configuring the "warnings" module to display "ResourceWarning" warnings. One way of doing that is by using the "-W" "default" command line option. When the debug mode is enabled: * Many non-threadsafe asyncio APIs (such as "loop.call_soon()" and "loop.call_at()" methods) raise an exception if they are called from a wrong thread. * The execution time of the I/O selector is logged if it takes too long to perform an I/O operation. * Callbacks taking longer than 100 milliseconds are logged. The "loop.slow_callback_duration" attribute can be used to set the minimum execution duration in seconds that is considered “slow”. Concurrency and Multithreading ============================== An event loop runs in a thread (typically the main thread) and executes all callbacks and Tasks in its thread. While a Task is running in the event loop, no other Tasks can run in the same thread. When a Task executes an "await" expression, the running Task gets suspended, and the event loop executes the next Task. To schedule a *callback* from another OS thread, the "loop.call_soon_threadsafe()" method should be used. Example: loop.call_soon_threadsafe(callback, *args) Almost all asyncio objects are not thread safe, which is typically not a problem unless there is code that works with them from outside of a Task or a callback. If there’s a need for such code to call a low- level asyncio API, the "loop.call_soon_threadsafe()" method should be used, e.g.: loop.call_soon_threadsafe(fut.cancel) To schedule a coroutine object from a different OS thread, the "run_coroutine_threadsafe()" function should be used. It returns a "concurrent.futures.Future" to access the result: async def coro_func(): return await asyncio.sleep(1, 42) # Later in another OS thread: future = asyncio.run_coroutine_threadsafe(coro_func(), loop) # Wait for the result: result = future.result() To handle signals the event loop must be run in the main thread. The "loop.run_in_executor()" method can be used with a "concurrent.futures.ThreadPoolExecutor" to execute blocking code in a different OS thread without blocking the OS thread that the event loop runs in. There is currently no way to schedule coroutines or callbacks directly from a different process (such as one started with "multiprocessing"). The Event Loop Methods section lists APIs that can read from pipes and watch file descriptors without blocking the event loop. In addition, asyncio’s Subprocess APIs provide a way to start a process and communicate with it from the event loop. Lastly, the aforementioned "loop.run_in_executor()" method can also be used with a "concurrent.futures.ProcessPoolExecutor" to execute code in a different process. Running Blocking Code ===================== Blocking (CPU-bound) code should not be called directly. For example, if a function performs a CPU-intensive calculation for 1 second, all concurrent asyncio Tasks and IO operations would be delayed by 1 second. An executor can be used to run a task in a different thread or even in a different process to avoid blocking the OS thread with the event loop. See the "loop.run_in_executor()" method for more details. Logging ======= asyncio uses the "logging" module and all logging is performed via the ""asyncio"" logger. The default log level is "logging.INFO", which can be easily adjusted: logging.getLogger("asyncio").setLevel(logging.WARNING) Network logging can block the event loop. It is recommended to use a separate thread for handling logs or use non-blocking IO. For example, see Dealing with handlers that block. Detect never-awaited coroutines =============================== When a coroutine function is called, but not awaited (e.g. "coro()" instead of "await coro()") or the coroutine is not scheduled with "asyncio.create_task()", asyncio will emit a "RuntimeWarning": import asyncio async def test(): print("never scheduled") async def main(): test() asyncio.run(main()) Output: test.py:7: RuntimeWarning: coroutine 'test' was never awaited test() Output in debug mode: test.py:7: RuntimeWarning: coroutine 'test' was never awaited Coroutine created at (most recent call last) File "../t.py", line 9, in asyncio.run(main(), debug=True) < .. > File "../t.py", line 7, in main test() test() The usual fix is to either await the coroutine or call the "asyncio.create_task()" function: async def main(): await test() Detect never-retrieved exceptions ================================= If a "Future.set_exception()" is called but the Future object is never awaited on, the exception would never be propagated to the user code. In this case, asyncio would emit a log message when the Future object is garbage collected. Example of an unhandled exception: import asyncio async def bug(): raise Exception("not consumed") async def main(): asyncio.create_task(bug()) asyncio.run(main()) Output: Task exception was never retrieved future: exception=Exception('not consumed')> Traceback (most recent call last): File "test.py", line 4, in bug raise Exception("not consumed") Exception: not consumed Enable the debug mode to get the traceback where the task was created: asyncio.run(main(), debug=True) Output in debug mode: Task exception was never retrieved future: exception=Exception('not consumed') created at asyncio/tasks.py:321> source_traceback: Object created at (most recent call last): File "../t.py", line 9, in asyncio.run(main(), debug=True) < .. > Traceback (most recent call last): File "../t.py", line 4, in bug raise Exception("not consumed") Exception: not consumed Event Loop ********** **Source code:** Lib/asyncio/events.py, Lib/asyncio/base_events.py ====================================================================== -[ Preface ]- The event loop is the core of every asyncio application. Event loops run asynchronous tasks and callbacks, perform network IO operations, and run subprocesses. Application developers should typically use the high-level asyncio functions, such as "asyncio.run()", and should rarely need to reference the loop object or call its methods. This section is intended mostly for authors of lower-level code, libraries, and frameworks, who need finer control over the event loop behavior. -[ Obtaining the Event Loop ]- The following low-level functions can be used to get, set, or create an event loop: asyncio.get_running_loop() Return the running event loop in the current OS thread. Raise a "RuntimeError" if there is no running event loop. This function can only be called from a coroutine or a callback. Added in version 3.7. asyncio.get_event_loop() Get the current event loop. When called from a coroutine or a callback (e.g. scheduled with call_soon or similar API), this function will always return the running event loop. If there is no running event loop set, the function will return the result of the "get_event_loop_policy().get_event_loop()" call. Because this function has rather complex behavior (especially when custom event loop policies are in use), using the "get_running_loop()" function is preferred to "get_event_loop()" in coroutines and callbacks. As noted above, consider using the higher-level "asyncio.run()" function, instead of using these lower level functions to manually create and close an event loop. Deprecated since version 3.12: Deprecation warning is emitted if there is no current event loop. In some future Python release this will become an error. asyncio.set_event_loop(loop) Set *loop* as the current event loop for the current OS thread. asyncio.new_event_loop() Create and return a new event loop object. Note that the behaviour of "get_event_loop()", "set_event_loop()", and "new_event_loop()" functions can be altered by setting a custom event loop policy. -[ Contents ]- This documentation page contains the following sections: * The Event Loop Methods section is the reference documentation of the event loop APIs; * The Callback Handles section documents the "Handle" and "TimerHandle" instances which are returned from scheduling methods such as "loop.call_soon()" and "loop.call_later()"; * The Server Objects section documents types returned from event loop methods like "loop.create_server()"; * The Event Loop Implementations section documents the "SelectorEventLoop" and "ProactorEventLoop" classes; * The Examples section showcases how to work with some event loop APIs. Event Loop Methods ================== Event loops have **low-level** APIs for the following: * Running and stopping the loop * Scheduling callbacks * Scheduling delayed callbacks * Creating Futures and Tasks * Opening network connections * Creating network servers * Transferring files * TLS Upgrade * Watching file descriptors * Working with socket objects directly * DNS * Working with pipes * Unix signals * Executing code in thread or process pools * Error Handling API * Enabling debug mode * Running Subprocesses Running and stopping the loop ----------------------------- loop.run_until_complete(future) Run until the *future* (an instance of "Future") has completed. If the argument is a coroutine object it is implicitly scheduled to run as a "asyncio.Task". Return the Future’s result or raise its exception. loop.run_forever() Run the event loop until "stop()" is called. If "stop()" is called before "run_forever()" is called, the loop will poll the I/O selector once with a timeout of zero, run all callbacks scheduled in response to I/O events (and those that were already scheduled), and then exit. If "stop()" is called while "run_forever()" is running, the loop will run the current batch of callbacks and then exit. Note that new callbacks scheduled by callbacks will not run in this case; instead, they will run the next time "run_forever()" or "run_until_complete()" is called. loop.stop() Stop the event loop. loop.is_running() Return "True" if the event loop is currently running. loop.is_closed() Return "True" if the event loop was closed. loop.close() Close the event loop. The loop must not be running when this function is called. Any pending callbacks will be discarded. This method clears all queues and shuts down the executor, but does not wait for the executor to finish. This method is idempotent and irreversible. No other methods should be called after the event loop is closed. async loop.shutdown_asyncgens() Schedule all currently open *asynchronous generator* objects to close with an "aclose()" call. After calling this method, the event loop will issue a warning if a new asynchronous generator is iterated. This should be used to reliably finalize all scheduled asynchronous generators. Note that there is no need to call this function when "asyncio.run()" is used. Example: try: loop.run_forever() finally: loop.run_until_complete(loop.shutdown_asyncgens()) loop.close() Added in version 3.6. async loop.shutdown_default_executor(timeout=None) Schedule the closure of the default executor and wait for it to join all of the threads in the "ThreadPoolExecutor". Once this method has been called, using the default executor with "loop.run_in_executor()" will raise a "RuntimeError". The *timeout* parameter specifies the amount of time (in "float" seconds) the executor will be given to finish joining. With the default, "None", the executor is allowed an unlimited amount of time. If the *timeout* is reached, a "RuntimeWarning" is emitted and the default executor is terminated without waiting for its threads to finish joining. Note: Do not call this method when using "asyncio.run()", as the latter handles default executor shutdown automatically. Added in version 3.9. Changed in version 3.12: Added the *timeout* parameter. Scheduling callbacks -------------------- loop.call_soon(callback, *args, context=None) Schedule the *callback* *callback* to be called with *args* arguments at the next iteration of the event loop. Return an instance of "asyncio.Handle", which can be used later to cancel the callback. Callbacks are called in the order in which they are registered. Each callback will be called exactly once. The optional keyword-only *context* argument specifies a custom "contextvars.Context" for the *callback* to run in. Callbacks use the current context when no *context* is provided. Unlike "call_soon_threadsafe()", this method is not thread-safe. loop.call_soon_threadsafe(callback, *args, context=None) A thread-safe variant of "call_soon()". When scheduling callbacks from another thread, this function *must* be used, since "call_soon()" is not thread-safe. This function is safe to be called from a reentrant context or signal handler, however, it is not safe or fruitful to use the returned handle in such contexts. Raises "RuntimeError" if called on a loop that’s been closed. This can happen on a secondary thread when the main application is shutting down. See the concurrency and multithreading section of the documentation. Changed in version 3.7: The *context* keyword-only parameter was added. See **PEP 567** for more details. Note: Most "asyncio" scheduling functions don’t allow passing keyword arguments. To do that, use "functools.partial()": # will schedule "print("Hello", flush=True)" loop.call_soon( functools.partial(print, "Hello", flush=True)) Using partial objects is usually more convenient than using lambdas, as asyncio can render partial objects better in debug and error messages. Scheduling delayed callbacks ---------------------------- Event loop provides mechanisms to schedule callback functions to be called at some point in the future. Event loop uses monotonic clocks to track time. loop.call_later(delay, callback, *args, context=None) Schedule *callback* to be called after the given *delay* number of seconds (can be either an int or a float). An instance of "asyncio.TimerHandle" is returned which can be used to cancel the callback. *callback* will be called exactly once. If two callbacks are scheduled for exactly the same time, the order in which they are called is undefined. The optional positional *args* will be passed to the callback when it is called. If you want the callback to be called with keyword arguments use "functools.partial()". An optional keyword-only *context* argument allows specifying a custom "contextvars.Context" for the *callback* to run in. The current context is used when no *context* is provided. Changed in version 3.7: The *context* keyword-only parameter was added. See **PEP 567** for more details. Changed in version 3.8: In Python 3.7 and earlier with the default event loop implementation, the *delay* could not exceed one day. This has been fixed in Python 3.8. loop.call_at(when, callback, *args, context=None) Schedule *callback* to be called at the given absolute timestamp *when* (an int or a float), using the same time reference as "loop.time()". This method’s behavior is the same as "call_later()". An instance of "asyncio.TimerHandle" is returned which can be used to cancel the callback. Changed in version 3.7: The *context* keyword-only parameter was added. See **PEP 567** for more details. Changed in version 3.8: In Python 3.7 and earlier with the default event loop implementation, the difference between *when* and the current time could not exceed one day. This has been fixed in Python 3.8. loop.time() Return the current time, as a "float" value, according to the event loop’s internal monotonic clock. Note: Changed in version 3.8: In Python 3.7 and earlier timeouts (relative *delay* or absolute *when*) should not exceed one day. This has been fixed in Python 3.8. See also: The "asyncio.sleep()" function. Creating Futures and Tasks -------------------------- loop.create_future() Create an "asyncio.Future" object attached to the event loop. This is the preferred way to create Futures in asyncio. This lets third-party event loops provide alternative implementations of the Future object (with better performance or instrumentation). Added in version 3.5.2. loop.create_task(coro, *, name=None, context=None, **kwargs) Schedule the execution of coroutine *coro*. Return a "Task" object. Third-party event loops can use their own subclass of "Task" for interoperability. In this case, the result type is a subclass of "Task". The full function signature is largely the same as that of the "Task" constructor (or factory) - all of the keyword arguments to this function are passed through to that interface, except *name*, or *context* if it is "None". If the *name* argument is provided and not "None", it is set as the name of the task using "Task.set_name()". An optional keyword-only *context* argument allows specifying a custom "contextvars.Context" for the *coro* to run in. The current context copy is created when no *context* is provided. Changed in version 3.8: Added the *name* parameter. Changed in version 3.11: Added the *context* parameter. Changed in version 3.13.3: Added "kwargs" which passes on arbitrary extra parameters, including "name" and "context". Changed in version 3.13.4: Rolled back the change that passes on *name* and *context* (if it is None), while still passing on other arbitrary keyword arguments (to avoid breaking backwards compatibility with 3.13.3). loop.set_task_factory(factory) Set a task factory that will be used by "loop.create_task()". If *factory* is "None" the default task factory will be set. Otherwise, *factory* must be a *callable* with the signature matching "(loop, coro, **kwargs)", where *loop* is a reference to the active event loop, and *coro* is a coroutine object. The callable must pass on all *kwargs*, and return a "asyncio.Task"-compatible object. Changed in version 3.13.3: Required that all *kwargs* are passed on to "asyncio.Task". Changed in version 3.13.4: *name* is no longer passed to task factories. *context* is no longer passed to task factories if it is "None". loop.get_task_factory() Return a task factory or "None" if the default one is in use. Opening network connections --------------------------- async loop.create_connection(protocol_factory, host=None, port=None, *, ssl=None, family=0, proto=0, flags=0, sock=None, local_addr=None, server_hostname=None, ssl_handshake_timeout=None, ssl_shutdown_timeout=None, happy_eyeballs_delay=None, interleave=None, all_errors=False) Open a streaming transport connection to a given address specified by *host* and *port*. The socket family can be either "AF_INET" or "AF_INET6" depending on *host* (or the *family* argument, if provided). The socket type will be "SOCK_STREAM". *protocol_factory* must be a callable returning an asyncio protocol implementation. This method will try to establish the connection in the background. When successful, it returns a "(transport, protocol)" pair. The chronological synopsis of the underlying operation is as follows: 1. The connection is established and a transport is created for it. 2. *protocol_factory* is called without arguments and is expected to return a protocol instance. 3. The protocol instance is coupled with the transport by calling its "connection_made()" method. 4. A "(transport, protocol)" tuple is returned on success. The created transport is an implementation-dependent bidirectional stream. Other arguments: * *ssl*: if given and not false, a SSL/TLS transport is created (by default a plain TCP transport is created). If *ssl* is a "ssl.SSLContext" object, this context is used to create the transport; if *ssl* is "True", a default context returned from "ssl.create_default_context()" is used. See also: SSL/TLS security considerations * *server_hostname* sets or overrides the hostname that the target server’s certificate will be matched against. Should only be passed if *ssl* is not "None". By default the value of the *host* argument is used. If *host* is empty, there is no default and you must pass a value for *server_hostname*. If *server_hostname* is an empty string, hostname matching is disabled (which is a serious security risk, allowing for potential man-in-the-middle attacks). * *family*, *proto*, *flags* are the optional address family, protocol and flags to be passed through to getaddrinfo() for *host* resolution. If given, these should all be integers from the corresponding "socket" module constants. * *happy_eyeballs_delay*, if given, enables Happy Eyeballs for this connection. It should be a floating-point number representing the amount of time in seconds to wait for a connection attempt to complete, before starting the next attempt in parallel. This is the “Connection Attempt Delay” as defined in **RFC 8305**. A sensible default value recommended by the RFC is "0.25" (250 milliseconds). * *interleave* controls address reordering when a host name resolves to multiple IP addresses. If "0" or unspecified, no reordering is done, and addresses are tried in the order returned by "getaddrinfo()". If a positive integer is specified, the addresses are interleaved by address family, and the given integer is interpreted as “First Address Family Count” as defined in **RFC 8305**. The default is "0" if *happy_eyeballs_delay* is not specified, and "1" if it is. * *sock*, if given, should be an existing, already connected "socket.socket" object to be used by the transport. If *sock* is given, none of *host*, *port*, *family*, *proto*, *flags*, *happy_eyeballs_delay*, *interleave* and *local_addr* should be specified. Note: The *sock* argument transfers ownership of the socket to the transport created. To close the socket, call the transport’s "close()" method. * *local_addr*, if given, is a "(local_host, local_port)" tuple used to bind the socket locally. The *local_host* and *local_port* are looked up using "getaddrinfo()", similarly to *host* and *port*. * *ssl_handshake_timeout* is (for a TLS connection) the time in seconds to wait for the TLS handshake to complete before aborting the connection. "60.0" seconds if "None" (default). * *ssl_shutdown_timeout* is the time in seconds to wait for the SSL shutdown to complete before aborting the connection. "30.0" seconds if "None" (default). * *all_errors* determines what exceptions are raised when a connection cannot be created. By default, only a single "Exception" is raised: the first exception if there is only one or all errors have same message, or a single "OSError" with the error messages combined. When "all_errors" is "True", an "ExceptionGroup" will be raised containing all exceptions (even if there is only one). Changed in version 3.5: Added support for SSL/TLS in "ProactorEventLoop". Changed in version 3.6: The socket option socket.TCP_NODELAY is set by default for all TCP connections. Changed in version 3.7: Added the *ssl_handshake_timeout* parameter. Changed in version 3.8: Added the *happy_eyeballs_delay* and *interleave* parameters.Happy Eyeballs Algorithm: Success with Dual-Stack Hosts. When a server’s IPv4 path and protocol are working, but the server’s IPv6 path and protocol are not working, a dual-stack client application experiences significant connection delay compared to an IPv4-only client. This is undesirable because it causes the dual-stack client to have a worse user experience. This document specifies requirements for algorithms that reduce this user-visible delay and provides an algorithm.For more information: https://datatracker.ietf.org/doc/html/rfc6555 Changed in version 3.11: Added the *ssl_shutdown_timeout* parameter. Changed in version 3.12: *all_errors* was added. See also: The "open_connection()" function is a high-level alternative API. It returns a pair of ("StreamReader", "StreamWriter") that can be used directly in async/await code. async loop.create_datagram_endpoint(protocol_factory, local_addr=None, remote_addr=None, *, family=0, proto=0, flags=0, reuse_port=None, allow_broadcast=None, sock=None) Create a datagram connection. The socket family can be either "AF_INET", "AF_INET6", or "AF_UNIX", depending on *host* (or the *family* argument, if provided). The socket type will be "SOCK_DGRAM". *protocol_factory* must be a callable returning a protocol implementation. A tuple of "(transport, protocol)" is returned on success. Other arguments: * *local_addr*, if given, is a "(local_host, local_port)" tuple used to bind the socket locally. The *local_host* and *local_port* are looked up using "getaddrinfo()". Note: On Windows, when using the proactor event loop with "local_addr=None", an "OSError" with "errno.WSAEINVAL" will be raised when running it. * *remote_addr*, if given, is a "(remote_host, remote_port)" tuple used to connect the socket to a remote address. The *remote_host* and *remote_port* are looked up using "getaddrinfo()". * *family*, *proto*, *flags* are the optional address family, protocol and flags to be passed through to "getaddrinfo()" for *host* resolution. If given, these should all be integers from the corresponding "socket" module constants. * *reuse_port* tells the kernel to allow this endpoint to be bound to the same port as other existing endpoints are bound to, so long as they all set this flag when being created. This option is not supported on Windows and some Unixes. If the socket.SO_REUSEPORT constant is not defined then this capability is unsupported. * *allow_broadcast* tells the kernel to allow this endpoint to send messages to the broadcast address. * *sock* can optionally be specified in order to use a preexisting, already connected, "socket.socket" object to be used by the transport. If specified, *local_addr* and *remote_addr* should be omitted (must be "None"). Note: The *sock* argument transfers ownership of the socket to the transport created. To close the socket, call the transport’s "close()" method. See UDP echo client protocol and UDP echo server protocol examples. Changed in version 3.4.4: The *family*, *proto*, *flags*, *reuse_address*, *reuse_port*, *allow_broadcast*, and *sock* parameters were added. Changed in version 3.8: Added support for Windows. Changed in version 3.8.1: The *reuse_address* parameter is no longer supported, as using socket.SO_REUSEADDR poses a significant security concern for UDP. Explicitly passing "reuse_address=True" will raise an exception.When multiple processes with differing UIDs assign sockets to an identical UDP socket address with "SO_REUSEADDR", incoming packets can become randomly distributed among the sockets.For supported platforms, *reuse_port* can be used as a replacement for similar functionality. With *reuse_port*, socket.SO_REUSEPORT is used instead, which specifically prevents processes with differing UIDs from assigning sockets to the same socket address. Changed in version 3.11: The *reuse_address* parameter, disabled since Python 3.8.1, 3.7.6 and 3.6.10, has been entirely removed. async loop.create_unix_connection(protocol_factory, path=None, *, ssl=None, sock=None, server_hostname=None, ssl_handshake_timeout=None, ssl_shutdown_timeout=None) Create a Unix connection. The socket family will be "AF_UNIX"; socket type will be "SOCK_STREAM". A tuple of "(transport, protocol)" is returned on success. *path* is the name of a Unix domain socket and is required, unless a *sock* parameter is specified. Abstract Unix sockets, "str", "bytes", and "Path" paths are supported. See the documentation of the "loop.create_connection()" method for information about arguments to this method. Availability: Unix. Changed in version 3.7: Added the *ssl_handshake_timeout* parameter. The *path* parameter can now be a *path-like object*. Changed in version 3.11: Added the *ssl_shutdown_timeout* parameter. Creating network servers ------------------------ async loop.create_server(protocol_factory, host=None, port=None, *, family=socket.AF_UNSPEC, flags=socket.AI_PASSIVE, sock=None, backlog=100, ssl=None, reuse_address=None, reuse_port=None, keep_alive=None, ssl_handshake_timeout=None, ssl_shutdown_timeout=None, start_serving=True) Create a TCP server (socket type "SOCK_STREAM") listening on *port* of the *host* address. Returns a "Server" object. Arguments: * *protocol_factory* must be a callable returning a protocol implementation. * The *host* parameter can be set to several types which determine where the server would be listening: * If *host* is a string, the TCP server is bound to a single network interface specified by *host*. * If *host* is a sequence of strings, the TCP server is bound to all network interfaces specified by the sequence. * If *host* is an empty string or "None", all interfaces are assumed and a list of multiple sockets will be returned (most likely one for IPv4 and another one for IPv6). * The *port* parameter can be set to specify which port the server should listen on. If "0" or "None" (the default), a random unused port will be selected (note that if *host* resolves to multiple network interfaces, a different random port will be selected for each interface). * *family* can be set to either "socket.AF_INET" or "AF_INET6" to force the socket to use IPv4 or IPv6. If not set, the *family* will be determined from host name (defaults to "AF_UNSPEC"). * *flags* is a bitmask for "getaddrinfo()". * *sock* can optionally be specified in order to use a preexisting socket object. If specified, *host* and *port* must not be specified. Note: The *sock* argument transfers ownership of the socket to the server created. To close the socket, call the server’s "close()" method. * *backlog* is the maximum number of queued connections passed to "listen()" (defaults to 100). * *ssl* can be set to an "SSLContext" instance to enable TLS over the accepted connections. * *reuse_address* tells the kernel to reuse a local socket in "TIME_WAIT" state, without waiting for its natural timeout to expire. If not specified will automatically be set to "True" on Unix. * *reuse_port* tells the kernel to allow this endpoint to be bound to the same port as other existing endpoints are bound to, so long as they all set this flag when being created. This option is not supported on Windows. * *keep_alive* set to "True" keeps connections active by enabling the periodic transmission of messages. Changed in version 3.13: Added the *keep_alive* parameter. * *ssl_handshake_timeout* is (for a TLS server) the time in seconds to wait for the TLS handshake to complete before aborting the connection. "60.0" seconds if "None" (default). * *ssl_shutdown_timeout* is the time in seconds to wait for the SSL shutdown to complete before aborting the connection. "30.0" seconds if "None" (default). * *start_serving* set to "True" (the default) causes the created server to start accepting connections immediately. When set to "False", the user should await on "Server.start_serving()" or "Server.serve_forever()" to make the server to start accepting connections. Changed in version 3.5: Added support for SSL/TLS in "ProactorEventLoop". Changed in version 3.5.1: The *host* parameter can be a sequence of strings. Changed in version 3.6: Added *ssl_handshake_timeout* and *start_serving* parameters. The socket option socket.TCP_NODELAY is set by default for all TCP connections. Changed in version 3.11: Added the *ssl_shutdown_timeout* parameter. See also: The "start_server()" function is a higher-level alternative API that returns a pair of "StreamReader" and "StreamWriter" that can be used in an async/await code. async loop.create_unix_server(protocol_factory, path=None, *, sock=None, backlog=100, ssl=None, ssl_handshake_timeout=None, ssl_shutdown_timeout=None, start_serving=True, cleanup_socket=True) Similar to "loop.create_server()" but works with the "AF_UNIX" socket family. *path* is the name of a Unix domain socket, and is required, unless a *sock* argument is provided. Abstract Unix sockets, "str", "bytes", and "Path" paths are supported. If *cleanup_socket* is true then the Unix socket will automatically be removed from the filesystem when the server is closed, unless the socket has been replaced after the server has been created. See the documentation of the "loop.create_server()" method for information about arguments to this method. Availability: Unix. Changed in version 3.7: Added the *ssl_handshake_timeout* and *start_serving* parameters. The *path* parameter can now be a "Path" object. Changed in version 3.11: Added the *ssl_shutdown_timeout* parameter. Changed in version 3.13: Added the *cleanup_socket* parameter. async loop.connect_accepted_socket(protocol_factory, sock, *, ssl=None, ssl_handshake_timeout=None, ssl_shutdown_timeout=None) Wrap an already accepted connection into a transport/protocol pair. This method can be used by servers that accept connections outside of asyncio but that use asyncio to handle them. Parameters: * *protocol_factory* must be a callable returning a protocol implementation. * *sock* is a preexisting socket object returned from "socket.accept". Note: The *sock* argument transfers ownership of the socket to the transport created. To close the socket, call the transport’s "close()" method. * *ssl* can be set to an "SSLContext" to enable SSL over the accepted connections. * *ssl_handshake_timeout* is (for an SSL connection) the time in seconds to wait for the SSL handshake to complete before aborting the connection. "60.0" seconds if "None" (default). * *ssl_shutdown_timeout* is the time in seconds to wait for the SSL shutdown to complete before aborting the connection. "30.0" seconds if "None" (default). Returns a "(transport, protocol)" pair. Added in version 3.5.3. Changed in version 3.7: Added the *ssl_handshake_timeout* parameter. Changed in version 3.11: Added the *ssl_shutdown_timeout* parameter. Transferring files ------------------ async loop.sendfile(transport, file, offset=0, count=None, *, fallback=True) Send a *file* over a *transport*. Return the total number of bytes sent. The method uses high-performance "os.sendfile()" if available. *file* must be a regular file object opened in binary mode. *offset* tells from where to start reading the file. If specified, *count* is the total number of bytes to transmit as opposed to sending the file until EOF is reached. File position is always updated, even when this method raises an error, and "file.tell()" can be used to obtain the actual number of bytes sent. *fallback* set to "True" makes asyncio to manually read and send the file when the platform does not support the sendfile system call (e.g. Windows or SSL socket on Unix). Raise "SendfileNotAvailableError" if the system does not support the *sendfile* syscall and *fallback* is "False". Added in version 3.7. TLS Upgrade ----------- async loop.start_tls(transport, protocol, sslcontext, *, server_side=False, server_hostname=None, ssl_handshake_timeout=None, ssl_shutdown_timeout=None) Upgrade an existing transport-based connection to TLS. Create a TLS coder/decoder instance and insert it between the *transport* and the *protocol*. The coder/decoder implements both *transport*-facing protocol and *protocol*-facing transport. Return the created two-interface instance. After *await*, the *protocol* must stop using the original *transport* and communicate with the returned object only because the coder caches *protocol*-side data and sporadically exchanges extra TLS session packets with *transport*. In some situations (e.g. when the passed transport is already closing) this may return "None". Parameters: * *transport* and *protocol* instances that methods like "create_server()" and "create_connection()" return. * *sslcontext*: a configured instance of "SSLContext". * *server_side* pass "True" when a server-side connection is being upgraded (like the one created by "create_server()"). * *server_hostname*: sets or overrides the host name that the target server’s certificate will be matched against. * *ssl_handshake_timeout* is (for a TLS connection) the time in seconds to wait for the TLS handshake to complete before aborting the connection. "60.0" seconds if "None" (default). * *ssl_shutdown_timeout* is the time in seconds to wait for the SSL shutdown to complete before aborting the connection. "30.0" seconds if "None" (default). Added in version 3.7. Changed in version 3.11: Added the *ssl_shutdown_timeout* parameter. Watching file descriptors ------------------------- loop.add_reader(fd, callback, *args) Start monitoring the *fd* file descriptor for read availability and invoke *callback* with the specified arguments once *fd* is available for reading. Any preexisting callback registered for *fd* is cancelled and replaced by *callback*. loop.remove_reader(fd) Stop monitoring the *fd* file descriptor for read availability. Returns "True" if *fd* was previously being monitored for reads. loop.add_writer(fd, callback, *args) Start monitoring the *fd* file descriptor for write availability and invoke *callback* with the specified arguments once *fd* is available for writing. Any preexisting callback registered for *fd* is cancelled and replaced by *callback*. Use "functools.partial()" to pass keyword arguments to *callback*. loop.remove_writer(fd) Stop monitoring the *fd* file descriptor for write availability. Returns "True" if *fd* was previously being monitored for writes. See also Platform Support section for some limitations of these methods. Working with socket objects directly ------------------------------------ In general, protocol implementations that use transport-based APIs such as "loop.create_connection()" and "loop.create_server()" are faster than implementations that work with sockets directly. However, there are some use cases when performance is not critical, and working with "socket" objects directly is more convenient. async loop.sock_recv(sock, nbytes) Receive up to *nbytes* from *sock*. Asynchronous version of "socket.recv()". Return the received data as a bytes object. *sock* must be a non-blocking socket. Changed in version 3.7: Even though this method was always documented as a coroutine method, releases before Python 3.7 returned a "Future". Since Python 3.7 this is an "async def" method. async loop.sock_recv_into(sock, buf) Receive data from *sock* into the *buf* buffer. Modeled after the blocking "socket.recv_into()" method. Return the number of bytes written to the buffer. *sock* must be a non-blocking socket. Added in version 3.7. async loop.sock_recvfrom(sock, bufsize) Receive a datagram of up to *bufsize* from *sock*. Asynchronous version of "socket.recvfrom()". Return a tuple of (received data, remote address). *sock* must be a non-blocking socket. Added in version 3.11. async loop.sock_recvfrom_into(sock, buf, nbytes=0) Receive a datagram of up to *nbytes* from *sock* into *buf*. Asynchronous version of "socket.recvfrom_into()". Return a tuple of (number of bytes received, remote address). *sock* must be a non-blocking socket. Added in version 3.11. async loop.sock_sendall(sock, data) Send *data* to the *sock* socket. Asynchronous version of "socket.sendall()". This method continues to send to the socket until either all data in *data* has been sent or an error occurs. "None" is returned on success. On error, an exception is raised. Additionally, there is no way to determine how much data, if any, was successfully processed by the receiving end of the connection. *sock* must be a non-blocking socket. Changed in version 3.7: Even though the method was always documented as a coroutine method, before Python 3.7 it returned a "Future". Since Python 3.7, this is an "async def" method. async loop.sock_sendto(sock, data, address) Send a datagram from *sock* to *address*. Asynchronous version of "socket.sendto()". Return the number of bytes sent. *sock* must be a non-blocking socket. Added in version 3.11. async loop.sock_connect(sock, address) Connect *sock* to a remote socket at *address*. Asynchronous version of "socket.connect()". *sock* must be a non-blocking socket. Changed in version 3.5.2: "address" no longer needs to be resolved. "sock_connect" will try to check if the *address* is already resolved by calling "socket.inet_pton()". If not, "loop.getaddrinfo()" will be used to resolve the *address*. See also: "loop.create_connection()" and "asyncio.open_connection()". async loop.sock_accept(sock) Accept a connection. Modeled after the blocking "socket.accept()" method. The socket must be bound to an address and listening for connections. The return value is a pair "(conn, address)" where *conn* is a *new* socket object usable to send and receive data on the connection, and *address* is the address bound to the socket on the other end of the connection. *sock* must be a non-blocking socket. Changed in version 3.7: Even though the method was always documented as a coroutine method, before Python 3.7 it returned a "Future". Since Python 3.7, this is an "async def" method. See also: "loop.create_server()" and "start_server()". async loop.sock_sendfile(sock, file, offset=0, count=None, *, fallback=True) Send a file using high-performance "os.sendfile" if possible. Return the total number of bytes sent. Asynchronous version of "socket.sendfile()". *sock* must be a non-blocking "socket.SOCK_STREAM" "socket". *file* must be a regular file object open in binary mode. *offset* tells from where to start reading the file. If specified, *count* is the total number of bytes to transmit as opposed to sending the file until EOF is reached. File position is always updated, even when this method raises an error, and "file.tell()" can be used to obtain the actual number of bytes sent. *fallback*, when set to "True", makes asyncio manually read and send the file when the platform does not support the sendfile syscall (e.g. Windows or SSL socket on Unix). Raise "SendfileNotAvailableError" if the system does not support *sendfile* syscall and *fallback* is "False". *sock* must be a non-blocking socket. Added in version 3.7. DNS --- async loop.getaddrinfo(host, port, *, family=0, type=0, proto=0, flags=0) Asynchronous version of "socket.getaddrinfo()". async loop.getnameinfo(sockaddr, flags=0) Asynchronous version of "socket.getnameinfo()". Note: Both *getaddrinfo* and *getnameinfo* internally utilize their synchronous versions through the loop’s default thread pool executor. When this executor is saturated, these methods may experience delays, which higher-level networking libraries may report as increased timeouts. To mitigate this, consider using a custom executor for other user tasks, or setting a default executor with a larger number of workers. Changed in version 3.7: Both *getaddrinfo* and *getnameinfo* methods were always documented to return a coroutine, but prior to Python 3.7 they were, in fact, returning "asyncio.Future" objects. Starting with Python 3.7 both methods are coroutines. Working with pipes ------------------ async loop.connect_read_pipe(protocol_factory, pipe) Register the read end of *pipe* in the event loop. *protocol_factory* must be a callable returning an asyncio protocol implementation. *pipe* is a *file-like object*. Return pair "(transport, protocol)", where *transport* supports the "ReadTransport" interface and *protocol* is an object instantiated by the *protocol_factory*. With "SelectorEventLoop" event loop, the *pipe* is set to non- blocking mode. async loop.connect_write_pipe(protocol_factory, pipe) Register the write end of *pipe* in the event loop. *protocol_factory* must be a callable returning an asyncio protocol implementation. *pipe* is *file-like object*. Return pair "(transport, protocol)", where *transport* supports "WriteTransport" interface and *protocol* is an object instantiated by the *protocol_factory*. With "SelectorEventLoop" event loop, the *pipe* is set to non- blocking mode. Note: "SelectorEventLoop" does not support the above methods on Windows. Use "ProactorEventLoop" instead for Windows. See also: The "loop.subprocess_exec()" and "loop.subprocess_shell()" methods. Unix signals ------------ loop.add_signal_handler(signum, callback, *args) Set *callback* as the handler for the *signum* signal. The callback will be invoked by *loop*, along with other queued callbacks and runnable coroutines of that event loop. Unlike signal handlers registered using "signal.signal()", a callback registered with this function is allowed to interact with the event loop. Raise "ValueError" if the signal number is invalid or uncatchable. Raise "RuntimeError" if there is a problem setting up the handler. Use "functools.partial()" to pass keyword arguments to *callback*. Like "signal.signal()", this function must be invoked in the main thread. loop.remove_signal_handler(sig) Remove the handler for the *sig* signal. Return "True" if the signal handler was removed, or "False" if no handler was set for the given signal. Availability: Unix. See also: The "signal" module. Executing code in thread or process pools ----------------------------------------- awaitable loop.run_in_executor(executor, func, *args) Arrange for *func* to be called in the specified executor. The *executor* argument should be an "concurrent.futures.Executor" instance. The default executor is used if *executor* is "None". The default executor can be set by "loop.set_default_executor()", otherwise, a "concurrent.futures.ThreadPoolExecutor" will be lazy- initialized and used by "run_in_executor()" if needed. Example: import asyncio import concurrent.futures def blocking_io(): # File operations (such as logging) can block the # event loop: run them in a thread pool. with open('/dev/urandom', 'rb') as f: return f.read(100) def cpu_bound(): # CPU-bound operations will block the event loop: # in general it is preferable to run them in a # process pool. return sum(i * i for i in range(10 ** 7)) async def main(): loop = asyncio.get_running_loop() ## Options: # 1. Run in the default loop's executor: result = await loop.run_in_executor( None, blocking_io) print('default thread pool', result) # 2. Run in a custom thread pool: with concurrent.futures.ThreadPoolExecutor() as pool: result = await loop.run_in_executor( pool, blocking_io) print('custom thread pool', result) # 3. Run in a custom process pool: with concurrent.futures.ProcessPoolExecutor() as pool: result = await loop.run_in_executor( pool, cpu_bound) print('custom process pool', result) if __name__ == '__main__': asyncio.run(main()) Note that the entry point guard ("if __name__ == '__main__'") is required for option 3 due to the peculiarities of "multiprocessing", which is used by "ProcessPoolExecutor". See Safe importing of main module. This method returns a "asyncio.Future" object. Use "functools.partial()" to pass keyword arguments to *func*. Changed in version 3.5.3: "loop.run_in_executor()" no longer configures the "max_workers" of the thread pool executor it creates, instead leaving it up to the thread pool executor ("ThreadPoolExecutor") to set the default. loop.set_default_executor(executor) Set *executor* as the default executor used by "run_in_executor()". *executor* must be an instance of "ThreadPoolExecutor". Changed in version 3.11: *executor* must be an instance of "ThreadPoolExecutor". Error Handling API ------------------ Allows customizing how exceptions are handled in the event loop. loop.set_exception_handler(handler) Set *handler* as the new event loop exception handler. If *handler* is "None", the default exception handler will be set. Otherwise, *handler* must be a callable with the signature matching "(loop, context)", where "loop" is a reference to the active event loop, and "context" is a "dict" object containing the details of the exception (see "call_exception_handler()" documentation for details about context). If the handler is called on behalf of a "Task" or "Handle", it is run in the "contextvars.Context" of that task or callback handle. Changed in version 3.12: The handler may be called in the "Context" of the task or handle where the exception originated. loop.get_exception_handler() Return the current exception handler, or "None" if no custom exception handler was set. Added in version 3.5.2. loop.default_exception_handler(context) Default exception handler. This is called when an exception occurs and no exception handler is set. This can be called by a custom exception handler that wants to defer to the default handler behavior. *context* parameter has the same meaning as in "call_exception_handler()". loop.call_exception_handler(context) Call the current event loop exception handler. *context* is a "dict" object containing the following keys (new keys may be introduced in future Python versions): * ‘message’: Error message; * ‘exception’ (optional): Exception object; * ‘future’ (optional): "asyncio.Future" instance; * ‘task’ (optional): "asyncio.Task" instance; * ‘handle’ (optional): "asyncio.Handle" instance; * ‘protocol’ (optional): Protocol instance; * ‘transport’ (optional): Transport instance; * ‘socket’ (optional): "socket.socket" instance; * ‘source_traceback’ (optional): Traceback of the source; * ‘handle_traceback’ (optional): Traceback of the handle; * ‘asyncgen’ (optional): Asynchronous generator that caused the exception. Note: This method should not be overloaded in subclassed event loops. For custom exception handling, use the "set_exception_handler()" method. Enabling debug mode ------------------- loop.get_debug() Get the debug mode ("bool") of the event loop. The default value is "True" if the environment variable "PYTHONASYNCIODEBUG" is set to a non-empty string, "False" otherwise. loop.set_debug(enabled: bool) Set the debug mode of the event loop. Changed in version 3.7: The new Python Development Mode can now also be used to enable the debug mode. loop.slow_callback_duration This attribute can be used to set the minimum execution duration in seconds that is considered “slow”. When debug mode is enabled, “slow” callbacks are logged. Default value is 100 milliseconds. See also: The debug mode of asyncio. Running Subprocesses -------------------- Methods described in this subsections are low-level. In regular async/await code consider using the high-level "asyncio.create_subprocess_shell()" and "asyncio.create_subprocess_exec()" convenience functions instead. Note: On Windows, the default event loop "ProactorEventLoop" supports subprocesses, whereas "SelectorEventLoop" does not. See Subprocess Support on Windows for details. async loop.subprocess_exec(protocol_factory, *args, stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE, **kwargs) Create a subprocess from one or more string arguments specified by *args*. *args* must be a list of strings represented by: * "str"; * or "bytes", encoded to the filesystem encoding. The first string specifies the program executable, and the remaining strings specify the arguments. Together, string arguments form the "argv" of the program. This is similar to the standard library "subprocess.Popen" class called with "shell=False" and the list of strings passed as the first argument; however, where "Popen" takes a single argument which is list of strings, *subprocess_exec* takes multiple string arguments. The *protocol_factory* must be a callable returning a subclass of the "asyncio.SubprocessProtocol" class. Other parameters: * *stdin* can be any of these: * a file-like object * an existing file descriptor (a positive integer), for example those created with "os.pipe()" * the "subprocess.PIPE" constant (default) which will create a new pipe and connect it, * the value "None" which will make the subprocess inherit the file descriptor from this process * the "subprocess.DEVNULL" constant which indicates that the special "os.devnull" file will be used * *stdout* can be any of these: * a file-like object * the "subprocess.PIPE" constant (default) which will create a new pipe and connect it, * the value "None" which will make the subprocess inherit the file descriptor from this process * the "subprocess.DEVNULL" constant which indicates that the special "os.devnull" file will be used * *stderr* can be any of these: * a file-like object * the "subprocess.PIPE" constant (default) which will create a new pipe and connect it, * the value "None" which will make the subprocess inherit the file descriptor from this process * the "subprocess.DEVNULL" constant which indicates that the special "os.devnull" file will be used * the "subprocess.STDOUT" constant which will connect the standard error stream to the process’ standard output stream * All other keyword arguments are passed to "subprocess.Popen" without interpretation, except for *bufsize*, *universal_newlines*, *shell*, *text*, *encoding* and *errors*, which should not be specified at all. The "asyncio" subprocess API does not support decoding the streams as text. "bytes.decode()" can be used to convert the bytes returned from the stream to text. If a file-like object passed as *stdin*, *stdout* or *stderr* represents a pipe, then the other side of this pipe should be registered with "connect_write_pipe()" or "connect_read_pipe()" for use with the event loop. See the constructor of the "subprocess.Popen" class for documentation on other arguments. Returns a pair of "(transport, protocol)", where *transport* conforms to the "asyncio.SubprocessTransport" base class and *protocol* is an object instantiated by the *protocol_factory*. async loop.subprocess_shell(protocol_factory, cmd, *, stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE, **kwargs) Create a subprocess from *cmd*, which can be a "str" or a "bytes" string encoded to the filesystem encoding, using the platform’s “shell” syntax. This is similar to the standard library "subprocess.Popen" class called with "shell=True". The *protocol_factory* must be a callable returning a subclass of the "SubprocessProtocol" class. See "subprocess_exec()" for more details about the remaining arguments. Returns a pair of "(transport, protocol)", where *transport* conforms to the "SubprocessTransport" base class and *protocol* is an object instantiated by the *protocol_factory*. Note: It is the application’s responsibility to ensure that all whitespace and special characters are quoted appropriately to avoid shell injection vulnerabilities. The "shlex.quote()" function can be used to properly escape whitespace and special characters in strings that are going to be used to construct shell commands. Callback Handles ================ class asyncio.Handle A callback wrapper object returned by "loop.call_soon()", "loop.call_soon_threadsafe()". get_context() Return the "contextvars.Context" object associated with the handle. Added in version 3.12. cancel() Cancel the callback. If the callback has already been canceled or executed, this method has no effect. cancelled() Return "True" if the callback was cancelled. Added in version 3.7. class asyncio.TimerHandle A callback wrapper object returned by "loop.call_later()", and "loop.call_at()". This class is a subclass of "Handle". when() Return a scheduled callback time as "float" seconds. The time is an absolute timestamp, using the same time reference as "loop.time()". Added in version 3.7. Server Objects ============== Server objects are created by "loop.create_server()", "loop.create_unix_server()", "start_server()", and "start_unix_server()" functions. Do not instantiate the "Server" class directly. class asyncio.Server *Server* objects are asynchronous context managers. When used in an "async with" statement, it’s guaranteed that the Server object is closed and not accepting new connections when the "async with" statement is completed: srv = await loop.create_server(...) async with srv: # some code # At this point, srv is closed and no longer accepts new connections. Changed in version 3.7: Server object is an asynchronous context manager since Python 3.7. Changed in version 3.11: This class was exposed publicly as "asyncio.Server" in Python 3.9.11, 3.10.3 and 3.11. close() Stop serving: close listening sockets and set the "sockets" attribute to "None". The sockets that represent existing incoming client connections are left open. The server is closed asynchronously; use the "wait_closed()" coroutine to wait until the server is closed (and no more connections are active). close_clients() Close all existing incoming client connections. Calls "close()" on all associated transports. "close()" should be called before "close_clients()" when closing the server to avoid races with new clients connecting. Added in version 3.13. abort_clients() Close all existing incoming client connections immediately, without waiting for pending operations to complete. Calls "abort()" on all associated transports. "close()" should be called before "abort_clients()" when closing the server to avoid races with new clients connecting. Added in version 3.13. get_loop() Return the event loop associated with the server object. Added in version 3.7. async start_serving() Start accepting connections. This method is idempotent, so it can be called when the server is already serving. The *start_serving* keyword-only parameter to "loop.create_server()" and "asyncio.start_server()" allows creating a Server object that is not accepting connections initially. In this case "Server.start_serving()", or "Server.serve_forever()" can be used to make the Server start accepting connections. Added in version 3.7. async serve_forever() Start accepting connections until the coroutine is cancelled. Cancellation of "serve_forever" task causes the server to be closed. This method can be called if the server is already accepting connections. Only one "serve_forever" task can exist per one *Server* object. Example: async def client_connected(reader, writer): # Communicate with the client with # reader/writer streams. For example: await reader.readline() async def main(host, port): srv = await asyncio.start_server( client_connected, host, port) await srv.serve_forever() asyncio.run(main('127.0.0.1', 0)) Added in version 3.7. is_serving() Return "True" if the server is accepting new connections. Added in version 3.7. async wait_closed() Wait until the "close()" method completes and all active connections have finished. sockets List of socket-like objects, "asyncio.trsock.TransportSocket", which the server is listening on. Changed in version 3.7: Prior to Python 3.7 "Server.sockets" used to return an internal list of server sockets directly. In 3.7 a copy of that list is returned. Event Loop Implementations ========================== asyncio ships with two different event loop implementations: "SelectorEventLoop" and "ProactorEventLoop". By default asyncio is configured to use "EventLoop". class asyncio.SelectorEventLoop A subclass of "AbstractEventLoop" based on the "selectors" module. Uses the most efficient *selector* available for the given platform. It is also possible to manually configure the exact selector implementation to be used: import asyncio import selectors class MyPolicy(asyncio.DefaultEventLoopPolicy): def new_event_loop(self): selector = selectors.SelectSelector() return asyncio.SelectorEventLoop(selector) asyncio.set_event_loop_policy(MyPolicy()) Availability: Unix, Windows. class asyncio.ProactorEventLoop A subclass of "AbstractEventLoop" for Windows that uses “I/O Completion Ports” (IOCP). Availability: Windows. See also: MSDN documentation on I/O Completion Ports. class asyncio.EventLoop An alias to the most efficient available subclass of "AbstractEventLoop" for the given platform. It is an alias to "SelectorEventLoop" on Unix and "ProactorEventLoop" on Windows. Added in version 3.13. class asyncio.AbstractEventLoop Abstract base class for asyncio-compliant event loops. The Event Loop Methods section lists all methods that an alternative implementation of "AbstractEventLoop" should have defined. Examples ======== Note that all examples in this section **purposefully** show how to use the low-level event loop APIs, such as "loop.run_forever()" and "loop.call_soon()". Modern asyncio applications rarely need to be written this way; consider using the high-level functions like "asyncio.run()". Hello World with call_soon() ---------------------------- An example using the "loop.call_soon()" method to schedule a callback. The callback displays ""Hello World"" and then stops the event loop: import asyncio def hello_world(loop): """A callback to print 'Hello World' and stop the event loop""" print('Hello World') loop.stop() loop = asyncio.new_event_loop() # Schedule a call to hello_world() loop.call_soon(hello_world, loop) # Blocking call interrupted by loop.stop() try: loop.run_forever() finally: loop.close() See also: A similar Hello World example created with a coroutine and the "run()" function. Display the current date with call_later() ------------------------------------------ An example of a callback displaying the current date every second. The callback uses the "loop.call_later()" method to reschedule itself after 5 seconds, and then stops the event loop: import asyncio import datetime def display_date(end_time, loop): print(datetime.datetime.now()) if (loop.time() + 1.0) < end_time: loop.call_later(1, display_date, end_time, loop) else: loop.stop() loop = asyncio.new_event_loop() # Schedule the first call to display_date() end_time = loop.time() + 5.0 loop.call_soon(display_date, end_time, loop) # Blocking call interrupted by loop.stop() try: loop.run_forever() finally: loop.close() See also: A similar current date example created with a coroutine and the "run()" function. Watch a file descriptor for read events --------------------------------------- Wait until a file descriptor received some data using the "loop.add_reader()" method and then close the event loop: import asyncio from socket import socketpair # Create a pair of connected file descriptors rsock, wsock = socketpair() loop = asyncio.new_event_loop() def reader(): data = rsock.recv(100) print("Received:", data.decode()) # We are done: unregister the file descriptor loop.remove_reader(rsock) # Stop the event loop loop.stop() # Register the file descriptor for read event loop.add_reader(rsock, reader) # Simulate the reception of data from the network loop.call_soon(wsock.send, 'abc'.encode()) try: # Run the event loop loop.run_forever() finally: # We are done. Close sockets and the event loop. rsock.close() wsock.close() loop.close() See also: * A similar example using transports, protocols, and the "loop.create_connection()" method. * Another similar example using the high-level "asyncio.open_connection()" function and streams. Set signal handlers for SIGINT and SIGTERM ------------------------------------------ (This "signals" example only works on Unix.) Register handlers for signals "SIGINT" and "SIGTERM" using the "loop.add_signal_handler()" method: import asyncio import functools import os import signal def ask_exit(signame, loop): print("got signal %s: exit" % signame) loop.stop() async def main(): loop = asyncio.get_running_loop() for signame in {'SIGINT', 'SIGTERM'}: loop.add_signal_handler( getattr(signal, signame), functools.partial(ask_exit, signame, loop)) await asyncio.sleep(3600) print("Event loop running for 1 hour, press Ctrl+C to interrupt.") print(f"pid {os.getpid()}: send SIGINT or SIGTERM to exit.") asyncio.run(main()) Exceptions ********** **Source code:** Lib/asyncio/exceptions.py ====================================================================== exception asyncio.TimeoutError A deprecated alias of "TimeoutError", raised when the operation has exceeded the given deadline. Changed in version 3.11: This class was made an alias of "TimeoutError". exception asyncio.CancelledError The operation has been cancelled. This exception can be caught to perform custom operations when asyncio Tasks are cancelled. In almost all situations the exception must be re-raised. Changed in version 3.8: "CancelledError" is now a subclass of "BaseException" rather than "Exception". exception asyncio.InvalidStateError Invalid internal state of "Task" or "Future". Can be raised in situations like setting a result value for a *Future* object that already has a result value set. exception asyncio.SendfileNotAvailableError The “sendfile” syscall is not available for the given socket or file type. A subclass of "RuntimeError". exception asyncio.IncompleteReadError The requested read operation did not complete fully. Raised by the asyncio stream APIs. This exception is a subclass of "EOFError". expected The total number ("int") of expected bytes. partial A string of "bytes" read before the end of stream was reached. exception asyncio.LimitOverrunError Reached the buffer size limit while looking for a separator. Raised by the asyncio stream APIs. consumed The total number of to be consumed bytes. Extending ********* The main direction for "asyncio" extending is writing custom *event loop* classes. Asyncio has helpers that could be used to simplify this task. Note: Third-parties should reuse existing asyncio code with caution, a new Python version is free to break backward compatibility in *internal* part of API. Writing a Custom Event Loop =========================== "asyncio.AbstractEventLoop" declares very many methods. Implementing all them from scratch is a tedious job. A loop can get many common methods implementation for free by inheriting from "asyncio.BaseEventLoop". In turn, the successor should implement a bunch of *private* methods declared but not implemented in "asyncio.BaseEventLoop". For example, "loop.create_connection()" checks arguments, resolves DNS addresses, and calls "loop._make_socket_transport()" that should be implemented by inherited class. The "_make_socket_transport()" method is not documented and is considered as an *internal* API. Future and Task private constructors ==================================== "asyncio.Future" and "asyncio.Task" should be never created directly, please use corresponding "loop.create_future()" and "loop.create_task()", or "asyncio.create_task()" factories instead. However, third-party *event loops* may *reuse* built-in future and task implementations for the sake of getting a complex and highly optimized code for free. For this purpose the following, *private* constructors are listed: Future.__init__(*, loop=None) Create a built-in future instance. *loop* is an optional event loop instance. Task.__init__(coro, *, loop=None, name=None, context=None) Create a built-in task instance. *loop* is an optional event loop instance. The rest of arguments are described in "loop.create_task()" description. Changed in version 3.11: *context* argument is added. Task lifetime support ===================== A third party task implementation should call the following functions to keep a task visible by "asyncio.all_tasks()" and "asyncio.current_task()": asyncio._register_task(task) Register a new *task* as managed by *asyncio*. Call the function from a task constructor. asyncio._unregister_task(task) Unregister a *task* from *asyncio* internal structures. The function should be called when a task is about to finish. asyncio._enter_task(loop, task) Switch the current task to the *task* argument. Call the function just before executing a portion of embedded *coroutine* ("coroutine.send()" or "coroutine.throw()"). asyncio._leave_task(loop, task) Switch the current task back from *task* to "None". Call the function just after "coroutine.send()" or "coroutine.throw()" execution. Futures ******* **Source code:** Lib/asyncio/futures.py, Lib/asyncio/base_futures.py ====================================================================== *Future* objects are used to bridge **low-level callback-based code** with high-level async/await code. Future Functions ================ asyncio.isfuture(obj) Return "True" if *obj* is either of: * an instance of "asyncio.Future", * an instance of "asyncio.Task", * a Future-like object with a "_asyncio_future_blocking" attribute. Added in version 3.5. asyncio.ensure_future(obj, *, loop=None) Return: * *obj* argument as is, if *obj* is a "Future", a "Task", or a Future-like object ("isfuture()" is used for the test.) * a "Task" object wrapping *obj*, if *obj* is a coroutine ("iscoroutine()" is used for the test); in this case the coroutine will be scheduled by "ensure_future()". * a "Task" object that would await on *obj*, if *obj* is an awaitable ("inspect.isawaitable()" is used for the test.) If *obj* is neither of the above a "TypeError" is raised. Important: See also the "create_task()" function which is the preferred way for creating new Tasks.Save a reference to the result of this function, to avoid a task disappearing mid-execution. Changed in version 3.5.1: The function accepts any *awaitable* object. Deprecated since version 3.10: Deprecation warning is emitted if *obj* is not a Future-like object and *loop* is not specified and there is no running event loop. asyncio.wrap_future(future, *, loop=None) Wrap a "concurrent.futures.Future" object in a "asyncio.Future" object. Deprecated since version 3.10: Deprecation warning is emitted if *future* is not a Future-like object and *loop* is not specified and there is no running event loop. Future Object ============= class asyncio.Future(*, loop=None) A Future represents an eventual result of an asynchronous operation. Not thread-safe. Future is an *awaitable* object. Coroutines can await on Future objects until they either have a result or an exception set, or until they are cancelled. A Future can be awaited multiple times and the result is same. Typically Futures are used to enable low-level callback-based code (e.g. in protocols implemented using asyncio transports) to interoperate with high-level async/await code. The rule of thumb is to never expose Future objects in user-facing APIs, and the recommended way to create a Future object is to call "loop.create_future()". This way alternative event loop implementations can inject their own optimized implementations of a Future object. Changed in version 3.7: Added support for the "contextvars" module. Deprecated since version 3.10: Deprecation warning is emitted if *loop* is not specified and there is no running event loop. result() Return the result of the Future. If the Future is *done* and has a result set by the "set_result()" method, the result value is returned. If the Future is *done* and has an exception set by the "set_exception()" method, this method raises the exception. If the Future has been *cancelled*, this method raises a "CancelledError" exception. If the Future’s result isn’t yet available, this method raises an "InvalidStateError" exception. set_result(result) Mark the Future as *done* and set its result. Raises an "InvalidStateError" error if the Future is already *done*. set_exception(exception) Mark the Future as *done* and set an exception. Raises an "InvalidStateError" error if the Future is already *done*. done() Return "True" if the Future is *done*. A Future is *done* if it was *cancelled* or if it has a result or an exception set with "set_result()" or "set_exception()" calls. cancelled() Return "True" if the Future was *cancelled*. The method is usually used to check if a Future is not *cancelled* before setting a result or an exception for it: if not fut.cancelled(): fut.set_result(42) add_done_callback(callback, *, context=None) Add a callback to be run when the Future is *done*. The *callback* is called with the Future object as its only argument. If the Future is already *done* when this method is called, the callback is scheduled with "loop.call_soon()". An optional keyword-only *context* argument allows specifying a custom "contextvars.Context" for the *callback* to run in. The current context is used when no *context* is provided. "functools.partial()" can be used to pass parameters to the callback, e.g.: # Call 'print("Future:", fut)' when "fut" is done. fut.add_done_callback( functools.partial(print, "Future:")) Changed in version 3.7: The *context* keyword-only parameter was added. See **PEP 567** for more details. remove_done_callback(callback) Remove *callback* from the callbacks list. Returns the number of callbacks removed, which is typically 1, unless a callback was added more than once. cancel(msg=None) Cancel the Future and schedule callbacks. If the Future is already *done* or *cancelled*, return "False". Otherwise, change the Future’s state to *cancelled*, schedule the callbacks, and return "True". Changed in version 3.9: Added the *msg* parameter. exception() Return the exception that was set on this Future. The exception (or "None" if no exception was set) is returned only if the Future is *done*. If the Future has been *cancelled*, this method raises a "CancelledError" exception. If the Future isn’t *done* yet, this method raises an "InvalidStateError" exception. get_loop() Return the event loop the Future object is bound to. Added in version 3.7. This example creates a Future object, creates and schedules an asynchronous Task to set result for the Future, and waits until the Future has a result: async def set_after(fut, delay, value): # Sleep for *delay* seconds. await asyncio.sleep(delay) # Set *value* as a result of *fut* Future. fut.set_result(value) async def main(): # Get the current event loop. loop = asyncio.get_running_loop() # Create a new Future object. fut = loop.create_future() # Run "set_after()" coroutine in a parallel Task. # We are using the low-level "loop.create_task()" API here because # we already have a reference to the event loop at hand. # Otherwise we could have just used "asyncio.create_task()". loop.create_task( set_after(fut, 1, '... world')) print('hello ...') # Wait until *fut* has a result (1 second) and print it. print(await fut) asyncio.run(main()) Important: The Future object was designed to mimic "concurrent.futures.Future". Key differences include: * unlike asyncio Futures, "concurrent.futures.Future" instances cannot be awaited. * "asyncio.Future.result()" and "asyncio.Future.exception()" do not accept the *timeout* argument. * "asyncio.Future.result()" and "asyncio.Future.exception()" raise an "InvalidStateError" exception when the Future is not *done*. * Callbacks registered with "asyncio.Future.add_done_callback()" are not called immediately. They are scheduled with "loop.call_soon()" instead. * asyncio Future is not compatible with the "concurrent.futures.wait()" and "concurrent.futures.as_completed()" functions. * "asyncio.Future.cancel()" accepts an optional "msg" argument, but "concurrent.futures.Future.cancel()" does not. Low-level API Index ******************* This page lists all low-level asyncio APIs. Obtaining the Event Loop ======================== +----------------------------------------------------+----------------------------------------------------+ | "asyncio.get_running_loop()" | The **preferred** function to get the running | | | event loop. | +----------------------------------------------------+----------------------------------------------------+ | "asyncio.get_event_loop()" | Get an event loop instance (running or current via | | | the current policy). | +----------------------------------------------------+----------------------------------------------------+ | "asyncio.set_event_loop()" | Set the event loop as current via the current | | | policy. | +----------------------------------------------------+----------------------------------------------------+ | "asyncio.new_event_loop()" | Create a new event loop. | +----------------------------------------------------+----------------------------------------------------+ -[ Examples ]- * Using asyncio.get_running_loop(). Event Loop Methods ================== See also the main documentation section about the Event Loop Methods. -[ Lifecycle ]- +----------------------------------------------------+----------------------------------------------------+ | "loop.run_until_complete()" | Run a Future/Task/awaitable until complete. | +----------------------------------------------------+----------------------------------------------------+ | "loop.run_forever()" | Run the event loop forever. | +----------------------------------------------------+----------------------------------------------------+ | "loop.stop()" | Stop the event loop. | +----------------------------------------------------+----------------------------------------------------+ | "loop.close()" | Close the event loop. | +----------------------------------------------------+----------------------------------------------------+ | "loop.is_running()" | Return "True" if the event loop is running. | +----------------------------------------------------+----------------------------------------------------+ | "loop.is_closed()" | Return "True" if the event loop is closed. | +----------------------------------------------------+----------------------------------------------------+ | "await" "loop.shutdown_asyncgens()" | Close asynchronous generators. | +----------------------------------------------------+----------------------------------------------------+ -[ Debugging ]- +----------------------------------------------------+----------------------------------------------------+ | "loop.set_debug()" | Enable or disable the debug mode. | +----------------------------------------------------+----------------------------------------------------+ | "loop.get_debug()" | Get the current debug mode. | +----------------------------------------------------+----------------------------------------------------+ -[ Scheduling Callbacks ]- +----------------------------------------------------+----------------------------------------------------+ | "loop.call_soon()" | Invoke a callback soon. | +----------------------------------------------------+----------------------------------------------------+ | "loop.call_soon_threadsafe()" | A thread-safe variant of "loop.call_soon()". | +----------------------------------------------------+----------------------------------------------------+ | "loop.call_later()" | Invoke a callback *after* the given time. | +----------------------------------------------------+----------------------------------------------------+ | "loop.call_at()" | Invoke a callback *at* the given time. | +----------------------------------------------------+----------------------------------------------------+ -[ Thread/Process Pool ]- +----------------------------------------------------+----------------------------------------------------+ | "await" "loop.run_in_executor()" | Run a CPU-bound or other blocking function in a | | | "concurrent.futures" executor. | +----------------------------------------------------+----------------------------------------------------+ | "loop.set_default_executor()" | Set the default executor for | | | "loop.run_in_executor()". | +----------------------------------------------------+----------------------------------------------------+ -[ Tasks and Futures ]- +----------------------------------------------------+----------------------------------------------------+ | "loop.create_future()" | Create a "Future" object. | +----------------------------------------------------+----------------------------------------------------+ | "loop.create_task()" | Schedule coroutine as a "Task". | +----------------------------------------------------+----------------------------------------------------+ | "loop.set_task_factory()" | Set a factory used by "loop.create_task()" to | | | create "Tasks". | +----------------------------------------------------+----------------------------------------------------+ | "loop.get_task_factory()" | Get the factory "loop.create_task()" uses to | | | create "Tasks". | +----------------------------------------------------+----------------------------------------------------+ -[ DNS ]- +----------------------------------------------------+----------------------------------------------------+ | "await" "loop.getaddrinfo()" | Asynchronous version of "socket.getaddrinfo()". | +----------------------------------------------------+----------------------------------------------------+ | "await" "loop.getnameinfo()" | Asynchronous version of "socket.getnameinfo()". | +----------------------------------------------------+----------------------------------------------------+ -[ Networking and IPC ]- +----------------------------------------------------+----------------------------------------------------+ | "await" "loop.create_connection()" | Open a TCP connection. | +----------------------------------------------------+----------------------------------------------------+ | "await" "loop.create_server()" | Create a TCP server. | +----------------------------------------------------+----------------------------------------------------+ | "await" "loop.create_unix_connection()" | Open a Unix socket connection. | +----------------------------------------------------+----------------------------------------------------+ | "await" "loop.create_unix_server()" | Create a Unix socket server. | +----------------------------------------------------+----------------------------------------------------+ | "await" "loop.connect_accepted_socket()" | Wrap a "socket" into a "(transport, protocol)" | | | pair. | +----------------------------------------------------+----------------------------------------------------+ | "await" "loop.create_datagram_endpoint()" | Open a datagram (UDP) connection. | +----------------------------------------------------+----------------------------------------------------+ | "await" "loop.sendfile()" | Send a file over a transport. | +----------------------------------------------------+----------------------------------------------------+ | "await" "loop.start_tls()" | Upgrade an existing connection to TLS. | +----------------------------------------------------+----------------------------------------------------+ | "await" "loop.connect_read_pipe()" | Wrap a read end of a pipe into a "(transport, | | | protocol)" pair. | +----------------------------------------------------+----------------------------------------------------+ | "await" "loop.connect_write_pipe()" | Wrap a write end of a pipe into a "(transport, | | | protocol)" pair. | +----------------------------------------------------+----------------------------------------------------+ -[ Sockets ]- +----------------------------------------------------+----------------------------------------------------+ | "await" "loop.sock_recv()" | Receive data from the "socket". | +----------------------------------------------------+----------------------------------------------------+ | "await" "loop.sock_recv_into()" | Receive data from the "socket" into a buffer. | +----------------------------------------------------+----------------------------------------------------+ | "await" "loop.sock_recvfrom()" | Receive a datagram from the "socket". | +----------------------------------------------------+----------------------------------------------------+ | "await" "loop.sock_recvfrom_into()" | Receive a datagram from the "socket" into a | | | buffer. | +----------------------------------------------------+----------------------------------------------------+ | "await" "loop.sock_sendall()" | Send data to the "socket". | +----------------------------------------------------+----------------------------------------------------+ | "await" "loop.sock_sendto()" | Send a datagram via the "socket" to the given | | | address. | +----------------------------------------------------+----------------------------------------------------+ | "await" "loop.sock_connect()" | Connect the "socket". | +----------------------------------------------------+----------------------------------------------------+ | "await" "loop.sock_accept()" | Accept a "socket" connection. | +----------------------------------------------------+----------------------------------------------------+ | "await" "loop.sock_sendfile()" | Send a file over the "socket". | +----------------------------------------------------+----------------------------------------------------+ | "loop.add_reader()" | Start watching a file descriptor for read | | | availability. | +----------------------------------------------------+----------------------------------------------------+ | "loop.remove_reader()" | Stop watching a file descriptor for read | | | availability. | +----------------------------------------------------+----------------------------------------------------+ | "loop.add_writer()" | Start watching a file descriptor for write | | | availability. | +----------------------------------------------------+----------------------------------------------------+ | "loop.remove_writer()" | Stop watching a file descriptor for write | | | availability. | +----------------------------------------------------+----------------------------------------------------+ -[ Unix Signals ]- +----------------------------------------------------+----------------------------------------------------+ | "loop.add_signal_handler()" | Add a handler for a "signal". | +----------------------------------------------------+----------------------------------------------------+ | "loop.remove_signal_handler()" | Remove a handler for a "signal". | +----------------------------------------------------+----------------------------------------------------+ -[ Subprocesses ]- +----------------------------------------------------+----------------------------------------------------+ | "loop.subprocess_exec()" | Spawn a subprocess. | +----------------------------------------------------+----------------------------------------------------+ | "loop.subprocess_shell()" | Spawn a subprocess from a shell command. | +----------------------------------------------------+----------------------------------------------------+ -[ Error Handling ]- +----------------------------------------------------+----------------------------------------------------+ | "loop.call_exception_handler()" | Call the exception handler. | +----------------------------------------------------+----------------------------------------------------+ | "loop.set_exception_handler()" | Set a new exception handler. | +----------------------------------------------------+----------------------------------------------------+ | "loop.get_exception_handler()" | Get the current exception handler. | +----------------------------------------------------+----------------------------------------------------+ | "loop.default_exception_handler()" | The default exception handler implementation. | +----------------------------------------------------+----------------------------------------------------+ -[ Examples ]- * Using asyncio.new_event_loop() and loop.run_forever(). * Using loop.call_later(). * Using "loop.create_connection()" to implement an echo-client. * Using "loop.create_connection()" to connect a socket. * Using add_reader() to watch an FD for read events. * Using loop.add_signal_handler(). * Using loop.subprocess_exec(). Transports ========== All transports implement the following methods: +----------------------------------------------------+----------------------------------------------------+ | "transport.close()" | Close the transport. | +----------------------------------------------------+----------------------------------------------------+ | "transport.is_closing()" | Return "True" if the transport is closing or is | | | closed. | +----------------------------------------------------+----------------------------------------------------+ | "transport.get_extra_info()" | Request for information about the transport. | +----------------------------------------------------+----------------------------------------------------+ | "transport.set_protocol()" | Set a new protocol. | +----------------------------------------------------+----------------------------------------------------+ | "transport.get_protocol()" | Return the current protocol. | +----------------------------------------------------+----------------------------------------------------+ Transports that can receive data (TCP and Unix connections, pipes, etc). Returned from methods like "loop.create_connection()", "loop.create_unix_connection()", "loop.connect_read_pipe()", etc: -[ Read Transports ]- +----------------------------------------------------+----------------------------------------------------+ | "transport.is_reading()" | Return "True" if the transport is receiving. | +----------------------------------------------------+----------------------------------------------------+ | "transport.pause_reading()" | Pause receiving. | +----------------------------------------------------+----------------------------------------------------+ | "transport.resume_reading()" | Resume receiving. | +----------------------------------------------------+----------------------------------------------------+ Transports that can Send data (TCP and Unix connections, pipes, etc). Returned from methods like "loop.create_connection()", "loop.create_unix_connection()", "loop.connect_write_pipe()", etc: -[ Write Transports ]- +----------------------------------------------------+----------------------------------------------------+ | "transport.write()" | Write data to the transport. | +----------------------------------------------------+----------------------------------------------------+ | "transport.writelines()" | Write buffers to the transport. | +----------------------------------------------------+----------------------------------------------------+ | "transport.can_write_eof()" | Return "True" if the transport supports sending | | | EOF. | +----------------------------------------------------+----------------------------------------------------+ | "transport.write_eof()" | Close and send EOF after flushing buffered data. | +----------------------------------------------------+----------------------------------------------------+ | "transport.abort()" | Close the transport immediately. | +----------------------------------------------------+----------------------------------------------------+ | "transport.get_write_buffer_size()" | Return the current size of the output buffer. | +----------------------------------------------------+----------------------------------------------------+ | "transport.get_write_buffer_limits()" | Return high and low water marks for write flow | | | control. | +----------------------------------------------------+----------------------------------------------------+ | "transport.set_write_buffer_limits()" | Set new high and low water marks for write flow | | | control. | +----------------------------------------------------+----------------------------------------------------+ Transports returned by "loop.create_datagram_endpoint()": -[ Datagram Transports ]- +----------------------------------------------------+----------------------------------------------------+ | "transport.sendto()" | Send data to the remote peer. | +----------------------------------------------------+----------------------------------------------------+ | "transport.abort()" | Close the transport immediately. | +----------------------------------------------------+----------------------------------------------------+ Low-level transport abstraction over subprocesses. Returned by "loop.subprocess_exec()" and "loop.subprocess_shell()": -[ Subprocess Transports ]- +----------------------------------------------------+----------------------------------------------------+ | "transport.get_pid()" | Return the subprocess process id. | +----------------------------------------------------+----------------------------------------------------+ | "transport.get_pipe_transport()" | Return the transport for the requested | | | communication pipe (*stdin*, *stdout*, or | | | *stderr*). | +----------------------------------------------------+----------------------------------------------------+ | "transport.get_returncode()" | Return the subprocess return code. | +----------------------------------------------------+----------------------------------------------------+ | "transport.kill()" | Kill the subprocess. | +----------------------------------------------------+----------------------------------------------------+ | "transport.send_signal()" | Send a signal to the subprocess. | +----------------------------------------------------+----------------------------------------------------+ | "transport.terminate()" | Stop the subprocess. | +----------------------------------------------------+----------------------------------------------------+ | "transport.close()" | Kill the subprocess and close all pipes. | +----------------------------------------------------+----------------------------------------------------+ Protocols ========= Protocol classes can implement the following **callback methods**: +----------------------------------------------------+----------------------------------------------------+ | "callback" "connection_made()" | Called when a connection is made. | +----------------------------------------------------+----------------------------------------------------+ | "callback" "connection_lost()" | Called when the connection is lost or closed. | +----------------------------------------------------+----------------------------------------------------+ | "callback" "pause_writing()" | Called when the transport’s buffer goes over the | | | high water mark. | +----------------------------------------------------+----------------------------------------------------+ | "callback" "resume_writing()" | Called when the transport’s buffer drains below | | | the low water mark. | +----------------------------------------------------+----------------------------------------------------+ -[ Streaming Protocols (TCP, Unix Sockets, Pipes) ]- +----------------------------------------------------+----------------------------------------------------+ | "callback" "data_received()" | Called when some data is received. | +----------------------------------------------------+----------------------------------------------------+ | "callback" "eof_received()" | Called when an EOF is received. | +----------------------------------------------------+----------------------------------------------------+ -[ Buffered Streaming Protocols ]- +----------------------------------------------------+----------------------------------------------------+ | "callback" "get_buffer()" | Called to allocate a new receive buffer. | +----------------------------------------------------+----------------------------------------------------+ | "callback" "buffer_updated()" | Called when the buffer was updated with the | | | received data. | +----------------------------------------------------+----------------------------------------------------+ | "callback" "eof_received()" | Called when an EOF is received. | +----------------------------------------------------+----------------------------------------------------+ -[ Datagram Protocols ]- +----------------------------------------------------+----------------------------------------------------+ | "callback" "datagram_received()" | Called when a datagram is received. | +----------------------------------------------------+----------------------------------------------------+ | "callback" "error_received()" | Called when a previous send or receive operation | | | raises an "OSError". | +----------------------------------------------------+----------------------------------------------------+ -[ Subprocess Protocols ]- +----------------------------------------------------+----------------------------------------------------+ | "callback" "pipe_data_received()" | Called when the child process writes data into its | | | *stdout* or *stderr* pipe. | +----------------------------------------------------+----------------------------------------------------+ | "callback" "pipe_connection_lost()" | Called when one of the pipes communicating with | | | the child process is closed. | +----------------------------------------------------+----------------------------------------------------+ | "callback" "process_exited()" | Called when the child process has exited. It can | | | be called before "pipe_data_received()" and | | | "pipe_connection_lost()" methods. | +----------------------------------------------------+----------------------------------------------------+ Event Loop Policies =================== Policies is a low-level mechanism to alter the behavior of functions like "asyncio.get_event_loop()". See also the main policies section for more details. -[ Accessing Policies ]- +----------------------------------------------------+----------------------------------------------------+ | "asyncio.get_event_loop_policy()" | Return the current process-wide policy. | +----------------------------------------------------+----------------------------------------------------+ | "asyncio.set_event_loop_policy()" | Set a new process-wide policy. | +----------------------------------------------------+----------------------------------------------------+ | "AbstractEventLoopPolicy" | Base class for policy objects. | +----------------------------------------------------+----------------------------------------------------+ Platform Support **************** The "asyncio" module is designed to be portable, but some platforms have subtle differences and limitations due to the platforms’ underlying architecture and capabilities. All Platforms ============= * "loop.add_reader()" and "loop.add_writer()" cannot be used to monitor file I/O. Windows ======= **Source code:** Lib/asyncio/proactor_events.py, Lib/asyncio/windows_events.py, Lib/asyncio/windows_utils.py ====================================================================== Changed in version 3.8: On Windows, "ProactorEventLoop" is now the default event loop. All event loops on Windows do not support the following methods: * "loop.create_unix_connection()" and "loop.create_unix_server()" are not supported. The "socket.AF_UNIX" socket family is specific to Unix. * "loop.add_signal_handler()" and "loop.remove_signal_handler()" are not supported. "SelectorEventLoop" has the following limitations: * "SelectSelector" is used to wait on socket events: it supports sockets and is limited to 512 sockets. * "loop.add_reader()" and "loop.add_writer()" only accept socket handles (e.g. pipe file descriptors are not supported). * Pipes are not supported, so the "loop.connect_read_pipe()" and "loop.connect_write_pipe()" methods are not implemented. * Subprocesses are not supported, i.e. "loop.subprocess_exec()" and "loop.subprocess_shell()" methods are not implemented. "ProactorEventLoop" has the following limitations: * The "loop.add_reader()" and "loop.add_writer()" methods are not supported. The resolution of the monotonic clock on Windows is usually around 15.6 milliseconds. The best resolution is 0.5 milliseconds. The resolution depends on the hardware (availability of HPET) and on the Windows configuration. Subprocess Support on Windows ----------------------------- On Windows, the default event loop "ProactorEventLoop" supports subprocesses, whereas "SelectorEventLoop" does not. The "policy.set_child_watcher()" function is also not supported, as "ProactorEventLoop" has a different mechanism to watch child processes. macOS ===== Modern macOS versions are fully supported. -[ macOS <= 10.8 ]- On macOS 10.6, 10.7 and 10.8, the default event loop uses "selectors.KqueueSelector", which does not support character devices on these versions. The "SelectorEventLoop" can be manually configured to use "SelectSelector" or "PollSelector" to support character devices on these older versions of macOS. Example: import asyncio import selectors selector = selectors.SelectSelector() loop = asyncio.SelectorEventLoop(selector) asyncio.set_event_loop(loop) Policies ******** An event loop policy is a global object used to get and set the current event loop, as well as create new event loops. The default policy can be replaced with built-in alternatives to use different event loop implementations, or substituted by a custom policy that can override these behaviors. The policy object gets and sets a separate event loop per *context*. This is per-thread by default, though custom policies could define *context* differently. Custom event loop policies can control the behavior of "get_event_loop()", "set_event_loop()", and "new_event_loop()". Policy objects should implement the APIs defined in the "AbstractEventLoopPolicy" abstract base class. Getting and Setting the Policy ============================== The following functions can be used to get and set the policy for the current process: asyncio.get_event_loop_policy() Return the current process-wide policy. asyncio.set_event_loop_policy(policy) Set the current process-wide policy to *policy*. If *policy* is set to "None", the default policy is restored. Policy Objects ============== The abstract event loop policy base class is defined as follows: class asyncio.AbstractEventLoopPolicy An abstract base class for asyncio policies. get_event_loop() Get the event loop for the current context. Return an event loop object implementing the "AbstractEventLoop" interface. This method should never return "None". Changed in version 3.6. set_event_loop(loop) Set the event loop for the current context to *loop*. new_event_loop() Create and return a new event loop object. This method should never return "None". get_child_watcher() Get a child process watcher object. Return a watcher object implementing the "AbstractChildWatcher" interface. This function is Unix specific. Deprecated since version 3.12. set_child_watcher(watcher) Set the current child process watcher to *watcher*. This function is Unix specific. Deprecated since version 3.12. asyncio ships with the following built-in policies: class asyncio.DefaultEventLoopPolicy The default asyncio policy. Uses "SelectorEventLoop" on Unix and "ProactorEventLoop" on Windows. There is no need to install the default policy manually. asyncio is configured to use the default policy automatically. Changed in version 3.8: On Windows, "ProactorEventLoop" is now used by default. Deprecated since version 3.12: The "get_event_loop()" method of the default asyncio policy now emits a "DeprecationWarning" if there is no current event loop set and it decides to create one. In some future Python release this will become an error. class asyncio.WindowsSelectorEventLoopPolicy An alternative event loop policy that uses the "SelectorEventLoop" event loop implementation. Availability: Windows. class asyncio.WindowsProactorEventLoopPolicy An alternative event loop policy that uses the "ProactorEventLoop" event loop implementation. Availability: Windows. Process Watchers ================ A process watcher allows customization of how an event loop monitors child processes on Unix. Specifically, the event loop needs to know when a child process has exited. In asyncio, child processes are created with "create_subprocess_exec()" and "loop.subprocess_exec()" functions. asyncio defines the "AbstractChildWatcher" abstract base class, which child watchers should implement, and has four different implementations: "ThreadedChildWatcher" (configured to be used by default), "MultiLoopChildWatcher", "SafeChildWatcher", and "FastChildWatcher". See also the Subprocess and Threads section. The following two functions can be used to customize the child process watcher implementation used by the asyncio event loop: asyncio.get_child_watcher() Return the current child watcher for the current policy. Deprecated since version 3.12. asyncio.set_child_watcher(watcher) Set the current child watcher to *watcher* for the current policy. *watcher* must implement methods defined in the "AbstractChildWatcher" base class. Deprecated since version 3.12. Note: Third-party event loops implementations might not support custom child watchers. For such event loops, using "set_child_watcher()" might be prohibited or have no effect. class asyncio.AbstractChildWatcher add_child_handler(pid, callback, *args) Register a new child handler. Arrange for "callback(pid, returncode, *args)" to be called when a process with PID equal to *pid* terminates. Specifying another callback for the same process replaces the previous handler. The *callback* callable must be thread-safe. remove_child_handler(pid) Removes the handler for process with PID equal to *pid*. The function returns "True" if the handler was successfully removed, "False" if there was nothing to remove. attach_loop(loop) Attach the watcher to an event loop. If the watcher was previously attached to an event loop, then it is first detached before attaching to the new loop. Note: loop may be "None". is_active() Return "True" if the watcher is ready to use. Spawning a subprocess with *inactive* current child watcher raises "RuntimeError". Added in version 3.8. close() Close the watcher. This method has to be called to ensure that underlying resources are cleaned-up. Deprecated since version 3.12. class asyncio.ThreadedChildWatcher This implementation starts a new waiting thread for every subprocess spawn. It works reliably even when the asyncio event loop is run in a non- main OS thread. There is no noticeable overhead when handling a big number of children (*O*(1) each time a child terminates), but starting a thread per process requires extra memory. This watcher is used by default. Added in version 3.8. class asyncio.MultiLoopChildWatcher This implementation registers a "SIGCHLD" signal handler on instantiation. That can break third-party code that installs a custom handler for "SIGCHLD" signal. The watcher avoids disrupting other code spawning processes by polling every process explicitly on a "SIGCHLD" signal. There is no limitation for running subprocesses from different threads once the watcher is installed. The solution is safe but it has a significant overhead when handling a big number of processes (*O*(*n*) each time a "SIGCHLD" is received). Added in version 3.8. Deprecated since version 3.12. class asyncio.SafeChildWatcher This implementation uses active event loop from the main thread to handle "SIGCHLD" signal. If the main thread has no running event loop another thread cannot spawn a subprocess ("RuntimeError" is raised). The watcher avoids disrupting other code spawning processes by polling every process explicitly on a "SIGCHLD" signal. This solution is as safe as "MultiLoopChildWatcher" and has the same *O*(*n*) complexity but requires a running event loop in the main thread to work. Deprecated since version 3.12. class asyncio.FastChildWatcher This implementation reaps every terminated processes by calling "os.waitpid(-1)" directly, possibly breaking other code spawning processes and waiting for their termination. There is no noticeable overhead when handling a big number of children (*O*(1) each time a child terminates). This solution requires a running event loop in the main thread to work, as "SafeChildWatcher". Deprecated since version 3.12. class asyncio.PidfdChildWatcher This implementation polls process file descriptors (pidfds) to await child process termination. In some respects, "PidfdChildWatcher" is a “Goldilocks” child watcher implementation. It doesn’t require signals or threads, doesn’t interfere with any processes launched outside the event loop, and scales linearly with the number of subprocesses launched by the event loop. The main disadvantage is that pidfds are specific to Linux, and only work on recent (5.3+) kernels. Added in version 3.9. Custom Policies =============== To implement a new event loop policy, it is recommended to subclass "DefaultEventLoopPolicy" and override the methods for which custom behavior is wanted, e.g.: class MyEventLoopPolicy(asyncio.DefaultEventLoopPolicy): def get_event_loop(self): """Get the event loop. This may be None or an instance of EventLoop. """ loop = super().get_event_loop() # Do something with loop ... return loop asyncio.set_event_loop_policy(MyEventLoopPolicy()) Transports and Protocols ************************ -[ Preface ]- Transports and Protocols are used by the **low-level** event loop APIs such as "loop.create_connection()". They use callback-based programming style and enable high-performance implementations of network or IPC protocols (e.g. HTTP). Essentially, transports and protocols should only be used in libraries and frameworks and never in high-level asyncio applications. This documentation page covers both Transports and Protocols. -[ Introduction ]- At the highest level, the transport is concerned with *how* bytes are transmitted, while the protocol determines *which* bytes to transmit (and to some extent when). A different way of saying the same thing: a transport is an abstraction for a socket (or similar I/O endpoint) while a protocol is an abstraction for an application, from the transport’s point of view. Yet another view is the transport and protocol interfaces together define an abstract interface for using network I/O and interprocess I/O. There is always a 1:1 relationship between transport and protocol objects: the protocol calls transport methods to send data, while the transport calls protocol methods to pass it data that has been received. Most of connection oriented event loop methods (such as "loop.create_connection()") usually accept a *protocol_factory* argument used to create a *Protocol* object for an accepted connection, represented by a *Transport* object. Such methods usually return a tuple of "(transport, protocol)". -[ Contents ]- This documentation page contains the following sections: * The Transports section documents asyncio "BaseTransport", "ReadTransport", "WriteTransport", "Transport", "DatagramTransport", and "SubprocessTransport" classes. * The Protocols section documents asyncio "BaseProtocol", "Protocol", "BufferedProtocol", "DatagramProtocol", and "SubprocessProtocol" classes. * The Examples section showcases how to work with transports, protocols, and low-level event loop APIs. Transports ========== **Source code:** Lib/asyncio/transports.py ====================================================================== Transports are classes provided by "asyncio" in order to abstract various kinds of communication channels. Transport objects are always instantiated by an asyncio event loop. asyncio implements transports for TCP, UDP, SSL, and subprocess pipes. The methods available on a transport depend on the transport’s kind. The transport classes are not thread safe. Transports Hierarchy -------------------- class asyncio.BaseTransport Base class for all transports. Contains methods that all asyncio transports share. class asyncio.WriteTransport(BaseTransport) A base transport for write-only connections. Instances of the *WriteTransport* class are returned from the "loop.connect_write_pipe()" event loop method and are also used by subprocess-related methods like "loop.subprocess_exec()". class asyncio.ReadTransport(BaseTransport) A base transport for read-only connections. Instances of the *ReadTransport* class are returned from the "loop.connect_read_pipe()" event loop method and are also used by subprocess-related methods like "loop.subprocess_exec()". class asyncio.Transport(WriteTransport, ReadTransport) Interface representing a bidirectional transport, such as a TCP connection. The user does not instantiate a transport directly; they call a utility function, passing it a protocol factory and other information necessary to create the transport and protocol. Instances of the *Transport* class are returned from or used by event loop methods like "loop.create_connection()", "loop.create_unix_connection()", "loop.create_server()", "loop.sendfile()", etc. class asyncio.DatagramTransport(BaseTransport) A transport for datagram (UDP) connections. Instances of the *DatagramTransport* class are returned from the "loop.create_datagram_endpoint()" event loop method. class asyncio.SubprocessTransport(BaseTransport) An abstraction to represent a connection between a parent and its child OS process. Instances of the *SubprocessTransport* class are returned from event loop methods "loop.subprocess_shell()" and "loop.subprocess_exec()". Base Transport -------------- BaseTransport.close() Close the transport. If the transport has a buffer for outgoing data, buffered data will be flushed asynchronously. No more data will be received. After all buffered data is flushed, the protocol’s "protocol.connection_lost()" method will be called with "None" as its argument. The transport should not be used once it is closed. BaseTransport.is_closing() Return "True" if the transport is closing or is closed. BaseTransport.get_extra_info(name, default=None) Return information about the transport or underlying resources it uses. *name* is a string representing the piece of transport-specific information to get. *default* is the value to return if the information is not available, or if the transport does not support querying it with the given third-party event loop implementation or on the current platform. For example, the following code attempts to get the underlying socket object of the transport: sock = transport.get_extra_info('socket') if sock is not None: print(sock.getsockopt(...)) Categories of information that can be queried on some transports: * socket: * "'peername'": the remote address to which the socket is connected, result of "socket.socket.getpeername()" ("None" on error) * "'socket'": "socket.socket" instance * "'sockname'": the socket’s own address, result of "socket.socket.getsockname()" * SSL socket: * "'compression'": the compression algorithm being used as a string, or "None" if the connection isn’t compressed; result of "ssl.SSLSocket.compression()" * "'cipher'": a three-value tuple containing the name of the cipher being used, the version of the SSL protocol that defines its use, and the number of secret bits being used; result of "ssl.SSLSocket.cipher()" * "'peercert'": peer certificate; result of "ssl.SSLSocket.getpeercert()" * "'sslcontext'": "ssl.SSLContext" instance * "'ssl_object'": "ssl.SSLObject" or "ssl.SSLSocket" instance * pipe: * "'pipe'": pipe object * subprocess: * "'subprocess'": "subprocess.Popen" instance BaseTransport.set_protocol(protocol) Set a new protocol. Switching protocol should only be done when both protocols are documented to support the switch. BaseTransport.get_protocol() Return the current protocol. Read-only Transports -------------------- ReadTransport.is_reading() Return "True" if the transport is receiving new data. Added in version 3.7. ReadTransport.pause_reading() Pause the receiving end of the transport. No data will be passed to the protocol’s "protocol.data_received()" method until "resume_reading()" is called. Changed in version 3.7: The method is idempotent, i.e. it can be called when the transport is already paused or closed. ReadTransport.resume_reading() Resume the receiving end. The protocol’s "protocol.data_received()" method will be called once again if some data is available for reading. Changed in version 3.7: The method is idempotent, i.e. it can be called when the transport is already reading. Write-only Transports --------------------- WriteTransport.abort() Close the transport immediately, without waiting for pending operations to complete. Buffered data will be lost. No more data will be received. The protocol’s "protocol.connection_lost()" method will eventually be called with "None" as its argument. WriteTransport.can_write_eof() Return "True" if the transport supports "write_eof()", "False" if not. WriteTransport.get_write_buffer_size() Return the current size of the output buffer used by the transport. WriteTransport.get_write_buffer_limits() Get the *high* and *low* watermarks for write flow control. Return a tuple "(low, high)" where *low* and *high* are positive number of bytes. Use "set_write_buffer_limits()" to set the limits. Added in version 3.4.2. WriteTransport.set_write_buffer_limits(high=None, low=None) Set the *high* and *low* watermarks for write flow control. These two values (measured in number of bytes) control when the protocol’s "protocol.pause_writing()" and "protocol.resume_writing()" methods are called. If specified, the low watermark must be less than or equal to the high watermark. Neither *high* nor *low* can be negative. "pause_writing()" is called when the buffer size becomes greater than or equal to the *high* value. If writing has been paused, "resume_writing()" is called when the buffer size becomes less than or equal to the *low* value. The defaults are implementation-specific. If only the high watermark is given, the low watermark defaults to an implementation-specific value less than or equal to the high watermark. Setting *high* to zero forces *low* to zero as well, and causes "pause_writing()" to be called whenever the buffer becomes non-empty. Setting *low* to zero causes "resume_writing()" to be called only once the buffer is empty. Use of zero for either limit is generally sub-optimal as it reduces opportunities for doing I/O and computation concurrently. Use "get_write_buffer_limits()" to get the limits. WriteTransport.write(data) Write some *data* bytes to the transport. This method does not block; it buffers the data and arranges for it to be sent out asynchronously. WriteTransport.writelines(list_of_data) Write a list (or any iterable) of data bytes to the transport. This is functionally equivalent to calling "write()" on each element yielded by the iterable, but may be implemented more efficiently. WriteTransport.write_eof() Close the write end of the transport after flushing all buffered data. Data may still be received. This method can raise "NotImplementedError" if the transport (e.g. SSL) doesn’t support half-closed connections. Datagram Transports ------------------- DatagramTransport.sendto(data, addr=None) Send the *data* bytes to the remote peer given by *addr* (a transport-dependent target address). If *addr* is "None", the data is sent to the target address given on transport creation. This method does not block; it buffers the data and arranges for it to be sent out asynchronously. Changed in version 3.13: This method can be called with an empty bytes object to send a zero-length datagram. The buffer size calculation used for flow control is also updated to account for the datagram header. DatagramTransport.abort() Close the transport immediately, without waiting for pending operations to complete. Buffered data will be lost. No more data will be received. The protocol’s "protocol.connection_lost()" method will eventually be called with "None" as its argument. Subprocess Transports --------------------- SubprocessTransport.get_pid() Return the subprocess process id as an integer. SubprocessTransport.get_pipe_transport(fd) Return the transport for the communication pipe corresponding to the integer file descriptor *fd*: * "0": readable streaming transport of the standard input (*stdin*), or "None" if the subprocess was not created with "stdin=PIPE" * "1": writable streaming transport of the standard output (*stdout*), or "None" if the subprocess was not created with "stdout=PIPE" * "2": writable streaming transport of the standard error (*stderr*), or "None" if the subprocess was not created with "stderr=PIPE" * other *fd*: "None" SubprocessTransport.get_returncode() Return the subprocess return code as an integer or "None" if it hasn’t returned, which is similar to the "subprocess.Popen.returncode" attribute. SubprocessTransport.kill() Kill the subprocess. On POSIX systems, the function sends SIGKILL to the subprocess. On Windows, this method is an alias for "terminate()". See also "subprocess.Popen.kill()". SubprocessTransport.send_signal(signal) Send the *signal* number to the subprocess, as in "subprocess.Popen.send_signal()". SubprocessTransport.terminate() Stop the subprocess. On POSIX systems, this method sends "SIGTERM" to the subprocess. On Windows, the Windows API function "TerminateProcess()" is called to stop the subprocess. See also "subprocess.Popen.terminate()". SubprocessTransport.close() Kill the subprocess by calling the "kill()" method. If the subprocess hasn’t returned yet, and close transports of *stdin*, *stdout*, and *stderr* pipes. Protocols ========= **Source code:** Lib/asyncio/protocols.py ====================================================================== asyncio provides a set of abstract base classes that should be used to implement network protocols. Those classes are meant to be used together with transports. Subclasses of abstract base protocol classes may implement some or all methods. All these methods are callbacks: they are called by transports on certain events, for example when some data is received. A base protocol method should be called by the corresponding transport. Base Protocols -------------- class asyncio.BaseProtocol Base protocol with methods that all protocols share. class asyncio.Protocol(BaseProtocol) The base class for implementing streaming protocols (TCP, Unix sockets, etc). class asyncio.BufferedProtocol(BaseProtocol) A base class for implementing streaming protocols with manual control of the receive buffer. class asyncio.DatagramProtocol(BaseProtocol) The base class for implementing datagram (UDP) protocols. class asyncio.SubprocessProtocol(BaseProtocol) The base class for implementing protocols communicating with child processes (unidirectional pipes). Base Protocol ------------- All asyncio protocols can implement Base Protocol callbacks. -[ Connection Callbacks ]- Connection callbacks are called on all protocols, exactly once per a successful connection. All other protocol callbacks can only be called between those two methods. BaseProtocol.connection_made(transport) Called when a connection is made. The *transport* argument is the transport representing the connection. The protocol is responsible for storing the reference to its transport. BaseProtocol.connection_lost(exc) Called when the connection is lost or closed. The argument is either an exception object or "None". The latter means a regular EOF is received, or the connection was aborted or closed by this side of the connection. -[ Flow Control Callbacks ]- Flow control callbacks can be called by transports to pause or resume writing performed by the protocol. See the documentation of the "set_write_buffer_limits()" method for more details. BaseProtocol.pause_writing() Called when the transport’s buffer goes over the high watermark. BaseProtocol.resume_writing() Called when the transport’s buffer drains below the low watermark. If the buffer size equals the high watermark, "pause_writing()" is not called: the buffer size must go strictly over. Conversely, "resume_writing()" is called when the buffer size is equal or lower than the low watermark. These end conditions are important to ensure that things go as expected when either mark is zero. Streaming Protocols ------------------- Event methods, such as "loop.create_server()", "loop.create_unix_server()", "loop.create_connection()", "loop.create_unix_connection()", "loop.connect_accepted_socket()", "loop.connect_read_pipe()", and "loop.connect_write_pipe()" accept factories that return streaming protocols. Protocol.data_received(data) Called when some data is received. *data* is a non-empty bytes object containing the incoming data. Whether the data is buffered, chunked or reassembled depends on the transport. In general, you shouldn’t rely on specific semantics and instead make your parsing generic and flexible. However, data is always received in the correct order. The method can be called an arbitrary number of times while a connection is open. However, "protocol.eof_received()" is called at most once. Once "eof_received()" is called, "data_received()" is not called anymore. Protocol.eof_received() Called when the other end signals it won’t send any more data (for example by calling "transport.write_eof()", if the other end also uses asyncio). This method may return a false value (including "None"), in which case the transport will close itself. Conversely, if this method returns a true value, the protocol used determines whether to close the transport. Since the default implementation returns "None", it implicitly closes the connection. Some transports, including SSL, don’t support half-closed connections, in which case returning true from this method will result in the connection being closed. State machine: start -> connection_made [-> data_received]* [-> eof_received]? -> connection_lost -> end Buffered Streaming Protocols ---------------------------- Added in version 3.7. Buffered Protocols can be used with any event loop method that supports Streaming Protocols. "BufferedProtocol" implementations allow explicit manual allocation and control of the receive buffer. Event loops can then use the buffer provided by the protocol to avoid unnecessary data copies. This can result in noticeable performance improvement for protocols that receive big amounts of data. Sophisticated protocol implementations can significantly reduce the number of buffer allocations. The following callbacks are called on "BufferedProtocol" instances: BufferedProtocol.get_buffer(sizehint) Called to allocate a new receive buffer. *sizehint* is the recommended minimum size for the returned buffer. It is acceptable to return smaller or larger buffers than what *sizehint* suggests. When set to -1, the buffer size can be arbitrary. It is an error to return a buffer with a zero size. "get_buffer()" must return an object implementing the buffer protocol. BufferedProtocol.buffer_updated(nbytes) Called when the buffer was updated with the received data. *nbytes* is the total number of bytes that were written to the buffer. BufferedProtocol.eof_received() See the documentation of the "protocol.eof_received()" method. "get_buffer()" can be called an arbitrary number of times during a connection. However, "protocol.eof_received()" is called at most once and, if called, "get_buffer()" and "buffer_updated()" won’t be called after it. State machine: start -> connection_made [-> get_buffer [-> buffer_updated]? ]* [-> eof_received]? -> connection_lost -> end Datagram Protocols ------------------ Datagram Protocol instances should be constructed by protocol factories passed to the "loop.create_datagram_endpoint()" method. DatagramProtocol.datagram_received(data, addr) Called when a datagram is received. *data* is a bytes object containing the incoming data. *addr* is the address of the peer sending the data; the exact format depends on the transport. DatagramProtocol.error_received(exc) Called when a previous send or receive operation raises an "OSError". *exc* is the "OSError" instance. This method is called in rare conditions, when the transport (e.g. UDP) detects that a datagram could not be delivered to its recipient. In many conditions though, undeliverable datagrams will be silently dropped. Note: On BSD systems (macOS, FreeBSD, etc.) flow control is not supported for datagram protocols, because there is no reliable way to detect send failures caused by writing too many packets.The socket always appears ‘ready’ and excess packets are dropped. An "OSError" with "errno" set to "errno.ENOBUFS" may or may not be raised; if it is raised, it will be reported to "DatagramProtocol.error_received()" but otherwise ignored. Subprocess Protocols -------------------- Subprocess Protocol instances should be constructed by protocol factories passed to the "loop.subprocess_exec()" and "loop.subprocess_shell()" methods. SubprocessProtocol.pipe_data_received(fd, data) Called when the child process writes data into its stdout or stderr pipe. *fd* is the integer file descriptor of the pipe. *data* is a non-empty bytes object containing the received data. SubprocessProtocol.pipe_connection_lost(fd, exc) Called when one of the pipes communicating with the child process is closed. *fd* is the integer file descriptor that was closed. SubprocessProtocol.process_exited() Called when the child process has exited. It can be called before "pipe_data_received()" and "pipe_connection_lost()" methods. Examples ======== TCP Echo Server --------------- Create a TCP echo server using the "loop.create_server()" method, send back received data, and close the connection: import asyncio class EchoServerProtocol(asyncio.Protocol): def connection_made(self, transport): peername = transport.get_extra_info('peername') print('Connection from {}'.format(peername)) self.transport = transport def data_received(self, data): message = data.decode() print('Data received: {!r}'.format(message)) print('Send: {!r}'.format(message)) self.transport.write(data) print('Close the client socket') self.transport.close() async def main(): # Get a reference to the event loop as we plan to use # low-level APIs. loop = asyncio.get_running_loop() server = await loop.create_server( EchoServerProtocol, '127.0.0.1', 8888) async with server: await server.serve_forever() asyncio.run(main()) See also: The TCP echo server using streams example uses the high-level "asyncio.start_server()" function. TCP Echo Client --------------- A TCP echo client using the "loop.create_connection()" method, sends data, and waits until the connection is closed: import asyncio class EchoClientProtocol(asyncio.Protocol): def __init__(self, message, on_con_lost): self.message = message self.on_con_lost = on_con_lost def connection_made(self, transport): transport.write(self.message.encode()) print('Data sent: {!r}'.format(self.message)) def data_received(self, data): print('Data received: {!r}'.format(data.decode())) def connection_lost(self, exc): print('The server closed the connection') self.on_con_lost.set_result(True) async def main(): # Get a reference to the event loop as we plan to use # low-level APIs. loop = asyncio.get_running_loop() on_con_lost = loop.create_future() message = 'Hello World!' transport, protocol = await loop.create_connection( lambda: EchoClientProtocol(message, on_con_lost), '127.0.0.1', 8888) # Wait until the protocol signals that the connection # is lost and close the transport. try: await on_con_lost finally: transport.close() asyncio.run(main()) See also: The TCP echo client using streams example uses the high-level "asyncio.open_connection()" function. UDP Echo Server --------------- A UDP echo server, using the "loop.create_datagram_endpoint()" method, sends back received data: import asyncio class EchoServerProtocol: def connection_made(self, transport): self.transport = transport def datagram_received(self, data, addr): message = data.decode() print('Received %r from %s' % (message, addr)) print('Send %r to %s' % (message, addr)) self.transport.sendto(data, addr) async def main(): print("Starting UDP server") # Get a reference to the event loop as we plan to use # low-level APIs. loop = asyncio.get_running_loop() # One protocol instance will be created to serve all # client requests. transport, protocol = await loop.create_datagram_endpoint( EchoServerProtocol, local_addr=('127.0.0.1', 9999)) try: await asyncio.sleep(3600) # Serve for 1 hour. finally: transport.close() asyncio.run(main()) UDP Echo Client --------------- A UDP echo client, using the "loop.create_datagram_endpoint()" method, sends data and closes the transport when it receives the answer: import asyncio class EchoClientProtocol: def __init__(self, message, on_con_lost): self.message = message self.on_con_lost = on_con_lost self.transport = None def connection_made(self, transport): self.transport = transport print('Send:', self.message) self.transport.sendto(self.message.encode()) def datagram_received(self, data, addr): print("Received:", data.decode()) print("Close the socket") self.transport.close() def error_received(self, exc): print('Error received:', exc) def connection_lost(self, exc): print("Connection closed") self.on_con_lost.set_result(True) async def main(): # Get a reference to the event loop as we plan to use # low-level APIs. loop = asyncio.get_running_loop() on_con_lost = loop.create_future() message = "Hello World!" transport, protocol = await loop.create_datagram_endpoint( lambda: EchoClientProtocol(message, on_con_lost), remote_addr=('127.0.0.1', 9999)) try: await on_con_lost finally: transport.close() asyncio.run(main()) Connecting Existing Sockets --------------------------- Wait until a socket receives data using the "loop.create_connection()" method with a protocol: import asyncio import socket class MyProtocol(asyncio.Protocol): def __init__(self, on_con_lost): self.transport = None self.on_con_lost = on_con_lost def connection_made(self, transport): self.transport = transport def data_received(self, data): print("Received:", data.decode()) # We are done: close the transport; # connection_lost() will be called automatically. self.transport.close() def connection_lost(self, exc): # The socket has been closed self.on_con_lost.set_result(True) async def main(): # Get a reference to the event loop as we plan to use # low-level APIs. loop = asyncio.get_running_loop() on_con_lost = loop.create_future() # Create a pair of connected sockets rsock, wsock = socket.socketpair() # Register the socket to wait for data. transport, protocol = await loop.create_connection( lambda: MyProtocol(on_con_lost), sock=rsock) # Simulate the reception of data from the network. loop.call_soon(wsock.send, 'abc'.encode()) try: await protocol.on_con_lost finally: transport.close() wsock.close() asyncio.run(main()) See also: The watch a file descriptor for read events example uses the low- level "loop.add_reader()" method to register an FD. The register an open socket to wait for data using streams example uses high-level streams created by the "open_connection()" function in a coroutine. loop.subprocess_exec() and SubprocessProtocol --------------------------------------------- An example of a subprocess protocol used to get the output of a subprocess and to wait for the subprocess exit. The subprocess is created by the "loop.subprocess_exec()" method: import asyncio import sys class DateProtocol(asyncio.SubprocessProtocol): def __init__(self, exit_future): self.exit_future = exit_future self.output = bytearray() self.pipe_closed = False self.exited = False def pipe_connection_lost(self, fd, exc): self.pipe_closed = True self.check_for_exit() def pipe_data_received(self, fd, data): self.output.extend(data) def process_exited(self): self.exited = True # process_exited() method can be called before # pipe_connection_lost() method: wait until both methods are # called. self.check_for_exit() def check_for_exit(self): if self.pipe_closed and self.exited: self.exit_future.set_result(True) async def get_date(): # Get a reference to the event loop as we plan to use # low-level APIs. loop = asyncio.get_running_loop() code = 'import datetime; print(datetime.datetime.now())' exit_future = asyncio.Future(loop=loop) # Create the subprocess controlled by DateProtocol; # redirect the standard output into a pipe. transport, protocol = await loop.subprocess_exec( lambda: DateProtocol(exit_future), sys.executable, '-c', code, stdin=None, stderr=None) # Wait for the subprocess exit using the process_exited() # method of the protocol. await exit_future # Close the stdout pipe. transport.close() # Read the output which was collected by the # pipe_data_received() method of the protocol. data = bytes(protocol.output) return data.decode('ascii').rstrip() date = asyncio.run(get_date()) print(f"Current date: {date}") See also the same example written using high-level APIs. Queues ****** **Source code:** Lib/asyncio/queues.py ====================================================================== asyncio queues are designed to be similar to classes of the "queue" module. Although asyncio queues are not thread-safe, they are designed to be used specifically in async/await code. Note that methods of asyncio queues don’t have a *timeout* parameter; use "asyncio.wait_for()" function to do queue operations with a timeout. See also the Examples section below. Queue ===== class asyncio.Queue(maxsize=0) A first in, first out (FIFO) queue. If *maxsize* is less than or equal to zero, the queue size is infinite. If it is an integer greater than "0", then "await put()" blocks when the queue reaches *maxsize* until an item is removed by "get()". Unlike the standard library threading "queue", the size of the queue is always known and can be returned by calling the "qsize()" method. Changed in version 3.10: Removed the *loop* parameter. This class is not thread safe. maxsize Number of items allowed in the queue. empty() Return "True" if the queue is empty, "False" otherwise. full() Return "True" if there are "maxsize" items in the queue. If the queue was initialized with "maxsize=0" (the default), then "full()" never returns "True". async get() Remove and return an item from the queue. If queue is empty, wait until an item is available. Raises "QueueShutDown" if the queue has been shut down and is empty, or if the queue has been shut down immediately. get_nowait() Return an item if one is immediately available, else raise "QueueEmpty". async join() Block until all items in the queue have been received and processed. The count of unfinished tasks goes up whenever an item is added to the queue. The count goes down whenever a consumer coroutine calls "task_done()" to indicate that the item was retrieved and all work on it is complete. When the count of unfinished tasks drops to zero, "join()" unblocks. async put(item) Put an item into the queue. If the queue is full, wait until a free slot is available before adding the item. Raises "QueueShutDown" if the queue has been shut down. put_nowait(item) Put an item into the queue without blocking. If no free slot is immediately available, raise "QueueFull". qsize() Return the number of items in the queue. shutdown(immediate=False) Put a "Queue" instance into a shutdown mode. The queue can no longer grow. Future calls to "put()" raise "QueueShutDown". Currently blocked callers of "put()" will be unblocked and will raise "QueueShutDown" in the formerly blocked thread. If *immediate* is false (the default), the queue can be wound down normally with "get()" calls to extract tasks that have already been loaded. And if "task_done()" is called for each remaining task, a pending "join()" will be unblocked normally. Once the queue is empty, future calls to "get()" will raise "QueueShutDown". If *immediate* is true, the queue is terminated immediately. The queue is drained to be completely empty and the count of unfinished tasks is reduced by the number of tasks drained. If unfinished tasks is zero, callers of "join()" are unblocked. Also, blocked callers of "get()" are unblocked and will raise "QueueShutDown" because the queue is empty. Use caution when using "join()" with *immediate* set to true. This unblocks the join even when no work has been done on the tasks, violating the usual invariant for joining a queue. Added in version 3.13. task_done() Indicate that a formerly enqueued work item is complete. Used by queue consumers. For each "get()" used to fetch a work item, a subsequent call to "task_done()" tells the queue that the processing on the work item is complete. If a "join()" is currently blocking, it will resume when all items have been processed (meaning that a "task_done()" call was received for every item that had been "put()" into the queue). Raises "ValueError" if called more times than there were items placed in the queue. Priority Queue ============== class asyncio.PriorityQueue A variant of "Queue"; retrieves entries in priority order (lowest first). Entries are typically tuples of the form "(priority_number, data)". LIFO Queue ========== class asyncio.LifoQueue A variant of "Queue" that retrieves most recently added entries first (last in, first out). Exceptions ========== exception asyncio.QueueEmpty This exception is raised when the "get_nowait()" method is called on an empty queue. exception asyncio.QueueFull Exception raised when the "put_nowait()" method is called on a queue that has reached its *maxsize*. exception asyncio.QueueShutDown Exception raised when "put()" or "get()" is called on a queue which has been shut down. Added in version 3.13. Examples ======== Queues can be used to distribute workload between several concurrent tasks: import asyncio import random import time async def worker(name, queue): while True: # Get a "work item" out of the queue. sleep_for = await queue.get() # Sleep for the "sleep_for" seconds. await asyncio.sleep(sleep_for) # Notify the queue that the "work item" has been processed. queue.task_done() print(f'{name} has slept for {sleep_for:.2f} seconds') async def main(): # Create a queue that we will use to store our "workload". queue = asyncio.Queue() # Generate random timings and put them into the queue. total_sleep_time = 0 for _ in range(20): sleep_for = random.uniform(0.05, 1.0) total_sleep_time += sleep_for queue.put_nowait(sleep_for) # Create three worker tasks to process the queue concurrently. tasks = [] for i in range(3): task = asyncio.create_task(worker(f'worker-{i}', queue)) tasks.append(task) # Wait until the queue is fully processed. started_at = time.monotonic() await queue.join() total_slept_for = time.monotonic() - started_at # Cancel our worker tasks. for task in tasks: task.cancel() # Wait until all worker tasks are cancelled. await asyncio.gather(*tasks, return_exceptions=True) print('====') print(f'3 workers slept in parallel for {total_slept_for:.2f} seconds') print(f'total expected sleep time: {total_sleep_time:.2f} seconds') asyncio.run(main()) Runners ******* **Source code:** Lib/asyncio/runners.py This section outlines high-level asyncio primitives to run asyncio code. They are built on top of an event loop with the aim to simplify async code usage for common wide-spread scenarios. * Running an asyncio Program * Runner context manager * Handling Keyboard Interruption Running an asyncio Program ========================== asyncio.run(coro, *, debug=None, loop_factory=None) Execute the *coroutine* *coro* and return the result. This function runs the passed coroutine, taking care of managing the asyncio event loop, *finalizing asynchronous generators*, and closing the executor. This function cannot be called when another asyncio event loop is running in the same thread. If *debug* is "True", the event loop will be run in debug mode. "False" disables debug mode explicitly. "None" is used to respect the global Debug Mode settings. If *loop_factory* is not "None", it is used to create a new event loop; otherwise "asyncio.new_event_loop()" is used. The loop is closed at the end. This function should be used as a main entry point for asyncio programs, and should ideally only be called once. It is recommended to use *loop_factory* to configure the event loop instead of policies. Passing "asyncio.EventLoop" allows running asyncio without the policy system. The executor is given a timeout duration of 5 minutes to shutdown. If the executor hasn’t finished within that duration, a warning is emitted and the executor is closed. Example: async def main(): await asyncio.sleep(1) print('hello') asyncio.run(main()) Added in version 3.7. Changed in version 3.9: Updated to use "loop.shutdown_default_executor()". Changed in version 3.10: *debug* is "None" by default to respect the global debug mode settings. Changed in version 3.12: Added *loop_factory* parameter. Runner context manager ====================== class asyncio.Runner(*, debug=None, loop_factory=None) A context manager that simplifies *multiple* async function calls in the same context. Sometimes several top-level async functions should be called in the same event loop and "contextvars.Context". If *debug* is "True", the event loop will be run in debug mode. "False" disables debug mode explicitly. "None" is used to respect the global Debug Mode settings. *loop_factory* could be used for overriding the loop creation. It is the responsibility of the *loop_factory* to set the created loop as the current one. By default "asyncio.new_event_loop()" is used and set as current event loop with "asyncio.set_event_loop()" if *loop_factory* is "None". Basically, "asyncio.run()" example can be rewritten with the runner usage: async def main(): await asyncio.sleep(1) print('hello') with asyncio.Runner() as runner: runner.run(main()) Added in version 3.11. run(coro, *, context=None) Run a *coroutine* *coro* in the embedded loop. Return the coroutine’s result or raise its exception. An optional keyword-only *context* argument allows specifying a custom "contextvars.Context" for the *coro* to run in. The runner’s default context is used if "None". This function cannot be called when another asyncio event loop is running in the same thread. close() Close the runner. Finalize asynchronous generators, shutdown default executor, close the event loop and release embedded "contextvars.Context". get_loop() Return the event loop associated with the runner instance. Note: "Runner" uses the lazy initialization strategy, its constructor doesn’t initialize underlying low-level structures.Embedded *loop* and *context* are created at the "with" body entering or the first call of "run()" or "get_loop()". Handling Keyboard Interruption ============================== Added in version 3.11. When "signal.SIGINT" is raised by "Ctrl"-"C", "KeyboardInterrupt" exception is raised in the main thread by default. However this doesn’t work with "asyncio" because it can interrupt asyncio internals and can hang the program from exiting. To mitigate this issue, "asyncio" handles "signal.SIGINT" as follows: 1. "asyncio.Runner.run()" installs a custom "signal.SIGINT" handler before any user code is executed and removes it when exiting from the function. 2. The "Runner" creates the main task for the passed coroutine for its execution. 3. When "signal.SIGINT" is raised by "Ctrl"-"C", the custom signal handler cancels the main task by calling "asyncio.Task.cancel()" which raises "asyncio.CancelledError" inside the main task. This causes the Python stack to unwind, "try/except" and "try/finally" blocks can be used for resource cleanup. After the main task is cancelled, "asyncio.Runner.run()" raises "KeyboardInterrupt". 4. A user could write a tight loop which cannot be interrupted by "asyncio.Task.cancel()", in which case the second following "Ctrl"-"C" immediately raises the "KeyboardInterrupt" without cancelling the main task. Streams ******* **Source code:** Lib/asyncio/streams.py ====================================================================== Streams are high-level async/await-ready primitives to work with network connections. Streams allow sending and receiving data without using callbacks or low-level protocols and transports. Here is an example of a TCP echo client written using asyncio streams: import asyncio async def tcp_echo_client(message): reader, writer = await asyncio.open_connection( '127.0.0.1', 8888) print(f'Send: {message!r}') writer.write(message.encode()) await writer.drain() data = await reader.read(100) print(f'Received: {data.decode()!r}') print('Close the connection') writer.close() await writer.wait_closed() asyncio.run(tcp_echo_client('Hello World!')) See also the Examples section below. -[ Stream Functions ]- The following top-level asyncio functions can be used to create and work with streams: async asyncio.open_connection(host=None, port=None, *, limit=None, ssl=None, family=0, proto=0, flags=0, sock=None, local_addr=None, server_hostname=None, ssl_handshake_timeout=None, ssl_shutdown_timeout=None, happy_eyeballs_delay=None, interleave=None) Establish a network connection and return a pair of "(reader, writer)" objects. The returned *reader* and *writer* objects are instances of "StreamReader" and "StreamWriter" classes. *limit* determines the buffer size limit used by the returned "StreamReader" instance. By default the *limit* is set to 64 KiB. The rest of the arguments are passed directly to "loop.create_connection()". Note: The *sock* argument transfers ownership of the socket to the "StreamWriter" created. To close the socket, call its "close()" method. Changed in version 3.7: Added the *ssl_handshake_timeout* parameter. Changed in version 3.8: Added the *happy_eyeballs_delay* and *interleave* parameters. Changed in version 3.10: Removed the *loop* parameter. Changed in version 3.11: Added the *ssl_shutdown_timeout* parameter. async asyncio.start_server(client_connected_cb, host=None, port=None, *, limit=None, family=socket.AF_UNSPEC, flags=socket.AI_PASSIVE, sock=None, backlog=100, ssl=None, reuse_address=None, reuse_port=None, keep_alive=None, ssl_handshake_timeout=None, ssl_shutdown_timeout=None, start_serving=True) Start a socket server. The *client_connected_cb* callback is called whenever a new client connection is established. It receives a "(reader, writer)" pair as two arguments, instances of the "StreamReader" and "StreamWriter" classes. *client_connected_cb* can be a plain callable or a coroutine function; if it is a coroutine function, it will be automatically scheduled as a "Task". *limit* determines the buffer size limit used by the returned "StreamReader" instance. By default the *limit* is set to 64 KiB. The rest of the arguments are passed directly to "loop.create_server()". Note: The *sock* argument transfers ownership of the socket to the server created. To close the socket, call the server’s "close()" method. Changed in version 3.7: Added the *ssl_handshake_timeout* and *start_serving* parameters. Changed in version 3.10: Removed the *loop* parameter. Changed in version 3.11: Added the *ssl_shutdown_timeout* parameter. Changed in version 3.13: Added the *keep_alive* parameter. -[ Unix Sockets ]- async asyncio.open_unix_connection(path=None, *, limit=None, ssl=None, sock=None, server_hostname=None, ssl_handshake_timeout=None, ssl_shutdown_timeout=None) Establish a Unix socket connection and return a pair of "(reader, writer)". Similar to "open_connection()" but operates on Unix sockets. See also the documentation of "loop.create_unix_connection()". Note: The *sock* argument transfers ownership of the socket to the "StreamWriter" created. To close the socket, call its "close()" method. Availability: Unix. Changed in version 3.7: Added the *ssl_handshake_timeout* parameter. The *path* parameter can now be a *path-like object* Changed in version 3.10: Removed the *loop* parameter. Changed in version 3.11: Added the *ssl_shutdown_timeout* parameter. async asyncio.start_unix_server(client_connected_cb, path=None, *, limit=None, sock=None, backlog=100, ssl=None, ssl_handshake_timeout=None, ssl_shutdown_timeout=None, start_serving=True, cleanup_socket=True) Start a Unix socket server. Similar to "start_server()" but works with Unix sockets. If *cleanup_socket* is true then the Unix socket will automatically be removed from the filesystem when the server is closed, unless the socket has been replaced after the server has been created. See also the documentation of "loop.create_unix_server()". Note: The *sock* argument transfers ownership of the socket to the server created. To close the socket, call the server’s "close()" method. Availability: Unix. Changed in version 3.7: Added the *ssl_handshake_timeout* and *start_serving* parameters. The *path* parameter can now be a *path-like object*. Changed in version 3.10: Removed the *loop* parameter. Changed in version 3.11: Added the *ssl_shutdown_timeout* parameter. Changed in version 3.13: Added the *cleanup_socket* parameter. StreamReader ============ class asyncio.StreamReader Represents a reader object that provides APIs to read data from the IO stream. As an *asynchronous iterable*, the object supports the "async for" statement. It is not recommended to instantiate *StreamReader* objects directly; use "open_connection()" and "start_server()" instead. feed_eof() Acknowledge the EOF. async read(n=-1) Read up to *n* bytes from the stream. If *n* is not provided or set to "-1", read until EOF, then return all read "bytes". If EOF was received and the internal buffer is empty, return an empty "bytes" object. If *n* is "0", return an empty "bytes" object immediately. If *n* is positive, return at most *n* available "bytes" as soon as at least 1 byte is available in the internal buffer. If EOF is received before any byte is read, return an empty "bytes" object. async readline() Read one line, where “line” is a sequence of bytes ending with "\n". If EOF is received and "\n" was not found, the method returns partially read data. If EOF is received and the internal buffer is empty, return an empty "bytes" object. async readexactly(n) Read exactly *n* bytes. Raise an "IncompleteReadError" if EOF is reached before *n* can be read. Use the "IncompleteReadError.partial" attribute to get the partially read data. async readuntil(separator=b'\n') Read data from the stream until *separator* is found. On success, the data and separator will be removed from the internal buffer (consumed). Returned data will include the separator at the end. If the amount of data read exceeds the configured stream limit, a "LimitOverrunError" exception is raised, and the data is left in the internal buffer and can be read again. If EOF is reached before the complete separator is found, an "IncompleteReadError" exception is raised, and the internal buffer is reset. The "IncompleteReadError.partial" attribute may contain a portion of the separator. The *separator* may also be a tuple of separators. In this case the return value will be the shortest possible that has any separator as the suffix. For the purposes of "LimitOverrunError", the shortest possible separator is considered to be the one that matched. Added in version 3.5.2. Changed in version 3.13: The *separator* parameter may now be a "tuple" of separators. at_eof() Return "True" if the buffer is empty and "feed_eof()" was called. StreamWriter ============ class asyncio.StreamWriter Represents a writer object that provides APIs to write data to the IO stream. It is not recommended to instantiate *StreamWriter* objects directly; use "open_connection()" and "start_server()" instead. write(data) The method attempts to write the *data* to the underlying socket immediately. If that fails, the data is queued in an internal write buffer until it can be sent. The method should be used along with the "drain()" method: stream.write(data) await stream.drain() writelines(data) The method writes a list (or any iterable) of bytes to the underlying socket immediately. If that fails, the data is queued in an internal write buffer until it can be sent. The method should be used along with the "drain()" method: stream.writelines(lines) await stream.drain() close() The method closes the stream and the underlying socket. The method should be used, though not mandatory, along with the "wait_closed()" method: stream.close() await stream.wait_closed() can_write_eof() Return "True" if the underlying transport supports the "write_eof()" method, "False" otherwise. write_eof() Close the write end of the stream after the buffered write data is flushed. transport Return the underlying asyncio transport. get_extra_info(name, default=None) Access optional transport information; see "BaseTransport.get_extra_info()" for details. async drain() Wait until it is appropriate to resume writing to the stream. Example: writer.write(data) await writer.drain() This is a flow control method that interacts with the underlying IO write buffer. When the size of the buffer reaches the high watermark, *drain()* blocks until the size of the buffer is drained down to the low watermark and writing can be resumed. When there is nothing to wait for, the "drain()" returns immediately. async start_tls(sslcontext, *, server_hostname=None, ssl_handshake_timeout=None, ssl_shutdown_timeout=None) Upgrade an existing stream-based connection to TLS. Parameters: * *sslcontext*: a configured instance of "SSLContext". * *server_hostname*: sets or overrides the host name that the target server’s certificate will be matched against. * *ssl_handshake_timeout* is the time in seconds to wait for the TLS handshake to complete before aborting the connection. "60.0" seconds if "None" (default). * *ssl_shutdown_timeout* is the time in seconds to wait for the SSL shutdown to complete before aborting the connection. "30.0" seconds if "None" (default). Added in version 3.11. Changed in version 3.12: Added the *ssl_shutdown_timeout* parameter. is_closing() Return "True" if the stream is closed or in the process of being closed. Added in version 3.7. async wait_closed() Wait until the stream is closed. Should be called after "close()" to wait until the underlying connection is closed, ensuring that all data has been flushed before e.g. exiting the program. Added in version 3.7. Examples ======== TCP echo client using streams ----------------------------- TCP echo client using the "asyncio.open_connection()" function: import asyncio async def tcp_echo_client(message): reader, writer = await asyncio.open_connection( '127.0.0.1', 8888) print(f'Send: {message!r}') writer.write(message.encode()) await writer.drain() data = await reader.read(100) print(f'Received: {data.decode()!r}') print('Close the connection') writer.close() await writer.wait_closed() asyncio.run(tcp_echo_client('Hello World!')) See also: The TCP echo client protocol example uses the low-level "loop.create_connection()" method. TCP echo server using streams ----------------------------- TCP echo server using the "asyncio.start_server()" function: import asyncio async def handle_echo(reader, writer): data = await reader.read(100) message = data.decode() addr = writer.get_extra_info('peername') print(f"Received {message!r} from {addr!r}") print(f"Send: {message!r}") writer.write(data) await writer.drain() print("Close the connection") writer.close() await writer.wait_closed() async def main(): server = await asyncio.start_server( handle_echo, '127.0.0.1', 8888) addrs = ', '.join(str(sock.getsockname()) for sock in server.sockets) print(f'Serving on {addrs}') async with server: await server.serve_forever() asyncio.run(main()) See also: The TCP echo server protocol example uses the "loop.create_server()" method. Get HTTP headers ---------------- Simple example querying HTTP headers of the URL passed on the command line: import asyncio import urllib.parse import sys async def print_http_headers(url): url = urllib.parse.urlsplit(url) if url.scheme == 'https': reader, writer = await asyncio.open_connection( url.hostname, 443, ssl=True) else: reader, writer = await asyncio.open_connection( url.hostname, 80) query = ( f"HEAD {url.path or '/'} HTTP/1.0\r\n" f"Host: {url.hostname}\r\n" f"\r\n" ) writer.write(query.encode('latin-1')) while True: line = await reader.readline() if not line: break line = line.decode('latin1').rstrip() if line: print(f'HTTP header> {line}') # Ignore the body, close the socket writer.close() await writer.wait_closed() url = sys.argv[1] asyncio.run(print_http_headers(url)) Usage: python example.py http://example.com/path/page.html or with HTTPS: python example.py https://example.com/path/page.html Register an open socket to wait for data using streams ------------------------------------------------------ Coroutine waiting until a socket receives data using the "open_connection()" function: import asyncio import socket async def wait_for_data(): # Get a reference to the current event loop because # we want to access low-level APIs. loop = asyncio.get_running_loop() # Create a pair of connected sockets. rsock, wsock = socket.socketpair() # Register the open socket to wait for data. reader, writer = await asyncio.open_connection(sock=rsock) # Simulate the reception of data from the network loop.call_soon(wsock.send, 'abc'.encode()) # Wait for data data = await reader.read(100) # Got data, we are done: close the socket print("Received:", data.decode()) writer.close() await writer.wait_closed() # Close the second socket wsock.close() asyncio.run(wait_for_data()) See also: The register an open socket to wait for data using a protocol example uses a low-level protocol and the "loop.create_connection()" method. The watch a file descriptor for read events example uses the low- level "loop.add_reader()" method to watch a file descriptor. Subprocesses ************ **Source code:** Lib/asyncio/subprocess.py, Lib/asyncio/base_subprocess.py ====================================================================== This section describes high-level async/await asyncio APIs to create and manage subprocesses. Here’s an example of how asyncio can run a shell command and obtain its result: import asyncio async def run(cmd): proc = await asyncio.create_subprocess_shell( cmd, stdout=asyncio.subprocess.PIPE, stderr=asyncio.subprocess.PIPE) stdout, stderr = await proc.communicate() print(f'[{cmd!r} exited with {proc.returncode}]') if stdout: print(f'[stdout]\n{stdout.decode()}') if stderr: print(f'[stderr]\n{stderr.decode()}') asyncio.run(run('ls /zzz')) will print: ['ls /zzz' exited with 1] [stderr] ls: /zzz: No such file or directory Because all asyncio subprocess functions are asynchronous and asyncio provides many tools to work with such functions, it is easy to execute and monitor multiple subprocesses in parallel. It is indeed trivial to modify the above example to run several commands simultaneously: async def main(): await asyncio.gather( run('ls /zzz'), run('sleep 1; echo "hello"')) asyncio.run(main()) See also the Examples subsection. Creating Subprocesses ===================== async asyncio.create_subprocess_exec(program, *args, stdin=None, stdout=None, stderr=None, limit=None, **kwds) Create a subprocess. The *limit* argument sets the buffer limit for "StreamReader" wrappers for "stdout" and "stderr" (if "subprocess.PIPE" is passed to *stdout* and *stderr* arguments). Return a "Process" instance. See the documentation of "loop.subprocess_exec()" for other parameters. Changed in version 3.10: Removed the *loop* parameter. async asyncio.create_subprocess_shell(cmd, stdin=None, stdout=None, stderr=None, limit=None, **kwds) Run the *cmd* shell command. The *limit* argument sets the buffer limit for "StreamReader" wrappers for "stdout" and "stderr" (if "subprocess.PIPE" is passed to *stdout* and *stderr* arguments). Return a "Process" instance. See the documentation of "loop.subprocess_shell()" for other parameters. Important: It is the application’s responsibility to ensure that all whitespace and special characters are quoted appropriately to avoid shell injection vulnerabilities. The "shlex.quote()" function can be used to properly escape whitespace and special shell characters in strings that are going to be used to construct shell commands. Changed in version 3.10: Removed the *loop* parameter. Note: Subprocesses are available for Windows if a "ProactorEventLoop" is used. See Subprocess Support on Windows for details. See also: asyncio also has the following *low-level* APIs to work with subprocesses: "loop.subprocess_exec()", "loop.subprocess_shell()", "loop.connect_read_pipe()", "loop.connect_write_pipe()", as well as the Subprocess Transports and Subprocess Protocols. Constants ========= asyncio.subprocess.PIPE Can be passed to the *stdin*, *stdout* or *stderr* parameters. If *PIPE* is passed to *stdin* argument, the "Process.stdin" attribute will point to a "StreamWriter" instance. If *PIPE* is passed to *stdout* or *stderr* arguments, the "Process.stdout" and "Process.stderr" attributes will point to "StreamReader" instances. asyncio.subprocess.STDOUT Special value that can be used as the *stderr* argument and indicates that standard error should be redirected into standard output. asyncio.subprocess.DEVNULL Special value that can be used as the *stdin*, *stdout* or *stderr* argument to process creation functions. It indicates that the special file "os.devnull" will be used for the corresponding subprocess stream. Interacting with Subprocesses ============================= Both "create_subprocess_exec()" and "create_subprocess_shell()" functions return instances of the *Process* class. *Process* is a high-level wrapper that allows communicating with subprocesses and watching for their completion. class asyncio.subprocess.Process An object that wraps OS processes created by the "create_subprocess_exec()" and "create_subprocess_shell()" functions. This class is designed to have a similar API to the "subprocess.Popen" class, but there are some notable differences: * unlike Popen, Process instances do not have an equivalent to the "poll()" method; * the "communicate()" and "wait()" methods don’t have a *timeout* parameter: use the "wait_for()" function; * the "Process.wait()" method is asynchronous, whereas "subprocess.Popen.wait()" method is implemented as a blocking busy loop; * the *universal_newlines* parameter is not supported. This class is not thread safe. See also the Subprocess and Threads section. async wait() Wait for the child process to terminate. Set and return the "returncode" attribute. Note: This method can deadlock when using "stdout=PIPE" or "stderr=PIPE" and the child process generates so much output that it blocks waiting for the OS pipe buffer to accept more data. Use the "communicate()" method when using pipes to avoid this condition. async communicate(input=None) Interact with process: 1. send data to *stdin* (if *input* is not "None"); 2. closes *stdin*; 3. read data from *stdout* and *stderr*, until EOF is reached; 4. wait for process to terminate. The optional *input* argument is the data ("bytes" object) that will be sent to the child process. Return a tuple "(stdout_data, stderr_data)". If either "BrokenPipeError" or "ConnectionResetError" exception is raised when writing *input* into *stdin*, the exception is ignored. This condition occurs when the process exits before all data are written into *stdin*. If it is desired to send data to the process’ *stdin*, the process needs to be created with "stdin=PIPE". Similarly, to get anything other than "None" in the result tuple, the process has to be created with "stdout=PIPE" and/or "stderr=PIPE" arguments. Note, that the data read is buffered in memory, so do not use this method if the data size is large or unlimited. Changed in version 3.12: *stdin* gets closed when "input=None" too. send_signal(signal) Sends the signal *signal* to the child process. Note: On Windows, "SIGTERM" is an alias for "terminate()". "CTRL_C_EVENT" and "CTRL_BREAK_EVENT" can be sent to processes started with a *creationflags* parameter which includes "CREATE_NEW_PROCESS_GROUP". terminate() Stop the child process. On POSIX systems this method sends "SIGTERM" to the child process. On Windows the Win32 API function "TerminateProcess()" is called to stop the child process. kill() Kill the child process. On POSIX systems this method sends "SIGKILL" to the child process. On Windows this method is an alias for "terminate()". stdin Standard input stream ("StreamWriter") or "None" if the process was created with "stdin=None". stdout Standard output stream ("StreamReader") or "None" if the process was created with "stdout=None". stderr Standard error stream ("StreamReader") or "None" if the process was created with "stderr=None". Warning: Use the "communicate()" method rather than "process.stdin.write()", "await process.stdout.read()" or "await process.stderr.read()". This avoids deadlocks due to streams pausing reading or writing and blocking the child process. pid Process identification number (PID). Note that for processes created by the "create_subprocess_shell()" function, this attribute is the PID of the spawned shell. returncode Return code of the process when it exits. A "None" value indicates that the process has not terminated yet. A negative value "-N" indicates that the child was terminated by signal "N" (POSIX only). Subprocess and Threads ---------------------- Standard asyncio event loop supports running subprocesses from different threads by default. On Windows subprocesses are provided by "ProactorEventLoop" only (default), "SelectorEventLoop" has no subprocess support. On UNIX *child watchers* are used for subprocess finish waiting, see Process Watchers for more info. Changed in version 3.8: UNIX switched to use "ThreadedChildWatcher" for spawning subprocesses from different threads without any limitation.Spawning a subprocess with *inactive* current child watcher raises "RuntimeError". Note that alternative event loop implementations might have own limitations; please refer to their documentation. See also: The Concurrency and multithreading in asyncio section. Examples -------- An example using the "Process" class to control a subprocess and the "StreamReader" class to read from its standard output. The subprocess is created by the "create_subprocess_exec()" function: import asyncio import sys async def get_date(): code = 'import datetime; print(datetime.datetime.now())' # Create the subprocess; redirect the standard output # into a pipe. proc = await asyncio.create_subprocess_exec( sys.executable, '-c', code, stdout=asyncio.subprocess.PIPE) # Read one line of output. data = await proc.stdout.readline() line = data.decode('ascii').rstrip() # Wait for the subprocess exit. await proc.wait() return line date = asyncio.run(get_date()) print(f"Current date: {date}") See also the same example written using low-level APIs. Synchronization Primitives ************************** **Source code:** Lib/asyncio/locks.py ====================================================================== asyncio synchronization primitives are designed to be similar to those of the "threading" module with two important caveats: * asyncio primitives are not thread-safe, therefore they should not be used for OS thread synchronization (use "threading" for that); * methods of these synchronization primitives do not accept the *timeout* argument; use the "asyncio.wait_for()" function to perform operations with timeouts. asyncio has the following basic synchronization primitives: * "Lock" * "Event" * "Condition" * "Semaphore" * "BoundedSemaphore" * "Barrier" ====================================================================== Lock ==== class asyncio.Lock Implements a mutex lock for asyncio tasks. Not thread-safe. An asyncio lock can be used to guarantee exclusive access to a shared resource. The preferred way to use a Lock is an "async with" statement: lock = asyncio.Lock() # ... later async with lock: # access shared state which is equivalent to: lock = asyncio.Lock() # ... later await lock.acquire() try: # access shared state finally: lock.release() Changed in version 3.10: Removed the *loop* parameter. async acquire() Acquire the lock. This method waits until the lock is *unlocked*, sets it to *locked* and returns "True". When more than one coroutine is blocked in "acquire()" waiting for the lock to be unlocked, only one coroutine eventually proceeds. Acquiring a lock is *fair*: the coroutine that proceeds will be the first coroutine that started waiting on the lock. release() Release the lock. When the lock is *locked*, reset it to *unlocked* and return. If the lock is *unlocked*, a "RuntimeError" is raised. locked() Return "True" if the lock is *locked*. Event ===== class asyncio.Event An event object. Not thread-safe. An asyncio event can be used to notify multiple asyncio tasks that some event has happened. An Event object manages an internal flag that can be set to *true* with the "set()" method and reset to *false* with the "clear()" method. The "wait()" method blocks until the flag is set to *true*. The flag is set to *false* initially. Changed in version 3.10: Removed the *loop* parameter. Example: async def waiter(event): print('waiting for it ...') await event.wait() print('... got it!') async def main(): # Create an Event object. event = asyncio.Event() # Spawn a Task to wait until 'event' is set. waiter_task = asyncio.create_task(waiter(event)) # Sleep for 1 second and set the event. await asyncio.sleep(1) event.set() # Wait until the waiter task is finished. await waiter_task asyncio.run(main()) async wait() Wait until the event is set. If the event is set, return "True" immediately. Otherwise block until another task calls "set()". set() Set the event. All tasks waiting for event to be set will be immediately awakened. clear() Clear (unset) the event. Tasks awaiting on "wait()" will now block until the "set()" method is called again. is_set() Return "True" if the event is set. Condition ========= class asyncio.Condition(lock=None) A Condition object. Not thread-safe. An asyncio condition primitive can be used by a task to wait for some event to happen and then get exclusive access to a shared resource. In essence, a Condition object combines the functionality of an "Event" and a "Lock". It is possible to have multiple Condition objects share one Lock, which allows coordinating exclusive access to a shared resource between different tasks interested in particular states of that shared resource. The optional *lock* argument must be a "Lock" object or "None". In the latter case a new Lock object is created automatically. Changed in version 3.10: Removed the *loop* parameter. The preferred way to use a Condition is an "async with" statement: cond = asyncio.Condition() # ... later async with cond: await cond.wait() which is equivalent to: cond = asyncio.Condition() # ... later await cond.acquire() try: await cond.wait() finally: cond.release() async acquire() Acquire the underlying lock. This method waits until the underlying lock is *unlocked*, sets it to *locked* and returns "True". notify(n=1) Wake up *n* tasks (1 by default) waiting on this condition. If fewer than *n* tasks are waiting they are all awakened. The lock must be acquired before this method is called and released shortly after. If called with an *unlocked* lock a "RuntimeError" error is raised. locked() Return "True" if the underlying lock is acquired. notify_all() Wake up all tasks waiting on this condition. This method acts like "notify()", but wakes up all waiting tasks. The lock must be acquired before this method is called and released shortly after. If called with an *unlocked* lock a "RuntimeError" error is raised. release() Release the underlying lock. When invoked on an unlocked lock, a "RuntimeError" is raised. async wait() Wait until notified. If the calling task has not acquired the lock when this method is called, a "RuntimeError" is raised. This method releases the underlying lock, and then blocks until it is awakened by a "notify()" or "notify_all()" call. Once awakened, the Condition re-acquires its lock and this method returns "True". Note that a task *may* return from this call spuriously, which is why the caller should always re-check the state and be prepared to "wait()" again. For this reason, you may prefer to use "wait_for()" instead. async wait_for(predicate) Wait until a predicate becomes *true*. The predicate must be a callable which result will be interpreted as a boolean value. The method will repeatedly "wait()" until the predicate evaluates to *true*. The final value is the return value. Semaphore ========= class asyncio.Semaphore(value=1) A Semaphore object. Not thread-safe. A semaphore manages an internal counter which is decremented by each "acquire()" call and incremented by each "release()" call. The counter can never go below zero; when "acquire()" finds that it is zero, it blocks, waiting until some task calls "release()". The optional *value* argument gives the initial value for the internal counter ("1" by default). If the given value is less than "0" a "ValueError" is raised. Changed in version 3.10: Removed the *loop* parameter. The preferred way to use a Semaphore is an "async with" statement: sem = asyncio.Semaphore(10) # ... later async with sem: # work with shared resource which is equivalent to: sem = asyncio.Semaphore(10) # ... later await sem.acquire() try: # work with shared resource finally: sem.release() async acquire() Acquire a semaphore. If the internal counter is greater than zero, decrement it by one and return "True" immediately. If it is zero, wait until a "release()" is called and return "True". locked() Returns "True" if semaphore can not be acquired immediately. release() Release a semaphore, incrementing the internal counter by one. Can wake up a task waiting to acquire the semaphore. Unlike "BoundedSemaphore", "Semaphore" allows making more "release()" calls than "acquire()" calls. BoundedSemaphore ================ class asyncio.BoundedSemaphore(value=1) A bounded semaphore object. Not thread-safe. Bounded Semaphore is a version of "Semaphore" that raises a "ValueError" in "release()" if it increases the internal counter above the initial *value*. Changed in version 3.10: Removed the *loop* parameter. Barrier ======= class asyncio.Barrier(parties) A barrier object. Not thread-safe. A barrier is a simple synchronization primitive that allows to block until *parties* number of tasks are waiting on it. Tasks can wait on the "wait()" method and would be blocked until the specified number of tasks end up waiting on "wait()". At that point all of the waiting tasks would unblock simultaneously. "async with" can be used as an alternative to awaiting on "wait()". The barrier can be reused any number of times. Example: async def example_barrier(): # barrier with 3 parties b = asyncio.Barrier(3) # create 2 new waiting tasks asyncio.create_task(b.wait()) asyncio.create_task(b.wait()) await asyncio.sleep(0) print(b) # The third .wait() call passes the barrier await b.wait() print(b) print("barrier passed") await asyncio.sleep(0) print(b) asyncio.run(example_barrier()) Result of this example is: barrier passed Added in version 3.11. async wait() Pass the barrier. When all the tasks party to the barrier have called this function, they are all unblocked simultaneously. When a waiting or blocked task in the barrier is cancelled, this task exits the barrier which stays in the same state. If the state of the barrier is “filling”, the number of waiting task decreases by 1. The return value is an integer in the range of 0 to "parties-1", different for each task. This can be used to select a task to do some special housekeeping, e.g.: ... async with barrier as position: if position == 0: # Only one task prints this print('End of *draining phase*') This method may raise a "BrokenBarrierError" exception if the barrier is broken or reset while a task is waiting. It could raise a "CancelledError" if a task is cancelled. async reset() Return the barrier to the default, empty state. Any tasks waiting on it will receive the "BrokenBarrierError" exception. If a barrier is broken it may be better to just leave it and create a new one. async abort() Put the barrier into a broken state. This causes any active or future calls to "wait()" to fail with the "BrokenBarrierError". Use this for example if one of the tasks needs to abort, to avoid infinite waiting tasks. parties The number of tasks required to pass the barrier. n_waiting The number of tasks currently waiting in the barrier while filling. broken A boolean that is "True" if the barrier is in the broken state. exception asyncio.BrokenBarrierError This exception, a subclass of "RuntimeError", is raised when the "Barrier" object is reset or broken. ====================================================================== Changed in version 3.9: Acquiring a lock using "await lock" or "yield from lock" and/or "with" statement ("with await lock", "with (yield from lock)") was removed. Use "async with lock" instead. Coroutines and Tasks ******************** This section outlines high-level asyncio APIs to work with coroutines and Tasks. * Coroutines * Awaitables * Creating Tasks * Task Cancellation * Task Groups * Sleeping * Running Tasks Concurrently * Eager Task Factory * Shielding From Cancellation * Timeouts * Waiting Primitives * Running in Threads * Scheduling From Other Threads * Introspection * Task Object Coroutines ========== **Source code:** Lib/asyncio/coroutines.py ====================================================================== *Coroutines* declared with the async/await syntax is the preferred way of writing asyncio applications. For example, the following snippet of code prints “hello”, waits 1 second, and then prints “world”: >>> import asyncio >>> async def main(): ... print('hello') ... await asyncio.sleep(1) ... print('world') >>> asyncio.run(main()) hello world Note that simply calling a coroutine will not schedule it to be executed: >>> main() To actually run a coroutine, asyncio provides the following mechanisms: * The "asyncio.run()" function to run the top-level entry point “main()” function (see the above example.) * Awaiting on a coroutine. The following snippet of code will print “hello” after waiting for 1 second, and then print “world” after waiting for *another* 2 seconds: import asyncio import time async def say_after(delay, what): await asyncio.sleep(delay) print(what) async def main(): print(f"started at {time.strftime('%X')}") await say_after(1, 'hello') await say_after(2, 'world') print(f"finished at {time.strftime('%X')}") asyncio.run(main()) Expected output: started at 17:13:52 hello world finished at 17:13:55 * The "asyncio.create_task()" function to run coroutines concurrently as asyncio "Tasks". Let’s modify the above example and run two "say_after" coroutines *concurrently*: async def main(): task1 = asyncio.create_task( say_after(1, 'hello')) task2 = asyncio.create_task( say_after(2, 'world')) print(f"started at {time.strftime('%X')}") # Wait until both tasks are completed (should take # around 2 seconds.) await task1 await task2 print(f"finished at {time.strftime('%X')}") Note that expected output now shows that the snippet runs 1 second faster than before: started at 17:14:32 hello world finished at 17:14:34 * The "asyncio.TaskGroup" class provides a more modern alternative to "create_task()". Using this API, the last example becomes: async def main(): async with asyncio.TaskGroup() as tg: task1 = tg.create_task( say_after(1, 'hello')) task2 = tg.create_task( say_after(2, 'world')) print(f"started at {time.strftime('%X')}") # The await is implicit when the context manager exits. print(f"finished at {time.strftime('%X')}") The timing and output should be the same as for the previous version. Added in version 3.11: "asyncio.TaskGroup". Awaitables ========== We say that an object is an **awaitable** object if it can be used in an "await" expression. Many asyncio APIs are designed to accept awaitables. There are three main types of *awaitable* objects: **coroutines**, **Tasks**, and **Futures**. -[ Coroutines ]- Python coroutines are *awaitables* and therefore can be awaited from other coroutines: import asyncio async def nested(): return 42 async def main(): # Nothing happens if we just call "nested()". # A coroutine object is created but not awaited, # so it *won't run at all*. nested() # will raise a "RuntimeWarning". # Let's do it differently now and await it: print(await nested()) # will print "42". asyncio.run(main()) Important: In this documentation the term “coroutine” can be used for two closely related concepts: * a *coroutine function*: an "async def" function; * a *coroutine object*: an object returned by calling a *coroutine function*. -[ Tasks ]- *Tasks* are used to schedule coroutines *concurrently*. When a coroutine is wrapped into a *Task* with functions like "asyncio.create_task()" the coroutine is automatically scheduled to run soon: import asyncio async def nested(): return 42 async def main(): # Schedule nested() to run soon concurrently # with "main()". task = asyncio.create_task(nested()) # "task" can now be used to cancel "nested()", or # can simply be awaited to wait until it is complete: await task asyncio.run(main()) -[ Futures ]- A "Future" is a special **low-level** awaitable object that represents an **eventual result** of an asynchronous operation. When a Future object is *awaited* it means that the coroutine will wait until the Future is resolved in some other place. Future objects in asyncio are needed to allow callback-based code to be used with async/await. Normally **there is no need** to create Future objects at the application level code. Future objects, sometimes exposed by libraries and some asyncio APIs, can be awaited: async def main(): await function_that_returns_a_future_object() # this is also valid: await asyncio.gather( function_that_returns_a_future_object(), some_python_coroutine() ) A good example of a low-level function that returns a Future object is "loop.run_in_executor()". Creating Tasks ============== **Source code:** Lib/asyncio/tasks.py ====================================================================== asyncio.create_task(coro, *, name=None, context=None) Wrap the *coro* coroutine into a "Task" and schedule its execution. Return the Task object. If *name* is not "None", it is set as the name of the task using "Task.set_name()". An optional keyword-only *context* argument allows specifying a custom "contextvars.Context" for the *coro* to run in. The current context copy is created when no *context* is provided. The task is executed in the loop returned by "get_running_loop()", "RuntimeError" is raised if there is no running loop in current thread. Note: "asyncio.TaskGroup.create_task()" is a new alternative leveraging structural concurrency; it allows for waiting for a group of related tasks with strong safety guarantees. Important: Save a reference to the result of this function, to avoid a task disappearing mid-execution. The event loop only keeps weak references to tasks. A task that isn’t referenced elsewhere may get garbage collected at any time, even before it’s done. For reliable “fire-and-forget” background tasks, gather them in a collection: background_tasks = set() for i in range(10): task = asyncio.create_task(some_coro(param=i)) # Add task to the set. This creates a strong reference. background_tasks.add(task) # To prevent keeping references to finished tasks forever, # make each task remove its own reference from the set after # completion: task.add_done_callback(background_tasks.discard) Added in version 3.7. Changed in version 3.8: Added the *name* parameter. Changed in version 3.11: Added the *context* parameter. Task Cancellation ================= Tasks can easily and safely be cancelled. When a task is cancelled, "asyncio.CancelledError" will be raised in the task at the next opportunity. It is recommended that coroutines use "try/finally" blocks to robustly perform clean-up logic. In case "asyncio.CancelledError" is explicitly caught, it should generally be propagated when clean-up is complete. "asyncio.CancelledError" directly subclasses "BaseException" so most code will not need to be aware of it. The asyncio components that enable structured concurrency, like "asyncio.TaskGroup" and "asyncio.timeout()", are implemented using cancellation internally and might misbehave if a coroutine swallows "asyncio.CancelledError". Similarly, user code should not generally call "uncancel". However, in cases when suppressing "asyncio.CancelledError" is truly desired, it is necessary to also call "uncancel()" to completely remove the cancellation state. Task Groups =========== Task groups combine a task creation API with a convenient and reliable way to wait for all tasks in the group to finish. class asyncio.TaskGroup An asynchronous context manager holding a group of tasks. Tasks can be added to the group using "create_task()". All tasks are awaited when the context manager exits. Added in version 3.11. create_task(coro, *, name=None, context=None) Create a task in this task group. The signature matches that of "asyncio.create_task()". If the task group is inactive (e.g. not yet entered, already finished, or in the process of shutting down), we will close the given "coro". Changed in version 3.13: Close the given coroutine if the task group is not active. Example: async def main(): async with asyncio.TaskGroup() as tg: task1 = tg.create_task(some_coro(...)) task2 = tg.create_task(another_coro(...)) print(f"Both tasks have completed now: {task1.result()}, {task2.result()}") The "async with" statement will wait for all tasks in the group to finish. While waiting, new tasks may still be added to the group (for example, by passing "tg" into one of the coroutines and calling "tg.create_task()" in that coroutine). Once the last task has finished and the "async with" block is exited, no new tasks may be added to the group. The first time any of the tasks belonging to the group fails with an exception other than "asyncio.CancelledError", the remaining tasks in the group are cancelled. No further tasks can then be added to the group. At this point, if the body of the "async with" statement is still active (i.e., "__aexit__()" hasn’t been called yet), the task directly containing the "async with" statement is also cancelled. The resulting "asyncio.CancelledError" will interrupt an "await", but it will not bubble out of the containing "async with" statement. Once all tasks have finished, if any tasks have failed with an exception other than "asyncio.CancelledError", those exceptions are combined in an "ExceptionGroup" or "BaseExceptionGroup" (as appropriate; see their documentation) which is then raised. Two base exceptions are treated specially: If any task fails with "KeyboardInterrupt" or "SystemExit", the task group still cancels the remaining tasks and waits for them, but then the initial "KeyboardInterrupt" or "SystemExit" is re-raised instead of "ExceptionGroup" or "BaseExceptionGroup". If the body of the "async with" statement exits with an exception (so "__aexit__()" is called with an exception set), this is treated the same as if one of the tasks failed: the remaining tasks are cancelled and then waited for, and non-cancellation exceptions are grouped into an exception group and raised. The exception passed into "__aexit__()", unless it is "asyncio.CancelledError", is also included in the exception group. The same special case is made for "KeyboardInterrupt" and "SystemExit" as in the previous paragraph. Task groups are careful not to mix up the internal cancellation used to “wake up” their "__aexit__()" with cancellation requests for the task in which they are running made by other parties. In particular, when one task group is syntactically nested in another, and both experience an exception in one of their child tasks simultaneously, the inner task group will process its exceptions, and then the outer task group will receive another cancellation and process its own exceptions. In the case where a task group is cancelled externally and also must raise an "ExceptionGroup", it will call the parent task’s "cancel()" method. This ensures that a "asyncio.CancelledError" will be raised at the next "await", so the cancellation is not lost. Task groups preserve the cancellation count reported by "asyncio.Task.cancelling()". Changed in version 3.13: Improved handling of simultaneous internal and external cancellations and correct preservation of cancellation counts. Terminating a Task Group ------------------------ While terminating a task group is not natively supported by the standard library, termination can be achieved by adding an exception- raising task to the task group and ignoring the raised exception: import asyncio from asyncio import TaskGroup class TerminateTaskGroup(Exception): """Exception raised to terminate a task group.""" async def force_terminate_task_group(): """Used to force termination of a task group.""" raise TerminateTaskGroup() async def job(task_id, sleep_time): print(f'Task {task_id}: start') await asyncio.sleep(sleep_time) print(f'Task {task_id}: done') async def main(): try: async with TaskGroup() as group: # spawn some tasks group.create_task(job(1, 0.5)) group.create_task(job(2, 1.5)) # sleep for 1 second await asyncio.sleep(1) # add an exception-raising task to force the group to terminate group.create_task(force_terminate_task_group()) except* TerminateTaskGroup: pass asyncio.run(main()) Expected output: Task 1: start Task 2: start Task 1: done Sleeping ======== async asyncio.sleep(delay, result=None) Block for *delay* seconds. If *result* is provided, it is returned to the caller when the coroutine completes. "sleep()" always suspends the current task, allowing other tasks to run. Setting the delay to 0 provides an optimized path to allow other tasks to run. This can be used by long-running functions to avoid blocking the event loop for the full duration of the function call. Example of coroutine displaying the current date every second for 5 seconds: import asyncio import datetime async def display_date(): loop = asyncio.get_running_loop() end_time = loop.time() + 5.0 while True: print(datetime.datetime.now()) if (loop.time() + 1.0) >= end_time: break await asyncio.sleep(1) asyncio.run(display_date()) Changed in version 3.10: Removed the *loop* parameter. Changed in version 3.13: Raises "ValueError" if *delay* is "nan". Running Tasks Concurrently ========================== awaitable asyncio.gather(*aws, return_exceptions=False) Run awaitable objects in the *aws* sequence *concurrently*. If any awaitable in *aws* is a coroutine, it is automatically scheduled as a Task. If all awaitables are completed successfully, the result is an aggregate list of returned values. The order of result values corresponds to the order of awaitables in *aws*. If *return_exceptions* is "False" (default), the first raised exception is immediately propagated to the task that awaits on "gather()". Other awaitables in the *aws* sequence **won’t be cancelled** and will continue to run. If *return_exceptions* is "True", exceptions are treated the same as successful results, and aggregated in the result list. If "gather()" is *cancelled*, all submitted awaitables (that have not completed yet) are also *cancelled*. If any Task or Future from the *aws* sequence is *cancelled*, it is treated as if it raised "CancelledError" – the "gather()" call is **not** cancelled in this case. This is to prevent the cancellation of one submitted Task/Future to cause other Tasks/Futures to be cancelled. Note: A new alternative to create and run tasks concurrently and wait for their completion is "asyncio.TaskGroup". *TaskGroup* provides stronger safety guarantees than *gather* for scheduling a nesting of subtasks: if a task (or a subtask, a task scheduled by a task) raises an exception, *TaskGroup* will, while *gather* will not, cancel the remaining scheduled tasks). Example: import asyncio async def factorial(name, number): f = 1 for i in range(2, number + 1): print(f"Task {name}: Compute factorial({number}), currently i={i}...") await asyncio.sleep(1) f *= i print(f"Task {name}: factorial({number}) = {f}") return f async def main(): # Schedule three calls *concurrently*: L = await asyncio.gather( factorial("A", 2), factorial("B", 3), factorial("C", 4), ) print(L) asyncio.run(main()) # Expected output: # # Task A: Compute factorial(2), currently i=2... # Task B: Compute factorial(3), currently i=2... # Task C: Compute factorial(4), currently i=2... # Task A: factorial(2) = 2 # Task B: Compute factorial(3), currently i=3... # Task C: Compute factorial(4), currently i=3... # Task B: factorial(3) = 6 # Task C: Compute factorial(4), currently i=4... # Task C: factorial(4) = 24 # [2, 6, 24] Note: If *return_exceptions* is false, cancelling gather() after it has been marked done won’t cancel any submitted awaitables. For instance, gather can be marked done after propagating an exception to the caller, therefore, calling "gather.cancel()" after catching an exception (raised by one of the awaitables) from gather won’t cancel any other awaitables. Changed in version 3.7: If the *gather* itself is cancelled, the cancellation is propagated regardless of *return_exceptions*. Changed in version 3.10: Removed the *loop* parameter. Deprecated since version 3.10: Deprecation warning is emitted if no positional arguments are provided or not all positional arguments are Future-like objects and there is no running event loop. Eager Task Factory ================== asyncio.eager_task_factory(loop, coro, *, name=None, context=None) A task factory for eager task execution. When using this factory (via "loop.set_task_factory(asyncio.eager_task_factory)"), coroutines begin execution synchronously during "Task" construction. Tasks are only scheduled on the event loop if they block. This can be a performance improvement as the overhead of loop scheduling is avoided for coroutines that complete synchronously. A common example where this is beneficial is coroutines which employ caching or memoization to avoid actual I/O when possible. Note: Immediate execution of the coroutine is a semantic change. If the coroutine returns or raises, the task is never scheduled to the event loop. If the coroutine execution blocks, the task is scheduled to the event loop. This change may introduce behavior changes to existing applications. For example, the application’s task execution order is likely to change. Added in version 3.12. asyncio.create_eager_task_factory(custom_task_constructor) Create an eager task factory, similar to "eager_task_factory()", using the provided *custom_task_constructor* when creating a new task instead of the default "Task". *custom_task_constructor* must be a *callable* with the signature matching the signature of "Task.__init__". The callable must return a "asyncio.Task"-compatible object. This function returns a *callable* intended to be used as a task factory of an event loop via "loop.set_task_factory(factory)"). Added in version 3.12. Shielding From Cancellation =========================== awaitable asyncio.shield(aw) Protect an awaitable object from being "cancelled". If *aw* is a coroutine it is automatically scheduled as a Task. The statement: task = asyncio.create_task(something()) res = await shield(task) is equivalent to: res = await something() *except* that if the coroutine containing it is cancelled, the Task running in "something()" is not cancelled. From the point of view of "something()", the cancellation did not happen. Although its caller is still cancelled, so the “await” expression still raises a "CancelledError". If "something()" is cancelled by other means (i.e. from within itself) that would also cancel "shield()". If it is desired to completely ignore cancellation (not recommended) the "shield()" function should be combined with a try/except clause, as follows: task = asyncio.create_task(something()) try: res = await shield(task) except CancelledError: res = None Important: Save a reference to tasks passed to this function, to avoid a task disappearing mid-execution. The event loop only keeps weak references to tasks. A task that isn’t referenced elsewhere may get garbage collected at any time, even before it’s done. Changed in version 3.10: Removed the *loop* parameter. Deprecated since version 3.10: Deprecation warning is emitted if *aw* is not Future-like object and there is no running event loop. Timeouts ======== asyncio.timeout(delay) Return an asynchronous context manager that can be used to limit the amount of time spent waiting on something. *delay* can either be "None", or a float/int number of seconds to wait. If *delay* is "None", no time limit will be applied; this can be useful if the delay is unknown when the context manager is created. In either case, the context manager can be rescheduled after creation using "Timeout.reschedule()". Example: async def main(): async with asyncio.timeout(10): await long_running_task() If "long_running_task" takes more than 10 seconds to complete, the context manager will cancel the current task and handle the resulting "asyncio.CancelledError" internally, transforming it into a "TimeoutError" which can be caught and handled. Note: The "asyncio.timeout()" context manager is what transforms the "asyncio.CancelledError" into a "TimeoutError", which means the "TimeoutError" can only be caught *outside* of the context manager. Example of catching "TimeoutError": async def main(): try: async with asyncio.timeout(10): await long_running_task() except TimeoutError: print("The long operation timed out, but we've handled it.") print("This statement will run regardless.") The context manager produced by "asyncio.timeout()" can be rescheduled to a different deadline and inspected. class asyncio.Timeout(when) An asynchronous context manager for cancelling overdue coroutines. "when" should be an absolute time at which the context should time out, as measured by the event loop’s clock: * If "when" is "None", the timeout will never trigger. * If "when < loop.time()", the timeout will trigger on the next iteration of the event loop. when() -> float | None Return the current deadline, or "None" if the current deadline is not set. reschedule(when: float | None) Reschedule the timeout. expired() -> bool Return whether the context manager has exceeded its deadline (expired). Example: async def main(): try: # We do not know the timeout when starting, so we pass ``None``. async with asyncio.timeout(None) as cm: # We know the timeout now, so we reschedule it. new_deadline = get_running_loop().time() + 10 cm.reschedule(new_deadline) await long_running_task() except TimeoutError: pass if cm.expired(): print("Looks like we haven't finished on time.") Timeout context managers can be safely nested. Added in version 3.11. asyncio.timeout_at(when) Similar to "asyncio.timeout()", except *when* is the absolute time to stop waiting, or "None". Example: async def main(): loop = get_running_loop() deadline = loop.time() + 20 try: async with asyncio.timeout_at(deadline): await long_running_task() except TimeoutError: print("The long operation timed out, but we've handled it.") print("This statement will run regardless.") Added in version 3.11. async asyncio.wait_for(aw, timeout) Wait for the *aw* awaitable to complete with a timeout. If *aw* is a coroutine it is automatically scheduled as a Task. *timeout* can either be "None" or a float or int number of seconds to wait for. If *timeout* is "None", block until the future completes. If a timeout occurs, it cancels the task and raises "TimeoutError". To avoid the task "cancellation", wrap it in "shield()". The function will wait until the future is actually cancelled, so the total wait time may exceed the *timeout*. If an exception happens during cancellation, it is propagated. If the wait is cancelled, the future *aw* is also cancelled. Example: async def eternity(): # Sleep for one hour await asyncio.sleep(3600) print('yay!') async def main(): # Wait for at most 1 second try: await asyncio.wait_for(eternity(), timeout=1.0) except TimeoutError: print('timeout!') asyncio.run(main()) # Expected output: # # timeout! Changed in version 3.7: When *aw* is cancelled due to a timeout, "wait_for" waits for *aw* to be cancelled. Previously, it raised "TimeoutError" immediately. Changed in version 3.10: Removed the *loop* parameter. Changed in version 3.11: Raises "TimeoutError" instead of "asyncio.TimeoutError". Waiting Primitives ================== async asyncio.wait(aws, *, timeout=None, return_when=ALL_COMPLETED) Run "Future" and "Task" instances in the *aws* iterable concurrently and block until the condition specified by *return_when*. The *aws* iterable must not be empty. Returns two sets of Tasks/Futures: "(done, pending)". Usage: done, pending = await asyncio.wait(aws) *timeout* (a float or int), if specified, can be used to control the maximum number of seconds to wait before returning. Note that this function does not raise "TimeoutError". Futures or Tasks that aren’t done when the timeout occurs are simply returned in the second set. *return_when* indicates when this function should return. It must be one of the following constants: +----------------------------------------------------+----------------------------------------------------+ | Constant | Description | |====================================================|====================================================| | asyncio.FIRST_COMPLETED | The function will return when any future finishes | | | or is cancelled. | +----------------------------------------------------+----------------------------------------------------+ | asyncio.FIRST_EXCEPTION | The function will return when any future finishes | | | by raising an exception. If no future raises an | | | exception then it is equivalent to | | | "ALL_COMPLETED". | +----------------------------------------------------+----------------------------------------------------+ | asyncio.ALL_COMPLETED | The function will return when all futures finish | | | or are cancelled. | +----------------------------------------------------+----------------------------------------------------+ Unlike "wait_for()", "wait()" does not cancel the futures when a timeout occurs. Changed in version 3.10: Removed the *loop* parameter. Changed in version 3.11: Passing coroutine objects to "wait()" directly is forbidden. Changed in version 3.12: Added support for generators yielding tasks. asyncio.as_completed(aws, *, timeout=None) Run awaitable objects in the *aws* iterable concurrently. The returned object can be iterated to obtain the results of the awaitables as they finish. The object returned by "as_completed()" can be iterated as an *asynchronous iterator* or a plain *iterator*. When asynchronous iteration is used, the originally-supplied awaitables are yielded if they are tasks or futures. This makes it easy to correlate previously-scheduled tasks with their results. Example: ipv4_connect = create_task(open_connection("127.0.0.1", 80)) ipv6_connect = create_task(open_connection("::1", 80)) tasks = [ipv4_connect, ipv6_connect] async for earliest_connect in as_completed(tasks): # earliest_connect is done. The result can be obtained by # awaiting it or calling earliest_connect.result() reader, writer = await earliest_connect if earliest_connect is ipv6_connect: print("IPv6 connection established.") else: print("IPv4 connection established.") During asynchronous iteration, implicitly-created tasks will be yielded for supplied awaitables that aren’t tasks or futures. When used as a plain iterator, each iteration yields a new coroutine that returns the result or raises the exception of the next completed awaitable. This pattern is compatible with Python versions older than 3.13: ipv4_connect = create_task(open_connection("127.0.0.1", 80)) ipv6_connect = create_task(open_connection("::1", 80)) tasks = [ipv4_connect, ipv6_connect] for next_connect in as_completed(tasks): # next_connect is not one of the original task objects. It must be # awaited to obtain the result value or raise the exception of the # awaitable that finishes next. reader, writer = await next_connect A "TimeoutError" is raised if the timeout occurs before all awaitables are done. This is raised by the "async for" loop during asynchronous iteration or by the coroutines yielded during plain iteration. Changed in version 3.10: Removed the *loop* parameter. Deprecated since version 3.10: Deprecation warning is emitted if not all awaitable objects in the *aws* iterable are Future-like objects and there is no running event loop. Changed in version 3.12: Added support for generators yielding tasks. Changed in version 3.13: The result can now be used as either an *asynchronous iterator* or as a plain *iterator* (previously it was only a plain iterator). Running in Threads ================== async asyncio.to_thread(func, /, *args, **kwargs) Asynchronously run function *func* in a separate thread. Any *args and **kwargs supplied for this function are directly passed to *func*. Also, the current "contextvars.Context" is propagated, allowing context variables from the event loop thread to be accessed in the separate thread. Return a coroutine that can be awaited to get the eventual result of *func*. This coroutine function is primarily intended to be used for executing IO-bound functions/methods that would otherwise block the event loop if they were run in the main thread. For example: def blocking_io(): print(f"start blocking_io at {time.strftime('%X')}") # Note that time.sleep() can be replaced with any blocking # IO-bound operation, such as file operations. time.sleep(1) print(f"blocking_io complete at {time.strftime('%X')}") async def main(): print(f"started main at {time.strftime('%X')}") await asyncio.gather( asyncio.to_thread(blocking_io), asyncio.sleep(1)) print(f"finished main at {time.strftime('%X')}") asyncio.run(main()) # Expected output: # # started main at 19:50:53 # start blocking_io at 19:50:53 # blocking_io complete at 19:50:54 # finished main at 19:50:54 Directly calling "blocking_io()" in any coroutine would block the event loop for its duration, resulting in an additional 1 second of run time. Instead, by using "asyncio.to_thread()", we can run it in a separate thread without blocking the event loop. Note: Due to the *GIL*, "asyncio.to_thread()" can typically only be used to make IO-bound functions non-blocking. However, for extension modules that release the GIL or alternative Python implementations that don’t have one, "asyncio.to_thread()" can also be used for CPU-bound functions. Added in version 3.9. Scheduling From Other Threads ============================= asyncio.run_coroutine_threadsafe(coro, loop) Submit a coroutine to the given event loop. Thread-safe. Return a "concurrent.futures.Future" to wait for the result from another OS thread. This function is meant to be called from a different OS thread than the one where the event loop is running. Example: # Create a coroutine coro = asyncio.sleep(1, result=3) # Submit the coroutine to a given loop future = asyncio.run_coroutine_threadsafe(coro, loop) # Wait for the result with an optional timeout argument assert future.result(timeout) == 3 If an exception is raised in the coroutine, the returned Future will be notified. It can also be used to cancel the task in the event loop: try: result = future.result(timeout) except TimeoutError: print('The coroutine took too long, cancelling the task...') future.cancel() except Exception as exc: print(f'The coroutine raised an exception: {exc!r}') else: print(f'The coroutine returned: {result!r}') See the concurrency and multithreading section of the documentation. Unlike other asyncio functions this function requires the *loop* argument to be passed explicitly. Added in version 3.5.1. Introspection ============= asyncio.current_task(loop=None) Return the currently running "Task" instance, or "None" if no task is running. If *loop* is "None" "get_running_loop()" is used to get the current loop. Added in version 3.7. asyncio.all_tasks(loop=None) Return a set of not yet finished "Task" objects run by the loop. If *loop* is "None", "get_running_loop()" is used for getting current loop. Added in version 3.7. asyncio.iscoroutine(obj) Return "True" if *obj* is a coroutine object. Added in version 3.4. Task Object =========== class asyncio.Task(coro, *, loop=None, name=None, context=None, eager_start=False) A "Future-like" object that runs a Python coroutine. Not thread- safe. Tasks are used to run coroutines in event loops. If a coroutine awaits on a Future, the Task suspends the execution of the coroutine and waits for the completion of the Future. When the Future is *done*, the execution of the wrapped coroutine resumes. Event loops use cooperative scheduling: an event loop runs one Task at a time. While a Task awaits for the completion of a Future, the event loop runs other Tasks, callbacks, or performs IO operations. Use the high-level "asyncio.create_task()" function to create Tasks, or the low-level "loop.create_task()" or "ensure_future()" functions. Manual instantiation of Tasks is discouraged. To cancel a running Task use the "cancel()" method. Calling it will cause the Task to throw a "CancelledError" exception into the wrapped coroutine. If a coroutine is awaiting on a Future object during cancellation, the Future object will be cancelled. "cancelled()" can be used to check if the Task was cancelled. The method returns "True" if the wrapped coroutine did not suppress the "CancelledError" exception and was actually cancelled. "asyncio.Task" inherits from "Future" all of its APIs except "Future.set_result()" and "Future.set_exception()". An optional keyword-only *context* argument allows specifying a custom "contextvars.Context" for the *coro* to run in. If no *context* is provided, the Task copies the current context and later runs its coroutine in the copied context. An optional keyword-only *eager_start* argument allows eagerly starting the execution of the "asyncio.Task" at task creation time. If set to "True" and the event loop is running, the task will start executing the coroutine immediately, until the first time the coroutine blocks. If the coroutine returns or raises without blocking, the task will be finished eagerly and will skip scheduling to the event loop. Changed in version 3.7: Added support for the "contextvars" module. Changed in version 3.8: Added the *name* parameter. Deprecated since version 3.10: Deprecation warning is emitted if *loop* is not specified and there is no running event loop. Changed in version 3.11: Added the *context* parameter. Changed in version 3.12: Added the *eager_start* parameter. done() Return "True" if the Task is *done*. A Task is *done* when the wrapped coroutine either returned a value, raised an exception, or the Task was cancelled. result() Return the result of the Task. If the Task is *done*, the result of the wrapped coroutine is returned (or if the coroutine raised an exception, that exception is re-raised.) If the Task has been *cancelled*, this method raises a "CancelledError" exception. If the Task’s result isn’t yet available, this method raises an "InvalidStateError" exception. exception() Return the exception of the Task. If the wrapped coroutine raised an exception that exception is returned. If the wrapped coroutine returned normally this method returns "None". If the Task has been *cancelled*, this method raises a "CancelledError" exception. If the Task isn’t *done* yet, this method raises an "InvalidStateError" exception. add_done_callback(callback, *, context=None) Add a callback to be run when the Task is *done*. This method should only be used in low-level callback-based code. See the documentation of "Future.add_done_callback()" for more details. remove_done_callback(callback) Remove *callback* from the callbacks list. This method should only be used in low-level callback-based code. See the documentation of "Future.remove_done_callback()" for more details. get_stack(*, limit=None) Return the list of stack frames for this Task. If the wrapped coroutine is not done, this returns the stack where it is suspended. If the coroutine has completed successfully or was cancelled, this returns an empty list. If the coroutine was terminated by an exception, this returns the list of traceback frames. The frames are always ordered from oldest to newest. Only one stack frame is returned for a suspended coroutine. The optional *limit* argument sets the maximum number of frames to return; by default all available frames are returned. The ordering of the returned list differs depending on whether a stack or a traceback is returned: the newest frames of a stack are returned, but the oldest frames of a traceback are returned. (This matches the behavior of the traceback module.) print_stack(*, limit=None, file=None) Print the stack or traceback for this Task. This produces output similar to that of the traceback module for the frames retrieved by "get_stack()". The *limit* argument is passed to "get_stack()" directly. The *file* argument is an I/O stream to which the output is written; by default output is written to "sys.stdout". get_coro() Return the coroutine object wrapped by the "Task". Note: This will return "None" for Tasks which have already completed eagerly. See the Eager Task Factory. Added in version 3.8. Changed in version 3.12: Newly added eager task execution means result may be "None". get_context() Return the "contextvars.Context" object associated with the task. Added in version 3.12. get_name() Return the name of the Task. If no name has been explicitly assigned to the Task, the default asyncio Task implementation generates a default name during instantiation. Added in version 3.8. set_name(value) Set the name of the Task. The *value* argument can be any object, which is then converted to a string. In the default Task implementation, the name will be visible in the "repr()" output of a task object. Added in version 3.8. cancel(msg=None) Request the Task to be cancelled. If the Task is already *done* or *cancelled*, return "False", otherwise, return "True". The method arranges for a "CancelledError" exception to be thrown into the wrapped coroutine on the next cycle of the event loop. The coroutine then has a chance to clean up or even deny the request by suppressing the exception with a "try" … … "except CancelledError" … "finally" block. Therefore, unlike "Future.cancel()", "Task.cancel()" does not guarantee that the Task will be cancelled, although suppressing cancellation completely is not common and is actively discouraged. Should the coroutine nevertheless decide to suppress the cancellation, it needs to call "Task.uncancel()" in addition to catching the exception. Changed in version 3.9: Added the *msg* parameter. Changed in version 3.11: The "msg" parameter is propagated from cancelled task to its awaiter. The following example illustrates how coroutines can intercept the cancellation request: async def cancel_me(): print('cancel_me(): before sleep') try: # Wait for 1 hour await asyncio.sleep(3600) except asyncio.CancelledError: print('cancel_me(): cancel sleep') raise finally: print('cancel_me(): after sleep') async def main(): # Create a "cancel_me" Task task = asyncio.create_task(cancel_me()) # Wait for 1 second await asyncio.sleep(1) task.cancel() try: await task except asyncio.CancelledError: print("main(): cancel_me is cancelled now") asyncio.run(main()) # Expected output: # # cancel_me(): before sleep # cancel_me(): cancel sleep # cancel_me(): after sleep # main(): cancel_me is cancelled now cancelled() Return "True" if the Task is *cancelled*. The Task is *cancelled* when the cancellation was requested with "cancel()" and the wrapped coroutine propagated the "CancelledError" exception thrown into it. uncancel() Decrement the count of cancellation requests to this Task. Returns the remaining number of cancellation requests. Note that once execution of a cancelled task completed, further calls to "uncancel()" are ineffective. Added in version 3.11. This method is used by asyncio’s internals and isn’t expected to be used by end-user code. In particular, if a Task gets successfully uncancelled, this allows for elements of structured concurrency like Task Groups and "asyncio.timeout()" to continue running, isolating cancellation to the respective structured block. For example: async def make_request_with_timeout(): try: async with asyncio.timeout(1): # Structured block affected by the timeout: await make_request() await make_another_request() except TimeoutError: log("There was a timeout") # Outer code not affected by the timeout: await unrelated_code() While the block with "make_request()" and "make_another_request()" might get cancelled due to the timeout, "unrelated_code()" should continue running even in case of the timeout. This is implemented with "uncancel()". "TaskGroup" context managers use "uncancel()" in a similar fashion. If end-user code is, for some reason, suppressing cancellation by catching "CancelledError", it needs to call this method to remove the cancellation state. When this method decrements the cancellation count to zero, the method checks if a previous "cancel()" call had arranged for "CancelledError" to be thrown into the task. If it hasn’t been thrown yet, that arrangement will be rescinded (by resetting the internal "_must_cancel" flag). Changed in version 3.13: Changed to rescind pending cancellation requests upon reaching zero. cancelling() Return the number of pending cancellation requests to this Task, i.e., the number of calls to "cancel()" less the number of "uncancel()" calls. Note that if this number is greater than zero but the Task is still executing, "cancelled()" will still return "False". This is because this number can be lowered by calling "uncancel()", which can lead to the task not being cancelled after all if the cancellation requests go down to zero. This method is used by asyncio’s internals and isn’t expected to be used by end-user code. See "uncancel()" for more details. Added in version 3.11. "asyncio" — Asynchronous I/O **************************** ====================================================================== Hello World! ^^^^^^^^^^^^ import asyncio async def main(): print('Hello ...') await asyncio.sleep(1) print('... World!') asyncio.run(main()) asyncio is a library to write **concurrent** code using the **async/await** syntax. asyncio is used as a foundation for multiple Python asynchronous frameworks that provide high-performance network and web-servers, database connection libraries, distributed task queues, etc. asyncio is often a perfect fit for IO-bound and high-level **structured** network code. See also: A Conceptual Overview of asyncio Explanation of the fundamentals of asyncio. asyncio provides a set of **high-level** APIs to: * run Python coroutines concurrently and have full control over their execution; * perform network IO and IPC; * control subprocesses; * distribute tasks via queues; * synchronize concurrent code; Additionally, there are **low-level** APIs for *library and framework developers* to: * create and manage event loops, which provide asynchronous APIs for networking, running subprocesses, handling OS signals, etc; * implement efficient protocols using transports; * bridge callback-based libraries and code with async/await syntax. Availability: not WASI. This module does not work or is not available on WebAssembly. See WebAssembly platforms for more information. -[ asyncio REPL ]- You can experiment with an "asyncio" concurrent context in the *REPL*: $ python -m asyncio asyncio REPL ... Use "await" directly instead of "asyncio.run()". Type "help", "copyright", "credits" or "license" for more information. >>> import asyncio >>> await asyncio.sleep(10, result='hello') 'hello' Raises an auditing event "cpython.run_stdin" with no arguments. Changed in version 3.12.5: (also 3.11.10, 3.10.15, 3.9.20, and 3.8.20) Emits audit events. Changed in version 3.13: Uses PyREPL if possible, in which case "PYTHONSTARTUP" is also executed. Emits audit events. -[ Reference ]- High-level APIs ^^^^^^^^^^^^^^^ * Runners * Coroutines and Tasks * Streams * Synchronization Primitives * Subprocesses * Queues * Exceptions Low-level APIs ^^^^^^^^^^^^^^ * Event Loop * Futures * Transports and Protocols * Policies * Platform Support * Extending Guides and Tutorials ^^^^^^^^^^^^^^^^^^^^ * High-level API Index * Low-level API Index * Developing with asyncio Note: The source code for asyncio can be found in Lib/asyncio/. "asyncore" — Asynchronous socket handler **************************************** Deprecated since version 3.6, removed in version 3.12. This module is no longer part of the Python standard library. It was removed in Python 3.12 after being deprecated in Python 3.6. The removal was decided in **PEP 594**. Applications should use the "asyncio" module instead. The last version of Python that provided the "asyncore" module was Python 3.11. "atexit" — Exit handlers ************************ ====================================================================== The "atexit" module defines functions to register and unregister cleanup functions. Functions thus registered are automatically executed upon normal interpreter termination. "atexit" runs these functions in the *reverse* order in which they were registered; if you register "A", "B", and "C", at interpreter termination time they will be run in the order "C", "B", "A". **Note:** The functions registered via this module are not called when the program is killed by a signal not handled by Python, when a Python fatal internal error is detected, or when "os._exit()" is called. **Note:** The effect of registering or unregistering functions from within a cleanup function is undefined. Changed in version 3.7: When used with C-API subinterpreters, registered functions are local to the interpreter they were registered in. atexit.register(func, *args, **kwargs) Register *func* as a function to be executed at termination. Any optional arguments that are to be passed to *func* must be passed as arguments to "register()". It is possible to register the same function and arguments more than once. At normal program termination (for instance, if "sys.exit()" is called or the main module’s execution completes), all functions registered are called in last in, first out order. The assumption is that lower level modules will normally be imported before higher level modules and thus must be cleaned up later. If an exception is raised during execution of the exit handlers, a traceback is printed (unless "SystemExit" is raised) and the exception information is saved. After all exit handlers have had a chance to run, the last exception to be raised is re-raised. This function returns *func*, which makes it possible to use it as a decorator. Warning: Starting new threads or calling "os.fork()" from a registered function can lead to race condition between the main Python runtime thread freeing thread states while internal "threading" routines or the new process try to use that state. This can lead to crashes rather than clean shutdown. Changed in version 3.12: Attempts to start a new thread or "os.fork()" a new process in a registered function now leads to "RuntimeError". atexit.unregister(func) Remove *func* from the list of functions to be run at interpreter shutdown. "unregister()" silently does nothing if *func* was not previously registered. If *func* has been registered more than once, every occurrence of that function in the "atexit" call stack will be removed. Equality comparisons ("==") are used internally during unregistration, so function references do not need to have matching identities. See also: Module "readline" Useful example of "atexit" to read and write "readline" history files. "atexit" Example ================ The following simple example demonstrates how a module can initialize a counter from a file when it is imported and save the counter’s updated value automatically when the program terminates without relying on the application making an explicit call into this module at termination. try: with open('counterfile') as infile: _count = int(infile.read()) except FileNotFoundError: _count = 0 def incrcounter(n): global _count _count = _count + n def savecounter(): with open('counterfile', 'w') as outfile: outfile.write('%d' % _count) import atexit atexit.register(savecounter) Positional and keyword arguments may also be passed to "register()" to be passed along to the registered function when it is called: def goodbye(name, adjective): print('Goodbye %s, it was %s to meet you.' % (name, adjective)) import atexit atexit.register(goodbye, 'Donny', 'nice') # or: atexit.register(goodbye, adjective='nice', name='Donny') Usage as a *decorator*: import atexit @atexit.register def goodbye(): print('You are now leaving the Python sector.') This only works with functions that can be called without arguments. "audioop" — Manipulate raw audio data ************************************* Deprecated since version 3.11, removed in version 3.13. This module is no longer part of the Python standard library. It was removed in Python 3.13 after being deprecated in Python 3.11. The removal was decided in **PEP 594**. The last version of Python that provided the "audioop" module was Python 3.12. Audit events table ****************** This table contains all events raised by "sys.audit()" or "PySys_Audit()" calls throughout the CPython runtime and the standard library. These calls were added in 3.8 or later (see **PEP 578**). See "sys.addaudithook()" and "PySys_AddAuditHook()" for information on handling these events. **CPython implementation detail:** This table is generated from the CPython documentation, and may not represent events raised by other implementations. See your runtime specific documentation for actual events raised. +--------------------------------+---------------------------------------------------------+-----------------+ | Audit event | Arguments | References | |================================|=========================================================|=================| | _thread.start_new_thread | "function", "args", "kwargs" | [1] | +--------------------------------+---------------------------------------------------------+-----------------+ | array.__new__ | "typecode", "initializer" | [1] | +--------------------------------+---------------------------------------------------------+-----------------+ | builtins.breakpoint | "breakpointhook" | [1] | +--------------------------------+---------------------------------------------------------+-----------------+ | builtins.id | "id" | [1] | +--------------------------------+---------------------------------------------------------+-----------------+ | builtins.input | "prompt" | [1] | +--------------------------------+---------------------------------------------------------+-----------------+ | builtins.input/result | "result" | [1] | +--------------------------------+---------------------------------------------------------+-----------------+ | code.__new__ | "code", "filename", "name", "argcount", | [1] | | | "posonlyargcount", "kwonlyargcount", "nlocals", | | | | "stacksize", "flags" | | +--------------------------------+---------------------------------------------------------+-----------------+ | compile | "source", "filename" | [1] | +--------------------------------+---------------------------------------------------------+-----------------+ | cpython.PyInterpreterState_Cl | | [1] | | ear | | | +--------------------------------+---------------------------------------------------------+-----------------+ | cpython.PyInterpreterState_New | | [1] | +--------------------------------+---------------------------------------------------------+-----------------+ | cpython._PySys_ClearAuditHooks | | [1] | +--------------------------------+---------------------------------------------------------+-----------------+ | cpython.run_command | "command" | [1] | +--------------------------------+---------------------------------------------------------+-----------------+ | cpython.run_file | "filename" | [1] | +--------------------------------+---------------------------------------------------------+-----------------+ | cpython.run_interactivehook | "hook" | [1] | +--------------------------------+---------------------------------------------------------+-----------------+ | cpython.run_module | "module-name" | [1] | +--------------------------------+---------------------------------------------------------+-----------------+ | cpython.run_startup | "filename" | [1] | +--------------------------------+---------------------------------------------------------+-----------------+ | cpython.run_stdin | | [1][2][3] | +--------------------------------+---------------------------------------------------------+-----------------+ | ctypes.addressof | "obj" | [1] | +--------------------------------+---------------------------------------------------------+-----------------+ | ctypes.call_function | "func_pointer", "arguments" | [1] | +--------------------------------+---------------------------------------------------------+-----------------+ | ctypes.cdata | "address" | [1] | +--------------------------------+---------------------------------------------------------+-----------------+ | ctypes.cdata/buffer | "pointer", "size", "offset" | [1][2] | +--------------------------------+---------------------------------------------------------+-----------------+ | ctypes.create_string_buffer | "init", "size" | [1] | +--------------------------------+---------------------------------------------------------+-----------------+ | ctypes.create_unicode_buffer | "init", "size" | [1] | +--------------------------------+---------------------------------------------------------+-----------------+ | ctypes.dlopen | "name" | [1] | +--------------------------------+---------------------------------------------------------+-----------------+ | ctypes.dlsym | "library", "name" | [1] | +--------------------------------+---------------------------------------------------------+-----------------+ | ctypes.dlsym/handle | "handle", "name" | [1] | +--------------------------------+---------------------------------------------------------+-----------------+ | ctypes.get_errno | | [1] | +--------------------------------+---------------------------------------------------------+-----------------+ | ctypes.get_last_error | | [1] | +--------------------------------+---------------------------------------------------------+-----------------+ | ctypes.set_errno | "errno" | [1] | +--------------------------------+---------------------------------------------------------+-----------------+ | ctypes.set_exception | "code" | [1] | +--------------------------------+---------------------------------------------------------+-----------------+ | ctypes.set_last_error | "error" | [1] | +--------------------------------+---------------------------------------------------------+-----------------+ | ctypes.string_at | "ptr", "size" | [1] | +--------------------------------+---------------------------------------------------------+-----------------+ | ctypes.wstring_at | "ptr", "size" | [1] | +--------------------------------+---------------------------------------------------------+-----------------+ | ensurepip.bootstrap | "root" | [1] | +--------------------------------+---------------------------------------------------------+-----------------+ | exec | "code_object" | [1][2] | +--------------------------------+---------------------------------------------------------+-----------------+ | fcntl.fcntl | "fd", "cmd", "arg" | [1] | +--------------------------------+---------------------------------------------------------+-----------------+ | fcntl.flock | "fd", "operation" | [1] | +--------------------------------+---------------------------------------------------------+-----------------+ | fcntl.ioctl | "fd", "request", "arg" | [1] | +--------------------------------+---------------------------------------------------------+-----------------+ | fcntl.lockf | "fd", "cmd", "len", "start", "whence" | [1] | +--------------------------------+---------------------------------------------------------+-----------------+ | ftplib.connect | "self", "host", "port" | [1] | +--------------------------------+---------------------------------------------------------+-----------------+ | ftplib.sendcmd | "self", "cmd" | [1][2] | +--------------------------------+---------------------------------------------------------+-----------------+ | function.__new__ | "code" | [1] | +--------------------------------+---------------------------------------------------------+-----------------+ | gc.get_objects | "generation" | [1] | +--------------------------------+---------------------------------------------------------+-----------------+ | gc.get_referents | "objs" | [1] | +--------------------------------+---------------------------------------------------------+-----------------+ | gc.get_referrers | "objs" | [1] | +--------------------------------+---------------------------------------------------------+-----------------+ | glob.glob | "pathname", "recursive" | [1][2] | +--------------------------------+---------------------------------------------------------+-----------------+ | glob.glob/2 | "pathname", "recursive", "root_dir", "dir_fd" | [1][2] | +--------------------------------+---------------------------------------------------------+-----------------+ | http.client.connect | "self", "host", "port" | [1] | +--------------------------------+---------------------------------------------------------+-----------------+ | http.client.send | "self", "data" | [1] | +--------------------------------+---------------------------------------------------------+-----------------+ | imaplib.open | "self", "host", "port" | [1] | +--------------------------------+---------------------------------------------------------+-----------------+ | imaplib.send | "self", "data" | [1] | +--------------------------------+---------------------------------------------------------+-----------------+ | import | "module", "filename", "sys.path", "sys.meta_path", | [1] | | | "sys.path_hooks" | | +--------------------------------+---------------------------------------------------------+-----------------+ | marshal.dumps | "value", "version" | [1] | +--------------------------------+---------------------------------------------------------+-----------------+ | marshal.load | | [1] | +--------------------------------+---------------------------------------------------------+-----------------+ | marshal.loads | "bytes" | [1] | +--------------------------------+---------------------------------------------------------+-----------------+ | mmap.__new__ | "fileno", "length", "access", "offset" | [1] | +--------------------------------+---------------------------------------------------------+-----------------+ | msvcrt.get_osfhandle | "fd" | [1] | +--------------------------------+---------------------------------------------------------+-----------------+ | msvcrt.locking | "fd", "mode", "nbytes" | [1] | +--------------------------------+---------------------------------------------------------+-----------------+ | msvcrt.open_osfhandle | "handle", "flags" | [1] | +--------------------------------+---------------------------------------------------------+-----------------+ | object.__delattr__ | "obj", "name" | [1] | +--------------------------------+---------------------------------------------------------+-----------------+ | object.__getattr__ | "obj", "name" | [1] | +--------------------------------+---------------------------------------------------------+-----------------+ | object.__setattr__ | "obj", "name", "value" | [1] | +--------------------------------+---------------------------------------------------------+-----------------+ | open | "path", "mode", "flags" | [1][2][3] | +--------------------------------+---------------------------------------------------------+-----------------+ | os.add_dll_directory | "path" | [1] | +--------------------------------+---------------------------------------------------------+-----------------+ | os.chdir | "path" | [1][2] | +--------------------------------+---------------------------------------------------------+-----------------+ | os.chflags | "path", "flags" | [1][2] | +--------------------------------+---------------------------------------------------------+-----------------+ | os.chmod | "path", "mode", "dir_fd" | [1][2][3] | +--------------------------------+---------------------------------------------------------+-----------------+ | os.chown | "path", "uid", "gid", "dir_fd" | [1][2][3] | +--------------------------------+---------------------------------------------------------+-----------------+ | os.exec | "path", "args", "env" | [1] | +--------------------------------+---------------------------------------------------------+-----------------+ | os.fork | | [1] | +--------------------------------+---------------------------------------------------------+-----------------+ | os.forkpty | | [1] | +--------------------------------+---------------------------------------------------------+-----------------+ | os.fwalk | "top", "topdown", "onerror", "follow_symlinks", | [1] | | | "dir_fd" | | +--------------------------------+---------------------------------------------------------+-----------------+ | os.getxattr | "path", "attribute" | [1] | +--------------------------------+---------------------------------------------------------+-----------------+ | os.kill | "pid", "sig" | [1] | +--------------------------------+---------------------------------------------------------+-----------------+ | os.killpg | "pgid", "sig" | [1] | +--------------------------------+---------------------------------------------------------+-----------------+ | os.link | "src", "dst", "src_dir_fd", "dst_dir_fd" | [1] | +--------------------------------+---------------------------------------------------------+-----------------+ | os.listdir | "path" | [1] | +--------------------------------+---------------------------------------------------------+-----------------+ | os.listdrives | | [1] | +--------------------------------+---------------------------------------------------------+-----------------+ | os.listmounts | "volume" | [1] | +--------------------------------+---------------------------------------------------------+-----------------+ | os.listvolumes | | [1] | +--------------------------------+---------------------------------------------------------+-----------------+ | os.listxattr | "path" | [1] | +--------------------------------+---------------------------------------------------------+-----------------+ | os.lockf | "fd", "cmd", "len" | [1] | +--------------------------------+---------------------------------------------------------+-----------------+ | os.mkdir | "path", "mode", "dir_fd" | [1][2] | +--------------------------------+---------------------------------------------------------+-----------------+ | os.posix_spawn | "path", "argv", "env" | [1][2] | +--------------------------------+---------------------------------------------------------+-----------------+ | os.putenv | "key", "value" | [1] | +--------------------------------+---------------------------------------------------------+-----------------+ | os.remove | "path", "dir_fd" | [1][2][3] | +--------------------------------+---------------------------------------------------------+-----------------+ | os.removexattr | "path", "attribute" | [1] | +--------------------------------+---------------------------------------------------------+-----------------+ | os.rename | "src", "dst", "src_dir_fd", "dst_dir_fd" | [1][2][3] | +--------------------------------+---------------------------------------------------------+-----------------+ | os.rmdir | "path", "dir_fd" | [1] | +--------------------------------+---------------------------------------------------------+-----------------+ | os.scandir | "path" | [1] | +--------------------------------+---------------------------------------------------------+-----------------+ | os.setxattr | "path", "attribute", "value", "flags" | [1] | +--------------------------------+---------------------------------------------------------+-----------------+ | os.spawn | "mode", "path", "args", "env" | [1] | +--------------------------------+---------------------------------------------------------+-----------------+ | os.startfile | "path", "operation" | [1] | +--------------------------------+---------------------------------------------------------+-----------------+ | os.startfile/2 | "path", "operation", "arguments", "cwd", "show_cmd" | [1] | +--------------------------------+---------------------------------------------------------+-----------------+ | os.symlink | "src", "dst", "dir_fd" | [1] | +--------------------------------+---------------------------------------------------------+-----------------+ | os.system | "command" | [1] | +--------------------------------+---------------------------------------------------------+-----------------+ | os.truncate | "fd", "length" | [1][2] | +--------------------------------+---------------------------------------------------------+-----------------+ | os.unsetenv | "key" | [1] | +--------------------------------+---------------------------------------------------------+-----------------+ | os.utime | "path", "times", "ns", "dir_fd" | [1] | +--------------------------------+---------------------------------------------------------+-----------------+ | os.walk | "top", "topdown", "onerror", "followlinks" | [1] | +--------------------------------+---------------------------------------------------------+-----------------+ | pathlib.Path.glob | "self", "pattern" | [1] | +--------------------------------+---------------------------------------------------------+-----------------+ | pathlib.Path.rglob | "self", "pattern" | [1] | +--------------------------------+---------------------------------------------------------+-----------------+ | pdb.Pdb | | [1] | +--------------------------------+---------------------------------------------------------+-----------------+ | pickle.find_class | "module", "name" | [1] | +--------------------------------+---------------------------------------------------------+-----------------+ | poplib.connect | "self", "host", "port" | [1][2] | +--------------------------------+---------------------------------------------------------+-----------------+ | poplib.putline | "self", "line" | [1][2] | +--------------------------------+---------------------------------------------------------+-----------------+ | pty.spawn | "argv" | [1] | +--------------------------------+---------------------------------------------------------+-----------------+ | resource.prlimit | "pid", "resource", "limits" | [1] | +--------------------------------+---------------------------------------------------------+-----------------+ | resource.setrlimit | "resource", "limits" | [1] | +--------------------------------+---------------------------------------------------------+-----------------+ | setopencodehook | | [1] | +--------------------------------+---------------------------------------------------------+-----------------+ | shutil.chown | "path", "user", "group" | [1] | +--------------------------------+---------------------------------------------------------+-----------------+ | shutil.copyfile | "src", "dst" | [1][2][3] | +--------------------------------+---------------------------------------------------------+-----------------+ | shutil.copymode | "src", "dst" | [1][2] | +--------------------------------+---------------------------------------------------------+-----------------+ | shutil.copystat | "src", "dst" | [1][2] | +--------------------------------+---------------------------------------------------------+-----------------+ | shutil.copytree | "src", "dst" | [1] | +--------------------------------+---------------------------------------------------------+-----------------+ | shutil.make_archive | "base_name", "format", "root_dir", "base_dir" | [1] | +--------------------------------+---------------------------------------------------------+-----------------+ | shutil.move | "src", "dst" | [1] | +--------------------------------+---------------------------------------------------------+-----------------+ | shutil.rmtree | "path", "dir_fd" | [1] | +--------------------------------+---------------------------------------------------------+-----------------+ | shutil.unpack_archive | "filename", "extract_dir", "format" | [1] | +--------------------------------+---------------------------------------------------------+-----------------+ | signal.pthread_kill | "thread_id", "signalnum" | [1] | +--------------------------------+---------------------------------------------------------+-----------------+ | smtplib.connect | "self", "host", "port" | [1] | +--------------------------------+---------------------------------------------------------+-----------------+ | smtplib.send | "self", "data" | [1] | +--------------------------------+---------------------------------------------------------+-----------------+ | socket.__new__ | "self", "family", "type", "protocol" | [1] | +--------------------------------+---------------------------------------------------------+-----------------+ | socket.bind | "self", "address" | [1] | +--------------------------------+---------------------------------------------------------+-----------------+ | socket.connect | "self", "address" | [1][2] | +--------------------------------+---------------------------------------------------------+-----------------+ | socket.getaddrinfo | "host", "port", "family", "type", "protocol" | [1] | +--------------------------------+---------------------------------------------------------+-----------------+ | socket.gethostbyaddr | "ip_address" | [1] | +--------------------------------+---------------------------------------------------------+-----------------+ | socket.gethostbyname | "hostname" | [1][2] | +--------------------------------+---------------------------------------------------------+-----------------+ | socket.gethostname | | [1] | +--------------------------------+---------------------------------------------------------+-----------------+ | socket.getnameinfo | "sockaddr" | [1] | +--------------------------------+---------------------------------------------------------+-----------------+ | socket.getservbyname | "servicename", "protocolname" | [1] | +--------------------------------+---------------------------------------------------------+-----------------+ | socket.getservbyport | "port", "protocolname" | [1] | +--------------------------------+---------------------------------------------------------+-----------------+ | socket.sendmsg | "self", "address" | [1] | +--------------------------------+---------------------------------------------------------+-----------------+ | socket.sendto | "self", "address" | [1] | +--------------------------------+---------------------------------------------------------+-----------------+ | socket.sethostname | "name" | [1] | +--------------------------------+---------------------------------------------------------+-----------------+ | sqlite3.connect | "database" | [1] | +--------------------------------+---------------------------------------------------------+-----------------+ | sqlite3.connect/handle | "connection_handle" | [1] | +--------------------------------+---------------------------------------------------------+-----------------+ | sqlite3.enable_load_extension | "connection", "enabled" | [1] | +--------------------------------+---------------------------------------------------------+-----------------+ | sqlite3.load_extension | "connection", "path" | [1] | +--------------------------------+---------------------------------------------------------+-----------------+ | subprocess.Popen | "executable", "args", "cwd", "env" | [1] | +--------------------------------+---------------------------------------------------------+-----------------+ | sys._current_exceptions | | [1] | +--------------------------------+---------------------------------------------------------+-----------------+ | sys._current_frames | | [1] | +--------------------------------+---------------------------------------------------------+-----------------+ | sys._getframe | "frame" | [1] | +--------------------------------+---------------------------------------------------------+-----------------+ | sys._getframemodulename | "depth" | [1] | +--------------------------------+---------------------------------------------------------+-----------------+ | sys.addaudithook | | [1][2] | +--------------------------------+---------------------------------------------------------+-----------------+ | sys.excepthook | "hook", "type", "value", "traceback" | [1] | +--------------------------------+---------------------------------------------------------+-----------------+ | sys.monitoring.register_callb | "func" | [1] | | ack | | | +--------------------------------+---------------------------------------------------------+-----------------+ | sys.set_asyncgen_hooks_finali | | [1] | | zer | | | +--------------------------------+---------------------------------------------------------+-----------------+ | sys.set_asyncgen_hooks_firsti | | [1] | | ter | | | +--------------------------------+---------------------------------------------------------+-----------------+ | sys.setprofile | | [1] | +--------------------------------+---------------------------------------------------------+-----------------+ | sys.settrace | | [1] | +--------------------------------+---------------------------------------------------------+-----------------+ | sys.unraisablehook | "hook", "unraisable" | [1] | +--------------------------------+---------------------------------------------------------+-----------------+ | syslog.closelog | | [1] | +--------------------------------+---------------------------------------------------------+-----------------+ | syslog.openlog | "ident", "logoption", "facility" | [1] | +--------------------------------+---------------------------------------------------------+-----------------+ | syslog.setlogmask | "maskpri" | [1] | +--------------------------------+---------------------------------------------------------+-----------------+ | syslog.syslog | "priority", "message" | [1] | +--------------------------------+---------------------------------------------------------+-----------------+ | tempfile.mkdtemp | "fullpath" | [1][2] | +--------------------------------+---------------------------------------------------------+-----------------+ | tempfile.mkstemp | "fullpath" | [1][2][3] | +--------------------------------+---------------------------------------------------------+-----------------+ | time.sleep | "secs" | [1] | +--------------------------------+---------------------------------------------------------+-----------------+ | urllib.Request | "fullurl", "data", "headers", "method" | [1] | +--------------------------------+---------------------------------------------------------+-----------------+ | webbrowser.open | "url" | [1] | +--------------------------------+---------------------------------------------------------+-----------------+ | winreg.ConnectRegistry | "computer_name", "key" | [1] | +--------------------------------+---------------------------------------------------------+-----------------+ | winreg.CreateKey | "key", "sub_key", "access" | [1][2] | +--------------------------------+---------------------------------------------------------+-----------------+ | winreg.DeleteKey | "key", "sub_key", "access" | [1][2] | +--------------------------------+---------------------------------------------------------+-----------------+ | winreg.DeleteValue | "key", "value" | [1] | +--------------------------------+---------------------------------------------------------+-----------------+ | winreg.DisableReflectionKey | "key" | [1] | +--------------------------------+---------------------------------------------------------+-----------------+ | winreg.EnableReflectionKey | "key" | [1] | +--------------------------------+---------------------------------------------------------+-----------------+ | winreg.EnumKey | "key", "index" | [1] | +--------------------------------+---------------------------------------------------------+-----------------+ | winreg.EnumValue | "key", "index" | [1] | +--------------------------------+---------------------------------------------------------+-----------------+ | winreg.ExpandEnvironmentStrin | "str" | [1] | | gs | | | +--------------------------------+---------------------------------------------------------+-----------------+ | winreg.LoadKey | "key", "sub_key", "file_name" | [1] | +--------------------------------+---------------------------------------------------------+-----------------+ | winreg.OpenKey | "key", "sub_key", "access" | [1] | +--------------------------------+---------------------------------------------------------+-----------------+ | winreg.OpenKey/result | "key" | [1][2][3] | +--------------------------------+---------------------------------------------------------+-----------------+ | winreg.PyHKEY.Detach | "key" | [1] | +--------------------------------+---------------------------------------------------------+-----------------+ | winreg.QueryInfoKey | "key" | [1] | +--------------------------------+---------------------------------------------------------+-----------------+ | winreg.QueryReflectionKey | "key" | [1] | +--------------------------------+---------------------------------------------------------+-----------------+ | winreg.QueryValue | "key", "sub_key", "value_name" | [1][2] | +--------------------------------+---------------------------------------------------------+-----------------+ | winreg.SaveKey | "key", "file_name" | [1] | +--------------------------------+---------------------------------------------------------+-----------------+ | winreg.SetValue | "key", "sub_key", "type", "value" | [1][2] | +--------------------------------+---------------------------------------------------------+-----------------+ The following events are raised internally and do not correspond to any public API of CPython: +----------------------------+---------------------------------------------+ | Audit event | Arguments | |============================|=============================================| | _winapi.CreateFile | "file_name", "desired_access", | | | "share_mode", "creation_disposition", | | | "flags_and_attributes" | +----------------------------+---------------------------------------------+ | _winapi.CreateJunction | "src_path", "dst_path" | +----------------------------+---------------------------------------------+ | _winapi.CreateNamedPipe | "name", "open_mode", "pipe_mode" | +----------------------------+---------------------------------------------+ | _winapi.CreatePipe | | +----------------------------+---------------------------------------------+ | _winapi.CreateProcess | "application_name", "command_line", | | | "current_directory" | +----------------------------+---------------------------------------------+ | _winapi.OpenProcess | "process_id", "desired_access" | +----------------------------+---------------------------------------------+ | _winapi.TerminateProcess | "handle", "exit_code" | +----------------------------+---------------------------------------------+ | ctypes.PyObj_FromPtr | "obj" | +----------------------------+---------------------------------------------+ "base64" — Base16, Base32, Base64, Base85 Data Encodings ******************************************************** **Source code:** Lib/base64.py ====================================================================== This module provides functions for encoding binary data to printable ASCII characters and decoding such encodings back to binary data. This includes the encodings specified in **RFC 4648** (Base64, Base32 and Base16) and the non-standard Base85 encodings. There are two interfaces provided by this module. The modern interface supports encoding *bytes-like objects* to ASCII "bytes", and decoding *bytes-like objects* or strings containing ASCII to "bytes". Both base-64 alphabets defined in **RFC 4648** (normal, and URL- and filesystem-safe) are supported. The legacy interface does not support decoding from strings, but it does provide functions for encoding and decoding to and from *file objects*. It only supports the Base64 standard alphabet, and it adds newlines every 76 characters as per **RFC 2045**. Note that if you are looking for **RFC 2045** support you probably want to be looking at the "email" package instead. Changed in version 3.3: ASCII-only Unicode strings are now accepted by the decoding functions of the modern interface. Changed in version 3.4: Any *bytes-like objects* are now accepted by all encoding and decoding functions in this module. Ascii85/Base85 support added. RFC 4648 Encodings ================== The **RFC 4648** encodings are suitable for encoding binary data so that it can be safely sent by email, used as parts of URLs, or included as part of an HTTP POST request. base64.b64encode(s, altchars=None) Encode the *bytes-like object* *s* using Base64 and return the encoded "bytes". Optional *altchars* must be a *bytes-like object* of length 2 which specifies an alternative alphabet for the "+" and "/" characters. This allows an application to e.g. generate URL or filesystem safe Base64 strings. The default is "None", for which the standard Base64 alphabet is used. May assert or raise a "ValueError" if the length of *altchars* is not 2. Raises a "TypeError" if *altchars* is not a *bytes-like object*. base64.b64decode(s, altchars=None, validate=False) Decode the Base64 encoded *bytes-like object* or ASCII string *s* and return the decoded "bytes". Optional *altchars* must be a *bytes-like object* or ASCII string of length 2 which specifies the alternative alphabet used instead of the "+" and "/" characters. A "binascii.Error" exception is raised if *s* is incorrectly padded. If *validate* is "False" (the default), characters that are neither in the normal base-64 alphabet nor the alternative alphabet are discarded prior to the padding check. If *validate* is "True", these non-alphabet characters in the input result in a "binascii.Error". For more information about the strict base64 check, see "binascii.a2b_base64()" May assert or raise a "ValueError" if the length of *altchars* is not 2. base64.standard_b64encode(s) Encode *bytes-like object* *s* using the standard Base64 alphabet and return the encoded "bytes". base64.standard_b64decode(s) Decode *bytes-like object* or ASCII string *s* using the standard Base64 alphabet and return the decoded "bytes". base64.urlsafe_b64encode(s) Encode *bytes-like object* *s* using the URL- and filesystem-safe alphabet, which substitutes "-" instead of "+" and "_" instead of "/" in the standard Base64 alphabet, and return the encoded "bytes". The result can still contain "=". base64.urlsafe_b64decode(s) Decode *bytes-like object* or ASCII string *s* using the URL- and filesystem-safe alphabet, which substitutes "-" instead of "+" and "_" instead of "/" in the standard Base64 alphabet, and return the decoded "bytes". base64.b32encode(s) Encode the *bytes-like object* *s* using Base32 and return the encoded "bytes". base64.b32decode(s, casefold=False, map01=None) Decode the Base32 encoded *bytes-like object* or ASCII string *s* and return the decoded "bytes". Optional *casefold* is a flag specifying whether a lowercase alphabet is acceptable as input. For security purposes, the default is "False". **RFC 4648** allows for optional mapping of the digit 0 (zero) to the letter O (oh), and for optional mapping of the digit 1 (one) to either the letter I (eye) or letter L (el). The optional argument *map01* when not "None", specifies which letter the digit 1 should be mapped to (when *map01* is not "None", the digit 0 is always mapped to the letter O). For security purposes the default is "None", so that 0 and 1 are not allowed in the input. A "binascii.Error" is raised if *s* is incorrectly padded or if there are non-alphabet characters present in the input. base64.b32hexencode(s) Similar to "b32encode()" but uses the Extended Hex Alphabet, as defined in **RFC 4648**. Added in version 3.10. base64.b32hexdecode(s, casefold=False) Similar to "b32decode()" but uses the Extended Hex Alphabet, as defined in **RFC 4648**. This version does not allow the digit 0 (zero) to the letter O (oh) and digit 1 (one) to either the letter I (eye) or letter L (el) mappings, all these characters are included in the Extended Hex Alphabet and are not interchangeable. Added in version 3.10. base64.b16encode(s) Encode the *bytes-like object* *s* using Base16 and return the encoded "bytes". base64.b16decode(s, casefold=False) Decode the Base16 encoded *bytes-like object* or ASCII string *s* and return the decoded "bytes". Optional *casefold* is a flag specifying whether a lowercase alphabet is acceptable as input. For security purposes, the default is "False". A "binascii.Error" is raised if *s* is incorrectly padded or if there are non-alphabet characters present in the input. Base85 Encodings ================ Base85 encoding is not formally specified but rather a de facto standard, thus different systems perform the encoding differently. The "a85encode()" and "b85encode()" functions in this module are two implementations of the de facto standard. You should call the function with the Base85 implementation used by the software you intend to work with. The two functions present in this module differ in how they handle the following: * Whether to include enclosing "<~" and "~>" markers * Whether to include newline characters * The set of ASCII characters used for encoding * Handling of null bytes Refer to the documentation of the individual functions for more information. base64.a85encode(b, *, foldspaces=False, wrapcol=0, pad=False, adobe=False) Encode the *bytes-like object* *b* using Ascii85 and return the encoded "bytes". *foldspaces* is an optional flag that uses the special short sequence ‘y’ instead of 4 consecutive spaces (ASCII 0x20) as supported by ‘btoa’. This feature is not supported by the “standard” Ascii85 encoding. *wrapcol* controls whether the output should have newline ("b'\n'") characters added to it. If this is non-zero, each output line will be at most this many characters long, excluding the trailing newline. *pad* controls whether the input is padded to a multiple of 4 before encoding. Note that the "btoa" implementation always pads. *adobe* controls whether the encoded byte sequence is framed with "<~" and "~>", which is used by the Adobe implementation. Added in version 3.4. base64.a85decode(b, *, foldspaces=False, adobe=False, ignorechars=b' \t\n\r\x0b') Decode the Ascii85 encoded *bytes-like object* or ASCII string *b* and return the decoded "bytes". *foldspaces* is a flag that specifies whether the ‘y’ short sequence should be accepted as shorthand for 4 consecutive spaces (ASCII 0x20). This feature is not supported by the “standard” Ascii85 encoding. *adobe* controls whether the input sequence is in Adobe Ascii85 format (i.e. is framed with <~ and ~>). *ignorechars* should be a *bytes-like object* or ASCII string containing characters to ignore from the input. This should only contain whitespace characters, and by default contains all whitespace characters in ASCII. Added in version 3.4. base64.b85encode(b, pad=False) Encode the *bytes-like object* *b* using base85 (as used in e.g. git-style binary diffs) and return the encoded "bytes". If *pad* is true, the input is padded with "b'\0'" so its length is a multiple of 4 bytes before encoding. Added in version 3.4. base64.b85decode(b) Decode the base85-encoded *bytes-like object* or ASCII string *b* and return the decoded "bytes". Padding is implicitly removed, if necessary. Added in version 3.4. base64.z85encode(s) Encode the *bytes-like object* *s* using Z85 (as used in ZeroMQ) and return the encoded "bytes". See Z85 specification for more information. Added in version 3.13. base64.z85decode(s) Decode the Z85-encoded *bytes-like object* or ASCII string *s* and return the decoded "bytes". See Z85 specification for more information. Added in version 3.13. Legacy Interface ================ base64.decode(input, output) Decode the contents of the binary *input* file and write the resulting binary data to the *output* file. *input* and *output* must be *file objects*. *input* will be read until "input.readline()" returns an empty bytes object. base64.decodebytes(s) Decode the *bytes-like object* *s*, which must contain one or more lines of base64 encoded data, and return the decoded "bytes". Added in version 3.1. base64.encode(input, output) Encode the contents of the binary *input* file and write the resulting base64 encoded data to the *output* file. *input* and *output* must be *file objects*. *input* will be read until "input.read()" returns an empty bytes object. "encode()" inserts a newline character ("b'\n'") after every 76 bytes of the output, as well as ensuring that the output always ends with a newline, as per **RFC 2045** (MIME). base64.encodebytes(s) Encode the *bytes-like object* *s*, which can contain arbitrary binary data, and return "bytes" containing the base64-encoded data, with newlines ("b'\n'") inserted after every 76 bytes of output, and ensuring that there is a trailing newline, as per **RFC 2045** (MIME). Added in version 3.1. An example usage of the module: >>> import base64 >>> encoded = base64.b64encode(b'data to be encoded') >>> encoded b'ZGF0YSB0byBiZSBlbmNvZGVk' >>> data = base64.b64decode(encoded) >>> data b'data to be encoded' Security Considerations ======================= A new security considerations section was added to **RFC 4648** (section 12); it’s recommended to review the security section for any code deployed to production. See also: Module "binascii" Support module containing ASCII-to-binary and binary-to-ASCII conversions. **RFC 1521** - MIME (Multipurpose Internet Mail Extensions) Part One: Mechanisms for Specifying and Describing the Format of Internet Message Bodies Section 5.2, “Base64 Content-Transfer-Encoding,” provides the definition of the base64 encoding. "bdb" — Debugger framework ************************** **Source code:** Lib/bdb.py ====================================================================== The "bdb" module handles basic debugger functions, like setting breakpoints or managing execution via the debugger. The following exception is defined: exception bdb.BdbQuit Exception raised by the "Bdb" class for quitting the debugger. The "bdb" module also defines two classes: class bdb.Breakpoint(self, file, line, temporary=False, cond=None, funcname=None) This class implements temporary breakpoints, ignore counts, disabling and (re-)enabling, and conditionals. Breakpoints are indexed by number through a list called "bpbynumber" and by "(file, line)" pairs through "bplist". The former points to a single instance of class "Breakpoint". The latter points to a list of such instances since there may be more than one breakpoint per line. When creating a breakpoint, its associated "file name" should be in canonical form. If a "funcname" is defined, a breakpoint "hit" will be counted when the first line of that function is executed. A "conditional" breakpoint always counts a "hit". "Breakpoint" instances have the following methods: deleteMe() Delete the breakpoint from the list associated to a file/line. If it is the last breakpoint in that position, it also deletes the entry for the file/line. enable() Mark the breakpoint as enabled. disable() Mark the breakpoint as disabled. bpformat() Return a string with all the information about the breakpoint, nicely formatted: * Breakpoint number. * Temporary status (del or keep). * File/line position. * Break condition. * Number of times to ignore. * Number of times hit. Added in version 3.2. bpprint(out=None) Print the output of "bpformat()" to the file *out*, or if it is "None", to standard output. "Breakpoint" instances have the following attributes: file File name of the "Breakpoint". line Line number of the "Breakpoint" within "file". temporary "True" if a "Breakpoint" at (file, line) is temporary. cond Condition for evaluating a "Breakpoint" at (file, line). funcname Function name that defines whether a "Breakpoint" is hit upon entering the function. enabled "True" if "Breakpoint" is enabled. bpbynumber Numeric index for a single instance of a "Breakpoint". bplist Dictionary of "Breakpoint" instances indexed by ("file", "line") tuples. ignore Number of times to ignore a "Breakpoint". hits Count of the number of times a "Breakpoint" has been hit. class bdb.Bdb(skip=None) The "Bdb" class acts as a generic Python debugger base class. This class takes care of the details of the trace facility; a derived class should implement user interaction. The standard debugger class ("pdb.Pdb") is an example. The *skip* argument, if given, must be an iterable of glob-style module name patterns. The debugger will not step into frames that originate in a module that matches one of these patterns. Whether a frame is considered to originate in a certain module is determined by the "__name__" in the frame globals. Changed in version 3.1: Added the *skip* parameter. The following methods of "Bdb" normally don’t need to be overridden. canonic(filename) Return canonical form of *filename*. For real file names, the canonical form is an operating-system- dependent, "case-normalized" "absolute path". A *filename* with angle brackets, such as """" generated in interactive mode, is returned unchanged. reset() Set the "botframe", "stopframe", "returnframe" and "quitting" attributes with values ready to start debugging. trace_dispatch(frame, event, arg) This function is installed as the trace function of debugged frames. Its return value is the new trace function (in most cases, that is, itself). The default implementation decides how to dispatch a frame, depending on the type of event (passed as a string) that is about to be executed. *event* can be one of the following: * ""line"": A new line of code is going to be executed. * ""call"": A function is about to be called, or another code block entered. * ""return"": A function or other code block is about to return. * ""exception"": An exception has occurred. * ""c_call"": A C function is about to be called. * ""c_return"": A C function has returned. * ""c_exception"": A C function has raised an exception. For the Python events, specialized functions (see below) are called. For the C events, no action is taken. The *arg* parameter depends on the previous event. See the documentation for "sys.settrace()" for more information on the trace function. For more information on code and frame objects, refer to The standard type hierarchy. dispatch_line(frame) If the debugger should stop on the current line, invoke the "user_line()" method (which should be overridden in subclasses). Raise a "BdbQuit" exception if the "quitting" flag is set (which can be set from "user_line()"). Return a reference to the "trace_dispatch()" method for further tracing in that scope. dispatch_call(frame, arg) If the debugger should stop on this function call, invoke the "user_call()" method (which should be overridden in subclasses). Raise a "BdbQuit" exception if the "quitting" flag is set (which can be set from "user_call()"). Return a reference to the "trace_dispatch()" method for further tracing in that scope. dispatch_return(frame, arg) If the debugger should stop on this function return, invoke the "user_return()" method (which should be overridden in subclasses). Raise a "BdbQuit" exception if the "quitting" flag is set (which can be set from "user_return()"). Return a reference to the "trace_dispatch()" method for further tracing in that scope. dispatch_exception(frame, arg) If the debugger should stop at this exception, invokes the "user_exception()" method (which should be overridden in subclasses). Raise a "BdbQuit" exception if the "quitting" flag is set (which can be set from "user_exception()"). Return a reference to the "trace_dispatch()" method for further tracing in that scope. Normally derived classes don’t override the following methods, but they may if they want to redefine the definition of stopping and breakpoints. is_skipped_line(module_name) Return "True" if *module_name* matches any skip pattern. stop_here(frame) Return "True" if *frame* is below the starting frame in the stack. break_here(frame) Return "True" if there is an effective breakpoint for this line. Check whether a line or function breakpoint exists and is in effect. Delete temporary breakpoints based on information from "effective()". break_anywhere(frame) Return "True" if any breakpoint exists for *frame*’s filename. Derived classes should override these methods to gain control over debugger operation. user_call(frame, argument_list) Called from "dispatch_call()" if a break might stop inside the called function. *argument_list* is not used anymore and will always be "None". The argument is kept for backwards compatibility. user_line(frame) Called from "dispatch_line()" when either "stop_here()" or "break_here()" returns "True". user_return(frame, return_value) Called from "dispatch_return()" when "stop_here()" returns "True". user_exception(frame, exc_info) Called from "dispatch_exception()" when "stop_here()" returns "True". do_clear(arg) Handle how a breakpoint must be removed when it is a temporary one. This method must be implemented by derived classes. Derived classes and clients can call the following methods to affect the stepping state. set_step() Stop after one line of code. set_next(frame) Stop on the next line in or below the given frame. set_return(frame) Stop when returning from the given frame. set_until(frame, lineno=None) Stop when the line with the *lineno* greater than the current one is reached or when returning from current frame. set_trace([frame]) Start debugging from *frame*. If *frame* is not specified, debugging starts from caller’s frame. Changed in version 3.13: "set_trace()" will enter the debugger immediately, rather than on the next line of code to be executed. set_continue() Stop only at breakpoints or when finished. If there are no breakpoints, set the system trace function to "None". set_quit() Set the "quitting" attribute to "True". This raises "BdbQuit" in the next call to one of the "dispatch_*()" methods. Derived classes and clients can call the following methods to manipulate breakpoints. These methods return a string containing an error message if something went wrong, or "None" if all is well. set_break(filename, lineno, temporary=False, cond=None, funcname=None) Set a new breakpoint. If the *lineno* line doesn’t exist for the *filename* passed as argument, return an error message. The *filename* should be in canonical form, as described in the "canonic()" method. clear_break(filename, lineno) Delete the breakpoints in *filename* and *lineno*. If none were set, return an error message. clear_bpbynumber(arg) Delete the breakpoint which has the index *arg* in the "Breakpoint.bpbynumber". If *arg* is not numeric or out of range, return an error message. clear_all_file_breaks(filename) Delete all breakpoints in *filename*. If none were set, return an error message. clear_all_breaks() Delete all existing breakpoints. If none were set, return an error message. get_bpbynumber(arg) Return a breakpoint specified by the given number. If *arg* is a string, it will be converted to a number. If *arg* is a non- numeric string, if the given breakpoint never existed or has been deleted, a "ValueError" is raised. Added in version 3.2. get_break(filename, lineno) Return "True" if there is a breakpoint for *lineno* in *filename*. get_breaks(filename, lineno) Return all breakpoints for *lineno* in *filename*, or an empty list if none are set. get_file_breaks(filename) Return all breakpoints in *filename*, or an empty list if none are set. get_all_breaks() Return all breakpoints that are set. Derived classes and clients can call the following methods to get a data structure representing a stack trace. get_stack(f, t) Return a list of (frame, lineno) tuples in a stack trace, and a size. The most recently called frame is last in the list. The size is the number of frames below the frame where the debugger was invoked. format_stack_entry(frame_lineno, lprefix=': ') Return a string with information about a stack entry, which is a "(frame, lineno)" tuple. The return string contains: * The canonical filename which contains the frame. * The function name or """". * The input arguments. * The return value. * The line of code (if it exists). The following two methods can be called by clients to use a debugger to debug a *statement*, given as a string. run(cmd, globals=None, locals=None) Debug a statement executed via the "exec()" function. *globals* defaults to "__main__.__dict__", *locals* defaults to *globals*. runeval(expr, globals=None, locals=None) Debug an expression executed via the "eval()" function. *globals* and *locals* have the same meaning as in "run()". runctx(cmd, globals, locals) For backwards compatibility. Calls the "run()" method. runcall(func, /, *args, **kwds) Debug a single function call, and return its result. Finally, the module defines the following functions: bdb.checkfuncname(b, frame) Return "True" if we should break here, depending on the way the "Breakpoint" *b* was set. If it was set via line number, it checks if "b.line" is the same as the one in *frame*. If the breakpoint was set via "function name", we have to check we are in the right *frame* (the right function) and if we are on its first executable line. bdb.effective(file, line, frame) Return "(active breakpoint, delete temporary flag)" or "(None, None)" as the breakpoint to act upon. The *active breakpoint* is the first entry in "bplist" for the ("file", "line") (which must exist) that is "enabled", for which "checkfuncname()" is true, and that has neither a false "condition" nor positive "ignore" count. The *flag*, meaning that a temporary breakpoint should be deleted, is "False" only when the "cond" cannot be evaluated (in which case, "ignore" count is ignored). If no such entry exists, then "(None, None)" is returned. bdb.set_trace() Start debugging with a "Bdb" instance from caller’s frame. Binary Data Services ******************** The modules described in this chapter provide some basic services operations for manipulation of binary data. Other operations on binary data, specifically in relation to file formats and network protocols, are described in the relevant sections. Some libraries described under Text Processing Services also work with either ASCII-compatible binary formats (for example, "re") or all binary data (for example, "difflib"). In addition, see the documentation for Python’s built-in binary data types in Binary Sequence Types — bytes, bytearray, memoryview. * "struct" — Interpret bytes as packed binary data * Functions and Exceptions * Format Strings * Byte Order, Size, and Alignment * Format Characters * Examples * Applications * Native Formats * Standard Formats * Classes * "codecs" — Codec registry and base classes * Codec Base Classes * Error Handlers * Stateless Encoding and Decoding * Incremental Encoding and Decoding * IncrementalEncoder Objects * IncrementalDecoder Objects * Stream Encoding and Decoding * StreamWriter Objects * StreamReader Objects * StreamReaderWriter Objects * StreamRecoder Objects * Encodings and Unicode * Standard Encodings * Python Specific Encodings * Text Encodings * Binary Transforms * Standalone Codec Functions * Text Transforms * "encodings" — Encodings package * "encodings.idna" — Internationalized Domain Names in Applications * "encodings.mbcs" — Windows ANSI codepage * "encodings.utf_8_sig" — UTF-8 codec with BOM signature "binascii" — Convert between binary and ASCII ********************************************* ====================================================================== The "binascii" module contains a number of methods to convert between binary and various ASCII-encoded binary representations. Normally, you will not use these functions directly but use wrapper modules like "base64" instead. The "binascii" module contains low-level functions written in C for greater speed that are used by the higher-level modules. Note: "a2b_*" functions accept Unicode strings containing only ASCII characters. Other functions only accept *bytes-like objects* (such as "bytes", "bytearray" and other objects that support the buffer protocol). Changed in version 3.3: ASCII-only unicode strings are now accepted by the "a2b_*" functions. The "binascii" module defines the following functions: binascii.a2b_uu(string) Convert a single line of uuencoded data back to binary and return the binary data. Lines normally contain 45 (binary) bytes, except for the last line. Line data may be followed by whitespace. binascii.b2a_uu(data, *, backtick=False) Convert binary data to a line of ASCII characters, the return value is the converted line, including a newline char. The length of *data* should be at most 45. If *backtick* is true, zeros are represented by "'`'" instead of spaces. Changed in version 3.7: Added the *backtick* parameter. binascii.a2b_base64(string, /, *, strict_mode=False) Convert a block of base64 data back to binary and return the binary data. More than one line may be passed at a time. If *strict_mode* is true, only valid base64 data will be converted. Invalid base64 data will raise "binascii.Error". Valid base64: * Conforms to **RFC 3548**. * Contains only characters from the base64 alphabet. * Contains no excess data after padding (including excess padding, newlines, etc.). * Does not start with a padding. Changed in version 3.11: Added the *strict_mode* parameter. binascii.b2a_base64(data, *, newline=True) Convert binary data to a line of ASCII characters in base64 coding. The return value is the converted line, including a newline char if *newline* is true. The output of this function conforms to **RFC 3548**. Changed in version 3.6: Added the *newline* parameter. binascii.a2b_qp(data, header=False) Convert a block of quoted-printable data back to binary and return the binary data. More than one line may be passed at a time. If the optional argument *header* is present and true, underscores will be decoded as spaces. binascii.b2a_qp(data, quotetabs=False, istext=True, header=False) Convert binary data to a line(s) of ASCII characters in quoted- printable encoding. The return value is the converted line(s). If the optional argument *quotetabs* is present and true, all tabs and spaces will be encoded. If the optional argument *istext* is present and true, newlines are not encoded but trailing whitespace will be encoded. If the optional argument *header* is present and true, spaces will be encoded as underscores per **RFC 1522**. If the optional argument *header* is present and false, newline characters will be encoded as well; otherwise linefeed conversion might corrupt the binary data stream. binascii.crc_hqx(data, value) Compute a 16-bit CRC value of *data*, starting with *value* as the initial CRC, and return the result. This uses the CRC-CCITT polynomial *x*^16 + *x*^12 + *x*^5 + 1, often represented as 0x1021. This CRC is used in the binhex4 format. binascii.crc32(data[, value]) Compute CRC-32, the unsigned 32-bit checksum of *data*, starting with an initial CRC of *value*. The default initial CRC is zero. The algorithm is consistent with the ZIP file checksum. Since the algorithm is designed for use as a checksum algorithm, it is not suitable for use as a general hash algorithm. Use as follows: print(binascii.crc32(b"hello world")) # Or, in two pieces: crc = binascii.crc32(b"hello") crc = binascii.crc32(b" world", crc) print('crc32 = {:#010x}'.format(crc)) Changed in version 3.0: The result is always unsigned. binascii.b2a_hex(data[, sep[, bytes_per_sep=1]]) binascii.hexlify(data[, sep[, bytes_per_sep=1]]) Return the hexadecimal representation of the binary *data*. Every byte of *data* is converted into the corresponding 2-digit hex representation. The returned bytes object is therefore twice as long as the length of *data*. Similar functionality (but returning a text string) is also conveniently accessible using the "bytes.hex()" method. If *sep* is specified, it must be a single character str or bytes object. It will be inserted in the output after every *bytes_per_sep* input bytes. Separator placement is counted from the right end of the output by default, if you wish to count from the left, supply a negative *bytes_per_sep* value. >>> import binascii >>> binascii.b2a_hex(b'\xb9\x01\xef') b'b901ef' >>> binascii.hexlify(b'\xb9\x01\xef', '-') b'b9-01-ef' >>> binascii.b2a_hex(b'\xb9\x01\xef', b'_', 2) b'b9_01ef' >>> binascii.b2a_hex(b'\xb9\x01\xef', b' ', -2) b'b901 ef' Changed in version 3.8: The *sep* and *bytes_per_sep* parameters were added. binascii.a2b_hex(hexstr) binascii.unhexlify(hexstr) Return the binary data represented by the hexadecimal string *hexstr*. This function is the inverse of "b2a_hex()". *hexstr* must contain an even number of hexadecimal digits (which can be upper or lower case), otherwise an "Error" exception is raised. Similar functionality (accepting only text string arguments, but more liberal towards whitespace) is also accessible using the "bytes.fromhex()" class method. exception binascii.Error Exception raised on errors. These are usually programming errors. exception binascii.Incomplete Exception raised on incomplete data. These are usually not programming errors, but may be handled by reading a little more data and trying again. See also: Module "base64" Support for RFC compliant base64-style encoding in base 16, 32, 64, and 85. Module "quopri" Support for quoted-printable encoding used in MIME email messages. "bisect" — Array bisection algorithm ************************************ **Source code:** Lib/bisect.py ====================================================================== This module provides support for maintaining a list in sorted order without having to sort the list after each insertion. For long lists of items with expensive comparison operations, this can be an improvement over linear searches or frequent resorting. The module is called "bisect" because it uses a basic bisection algorithm to do its work. Unlike other bisection tools that search for a specific value, the functions in this module are designed to locate an insertion point. Accordingly, the functions never call an "__eq__()" method to determine whether a value has been found. Instead, the functions only call the "__lt__()" method and will return an insertion point between values in an array. Note: The functions in this module are not thread-safe. If multiple threads concurrently use "bisect" functions on the same sequence, this may result in undefined behaviour. Likewise, if the provided sequence is mutated by a different thread while a "bisect" function is operating on it, the result is undefined. For example, using "insort_left()" on the same list from multiple threads may result in the list becoming unsorted. The following functions are provided: bisect.bisect_left(a, x, lo=0, hi=len(a), *, key=None) Locate the insertion point for *x* in *a* to maintain sorted order. The parameters *lo* and *hi* may be used to specify a subset of the list which should be considered; by default the entire list is used. If *x* is already present in *a*, the insertion point will be before (to the left of) any existing entries. The return value is suitable for use as the first parameter to "list.insert()" assuming that *a* is already sorted. The returned insertion point *ip* partitions the array *a* into two slices such that "all(elem < x for elem in a[lo : ip])" is true for the left slice and "all(elem >= x for elem in a[ip : hi])" is true for the right slice. *key* specifies a *key function* of one argument that is used to extract a comparison key from each element in the array. To support searching complex records, the key function is not applied to the *x* value. If *key* is "None", the elements are compared directly and no key function is called. Changed in version 3.10: Added the *key* parameter. bisect.bisect_right(a, x, lo=0, hi=len(a), *, key=None) bisect.bisect(a, x, lo=0, hi=len(a), *, key=None) Similar to "bisect_left()", but returns an insertion point which comes after (to the right of) any existing entries of *x* in *a*. The returned insertion point *ip* partitions the array *a* into two slices such that "all(elem <= x for elem in a[lo : ip])" is true for the left slice and "all(elem > x for elem in a[ip : hi])" is true for the right slice. Changed in version 3.10: Added the *key* parameter. bisect.insort_left(a, x, lo=0, hi=len(a), *, key=None) Insert *x* in *a* in sorted order. This function first runs "bisect_left()" to locate an insertion point. Next, it runs the "insert()" method on *a* to insert *x* at the appropriate position to maintain sort order. To support inserting records in a table, the *key* function (if any) is applied to *x* for the search step but not for the insertion step. Keep in mind that the *O*(log *n*) search is dominated by the slow *O*(*n*) insertion step. Changed in version 3.10: Added the *key* parameter. bisect.insort_right(a, x, lo=0, hi=len(a), *, key=None) bisect.insort(a, x, lo=0, hi=len(a), *, key=None) Similar to "insort_left()", but inserting *x* in *a* after any existing entries of *x*. This function first runs "bisect_right()" to locate an insertion point. Next, it runs the "insert()" method on *a* to insert *x* at the appropriate position to maintain sort order. To support inserting records in a table, the *key* function (if any) is applied to *x* for the search step but not for the insertion step. Keep in mind that the *O*(log *n*) search is dominated by the slow *O*(*n*) insertion step. Changed in version 3.10: Added the *key* parameter. Performance Notes ================= When writing time sensitive code using *bisect()* and *insort()*, keep these thoughts in mind: * Bisection is effective for searching ranges of values. For locating specific values, dictionaries are more performant. * The *insort()* functions are *O*(*n*) because the logarithmic search step is dominated by the linear time insertion step. * The search functions are stateless and discard key function results after they are used. Consequently, if the search functions are used in a loop, the key function may be called again and again on the same array elements. If the key function isn’t fast, consider wrapping it with "functools.cache()" to avoid duplicate computations. Alternatively, consider searching an array of precomputed keys to locate the insertion point (as shown in the examples section below). See also: * Sorted Collections is a high performance module that uses *bisect* to managed sorted collections of data. * The SortedCollection recipe uses bisect to build a full-featured collection class with straight-forward search methods and support for a key-function. The keys are precomputed to save unnecessary calls to the key function during searches. Searching Sorted Lists ====================== The above bisect functions are useful for finding insertion points but can be tricky or awkward to use for common searching tasks. The following five functions show how to transform them into the standard lookups for sorted lists: def index(a, x): 'Locate the leftmost value exactly equal to x' i = bisect_left(a, x) if i != len(a) and a[i] == x: return i raise ValueError def find_lt(a, x): 'Find rightmost value less than x' i = bisect_left(a, x) if i: return a[i-1] raise ValueError def find_le(a, x): 'Find rightmost value less than or equal to x' i = bisect_right(a, x) if i: return a[i-1] raise ValueError def find_gt(a, x): 'Find leftmost value greater than x' i = bisect_right(a, x) if i != len(a): return a[i] raise ValueError def find_ge(a, x): 'Find leftmost item greater than or equal to x' i = bisect_left(a, x) if i != len(a): return a[i] raise ValueError Examples ======== The "bisect()" function can be useful for numeric table lookups. This example uses "bisect()" to look up a letter grade for an exam score (say) based on a set of ordered numeric breakpoints: 90 and up is an ‘A’, 80 to 89 is a ‘B’, and so on: >>> def grade(score, breakpoints=[60, 70, 80, 90], grades='FDCBA'): ... i = bisect(breakpoints, score) ... return grades[i] ... >>> [grade(score) for score in [33, 99, 77, 70, 89, 90, 100]] ['F', 'A', 'C', 'C', 'B', 'A', 'A'] The "bisect()" and "insort()" functions also work with lists of tuples. The *key* argument can serve to extract the field used for ordering records in a table: >>> from collections import namedtuple >>> from operator import attrgetter >>> from bisect import bisect, insort >>> from pprint import pprint >>> Movie = namedtuple('Movie', ('name', 'released', 'director')) >>> movies = [ ... Movie('Jaws', 1975, 'Spielberg'), ... Movie('Titanic', 1997, 'Cameron'), ... Movie('The Birds', 1963, 'Hitchcock'), ... Movie('Aliens', 1986, 'Cameron') ... ] >>> # Find the first movie released after 1960 >>> by_year = attrgetter('released') >>> movies.sort(key=by_year) >>> movies[bisect(movies, 1960, key=by_year)] Movie(name='The Birds', released=1963, director='Hitchcock') >>> # Insert a movie while maintaining sort order >>> romance = Movie('Love Story', 1970, 'Hiller') >>> insort(movies, romance, key=by_year) >>> pprint(movies) [Movie(name='The Birds', released=1963, director='Hitchcock'), Movie(name='Love Story', released=1970, director='Hiller'), Movie(name='Jaws', released=1975, director='Spielberg'), Movie(name='Aliens', released=1986, director='Cameron'), Movie(name='Titanic', released=1997, director='Cameron')] If the key function is expensive, it is possible to avoid repeated function calls by searching a list of precomputed keys to find the index of a record: >>> data = [('red', 5), ('blue', 1), ('yellow', 8), ('black', 0)] >>> data.sort(key=lambda r: r[1]) # Or use operator.itemgetter(1). >>> keys = [r[1] for r in data] # Precompute a list of keys. >>> data[bisect_left(keys, 0)] ('black', 0) >>> data[bisect_left(keys, 1)] ('blue', 1) >>> data[bisect_left(keys, 5)] ('red', 5) >>> data[bisect_left(keys, 8)] ('yellow', 8) "builtins" — Built-in objects ***************************** ====================================================================== This module provides direct access to all ‘built-in’ identifiers of Python; for example, "builtins.open" is the full name for the built-in function "open()". This module is not normally accessed explicitly by most applications, but can be useful in modules that provide objects with the same name as a built-in value, but in which the built-in of that name is also needed. For example, in a module that wants to implement an "open()" function that wraps the built-in "open()", this module can be used directly: import builtins def open(path): f = builtins.open(path, 'r') return UpperCaser(f) class UpperCaser: '''Wrapper around a file that converts output to uppercase.''' def __init__(self, f): self._f = f def read(self, count=-1): return self._f.read(count).upper() # ... As an implementation detail, most modules have the name "__builtins__" made available as part of their globals. The value of "__builtins__" is normally either this module or the value of this module’s "__dict__" attribute. Since this is an implementation detail, it may not be used by alternate implementations of Python. See also: * Built-in Constants * Built-in Exceptions * Built-in Functions * Built-in Types "bz2" — Support for **bzip2** compression ***************************************** **Source code:** Lib/bz2.py ====================================================================== This module provides a comprehensive interface for compressing and decompressing data using the bzip2 compression algorithm. The "bz2" module contains: * The "open()" function and "BZ2File" class for reading and writing compressed files. * The "BZ2Compressor" and "BZ2Decompressor" classes for incremental (de)compression. * The "compress()" and "decompress()" functions for one-shot (de)compression. (De)compression of files ======================== bz2.open(filename, mode='rb', compresslevel=9, encoding=None, errors=None, newline=None) Open a bzip2-compressed file in binary or text mode, returning a *file object*. As with the constructor for "BZ2File", the *filename* argument can be an actual filename (a "str" or "bytes" object), or an existing file object to read from or write to. The *mode* argument can be any of "'r'", "'rb'", "'w'", "'wb'", "'x'", "'xb'", "'a'" or "'ab'" for binary mode, or "'rt'", "'wt'", "'xt'", or "'at'" for text mode. The default is "'rb'". The *compresslevel* argument is an integer from 1 to 9, as for the "BZ2File" constructor. For binary mode, this function is equivalent to the "BZ2File" constructor: "BZ2File(filename, mode, compresslevel=compresslevel)". In this case, the *encoding*, *errors* and *newline* arguments must not be provided. For text mode, a "BZ2File" object is created, and wrapped in an "io.TextIOWrapper" instance with the specified encoding, error handling behavior, and line ending(s). Added in version 3.3. Changed in version 3.4: The "'x'" (exclusive creation) mode was added. Changed in version 3.6: Accepts a *path-like object*. class bz2.BZ2File(filename, mode='r', *, compresslevel=9) Open a bzip2-compressed file in binary mode. If *filename* is a "str" or "bytes" object, open the named file directly. Otherwise, *filename* should be a *file object*, which will be used to read or write the compressed data. The *mode* argument can be either "'r'" for reading (default), "'w'" for overwriting, "'x'" for exclusive creation, or "'a'" for appending. These can equivalently be given as "'rb'", "'wb'", "'xb'" and "'ab'" respectively. If *filename* is a file object (rather than an actual file name), a mode of "'w'" does not truncate the file, and is instead equivalent to "'a'". If *mode* is "'w'" or "'a'", *compresslevel* can be an integer between "1" and "9" specifying the level of compression: "1" produces the least compression, and "9" (default) produces the most compression. If *mode* is "'r'", the input file may be the concatenation of multiple compressed streams. "BZ2File" provides all of the members specified by the "io.BufferedIOBase", except for "detach()" and "truncate()". Iteration and the "with" statement are supported. "BZ2File" also provides the following methods and attributes: peek([n]) Return buffered data without advancing the file position. At least one byte of data will be returned (unless at EOF). The exact number of bytes returned is unspecified. Note: While calling "peek()" does not change the file position of the "BZ2File", it may change the position of the underlying file object (e.g. if the "BZ2File" was constructed by passing a file object for *filename*). Added in version 3.3. fileno() Return the file descriptor for the underlying file. Added in version 3.3. readable() Return whether the file was opened for reading. Added in version 3.3. seekable() Return whether the file supports seeking. Added in version 3.3. writable() Return whether the file was opened for writing. Added in version 3.3. read1(size=-1) Read up to *size* uncompressed bytes, while trying to avoid making multiple reads from the underlying stream. Reads up to a buffer’s worth of data if size is negative. Returns "b''" if the file is at EOF. Added in version 3.3. readinto(b) Read bytes into *b*. Returns the number of bytes read (0 for EOF). Added in version 3.3. mode "'rb'" for reading and "'wb'" for writing. Added in version 3.13. name The bzip2 file name. Equivalent to the "name" attribute of the underlying *file object*. Added in version 3.13. Changed in version 3.1: Support for the "with" statement was added. Changed in version 3.3: Support was added for *filename* being a *file object* instead of an actual filename.The "'a'" (append) mode was added, along with support for reading multi-stream files. Changed in version 3.4: The "'x'" (exclusive creation) mode was added. Changed in version 3.5: The "read()" method now accepts an argument of "None". Changed in version 3.6: Accepts a *path-like object*. Changed in version 3.9: The *buffering* parameter has been removed. It was ignored and deprecated since Python 3.0. Pass an open file object to control how the file is opened.The *compresslevel* parameter became keyword-only. Changed in version 3.10: This class is thread unsafe in the face of multiple simultaneous readers or writers, just like its equivalent classes in "gzip" and "lzma" have always been. Incremental (de)compression =========================== class bz2.BZ2Compressor(compresslevel=9) Create a new compressor object. This object may be used to compress data incrementally. For one-shot compression, use the "compress()" function instead. *compresslevel*, if given, must be an integer between "1" and "9". The default is "9". compress(data) Provide data to the compressor object. Returns a chunk of compressed data if possible, or an empty byte string otherwise. When you have finished providing data to the compressor, call the "flush()" method to finish the compression process. flush() Finish the compression process. Returns the compressed data left in internal buffers. The compressor object may not be used after this method has been called. class bz2.BZ2Decompressor Create a new decompressor object. This object may be used to decompress data incrementally. For one-shot compression, use the "decompress()" function instead. Note: This class does not transparently handle inputs containing multiple compressed streams, unlike "decompress()" and "BZ2File". If you need to decompress a multi-stream input with "BZ2Decompressor", you must use a new decompressor for each stream. decompress(data, max_length=-1) Decompress *data* (a *bytes-like object*), returning uncompressed data as bytes. Some of *data* may be buffered internally, for use in later calls to "decompress()". The returned data should be concatenated with the output of any previous calls to "decompress()". If *max_length* is nonnegative, returns at most *max_length* bytes of decompressed data. If this limit is reached and further output can be produced, the "needs_input" attribute will be set to "False". In this case, the next call to "decompress()" may provide *data* as "b''" to obtain more of the output. If all of the input data was decompressed and returned (either because this was less than *max_length* bytes, or because *max_length* was negative), the "needs_input" attribute will be set to "True". Attempting to decompress data after the end of stream is reached raises an "EOFError". Any data found after the end of the stream is ignored and saved in the "unused_data" attribute. Changed in version 3.5: Added the *max_length* parameter. eof "True" if the end-of-stream marker has been reached. Added in version 3.3. unused_data Data found after the end of the compressed stream. If this attribute is accessed before the end of the stream has been reached, its value will be "b''". needs_input "False" if the "decompress()" method can provide more decompressed data before requiring new uncompressed input. Added in version 3.5. One-shot (de)compression ======================== bz2.compress(data, compresslevel=9) Compress *data*, a *bytes-like object*. *compresslevel*, if given, must be an integer between "1" and "9". The default is "9". For incremental compression, use a "BZ2Compressor" instead. bz2.decompress(data) Decompress *data*, a *bytes-like object*. If *data* is the concatenation of multiple compressed streams, decompress all of the streams. For incremental decompression, use a "BZ2Decompressor" instead. Changed in version 3.3: Support for multi-stream inputs was added. Examples of usage ================= Below are some examples of typical usage of the "bz2" module. Using "compress()" and "decompress()" to demonstrate round-trip compression: >>> import bz2 >>> data = b"""\ ... Donec rhoncus quis sapien sit amet molestie. Fusce scelerisque vel augue ... nec ullamcorper. Nam rutrum pretium placerat. Aliquam vel tristique lorem, ... sit amet cursus ante. In interdum laoreet mi, sit amet ultrices purus ... pulvinar a. Nam gravida euismod magna, non varius justo tincidunt feugiat. ... Aliquam pharetra lacus non risus vehicula rutrum. Maecenas aliquam leo ... felis. Pellentesque semper nunc sit amet nibh ullamcorper, ac elementum ... dolor luctus. Curabitur lacinia mi ornare consectetur vestibulum.""" >>> c = bz2.compress(data) >>> len(data) / len(c) # Data compression ratio 1.513595166163142 >>> d = bz2.decompress(c) >>> data == d # Check equality to original object after round-trip True Using "BZ2Compressor" for incremental compression: >>> import bz2 >>> def gen_data(chunks=10, chunksize=1000): ... """Yield incremental blocks of chunksize bytes.""" ... for _ in range(chunks): ... yield b"z" * chunksize ... >>> comp = bz2.BZ2Compressor() >>> out = b"" >>> for chunk in gen_data(): ... # Provide data to the compressor object ... out = out + comp.compress(chunk) ... >>> # Finish the compression process. Call this once you have >>> # finished providing data to the compressor. >>> out = out + comp.flush() The example above uses a very “nonrandom” stream of data (a stream of "b"z"" chunks). Random data tends to compress poorly, while ordered, repetitive data usually yields a high compression ratio. Writing and reading a bzip2-compressed file in binary mode: >>> import bz2 >>> data = b"""\ ... Donec rhoncus quis sapien sit amet molestie. Fusce scelerisque vel augue ... nec ullamcorper. Nam rutrum pretium placerat. Aliquam vel tristique lorem, ... sit amet cursus ante. In interdum laoreet mi, sit amet ultrices purus ... pulvinar a. Nam gravida euismod magna, non varius justo tincidunt feugiat. ... Aliquam pharetra lacus non risus vehicula rutrum. Maecenas aliquam leo ... felis. Pellentesque semper nunc sit amet nibh ullamcorper, ac elementum ... dolor luctus. Curabitur lacinia mi ornare consectetur vestibulum.""" >>> with bz2.open("myfile.bz2", "wb") as f: ... # Write compressed data to file ... unused = f.write(data) ... >>> with bz2.open("myfile.bz2", "rb") as f: ... # Decompress data from file ... content = f.read() ... >>> content == data # Check equality to original object after round-trip True "calendar" — General calendar-related functions *********************************************** **Source code:** Lib/calendar.py ====================================================================== This module allows you to output calendars like the Unix **cal** program, and provides additional useful functions related to the calendar. By default, these calendars have Monday as the first day of the week, and Sunday as the last (the European convention). Use "setfirstweekday()" to set the first day of the week to Sunday (6) or to any other weekday. Parameters that specify dates are given as integers. For related functionality, see also the "datetime" and "time" modules. The functions and classes defined in this module use an idealized calendar, the current Gregorian calendar extended indefinitely in both directions. This matches the definition of the “proleptic Gregorian” calendar in Dershowitz and Reingold’s book “Calendrical Calculations”, where it’s the base calendar for all computations. Zero and negative years are interpreted as prescribed by the ISO 8601 standard. Year 0 is 1 BC, year -1 is 2 BC, and so on. class calendar.Calendar(firstweekday=0) Creates a "Calendar" object. *firstweekday* is an integer specifying the first day of the week. "MONDAY" is "0" (the default), "SUNDAY" is "6". A "Calendar" object provides several methods that can be used for preparing the calendar data for formatting. This class doesn’t do any formatting itself. This is the job of subclasses. "Calendar" instances have the following methods and attributes: firstweekday The first weekday as an integer (0–6). This property can also be set and read using "setfirstweekday()" and "getfirstweekday()" respectively. getfirstweekday() Return an "int" for the current first weekday (0–6). Identical to reading the "firstweekday" property. setfirstweekday(firstweekday) Set the first weekday to *firstweekday*, passed as an "int" (0–6) Identical to setting the "firstweekday" property. iterweekdays() Return an iterator for the week day numbers that will be used for one week. The first value from the iterator will be the same as the value of the "firstweekday" property. itermonthdates(year, month) Return an iterator for the month *month* (1–12) in the year *year*. This iterator will return all days (as "datetime.date" objects) for the month and all days before the start of the month or after the end of the month that are required to get a complete week. itermonthdays(year, month) Return an iterator for the month *month* in the year *year* similar to "itermonthdates()", but not restricted by the "datetime.date" range. Days returned will simply be day of the month numbers. For the days outside of the specified month, the day number is "0". itermonthdays2(year, month) Return an iterator for the month *month* in the year *year* similar to "itermonthdates()", but not restricted by the "datetime.date" range. Days returned will be tuples consisting of a day of the month number and a week day number. itermonthdays3(year, month) Return an iterator for the month *month* in the year *year* similar to "itermonthdates()", but not restricted by the "datetime.date" range. Days returned will be tuples consisting of a year, a month and a day of the month numbers. Added in version 3.7. itermonthdays4(year, month) Return an iterator for the month *month* in the year *year* similar to "itermonthdates()", but not restricted by the "datetime.date" range. Days returned will be tuples consisting of a year, a month, a day of the month, and a day of the week numbers. Added in version 3.7. monthdatescalendar(year, month) Return a list of the weeks in the month *month* of the *year* as full weeks. Weeks are lists of seven "datetime.date" objects. monthdays2calendar(year, month) Return a list of the weeks in the month *month* of the *year* as full weeks. Weeks are lists of seven tuples of day numbers and weekday numbers. monthdayscalendar(year, month) Return a list of the weeks in the month *month* of the *year* as full weeks. Weeks are lists of seven day numbers. yeardatescalendar(year, width=3) Return the data for the specified year ready for formatting. The return value is a list of month rows. Each month row contains up to *width* months (defaulting to 3). Each month contains between 4 and 6 weeks and each week contains 1–7 days. Days are "datetime.date" objects. yeardays2calendar(year, width=3) Return the data for the specified year ready for formatting (similar to "yeardatescalendar()"). Entries in the week lists are tuples of day numbers and weekday numbers. Day numbers outside this month are zero. yeardayscalendar(year, width=3) Return the data for the specified year ready for formatting (similar to "yeardatescalendar()"). Entries in the week lists are day numbers. Day numbers outside this month are zero. class calendar.TextCalendar(firstweekday=0) This class can be used to generate plain text calendars. "TextCalendar" instances have the following methods: formatday(theday, weekday, width) Return a string representing a single day formatted with the given *width*. If *theday* is "0", return a string of spaces of the specified width, representing an empty day. The *weekday* parameter is unused. formatweek(theweek, w=0) Return a single week in a string with no newline. If *w* is provided, it specifies the width of the date columns, which are centered. Depends on the first weekday as specified in the constructor or set by the "setfirstweekday()" method. formatweekday(weekday, width) Return a string representing the name of a single weekday formatted to the specified *width*. The *weekday* parameter is an integer representing the day of the week, where "0" is Monday and "6" is Sunday. formatweekheader(width) Return a string containing the header row of weekday names, formatted with the given *width* for each column. The names depend on the locale settings and are padded to the specified width. formatmonth(theyear, themonth, w=0, l=0) Return a month’s calendar in a multi-line string. If *w* is provided, it specifies the width of the date columns, which are centered. If *l* is given, it specifies the number of lines that each week will use. Depends on the first weekday as specified in the constructor or set by the "setfirstweekday()" method. formatmonthname(theyear, themonth, width=0, withyear=True) Return a string representing the month’s name centered within the specified *width*. If *withyear* is "True", include the year in the output. The *theyear* and *themonth* parameters specify the year and month for the name to be formatted respectively. prmonth(theyear, themonth, w=0, l=0) Print a month’s calendar as returned by "formatmonth()". formatyear(theyear, w=2, l=1, c=6, m=3) Return a *m*-column calendar for an entire year as a multi-line string. Optional parameters *w*, *l*, and *c* are for date column width, lines per week, and number of spaces between month columns, respectively. Depends on the first weekday as specified in the constructor or set by the "setfirstweekday()" method. The earliest year for which a calendar can be generated is platform-dependent. pryear(theyear, w=2, l=1, c=6, m=3) Print the calendar for an entire year as returned by "formatyear()". class calendar.HTMLCalendar(firstweekday=0) This class can be used to generate HTML calendars. "HTMLCalendar" instances have the following methods: formatmonth(theyear, themonth, withyear=True) Return a month’s calendar as an HTML table. If *withyear* is true the year will be included in the header, otherwise just the month name will be used. formatyear(theyear, width=3) Return a year’s calendar as an HTML table. *width* (defaulting to 3) specifies the number of months per row. formatyearpage(theyear, width=3, css='calendar.css', encoding=None) Return a year’s calendar as a complete HTML page. *width* (defaulting to 3) specifies the number of months per row. *css* is the name for the cascading style sheet to be used. "None" can be passed if no style sheet should be used. *encoding* specifies the encoding to be used for the output (defaulting to the system default encoding). formatmonthname(theyear, themonth, withyear=True) Return a month name as an HTML table row. If *withyear* is true the year will be included in the row, otherwise just the month name will be used. "HTMLCalendar" has the following attributes you can override to customize the CSS classes used by the calendar: cssclasses A list of CSS classes used for each weekday. The default class list is: cssclasses = ["mon", "tue", "wed", "thu", "fri", "sat", "sun"] more styles can be added for each day: cssclasses = ["mon text-bold", "tue", "wed", "thu", "fri", "sat", "sun red"] Note that the length of this list must be seven items. cssclass_noday The CSS class for a weekday occurring in the previous or coming month. Added in version 3.7. cssclasses_weekday_head A list of CSS classes used for weekday names in the header row. The default is the same as "cssclasses". Added in version 3.7. cssclass_month_head The month’s head CSS class (used by "formatmonthname()"). The default value is ""month"". Added in version 3.7. cssclass_month The CSS class for the whole month’s table (used by "formatmonth()"). The default value is ""month"". Added in version 3.7. cssclass_year The CSS class for the whole year’s table of tables (used by "formatyear()"). The default value is ""year"". Added in version 3.7. cssclass_year_head The CSS class for the table head for the whole year (used by "formatyear()"). The default value is ""year"". Added in version 3.7. Note that although the naming for the above described class attributes is singular (e.g. "cssclass_month" "cssclass_noday"), one can replace the single CSS class with a space separated list of CSS classes, for example: "text-bold text-red" Here is an example how "HTMLCalendar" can be customized: class CustomHTMLCal(calendar.HTMLCalendar): cssclasses = [style + " text-nowrap" for style in calendar.HTMLCalendar.cssclasses] cssclass_month_head = "text-center month-head" cssclass_month = "text-center month" cssclass_year = "text-italic lead" class calendar.LocaleTextCalendar(firstweekday=0, locale=None) This subclass of "TextCalendar" can be passed a locale name in the constructor and will return month and weekday names in the specified locale. class calendar.LocaleHTMLCalendar(firstweekday=0, locale=None) This subclass of "HTMLCalendar" can be passed a locale name in the constructor and will return month and weekday names in the specified locale. Note: The constructor, "formatweekday()" and "formatmonthname()" methods of these two classes temporarily change the "LC_TIME" locale to the given *locale*. Because the current locale is a process-wide setting, they are not thread-safe. For simple text calendars this module provides the following functions. calendar.setfirstweekday(weekday) Sets the weekday ("0" is Monday, "6" is Sunday) to start each week. The values "MONDAY", "TUESDAY", "WEDNESDAY", "THURSDAY", "FRIDAY", "SATURDAY", and "SUNDAY" are provided for convenience. For example, to set the first weekday to Sunday: import calendar calendar.setfirstweekday(calendar.SUNDAY) calendar.firstweekday() Returns the current setting for the weekday to start each week. calendar.isleap(year) Returns "True" if *year* is a leap year, otherwise "False". calendar.leapdays(y1, y2) Returns the number of leap years in the range from *y1* to *y2* (exclusive), where *y1* and *y2* are years. This function works for ranges spanning a century change. calendar.weekday(year, month, day) Returns the day of the week ("0" is Monday) for *year* ("1970"–…), *month* ("1"–"12"), *day* ("1"–"31"). calendar.weekheader(n) Return a header containing abbreviated weekday names. *n* specifies the width in characters for one weekday. calendar.monthrange(year, month) Returns weekday of first day of the month and number of days in month, for the specified *year* and *month*. calendar.monthcalendar(year, month) Returns a matrix representing a month’s calendar. Each row represents a week; days outside of the month are represented by zeros. Each week begins with Monday unless set by "setfirstweekday()". calendar.prmonth(theyear, themonth, w=0, l=0) Prints a month’s calendar as returned by "month()". calendar.month(theyear, themonth, w=0, l=0) Returns a month’s calendar in a multi-line string using the "formatmonth()" of the "TextCalendar" class. calendar.prcal(year, w=0, l=0, c=6, m=3) Prints the calendar for an entire year as returned by "calendar()". calendar.calendar(year, w=2, l=1, c=6, m=3) Returns a 3-column calendar for an entire year as a multi-line string using the "formatyear()" of the "TextCalendar" class. calendar.timegm(tuple) An unrelated but handy function that takes a time tuple such as returned by the "gmtime()" function in the "time" module, and returns the corresponding Unix timestamp value, assuming an epoch of 1970, and the POSIX encoding. In fact, "time.gmtime()" and "timegm()" are each others’ inverse. The "calendar" module exports the following data attributes: calendar.day_name A sequence that represents the days of the week in the current locale, where Monday is day number 0. >>> import calendar >>> list(calendar.day_name) ['Monday', 'Tuesday', 'Wednesday', 'Thursday', 'Friday', 'Saturday', 'Sunday'] calendar.day_abbr A sequence that represents the abbreviated days of the week in the current locale, where Mon is day number 0. >>> import calendar >>> list(calendar.day_abbr) ['Mon', 'Tue', 'Wed', 'Thu', 'Fri', 'Sat', 'Sun'] calendar.MONDAY calendar.TUESDAY calendar.WEDNESDAY calendar.THURSDAY calendar.FRIDAY calendar.SATURDAY calendar.SUNDAY Aliases for the days of the week, where "MONDAY" is "0" and "SUNDAY" is "6". Added in version 3.12. class calendar.Day Enumeration defining days of the week as integer constants. The members of this enumeration are exported to the module scope as "MONDAY" through "SUNDAY". Added in version 3.12. calendar.month_name A sequence that represents the months of the year in the current locale. This follows normal convention of January being month number 1, so it has a length of 13 and "month_name[0]" is the empty string. >>> import calendar >>> list(calendar.month_name) ['', 'January', 'February', 'March', 'April', 'May', 'June', 'July', 'August', 'September', 'October', 'November', 'December'] calendar.month_abbr A sequence that represents the abbreviated months of the year in the current locale. This follows normal convention of January being month number 1, so it has a length of 13 and "month_abbr[0]" is the empty string. >>> import calendar >>> list(calendar.month_abbr) ['', 'Jan', 'Feb', 'Mar', 'Apr', 'May', 'Jun', 'Jul', 'Aug', 'Sep', 'Oct', 'Nov', 'Dec'] calendar.JANUARY calendar.FEBRUARY calendar.MARCH calendar.APRIL calendar.MAY calendar.JUNE calendar.JULY calendar.AUGUST calendar.SEPTEMBER calendar.OCTOBER calendar.NOVEMBER calendar.DECEMBER Aliases for the months of the year, where "JANUARY" is "1" and "DECEMBER" is "12". Added in version 3.12. class calendar.Month Enumeration defining months of the year as integer constants. The members of this enumeration are exported to the module scope as "JANUARY" through "DECEMBER". Added in version 3.12. The "calendar" module defines the following exceptions: exception calendar.IllegalMonthError(month) A subclass of "ValueError", raised when the given month number is outside of the range 1-12 (inclusive). month The invalid month number. exception calendar.IllegalWeekdayError(weekday) A subclass of "ValueError", raised when the given weekday number is outside of the range 0-6 (inclusive). weekday The invalid weekday number. See also: Module "datetime" Object-oriented interface to dates and times with similar functionality to the "time" module. Module "time" Low-level time related functions. Command-Line Usage ================== Added in version 2.5. The "calendar" module can be executed as a script from the command line to interactively print a calendar. python -m calendar [-h] [-L LOCALE] [-e ENCODING] [-t {text,html}] [-w WIDTH] [-l LINES] [-s SPACING] [-m MONTHS] [-c CSS] [-f FIRST_WEEKDAY] [year] [month] For example, to print a calendar for the year 2000: $ python -m calendar 2000 2000 January February March Mo Tu We Th Fr Sa Su Mo Tu We Th Fr Sa Su Mo Tu We Th Fr Sa Su 1 2 1 2 3 4 5 6 1 2 3 4 5 3 4 5 6 7 8 9 7 8 9 10 11 12 13 6 7 8 9 10 11 12 10 11 12 13 14 15 16 14 15 16 17 18 19 20 13 14 15 16 17 18 19 17 18 19 20 21 22 23 21 22 23 24 25 26 27 20 21 22 23 24 25 26 24 25 26 27 28 29 30 28 29 27 28 29 30 31 31 April May June Mo Tu We Th Fr Sa Su Mo Tu We Th Fr Sa Su Mo Tu We Th Fr Sa Su 1 2 1 2 3 4 5 6 7 1 2 3 4 3 4 5 6 7 8 9 8 9 10 11 12 13 14 5 6 7 8 9 10 11 10 11 12 13 14 15 16 15 16 17 18 19 20 21 12 13 14 15 16 17 18 17 18 19 20 21 22 23 22 23 24 25 26 27 28 19 20 21 22 23 24 25 24 25 26 27 28 29 30 29 30 31 26 27 28 29 30 July August September Mo Tu We Th Fr Sa Su Mo Tu We Th Fr Sa Su Mo Tu We Th Fr Sa Su 1 2 1 2 3 4 5 6 1 2 3 3 4 5 6 7 8 9 7 8 9 10 11 12 13 4 5 6 7 8 9 10 10 11 12 13 14 15 16 14 15 16 17 18 19 20 11 12 13 14 15 16 17 17 18 19 20 21 22 23 21 22 23 24 25 26 27 18 19 20 21 22 23 24 24 25 26 27 28 29 30 28 29 30 31 25 26 27 28 29 30 31 October November December Mo Tu We Th Fr Sa Su Mo Tu We Th Fr Sa Su Mo Tu We Th Fr Sa Su 1 1 2 3 4 5 1 2 3 2 3 4 5 6 7 8 6 7 8 9 10 11 12 4 5 6 7 8 9 10 9 10 11 12 13 14 15 13 14 15 16 17 18 19 11 12 13 14 15 16 17 16 17 18 19 20 21 22 20 21 22 23 24 25 26 18 19 20 21 22 23 24 23 24 25 26 27 28 29 27 28 29 30 25 26 27 28 29 30 31 30 31 The following options are accepted: --help, -h Show the help message and exit. --locale LOCALE, -L LOCALE The locale to use for month and weekday names. Defaults to English. --encoding ENCODING, -e ENCODING The encoding to use for output. "--encoding" is required if "-- locale" is set. --type {text,html}, -t {text,html} Print the calendar to the terminal as text, or as an HTML document. --first-weekday FIRST_WEEKDAY, -f FIRST_WEEKDAY The weekday to start each week. Must be a number between 0 (Monday) and 6 (Sunday). Defaults to 0. Added in version 3.13. year The year to print the calendar for. Defaults to the current year. month The month of the specified "year" to print the calendar for. Must be a number between 1 and 12, and may only be used in text mode. Defaults to printing a calendar for the full year. *Text-mode options:* --width WIDTH, -w WIDTH The width of the date column in terminal columns. The date is printed centred in the column. Any value lower than 2 is ignored. Defaults to 2. --lines LINES, -l LINES The number of lines for each week in terminal rows. The date is printed top-aligned. Any value lower than 1 is ignored. Defaults to 1. --spacing SPACING, -s SPACING The space between months in columns. Any value lower than 2 is ignored. Defaults to 6. --months MONTHS, -m MONTHS The number of months printed per row. Defaults to 3. *HTML-mode options:* --css CSS, -c CSS The path of a CSS stylesheet to use for the calendar. This must either be relative to the generated HTML, or an absolute HTTP or "file:///" URL. "cgi" — Common Gateway Interface support **************************************** Deprecated since version 3.11, removed in version 3.13. This module is no longer part of the Python standard library. It was removed in Python 3.13 after being deprecated in Python 3.11. The removal was decided in **PEP 594**. A fork of the module on PyPI can be used instead: legacy-cgi. This is a copy of the cgi module, no longer maintained or supported by the core Python team. The last version of Python that provided the "cgi" module was Python 3.12. "cgitb" — Traceback manager for CGI scripts ******************************************* Deprecated since version 3.11, removed in version 3.13. This module is no longer part of the Python standard library. It was removed in Python 3.13 after being deprecated in Python 3.11. The removal was decided in **PEP 594**. A fork of the module on PyPI can now be used instead: legacy-cgi. This is a copy of the cgi module, no longer maintained or supported by the core Python team. The last version of Python that provided the "cgitb" module was Python 3.12. "chunk" — Read IFF chunked data ******************************* Deprecated since version 3.11, removed in version 3.13. This module is no longer part of the Python standard library. It was removed in Python 3.13 after being deprecated in Python 3.11. The removal was decided in **PEP 594**. The last version of Python that provided the "chunk" module was Python 3.12. "cmath" — Mathematical functions for complex numbers **************************************************** ====================================================================== This module provides access to mathematical functions for complex numbers. The functions in this module accept integers, floating-point numbers or complex numbers as arguments. They will also accept any Python object that has either a "__complex__()" or a "__float__()" method: these methods are used to convert the object to a complex or floating-point number, respectively, and the function is then applied to the result of the conversion. Note: For functions involving branch cuts, we have the problem of deciding how to define those functions on the cut itself. Following Kahan’s “Branch cuts for complex elementary functions” paper, as well as Annex G of C99 and later C standards, we use the sign of zero to distinguish one side of the branch cut from the other: for a branch cut along (a portion of) the real axis we look at the sign of the imaginary part, while for a branch cut along the imaginary axis we look at the sign of the real part.For example, the "cmath.sqrt()" function has a branch cut along the negative real axis. An argument of "complex(-2.0, -0.0)" is treated as though it lies *below* the branch cut, and so gives a result on the negative imaginary axis: >>> cmath.sqrt(complex(-2.0, -0.0)) -1.4142135623730951j But an argument of "complex(-2.0, 0.0)" is treated as though it lies above the branch cut: >>> cmath.sqrt(complex(-2.0, 0.0)) 1.4142135623730951j +------------------------------------------------------+--------------------------------------------------------------------+ | **Conversions to and from polar coordinates** | +------------------------------------------------------+--------------------------------------------------------------------+ | "phase(z)" | Return the phase of *z* | +------------------------------------------------------+--------------------------------------------------------------------+ | "polar(z)" | Return the representation of *z* in polar coordinates | +------------------------------------------------------+--------------------------------------------------------------------+ | "rect(r, phi)" | Return the complex number *z* with polar coordinates *r* and *phi* | +------------------------------------------------------+--------------------------------------------------------------------+ | **Power and logarithmic functions** | +------------------------------------------------------+--------------------------------------------------------------------+ | "exp(z)" | Return *e* raised to the power *z* | +------------------------------------------------------+--------------------------------------------------------------------+ | "log(z[, base])" | Return the logarithm of *z* to the given *base* (*e* by default) | +------------------------------------------------------+--------------------------------------------------------------------+ | "log10(z)" | Return the base-10 logarithm of *z* | +------------------------------------------------------+--------------------------------------------------------------------+ | "sqrt(z)" | Return the square root of *z* | +------------------------------------------------------+--------------------------------------------------------------------+ | **Trigonometric functions** | +------------------------------------------------------+--------------------------------------------------------------------+ | "acos(z)" | Return the arc cosine of *z* | +------------------------------------------------------+--------------------------------------------------------------------+ | "asin(z)" | Return the arc sine of *z* | +------------------------------------------------------+--------------------------------------------------------------------+ | "atan(z)" | Return the arc tangent of *z* | +------------------------------------------------------+--------------------------------------------------------------------+ | "cos(z)" | Return the cosine of *z* | +------------------------------------------------------+--------------------------------------------------------------------+ | "sin(z)" | Return the sine of *z* | +------------------------------------------------------+--------------------------------------------------------------------+ | "tan(z)" | Return the tangent of *z* | +------------------------------------------------------+--------------------------------------------------------------------+ | **Hyperbolic functions** | +------------------------------------------------------+--------------------------------------------------------------------+ | "acosh(z)" | Return the inverse hyperbolic cosine of *z* | +------------------------------------------------------+--------------------------------------------------------------------+ | "asinh(z)" | Return the inverse hyperbolic sine of *z* | +------------------------------------------------------+--------------------------------------------------------------------+ | "atanh(z)" | Return the inverse hyperbolic tangent of *z* | +------------------------------------------------------+--------------------------------------------------------------------+ | "cosh(z)" | Return the hyperbolic cosine of *z* | +------------------------------------------------------+--------------------------------------------------------------------+ | "sinh(z)" | Return the hyperbolic sine of *z* | +------------------------------------------------------+--------------------------------------------------------------------+ | "tanh(z)" | Return the hyperbolic tangent of *z* | +------------------------------------------------------+--------------------------------------------------------------------+ | **Classification functions** | +------------------------------------------------------+--------------------------------------------------------------------+ | "isfinite(z)" | Check if all components of *z* are finite | +------------------------------------------------------+--------------------------------------------------------------------+ | "isinf(z)" | Check if any component of *z* is infinite | +------------------------------------------------------+--------------------------------------------------------------------+ | "isnan(z)" | Check if any component of *z* is a NaN | +------------------------------------------------------+--------------------------------------------------------------------+ | "isclose(a, b, *, rel_tol, abs_tol)" | Check if the values *a* and *b* are close to each other | +------------------------------------------------------+--------------------------------------------------------------------+ | **Constants** | +------------------------------------------------------+--------------------------------------------------------------------+ | "pi" | *π* = 3.141592… | +------------------------------------------------------+--------------------------------------------------------------------+ | "e" | *e* = 2.718281… | +------------------------------------------------------+--------------------------------------------------------------------+ | "tau" | *τ* = 2*π* = 6.283185… | +------------------------------------------------------+--------------------------------------------------------------------+ | "inf" | Positive infinity | +------------------------------------------------------+--------------------------------------------------------------------+ | "infj" | Pure imaginary infinity | +------------------------------------------------------+--------------------------------------------------------------------+ | "nan" | “Not a number” (NaN) | +------------------------------------------------------+--------------------------------------------------------------------+ | "nanj" | Pure imaginary NaN | +------------------------------------------------------+--------------------------------------------------------------------+ Conversions to and from polar coordinates ========================================= A Python complex number "z" is stored internally using *rectangular* or *Cartesian* coordinates. It is completely determined by its *real part* "z.real" and its *imaginary part* "z.imag". *Polar coordinates* give an alternative way to represent a complex number. In polar coordinates, a complex number *z* is defined by the modulus *r* and the phase angle *phi*. The modulus *r* is the distance from *z* to the origin, while the phase *phi* is the counterclockwise angle, measured in radians, from the positive x-axis to the line segment that joins the origin to *z*. The following functions can be used to convert from the native rectangular coordinates to polar coordinates and back. cmath.phase(z) Return the phase of *z* (also known as the *argument* of *z*), as a float. "phase(z)" is equivalent to "math.atan2(z.imag, z.real)". The result lies in the range [-*π*, *π*], and the branch cut for this operation lies along the negative real axis. The sign of the result is the same as the sign of "z.imag", even when "z.imag" is zero: >>> phase(complex(-1.0, 0.0)) 3.141592653589793 >>> phase(complex(-1.0, -0.0)) -3.141592653589793 Note: The modulus (absolute value) of a complex number *z* can be computed using the built-in "abs()" function. There is no separate "cmath" module function for this operation. cmath.polar(z) Return the representation of *z* in polar coordinates. Returns a pair "(r, phi)" where *r* is the modulus of *z* and *phi* is the phase of *z*. "polar(z)" is equivalent to "(abs(z), phase(z))". cmath.rect(r, phi) Return the complex number *z* with polar coordinates *r* and *phi*. Equivalent to "complex(r * math.cos(phi), r * math.sin(phi))". Power and logarithmic functions =============================== cmath.exp(z) Return *e* raised to the power *z*, where *e* is the base of natural logarithms. cmath.log(z[, base]) Return the logarithm of *z* to the given *base*. If the *base* is not specified, returns the natural logarithm of *z*. There is one branch cut, from 0 along the negative real axis to -∞. cmath.log10(z) Return the base-10 logarithm of *z*. This has the same branch cut as "log()". cmath.sqrt(z) Return the square root of *z*. This has the same branch cut as "log()". Trigonometric functions ======================= cmath.acos(z) Return the arc cosine of *z*. There are two branch cuts: One extends right from 1 along the real axis to ∞. The other extends left from -1 along the real axis to -∞. cmath.asin(z) Return the arc sine of *z*. This has the same branch cuts as "acos()". cmath.atan(z) Return the arc tangent of *z*. There are two branch cuts: One extends from "1j" along the imaginary axis to "∞j". The other extends from "-1j" along the imaginary axis to "-∞j". cmath.cos(z) Return the cosine of *z*. cmath.sin(z) Return the sine of *z*. cmath.tan(z) Return the tangent of *z*. Hyperbolic functions ==================== cmath.acosh(z) Return the inverse hyperbolic cosine of *z*. There is one branch cut, extending left from 1 along the real axis to -∞. cmath.asinh(z) Return the inverse hyperbolic sine of *z*. There are two branch cuts: One extends from "1j" along the imaginary axis to "∞j". The other extends from "-1j" along the imaginary axis to "-∞j". cmath.atanh(z) Return the inverse hyperbolic tangent of *z*. There are two branch cuts: One extends from "1" along the real axis to "∞". The other extends from "-1" along the real axis to "-∞". cmath.cosh(z) Return the hyperbolic cosine of *z*. cmath.sinh(z) Return the hyperbolic sine of *z*. cmath.tanh(z) Return the hyperbolic tangent of *z*. Classification functions ======================== cmath.isfinite(z) Return "True" if both the real and imaginary parts of *z* are finite, and "False" otherwise. Added in version 3.2. cmath.isinf(z) Return "True" if either the real or the imaginary part of *z* is an infinity, and "False" otherwise. cmath.isnan(z) Return "True" if either the real or the imaginary part of *z* is a NaN, and "False" otherwise. cmath.isclose(a, b, *, rel_tol=1e-09, abs_tol=0.0) Return "True" if the values *a* and *b* are close to each other and "False" otherwise. Whether or not two values are considered close is determined according to given absolute and relative tolerances. If no errors occur, the result will be: "abs(a-b) <= max(rel_tol * max(abs(a), abs(b)), abs_tol)". *rel_tol* is the relative tolerance – it is the maximum allowed difference between *a* and *b*, relative to the larger absolute value of *a* or *b*. For example, to set a tolerance of 5%, pass "rel_tol=0.05". The default tolerance is "1e-09", which assures that the two values are the same within about 9 decimal digits. *rel_tol* must be nonnegative and less than "1.0". *abs_tol* is the absolute tolerance; it defaults to "0.0" and it must be nonnegative. When comparing "x" to "0.0", "isclose(x, 0)" is computed as "abs(x) <= rel_tol * abs(x)", which is "False" for any "x" and rel_tol less than "1.0". So add an appropriate positive abs_tol argument to the call. The IEEE 754 special values of "NaN", "inf", and "-inf" will be handled according to IEEE rules. Specifically, "NaN" is not considered close to any other value, including "NaN". "inf" and "-inf" are only considered close to themselves. Added in version 3.5. See also: **PEP 485** – A function for testing approximate equality Constants ========= cmath.pi The mathematical constant *π*, as a float. cmath.e The mathematical constant *e*, as a float. cmath.tau The mathematical constant *τ*, as a float. Added in version 3.6. cmath.inf Floating-point positive infinity. Equivalent to "float('inf')". Added in version 3.6. cmath.infj Complex number with zero real part and positive infinity imaginary part. Equivalent to "complex(0.0, float('inf'))". Added in version 3.6. cmath.nan A floating-point “not a number” (NaN) value. Equivalent to "float('nan')". Added in version 3.6. cmath.nanj Complex number with zero real part and NaN imaginary part. Equivalent to "complex(0.0, float('nan'))". Added in version 3.6. Note that the selection of functions is similar, but not identical, to that in module "math". The reason for having two modules is that some users aren’t interested in complex numbers, and perhaps don’t even know what they are. They would rather have "math.sqrt(-1)" raise an exception than return a complex number. Also note that the functions defined in "cmath" always return a complex number, even if the answer can be expressed as a real number (in which case the complex number has an imaginary part of zero). A note on branch cuts: They are curves along which the given function fails to be continuous. They are a necessary feature of many complex functions. It is assumed that if you need to compute with complex functions, you will understand about branch cuts. Consult almost any (not too elementary) book on complex variables for enlightenment. For information of the proper choice of branch cuts for numerical purposes, a good reference should be the following: See also: Kahan, W: Branch cuts for complex elementary functions; or, Much ado about nothing’s sign bit. In Iserles, A., and Powell, M. (eds.), The state of the art in numerical analysis. Clarendon Press (1987) pp165–211. "cmd" — Support for line-oriented command interpreters ****************************************************** **Source code:** Lib/cmd.py ====================================================================== The "Cmd" class provides a simple framework for writing line-oriented command interpreters. These are often useful for test harnesses, administrative tools, and prototypes that will later be wrapped in a more sophisticated interface. class cmd.Cmd(completekey='tab', stdin=None, stdout=None) A "Cmd" instance or subclass instance is a line-oriented interpreter framework. There is no good reason to instantiate "Cmd" itself; rather, it’s useful as a superclass of an interpreter class you define yourself in order to inherit "Cmd"’s methods and encapsulate action methods. The optional argument *completekey* is the "readline" name of a completion key; it defaults to "Tab". If *completekey* is not "None" and "readline" is available, command completion is done automatically. The default, "'tab'", is treated specially, so that it refers to the "Tab" key on every "readline.backend". Specifically, if "readline.backend" is "editline", "Cmd" will use "'^I'" instead of "'tab'". Note that other values are not treated this way, and might only work with a specific backend. The optional arguments *stdin* and *stdout* specify the input and output file objects that the Cmd instance or subclass instance will use for input and output. If not specified, they will default to "sys.stdin" and "sys.stdout". If you want a given *stdin* to be used, make sure to set the instance’s "use_rawinput" attribute to "False", otherwise *stdin* will be ignored. Changed in version 3.13: "completekey='tab'" is replaced by "'^I'" for "editline". Cmd Objects =========== A "Cmd" instance has the following methods: Cmd.cmdloop(intro=None) Repeatedly issue a prompt, accept input, parse an initial prefix off the received input, and dispatch to action methods, passing them the remainder of the line as argument. The optional argument is a banner or intro string to be issued before the first prompt (this overrides the "intro" class attribute). If the "readline" module is loaded, input will automatically inherit **bash**-like history-list editing (e.g. "Control"-"P" scrolls back to the last command, "Control"-"N" forward to the next one, "Control"-"F" moves the cursor to the right non-destructively, "Control"-"B" moves the cursor to the left non-destructively, etc.). An end-of-file on input is passed back as the string "'EOF'". An interpreter instance will recognize a command name "foo" if and only if it has a method "do_foo()". As a special case, a line beginning with the character "'?'" is dispatched to the method "do_help()". As another special case, a line beginning with the character "'!'" is dispatched to the method "do_shell()" (if such a method is defined). This method will return when the "postcmd()" method returns a true value. The *stop* argument to "postcmd()" is the return value from the command’s corresponding "do_*()" method. If completion is enabled, completing commands will be done automatically, and completing of commands args is done by calling "complete_foo()" with arguments *text*, *line*, *begidx*, and *endidx*. *text* is the string prefix we are attempting to match: all returned matches must begin with it. *line* is the current input line with leading whitespace removed, *begidx* and *endidx* are the beginning and ending indexes of the prefix text, which could be used to provide different completion depending upon which position the argument is in. Cmd.do_help(arg) All subclasses of "Cmd" inherit a predefined "do_help()". This method, called with an argument "'bar'", invokes the corresponding method "help_bar()", and if that is not present, prints the docstring of "do_bar()", if available. With no argument, "do_help()" lists all available help topics (that is, all commands with corresponding "help_*()" methods or commands that have docstrings), and also lists any undocumented commands. Cmd.onecmd(str) Interpret the argument as though it had been typed in response to the prompt. This may be overridden, but should not normally need to be; see the "precmd()" and "postcmd()" methods for useful execution hooks. The return value is a flag indicating whether interpretation of commands by the interpreter should stop. If there is a "do_*()" method for the command *str*, the return value of that method is returned, otherwise the return value from the "default()" method is returned. Cmd.emptyline() Method called when an empty line is entered in response to the prompt. If this method is not overridden, it repeats the last nonempty command entered. Cmd.default(line) Method called on an input line when the command prefix is not recognized. If this method is not overridden, it prints an error message and returns. Cmd.completedefault(text, line, begidx, endidx) Method called to complete an input line when no command-specific "complete_*()" method is available. By default, it returns an empty list. Cmd.columnize(list, displaywidth=80) Method called to display a list of strings as a compact set of columns. Each column is only as wide as necessary. Columns are separated by two spaces for readability. Cmd.precmd(line) Hook method executed just before the command line *line* is interpreted, but after the input prompt is generated and issued. This method is a stub in "Cmd"; it exists to be overridden by subclasses. The return value is used as the command which will be executed by the "onecmd()" method; the "precmd()" implementation may re-write the command or simply return *line* unchanged. Cmd.postcmd(stop, line) Hook method executed just after a command dispatch is finished. This method is a stub in "Cmd"; it exists to be overridden by subclasses. *line* is the command line which was executed, and *stop* is a flag which indicates whether execution will be terminated after the call to "postcmd()"; this will be the return value of the "onecmd()" method. The return value of this method will be used as the new value for the internal flag which corresponds to *stop*; returning false will cause interpretation to continue. Cmd.preloop() Hook method executed once when "cmdloop()" is called. This method is a stub in "Cmd"; it exists to be overridden by subclasses. Cmd.postloop() Hook method executed once when "cmdloop()" is about to return. This method is a stub in "Cmd"; it exists to be overridden by subclasses. Instances of "Cmd" subclasses have some public instance variables: Cmd.prompt The prompt issued to solicit input. Cmd.identchars The string of characters accepted for the command prefix. Cmd.lastcmd The last nonempty command prefix seen. Cmd.cmdqueue A list of queued input lines. The cmdqueue list is checked in "cmdloop()" when new input is needed; if it is nonempty, its elements will be processed in order, as if entered at the prompt. Cmd.intro A string to issue as an intro or banner. May be overridden by giving the "cmdloop()" method an argument. Cmd.doc_header The header to issue if the help output has a section for documented commands. Cmd.misc_header The header to issue if the help output has a section for miscellaneous help topics (that is, there are "help_*()" methods without corresponding "do_*()" methods). Cmd.undoc_header The header to issue if the help output has a section for undocumented commands (that is, there are "do_*()" methods without corresponding "help_*()" methods). Cmd.ruler The character used to draw separator lines under the help-message headers. If empty, no ruler line is drawn. It defaults to "'='". Cmd.use_rawinput A flag, defaulting to true. If true, "cmdloop()" uses "input()" to display a prompt and read the next command; if false, "sys.stdout.write()" and "sys.stdin.readline()" are used. (This means that by importing "readline", on systems that support it, the interpreter will automatically support **Emacs**-like line editing and command-history keystrokes.) Cmd Example =========== The "cmd" module is mainly useful for building custom shells that let a user work with a program interactively. This section presents a simple example of how to build a shell around a few of the commands in the "turtle" module. Basic turtle commands such as "forward()" are added to a "Cmd" subclass with method named "do_forward()". The argument is converted to a number and dispatched to the turtle module. The docstring is used in the help utility provided by the shell. The example also includes a basic record and playback facility implemented with the "precmd()" method which is responsible for converting the input to lowercase and writing the commands to a file. The "do_playback()" method reads the file and adds the recorded commands to the "cmdqueue" for immediate playback: import cmd, sys from turtle import * class TurtleShell(cmd.Cmd): intro = 'Welcome to the turtle shell. Type help or ? to list commands.\n' prompt = '(turtle) ' file = None # ----- basic turtle commands ----- def do_forward(self, arg): 'Move the turtle forward by the specified distance: FORWARD 10' forward(*parse(arg)) def do_right(self, arg): 'Turn turtle right by given number of degrees: RIGHT 20' right(*parse(arg)) def do_left(self, arg): 'Turn turtle left by given number of degrees: LEFT 90' left(*parse(arg)) def do_goto(self, arg): 'Move turtle to an absolute position with changing orientation. GOTO 100 200' goto(*parse(arg)) def do_home(self, arg): 'Return turtle to the home position: HOME' home() def do_circle(self, arg): 'Draw circle with given radius an options extent and steps: CIRCLE 50' circle(*parse(arg)) def do_position(self, arg): 'Print the current turtle position: POSITION' print('Current position is %d %d\n' % position()) def do_heading(self, arg): 'Print the current turtle heading in degrees: HEADING' print('Current heading is %d\n' % (heading(),)) def do_color(self, arg): 'Set the color: COLOR BLUE' color(arg.lower()) def do_undo(self, arg): 'Undo (repeatedly) the last turtle action(s): UNDO' def do_reset(self, arg): 'Clear the screen and return turtle to center: RESET' reset() def do_bye(self, arg): 'Stop recording, close the turtle window, and exit: BYE' print('Thank you for using Turtle') self.close() bye() return True # ----- record and playback ----- def do_record(self, arg): 'Save future commands to filename: RECORD rose.cmd' self.file = open(arg, 'w') def do_playback(self, arg): 'Playback commands from a file: PLAYBACK rose.cmd' self.close() with open(arg) as f: self.cmdqueue.extend(f.read().splitlines()) def precmd(self, line): line = line.lower() if self.file and 'playback' not in line: print(line, file=self.file) return line def close(self): if self.file: self.file.close() self.file = None def parse(arg): 'Convert a series of zero or more numbers to an argument tuple' return tuple(map(int, arg.split())) if __name__ == '__main__': TurtleShell().cmdloop() Here is a sample session with the turtle shell showing the help functions, using blank lines to repeat commands, and the simple record and playback facility: Welcome to the turtle shell. Type help or ? to list commands. (turtle) ? Documented commands (type help ): ======================================== bye color goto home playback record right circle forward heading left position reset undo (turtle) help forward Move the turtle forward by the specified distance: FORWARD 10 (turtle) record spiral.cmd (turtle) position Current position is 0 0 (turtle) heading Current heading is 0 (turtle) reset (turtle) circle 20 (turtle) right 30 (turtle) circle 40 (turtle) right 30 (turtle) circle 60 (turtle) right 30 (turtle) circle 80 (turtle) right 30 (turtle) circle 100 (turtle) right 30 (turtle) circle 120 (turtle) right 30 (turtle) circle 120 (turtle) heading Current heading is 180 (turtle) forward 100 (turtle) (turtle) right 90 (turtle) forward 100 (turtle) (turtle) right 90 (turtle) forward 400 (turtle) right 90 (turtle) forward 500 (turtle) right 90 (turtle) forward 400 (turtle) right 90 (turtle) forward 300 (turtle) playback spiral.cmd Current position is 0 0 Current heading is 0 Current heading is 180 (turtle) bye Thank you for using Turtle Modules command-line interface (CLI) ************************************ The following modules have a command-line interface. * ast * asyncio * "base64" * calendar * "code" * compileall * "cProfile": see profile * difflib * dis * doctest * "encodings.rot_13" * "ensurepip" * "filecmp" * "fileinput" * "ftplib" * gzip * http.server * "idlelib" * inspect * json.tool * "mimetypes" * "pdb" * "pickle" * pickletools * platform * "poplib" * profile * "pstats" * py_compile * "pyclbr" * "pydoc" * "quopri" * random * "runpy" * site * sqlite3 * symtable * sysconfig * "tabnanny" * tarfile * "this" * timeit * tokenize * trace * "turtledemo" * unittest * uuid * "venv" * "webbrowser" * zipapp * zipfile See also the Python command-line interface. Command Line Interface Libraries ******************************** The modules described in this chapter assist with implementing command line and terminal interfaces for applications. Here’s an overview: * "argparse" — Parser for command-line options, arguments and subcommands * "optparse" — Parser for command line options * "getpass" — Portable password input * "fileinput" — Iterate over lines from multiple input streams * "curses" — Terminal handling for character-cell displays * "curses.textpad" — Text input widget for curses programs * "curses.ascii" — Utilities for ASCII characters * "curses.panel" — A panel stack extension for curses "code" — Interpreter base classes ********************************* **Source code:** Lib/code.py ====================================================================== The "code" module provides facilities to implement read-eval-print loops in Python. Two classes and convenience functions are included which can be used to build applications which provide an interactive interpreter prompt. class code.InteractiveInterpreter(locals=None) This class deals with parsing and interpreter state (the user’s namespace); it does not deal with input buffering or prompting or input file naming (the filename is always passed in explicitly). The optional *locals* argument specifies a mapping to use as the namespace in which code will be executed; it defaults to a newly created dictionary with key "'__name__'" set to "'__console__'" and key "'__doc__'" set to "None". Note that functions and classes objects created under an "InteractiveInterpreter" instance will belong to the namespace specified by *locals*. They are only pickleable if *locals* is the namespace of an existing module. class code.InteractiveConsole(locals=None, filename='', local_exit=False) Closely emulate the behavior of the interactive Python interpreter. This class builds on "InteractiveInterpreter" and adds prompting using the familiar "sys.ps1" and "sys.ps2", and input buffering. If *local_exit* is true, "exit()" and "quit()" in the console will not raise "SystemExit", but instead return to the calling code. Changed in version 3.13: Added *local_exit* parameter. code.interact(banner=None, readfunc=None, local=None, exitmsg=None, local_exit=False) Convenience function to run a read-eval-print loop. This creates a new instance of "InteractiveConsole" and sets *readfunc* to be used as the "InteractiveConsole.raw_input()" method, if provided. If *local* is provided, it is passed to the "InteractiveConsole" constructor for use as the default namespace for the interpreter loop. If *local_exit* is provided, it is passed to the "InteractiveConsole" constructor. The "interact()" method of the instance is then run with *banner* and *exitmsg* passed as the banner and exit message to use, if provided. The console object is discarded after use. Changed in version 3.6: Added *exitmsg* parameter. Changed in version 3.13: Added *local_exit* parameter. code.compile_command(source, filename='', symbol='single') This function is useful for programs that want to emulate Python’s interpreter main loop (a.k.a. the read-eval-print loop). The tricky part is to determine when the user has entered an incomplete command that can be completed by entering more text (as opposed to a complete command or a syntax error). This function *almost* always makes the same decision as the real interpreter main loop. *source* is the source string; *filename* is the optional filename from which source was read, defaulting to "''"; and *symbol* is the optional grammar start symbol, which should be "'single'" (the default), "'eval'" or "'exec'". Returns a code object (the same as "compile(source, filename, symbol)") if the command is complete and valid; "None" if the command is incomplete; raises "SyntaxError" if the command is complete and contains a syntax error, or raises "OverflowError" or "ValueError" if the command contains an invalid literal. Interactive Interpreter Objects =============================== InteractiveInterpreter.runsource(source, filename='', symbol='single') Compile and run some source in the interpreter. Arguments are the same as for "compile_command()"; the default for *filename* is "''", and for *symbol* is "'single'". One of several things can happen: * The input is incorrect; "compile_command()" raised an exception ("SyntaxError" or "OverflowError"). A syntax traceback will be printed by calling the "showsyntaxerror()" method. "runsource()" returns "False". * The input is incomplete, and more input is required; "compile_command()" returned "None". "runsource()" returns "True". * The input is complete; "compile_command()" returned a code object. The code is executed by calling the "runcode()" (which also handles run-time exceptions, except for "SystemExit"). "runsource()" returns "False". The return value can be used to decide whether to use "sys.ps1" or "sys.ps2" to prompt the next line. InteractiveInterpreter.runcode(code) Execute a code object. When an exception occurs, "showtraceback()" is called to display a traceback. All exceptions are caught except "SystemExit", which is allowed to propagate. A note about "KeyboardInterrupt": this exception may occur elsewhere in this code, and may not always be caught. The caller should be prepared to deal with it. InteractiveInterpreter.showsyntaxerror(filename=None) Display the syntax error that just occurred. This does not display a stack trace because there isn’t one for syntax errors. If *filename* is given, it is stuffed into the exception instead of the default filename provided by Python’s parser, because it always uses "''" when reading from a string. The output is written by the "write()" method. InteractiveInterpreter.showtraceback() Display the exception that just occurred. We remove the first stack item because it is within the interpreter object implementation. The output is written by the "write()" method. Changed in version 3.5: The full chained traceback is displayed instead of just the primary traceback. InteractiveInterpreter.write(data) Write a string to the standard error stream ("sys.stderr"). Derived classes should override this to provide the appropriate output handling as needed. Interactive Console Objects =========================== The "InteractiveConsole" class is a subclass of "InteractiveInterpreter", and so offers all the methods of the interpreter objects as well as the following additions. InteractiveConsole.interact(banner=None, exitmsg=None) Closely emulate the interactive Python console. The optional *banner* argument specify the banner to print before the first interaction; by default it prints a banner similar to the one printed by the standard Python interpreter, followed by the class name of the console object in parentheses (so as not to confuse this with the real interpreter – since it’s so close!). The optional *exitmsg* argument specifies an exit message printed when exiting. Pass the empty string to suppress the exit message. If *exitmsg* is not given or "None", a default message is printed. Changed in version 3.4: To suppress printing any banner, pass an empty string. Changed in version 3.6: Print an exit message when exiting. InteractiveConsole.push(line) Push a line of source text to the interpreter. The line should not have a trailing newline; it may have internal newlines. The line is appended to a buffer and the interpreter’s "runsource()" method is called with the concatenated contents of the buffer as source. If this indicates that the command was executed or invalid, the buffer is reset; otherwise, the command is incomplete, and the buffer is left as it was after the line was appended. The return value is "True" if more input is required, "False" if the line was dealt with in some way (this is the same as "runsource()"). InteractiveConsole.resetbuffer() Remove any unhandled source text from the input buffer. InteractiveConsole.raw_input(prompt='') Write a prompt and read a line. The returned line does not include the trailing newline. When the user enters the EOF key sequence, "EOFError" is raised. The base implementation reads from "sys.stdin"; a subclass may replace this with a different implementation. "codeop" — Compile Python code ****************************** **Source code:** Lib/codeop.py ====================================================================== The "codeop" module provides utilities upon which the Python read- eval-print loop can be emulated, as is done in the "code" module. As a result, you probably don’t want to use the module directly; if you want to include such a loop in your program you probably want to use the "code" module instead. There are two parts to this job: 1. Being able to tell if a line of input completes a Python statement: in short, telling whether to print ‘">>>"’ or ‘"..."’ next. 2. Remembering which future statements the user has entered, so subsequent input can be compiled with these in effect. The "codeop" module provides a way of doing each of these things, and a way of doing them both. To do just the former: codeop.compile_command(source, filename='', symbol='single') Tries to compile *source*, which should be a string of Python code and return a code object if *source* is valid Python code. In that case, the filename attribute of the code object will be *filename*, which defaults to "''". Returns "None" if *source* is *not* valid Python code, but is a prefix of valid Python code. If there is a problem with *source*, an exception will be raised. "SyntaxError" is raised if there is invalid Python syntax, and "OverflowError" or "ValueError" if there is an invalid literal. The *symbol* argument determines whether *source* is compiled as a statement ("'single'", the default), as a sequence of *statement* ("'exec'") or as an *expression* ("'eval'"). Any other value will cause "ValueError" to be raised. Note: It is possible (but not likely) that the parser stops parsing with a successful outcome before reaching the end of the source; in this case, trailing symbols may be ignored instead of causing an error. For example, a backslash followed by two newlines may be followed by arbitrary garbage. This will be fixed once the API for the parser is better. class codeop.Compile Instances of this class have "__call__()" methods identical in signature to the built-in function "compile()", but with the difference that if the instance compiles program text containing a "__future__" statement, the instance ‘remembers’ and compiles all subsequent program texts with the statement in force. class codeop.CommandCompiler Instances of this class have "__call__()" methods identical in signature to "compile_command()"; the difference is that if the instance compiles program text containing a "__future__" statement, the instance ‘remembers’ and compiles all subsequent program texts with the statement in force. "collections.abc" — Abstract Base Classes for Containers ******************************************************** Added in version 3.3: Formerly, this module was part of the "collections" module. **Source code:** Lib/_collections_abc.py ====================================================================== This module provides *abstract base classes* that can be used to test whether a class provides a particular interface; for example, whether it is *hashable* or whether it is a *mapping*. An "issubclass()" or "isinstance()" test for an interface works in one of three ways. 1. A newly written class can inherit directly from one of the abstract base classes. The class must supply the required abstract methods. The remaining mixin methods come from inheritance and can be overridden if desired. Other methods may be added as needed: class C(Sequence): # Direct inheritance def __init__(self): ... # Extra method not required by the ABC def __getitem__(self, index): ... # Required abstract method def __len__(self): ... # Required abstract method def count(self, value): ... # Optionally override a mixin method >>> issubclass(C, Sequence) True >>> isinstance(C(), Sequence) True 2. Existing classes and built-in classes can be registered as “virtual subclasses” of the ABCs. Those classes should define the full API including all of the abstract methods and all of the mixin methods. This lets users rely on "issubclass()" or "isinstance()" tests to determine whether the full interface is supported. The exception to this rule is for methods that are automatically inferred from the rest of the API: class D: # No inheritance def __init__(self): ... # Extra method not required by the ABC def __getitem__(self, index): ... # Abstract method def __len__(self): ... # Abstract method def count(self, value): ... # Mixin method def index(self, value): ... # Mixin method Sequence.register(D) # Register instead of inherit >>> issubclass(D, Sequence) True >>> isinstance(D(), Sequence) True In this example, class "D" does not need to define "__contains__", "__iter__", and "__reversed__" because the in-operator, the *iteration* logic, and the "reversed()" function automatically fall back to using "__getitem__" and "__len__". 3. Some simple interfaces are directly recognizable by the presence of the required methods (unless those methods have been set to "None"): class E: def __iter__(self): ... def __next__(self): ... >>> issubclass(E, Iterable) True >>> isinstance(E(), Iterable) True Complex interfaces do not support this last technique because an interface is more than just the presence of method names. Interfaces specify semantics and relationships between methods that cannot be inferred solely from the presence of specific method names. For example, knowing that a class supplies "__getitem__", "__len__", and "__iter__" is insufficient for distinguishing a "Sequence" from a "Mapping". Added in version 3.9: These abstract classes now support "[]". See Generic Alias Type and **PEP 585**. Collections Abstract Base Classes ================================= The collections module offers the following *ABCs*: +--------------------------------+------------------------+-------------------------+------------------------------------------------------+ | ABC | Inherits from | Abstract Methods | Mixin Methods | |================================|========================|=========================|======================================================| | "Container" [1] | | "__contains__" | | +--------------------------------+------------------------+-------------------------+------------------------------------------------------+ | "Hashable" [1] | | "__hash__" | | +--------------------------------+------------------------+-------------------------+------------------------------------------------------+ | "Iterable" [1] [2] | | "__iter__" | | +--------------------------------+------------------------+-------------------------+------------------------------------------------------+ | "Iterator" [1] | "Iterable" | "__next__" | "__iter__" | +--------------------------------+------------------------+-------------------------+------------------------------------------------------+ | "Reversible" [1] | "Iterable" | "__reversed__" | | +--------------------------------+------------------------+-------------------------+------------------------------------------------------+ | "Generator" [1] | "Iterator" | "send", "throw" | "close", "__iter__", "__next__" | +--------------------------------+------------------------+-------------------------+------------------------------------------------------+ | "Sized" [1] | | "__len__" | | +--------------------------------+------------------------+-------------------------+------------------------------------------------------+ | "Callable" [1] | | "__call__" | | +--------------------------------+------------------------+-------------------------+------------------------------------------------------+ | "Collection" [1] | "Sized", "Iterable", | "__contains__", | | | | "Container" | "__iter__", "__len__" | | +--------------------------------+------------------------+-------------------------+------------------------------------------------------+ | "Sequence" | "Reversible", | "__getitem__", | "__contains__", "__iter__", "__reversed__", "index", | | | "Collection" | "__len__" | and "count" | +--------------------------------+------------------------+-------------------------+------------------------------------------------------+ | "MutableSequence" | "Sequence" | "__getitem__", | Inherited "Sequence" methods and "append", "clear", | | | | "__setitem__", | "reverse", "extend", "pop", "remove", and "__iadd__" | | | | "__delitem__", | | | | | "__len__", "insert" | | +--------------------------------+------------------------+-------------------------+------------------------------------------------------+ | "ByteString" | "Sequence" | "__getitem__", | Inherited "Sequence" methods | | | | "__len__" | | +--------------------------------+------------------------+-------------------------+------------------------------------------------------+ | "Set" | "Collection" | "__contains__", | "__le__", "__lt__", "__eq__", "__ne__", "__gt__", | | | | "__iter__", "__len__" | "__ge__", "__and__", "__or__", "__sub__", | | | | | "__rsub__", "__xor__", "__rxor__" and "isdisjoint" | +--------------------------------+------------------------+-------------------------+------------------------------------------------------+ | "MutableSet" | "Set" | "__contains__", | Inherited "Set" methods and "clear", "pop", | | | | "__iter__", "__len__", | "remove", "__ior__", "__iand__", "__ixor__", and | | | | "add", "discard" | "__isub__" | +--------------------------------+------------------------+-------------------------+------------------------------------------------------+ | "Mapping" | "Collection" | "__getitem__", | "__contains__", "keys", "items", "values", "get", | | | | "__iter__", "__len__" | "__eq__", and "__ne__" | +--------------------------------+------------------------+-------------------------+------------------------------------------------------+ | "MutableMapping" | "Mapping" | "__getitem__", | Inherited "Mapping" methods and "pop", "popitem", | | | | "__setitem__", | "clear", "update", and "setdefault" | | | | "__delitem__", | | | | | "__iter__", "__len__" | | +--------------------------------+------------------------+-------------------------+------------------------------------------------------+ | "MappingView" | "Sized" | | "__init__", "__len__" and "__repr__" | +--------------------------------+------------------------+-------------------------+------------------------------------------------------+ | "ItemsView" | "MappingView", "Set" | | "__contains__", "__iter__" | +--------------------------------+------------------------+-------------------------+------------------------------------------------------+ | "KeysView" | "MappingView", "Set" | | "__contains__", "__iter__" | +--------------------------------+------------------------+-------------------------+------------------------------------------------------+ | "ValuesView" | "MappingView", | | "__contains__", "__iter__" | | | "Collection" | | | +--------------------------------+------------------------+-------------------------+------------------------------------------------------+ | "Awaitable" [1] | | "__await__" | | +--------------------------------+------------------------+-------------------------+------------------------------------------------------+ | "Coroutine" [1] | "Awaitable" | "send", "throw" | "close" | +--------------------------------+------------------------+-------------------------+------------------------------------------------------+ | "AsyncIterable" [1] | | "__aiter__" | | +--------------------------------+------------------------+-------------------------+------------------------------------------------------+ | "AsyncIterator" [1] | "AsyncIterable" | "__anext__" | "__aiter__" | +--------------------------------+------------------------+-------------------------+------------------------------------------------------+ | "AsyncGenerator" [1] | "AsyncIterator" | "asend", "athrow" | "aclose", "__aiter__", "__anext__" | +--------------------------------+------------------------+-------------------------+------------------------------------------------------+ | "Buffer" [1] | | "__buffer__" | | +--------------------------------+------------------------+-------------------------+------------------------------------------------------+ -[ Footnotes ]- [1] These ABCs override "__subclasshook__()" to support testing an interface by verifying the required methods are present and have not been set to "None". This only works for simple interfaces. More complex interfaces require registration or direct subclassing. [2] Checking "isinstance(obj, Iterable)" detects classes that are registered as "Iterable" or that have an "__iter__()" method, but it does not detect classes that iterate with the "__getitem__()" method. The only reliable way to determine whether an object is *iterable* is to call "iter(obj)". Collections Abstract Base Classes – Detailed Descriptions ========================================================= class collections.abc.Container ABC for classes that provide the "__contains__()" method. class collections.abc.Hashable ABC for classes that provide the "__hash__()" method. class collections.abc.Sized ABC for classes that provide the "__len__()" method. class collections.abc.Callable ABC for classes that provide the "__call__()" method. See Annotating callable objects for details on how to use "Callable" in type annotations. class collections.abc.Iterable ABC for classes that provide the "__iter__()" method. Checking "isinstance(obj, Iterable)" detects classes that are registered as "Iterable" or that have an "__iter__()" method, but it does not detect classes that iterate with the "__getitem__()" method. The only reliable way to determine whether an object is *iterable* is to call "iter(obj)". class collections.abc.Collection ABC for sized iterable container classes. Added in version 3.6. class collections.abc.Iterator ABC for classes that provide the "__iter__()" and "__next__()" methods. See also the definition of *iterator*. class collections.abc.Reversible ABC for iterable classes that also provide the "__reversed__()" method. Added in version 3.6. class collections.abc.Generator ABC for *generator* classes that implement the protocol defined in **PEP 342** that extends *iterators* with the "send()", "throw()" and "close()" methods. See Annotating generators and coroutines for details on using "Generator" in type annotations. Added in version 3.5. class collections.abc.Sequence class collections.abc.MutableSequence class collections.abc.ByteString ABCs for read-only and mutable *sequences*. Implementation note: Some of the mixin methods, such as "__iter__()", "__reversed__()" and "index()", make repeated calls to the underlying "__getitem__()" method. Consequently, if "__getitem__()" is implemented with constant access speed, the mixin methods will have linear performance; however, if the underlying method is linear (as it would be with a linked list), the mixins will have quadratic performance and will likely need to be overridden. Changed in version 3.5: The index() method added support for *stop* and *start* arguments. Deprecated since version 3.12, will be removed in version 3.14: The "ByteString" ABC has been deprecated. For use in typing, prefer a union, like "bytes | bytearray", or "collections.abc.Buffer". For use as an ABC, prefer "Sequence" or "collections.abc.Buffer". class collections.abc.Set class collections.abc.MutableSet ABCs for read-only and mutable sets. class collections.abc.Mapping class collections.abc.MutableMapping ABCs for read-only and mutable *mappings*. class collections.abc.MappingView class collections.abc.ItemsView class collections.abc.KeysView class collections.abc.ValuesView ABCs for mapping, items, keys, and values *views*. class collections.abc.Awaitable ABC for *awaitable* objects, which can be used in "await" expressions. Custom implementations must provide the "__await__()" method. *Coroutine* objects and instances of the "Coroutine" ABC are all instances of this ABC. Note: In CPython, generator-based coroutines (*generators* decorated with "@types.coroutine") are *awaitables*, even though they do not have an "__await__()" method. Using "isinstance(gencoro, Awaitable)" for them will return "False". Use "inspect.isawaitable()" to detect them. Added in version 3.5. class collections.abc.Coroutine ABC for *coroutine* compatible classes. These implement the following methods, defined in Coroutine Objects: "send()", "throw()", and "close()". Custom implementations must also implement "__await__()". All "Coroutine" instances are also instances of "Awaitable". Note: In CPython, generator-based coroutines (*generators* decorated with "@types.coroutine") are *awaitables*, even though they do not have an "__await__()" method. Using "isinstance(gencoro, Coroutine)" for them will return "False". Use "inspect.isawaitable()" to detect them. See Annotating generators and coroutines for details on using "Coroutine" in type annotations. The variance and order of type parameters correspond to those of "Generator". Added in version 3.5. class collections.abc.AsyncIterable ABC for classes that provide an "__aiter__" method. See also the definition of *asynchronous iterable*. Added in version 3.5. class collections.abc.AsyncIterator ABC for classes that provide "__aiter__" and "__anext__" methods. See also the definition of *asynchronous iterator*. Added in version 3.5. class collections.abc.AsyncGenerator ABC for *asynchronous generator* classes that implement the protocol defined in **PEP 525** and **PEP 492**. See Annotating generators and coroutines for details on using "AsyncGenerator" in type annotations. Added in version 3.6. class collections.abc.Buffer ABC for classes that provide the "__buffer__()" method, implementing the buffer protocol. See **PEP 688**. Added in version 3.12. Examples and Recipes ==================== ABCs allow us to ask classes or instances if they provide particular functionality, for example: size = None if isinstance(myvar, collections.abc.Sized): size = len(myvar) Several of the ABCs are also useful as mixins that make it easier to develop classes supporting container APIs. For example, to write a class supporting the full "Set" API, it is only necessary to supply the three underlying abstract methods: "__contains__()", "__iter__()", and "__len__()". The ABC supplies the remaining methods such as "__and__()" and "isdisjoint()": class ListBasedSet(collections.abc.Set): ''' Alternate set implementation favoring space over speed and not requiring the set elements to be hashable. ''' def __init__(self, iterable): self.elements = lst = [] for value in iterable: if value not in lst: lst.append(value) def __iter__(self): return iter(self.elements) def __contains__(self, value): return value in self.elements def __len__(self): return len(self.elements) s1 = ListBasedSet('abcdef') s2 = ListBasedSet('defghi') overlap = s1 & s2 # The __and__() method is supported automatically Notes on using "Set" and "MutableSet" as a mixin: 1. Since some set operations create new sets, the default mixin methods need a way to create new instances from an *iterable*. The class constructor is assumed to have a signature in the form "ClassName(iterable)". That assumption is factored-out to an internal "classmethod" called "_from_iterable()" which calls "cls(iterable)" to produce a new set. If the "Set" mixin is being used in a class with a different constructor signature, you will need to override "_from_iterable()" with a classmethod or regular method that can construct new instances from an iterable argument. 2. To override the comparisons (presumably for speed, as the semantics are fixed), redefine "__le__()" and "__ge__()", then the other operations will automatically follow suit. 3. The "Set" mixin provides a "_hash()" method to compute a hash value for the set; however, "__hash__()" is not defined because not all sets are *hashable* or immutable. To add set hashability using mixins, inherit from both "Set()" and "Hashable()", then define "__hash__ = Set._hash". See also: * OrderedSet recipe for an example built on "MutableSet". * For more about ABCs, see the "abc" module and **PEP 3119**. "collections" — Container datatypes *********************************** **Source code:** Lib/collections/__init__.py ====================================================================== This module implements specialized container datatypes providing alternatives to Python’s general purpose built-in containers, "dict", "list", "set", and "tuple". +-----------------------+----------------------------------------------------------------------+ | "namedtuple()" | factory function for creating tuple subclasses with named fields | +-----------------------+----------------------------------------------------------------------+ | "deque" | list-like container with fast appends and pops on either end | +-----------------------+----------------------------------------------------------------------+ | "ChainMap" | dict-like class for creating a single view of multiple mappings | +-----------------------+----------------------------------------------------------------------+ | "Counter" | dict subclass for counting *hashable* objects | +-----------------------+----------------------------------------------------------------------+ | "OrderedDict" | dict subclass that remembers the order entries were added | +-----------------------+----------------------------------------------------------------------+ | "defaultdict" | dict subclass that calls a factory function to supply missing values | +-----------------------+----------------------------------------------------------------------+ | "UserDict" | wrapper around dictionary objects for easier dict subclassing | +-----------------------+----------------------------------------------------------------------+ | "UserList" | wrapper around list objects for easier list subclassing | +-----------------------+----------------------------------------------------------------------+ | "UserString" | wrapper around string objects for easier string subclassing | +-----------------------+----------------------------------------------------------------------+ "ChainMap" objects ================== Added in version 3.3. A "ChainMap" class is provided for quickly linking a number of mappings so they can be treated as a single unit. It is often much faster than creating a new dictionary and running multiple "update()" calls. The class can be used to simulate nested scopes and is useful in templating. class collections.ChainMap(*maps) A "ChainMap" groups multiple dicts or other mappings together to create a single, updateable view. If no *maps* are specified, a single empty dictionary is provided so that a new chain always has at least one mapping. The underlying mappings are stored in a list. That list is public and can be accessed or updated using the *maps* attribute. There is no other state. Lookups search the underlying mappings successively until a key is found. In contrast, writes, updates, and deletions only operate on the first mapping. A "ChainMap" incorporates the underlying mappings by reference. So, if one of the underlying mappings gets updated, those changes will be reflected in "ChainMap". All of the usual dictionary methods are supported. In addition, there is a *maps* attribute, a method for creating new subcontexts, and a property for accessing all but the first mapping: maps A user updateable list of mappings. The list is ordered from first-searched to last-searched. It is the only stored state and can be modified to change which mappings are searched. The list should always contain at least one mapping. new_child(m=None, **kwargs) Returns a new "ChainMap" containing a new map followed by all of the maps in the current instance. If "m" is specified, it becomes the new map at the front of the list of mappings; if not specified, an empty dict is used, so that a call to "d.new_child()" is equivalent to: "ChainMap({}, *d.maps)". If any keyword arguments are specified, they update passed map or new empty dict. This method is used for creating subcontexts that can be updated without altering values in any of the parent mappings. Changed in version 3.4: The optional "m" parameter was added. Changed in version 3.10: Keyword arguments support was added. parents Property returning a new "ChainMap" containing all of the maps in the current instance except the first one. This is useful for skipping the first map in the search. Use cases are similar to those for the "nonlocal" keyword used in *nested scopes*. The use cases also parallel those for the built-in "super()" function. A reference to "d.parents" is equivalent to: "ChainMap(*d.maps[1:])". Note, the iteration order of a "ChainMap" is determined by scanning the mappings last to first: >>> baseline = {'music': 'bach', 'art': 'rembrandt'} >>> adjustments = {'art': 'van gogh', 'opera': 'carmen'} >>> list(ChainMap(adjustments, baseline)) ['music', 'art', 'opera'] This gives the same ordering as a series of "dict.update()" calls starting with the last mapping: >>> combined = baseline.copy() >>> combined.update(adjustments) >>> list(combined) ['music', 'art', 'opera'] Changed in version 3.9: Added support for "|" and "|=" operators, specified in **PEP 584**. See also: * The MultiContext class in the Enthought CodeTools package has options to support writing to any mapping in the chain. * Django’s Context class for templating is a read-only chain of mappings. It also features pushing and popping of contexts similar to the "new_child()" method and the "parents" property. * The Nested Contexts recipe has options to control whether writes and other mutations apply only to the first mapping or to any mapping in the chain. * A greatly simplified read-only version of Chainmap. "ChainMap" Examples and Recipes ------------------------------- This section shows various approaches to working with chained maps. Example of simulating Python’s internal lookup chain: import builtins pylookup = ChainMap(locals(), globals(), vars(builtins)) Example of letting user specified command-line arguments take precedence over environment variables which in turn take precedence over default values: import os, argparse defaults = {'color': 'red', 'user': 'guest'} parser = argparse.ArgumentParser() parser.add_argument('-u', '--user') parser.add_argument('-c', '--color') namespace = parser.parse_args() command_line_args = {k: v for k, v in vars(namespace).items() if v is not None} combined = ChainMap(command_line_args, os.environ, defaults) print(combined['color']) print(combined['user']) Example patterns for using the "ChainMap" class to simulate nested contexts: c = ChainMap() # Create root context d = c.new_child() # Create nested child context e = c.new_child() # Child of c, independent from d e.maps[0] # Current context dictionary -- like Python's locals() e.maps[-1] # Root context -- like Python's globals() e.parents # Enclosing context chain -- like Python's nonlocals d['x'] = 1 # Set value in current context d['x'] # Get first key in the chain of contexts del d['x'] # Delete from current context list(d) # All nested values k in d # Check all nested values len(d) # Number of nested values d.items() # All nested items dict(d) # Flatten into a regular dictionary The "ChainMap" class only makes updates (writes and deletions) to the first mapping in the chain while lookups will search the full chain. However, if deep writes and deletions are desired, it is easy to make a subclass that updates keys found deeper in the chain: class DeepChainMap(ChainMap): 'Variant of ChainMap that allows direct updates to inner scopes' def __setitem__(self, key, value): for mapping in self.maps: if key in mapping: mapping[key] = value return self.maps[0][key] = value def __delitem__(self, key): for mapping in self.maps: if key in mapping: del mapping[key] return raise KeyError(key) >>> d = DeepChainMap({'zebra': 'black'}, {'elephant': 'blue'}, {'lion': 'yellow'}) >>> d['lion'] = 'orange' # update an existing key two levels down >>> d['snake'] = 'red' # new keys get added to the topmost dict >>> del d['elephant'] # remove an existing key one level down >>> d # display result DeepChainMap({'zebra': 'black', 'snake': 'red'}, {}, {'lion': 'orange'}) "Counter" objects ================= A counter tool is provided to support convenient and rapid tallies. For example: >>> # Tally occurrences of words in a list >>> cnt = Counter() >>> for word in ['red', 'blue', 'red', 'green', 'blue', 'blue']: ... cnt[word] += 1 ... >>> cnt Counter({'blue': 3, 'red': 2, 'green': 1}) >>> # Find the ten most common words in Hamlet >>> import re >>> words = re.findall(r'\w+', open('hamlet.txt').read().lower()) >>> Counter(words).most_common(10) [('the', 1143), ('and', 966), ('to', 762), ('of', 669), ('i', 631), ('you', 554), ('a', 546), ('my', 514), ('hamlet', 471), ('in', 451)] class collections.Counter([iterable-or-mapping]) A "Counter" is a "dict" subclass for counting *hashable* objects. It is a collection where elements are stored as dictionary keys and their counts are stored as dictionary values. Counts are allowed to be any integer value including zero or negative counts. The "Counter" class is similar to bags or multisets in other languages. Elements are counted from an *iterable* or initialized from another *mapping* (or counter): >>> c = Counter() # a new, empty counter >>> c = Counter('gallahad') # a new counter from an iterable >>> c = Counter({'red': 4, 'blue': 2}) # a new counter from a mapping >>> c = Counter(cats=4, dogs=8) # a new counter from keyword args Counter objects have a dictionary interface except that they return a zero count for missing items instead of raising a "KeyError": >>> c = Counter(['eggs', 'ham']) >>> c['bacon'] # count of a missing element is zero 0 Setting a count to zero does not remove an element from a counter. Use "del" to remove it entirely: >>> c['sausage'] = 0 # counter entry with a zero count >>> del c['sausage'] # del actually removes the entry Added in version 3.1. Changed in version 3.7: As a "dict" subclass, "Counter" inherited the capability to remember insertion order. Math operations on *Counter* objects also preserve order. Results are ordered according to when an element is first encountered in the left operand and then by the order encountered in the right operand. Counter objects support additional methods beyond those available for all dictionaries: elements() Return an iterator over elements repeating each as many times as its count. Elements are returned in the order first encountered. If an element’s count is less than one, "elements()" will ignore it. >>> c = Counter(a=4, b=2, c=0, d=-2) >>> sorted(c.elements()) ['a', 'a', 'a', 'a', 'b', 'b'] most_common([n]) Return a list of the *n* most common elements and their counts from the most common to the least. If *n* is omitted or "None", "most_common()" returns *all* elements in the counter. Elements with equal counts are ordered in the order first encountered: >>> Counter('abracadabra').most_common(3) [('a', 5), ('b', 2), ('r', 2)] subtract([iterable-or-mapping]) Elements are subtracted from an *iterable* or from another *mapping* (or counter). Like "dict.update()" but subtracts counts instead of replacing them. Both inputs and outputs may be zero or negative. >>> c = Counter(a=4, b=2, c=0, d=-2) >>> d = Counter(a=1, b=2, c=3, d=4) >>> c.subtract(d) >>> c Counter({'a': 3, 'b': 0, 'c': -3, 'd': -6}) Added in version 3.2. total() Compute the sum of the counts. >>> c = Counter(a=10, b=5, c=0) >>> c.total() 15 Added in version 3.10. The usual dictionary methods are available for "Counter" objects except for two which work differently for counters. fromkeys(iterable) This class method is not implemented for "Counter" objects. update([iterable-or-mapping]) Elements are counted from an *iterable* or added-in from another *mapping* (or counter). Like "dict.update()" but adds counts instead of replacing them. Also, the *iterable* is expected to be a sequence of elements, not a sequence of "(key, value)" pairs. Counters support rich comparison operators for equality, subset, and superset relationships: "==", "!=", "<", "<=", ">", ">=". All of those tests treat missing elements as having zero counts so that "Counter(a=1) == Counter(a=1, b=0)" returns true. Changed in version 3.10: Rich comparison operations were added. Changed in version 3.10: In equality tests, missing elements are treated as having zero counts. Formerly, "Counter(a=3)" and "Counter(a=3, b=0)" were considered distinct. Common patterns for working with "Counter" objects: c.total() # total of all counts c.clear() # reset all counts list(c) # list unique elements set(c) # convert to a set dict(c) # convert to a regular dictionary c.items() # access the (elem, cnt) pairs Counter(dict(list_of_pairs)) # convert from a list of (elem, cnt) pairs c.most_common()[:-n-1:-1] # n least common elements +c # remove zero and negative counts Several mathematical operations are provided for combining "Counter" objects to produce multisets (counters that have counts greater than zero). Addition and subtraction combine counters by adding or subtracting the counts of corresponding elements. Intersection and union return the minimum and maximum of corresponding counts. Equality and inclusion compare corresponding counts. Each operation can accept inputs with signed counts, but the output will exclude results with counts of zero or less. >>> c = Counter(a=3, b=1) >>> d = Counter(a=1, b=2) >>> c + d # add two counters together: c[x] + d[x] Counter({'a': 4, 'b': 3}) >>> c - d # subtract (keeping only positive counts) Counter({'a': 2}) >>> c & d # intersection: min(c[x], d[x]) Counter({'a': 1, 'b': 1}) >>> c | d # union: max(c[x], d[x]) Counter({'a': 3, 'b': 2}) >>> c == d # equality: c[x] == d[x] False >>> c <= d # inclusion: c[x] <= d[x] False Unary addition and subtraction are shortcuts for adding an empty counter or subtracting from an empty counter. >>> c = Counter(a=2, b=-4) >>> +c Counter({'a': 2}) >>> -c Counter({'b': 4}) Added in version 3.3: Added support for unary plus, unary minus, and in-place multiset operations. Note: Counters were primarily designed to work with positive integers to represent running counts; however, care was taken to not unnecessarily preclude use cases needing other types or negative values. To help with those use cases, this section documents the minimum range and type restrictions. * The "Counter" class itself is a dictionary subclass with no restrictions on its keys and values. The values are intended to be numbers representing counts, but you *could* store anything in the value field. * The "most_common()" method requires only that the values be orderable. * For in-place operations such as "c[key] += 1", the value type need only support addition and subtraction. So fractions, floats, and decimals would work and negative values are supported. The same is also true for "update()" and "subtract()" which allow negative and zero values for both inputs and outputs. * The multiset methods are designed only for use cases with positive values. The inputs may be negative or zero, but only outputs with positive values are created. There are no type restrictions, but the value type needs to support addition, subtraction, and comparison. * The "elements()" method requires integer counts. It ignores zero and negative counts. See also: * Bag class in Smalltalk. * Wikipedia entry for Multisets. * C++ multisets tutorial with examples. * For mathematical operations on multisets and their use cases, see *Knuth, Donald. The Art of Computer Programming Volume II, Section 4.6.3, Exercise 19*. * To enumerate all distinct multisets of a given size over a given set of elements, see "itertools.combinations_with_replacement()": map(Counter, combinations_with_replacement('ABC', 2)) # --> AA AB AC BB BC CC "deque" objects =============== class collections.deque([iterable[, maxlen]]) Returns a new deque object initialized left-to-right (using "append()") with data from *iterable*. If *iterable* is not specified, the new deque is empty. Deques are a generalization of stacks and queues (the name is pronounced “deck” and is short for “double-ended queue”). Deques support thread-safe, memory efficient appends and pops from either side of the deque with approximately the same *O*(1) performance in either direction. Though "list" objects support similar operations, they are optimized for fast fixed-length operations and incur *O*(*n*) memory movement costs for "pop(0)" and "insert(0, v)" operations which change both the size and position of the underlying data representation. If *maxlen* is not specified or is "None", deques may grow to an arbitrary length. Otherwise, the deque is bounded to the specified maximum length. Once a bounded length deque is full, when new items are added, a corresponding number of items are discarded from the opposite end. Bounded length deques provide functionality similar to the "tail" filter in Unix. They are also useful for tracking transactions and other pools of data where only the most recent activity is of interest. Deque objects support the following methods: append(x) Add *x* to the right side of the deque. appendleft(x) Add *x* to the left side of the deque. clear() Remove all elements from the deque leaving it with length 0. copy() Create a shallow copy of the deque. Added in version 3.5. count(x) Count the number of deque elements equal to *x*. Added in version 3.2. extend(iterable) Extend the right side of the deque by appending elements from the iterable argument. extendleft(iterable) Extend the left side of the deque by appending elements from *iterable*. Note, the series of left appends results in reversing the order of elements in the iterable argument. index(x[, start[, stop]]) Return the position of *x* in the deque (at or after index *start* and before index *stop*). Returns the first match or raises "ValueError" if not found. Added in version 3.5. insert(i, x) Insert *x* into the deque at position *i*. If the insertion would cause a bounded deque to grow beyond *maxlen*, an "IndexError" is raised. Added in version 3.5. pop() Remove and return an element from the right side of the deque. If no elements are present, raises an "IndexError". popleft() Remove and return an element from the left side of the deque. If no elements are present, raises an "IndexError". remove(value) Remove the first occurrence of *value*. If not found, raises a "ValueError". reverse() Reverse the elements of the deque in-place and then return "None". Added in version 3.2. rotate(n=1) Rotate the deque *n* steps to the right. If *n* is negative, rotate to the left. When the deque is not empty, rotating one step to the right is equivalent to "d.appendleft(d.pop())", and rotating one step to the left is equivalent to "d.append(d.popleft())". Deque objects also provide one read-only attribute: maxlen Maximum size of a deque or "None" if unbounded. Added in version 3.1. In addition to the above, deques support iteration, pickling, "len(d)", "reversed(d)", "copy.copy(d)", "copy.deepcopy(d)", membership testing with the "in" operator, and subscript references such as "d[0]" to access the first element. Indexed access is *O*(1) at both ends but slows to *O*(*n*) in the middle. For fast random access, use lists instead. Starting in version 3.5, deques support "__add__()", "__mul__()", and "__imul__()". Example: >>> from collections import deque >>> d = deque('ghi') # make a new deque with three items >>> for elem in d: # iterate over the deque's elements ... print(elem.upper()) G H I >>> d.append('j') # add a new entry to the right side >>> d.appendleft('f') # add a new entry to the left side >>> d # show the representation of the deque deque(['f', 'g', 'h', 'i', 'j']) >>> d.pop() # return and remove the rightmost item 'j' >>> d.popleft() # return and remove the leftmost item 'f' >>> list(d) # list the contents of the deque ['g', 'h', 'i'] >>> d[0] # peek at leftmost item 'g' >>> d[-1] # peek at rightmost item 'i' >>> list(reversed(d)) # list the contents of a deque in reverse ['i', 'h', 'g'] >>> 'h' in d # search the deque True >>> d.extend('jkl') # add multiple elements at once >>> d deque(['g', 'h', 'i', 'j', 'k', 'l']) >>> d.rotate(1) # right rotation >>> d deque(['l', 'g', 'h', 'i', 'j', 'k']) >>> d.rotate(-1) # left rotation >>> d deque(['g', 'h', 'i', 'j', 'k', 'l']) >>> deque(reversed(d)) # make a new deque in reverse order deque(['l', 'k', 'j', 'i', 'h', 'g']) >>> d.clear() # empty the deque >>> d.pop() # cannot pop from an empty deque Traceback (most recent call last): File "", line 1, in -toplevel- d.pop() IndexError: pop from an empty deque >>> d.extendleft('abc') # extendleft() reverses the input order >>> d deque(['c', 'b', 'a']) "deque" Recipes --------------- This section shows various approaches to working with deques. Bounded length deques provide functionality similar to the "tail" filter in Unix: def tail(filename, n=10): 'Return the last n lines of a file' with open(filename) as f: return deque(f, n) Another approach to using deques is to maintain a sequence of recently added elements by appending to the right and popping to the left: def moving_average(iterable, n=3): # moving_average([40, 30, 50, 46, 39, 44]) --> 40.0 42.0 45.0 43.0 # https://en.wikipedia.org/wiki/Moving_average it = iter(iterable) d = deque(itertools.islice(it, n-1)) d.appendleft(0) s = sum(d) for elem in it: s += elem - d.popleft() d.append(elem) yield s / n A round-robin scheduler can be implemented with input iterators stored in a "deque". Values are yielded from the active iterator in position zero. If that iterator is exhausted, it can be removed with "popleft()"; otherwise, it can be cycled back to the end with the "rotate()" method: def roundrobin(*iterables): "roundrobin('ABC', 'D', 'EF') --> A D E B F C" iterators = deque(map(iter, iterables)) while iterators: try: while True: yield next(iterators[0]) iterators.rotate(-1) except StopIteration: # Remove an exhausted iterator. iterators.popleft() The "rotate()" method provides a way to implement "deque" slicing and deletion. For example, a pure Python implementation of "del d[n]" relies on the "rotate()" method to position elements to be popped: def delete_nth(d, n): d.rotate(-n) d.popleft() d.rotate(n) To implement "deque" slicing, use a similar approach applying "rotate()" to bring a target element to the left side of the deque. Remove old entries with "popleft()", add new entries with "extend()", and then reverse the rotation. With minor variations on that approach, it is easy to implement Forth style stack manipulations such as "dup", "drop", "swap", "over", "pick", "rot", and "roll". "defaultdict" objects ===================== class collections.defaultdict(default_factory=None, /[, ...]) Return a new dictionary-like object. "defaultdict" is a subclass of the built-in "dict" class. It overrides one method and adds one writable instance variable. The remaining functionality is the same as for the "dict" class and is not documented here. The first argument provides the initial value for the "default_factory" attribute; it defaults to "None". All remaining arguments are treated the same as if they were passed to the "dict" constructor, including keyword arguments. "defaultdict" objects support the following method in addition to the standard "dict" operations: __missing__(key) If the "default_factory" attribute is "None", this raises a "KeyError" exception with the *key* as argument. If "default_factory" is not "None", it is called without arguments to provide a default value for the given *key*, this value is inserted in the dictionary for the *key*, and returned. If calling "default_factory" raises an exception this exception is propagated unchanged. This method is called by the "__getitem__()" method of the "dict" class when the requested key is not found; whatever it returns or raises is then returned or raised by "__getitem__()". Note that "__missing__()" is *not* called for any operations besides "__getitem__()". This means that "get()" will, like normal dictionaries, return "None" as a default rather than using "default_factory". "defaultdict" objects support the following instance variable: default_factory This attribute is used by the "__missing__()" method; it is initialized from the first argument to the constructor, if present, or to "None", if absent. Changed in version 3.9: Added merge ("|") and update ("|=") operators, specified in **PEP 584**. "defaultdict" Examples ---------------------- Using "list" as the "default_factory", it is easy to group a sequence of key-value pairs into a dictionary of lists: >>> s = [('yellow', 1), ('blue', 2), ('yellow', 3), ('blue', 4), ('red', 1)] >>> d = defaultdict(list) >>> for k, v in s: ... d[k].append(v) ... >>> sorted(d.items()) [('blue', [2, 4]), ('red', [1]), ('yellow', [1, 3])] When each key is encountered for the first time, it is not already in the mapping; so an entry is automatically created using the "default_factory" function which returns an empty "list". The "list.append()" operation then attaches the value to the new list. When keys are encountered again, the look-up proceeds normally (returning the list for that key) and the "list.append()" operation adds another value to the list. This technique is simpler and faster than an equivalent technique using "dict.setdefault()": >>> d = {} >>> for k, v in s: ... d.setdefault(k, []).append(v) ... >>> sorted(d.items()) [('blue', [2, 4]), ('red', [1]), ('yellow', [1, 3])] Setting the "default_factory" to "int" makes the "defaultdict" useful for counting (like a bag or multiset in other languages): >>> s = 'mississippi' >>> d = defaultdict(int) >>> for k in s: ... d[k] += 1 ... >>> sorted(d.items()) [('i', 4), ('m', 1), ('p', 2), ('s', 4)] When a letter is first encountered, it is missing from the mapping, so the "default_factory" function calls "int()" to supply a default count of zero. The increment operation then builds up the count for each letter. The function "int()" which always returns zero is just a special case of constant functions. A faster and more flexible way to create constant functions is to use a lambda function which can supply any constant value (not just zero): >>> def constant_factory(value): ... return lambda: value ... >>> d = defaultdict(constant_factory('')) >>> d.update(name='John', action='ran') >>> '%(name)s %(action)s to %(object)s' % d 'John ran to ' Setting the "default_factory" to "set" makes the "defaultdict" useful for building a dictionary of sets: >>> s = [('red', 1), ('blue', 2), ('red', 3), ('blue', 4), ('red', 1), ('blue', 4)] >>> d = defaultdict(set) >>> for k, v in s: ... d[k].add(v) ... >>> sorted(d.items()) [('blue', {2, 4}), ('red', {1, 3})] "namedtuple()" Factory Function for Tuples with Named Fields ============================================================ Named tuples assign meaning to each position in a tuple and allow for more readable, self-documenting code. They can be used wherever regular tuples are used, and they add the ability to access fields by name instead of position index. collections.namedtuple(typename, field_names, *, rename=False, defaults=None, module=None) Returns a new tuple subclass named *typename*. The new subclass is used to create tuple-like objects that have fields accessible by attribute lookup as well as being indexable and iterable. Instances of the subclass also have a helpful docstring (with *typename* and *field_names*) and a helpful "__repr__()" method which lists the tuple contents in a "name=value" format. The *field_names* are a sequence of strings such as "['x', 'y']". Alternatively, *field_names* can be a single string with each fieldname separated by whitespace and/or commas, for example "'x y'" or "'x, y'". Any valid Python identifier may be used for a fieldname except for names starting with an underscore. Valid identifiers consist of letters, digits, and underscores but do not start with a digit or underscore and cannot be a "keyword" such as *class*, *for*, *return*, *global*, *pass*, or *raise*. If *rename* is true, invalid fieldnames are automatically replaced with positional names. For example, "['abc', 'def', 'ghi', 'abc']" is converted to "['abc', '_1', 'ghi', '_3']", eliminating the keyword "def" and the duplicate fieldname "abc". *defaults* can be "None" or an *iterable* of default values. Since fields with a default value must come after any fields without a default, the *defaults* are applied to the rightmost parameters. For example, if the fieldnames are "['x', 'y', 'z']" and the defaults are "(1, 2)", then "x" will be a required argument, "y" will default to "1", and "z" will default to "2". If *module* is defined, the "__module__" attribute of the named tuple is set to that value. Named tuple instances do not have per-instance dictionaries, so they are lightweight and require no more memory than regular tuples. To support pickling, the named tuple class should be assigned to a variable that matches *typename*. Changed in version 3.1: Added support for *rename*. Changed in version 3.6: The *verbose* and *rename* parameters became keyword-only arguments. Changed in version 3.6: Added the *module* parameter. Changed in version 3.7: Removed the *verbose* parameter and the "_source" attribute. Changed in version 3.7: Added the *defaults* parameter and the "_field_defaults" attribute. >>> # Basic example >>> Point = namedtuple('Point', ['x', 'y']) >>> p = Point(11, y=22) # instantiate with positional or keyword arguments >>> p[0] + p[1] # indexable like the plain tuple (11, 22) 33 >>> x, y = p # unpack like a regular tuple >>> x, y (11, 22) >>> p.x + p.y # fields also accessible by name 33 >>> p # readable __repr__ with a name=value style Point(x=11, y=22) Named tuples are especially useful for assigning field names to result tuples returned by the "csv" or "sqlite3" modules: EmployeeRecord = namedtuple('EmployeeRecord', 'name, age, title, department, paygrade') import csv for emp in map(EmployeeRecord._make, csv.reader(open("employees.csv", "rb"))): print(emp.name, emp.title) import sqlite3 conn = sqlite3.connect('/companydata') cursor = conn.cursor() cursor.execute('SELECT name, age, title, department, paygrade FROM employees') for emp in map(EmployeeRecord._make, cursor.fetchall()): print(emp.name, emp.title) In addition to the methods inherited from tuples, named tuples support three additional methods and two attributes. To prevent conflicts with field names, the method and attribute names start with an underscore. classmethod somenamedtuple._make(iterable) Class method that makes a new instance from an existing sequence or iterable. >>> t = [11, 22] >>> Point._make(t) Point(x=11, y=22) somenamedtuple._asdict() Return a new "dict" which maps field names to their corresponding values: >>> p = Point(x=11, y=22) >>> p._asdict() {'x': 11, 'y': 22} Changed in version 3.1: Returns an "OrderedDict" instead of a regular "dict". Changed in version 3.8: Returns a regular "dict" instead of an "OrderedDict". As of Python 3.7, regular dicts are guaranteed to be ordered. If the extra features of "OrderedDict" are required, the suggested remediation is to cast the result to the desired type: "OrderedDict(nt._asdict())". somenamedtuple._replace(**kwargs) Return a new instance of the named tuple replacing specified fields with new values: >>> p = Point(x=11, y=22) >>> p._replace(x=33) Point(x=33, y=22) >>> for partnum, record in inventory.items(): ... inventory[partnum] = record._replace(price=newprices[partnum], timestamp=time.now()) Named tuples are also supported by generic function "copy.replace()". Changed in version 3.13: Raise "TypeError" instead of "ValueError" for invalid keyword arguments. somenamedtuple._fields Tuple of strings listing the field names. Useful for introspection and for creating new named tuple types from existing named tuples. >>> p._fields # view the field names ('x', 'y') >>> Color = namedtuple('Color', 'red green blue') >>> Pixel = namedtuple('Pixel', Point._fields + Color._fields) >>> Pixel(11, 22, 128, 255, 0) Pixel(x=11, y=22, red=128, green=255, blue=0) somenamedtuple._field_defaults Dictionary mapping field names to default values. >>> Account = namedtuple('Account', ['type', 'balance'], defaults=[0]) >>> Account._field_defaults {'balance': 0} >>> Account('premium') Account(type='premium', balance=0) To retrieve a field whose name is stored in a string, use the "getattr()" function: >>> getattr(p, 'x') 11 To convert a dictionary to a named tuple, use the double-star-operator (as described in Unpacking Argument Lists): >>> d = {'x': 11, 'y': 22} >>> Point(**d) Point(x=11, y=22) Since a named tuple is a regular Python class, it is easy to add or change functionality with a subclass. Here is how to add a calculated field and a fixed-width print format: >>> class Point(namedtuple('Point', ['x', 'y'])): ... __slots__ = () ... @property ... def hypot(self): ... return (self.x ** 2 + self.y ** 2) ** 0.5 ... def __str__(self): ... return 'Point: x=%6.3f y=%6.3f hypot=%6.3f' % (self.x, self.y, self.hypot) >>> for p in Point(3, 4), Point(14, 5/7): ... print(p) Point: x= 3.000 y= 4.000 hypot= 5.000 Point: x=14.000 y= 0.714 hypot=14.018 The subclass shown above sets "__slots__" to an empty tuple. This helps keep memory requirements low by preventing the creation of instance dictionaries. Subclassing is not useful for adding new, stored fields. Instead, simply create a new named tuple type from the "_fields" attribute: >>> Point3D = namedtuple('Point3D', Point._fields + ('z',)) Docstrings can be customized by making direct assignments to the "__doc__" fields: >>> Book = namedtuple('Book', ['id', 'title', 'authors']) >>> Book.__doc__ += ': Hardcover book in active collection' >>> Book.id.__doc__ = '13-digit ISBN' >>> Book.title.__doc__ = 'Title of first printing' >>> Book.authors.__doc__ = 'List of authors sorted by last name' Changed in version 3.5: Property docstrings became writeable. See also: * See "typing.NamedTuple" for a way to add type hints for named tuples. It also provides an elegant notation using the "class" keyword: class Component(NamedTuple): part_number: int weight: float description: Optional[str] = None * See "types.SimpleNamespace()" for a mutable namespace based on an underlying dictionary instead of a tuple. * The "dataclasses" module provides a decorator and functions for automatically adding generated special methods to user-defined classes. "OrderedDict" objects ===================== Ordered dictionaries are just like regular dictionaries but have some extra capabilities relating to ordering operations. They have become less important now that the built-in "dict" class gained the ability to remember insertion order (this new behavior became guaranteed in Python 3.7). Some differences from "dict" still remain: * The regular "dict" was designed to be very good at mapping operations. Tracking insertion order was secondary. * The "OrderedDict" was designed to be good at reordering operations. Space efficiency, iteration speed, and the performance of update operations were secondary. * The "OrderedDict" algorithm can handle frequent reordering operations better than "dict". As shown in the recipes below, this makes it suitable for implementing various kinds of LRU caches. * The equality operation for "OrderedDict" checks for matching order. A regular "dict" can emulate the order sensitive equality test with "p == q and all(k1 == k2 for k1, k2 in zip(p, q))". * The "popitem()" method of "OrderedDict" has a different signature. It accepts an optional argument to specify which item is popped. A regular "dict" can emulate OrderedDict’s "od.popitem(last=True)" with "d.popitem()" which is guaranteed to pop the rightmost (last) item. A regular "dict" can emulate OrderedDict’s "od.popitem(last=False)" with "(k := next(iter(d)), d.pop(k))" which will return and remove the leftmost (first) item if it exists. * "OrderedDict" has a "move_to_end()" method to efficiently reposition an element to an endpoint. A regular "dict" can emulate OrderedDict’s "od.move_to_end(k, last=True)" with "d[k] = d.pop(k)" which will move the key and its associated value to the rightmost (last) position. A regular "dict" does not have an efficient equivalent for OrderedDict’s "od.move_to_end(k, last=False)" which moves the key and its associated value to the leftmost (first) position. * Until Python 3.8, "dict" lacked a "__reversed__()" method. class collections.OrderedDict([items]) Return an instance of a "dict" subclass that has methods specialized for rearranging dictionary order. Added in version 3.1. popitem(last=True) The "popitem()" method for ordered dictionaries returns and removes a (key, value) pair. The pairs are returned in LIFO (last-in, first-out) order if *last* is true or FIFO (first-in, first-out) order if false. move_to_end(key, last=True) Move an existing *key* to either end of an ordered dictionary. The item is moved to the right end if *last* is true (the default) or to the beginning if *last* is false. Raises "KeyError" if the *key* does not exist: >>> d = OrderedDict.fromkeys('abcde') >>> d.move_to_end('b') >>> ''.join(d) 'acdeb' >>> d.move_to_end('b', last=False) >>> ''.join(d) 'bacde' Added in version 3.2. In addition to the usual mapping methods, ordered dictionaries also support reverse iteration using "reversed()". Equality tests between "OrderedDict" objects are order-sensitive and are roughly equivalent to "list(od1.items())==list(od2.items())". Equality tests between "OrderedDict" objects and other "Mapping" objects are order-insensitive like regular dictionaries. This allows "OrderedDict" objects to be substituted anywhere a regular dictionary is used. Changed in version 3.5: The items, keys, and values *views* of "OrderedDict" now support reverse iteration using "reversed()". Changed in version 3.6: With the acceptance of **PEP 468**, order is retained for keyword arguments passed to the "OrderedDict" constructor and its "update()" method. Changed in version 3.9: Added merge ("|") and update ("|=") operators, specified in **PEP 584**. "OrderedDict" Examples and Recipes ---------------------------------- It is straightforward to create an ordered dictionary variant that remembers the order the keys were *last* inserted. If a new entry overwrites an existing entry, the original insertion position is changed and moved to the end: class LastUpdatedOrderedDict(OrderedDict): 'Store items in the order the keys were last added' def __setitem__(self, key, value): super().__setitem__(key, value) self.move_to_end(key) An "OrderedDict" would also be useful for implementing variants of "functools.lru_cache()": from collections import OrderedDict from time import time class TimeBoundedLRU: "LRU Cache that invalidates and refreshes old entries." def __init__(self, func, maxsize=128, maxage=30): self.cache = OrderedDict() # { args : (timestamp, result)} self.func = func self.maxsize = maxsize self.maxage = maxage def __call__(self, *args): if args in self.cache: self.cache.move_to_end(args) timestamp, result = self.cache[args] if time() - timestamp <= self.maxage: return result result = self.func(*args) self.cache[args] = time(), result if len(self.cache) > self.maxsize: self.cache.popitem(last=False) return result class MultiHitLRUCache: """ LRU cache that defers caching a result until it has been requested multiple times. To avoid flushing the LRU cache with one-time requests, we don't cache until a request has been made more than once. """ def __init__(self, func, maxsize=128, maxrequests=4096, cache_after=1): self.requests = OrderedDict() # { uncached_key : request_count } self.cache = OrderedDict() # { cached_key : function_result } self.func = func self.maxrequests = maxrequests # max number of uncached requests self.maxsize = maxsize # max number of stored return values self.cache_after = cache_after def __call__(self, *args): if args in self.cache: self.cache.move_to_end(args) return self.cache[args] result = self.func(*args) self.requests[args] = self.requests.get(args, 0) + 1 if self.requests[args] <= self.cache_after: self.requests.move_to_end(args) if len(self.requests) > self.maxrequests: self.requests.popitem(last=False) else: self.requests.pop(args, None) self.cache[args] = result if len(self.cache) > self.maxsize: self.cache.popitem(last=False) return result "UserDict" objects ================== The class, "UserDict" acts as a wrapper around dictionary objects. The need for this class has been partially supplanted by the ability to subclass directly from "dict"; however, this class can be easier to work with because the underlying dictionary is accessible as an attribute. class collections.UserDict([initialdata]) Class that simulates a dictionary. The instance’s contents are kept in a regular dictionary, which is accessible via the "data" attribute of "UserDict" instances. If *initialdata* is provided, "data" is initialized with its contents; note that a reference to *initialdata* will not be kept, allowing it to be used for other purposes. In addition to supporting the methods and operations of mappings, "UserDict" instances provide the following attribute: data A real dictionary used to store the contents of the "UserDict" class. "UserList" objects ================== This class acts as a wrapper around list objects. It is a useful base class for your own list-like classes which can inherit from them and override existing methods or add new ones. In this way, one can add new behaviors to lists. The need for this class has been partially supplanted by the ability to subclass directly from "list"; however, this class can be easier to work with because the underlying list is accessible as an attribute. class collections.UserList([list]) Class that simulates a list. The instance’s contents are kept in a regular list, which is accessible via the "data" attribute of "UserList" instances. The instance’s contents are initially set to a copy of *list*, defaulting to the empty list "[]". *list* can be any iterable, for example a real Python list or a "UserList" object. In addition to supporting the methods and operations of mutable sequences, "UserList" instances provide the following attribute: data A real "list" object used to store the contents of the "UserList" class. **Subclassing requirements:** Subclasses of "UserList" are expected to offer a constructor which can be called with either no arguments or one argument. List operations which return a new sequence attempt to create an instance of the actual implementation class. To do so, it assumes that the constructor can be called with a single parameter, which is a sequence object used as a data source. If a derived class does not wish to comply with this requirement, all of the special methods supported by this class will need to be overridden; please consult the sources for information about the methods which need to be provided in that case. "UserString" objects ==================== The class, "UserString" acts as a wrapper around string objects. The need for this class has been partially supplanted by the ability to subclass directly from "str"; however, this class can be easier to work with because the underlying string is accessible as an attribute. class collections.UserString(seq) Class that simulates a string object. The instance’s content is kept in a regular string object, which is accessible via the "data" attribute of "UserString" instances. The instance’s contents are initially set to a copy of *seq*. The *seq* argument can be any object which can be converted into a string using the built-in "str()" function. In addition to supporting the methods and operations of strings, "UserString" instances provide the following attribute: data A real "str" object used to store the contents of the "UserString" class. Changed in version 3.5: New methods "__getnewargs__", "__rmod__", "casefold", "format_map", "isprintable", and "maketrans". "colorsys" — Conversions between color systems ********************************************** **Source code:** Lib/colorsys.py ====================================================================== The "colorsys" module defines bidirectional conversions of color values between colors expressed in the RGB (Red Green Blue) color space used in computer monitors and three other coordinate systems: YIQ, HLS (Hue Lightness Saturation) and HSV (Hue Saturation Value). Coordinates in all of these color spaces are floating-point values. In the YIQ space, the Y coordinate is between 0 and 1, but the I and Q coordinates can be positive or negative. In all other spaces, the coordinates are all between 0 and 1. See also: More information about color spaces can be found at https://poynton.ca/ColorFAQ.html and https://www.cambridgeincolour.com/tutorials/color-spaces.htm. The "colorsys" module defines the following functions: colorsys.rgb_to_yiq(r, g, b) Convert the color from RGB coordinates to YIQ coordinates. colorsys.yiq_to_rgb(y, i, q) Convert the color from YIQ coordinates to RGB coordinates. colorsys.rgb_to_hls(r, g, b) Convert the color from RGB coordinates to HLS coordinates. colorsys.hls_to_rgb(h, l, s) Convert the color from HLS coordinates to RGB coordinates. colorsys.rgb_to_hsv(r, g, b) Convert the color from RGB coordinates to HSV coordinates. colorsys.hsv_to_rgb(h, s, v) Convert the color from HSV coordinates to RGB coordinates. Example: >>> import colorsys >>> colorsys.rgb_to_hsv(0.2, 0.4, 0.4) (0.5, 0.5, 0.4) >>> colorsys.hsv_to_rgb(0.5, 0.5, 0.4) (0.2, 0.4, 0.4) "compileall" — Byte-compile Python libraries ******************************************** **Source code:** Lib/compileall.py ====================================================================== This module provides some utility functions to support installing Python libraries. These functions compile Python source files in a directory tree. This module can be used to create the cached byte-code files at library installation time, which makes them available for use even by users who don’t have write permission to the library directories. Availability: not WASI. This module does not work or is not available on WebAssembly. See WebAssembly platforms for more information. Command-line use ================ This module can work as a script (using **python -m compileall**) to compile Python sources. directory ... file ... Positional arguments are files to compile or directories that contain source files, traversed recursively. If no argument is given, behave as if the command line was "-l **". -l Do not recurse into subdirectories, only compile source code files directly contained in the named or implied directories. -f Force rebuild even if timestamps are up-to-date. -q Do not print the list of files compiled. If passed once, error messages will still be printed. If passed twice ("-qq"), all output is suppressed. -d destdir Directory prepended to the path to each file being compiled. This will appear in compilation time tracebacks, and is also compiled in to the byte-code file, where it will be used in tracebacks and other messages in cases where the source file does not exist at the time the byte-code file is executed. -s strip_prefix Remove the given prefix from paths recorded in the ".pyc" files. Paths are made relative to the prefix. This option can be used with "-p" but not with "-d". -p prepend_prefix Prepend the given prefix to paths recorded in the ".pyc" files. Use "-p /" to make the paths absolute. This option can be used with "-s" but not with "-d". -x regex regex is used to search the full path to each file considered for compilation, and if the regex produces a match, the file is skipped. -i list Read the file "list" and add each line that it contains to the list of files and directories to compile. If "list" is "-", read lines from "stdin". -b Write the byte-code files to their legacy locations and names, which may overwrite byte-code files created by another version of Python. The default is to write files to their **PEP 3147** locations and names, which allows byte-code files from multiple versions of Python to coexist. -r Control the maximum recursion level for subdirectories. If this is given, then "-l" option will not be taken into account. **python -m compileall -r 0** is equivalent to **python -m compileall -l**. -j N Use *N* workers to compile the files within the given directory. If "0" is used, then the result of "os.process_cpu_count()" will be used. --invalidation-mode [timestamp|checked-hash|unchecked-hash] Control how the generated byte-code files are invalidated at runtime. The "timestamp" value, means that ".pyc" files with the source timestamp and size embedded will be generated. The "checked- hash" and "unchecked-hash" values cause hash-based pycs to be generated. Hash-based pycs embed a hash of the source file contents rather than a timestamp. See Cached bytecode invalidation for more information on how Python validates bytecode cache files at runtime. The default is "timestamp" if the "SOURCE_DATE_EPOCH" environment variable is not set, and "checked-hash" if the "SOURCE_DATE_EPOCH" environment variable is set. -o level Compile with the given optimization level. May be used multiple times to compile for multiple levels at a time (for example, "compileall -o 1 -o 2"). -e dir Ignore symlinks pointing outside the given directory. --hardlink-dupes If two ".pyc" files with different optimization level have the same content, use hard links to consolidate duplicate files. Changed in version 3.2: Added the "-i", "-b" and "-h" options. Changed in version 3.5: Added the "-j", "-r", and "-qq" options. "-q" option was changed to a multilevel value. "-b" will always produce a byte-code file ending in ".pyc", never ".pyo". Changed in version 3.7: Added the "--invalidation-mode" option. Changed in version 3.9: Added the "-s", "-p", "-e" and "--hardlink- dupes" options. Raised the default recursion limit from 10 to "sys.getrecursionlimit()". Added the possibility to specify the "-o" option multiple times. There is no command-line option to control the optimization level used by the "compile()" function, because the Python interpreter itself already provides the option: **python -O -m compileall**. Similarly, the "compile()" function respects the "sys.pycache_prefix" setting. The generated bytecode cache will only be useful if "compile()" is run with the same "sys.pycache_prefix" (if any) that will be used at runtime. Public functions ================ compileall.compile_dir(dir, maxlevels=sys.getrecursionlimit(), ddir=None, force=False, rx=None, quiet=0, legacy=False, optimize=-1, workers=1, invalidation_mode=None, *, stripdir=None, prependdir=None, limit_sl_dest=None, hardlink_dupes=False) Recursively descend the directory tree named by *dir*, compiling all ".py" files along the way. Return a true value if all the files compiled successfully, and a false value otherwise. The *maxlevels* parameter is used to limit the depth of the recursion; it defaults to "sys.getrecursionlimit()". If *ddir* is given, it is prepended to the path to each file being compiled for use in compilation time tracebacks, and is also compiled in to the byte-code file, where it will be used in tracebacks and other messages in cases where the source file does not exist at the time the byte-code file is executed. If *force* is true, modules are re-compiled even if the timestamps are up to date. If *rx* is given, its "search" method is called on the complete path to each file considered for compilation, and if it returns a true value, the file is skipped. This can be used to exclude files matching a regular expression, given as a re.Pattern object. If *quiet* is "False" or "0" (the default), the filenames and other information are printed to standard out. Set to "1", only errors are printed. Set to "2", all output is suppressed. If *legacy* is true, byte-code files are written to their legacy locations and names, which may overwrite byte-code files created by another version of Python. The default is to write files to their **PEP 3147** locations and names, which allows byte-code files from multiple versions of Python to coexist. *optimize* specifies the optimization level for the compiler. It is passed to the built-in "compile()" function. Accepts also a sequence of optimization levels which lead to multiple compilations of one ".py" file in one call. The argument *workers* specifies how many workers are used to compile files in parallel. The default is to not use multiple workers. If the platform can’t use multiple workers and *workers* argument is given, then sequential compilation will be used as a fallback. If *workers* is 0, the number of cores in the system is used. If *workers* is lower than "0", a "ValueError" will be raised. *invalidation_mode* should be a member of the "py_compile.PycInvalidationMode" enum and controls how the generated pycs are invalidated at runtime. The *stripdir*, *prependdir* and *limit_sl_dest* arguments correspond to the "-s", "-p" and "-e" options described above. They may be specified as "str" or "os.PathLike". If *hardlink_dupes* is true and two ".pyc" files with different optimization level have the same content, use hard links to consolidate duplicate files. Changed in version 3.2: Added the *legacy* and *optimize* parameter. Changed in version 3.5: Added the *workers* parameter. Changed in version 3.5: *quiet* parameter was changed to a multilevel value. Changed in version 3.5: The *legacy* parameter only writes out ".pyc" files, not ".pyo" files no matter what the value of *optimize* is. Changed in version 3.6: Accepts a *path-like object*. Changed in version 3.7: The *invalidation_mode* parameter was added. Changed in version 3.7.2: The *invalidation_mode* parameter’s default value is updated to "None". Changed in version 3.8: Setting *workers* to 0 now chooses the optimal number of cores. Changed in version 3.9: Added *stripdir*, *prependdir*, *limit_sl_dest* and *hardlink_dupes* arguments. Default value of *maxlevels* was changed from "10" to "sys.getrecursionlimit()" compileall.compile_file(fullname, ddir=None, force=False, rx=None, quiet=0, legacy=False, optimize=-1, invalidation_mode=None, *, stripdir=None, prependdir=None, limit_sl_dest=None, hardlink_dupes=False) Compile the file with path *fullname*. Return a true value if the file compiled successfully, and a false value otherwise. If *ddir* is given, it is prepended to the path to the file being compiled for use in compilation time tracebacks, and is also compiled in to the byte-code file, where it will be used in tracebacks and other messages in cases where the source file does not exist at the time the byte-code file is executed. If *rx* is given, its "search" method is passed the full path name to the file being compiled, and if it returns a true value, the file is not compiled and "True" is returned. This can be used to exclude files matching a regular expression, given as a re.Pattern object. If *quiet* is "False" or "0" (the default), the filenames and other information are printed to standard out. Set to "1", only errors are printed. Set to "2", all output is suppressed. If *legacy* is true, byte-code files are written to their legacy locations and names, which may overwrite byte-code files created by another version of Python. The default is to write files to their **PEP 3147** locations and names, which allows byte-code files from multiple versions of Python to coexist. *optimize* specifies the optimization level for the compiler. It is passed to the built-in "compile()" function. Accepts also a sequence of optimization levels which lead to multiple compilations of one ".py" file in one call. *invalidation_mode* should be a member of the "py_compile.PycInvalidationMode" enum and controls how the generated pycs are invalidated at runtime. The *stripdir*, *prependdir* and *limit_sl_dest* arguments correspond to the "-s", "-p" and "-e" options described above. They may be specified as "str" or "os.PathLike". If *hardlink_dupes* is true and two ".pyc" files with different optimization level have the same content, use hard links to consolidate duplicate files. Added in version 3.2. Changed in version 3.5: *quiet* parameter was changed to a multilevel value. Changed in version 3.5: The *legacy* parameter only writes out ".pyc" files, not ".pyo" files no matter what the value of *optimize* is. Changed in version 3.7: The *invalidation_mode* parameter was added. Changed in version 3.7.2: The *invalidation_mode* parameter’s default value is updated to "None". Changed in version 3.9: Added *stripdir*, *prependdir*, *limit_sl_dest* and *hardlink_dupes* arguments. compileall.compile_path(skip_curdir=True, maxlevels=0, force=False, quiet=0, legacy=False, optimize=-1, invalidation_mode=None) Byte-compile all the ".py" files found along "sys.path". Return a true value if all the files compiled successfully, and a false value otherwise. If *skip_curdir* is true (the default), the current directory is not included in the search. All other parameters are passed to the "compile_dir()" function. Note that unlike the other compile functions, "maxlevels" defaults to "0". Changed in version 3.2: Added the *legacy* and *optimize* parameter. Changed in version 3.5: *quiet* parameter was changed to a multilevel value. Changed in version 3.5: The *legacy* parameter only writes out ".pyc" files, not ".pyo" files no matter what the value of *optimize* is. Changed in version 3.7: The *invalidation_mode* parameter was added. Changed in version 3.7.2: The *invalidation_mode* parameter’s default value is updated to "None". To force a recompile of all the ".py" files in the "Lib/" subdirectory and all its subdirectories: import compileall compileall.compile_dir('Lib/', force=True) # Perform same compilation, excluding files in .svn directories. import re compileall.compile_dir('Lib/', rx=re.compile(r'[/\\][.]svn'), force=True) # pathlib.Path objects can also be used. import pathlib compileall.compile_dir(pathlib.Path('Lib/'), force=True) See also: Module "py_compile" Byte-compile a single source file. Concurrent Execution ******************** The modules described in this chapter provide support for concurrent execution of code. The appropriate choice of tool will depend on the task to be executed (CPU bound vs IO bound) and preferred style of development (event driven cooperative multitasking vs preemptive multitasking). Here’s an overview: * "threading" — Thread-based parallelism * Introduction * GIL and performance considerations * Reference * Thread-local data * Thread objects * Lock objects * RLock objects * Condition objects * Semaphore objects * "Semaphore" example * Event objects * Timer objects * Barrier objects * Using locks, conditions, and semaphores in the "with" statement * "multiprocessing" — Process-based parallelism * Introduction * The "Process" class * Contexts and start methods * Exchanging objects between processes * Synchronization between processes * Sharing state between processes * Using a pool of workers * Reference * "Process" and exceptions * Pipes and Queues * Miscellaneous * Connection Objects * Synchronization primitives * Shared "ctypes" Objects * The "multiprocessing.sharedctypes" module * Managers * Customized managers * Using a remote manager * Proxy Objects * Cleanup * Process Pools * Listeners and Clients * Address Formats * Authentication keys * Logging * The "multiprocessing.dummy" module * Programming guidelines * All start methods * The *spawn* and *forkserver* start methods * Examples * "multiprocessing.shared_memory" — Shared memory for direct access across processes * The "concurrent" package * "concurrent.futures" — Launching parallel tasks * Executor Objects * ThreadPoolExecutor * ThreadPoolExecutor Example * ProcessPoolExecutor * ProcessPoolExecutor Example * Future Objects * Module Functions * Exception classes * "subprocess" — Subprocess management * Using the "subprocess" Module * Frequently Used Arguments * Popen Constructor * Exceptions * Security Considerations * Popen Objects * Windows Popen Helpers * Windows Constants * Older high-level API * Replacing Older Functions with the "subprocess" Module * Replacing **/bin/sh** shell command substitution * Replacing shell pipeline * Replacing "os.system()" * Replacing the "os.spawn" family * Replacing "os.popen()", "os.popen2()", "os.popen3()" * Replacing functions from the "popen2" module * Legacy Shell Invocation Functions * Notes * Timeout Behavior * Converting an argument sequence to a string on Windows * Disabling use of "vfork()" or "posix_spawn()" * "sched" — Event scheduler * Scheduler Objects * "queue" — A synchronized queue class * Queue Objects * Waiting for task completion * Terminating queues * SimpleQueue Objects * "contextvars" — Context Variables * Context Variables * Manual Context Management * asyncio support The following are support modules for some of the above services: * "_thread" — Low-level threading API "concurrent.futures" — Launching parallel tasks *********************************************** Added in version 3.2. **Source code:** Lib/concurrent/futures/thread.py and Lib/concurrent/futures/process.py ====================================================================== The "concurrent.futures" module provides a high-level interface for asynchronously executing callables. The asynchronous execution can be performed with threads, using "ThreadPoolExecutor", or separate processes, using "ProcessPoolExecutor". Both implement the same interface, which is defined by the abstract "Executor" class. Availability: not WASI. This module does not work or is not available on WebAssembly. See WebAssembly platforms for more information. Executor Objects ================ class concurrent.futures.Executor An abstract class that provides methods to execute calls asynchronously. It should not be used directly, but through its concrete subclasses. submit(fn, /, *args, **kwargs) Schedules the callable, *fn*, to be executed as "fn(*args, **kwargs)" and returns a "Future" object representing the execution of the callable. with ThreadPoolExecutor(max_workers=1) as executor: future = executor.submit(pow, 323, 1235) print(future.result()) map(fn, *iterables, timeout=None, chunksize=1) Similar to "map(fn, *iterables)" except: * the *iterables* are collected immediately rather than lazily; * *fn* is executed asynchronously and several calls to *fn* may be made concurrently. The returned iterator raises a "TimeoutError" if "__next__()" is called and the result isn’t available after *timeout* seconds from the original call to "Executor.map()". *timeout* can be an int or a float. If *timeout* is not specified or "None", there is no limit to the wait time. If a *fn* call raises an exception, then that exception will be raised when its value is retrieved from the iterator. When using "ProcessPoolExecutor", this method chops *iterables* into a number of chunks which it submits to the pool as separate tasks. The (approximate) size of these chunks can be specified by setting *chunksize* to a positive integer. For very long iterables, using a large value for *chunksize* can significantly improve performance compared to the default size of 1. With "ThreadPoolExecutor", *chunksize* has no effect. Changed in version 3.5: Added the *chunksize* argument. shutdown(wait=True, *, cancel_futures=False) Signal the executor that it should free any resources that it is using when the currently pending futures are done executing. Calls to "Executor.submit()" and "Executor.map()" made after shutdown will raise "RuntimeError". If *wait* is "True" then this method will not return until all the pending futures are done executing and the resources associated with the executor have been freed. If *wait* is "False" then this method will return immediately and the resources associated with the executor will be freed when all pending futures are done executing. Regardless of the value of *wait*, the entire Python program will not exit until all pending futures are done executing. If *cancel_futures* is "True", this method will cancel all pending futures that the executor has not started running. Any futures that are completed or running won’t be cancelled, regardless of the value of *cancel_futures*. If both *cancel_futures* and *wait* are "True", all futures that the executor has started running will be completed prior to this method returning. The remaining futures are cancelled. You can avoid having to call this method explicitly if you use the "with" statement, which will shutdown the "Executor" (waiting as if "Executor.shutdown()" were called with *wait* set to "True"): import shutil with ThreadPoolExecutor(max_workers=4) as e: e.submit(shutil.copy, 'src1.txt', 'dest1.txt') e.submit(shutil.copy, 'src2.txt', 'dest2.txt') e.submit(shutil.copy, 'src3.txt', 'dest3.txt') e.submit(shutil.copy, 'src4.txt', 'dest4.txt') Changed in version 3.9: Added *cancel_futures*. ThreadPoolExecutor ================== "ThreadPoolExecutor" is an "Executor" subclass that uses a pool of threads to execute calls asynchronously. Deadlocks can occur when the callable associated with a "Future" waits on the results of another "Future". For example: import time def wait_on_b(): time.sleep(5) print(b.result()) # b will never complete because it is waiting on a. return 5 def wait_on_a(): time.sleep(5) print(a.result()) # a will never complete because it is waiting on b. return 6 executor = ThreadPoolExecutor(max_workers=2) a = executor.submit(wait_on_b) b = executor.submit(wait_on_a) And: def wait_on_future(): f = executor.submit(pow, 5, 2) # This will never complete because there is only one worker thread and # it is executing this function. print(f.result()) executor = ThreadPoolExecutor(max_workers=1) executor.submit(wait_on_future) class concurrent.futures.ThreadPoolExecutor(max_workers=None, thread_name_prefix='', initializer=None, initargs=()) An "Executor" subclass that uses a pool of at most *max_workers* threads to execute calls asynchronously. All threads enqueued to "ThreadPoolExecutor" will be joined before the interpreter can exit. Note that the exit handler which does this is executed *before* any exit handlers added using "atexit". This means exceptions in the main thread must be caught and handled in order to signal threads to exit gracefully. For this reason, it is recommended that "ThreadPoolExecutor" not be used for long- running tasks. *initializer* is an optional callable that is called at the start of each worker thread; *initargs* is a tuple of arguments passed to the initializer. Should *initializer* raise an exception, all currently pending jobs will raise a "BrokenThreadPool", as well as any attempt to submit more jobs to the pool. Changed in version 3.5: If *max_workers* is "None" or not given, it will default to the number of processors on the machine, multiplied by "5", assuming that "ThreadPoolExecutor" is often used to overlap I/O instead of CPU work and the number of workers should be higher than the number of workers for "ProcessPoolExecutor". Changed in version 3.6: Added the *thread_name_prefix* parameter to allow users to control the "threading.Thread" names for worker threads created by the pool for easier debugging. Changed in version 3.7: Added the *initializer* and *initargs* arguments. Changed in version 3.8: Default value of *max_workers* is changed to "min(32, os.cpu_count() + 4)". This default value preserves at least 5 workers for I/O bound tasks. It utilizes at most 32 CPU cores for CPU bound tasks which release the GIL. And it avoids using very large resources implicitly on many-core machines.ThreadPoolExecutor now reuses idle worker threads before starting *max_workers* worker threads too. Changed in version 3.13: Default value of *max_workers* is changed to "min(32, (os.process_cpu_count() or 1) + 4)". ThreadPoolExecutor Example -------------------------- import concurrent.futures import urllib.request URLS = ['http://www.foxnews.com/', 'http://www.cnn.com/', 'http://europe.wsj.com/', 'http://www.bbc.co.uk/', 'http://nonexistent-subdomain.python.org/'] # Retrieve a single page and report the URL and contents def load_url(url, timeout): with urllib.request.urlopen(url, timeout=timeout) as conn: return conn.read() # We can use a with statement to ensure threads are cleaned up promptly with concurrent.futures.ThreadPoolExecutor(max_workers=5) as executor: # Start the load operations and mark each future with its URL future_to_url = {executor.submit(load_url, url, 60): url for url in URLS} for future in concurrent.futures.as_completed(future_to_url): url = future_to_url[future] try: data = future.result() except Exception as exc: print('%r generated an exception: %s' % (url, exc)) else: print('%r page is %d bytes' % (url, len(data))) ProcessPoolExecutor =================== The "ProcessPoolExecutor" class is an "Executor" subclass that uses a pool of processes to execute calls asynchronously. "ProcessPoolExecutor" uses the "multiprocessing" module, which allows it to side-step the *Global Interpreter Lock* but also means that only picklable objects can be executed and returned. The "__main__" module must be importable by worker subprocesses. This means that "ProcessPoolExecutor" will not work in the interactive interpreter. Calling "Executor" or "Future" methods from a callable submitted to a "ProcessPoolExecutor" will result in deadlock. Note that the restrictions on functions and arguments needing to picklable as per "multiprocessing.Process" apply when using "submit()" and "map()" on a "ProcessPoolExecutor". A function defined in a REPL or a lambda should not be expected to work. class concurrent.futures.ProcessPoolExecutor(max_workers=None, mp_context=None, initializer=None, initargs=(), max_tasks_per_child=None) An "Executor" subclass that executes calls asynchronously using a pool of at most *max_workers* processes. If *max_workers* is "None" or not given, it will default to "os.process_cpu_count()". If *max_workers* is less than or equal to "0", then a "ValueError" will be raised. On Windows, *max_workers* must be less than or equal to "61". If it is not then "ValueError" will be raised. If *max_workers* is "None", then the default chosen will be at most "61", even if more processors are available. *mp_context* can be a "multiprocessing" context or "None". It will be used to launch the workers. If *mp_context* is "None" or not given, the default "multiprocessing" context is used. See Contexts and start methods. *initializer* is an optional callable that is called at the start of each worker process; *initargs* is a tuple of arguments passed to the initializer. Should *initializer* raise an exception, all currently pending jobs will raise a "BrokenProcessPool", as well as any attempt to submit more jobs to the pool. *max_tasks_per_child* is an optional argument that specifies the maximum number of tasks a single process can execute before it will exit and be replaced with a fresh worker process. By default *max_tasks_per_child* is "None" which means worker processes will live as long as the pool. When a max is specified, the “spawn” multiprocessing start method will be used by default in absence of a *mp_context* parameter. This feature is incompatible with the “fork” start method. Changed in version 3.3: When one of the worker processes terminates abruptly, a "BrokenProcessPool" error is now raised. Previously, behaviour was undefined but operations on the executor or its futures would often freeze or deadlock. Changed in version 3.7: The *mp_context* argument was added to allow users to control the start_method for worker processes created by the pool.Added the *initializer* and *initargs* arguments. Note: The default "multiprocessing" start method (see Contexts and start methods) will change away from *fork* in Python 3.14. Code that requires *fork* be used for their "ProcessPoolExecutor" should explicitly specify that by passing a "mp_context=multiprocessing.get_context("fork")" parameter. Changed in version 3.11: The *max_tasks_per_child* argument was added to allow users to control the lifetime of workers in the pool. Changed in version 3.12: On POSIX systems, if your application has multiple threads and the "multiprocessing" context uses the ""fork"" start method: The "os.fork()" function called internally to spawn workers may raise a "DeprecationWarning". Pass a *mp_context* configured to use a different start method. See the "os.fork()" documentation for further explanation. Changed in version 3.13: *max_workers* uses "os.process_cpu_count()" by default, instead of "os.cpu_count()". ProcessPoolExecutor Example --------------------------- import concurrent.futures import math PRIMES = [ 112272535095293, 112582705942171, 112272535095293, 115280095190773, 115797848077099, 1099726899285419] def is_prime(n): if n < 2: return False if n == 2: return True if n % 2 == 0: return False sqrt_n = int(math.floor(math.sqrt(n))) for i in range(3, sqrt_n + 1, 2): if n % i == 0: return False return True def main(): with concurrent.futures.ProcessPoolExecutor() as executor: for number, prime in zip(PRIMES, executor.map(is_prime, PRIMES)): print('%d is prime: %s' % (number, prime)) if __name__ == '__main__': main() Future Objects ============== The "Future" class encapsulates the asynchronous execution of a callable. "Future" instances are created by "Executor.submit()". class concurrent.futures.Future Encapsulates the asynchronous execution of a callable. "Future" instances are created by "Executor.submit()" and should not be created directly except for testing. cancel() Attempt to cancel the call. If the call is currently being executed or finished running and cannot be cancelled then the method will return "False", otherwise the call will be cancelled and the method will return "True". cancelled() Return "True" if the call was successfully cancelled. running() Return "True" if the call is currently being executed and cannot be cancelled. done() Return "True" if the call was successfully cancelled or finished running. result(timeout=None) Return the value returned by the call. If the call hasn’t yet completed then this method will wait up to *timeout* seconds. If the call hasn’t completed in *timeout* seconds, then a "TimeoutError" will be raised. *timeout* can be an int or float. If *timeout* is not specified or "None", there is no limit to the wait time. If the future is cancelled before completing then "CancelledError" will be raised. If the call raised an exception, this method will raise the same exception. exception(timeout=None) Return the exception raised by the call. If the call hasn’t yet completed then this method will wait up to *timeout* seconds. If the call hasn’t completed in *timeout* seconds, then a "TimeoutError" will be raised. *timeout* can be an int or float. If *timeout* is not specified or "None", there is no limit to the wait time. If the future is cancelled before completing then "CancelledError" will be raised. If the call completed without raising, "None" is returned. add_done_callback(fn) Attaches the callable *fn* to the future. *fn* will be called, with the future as its only argument, when the future is cancelled or finishes running. Added callables are called in the order that they were added and are always called in a thread belonging to the process that added them. If the callable raises an "Exception" subclass, it will be logged and ignored. If the callable raises a "BaseException" subclass, the behavior is undefined. If the future has already completed or been cancelled, *fn* will be called immediately. The following "Future" methods are meant for use in unit tests and "Executor" implementations. set_running_or_notify_cancel() This method should only be called by "Executor" implementations before executing the work associated with the "Future" and by unit tests. If the method returns "False" then the "Future" was cancelled, i.e. "Future.cancel()" was called and returned "True". Any threads waiting on the "Future" completing (i.e. through "as_completed()" or "wait()") will be woken up. If the method returns "True" then the "Future" was not cancelled and has been put in the running state, i.e. calls to "Future.running()" will return "True". This method can only be called once and cannot be called after "Future.set_result()" or "Future.set_exception()" have been called. set_result(result) Sets the result of the work associated with the "Future" to *result*. This method should only be used by "Executor" implementations and unit tests. Changed in version 3.8: This method raises "concurrent.futures.InvalidStateError" if the "Future" is already done. set_exception(exception) Sets the result of the work associated with the "Future" to the "Exception" *exception*. This method should only be used by "Executor" implementations and unit tests. Changed in version 3.8: This method raises "concurrent.futures.InvalidStateError" if the "Future" is already done. Module Functions ================ concurrent.futures.wait(fs, timeout=None, return_when=ALL_COMPLETED) Wait for the "Future" instances (possibly created by different "Executor" instances) given by *fs* to complete. Duplicate futures given to *fs* are removed and will be returned only once. Returns a named 2-tuple of sets. The first set, named "done", contains the futures that completed (finished or cancelled futures) before the wait completed. The second set, named "not_done", contains the futures that did not complete (pending or running futures). *timeout* can be used to control the maximum number of seconds to wait before returning. *timeout* can be an int or float. If *timeout* is not specified or "None", there is no limit to the wait time. *return_when* indicates when this function should return. It must be one of the following constants: +----------------------------------------------------+----------------------------------------------------+ | Constant | Description | |====================================================|====================================================| | concurrent.futures.FIRST_COMPLETED | The function will return when any future finishes | | | or is cancelled. | +----------------------------------------------------+----------------------------------------------------+ | concurrent.futures.FIRST_EXCEPTION | The function will return when any future finishes | | | by raising an exception. If no future raises an | | | exception then it is equivalent to | | | "ALL_COMPLETED". | +----------------------------------------------------+----------------------------------------------------+ | concurrent.futures.ALL_COMPLETED | The function will return when all futures finish | | | or are cancelled. | +----------------------------------------------------+----------------------------------------------------+ concurrent.futures.as_completed(fs, timeout=None) Returns an iterator over the "Future" instances (possibly created by different "Executor" instances) given by *fs* that yields futures as they complete (finished or cancelled futures). Any futures given by *fs* that are duplicated will be returned once. Any futures that completed before "as_completed()" is called will be yielded first. The returned iterator raises a "TimeoutError" if "__next__()" is called and the result isn’t available after *timeout* seconds from the original call to "as_completed()". *timeout* can be an int or float. If *timeout* is not specified or "None", there is no limit to the wait time. See also: **PEP 3148** – futures - execute computations asynchronously The proposal which described this feature for inclusion in the Python standard library. Exception classes ================= exception concurrent.futures.CancelledError Raised when a future is cancelled. exception concurrent.futures.TimeoutError A deprecated alias of "TimeoutError", raised when a future operation exceeds the given timeout. Changed in version 3.11: This class was made an alias of "TimeoutError". exception concurrent.futures.BrokenExecutor Derived from "RuntimeError", this exception class is raised when an executor is broken for some reason, and cannot be used to submit or execute new tasks. Added in version 3.7. exception concurrent.futures.InvalidStateError Raised when an operation is performed on a future that is not allowed in the current state. Added in version 3.8. exception concurrent.futures.thread.BrokenThreadPool Derived from "BrokenExecutor", this exception class is raised when one of the workers of a "ThreadPoolExecutor" has failed initializing. Added in version 3.7. exception concurrent.futures.process.BrokenProcessPool Derived from "BrokenExecutor" (formerly "RuntimeError"), this exception class is raised when one of the workers of a "ProcessPoolExecutor" has terminated in a non-clean fashion (for example, if it was killed from the outside). Added in version 3.3. The "concurrent" package ************************ Currently, there is only one module in this package: * "concurrent.futures" – Launching parallel tasks "configparser" — Configuration file parser ****************************************** **Source code:** Lib/configparser.py ====================================================================== This module provides the "ConfigParser" class which implements a basic configuration language which provides a structure similar to what’s found in Microsoft Windows INI files. You can use this to write Python programs which can be customized by end users easily. Note: This library does *not* interpret or write the value-type prefixes used in the Windows Registry extended version of INI syntax. See also: Module "tomllib" TOML is a well-specified format for application configuration files. It is specifically designed to be an improved version of INI. Module "shlex" Support for creating Unix shell-like mini-languages which can also be used for application configuration files. Module "json" The "json" module implements a subset of JavaScript syntax which is sometimes used for configuration, but does not support comments. Quick Start =========== Let’s take a very basic configuration file that looks like this: [DEFAULT] ServerAliveInterval = 45 Compression = yes CompressionLevel = 9 ForwardX11 = yes [forge.example] User = hg [topsecret.server.example] Port = 50022 ForwardX11 = no The structure of INI files is described in the following section. Essentially, the file consists of sections, each of which contains keys with values. "configparser" classes can read and write such files. Let’s start by creating the above configuration file programmatically. >>> import configparser >>> config = configparser.ConfigParser() >>> config['DEFAULT'] = {'ServerAliveInterval': '45', ... 'Compression': 'yes', ... 'CompressionLevel': '9'} >>> config['forge.example'] = {} >>> config['forge.example']['User'] = 'hg' >>> config['topsecret.server.example'] = {} >>> topsecret = config['topsecret.server.example'] >>> topsecret['Port'] = '50022' # mutates the parser >>> topsecret['ForwardX11'] = 'no' # same here >>> config['DEFAULT']['ForwardX11'] = 'yes' >>> with open('example.ini', 'w') as configfile: ... config.write(configfile) ... As you can see, we can treat a config parser much like a dictionary. There are differences, outlined later, but the behavior is very close to what you would expect from a dictionary. Now that we have created and saved a configuration file, let’s read it back and explore the data it holds. >>> config = configparser.ConfigParser() >>> config.sections() [] >>> config.read('example.ini') ['example.ini'] >>> config.sections() ['forge.example', 'topsecret.server.example'] >>> 'forge.example' in config True >>> 'python.org' in config False >>> config['forge.example']['User'] 'hg' >>> config['DEFAULT']['Compression'] 'yes' >>> topsecret = config['topsecret.server.example'] >>> topsecret['ForwardX11'] 'no' >>> topsecret['Port'] '50022' >>> for key in config['forge.example']: ... print(key) user compressionlevel serveraliveinterval compression forwardx11 >>> config['forge.example']['ForwardX11'] 'yes' As we can see above, the API is pretty straightforward. The only bit of magic involves the "DEFAULT" section which provides default values for all other sections [1]. Note also that keys in sections are case- insensitive and stored in lowercase [1]. It is possible to read several configurations into a single "ConfigParser", where the most recently added configuration has the highest priority. Any conflicting keys are taken from the more recent configuration while the previously existing keys are retained. The example below reads in an "override.ini" file, which will override any conflicting keys from the "example.ini" file. [DEFAULT] ServerAliveInterval = -1 >>> config_override = configparser.ConfigParser() >>> config_override['DEFAULT'] = {'ServerAliveInterval': '-1'} >>> with open('override.ini', 'w') as configfile: ... config_override.write(configfile) ... >>> config_override = configparser.ConfigParser() >>> config_override.read(['example.ini', 'override.ini']) ['example.ini', 'override.ini'] >>> print(config_override.get('DEFAULT', 'ServerAliveInterval')) -1 This behaviour is equivalent to a "ConfigParser.read()" call with several files passed to the *filenames* parameter. Supported Datatypes =================== Config parsers do not guess datatypes of values in configuration files, always storing them internally as strings. This means that if you need other datatypes, you should convert on your own: >>> int(topsecret['Port']) 50022 >>> float(topsecret['CompressionLevel']) 9.0 Since this task is so common, config parsers provide a range of handy getter methods to handle integers, floats and booleans. The last one is the most interesting because simply passing the value to "bool()" would do no good since "bool('False')" is still "True". This is why config parsers also provide "getboolean()". This method is case- insensitive and recognizes Boolean values from "'yes'"/"'no'", "'on'"/"'off'", "'true'"/"'false'" and "'1'"/"'0'" [1]. For example: >>> topsecret.getboolean('ForwardX11') False >>> config['forge.example'].getboolean('ForwardX11') True >>> config.getboolean('forge.example', 'Compression') True Apart from "getboolean()", config parsers also provide equivalent "getint()" and "getfloat()" methods. You can register your own converters and customize the provided ones. [1] Fallback Values =============== As with a dictionary, you can use a section’s "get()" method to provide fallback values: >>> topsecret.get('Port') '50022' >>> topsecret.get('CompressionLevel') '9' >>> topsecret.get('Cipher') >>> topsecret.get('Cipher', '3des-cbc') '3des-cbc' Please note that default values have precedence over fallback values. For instance, in our example the "'CompressionLevel'" key was specified only in the "'DEFAULT'" section. If we try to get it from the section "'topsecret.server.example'", we will always get the default, even if we specify a fallback: >>> topsecret.get('CompressionLevel', '3') '9' One more thing to be aware of is that the parser-level "get()" method provides a custom, more complex interface, maintained for backwards compatibility. When using this method, a fallback value can be provided via the "fallback" keyword-only argument: >>> config.get('forge.example', 'monster', ... fallback='No such things as monsters') 'No such things as monsters' The same "fallback" argument can be used with the "getint()", "getfloat()" and "getboolean()" methods, for example: >>> 'BatchMode' in topsecret False >>> topsecret.getboolean('BatchMode', fallback=True) True >>> config['DEFAULT']['BatchMode'] = 'no' >>> topsecret.getboolean('BatchMode', fallback=True) False Supported INI File Structure ============================ A configuration file consists of sections, each led by a "[section]" header, followed by key/value entries separated by a specific string ("=" or ":" by default [1]). By default, section names are case sensitive but keys are not [1]. Leading and trailing whitespace is removed from keys and values. Values can be omitted if the parser is configured to allow it [1], in which case the key/value delimiter may also be left out. Values can also span multiple lines, as long as they are indented deeper than the first line of the value. Depending on the parser’s mode, blank lines may be treated as parts of multiline values or ignored. By default, a valid section name can be any string that does not contain ‘\n’. To change this, see "ConfigParser.SECTCRE". The first section name may be omitted if the parser is configured to allow an unnamed top level section with "allow_unnamed_section=True". In this case, the keys/values may be retrieved by "UNNAMED_SECTION" as in "config[UNNAMED_SECTION]". Configuration files may include comments, prefixed by specific characters ("#" and ";" by default [1]). Comments may appear on their own on an otherwise empty line, possibly indented. [1] For example: [Simple Values] key=value spaces in keys=allowed spaces in values=allowed as well spaces around the delimiter = obviously you can also use : to delimit keys from values [All Values Are Strings] values like this: 1000000 or this: 3.14159265359 are they treated as numbers? : no integers, floats and booleans are held as: strings can use the API to get converted values directly: true [Multiline Values] chorus: I'm a lumberjack, and I'm okay I sleep all night and I work all day [No Values] key_without_value empty string value here = [You can use comments] # like this ; or this # By default only in an empty line. # Inline comments can be harmful because they prevent users # from using the delimiting characters as parts of values. # That being said, this can be customized. [Sections Can Be Indented] can_values_be_as_well = True does_that_mean_anything_special = False purpose = formatting for readability multiline_values = are handled just fine as long as they are indented deeper than the first line of a value # Did I mention we can indent comments, too? Unnamed Sections ================ The name of the first section (or unique) may be omitted and values retrieved by the "UNNAMED_SECTION" attribute. >>> config = """ ... option = value ... ... [ Section 2 ] ... another = val ... """ >>> unnamed = configparser.ConfigParser(allow_unnamed_section=True) >>> unnamed.read_string(config) >>> unnamed.get(configparser.UNNAMED_SECTION, 'option') 'value' Interpolation of values ======================= On top of the core functionality, "ConfigParser" supports interpolation. This means values can be preprocessed before returning them from "get()" calls. class configparser.BasicInterpolation The default implementation used by "ConfigParser". It enables values to contain format strings which refer to other values in the same section, or values in the special default section [1]. Additional default values can be provided on initialization. For example: [Paths] home_dir: /Users my_dir: %(home_dir)s/lumberjack my_pictures: %(my_dir)s/Pictures [Escape] # use a %% to escape the % sign (% is the only character that needs to be escaped): gain: 80%% In the example above, "ConfigParser" with *interpolation* set to "BasicInterpolation()" would resolve "%(home_dir)s" to the value of "home_dir" ("/Users" in this case). "%(my_dir)s" in effect would resolve to "/Users/lumberjack". All interpolations are done on demand so keys used in the chain of references do not have to be specified in any specific order in the configuration file. With "interpolation" set to "None", the parser would simply return "%(my_dir)s/Pictures" as the value of "my_pictures" and "%(home_dir)s/lumberjack" as the value of "my_dir". class configparser.ExtendedInterpolation An alternative handler for interpolation which implements a more advanced syntax, used for instance in "zc.buildout". Extended interpolation is using "${section:option}" to denote a value from a foreign section. Interpolation can span multiple levels. For convenience, if the "section:" part is omitted, interpolation defaults to the current section (and possibly the default values from the special section). For example, the configuration specified above with basic interpolation, would look like this with extended interpolation: [Paths] home_dir: /Users my_dir: ${home_dir}/lumberjack my_pictures: ${my_dir}/Pictures [Escape] # use a $$ to escape the $ sign ($ is the only character that needs to be escaped): cost: $$80 Values from other sections can be fetched as well: [Common] home_dir: /Users library_dir: /Library system_dir: /System macports_dir: /opt/local [Frameworks] Python: 3.2 path: ${Common:system_dir}/Library/Frameworks/ [Arthur] nickname: Two Sheds last_name: Jackson my_dir: ${Common:home_dir}/twosheds my_pictures: ${my_dir}/Pictures python_dir: ${Frameworks:path}/Python/Versions/${Frameworks:Python} Mapping Protocol Access ======================= Added in version 3.2. Mapping protocol access is a generic name for functionality that enables using custom objects as if they were dictionaries. In case of "configparser", the mapping interface implementation is using the "parser['section']['option']" notation. "parser['section']" in particular returns a proxy for the section’s data in the parser. This means that the values are not copied but they are taken from the original parser on demand. What’s even more important is that when values are changed on a section proxy, they are actually mutated in the original parser. "configparser" objects behave as close to actual dictionaries as possible. The mapping interface is complete and adheres to the "MutableMapping" ABC. However, there are a few differences that should be taken into account: * By default, all keys in sections are accessible in a case- insensitive manner [1]. E.g. "for option in parser["section"]" yields only "optionxform"’ed option key names. This means lowercased keys by default. At the same time, for a section that holds the key "'a'", both expressions return "True": "a" in parser["section"] "A" in parser["section"] * All sections include "DEFAULTSECT" values as well which means that ".clear()" on a section may not leave the section visibly empty. This is because default values cannot be deleted from the section (because technically they are not there). If they are overridden in the section, deleting causes the default value to be visible again. Trying to delete a default value causes a "KeyError". * "DEFAULTSECT" cannot be removed from the parser: * trying to delete it raises "ValueError", * "parser.clear()" leaves it intact, * "parser.popitem()" never returns it. * "parser.get(section, option, **kwargs)" - the second argument is **not** a fallback value. Note however that the section-level "get()" methods are compatible both with the mapping protocol and the classic configparser API. * "parser.items()" is compatible with the mapping protocol (returns a list of *section_name*, *section_proxy* pairs including the DEFAULTSECT). However, this method can also be invoked with arguments: "parser.items(section, raw, vars)". The latter call returns a list of *option*, *value* pairs for a specified "section", with all interpolations expanded (unless "raw=True" is provided). The mapping protocol is implemented on top of the existing legacy API so that subclasses overriding the original interface still should have mappings working as expected. Customizing Parser Behaviour ============================ There are nearly as many INI format variants as there are applications using it. "configparser" goes a long way to provide support for the largest sensible set of INI styles available. The default functionality is mainly dictated by historical background and it’s very likely that you will want to customize some of the features. The most common way to change the way a specific config parser works is to use the "__init__()" options: * *defaults*, default value: "None" This option accepts a dictionary of key-value pairs which will be initially put in the "DEFAULT" section. This makes for an elegant way to support concise configuration files that don’t specify values which are the same as the documented default. Hint: if you want to specify default values for a specific section, use "read_dict()" before you read the actual file. * *dict_type*, default value: "dict" This option has a major impact on how the mapping protocol will behave and how the written configuration files look. With the standard dictionary, every section is stored in the order they were added to the parser. Same goes for options within sections. An alternative dictionary type can be used for example to sort sections and options on write-back. Please note: there are ways to add a set of key-value pairs in a single operation. When you use a regular dictionary in those operations, the order of the keys will be ordered. For example: >>> parser = configparser.ConfigParser() >>> parser.read_dict({'section1': {'key1': 'value1', ... 'key2': 'value2', ... 'key3': 'value3'}, ... 'section2': {'keyA': 'valueA', ... 'keyB': 'valueB', ... 'keyC': 'valueC'}, ... 'section3': {'foo': 'x', ... 'bar': 'y', ... 'baz': 'z'} ... }) >>> parser.sections() ['section1', 'section2', 'section3'] >>> [option for option in parser['section3']] ['foo', 'bar', 'baz'] * *allow_no_value*, default value: "False" Some configuration files are known to include settings without values, but which otherwise conform to the syntax supported by "configparser". The *allow_no_value* parameter to the constructor can be used to indicate that such values should be accepted: >>> import configparser >>> sample_config = """ ... [mysqld] ... user = mysql ... pid-file = /var/run/mysqld/mysqld.pid ... skip-external-locking ... old_passwords = 1 ... skip-bdb ... # we don't need ACID today ... skip-innodb ... """ >>> config = configparser.ConfigParser(allow_no_value=True) >>> config.read_string(sample_config) >>> # Settings with values are treated as before: >>> config["mysqld"]["user"] 'mysql' >>> # Settings without values provide None: >>> config["mysqld"]["skip-bdb"] >>> # Settings which aren't specified still raise an error: >>> config["mysqld"]["does-not-exist"] Traceback (most recent call last): ... KeyError: 'does-not-exist' * *delimiters*, default value: "('=', ':')" Delimiters are substrings that delimit keys from values within a section. The first occurrence of a delimiting substring on a line is considered a delimiter. This means values (but not keys) can contain the delimiters. See also the *space_around_delimiters* argument to "ConfigParser.write()". * *comment_prefixes*, default value: "('#', ';')" * *inline_comment_prefixes*, default value: "None" Comment prefixes are strings that indicate the start of a valid comment within a config file. *comment_prefixes* are used only on otherwise empty lines (optionally indented) whereas *inline_comment_prefixes* can be used after every valid value (e.g. section names, options and empty lines as well). By default inline comments are disabled and "'#'" and "';'" are used as prefixes for whole line comments. Changed in version 3.2: In previous versions of "configparser" behaviour matched "comment_prefixes=('#',';')" and "inline_comment_prefixes=(';',)". Please note that config parsers don’t support escaping of comment prefixes so using *inline_comment_prefixes* may prevent users from specifying option values with characters used as comment prefixes. When in doubt, avoid setting *inline_comment_prefixes*. In any circumstances, the only way of storing comment prefix characters at the beginning of a line in multiline values is to interpolate the prefix, for example: >>> from configparser import ConfigParser, ExtendedInterpolation >>> parser = ConfigParser(interpolation=ExtendedInterpolation()) >>> # the default BasicInterpolation could be used as well >>> parser.read_string(""" ... [DEFAULT] ... hash = # ... ... [hashes] ... shebang = ... ${hash}!/usr/bin/env python ... ${hash} -*- coding: utf-8 -*- ... ... extensions = ... enabled_extension ... another_extension ... #disabled_by_comment ... yet_another_extension ... ... interpolation not necessary = if # is not at line start ... even in multiline values = line #1 ... line #2 ... line #3 ... """) >>> print(parser['hashes']['shebang']) #!/usr/bin/env python # -*- coding: utf-8 -*- >>> print(parser['hashes']['extensions']) enabled_extension another_extension yet_another_extension >>> print(parser['hashes']['interpolation not necessary']) if # is not at line start >>> print(parser['hashes']['even in multiline values']) line #1 line #2 line #3 * *strict*, default value: "True" When set to "True", the parser will not allow for any section or option duplicates while reading from a single source (using "read_file()", "read_string()" or "read_dict()"). It is recommended to use strict parsers in new applications. Changed in version 3.2: In previous versions of "configparser" behaviour matched "strict=False". * *empty_lines_in_values*, default value: "True" In config parsers, values can span multiple lines as long as they are indented more than the key that holds them. By default parsers also let empty lines to be parts of values. At the same time, keys can be arbitrarily indented themselves to improve readability. In consequence, when configuration files get big and complex, it is easy for the user to lose track of the file structure. Take for instance: [Section] key = multiline value with a gotcha this = is still a part of the multiline value of 'key' This can be especially problematic for the user to see if she’s using a proportional font to edit the file. That is why when your application does not need values with empty lines, you should consider disallowing them. This will make empty lines split keys every time. In the example above, it would produce two keys, "key" and "this". * *default_section*, default value: "configparser.DEFAULTSECT" (that is: ""DEFAULT"") The convention of allowing a special section of default values for other sections or interpolation purposes is a powerful concept of this library, letting users create complex declarative configurations. This section is normally called ""DEFAULT"" but this can be customized to point to any other valid section name. Some typical values include: ""general"" or ""common"". The name provided is used for recognizing default sections when reading from any source and is used when writing configuration back to a file. Its current value can be retrieved using the "parser_instance.default_section" attribute and may be modified at runtime (i.e. to convert files from one format to another). * *interpolation*, default value: "configparser.BasicInterpolation" Interpolation behaviour may be customized by providing a custom handler through the *interpolation* argument. "None" can be used to turn off interpolation completely, "ExtendedInterpolation()" provides a more advanced variant inspired by "zc.buildout". More on the subject in the dedicated documentation section. "RawConfigParser" has a default value of "None". * *converters*, default value: not set Config parsers provide option value getters that perform type conversion. By default "getint()", "getfloat()", and "getboolean()" are implemented. Should other getters be desirable, users may define them in a subclass or pass a dictionary where each key is a name of the converter and each value is a callable implementing said conversion. For instance, passing "{'decimal': decimal.Decimal}" would add "getdecimal()" on both the parser object and all section proxies. In other words, it will be possible to write both "parser_instance.getdecimal('section', 'key', fallback=0)" and "parser_instance['section'].getdecimal('key', 0)". If the converter needs to access the state of the parser, it can be implemented as a method on a config parser subclass. If the name of this method starts with "get", it will be available on all section proxies, in the dict-compatible form (see the "getdecimal()" example above). More advanced customization may be achieved by overriding default values of these parser attributes. The defaults are defined on the classes, so they may be overridden by subclasses or by attribute assignment. ConfigParser.BOOLEAN_STATES By default when using "getboolean()", config parsers consider the following values "True": "'1'", "'yes'", "'true'", "'on'" and the following values "False": "'0'", "'no'", "'false'", "'off'". You can override this by specifying a custom dictionary of strings and their Boolean outcomes. For example: >>> custom = configparser.ConfigParser() >>> custom['section1'] = {'funky': 'nope'} >>> custom['section1'].getboolean('funky') Traceback (most recent call last): ... ValueError: Not a boolean: nope >>> custom.BOOLEAN_STATES = {'sure': True, 'nope': False} >>> custom['section1'].getboolean('funky') False Other typical Boolean pairs include "accept"/"reject" or "enabled"/"disabled". ConfigParser.optionxform(option) This method transforms option names on every read, get, or set operation. The default converts the name to lowercase. This also means that when a configuration file gets written, all keys will be lowercase. Override this method if that’s unsuitable. For example: >>> config = """ ... [Section1] ... Key = Value ... ... [Section2] ... AnotherKey = Value ... """ >>> typical = configparser.ConfigParser() >>> typical.read_string(config) >>> list(typical['Section1'].keys()) ['key'] >>> list(typical['Section2'].keys()) ['anotherkey'] >>> custom = configparser.RawConfigParser() >>> custom.optionxform = lambda option: option >>> custom.read_string(config) >>> list(custom['Section1'].keys()) ['Key'] >>> list(custom['Section2'].keys()) ['AnotherKey'] Note: The optionxform function transforms option names to a canonical form. This should be an idempotent function: if the name is already in canonical form, it should be returned unchanged. ConfigParser.SECTCRE A compiled regular expression used to parse section headers. The default matches "[section]" to the name ""section"". Whitespace is considered part of the section name, thus "[ larch ]" will be read as a section of name "" larch "". Override this attribute if that’s unsuitable. For example: >>> import re >>> config = """ ... [Section 1] ... option = value ... ... [ Section 2 ] ... another = val ... """ >>> typical = configparser.ConfigParser() >>> typical.read_string(config) >>> typical.sections() ['Section 1', ' Section 2 '] >>> custom = configparser.ConfigParser() >>> custom.SECTCRE = re.compile(r"\[ *(?P
[^]]+?) *\]") >>> custom.read_string(config) >>> custom.sections() ['Section 1', 'Section 2'] Note: While ConfigParser objects also use an "OPTCRE" attribute for recognizing option lines, it’s not recommended to override it because that would interfere with constructor options *allow_no_value* and *delimiters*. Legacy API Examples =================== Mainly because of backwards compatibility concerns, "configparser" provides also a legacy API with explicit "get"/"set" methods. While there are valid use cases for the methods outlined below, mapping protocol access is preferred for new projects. The legacy API is at times more advanced, low-level and downright counterintuitive. An example of writing to a configuration file: import configparser config = configparser.RawConfigParser() # Please note that using RawConfigParser's set functions, you can assign # non-string values to keys internally, but will receive an error when # attempting to write to a file or when you get it in non-raw mode. Setting # values using the mapping protocol or ConfigParser's set() does not allow # such assignments to take place. config.add_section('Section1') config.set('Section1', 'an_int', '15') config.set('Section1', 'a_bool', 'true') config.set('Section1', 'a_float', '3.1415') config.set('Section1', 'baz', 'fun') config.set('Section1', 'bar', 'Python') config.set('Section1', 'foo', '%(bar)s is %(baz)s!') # Writing our configuration file to 'example.cfg' with open('example.cfg', 'w') as configfile: config.write(configfile) An example of reading the configuration file again: import configparser config = configparser.RawConfigParser() config.read('example.cfg') # getfloat() raises an exception if the value is not a float # getint() and getboolean() also do this for their respective types a_float = config.getfloat('Section1', 'a_float') an_int = config.getint('Section1', 'an_int') print(a_float + an_int) # Notice that the next output does not interpolate '%(bar)s' or '%(baz)s'. # This is because we are using a RawConfigParser(). if config.getboolean('Section1', 'a_bool'): print(config.get('Section1', 'foo')) To get interpolation, use "ConfigParser": import configparser cfg = configparser.ConfigParser() cfg.read('example.cfg') # Set the optional *raw* argument of get() to True if you wish to disable # interpolation in a single get operation. print(cfg.get('Section1', 'foo', raw=False)) # -> "Python is fun!" print(cfg.get('Section1', 'foo', raw=True)) # -> "%(bar)s is %(baz)s!" # The optional *vars* argument is a dict with members that will take # precedence in interpolation. print(cfg.get('Section1', 'foo', vars={'bar': 'Documentation', 'baz': 'evil'})) # The optional *fallback* argument can be used to provide a fallback value print(cfg.get('Section1', 'foo')) # -> "Python is fun!" print(cfg.get('Section1', 'foo', fallback='Monty is not.')) # -> "Python is fun!" print(cfg.get('Section1', 'monster', fallback='No such things as monsters.')) # -> "No such things as monsters." # A bare print(cfg.get('Section1', 'monster')) would raise NoOptionError # but we can also use: print(cfg.get('Section1', 'monster', fallback=None)) # -> None Default values are available in both types of ConfigParsers. They are used in interpolation if an option used is not defined elsewhere. import configparser # New instance with 'bar' and 'baz' defaulting to 'Life' and 'hard' each config = configparser.ConfigParser({'bar': 'Life', 'baz': 'hard'}) config.read('example.cfg') print(config.get('Section1', 'foo')) # -> "Python is fun!" config.remove_option('Section1', 'bar') config.remove_option('Section1', 'baz') print(config.get('Section1', 'foo')) # -> "Life is hard!" ConfigParser Objects ==================== class configparser.ConfigParser(defaults=None, dict_type=dict, allow_no_value=False, *, delimiters=('=', ':'), comment_prefixes=('#', ';'), inline_comment_prefixes=None, strict=True, empty_lines_in_values=True, default_section=configparser.DEFAULTSECT, interpolation=BasicInterpolation(), converters={}, allow_unnamed_section=False) The main configuration parser. When *defaults* is given, it is initialized into the dictionary of intrinsic defaults. When *dict_type* is given, it will be used to create the dictionary objects for the list of sections, for the options within a section, and for the default values. When *delimiters* is given, it is used as the set of substrings that divide keys from values. When *comment_prefixes* is given, it will be used as the set of substrings that prefix comments in otherwise empty lines. Comments can be indented. When *inline_comment_prefixes* is given, it will be used as the set of substrings that prefix comments in non-empty lines. When *strict* is "True" (the default), the parser won’t allow for any section or option duplicates while reading from a single source (file, string or dictionary), raising "DuplicateSectionError" or "DuplicateOptionError". When *empty_lines_in_values* is "False" (default: "True"), each empty line marks the end of an option. Otherwise, internal empty lines of a multiline option are kept as part of the value. When *allow_no_value* is "True" (default: "False"), options without values are accepted; the value held for these is "None" and they are serialized without the trailing delimiter. When *default_section* is given, it specifies the name for the special section holding default values for other sections and interpolation purposes (normally named ""DEFAULT""). This value can be retrieved and changed at runtime using the "default_section" instance attribute. This won’t re-evaluate an already parsed config file, but will be used when writing parsed settings to a new config file. Interpolation behaviour may be customized by providing a custom handler through the *interpolation* argument. "None" can be used to turn off interpolation completely, "ExtendedInterpolation()" provides a more advanced variant inspired by "zc.buildout". More on the subject in the dedicated documentation section. All option names used in interpolation will be passed through the "optionxform()" method just like any other option name reference. For example, using the default implementation of "optionxform()" (which converts option names to lower case), the values "foo %(bar)s" and "foo %(BAR)s" are equivalent. When *converters* is given, it should be a dictionary where each key represents the name of a type converter and each value is a callable implementing the conversion from string to the desired datatype. Every converter gets its own corresponding "get*()" method on the parser object and section proxies. When *allow_unnamed_section* is "True" (default: "False"), the first section name can be omitted. See the “Unnamed Sections” section. It is possible to read several configurations into a single "ConfigParser", where the most recently added configuration has the highest priority. Any conflicting keys are taken from the more recent configuration while the previously existing keys are retained. The example below reads in an "override.ini" file, which will override any conflicting keys from the "example.ini" file. [DEFAULT] ServerAliveInterval = -1 >>> config_override = configparser.ConfigParser() >>> config_override['DEFAULT'] = {'ServerAliveInterval': '-1'} >>> with open('override.ini', 'w') as configfile: ... config_override.write(configfile) ... >>> config_override = configparser.ConfigParser() >>> config_override.read(['example.ini', 'override.ini']) ['example.ini', 'override.ini'] >>> print(config_override.get('DEFAULT', 'ServerAliveInterval')) -1 Changed in version 3.1: The default *dict_type* is "collections.OrderedDict". Changed in version 3.2: *allow_no_value*, *delimiters*, *comment_prefixes*, *strict*, *empty_lines_in_values*, *default_section* and *interpolation* were added. Changed in version 3.5: The *converters* argument was added. Changed in version 3.7: The *defaults* argument is read with "read_dict()", providing consistent behavior across the parser: non-string keys and values are implicitly converted to strings. Changed in version 3.8: The default *dict_type* is "dict", since it now preserves insertion order. Changed in version 3.13: Raise a "MultilineContinuationError" when *allow_no_value* is "True", and a key without a value is continued with an indented line. Changed in version 3.13: The *allow_unnamed_section* argument was added. defaults() Return a dictionary containing the instance-wide defaults. sections() Return a list of the sections available; the *default section* is not included in the list. add_section(section) Add a section named *section* to the instance. If a section by the given name already exists, "DuplicateSectionError" is raised. If the *default section* name is passed, "ValueError" is raised. The name of the section must be a string; if not, "TypeError" is raised. Changed in version 3.2: Non-string section names raise "TypeError". has_section(section) Indicates whether the named *section* is present in the configuration. The *default section* is not acknowledged. options(section) Return a list of options available in the specified *section*. has_option(section, option) If the given *section* exists, and contains the given *option*, return "True"; otherwise return "False". If the specified *section* is "None" or an empty string, DEFAULT is assumed. read(filenames, encoding=None) Attempt to read and parse an iterable of filenames, returning a list of filenames which were successfully parsed. If *filenames* is a string, a "bytes" object or a *path-like object*, it is treated as a single filename. If a file named in *filenames* cannot be opened, that file will be ignored. This is designed so that you can specify an iterable of potential configuration file locations (for example, the current directory, the user’s home directory, and some system-wide directory), and all existing configuration files in the iterable will be read. If none of the named files exist, the "ConfigParser" instance will contain an empty dataset. An application which requires initial values to be loaded from a file should load the required file or files using "read_file()" before calling "read()" for any optional files: import configparser, os config = configparser.ConfigParser() config.read_file(open('defaults.cfg')) config.read(['site.cfg', os.path.expanduser('~/.myapp.cfg')], encoding='cp1250') Changed in version 3.2: Added the *encoding* parameter. Previously, all files were read using the default encoding for "open()". Changed in version 3.6.1: The *filenames* parameter accepts a *path-like object*. Changed in version 3.7: The *filenames* parameter accepts a "bytes" object. read_file(f, source=None) Read and parse configuration data from *f* which must be an iterable yielding Unicode strings (for example files opened in text mode). Optional argument *source* specifies the name of the file being read. If not given and *f* has a "name" attribute, that is used for *source*; the default is "''". Added in version 3.2: Replaces "readfp()". read_string(string, source='') Parse configuration data from a string. Optional argument *source* specifies a context-specific name of the string passed. If not given, "''" is used. This should commonly be a filesystem path or a URL. Added in version 3.2. read_dict(dictionary, source='') Load configuration from any object that provides a dict-like "items()" method. Keys are section names, values are dictionaries with keys and values that should be present in the section. If the used dictionary type preserves order, sections and their keys will be added in order. Values are automatically converted to strings. Optional argument *source* specifies a context-specific name of the dictionary passed. If not given, "" is used. This method can be used to copy state between parsers. Added in version 3.2. get(section, option, *, raw=False, vars=None[, fallback]) Get an *option* value for the named *section*. If *vars* is provided, it must be a dictionary. The *option* is looked up in *vars* (if provided), *section*, and in *DEFAULTSECT* in that order. If the key is not found and *fallback* is provided, it is used as a fallback value. "None" can be provided as a *fallback* value. All the "'%'" interpolations are expanded in the return values, unless the *raw* argument is true. Values for interpolation keys are looked up in the same manner as the option. Changed in version 3.2: Arguments *raw*, *vars* and *fallback* are keyword only to protect users from trying to use the third argument as the *fallback* fallback (especially when using the mapping protocol). getint(section, option, *, raw=False, vars=None[, fallback]) A convenience method which coerces the *option* in the specified *section* to an integer. See "get()" for explanation of *raw*, *vars* and *fallback*. getfloat(section, option, *, raw=False, vars=None[, fallback]) A convenience method which coerces the *option* in the specified *section* to a floating-point number. See "get()" for explanation of *raw*, *vars* and *fallback*. getboolean(section, option, *, raw=False, vars=None[, fallback]) A convenience method which coerces the *option* in the specified *section* to a Boolean value. Note that the accepted values for the option are "'1'", "'yes'", "'true'", and "'on'", which cause this method to return "True", and "'0'", "'no'", "'false'", and "'off'", which cause it to return "False". These string values are checked in a case-insensitive manner. Any other value will cause it to raise "ValueError". See "get()" for explanation of *raw*, *vars* and *fallback*. items(raw=False, vars=None) items(section, raw=False, vars=None) When *section* is not given, return a list of *section_name*, *section_proxy* pairs, including DEFAULTSECT. Otherwise, return a list of *name*, *value* pairs for the options in the given *section*. Optional arguments have the same meaning as for the "get()" method. Changed in version 3.8: Items present in *vars* no longer appear in the result. The previous behaviour mixed actual parser options with variables provided for interpolation. set(section, option, value) If the given section exists, set the given option to the specified value; otherwise raise "NoSectionError". *option* and *value* must be strings; if not, "TypeError" is raised. write(fileobject, space_around_delimiters=True) Write a representation of the configuration to the specified *file object*, which must be opened in text mode (accepting strings). This representation can be parsed by a future "read()" call. If *space_around_delimiters* is true, delimiters between keys and values are surrounded by spaces. Note: Comments in the original configuration file are not preserved when writing the configuration back. What is considered a comment, depends on the given values for *comment_prefix* and *inline_comment_prefix*. remove_option(section, option) Remove the specified *option* from the specified *section*. If the section does not exist, raise "NoSectionError". If the option existed to be removed, return "True"; otherwise return "False". remove_section(section) Remove the specified *section* from the configuration. If the section in fact existed, return "True". Otherwise return "False". optionxform(option) Transforms the option name *option* as found in an input file or as passed in by client code to the form that should be used in the internal structures. The default implementation returns a lower-case version of *option*; subclasses may override this or client code can set an attribute of this name on instances to affect this behavior. You don’t need to subclass the parser to use this method, you can also set it on an instance, to a function that takes a string argument and returns a string. Setting it to "str", for example, would make option names case sensitive: cfgparser = ConfigParser() cfgparser.optionxform = str Note that when reading configuration files, whitespace around the option names is stripped before "optionxform()" is called. configparser.UNNAMED_SECTION A special object representing a section name used to reference the unnamed section (see Unnamed Sections). configparser.MAX_INTERPOLATION_DEPTH The maximum depth for recursive interpolation for "get()" when the *raw* parameter is false. This is relevant only when the default *interpolation* is used. RawConfigParser Objects ======================= class configparser.RawConfigParser(defaults=None, dict_type=dict, allow_no_value=False, *, delimiters=('=', ':'), comment_prefixes=('#', ';'), inline_comment_prefixes=None, strict=True, empty_lines_in_values=True, default_section=configparser.DEFAULTSECT, interpolation=BasicInterpolation(), converters={}, allow_unnamed_section=False) Legacy variant of the "ConfigParser". It has interpolation disabled by default and allows for non-string section names, option names, and values via its unsafe "add_section" and "set" methods, as well as the legacy "defaults=" keyword argument handling. Changed in version 3.2: *allow_no_value*, *delimiters*, *comment_prefixes*, *strict*, *empty_lines_in_values*, *default_section* and *interpolation* were added. Changed in version 3.5: The *converters* argument was added. Changed in version 3.8: The default *dict_type* is "dict", since it now preserves insertion order. Changed in version 3.13: The *allow_unnamed_section* argument was added. Note: Consider using "ConfigParser" instead which checks types of the values to be stored internally. If you don’t want interpolation, you can use "ConfigParser(interpolation=None)". add_section(section) Add a section named *section* to the instance. If a section by the given name already exists, "DuplicateSectionError" is raised. If the *default section* name is passed, "ValueError" is raised. Type of *section* is not checked which lets users create non- string named sections. This behaviour is unsupported and may cause internal errors. set(section, option, value) If the given section exists, set the given option to the specified value; otherwise raise "NoSectionError". While it is possible to use "RawConfigParser" (or "ConfigParser" with *raw* parameters set to true) for *internal* storage of non-string values, full functionality (including interpolation and output to files) can only be achieved using string values. This method lets users assign non-string values to keys internally. This behaviour is unsupported and will cause errors when attempting to write to a file or get it in non-raw mode. **Use the mapping protocol API** which does not allow such assignments to take place. Exceptions ========== exception configparser.Error Base class for all other "configparser" exceptions. exception configparser.NoSectionError Exception raised when a specified section is not found. exception configparser.DuplicateSectionError Exception raised if "add_section()" is called with the name of a section that is already present or in strict parsers when a section if found more than once in a single input file, string or dictionary. Changed in version 3.2: Added the optional *source* and *lineno* attributes and parameters to "__init__()". exception configparser.DuplicateOptionError Exception raised by strict parsers if a single option appears twice during reading from a single file, string or dictionary. This catches misspellings and case sensitivity-related errors, e.g. a dictionary may have two keys representing the same case-insensitive configuration key. exception configparser.NoOptionError Exception raised when a specified option is not found in the specified section. exception configparser.InterpolationError Base class for exceptions raised when problems occur performing string interpolation. exception configparser.InterpolationDepthError Exception raised when string interpolation cannot be completed because the number of iterations exceeds "MAX_INTERPOLATION_DEPTH". Subclass of "InterpolationError". exception configparser.InterpolationMissingOptionError Exception raised when an option referenced from a value does not exist. Subclass of "InterpolationError". exception configparser.InterpolationSyntaxError Exception raised when the source text into which substitutions are made does not conform to the required syntax. Subclass of "InterpolationError". exception configparser.MissingSectionHeaderError Exception raised when attempting to parse a file which has no section headers. exception configparser.ParsingError Exception raised when errors occur attempting to parse a file. Changed in version 3.12: The "filename" attribute and "__init__()" constructor argument were removed. They have been available using the name "source" since 3.2. exception configparser.MultilineContinuationError Exception raised when a key without a corresponding value is continued with an indented line. Added in version 3.13. -[ Footnotes ]- [1] Config parsers allow for heavy customization. If you are interested in changing the behaviour outlined by the footnote reference, consult the Customizing Parser Behaviour section. Built-in Constants ****************** A small number of constants live in the built-in namespace. They are: False The false value of the "bool" type. Assignments to "False" are illegal and raise a "SyntaxError". True The true value of the "bool" type. Assignments to "True" are illegal and raise a "SyntaxError". None An object frequently used to represent the absence of a value, as when default arguments are not passed to a function. Assignments to "None" are illegal and raise a "SyntaxError". "None" is the sole instance of the "NoneType" type. NotImplemented A special value which should be returned by the binary special methods (e.g. "__eq__()", "__lt__()", "__add__()", "__rsub__()", etc.) to indicate that the operation is not implemented with respect to the other type; may be returned by the in-place binary special methods (e.g. "__imul__()", "__iand__()", etc.) for the same purpose. It should not be evaluated in a boolean context. "NotImplemented" is the sole instance of the "types.NotImplementedType" type. Note: When a binary (or in-place) method returns "NotImplemented" the interpreter will try the reflected operation on the other type (or some other fallback, depending on the operator). If all attempts return "NotImplemented", the interpreter will raise an appropriate exception. Incorrectly returning "NotImplemented" will result in a misleading error message or the "NotImplemented" value being returned to Python code.See Implementing the arithmetic operations for examples. Caution: "NotImplemented" and "NotImplementedError" are not interchangeable. This constant should only be used as described above; see "NotImplementedError" for details on correct usage of the exception. Changed in version 3.9: Evaluating "NotImplemented" in a boolean context is deprecated. While it currently evaluates as true, it will emit a "DeprecationWarning". It will raise a "TypeError" in a future version of Python. Ellipsis The same as the ellipsis literal “"..."”. Special value used mostly in conjunction with extended slicing syntax for user-defined container data types. "Ellipsis" is the sole instance of the "types.EllipsisType" type. __debug__ This constant is true if Python was not started with an "-O" option. See also the "assert" statement. Note: The names "None", "False", "True" and "__debug__" cannot be reassigned (assignments to them, even as an attribute name, raise "SyntaxError"), so they can be considered “true” constants. Constants added by the "site" module ==================================== The "site" module (which is imported automatically during startup, except if the "-S" command-line option is given) adds several constants to the built-in namespace. They are useful for the interactive interpreter shell and should not be used in programs. quit(code=None) exit(code=None) Objects that when printed, print a message like “Use quit() or Ctrl-D (i.e. EOF) to exit”, and when called, raise "SystemExit" with the specified exit code. help Object that when printed, prints the message “Type help() for interactive help, or help(object) for help about object.”, and when called, acts as described "elsewhere". copyright credits Objects that when printed or called, print the text of copyright or credits, respectively. license Object that when printed, prints the message “Type license() to see the full license text”, and when called, displays the full license text in a pager-like fashion (one screen at a time). "contextlib" — Utilities for "with"-statement contexts ****************************************************** **Source code:** Lib/contextlib.py ====================================================================== This module provides utilities for common tasks involving the "with" statement. For more information see also Context Manager Types and With Statement Context Managers. Utilities ========= Functions and classes provided: class contextlib.AbstractContextManager An *abstract base class* for classes that implement "object.__enter__()" and "object.__exit__()". A default implementation for "object.__enter__()" is provided which returns "self" while "object.__exit__()" is an abstract method which by default returns "None". See also the definition of Context Manager Types. Added in version 3.6. class contextlib.AbstractAsyncContextManager An *abstract base class* for classes that implement "object.__aenter__()" and "object.__aexit__()". A default implementation for "object.__aenter__()" is provided which returns "self" while "object.__aexit__()" is an abstract method which by default returns "None". See also the definition of Asynchronous Context Managers. Added in version 3.7. @contextlib.contextmanager This function is a *decorator* that can be used to define a factory function for "with" statement context managers, without needing to create a class or separate "__enter__()" and "__exit__()" methods. While many objects natively support use in with statements, sometimes a resource needs to be managed that isn’t a context manager in its own right, and doesn’t implement a "close()" method for use with "contextlib.closing". An abstract example would be the following to ensure correct resource management: from contextlib import contextmanager @contextmanager def managed_resource(*args, **kwds): # Code to acquire resource, e.g.: resource = acquire_resource(*args, **kwds) try: yield resource finally: # Code to release resource, e.g.: release_resource(resource) The function can then be used like this: >>> with managed_resource(timeout=3600) as resource: ... # Resource is released at the end of this block, ... # even if code in the block raises an exception The function being decorated must return a *generator*-iterator when called. This iterator must yield exactly one value, which will be bound to the targets in the "with" statement’s "as" clause, if any. At the point where the generator yields, the block nested in the "with" statement is executed. The generator is then resumed after the block is exited. If an unhandled exception occurs in the block, it is reraised inside the generator at the point where the yield occurred. Thus, you can use a "try"…"except"…"finally" statement to trap the error (if any), or ensure that some cleanup takes place. If an exception is trapped merely in order to log it or to perform some action (rather than to suppress it entirely), the generator must reraise that exception. Otherwise the generator context manager will indicate to the "with" statement that the exception has been handled, and execution will resume with the statement immediately following the "with" statement. "contextmanager()" uses "ContextDecorator" so the context managers it creates can be used as decorators as well as in "with" statements. When used as a decorator, a new generator instance is implicitly created on each function call (this allows the otherwise “one-shot” context managers created by "contextmanager()" to meet the requirement that context managers support multiple invocations in order to be used as decorators). Changed in version 3.2: Use of "ContextDecorator". @contextlib.asynccontextmanager Similar to "contextmanager()", but creates an asynchronous context manager. This function is a *decorator* that can be used to define a factory function for "async with" statement asynchronous context managers, without needing to create a class or separate "__aenter__()" and "__aexit__()" methods. It must be applied to an *asynchronous generator* function. A simple example: from contextlib import asynccontextmanager @asynccontextmanager async def get_connection(): conn = await acquire_db_connection() try: yield conn finally: await release_db_connection(conn) async def get_all_users(): async with get_connection() as conn: return conn.query('SELECT ...') Added in version 3.7. Context managers defined with "asynccontextmanager()" can be used either as decorators or with "async with" statements: import time from contextlib import asynccontextmanager @asynccontextmanager async def timeit(): now = time.monotonic() try: yield finally: print(f'it took {time.monotonic() - now}s to run') @timeit() async def main(): # ... async code ... When used as a decorator, a new generator instance is implicitly created on each function call. This allows the otherwise “one-shot” context managers created by "asynccontextmanager()" to meet the requirement that context managers support multiple invocations in order to be used as decorators. Changed in version 3.10: Async context managers created with "asynccontextmanager()" can be used as decorators. contextlib.closing(thing) Return a context manager that closes *thing* upon completion of the block. This is basically equivalent to: from contextlib import contextmanager @contextmanager def closing(thing): try: yield thing finally: thing.close() And lets you write code like this: from contextlib import closing from urllib.request import urlopen with closing(urlopen('https://www.python.org')) as page: for line in page: print(line) without needing to explicitly close "page". Even if an error occurs, "page.close()" will be called when the "with" block is exited. Note: Most types managing resources support the *context manager* protocol, which closes *thing* on leaving the "with" statement. As such, "closing()" is most useful for third party types that don’t support context managers. This example is purely for illustration purposes, as "urlopen()" would normally be used in a context manager. contextlib.aclosing(thing) Return an async context manager that calls the "aclose()" method of *thing* upon completion of the block. This is basically equivalent to: from contextlib import asynccontextmanager @asynccontextmanager async def aclosing(thing): try: yield thing finally: await thing.aclose() Significantly, "aclosing()" supports deterministic cleanup of async generators when they happen to exit early by "break" or an exception. For example: from contextlib import aclosing async with aclosing(my_generator()) as values: async for value in values: if value == 42: break This pattern ensures that the generator’s async exit code is executed in the same context as its iterations (so that exceptions and context variables work as expected, and the exit code isn’t run after the lifetime of some task it depends on). Added in version 3.10. contextlib.nullcontext(enter_result=None) Return a context manager that returns *enter_result* from "__enter__", but otherwise does nothing. It is intended to be used as a stand-in for an optional context manager, for example: def myfunction(arg, ignore_exceptions=False): if ignore_exceptions: # Use suppress to ignore all exceptions. cm = contextlib.suppress(Exception) else: # Do not ignore any exceptions, cm has no effect. cm = contextlib.nullcontext() with cm: # Do something An example using *enter_result*: def process_file(file_or_path): if isinstance(file_or_path, str): # If string, open file cm = open(file_or_path) else: # Caller is responsible for closing file cm = nullcontext(file_or_path) with cm as file: # Perform processing on the file It can also be used as a stand-in for asynchronous context managers: async def send_http(session=None): if not session: # If no http session, create it with aiohttp cm = aiohttp.ClientSession() else: # Caller is responsible for closing the session cm = nullcontext(session) async with cm as session: # Send http requests with session Added in version 3.7. Changed in version 3.10: *asynchronous context manager* support was added. contextlib.suppress(*exceptions) Return a context manager that suppresses any of the specified exceptions if they occur in the body of a "with" statement and then resumes execution with the first statement following the end of the "with" statement. As with any other mechanism that completely suppresses exceptions, this context manager should be used only to cover very specific errors where silently continuing with program execution is known to be the right thing to do. For example: from contextlib import suppress with suppress(FileNotFoundError): os.remove('somefile.tmp') with suppress(FileNotFoundError): os.remove('someotherfile.tmp') This code is equivalent to: try: os.remove('somefile.tmp') except FileNotFoundError: pass try: os.remove('someotherfile.tmp') except FileNotFoundError: pass This context manager is reentrant. If the code within the "with" block raises a "BaseExceptionGroup", suppressed exceptions are removed from the group. Any exceptions of the group which are not suppressed are re-raised in a new group which is created using the original group’s "derive()" method. Added in version 3.4. Changed in version 3.12: "suppress" now supports suppressing exceptions raised as part of a "BaseExceptionGroup". contextlib.redirect_stdout(new_target) Context manager for temporarily redirecting "sys.stdout" to another file or file-like object. This tool adds flexibility to existing functions or classes whose output is hardwired to stdout. For example, the output of "help()" normally is sent to *sys.stdout*. You can capture that output in a string by redirecting the output to an "io.StringIO" object. The replacement stream is returned from the "__enter__" method and so is available as the target of the "with" statement: with redirect_stdout(io.StringIO()) as f: help(pow) s = f.getvalue() To send the output of "help()" to a file on disk, redirect the output to a regular file: with open('help.txt', 'w') as f: with redirect_stdout(f): help(pow) To send the output of "help()" to *sys.stderr*: with redirect_stdout(sys.stderr): help(pow) Note that the global side effect on "sys.stdout" means that this context manager is not suitable for use in library code and most threaded applications. It also has no effect on the output of subprocesses. However, it is still a useful approach for many utility scripts. This context manager is reentrant. Added in version 3.4. contextlib.redirect_stderr(new_target) Similar to "redirect_stdout()" but redirecting "sys.stderr" to another file or file-like object. This context manager is reentrant. Added in version 3.5. contextlib.chdir(path) Non parallel-safe context manager to change the current working directory. As this changes a global state, the working directory, it is not suitable for use in most threaded or async contexts. It is also not suitable for most non-linear code execution, like generators, where the program execution is temporarily relinquished – unless explicitly desired, you should not yield when this context manager is active. This is a simple wrapper around "chdir()", it changes the current working directory upon entering and restores the old one on exit. This context manager is reentrant. Added in version 3.11. class contextlib.ContextDecorator A base class that enables a context manager to also be used as a decorator. Context managers inheriting from "ContextDecorator" have to implement "__enter__" and "__exit__" as normal. "__exit__" retains its optional exception handling even when used as a decorator. "ContextDecorator" is used by "contextmanager()", so you get this functionality automatically. Example of "ContextDecorator": from contextlib import ContextDecorator class mycontext(ContextDecorator): def __enter__(self): print('Starting') return self def __exit__(self, *exc): print('Finishing') return False The class can then be used like this: >>> @mycontext() ... def function(): ... print('The bit in the middle') ... >>> function() Starting The bit in the middle Finishing >>> with mycontext(): ... print('The bit in the middle') ... Starting The bit in the middle Finishing This change is just syntactic sugar for any construct of the following form: def f(): with cm(): # Do stuff "ContextDecorator" lets you instead write: @cm() def f(): # Do stuff It makes it clear that the "cm" applies to the whole function, rather than just a piece of it (and saving an indentation level is nice, too). Existing context managers that already have a base class can be extended by using "ContextDecorator" as a mixin class: from contextlib import ContextDecorator class mycontext(ContextBaseClass, ContextDecorator): def __enter__(self): return self def __exit__(self, *exc): return False Note: As the decorated function must be able to be called multiple times, the underlying context manager must support use in multiple "with" statements. If this is not the case, then the original construct with the explicit "with" statement inside the function should be used. Added in version 3.2. class contextlib.AsyncContextDecorator Similar to "ContextDecorator" but only for asynchronous functions. Example of "AsyncContextDecorator": from asyncio import run from contextlib import AsyncContextDecorator class mycontext(AsyncContextDecorator): async def __aenter__(self): print('Starting') return self async def __aexit__(self, *exc): print('Finishing') return False The class can then be used like this: >>> @mycontext() ... async def function(): ... print('The bit in the middle') ... >>> run(function()) Starting The bit in the middle Finishing >>> async def function(): ... async with mycontext(): ... print('The bit in the middle') ... >>> run(function()) Starting The bit in the middle Finishing Added in version 3.10. class contextlib.ExitStack A context manager that is designed to make it easy to programmatically combine other context managers and cleanup functions, especially those that are optional or otherwise driven by input data. For example, a set of files may easily be handled in a single with statement as follows: with ExitStack() as stack: files = [stack.enter_context(open(fname)) for fname in filenames] # All opened files will automatically be closed at the end of # the with statement, even if attempts to open files later # in the list raise an exception The "__enter__()" method returns the "ExitStack" instance, and performs no additional operations. Each instance maintains a stack of registered callbacks that are called in reverse order when the instance is closed (either explicitly or implicitly at the end of a "with" statement). Note that callbacks are *not* invoked implicitly when the context stack instance is garbage collected. This stack model is used so that context managers that acquire their resources in their "__init__" method (such as file objects) can be handled correctly. Since registered callbacks are invoked in the reverse order of registration, this ends up behaving as if multiple nested "with" statements had been used with the registered set of callbacks. This even extends to exception handling - if an inner callback suppresses or replaces an exception, then outer callbacks will be passed arguments based on that updated state. This is a relatively low level API that takes care of the details of correctly unwinding the stack of exit callbacks. It provides a suitable foundation for higher level context managers that manipulate the exit stack in application specific ways. Added in version 3.3. enter_context(cm) Enters a new context manager and adds its "__exit__()" method to the callback stack. The return value is the result of the context manager’s own "__enter__()" method. These context managers may suppress exceptions just as they normally would if used directly as part of a "with" statement. Changed in version 3.11: Raises "TypeError" instead of "AttributeError" if *cm* is not a context manager. push(exit) Adds a context manager’s "__exit__()" method to the callback stack. As "__enter__" is *not* invoked, this method can be used to cover part of an "__enter__()" implementation with a context manager’s own "__exit__()" method. If passed an object that is not a context manager, this method assumes it is a callback with the same signature as a context manager’s "__exit__()" method and adds it directly to the callback stack. By returning true values, these callbacks can suppress exceptions the same way context manager "__exit__()" methods can. The passed in object is returned from the function, allowing this method to be used as a function decorator. callback(callback, /, *args, **kwds) Accepts an arbitrary callback function and arguments and adds it to the callback stack. Unlike the other methods, callbacks added this way cannot suppress exceptions (as they are never passed the exception details). The passed in callback is returned from the function, allowing this method to be used as a function decorator. pop_all() Transfers the callback stack to a fresh "ExitStack" instance and returns it. No callbacks are invoked by this operation - instead, they will now be invoked when the new stack is closed (either explicitly or implicitly at the end of a "with" statement). For example, a group of files can be opened as an “all or nothing” operation as follows: with ExitStack() as stack: files = [stack.enter_context(open(fname)) for fname in filenames] # Hold onto the close method, but don't call it yet. close_files = stack.pop_all().close # If opening any file fails, all previously opened files will be # closed automatically. If all files are opened successfully, # they will remain open even after the with statement ends. # close_files() can then be invoked explicitly to close them all. close() Immediately unwinds the callback stack, invoking callbacks in the reverse order of registration. For any context managers and exit callbacks registered, the arguments passed in will indicate that no exception occurred. class contextlib.AsyncExitStack An asynchronous context manager, similar to "ExitStack", that supports combining both synchronous and asynchronous context managers, as well as having coroutines for cleanup logic. The "close()" method is not implemented; "aclose()" must be used instead. async enter_async_context(cm) Similar to "ExitStack.enter_context()" but expects an asynchronous context manager. Changed in version 3.11: Raises "TypeError" instead of "AttributeError" if *cm* is not an asynchronous context manager. push_async_exit(exit) Similar to "ExitStack.push()" but expects either an asynchronous context manager or a coroutine function. push_async_callback(callback, /, *args, **kwds) Similar to "ExitStack.callback()" but expects a coroutine function. async aclose() Similar to "ExitStack.close()" but properly handles awaitables. Continuing the example for "asynccontextmanager()": async with AsyncExitStack() as stack: connections = [await stack.enter_async_context(get_connection()) for i in range(5)] # All opened connections will automatically be released at the end of # the async with statement, even if attempts to open a connection # later in the list raise an exception. Added in version 3.7. Examples and Recipes ==================== This section describes some examples and recipes for making effective use of the tools provided by "contextlib". Supporting a variable number of context managers ------------------------------------------------ The primary use case for "ExitStack" is the one given in the class documentation: supporting a variable number of context managers and other cleanup operations in a single "with" statement. The variability may come from the number of context managers needed being driven by user input (such as opening a user specified collection of files), or from some of the context managers being optional: with ExitStack() as stack: for resource in resources: stack.enter_context(resource) if need_special_resource(): special = acquire_special_resource() stack.callback(release_special_resource, special) # Perform operations that use the acquired resources As shown, "ExitStack" also makes it quite easy to use "with" statements to manage arbitrary resources that don’t natively support the context management protocol. Catching exceptions from "__enter__" methods -------------------------------------------- It is occasionally desirable to catch exceptions from an "__enter__" method implementation, *without* inadvertently catching exceptions from the "with" statement body or the context manager’s "__exit__" method. By using "ExitStack" the steps in the context management protocol can be separated slightly in order to allow this: stack = ExitStack() try: x = stack.enter_context(cm) except Exception: # handle __enter__ exception else: with stack: # Handle normal case Actually needing to do this is likely to indicate that the underlying API should be providing a direct resource management interface for use with "try"/"except"/"finally" statements, but not all APIs are well designed in that regard. When a context manager is the only resource management API provided, then "ExitStack" can make it easier to handle various situations that can’t be handled directly in a "with" statement. Cleaning up in an "__enter__" implementation -------------------------------------------- As noted in the documentation of "ExitStack.push()", this method can be useful in cleaning up an already allocated resource if later steps in the "__enter__()" implementation fail. Here’s an example of doing this for a context manager that accepts resource acquisition and release functions, along with an optional validation function, and maps them to the context management protocol: from contextlib import contextmanager, AbstractContextManager, ExitStack class ResourceManager(AbstractContextManager): def __init__(self, acquire_resource, release_resource, check_resource_ok=None): self.acquire_resource = acquire_resource self.release_resource = release_resource if check_resource_ok is None: def check_resource_ok(resource): return True self.check_resource_ok = check_resource_ok @contextmanager def _cleanup_on_error(self): with ExitStack() as stack: stack.push(self) yield # The validation check passed and didn't raise an exception # Accordingly, we want to keep the resource, and pass it # back to our caller stack.pop_all() def __enter__(self): resource = self.acquire_resource() with self._cleanup_on_error(): if not self.check_resource_ok(resource): msg = "Failed validation for {!r}" raise RuntimeError(msg.format(resource)) return resource def __exit__(self, *exc_details): # We don't need to duplicate any of our resource release logic self.release_resource() Replacing any use of "try-finally" and flag variables ----------------------------------------------------- A pattern you will sometimes see is a "try-finally" statement with a flag variable to indicate whether or not the body of the "finally" clause should be executed. In its simplest form (that can’t already be handled just by using an "except" clause instead), it looks something like this: cleanup_needed = True try: result = perform_operation() if result: cleanup_needed = False finally: if cleanup_needed: cleanup_resources() As with any "try" statement based code, this can cause problems for development and review, because the setup code and the cleanup code can end up being separated by arbitrarily long sections of code. "ExitStack" makes it possible to instead register a callback for execution at the end of a "with" statement, and then later decide to skip executing that callback: from contextlib import ExitStack with ExitStack() as stack: stack.callback(cleanup_resources) result = perform_operation() if result: stack.pop_all() This allows the intended cleanup behaviour to be made explicit up front, rather than requiring a separate flag variable. If a particular application uses this pattern a lot, it can be simplified even further by means of a small helper class: from contextlib import ExitStack class Callback(ExitStack): def __init__(self, callback, /, *args, **kwds): super().__init__() self.callback(callback, *args, **kwds) def cancel(self): self.pop_all() with Callback(cleanup_resources) as cb: result = perform_operation() if result: cb.cancel() If the resource cleanup isn’t already neatly bundled into a standalone function, then it is still possible to use the decorator form of "ExitStack.callback()" to declare the resource cleanup in advance: from contextlib import ExitStack with ExitStack() as stack: @stack.callback def cleanup_resources(): ... result = perform_operation() if result: stack.pop_all() Due to the way the decorator protocol works, a callback function declared this way cannot take any parameters. Instead, any resources to be released must be accessed as closure variables. Using a context manager as a function decorator ----------------------------------------------- "ContextDecorator" makes it possible to use a context manager in both an ordinary "with" statement and also as a function decorator. For example, it is sometimes useful to wrap functions or groups of statements with a logger that can track the time of entry and time of exit. Rather than writing both a function decorator and a context manager for the task, inheriting from "ContextDecorator" provides both capabilities in a single definition: from contextlib import ContextDecorator import logging logging.basicConfig(level=logging.INFO) class track_entry_and_exit(ContextDecorator): def __init__(self, name): self.name = name def __enter__(self): logging.info('Entering: %s', self.name) def __exit__(self, exc_type, exc, exc_tb): logging.info('Exiting: %s', self.name) Instances of this class can be used as both a context manager: with track_entry_and_exit('widget loader'): print('Some time consuming activity goes here') load_widget() And also as a function decorator: @track_entry_and_exit('widget loader') def activity(): print('Some time consuming activity goes here') load_widget() Note that there is one additional limitation when using context managers as function decorators: there’s no way to access the return value of "__enter__()". If that value is needed, then it is still necessary to use an explicit "with" statement. See also: **PEP 343** - The “with” statement The specification, background, and examples for the Python "with" statement. Single use, reusable and reentrant context managers =================================================== Most context managers are written in a way that means they can only be used effectively in a "with" statement once. These single use context managers must be created afresh each time they’re used - attempting to use them a second time will trigger an exception or otherwise not work correctly. This common limitation means that it is generally advisable to create context managers directly in the header of the "with" statement where they are used (as shown in all of the usage examples above). Files are an example of effectively single use context managers, since the first "with" statement will close the file, preventing any further IO operations using that file object. Context managers created using "contextmanager()" are also single use context managers, and will complain about the underlying generator failing to yield if an attempt is made to use them a second time: >>> from contextlib import contextmanager >>> @contextmanager ... def singleuse(): ... print("Before") ... yield ... print("After") ... >>> cm = singleuse() >>> with cm: ... pass ... Before After >>> with cm: ... pass ... Traceback (most recent call last): ... RuntimeError: generator didn't yield Reentrant context managers -------------------------- More sophisticated context managers may be “reentrant”. These context managers can not only be used in multiple "with" statements, but may also be used *inside* a "with" statement that is already using the same context manager. "threading.RLock" is an example of a reentrant context manager, as are "suppress()", "redirect_stdout()", and "chdir()". Here’s a very simple example of reentrant use: >>> from contextlib import redirect_stdout >>> from io import StringIO >>> stream = StringIO() >>> write_to_stream = redirect_stdout(stream) >>> with write_to_stream: ... print("This is written to the stream rather than stdout") ... with write_to_stream: ... print("This is also written to the stream") ... >>> print("This is written directly to stdout") This is written directly to stdout >>> print(stream.getvalue()) This is written to the stream rather than stdout This is also written to the stream Real world examples of reentrancy are more likely to involve multiple functions calling each other and hence be far more complicated than this example. Note also that being reentrant is *not* the same thing as being thread safe. "redirect_stdout()", for example, is definitely not thread safe, as it makes a global modification to the system state by binding "sys.stdout" to a different stream. Reusable context managers ------------------------- Distinct from both single use and reentrant context managers are “reusable” context managers (or, to be completely explicit, “reusable, but not reentrant” context managers, since reentrant context managers are also reusable). These context managers support being used multiple times, but will fail (or otherwise not work correctly) if the specific context manager instance has already been used in a containing with statement. "threading.Lock" is an example of a reusable, but not reentrant, context manager (for a reentrant lock, it is necessary to use "threading.RLock" instead). Another example of a reusable, but not reentrant, context manager is "ExitStack", as it invokes *all* currently registered callbacks when leaving any with statement, regardless of where those callbacks were added: >>> from contextlib import ExitStack >>> stack = ExitStack() >>> with stack: ... stack.callback(print, "Callback: from first context") ... print("Leaving first context") ... Leaving first context Callback: from first context >>> with stack: ... stack.callback(print, "Callback: from second context") ... print("Leaving second context") ... Leaving second context Callback: from second context >>> with stack: ... stack.callback(print, "Callback: from outer context") ... with stack: ... stack.callback(print, "Callback: from inner context") ... print("Leaving inner context") ... print("Leaving outer context") ... Leaving inner context Callback: from inner context Callback: from outer context Leaving outer context As the output from the example shows, reusing a single stack object across multiple with statements works correctly, but attempting to nest them will cause the stack to be cleared at the end of the innermost with statement, which is unlikely to be desirable behaviour. Using separate "ExitStack" instances instead of reusing a single instance avoids that problem: >>> from contextlib import ExitStack >>> with ExitStack() as outer_stack: ... outer_stack.callback(print, "Callback: from outer context") ... with ExitStack() as inner_stack: ... inner_stack.callback(print, "Callback: from inner context") ... print("Leaving inner context") ... print("Leaving outer context") ... Leaving inner context Callback: from inner context Leaving outer context Callback: from outer context "contextvars" — Context Variables ********************************* ====================================================================== This module provides APIs to manage, store, and access context-local state. The "ContextVar" class is used to declare and work with *Context Variables*. The "copy_context()" function and the "Context" class should be used to manage the current context in asynchronous frameworks. Context managers that have state should use Context Variables instead of "threading.local()" to prevent their state from bleeding to other code unexpectedly, when used in concurrent code. See also **PEP 567** for additional details. Added in version 3.7. Context Variables ================= class contextvars.ContextVar(name[, *, default]) This class is used to declare a new Context Variable, e.g.: var: ContextVar[int] = ContextVar('var', default=42) The required *name* parameter is used for introspection and debug purposes. The optional keyword-only *default* parameter is returned by "ContextVar.get()" when no value for the variable is found in the current context. **Important:** Context Variables should be created at the top module level and never in closures. "Context" objects hold strong references to context variables which prevents context variables from being properly garbage collected. name The name of the variable. This is a read-only property. Added in version 3.7.1. get([default]) Return a value for the context variable for the current context. If there is no value for the variable in the current context, the method will: * return the value of the *default* argument of the method, if provided; or * return the default value for the context variable, if it was created with one; or * raise a "LookupError". set(value) Call to set a new value for the context variable in the current context. The required *value* argument is the new value for the context variable. Returns a "Token" object that can be used to restore the variable to its previous value via the "ContextVar.reset()" method. reset(token) Reset the context variable to the value it had before the "ContextVar.set()" that created the *token* was used. For example: var = ContextVar('var') token = var.set('new value') # code that uses 'var'; var.get() returns 'new value'. var.reset(token) # After the reset call the var has no value again, so # var.get() would raise a LookupError. class contextvars.Token *Token* objects are returned by the "ContextVar.set()" method. They can be passed to the "ContextVar.reset()" method to revert the value of the variable to what it was before the corresponding *set*. var A read-only property. Points to the "ContextVar" object that created the token. old_value A read-only property. Set to the value the variable had before the "ContextVar.set()" method call that created the token. It points to "Token.MISSING" if the variable was not set before the call. MISSING A marker object used by "Token.old_value". Manual Context Management ========================= contextvars.copy_context() Returns a copy of the current "Context" object. The following snippet gets a copy of the current context and prints all variables and their values that are set in it: ctx: Context = copy_context() print(list(ctx.items())) The function has an *O*(1) complexity, i.e. works equally fast for contexts with a few context variables and for contexts that have a lot of them. class contextvars.Context A mapping of "ContextVars" to their values. "Context()" creates an empty context with no values in it. To get a copy of the current context use the "copy_context()" function. Each thread has its own effective stack of "Context" objects. The *current context* is the "Context" object at the top of the current thread’s stack. All "Context" objects in the stacks are considered to be *entered*. *Entering* a context, which can be done by calling its "run()" method, makes the context the current context by pushing it onto the top of the current thread’s context stack. *Exiting* from the current context, which can be done by returning from the callback passed to the "run()" method, restores the current context to what it was before the context was entered by popping the context off the top of the context stack. Since each thread has its own context stack, "ContextVar" objects behave in a similar fashion to "threading.local()" when values are assigned in different threads. Attempting to enter an already entered context, including contexts entered in other threads, raises a "RuntimeError". After exiting a context, it can later be re-entered (from any thread). Any changes to "ContextVar" values via the "ContextVar.set()" method are recorded in the current context. The "ContextVar.get()" method returns the value associated with the current context. Exiting a context effectively reverts any changes made to context variables while the context was entered (if needed, the values can be restored by re-entering the context). Context implements the "collections.abc.Mapping" interface. run(callable, *args, **kwargs) Enters the Context, executes "callable(*args, **kwargs)", then exits the Context. Returns *callable*’s return value, or propagates an exception if one occurred. Example: import contextvars var = contextvars.ContextVar('var') var.set('spam') print(var.get()) # 'spam' ctx = contextvars.copy_context() def main(): # 'var' was set to 'spam' before # calling 'copy_context()' and 'ctx.run(main)', so: print(var.get()) # 'spam' print(ctx[var]) # 'spam' var.set('ham') # Now, after setting 'var' to 'ham': print(var.get()) # 'ham' print(ctx[var]) # 'ham' # Any changes that the 'main' function makes to 'var' # will be contained in 'ctx'. ctx.run(main) # The 'main()' function was run in the 'ctx' context, # so changes to 'var' are contained in it: print(ctx[var]) # 'ham' # However, outside of 'ctx', 'var' is still set to 'spam': print(var.get()) # 'spam' copy() Return a shallow copy of the context object. var in context Return "True" if the *context* has a value for *var* set; return "False" otherwise. context[var] Return the value of the *var* "ContextVar" variable. If the variable is not set in the context object, a "KeyError" is raised. get(var[, default]) Return the value for *var* if *var* has the value in the context object. Return *default* otherwise. If *default* is not given, return "None". iter(context) Return an iterator over the variables stored in the context object. len(proxy) Return the number of variables set in the context object. keys() Return a list of all variables in the context object. values() Return a list of all variables’ values in the context object. items() Return a list of 2-tuples containing all variables and their values in the context object. asyncio support =============== Context variables are natively supported in "asyncio" and are ready to be used without any extra configuration. For example, here is a simple echo server, that uses a context variable to make the address of a remote client available in the Task that handles that client: import asyncio import contextvars client_addr_var = contextvars.ContextVar('client_addr') def render_goodbye(): # The address of the currently handled client can be accessed # without passing it explicitly to this function. client_addr = client_addr_var.get() return f'Good bye, client @ {client_addr}\r\n'.encode() async def handle_request(reader, writer): addr = writer.transport.get_extra_info('socket').getpeername() client_addr_var.set(addr) # In any code that we call is now possible to get # client's address by calling 'client_addr_var.get()'. while True: line = await reader.readline() print(line) if not line.strip(): break writer.write(b'HTTP/1.1 200 OK\r\n') # status line writer.write(b'\r\n') # headers writer.write(render_goodbye()) # body writer.close() async def main(): srv = await asyncio.start_server( handle_request, '127.0.0.1', 8081) async with srv: await srv.serve_forever() asyncio.run(main()) # To test it you can use telnet or curl: # telnet 127.0.0.1 8081 # curl 127.0.0.1:8081 "copy" — Shallow and deep copy operations ***************************************** **Source code:** Lib/copy.py ====================================================================== Assignment statements in Python do not copy objects, they create bindings between a target and an object. For collections that are mutable or contain mutable items, a copy is sometimes needed so one can change one copy without changing the other. This module provides generic shallow and deep copy operations (explained below). Interface summary: copy.copy(obj) Return a shallow copy of *obj*. copy.deepcopy(obj[, memo]) Return a deep copy of *obj*. copy.replace(obj, /, **changes) Creates a new object of the same type as *obj*, replacing fields with values from *changes*. Added in version 3.13. exception copy.Error Raised for module specific errors. The difference between shallow and deep copying is only relevant for compound objects (objects that contain other objects, like lists or class instances): * A *shallow copy* constructs a new compound object and then (to the extent possible) inserts *references* into it to the objects found in the original. * A *deep copy* constructs a new compound object and then, recursively, inserts *copies* into it of the objects found in the original. Two problems often exist with deep copy operations that don’t exist with shallow copy operations: * Recursive objects (compound objects that, directly or indirectly, contain a reference to themselves) may cause a recursive loop. * Because deep copy copies everything it may copy too much, such as data which is intended to be shared between copies. The "deepcopy()" function avoids these problems by: * keeping a "memo" dictionary of objects already copied during the current copying pass; and * letting user-defined classes override the copying operation or the set of components copied. This module does not copy types like module, method, stack trace, stack frame, file, socket, window, or any similar types. It does “copy” functions and classes (shallow and deeply), by returning the original object unchanged; this is compatible with the way these are treated by the "pickle" module. Shallow copies of dictionaries can be made using "dict.copy()", and of lists by assigning a slice of the entire list, for example, "copied_list = original_list[:]". Classes can use the same interfaces to control copying that they use to control pickling. See the description of module "pickle" for information on these methods. In fact, the "copy" module uses the registered pickle functions from the "copyreg" module. In order for a class to define its own copy implementation, it can define special methods "__copy__()" and "__deepcopy__()". object.__copy__(self) Called to implement the shallow copy operation; no additional arguments are passed. object.__deepcopy__(self, memo) Called to implement the deep copy operation; it is passed one argument, the *memo* dictionary. If the "__deepcopy__" implementation needs to make a deep copy of a component, it should call the "deepcopy()" function with the component as first argument and the *memo* dictionary as second argument. The *memo* dictionary should be treated as an opaque object. Function "copy.replace()" is more limited than "copy()" and "deepcopy()", and only supports named tuples created by "namedtuple()", "dataclasses", and other classes which define method "__replace__()". object.__replace__(self, /, **changes) This method should create a new object of the same type, replacing fields with values from *changes*. Added in version 3.13. See also: Module "pickle" Discussion of the special methods used to support object state retrieval and restoration. "copyreg" — Register "pickle" support functions *********************************************** **Source code:** Lib/copyreg.py ====================================================================== The "copyreg" module offers a way to define functions used while pickling specific objects. The "pickle" and "copy" modules use those functions when pickling/copying those objects. The module provides configuration information about object constructors which are not classes. Such constructors may be factory functions or class instances. copyreg.constructor(object) Declares *object* to be a valid constructor. If *object* is not callable (and hence not valid as a constructor), raises "TypeError". copyreg.pickle(type, function, constructor_ob=None) Declares that *function* should be used as a “reduction” function for objects of type *type*. *function* must return either a string or a tuple containing between two and six elements. See the "dispatch_table" for more details on the interface of *function*. The *constructor_ob* parameter is a legacy feature and is now ignored, but if passed it must be a callable. Note that the "dispatch_table" attribute of a pickler object or subclass of "pickle.Pickler" can also be used for declaring reduction functions. Example ======= The example below would like to show how to register a pickle function and how it will be used: >>> import copyreg, copy, pickle >>> class C: ... def __init__(self, a): ... self.a = a ... >>> def pickle_c(c): ... print("pickling a C instance...") ... return C, (c.a,) ... >>> copyreg.pickle(C, pickle_c) >>> c = C(1) >>> d = copy.copy(c) pickling a C instance... >>> p = pickle.dumps(c) pickling a C instance... "crypt" — Function to check Unix passwords ****************************************** Deprecated since version 3.11, removed in version 3.13. This module is no longer part of the Python standard library. It was removed in Python 3.13 after being deprecated in Python 3.11. The removal was decided in **PEP 594**. Applications can use the "hashlib" module from the standard library. Other possible replacements are third-party libraries from PyPI: legacycrypt, bcrypt, argon2-cffi, or passlib. These are not supported or maintained by the Python core team. The last version of Python that provided the "crypt" module was Python 3.12. Cryptographic Services ********************** The modules described in this chapter implement various algorithms of a cryptographic nature. They are available at the discretion of the installation. Here’s an overview: * "hashlib" — Secure hashes and message digests * Hash algorithms * Usage * Constructors * Attributes * Hash Objects * SHAKE variable length digests * File hashing * Key derivation * BLAKE2 * Creating hash objects * Constants * Examples * Simple hashing * Using different digest sizes * Keyed hashing * Randomized hashing * Personalization * Tree mode * Credits * "hmac" — Keyed-Hashing for Message Authentication * "secrets" — Generate secure random numbers for managing secrets * Random numbers * Generating tokens * How many bytes should tokens use? * Other functions * Recipes and best practices "csv" — CSV File Reading and Writing ************************************ **Source code:** Lib/csv.py ====================================================================== The so-called CSV (Comma Separated Values) format is the most common import and export format for spreadsheets and databases. CSV format was used for many years prior to attempts to describe the format in a standardized way in **RFC 4180**. The lack of a well-defined standard means that subtle differences often exist in the data produced and consumed by different applications. These differences can make it annoying to process CSV files from multiple sources. Still, while the delimiters and quoting characters vary, the overall format is similar enough that it is possible to write a single module which can efficiently manipulate such data, hiding the details of reading and writing the data from the programmer. The "csv" module implements classes to read and write tabular data in CSV format. It allows programmers to say, “write this data in the format preferred by Excel,” or “read data from this file which was generated by Excel,” without knowing the precise details of the CSV format used by Excel. Programmers can also describe the CSV formats understood by other applications or define their own special-purpose CSV formats. The "csv" module’s "reader" and "writer" objects read and write sequences. Programmers can also read and write data in dictionary form using the "DictReader" and "DictWriter" classes. See also: **PEP 305** - CSV File API The Python Enhancement Proposal which proposed this addition to Python. Module Contents =============== The "csv" module defines the following functions: csv.reader(csvfile, /, dialect='excel', **fmtparams) Return a reader object that will process lines from the given *csvfile*. A csvfile must be an iterable of strings, each in the reader’s defined csv format. A csvfile is most commonly a file-like object or list. If *csvfile* is a file object, it should be opened with "newline=''". [1] An optional *dialect* parameter can be given which is used to define a set of parameters specific to a particular CSV dialect. It may be an instance of a subclass of the "Dialect" class or one of the strings returned by the "list_dialects()" function. The other optional *fmtparams* keyword arguments can be given to override individual formatting parameters in the current dialect. For full details about the dialect and formatting parameters, see section Dialects and Formatting Parameters. Each row read from the csv file is returned as a list of strings. No automatic data type conversion is performed unless the "QUOTE_NONNUMERIC" format option is specified (in which case unquoted fields are transformed into floats). A short usage example: >>> import csv >>> with open('eggs.csv', newline='') as csvfile: ... spamreader = csv.reader(csvfile, delimiter=' ', quotechar='|') ... for row in spamreader: ... print(', '.join(row)) Spam, Spam, Spam, Spam, Spam, Baked Beans Spam, Lovely Spam, Wonderful Spam csv.writer(csvfile, /, dialect='excel', **fmtparams) Return a writer object responsible for converting the user’s data into delimited strings on the given file-like object. *csvfile* can be any object with a "write()" method. If *csvfile* is a file object, it should be opened with "newline=''" [1]. An optional *dialect* parameter can be given which is used to define a set of parameters specific to a particular CSV dialect. It may be an instance of a subclass of the "Dialect" class or one of the strings returned by the "list_dialects()" function. The other optional *fmtparams* keyword arguments can be given to override individual formatting parameters in the current dialect. For full details about dialects and formatting parameters, see the Dialects and Formatting Parameters section. To make it as easy as possible to interface with modules which implement the DB API, the value "None" is written as the empty string. While this isn’t a reversible transformation, it makes it easier to dump SQL NULL data values to CSV files without preprocessing the data returned from a "cursor.fetch*" call. All other non-string data are stringified with "str()" before being written. A short usage example: import csv with open('eggs.csv', 'w', newline='') as csvfile: spamwriter = csv.writer(csvfile, delimiter=' ', quotechar='|', quoting=csv.QUOTE_MINIMAL) spamwriter.writerow(['Spam'] * 5 + ['Baked Beans']) spamwriter.writerow(['Spam', 'Lovely Spam', 'Wonderful Spam']) csv.register_dialect(name[, dialect[, **fmtparams]]) Associate *dialect* with *name*. *name* must be a string. The dialect can be specified either by passing a sub-class of "Dialect", or by *fmtparams* keyword arguments, or both, with keyword arguments overriding parameters of the dialect. For full details about dialects and formatting parameters, see section Dialects and Formatting Parameters. csv.unregister_dialect(name) Delete the dialect associated with *name* from the dialect registry. An "Error" is raised if *name* is not a registered dialect name. csv.get_dialect(name) Return the dialect associated with *name*. An "Error" is raised if *name* is not a registered dialect name. This function returns an immutable "Dialect". csv.list_dialects() Return the names of all registered dialects. csv.field_size_limit([new_limit]) Returns the current maximum field size allowed by the parser. If *new_limit* is given, this becomes the new limit. The "csv" module defines the following classes: class csv.DictReader(f, fieldnames=None, restkey=None, restval=None, dialect='excel', *args, **kwds) Create an object that operates like a regular reader but maps the information in each row to a "dict" whose keys are given by the optional *fieldnames* parameter. The *fieldnames* parameter is a *sequence*. If *fieldnames* is omitted, the values in the first row of file *f* will be used as the fieldnames and will be omitted from the results. If *fieldnames* is provided, they will be used and the first row will be included in the results. Regardless of how the fieldnames are determined, the dictionary preserves their original ordering. If a row has more fields than fieldnames, the remaining data is put in a list and stored with the fieldname specified by *restkey* (which defaults to "None"). If a non-blank row has fewer fields than fieldnames, the missing values are filled-in with the value of *restval* (which defaults to "None"). All other optional or keyword arguments are passed to the underlying "reader" instance. If the argument passed to *fieldnames* is an iterator, it will be coerced to a "list". Changed in version 3.6: Returned rows are now of type "OrderedDict". Changed in version 3.8: Returned rows are now of type "dict". A short usage example: >>> import csv >>> with open('names.csv', newline='') as csvfile: ... reader = csv.DictReader(csvfile) ... for row in reader: ... print(row['first_name'], row['last_name']) ... Eric Idle John Cleese >>> print(row) {'first_name': 'John', 'last_name': 'Cleese'} class csv.DictWriter(f, fieldnames, restval='', extrasaction='raise', dialect='excel', *args, **kwds) Create an object which operates like a regular writer but maps dictionaries onto output rows. The *fieldnames* parameter is a "sequence" of keys that identify the order in which values in the dictionary passed to the "writerow()" method are written to file *f*. The optional *restval* parameter specifies the value to be written if the dictionary is missing a key in *fieldnames*. If the dictionary passed to the "writerow()" method contains a key not found in *fieldnames*, the optional *extrasaction* parameter indicates what action to take. If it is set to "'raise'", the default value, a "ValueError" is raised. If it is set to "'ignore'", extra values in the dictionary are ignored. Any other optional or keyword arguments are passed to the underlying "writer" instance. Note that unlike the "DictReader" class, the *fieldnames* parameter of the "DictWriter" class is not optional. If the argument passed to *fieldnames* is an iterator, it will be coerced to a "list". A short usage example: import csv with open('names.csv', 'w', newline='') as csvfile: fieldnames = ['first_name', 'last_name'] writer = csv.DictWriter(csvfile, fieldnames=fieldnames) writer.writeheader() writer.writerow({'first_name': 'Baked', 'last_name': 'Beans'}) writer.writerow({'first_name': 'Lovely', 'last_name': 'Spam'}) writer.writerow({'first_name': 'Wonderful', 'last_name': 'Spam'}) class csv.Dialect The "Dialect" class is a container class whose attributes contain information for how to handle doublequotes, whitespace, delimiters, etc. Due to the lack of a strict CSV specification, different applications produce subtly different CSV data. "Dialect" instances define how "reader" and "writer" instances behave. All available "Dialect" names are returned by "list_dialects()", and they can be registered with specific "reader" and "writer" classes through their initializer ("__init__") functions like this: import csv with open('students.csv', 'w', newline='') as csvfile: writer = csv.writer(csvfile, dialect='unix') class csv.excel The "excel" class defines the usual properties of an Excel- generated CSV file. It is registered with the dialect name "'excel'". class csv.excel_tab The "excel_tab" class defines the usual properties of an Excel- generated TAB-delimited file. It is registered with the dialect name "'excel-tab'". class csv.unix_dialect The "unix_dialect" class defines the usual properties of a CSV file generated on UNIX systems, i.e. using "'\n'" as line terminator and quoting all fields. It is registered with the dialect name "'unix'". Added in version 3.2. class csv.Sniffer The "Sniffer" class is used to deduce the format of a CSV file. The "Sniffer" class provides two methods: sniff(sample, delimiters=None) Analyze the given *sample* and return a "Dialect" subclass reflecting the parameters found. If the optional *delimiters* parameter is given, it is interpreted as a string containing possible valid delimiter characters. has_header(sample) Analyze the sample text (presumed to be in CSV format) and return "True" if the first row appears to be a series of column headers. Inspecting each column, one of two key criteria will be considered to estimate if the sample contains a header: * the second through n-th rows contain numeric values * the second through n-th rows contain strings where at least one value’s length differs from that of the putative header of that column. Twenty rows after the first row are sampled; if more than half of columns + rows meet the criteria, "True" is returned. Note: This method is a rough heuristic and may produce both false positives and negatives. An example for "Sniffer" use: with open('example.csv', newline='') as csvfile: dialect = csv.Sniffer().sniff(csvfile.read(1024)) csvfile.seek(0) reader = csv.reader(csvfile, dialect) # ... process CSV file contents here ... The "csv" module defines the following constants: csv.QUOTE_ALL Instructs "writer" objects to quote all fields. csv.QUOTE_MINIMAL Instructs "writer" objects to only quote those fields which contain special characters such as *delimiter*, *quotechar*, "'\r'", "'\n'" or any of the characters in *lineterminator*. csv.QUOTE_NONNUMERIC Instructs "writer" objects to quote all non-numeric fields. Instructs "reader" objects to convert all non-quoted fields to type "float". Note: Some numeric types, such as "bool", "Fraction", or "IntEnum", have a string representation that cannot be converted to "float". They cannot be read in the "QUOTE_NONNUMERIC" and "QUOTE_STRINGS" modes. csv.QUOTE_NONE Instructs "writer" objects to never quote fields. When the current *delimiter*, *quotechar*, *escapechar*, "'\r'", "'\n'" or any of the characters in *lineterminator* occurs in output data it is preceded by the current *escapechar* character. If *escapechar* is not set, the writer will raise "Error" if any characters that require escaping are encountered. Set *quotechar* to "None" to prevent its escaping. Instructs "reader" objects to perform no special processing of quote characters. csv.QUOTE_NOTNULL Instructs "writer" objects to quote all fields which are not "None". This is similar to "QUOTE_ALL", except that if a field value is "None" an empty (unquoted) string is written. Instructs "reader" objects to interpret an empty (unquoted) field as "None" and to otherwise behave as "QUOTE_ALL". Added in version 3.12. csv.QUOTE_STRINGS Instructs "writer" objects to always place quotes around fields which are strings. This is similar to "QUOTE_NONNUMERIC", except that if a field value is "None" an empty (unquoted) string is written. Instructs "reader" objects to interpret an empty (unquoted) string as "None" and to otherwise behave as "QUOTE_NONNUMERIC". Added in version 3.12. The "csv" module defines the following exception: exception csv.Error Raised by any of the functions when an error is detected. Dialects and Formatting Parameters ================================== To make it easier to specify the format of input and output records, specific formatting parameters are grouped together into dialects. A dialect is a subclass of the "Dialect" class containing various attributes describing the format of the CSV file. When creating "reader" or "writer" objects, the programmer can specify a string or a subclass of the "Dialect" class as the dialect parameter. In addition to, or instead of, the *dialect* parameter, the programmer can also specify individual formatting parameters, which have the same names as the attributes defined below for the "Dialect" class. Dialects support the following attributes: Dialect.delimiter A one-character string used to separate fields. It defaults to "','". Dialect.doublequote Controls how instances of *quotechar* appearing inside a field should themselves be quoted. When "True", the character is doubled. When "False", the *escapechar* is used as a prefix to the *quotechar*. It defaults to "True". On output, if *doublequote* is "False" and no *escapechar* is set, "Error" is raised if a *quotechar* is found in a field. Dialect.escapechar A one-character string used by the writer to escape characters that require escaping: * the *delimiter*, the *quotechar*, "'\r'", "'\n'" and any of the characters in *lineterminator* are escaped if *quoting* is set to "QUOTE_NONE"; * the *quotechar* is escaped if *doublequote* is "False"; * the *escapechar* itself. On reading, the *escapechar* removes any special meaning from the following character. It defaults to "None", which disables escaping. Changed in version 3.11: An empty *escapechar* is not allowed. Dialect.lineterminator The string used to terminate lines produced by the "writer". It defaults to "'\r\n'". Note: The "reader" is hard-coded to recognise either "'\r'" or "'\n'" as end-of-line, and ignores *lineterminator*. This behavior may change in the future. Dialect.quotechar A one-character string used to quote fields containing special characters, such as the *delimiter* or the *quotechar*, or which contain new-line characters ("'\r'", "'\n'" or any of the characters in *lineterminator*). It defaults to "'"'". Can be set to "None" to prevent escaping "'"'" if *quoting* is set to "QUOTE_NONE". Changed in version 3.11: An empty *quotechar* is not allowed. Dialect.quoting Controls when quotes should be generated by the writer and recognised by the reader. It can take on any of the QUOTE_* constants and defaults to "QUOTE_MINIMAL" if *quotechar* is not "None", and "QUOTE_NONE" otherwise. Dialect.skipinitialspace When "True", spaces immediately following the *delimiter* are ignored. The default is "False". Dialect.strict When "True", raise exception "Error" on bad CSV input. The default is "False". Reader Objects ============== Reader objects ("DictReader" instances and objects returned by the "reader()" function) have the following public methods: csvreader.__next__() Return the next row of the reader’s iterable object as a list (if the object was returned from "reader()") or a dict (if it is a "DictReader" instance), parsed according to the current "Dialect". Usually you should call this as "next(reader)". Reader objects have the following public attributes: csvreader.dialect A read-only description of the dialect in use by the parser. csvreader.line_num The number of lines read from the source iterator. This is not the same as the number of records returned, as records can span multiple lines. DictReader objects have the following public attribute: DictReader.fieldnames If not passed as a parameter when creating the object, this attribute is initialized upon first access or when the first record is read from the file. Writer Objects ============== "writer" objects ("DictWriter" instances and objects returned by the "writer()" function) have the following public methods. A *row* must be an iterable of strings or numbers for "writer" objects and a dictionary mapping fieldnames to strings or numbers (by passing them through "str()" first) for "DictWriter" objects. Note that complex numbers are written out surrounded by parens. This may cause some problems for other programs which read CSV files (assuming they support complex numbers at all). csvwriter.writerow(row) Write the *row* parameter to the writer’s file object, formatted according to the current "Dialect". Return the return value of the call to the *write* method of the underlying file object. Changed in version 3.5: Added support of arbitrary iterables. csvwriter.writerows(rows) Write all elements in *rows* (an iterable of *row* objects as described above) to the writer’s file object, formatted according to the current dialect. Writer objects have the following public attribute: csvwriter.dialect A read-only description of the dialect in use by the writer. DictWriter objects have the following public method: DictWriter.writeheader() Write a row with the field names (as specified in the constructor) to the writer’s file object, formatted according to the current dialect. Return the return value of the "csvwriter.writerow()" call used internally. Added in version 3.2. Changed in version 3.8: "writeheader()" now also returns the value returned by the "csvwriter.writerow()" method it uses internally. Examples ======== The simplest example of reading a CSV file: import csv with open('some.csv', newline='') as f: reader = csv.reader(f) for row in reader: print(row) Reading a file with an alternate format: import csv with open('passwd', newline='') as f: reader = csv.reader(f, delimiter=':', quoting=csv.QUOTE_NONE) for row in reader: print(row) The corresponding simplest possible writing example is: import csv with open('some.csv', 'w', newline='') as f: writer = csv.writer(f) writer.writerows(someiterable) Since "open()" is used to open a CSV file for reading, the file will by default be decoded into unicode using the system default encoding (see "locale.getencoding()"). To decode a file using a different encoding, use the "encoding" argument of open: import csv with open('some.csv', newline='', encoding='utf-8') as f: reader = csv.reader(f) for row in reader: print(row) The same applies to writing in something other than the system default encoding: specify the encoding argument when opening the output file. Registering a new dialect: import csv csv.register_dialect('unixpwd', delimiter=':', quoting=csv.QUOTE_NONE) with open('passwd', newline='') as f: reader = csv.reader(f, 'unixpwd') A slightly more advanced use of the reader — catching and reporting errors: import csv, sys filename = 'some.csv' with open(filename, newline='') as f: reader = csv.reader(f) try: for row in reader: print(row) except csv.Error as e: sys.exit(f'file {filename}, line {reader.line_num}: {e}') And while the module doesn’t directly support parsing strings, it can easily be done: import csv for row in csv.reader(['one,two,three']): print(row) -[ Footnotes ]- [1] If "newline=''" is not specified, newlines embedded inside quoted fields will not be interpreted correctly, and on platforms that use "\r\n" linendings on write an extra "\r" will be added. It should always be safe to specify "newline=''", since the csv module does its own (*universal*) newline handling. "ctypes" — A foreign function library for Python ************************************************ **Source code:** Lib/ctypes ====================================================================== "ctypes" is a foreign function library for Python. It provides C compatible data types, and allows calling functions in DLLs or shared libraries. It can be used to wrap these libraries in pure Python. ctypes tutorial =============== Note: The code samples in this tutorial use "doctest" to make sure that they actually work. Since some code samples behave differently under Linux, Windows, or macOS, they contain doctest directives in comments. Note: Some code samples reference the ctypes "c_int" type. On platforms where "sizeof(long) == sizeof(int)" it is an alias to "c_long". So, you should not be confused if "c_long" is printed if you would expect "c_int" — they are actually the same type. Loading dynamic link libraries ------------------------------ "ctypes" exports the *cdll*, and on Windows *windll* and *oledll* objects, for loading dynamic link libraries. You load libraries by accessing them as attributes of these objects. *cdll* loads libraries which export functions using the standard "cdecl" calling convention, while *windll* libraries call functions using the "stdcall" calling convention. *oledll* also uses the "stdcall" calling convention, and assumes the functions return a Windows "HRESULT" error code. The error code is used to automatically raise an "OSError" exception when the function call fails. Changed in version 3.3: Windows errors used to raise "WindowsError", which is now an alias of "OSError". Here are some examples for Windows. Note that "msvcrt" is the MS standard C library containing most standard C functions, and uses the "cdecl" calling convention: >>> from ctypes import * >>> print(windll.kernel32) >>> print(cdll.msvcrt) >>> libc = cdll.msvcrt >>> Windows appends the usual ".dll" file suffix automatically. Note: Accessing the standard C library through "cdll.msvcrt" will use an outdated version of the library that may be incompatible with the one being used by Python. Where possible, use native Python functionality, or else import and use the "msvcrt" module. On Linux, it is required to specify the filename *including* the extension to load a library, so attribute access can not be used to load libraries. Either the "LoadLibrary()" method of the dll loaders should be used, or you should load the library by creating an instance of CDLL by calling the constructor: >>> cdll.LoadLibrary("libc.so.6") >>> libc = CDLL("libc.so.6") >>> libc >>> Accessing functions from loaded dlls ------------------------------------ Functions are accessed as attributes of dll objects: >>> libc.printf <_FuncPtr object at 0x...> >>> print(windll.kernel32.GetModuleHandleA) <_FuncPtr object at 0x...> >>> print(windll.kernel32.MyOwnFunction) Traceback (most recent call last): File "", line 1, in File "ctypes.py", line 239, in __getattr__ func = _StdcallFuncPtr(name, self) AttributeError: function 'MyOwnFunction' not found >>> Note that win32 system dlls like "kernel32" and "user32" often export ANSI as well as UNICODE versions of a function. The UNICODE version is exported with a "W" appended to the name, while the ANSI version is exported with an "A" appended to the name. The win32 "GetModuleHandle" function, which returns a *module handle* for a given module name, has the following C prototype, and a macro is used to expose one of them as "GetModuleHandle" depending on whether UNICODE is defined or not: /* ANSI version */ HMODULE GetModuleHandleA(LPCSTR lpModuleName); /* UNICODE version */ HMODULE GetModuleHandleW(LPCWSTR lpModuleName); *windll* does not try to select one of them by magic, you must access the version you need by specifying "GetModuleHandleA" or "GetModuleHandleW" explicitly, and then call it with bytes or string objects respectively. Sometimes, dlls export functions with names which aren’t valid Python identifiers, like ""??2@YAPAXI@Z"". In this case you have to use "getattr()" to retrieve the function: >>> getattr(cdll.msvcrt, "??2@YAPAXI@Z") <_FuncPtr object at 0x...> >>> On Windows, some dlls export functions not by name but by ordinal. These functions can be accessed by indexing the dll object with the ordinal number: >>> cdll.kernel32[1] <_FuncPtr object at 0x...> >>> cdll.kernel32[0] Traceback (most recent call last): File "", line 1, in File "ctypes.py", line 310, in __getitem__ func = _StdcallFuncPtr(name, self) AttributeError: function ordinal 0 not found >>> Calling functions ----------------- You can call these functions like any other Python callable. This example uses the "rand()" function, which takes no arguments and returns a pseudo-random integer: >>> print(libc.rand()) 1804289383 On Windows, you can call the "GetModuleHandleA()" function, which returns a win32 module handle (passing "None" as single argument to call it with a "NULL" pointer): >>> print(hex(windll.kernel32.GetModuleHandleA(None))) 0x1d000000 >>> "ValueError" is raised when you call an "stdcall" function with the "cdecl" calling convention, or vice versa: >>> cdll.kernel32.GetModuleHandleA(None) Traceback (most recent call last): File "", line 1, in ValueError: Procedure probably called with not enough arguments (4 bytes missing) >>> >>> windll.msvcrt.printf(b"spam") Traceback (most recent call last): File "", line 1, in ValueError: Procedure probably called with too many arguments (4 bytes in excess) >>> To find out the correct calling convention you have to look into the C header file or the documentation for the function you want to call. On Windows, "ctypes" uses win32 structured exception handling to prevent crashes from general protection faults when functions are called with invalid argument values: >>> windll.kernel32.GetModuleHandleA(32) Traceback (most recent call last): File "", line 1, in OSError: exception: access violation reading 0x00000020 >>> There are, however, enough ways to crash Python with "ctypes", so you should be careful anyway. The "faulthandler" module can be helpful in debugging crashes (e.g. from segmentation faults produced by erroneous C library calls). "None", integers, bytes objects and (unicode) strings are the only native Python objects that can directly be used as parameters in these function calls. "None" is passed as a C "NULL" pointer, bytes objects and strings are passed as pointer to the memory block that contains their data (char* or wchar_t*). Python integers are passed as the platform’s default C int type, their value is masked to fit into the C type. Before we move on calling functions with other parameter types, we have to learn more about "ctypes" data types. Fundamental data types ---------------------- "ctypes" defines a number of primitive C compatible data types: +------------------------+--------------------------------------------+------------------------------+ | ctypes type | C type | Python type | |========================|============================================|==============================| | "c_bool" | _Bool | bool (1) | +------------------------+--------------------------------------------+------------------------------+ | "c_char" | char | 1-character bytes object | +------------------------+--------------------------------------------+------------------------------+ | "c_wchar" | "wchar_t" | 1-character string | +------------------------+--------------------------------------------+------------------------------+ | "c_byte" | char | int | +------------------------+--------------------------------------------+------------------------------+ | "c_ubyte" | unsigned char | int | +------------------------+--------------------------------------------+------------------------------+ | "c_short" | short | int | +------------------------+--------------------------------------------+------------------------------+ | "c_ushort" | unsigned short | int | +------------------------+--------------------------------------------+------------------------------+ | "c_int" | int | int | +------------------------+--------------------------------------------+------------------------------+ | "c_int8" | "int8_t" | int | +------------------------+--------------------------------------------+------------------------------+ | "c_int16" | "int16_t" | int | +------------------------+--------------------------------------------+------------------------------+ | "c_int32" | "int32_t" | int | +------------------------+--------------------------------------------+------------------------------+ | "c_int64" | "int64_t" | int | +------------------------+--------------------------------------------+------------------------------+ | "c_uint" | unsigned int | int | +------------------------+--------------------------------------------+------------------------------+ | "c_uint8" | "uint8_t" | int | +------------------------+--------------------------------------------+------------------------------+ | "c_uint16" | "uint16_t" | int | +------------------------+--------------------------------------------+------------------------------+ | "c_uint32" | "uint32_t" | int | +------------------------+--------------------------------------------+------------------------------+ | "c_uint64" | "uint64_t" | int | +------------------------+--------------------------------------------+------------------------------+ | "c_long" | long | int | +------------------------+--------------------------------------------+------------------------------+ | "c_ulong" | unsigned long | int | +------------------------+--------------------------------------------+------------------------------+ | "c_longlong" | __int64 or long long | int | +------------------------+--------------------------------------------+------------------------------+ | "c_ulonglong" | unsigned __int64 or unsigned long long | int | +------------------------+--------------------------------------------+------------------------------+ | "c_size_t" | "size_t" | int | +------------------------+--------------------------------------------+------------------------------+ | "c_ssize_t" | "ssize_t" or Py_ssize_t | int | +------------------------+--------------------------------------------+------------------------------+ | "c_time_t" | "time_t" | int | +------------------------+--------------------------------------------+------------------------------+ | "c_float" | float | float | +------------------------+--------------------------------------------+------------------------------+ | "c_double" | double | float | +------------------------+--------------------------------------------+------------------------------+ | "c_longdouble" | long double | float | +------------------------+--------------------------------------------+------------------------------+ | "c_char_p" | char* (NUL terminated) | bytes object or "None" | +------------------------+--------------------------------------------+------------------------------+ | "c_wchar_p" | wchar_t* (NUL terminated) | string or "None" | +------------------------+--------------------------------------------+------------------------------+ | "c_void_p" | void* | int or "None" | +------------------------+--------------------------------------------+------------------------------+ 1. The constructor accepts any object with a truth value. All these types can be created by calling them with an optional initializer of the correct type and value: >>> c_int() c_long(0) >>> c_wchar_p("Hello, World") c_wchar_p(140018365411392) >>> c_ushort(-3) c_ushort(65533) >>> Since these types are mutable, their value can also be changed afterwards: >>> i = c_int(42) >>> print(i) c_long(42) >>> print(i.value) 42 >>> i.value = -99 >>> print(i.value) -99 >>> Assigning a new value to instances of the pointer types "c_char_p", "c_wchar_p", and "c_void_p" changes the *memory location* they point to, *not the contents* of the memory block (of course not, because Python string objects are immutable): >>> s = "Hello, World" >>> c_s = c_wchar_p(s) >>> print(c_s) c_wchar_p(139966785747344) >>> print(c_s.value) Hello World >>> c_s.value = "Hi, there" >>> print(c_s) # the memory location has changed c_wchar_p(139966783348904) >>> print(c_s.value) Hi, there >>> print(s) # first object is unchanged Hello, World >>> You should be careful, however, not to pass them to functions expecting pointers to mutable memory. If you need mutable memory blocks, ctypes has a "create_string_buffer()" function which creates these in various ways. The current memory block contents can be accessed (or changed) with the "raw" property; if you want to access it as NUL terminated string, use the "value" property: >>> from ctypes import * >>> p = create_string_buffer(3) # create a 3 byte buffer, initialized to NUL bytes >>> print(sizeof(p), repr(p.raw)) 3 b'\x00\x00\x00' >>> p = create_string_buffer(b"Hello") # create a buffer containing a NUL terminated string >>> print(sizeof(p), repr(p.raw)) 6 b'Hello\x00' >>> print(repr(p.value)) b'Hello' >>> p = create_string_buffer(b"Hello", 10) # create a 10 byte buffer >>> print(sizeof(p), repr(p.raw)) 10 b'Hello\x00\x00\x00\x00\x00' >>> p.value = b"Hi" >>> print(sizeof(p), repr(p.raw)) 10 b'Hi\x00lo\x00\x00\x00\x00\x00' >>> The "create_string_buffer()" function replaces the old "c_buffer()" function (which is still available as an alias). To create a mutable memory block containing unicode characters of the C type "wchar_t", use the "create_unicode_buffer()" function. Calling functions, continued ---------------------------- Note that printf prints to the real standard output channel, *not* to "sys.stdout", so these examples will only work at the console prompt, not from within *IDLE* or *PythonWin*: >>> printf = libc.printf >>> printf(b"Hello, %s\n", b"World!") Hello, World! 14 >>> printf(b"Hello, %S\n", "World!") Hello, World! 14 >>> printf(b"%d bottles of beer\n", 42) 42 bottles of beer 19 >>> printf(b"%f bottles of beer\n", 42.5) Traceback (most recent call last): File "", line 1, in ctypes.ArgumentError: argument 2: TypeError: Don't know how to convert parameter 2 >>> As has been mentioned before, all Python types except integers, strings, and bytes objects have to be wrapped in their corresponding "ctypes" type, so that they can be converted to the required C data type: >>> printf(b"An int %d, a double %f\n", 1234, c_double(3.14)) An int 1234, a double 3.140000 31 >>> Calling variadic functions -------------------------- On a lot of platforms calling variadic functions through ctypes is exactly the same as calling functions with a fixed number of parameters. On some platforms, and in particular ARM64 for Apple Platforms, the calling convention for variadic functions is different than that for regular functions. On those platforms it is required to specify the "argtypes" attribute for the regular, non-variadic, function arguments: libc.printf.argtypes = [ctypes.c_char_p] Because specifying the attribute does not inhibit portability it is advised to always specify "argtypes" for all variadic functions. Calling functions with your own custom data types ------------------------------------------------- You can also customize "ctypes" argument conversion to allow instances of your own classes be used as function arguments. "ctypes" looks for an "_as_parameter_" attribute and uses this as the function argument. The attribute must be an integer, string, bytes, a "ctypes" instance, or an object with an "_as_parameter_" attribute: >>> class Bottles: ... def __init__(self, number): ... self._as_parameter_ = number ... >>> bottles = Bottles(42) >>> printf(b"%d bottles of beer\n", bottles) 42 bottles of beer 19 >>> If you don’t want to store the instance’s data in the "_as_parameter_" instance variable, you could define a "property" which makes the attribute available on request. Specifying the required argument types (function prototypes) ------------------------------------------------------------ It is possible to specify the required argument types of functions exported from DLLs by setting the "argtypes" attribute. "argtypes" must be a sequence of C data types (the "printf()" function is probably not a good example here, because it takes a variable number and different types of parameters depending on the format string, on the other hand this is quite handy to experiment with this feature): >>> printf.argtypes = [c_char_p, c_char_p, c_int, c_double] >>> printf(b"String '%s', Int %d, Double %f\n", b"Hi", 10, 2.2) String 'Hi', Int 10, Double 2.200000 37 >>> Specifying a format protects against incompatible argument types (just as a prototype for a C function), and tries to convert the arguments to valid types: >>> printf(b"%d %d %d", 1, 2, 3) Traceback (most recent call last): File "", line 1, in ctypes.ArgumentError: argument 2: TypeError: 'int' object cannot be interpreted as ctypes.c_char_p >>> printf(b"%s %d %f\n", b"X", 2, 3) X 2 3.000000 13 >>> If you have defined your own classes which you pass to function calls, you have to implement a "from_param()" class method for them to be able to use them in the "argtypes" sequence. The "from_param()" class method receives the Python object passed to the function call, it should do a typecheck or whatever is needed to make sure this object is acceptable, and then return the object itself, its "_as_parameter_" attribute, or whatever you want to pass as the C function argument in this case. Again, the result should be an integer, string, bytes, a "ctypes" instance, or an object with an "_as_parameter_" attribute. Return types ------------ By default functions are assumed to return the C int type. Other return types can be specified by setting the "restype" attribute of the function object. The C prototype of "time()" is "time_t time(time_t *)". Because "time_t" might be of a different type than the default return type int, you should specify the "restype" attribute: >>> libc.time.restype = c_time_t The argument types can be specified using "argtypes": >>> libc.time.argtypes = (POINTER(c_time_t),) To call the function with a "NULL" pointer as first argument, use "None": >>> print(libc.time(None)) 1150640792 Here is a more advanced example, it uses the "strchr()" function, which expects a string pointer and a char, and returns a pointer to a string: >>> strchr = libc.strchr >>> strchr(b"abcdef", ord("d")) 8059983 >>> strchr.restype = c_char_p # c_char_p is a pointer to a string >>> strchr(b"abcdef", ord("d")) b'def' >>> print(strchr(b"abcdef", ord("x"))) None >>> If you want to avoid the "ord("x")" calls above, you can set the "argtypes" attribute, and the second argument will be converted from a single character Python bytes object into a C char: >>> strchr.restype = c_char_p >>> strchr.argtypes = [c_char_p, c_char] >>> strchr(b"abcdef", b"d") b'def' >>> strchr(b"abcdef", b"def") Traceback (most recent call last): ctypes.ArgumentError: argument 2: TypeError: one character bytes, bytearray or integer expected >>> print(strchr(b"abcdef", b"x")) None >>> strchr(b"abcdef", b"d") b'def' >>> You can also use a callable Python object (a function or a class for example) as the "restype" attribute, if the foreign function returns an integer. The callable will be called with the *integer* the C function returns, and the result of this call will be used as the result of your function call. This is useful to check for error return values and automatically raise an exception: >>> GetModuleHandle = windll.kernel32.GetModuleHandleA >>> def ValidHandle(value): ... if value == 0: ... raise WinError() ... return value ... >>> >>> GetModuleHandle.restype = ValidHandle >>> GetModuleHandle(None) 486539264 >>> GetModuleHandle("something silly") Traceback (most recent call last): File "", line 1, in File "", line 3, in ValidHandle OSError: [Errno 126] The specified module could not be found. >>> "WinError" is a function which will call Windows "FormatMessage()" api to get the string representation of an error code, and *returns* an exception. "WinError" takes an optional error code parameter, if no one is used, it calls "GetLastError()" to retrieve it. Please note that a much more powerful error checking mechanism is available through the "errcheck" attribute; see the reference manual for details. Passing pointers (or: passing parameters by reference) ------------------------------------------------------ Sometimes a C api function expects a *pointer* to a data type as parameter, probably to write into the corresponding location, or if the data is too large to be passed by value. This is also known as *passing parameters by reference*. "ctypes" exports the "byref()" function which is used to pass parameters by reference. The same effect can be achieved with the "pointer()" function, although "pointer()" does a lot more work since it constructs a real pointer object, so it is faster to use "byref()" if you don’t need the pointer object in Python itself: >>> i = c_int() >>> f = c_float() >>> s = create_string_buffer(b'\000' * 32) >>> print(i.value, f.value, repr(s.value)) 0 0.0 b'' >>> libc.sscanf(b"1 3.14 Hello", b"%d %f %s", ... byref(i), byref(f), s) 3 >>> print(i.value, f.value, repr(s.value)) 1 3.1400001049 b'Hello' >>> Structures and unions --------------------- Structures and unions must derive from the "Structure" and "Union" base classes which are defined in the "ctypes" module. Each subclass must define a "_fields_" attribute. "_fields_" must be a list of *2-tuples*, containing a *field name* and a *field type*. The field type must be a "ctypes" type like "c_int", or any other derived "ctypes" type: structure, union, array, pointer. Here is a simple example of a POINT structure, which contains two integers named *x* and *y*, and also shows how to initialize a structure in the constructor: >>> from ctypes import * >>> class POINT(Structure): ... _fields_ = [("x", c_int), ... ("y", c_int)] ... >>> point = POINT(10, 20) >>> print(point.x, point.y) 10 20 >>> point = POINT(y=5) >>> print(point.x, point.y) 0 5 >>> POINT(1, 2, 3) Traceback (most recent call last): File "", line 1, in TypeError: too many initializers >>> You can, however, build much more complicated structures. A structure can itself contain other structures by using a structure as a field type. Here is a RECT structure which contains two POINTs named *upperleft* and *lowerright*: >>> class RECT(Structure): ... _fields_ = [("upperleft", POINT), ... ("lowerright", POINT)] ... >>> rc = RECT(point) >>> print(rc.upperleft.x, rc.upperleft.y) 0 5 >>> print(rc.lowerright.x, rc.lowerright.y) 0 0 >>> Nested structures can also be initialized in the constructor in several ways: >>> r = RECT(POINT(1, 2), POINT(3, 4)) >>> r = RECT((1, 2), (3, 4)) Field *descriptor*s can be retrieved from the *class*, they are useful for debugging because they can provide useful information: >>> print(POINT.x) >>> print(POINT.y) >>> Warning: "ctypes" does not support passing unions or structures with bit- fields to functions by value. While this may work on 32-bit x86, it’s not guaranteed by the library to work in the general case. Unions and structures with bit-fields should always be passed to functions by pointer. Structure/union alignment and byte order ---------------------------------------- By default, Structure and Union fields are aligned in the same way the C compiler does it. It is possible to override this behavior by specifying a "_pack_" class attribute in the subclass definition. This must be set to a positive integer and specifies the maximum alignment for the fields. This is what "#pragma pack(n)" also does in MSVC. It is also possible to set a minimum alignment for how the subclass itself is packed in the same way "#pragma align(n)" works in MSVC. This can be achieved by specifying a "_align_" class attribute in the subclass definition. "ctypes" uses the native byte order for Structures and Unions. To build structures with non-native byte order, you can use one of the "BigEndianStructure", "LittleEndianStructure", "BigEndianUnion", and "LittleEndianUnion" base classes. These classes cannot contain pointer fields. Bit fields in structures and unions ----------------------------------- It is possible to create structures and unions containing bit fields. Bit fields are only possible for integer fields, the bit width is specified as the third item in the "_fields_" tuples: >>> class Int(Structure): ... _fields_ = [("first_16", c_int, 16), ... ("second_16", c_int, 16)] ... >>> print(Int.first_16) >>> print(Int.second_16) >>> Arrays ------ Arrays are sequences, containing a fixed number of instances of the same type. The recommended way to create array types is by multiplying a data type with a positive integer: TenPointsArrayType = POINT * 10 Here is an example of a somewhat artificial data type, a structure containing 4 POINTs among other stuff: >>> from ctypes import * >>> class POINT(Structure): ... _fields_ = ("x", c_int), ("y", c_int) ... >>> class MyStruct(Structure): ... _fields_ = [("a", c_int), ... ("b", c_float), ... ("point_array", POINT * 4)] >>> >>> print(len(MyStruct().point_array)) 4 >>> Instances are created in the usual way, by calling the class: arr = TenPointsArrayType() for pt in arr: print(pt.x, pt.y) The above code print a series of "0 0" lines, because the array contents is initialized to zeros. Initializers of the correct type can also be specified: >>> from ctypes import * >>> TenIntegers = c_int * 10 >>> ii = TenIntegers(1, 2, 3, 4, 5, 6, 7, 8, 9, 10) >>> print(ii) >>> for i in ii: print(i, end=" ") ... 1 2 3 4 5 6 7 8 9 10 >>> Pointers -------- Pointer instances are created by calling the "pointer()" function on a "ctypes" type: >>> from ctypes import * >>> i = c_int(42) >>> pi = pointer(i) >>> Pointer instances have a "contents" attribute which returns the object to which the pointer points, the "i" object above: >>> pi.contents c_long(42) >>> Note that "ctypes" does not have OOR (original object return), it constructs a new, equivalent object each time you retrieve an attribute: >>> pi.contents is i False >>> pi.contents is pi.contents False >>> Assigning another "c_int" instance to the pointer’s contents attribute would cause the pointer to point to the memory location where this is stored: >>> i = c_int(99) >>> pi.contents = i >>> pi.contents c_long(99) >>> Pointer instances can also be indexed with integers: >>> pi[0] 99 >>> Assigning to an integer index changes the pointed to value: >>> print(i) c_long(99) >>> pi[0] = 22 >>> print(i) c_long(22) >>> It is also possible to use indexes different from 0, but you must know what you’re doing, just as in C: You can access or change arbitrary memory locations. Generally you only use this feature if you receive a pointer from a C function, and you *know* that the pointer actually points to an array instead of a single item. Behind the scenes, the "pointer()" function does more than simply create pointer instances, it has to create pointer *types* first. This is done with the "POINTER()" function, which accepts any "ctypes" type, and returns a new type: >>> PI = POINTER(c_int) >>> PI >>> PI(42) Traceback (most recent call last): File "", line 1, in TypeError: expected c_long instead of int >>> PI(c_int(42)) >>> Calling the pointer type without an argument creates a "NULL" pointer. "NULL" pointers have a "False" boolean value: >>> null_ptr = POINTER(c_int)() >>> print(bool(null_ptr)) False >>> "ctypes" checks for "NULL" when dereferencing pointers (but dereferencing invalid non-"NULL" pointers would crash Python): >>> null_ptr[0] Traceback (most recent call last): .... ValueError: NULL pointer access >>> >>> null_ptr[0] = 1234 Traceback (most recent call last): .... ValueError: NULL pointer access >>> Type conversions ---------------- Usually, ctypes does strict type checking. This means, if you have "POINTER(c_int)" in the "argtypes" list of a function or as the type of a member field in a structure definition, only instances of exactly the same type are accepted. There are some exceptions to this rule, where ctypes accepts other objects. For example, you can pass compatible array instances instead of pointer types. So, for "POINTER(c_int)", ctypes accepts an array of c_int: >>> class Bar(Structure): ... _fields_ = [("count", c_int), ("values", POINTER(c_int))] ... >>> bar = Bar() >>> bar.values = (c_int * 3)(1, 2, 3) >>> bar.count = 3 >>> for i in range(bar.count): ... print(bar.values[i]) ... 1 2 3 >>> In addition, if a function argument is explicitly declared to be a pointer type (such as "POINTER(c_int)") in "argtypes", an object of the pointed type ("c_int" in this case) can be passed to the function. ctypes will apply the required "byref()" conversion in this case automatically. To set a POINTER type field to "NULL", you can assign "None": >>> bar.values = None >>> Sometimes you have instances of incompatible types. In C, you can cast one type into another type. "ctypes" provides a "cast()" function which can be used in the same way. The "Bar" structure defined above accepts "POINTER(c_int)" pointers or "c_int" arrays for its "values" field, but not instances of other types: >>> bar.values = (c_byte * 4)() Traceback (most recent call last): File "", line 1, in TypeError: incompatible types, c_byte_Array_4 instance instead of LP_c_long instance >>> For these cases, the "cast()" function is handy. The "cast()" function can be used to cast a ctypes instance into a pointer to a different ctypes data type. "cast()" takes two parameters, a ctypes object that is or can be converted to a pointer of some kind, and a ctypes pointer type. It returns an instance of the second argument, which references the same memory block as the first argument: >>> a = (c_byte * 4)() >>> cast(a, POINTER(c_int)) >>> So, "cast()" can be used to assign to the "values" field of "Bar" the structure: >>> bar = Bar() >>> bar.values = cast((c_byte * 4)(), POINTER(c_int)) >>> print(bar.values[0]) 0 >>> Incomplete Types ---------------- *Incomplete Types* are structures, unions or arrays whose members are not yet specified. In C, they are specified by forward declarations, which are defined later: struct cell; /* forward declaration */ struct cell { char *name; struct cell *next; }; The straightforward translation into ctypes code would be this, but it does not work: >>> class cell(Structure): ... _fields_ = [("name", c_char_p), ... ("next", POINTER(cell))] ... Traceback (most recent call last): File "", line 1, in File "", line 2, in cell NameError: name 'cell' is not defined >>> because the new "class cell" is not available in the class statement itself. In "ctypes", we can define the "cell" class and set the "_fields_" attribute later, after the class statement: >>> from ctypes import * >>> class cell(Structure): ... pass ... >>> cell._fields_ = [("name", c_char_p), ... ("next", POINTER(cell))] >>> Let’s try it. We create two instances of "cell", and let them point to each other, and finally follow the pointer chain a few times: >>> c1 = cell() >>> c1.name = b"foo" >>> c2 = cell() >>> c2.name = b"bar" >>> c1.next = pointer(c2) >>> c2.next = pointer(c1) >>> p = c1 >>> for i in range(8): ... print(p.name, end=" ") ... p = p.next[0] ... foo bar foo bar foo bar foo bar >>> Callback functions ------------------ "ctypes" allows creating C callable function pointers from Python callables. These are sometimes called *callback functions*. First, you must create a class for the callback function. The class knows the calling convention, the return type, and the number and types of arguments this function will receive. The "CFUNCTYPE()" factory function creates types for callback functions using the "cdecl" calling convention. On Windows, the "WINFUNCTYPE()" factory function creates types for callback functions using the "stdcall" calling convention. Both of these factory functions are called with the result type as first argument, and the callback functions expected argument types as the remaining arguments. I will present an example here which uses the standard C library’s "qsort()" function, that is used to sort items with the help of a callback function. "qsort()" will be used to sort an array of integers: >>> IntArray5 = c_int * 5 >>> ia = IntArray5(5, 1, 7, 33, 99) >>> qsort = libc.qsort >>> qsort.restype = None >>> "qsort()" must be called with a pointer to the data to sort, the number of items in the data array, the size of one item, and a pointer to the comparison function, the callback. The callback will then be called with two pointers to items, and it must return a negative integer if the first item is smaller than the second, a zero if they are equal, and a positive integer otherwise. So our callback function receives pointers to integers, and must return an integer. First we create the "type" for the callback function: >>> CMPFUNC = CFUNCTYPE(c_int, POINTER(c_int), POINTER(c_int)) >>> To get started, here is a simple callback that shows the values it gets passed: >>> def py_cmp_func(a, b): ... print("py_cmp_func", a[0], b[0]) ... return 0 ... >>> cmp_func = CMPFUNC(py_cmp_func) >>> The result: >>> qsort(ia, len(ia), sizeof(c_int), cmp_func) py_cmp_func 5 1 py_cmp_func 33 99 py_cmp_func 7 33 py_cmp_func 5 7 py_cmp_func 1 7 >>> Now we can actually compare the two items and return a useful result: >>> def py_cmp_func(a, b): ... print("py_cmp_func", a[0], b[0]) ... return a[0] - b[0] ... >>> >>> qsort(ia, len(ia), sizeof(c_int), CMPFUNC(py_cmp_func)) py_cmp_func 5 1 py_cmp_func 33 99 py_cmp_func 7 33 py_cmp_func 1 7 py_cmp_func 5 7 >>> As we can easily check, our array is sorted now: >>> for i in ia: print(i, end=" ") ... 1 5 7 33 99 >>> The function factories can be used as decorator factories, so we may as well write: >>> @CFUNCTYPE(c_int, POINTER(c_int), POINTER(c_int)) ... def py_cmp_func(a, b): ... print("py_cmp_func", a[0], b[0]) ... return a[0] - b[0] ... >>> qsort(ia, len(ia), sizeof(c_int), py_cmp_func) py_cmp_func 5 1 py_cmp_func 33 99 py_cmp_func 7 33 py_cmp_func 1 7 py_cmp_func 5 7 >>> Note: Make sure you keep references to "CFUNCTYPE()" objects as long as they are used from C code. "ctypes" doesn’t, and if you don’t, they may be garbage collected, crashing your program when a callback is made.Also, note that if the callback function is called in a thread created outside of Python’s control (e.g. by the foreign code that calls the callback), ctypes creates a new dummy Python thread on every invocation. This behavior is correct for most purposes, but it means that values stored with "threading.local" will *not* survive across different callbacks, even when those calls are made from the same C thread. Accessing values exported from dlls ----------------------------------- Some shared libraries not only export functions, they also export variables. An example in the Python library itself is the "Py_Version", Python runtime version number encoded in a single constant integer. "ctypes" can access values like this with the "in_dll()" class methods of the type. *pythonapi* is a predefined symbol giving access to the Python C api: >>> version = ctypes.c_int.in_dll(ctypes.pythonapi, "Py_Version") >>> print(hex(version.value)) 0x30c00a0 An extended example which also demonstrates the use of pointers accesses the "PyImport_FrozenModules" pointer exported by Python. Quoting the docs for that value: This pointer is initialized to point to an array of "_frozen" records, terminated by one whose members are all "NULL" or zero. When a frozen module is imported, it is searched in this table. Third-party code could play tricks with this to provide a dynamically created collection of frozen modules. So manipulating this pointer could even prove useful. To restrict the example size, we show only how this table can be read with "ctypes": >>> from ctypes import * >>> >>> class struct_frozen(Structure): ... _fields_ = [("name", c_char_p), ... ("code", POINTER(c_ubyte)), ... ("size", c_int), ... ("get_code", POINTER(c_ubyte)), # Function pointer ... ] ... >>> We have defined the "_frozen" data type, so we can get the pointer to the table: >>> FrozenTable = POINTER(struct_frozen) >>> table = FrozenTable.in_dll(pythonapi, "_PyImport_FrozenBootstrap") >>> Since "table" is a "pointer" to the array of "struct_frozen" records, we can iterate over it, but we just have to make sure that our loop terminates, because pointers have no size. Sooner or later it would probably crash with an access violation or whatever, so it’s better to break out of the loop when we hit the "NULL" entry: >>> for item in table: ... if item.name is None: ... break ... print(item.name.decode("ascii"), item.size) ... _frozen_importlib 31764 _frozen_importlib_external 41499 zipimport 12345 >>> The fact that standard Python has a frozen module and a frozen package (indicated by the negative "size" member) is not well known, it is only used for testing. Try it out with "import __hello__" for example. Surprises --------- There are some edges in "ctypes" where you might expect something other than what actually happens. Consider the following example: >>> from ctypes import * >>> class POINT(Structure): ... _fields_ = ("x", c_int), ("y", c_int) ... >>> class RECT(Structure): ... _fields_ = ("a", POINT), ("b", POINT) ... >>> p1 = POINT(1, 2) >>> p2 = POINT(3, 4) >>> rc = RECT(p1, p2) >>> print(rc.a.x, rc.a.y, rc.b.x, rc.b.y) 1 2 3 4 >>> # now swap the two points >>> rc.a, rc.b = rc.b, rc.a >>> print(rc.a.x, rc.a.y, rc.b.x, rc.b.y) 3 4 3 4 >>> Hm. We certainly expected the last statement to print "3 4 1 2". What happened? Here are the steps of the "rc.a, rc.b = rc.b, rc.a" line above: >>> temp0, temp1 = rc.b, rc.a >>> rc.a = temp0 >>> rc.b = temp1 >>> Note that "temp0" and "temp1" are objects still using the internal buffer of the "rc" object above. So executing "rc.a = temp0" copies the buffer contents of "temp0" into "rc" ‘s buffer. This, in turn, changes the contents of "temp1". So, the last assignment "rc.b = temp1", doesn’t have the expected effect. Keep in mind that retrieving sub-objects from Structure, Unions, and Arrays doesn’t *copy* the sub-object, instead it retrieves a wrapper object accessing the root-object’s underlying buffer. Another example that may behave differently from what one would expect is this: >>> s = c_char_p() >>> s.value = b"abc def ghi" >>> s.value b'abc def ghi' >>> s.value is s.value False >>> Note: Objects instantiated from "c_char_p" can only have their value set to bytes or integers. Why is it printing "False"? ctypes instances are objects containing a memory block plus some *descriptor*s accessing the contents of the memory. Storing a Python object in the memory block does not store the object itself, instead the "contents" of the object is stored. Accessing the contents again constructs a new Python object each time! Variable-sized data types ------------------------- "ctypes" provides some support for variable-sized arrays and structures. The "resize()" function can be used to resize the memory buffer of an existing ctypes object. The function takes the object as first argument, and the requested size in bytes as the second argument. The memory block cannot be made smaller than the natural memory block specified by the objects type, a "ValueError" is raised if this is tried: >>> short_array = (c_short * 4)() >>> print(sizeof(short_array)) 8 >>> resize(short_array, 4) Traceback (most recent call last): ... ValueError: minimum size is 8 >>> resize(short_array, 32) >>> sizeof(short_array) 32 >>> sizeof(type(short_array)) 8 >>> This is nice and fine, but how would one access the additional elements contained in this array? Since the type still only knows about 4 elements, we get errors accessing other elements: >>> short_array[:] [0, 0, 0, 0] >>> short_array[7] Traceback (most recent call last): ... IndexError: invalid index >>> Another way to use variable-sized data types with "ctypes" is to use the dynamic nature of Python, and (re-)define the data type after the required size is already known, on a case by case basis. ctypes reference ================ Finding shared libraries ------------------------ When programming in a compiled language, shared libraries are accessed when compiling/linking a program, and when the program is run. The purpose of the "find_library()" function is to locate a library in a way similar to what the compiler or runtime loader does (on platforms with several versions of a shared library the most recent should be loaded), while the ctypes library loaders act like when a program is run, and call the runtime loader directly. The "ctypes.util" module provides a function which can help to determine the library to load. ctypes.util.find_library(name) Try to find a library and return a pathname. *name* is the library name without any prefix like *lib*, suffix like ".so", ".dylib" or version number (this is the form used for the posix linker option "-l"). If no library can be found, returns "None". The exact functionality is system dependent. On Linux, "find_library()" tries to run external programs ("/sbin/ldconfig", "gcc", "objdump" and "ld") to find the library file. It returns the filename of the library file. Changed in version 3.6: On Linux, the value of the environment variable "LD_LIBRARY_PATH" is used when searching for libraries, if a library cannot be found by any other means. Here are some examples: >>> from ctypes.util import find_library >>> find_library("m") 'libm.so.6' >>> find_library("c") 'libc.so.6' >>> find_library("bz2") 'libbz2.so.1.0' >>> On macOS and Android, "find_library()" uses the system’s standard naming schemes and paths to locate the library, and returns a full pathname if successful: >>> from ctypes.util import find_library >>> find_library("c") '/usr/lib/libc.dylib' >>> find_library("m") '/usr/lib/libm.dylib' >>> find_library("bz2") '/usr/lib/libbz2.dylib' >>> find_library("AGL") '/System/Library/Frameworks/AGL.framework/AGL' >>> On Windows, "find_library()" searches along the system search path, and returns the full pathname, but since there is no predefined naming scheme a call like "find_library("c")" will fail and return "None". If wrapping a shared library with "ctypes", it *may* be better to determine the shared library name at development time, and hardcode that into the wrapper module instead of using "find_library()" to locate the library at runtime. Loading shared libraries ------------------------ There are several ways to load shared libraries into the Python process. One way is to instantiate one of the following classes: class ctypes.CDLL(name, mode=DEFAULT_MODE, handle=None, use_errno=False, use_last_error=False, winmode=None) Instances of this class represent loaded shared libraries. Functions in these libraries use the standard C calling convention, and are assumed to return int. On Windows creating a "CDLL" instance may fail even if the DLL name exists. When a dependent DLL of the loaded DLL is not found, a "OSError" error is raised with the message *“[WinError 126] The specified module could not be found”.* This error message does not contain the name of the missing DLL because the Windows API does not return this information making this error hard to diagnose. To resolve this error and determine which DLL is not found, you need to find the list of dependent DLLs and determine which one is not found using Windows debugging and tracing tools. Changed in version 3.12: The *name* parameter can now be a *path- like object*. See also: Microsoft DUMPBIN tool – A tool to find DLL dependents. class ctypes.OleDLL(name, mode=DEFAULT_MODE, handle=None, use_errno=False, use_last_error=False, winmode=None) Instances of this class represent loaded shared libraries, functions in these libraries use the "stdcall" calling convention, and are assumed to return the windows specific "HRESULT" code. "HRESULT" values contain information specifying whether the function call failed or succeeded, together with additional error code. If the return value signals a failure, an "OSError" is automatically raised. Availability: Windows Changed in version 3.3: "WindowsError" used to be raised, which is now an alias of "OSError". Changed in version 3.12: The *name* parameter can now be a *path- like object*. class ctypes.WinDLL(name, mode=DEFAULT_MODE, handle=None, use_errno=False, use_last_error=False, winmode=None) Instances of this class represent loaded shared libraries, functions in these libraries use the "stdcall" calling convention, and are assumed to return int by default. Availability: Windows Changed in version 3.12: The *name* parameter can now be a *path- like object*. The Python *global interpreter lock* is released before calling any function exported by these libraries, and reacquired afterwards. class ctypes.PyDLL(name, mode=DEFAULT_MODE, handle=None) Instances of this class behave like "CDLL" instances, except that the Python GIL is *not* released during the function call, and after the function execution the Python error flag is checked. If the error flag is set, a Python exception is raised. Thus, this is only useful to call Python C api functions directly. Changed in version 3.12: The *name* parameter can now be a *path- like object*. All these classes can be instantiated by calling them with at least one argument, the pathname of the shared library. If you have an existing handle to an already loaded shared library, it can be passed as the "handle" named parameter, otherwise the underlying platform’s "dlopen()" or "LoadLibrary()" function is used to load the library into the process, and to get a handle to it. The *mode* parameter can be used to specify how the library is loaded. For details, consult the *dlopen(3)* manpage. On Windows, *mode* is ignored. On posix systems, RTLD_NOW is always added, and is not configurable. The *use_errno* parameter, when set to true, enables a ctypes mechanism that allows accessing the system "errno" error number in a safe way. "ctypes" maintains a thread-local copy of the system’s "errno" variable; if you call foreign functions created with "use_errno=True" then the "errno" value before the function call is swapped with the ctypes private copy, the same happens immediately after the function call. The function "ctypes.get_errno()" returns the value of the ctypes private copy, and the function "ctypes.set_errno()" changes the ctypes private copy to a new value and returns the former value. The *use_last_error* parameter, when set to true, enables the same mechanism for the Windows error code which is managed by the "GetLastError()" and "SetLastError()" Windows API functions; "ctypes.get_last_error()" and "ctypes.set_last_error()" are used to request and change the ctypes private copy of the windows error code. The *winmode* parameter is used on Windows to specify how the library is loaded (since *mode* is ignored). It takes any value that is valid for the Win32 API "LoadLibraryEx" flags parameter. When omitted, the default is to use the flags that result in the most secure DLL load, which avoids issues such as DLL hijacking. Passing the full path to the DLL is the safest way to ensure the correct library and dependencies are loaded. Changed in version 3.8: Added *winmode* parameter. ctypes.RTLD_GLOBAL Flag to use as *mode* parameter. On platforms where this flag is not available, it is defined as the integer zero. ctypes.RTLD_LOCAL Flag to use as *mode* parameter. On platforms where this is not available, it is the same as *RTLD_GLOBAL*. ctypes.DEFAULT_MODE The default mode which is used to load shared libraries. On OSX 10.3, this is *RTLD_GLOBAL*, otherwise it is the same as *RTLD_LOCAL*. Instances of these classes have no public methods. Functions exported by the shared library can be accessed as attributes or by index. Please note that accessing the function through an attribute caches the result and therefore accessing it repeatedly returns the same object each time. On the other hand, accessing it through an index returns a new object each time: >>> from ctypes import CDLL >>> libc = CDLL("libc.so.6") # On Linux >>> libc.time == libc.time True >>> libc['time'] == libc['time'] False The following public attributes are available, their name starts with an underscore to not clash with exported function names: PyDLL._handle The system handle used to access the library. PyDLL._name The name of the library passed in the constructor. Shared libraries can also be loaded by using one of the prefabricated objects, which are instances of the "LibraryLoader" class, either by calling the "LoadLibrary()" method, or by retrieving the library as attribute of the loader instance. class ctypes.LibraryLoader(dlltype) Class which loads shared libraries. *dlltype* should be one of the "CDLL", "PyDLL", "WinDLL", or "OleDLL" types. "__getattr__()" has special behavior: It allows loading a shared library by accessing it as attribute of a library loader instance. The result is cached, so repeated attribute accesses return the same library each time. LoadLibrary(name) Load a shared library into the process and return it. This method always returns a new instance of the library. These prefabricated library loaders are available: ctypes.cdll Creates "CDLL" instances. ctypes.windll Creates "WinDLL" instances. Availability: Windows ctypes.oledll Creates "OleDLL" instances. Availability: Windows ctypes.pydll Creates "PyDLL" instances. For accessing the C Python api directly, a ready-to-use Python shared library object is available: ctypes.pythonapi An instance of "PyDLL" that exposes Python C API functions as attributes. Note that all these functions are assumed to return C int, which is of course not always the truth, so you have to assign the correct "restype" attribute to use these functions. Loading a library through any of these objects raises an auditing event "ctypes.dlopen" with string argument "name", the name used to load the library. Accessing a function on a loaded library raises an auditing event "ctypes.dlsym" with arguments "library" (the library object) and "name" (the symbol’s name as a string or integer). In cases when only the library handle is available rather than the object, accessing a function raises an auditing event "ctypes.dlsym/handle" with arguments "handle" (the raw library handle) and "name". Foreign functions ----------------- As explained in the previous section, foreign functions can be accessed as attributes of loaded shared libraries. The function objects created in this way by default accept any number of arguments, accept any ctypes data instances as arguments, and return the default result type specified by the library loader. They are instances of a private local class "_FuncPtr" (not exposed in "ctypes") which inherits from the private "_CFuncPtr" class: >>> import ctypes >>> lib = ctypes.CDLL(None) >>> issubclass(lib._FuncPtr, ctypes._CFuncPtr) True >>> lib._FuncPtr is ctypes._CFuncPtr False class ctypes._CFuncPtr Base class for C callable foreign functions. Instances of foreign functions are also C compatible data types; they represent C function pointers. This behavior can be customized by assigning to special attributes of the foreign function object. restype Assign a ctypes type to specify the result type of the foreign function. Use "None" for void, a function not returning anything. It is possible to assign a callable Python object that is not a ctypes type, in this case the function is assumed to return a C int, and the callable will be called with this integer, allowing further processing or error checking. Using this is deprecated, for more flexible post processing or error checking use a ctypes data type as "restype" and assign a callable to the "errcheck" attribute. argtypes Assign a tuple of ctypes types to specify the argument types that the function accepts. Functions using the "stdcall" calling convention can only be called with the same number of arguments as the length of this tuple; functions using the C calling convention accept additional, unspecified arguments as well. When a foreign function is called, each actual argument is passed to the "from_param()" class method of the items in the "argtypes" tuple, this method allows adapting the actual argument to an object that the foreign function accepts. For example, a "c_char_p" item in the "argtypes" tuple will convert a string passed as argument into a bytes object using ctypes conversion rules. New: It is now possible to put items in argtypes which are not ctypes types, but each item must have a "from_param()" method which returns a value usable as argument (integer, string, ctypes instance). This allows defining adapters that can adapt custom objects as function parameters. errcheck Assign a Python function or another callable to this attribute. The callable will be called with three or more arguments: callable(result, func, arguments) *result* is what the foreign function returns, as specified by the "restype" attribute. *func* is the foreign function object itself, this allows reusing the same callable object to check or post process the results of several functions. *arguments* is a tuple containing the parameters originally passed to the function call, this allows specializing the behavior on the arguments used. The object that this function returns will be returned from the foreign function call, but it can also check the result value and raise an exception if the foreign function call failed. exception ctypes.ArgumentError This exception is raised when a foreign function call cannot convert one of the passed arguments. On Windows, when a foreign function call raises a system exception (for example, due to an access violation), it will be captured and replaced with a suitable Python exception. Further, an auditing event "ctypes.set_exception" with argument "code" will be raised, allowing an audit hook to replace the exception with its own. Some ways to invoke foreign function calls may raise an auditing event "ctypes.call_function" with arguments "function pointer" and "arguments". Function prototypes ------------------- Foreign functions can also be created by instantiating function prototypes. Function prototypes are similar to function prototypes in C; they describe a function (return type, argument types, calling convention) without defining an implementation. The factory functions must be called with the desired result type and the argument types of the function, and can be used as decorator factories, and as such, be applied to functions through the "@wrapper" syntax. See Callback functions for examples. ctypes.CFUNCTYPE(restype, *argtypes, use_errno=False, use_last_error=False) The returned function prototype creates functions that use the standard C calling convention. The function will release the GIL during the call. If *use_errno* is set to true, the ctypes private copy of the system "errno" variable is exchanged with the real "errno" value before and after the call; *use_last_error* does the same for the Windows error code. ctypes.WINFUNCTYPE(restype, *argtypes, use_errno=False, use_last_error=False) The returned function prototype creates functions that use the "stdcall" calling convention. The function will release the GIL during the call. *use_errno* and *use_last_error* have the same meaning as above. Availability: Windows ctypes.PYFUNCTYPE(restype, *argtypes) The returned function prototype creates functions that use the Python calling convention. The function will *not* release the GIL during the call. Function prototypes created by these factory functions can be instantiated in different ways, depending on the type and number of the parameters in the call: prototype(address) Returns a foreign function at the specified address which must be an integer. prototype(callable) Create a C callable function (a callback function) from a Python *callable*. prototype(func_spec[, paramflags]) Returns a foreign function exported by a shared library. *func_spec* must be a 2-tuple "(name_or_ordinal, library)". The first item is the name of the exported function as string, or the ordinal of the exported function as small integer. The second item is the shared library instance. prototype(vtbl_index, name[, paramflags[, iid]]) Returns a foreign function that will call a COM method. *vtbl_index* is the index into the virtual function table, a small non-negative integer. *name* is name of the COM method. *iid* is an optional pointer to the interface identifier which is used in extended error reporting. COM methods use a special calling convention: They require a pointer to the COM interface as first argument, in addition to those parameters that are specified in the "argtypes" tuple. The optional *paramflags* parameter creates foreign function wrappers with much more functionality than the features described above. *paramflags* must be a tuple of the same length as "argtypes". Each item in this tuple contains further information about a parameter, it must be a tuple containing one, two, or three items. The first item is an integer containing a combination of direction flags for the parameter: 1 Specifies an input parameter to the function. 2 Output parameter. The foreign function fills in a value. 4 Input parameter which defaults to the integer zero. The optional second item is the parameter name as string. If this is specified, the foreign function can be called with named parameters. The optional third item is the default value for this parameter. The following example demonstrates how to wrap the Windows "MessageBoxW" function so that it supports default parameters and named arguments. The C declaration from the windows header file is this: WINUSERAPI int WINAPI MessageBoxW( HWND hWnd, LPCWSTR lpText, LPCWSTR lpCaption, UINT uType); Here is the wrapping with "ctypes": >>> from ctypes import c_int, WINFUNCTYPE, windll >>> from ctypes.wintypes import HWND, LPCWSTR, UINT >>> prototype = WINFUNCTYPE(c_int, HWND, LPCWSTR, LPCWSTR, UINT) >>> paramflags = (1, "hwnd", 0), (1, "text", "Hi"), (1, "caption", "Hello from ctypes"), (1, "flags", 0) >>> MessageBox = prototype(("MessageBoxW", windll.user32), paramflags) The "MessageBox" foreign function can now be called in these ways: >>> MessageBox() >>> MessageBox(text="Spam, spam, spam") >>> MessageBox(flags=2, text="foo bar") A second example demonstrates output parameters. The win32 "GetWindowRect" function retrieves the dimensions of a specified window by copying them into "RECT" structure that the caller has to supply. Here is the C declaration: WINUSERAPI BOOL WINAPI GetWindowRect( HWND hWnd, LPRECT lpRect); Here is the wrapping with "ctypes": >>> from ctypes import POINTER, WINFUNCTYPE, windll, WinError >>> from ctypes.wintypes import BOOL, HWND, RECT >>> prototype = WINFUNCTYPE(BOOL, HWND, POINTER(RECT)) >>> paramflags = (1, "hwnd"), (2, "lprect") >>> GetWindowRect = prototype(("GetWindowRect", windll.user32), paramflags) >>> Functions with output parameters will automatically return the output parameter value if there is a single one, or a tuple containing the output parameter values when there are more than one, so the GetWindowRect function now returns a RECT instance, when called. Output parameters can be combined with the "errcheck" protocol to do further output processing and error checking. The win32 "GetWindowRect" api function returns a "BOOL" to signal success or failure, so this function could do the error checking, and raises an exception when the api call failed: >>> def errcheck(result, func, args): ... if not result: ... raise WinError() ... return args ... >>> GetWindowRect.errcheck = errcheck >>> If the "errcheck" function returns the argument tuple it receives unchanged, "ctypes" continues the normal processing it does on the output parameters. If you want to return a tuple of window coordinates instead of a "RECT" instance, you can retrieve the fields in the function and return them instead, the normal processing will no longer take place: >>> def errcheck(result, func, args): ... if not result: ... raise WinError() ... rc = args[1] ... return rc.left, rc.top, rc.bottom, rc.right ... >>> GetWindowRect.errcheck = errcheck >>> Utility functions ----------------- ctypes.addressof(obj) Returns the address of the memory buffer as integer. *obj* must be an instance of a ctypes type. Raises an auditing event "ctypes.addressof" with argument "obj". ctypes.alignment(obj_or_type) Returns the alignment requirements of a ctypes type. *obj_or_type* must be a ctypes type or instance. ctypes.byref(obj[, offset]) Returns a light-weight pointer to *obj*, which must be an instance of a ctypes type. *offset* defaults to zero, and must be an integer that will be added to the internal pointer value. "byref(obj, offset)" corresponds to this C code: (((char *)&obj) + offset) The returned object can only be used as a foreign function call parameter. It behaves similar to "pointer(obj)", but the construction is a lot faster. ctypes.cast(obj, type) This function is similar to the cast operator in C. It returns a new instance of *type* which points to the same memory block as *obj*. *type* must be a pointer type, and *obj* must be an object that can be interpreted as a pointer. ctypes.create_string_buffer(init, size=None) ctypes.create_string_buffer(size) This function creates a mutable character buffer. The returned object is a ctypes array of "c_char". If *size* is given (and not "None"), it must be an "int". It specifies the size of the returned array. If the *init* argument is given, it must be "bytes". It is used to initialize the array items. Bytes not initialized this way are set to zero (NUL). If *size* is not given (or if it is "None"), the buffer is made one element larger than *init*, effectively adding a NUL terminator. If both arguments are given, *size* must not be less than "len(init)". Warning: If *size* is equal to "len(init)", a NUL terminator is not added. Do not treat such a buffer as a C string. For example: >>> bytes(create_string_buffer(2)) b'\x00\x00' >>> bytes(create_string_buffer(b'ab')) b'ab\x00' >>> bytes(create_string_buffer(b'ab', 2)) b'ab' >>> bytes(create_string_buffer(b'ab', 4)) b'ab\x00\x00' >>> bytes(create_string_buffer(b'abcdef', 2)) Traceback (most recent call last): ... ValueError: byte string too long Raises an auditing event "ctypes.create_string_buffer" with arguments "init", "size". ctypes.create_unicode_buffer(init, size=None) ctypes.create_unicode_buffer(size) This function creates a mutable unicode character buffer. The returned object is a ctypes array of "c_wchar". The function takes the same arguments as "create_string_buffer()" except *init* must be a string and *size* counts "c_wchar". Raises an auditing event "ctypes.create_unicode_buffer" with arguments "init", "size". ctypes.DllCanUnloadNow() This function is a hook which allows implementing in-process COM servers with ctypes. It is called from the DllCanUnloadNow function that the _ctypes extension dll exports. Availability: Windows ctypes.DllGetClassObject() This function is a hook which allows implementing in-process COM servers with ctypes. It is called from the DllGetClassObject function that the "_ctypes" extension dll exports. Availability: Windows ctypes.util.find_library(name) Try to find a library and return a pathname. *name* is the library name without any prefix like "lib", suffix like ".so", ".dylib" or version number (this is the form used for the posix linker option "-l"). If no library can be found, returns "None". The exact functionality is system dependent. ctypes.util.find_msvcrt() Returns the filename of the VC runtime library used by Python, and by the extension modules. If the name of the library cannot be determined, "None" is returned. If you need to free memory, for example, allocated by an extension module with a call to the "free(void *)", it is important that you use the function in the same library that allocated the memory. Availability: Windows ctypes.FormatError([code]) Returns a textual description of the error code *code*. If no error code is specified, the last error code is used by calling the Windows api function GetLastError. Availability: Windows ctypes.GetLastError() Returns the last error code set by Windows in the calling thread. This function calls the Windows "GetLastError()" function directly, it does not return the ctypes-private copy of the error code. Availability: Windows ctypes.get_errno() Returns the current value of the ctypes-private copy of the system "errno" variable in the calling thread. Raises an auditing event "ctypes.get_errno" with no arguments. ctypes.get_last_error() Returns the current value of the ctypes-private copy of the system "LastError" variable in the calling thread. Availability: Windows Raises an auditing event "ctypes.get_last_error" with no arguments. ctypes.memmove(dst, src, count) Same as the standard C memmove library function: copies *count* bytes from *src* to *dst*. *dst* and *src* must be integers or ctypes instances that can be converted to pointers. ctypes.memset(dst, c, count) Same as the standard C memset library function: fills the memory block at address *dst* with *count* bytes of value *c*. *dst* must be an integer specifying an address, or a ctypes instance. ctypes.POINTER(type, /) Create and return a new ctypes pointer type. Pointer types are cached and reused internally, so calling this function repeatedly is cheap. *type* must be a ctypes type. ctypes.pointer(obj, /) Create a new pointer instance, pointing to *obj*. The returned object is of the type "POINTER(type(obj))". Note: If you just want to pass a pointer to an object to a foreign function call, you should use "byref(obj)" which is much faster. ctypes.resize(obj, size) This function resizes the internal memory buffer of *obj*, which must be an instance of a ctypes type. It is not possible to make the buffer smaller than the native size of the objects type, as given by "sizeof(type(obj))", but it is possible to enlarge the buffer. ctypes.set_errno(value) Set the current value of the ctypes-private copy of the system "errno" variable in the calling thread to *value* and return the previous value. Raises an auditing event "ctypes.set_errno" with argument "errno". ctypes.set_last_error(value) Sets the current value of the ctypes-private copy of the system "LastError" variable in the calling thread to *value* and return the previous value. Availability: Windows Raises an auditing event "ctypes.set_last_error" with argument "error". ctypes.sizeof(obj_or_type) Returns the size in bytes of a ctypes type or instance memory buffer. Does the same as the C "sizeof" operator. ctypes.string_at(ptr, size=-1) Return the byte string at *void *ptr*. If *size* is specified, it is used as size, otherwise the string is assumed to be zero- terminated. Raises an auditing event "ctypes.string_at" with arguments "ptr", "size". ctypes.WinError(code=None, descr=None) This function is probably the worst-named thing in ctypes. It creates an instance of "OSError". If *code* is not specified, "GetLastError" is called to determine the error code. If *descr* is not specified, "FormatError()" is called to get a textual description of the error. Availability: Windows Changed in version 3.3: An instance of "WindowsError" used to be created, which is now an alias of "OSError". ctypes.wstring_at(ptr, size=-1) Return the wide-character string at *void *ptr*. If *size* is specified, it is used as the number of characters of the string, otherwise the string is assumed to be zero-terminated. Raises an auditing event "ctypes.wstring_at" with arguments "ptr", "size". Data types ---------- class ctypes._CData This non-public class is the common base class of all ctypes data types. Among other things, all ctypes type instances contain a memory block that hold C compatible data; the address of the memory block is returned by the "addressof()" helper function. Another instance variable is exposed as "_objects"; this contains other Python objects that need to be kept alive in case the memory block contains pointers. Common methods of ctypes data types, these are all class methods (to be exact, they are methods of the *metaclass*): from_buffer(source[, offset]) This method returns a ctypes instance that shares the buffer of the *source* object. The *source* object must support the writeable buffer interface. The optional *offset* parameter specifies an offset into the source buffer in bytes; the default is zero. If the source buffer is not large enough a "ValueError" is raised. Raises an auditing event "ctypes.cdata/buffer" with arguments "pointer", "size", "offset". from_buffer_copy(source[, offset]) This method creates a ctypes instance, copying the buffer from the *source* object buffer which must be readable. The optional *offset* parameter specifies an offset into the source buffer in bytes; the default is zero. If the source buffer is not large enough a "ValueError" is raised. Raises an auditing event "ctypes.cdata/buffer" with arguments "pointer", "size", "offset". from_address(address) This method returns a ctypes type instance using the memory specified by *address* which must be an integer. This method, and others that indirectly call this method, raises an auditing event "ctypes.cdata" with argument "address". from_param(obj) This method adapts *obj* to a ctypes type. It is called with the actual object used in a foreign function call when the type is present in the foreign function’s "argtypes" tuple; it must return an object that can be used as a function call parameter. All ctypes data types have a default implementation of this classmethod that normally returns *obj* if that is an instance of the type. Some types accept other objects as well. in_dll(library, name) This method returns a ctypes type instance exported by a shared library. *name* is the name of the symbol that exports the data, *library* is the loaded shared library. Common instance variables of ctypes data types: _b_base_ Sometimes ctypes data instances do not own the memory block they contain, instead they share part of the memory block of a base object. The "_b_base_" read-only member is the root ctypes object that owns the memory block. _b_needsfree_ This read-only variable is true when the ctypes data instance has allocated the memory block itself, false otherwise. _objects This member is either "None" or a dictionary containing Python objects that need to be kept alive so that the memory block contents is kept valid. This object is only exposed for debugging; never modify the contents of this dictionary. Fundamental data types ---------------------- class ctypes._SimpleCData This non-public class is the base class of all fundamental ctypes data types. It is mentioned here because it contains the common attributes of the fundamental ctypes data types. "_SimpleCData" is a subclass of "_CData", so it inherits their methods and attributes. ctypes data types that are not and do not contain pointers can now be pickled. Instances have a single attribute: value This attribute contains the actual value of the instance. For integer and pointer types, it is an integer, for character types, it is a single character bytes object or string, for character pointer types it is a Python bytes object or string. When the "value" attribute is retrieved from a ctypes instance, usually a new object is returned each time. "ctypes" does *not* implement original object return, always a new object is constructed. The same is true for all other ctypes object instances. Fundamental data types, when returned as foreign function call results, or, for example, by retrieving structure field members or array items, are transparently converted to native Python types. In other words, if a foreign function has a "restype" of "c_char_p", you will always receive a Python bytes object, *not* a "c_char_p" instance. Subclasses of fundamental data types do *not* inherit this behavior. So, if a foreign functions "restype" is a subclass of "c_void_p", you will receive an instance of this subclass from the function call. Of course, you can get the value of the pointer by accessing the "value" attribute. These are the fundamental ctypes data types: class ctypes.c_byte Represents the C signed char datatype, and interprets the value as small integer. The constructor accepts an optional integer initializer; no overflow checking is done. class ctypes.c_char Represents the C char datatype, and interprets the value as a single character. The constructor accepts an optional string initializer, the length of the string must be exactly one character. class ctypes.c_char_p Represents the C char* datatype when it points to a zero-terminated string. For a general character pointer that may also point to binary data, "POINTER(c_char)" must be used. The constructor accepts an integer address, or a bytes object. class ctypes.c_double Represents the C double datatype. The constructor accepts an optional float initializer. class ctypes.c_longdouble Represents the C long double datatype. The constructor accepts an optional float initializer. On platforms where "sizeof(long double) == sizeof(double)" it is an alias to "c_double". class ctypes.c_float Represents the C float datatype. The constructor accepts an optional float initializer. class ctypes.c_int Represents the C signed int datatype. The constructor accepts an optional integer initializer; no overflow checking is done. On platforms where "sizeof(int) == sizeof(long)" it is an alias to "c_long". class ctypes.c_int8 Represents the C 8-bit signed int datatype. It is an alias for "c_byte". class ctypes.c_int16 Represents the C 16-bit signed int datatype. Usually an alias for "c_short". class ctypes.c_int32 Represents the C 32-bit signed int datatype. Usually an alias for "c_int". class ctypes.c_int64 Represents the C 64-bit signed int datatype. Usually an alias for "c_longlong". class ctypes.c_long Represents the C signed long datatype. The constructor accepts an optional integer initializer; no overflow checking is done. class ctypes.c_longlong Represents the C signed long long datatype. The constructor accepts an optional integer initializer; no overflow checking is done. class ctypes.c_short Represents the C signed short datatype. The constructor accepts an optional integer initializer; no overflow checking is done. class ctypes.c_size_t Represents the C "size_t" datatype. class ctypes.c_ssize_t Represents the C "ssize_t" datatype. Added in version 3.2. class ctypes.c_time_t Represents the C "time_t" datatype. Added in version 3.12. class ctypes.c_ubyte Represents the C unsigned char datatype, it interprets the value as small integer. The constructor accepts an optional integer initializer; no overflow checking is done. class ctypes.c_uint Represents the C unsigned int datatype. The constructor accepts an optional integer initializer; no overflow checking is done. On platforms where "sizeof(int) == sizeof(long)" it is an alias for "c_ulong". class ctypes.c_uint8 Represents the C 8-bit unsigned int datatype. It is an alias for "c_ubyte". class ctypes.c_uint16 Represents the C 16-bit unsigned int datatype. Usually an alias for "c_ushort". class ctypes.c_uint32 Represents the C 32-bit unsigned int datatype. Usually an alias for "c_uint". class ctypes.c_uint64 Represents the C 64-bit unsigned int datatype. Usually an alias for "c_ulonglong". class ctypes.c_ulong Represents the C unsigned long datatype. The constructor accepts an optional integer initializer; no overflow checking is done. class ctypes.c_ulonglong Represents the C unsigned long long datatype. The constructor accepts an optional integer initializer; no overflow checking is done. class ctypes.c_ushort Represents the C unsigned short datatype. The constructor accepts an optional integer initializer; no overflow checking is done. class ctypes.c_void_p Represents the C void* type. The value is represented as integer. The constructor accepts an optional integer initializer. class ctypes.c_wchar Represents the C "wchar_t" datatype, and interprets the value as a single character unicode string. The constructor accepts an optional string initializer, the length of the string must be exactly one character. class ctypes.c_wchar_p Represents the C wchar_t* datatype, which must be a pointer to a zero-terminated wide character string. The constructor accepts an integer address, or a string. class ctypes.c_bool Represent the C bool datatype (more accurately, _Bool from C99). Its value can be "True" or "False", and the constructor accepts any object that has a truth value. class ctypes.HRESULT Represents a "HRESULT" value, which contains success or error information for a function or method call. Availability: Windows class ctypes.py_object Represents the C PyObject* datatype. Calling this without an argument creates a "NULL" PyObject* pointer. The "ctypes.wintypes" module provides quite some other Windows specific data types, for example "HWND", "WPARAM", or "DWORD". Some useful structures like "MSG" or "RECT" are also defined. Structured data types --------------------- class ctypes.Union(*args, **kw) Abstract base class for unions in native byte order. class ctypes.BigEndianUnion(*args, **kw) Abstract base class for unions in *big endian* byte order. Added in version 3.11. class ctypes.LittleEndianUnion(*args, **kw) Abstract base class for unions in *little endian* byte order. Added in version 3.11. class ctypes.BigEndianStructure(*args, **kw) Abstract base class for structures in *big endian* byte order. class ctypes.LittleEndianStructure(*args, **kw) Abstract base class for structures in *little endian* byte order. Structures and unions with non-native byte order cannot contain pointer type fields, or any other data types containing pointer type fields. class ctypes.Structure(*args, **kw) Abstract base class for structures in *native* byte order. Concrete structure and union types must be created by subclassing one of these types, and at least define a "_fields_" class variable. "ctypes" will create *descriptor*s which allow reading and writing the fields by direct attribute accesses. These are the _fields_ A sequence defining the structure fields. The items must be 2-tuples or 3-tuples. The first item is the name of the field, the second item specifies the type of the field; it can be any ctypes data type. For integer type fields like "c_int", a third optional item can be given. It must be a small positive integer defining the bit width of the field. Field names must be unique within one structure or union. This is not checked, only one field can be accessed when names are repeated. It is possible to define the "_fields_" class variable *after* the class statement that defines the Structure subclass, this allows creating data types that directly or indirectly reference themselves: class List(Structure): pass List._fields_ = [("pnext", POINTER(List)), ... ] The "_fields_" class variable must, however, be defined before the type is first used (an instance is created, "sizeof()" is called on it, and so on). Later assignments to the "_fields_" class variable will raise an AttributeError. It is possible to define sub-subclasses of structure types, they inherit the fields of the base class plus the "_fields_" defined in the sub-subclass, if any. _pack_ An optional small integer that allows overriding the alignment of structure fields in the instance. "_pack_" must already be defined when "_fields_" is assigned, otherwise it will have no effect. Setting this attribute to 0 is the same as not setting it at all. _align_ An optional small integer that allows overriding the alignment of the structure when being packed or unpacked to/from memory. Setting this attribute to 0 is the same as not setting it at all. Added in version 3.13. _anonymous_ An optional sequence that lists the names of unnamed (anonymous) fields. "_anonymous_" must be already defined when "_fields_" is assigned, otherwise it will have no effect. The fields listed in this variable must be structure or union type fields. "ctypes" will create descriptors in the structure type that allows accessing the nested fields directly, without the need to create the structure or union field. Here is an example type (Windows): class _U(Union): _fields_ = [("lptdesc", POINTER(TYPEDESC)), ("lpadesc", POINTER(ARRAYDESC)), ("hreftype", HREFTYPE)] class TYPEDESC(Structure): _anonymous_ = ("u",) _fields_ = [("u", _U), ("vt", VARTYPE)] The "TYPEDESC" structure describes a COM data type, the "vt" field specifies which one of the union fields is valid. Since the "u" field is defined as anonymous field, it is now possible to access the members directly off the TYPEDESC instance. "td.lptdesc" and "td.u.lptdesc" are equivalent, but the former is faster since it does not need to create a temporary union instance: td = TYPEDESC() td.vt = VT_PTR td.lptdesc = POINTER(some_type) td.u.lptdesc = POINTER(some_type) It is possible to define sub-subclasses of structures, they inherit the fields of the base class. If the subclass definition has a separate "_fields_" variable, the fields specified in this are appended to the fields of the base class. Structure and union constructors accept both positional and keyword arguments. Positional arguments are used to initialize member fields in the same order as they are appear in "_fields_". Keyword arguments in the constructor are interpreted as attribute assignments, so they will initialize "_fields_" with the same name, or create new attributes for names not present in "_fields_". Arrays and pointers ------------------- class ctypes.Array(*args) Abstract base class for arrays. The recommended way to create concrete array types is by multiplying any "ctypes" data type with a non-negative integer. Alternatively, you can subclass this type and define "_length_" and "_type_" class variables. Array elements can be read and written using standard subscript and slice accesses; for slice reads, the resulting object is *not* itself an "Array". _length_ A positive integer specifying the number of elements in the array. Out-of-range subscripts result in an "IndexError". Will be returned by "len()". _type_ Specifies the type of each element in the array. Array subclass constructors accept positional arguments, used to initialize the elements in order. ctypes.ARRAY(type, length) Create an array. Equivalent to "type * length", where *type* is a "ctypes" data type and *length* an integer. This function is *soft deprecated* in favor of multiplication. There are no plans to remove it. class ctypes._Pointer Private, abstract base class for pointers. Concrete pointer types are created by calling "POINTER()" with the type that will be pointed to; this is done automatically by "pointer()". If a pointer points to an array, its elements can be read and written using standard subscript and slice accesses. Pointer objects have no size, so "len()" will raise "TypeError". Negative subscripts will read from the memory *before* the pointer (as in C), and out-of-range subscripts will probably crash with an access violation (if you’re lucky). _type_ Specifies the type pointed to. contents Returns the object to which to pointer points. Assigning to this attribute changes the pointer to point to the assigned object. "curses.ascii" — Utilities for ASCII characters *********************************************** **Source code:** Lib/curses/ascii.py ====================================================================== The "curses.ascii" module supplies name constants for ASCII characters and functions to test membership in various ASCII character classes. The constants supplied are names for control characters as follows: +-----------------+------------------------------------------------+ | Name | Meaning | |=================|================================================| | curses.ascii.N | | | UL | | +-----------------+------------------------------------------------+ | curses.ascii.S | Start of heading, console interrupt | | OH | | +-----------------+------------------------------------------------+ | curses.ascii.S | Start of text | | TX | | +-----------------+------------------------------------------------+ | curses.ascii.E | End of text | | TX | | +-----------------+------------------------------------------------+ | curses.ascii.E | End of transmission | | OT | | +-----------------+------------------------------------------------+ | curses.ascii.E | Enquiry, goes with "ACK" flow control | | NQ | | +-----------------+------------------------------------------------+ | curses.ascii.A | Acknowledgement | | CK | | +-----------------+------------------------------------------------+ | curses.ascii.B | Bell | | EL | | +-----------------+------------------------------------------------+ | curses.ascii.BS | Backspace | +-----------------+------------------------------------------------+ | curses.ascii.T | Tab | | AB | | +-----------------+------------------------------------------------+ | curses.ascii.HT | Alias for "TAB": “Horizontal tab” | +-----------------+------------------------------------------------+ | curses.ascii.LF | Line feed | +-----------------+------------------------------------------------+ | curses.ascii.NL | Alias for "LF": “New line” | +-----------------+------------------------------------------------+ | curses.ascii.VT | Vertical tab | +-----------------+------------------------------------------------+ | curses.ascii.FF | Form feed | +-----------------+------------------------------------------------+ | curses.ascii.CR | Carriage return | +-----------------+------------------------------------------------+ | curses.ascii.SO | Shift-out, begin alternate character set | +-----------------+------------------------------------------------+ | curses.ascii.SI | Shift-in, resume default character set | +-----------------+------------------------------------------------+ | curses.ascii.D | Data-link escape | | LE | | +-----------------+------------------------------------------------+ | curses.ascii.D | XON, for flow control | | C1 | | +-----------------+------------------------------------------------+ | curses.ascii.D | Device control 2, block-mode flow control | | C2 | | +-----------------+------------------------------------------------+ | curses.ascii.D | XOFF, for flow control | | C3 | | +-----------------+------------------------------------------------+ | curses.ascii.D | Device control 4 | | C4 | | +-----------------+------------------------------------------------+ | curses.ascii.N | Negative acknowledgement | | AK | | +-----------------+------------------------------------------------+ | curses.ascii.S | Synchronous idle | | YN | | +-----------------+------------------------------------------------+ | curses.ascii.E | End transmission block | | TB | | +-----------------+------------------------------------------------+ | curses.ascii.C | Cancel | | AN | | +-----------------+------------------------------------------------+ | curses.ascii.EM | End of medium | +-----------------+------------------------------------------------+ | curses.ascii.S | Substitute | | UB | | +-----------------+------------------------------------------------+ | curses.ascii.E | Escape | | SC | | +-----------------+------------------------------------------------+ | curses.ascii.FS | File separator | +-----------------+------------------------------------------------+ | curses.ascii.GS | Group separator | +-----------------+------------------------------------------------+ | curses.ascii.RS | Record separator, block-mode terminator | +-----------------+------------------------------------------------+ | curses.ascii.US | Unit separator | +-----------------+------------------------------------------------+ | curses.ascii.SP | Space | +-----------------+------------------------------------------------+ | curses.ascii.D | Delete | | EL | | +-----------------+------------------------------------------------+ Note that many of these have little practical significance in modern usage. The mnemonics derive from teleprinter conventions that predate digital computers. The module supplies the following functions, patterned on those in the standard C library: curses.ascii.isalnum(c) Checks for an ASCII alphanumeric character; it is equivalent to "isalpha(c) or isdigit(c)". curses.ascii.isalpha(c) Checks for an ASCII alphabetic character; it is equivalent to "isupper(c) or islower(c)". curses.ascii.isascii(c) Checks for a character value that fits in the 7-bit ASCII set. curses.ascii.isblank(c) Checks for an ASCII whitespace character; space or horizontal tab. curses.ascii.iscntrl(c) Checks for an ASCII control character (in the range 0x00 to 0x1f or 0x7f). curses.ascii.isdigit(c) Checks for an ASCII decimal digit, "'0'" through "'9'". This is equivalent to "c in string.digits". curses.ascii.isgraph(c) Checks for ASCII any printable character except space. curses.ascii.islower(c) Checks for an ASCII lower-case character. curses.ascii.isprint(c) Checks for any ASCII printable character including space. curses.ascii.ispunct(c) Checks for any printable ASCII character which is not a space or an alphanumeric character. curses.ascii.isspace(c) Checks for ASCII white-space characters; space, line feed, carriage return, form feed, horizontal tab, vertical tab. curses.ascii.isupper(c) Checks for an ASCII uppercase letter. curses.ascii.isxdigit(c) Checks for an ASCII hexadecimal digit. This is equivalent to "c in string.hexdigits". curses.ascii.isctrl(c) Checks for an ASCII control character (ordinal values 0 to 31). curses.ascii.ismeta(c) Checks for a non-ASCII character (ordinal values 0x80 and above). These functions accept either integers or single-character strings; when the argument is a string, it is first converted using the built- in function "ord()". Note that all these functions check ordinal bit values derived from the character of the string you pass in; they do not actually know anything about the host machine’s character encoding. The following two functions take either a single-character string or integer byte value; they return a value of the same type. curses.ascii.ascii(c) Return the ASCII value corresponding to the low 7 bits of *c*. curses.ascii.ctrl(c) Return the control character corresponding to the given character (the character bit value is bitwise-anded with 0x1f). curses.ascii.alt(c) Return the 8-bit character corresponding to the given ASCII character (the character bit value is bitwise-ored with 0x80). The following function takes either a single-character string or integer value; it returns a string. curses.ascii.unctrl(c) Return a string representation of the ASCII character *c*. If *c* is printable, this string is the character itself. If the character is a control character (0x00–0x1f) the string consists of a caret ("'^'") followed by the corresponding uppercase letter. If the character is an ASCII delete (0x7f) the string is "'^?'". If the character has its meta bit (0x80) set, the meta bit is stripped, the preceding rules applied, and "'!'" prepended to the result. curses.ascii.controlnames A 33-element string array that contains the ASCII mnemonics for the thirty-two ASCII control characters from 0 (NUL) to 0x1f (US), in order, plus the mnemonic "SP" for the space character. "curses.panel" — A panel stack extension for curses *************************************************** ====================================================================== Panels are windows with the added feature of depth, so they can be stacked on top of each other, and only the visible portions of each window will be displayed. Panels can be added, moved up or down in the stack, and removed. Functions ========= The module "curses.panel" defines the following functions: curses.panel.bottom_panel() Returns the bottom panel in the panel stack. curses.panel.new_panel(win) Returns a panel object, associating it with the given window *win*. Be aware that you need to keep the returned panel object referenced explicitly. If you don’t, the panel object is garbage collected and removed from the panel stack. curses.panel.top_panel() Returns the top panel in the panel stack. curses.panel.update_panels() Updates the virtual screen after changes in the panel stack. This does not call "curses.doupdate()", so you’ll have to do this yourself. Panel Objects ============= Panel objects, as returned by "new_panel()" above, are windows with a stacking order. There’s always a window associated with a panel which determines the content, while the panel methods are responsible for the window’s depth in the panel stack. Panel objects have the following methods: Panel.above() Returns the panel above the current panel. Panel.below() Returns the panel below the current panel. Panel.bottom() Push the panel to the bottom of the stack. Panel.hidden() Returns "True" if the panel is hidden (not visible), "False" otherwise. Panel.hide() Hide the panel. This does not delete the object, it just makes the window on screen invisible. Panel.move(y, x) Move the panel to the screen coordinates "(y, x)". Panel.replace(win) Change the window associated with the panel to the window *win*. Panel.set_userptr(obj) Set the panel’s user pointer to *obj*. This is used to associate an arbitrary piece of data with the panel, and can be any Python object. Panel.show() Display the panel (which might have been hidden). Panel.top() Push panel to the top of the stack. Panel.userptr() Returns the user pointer for the panel. This might be any Python object. Panel.window() Returns the window object associated with the panel. "curses" — Terminal handling for character-cell displays ******************************************************** **Source code:** Lib/curses ====================================================================== The "curses" module provides an interface to the curses library, the de-facto standard for portable advanced terminal handling. While curses is most widely used in the Unix environment, versions are available for Windows, DOS, and possibly other systems as well. This extension module is designed to match the API of ncurses, an open- source curses library hosted on Linux and the BSD variants of Unix. Availability: not Android, not iOS, not WASI. This module is not supported on mobile platforms or WebAssembly platforms. Note: Whenever the documentation mentions a *character* it can be specified as an integer, a one-character Unicode string or a one- byte byte string.Whenever the documentation mentions a *character string* it can be specified as a Unicode string or a byte string. See also: Module "curses.ascii" Utilities for working with ASCII characters, regardless of your locale settings. Module "curses.panel" A panel stack extension that adds depth to curses windows. Module "curses.textpad" Editable text widget for curses supporting **Emacs**-like bindings. Curses Programming with Python Tutorial material on using curses with Python, by Andrew Kuchling and Eric Raymond. Functions ========= The module "curses" defines the following exception: exception curses.error Exception raised when a curses library function returns an error. Note: Whenever *x* or *y* arguments to a function or a method are optional, they default to the current cursor location. Whenever *attr* is optional, it defaults to "A_NORMAL". The module "curses" defines the following functions: curses.baudrate() Return the output speed of the terminal in bits per second. On software terminal emulators it will have a fixed high value. Included for historical reasons; in former times, it was used to write output loops for time delays and occasionally to change interfaces depending on the line speed. curses.beep() Emit a short attention sound. curses.can_change_color() Return "True" or "False", depending on whether the programmer can change the colors displayed by the terminal. curses.cbreak() Enter cbreak mode. In cbreak mode (sometimes called “rare” mode) normal tty line buffering is turned off and characters are available to be read one by one. However, unlike raw mode, special characters (interrupt, quit, suspend, and flow control) retain their effects on the tty driver and calling program. Calling first "raw()" then "cbreak()" leaves the terminal in cbreak mode. curses.color_content(color_number) Return the intensity of the red, green, and blue (RGB) components in the color *color_number*, which must be between "0" and "COLORS - 1". Return a 3-tuple, containing the R,G,B values for the given color, which will be between "0" (no component) and "1000" (maximum amount of component). curses.color_pair(pair_number) Return the attribute value for displaying text in the specified color pair. Only the first 256 color pairs are supported. This attribute value can be combined with "A_STANDOUT", "A_REVERSE", and the other "A_*" attributes. "pair_number()" is the counterpart to this function. curses.curs_set(visibility) Set the cursor state. *visibility* can be set to "0", "1", or "2", for invisible, normal, or very visible. If the terminal supports the visibility requested, return the previous cursor state; otherwise raise an exception. On many terminals, the “visible” mode is an underline cursor and the “very visible” mode is a block cursor. curses.def_prog_mode() Save the current terminal mode as the “program” mode, the mode when the running program is using curses. (Its counterpart is the “shell” mode, for when the program is not in curses.) Subsequent calls to "reset_prog_mode()" will restore this mode. curses.def_shell_mode() Save the current terminal mode as the “shell” mode, the mode when the running program is not using curses. (Its counterpart is the “program” mode, when the program is using curses capabilities.) Subsequent calls to "reset_shell_mode()" will restore this mode. curses.delay_output(ms) Insert an *ms* millisecond pause in output. curses.doupdate() Update the physical screen. The curses library keeps two data structures, one representing the current physical screen contents and a virtual screen representing the desired next state. The "doupdate()" ground updates the physical screen to match the virtual screen. The virtual screen may be updated by a "noutrefresh()" call after write operations such as "addstr()" have been performed on a window. The normal "refresh()" call is simply "noutrefresh()" followed by "doupdate()"; if you have to update multiple windows, you can speed performance and perhaps reduce screen flicker by issuing "noutrefresh()" calls on all windows, followed by a single "doupdate()". curses.echo() Enter echo mode. In echo mode, each character input is echoed to the screen as it is entered. curses.endwin() De-initialize the library, and return terminal to normal status. curses.erasechar() Return the user’s current erase character as a one-byte bytes object. Under Unix operating systems this is a property of the controlling tty of the curses program, and is not set by the curses library itself. curses.filter() The "filter()" routine, if used, must be called before "initscr()" is called. The effect is that, during those calls, "LINES" is set to "1"; the capabilities "clear", "cup", "cud", "cud1", "cuu1", "cuu", "vpa" are disabled; and the "home" string is set to the value of "cr". The effect is that the cursor is confined to the current line, and so are screen updates. This may be used for enabling character-at-a-time line editing without touching the rest of the screen. curses.flash() Flash the screen. That is, change it to reverse-video and then change it back in a short interval. Some people prefer such as ‘visible bell’ to the audible attention signal produced by "beep()". curses.flushinp() Flush all input buffers. This throws away any typeahead that has been typed by the user and has not yet been processed by the program. curses.getmouse() After "getch()" returns "KEY_MOUSE" to signal a mouse event, this method should be called to retrieve the queued mouse event, represented as a 5-tuple "(id, x, y, z, bstate)". *id* is an ID value used to distinguish multiple devices, and *x*, *y*, *z* are the event’s coordinates. (*z* is currently unused.) *bstate* is an integer value whose bits will be set to indicate the type of event, and will be the bitwise OR of one or more of the following constants, where *n* is the button number from 1 to 5: "BUTTONn_PRESSED", "BUTTONn_RELEASED", "BUTTONn_CLICKED", "BUTTONn_DOUBLE_CLICKED", "BUTTONn_TRIPLE_CLICKED", "BUTTON_SHIFT", "BUTTON_CTRL", "BUTTON_ALT". Changed in version 3.10: The "BUTTON5_*" constants are now exposed if they are provided by the underlying curses library. curses.getsyx() Return the current coordinates of the virtual screen cursor as a tuple "(y, x)". If "leaveok" is currently "True", then return "(-1, -1)". curses.getwin(file) Read window related data stored in the file by an earlier "window.putwin()" call. The routine then creates and initializes a new window using that data, returning the new window object. curses.has_colors() Return "True" if the terminal can display colors; otherwise, return "False". curses.has_extended_color_support() Return "True" if the module supports extended colors; otherwise, return "False". Extended color support allows more than 256 color pairs for terminals that support more than 16 colors (e.g. xterm- 256color). Extended color support requires ncurses version 6.1 or later. Added in version 3.10. curses.has_ic() Return "True" if the terminal has insert- and delete-character capabilities. This function is included for historical reasons only, as all modern software terminal emulators have such capabilities. curses.has_il() Return "True" if the terminal has insert- and delete-line capabilities, or can simulate them using scrolling regions. This function is included for historical reasons only, as all modern software terminal emulators have such capabilities. curses.has_key(ch) Take a key value *ch*, and return "True" if the current terminal type recognizes a key with that value. curses.halfdelay(tenths) Used for half-delay mode, which is similar to cbreak mode in that characters typed by the user are immediately available to the program. However, after blocking for *tenths* tenths of seconds, raise an exception if nothing has been typed. The value of *tenths* must be a number between "1" and "255". Use "nocbreak()" to leave half-delay mode. curses.init_color(color_number, r, g, b) Change the definition of a color, taking the number of the color to be changed followed by three RGB values (for the amounts of red, green, and blue components). The value of *color_number* must be between "0" and "COLORS - 1". Each of *r*, *g*, *b*, must be a value between "0" and "1000". When "init_color()" is used, all occurrences of that color on the screen immediately change to the new definition. This function is a no-op on most terminals; it is active only if "can_change_color()" returns "True". curses.init_pair(pair_number, fg, bg) Change the definition of a color-pair. It takes three arguments: the number of the color-pair to be changed, the foreground color number, and the background color number. The value of *pair_number* must be between "1" and "COLOR_PAIRS - 1" (the "0" color pair is wired to white on black and cannot be changed). The value of *fg* and *bg* arguments must be between "0" and "COLORS - 1", or, after calling "use_default_colors()", "-1". If the color- pair was previously initialized, the screen is refreshed and all occurrences of that color-pair are changed to the new definition. curses.initscr() Initialize the library. Return a window object which represents the whole screen. Note: If there is an error opening the terminal, the underlying curses library may cause the interpreter to exit. curses.is_term_resized(nlines, ncols) Return "True" if "resize_term()" would modify the window structure, "False" otherwise. curses.isendwin() Return "True" if "endwin()" has been called (that is, the curses library has been deinitialized). curses.keyname(k) Return the name of the key numbered *k* as a bytes object. The name of a key generating printable ASCII character is the key’s character. The name of a control-key combination is a two-byte bytes object consisting of a caret ("b'^'") followed by the corresponding printable ASCII character. The name of an alt-key combination (128–255) is a bytes object consisting of the prefix "b'M-'" followed by the name of the corresponding ASCII character. curses.killchar() Return the user’s current line kill character as a one-byte bytes object. Under Unix operating systems this is a property of the controlling tty of the curses program, and is not set by the curses library itself. curses.longname() Return a bytes object containing the terminfo long name field describing the current terminal. The maximum length of a verbose description is 128 characters. It is defined only after the call to "initscr()". curses.meta(flag) If *flag* is "True", allow 8-bit characters to be input. If *flag* is "False", allow only 7-bit chars. curses.mouseinterval(interval) Set the maximum time in milliseconds that can elapse between press and release events in order for them to be recognized as a click, and return the previous interval value. The default value is 200 milliseconds, or one fifth of a second. curses.mousemask(mousemask) Set the mouse events to be reported, and return a tuple "(availmask, oldmask)". *availmask* indicates which of the specified mouse events can be reported; on complete failure it returns "0". *oldmask* is the previous value of the given window’s mouse event mask. If this function is never called, no mouse events are ever reported. curses.napms(ms) Sleep for *ms* milliseconds. curses.newpad(nlines, ncols) Create and return a pointer to a new pad data structure with the given number of lines and columns. Return a pad as a window object. A pad is like a window, except that it is not restricted by the screen size, and is not necessarily associated with a particular part of the screen. Pads can be used when a large window is needed, and only a part of the window will be on the screen at one time. Automatic refreshes of pads (such as from scrolling or echoing of input) do not occur. The "refresh()" and "noutrefresh()" methods of a pad require 6 arguments to specify the part of the pad to be displayed and the location on the screen to be used for the display. The arguments are *pminrow*, *pmincol*, *sminrow*, *smincol*, *smaxrow*, *smaxcol*; the *p* arguments refer to the upper left corner of the pad region to be displayed and the *s* arguments define a clipping box on the screen within which the pad region is to be displayed. curses.newwin(nlines, ncols) curses.newwin(nlines, ncols, begin_y, begin_x) Return a new window, whose left-upper corner is at "(begin_y, begin_x)", and whose height/width is *nlines*/*ncols*. By default, the window will extend from the specified position to the lower right corner of the screen. curses.nl() Enter newline mode. This mode translates the return key into newline on input, and translates newline into return and line-feed on output. Newline mode is initially on. curses.nocbreak() Leave cbreak mode. Return to normal “cooked” mode with line buffering. curses.noecho() Leave echo mode. Echoing of input characters is turned off. curses.nonl() Leave newline mode. Disable translation of return into newline on input, and disable low-level translation of newline into newline/return on output (but this does not change the behavior of "addch('\n')", which always does the equivalent of return and line feed on the virtual screen). With translation off, curses can sometimes speed up vertical motion a little; also, it will be able to detect the return key on input. curses.noqiflush() When the "noqiflush()" routine is used, normal flush of input and output queues associated with the "INTR", "QUIT" and "SUSP" characters will not be done. You may want to call "noqiflush()" in a signal handler if you want output to continue as though the interrupt had not occurred, after the handler exits. curses.noraw() Leave raw mode. Return to normal “cooked” mode with line buffering. curses.pair_content(pair_number) Return a tuple "(fg, bg)" containing the colors for the requested color pair. The value of *pair_number* must be between "0" and "COLOR_PAIRS - 1". curses.pair_number(attr) Return the number of the color-pair set by the attribute value *attr*. "color_pair()" is the counterpart to this function. curses.putp(str) Equivalent to "tputs(str, 1, putchar)"; emit the value of a specified terminfo capability for the current terminal. Note that the output of "putp()" always goes to standard output. curses.qiflush([flag]) If *flag* is "False", the effect is the same as calling "noqiflush()". If *flag* is "True", or no argument is provided, the queues will be flushed when these control characters are read. curses.raw() Enter raw mode. In raw mode, normal line buffering and processing of interrupt, quit, suspend, and flow control keys are turned off; characters are presented to curses input functions one by one. curses.reset_prog_mode() Restore the terminal to “program” mode, as previously saved by "def_prog_mode()". curses.reset_shell_mode() Restore the terminal to “shell” mode, as previously saved by "def_shell_mode()". curses.resetty() Restore the state of the terminal modes to what it was at the last call to "savetty()". curses.resize_term(nlines, ncols) Backend function used by "resizeterm()", performing most of the work; when resizing the windows, "resize_term()" blank-fills the areas that are extended. The calling application should fill in these areas with appropriate data. The "resize_term()" function attempts to resize all windows. However, due to the calling convention of pads, it is not possible to resize these without additional interaction with the application. curses.resizeterm(nlines, ncols) Resize the standard and current windows to the specified dimensions, and adjusts other bookkeeping data used by the curses library that record the window dimensions (in particular the SIGWINCH handler). curses.savetty() Save the current state of the terminal modes in a buffer, usable by "resetty()". curses.get_escdelay() Retrieves the value set by "set_escdelay()". Added in version 3.9. curses.set_escdelay(ms) Sets the number of milliseconds to wait after reading an escape character, to distinguish between an individual escape character entered on the keyboard from escape sequences sent by cursor and function keys. Added in version 3.9. curses.get_tabsize() Retrieves the value set by "set_tabsize()". Added in version 3.9. curses.set_tabsize(size) Sets the number of columns used by the curses library when converting a tab character to spaces as it adds the tab to a window. Added in version 3.9. curses.setsyx(y, x) Set the virtual screen cursor to *y*, *x*. If *y* and *x* are both "-1", then "leaveok" is set "True". curses.setupterm(term=None, fd=-1) Initialize the terminal. *term* is a string giving the terminal name, or "None"; if omitted or "None", the value of the "TERM" environment variable will be used. *fd* is the file descriptor to which any initialization sequences will be sent; if not supplied or "-1", the file descriptor for "sys.stdout" will be used. curses.start_color() Must be called if the programmer wants to use colors, and before any other color manipulation routine is called. It is good practice to call this routine right after "initscr()". "start_color()" initializes eight basic colors (black, red, green, yellow, blue, magenta, cyan, and white), and two global variables in the "curses" module, "COLORS" and "COLOR_PAIRS", containing the maximum number of colors and color-pairs the terminal can support. It also restores the colors on the terminal to the values they had when the terminal was just turned on. curses.termattrs() Return a logical OR of all video attributes supported by the terminal. This information is useful when a curses program needs complete control over the appearance of the screen. curses.termname() Return the value of the environment variable "TERM", as a bytes object, truncated to 14 characters. curses.tigetflag(capname) Return the value of the Boolean capability corresponding to the terminfo capability name *capname* as an integer. Return the value "-1" if *capname* is not a Boolean capability, or "0" if it is canceled or absent from the terminal description. curses.tigetnum(capname) Return the value of the numeric capability corresponding to the terminfo capability name *capname* as an integer. Return the value "-2" if *capname* is not a numeric capability, or "-1" if it is canceled or absent from the terminal description. curses.tigetstr(capname) Return the value of the string capability corresponding to the terminfo capability name *capname* as a bytes object. Return "None" if *capname* is not a terminfo “string capability”, or is canceled or absent from the terminal description. curses.tparm(str[, ...]) Instantiate the bytes object *str* with the supplied parameters, where *str* should be a parameterized string obtained from the terminfo database. E.g. "tparm(tigetstr("cup"), 5, 3)" could result in "b'\033[6;4H'", the exact result depending on terminal type. curses.typeahead(fd) Specify that the file descriptor *fd* be used for typeahead checking. If *fd* is "-1", then no typeahead checking is done. The curses library does “line-breakout optimization” by looking for typeahead periodically while updating the screen. If input is found, and it is coming from a tty, the current update is postponed until refresh or doupdate is called again, allowing faster response to commands typed in advance. This function allows specifying a different file descriptor for typeahead checking. curses.unctrl(ch) Return a bytes object which is a printable representation of the character *ch*. Control characters are represented as a caret followed by the character, for example as "b'^C'". Printing characters are left as they are. curses.ungetch(ch) Push *ch* so the next "getch()" will return it. Note: Only one *ch* can be pushed before "getch()" is called. curses.update_lines_cols() Update the "LINES" and "COLS" module variables. Useful for detecting manual screen resize. Added in version 3.5. curses.unget_wch(ch) Push *ch* so the next "get_wch()" will return it. Note: Only one *ch* can be pushed before "get_wch()" is called. Added in version 3.3. curses.ungetmouse(id, x, y, z, bstate) Push a "KEY_MOUSE" event onto the input queue, associating the given state data with it. curses.use_env(flag) If used, this function should be called before "initscr()" or newterm are called. When *flag* is "False", the values of lines and columns specified in the terminfo database will be used, even if environment variables "LINES" and "COLUMNS" (used by default) are set, or if curses is running in a window (in which case default behavior would be to use the window size if "LINES" and "COLUMNS" are not set). curses.use_default_colors() Allow use of default values for colors on terminals supporting this feature. Use this to support transparency in your application. The default color is assigned to the color number "-1". After calling this function, "init_pair(x, curses.COLOR_RED, -1)" initializes, for instance, color pair *x* to a red foreground color on the default background. curses.wrapper(func, /, *args, **kwargs) Initialize curses and call another callable object, *func*, which should be the rest of your curses-using application. If the application raises an exception, this function will restore the terminal to a sane state before re-raising the exception and generating a traceback. The callable object *func* is then passed the main window ‘stdscr’ as its first argument, followed by any other arguments passed to "wrapper()". Before calling *func*, "wrapper()" turns on cbreak mode, turns off echo, enables the terminal keypad, and initializes colors if the terminal has color support. On exit (whether normally or by exception) it restores cooked mode, turns on echo, and disables the terminal keypad. Window Objects ============== Window objects, as returned by "initscr()" and "newwin()" above, have the following methods and attributes: window.addch(ch[, attr]) window.addch(y, x, ch[, attr]) Paint character *ch* at "(y, x)" with attributes *attr*, overwriting any character previously painted at that location. By default, the character position and attributes are the current settings for the window object. Note: Writing outside the window, subwindow, or pad raises a "curses.error". Attempting to write to the lower right corner of a window, subwindow, or pad will cause an exception to be raised after the character is printed. window.addnstr(str, n[, attr]) window.addnstr(y, x, str, n[, attr]) Paint at most *n* characters of the character string *str* at "(y, x)" with attributes *attr*, overwriting anything previously on the display. window.addstr(str[, attr]) window.addstr(y, x, str[, attr]) Paint the character string *str* at "(y, x)" with attributes *attr*, overwriting anything previously on the display. Note: * Writing outside the window, subwindow, or pad raises "curses.error". Attempting to write to the lower right corner of a window, subwindow, or pad will cause an exception to be raised after the string is printed. * A bug in ncurses, the backend for this Python module, can cause SegFaults when resizing windows. This is fixed in ncurses-6.1-20190511. If you are stuck with an earlier ncurses, you can avoid triggering this if you do not call "addstr()" with a *str* that has embedded newlines. Instead, call "addstr()" separately for each line. window.attroff(attr) Remove attribute *attr* from the “background” set applied to all writes to the current window. window.attron(attr) Add attribute *attr* from the “background” set applied to all writes to the current window. window.attrset(attr) Set the “background” set of attributes to *attr*. This set is initially "0" (no attributes). window.bkgd(ch[, attr]) Set the background property of the window to the character *ch*, with attributes *attr*. The change is then applied to every character position in that window: * The attribute of every character in the window is changed to the new background attribute. * Wherever the former background character appears, it is changed to the new background character. window.bkgdset(ch[, attr]) Set the window’s background. A window’s background consists of a character and any combination of attributes. The attribute part of the background is combined (OR’ed) with all non-blank characters that are written into the window. Both the character and attribute parts of the background are combined with the blank characters. The background becomes a property of the character and moves with the character through any scrolling and insert/delete line/character operations. window.border([ls[, rs[, ts[, bs[, tl[, tr[, bl[, br]]]]]]]]) Draw a border around the edges of the window. Each parameter specifies the character to use for a specific part of the border; see the table below for more details. Note: A "0" value for any parameter will cause the default character to be used for that parameter. Keyword parameters can *not* be used. The defaults are listed in this table: +-------------+-----------------------+-------------------------+ | Parameter | Description | Default value | |=============|=======================|=========================| | *ls* | Left side | "ACS_VLINE" | +-------------+-----------------------+-------------------------+ | *rs* | Right side | "ACS_VLINE" | +-------------+-----------------------+-------------------------+ | *ts* | Top | "ACS_HLINE" | +-------------+-----------------------+-------------------------+ | *bs* | Bottom | "ACS_HLINE" | +-------------+-----------------------+-------------------------+ | *tl* | Upper-left corner | "ACS_ULCORNER" | +-------------+-----------------------+-------------------------+ | *tr* | Upper-right corner | "ACS_URCORNER" | +-------------+-----------------------+-------------------------+ | *bl* | Bottom-left corner | "ACS_LLCORNER" | +-------------+-----------------------+-------------------------+ | *br* | Bottom-right corner | "ACS_LRCORNER" | +-------------+-----------------------+-------------------------+ window.box([vertch, horch]) Similar to "border()", but both *ls* and *rs* are *vertch* and both *ts* and *bs* are *horch*. The default corner characters are always used by this function. window.chgat(attr) window.chgat(num, attr) window.chgat(y, x, attr) window.chgat(y, x, num, attr) Set the attributes of *num* characters at the current cursor position, or at position "(y, x)" if supplied. If *num* is not given or is "-1", the attribute will be set on all the characters to the end of the line. This function moves cursor to position "(y, x)" if supplied. The changed line will be touched using the "touchline()" method so that the contents will be redisplayed by the next window refresh. window.clear() Like "erase()", but also cause the whole window to be repainted upon next call to "refresh()". window.clearok(flag) If *flag* is "True", the next call to "refresh()" will clear the window completely. window.clrtobot() Erase from cursor to the end of the window: all lines below the cursor are deleted, and then the equivalent of "clrtoeol()" is performed. window.clrtoeol() Erase from cursor to the end of the line. window.cursyncup() Update the current cursor position of all the ancestors of the window to reflect the current cursor position of the window. window.delch([y, x]) Delete any character at "(y, x)". window.deleteln() Delete the line under the cursor. All following lines are moved up by one line. window.derwin(begin_y, begin_x) window.derwin(nlines, ncols, begin_y, begin_x) An abbreviation for “derive window”, "derwin()" is the same as calling "subwin()", except that *begin_y* and *begin_x* are relative to the origin of the window, rather than relative to the entire screen. Return a window object for the derived window. window.echochar(ch[, attr]) Add character *ch* with attribute *attr*, and immediately call "refresh()" on the window. window.enclose(y, x) Test whether the given pair of screen-relative character-cell coordinates are enclosed by the given window, returning "True" or "False". It is useful for determining what subset of the screen windows enclose the location of a mouse event. Changed in version 3.10: Previously it returned "1" or "0" instead of "True" or "False". window.encoding Encoding used to encode method arguments (Unicode strings and characters). The encoding attribute is inherited from the parent window when a subwindow is created, for example with "window.subwin()". By default, current locale encoding is used (see "locale.getencoding()"). Added in version 3.3. window.erase() Clear the window. window.getbegyx() Return a tuple "(y, x)" of coordinates of upper-left corner. window.getbkgd() Return the given window’s current background character/attribute pair. window.getch([y, x]) Get a character. Note that the integer returned does *not* have to be in ASCII range: function keys, keypad keys and so on are represented by numbers higher than 255. In no-delay mode, return "-1" if there is no input, otherwise wait until a key is pressed. window.get_wch([y, x]) Get a wide character. Return a character for most keys, or an integer for function keys, keypad keys, and other special keys. In no-delay mode, raise an exception if there is no input. Added in version 3.3. window.getkey([y, x]) Get a character, returning a string instead of an integer, as "getch()" does. Function keys, keypad keys and other special keys return a multibyte string containing the key name. In no-delay mode, raise an exception if there is no input. window.getmaxyx() Return a tuple "(y, x)" of the height and width of the window. window.getparyx() Return the beginning coordinates of this window relative to its parent window as a tuple "(y, x)". Return "(-1, -1)" if this window has no parent. window.getstr() window.getstr(n) window.getstr(y, x) window.getstr(y, x, n) Read a bytes object from the user, with primitive line editing capacity. window.getyx() Return a tuple "(y, x)" of current cursor position relative to the window’s upper-left corner. window.hline(ch, n) window.hline(y, x, ch, n) Display a horizontal line starting at "(y, x)" with length *n* consisting of the character *ch*. window.idcok(flag) If *flag* is "False", curses no longer considers using the hardware insert/delete character feature of the terminal; if *flag* is "True", use of character insertion and deletion is enabled. When curses is first initialized, use of character insert/delete is enabled by default. window.idlok(flag) If *flag* is "True", "curses" will try and use hardware line editing facilities. Otherwise, line insertion/deletion are disabled. window.immedok(flag) If *flag* is "True", any change in the window image automatically causes the window to be refreshed; you no longer have to call "refresh()" yourself. However, it may degrade performance considerably, due to repeated calls to wrefresh. This option is disabled by default. window.inch([y, x]) Return the character at the given position in the window. The bottom 8 bits are the character proper, and upper bits are the attributes. window.insch(ch[, attr]) window.insch(y, x, ch[, attr]) Paint character *ch* at "(y, x)" with attributes *attr*, moving the line from position *x* right by one character. window.insdelln(nlines) Insert *nlines* lines into the specified window above the current line. The *nlines* bottom lines are lost. For negative *nlines*, delete *nlines* lines starting with the one under the cursor, and move the remaining lines up. The bottom *nlines* lines are cleared. The current cursor position remains the same. window.insertln() Insert a blank line under the cursor. All following lines are moved down by one line. window.insnstr(str, n[, attr]) window.insnstr(y, x, str, n[, attr]) Insert a character string (as many characters as will fit on the line) before the character under the cursor, up to *n* characters. If *n* is zero or negative, the entire string is inserted. All characters to the right of the cursor are shifted right, with the rightmost characters on the line being lost. The cursor position does not change (after moving to *y*, *x*, if specified). window.insstr(str[, attr]) window.insstr(y, x, str[, attr]) Insert a character string (as many characters as will fit on the line) before the character under the cursor. All characters to the right of the cursor are shifted right, with the rightmost characters on the line being lost. The cursor position does not change (after moving to *y*, *x*, if specified). window.instr([n]) window.instr(y, x[, n]) Return a bytes object of characters, extracted from the window starting at the current cursor position, or at *y*, *x* if specified. Attributes are stripped from the characters. If *n* is specified, "instr()" returns a string at most *n* characters long (exclusive of the trailing NUL). window.is_linetouched(line) Return "True" if the specified line was modified since the last call to "refresh()"; otherwise return "False". Raise a "curses.error" exception if *line* is not valid for the given window. window.is_wintouched() Return "True" if the specified window was modified since the last call to "refresh()"; otherwise return "False". window.keypad(flag) If *flag* is "True", escape sequences generated by some keys (keypad, function keys) will be interpreted by "curses". If *flag* is "False", escape sequences will be left as is in the input stream. window.leaveok(flag) If *flag* is "True", cursor is left where it is on update, instead of being at “cursor position.” This reduces cursor movement where possible. If possible the cursor will be made invisible. If *flag* is "False", cursor will always be at “cursor position” after an update. window.move(new_y, new_x) Move cursor to "(new_y, new_x)". window.mvderwin(y, x) Move the window inside its parent window. The screen-relative parameters of the window are not changed. This routine is used to display different parts of the parent window at the same physical position on the screen. window.mvwin(new_y, new_x) Move the window so its upper-left corner is at "(new_y, new_x)". window.nodelay(flag) If *flag* is "True", "getch()" will be non-blocking. window.notimeout(flag) If *flag* is "True", escape sequences will not be timed out. If *flag* is "False", after a few milliseconds, an escape sequence will not be interpreted, and will be left in the input stream as is. window.noutrefresh() Mark for refresh but wait. This function updates the data structure representing the desired state of the window, but does not force an update of the physical screen. To accomplish that, call "doupdate()". window.overlay(destwin[, sminrow, smincol, dminrow, dmincol, dmaxrow, dmaxcol]) Overlay the window on top of *destwin*. The windows need not be the same size, only the overlapping region is copied. This copy is non- destructive, which means that the current background character does not overwrite the old contents of *destwin*. To get fine-grained control over the copied region, the second form of "overlay()" can be used. *sminrow* and *smincol* are the upper- left coordinates of the source window, and the other variables mark a rectangle in the destination window. window.overwrite(destwin[, sminrow, smincol, dminrow, dmincol, dmaxrow, dmaxcol]) Overwrite the window on top of *destwin*. The windows need not be the same size, in which case only the overlapping region is copied. This copy is destructive, which means that the current background character overwrites the old contents of *destwin*. To get fine-grained control over the copied region, the second form of "overwrite()" can be used. *sminrow* and *smincol* are the upper-left coordinates of the source window, the other variables mark a rectangle in the destination window. window.putwin(file) Write all data associated with the window into the provided file object. This information can be later retrieved using the "getwin()" function. window.redrawln(beg, num) Indicate that the *num* screen lines, starting at line *beg*, are corrupted and should be completely redrawn on the next "refresh()" call. window.redrawwin() Touch the entire window, causing it to be completely redrawn on the next "refresh()" call. window.refresh([pminrow, pmincol, sminrow, smincol, smaxrow, smaxcol]) Update the display immediately (sync actual screen with previous drawing/deleting methods). The 6 optional arguments can only be specified when the window is a pad created with "newpad()". The additional parameters are needed to indicate what part of the pad and screen are involved. *pminrow* and *pmincol* specify the upper left-hand corner of the rectangle to be displayed in the pad. *sminrow*, *smincol*, *smaxrow*, and *smaxcol* specify the edges of the rectangle to be displayed on the screen. The lower right-hand corner of the rectangle to be displayed in the pad is calculated from the screen coordinates, since the rectangles must be the same size. Both rectangles must be entirely contained within their respective structures. Negative values of *pminrow*, *pmincol*, *sminrow*, or *smincol* are treated as if they were zero. window.resize(nlines, ncols) Reallocate storage for a curses window to adjust its dimensions to the specified values. If either dimension is larger than the current values, the window’s data is filled with blanks that have the current background rendition (as set by "bkgdset()") merged into them. window.scroll([lines=1]) Scroll the screen or scrolling region upward by *lines* lines. window.scrollok(flag) Control what happens when the cursor of a window is moved off the edge of the window or scrolling region, either as a result of a newline action on the bottom line, or typing the last character of the last line. If *flag* is "False", the cursor is left on the bottom line. If *flag* is "True", the window is scrolled up one line. Note that in order to get the physical scrolling effect on the terminal, it is also necessary to call "idlok()". window.setscrreg(top, bottom) Set the scrolling region from line *top* to line *bottom*. All scrolling actions will take place in this region. window.standend() Turn off the standout attribute. On some terminals this has the side effect of turning off all attributes. window.standout() Turn on attribute *A_STANDOUT*. window.subpad(begin_y, begin_x) window.subpad(nlines, ncols, begin_y, begin_x) Return a sub-window, whose upper-left corner is at "(begin_y, begin_x)", and whose width/height is *ncols*/*nlines*. window.subwin(begin_y, begin_x) window.subwin(nlines, ncols, begin_y, begin_x) Return a sub-window, whose upper-left corner is at "(begin_y, begin_x)", and whose width/height is *ncols*/*nlines*. By default, the sub-window will extend from the specified position to the lower right corner of the window. window.syncdown() Touch each location in the window that has been touched in any of its ancestor windows. This routine is called by "refresh()", so it should almost never be necessary to call it manually. window.syncok(flag) If *flag* is "True", then "syncup()" is called automatically whenever there is a change in the window. window.syncup() Touch all locations in ancestors of the window that have been changed in the window. window.timeout(delay) Set blocking or non-blocking read behavior for the window. If *delay* is negative, blocking read is used (which will wait indefinitely for input). If *delay* is zero, then non-blocking read is used, and "getch()" will return "-1" if no input is waiting. If *delay* is positive, then "getch()" will block for *delay* milliseconds, and return "-1" if there is still no input at the end of that time. window.touchline(start, count[, changed]) Pretend *count* lines have been changed, starting with line *start*. If *changed* is supplied, it specifies whether the affected lines are marked as having been changed (*changed*"=True") or unchanged (*changed*"=False"). window.touchwin() Pretend the whole window has been changed, for purposes of drawing optimizations. window.untouchwin() Mark all lines in the window as unchanged since the last call to "refresh()". window.vline(ch, n[, attr]) window.vline(y, x, ch, n[, attr]) Display a vertical line starting at "(y, x)" with length *n* consisting of the character *ch* with attributes *attr*. Constants ========= The "curses" module defines the following data members: curses.ERR Some curses routines that return an integer, such as "getch()", return "ERR" upon failure. curses.OK Some curses routines that return an integer, such as "napms()", return "OK" upon success. curses.version curses.__version__ A bytes object representing the current version of the module. curses.ncurses_version A named tuple containing the three components of the ncurses library version: *major*, *minor*, and *patch*. All values are integers. The components can also be accessed by name, so "curses.ncurses_version[0]" is equivalent to "curses.ncurses_version.major" and so on. Availability: if the ncurses library is used. Added in version 3.8. curses.COLORS The maximum number of colors the terminal can support. It is defined only after the call to "start_color()". curses.COLOR_PAIRS The maximum number of color pairs the terminal can support. It is defined only after the call to "start_color()". curses.COLS The width of the screen, i.e., the number of columns. It is defined only after the call to "initscr()". Updated by "update_lines_cols()", "resizeterm()" and "resize_term()". curses.LINES The height of the screen, i.e., the number of lines. It is defined only after the call to "initscr()". Updated by "update_lines_cols()", "resizeterm()" and "resize_term()". Some constants are available to specify character cell attributes. The exact constants available are system dependent. +--------------------------+---------------------------------+ | Attribute | Meaning | |==========================|=================================| | curses.A_ALTCHARSET | Alternate character set mode | +--------------------------+---------------------------------+ | curses.A_BLINK | Blink mode | +--------------------------+---------------------------------+ | curses.A_BOLD | Bold mode | +--------------------------+---------------------------------+ | curses.A_DIM | Dim mode | +--------------------------+---------------------------------+ | curses.A_INVIS | Invisible or blank mode | +--------------------------+---------------------------------+ | curses.A_ITALIC | Italic mode | +--------------------------+---------------------------------+ | curses.A_NORMAL | Normal attribute | +--------------------------+---------------------------------+ | curses.A_PROTECT | Protected mode | +--------------------------+---------------------------------+ | curses.A_REVERSE | Reverse background and | | | foreground colors | +--------------------------+---------------------------------+ | curses.A_STANDOUT | Standout mode | +--------------------------+---------------------------------+ | curses.A_UNDERLINE | Underline mode | +--------------------------+---------------------------------+ | curses.A_HORIZONTAL | Horizontal highlight | +--------------------------+---------------------------------+ | curses.A_LEFT | Left highlight | +--------------------------+---------------------------------+ | curses.A_LOW | Low highlight | +--------------------------+---------------------------------+ | curses.A_RIGHT | Right highlight | +--------------------------+---------------------------------+ | curses.A_TOP | Top highlight | +--------------------------+---------------------------------+ | curses.A_VERTICAL | Vertical highlight | +--------------------------+---------------------------------+ Added in version 3.7: "A_ITALIC" was added. Several constants are available to extract corresponding attributes returned by some methods. +---------------------------+---------------------------------+ | Bit-mask | Meaning | |===========================|=================================| | curses.A_ATTRIBUTES | Bit-mask to extract attributes | +---------------------------+---------------------------------+ | curses.A_CHARTEXT | Bit-mask to extract a character | +---------------------------+---------------------------------+ | curses.A_COLOR | Bit-mask to extract color-pair | | | field information | +---------------------------+---------------------------------+ Keys are referred to by integer constants with names starting with "KEY_". The exact keycaps available are system dependent. +---------------------------+----------------------------------------------+ | Key constant | Key | |===========================|==============================================| | curses.KEY_MIN | Minimum key value | +---------------------------+----------------------------------------------+ | curses.KEY_BREAK | Break key (unreliable) | +---------------------------+----------------------------------------------+ | curses.KEY_DOWN | Down-arrow | +---------------------------+----------------------------------------------+ | curses.KEY_UP | Up-arrow | +---------------------------+----------------------------------------------+ | curses.KEY_LEFT | Left-arrow | +---------------------------+----------------------------------------------+ | curses.KEY_RIGHT | Right-arrow | +---------------------------+----------------------------------------------+ | curses.KEY_HOME | Home key (upward+left arrow) | +---------------------------+----------------------------------------------+ | curses.KEY_BACKSPACE | Backspace (unreliable) | +---------------------------+----------------------------------------------+ | curses.KEY_F0 | Function keys. Up to 64 function keys are | | | supported. | +---------------------------+----------------------------------------------+ | curses.KEY_Fn | Value of function key *n* | +---------------------------+----------------------------------------------+ | curses.KEY_DL | Delete line | +---------------------------+----------------------------------------------+ | curses.KEY_IL | Insert line | +---------------------------+----------------------------------------------+ | curses.KEY_DC | Delete character | +---------------------------+----------------------------------------------+ | curses.KEY_IC | Insert char or enter insert mode | +---------------------------+----------------------------------------------+ | curses.KEY_EIC | Exit insert char mode | +---------------------------+----------------------------------------------+ | curses.KEY_CLEAR | Clear screen | +---------------------------+----------------------------------------------+ | curses.KEY_EOS | Clear to end of screen | +---------------------------+----------------------------------------------+ | curses.KEY_EOL | Clear to end of line | +---------------------------+----------------------------------------------+ | curses.KEY_SF | Scroll 1 line forward | +---------------------------+----------------------------------------------+ | curses.KEY_SR | Scroll 1 line backward (reverse) | +---------------------------+----------------------------------------------+ | curses.KEY_NPAGE | Next page | +---------------------------+----------------------------------------------+ | curses.KEY_PPAGE | Previous page | +---------------------------+----------------------------------------------+ | curses.KEY_STAB | Set tab | +---------------------------+----------------------------------------------+ | curses.KEY_CTAB | Clear tab | +---------------------------+----------------------------------------------+ | curses.KEY_CATAB | Clear all tabs | +---------------------------+----------------------------------------------+ | curses.KEY_ENTER | Enter or send (unreliable) | +---------------------------+----------------------------------------------+ | curses.KEY_SRESET | Soft (partial) reset (unreliable) | +---------------------------+----------------------------------------------+ | curses.KEY_RESET | Reset or hard reset (unreliable) | +---------------------------+----------------------------------------------+ | curses.KEY_PRINT | Print | +---------------------------+----------------------------------------------+ | curses.KEY_LL | Home down or bottom (lower left) | +---------------------------+----------------------------------------------+ | curses.KEY_A1 | Upper left of keypad | +---------------------------+----------------------------------------------+ | curses.KEY_A3 | Upper right of keypad | +---------------------------+----------------------------------------------+ | curses.KEY_B2 | Center of keypad | +---------------------------+----------------------------------------------+ | curses.KEY_C1 | Lower left of keypad | +---------------------------+----------------------------------------------+ | curses.KEY_C3 | Lower right of keypad | +---------------------------+----------------------------------------------+ | curses.KEY_BTAB | Back tab | +---------------------------+----------------------------------------------+ | curses.KEY_BEG | Beg (beginning) | +---------------------------+----------------------------------------------+ | curses.KEY_CANCEL | Cancel | +---------------------------+----------------------------------------------+ | curses.KEY_CLOSE | Close | +---------------------------+----------------------------------------------+ | curses.KEY_COMMAND | Cmd (command) | +---------------------------+----------------------------------------------+ | curses.KEY_COPY | Copy | +---------------------------+----------------------------------------------+ | curses.KEY_CREATE | Create | +---------------------------+----------------------------------------------+ | curses.KEY_END | End | +---------------------------+----------------------------------------------+ | curses.KEY_EXIT | Exit | +---------------------------+----------------------------------------------+ | curses.KEY_FIND | Find | +---------------------------+----------------------------------------------+ | curses.KEY_HELP | Help | +---------------------------+----------------------------------------------+ | curses.KEY_MARK | Mark | +---------------------------+----------------------------------------------+ | curses.KEY_MESSAGE | Message | +---------------------------+----------------------------------------------+ | curses.KEY_MOVE | Move | +---------------------------+----------------------------------------------+ | curses.KEY_NEXT | Next | +---------------------------+----------------------------------------------+ | curses.KEY_OPEN | Open | +---------------------------+----------------------------------------------+ | curses.KEY_OPTIONS | Options | +---------------------------+----------------------------------------------+ | curses.KEY_PREVIOUS | Prev (previous) | +---------------------------+----------------------------------------------+ | curses.KEY_REDO | Redo | +---------------------------+----------------------------------------------+ | curses.KEY_REFERENCE | Ref (reference) | +---------------------------+----------------------------------------------+ | curses.KEY_REFRESH | Refresh | +---------------------------+----------------------------------------------+ | curses.KEY_REPLACE | Replace | +---------------------------+----------------------------------------------+ | curses.KEY_RESTART | Restart | +---------------------------+----------------------------------------------+ | curses.KEY_RESUME | Resume | +---------------------------+----------------------------------------------+ | curses.KEY_SAVE | Save | +---------------------------+----------------------------------------------+ | curses.KEY_SBEG | Shifted Beg (beginning) | +---------------------------+----------------------------------------------+ | curses.KEY_SCANCEL | Shifted Cancel | +---------------------------+----------------------------------------------+ | curses.KEY_SCOMMAND | Shifted Command | +---------------------------+----------------------------------------------+ | curses.KEY_SCOPY | Shifted Copy | +---------------------------+----------------------------------------------+ | curses.KEY_SCREATE | Shifted Create | +---------------------------+----------------------------------------------+ | curses.KEY_SDC | Shifted Delete char | +---------------------------+----------------------------------------------+ | curses.KEY_SDL | Shifted Delete line | +---------------------------+----------------------------------------------+ | curses.KEY_SELECT | Select | +---------------------------+----------------------------------------------+ | curses.KEY_SEND | Shifted End | +---------------------------+----------------------------------------------+ | curses.KEY_SEOL | Shifted Clear line | +---------------------------+----------------------------------------------+ | curses.KEY_SEXIT | Shifted Exit | +---------------------------+----------------------------------------------+ | curses.KEY_SFIND | Shifted Find | +---------------------------+----------------------------------------------+ | curses.KEY_SHELP | Shifted Help | +---------------------------+----------------------------------------------+ | curses.KEY_SHOME | Shifted Home | +---------------------------+----------------------------------------------+ | curses.KEY_SIC | Shifted Input | +---------------------------+----------------------------------------------+ | curses.KEY_SLEFT | Shifted Left arrow | +---------------------------+----------------------------------------------+ | curses.KEY_SMESSAGE | Shifted Message | +---------------------------+----------------------------------------------+ | curses.KEY_SMOVE | Shifted Move | +---------------------------+----------------------------------------------+ | curses.KEY_SNEXT | Shifted Next | +---------------------------+----------------------------------------------+ | curses.KEY_SOPTIONS | Shifted Options | +---------------------------+----------------------------------------------+ | curses.KEY_SPREVIOUS | Shifted Prev | +---------------------------+----------------------------------------------+ | curses.KEY_SPRINT | Shifted Print | +---------------------------+----------------------------------------------+ | curses.KEY_SREDO | Shifted Redo | +---------------------------+----------------------------------------------+ | curses.KEY_SREPLACE | Shifted Replace | +---------------------------+----------------------------------------------+ | curses.KEY_SRIGHT | Shifted Right arrow | +---------------------------+----------------------------------------------+ | curses.KEY_SRSUME | Shifted Resume | +---------------------------+----------------------------------------------+ | curses.KEY_SSAVE | Shifted Save | +---------------------------+----------------------------------------------+ | curses.KEY_SSUSPEND | Shifted Suspend | +---------------------------+----------------------------------------------+ | curses.KEY_SUNDO | Shifted Undo | +---------------------------+----------------------------------------------+ | curses.KEY_SUSPEND | Suspend | +---------------------------+----------------------------------------------+ | curses.KEY_UNDO | Undo | +---------------------------+----------------------------------------------+ | curses.KEY_MOUSE | Mouse event has occurred | +---------------------------+----------------------------------------------+ | curses.KEY_RESIZE | Terminal resize event | +---------------------------+----------------------------------------------+ | curses.KEY_MAX | Maximum key value | +---------------------------+----------------------------------------------+ On VT100s and their software emulations, such as X terminal emulators, there are normally at least four function keys ("KEY_F1", "KEY_F2", "KEY_F3", "KEY_F4") available, and the arrow keys mapped to "KEY_UP", "KEY_DOWN", "KEY_LEFT" and "KEY_RIGHT" in the obvious way. If your machine has a PC keyboard, it is safe to expect arrow keys and twelve function keys (older PC keyboards may have only ten function keys); also, the following keypad mappings are standard: +--------------------+-------------+ | Keycap | Constant | |====================|=============| | "Insert" | KEY_IC | +--------------------+-------------+ | "Delete" | KEY_DC | +--------------------+-------------+ | "Home" | KEY_HOME | +--------------------+-------------+ | "End" | KEY_END | +--------------------+-------------+ | "Page Up" | KEY_PPAGE | +--------------------+-------------+ | "Page Down" | KEY_NPAGE | +--------------------+-------------+ The following table lists characters from the alternate character set. These are inherited from the VT100 terminal, and will generally be available on software emulations such as X terminals. When there is no graphic available, curses falls back on a crude printable ASCII approximation. Note: These are available only after "initscr()" has been called. +--------------------------+--------------------------------------------+ | ACS code | Meaning | |==========================|============================================| | curses.ACS_BBSS | alternate name for upper right corner | +--------------------------+--------------------------------------------+ | curses.ACS_BLOCK | solid square block | +--------------------------+--------------------------------------------+ | curses.ACS_BOARD | board of squares | +--------------------------+--------------------------------------------+ | curses.ACS_BSBS | alternate name for horizontal line | +--------------------------+--------------------------------------------+ | curses.ACS_BSSB | alternate name for upper left corner | +--------------------------+--------------------------------------------+ | curses.ACS_BSSS | alternate name for top tee | +--------------------------+--------------------------------------------+ | curses.ACS_BTEE | bottom tee | +--------------------------+--------------------------------------------+ | curses.ACS_BULLET | bullet | +--------------------------+--------------------------------------------+ | curses.ACS_CKBOARD | checker board (stipple) | +--------------------------+--------------------------------------------+ | curses.ACS_DARROW | arrow pointing down | +--------------------------+--------------------------------------------+ | curses.ACS_DEGREE | degree symbol | +--------------------------+--------------------------------------------+ | curses.ACS_DIAMOND | diamond | +--------------------------+--------------------------------------------+ | curses.ACS_GEQUAL | greater-than-or-equal-to | +--------------------------+--------------------------------------------+ | curses.ACS_HLINE | horizontal line | +--------------------------+--------------------------------------------+ | curses.ACS_LANTERN | lantern symbol | +--------------------------+--------------------------------------------+ | curses.ACS_LARROW | left arrow | +--------------------------+--------------------------------------------+ | curses.ACS_LEQUAL | less-than-or-equal-to | +--------------------------+--------------------------------------------+ | curses.ACS_LLCORNER | lower left-hand corner | +--------------------------+--------------------------------------------+ | curses.ACS_LRCORNER | lower right-hand corner | +--------------------------+--------------------------------------------+ | curses.ACS_LTEE | left tee | +--------------------------+--------------------------------------------+ | curses.ACS_NEQUAL | not-equal sign | +--------------------------+--------------------------------------------+ | curses.ACS_PI | letter pi | +--------------------------+--------------------------------------------+ | curses.ACS_PLMINUS | plus-or-minus sign | +--------------------------+--------------------------------------------+ | curses.ACS_PLUS | big plus sign | +--------------------------+--------------------------------------------+ | curses.ACS_RARROW | right arrow | +--------------------------+--------------------------------------------+ | curses.ACS_RTEE | right tee | +--------------------------+--------------------------------------------+ | curses.ACS_S1 | scan line 1 | +--------------------------+--------------------------------------------+ | curses.ACS_S3 | scan line 3 | +--------------------------+--------------------------------------------+ | curses.ACS_S7 | scan line 7 | +--------------------------+--------------------------------------------+ | curses.ACS_S9 | scan line 9 | +--------------------------+--------------------------------------------+ | curses.ACS_SBBS | alternate name for lower right corner | +--------------------------+--------------------------------------------+ | curses.ACS_SBSB | alternate name for vertical line | +--------------------------+--------------------------------------------+ | curses.ACS_SBSS | alternate name for right tee | +--------------------------+--------------------------------------------+ | curses.ACS_SSBB | alternate name for lower left corner | +--------------------------+--------------------------------------------+ | curses.ACS_SSBS | alternate name for bottom tee | +--------------------------+--------------------------------------------+ | curses.ACS_SSSB | alternate name for left tee | +--------------------------+--------------------------------------------+ | curses.ACS_SSSS | alternate name for crossover or big plus | +--------------------------+--------------------------------------------+ | curses.ACS_STERLING | pound sterling | +--------------------------+--------------------------------------------+ | curses.ACS_TTEE | top tee | +--------------------------+--------------------------------------------+ | curses.ACS_UARROW | up arrow | +--------------------------+--------------------------------------------+ | curses.ACS_ULCORNER | upper left corner | +--------------------------+--------------------------------------------+ | curses.ACS_URCORNER | upper right corner | +--------------------------+--------------------------------------------+ | curses.ACS_VLINE | vertical line | +--------------------------+--------------------------------------------+ The following table lists mouse button constants used by "getmouse()": +------------------------------------+-----------------------------------------------+ | Mouse button constant | Meaning | |====================================|===============================================| | curses.BUTTONn_PRESSED | Mouse button *n* pressed | +------------------------------------+-----------------------------------------------+ | curses.BUTTONn_RELEASED | Mouse button *n* released | +------------------------------------+-----------------------------------------------+ | curses.BUTTONn_CLICKED | Mouse button *n* clicked | +------------------------------------+-----------------------------------------------+ | curses.BUTTONn_DOUBLE_CLICKED | Mouse button *n* double clicked | +------------------------------------+-----------------------------------------------+ | curses.BUTTONn_TRIPLE_CLICKED | Mouse button *n* triple clicked | +------------------------------------+-----------------------------------------------+ | curses.BUTTON_SHIFT | Shift was down during button state change | +------------------------------------+-----------------------------------------------+ | curses.BUTTON_CTRL | Control was down during button state change | +------------------------------------+-----------------------------------------------+ | curses.BUTTON_ALT | Control was down during button state change | +------------------------------------+-----------------------------------------------+ Changed in version 3.10: The "BUTTON5_*" constants are now exposed if they are provided by the underlying curses library. The following table lists the predefined colors: +---------------------------+------------------------------+ | Constant | Color | |===========================|==============================| | curses.COLOR_BLACK | Black | +---------------------------+------------------------------+ | curses.COLOR_BLUE | Blue | +---------------------------+------------------------------+ | curses.COLOR_CYAN | Cyan (light greenish blue) | +---------------------------+------------------------------+ | curses.COLOR_GREEN | Green | +---------------------------+------------------------------+ | curses.COLOR_MAGENTA | Magenta (purplish red) | +---------------------------+------------------------------+ | curses.COLOR_RED | Red | +---------------------------+------------------------------+ | curses.COLOR_WHITE | White | +---------------------------+------------------------------+ | curses.COLOR_YELLOW | Yellow | +---------------------------+------------------------------+ "curses.textpad" — Text input widget for curses programs ******************************************************** The "curses.textpad" module provides a "Textbox" class that handles elementary text editing in a curses window, supporting a set of keybindings resembling those of Emacs (thus, also of Netscape Navigator, BBedit 6.x, FrameMaker, and many other programs). The module also provides a rectangle-drawing function useful for framing text boxes or for other purposes. The module "curses.textpad" defines the following function: curses.textpad.rectangle(win, uly, ulx, lry, lrx) Draw a rectangle. The first argument must be a window object; the remaining arguments are coordinates relative to that window. The second and third arguments are the y and x coordinates of the upper left hand corner of the rectangle to be drawn; the fourth and fifth arguments are the y and x coordinates of the lower right hand corner. The rectangle will be drawn using VT100/IBM PC forms characters on terminals that make this possible (including xterm and most other software terminal emulators). Otherwise it will be drawn with ASCII dashes, vertical bars, and plus signs. Textbox objects =============== You can instantiate a "Textbox" object as follows: class curses.textpad.Textbox(win) Return a textbox widget object. The *win* argument should be a curses window object in which the textbox is to be contained. The edit cursor of the textbox is initially located at the upper left hand corner of the containing window, with coordinates "(0, 0)". The instance’s "stripspaces" flag is initially on. "Textbox" objects have the following methods: edit([validator]) This is the entry point you will normally use. It accepts editing keystrokes until one of the termination keystrokes is entered. If *validator* is supplied, it must be a function. It will be called for each keystroke entered with the keystroke as a parameter; command dispatch is done on the result. This method returns the window contents as a string; whether blanks in the window are included is affected by the "stripspaces" attribute. do_command(ch) Process a single command keystroke. Here are the supported special keystrokes: +--------------------+---------------------------------------------+ | Keystroke | Action | |====================|=============================================| | "Control"-"A" | Go to left edge of window. | +--------------------+---------------------------------------------+ | "Control"-"B" | Cursor left, wrapping to previous line if | | | appropriate. | +--------------------+---------------------------------------------+ | "Control"-"D" | Delete character under cursor. | +--------------------+---------------------------------------------+ | "Control"-"E" | Go to right edge (stripspaces off) or end | | | of line (stripspaces on). | +--------------------+---------------------------------------------+ | "Control"-"F" | Cursor right, wrapping to next line when | | | appropriate. | +--------------------+---------------------------------------------+ | "Control"-"G" | Terminate, returning the window contents. | +--------------------+---------------------------------------------+ | "Control"-"H" | Delete character backward. | +--------------------+---------------------------------------------+ | "Control"-"J" | Terminate if the window is 1 line, | | | otherwise insert newline. | +--------------------+---------------------------------------------+ | "Control"-"K" | If line is blank, delete it, otherwise | | | clear to end of line. | +--------------------+---------------------------------------------+ | "Control"-"L" | Refresh screen. | +--------------------+---------------------------------------------+ | "Control"-"N" | Cursor down; move down one line. | +--------------------+---------------------------------------------+ | "Control"-"O" | Insert a blank line at cursor location. | +--------------------+---------------------------------------------+ | "Control"-"P" | Cursor up; move up one line. | +--------------------+---------------------------------------------+ Move operations do nothing if the cursor is at an edge where the movement is not possible. The following synonyms are supported where possible: +----------------------------------+--------------------+ | Constant | Keystroke | |==================================|====================| | "KEY_LEFT" | "Control"-"B" | +----------------------------------+--------------------+ | "KEY_RIGHT" | "Control"-"F" | +----------------------------------+--------------------+ | "KEY_UP" | "Control"-"P" | +----------------------------------+--------------------+ | "KEY_DOWN" | "Control"-"N" | +----------------------------------+--------------------+ | "KEY_BACKSPACE" | "Control"-"h" | +----------------------------------+--------------------+ All other keystrokes are treated as a command to insert the given character and move right (with line wrapping). gather() Return the window contents as a string; whether blanks in the window are included is affected by the "stripspaces" member. stripspaces This attribute is a flag which controls the interpretation of blanks in the window. When it is on, trailing blanks on each line are ignored; any cursor motion that would land the cursor on a trailing blank goes to the end of that line instead, and trailing blanks are stripped when the window contents are gathered. Custom Python Interpreters ************************** The modules described in this chapter allow writing interfaces similar to Python’s interactive interpreter. If you want a Python interpreter that supports some special feature in addition to the Python language, you should look at the "code" module. (The "codeop" module is lower- level, used to support compiling a possibly incomplete chunk of Python code.) The full list of modules described in this chapter is: * "code" — Interpreter base classes * Interactive Interpreter Objects * Interactive Console Objects * "codeop" — Compile Python code "dataclasses" — Data Classes **************************** **Source code:** Lib/dataclasses.py ====================================================================== This module provides a decorator and functions for automatically adding generated *special methods* such as "__init__()" and "__repr__()" to user-defined classes. It was originally described in **PEP 557**. The member variables to use in these generated methods are defined using **PEP 526** type annotations. For example, this code: from dataclasses import dataclass @dataclass class InventoryItem: """Class for keeping track of an item in inventory.""" name: str unit_price: float quantity_on_hand: int = 0 def total_cost(self) -> float: return self.unit_price * self.quantity_on_hand will add, among other things, a "__init__()" that looks like: def __init__(self, name: str, unit_price: float, quantity_on_hand: int = 0): self.name = name self.unit_price = unit_price self.quantity_on_hand = quantity_on_hand Note that this method is automatically added to the class: it is not directly specified in the "InventoryItem" definition shown above. Added in version 3.7. Module contents =============== @dataclasses.dataclass(*, init=True, repr=True, eq=True, order=False, unsafe_hash=False, frozen=False, match_args=True, kw_only=False, slots=False, weakref_slot=False) This function is a *decorator* that is used to add generated *special methods* to classes, as described below. The "@dataclass" decorator examines the class to find "field"s. A "field" is defined as a class variable that has a *type annotation*. With two exceptions described below, nothing in "@dataclass" examines the type specified in the variable annotation. The order of the fields in all of the generated methods is the order in which they appear in the class definition. The "@dataclass" decorator will add various “dunder” methods to the class, described below. If any of the added methods already exist in the class, the behavior depends on the parameter, as documented below. The decorator returns the same class that it is called on; no new class is created. If "@dataclass" is used just as a simple decorator with no parameters, it acts as if it has the default values documented in this signature. That is, these three uses of "@dataclass" are equivalent: @dataclass class C: ... @dataclass() class C: ... @dataclass(init=True, repr=True, eq=True, order=False, unsafe_hash=False, frozen=False, match_args=True, kw_only=False, slots=False, weakref_slot=False) class C: ... The parameters to "@dataclass" are: * *init*: If true (the default), a "__init__()" method will be generated. If the class already defines "__init__()", this parameter is ignored. * *repr*: If true (the default), a "__repr__()" method will be generated. The generated repr string will have the class name and the name and repr of each field, in the order they are defined in the class. Fields that are marked as being excluded from the repr are not included. For example: "InventoryItem(name='widget', unit_price=3.0, quantity_on_hand=10)". If the class already defines "__repr__()", this parameter is ignored. * *eq*: If true (the default), an "__eq__()" method will be generated. This method compares the class as if it were a tuple of its fields, in order. Both instances in the comparison must be of the identical type. If the class already defines "__eq__()", this parameter is ignored. * *order*: If true (the default is "False"), "__lt__()", "__le__()", "__gt__()", and "__ge__()" methods will be generated. These compare the class as if it were a tuple of its fields, in order. Both instances in the comparison must be of the identical type. If *order* is true and *eq* is false, a "ValueError" is raised. If the class already defines any of "__lt__()", "__le__()", "__gt__()", or "__ge__()", then "TypeError" is raised. * *unsafe_hash*: If true, force "dataclasses" to create a "__hash__()" method, even though it may not be safe to do so. Otherwise, generate a "__hash__()" method according to how *eq* and *frozen* are set. The default value is "False". "__hash__()" is used by built-in "hash()", and when objects are added to hashed collections such as dictionaries and sets. Having a "__hash__()" implies that instances of the class are immutable. Mutability is a complicated property that depends on the programmer’s intent, the existence and behavior of "__eq__()", and the values of the *eq* and *frozen* flags in the "@dataclass" decorator. By default, "@dataclass" will not implicitly add a "__hash__()" method unless it is safe to do so. Neither will it add or change an existing explicitly defined "__hash__()" method. Setting the class attribute "__hash__ = None" has a specific meaning to Python, as described in the "__hash__()" documentation. If "__hash__()" is not explicitly defined, or if it is set to "None", then "@dataclass" *may* add an implicit "__hash__()" method. Although not recommended, you can force "@dataclass" to create a "__hash__()" method with "unsafe_hash=True". This might be the case if your class is logically immutable but can still be mutated. This is a specialized use case and should be considered carefully. Here are the rules governing implicit creation of a "__hash__()" method. Note that you cannot both have an explicit "__hash__()" method in your dataclass and set "unsafe_hash=True"; this will result in a "TypeError". If *eq* and *frozen* are both true, by default "@dataclass" will generate a "__hash__()" method for you. If *eq* is true and *frozen* is false, "__hash__()" will be set to "None", marking it unhashable (which it is, since it is mutable). If *eq* is false, "__hash__()" will be left untouched meaning the "__hash__()" method of the superclass will be used (if the superclass is "object", this means it will fall back to id-based hashing). * *frozen*: If true (the default is "False"), assigning to fields will generate an exception. This emulates read-only frozen instances. If "__setattr__()" or "__delattr__()" is defined in the class, then "TypeError" is raised. See the discussion below. * *match_args*: If true (the default is "True"), the "__match_args__" tuple will be created from the list of non keyword-only parameters to the generated "__init__()" method (even if "__init__()" is not generated, see above). If false, or if "__match_args__" is already defined in the class, then "__match_args__" will not be generated. Added in version 3.10. * *kw_only*: If true (the default value is "False"), then all fields will be marked as keyword-only. If a field is marked as keyword-only, then the only effect is that the "__init__()" parameter generated from a keyword-only field must be specified with a keyword when "__init__()" is called. See the *parameter* glossary entry for details. Also see the "KW_ONLY" section. Keyword-only fields are not included in "__match_args__". Added in version 3.10. * *slots*: If true (the default is "False"), "__slots__" attribute will be generated and new class will be returned instead of the original one. If "__slots__" is already defined in the class, then "TypeError" is raised. Warning: Calling no-arg "super()" in dataclasses using "slots=True" will result in the following exception being raised: "TypeError: super(type, obj): obj must be an instance or subtype of type". The two-arg "super()" is a valid workaround. See gh-90562 for full details. Warning: Passing parameters to a base class "__init_subclass__()" when using "slots=True" will result in a "TypeError". Either use "__init_subclass__" with no parameters or use default values as a workaround. See gh-91126 for full details. Added in version 3.10. Changed in version 3.11: If a field name is already included in the "__slots__" of a base class, it will not be included in the generated "__slots__" to prevent overriding them. Therefore, do not use "__slots__" to retrieve the field names of a dataclass. Use "fields()" instead. To be able to determine inherited slots, base class "__slots__" may be any iterable, but *not* an iterator. * *weakref_slot*: If true (the default is "False"), add a slot named “__weakref__”, which is required to make an instance "weakref-able". It is an error to specify "weakref_slot=True" without also specifying "slots=True". Added in version 3.11. "field"s may optionally specify a default value, using normal Python syntax: @dataclass class C: a: int # 'a' has no default value b: int = 0 # assign a default value for 'b' In this example, both "a" and "b" will be included in the added "__init__()" method, which will be defined as: def __init__(self, a: int, b: int = 0): "TypeError" will be raised if a field without a default value follows a field with a default value. This is true whether this occurs in a single class, or as a result of class inheritance. dataclasses.field(*, default=MISSING, default_factory=MISSING, init=True, repr=True, hash=None, compare=True, metadata=None, kw_only=MISSING) For common and simple use cases, no other functionality is required. There are, however, some dataclass features that require additional per-field information. To satisfy this need for additional information, you can replace the default field value with a call to the provided "field()" function. For example: @dataclass class C: mylist: list[int] = field(default_factory=list) c = C() c.mylist += [1, 2, 3] As shown above, the "MISSING" value is a sentinel object used to detect if some parameters are provided by the user. This sentinel is used because "None" is a valid value for some parameters with a distinct meaning. No code should directly use the "MISSING" value. The parameters to "field()" are: * *default*: If provided, this will be the default value for this field. This is needed because the "field()" call itself replaces the normal position of the default value. * *default_factory*: If provided, it must be a zero-argument callable that will be called when a default value is needed for this field. Among other purposes, this can be used to specify fields with mutable default values, as discussed below. It is an error to specify both *default* and *default_factory*. * *init*: If true (the default), this field is included as a parameter to the generated "__init__()" method. * *repr*: If true (the default), this field is included in the string returned by the generated "__repr__()" method. * *hash*: This can be a bool or "None". If true, this field is included in the generated "__hash__()" method. If false, this field is excluded from the generated "__hash__()". If "None" (the default), use the value of *compare*: this would normally be the expected behavior, since a field should be included in the hash if it’s used for comparisons. Setting this value to anything other than "None" is discouraged. One possible reason to set "hash=False" but "compare=True" would be if a field is expensive to compute a hash value for, that field is needed for equality testing, and there are other fields that contribute to the type’s hash value. Even if a field is excluded from the hash, it will still be used for comparisons. * *compare*: If true (the default), this field is included in the generated equality and comparison methods ("__eq__()", "__gt__()", et al.). * *metadata*: This can be a mapping or "None". "None" is treated as an empty dict. This value is wrapped in "MappingProxyType()" to make it read-only, and exposed on the "Field" object. It is not used at all by Data Classes, and is provided as a third-party extension mechanism. Multiple third-parties can each have their own key, to use as a namespace in the metadata. * *kw_only*: If true, this field will be marked as keyword-only. This is used when the generated "__init__()" method’s parameters are computed. Keyword-only fields are also not included in "__match_args__". Added in version 3.10. If the default value of a field is specified by a call to "field()", then the class attribute for this field will be replaced by the specified *default* value. If *default* is not provided, then the class attribute will be deleted. The intent is that after the "@dataclass" decorator runs, the class attributes will all contain the default values for the fields, just as if the default value itself were specified. For example, after: @dataclass class C: x: int y: int = field(repr=False) z: int = field(repr=False, default=10) t: int = 20 The class attribute "C.z" will be "10", the class attribute "C.t" will be "20", and the class attributes "C.x" and "C.y" will not be set. class dataclasses.Field "Field" objects describe each defined field. These objects are created internally, and are returned by the "fields()" module-level method (see below). Users should never instantiate a "Field" object directly. Its documented attributes are: * "name": The name of the field. * "type": The type of the field. * "default", "default_factory", "init", "repr", "hash", "compare", "metadata", and "kw_only" have the identical meaning and values as they do in the "field()" function. Other attributes may exist, but they are private and must not be inspected or relied on. class dataclasses.InitVar "InitVar[T]" type annotations describe variables that are init- only. Fields annotated with "InitVar" are considered pseudo-fields, and thus are neither returned by the "fields()" function nor used in any way except adding them as parameters to "__init__()" and an optional "__post_init__()". dataclasses.fields(class_or_instance) Returns a tuple of "Field" objects that define the fields for this dataclass. Accepts either a dataclass, or an instance of a dataclass. Raises "TypeError" if not passed a dataclass or instance of one. Does not return pseudo-fields which are "ClassVar" or "InitVar". dataclasses.asdict(obj, *, dict_factory=dict) Converts the dataclass *obj* to a dict (by using the factory function *dict_factory*). Each dataclass is converted to a dict of its fields, as "name: value" pairs. dataclasses, dicts, lists, and tuples are recursed into. Other objects are copied with "copy.deepcopy()". Example of using "asdict()" on nested dataclasses: @dataclass class Point: x: int y: int @dataclass class C: mylist: list[Point] p = Point(10, 20) assert asdict(p) == {'x': 10, 'y': 20} c = C([Point(0, 0), Point(10, 4)]) assert asdict(c) == {'mylist': [{'x': 0, 'y': 0}, {'x': 10, 'y': 4}]} To create a shallow copy, the following workaround may be used: {field.name: getattr(obj, field.name) for field in fields(obj)} "asdict()" raises "TypeError" if *obj* is not a dataclass instance. dataclasses.astuple(obj, *, tuple_factory=tuple) Converts the dataclass *obj* to a tuple (by using the factory function *tuple_factory*). Each dataclass is converted to a tuple of its field values. dataclasses, dicts, lists, and tuples are recursed into. Other objects are copied with "copy.deepcopy()". Continuing from the previous example: assert astuple(p) == (10, 20) assert astuple(c) == ([(0, 0), (10, 4)],) To create a shallow copy, the following workaround may be used: tuple(getattr(obj, field.name) for field in dataclasses.fields(obj)) "astuple()" raises "TypeError" if *obj* is not a dataclass instance. dataclasses.make_dataclass(cls_name, fields, *, bases=(), namespace=None, init=True, repr=True, eq=True, order=False, unsafe_hash=False, frozen=False, match_args=True, kw_only=False, slots=False, weakref_slot=False, module=None) Creates a new dataclass with name *cls_name*, fields as defined in *fields*, base classes as given in *bases*, and initialized with a namespace as given in *namespace*. *fields* is an iterable whose elements are each either "name", "(name, type)", or "(name, type, Field)". If just "name" is supplied, "typing.Any" is used for "type". The values of *init*, *repr*, *eq*, *order*, *unsafe_hash*, *frozen*, *match_args*, *kw_only*, *slots*, and *weakref_slot* have the same meaning as they do in "@dataclass". If *module* is defined, the "__module__" attribute of the dataclass is set to that value. By default, it is set to the module name of the caller. This function is not strictly required, because any Python mechanism for creating a new class with "__annotations__" can then apply the "@dataclass" function to convert that class to a dataclass. This function is provided as a convenience. For example: C = make_dataclass('C', [('x', int), 'y', ('z', int, field(default=5))], namespace={'add_one': lambda self: self.x + 1}) Is equivalent to: @dataclass class C: x: int y: 'typing.Any' z: int = 5 def add_one(self): return self.x + 1 dataclasses.replace(obj, /, **changes) Creates a new object of the same type as *obj*, replacing fields with values from *changes*. If *obj* is not a Data Class, raises "TypeError". If keys in *changes* are not field names of the given dataclass, raises "TypeError". The newly returned object is created by calling the "__init__()" method of the dataclass. This ensures that "__post_init__()", if present, is also called. Init-only variables without default values, if any exist, must be specified on the call to "replace()" so that they can be passed to "__init__()" and "__post_init__()". It is an error for *changes* to contain any fields that are defined as having "init=False". A "ValueError" will be raised in this case. Be forewarned about how "init=False" fields work during a call to "replace()". They are not copied from the source object, but rather are initialized in "__post_init__()", if they’re initialized at all. It is expected that "init=False" fields will be rarely and judiciously used. If they are used, it might be wise to have alternate class constructors, or perhaps a custom "replace()" (or similarly named) method which handles instance copying. Dataclass instances are also supported by generic function "copy.replace()". dataclasses.is_dataclass(obj) Return "True" if its parameter is a dataclass (including subclasses of a dataclass) or an instance of one, otherwise return "False". If you need to know if a class is an instance of a dataclass (and not a dataclass itself), then add a further check for "not isinstance(obj, type)": def is_dataclass_instance(obj): return is_dataclass(obj) and not isinstance(obj, type) dataclasses.MISSING A sentinel value signifying a missing default or default_factory. dataclasses.KW_ONLY A sentinel value used as a type annotation. Any fields after a pseudo-field with the type of "KW_ONLY" are marked as keyword-only fields. Note that a pseudo-field of type "KW_ONLY" is otherwise completely ignored. This includes the name of such a field. By convention, a name of "_" is used for a "KW_ONLY" field. Keyword- only fields signify "__init__()" parameters that must be specified as keywords when the class is instantiated. In this example, the fields "y" and "z" will be marked as keyword- only fields: @dataclass class Point: x: float _: KW_ONLY y: float z: float p = Point(0, y=1.5, z=2.0) In a single dataclass, it is an error to specify more than one field whose type is "KW_ONLY". Added in version 3.10. exception dataclasses.FrozenInstanceError Raised when an implicitly defined "__setattr__()" or "__delattr__()" is called on a dataclass which was defined with "frozen=True". It is a subclass of "AttributeError". Post-init processing ==================== dataclasses.__post_init__() When defined on the class, it will be called by the generated "__init__()", normally as "self.__post_init__()". However, if any "InitVar" fields are defined, they will also be passed to "__post_init__()" in the order they were defined in the class. If no "__init__()" method is generated, then "__post_init__()" will not automatically be called. Among other uses, this allows for initializing field values that depend on one or more other fields. For example: @dataclass class C: a: float b: float c: float = field(init=False) def __post_init__(self): self.c = self.a + self.b The "__init__()" method generated by "@dataclass" does not call base class "__init__()" methods. If the base class has an "__init__()" method that has to be called, it is common to call this method in a "__post_init__()" method: class Rectangle: def __init__(self, height, width): self.height = height self.width = width @dataclass class Square(Rectangle): side: float def __post_init__(self): super().__init__(self.side, self.side) Note, however, that in general the dataclass-generated "__init__()" methods don’t need to be called, since the derived dataclass will take care of initializing all fields of any base class that is a dataclass itself. See the section below on init-only variables for ways to pass parameters to "__post_init__()". Also see the warning about how "replace()" handles "init=False" fields. Class variables =============== One of the few places where "@dataclass" actually inspects the type of a field is to determine if a field is a class variable as defined in **PEP 526**. It does this by checking if the type of the field is "typing.ClassVar". If a field is a "ClassVar", it is excluded from consideration as a field and is ignored by the dataclass mechanisms. Such "ClassVar" pseudo-fields are not returned by the module-level "fields()" function. Init-only variables =================== Another place where "@dataclass" inspects a type annotation is to determine if a field is an init-only variable. It does this by seeing if the type of a field is of type "InitVar". If a field is an "InitVar", it is considered a pseudo-field called an init-only field. As it is not a true field, it is not returned by the module-level "fields()" function. Init-only fields are added as parameters to the generated "__init__()" method, and are passed to the optional "__post_init__()" method. They are not otherwise used by dataclasses. For example, suppose a field will be initialized from a database, if a value is not provided when creating the class: @dataclass class C: i: int j: int | None = None database: InitVar[DatabaseType | None] = None def __post_init__(self, database): if self.j is None and database is not None: self.j = database.lookup('j') c = C(10, database=my_database) In this case, "fields()" will return "Field" objects for "i" and "j", but not for "database". Frozen instances ================ It is not possible to create truly immutable Python objects. However, by passing "frozen=True" to the "@dataclass" decorator you can emulate immutability. In that case, dataclasses will add "__setattr__()" and "__delattr__()" methods to the class. These methods will raise a "FrozenInstanceError" when invoked. There is a tiny performance penalty when using "frozen=True": "__init__()" cannot use simple assignment to initialize fields, and must use "object.__setattr__()". Inheritance =========== When the dataclass is being created by the "@dataclass" decorator, it looks through all of the class’s base classes in reverse MRO (that is, starting at "object") and, for each dataclass that it finds, adds the fields from that base class to an ordered mapping of fields. After all of the base class fields are added, it adds its own fields to the ordered mapping. All of the generated methods will use this combined, calculated ordered mapping of fields. Because the fields are in insertion order, derived classes override base classes. An example: @dataclass class Base: x: Any = 15.0 y: int = 0 @dataclass class C(Base): z: int = 10 x: int = 15 The final list of fields is, in order, "x", "y", "z". The final type of "x" is "int", as specified in class "C". The generated "__init__()" method for "C" will look like: def __init__(self, x: int = 15, y: int = 0, z: int = 10): Re-ordering of keyword-only parameters in "__init__()" ====================================================== After the parameters needed for "__init__()" are computed, any keyword-only parameters are moved to come after all regular (non- keyword-only) parameters. This is a requirement of how keyword-only parameters are implemented in Python: they must come after non- keyword-only parameters. In this example, "Base.y", "Base.w", and "D.t" are keyword-only fields, and "Base.x" and "D.z" are regular fields: @dataclass class Base: x: Any = 15.0 _: KW_ONLY y: int = 0 w: int = 1 @dataclass class D(Base): z: int = 10 t: int = field(kw_only=True, default=0) The generated "__init__()" method for "D" will look like: def __init__(self, x: Any = 15.0, z: int = 10, *, y: int = 0, w: int = 1, t: int = 0): Note that the parameters have been re-ordered from how they appear in the list of fields: parameters derived from regular fields are followed by parameters derived from keyword-only fields. The relative ordering of keyword-only parameters is maintained in the re-ordered "__init__()" parameter list. Default factory functions ========================= If a "field()" specifies a *default_factory*, it is called with zero arguments when a default value for the field is needed. For example, to create a new instance of a list, use: mylist: list = field(default_factory=list) If a field is excluded from "__init__()" (using "init=False") and the field also specifies *default_factory*, then the default factory function will always be called from the generated "__init__()" function. This happens because there is no other way to give the field an initial value. Mutable default values ====================== Python stores default member variable values in class attributes. Consider this example, not using dataclasses: class C: x = [] def add(self, element): self.x.append(element) o1 = C() o2 = C() o1.add(1) o2.add(2) assert o1.x == [1, 2] assert o1.x is o2.x Note that the two instances of class "C" share the same class variable "x", as expected. Using dataclasses, *if* this code was valid: @dataclass class D: x: list = [] # This code raises ValueError def add(self, element): self.x.append(element) it would generate code similar to: class D: x = [] def __init__(self, x=x): self.x = x def add(self, element): self.x.append(element) assert D().x is D().x This has the same issue as the original example using class "C". That is, two instances of class "D" that do not specify a value for "x" when creating a class instance will share the same copy of "x". Because dataclasses just use normal Python class creation they also share this behavior. There is no general way for Data Classes to detect this condition. Instead, the "@dataclass" decorator will raise a "ValueError" if it detects an unhashable default parameter. The assumption is that if a value is unhashable, it is mutable. This is a partial solution, but it does protect against many common errors. Using default factory functions is a way to create new instances of mutable types as default values for fields: @dataclass class D: x: list = field(default_factory=list) assert D().x is not D().x Changed in version 3.11: Instead of looking for and disallowing objects of type "list", "dict", or "set", unhashable objects are now not allowed as default values. Unhashability is used to approximate mutability. Descriptor-typed fields ======================= Fields that are assigned descriptor objects as their default value have the following special behaviors: * The value for the field passed to the dataclass’s "__init__()" method is passed to the descriptor’s "__set__()" method rather than overwriting the descriptor object. * Similarly, when getting or setting the field, the descriptor’s "__get__()" or "__set__()" method is called rather than returning or overwriting the descriptor object. * To determine whether a field contains a default value, "@dataclass" will call the descriptor’s "__get__()" method using its class access form: "descriptor.__get__(obj=None, type=cls)". If the descriptor returns a value in this case, it will be used as the field’s default. On the other hand, if the descriptor raises "AttributeError" in this situation, no default value will be provided for the field. class IntConversionDescriptor: def __init__(self, *, default): self._default = default def __set_name__(self, owner, name): self._name = "_" + name def __get__(self, obj, type): if obj is None: return self._default return getattr(obj, self._name, self._default) def __set__(self, obj, value): setattr(obj, self._name, int(value)) @dataclass class InventoryItem: quantity_on_hand: IntConversionDescriptor = IntConversionDescriptor(default=100) i = InventoryItem() print(i.quantity_on_hand) # 100 i.quantity_on_hand = 2.5 # calls __set__ with 2.5 print(i.quantity_on_hand) # 2 Note that if a field is annotated with a descriptor type, but is not assigned a descriptor object as its default value, the field will act like a normal field. Data Types ********** The modules described in this chapter provide a variety of specialized data types such as dates and times, fixed-type arrays, heap queues, double-ended queues, and enumerations. Python also provides some built-in data types, in particular, "dict", "list", "set" and "frozenset", and "tuple". The "str" class is used to hold Unicode strings, and the "bytes" and "bytearray" classes are used to hold binary data. The following modules are documented in this chapter: * "datetime" — Basic date and time types * Aware and Naive Objects * Constants * Available Types * Common Properties * Determining if an Object is Aware or Naive * "timedelta" Objects * Examples of usage: "timedelta" * "date" Objects * Examples of Usage: "date" * "datetime" Objects * Examples of Usage: "datetime" * "time" Objects * Examples of Usage: "time" * "tzinfo" Objects * "timezone" Objects * "strftime()" and "strptime()" Behavior * "strftime()" and "strptime()" Format Codes * Technical Detail * "zoneinfo" — IANA time zone support * Using "ZoneInfo" * Data sources * Configuring the data sources * Compile-time configuration * Environment configuration * Runtime configuration * The "ZoneInfo" class * String representations * Pickle serialization * Functions * Globals * Exceptions and warnings * "calendar" — General calendar-related functions * Command-Line Usage * "collections" — Container datatypes * "ChainMap" objects * "ChainMap" Examples and Recipes * "Counter" objects * "deque" objects * "deque" Recipes * "defaultdict" objects * "defaultdict" Examples * "namedtuple()" Factory Function for Tuples with Named Fields * "OrderedDict" objects * "OrderedDict" Examples and Recipes * "UserDict" objects * "UserList" objects * "UserString" objects * "collections.abc" — Abstract Base Classes for Containers * Collections Abstract Base Classes * Collections Abstract Base Classes – Detailed Descriptions * Examples and Recipes * "heapq" — Heap queue algorithm * Basic Examples * Priority Queue Implementation Notes * Theory * "bisect" — Array bisection algorithm * Performance Notes * Searching Sorted Lists * Examples * "array" — Efficient arrays of numeric values * "weakref" — Weak references * Weak Reference Objects * Example * Finalizer Objects * Comparing finalizers with "__del__()" methods * "types" — Dynamic type creation and names for built-in types * Dynamic Type Creation * Standard Interpreter Types * Additional Utility Classes and Functions * Coroutine Utility Functions * "copy" — Shallow and deep copy operations * "pprint" — Data pretty printer * Functions * PrettyPrinter Objects * Example * "reprlib" — Alternate "repr()" implementation * Repr Objects * Subclassing Repr Objects * "enum" — Support for enumerations * Module Contents * Data Types * Supported "__dunder__" names * Supported "_sunder_" names * Utilities and Decorators * Notes * "graphlib" — Functionality to operate with graph-like structures * Exceptions "datetime" — Basic date and time types ************************************** **Source code:** Lib/datetime.py ====================================================================== The "datetime" module supplies classes for manipulating dates and times. While date and time arithmetic is supported, the focus of the implementation is on efficient attribute extraction for output formatting and manipulation. Tip: Skip to the format codes. See also: Module "calendar" General calendar related functions. Module "time" Time access and conversions. Module "zoneinfo" Concrete time zones representing the IANA time zone database. Package dateutil Third-party library with expanded time zone and parsing support. Package DateType Third-party library that introduces distinct static types to e.g. allow *static type checkers* to differentiate between naive and aware datetimes. Aware and Naive Objects ======================= Date and time objects may be categorized as “aware” or “naive” depending on whether or not they include time zone information. With sufficient knowledge of applicable algorithmic and political time adjustments, such as time zone and daylight saving time information, an **aware** object can locate itself relative to other aware objects. An aware object represents a specific moment in time that is not open to interpretation. [1] A **naive** object does not contain enough information to unambiguously locate itself relative to other date/time objects. Whether a naive object represents Coordinated Universal Time (UTC), local time, or time in some other time zone is purely up to the program, just like it is up to the program whether a particular number represents metres, miles, or mass. Naive objects are easy to understand and to work with, at the cost of ignoring some aspects of reality. For applications requiring aware objects, "datetime" and "time" objects have an optional time zone information attribute, "tzinfo", that can be set to an instance of a subclass of the abstract "tzinfo" class. These "tzinfo" objects capture information about the offset from UTC time, the time zone name, and whether daylight saving time is in effect. Only one concrete "tzinfo" class, the "timezone" class, is supplied by the "datetime" module. The "timezone" class can represent simple time zones with fixed offsets from UTC, such as UTC itself or North American EST and EDT time zones. Supporting time zones at deeper levels of detail is up to the application. The rules for time adjustment across the world are more political than rational, change frequently, and there is no standard suitable for every application aside from UTC. Constants ========= The "datetime" module exports the following constants: datetime.MINYEAR The smallest year number allowed in a "date" or "datetime" object. "MINYEAR" is 1. datetime.MAXYEAR The largest year number allowed in a "date" or "datetime" object. "MAXYEAR" is 9999. datetime.UTC Alias for the UTC time zone singleton "datetime.timezone.utc". Added in version 3.11. Available Types =============== class datetime.date An idealized naive date, assuming the current Gregorian calendar always was, and always will be, in effect. Attributes: "year", "month", and "day". class datetime.time An idealized time, independent of any particular day, assuming that every day has exactly 24*60*60 seconds. (There is no notion of “leap seconds” here.) Attributes: "hour", "minute", "second", "microsecond", and "tzinfo". class datetime.datetime A combination of a date and a time. Attributes: "year", "month", "day", "hour", "minute", "second", "microsecond", and "tzinfo". class datetime.timedelta A duration expressing the difference between two "datetime" or "date" instances to microsecond resolution. class datetime.tzinfo An abstract base class for time zone information objects. These are used by the "datetime" and "time" classes to provide a customizable notion of time adjustment (for example, to account for time zone and/or daylight saving time). class datetime.timezone A class that implements the "tzinfo" abstract base class as a fixed offset from the UTC. Added in version 3.2. Objects of these types are immutable. Subclass relationships: object timedelta tzinfo timezone time date datetime Common Properties ----------------- The "date", "datetime", "time", and "timezone" types share these common features: * Objects of these types are immutable. * Objects of these types are *hashable*, meaning that they can be used as dictionary keys. * Objects of these types support efficient pickling via the "pickle" module. Determining if an Object is Aware or Naive ------------------------------------------ Objects of the "date" type are always naive. An object of type "time" or "datetime" may be aware or naive. A "datetime" object "d" is aware if both of the following hold: 1. "d.tzinfo" is not "None" 2. "d.tzinfo.utcoffset(d)" does not return "None" Otherwise, "d" is naive. A "time" object "t" is aware if both of the following hold: 1. "t.tzinfo" is not "None" 2. "t.tzinfo.utcoffset(None)" does not return "None". Otherwise, "t" is naive. The distinction between aware and naive doesn’t apply to "timedelta" objects. "timedelta" Objects =================== A "timedelta" object represents a duration, the difference between two "datetime" or "date" instances. class datetime.timedelta(days=0, seconds=0, microseconds=0, milliseconds=0, minutes=0, hours=0, weeks=0) All arguments are optional and default to 0. Arguments may be integers or floats, and may be positive or negative. Only *days*, *seconds* and *microseconds* are stored internally. Arguments are converted to those units: * A millisecond is converted to 1000 microseconds. * A minute is converted to 60 seconds. * An hour is converted to 3600 seconds. * A week is converted to 7 days. and days, seconds and microseconds are then normalized so that the representation is unique, with * "0 <= microseconds < 1000000" * "0 <= seconds < 3600*24" (the number of seconds in one day) * "-999999999 <= days <= 999999999" The following example illustrates how any arguments besides *days*, *seconds* and *microseconds* are “merged” and normalized into those three resulting attributes: >>> from datetime import timedelta >>> delta = timedelta( ... days=50, ... seconds=27, ... microseconds=10, ... milliseconds=29000, ... minutes=5, ... hours=8, ... weeks=2 ... ) >>> # Only days, seconds, and microseconds remain >>> delta datetime.timedelta(days=64, seconds=29156, microseconds=10) If any argument is a float and there are fractional microseconds, the fractional microseconds left over from all arguments are combined and their sum is rounded to the nearest microsecond using round-half-to-even tiebreaker. If no argument is a float, the conversion and normalization processes are exact (no information is lost). If the normalized value of days lies outside the indicated range, "OverflowError" is raised. Note that normalization of negative values may be surprising at first. For example: >>> from datetime import timedelta >>> d = timedelta(microseconds=-1) >>> (d.days, d.seconds, d.microseconds) (-1, 86399, 999999) Since the string representation of "timedelta" objects can be confusing, use the following recipe to produce a more readable format: >>> def pretty_timedelta(td): ... if td.days >= 0: ... return str(td) ... return f'-({-td!s})' ... >>> d = timedelta(hours=-1) >>> str(d) # not human-friendly '-1 day, 23:00:00' >>> pretty_timedelta(d) '-(1:00:00)' Class attributes: timedelta.min The most negative "timedelta" object, "timedelta(-999999999)". timedelta.max The most positive "timedelta" object, "timedelta(days=999999999, hours=23, minutes=59, seconds=59, microseconds=999999)". timedelta.resolution The smallest possible difference between non-equal "timedelta" objects, "timedelta(microseconds=1)". Note that, because of normalization, "timedelta.max" is greater than "-timedelta.min". "-timedelta.max" is not representable as a "timedelta" object. Instance attributes (read-only): timedelta.days Between -999,999,999 and 999,999,999 inclusive. timedelta.seconds Between 0 and 86,399 inclusive. Caution: It is a somewhat common bug for code to unintentionally use this attribute when it is actually intended to get a "total_seconds()" value instead: >>> from datetime import timedelta >>> duration = timedelta(seconds=11235813) >>> duration.days, duration.seconds (130, 3813) >>> duration.total_seconds() 11235813.0 timedelta.microseconds Between 0 and 999,999 inclusive. Supported operations: +----------------------------------+-------------------------------------------------+ | Operation | Result | |==================================|=================================================| | "t1 = t2 + t3" | Sum of "t2" and "t3". Afterwards "t1 - t2 == | | | t3" and "t1 - t3 == t2" are true. (1) | +----------------------------------+-------------------------------------------------+ | "t1 = t2 - t3" | Difference of "t2" and "t3". Afterwards "t1 == | | | t2 - t3" and "t2 == t1 + t3" are true. (1)(6) | +----------------------------------+-------------------------------------------------+ | "t1 = t2 * i or t1 = i * t2" | Delta multiplied by an integer. Afterwards "t1 | | | // i == t2" is true, provided "i != 0". | +----------------------------------+-------------------------------------------------+ | | In general, "t1 * i == t1 * (i-1) + t1" is | | | true. (1) | +----------------------------------+-------------------------------------------------+ | "t1 = t2 * f or t1 = f * t2" | Delta multiplied by a float. The result is | | | rounded to the nearest multiple of | | | timedelta.resolution using round-half-to-even. | +----------------------------------+-------------------------------------------------+ | "f = t2 / t3" | Division (3) of overall duration "t2" by | | | interval unit "t3". Returns a "float" object. | +----------------------------------+-------------------------------------------------+ | "t1 = t2 / f or t1 = t2 / i" | Delta divided by a float or an int. The result | | | is rounded to the nearest multiple of | | | timedelta.resolution using round-half-to-even. | +----------------------------------+-------------------------------------------------+ | "t1 = t2 // i" or "t1 = t2 // | The floor is computed and the remainder (if | | t3" | any) is thrown away. In the second case, an | | | integer is returned. (3) | +----------------------------------+-------------------------------------------------+ | "t1 = t2 % t3" | The remainder is computed as a "timedelta" | | | object. (3) | +----------------------------------+-------------------------------------------------+ | "q, r = divmod(t1, t2)" | Computes the quotient and the remainder: "q = | | | t1 // t2" (3) and "r = t1 % t2". "q" is an | | | integer and "r" is a "timedelta" object. | +----------------------------------+-------------------------------------------------+ | "+t1" | Returns a "timedelta" object with the same | | | value. (2) | +----------------------------------+-------------------------------------------------+ | "-t1" | Equivalent to "timedelta(-t1.days, -t1.seconds, | | | -t1.microseconds)", and to "t1 * -1". (1)(4) | +----------------------------------+-------------------------------------------------+ | "abs(t)" | Equivalent to "+t" when "t.days >= 0", and to | | | "-t" when "t.days < 0". (2) | +----------------------------------+-------------------------------------------------+ | "str(t)" | Returns a string in the form "[D day[s], | | | ][H]H:MM:SS[.UUUUUU]", where D is negative for | | | negative "t". (5) | +----------------------------------+-------------------------------------------------+ | "repr(t)" | Returns a string representation of the | | | "timedelta" object as a constructor call with | | | canonical attribute values. | +----------------------------------+-------------------------------------------------+ Notes: 1. This is exact but may overflow. 2. This is exact and cannot overflow. 3. Division by zero raises "ZeroDivisionError". 4. "-timedelta.max" is not representable as a "timedelta" object. 5. String representations of "timedelta" objects are normalized similarly to their internal representation. This leads to somewhat unusual results for negative timedeltas. For example: >>> timedelta(hours=-5) datetime.timedelta(days=-1, seconds=68400) >>> print(_) -1 day, 19:00:00 6. The expression "t2 - t3" will always be equal to the expression "t2 + (-t3)" except when t3 is equal to "timedelta.max"; in that case the former will produce a result while the latter will overflow. In addition to the operations listed above, "timedelta" objects support certain additions and subtractions with "date" and "datetime" objects (see below). Changed in version 3.2: Floor division and true division of a "timedelta" object by another "timedelta" object are now supported, as are remainder operations and the "divmod()" function. True division and multiplication of a "timedelta" object by a "float" object are now supported. "timedelta" objects support equality and order comparisons. In Boolean contexts, a "timedelta" object is considered to be true if and only if it isn’t equal to "timedelta(0)". Instance methods: timedelta.total_seconds() Return the total number of seconds contained in the duration. Equivalent to "td / timedelta(seconds=1)". For interval units other than seconds, use the division form directly (e.g. "td / timedelta(microseconds=1)"). Note that for very large time intervals (greater than 270 years on most platforms) this method will lose microsecond accuracy. Added in version 3.2. Examples of usage: "timedelta" ------------------------------ An additional example of normalization: >>> # Components of another_year add up to exactly 365 days >>> from datetime import timedelta >>> year = timedelta(days=365) >>> another_year = timedelta(weeks=40, days=84, hours=23, ... minutes=50, seconds=600) >>> year == another_year True >>> year.total_seconds() 31536000.0 Examples of "timedelta" arithmetic: >>> from datetime import timedelta >>> year = timedelta(days=365) >>> ten_years = 10 * year >>> ten_years datetime.timedelta(days=3650) >>> ten_years.days // 365 10 >>> nine_years = ten_years - year >>> nine_years datetime.timedelta(days=3285) >>> three_years = nine_years // 3 >>> three_years, three_years.days // 365 (datetime.timedelta(days=1095), 3) "date" Objects ============== A "date" object represents a date (year, month and day) in an idealized calendar, the current Gregorian calendar indefinitely extended in both directions. January 1 of year 1 is called day number 1, January 2 of year 1 is called day number 2, and so on. [2] class datetime.date(year, month, day) All arguments are required. Arguments must be integers, in the following ranges: * "MINYEAR <= year <= MAXYEAR" * "1 <= month <= 12" * "1 <= day <= number of days in the given month and year" If an argument outside those ranges is given, "ValueError" is raised. Other constructors, all class methods: classmethod date.today() Return the current local date. This is equivalent to "date.fromtimestamp(time.time())". classmethod date.fromtimestamp(timestamp) Return the local date corresponding to the POSIX timestamp, such as is returned by "time.time()". This may raise "OverflowError", if the timestamp is out of the range of values supported by the platform C "localtime()" function, and "OSError" on "localtime()" failure. It’s common for this to be restricted to years from 1970 through 2038. Note that on non-POSIX systems that include leap seconds in their notion of a timestamp, leap seconds are ignored by "fromtimestamp()". Changed in version 3.3: Raise "OverflowError" instead of "ValueError" if the timestamp is out of the range of values supported by the platform C "localtime()" function. Raise "OSError" instead of "ValueError" on "localtime()" failure. classmethod date.fromordinal(ordinal) Return the date corresponding to the proleptic Gregorian ordinal, where January 1 of year 1 has ordinal 1. "ValueError" is raised unless "1 <= ordinal <= date.max.toordinal()". For any date "d", "date.fromordinal(d.toordinal()) == d". classmethod date.fromisoformat(date_string) Return a "date" corresponding to a *date_string* given in any valid ISO 8601 format, with the following exceptions: 1. Reduced precision dates are not currently supported ("YYYY-MM", "YYYY"). 2. Extended date representations are not currently supported ("±YYYYYY-MM-DD"). 3. Ordinal dates are not currently supported ("YYYY-OOO"). Examples: >>> from datetime import date >>> date.fromisoformat('2019-12-04') datetime.date(2019, 12, 4) >>> date.fromisoformat('20191204') datetime.date(2019, 12, 4) >>> date.fromisoformat('2021-W01-1') datetime.date(2021, 1, 4) Added in version 3.7. Changed in version 3.11: Previously, this method only supported the format "YYYY-MM-DD". classmethod date.fromisocalendar(year, week, day) Return a "date" corresponding to the ISO calendar date specified by year, week and day. This is the inverse of the function "date.isocalendar()". Added in version 3.8. Class attributes: date.min The earliest representable date, "date(MINYEAR, 1, 1)". date.max The latest representable date, "date(MAXYEAR, 12, 31)". date.resolution The smallest possible difference between non-equal date objects, "timedelta(days=1)". Instance attributes (read-only): date.year Between "MINYEAR" and "MAXYEAR" inclusive. date.month Between 1 and 12 inclusive. date.day Between 1 and the number of days in the given month of the given year. Supported operations: +---------------------------------+------------------------------------------------+ | Operation | Result | |=================================|================================================| | "date2 = date1 + timedelta" | "date2" will be "timedelta.days" days after | | | "date1". (1) | +---------------------------------+------------------------------------------------+ | "date2 = date1 - timedelta" | Computes "date2" such that "date2 + timedelta | | | == date1". (2) | +---------------------------------+------------------------------------------------+ | "timedelta = date1 - date2" | (3) | +---------------------------------+------------------------------------------------+ | "date1 == date2" "date1 != | Equality comparison. (4) | | date2" | | +---------------------------------+------------------------------------------------+ | "date1 < date2" "date1 > date2" | Order comparison. (5) | | "date1 <= date2" "date1 >= | | | date2" | | +---------------------------------+------------------------------------------------+ Notes: 1. *date2* is moved forward in time if "timedelta.days > 0", or backward if "timedelta.days < 0". Afterward "date2 - date1 == timedelta.days". "timedelta.seconds" and "timedelta.microseconds" are ignored. "OverflowError" is raised if "date2.year" would be smaller than "MINYEAR" or larger than "MAXYEAR". 2. "timedelta.seconds" and "timedelta.microseconds" are ignored. 3. This is exact, and cannot overflow. "timedelta.seconds" and "timedelta.microseconds" are 0, and "date2 + timedelta == date1" after. 4. "date" objects are equal if they represent the same date. "date" objects that are not also "datetime" instances are never equal to "datetime" objects, even if they represent the same date. 5. *date1* is considered less than *date2* when *date1* precedes *date2* in time. In other words, "date1 < date2" if and only if "date1.toordinal() < date2.toordinal()". Order comparison between a "date" object that is not also a "datetime" instance and a "datetime" object raises "TypeError". Changed in version 3.13: Comparison between "datetime" object and an instance of the "date" subclass that is not a "datetime" subclass no longer converts the latter to "date", ignoring the time part and the time zone. The default behavior can be changed by overriding the special comparison methods in subclasses. In Boolean contexts, all "date" objects are considered to be true. Instance methods: date.replace(year=self.year, month=self.month, day=self.day) Return a new "date" object with the same values, but with specified parameters updated. Example: >>> from datetime import date >>> d = date(2002, 12, 31) >>> d.replace(day=26) datetime.date(2002, 12, 26) The generic function "copy.replace()" also supports "date" objects. date.timetuple() Return a "time.struct_time" such as returned by "time.localtime()". The hours, minutes and seconds are 0, and the DST flag is -1. "d.timetuple()" is equivalent to: time.struct_time((d.year, d.month, d.day, 0, 0, 0, d.weekday(), yday, -1)) where "yday = d.toordinal() - date(d.year, 1, 1).toordinal() + 1" is the day number within the current year starting with 1 for January 1st. date.toordinal() Return the proleptic Gregorian ordinal of the date, where January 1 of year 1 has ordinal 1. For any "date" object "d", "date.fromordinal(d.toordinal()) == d". date.weekday() Return the day of the week as an integer, where Monday is 0 and Sunday is 6. For example, "date(2002, 12, 4).weekday() == 2", a Wednesday. See also "isoweekday()". date.isoweekday() Return the day of the week as an integer, where Monday is 1 and Sunday is 7. For example, "date(2002, 12, 4).isoweekday() == 3", a Wednesday. See also "weekday()", "isocalendar()". date.isocalendar() Return a *named tuple* object with three components: "year", "week" and "weekday". The ISO calendar is a widely used variant of the Gregorian calendar. [3] The ISO year consists of 52 or 53 full weeks, and where a week starts on a Monday and ends on a Sunday. The first week of an ISO year is the first (Gregorian) calendar week of a year containing a Thursday. This is called week number 1, and the ISO year of that Thursday is the same as its Gregorian year. For example, 2004 begins on a Thursday, so the first week of ISO year 2004 begins on Monday, 29 Dec 2003 and ends on Sunday, 4 Jan 2004: >>> from datetime import date >>> date(2003, 12, 29).isocalendar() datetime.IsoCalendarDate(year=2004, week=1, weekday=1) >>> date(2004, 1, 4).isocalendar() datetime.IsoCalendarDate(year=2004, week=1, weekday=7) Changed in version 3.9: Result changed from a tuple to a *named tuple*. date.isoformat() Return a string representing the date in ISO 8601 format, "YYYY-MM- DD": >>> from datetime import date >>> date(2002, 12, 4).isoformat() '2002-12-04' date.__str__() For a date "d", "str(d)" is equivalent to "d.isoformat()". date.ctime() Return a string representing the date: >>> from datetime import date >>> date(2002, 12, 4).ctime() 'Wed Dec 4 00:00:00 2002' "d.ctime()" is equivalent to: time.ctime(time.mktime(d.timetuple())) on platforms where the native C "ctime()" function (which "time.ctime()" invokes, but which "date.ctime()" does not invoke) conforms to the C standard. date.strftime(format) Return a string representing the date, controlled by an explicit format string. Format codes referring to hours, minutes or seconds will see 0 values. See also strftime() and strptime() Behavior and "date.isoformat()". date.__format__(format) Same as "date.strftime()". This makes it possible to specify a format string for a "date" object in formatted string literals and when using "str.format()". See also strftime() and strptime() Behavior and "date.isoformat()". Examples of Usage: "date" ------------------------- Example of counting days to an event: >>> import time >>> from datetime import date >>> today = date.today() >>> today datetime.date(2007, 12, 5) >>> today == date.fromtimestamp(time.time()) True >>> my_birthday = date(today.year, 6, 24) >>> if my_birthday < today: ... my_birthday = my_birthday.replace(year=today.year + 1) ... >>> my_birthday datetime.date(2008, 6, 24) >>> time_to_birthday = abs(my_birthday - today) >>> time_to_birthday.days 202 More examples of working with "date": >>> from datetime import date >>> d = date.fromordinal(730920) # 730920th day after 1. 1. 0001 >>> d datetime.date(2002, 3, 11) >>> # Methods related to formatting string output >>> d.isoformat() '2002-03-11' >>> d.strftime("%d/%m/%y") '11/03/02' >>> d.strftime("%A %d. %B %Y") 'Monday 11. March 2002' >>> d.ctime() 'Mon Mar 11 00:00:00 2002' >>> 'The {1} is {0:%d}, the {2} is {0:%B}.'.format(d, "day", "month") 'The day is 11, the month is March.' >>> # Methods for to extracting 'components' under different calendars >>> t = d.timetuple() >>> for i in t: ... print(i) 2002 # year 3 # month 11 # day 0 0 0 0 # weekday (0 = Monday) 70 # 70th day in the year -1 >>> ic = d.isocalendar() >>> for i in ic: ... print(i) 2002 # ISO year 11 # ISO week number 1 # ISO day number ( 1 = Monday ) >>> # A date object is immutable; all operations produce a new object >>> d.replace(year=2005) datetime.date(2005, 3, 11) "datetime" Objects ================== A "datetime" object is a single object containing all the information from a "date" object and a "time" object. Like a "date" object, "datetime" assumes the current Gregorian calendar extended in both directions; like a "time" object, "datetime" assumes there are exactly 3600*24 seconds in every day. Constructor: class datetime.datetime(year, month, day, hour=0, minute=0, second=0, microsecond=0, tzinfo=None, *, fold=0) The *year*, *month* and *day* arguments are required. *tzinfo* may be "None", or an instance of a "tzinfo" subclass. The remaining arguments must be integers in the following ranges: * "MINYEAR <= year <= MAXYEAR", * "1 <= month <= 12", * "1 <= day <= number of days in the given month and year", * "0 <= hour < 24", * "0 <= minute < 60", * "0 <= second < 60", * "0 <= microsecond < 1000000", * "fold in [0, 1]". If an argument outside those ranges is given, "ValueError" is raised. Changed in version 3.6: Added the *fold* parameter. Other constructors, all class methods: classmethod datetime.today() Return the current local date and time, with "tzinfo" "None". Equivalent to: datetime.fromtimestamp(time.time()) See also "now()", "fromtimestamp()". This method is functionally equivalent to "now()", but without a "tz" parameter. classmethod datetime.now(tz=None) Return the current local date and time. If optional argument *tz* is "None" or not specified, this is like "today()", but, if possible, supplies more precision than can be gotten from going through a "time.time()" timestamp (for example, this may be possible on platforms supplying the C "gettimeofday()" function). If *tz* is not "None", it must be an instance of a "tzinfo" subclass, and the current date and time are converted to *tz*’s time zone. This function is preferred over "today()" and "utcnow()". Note: Subsequent calls to "datetime.now()" may return the same instant depending on the precision of the underlying clock. classmethod datetime.utcnow() Return the current UTC date and time, with "tzinfo" "None". This is like "now()", but returns the current UTC date and time, as a naive "datetime" object. An aware current UTC datetime can be obtained by calling "datetime.now(timezone.utc)". See also "now()". Warning: Because naive "datetime" objects are treated by many "datetime" methods as local times, it is preferred to use aware datetimes to represent times in UTC. As such, the recommended way to create an object representing the current time in UTC is by calling "datetime.now(timezone.utc)". Deprecated since version 3.12: Use "datetime.now()" with "UTC" instead. classmethod datetime.fromtimestamp(timestamp, tz=None) Return the local date and time corresponding to the POSIX timestamp, such as is returned by "time.time()". If optional argument *tz* is "None" or not specified, the timestamp is converted to the platform’s local date and time, and the returned "datetime" object is naive. If *tz* is not "None", it must be an instance of a "tzinfo" subclass, and the timestamp is converted to *tz*’s time zone. "fromtimestamp()" may raise "OverflowError", if the timestamp is out of the range of values supported by the platform C "localtime()" or "gmtime()" functions, and "OSError" on "localtime()" or "gmtime()" failure. It’s common for this to be restricted to years in 1970 through 2038. Note that on non-POSIX systems that include leap seconds in their notion of a timestamp, leap seconds are ignored by "fromtimestamp()", and then it’s possible to have two timestamps differing by a second that yield identical "datetime" objects. This method is preferred over "utcfromtimestamp()". Changed in version 3.3: Raise "OverflowError" instead of "ValueError" if the timestamp is out of the range of values supported by the platform C "localtime()" or "gmtime()" functions. Raise "OSError" instead of "ValueError" on "localtime()" or "gmtime()" failure. Changed in version 3.6: "fromtimestamp()" may return instances with "fold" set to 1. classmethod datetime.utcfromtimestamp(timestamp) Return the UTC "datetime" corresponding to the POSIX timestamp, with "tzinfo" "None". (The resulting object is naive.) This may raise "OverflowError", if the timestamp is out of the range of values supported by the platform C "gmtime()" function, and "OSError" on "gmtime()" failure. It’s common for this to be restricted to years in 1970 through 2038. To get an aware "datetime" object, call "fromtimestamp()": datetime.fromtimestamp(timestamp, timezone.utc) On the POSIX compliant platforms, it is equivalent to the following expression: datetime(1970, 1, 1, tzinfo=timezone.utc) + timedelta(seconds=timestamp) except the latter formula always supports the full years range: between "MINYEAR" and "MAXYEAR" inclusive. Warning: Because naive "datetime" objects are treated by many "datetime" methods as local times, it is preferred to use aware datetimes to represent times in UTC. As such, the recommended way to create an object representing a specific timestamp in UTC is by calling "datetime.fromtimestamp(timestamp, tz=timezone.utc)". Changed in version 3.3: Raise "OverflowError" instead of "ValueError" if the timestamp is out of the range of values supported by the platform C "gmtime()" function. Raise "OSError" instead of "ValueError" on "gmtime()" failure. Deprecated since version 3.12: Use "datetime.fromtimestamp()" with "UTC" instead. classmethod datetime.fromordinal(ordinal) Return the "datetime" corresponding to the proleptic Gregorian ordinal, where January 1 of year 1 has ordinal 1. "ValueError" is raised unless "1 <= ordinal <= datetime.max.toordinal()". The hour, minute, second and microsecond of the result are all 0, and "tzinfo" is "None". classmethod datetime.combine(date, time, tzinfo=time.tzinfo) Return a new "datetime" object whose date components are equal to the given "date" object’s, and whose time components are equal to the given "time" object’s. If the *tzinfo* argument is provided, its value is used to set the "tzinfo" attribute of the result, otherwise the "tzinfo" attribute of the *time* argument is used. If the *date* argument is a "datetime" object, its time components and "tzinfo" attributes are ignored. For any "datetime" object "d", "d == datetime.combine(d.date(), d.time(), d.tzinfo)". Changed in version 3.6: Added the *tzinfo* argument. classmethod datetime.fromisoformat(date_string) Return a "datetime" corresponding to a *date_string* in any valid ISO 8601 format, with the following exceptions: 1. Time zone offsets may have fractional seconds. 2. The "T" separator may be replaced by any single unicode character. 3. Fractional hours and minutes are not supported. 4. Reduced precision dates are not currently supported ("YYYY-MM", "YYYY"). 5. Extended date representations are not currently supported ("±YYYYYY-MM-DD"). 6. Ordinal dates are not currently supported ("YYYY-OOO"). Examples: >>> from datetime import datetime >>> datetime.fromisoformat('2011-11-04') datetime.datetime(2011, 11, 4, 0, 0) >>> datetime.fromisoformat('20111104') datetime.datetime(2011, 11, 4, 0, 0) >>> datetime.fromisoformat('2011-11-04T00:05:23') datetime.datetime(2011, 11, 4, 0, 5, 23) >>> datetime.fromisoformat('2011-11-04T00:05:23Z') datetime.datetime(2011, 11, 4, 0, 5, 23, tzinfo=datetime.timezone.utc) >>> datetime.fromisoformat('20111104T000523') datetime.datetime(2011, 11, 4, 0, 5, 23) >>> datetime.fromisoformat('2011-W01-2T00:05:23.283') datetime.datetime(2011, 1, 4, 0, 5, 23, 283000) >>> datetime.fromisoformat('2011-11-04 00:05:23.283') datetime.datetime(2011, 11, 4, 0, 5, 23, 283000) >>> datetime.fromisoformat('2011-11-04 00:05:23.283+00:00') datetime.datetime(2011, 11, 4, 0, 5, 23, 283000, tzinfo=datetime.timezone.utc) >>> datetime.fromisoformat('2011-11-04T00:05:23+04:00') datetime.datetime(2011, 11, 4, 0, 5, 23, tzinfo=datetime.timezone(datetime.timedelta(seconds=14400))) Added in version 3.7. Changed in version 3.11: Previously, this method only supported formats that could be emitted by "date.isoformat()" or "datetime.isoformat()". classmethod datetime.fromisocalendar(year, week, day) Return a "datetime" corresponding to the ISO calendar date specified by year, week and day. The non-date components of the datetime are populated with their normal default values. This is the inverse of the function "datetime.isocalendar()". Added in version 3.8. classmethod datetime.strptime(date_string, format) Return a "datetime" corresponding to *date_string*, parsed according to *format*. If *format* does not contain microseconds or time zone information, this is equivalent to: datetime(*(time.strptime(date_string, format)[0:6])) "ValueError" is raised if the date_string and format can’t be parsed by "time.strptime()" or if it returns a value which isn’t a time tuple. See also strftime() and strptime() Behavior and "datetime.fromisoformat()". Changed in version 3.13: If *format* specifies a day of month without a year a "DeprecationWarning" is now emitted. This is to avoid a quadrennial leap year bug in code seeking to parse only a month and day as the default year used in absence of one in the format is not a leap year. Such *format* values may raise an error as of Python 3.15. The workaround is to always include a year in your *format*. If parsing *date_string* values that do not have a year, explicitly add a year that is a leap year before parsing: >>> from datetime import datetime >>> date_string = "02/29" >>> when = datetime.strptime(f"{date_string};1984", "%m/%d;%Y") # Avoids leap year bug. >>> when.strftime("%B %d") 'February 29' Class attributes: datetime.min The earliest representable "datetime", "datetime(MINYEAR, 1, 1, tzinfo=None)". datetime.max The latest representable "datetime", "datetime(MAXYEAR, 12, 31, 23, 59, 59, 999999, tzinfo=None)". datetime.resolution The smallest possible difference between non-equal "datetime" objects, "timedelta(microseconds=1)". Instance attributes (read-only): datetime.year Between "MINYEAR" and "MAXYEAR" inclusive. datetime.month Between 1 and 12 inclusive. datetime.day Between 1 and the number of days in the given month of the given year. datetime.hour In "range(24)". datetime.minute In "range(60)". datetime.second In "range(60)". datetime.microsecond In "range(1000000)". datetime.tzinfo The object passed as the *tzinfo* argument to the "datetime" constructor, or "None" if none was passed. datetime.fold In "[0, 1]". Used to disambiguate wall times during a repeated interval. (A repeated interval occurs when clocks are rolled back at the end of daylight saving time or when the UTC offset for the current zone is decreased for political reasons.) The values 0 and 1 represent, respectively, the earlier and later of the two moments with the same wall time representation. Added in version 3.6. Supported operations: +-----------------------------------------+----------------------------------+ | Operation | Result | |=========================================|==================================| | "datetime2 = datetime1 + timedelta" | (1) | +-----------------------------------------+----------------------------------+ | "datetime2 = datetime1 - timedelta" | (2) | +-----------------------------------------+----------------------------------+ | "timedelta = datetime1 - datetime2" | (3) | +-----------------------------------------+----------------------------------+ | "datetime1 == datetime2" "datetime1 != | Equality comparison. (4) | | datetime2" | | +-----------------------------------------+----------------------------------+ | "datetime1 < datetime2" "datetime1 > | Order comparison. (5) | | datetime2" "datetime1 <= datetime2" | | | "datetime1 >= datetime2" | | +-----------------------------------------+----------------------------------+ 1. "datetime2" is a duration of "timedelta" removed from "datetime1", moving forward in time if "timedelta.days > 0", or backward if "timedelta.days < 0". The result has the same "tzinfo" attribute as the input datetime, and "datetime2 - datetime1 == timedelta" after. "OverflowError" is raised if "datetime2.year" would be smaller than "MINYEAR" or larger than "MAXYEAR". Note that no time zone adjustments are done even if the input is an aware object. 2. Computes the "datetime2" such that "datetime2 + timedelta == datetime1". As for addition, the result has the same "tzinfo" attribute as the input datetime, and no time zone adjustments are done even if the input is aware. 3. Subtraction of a "datetime" from a "datetime" is defined only if both operands are naive, or if both are aware. If one is aware and the other is naive, "TypeError" is raised. If both are naive, or both are aware and have the same "tzinfo" attribute, the "tzinfo" attributes are ignored, and the result is a "timedelta" object "t" such that "datetime2 + t == datetime1". No time zone adjustments are done in this case. If both are aware and have different "tzinfo" attributes, "a-b" acts as if "a" and "b" were first converted to naive UTC datetimes. The result is "(a.replace(tzinfo=None) - a.utcoffset()) - (b.replace(tzinfo=None) - b.utcoffset())" except that the implementation never overflows. 4. "datetime" objects are equal if they represent the same date and time, taking into account the time zone. Naive and aware "datetime" objects are never equal. If both comparands are aware, and have the same "tzinfo" attribute, the "tzinfo" and "fold" attributes are ignored and the base datetimes are compared. If both comparands are aware and have different "tzinfo" attributes, the comparison acts as comparands were first converted to UTC datetimes except that the implementation never overflows. "datetime" instances in a repeated interval are never equal to "datetime" instances in other time zone. 5. *datetime1* is considered less than *datetime2* when *datetime1* precedes *datetime2* in time, taking into account the time zone. Order comparison between naive and aware "datetime" objects raises "TypeError". If both comparands are aware, and have the same "tzinfo" attribute, the "tzinfo" and "fold" attributes are ignored and the base datetimes are compared. If both comparands are aware and have different "tzinfo" attributes, the comparison acts as comparands were first converted to UTC datetimes except that the implementation never overflows. Changed in version 3.3: Equality comparisons between aware and naive "datetime" instances don’t raise "TypeError". Changed in version 3.13: Comparison between "datetime" object and an instance of the "date" subclass that is not a "datetime" subclass no longer converts the latter to "date", ignoring the time part and the time zone. The default behavior can be changed by overriding the special comparison methods in subclasses. Instance methods: datetime.date() Return "date" object with same year, month and day. datetime.time() Return "time" object with same hour, minute, second, microsecond and fold. "tzinfo" is "None". See also method "timetz()". Changed in version 3.6: The fold value is copied to the returned "time" object. datetime.timetz() Return "time" object with same hour, minute, second, microsecond, fold, and tzinfo attributes. See also method "time()". Changed in version 3.6: The fold value is copied to the returned "time" object. datetime.replace(year=self.year, month=self.month, day=self.day, hour=self.hour, minute=self.minute, second=self.second, microsecond=self.microsecond, tzinfo=self.tzinfo, *, fold=0) Return a new "datetime" object with the same attributes, but with specified parameters updated. Note that "tzinfo=None" can be specified to create a naive datetime from an aware datetime with no conversion of date and time data. "datetime" objects are also supported by generic function "copy.replace()". Changed in version 3.6: Added the *fold* parameter. datetime.astimezone(tz=None) Return a "datetime" object with new "tzinfo" attribute *tz*, adjusting the date and time data so the result is the same UTC time as *self*, but in *tz*’s local time. If provided, *tz* must be an instance of a "tzinfo" subclass, and its "utcoffset()" and "dst()" methods must not return "None". If *self* is naive, it is presumed to represent time in the system time zone. If called without arguments (or with "tz=None") the system local time zone is assumed for the target time zone. The ".tzinfo" attribute of the converted datetime instance will be set to an instance of "timezone" with the zone name and offset obtained from the OS. If "self.tzinfo" is *tz*, "self.astimezone(tz)" is equal to *self*: no adjustment of date or time data is performed. Else the result is local time in the time zone *tz*, representing the same UTC time as *self*: after "astz = dt.astimezone(tz)", "astz - astz.utcoffset()" will have the same date and time data as "dt - dt.utcoffset()". If you merely want to attach a "timezone" object *tz* to a datetime *dt* without adjustment of date and time data, use "dt.replace(tzinfo=tz)". If you merely want to remove the "timezone" object from an aware datetime *dt* without conversion of date and time data, use "dt.replace(tzinfo=None)". Note that the default "tzinfo.fromutc()" method can be overridden in a "tzinfo" subclass to affect the result returned by "astimezone()". Ignoring error cases, "astimezone()" acts like: def astimezone(self, tz): if self.tzinfo is tz: return self # Convert self to UTC, and attach the new timezone object. utc = (self - self.utcoffset()).replace(tzinfo=tz) # Convert from UTC to tz's local time. return tz.fromutc(utc) Changed in version 3.3: *tz* now can be omitted. Changed in version 3.6: The "astimezone()" method can now be called on naive instances that are presumed to represent system local time. datetime.utcoffset() If "tzinfo" is "None", returns "None", else returns "self.tzinfo.utcoffset(self)", and raises an exception if the latter doesn’t return "None" or a "timedelta" object with magnitude less than one day. Changed in version 3.7: The UTC offset is not restricted to a whole number of minutes. datetime.dst() If "tzinfo" is "None", returns "None", else returns "self.tzinfo.dst(self)", and raises an exception if the latter doesn’t return "None" or a "timedelta" object with magnitude less than one day. Changed in version 3.7: The DST offset is not restricted to a whole number of minutes. datetime.tzname() If "tzinfo" is "None", returns "None", else returns "self.tzinfo.tzname(self)", raises an exception if the latter doesn’t return "None" or a string object, datetime.timetuple() Return a "time.struct_time" such as returned by "time.localtime()". "d.timetuple()" is equivalent to: time.struct_time((d.year, d.month, d.day, d.hour, d.minute, d.second, d.weekday(), yday, dst)) where "yday = d.toordinal() - date(d.year, 1, 1).toordinal() + 1" is the day number within the current year starting with 1 for January 1st. The "tm_isdst" flag of the result is set according to the "dst()" method: "tzinfo" is "None" or "dst()" returns "None", "tm_isdst" is set to "-1"; else if "dst()" returns a non-zero value, "tm_isdst" is set to 1; else "tm_isdst" is set to 0. datetime.utctimetuple() If "datetime" instance "d" is naive, this is the same as "d.timetuple()" except that "tm_isdst" is forced to 0 regardless of what "d.dst()" returns. DST is never in effect for a UTC time. If "d" is aware, "d" is normalized to UTC time, by subtracting "d.utcoffset()", and a "time.struct_time" for the normalized time is returned. "tm_isdst" is forced to 0. Note that an "OverflowError" may be raised if "d.year" was "MINYEAR" or "MAXYEAR" and UTC adjustment spills over a year boundary. Warning: Because naive "datetime" objects are treated by many "datetime" methods as local times, it is preferred to use aware datetimes to represent times in UTC; as a result, using "datetime.utctimetuple()" may give misleading results. If you have a naive "datetime" representing UTC, use "datetime.replace(tzinfo=timezone.utc)" to make it aware, at which point you can use "datetime.timetuple()". datetime.toordinal() Return the proleptic Gregorian ordinal of the date. The same as "self.date().toordinal()". datetime.timestamp() Return POSIX timestamp corresponding to the "datetime" instance. The return value is a "float" similar to that returned by "time.time()". Naive "datetime" instances are assumed to represent local time and this method relies on the platform C "mktime()" function to perform the conversion. Since "datetime" supports wider range of values than "mktime()" on many platforms, this method may raise "OverflowError" or "OSError" for times far in the past or far in the future. For aware "datetime" instances, the return value is computed as: (dt - datetime(1970, 1, 1, tzinfo=timezone.utc)).total_seconds() Added in version 3.3. Changed in version 3.6: The "timestamp()" method uses the "fold" attribute to disambiguate the times during a repeated interval. Note: There is no method to obtain the POSIX timestamp directly from a naive "datetime" instance representing UTC time. If your application uses this convention and your system time zone is not set to UTC, you can obtain the POSIX timestamp by supplying "tzinfo=timezone.utc": timestamp = dt.replace(tzinfo=timezone.utc).timestamp() or by calculating the timestamp directly: timestamp = (dt - datetime(1970, 1, 1)) / timedelta(seconds=1) datetime.weekday() Return the day of the week as an integer, where Monday is 0 and Sunday is 6. The same as "self.date().weekday()". See also "isoweekday()". datetime.isoweekday() Return the day of the week as an integer, where Monday is 1 and Sunday is 7. The same as "self.date().isoweekday()". See also "weekday()", "isocalendar()". datetime.isocalendar() Return a *named tuple* with three components: "year", "week" and "weekday". The same as "self.date().isocalendar()". datetime.isoformat(sep='T', timespec='auto') Return a string representing the date and time in ISO 8601 format: * "YYYY-MM-DDTHH:MM:SS.ffffff", if "microsecond" is not 0 * "YYYY-MM-DDTHH:MM:SS", if "microsecond" is 0 If "utcoffset()" does not return "None", a string is appended, giving the UTC offset: * "YYYY-MM-DDTHH:MM:SS.ffffff+HH:MM[:SS[.ffffff]]", if "microsecond" is not 0 * "YYYY-MM-DDTHH:MM:SS+HH:MM[:SS[.ffffff]]", if "microsecond" is 0 Examples: >>> from datetime import datetime, timezone >>> datetime(2019, 5, 18, 15, 17, 8, 132263).isoformat() '2019-05-18T15:17:08.132263' >>> datetime(2019, 5, 18, 15, 17, tzinfo=timezone.utc).isoformat() '2019-05-18T15:17:00+00:00' The optional argument *sep* (default "'T'") is a one-character separator, placed between the date and time portions of the result. For example: >>> from datetime import tzinfo, timedelta, datetime >>> class TZ(tzinfo): ... """A time zone with an arbitrary, constant -06:39 offset.""" ... def utcoffset(self, dt): ... return timedelta(hours=-6, minutes=-39) ... >>> datetime(2002, 12, 25, tzinfo=TZ()).isoformat(' ') '2002-12-25 00:00:00-06:39' >>> datetime(2009, 11, 27, microsecond=100, tzinfo=TZ()).isoformat() '2009-11-27T00:00:00.000100-06:39' The optional argument *timespec* specifies the number of additional components of the time to include (the default is "'auto'"). It can be one of the following: * "'auto'": Same as "'seconds'" if "microsecond" is 0, same as "'microseconds'" otherwise. * "'hours'": Include the "hour" in the two-digit "HH" format. * "'minutes'": Include "hour" and "minute" in "HH:MM" format. * "'seconds'": Include "hour", "minute", and "second" in "HH:MM:SS" format. * "'milliseconds'": Include full time, but truncate fractional second part to milliseconds. "HH:MM:SS.sss" format. * "'microseconds'": Include full time in "HH:MM:SS.ffffff" format. Note: Excluded time components are truncated, not rounded. "ValueError" will be raised on an invalid *timespec* argument: >>> from datetime import datetime >>> datetime.now().isoformat(timespec='minutes') '2002-12-25T00:00' >>> dt = datetime(2015, 1, 1, 12, 30, 59, 0) >>> dt.isoformat(timespec='microseconds') '2015-01-01T12:30:59.000000' Changed in version 3.6: Added the *timespec* parameter. datetime.__str__() For a "datetime" instance "d", "str(d)" is equivalent to "d.isoformat(' ')". datetime.ctime() Return a string representing the date and time: >>> from datetime import datetime >>> datetime(2002, 12, 4, 20, 30, 40).ctime() 'Wed Dec 4 20:30:40 2002' The output string will *not* include time zone information, regardless of whether the input is aware or naive. "d.ctime()" is equivalent to: time.ctime(time.mktime(d.timetuple())) on platforms where the native C "ctime()" function (which "time.ctime()" invokes, but which "datetime.ctime()" does not invoke) conforms to the C standard. datetime.strftime(format) Return a string representing the date and time, controlled by an explicit format string. See also strftime() and strptime() Behavior and "datetime.isoformat()". datetime.__format__(format) Same as "datetime.strftime()". This makes it possible to specify a format string for a "datetime" object in formatted string literals and when using "str.format()". See also strftime() and strptime() Behavior and "datetime.isoformat()". Examples of Usage: "datetime" ----------------------------- Examples of working with "datetime" objects: >>> from datetime import datetime, date, time, timezone >>> # Using datetime.combine() >>> d = date(2005, 7, 14) >>> t = time(12, 30) >>> datetime.combine(d, t) datetime.datetime(2005, 7, 14, 12, 30) >>> # Using datetime.now() >>> datetime.now() datetime.datetime(2007, 12, 6, 16, 29, 43, 79043) # GMT +1 >>> datetime.now(timezone.utc) datetime.datetime(2007, 12, 6, 15, 29, 43, 79060, tzinfo=datetime.timezone.utc) >>> # Using datetime.strptime() >>> dt = datetime.strptime("21/11/06 16:30", "%d/%m/%y %H:%M") >>> dt datetime.datetime(2006, 11, 21, 16, 30) >>> # Using datetime.timetuple() to get tuple of all attributes >>> tt = dt.timetuple() >>> for it in tt: ... print(it) ... 2006 # year 11 # month 21 # day 16 # hour 30 # minute 0 # second 1 # weekday (0 = Monday) 325 # number of days since 1st January -1 # dst - method tzinfo.dst() returned None >>> # Date in ISO format >>> ic = dt.isocalendar() >>> for it in ic: ... print(it) ... 2006 # ISO year 47 # ISO week 2 # ISO weekday >>> # Formatting a datetime >>> dt.strftime("%A, %d. %B %Y %I:%M%p") 'Tuesday, 21. November 2006 04:30PM' >>> 'The {1} is {0:%d}, the {2} is {0:%B}, the {3} is {0:%I:%M%p}.'.format(dt, "day", "month", "time") 'The day is 21, the month is November, the time is 04:30PM.' The example below defines a "tzinfo" subclass capturing time zone information for Kabul, Afghanistan, which used +4 UTC until 1945 and then +4:30 UTC thereafter: from datetime import timedelta, datetime, tzinfo, timezone class KabulTz(tzinfo): # Kabul used +4 until 1945, when they moved to +4:30 UTC_MOVE_DATE = datetime(1944, 12, 31, 20, tzinfo=timezone.utc) def utcoffset(self, dt): if dt.year < 1945: return timedelta(hours=4) elif (1945, 1, 1, 0, 0) <= dt.timetuple()[:5] < (1945, 1, 1, 0, 30): # An ambiguous ("imaginary") half-hour range representing # a 'fold' in time due to the shift from +4 to +4:30. # If dt falls in the imaginary range, use fold to decide how # to resolve. See PEP495. return timedelta(hours=4, minutes=(30 if dt.fold else 0)) else: return timedelta(hours=4, minutes=30) def fromutc(self, dt): # Follow same validations as in datetime.tzinfo if not isinstance(dt, datetime): raise TypeError("fromutc() requires a datetime argument") if dt.tzinfo is not self: raise ValueError("dt.tzinfo is not self") # A custom implementation is required for fromutc as # the input to this function is a datetime with utc values # but with a tzinfo set to self. # See datetime.astimezone or fromtimestamp. if dt.replace(tzinfo=timezone.utc) >= self.UTC_MOVE_DATE: return dt + timedelta(hours=4, minutes=30) else: return dt + timedelta(hours=4) def dst(self, dt): # Kabul does not observe daylight saving time. return timedelta(0) def tzname(self, dt): if dt >= self.UTC_MOVE_DATE: return "+04:30" return "+04" Usage of "KabulTz" from above: >>> tz1 = KabulTz() >>> # Datetime before the change >>> dt1 = datetime(1900, 11, 21, 16, 30, tzinfo=tz1) >>> print(dt1.utcoffset()) 4:00:00 >>> # Datetime after the change >>> dt2 = datetime(2006, 6, 14, 13, 0, tzinfo=tz1) >>> print(dt2.utcoffset()) 4:30:00 >>> # Convert datetime to another time zone >>> dt3 = dt2.astimezone(timezone.utc) >>> dt3 datetime.datetime(2006, 6, 14, 8, 30, tzinfo=datetime.timezone.utc) >>> dt2 datetime.datetime(2006, 6, 14, 13, 0, tzinfo=KabulTz()) >>> dt2 == dt3 True "time" Objects ============== A "time" object represents a (local) time of day, independent of any particular day, and subject to adjustment via a "tzinfo" object. class datetime.time(hour=0, minute=0, second=0, microsecond=0, tzinfo=None, *, fold=0) All arguments are optional. *tzinfo* may be "None", or an instance of a "tzinfo" subclass. The remaining arguments must be integers in the following ranges: * "0 <= hour < 24", * "0 <= minute < 60", * "0 <= second < 60", * "0 <= microsecond < 1000000", * "fold in [0, 1]". If an argument outside those ranges is given, "ValueError" is raised. All default to 0 except *tzinfo*, which defaults to "None". Class attributes: time.min The earliest representable "time", "time(0, 0, 0, 0)". time.max The latest representable "time", "time(23, 59, 59, 999999)". time.resolution The smallest possible difference between non-equal "time" objects, "timedelta(microseconds=1)", although note that arithmetic on "time" objects is not supported. Instance attributes (read-only): time.hour In "range(24)". time.minute In "range(60)". time.second In "range(60)". time.microsecond In "range(1000000)". time.tzinfo The object passed as the tzinfo argument to the "time" constructor, or "None" if none was passed. time.fold In "[0, 1]". Used to disambiguate wall times during a repeated interval. (A repeated interval occurs when clocks are rolled back at the end of daylight saving time or when the UTC offset for the current zone is decreased for political reasons.) The values 0 and 1 represent, respectively, the earlier and later of the two moments with the same wall time representation. Added in version 3.6. "time" objects support equality and order comparisons, where "a" is considered less than "b" when "a" precedes "b" in time. Naive and aware "time" objects are never equal. Order comparison between naive and aware "time" objects raises "TypeError". If both comparands are aware, and have the same "tzinfo" attribute, the "tzinfo" and "fold" attributes are ignored and the base times are compared. If both comparands are aware and have different "tzinfo" attributes, the comparands are first adjusted by subtracting their UTC offsets (obtained from "self.utcoffset()"). Changed in version 3.3: Equality comparisons between aware and naive "time" instances don’t raise "TypeError". In Boolean contexts, a "time" object is always considered to be true. Changed in version 3.5: Before Python 3.5, a "time" object was considered to be false if it represented midnight in UTC. This behavior was considered obscure and error-prone and has been removed in Python 3.5. See bpo-13936 for full details. Other constructor: classmethod time.fromisoformat(time_string) Return a "time" corresponding to a *time_string* in any valid ISO 8601 format, with the following exceptions: 1. Time zone offsets may have fractional seconds. 2. The leading "T", normally required in cases where there may be ambiguity between a date and a time, is not required. 3. Fractional seconds may have any number of digits (anything beyond 6 will be truncated). 4. Fractional hours and minutes are not supported. Examples: >>> from datetime import time >>> time.fromisoformat('04:23:01') datetime.time(4, 23, 1) >>> time.fromisoformat('T04:23:01') datetime.time(4, 23, 1) >>> time.fromisoformat('T042301') datetime.time(4, 23, 1) >>> time.fromisoformat('04:23:01.000384') datetime.time(4, 23, 1, 384) >>> time.fromisoformat('04:23:01,000384') datetime.time(4, 23, 1, 384) >>> time.fromisoformat('04:23:01+04:00') datetime.time(4, 23, 1, tzinfo=datetime.timezone(datetime.timedelta(seconds=14400))) >>> time.fromisoformat('04:23:01Z') datetime.time(4, 23, 1, tzinfo=datetime.timezone.utc) >>> time.fromisoformat('04:23:01+00:00') datetime.time(4, 23, 1, tzinfo=datetime.timezone.utc) Added in version 3.7. Changed in version 3.11: Previously, this method only supported formats that could be emitted by "time.isoformat()". Instance methods: time.replace(hour=self.hour, minute=self.minute, second=self.second, microsecond=self.microsecond, tzinfo=self.tzinfo, *, fold=0) Return a new "time" with the same values, but with specified parameters updated. Note that "tzinfo=None" can be specified to create a naive "time" from an aware "time", without conversion of the time data. "time" objects are also supported by generic function "copy.replace()". Changed in version 3.6: Added the *fold* parameter. time.isoformat(timespec='auto') Return a string representing the time in ISO 8601 format, one of: * "HH:MM:SS.ffffff", if "microsecond" is not 0 * "HH:MM:SS", if "microsecond" is 0 * "HH:MM:SS.ffffff+HH:MM[:SS[.ffffff]]", if "utcoffset()" does not return "None" * "HH:MM:SS+HH:MM[:SS[.ffffff]]", if "microsecond" is 0 and "utcoffset()" does not return "None" The optional argument *timespec* specifies the number of additional components of the time to include (the default is "'auto'"). It can be one of the following: * "'auto'": Same as "'seconds'" if "microsecond" is 0, same as "'microseconds'" otherwise. * "'hours'": Include the "hour" in the two-digit "HH" format. * "'minutes'": Include "hour" and "minute" in "HH:MM" format. * "'seconds'": Include "hour", "minute", and "second" in "HH:MM:SS" format. * "'milliseconds'": Include full time, but truncate fractional second part to milliseconds. "HH:MM:SS.sss" format. * "'microseconds'": Include full time in "HH:MM:SS.ffffff" format. Note: Excluded time components are truncated, not rounded. "ValueError" will be raised on an invalid *timespec* argument. Example: >>> from datetime import time >>> time(hour=12, minute=34, second=56, microsecond=123456).isoformat(timespec='minutes') '12:34' >>> dt = time(hour=12, minute=34, second=56, microsecond=0) >>> dt.isoformat(timespec='microseconds') '12:34:56.000000' >>> dt.isoformat(timespec='auto') '12:34:56' Changed in version 3.6: Added the *timespec* parameter. time.__str__() For a time "t", "str(t)" is equivalent to "t.isoformat()". time.strftime(format) Return a string representing the time, controlled by an explicit format string. See also strftime() and strptime() Behavior and "time.isoformat()". time.__format__(format) Same as "time.strftime()". This makes it possible to specify a format string for a "time" object in formatted string literals and when using "str.format()". See also strftime() and strptime() Behavior and "time.isoformat()". time.utcoffset() If "tzinfo" is "None", returns "None", else returns "self.tzinfo.utcoffset(None)", and raises an exception if the latter doesn’t return "None" or a "timedelta" object with magnitude less than one day. Changed in version 3.7: The UTC offset is not restricted to a whole number of minutes. time.dst() If "tzinfo" is "None", returns "None", else returns "self.tzinfo.dst(None)", and raises an exception if the latter doesn’t return "None", or a "timedelta" object with magnitude less than one day. Changed in version 3.7: The DST offset is not restricted to a whole number of minutes. time.tzname() If "tzinfo" is "None", returns "None", else returns "self.tzinfo.tzname(None)", or raises an exception if the latter doesn’t return "None" or a string object. Examples of Usage: "time" ------------------------- Examples of working with a "time" object: >>> from datetime import time, tzinfo, timedelta >>> class TZ1(tzinfo): ... def utcoffset(self, dt): ... return timedelta(hours=1) ... def dst(self, dt): ... return timedelta(0) ... def tzname(self,dt): ... return "+01:00" ... def __repr__(self): ... return f"{self.__class__.__name__}()" ... >>> t = time(12, 10, 30, tzinfo=TZ1()) >>> t datetime.time(12, 10, 30, tzinfo=TZ1()) >>> t.isoformat() '12:10:30+01:00' >>> t.dst() datetime.timedelta(0) >>> t.tzname() '+01:00' >>> t.strftime("%H:%M:%S %Z") '12:10:30 +01:00' >>> 'The {} is {:%H:%M}.'.format("time", t) 'The time is 12:10.' "tzinfo" Objects ================ class datetime.tzinfo This is an abstract base class, meaning that this class should not be instantiated directly. Define a subclass of "tzinfo" to capture information about a particular time zone. An instance of (a concrete subclass of) "tzinfo" can be passed to the constructors for "datetime" and "time" objects. The latter objects view their attributes as being in local time, and the "tzinfo" object supports methods revealing offset of local time from UTC, the name of the time zone, and DST offset, all relative to a date or time object passed to them. You need to derive a concrete subclass, and (at least) supply implementations of the standard "tzinfo" methods needed by the "datetime" methods you use. The "datetime" module provides "timezone", a simple concrete subclass of "tzinfo" which can represent time zones with fixed offset from UTC such as UTC itself or North American EST and EDT. Special requirement for pickling: A "tzinfo" subclass must have an "__init__()" method that can be called with no arguments, otherwise it can be pickled but possibly not unpickled again. This is a technical requirement that may be relaxed in the future. A concrete subclass of "tzinfo" may need to implement the following methods. Exactly which methods are needed depends on the uses made of aware "datetime" objects. If in doubt, simply implement all of them. tzinfo.utcoffset(dt) Return offset of local time from UTC, as a "timedelta" object that is positive east of UTC. If local time is west of UTC, this should be negative. This represents the *total* offset from UTC; for example, if a "tzinfo" object represents both time zone and DST adjustments, "utcoffset()" should return their sum. If the UTC offset isn’t known, return "None". Else the value returned must be a "timedelta" object strictly between "-timedelta(hours=24)" and "timedelta(hours=24)" (the magnitude of the offset must be less than one day). Most implementations of "utcoffset()" will probably look like one of these two: return CONSTANT # fixed-offset class return CONSTANT + self.dst(dt) # daylight-aware class If "utcoffset()" does not return "None", "dst()" should not return "None" either. The default implementation of "utcoffset()" raises "NotImplementedError". Changed in version 3.7: The UTC offset is not restricted to a whole number of minutes. tzinfo.dst(dt) Return the daylight saving time (DST) adjustment, as a "timedelta" object or "None" if DST information isn’t known. Return "timedelta(0)" if DST is not in effect. If DST is in effect, return the offset as a "timedelta" object (see "utcoffset()" for details). Note that DST offset, if applicable, has already been added to the UTC offset returned by "utcoffset()", so there’s no need to consult "dst()" unless you’re interested in obtaining DST info separately. For example, "datetime.timetuple()" calls its "tzinfo" attribute’s "dst()" method to determine how the "tm_isdst" flag should be set, and "tzinfo.fromutc()" calls "dst()" to account for DST changes when crossing time zones. An instance *tz* of a "tzinfo" subclass that models both standard and daylight times must be consistent in this sense: "tz.utcoffset(dt) - tz.dst(dt)" must return the same result for every "datetime" *dt* with "dt.tzinfo == tz". For sane "tzinfo" subclasses, this expression yields the time zone’s “standard offset”, which should not depend on the date or the time, but only on geographic location. The implementation of "datetime.astimezone()" relies on this, but cannot detect violations; it’s the programmer’s responsibility to ensure it. If a "tzinfo" subclass cannot guarantee this, it may be able to override the default implementation of "tzinfo.fromutc()" to work correctly with "astimezone()" regardless. Most implementations of "dst()" will probably look like one of these two: def dst(self, dt): # a fixed-offset class: doesn't account for DST return timedelta(0) or: def dst(self, dt): # Code to set dston and dstoff to the time zone's DST # transition times based on the input dt.year, and expressed # in standard local time. if dston <= dt.replace(tzinfo=None) < dstoff: return timedelta(hours=1) else: return timedelta(0) The default implementation of "dst()" raises "NotImplementedError". Changed in version 3.7: The DST offset is not restricted to a whole number of minutes. tzinfo.tzname(dt) Return the time zone name corresponding to the "datetime" object *dt*, as a string. Nothing about string names is defined by the "datetime" module, and there’s no requirement that it mean anything in particular. For example, ""GMT"", ""UTC"", ""-500"", ""-5:00"", ""EDT"", ""US/Eastern"", ""America/New York"" are all valid replies. Return "None" if a string name isn’t known. Note that this is a method rather than a fixed string primarily because some "tzinfo" subclasses will wish to return different names depending on the specific value of *dt* passed, especially if the "tzinfo" class is accounting for daylight time. The default implementation of "tzname()" raises "NotImplementedError". These methods are called by a "datetime" or "time" object, in response to their methods of the same names. A "datetime" object passes itself as the argument, and a "time" object passes "None" as the argument. A "tzinfo" subclass’s methods should therefore be prepared to accept a *dt* argument of "None", or of class "datetime". When "None" is passed, it’s up to the class designer to decide the best response. For example, returning "None" is appropriate if the class wishes to say that time objects don’t participate in the "tzinfo" protocols. It may be more useful for "utcoffset(None)" to return the standard UTC offset, as there is no other convention for discovering the standard offset. When a "datetime" object is passed in response to a "datetime" method, "dt.tzinfo" is the same object as *self*. "tzinfo" methods can rely on this, unless user code calls "tzinfo" methods directly. The intent is that the "tzinfo" methods interpret *dt* as being in local time, and not need worry about objects in other time zones. There is one more "tzinfo" method that a subclass may wish to override: tzinfo.fromutc(dt) This is called from the default "datetime.astimezone()" implementation. When called from that, "dt.tzinfo" is *self*, and *dt*’s date and time data are to be viewed as expressing a UTC time. The purpose of "fromutc()" is to adjust the date and time data, returning an equivalent datetime in *self*’s local time. Most "tzinfo" subclasses should be able to inherit the default "fromutc()" implementation without problems. It’s strong enough to handle fixed-offset time zones, and time zones accounting for both standard and daylight time, and the latter even if the DST transition times differ in different years. An example of a time zone the default "fromutc()" implementation may not handle correctly in all cases is one where the standard offset (from UTC) depends on the specific date and time passed, which can happen for political reasons. The default implementations of "astimezone()" and "fromutc()" may not produce the result you want if the result is one of the hours straddling the moment the standard offset changes. Skipping code for error cases, the default "fromutc()" implementation acts like: def fromutc(self, dt): # raise ValueError error if dt.tzinfo is not self dtoff = dt.utcoffset() dtdst = dt.dst() # raise ValueError if dtoff is None or dtdst is None delta = dtoff - dtdst # this is self's standard offset if delta: dt += delta # convert to standard local time dtdst = dt.dst() # raise ValueError if dtdst is None if dtdst: return dt + dtdst else: return dt In the following "tzinfo_examples.py" file there are some examples of "tzinfo" classes: from datetime import tzinfo, timedelta, datetime ZERO = timedelta(0) HOUR = timedelta(hours=1) SECOND = timedelta(seconds=1) # A class capturing the platform's idea of local time. # (May result in wrong values on historical times in # timezones where UTC offset and/or the DST rules had # changed in the past.) import time as _time STDOFFSET = timedelta(seconds = -_time.timezone) if _time.daylight: DSTOFFSET = timedelta(seconds = -_time.altzone) else: DSTOFFSET = STDOFFSET DSTDIFF = DSTOFFSET - STDOFFSET class LocalTimezone(tzinfo): def fromutc(self, dt): assert dt.tzinfo is self stamp = (dt - datetime(1970, 1, 1, tzinfo=self)) // SECOND args = _time.localtime(stamp)[:6] dst_diff = DSTDIFF // SECOND # Detect fold fold = (args == _time.localtime(stamp - dst_diff)) return datetime(*args, microsecond=dt.microsecond, tzinfo=self, fold=fold) def utcoffset(self, dt): if self._isdst(dt): return DSTOFFSET else: return STDOFFSET def dst(self, dt): if self._isdst(dt): return DSTDIFF else: return ZERO def tzname(self, dt): return _time.tzname[self._isdst(dt)] def _isdst(self, dt): tt = (dt.year, dt.month, dt.day, dt.hour, dt.minute, dt.second, dt.weekday(), 0, 0) stamp = _time.mktime(tt) tt = _time.localtime(stamp) return tt.tm_isdst > 0 Local = LocalTimezone() # A complete implementation of current DST rules for major US time zones. def first_sunday_on_or_after(dt): days_to_go = 6 - dt.weekday() if days_to_go: dt += timedelta(days_to_go) return dt # US DST Rules # # This is a simplified (i.e., wrong for a few cases) set of rules for US # DST start and end times. For a complete and up-to-date set of DST rules # and timezone definitions, visit the Olson Database (or try pytz): # http://www.twinsun.com/tz/tz-link.htm # https://sourceforge.net/projects/pytz/ (might not be up-to-date) # # In the US, since 2007, DST starts at 2am (standard time) on the second # Sunday in March, which is the first Sunday on or after Mar 8. DSTSTART_2007 = datetime(1, 3, 8, 2) # and ends at 2am (DST time) on the first Sunday of Nov. DSTEND_2007 = datetime(1, 11, 1, 2) # From 1987 to 2006, DST used to start at 2am (standard time) on the first # Sunday in April and to end at 2am (DST time) on the last # Sunday of October, which is the first Sunday on or after Oct 25. DSTSTART_1987_2006 = datetime(1, 4, 1, 2) DSTEND_1987_2006 = datetime(1, 10, 25, 2) # From 1967 to 1986, DST used to start at 2am (standard time) on the last # Sunday in April (the one on or after April 24) and to end at 2am (DST time) # on the last Sunday of October, which is the first Sunday # on or after Oct 25. DSTSTART_1967_1986 = datetime(1, 4, 24, 2) DSTEND_1967_1986 = DSTEND_1987_2006 def us_dst_range(year): # Find start and end times for US DST. For years before 1967, return # start = end for no DST. if 2006 < year: dststart, dstend = DSTSTART_2007, DSTEND_2007 elif 1986 < year < 2007: dststart, dstend = DSTSTART_1987_2006, DSTEND_1987_2006 elif 1966 < year < 1987: dststart, dstend = DSTSTART_1967_1986, DSTEND_1967_1986 else: return (datetime(year, 1, 1), ) * 2 start = first_sunday_on_or_after(dststart.replace(year=year)) end = first_sunday_on_or_after(dstend.replace(year=year)) return start, end class USTimeZone(tzinfo): def __init__(self, hours, reprname, stdname, dstname): self.stdoffset = timedelta(hours=hours) self.reprname = reprname self.stdname = stdname self.dstname = dstname def __repr__(self): return self.reprname def tzname(self, dt): if self.dst(dt): return self.dstname else: return self.stdname def utcoffset(self, dt): return self.stdoffset + self.dst(dt) def dst(self, dt): if dt is None or dt.tzinfo is None: # An exception may be sensible here, in one or both cases. # It depends on how you want to treat them. The default # fromutc() implementation (called by the default astimezone() # implementation) passes a datetime with dt.tzinfo is self. return ZERO assert dt.tzinfo is self start, end = us_dst_range(dt.year) # Can't compare naive to aware objects, so strip the timezone from # dt first. dt = dt.replace(tzinfo=None) if start + HOUR <= dt < end - HOUR: # DST is in effect. return HOUR if end - HOUR <= dt < end: # Fold (an ambiguous hour): use dt.fold to disambiguate. return ZERO if dt.fold else HOUR if start <= dt < start + HOUR: # Gap (a non-existent hour): reverse the fold rule. return HOUR if dt.fold else ZERO # DST is off. return ZERO def fromutc(self, dt): assert dt.tzinfo is self start, end = us_dst_range(dt.year) start = start.replace(tzinfo=self) end = end.replace(tzinfo=self) std_time = dt + self.stdoffset dst_time = std_time + HOUR if end <= dst_time < end + HOUR: # Repeated hour return std_time.replace(fold=1) if std_time < start or dst_time >= end: # Standard time return std_time if start <= std_time < end - HOUR: # Daylight saving time return dst_time Eastern = USTimeZone(-5, "Eastern", "EST", "EDT") Central = USTimeZone(-6, "Central", "CST", "CDT") Mountain = USTimeZone(-7, "Mountain", "MST", "MDT") Pacific = USTimeZone(-8, "Pacific", "PST", "PDT") Note that there are unavoidable subtleties twice per year in a "tzinfo" subclass accounting for both standard and daylight time, at the DST transition points. For concreteness, consider US Eastern (UTC -0500), where EDT begins the minute after 1:59 (EST) on the second Sunday in March, and ends the minute after 1:59 (EDT) on the first Sunday in November: UTC 3:MM 4:MM 5:MM 6:MM 7:MM 8:MM EST 22:MM 23:MM 0:MM 1:MM 2:MM 3:MM EDT 23:MM 0:MM 1:MM 2:MM 3:MM 4:MM start 22:MM 23:MM 0:MM 1:MM 3:MM 4:MM end 23:MM 0:MM 1:MM 1:MM 2:MM 3:MM When DST starts (the “start” line), the local wall clock leaps from 1:59 to 3:00. A wall time of the form 2:MM doesn’t really make sense on that day, so "astimezone(Eastern)" won’t deliver a result with "hour == 2" on the day DST begins. For example, at the Spring forward transition of 2016, we get: >>> from datetime import datetime, timezone >>> from tzinfo_examples import HOUR, Eastern >>> u0 = datetime(2016, 3, 13, 5, tzinfo=timezone.utc) >>> for i in range(4): ... u = u0 + i*HOUR ... t = u.astimezone(Eastern) ... print(u.time(), 'UTC =', t.time(), t.tzname()) ... 05:00:00 UTC = 00:00:00 EST 06:00:00 UTC = 01:00:00 EST 07:00:00 UTC = 03:00:00 EDT 08:00:00 UTC = 04:00:00 EDT When DST ends (the “end” line), there’s a potentially worse problem: there’s an hour that can’t be spelled unambiguously in local wall time: the last hour of daylight time. In Eastern, that’s times of the form 5:MM UTC on the day daylight time ends. The local wall clock leaps from 1:59 (daylight time) back to 1:00 (standard time) again. Local times of the form 1:MM are ambiguous. "astimezone()" mimics the local clock’s behavior by mapping two adjacent UTC hours into the same local hour then. In the Eastern example, UTC times of the form 5:MM and 6:MM both map to 1:MM when converted to Eastern, but earlier times have the "fold" attribute set to 0 and the later times have it set to 1. For example, at the Fall back transition of 2016, we get: >>> u0 = datetime(2016, 11, 6, 4, tzinfo=timezone.utc) >>> for i in range(4): ... u = u0 + i*HOUR ... t = u.astimezone(Eastern) ... print(u.time(), 'UTC =', t.time(), t.tzname(), t.fold) ... 04:00:00 UTC = 00:00:00 EDT 0 05:00:00 UTC = 01:00:00 EDT 0 06:00:00 UTC = 01:00:00 EST 1 07:00:00 UTC = 02:00:00 EST 0 Note that the "datetime" instances that differ only by the value of the "fold" attribute are considered equal in comparisons. Applications that can’t bear wall-time ambiguities should explicitly check the value of the "fold" attribute or avoid using hybrid "tzinfo" subclasses; there are no ambiguities when using "timezone", or any other fixed-offset "tzinfo" subclass (such as a class representing only EST (fixed offset -5 hours), or only EDT (fixed offset -4 hours)). See also: "zoneinfo" The "datetime" module has a basic "timezone" class (for handling arbitrary fixed offsets from UTC) and its "timezone.utc" attribute (a UTC "timezone" instance). "zoneinfo" brings the *IANA time zone database* (also known as the Olson database) to Python, and its usage is recommended. IANA time zone database The Time Zone Database (often called tz, tzdata or zoneinfo) contains code and data that represent the history of local time for many representative locations around the globe. It is updated periodically to reflect changes made by political bodies to time zone boundaries, UTC offsets, and daylight-saving rules. "timezone" Objects ================== The "timezone" class is a subclass of "tzinfo", each instance of which represents a time zone defined by a fixed offset from UTC. Objects of this class cannot be used to represent time zone information in the locations where different offsets are used in different days of the year or where historical changes have been made to civil time. class datetime.timezone(offset, name=None) The *offset* argument must be specified as a "timedelta" object representing the difference between the local time and UTC. It must be strictly between "-timedelta(hours=24)" and "timedelta(hours=24)", otherwise "ValueError" is raised. The *name* argument is optional. If specified it must be a string that will be used as the value returned by the "datetime.tzname()" method. Added in version 3.2. Changed in version 3.7: The UTC offset is not restricted to a whole number of minutes. timezone.utcoffset(dt) Return the fixed value specified when the "timezone" instance is constructed. The *dt* argument is ignored. The return value is a "timedelta" instance equal to the difference between the local time and UTC. Changed in version 3.7: The UTC offset is not restricted to a whole number of minutes. timezone.tzname(dt) Return the fixed value specified when the "timezone" instance is constructed. If *name* is not provided in the constructor, the name returned by "tzname(dt)" is generated from the value of the "offset" as follows. If *offset* is "timedelta(0)", the name is “UTC”, otherwise it is a string in the format "UTC±HH:MM", where ± is the sign of "offset", HH and MM are two digits of "offset.hours" and "offset.minutes" respectively. Changed in version 3.6: Name generated from "offset=timedelta(0)" is now plain "'UTC'", not "'UTC+00:00'". timezone.dst(dt) Always returns "None". timezone.fromutc(dt) Return "dt + offset". The *dt* argument must be an aware "datetime" instance, with "tzinfo" set to "self". Class attributes: timezone.utc The UTC time zone, "timezone(timedelta(0))". "strftime()" and "strptime()" Behavior ====================================== "date", "datetime", and "time" objects all support a "strftime(format)" method, to create a string representing the time under the control of an explicit format string. Conversely, the "datetime.strptime()" class method creates a "datetime" object from a string representing a date and time and a corresponding format string. The table below provides a high-level comparison of "strftime()" versus "strptime()": +------------------+----------------------------------------------------------+--------------------------------------------------------------------------------+ | | "strftime" | "strptime" | |==================|==========================================================|================================================================================| | Usage | Convert object to a string according to a given format | Parse a string into a "datetime" object given a corresponding format | +------------------+----------------------------------------------------------+--------------------------------------------------------------------------------+ | Type of method | Instance method | Class method | +------------------+----------------------------------------------------------+--------------------------------------------------------------------------------+ | Method of | "date"; "datetime"; "time" | "datetime" | +------------------+----------------------------------------------------------+--------------------------------------------------------------------------------+ | Signature | "strftime(format)" | "strptime(date_string, format)" | +------------------+----------------------------------------------------------+--------------------------------------------------------------------------------+ "strftime()" and "strptime()" Format Codes ------------------------------------------ These methods accept format codes that can be used to parse and format dates: >>> datetime.strptime('31/01/22 23:59:59.999999', ... '%d/%m/%y %H:%M:%S.%f') datetime.datetime(2022, 1, 31, 23, 59, 59, 999999) >>> _.strftime('%a %d %b %Y, %I:%M%p') 'Mon 31 Jan 2022, 11:59PM' The following is a list of all the format codes that the 1989 C standard requires, and these work on all platforms with a standard C implementation. +-------------+----------------------------------+--------------------------+---------+ | Directive | Meaning | Example | Notes | |=============|==================================|==========================|=========| | "%a" | Weekday as locale’s abbreviated | Sun, Mon, …, Sat | (1) | | | name. | (en_US); So, Mo, …, Sa | | | | | (de_DE) | | +-------------+----------------------------------+--------------------------+---------+ | "%A" | Weekday as locale’s full name. | Sunday, Monday, …, | (1) | | | | Saturday (en_US); | | | | | Sonntag, Montag, …, | | | | | Samstag (de_DE) | | +-------------+----------------------------------+--------------------------+---------+ | "%w" | Weekday as a decimal number, | 0, 1, …, 6 | | | | where 0 is Sunday and 6 is | | | | | Saturday. | | | +-------------+----------------------------------+--------------------------+---------+ | "%d" | Day of the month as a zero- | 01, 02, …, 31 | (9) | | | padded decimal number. | | | +-------------+----------------------------------+--------------------------+---------+ | "%b" | Month as locale’s abbreviated | Jan, Feb, …, Dec | (1) | | | name. | (en_US); Jan, Feb, …, | | | | | Dez (de_DE) | | +-------------+----------------------------------+--------------------------+---------+ | "%B" | Month as locale’s full name. | January, February, …, | (1) | | | | December (en_US); | | | | | Januar, Februar, …, | | | | | Dezember (de_DE) | | +-------------+----------------------------------+--------------------------+---------+ | "%m" | Month as a zero-padded decimal | 01, 02, …, 12 | (9) | | | number. | | | +-------------+----------------------------------+--------------------------+---------+ | "%y" | Year without century as a zero- | 00, 01, …, 99 | (9) | | | padded decimal number. | | | +-------------+----------------------------------+--------------------------+---------+ | "%Y" | Year with century as a decimal | 0001, 0002, …, 2013, | (2) | | | number. | 2014, …, 9998, 9999 | | +-------------+----------------------------------+--------------------------+---------+ | "%H" | Hour (24-hour clock) as a zero- | 00, 01, …, 23 | (9) | | | padded decimal number. | | | +-------------+----------------------------------+--------------------------+---------+ | "%I" | Hour (12-hour clock) as a zero- | 01, 02, …, 12 | (9) | | | padded decimal number. | | | +-------------+----------------------------------+--------------------------+---------+ | "%p" | Locale’s equivalent of either AM | AM, PM (en_US); am, pm | (1), | | | or PM. | (de_DE) | (3) | +-------------+----------------------------------+--------------------------+---------+ | "%M" | Minute as a zero-padded decimal | 00, 01, …, 59 | (9) | | | number. | | | +-------------+----------------------------------+--------------------------+---------+ | "%S" | Second as a zero-padded decimal | 00, 01, …, 59 | (4), | | | number. | | (9) | +-------------+----------------------------------+--------------------------+---------+ | "%f" | Microsecond as a decimal number, | 000000, 000001, …, | (5) | | | zero-padded to 6 digits. | 999999 | | +-------------+----------------------------------+--------------------------+---------+ | "%z" | UTC offset in the form | (empty), +0000, -0400, | (6) | | | "±HHMM[SS[.ffffff]]" (empty | +1030, +063415, | | | | string if the object is naive). | -030712.345216 | | +-------------+----------------------------------+--------------------------+---------+ | "%Z" | Time zone name (empty string if | (empty), UTC, GMT | (6) | | | the object is naive). | | | +-------------+----------------------------------+--------------------------+---------+ | "%j" | Day of the year as a zero-padded | 001, 002, …, 366 | (9) | | | decimal number. | | | +-------------+----------------------------------+--------------------------+---------+ | "%U" | Week number of the year (Sunday | 00, 01, …, 53 | (7), | | | as the first day of the week) as | | (9) | | | a zero-padded decimal number. | | | | | All days in a new year preceding | | | | | the first Sunday are considered | | | | | to be in week 0. | | | +-------------+----------------------------------+--------------------------+---------+ | "%W" | Week number of the year (Monday | 00, 01, …, 53 | (7), | | | as the first day of the week) as | | (9) | | | a zero-padded decimal number. | | | | | All days in a new year preceding | | | | | the first Monday are considered | | | | | to be in week 0. | | | +-------------+----------------------------------+--------------------------+---------+ | "%c" | Locale’s appropriate date and | Tue Aug 16 21:30:00 1988 | (1) | | | time representation. | (en_US); Di 16 Aug | | | | | 21:30:00 1988 (de_DE) | | +-------------+----------------------------------+--------------------------+---------+ | "%x" | Locale’s appropriate date | 08/16/88 (None); | (1) | | | representation. | 08/16/1988 (en_US); | | | | | 16.08.1988 (de_DE) | | +-------------+----------------------------------+--------------------------+---------+ | "%X" | Locale’s appropriate time | 21:30:00 (en_US); | (1) | | | representation. | 21:30:00 (de_DE) | | +-------------+----------------------------------+--------------------------+---------+ | "%%" | A literal "'%'" character. | % | | +-------------+----------------------------------+--------------------------+---------+ Several additional directives not required by the C89 standard are included for convenience. These parameters all correspond to ISO 8601 date values. +-------------+----------------------------------+--------------------------+---------+ | Directive | Meaning | Example | Notes | |=============|==================================|==========================|=========| | "%G" | ISO 8601 year with century | 0001, 0002, …, 2013, | (8) | | | representing the year that | 2014, …, 9998, 9999 | | | | contains the greater part of the | | | | | ISO week ("%V"). | | | +-------------+----------------------------------+--------------------------+---------+ | "%u" | ISO 8601 weekday as a decimal | 1, 2, …, 7 | | | | number where 1 is Monday. | | | +-------------+----------------------------------+--------------------------+---------+ | "%V" | ISO 8601 week as a decimal | 01, 02, …, 53 | (8), | | | number with Monday as the first | | (9) | | | day of the week. Week 01 is the | | | | | week containing Jan 4. | | | +-------------+----------------------------------+--------------------------+---------+ | "%:z" | UTC offset in the form | (empty), +00:00, -04:00, | (6) | | | "±HH:MM[:SS[.ffffff]]" (empty | +10:30, +06:34:15, | | | | string if the object is naive). | -03:07:12.345216 | | +-------------+----------------------------------+--------------------------+---------+ These may not be available on all platforms when used with the "strftime()" method. The ISO 8601 year and ISO 8601 week directives are not interchangeable with the year and week number directives above. Calling "strptime()" with incomplete or ambiguous ISO 8601 directives will raise a "ValueError". The full set of format codes supported varies across platforms, because Python calls the platform C library’s "strftime()" function, and platform variations are common. To see the full set of format codes supported on your platform, consult the *strftime(3)* documentation. There are also differences between platforms in handling of unsupported format specifiers. Added in version 3.6: "%G", "%u" and "%V" were added. Added in version 3.12: "%:z" was added. Technical Detail ---------------- Broadly speaking, "d.strftime(fmt)" acts like the "time" module’s "time.strftime(fmt, d.timetuple())" although not all objects support a "timetuple()" method. For the "datetime.strptime()" class method, the default value is "1900-01-01T00:00:00.000": any components not specified in the format string will be pulled from the default value. [4] Using "datetime.strptime(date_string, format)" is equivalent to: datetime(*(time.strptime(date_string, format)[0:6])) except when the format includes sub-second components or time zone offset information, which are supported in "datetime.strptime" but are discarded by "time.strptime". For "time" objects, the format codes for year, month, and day should not be used, as "time" objects have no such values. If they’re used anyway, 1900 is substituted for the year, and 1 for the month and day. For "date" objects, the format codes for hours, minutes, seconds, and microseconds should not be used, as "date" objects have no such values. If they’re used anyway, 0 is substituted for them. For the same reason, handling of format strings containing Unicode code points that can’t be represented in the charset of the current locale is also platform-dependent. On some platforms such code points are preserved intact in the output, while on others "strftime" may raise "UnicodeError" or return an empty string instead. Notes: 1. Because the format depends on the current locale, care should be taken when making assumptions about the output value. Field orderings will vary (for example, “month/day/year” versus “day/month/year”), and the output may contain non-ASCII characters. 2. The "strptime()" method can parse years in the full [1, 9999] range, but years < 1000 must be zero-filled to 4-digit width. Changed in version 3.2: In previous versions, "strftime()" method was restricted to years >= 1900. Changed in version 3.3: In version 3.2, "strftime()" method was restricted to years >= 1000. 3. When used with the "strptime()" method, the "%p" directive only affects the output hour field if the "%I" directive is used to parse the hour. 4. Unlike the "time" module, the "datetime" module does not support leap seconds. 5. When used with the "strptime()" method, the "%f" directive accepts from one to six digits and zero pads on the right. "%f" is an extension to the set of format characters in the C standard (but implemented separately in datetime objects, and therefore always available). 6. For a naive object, the "%z", "%:z" and "%Z" format codes are replaced by empty strings. For an aware object: "%z" "utcoffset()" is transformed into a string of the form "±HHMM[SS[.ffffff]]", where "HH" is a 2-digit string giving the number of UTC offset hours, "MM" is a 2-digit string giving the number of UTC offset minutes, "SS" is a 2-digit string giving the number of UTC offset seconds and "ffffff" is a 6-digit string giving the number of UTC offset microseconds. The "ffffff" part is omitted when the offset is a whole number of seconds and both the "ffffff" and the "SS" part is omitted when the offset is a whole number of minutes. For example, if "utcoffset()" returns "timedelta(hours=-3, minutes=-30)", "%z" is replaced with the string "'-0330'". Changed in version 3.7: The UTC offset is not restricted to a whole number of minutes. Changed in version 3.7: When the "%z" directive is provided to the "strptime()" method, the UTC offsets can have a colon as a separator between hours, minutes and seconds. For example, "'+01:00:00'" will be parsed as an offset of one hour. In addition, providing "'Z'" is identical to "'+00:00'". "%:z" Behaves exactly as "%z", but has a colon separator added between hours, minutes and seconds. "%Z" In "strftime()", "%Z" is replaced by an empty string if "tzname()" returns "None"; otherwise "%Z" is replaced by the returned value, which must be a string. "strptime()" only accepts certain values for "%Z": 1. any value in "time.tzname" for your machine’s locale 2. the hard-coded values "UTC" and "GMT" So someone living in Japan may have "JST", "UTC", and "GMT" as valid values, but probably not "EST". It will raise "ValueError" for invalid values. Changed in version 3.2: When the "%z" directive is provided to the "strptime()" method, an aware "datetime" object will be produced. The "tzinfo" of the result will be set to a "timezone" instance. 7. When used with the "strptime()" method, "%U" and "%W" are only used in calculations when the day of the week and the calendar year ("%Y") are specified. 8. Similar to "%U" and "%W", "%V" is only used in calculations when the day of the week and the ISO year ("%G") are specified in a "strptime()" format string. Also note that "%G" and "%Y" are not interchangeable. 9. When used with the "strptime()" method, the leading zero is optional for formats "%d", "%m", "%H", "%I", "%M", "%S", "%j", "%U", "%W", and "%V". Format "%y" does require a leading zero. 10. When parsing a month and day using "strptime()", always include a year in the format. If the value you need to parse lacks a year, append an explicit dummy leap year. Otherwise your code will raise an exception when it encounters leap day because the default year used by the parser is not a leap year. Users run into this bug every four years… >>> month_day = "02/29" >>> datetime.strptime(f"{month_day};1984", "%m/%d;%Y") # No leap year bug. datetime.datetime(1984, 2, 29, 0, 0) Deprecated since version 3.13, will be removed in version 3.15: "strptime()" calls using a format string containing a day of month without a year now emit a "DeprecationWarning". In 3.15 or later we may change this into an error or change the default year to a leap year. See gh-70647. -[ Footnotes ]- [1] If, that is, we ignore the effects of Relativity [2] This matches the definition of the “proleptic Gregorian” calendar in Dershowitz and Reingold’s book *Calendrical Calculations*, where it’s the base calendar for all computations. See the book for algorithms for converting between proleptic Gregorian ordinals and many other calendar systems. [3] See R. H. van Gent’s guide to the mathematics of the ISO 8601 calendar for a good explanation. [4] Passing "datetime.strptime('Feb 29', '%b %d')" will fail since 1900 is not a leap year. "dbm" — Interfaces to Unix “databases” ************************************** **Source code:** Lib/dbm/__init__.py ====================================================================== "dbm" is a generic interface to variants of the DBM database: * "dbm.sqlite3" * "dbm.gnu" * "dbm.ndbm" If none of these modules are installed, the slow-but-simple implementation in module "dbm.dumb" will be used. There is a third party interface to the Oracle Berkeley DB. exception dbm.error A tuple containing the exceptions that can be raised by each of the supported modules, with a unique exception also named "dbm.error" as the first item — the latter is used when "dbm.error" is raised. dbm.whichdb(filename) This function attempts to guess which of the several simple database modules available — "dbm.sqlite3", "dbm.gnu", "dbm.ndbm", or "dbm.dumb" — should be used to open a given file. Return one of the following values: * "None" if the file can’t be opened because it’s unreadable or doesn’t exist * the empty string ("''") if the file’s format can’t be guessed * a string containing the required module name, such as "'dbm.ndbm'" or "'dbm.gnu'" Changed in version 3.11: *filename* accepts a *path-like object*. dbm.open(file, flag='r', mode=0o666) Open a database and return the corresponding database object. Parameters: * **file** (*path-like object*) – The database file to open. If the database file already exists, the "whichdb()" function is used to determine its type and the appropriate module is used; if it does not exist, the first submodule listed above that can be imported is used. * **flag** (*str*) – * "'r'" (default): Open existing database for reading only. * "'w'": Open existing database for reading and writing. * "'c'": Open database for reading and writing, creating it if it doesn’t exist. * "'n'": Always create a new, empty database, open for reading and writing. * **mode** (*int*) – The Unix file access mode of the file (default: octal "0o666"), used only when the database has to be created. Changed in version 3.11: *file* accepts a *path-like object*. The object returned by "open()" supports the same basic functionality as a "dict"; keys and their corresponding values can be stored, retrieved, and deleted, and the "in" operator and the "keys()" method are available, as well as "get()" and "setdefault()" methods. Key and values are always stored as "bytes". This means that when strings are used they are implicitly converted to the default encoding before being stored. These objects also support being used in a "with" statement, which will automatically close them when done. Changed in version 3.2: "get()" and "setdefault()" methods are now available for all "dbm" backends. Changed in version 3.4: Added native support for the context management protocol to the objects returned by "open()". Changed in version 3.8: Deleting a key from a read-only database raises a database module specific exception instead of "KeyError". The following example records some hostnames and a corresponding title, and then prints out the contents of the database: import dbm # Open database, creating it if necessary. with dbm.open('cache', 'c') as db: # Record some values db[b'hello'] = b'there' db['www.python.org'] = 'Python Website' db['www.cnn.com'] = 'Cable News Network' # Note that the keys are considered bytes now. assert db[b'www.python.org'] == b'Python Website' # Notice how the value is now in bytes. assert db['www.cnn.com'] == b'Cable News Network' # Often-used methods of the dict interface work too. print(db.get('python.org', b'not present')) # Storing a non-string key or value will raise an exception (most # likely a TypeError). db['www.yahoo.com'] = 4 # db is automatically closed when leaving the with statement. See also: Module "shelve" Persistence module which stores non-string data. The individual submodules are described in the following sections. "dbm.sqlite3" — SQLite backend for dbm ====================================== Added in version 3.13. **Source code:** Lib/dbm/sqlite3.py ====================================================================== This module uses the standard library "sqlite3" module to provide an SQLite backend for the "dbm" module. The files created by "dbm.sqlite3" can thus be opened by "sqlite3", or any other SQLite browser, including the SQLite CLI. Availability: not WASI. This module does not work or is not available on WebAssembly. See WebAssembly platforms for more information. dbm.sqlite3.open(filename, /, flag='r', mode=0o666) Open an SQLite database. The returned object behaves like a *mapping*, implements a "close()" method, and supports a “closing” context manager via the "with" keyword. Parameters: * **filename** (*path-like object*) – The path to the database to be opened. * **flag** (*str*) – * "'r'" (default): Open existing database for reading only. * "'w'": Open existing database for reading and writing. * "'c'": Open database for reading and writing, creating it if it doesn’t exist. * "'n'": Always create a new, empty database, open for reading and writing. * **mode** – The Unix file access mode of the file (default: octal "0o666"), used only when the database has to be created. "dbm.gnu" — GNU database manager ================================ **Source code:** Lib/dbm/gnu.py ====================================================================== The "dbm.gnu" module provides an interface to the GDBM (GNU dbm) library, similar to the "dbm.ndbm" module, but with additional functionality like crash tolerance. Note: The file formats created by "dbm.gnu" and "dbm.ndbm" are incompatible and can not be used interchangeably. Availability: not Android, not iOS, not WASI. This module is not supported on mobile platforms or WebAssembly platforms. exception dbm.gnu.error Raised on "dbm.gnu"-specific errors, such as I/O errors. "KeyError" is raised for general mapping errors like specifying an incorrect key. dbm.gnu.open(filename, flag='r', mode=0o666, /) Open a GDBM database and return a "gdbm" object. Parameters: * **filename** (*path-like object*) – The database file to open. * **flag** (*str*) – * "'r'" (default): Open existing database for reading only. * "'w'": Open existing database for reading and writing. * "'c'": Open database for reading and writing, creating it if it doesn’t exist. * "'n'": Always create a new, empty database, open for reading and writing. The following additional characters may be appended to control how the database is opened: * "'f'": Open the database in fast mode. Writes to the database will not be synchronized. * "'s'": Synchronized mode. Changes to the database will be written immediately to the file. * "'u'": Do not lock database. Not all flags are valid for all versions of GDBM. See the "open_flags" member for a list of supported flag characters. * **mode** (*int*) – The Unix file access mode of the file (default: octal "0o666"), used only when the database has to be created. Raises: **error** – If an invalid *flag* argument is passed. Changed in version 3.11: *filename* accepts a *path-like object*. dbm.gnu.open_flags A string of characters the *flag* parameter of "open()" supports. "gdbm" objects behave similar to *mappings*, but "items()" and "values()" methods are not supported. The following methods are also provided: gdbm.firstkey() It’s possible to loop over every key in the database using this method and the "nextkey()" method. The traversal is ordered by GDBM’s internal hash values, and won’t be sorted by the key values. This method returns the starting key. gdbm.nextkey(key) Returns the key that follows *key* in the traversal. The following code prints every key in the database "db", without having to create a list in memory that contains them all: k = db.firstkey() while k is not None: print(k) k = db.nextkey(k) gdbm.reorganize() If you have carried out a lot of deletions and would like to shrink the space used by the GDBM file, this routine will reorganize the database. "gdbm" objects will not shorten the length of a database file except by using this reorganization; otherwise, deleted file space will be kept and reused as new (key, value) pairs are added. gdbm.sync() When the database has been opened in fast mode, this method forces any unwritten data to be written to the disk. gdbm.close() Close the GDBM database. gdbm.clear() Remove all items from the GDBM database. Added in version 3.13. "dbm.ndbm" — New Database Manager ================================= **Source code:** Lib/dbm/ndbm.py ====================================================================== The "dbm.ndbm" module provides an interface to the NDBM (New Database Manager) library. This module can be used with the “classic” NDBM interface or the GDBM (GNU dbm) compatibility interface. Note: The file formats created by "dbm.gnu" and "dbm.ndbm" are incompatible and can not be used interchangeably. Warning: The NDBM library shipped as part of macOS has an undocumented limitation on the size of values, which can result in corrupted database files when storing values larger than this limit. Reading such corrupted files can result in a hard crash (segmentation fault). Availability: not Android, not iOS, not WASI. This module is not supported on mobile platforms or WebAssembly platforms. exception dbm.ndbm.error Raised on "dbm.ndbm"-specific errors, such as I/O errors. "KeyError" is raised for general mapping errors like specifying an incorrect key. dbm.ndbm.library Name of the NDBM implementation library used. dbm.ndbm.open(filename, flag='r', mode=0o666, /) Open an NDBM database and return an "ndbm" object. Parameters: * **filename** (*path-like object*) – The basename of the database file (without the ".dir" or ".pag" extensions). * **flag** (*str*) – * "'r'" (default): Open existing database for reading only. * "'w'": Open existing database for reading and writing. * "'c'": Open database for reading and writing, creating it if it doesn’t exist. * "'n'": Always create a new, empty database, open for reading and writing. * **mode** (*int*) – The Unix file access mode of the file (default: octal "0o666"), used only when the database has to be created. "ndbm" objects behave similar to *mappings*, but "items()" and "values()" methods are not supported. The following methods are also provided: Changed in version 3.11: Accepts *path-like object* for filename. ndbm.close() Close the NDBM database. ndbm.clear() Remove all items from the NDBM database. Added in version 3.13. "dbm.dumb" — Portable DBM implementation ======================================== **Source code:** Lib/dbm/dumb.py Note: The "dbm.dumb" module is intended as a last resort fallback for the "dbm" module when a more robust module is not available. The "dbm.dumb" module is not written for speed and is not nearly as heavily used as the other database modules. ====================================================================== The "dbm.dumb" module provides a persistent "dict"-like interface which is written entirely in Python. Unlike other "dbm" backends, such as "dbm.gnu", no external library is required. The "dbm.dumb" module defines the following: exception dbm.dumb.error Raised on "dbm.dumb"-specific errors, such as I/O errors. "KeyError" is raised for general mapping errors like specifying an incorrect key. dbm.dumb.open(filename, flag='c', mode=0o666) Open a "dbm.dumb" database. The returned database object behaves similar to a *mapping*, in addition to providing "sync()" and "close()" methods. Parameters: * **filename** – The basename of the database file (without extensions). A new database creates the following files: * "*filename*.dat" * "*filename*.dir" * **flag** (*str*) – * "'r'": Open existing database for reading only. * "'w'": Open existing database for reading and writing. * "'c'" (default): Open database for reading and writing, creating it if it doesn’t exist. * "'n'": Always create a new, empty database, open for reading and writing. * **mode** (*int*) – The Unix file access mode of the file (default: octal "0o666"), used only when the database has to be created. Warning: It is possible to crash the Python interpreter when loading a database with a sufficiently large/complex entry due to stack depth limitations in Python’s AST compiler. Changed in version 3.5: "open()" always creates a new database when *flag* is "'n'". Changed in version 3.8: A database opened read-only if *flag* is "'r'". A database is not created if it does not exist if *flag* is "'r'" or "'w'". Changed in version 3.11: *filename* accepts a *path-like object*. In addition to the methods provided by the "collections.abc.MutableMapping" class, the following methods are provided: dumbdbm.sync() Synchronize the on-disk directory and data files. This method is called by the "shelve.Shelf.sync()" method. dumbdbm.close() Close the database. Debugging and Profiling *********************** These libraries help you with Python development: the debugger enables you to step through code, analyze stack frames and set breakpoints etc., and the profilers run code and give you a detailed breakdown of execution times, allowing you to identify bottlenecks in your programs. Auditing events provide visibility into runtime behaviors that would otherwise require intrusive debugging or patching. * Audit events table * "bdb" — Debugger framework * "faulthandler" — Dump the Python traceback * Dumping the traceback * Fault handler state * Dumping the tracebacks after a timeout * Dumping the traceback on a user signal * Issue with file descriptors * Example * "pdb" — The Python Debugger * Debugger Commands * The Python Profilers * Introduction to the profilers * Instant User’s Manual * "profile" and "cProfile" Module Reference * The "Stats" Class * What Is Deterministic Profiling? * Limitations * Calibration * Using a custom timer * "timeit" — Measure execution time of small code snippets * Basic Examples * Python Interface * Command-Line Interface * Examples * "trace" — Trace or track Python statement execution * Command-Line Usage * Main options * Modifiers * Filters * Programmatic Interface * "tracemalloc" — Trace memory allocations * Examples * Display the top 10 * Compute differences * Get the traceback of a memory block * Pretty top * Record the current and peak size of all traced memory blocks * API * Functions * DomainFilter * Filter * Frame * Snapshot * Statistic * StatisticDiff * Trace * Traceback "decimal" — Decimal fixed-point and floating-point arithmetic ************************************************************* **Source code:** Lib/decimal.py ====================================================================== The "decimal" module provides support for fast correctly rounded decimal floating-point arithmetic. It offers several advantages over the "float" datatype: * Decimal “is based on a floating-point model which was designed with people in mind, and necessarily has a paramount guiding principle – computers must provide an arithmetic that works in the same way as the arithmetic that people learn at school.” – excerpt from the decimal arithmetic specification. * Decimal numbers can be represented exactly. In contrast, numbers like "1.1" and "2.2" do not have exact representations in binary floating point. End users typically would not expect "1.1 + 2.2" to display as "3.3000000000000003" as it does with binary floating point. * The exactness carries over into arithmetic. In decimal floating point, "0.1 + 0.1 + 0.1 - 0.3" is exactly equal to zero. In binary floating point, the result is "5.5511151231257827e-017". While near to zero, the differences prevent reliable equality testing and differences can accumulate. For this reason, decimal is preferred in accounting applications which have strict equality invariants. * The decimal module incorporates a notion of significant places so that "1.30 + 1.20" is "2.50". The trailing zero is kept to indicate significance. This is the customary presentation for monetary applications. For multiplication, the “schoolbook” approach uses all the figures in the multiplicands. For instance, "1.3 * 1.2" gives "1.56" while "1.30 * 1.20" gives "1.5600". * Unlike hardware based binary floating point, the decimal module has a user alterable precision (defaulting to 28 places) which can be as large as needed for a given problem: >>> from decimal import * >>> getcontext().prec = 6 >>> Decimal(1) / Decimal(7) Decimal('0.142857') >>> getcontext().prec = 28 >>> Decimal(1) / Decimal(7) Decimal('0.1428571428571428571428571429') * Both binary and decimal floating point are implemented in terms of published standards. While the built-in float type exposes only a modest portion of its capabilities, the decimal module exposes all required parts of the standard. When needed, the programmer has full control over rounding and signal handling. This includes an option to enforce exact arithmetic by using exceptions to block any inexact operations. * The decimal module was designed to support “without prejudice, both exact unrounded decimal arithmetic (sometimes called fixed-point arithmetic) and rounded floating-point arithmetic.” – excerpt from the decimal arithmetic specification. The module design is centered around three concepts: the decimal number, the context for arithmetic, and signals. A decimal number is immutable. It has a sign, coefficient digits, and an exponent. To preserve significance, the coefficient digits do not truncate trailing zeros. Decimals also include special values such as "Infinity", "-Infinity", and "NaN". The standard also differentiates "-0" from "+0". The context for arithmetic is an environment specifying precision, rounding rules, limits on exponents, flags indicating the results of operations, and trap enablers which determine whether signals are treated as exceptions. Rounding options include "ROUND_CEILING", "ROUND_DOWN", "ROUND_FLOOR", "ROUND_HALF_DOWN", "ROUND_HALF_EVEN", "ROUND_HALF_UP", "ROUND_UP", and "ROUND_05UP". Signals are groups of exceptional conditions arising during the course of computation. Depending on the needs of the application, signals may be ignored, considered as informational, or treated as exceptions. The signals in the decimal module are: "Clamped", "InvalidOperation", "DivisionByZero", "Inexact", "Rounded", "Subnormal", "Overflow", "Underflow" and "FloatOperation". For each signal there is a flag and a trap enabler. When a signal is encountered, its flag is set to one, then, if the trap enabler is set to one, an exception is raised. Flags are sticky, so the user needs to reset them before monitoring a calculation. See also: * IBM’s General Decimal Arithmetic Specification, The General Decimal Arithmetic Specification. Quick-start tutorial ==================== The usual start to using decimals is importing the module, viewing the current context with "getcontext()" and, if necessary, setting new values for precision, rounding, or enabled traps: >>> from decimal import * >>> getcontext() Context(prec=28, rounding=ROUND_HALF_EVEN, Emin=-999999, Emax=999999, capitals=1, clamp=0, flags=[], traps=[Overflow, DivisionByZero, InvalidOperation]) >>> getcontext().prec = 7 # Set a new precision Decimal instances can be constructed from integers, strings, floats, or tuples. Construction from an integer or a float performs an exact conversion of the value of that integer or float. Decimal numbers include special values such as "NaN" which stands for “Not a number”, positive and negative "Infinity", and "-0": >>> getcontext().prec = 28 >>> Decimal(10) Decimal('10') >>> Decimal('3.14') Decimal('3.14') >>> Decimal(3.14) Decimal('3.140000000000000124344978758017532527446746826171875') >>> Decimal((0, (3, 1, 4), -2)) Decimal('3.14') >>> Decimal(str(2.0 ** 0.5)) Decimal('1.4142135623730951') >>> Decimal(2) ** Decimal('0.5') Decimal('1.414213562373095048801688724') >>> Decimal('NaN') Decimal('NaN') >>> Decimal('-Infinity') Decimal('-Infinity') If the "FloatOperation" signal is trapped, accidental mixing of decimals and floats in constructors or ordering comparisons raises an exception: >>> c = getcontext() >>> c.traps[FloatOperation] = True >>> Decimal(3.14) Traceback (most recent call last): File "", line 1, in decimal.FloatOperation: [] >>> Decimal('3.5') < 3.7 Traceback (most recent call last): File "", line 1, in decimal.FloatOperation: [] >>> Decimal('3.5') == 3.5 True Added in version 3.3. The significance of a new Decimal is determined solely by the number of digits input. Context precision and rounding only come into play during arithmetic operations. >>> getcontext().prec = 6 >>> Decimal('3.0') Decimal('3.0') >>> Decimal('3.1415926535') Decimal('3.1415926535') >>> Decimal('3.1415926535') + Decimal('2.7182818285') Decimal('5.85987') >>> getcontext().rounding = ROUND_UP >>> Decimal('3.1415926535') + Decimal('2.7182818285') Decimal('5.85988') If the internal limits of the C version are exceeded, constructing a decimal raises "InvalidOperation": >>> Decimal("1e9999999999999999999") Traceback (most recent call last): File "", line 1, in decimal.InvalidOperation: [] Changed in version 3.3. Decimals interact well with much of the rest of Python. Here is a small decimal floating-point flying circus: >>> data = list(map(Decimal, '1.34 1.87 3.45 2.35 1.00 0.03 9.25'.split())) >>> max(data) Decimal('9.25') >>> min(data) Decimal('0.03') >>> sorted(data) [Decimal('0.03'), Decimal('1.00'), Decimal('1.34'), Decimal('1.87'), Decimal('2.35'), Decimal('3.45'), Decimal('9.25')] >>> sum(data) Decimal('19.29') >>> a,b,c = data[:3] >>> str(a) '1.34' >>> float(a) 1.34 >>> round(a, 1) Decimal('1.3') >>> int(a) 1 >>> a * 5 Decimal('6.70') >>> a * b Decimal('2.5058') >>> c % a Decimal('0.77') And some mathematical functions are also available to Decimal: >>> getcontext().prec = 28 >>> Decimal(2).sqrt() Decimal('1.414213562373095048801688724') >>> Decimal(1).exp() Decimal('2.718281828459045235360287471') >>> Decimal('10').ln() Decimal('2.302585092994045684017991455') >>> Decimal('10').log10() Decimal('1') The "quantize()" method rounds a number to a fixed exponent. This method is useful for monetary applications that often round results to a fixed number of places: >>> Decimal('7.325').quantize(Decimal('.01'), rounding=ROUND_DOWN) Decimal('7.32') >>> Decimal('7.325').quantize(Decimal('1.'), rounding=ROUND_UP) Decimal('8') As shown above, the "getcontext()" function accesses the current context and allows the settings to be changed. This approach meets the needs of most applications. For more advanced work, it may be useful to create alternate contexts using the Context() constructor. To make an alternate active, use the "setcontext()" function. In accordance with the standard, the "decimal" module provides two ready to use standard contexts, "BasicContext" and "ExtendedContext". The former is especially useful for debugging because many of the traps are enabled: >>> myothercontext = Context(prec=60, rounding=ROUND_HALF_DOWN) >>> setcontext(myothercontext) >>> Decimal(1) / Decimal(7) Decimal('0.142857142857142857142857142857142857142857142857142857142857') >>> ExtendedContext Context(prec=9, rounding=ROUND_HALF_EVEN, Emin=-999999, Emax=999999, capitals=1, clamp=0, flags=[], traps=[]) >>> setcontext(ExtendedContext) >>> Decimal(1) / Decimal(7) Decimal('0.142857143') >>> Decimal(42) / Decimal(0) Decimal('Infinity') >>> setcontext(BasicContext) >>> Decimal(42) / Decimal(0) Traceback (most recent call last): File "", line 1, in -toplevel- Decimal(42) / Decimal(0) DivisionByZero: x / 0 Contexts also have signal flags for monitoring exceptional conditions encountered during computations. The flags remain set until explicitly cleared, so it is best to clear the flags before each set of monitored computations by using the "clear_flags()" method. >>> setcontext(ExtendedContext) >>> getcontext().clear_flags() >>> Decimal(355) / Decimal(113) Decimal('3.14159292') >>> getcontext() Context(prec=9, rounding=ROUND_HALF_EVEN, Emin=-999999, Emax=999999, capitals=1, clamp=0, flags=[Inexact, Rounded], traps=[]) The *flags* entry shows that the rational approximation to pi was rounded (digits beyond the context precision were thrown away) and that the result is inexact (some of the discarded digits were non- zero). Individual traps are set using the dictionary in the "traps" attribute of a context: >>> setcontext(ExtendedContext) >>> Decimal(1) / Decimal(0) Decimal('Infinity') >>> getcontext().traps[DivisionByZero] = 1 >>> Decimal(1) / Decimal(0) Traceback (most recent call last): File "", line 1, in -toplevel- Decimal(1) / Decimal(0) DivisionByZero: x / 0 Most programs adjust the current context only once, at the beginning of the program. And, in many applications, data is converted to "Decimal" with a single cast inside a loop. With context set and decimals created, the bulk of the program manipulates the data no differently than with other Python numeric types. Decimal objects =============== class decimal.Decimal(value='0', context=None) Construct a new "Decimal" object based from *value*. *value* can be an integer, string, tuple, "float", or another "Decimal" object. If no *value* is given, returns "Decimal('0')". If *value* is a string, it should conform to the decimal numeric string syntax after leading and trailing whitespace characters, as well as underscores throughout, are removed: sign ::= '+' | '-' digit ::= '0' | '1' | '2' | '3' | '4' | '5' | '6' | '7' | '8' | '9' indicator ::= 'e' | 'E' digits ::= digit [digit]... decimal-part ::= digits '.' [digits] | ['.'] digits exponent-part ::= indicator [sign] digits infinity ::= 'Infinity' | 'Inf' nan ::= 'NaN' [digits] | 'sNaN' [digits] numeric-value ::= decimal-part [exponent-part] | infinity numeric-string ::= [sign] numeric-value | [sign] nan Other Unicode decimal digits are also permitted where "digit" appears above. These include decimal digits from various other alphabets (for example, Arabic-Indic and Devanāgarī digits) along with the fullwidth digits "'\uff10'" through "'\uff19'". Case is not significant, so, for example, "inf", "Inf", "INFINITY", and "iNfINity" are all acceptable spellings for positive infinity. If *value* is a "tuple", it should have three components, a sign ("0" for positive or "1" for negative), a "tuple" of digits, and an integer exponent. For example, "Decimal((0, (1, 4, 1, 4), -3))" returns "Decimal('1.414')". If *value* is a "float", the binary floating-point value is losslessly converted to its exact decimal equivalent. This conversion can often require 53 or more digits of precision. For example, "Decimal(float('1.1'))" converts to "Decimal('1.100000000000000088817841970012523233890533447265625')". The *context* precision does not affect how many digits are stored. That is determined exclusively by the number of digits in *value*. For example, "Decimal('3.00000')" records all five zeros even if the context precision is only three. The purpose of the *context* argument is determining what to do if *value* is a malformed string. If the context traps "InvalidOperation", an exception is raised; otherwise, the constructor returns a new Decimal with the value of "NaN". Once constructed, "Decimal" objects are immutable. Changed in version 3.2: The argument to the constructor is now permitted to be a "float" instance. Changed in version 3.3: "float" arguments raise an exception if the "FloatOperation" trap is set. By default the trap is off. Changed in version 3.6: Underscores are allowed for grouping, as with integral and floating-point literals in code. Decimal floating-point objects share many properties with the other built-in numeric types such as "float" and "int". All of the usual math operations and special methods apply. Likewise, decimal objects can be copied, pickled, printed, used as dictionary keys, used as set elements, compared, sorted, and coerced to another type (such as "float" or "int"). There are some small differences between arithmetic on Decimal objects and arithmetic on integers and floats. When the remainder operator "%" is applied to Decimal objects, the sign of the result is the sign of the *dividend* rather than the sign of the divisor: >>> (-7) % 4 1 >>> Decimal(-7) % Decimal(4) Decimal('-3') The integer division operator "//" behaves analogously, returning the integer part of the true quotient (truncating towards zero) rather than its floor, so as to preserve the usual identity "x == (x // y) * y + x % y": >>> -7 // 4 -2 >>> Decimal(-7) // Decimal(4) Decimal('-1') The "%" and "//" operators implement the "remainder" and "divide- integer" operations (respectively) as described in the specification. Decimal objects cannot generally be combined with floats or instances of "fractions.Fraction" in arithmetic operations: an attempt to add a "Decimal" to a "float", for example, will raise a "TypeError". However, it is possible to use Python’s comparison operators to compare a "Decimal" instance "x" with another number "y". This avoids confusing results when doing equality comparisons between numbers of different types. Changed in version 3.2: Mixed-type comparisons between "Decimal" instances and other numeric types are now fully supported. In addition to the standard numeric properties, decimal floating- point objects also have a number of specialized methods: adjusted() Return the adjusted exponent after shifting out the coefficient’s rightmost digits until only the lead digit remains: "Decimal('321e+5').adjusted()" returns seven. Used for determining the position of the most significant digit with respect to the decimal point. as_integer_ratio() Return a pair "(n, d)" of integers that represent the given "Decimal" instance as a fraction, in lowest terms and with a positive denominator: >>> Decimal('-3.14').as_integer_ratio() (-157, 50) The conversion is exact. Raise OverflowError on infinities and ValueError on NaNs. Added in version 3.6. as_tuple() Return a *named tuple* representation of the number: "DecimalTuple(sign, digits, exponent)". canonical() Return the canonical encoding of the argument. Currently, the encoding of a "Decimal" instance is always canonical, so this operation returns its argument unchanged. compare(other, context=None) Compare the values of two Decimal instances. "compare()" returns a Decimal instance, and if either operand is a NaN then the result is a NaN: a or b is a NaN ==> Decimal('NaN') a < b ==> Decimal('-1') a == b ==> Decimal('0') a > b ==> Decimal('1') compare_signal(other, context=None) This operation is identical to the "compare()" method, except that all NaNs signal. That is, if neither operand is a signaling NaN then any quiet NaN operand is treated as though it were a signaling NaN. compare_total(other, context=None) Compare two operands using their abstract representation rather than their numerical value. Similar to the "compare()" method, but the result gives a total ordering on "Decimal" instances. Two "Decimal" instances with the same numeric value but different representations compare unequal in this ordering: >>> Decimal('12.0').compare_total(Decimal('12')) Decimal('-1') Quiet and signaling NaNs are also included in the total ordering. The result of this function is "Decimal('0')" if both operands have the same representation, "Decimal('-1')" if the first operand is lower in the total order than the second, and "Decimal('1')" if the first operand is higher in the total order than the second operand. See the specification for details of the total order. This operation is unaffected by context and is quiet: no flags are changed and no rounding is performed. As an exception, the C version may raise InvalidOperation if the second operand cannot be converted exactly. compare_total_mag(other, context=None) Compare two operands using their abstract representation rather than their value as in "compare_total()", but ignoring the sign of each operand. "x.compare_total_mag(y)" is equivalent to "x.copy_abs().compare_total(y.copy_abs())". This operation is unaffected by context and is quiet: no flags are changed and no rounding is performed. As an exception, the C version may raise InvalidOperation if the second operand cannot be converted exactly. conjugate() Just returns self, this method is only to comply with the Decimal Specification. copy_abs() Return the absolute value of the argument. This operation is unaffected by the context and is quiet: no flags are changed and no rounding is performed. copy_negate() Return the negation of the argument. This operation is unaffected by the context and is quiet: no flags are changed and no rounding is performed. copy_sign(other, context=None) Return a copy of the first operand with the sign set to be the same as the sign of the second operand. For example: >>> Decimal('2.3').copy_sign(Decimal('-1.5')) Decimal('-2.3') This operation is unaffected by context and is quiet: no flags are changed and no rounding is performed. As an exception, the C version may raise InvalidOperation if the second operand cannot be converted exactly. exp(context=None) Return the value of the (natural) exponential function "e**x" at the given number. The result is correctly rounded using the "ROUND_HALF_EVEN" rounding mode. >>> Decimal(1).exp() Decimal('2.718281828459045235360287471') >>> Decimal(321).exp() Decimal('2.561702493119680037517373933E+139') classmethod from_float(f) Alternative constructor that only accepts instances of "float" or "int". Note "Decimal.from_float(0.1)" is not the same as "Decimal('0.1')". Since 0.1 is not exactly representable in binary floating point, the value is stored as the nearest representable value which is "0x1.999999999999ap-4". That equivalent value in decimal is "0.1000000000000000055511151231257827021181583404541015625". Note: From Python 3.2 onwards, a "Decimal" instance can also be constructed directly from a "float". >>> Decimal.from_float(0.1) Decimal('0.1000000000000000055511151231257827021181583404541015625') >>> Decimal.from_float(float('nan')) Decimal('NaN') >>> Decimal.from_float(float('inf')) Decimal('Infinity') >>> Decimal.from_float(float('-inf')) Decimal('-Infinity') Added in version 3.1. fma(other, third, context=None) Fused multiply-add. Return self*other+third with no rounding of the intermediate product self*other. >>> Decimal(2).fma(3, 5) Decimal('11') is_canonical() Return "True" if the argument is canonical and "False" otherwise. Currently, a "Decimal" instance is always canonical, so this operation always returns "True". is_finite() Return "True" if the argument is a finite number, and "False" if the argument is an infinity or a NaN. is_infinite() Return "True" if the argument is either positive or negative infinity and "False" otherwise. is_nan() Return "True" if the argument is a (quiet or signaling) NaN and "False" otherwise. is_normal(context=None) Return "True" if the argument is a *normal* finite number. Return "False" if the argument is zero, subnormal, infinite or a NaN. is_qnan() Return "True" if the argument is a quiet NaN, and "False" otherwise. is_signed() Return "True" if the argument has a negative sign and "False" otherwise. Note that zeros and NaNs can both carry signs. is_snan() Return "True" if the argument is a signaling NaN and "False" otherwise. is_subnormal(context=None) Return "True" if the argument is subnormal, and "False" otherwise. is_zero() Return "True" if the argument is a (positive or negative) zero and "False" otherwise. ln(context=None) Return the natural (base e) logarithm of the operand. The result is correctly rounded using the "ROUND_HALF_EVEN" rounding mode. log10(context=None) Return the base ten logarithm of the operand. The result is correctly rounded using the "ROUND_HALF_EVEN" rounding mode. logb(context=None) For a nonzero number, return the adjusted exponent of its operand as a "Decimal" instance. If the operand is a zero then "Decimal('-Infinity')" is returned and the "DivisionByZero" flag is raised. If the operand is an infinity then "Decimal('Infinity')" is returned. logical_and(other, context=None) "logical_and()" is a logical operation which takes two *logical operands* (see Logical operands). The result is the digit-wise "and" of the two operands. logical_invert(context=None) "logical_invert()" is a logical operation. The result is the digit-wise inversion of the operand. logical_or(other, context=None) "logical_or()" is a logical operation which takes two *logical operands* (see Logical operands). The result is the digit-wise "or" of the two operands. logical_xor(other, context=None) "logical_xor()" is a logical operation which takes two *logical operands* (see Logical operands). The result is the digit-wise exclusive or of the two operands. max(other, context=None) Like "max(self, other)" except that the context rounding rule is applied before returning and that "NaN" values are either signaled or ignored (depending on the context and whether they are signaling or quiet). max_mag(other, context=None) Similar to the "max()" method, but the comparison is done using the absolute values of the operands. min(other, context=None) Like "min(self, other)" except that the context rounding rule is applied before returning and that "NaN" values are either signaled or ignored (depending on the context and whether they are signaling or quiet). min_mag(other, context=None) Similar to the "min()" method, but the comparison is done using the absolute values of the operands. next_minus(context=None) Return the largest number representable in the given context (or in the current thread’s context if no context is given) that is smaller than the given operand. next_plus(context=None) Return the smallest number representable in the given context (or in the current thread’s context if no context is given) that is larger than the given operand. next_toward(other, context=None) If the two operands are unequal, return the number closest to the first operand in the direction of the second operand. If both operands are numerically equal, return a copy of the first operand with the sign set to be the same as the sign of the second operand. normalize(context=None) Used for producing canonical values of an equivalence class within either the current context or the specified context. This has the same semantics as the unary plus operation, except that if the final result is finite it is reduced to its simplest form, with all trailing zeros removed and its sign preserved. That is, while the coefficient is non-zero and a multiple of ten the coefficient is divided by ten and the exponent is incremented by 1. Otherwise (the coefficient is zero) the exponent is set to 0. In all cases the sign is unchanged. For example, "Decimal('32.100')" and "Decimal('0.321000e+2')" both normalize to the equivalent value "Decimal('32.1')". Note that rounding is applied *before* reducing to simplest form. In the latest versions of the specification, this operation is also known as "reduce". number_class(context=None) Return a string describing the *class* of the operand. The returned value is one of the following ten strings. * ""-Infinity"", indicating that the operand is negative infinity. * ""-Normal"", indicating that the operand is a negative normal number. * ""-Subnormal"", indicating that the operand is negative and subnormal. * ""-Zero"", indicating that the operand is a negative zero. * ""+Zero"", indicating that the operand is a positive zero. * ""+Subnormal"", indicating that the operand is positive and subnormal. * ""+Normal"", indicating that the operand is a positive normal number. * ""+Infinity"", indicating that the operand is positive infinity. * ""NaN"", indicating that the operand is a quiet NaN (Not a Number). * ""sNaN"", indicating that the operand is a signaling NaN. quantize(exp, rounding=None, context=None) Return a value equal to the first operand after rounding and having the exponent of the second operand. >>> Decimal('1.41421356').quantize(Decimal('1.000')) Decimal('1.414') Unlike other operations, if the length of the coefficient after the quantize operation would be greater than precision, then an "InvalidOperation" is signaled. This guarantees that, unless there is an error condition, the quantized exponent is always equal to that of the right-hand operand. Also unlike other operations, quantize never signals Underflow, even if the result is subnormal and inexact. If the exponent of the second operand is larger than that of the first then rounding may be necessary. In this case, the rounding mode is determined by the "rounding" argument if given, else by the given "context" argument; if neither argument is given the rounding mode of the current thread’s context is used. An error is returned whenever the resulting exponent is greater than "Emax" or less than "Etiny()". radix() Return "Decimal(10)", the radix (base) in which the "Decimal" class does all its arithmetic. Included for compatibility with the specification. remainder_near(other, context=None) Return the remainder from dividing *self* by *other*. This differs from "self % other" in that the sign of the remainder is chosen so as to minimize its absolute value. More precisely, the return value is "self - n * other" where "n" is the integer nearest to the exact value of "self / other", and if two integers are equally near then the even one is chosen. If the result is zero then its sign will be the sign of *self*. >>> Decimal(18).remainder_near(Decimal(10)) Decimal('-2') >>> Decimal(25).remainder_near(Decimal(10)) Decimal('5') >>> Decimal(35).remainder_near(Decimal(10)) Decimal('-5') rotate(other, context=None) Return the result of rotating the digits of the first operand by an amount specified by the second operand. The second operand must be an integer in the range -precision through precision. The absolute value of the second operand gives the number of places to rotate. If the second operand is positive then rotation is to the left; otherwise rotation is to the right. The coefficient of the first operand is padded on the left with zeros to length precision if necessary. The sign and exponent of the first operand are unchanged. same_quantum(other, context=None) Test whether self and other have the same exponent or whether both are "NaN". This operation is unaffected by context and is quiet: no flags are changed and no rounding is performed. As an exception, the C version may raise InvalidOperation if the second operand cannot be converted exactly. scaleb(other, context=None) Return the first operand with exponent adjusted by the second. Equivalently, return the first operand multiplied by "10**other". The second operand must be an integer. shift(other, context=None) Return the result of shifting the digits of the first operand by an amount specified by the second operand. The second operand must be an integer in the range -precision through precision. The absolute value of the second operand gives the number of places to shift. If the second operand is positive then the shift is to the left; otherwise the shift is to the right. Digits shifted into the coefficient are zeros. The sign and exponent of the first operand are unchanged. sqrt(context=None) Return the square root of the argument to full precision. to_eng_string(context=None) Convert to a string, using engineering notation if an exponent is needed. Engineering notation has an exponent which is a multiple of 3. This can leave up to 3 digits to the left of the decimal place and may require the addition of either one or two trailing zeros. For example, this converts "Decimal('123E+1')" to "Decimal('1.23E+3')". to_integral(rounding=None, context=None) Identical to the "to_integral_value()" method. The "to_integral" name has been kept for compatibility with older versions. to_integral_exact(rounding=None, context=None) Round to the nearest integer, signaling "Inexact" or "Rounded" as appropriate if rounding occurs. The rounding mode is determined by the "rounding" parameter if given, else by the given "context". If neither parameter is given then the rounding mode of the current context is used. to_integral_value(rounding=None, context=None) Round to the nearest integer without signaling "Inexact" or "Rounded". If given, applies *rounding*; otherwise, uses the rounding method in either the supplied *context* or the current context. Decimal numbers can be rounded using the "round()" function: round(number) round(number, ndigits) If *ndigits* is not given or "None", returns the nearest "int" to *number*, rounding ties to even, and ignoring the rounding mode of the "Decimal" context. Raises "OverflowError" if *number* is an infinity or "ValueError" if it is a (quiet or signaling) NaN. If *ndigits* is an "int", the context’s rounding mode is respected and a "Decimal" representing *number* rounded to the nearest multiple of "Decimal('1E-ndigits')" is returned; in this case, "round(number, ndigits)" is equivalent to "self.quantize(Decimal('1E-ndigits'))". Returns "Decimal('NaN')" if *number* is a quiet NaN. Raises "InvalidOperation" if *number* is an infinity, a signaling NaN, or if the length of the coefficient after the quantize operation would be greater than the current context’s precision. In other words, for the non-corner cases: * if *ndigits* is positive, return *number* rounded to *ndigits* decimal places; * if *ndigits* is zero, return *number* rounded to the nearest integer; * if *ndigits* is negative, return *number* rounded to the nearest multiple of "10**abs(ndigits)". For example: >>> from decimal import Decimal, getcontext, ROUND_DOWN >>> getcontext().rounding = ROUND_DOWN >>> round(Decimal('3.75')) # context rounding ignored 4 >>> round(Decimal('3.5')) # round-ties-to-even 4 >>> round(Decimal('3.75'), 0) # uses the context rounding Decimal('3') >>> round(Decimal('3.75'), 1) Decimal('3.7') >>> round(Decimal('3.75'), -1) Decimal('0E+1') Logical operands ---------------- The "logical_and()", "logical_invert()", "logical_or()", and "logical_xor()" methods expect their arguments to be *logical operands*. A *logical operand* is a "Decimal" instance whose exponent and sign are both zero, and whose digits are all either "0" or "1". Context objects =============== Contexts are environments for arithmetic operations. They govern precision, set rules for rounding, determine which signals are treated as exceptions, and limit the range for exponents. Each thread has its own current context which is accessed or changed using the "getcontext()" and "setcontext()" functions: decimal.getcontext() Return the current context for the active thread. decimal.setcontext(c) Set the current context for the active thread to *c*. You can also use the "with" statement and the "localcontext()" function to temporarily change the active context. decimal.localcontext(ctx=None, **kwargs) Return a context manager that will set the current context for the active thread to a copy of *ctx* on entry to the with-statement and restore the previous context when exiting the with-statement. If no context is specified, a copy of the current context is used. The *kwargs* argument is used to set the attributes of the new context. For example, the following code sets the current decimal precision to 42 places, performs a calculation, and then automatically restores the previous context: from decimal import localcontext with localcontext() as ctx: ctx.prec = 42 # Perform a high precision calculation s = calculate_something() s = +s # Round the final result back to the default precision Using keyword arguments, the code would be the following: from decimal import localcontext with localcontext(prec=42) as ctx: s = calculate_something() s = +s Raises "TypeError" if *kwargs* supplies an attribute that "Context" doesn’t support. Raises either "TypeError" or "ValueError" if *kwargs* supplies an invalid value for an attribute. Changed in version 3.11: "localcontext()" now supports setting context attributes through the use of keyword arguments. New contexts can also be created using the "Context" constructor described below. In addition, the module provides three pre-made contexts: decimal.BasicContext This is a standard context defined by the General Decimal Arithmetic Specification. Precision is set to nine. Rounding is set to "ROUND_HALF_UP". All flags are cleared. All traps are enabled (treated as exceptions) except "Inexact", "Rounded", and "Subnormal". Because many of the traps are enabled, this context is useful for debugging. decimal.ExtendedContext This is a standard context defined by the General Decimal Arithmetic Specification. Precision is set to nine. Rounding is set to "ROUND_HALF_EVEN". All flags are cleared. No traps are enabled (so that exceptions are not raised during computations). Because the traps are disabled, this context is useful for applications that prefer to have result value of "NaN" or "Infinity" instead of raising exceptions. This allows an application to complete a run in the presence of conditions that would otherwise halt the program. decimal.DefaultContext This context is used by the "Context" constructor as a prototype for new contexts. Changing a field (such a precision) has the effect of changing the default for new contexts created by the "Context" constructor. This context is most useful in multi-threaded environments. Changing one of the fields before threads are started has the effect of setting system-wide defaults. Changing the fields after threads have started is not recommended as it would require thread synchronization to prevent race conditions. In single threaded environments, it is preferable to not use this context at all. Instead, simply create contexts explicitly as described below. The default values are "Context.prec"="28", "Context.rounding"="ROUND_HALF_EVEN", and enabled traps for "Overflow", "InvalidOperation", and "DivisionByZero". In addition to the three supplied contexts, new contexts can be created with the "Context" constructor. class decimal.Context(prec=None, rounding=None, Emin=None, Emax=None, capitals=None, clamp=None, flags=None, traps=None) Creates a new context. If a field is not specified or is "None", the default values are copied from the "DefaultContext". If the *flags* field is not specified or is "None", all flags are cleared. prec An integer in the range ["1", "MAX_PREC"] that sets the precision for arithmetic operations in the context. rounding One of the constants listed in the section Rounding Modes. traps flags Lists of any signals to be set. Generally, new contexts should only set traps and leave the flags clear. Emin Emax Integers specifying the outer limits allowable for exponents. *Emin* must be in the range ["MIN_EMIN", "0"], *Emax* in the range ["0", "MAX_EMAX"]. capitals Either "0" or "1" (the default). If set to "1", exponents are printed with a capital "E"; otherwise, a lowercase "e" is used: "Decimal('6.02e+23')". clamp Either "0" (the default) or "1". If set to "1", the exponent "e" of a "Decimal" instance representable in this context is strictly limited to the range "Emin - prec + 1 <= e <= Emax - prec + 1". If *clamp* is "0" then a weaker condition holds: the adjusted exponent of the "Decimal" instance is at most "Emax". When *clamp* is "1", a large normal number will, where possible, have its exponent reduced and a corresponding number of zeros added to its coefficient, in order to fit the exponent constraints; this preserves the value of the number but loses information about significant trailing zeros. For example: >>> Context(prec=6, Emax=999, clamp=1).create_decimal('1.23e999') Decimal('1.23000E+999') A *clamp* value of "1" allows compatibility with the fixed-width decimal interchange formats specified in IEEE 754. The "Context" class defines several general purpose methods as well as a large number of methods for doing arithmetic directly in a given context. In addition, for each of the "Decimal" methods described above (with the exception of the "adjusted()" and "as_tuple()" methods) there is a corresponding "Context" method. For example, for a "Context" instance "C" and "Decimal" instance "x", "C.exp(x)" is equivalent to "x.exp(context=C)". Each "Context" method accepts a Python integer (an instance of "int") anywhere that a Decimal instance is accepted. clear_flags() Resets all of the flags to "0". clear_traps() Resets all of the traps to "0". Added in version 3.3. copy() Return a duplicate of the context. copy_decimal(num) Return a copy of the Decimal instance num. create_decimal(num) Creates a new Decimal instance from *num* but using *self* as context. Unlike the "Decimal" constructor, the context precision, rounding method, flags, and traps are applied to the conversion. This is useful because constants are often given to a greater precision than is needed by the application. Another benefit is that rounding immediately eliminates unintended effects from digits beyond the current precision. In the following example, using unrounded inputs means that adding zero to a sum can change the result: >>> getcontext().prec = 3 >>> Decimal('3.4445') + Decimal('1.0023') Decimal('4.45') >>> Decimal('3.4445') + Decimal(0) + Decimal('1.0023') Decimal('4.44') This method implements the to-number operation of the IBM specification. If the argument is a string, no leading or trailing whitespace or underscores are permitted. create_decimal_from_float(f) Creates a new Decimal instance from a float *f* but rounding using *self* as the context. Unlike the "Decimal.from_float()" class method, the context precision, rounding method, flags, and traps are applied to the conversion. >>> context = Context(prec=5, rounding=ROUND_DOWN) >>> context.create_decimal_from_float(math.pi) Decimal('3.1415') >>> context = Context(prec=5, traps=[Inexact]) >>> context.create_decimal_from_float(math.pi) Traceback (most recent call last): ... decimal.Inexact: None Added in version 3.1. Etiny() Returns a value equal to "Emin - prec + 1" which is the minimum exponent value for subnormal results. When underflow occurs, the exponent is set to "Etiny". Etop() Returns a value equal to "Emax - prec + 1". The usual approach to working with decimals is to create "Decimal" instances and then apply arithmetic operations which take place within the current context for the active thread. An alternative approach is to use context methods for calculating within a specific context. The methods are similar to those for the "Decimal" class and are only briefly recounted here. abs(x) Returns the absolute value of *x*. add(x, y) Return the sum of *x* and *y*. canonical(x) Returns the same Decimal object *x*. compare(x, y) Compares *x* and *y* numerically. compare_signal(x, y) Compares the values of the two operands numerically. compare_total(x, y) Compares two operands using their abstract representation. compare_total_mag(x, y) Compares two operands using their abstract representation, ignoring sign. copy_abs(x) Returns a copy of *x* with the sign set to 0. copy_negate(x) Returns a copy of *x* with the sign inverted. copy_sign(x, y) Copies the sign from *y* to *x*. divide(x, y) Return *x* divided by *y*. divide_int(x, y) Return *x* divided by *y*, truncated to an integer. divmod(x, y) Divides two numbers and returns the integer part of the result. exp(x) Returns "e ** x". fma(x, y, z) Returns *x* multiplied by *y*, plus *z*. is_canonical(x) Returns "True" if *x* is canonical; otherwise returns "False". is_finite(x) Returns "True" if *x* is finite; otherwise returns "False". is_infinite(x) Returns "True" if *x* is infinite; otherwise returns "False". is_nan(x) Returns "True" if *x* is a qNaN or sNaN; otherwise returns "False". is_normal(x) Returns "True" if *x* is a normal number; otherwise returns "False". is_qnan(x) Returns "True" if *x* is a quiet NaN; otherwise returns "False". is_signed(x) Returns "True" if *x* is negative; otherwise returns "False". is_snan(x) Returns "True" if *x* is a signaling NaN; otherwise returns "False". is_subnormal(x) Returns "True" if *x* is subnormal; otherwise returns "False". is_zero(x) Returns "True" if *x* is a zero; otherwise returns "False". ln(x) Returns the natural (base e) logarithm of *x*. log10(x) Returns the base 10 logarithm of *x*. logb(x) Returns the exponent of the magnitude of the operand’s MSD. logical_and(x, y) Applies the logical operation *and* between each operand’s digits. logical_invert(x) Invert all the digits in *x*. logical_or(x, y) Applies the logical operation *or* between each operand’s digits. logical_xor(x, y) Applies the logical operation *xor* between each operand’s digits. max(x, y) Compares two values numerically and returns the maximum. max_mag(x, y) Compares the values numerically with their sign ignored. min(x, y) Compares two values numerically and returns the minimum. min_mag(x, y) Compares the values numerically with their sign ignored. minus(x) Minus corresponds to the unary prefix minus operator in Python. multiply(x, y) Return the product of *x* and *y*. next_minus(x) Returns the largest representable number smaller than *x*. next_plus(x) Returns the smallest representable number larger than *x*. next_toward(x, y) Returns the number closest to *x*, in direction towards *y*. normalize(x) Reduces *x* to its simplest form. number_class(x) Returns an indication of the class of *x*. plus(x) Plus corresponds to the unary prefix plus operator in Python. This operation applies the context precision and rounding, so it is *not* an identity operation. power(x, y, modulo=None) Return "x" to the power of "y", reduced modulo "modulo" if given. With two arguments, compute "x**y". If "x" is negative then "y" must be integral. The result will be inexact unless "y" is integral and the result is finite and can be expressed exactly in ‘precision’ digits. The rounding mode of the context is used. Results are always correctly rounded in the Python version. "Decimal(0) ** Decimal(0)" results in "InvalidOperation", and if "InvalidOperation" is not trapped, then results in "Decimal('NaN')". Changed in version 3.3: The C module computes "power()" in terms of the correctly rounded "exp()" and "ln()" functions. The result is well-defined but only “almost always correctly rounded”. With three arguments, compute "(x**y) % modulo". For the three argument form, the following restrictions on the arguments hold: * all three arguments must be integral * "y" must be nonnegative * at least one of "x" or "y" must be nonzero * "modulo" must be nonzero and have at most ‘precision’ digits The value resulting from "Context.power(x, y, modulo)" is equal to the value that would be obtained by computing "(x**y) % modulo" with unbounded precision, but is computed more efficiently. The exponent of the result is zero, regardless of the exponents of "x", "y" and "modulo". The result is always exact. quantize(x, y) Returns a value equal to *x* (rounded), having the exponent of *y*. radix() Just returns 10, as this is Decimal, :) remainder(x, y) Returns the remainder from integer division. The sign of the result, if non-zero, is the same as that of the original dividend. remainder_near(x, y) Returns "x - y * n", where *n* is the integer nearest the exact value of "x / y" (if the result is 0 then its sign will be the sign of *x*). rotate(x, y) Returns a rotated copy of *x*, *y* times. same_quantum(x, y) Returns "True" if the two operands have the same exponent. scaleb(x, y) Returns the first operand after adding the second value its exp. shift(x, y) Returns a shifted copy of *x*, *y* times. sqrt(x) Square root of a non-negative number to context precision. subtract(x, y) Return the difference between *x* and *y*. to_eng_string(x) Convert to a string, using engineering notation if an exponent is needed. Engineering notation has an exponent which is a multiple of 3. This can leave up to 3 digits to the left of the decimal place and may require the addition of either one or two trailing zeros. to_integral_exact(x) Rounds to an integer. to_sci_string(x) Converts a number to a string using scientific notation. Constants ========= The constants in this section are only relevant for the C module. They are also included in the pure Python version for compatibility. +-----------------------+-----------------------+---------------------------------+ | | 32-bit | 64-bit | |=======================|=======================|=================================| | decimal.MAX_PREC | "425000000" | "999999999999999999" | +-----------------------+-----------------------+---------------------------------+ | decimal.MAX_EMAX | "425000000" | "999999999999999999" | +-----------------------+-----------------------+---------------------------------+ | decimal.MIN_EMIN | "-425000000" | "-999999999999999999" | +-----------------------+-----------------------+---------------------------------+ | decimal.MIN_ETINY | "-849999999" | "-1999999999999999997" | +-----------------------+-----------------------+---------------------------------+ decimal.HAVE_THREADS The value is "True". Deprecated, because Python now always has threads. Deprecated since version 3.9. decimal.HAVE_CONTEXTVAR The default value is "True". If Python is "configured using the --without-decimal-contextvar option", the C version uses a thread- local rather than a coroutine-local context and the value is "False". This is slightly faster in some nested context scenarios. Added in version 3.8.3. Rounding modes ============== decimal.ROUND_CEILING Round towards "Infinity". decimal.ROUND_DOWN Round towards zero. decimal.ROUND_FLOOR Round towards "-Infinity". decimal.ROUND_HALF_DOWN Round to nearest with ties going towards zero. decimal.ROUND_HALF_EVEN Round to nearest with ties going to nearest even integer. decimal.ROUND_HALF_UP Round to nearest with ties going away from zero. decimal.ROUND_UP Round away from zero. decimal.ROUND_05UP Round away from zero if last digit after rounding towards zero would have been 0 or 5; otherwise round towards zero. Signals ======= Signals represent conditions that arise during computation. Each corresponds to one context flag and one context trap enabler. The context flag is set whenever the condition is encountered. After the computation, flags may be checked for informational purposes (for instance, to determine whether a computation was exact). After checking the flags, be sure to clear all flags before starting the next computation. If the context’s trap enabler is set for the signal, then the condition causes a Python exception to be raised. For example, if the "DivisionByZero" trap is set, then a "DivisionByZero" exception is raised upon encountering the condition. class decimal.Clamped Altered an exponent to fit representation constraints. Typically, clamping occurs when an exponent falls outside the context’s "Emin" and "Emax" limits. If possible, the exponent is reduced to fit by adding zeros to the coefficient. class decimal.DecimalException Base class for other signals and a subclass of "ArithmeticError". class decimal.DivisionByZero Signals the division of a non-infinite number by zero. Can occur with division, modulo division, or when raising a number to a negative power. If this signal is not trapped, returns "Infinity" or "-Infinity" with the sign determined by the inputs to the calculation. class decimal.Inexact Indicates that rounding occurred and the result is not exact. Signals when non-zero digits were discarded during rounding. The rounded result is returned. The signal flag or trap is used to detect when results are inexact. class decimal.InvalidOperation An invalid operation was performed. Indicates that an operation was requested that does not make sense. If not trapped, returns "NaN". Possible causes include: Infinity - Infinity 0 * Infinity Infinity / Infinity x % 0 Infinity % x sqrt(-x) and x > 0 0 ** 0 x ** (non-integer) x ** Infinity class decimal.Overflow Numerical overflow. Indicates the exponent is larger than "Context.Emax" after rounding has occurred. If not trapped, the result depends on the rounding mode, either pulling inward to the largest representable finite number or rounding outward to "Infinity". In either case, "Inexact" and "Rounded" are also signaled. class decimal.Rounded Rounding occurred though possibly no information was lost. Signaled whenever rounding discards digits; even if those digits are zero (such as rounding "5.00" to "5.0"). If not trapped, returns the result unchanged. This signal is used to detect loss of significant digits. class decimal.Subnormal Exponent was lower than "Emin" prior to rounding. Occurs when an operation result is subnormal (the exponent is too small). If not trapped, returns the result unchanged. class decimal.Underflow Numerical underflow with result rounded to zero. Occurs when a subnormal result is pushed to zero by rounding. "Inexact" and "Subnormal" are also signaled. class decimal.FloatOperation Enable stricter semantics for mixing floats and Decimals. If the signal is not trapped (default), mixing floats and Decimals is permitted in the "Decimal" constructor, "create_decimal()" and all comparison operators. Both conversion and comparisons are exact. Any occurrence of a mixed operation is silently recorded by setting "FloatOperation" in the context flags. Explicit conversions with "from_float()" or "create_decimal_from_float()" do not set the flag. Otherwise (the signal is trapped), only equality comparisons and explicit conversions are silent. All other mixed operations raise "FloatOperation". The following table summarizes the hierarchy of signals: exceptions.ArithmeticError(exceptions.Exception) DecimalException Clamped DivisionByZero(DecimalException, exceptions.ZeroDivisionError) Inexact Overflow(Inexact, Rounded) Underflow(Inexact, Rounded, Subnormal) InvalidOperation Rounded Subnormal FloatOperation(DecimalException, exceptions.TypeError) Floating-point notes ==================== Mitigating round-off error with increased precision --------------------------------------------------- The use of decimal floating point eliminates decimal representation error (making it possible to represent "0.1" exactly); however, some operations can still incur round-off error when non-zero digits exceed the fixed precision. The effects of round-off error can be amplified by the addition or subtraction of nearly offsetting quantities resulting in loss of significance. Knuth provides two instructive examples where rounded floating-point arithmetic with insufficient precision causes the breakdown of the associative and distributive properties of addition: # Examples from Seminumerical Algorithms, Section 4.2.2. >>> from decimal import Decimal, getcontext >>> getcontext().prec = 8 >>> u, v, w = Decimal(11111113), Decimal(-11111111), Decimal('7.51111111') >>> (u + v) + w Decimal('9.5111111') >>> u + (v + w) Decimal('10') >>> u, v, w = Decimal(20000), Decimal(-6), Decimal('6.0000003') >>> (u*v) + (u*w) Decimal('0.01') >>> u * (v+w) Decimal('0.0060000') The "decimal" module makes it possible to restore the identities by expanding the precision sufficiently to avoid loss of significance: >>> getcontext().prec = 20 >>> u, v, w = Decimal(11111113), Decimal(-11111111), Decimal('7.51111111') >>> (u + v) + w Decimal('9.51111111') >>> u + (v + w) Decimal('9.51111111') >>> >>> u, v, w = Decimal(20000), Decimal(-6), Decimal('6.0000003') >>> (u*v) + (u*w) Decimal('0.0060000') >>> u * (v+w) Decimal('0.0060000') Special values -------------- The number system for the "decimal" module provides special values including "NaN", "sNaN", "-Infinity", "Infinity", and two zeros, "+0" and "-0". Infinities can be constructed directly with: "Decimal('Infinity')". Also, they can arise from dividing by zero when the "DivisionByZero" signal is not trapped. Likewise, when the "Overflow" signal is not trapped, infinity can result from rounding beyond the limits of the largest representable number. The infinities are signed (affine) and can be used in arithmetic operations where they get treated as very large, indeterminate numbers. For instance, adding a constant to infinity gives another infinite result. Some operations are indeterminate and return "NaN", or if the "InvalidOperation" signal is trapped, raise an exception. For example, "0/0" returns "NaN" which means “not a number”. This variety of "NaN" is quiet and, once created, will flow through other computations always resulting in another "NaN". This behavior can be useful for a series of computations that occasionally have missing inputs — it allows the calculation to proceed while flagging specific results as invalid. A variant is "sNaN" which signals rather than remaining quiet after every operation. This is a useful return value when an invalid result needs to interrupt a calculation for special handling. The behavior of Python’s comparison operators can be a little surprising where a "NaN" is involved. A test for equality where one of the operands is a quiet or signaling "NaN" always returns "False" (even when doing "Decimal('NaN')==Decimal('NaN')"), while a test for inequality always returns "True". An attempt to compare two Decimals using any of the "<", "<=", ">" or ">=" operators will raise the "InvalidOperation" signal if either operand is a "NaN", and return "False" if this signal is not trapped. Note that the General Decimal Arithmetic specification does not specify the behavior of direct comparisons; these rules for comparisons involving a "NaN" were taken from the IEEE 854 standard (see Table 3 in section 5.7). To ensure strict standards-compliance, use the "compare()" and "compare_signal()" methods instead. The signed zeros can result from calculations that underflow. They keep the sign that would have resulted if the calculation had been carried out to greater precision. Since their magnitude is zero, both positive and negative zeros are treated as equal and their sign is informational. In addition to the two signed zeros which are distinct yet equal, there are various representations of zero with differing precisions yet equivalent in value. This takes a bit of getting used to. For an eye accustomed to normalized floating-point representations, it is not immediately obvious that the following calculation returns a value equal to zero: >>> 1 / Decimal('Infinity') Decimal('0E-1000026') Working with threads ==================== The "getcontext()" function accesses a different "Context" object for each thread. Having separate thread contexts means that threads may make changes (such as "getcontext().prec=10") without interfering with other threads. Likewise, the "setcontext()" function automatically assigns its target to the current thread. If "setcontext()" has not been called before "getcontext()", then "getcontext()" will automatically create a new context for use in the current thread. The new context is copied from a prototype context called *DefaultContext*. To control the defaults so that each thread will use the same values throughout the application, directly modify the *DefaultContext* object. This should be done *before* any threads are started so that there won’t be a race condition between threads calling "getcontext()". For example: # Set applicationwide defaults for all threads about to be launched DefaultContext.prec = 12 DefaultContext.rounding = ROUND_DOWN DefaultContext.traps = ExtendedContext.traps.copy() DefaultContext.traps[InvalidOperation] = 1 setcontext(DefaultContext) # Afterwards, the threads can be started t1.start() t2.start() t3.start() . . . Recipes ======= Here are a few recipes that serve as utility functions and that demonstrate ways to work with the "Decimal" class: def moneyfmt(value, places=2, curr='', sep=',', dp='.', pos='', neg='-', trailneg=''): """Convert Decimal to a money formatted string. places: required number of places after the decimal point curr: optional currency symbol before the sign (may be blank) sep: optional grouping separator (comma, period, space, or blank) dp: decimal point indicator (comma or period) only specify as blank when places is zero pos: optional sign for positive numbers: '+', space or blank neg: optional sign for negative numbers: '-', '(', space or blank trailneg:optional trailing minus indicator: '-', ')', space or blank >>> d = Decimal('-1234567.8901') >>> moneyfmt(d, curr='$') '-$1,234,567.89' >>> moneyfmt(d, places=0, sep='.', dp='', neg='', trailneg='-') '1.234.568-' >>> moneyfmt(d, curr='$', neg='(', trailneg=')') '($1,234,567.89)' >>> moneyfmt(Decimal(123456789), sep=' ') '123 456 789.00' >>> moneyfmt(Decimal('-0.02'), neg='<', trailneg='>') '<0.02>' """ q = Decimal(10) ** -places # 2 places --> '0.01' sign, digits, exp = value.quantize(q).as_tuple() result = [] digits = list(map(str, digits)) build, next = result.append, digits.pop if sign: build(trailneg) for i in range(places): build(next() if digits else '0') if places: build(dp) if not digits: build('0') i = 0 while digits: build(next()) i += 1 if i == 3 and digits: i = 0 build(sep) build(curr) build(neg if sign else pos) return ''.join(reversed(result)) def pi(): """Compute Pi to the current precision. >>> print(pi()) 3.141592653589793238462643383 """ getcontext().prec += 2 # extra digits for intermediate steps three = Decimal(3) # substitute "three=3.0" for regular floats lasts, t, s, n, na, d, da = 0, three, 3, 1, 0, 0, 24 while s != lasts: lasts = s n, na = n+na, na+8 d, da = d+da, da+32 t = (t * n) / d s += t getcontext().prec -= 2 return +s # unary plus applies the new precision def exp(x): """Return e raised to the power of x. Result type matches input type. >>> print(exp(Decimal(1))) 2.718281828459045235360287471 >>> print(exp(Decimal(2))) 7.389056098930650227230427461 >>> print(exp(2.0)) 7.38905609893 >>> print(exp(2+0j)) (7.38905609893+0j) """ getcontext().prec += 2 i, lasts, s, fact, num = 0, 0, 1, 1, 1 while s != lasts: lasts = s i += 1 fact *= i num *= x s += num / fact getcontext().prec -= 2 return +s def cos(x): """Return the cosine of x as measured in radians. The Taylor series approximation works best for a small value of x. For larger values, first compute x = x % (2 * pi). >>> print(cos(Decimal('0.5'))) 0.8775825618903727161162815826 >>> print(cos(0.5)) 0.87758256189 >>> print(cos(0.5+0j)) (0.87758256189+0j) """ getcontext().prec += 2 i, lasts, s, fact, num, sign = 0, 0, 1, 1, 1, 1 while s != lasts: lasts = s i += 2 fact *= i * (i-1) num *= x * x sign *= -1 s += num / fact * sign getcontext().prec -= 2 return +s def sin(x): """Return the sine of x as measured in radians. The Taylor series approximation works best for a small value of x. For larger values, first compute x = x % (2 * pi). >>> print(sin(Decimal('0.5'))) 0.4794255386042030002732879352 >>> print(sin(0.5)) 0.479425538604 >>> print(sin(0.5+0j)) (0.479425538604+0j) """ getcontext().prec += 2 i, lasts, s, fact, num, sign = 1, 0, x, 1, x, 1 while s != lasts: lasts = s i += 2 fact *= i * (i-1) num *= x * x sign *= -1 s += num / fact * sign getcontext().prec -= 2 return +s Decimal FAQ =========== Q. It is cumbersome to type "decimal.Decimal('1234.5')". Is there a way to minimize typing when using the interactive interpreter? A. Some users abbreviate the constructor to just a single letter: >>> D = decimal.Decimal >>> D('1.23') + D('3.45') Decimal('4.68') Q. In a fixed-point application with two decimal places, some inputs have many places and need to be rounded. Others are not supposed to have excess digits and need to be validated. What methods should be used? A. The "quantize()" method rounds to a fixed number of decimal places. If the "Inexact" trap is set, it is also useful for validation: >>> TWOPLACES = Decimal(10) ** -2 # same as Decimal('0.01') >>> # Round to two places >>> Decimal('3.214').quantize(TWOPLACES) Decimal('3.21') >>> # Validate that a number does not exceed two places >>> Decimal('3.21').quantize(TWOPLACES, context=Context(traps=[Inexact])) Decimal('3.21') >>> Decimal('3.214').quantize(TWOPLACES, context=Context(traps=[Inexact])) Traceback (most recent call last): ... Inexact: None Q. Once I have valid two place inputs, how do I maintain that invariant throughout an application? A. Some operations like addition, subtraction, and multiplication by an integer will automatically preserve fixed point. Others operations, like division and non-integer multiplication, will change the number of decimal places and need to be followed-up with a "quantize()" step: >>> a = Decimal('102.72') # Initial fixed-point values >>> b = Decimal('3.17') >>> a + b # Addition preserves fixed-point Decimal('105.89') >>> a - b Decimal('99.55') >>> a * 42 # So does integer multiplication Decimal('4314.24') >>> (a * b).quantize(TWOPLACES) # Must quantize non-integer multiplication Decimal('325.62') >>> (b / a).quantize(TWOPLACES) # And quantize division Decimal('0.03') In developing fixed-point applications, it is convenient to define functions to handle the "quantize()" step: >>> def mul(x, y, fp=TWOPLACES): ... return (x * y).quantize(fp) ... >>> def div(x, y, fp=TWOPLACES): ... return (x / y).quantize(fp) >>> mul(a, b) # Automatically preserve fixed-point Decimal('325.62') >>> div(b, a) Decimal('0.03') Q. There are many ways to express the same value. The numbers "200", "200.000", "2E2", and ".02E+4" all have the same value at various precisions. Is there a way to transform them to a single recognizable canonical value? A. The "normalize()" method maps all equivalent values to a single representative: >>> values = map(Decimal, '200 200.000 2E2 .02E+4'.split()) >>> [v.normalize() for v in values] [Decimal('2E+2'), Decimal('2E+2'), Decimal('2E+2'), Decimal('2E+2')] Q. When does rounding occur in a computation? A. It occurs *after* the computation. The philosophy of the decimal specification is that numbers are considered exact and are created independent of the current context. They can even have greater precision than current context. Computations process with those exact inputs and then rounding (or other context operations) is applied to the *result* of the computation: >>> getcontext().prec = 5 >>> pi = Decimal('3.1415926535') # More than 5 digits >>> pi # All digits are retained Decimal('3.1415926535') >>> pi + 0 # Rounded after an addition Decimal('3.1416') >>> pi - Decimal('0.00005') # Subtract unrounded numbers, then round Decimal('3.1415') >>> pi + 0 - Decimal('0.00005'). # Intermediate values are rounded Decimal('3.1416') Q. Some decimal values always print with exponential notation. Is there a way to get a non-exponential representation? A. For some values, exponential notation is the only way to express the number of significant places in the coefficient. For example, expressing "5.0E+3" as "5000" keeps the value constant but cannot show the original’s two-place significance. If an application does not care about tracking significance, it is easy to remove the exponent and trailing zeroes, losing significance, but keeping the value unchanged: >>> def remove_exponent(d): ... return d.quantize(Decimal(1)) if d == d.to_integral() else d.normalize() >>> remove_exponent(Decimal('5E+3')) Decimal('5000') Q. Is there a way to convert a regular float to a "Decimal"? A. Yes, any binary floating-point number can be exactly expressed as a Decimal though an exact conversion may take more precision than intuition would suggest: >>> Decimal(math.pi) Decimal('3.141592653589793115997963468544185161590576171875') Q. Within a complex calculation, how can I make sure that I haven’t gotten a spurious result because of insufficient precision or rounding anomalies. A. The decimal module makes it easy to test results. A best practice is to re-run calculations using greater precision and with various rounding modes. Widely differing results indicate insufficient precision, rounding mode issues, ill-conditioned inputs, or a numerically unstable algorithm. Q. I noticed that context precision is applied to the results of operations but not to the inputs. Is there anything to watch out for when mixing values of different precisions? A. Yes. The principle is that all values are considered to be exact and so is the arithmetic on those values. Only the results are rounded. The advantage for inputs is that “what you type is what you get”. A disadvantage is that the results can look odd if you forget that the inputs haven’t been rounded: >>> getcontext().prec = 3 >>> Decimal('3.104') + Decimal('2.104') Decimal('5.21') >>> Decimal('3.104') + Decimal('0.000') + Decimal('2.104') Decimal('5.20') The solution is either to increase precision or to force rounding of inputs using the unary plus operation: >>> getcontext().prec = 3 >>> +Decimal('1.23456789') # unary plus triggers rounding Decimal('1.23') Alternatively, inputs can be rounded upon creation using the "Context.create_decimal()" method: >>> Context(prec=5, rounding=ROUND_DOWN).create_decimal('1.2345678') Decimal('1.2345') Q. Is the CPython implementation fast for large numbers? A. Yes. In the CPython and PyPy3 implementations, the C/CFFI versions of the decimal module integrate the high speed libmpdec library for arbitrary precision correctly rounded decimal floating-point arithmetic [1]. "libmpdec" uses Karatsuba multiplication for medium- sized numbers and the Number Theoretic Transform for very large numbers. The context must be adapted for exact arbitrary precision arithmetic. "Emin" and "Emax" should always be set to the maximum values, "clamp" should always be 0 (the default). Setting "prec" requires some care. The easiest approach for trying out bignum arithmetic is to use the maximum value for "prec" as well [2]: >>> setcontext(Context(prec=MAX_PREC, Emax=MAX_EMAX, Emin=MIN_EMIN)) >>> x = Decimal(2) ** 256 >>> x / 128 Decimal('904625697166532776746648320380374280103671755200316906558262375061821325312') For inexact results, "MAX_PREC" is far too large on 64-bit platforms and the available memory will be insufficient: >>> Decimal(1) / 3 Traceback (most recent call last): File "", line 1, in MemoryError On systems with overallocation (e.g. Linux), a more sophisticated approach is to adjust "prec" to the amount of available RAM. Suppose that you have 8GB of RAM and expect 10 simultaneous operands using a maximum of 500MB each: >>> import sys >>> >>> # Maximum number of digits for a single operand using 500MB in 8-byte words >>> # with 19 digits per word (4-byte and 9 digits for the 32-bit build): >>> maxdigits = 19 * ((500 * 1024**2) // 8) >>> >>> # Check that this works: >>> c = Context(prec=maxdigits, Emax=MAX_EMAX, Emin=MIN_EMIN) >>> c.traps[Inexact] = True >>> setcontext(c) >>> >>> # Fill the available precision with nines: >>> x = Decimal(0).logical_invert() * 9 >>> sys.getsizeof(x) 524288112 >>> x + 2 Traceback (most recent call last): File "", line 1, in decimal.Inexact: [] In general (and especially on systems without overallocation), it is recommended to estimate even tighter bounds and set the "Inexact" trap if all calculations are expected to be exact. [1] Added in version 3.3. [2] Changed in version 3.9: This approach now works for all exact results except for non-integer powers. Development Tools ***************** The modules described in this chapter help you write software. For example, the "pydoc" module takes a module and generates documentation based on the module’s contents. The "doctest" and "unittest" modules contains frameworks for writing unit tests that automatically exercise code and verify that the expected output is produced. The list of modules described in this chapter is: * "typing" — Support for type hints * Specification for the Python Type System * Type aliases * NewType * Annotating callable objects * Generics * Annotating tuples * The type of class objects * Annotating generators and coroutines * User-defined generic types * The "Any" type * Nominal vs structural subtyping * Module contents * Special typing primitives * Special types * Special forms * Building generic types and type aliases * Other special directives * Protocols * ABCs for working with IO * Functions and decorators * Introspection helpers * Constant * Deprecated aliases * Aliases to built-in types * Aliases to types in "collections" * Aliases to other concrete types * Aliases to container ABCs in "collections.abc" * Aliases to asynchronous ABCs in "collections.abc" * Aliases to other ABCs in "collections.abc" * Aliases to "contextlib" ABCs * Deprecation Timeline of Major Features * "pydoc" — Documentation generator and online help system * Python Development Mode * Effects of the Python Development Mode * ResourceWarning Example * Bad file descriptor error example * "doctest" — Test interactive Python examples * Simple Usage: Checking Examples in Docstrings * Simple Usage: Checking Examples in a Text File * Command-line Usage * How It Works * Which Docstrings Are Examined? * How are Docstring Examples Recognized? * What’s the Execution Context? * What About Exceptions? * Option Flags * Directives * Warnings * Basic API * Unittest API * Advanced API * DocTest Objects * Example Objects * DocTestFinder objects * DocTestParser objects * TestResults objects * DocTestRunner objects * OutputChecker objects * Debugging * Soapbox * "unittest" — Unit testing framework * Basic example * Command-Line Interface * Command-line options * Test Discovery * Organizing test code * Re-using old test code * Skipping tests and expected failures * Distinguishing test iterations using subtests * Classes and functions * Test cases * Grouping tests * Loading and running tests * load_tests Protocol * Class and Module Fixtures * setUpClass and tearDownClass * setUpModule and tearDownModule * Signal Handling * "unittest.mock" — mock object library * Quick Guide * The Mock Class * Calling * Deleting Attributes * Mock names and the name attribute * Attaching Mocks as Attributes * The patchers * patch * patch.object * patch.dict * patch.multiple * patch methods: start and stop * patch builtins * TEST_PREFIX * Nesting Patch Decorators * Where to patch * Patching Descriptors and Proxy Objects * MagicMock and magic method support * Mocking Magic Methods * Magic Mock * Helpers * sentinel * DEFAULT * call * create_autospec * ANY * FILTER_DIR * mock_open * Autospeccing * Sealing mocks * Order of precedence of "side_effect", "return_value" and *wraps* * "unittest.mock" — getting started * Using Mock * Mock Patching Methods * Mock for Method Calls on an Object * Mocking Classes * Naming your mocks * Tracking all Calls * Setting Return Values and Attributes * Raising exceptions with mocks * Side effect functions and iterables * Mocking asynchronous iterators * Mocking asynchronous context manager * Creating a Mock from an Existing Object * Using side_effect to return per file content * Patch Decorators * Further Examples * Mocking chained calls * Partial mocking * Mocking a Generator Method * Applying the same patch to every test method * Mocking Unbound Methods * Checking multiple calls with mock * Coping with mutable arguments * Nesting Patches * Mocking a dictionary with MagicMock * Mock subclasses and their attributes * Mocking imports with patch.dict * Tracking order of calls and less verbose call assertions * More complex argument matching * "test" — Regression tests package for Python * Writing Unit Tests for the "test" package * Running tests using the command-line interface * "test.support" — Utilities for the Python test suite * "test.support.socket_helper" — Utilities for socket tests * "test.support.script_helper" — Utilities for the Python execution tests * "test.support.bytecode_helper" — Support tools for testing correct bytecode generation * "test.support.threading_helper" — Utilities for threading tests * "test.support.os_helper" — Utilities for os tests * "test.support.import_helper" — Utilities for import tests * "test.support.warnings_helper" — Utilities for warnings tests Python Development Mode *********************** Added in version 3.7. The Python Development Mode introduces additional runtime checks that are too expensive to be enabled by default. It should not be more verbose than the default if the code is correct; new warnings are only emitted when an issue is detected. It can be enabled using the "-X dev" command line option or by setting the "PYTHONDEVMODE" environment variable to "1". See also Python debug build. Effects of the Python Development Mode ====================================== Enabling the Python Development Mode is similar to the following command, but with additional effects described below: PYTHONMALLOC=debug PYTHONASYNCIODEBUG=1 python -W default -X faulthandler Effects of the Python Development Mode: * Add "default" warning filter. The following warnings are shown: * "DeprecationWarning" * "ImportWarning" * "PendingDeprecationWarning" * "ResourceWarning" Normally, the above warnings are filtered by the default warning filters. It behaves as if the "-W default" command line option is used. Use the "-W error" command line option or set the "PYTHONWARNINGS" environment variable to "error" to treat warnings as errors. * Install debug hooks on memory allocators to check for: * Buffer underflow * Buffer overflow * Memory allocator API violation * Unsafe usage of the GIL See the "PyMem_SetupDebugHooks()" C function. It behaves as if the "PYTHONMALLOC" environment variable is set to "debug". To enable the Python Development Mode without installing debug hooks on memory allocators, set the "PYTHONMALLOC" environment variable to "default". * Call "faulthandler.enable()" at Python startup to install handlers for the "SIGSEGV", "SIGFPE", "SIGABRT", "SIGBUS" and "SIGILL" signals to dump the Python traceback on a crash. It behaves as if the "-X faulthandler" command line option is used or if the "PYTHONFAULTHANDLER" environment variable is set to "1". * Enable asyncio debug mode. For example, "asyncio" checks for coroutines that were not awaited and logs them. It behaves as if the "PYTHONASYNCIODEBUG" environment variable is set to "1". * Check the *encoding* and *errors* arguments for string encoding and decoding operations. Examples: "open()", "str.encode()" and "bytes.decode()". By default, for best performance, the *errors* argument is only checked at the first encoding/decoding error and the *encoding* argument is sometimes ignored for empty strings. * The "io.IOBase" destructor logs "close()" exceptions. * Set the "dev_mode" attribute of "sys.flags" to "True". The Python Development Mode does not enable the "tracemalloc" module by default, because the overhead cost (to performance and memory) would be too large. Enabling the "tracemalloc" module provides additional information on the origin of some errors. For example, "ResourceWarning" logs the traceback where the resource was allocated, and a buffer overflow error logs the traceback where the memory block was allocated. The Python Development Mode does not prevent the "-O" command line option from removing "assert" statements nor from setting "__debug__" to "False". The Python Development Mode can only be enabled at the Python startup. Its value can be read from "sys.flags.dev_mode". Changed in version 3.8: The "io.IOBase" destructor now logs "close()" exceptions. Changed in version 3.9: The *encoding* and *errors* arguments are now checked for string encoding and decoding operations. ResourceWarning Example ======================= Example of a script counting the number of lines of the text file specified in the command line: import sys def main(): fp = open(sys.argv[1]) nlines = len(fp.readlines()) print(nlines) # The file is closed implicitly if __name__ == "__main__": main() The script does not close the file explicitly. By default, Python does not emit any warning. Example using README.txt, which has 269 lines: $ python script.py README.txt 269 Enabling the Python Development Mode displays a "ResourceWarning" warning: $ python -X dev script.py README.txt 269 script.py:10: ResourceWarning: unclosed file <_io.TextIOWrapper name='README.rst' mode='r' encoding='UTF-8'> main() ResourceWarning: Enable tracemalloc to get the object allocation traceback In addition, enabling "tracemalloc" shows the line where the file was opened: $ python -X dev -X tracemalloc=5 script.py README.rst 269 script.py:10: ResourceWarning: unclosed file <_io.TextIOWrapper name='README.rst' mode='r' encoding='UTF-8'> main() Object allocated at (most recent call last): File "script.py", lineno 10 main() File "script.py", lineno 4 fp = open(sys.argv[1]) The fix is to close explicitly the file. Example using a context manager: def main(): # Close the file explicitly when exiting the with block with open(sys.argv[1]) as fp: nlines = len(fp.readlines()) print(nlines) Not closing a resource explicitly can leave a resource open for way longer than expected; it can cause severe issues upon exiting Python. It is bad in CPython, but it is even worse in PyPy. Closing resources explicitly makes an application more deterministic and more reliable. Bad file descriptor error example ================================= Script displaying the first line of itself: import os def main(): fp = open(__file__) firstline = fp.readline() print(firstline.rstrip()) os.close(fp.fileno()) # The file is closed implicitly main() By default, Python does not emit any warning: $ python script.py import os The Python Development Mode shows a "ResourceWarning" and logs a “Bad file descriptor” error when finalizing the file object: $ python -X dev script.py import os script.py:10: ResourceWarning: unclosed file <_io.TextIOWrapper name='script.py' mode='r' encoding='UTF-8'> main() ResourceWarning: Enable tracemalloc to get the object allocation traceback Exception ignored in: <_io.TextIOWrapper name='script.py' mode='r' encoding='UTF-8'> Traceback (most recent call last): File "script.py", line 10, in main() OSError: [Errno 9] Bad file descriptor "os.close(fp.fileno())" closes the file descriptor. When the file object finalizer tries to close the file descriptor again, it fails with the "Bad file descriptor" error. A file descriptor must be closed only once. In the worst case scenario, closing it twice can lead to a crash (see bpo-18748 for an example). The fix is to remove the "os.close(fp.fileno())" line, or open the file with "closefd=False". Tkinter Dialogs *************** "tkinter.simpledialog" — Standard Tkinter input dialogs ======================================================= **Source code:** Lib/tkinter/simpledialog.py ====================================================================== The "tkinter.simpledialog" module contains convenience classes and functions for creating simple modal dialogs to get a value from the user. tkinter.simpledialog.askfloat(title, prompt, **kw) tkinter.simpledialog.askinteger(title, prompt, **kw) tkinter.simpledialog.askstring(title, prompt, **kw) The above three functions provide dialogs that prompt the user to enter a value of the desired type. class tkinter.simpledialog.Dialog(parent, title=None) The base class for custom dialogs. body(master) Override to construct the dialog’s interface and return the widget that should have initial focus. buttonbox() Default behaviour adds OK and Cancel buttons. Override for custom button layouts. "tkinter.filedialog" — File selection dialogs ============================================= **Source code:** Lib/tkinter/filedialog.py ====================================================================== The "tkinter.filedialog" module provides classes and factory functions for creating file/directory selection windows. Native Load/Save Dialogs ------------------------ The following classes and functions provide file dialog windows that combine a native look-and-feel with configuration options to customize behaviour. The following keyword arguments are applicable to the classes and functions listed below: *parent* - the window to place the dialog on top of *title* - the title of the window *initialdir* - the directory that the dialog starts in *initialfile* - the file selected upon opening of the dialog *filetypes* - a sequence of (label, pattern) tuples, ‘*’ wildcard is allowed *defaultextension* - default extension to append to file (save dialogs) *multiple* - when true, selection of multiple items is allowed **Static factory functions** The below functions when called create a modal, native look-and-feel dialog, wait for the user’s selection, then return the selected value(s) or "None" to the caller. tkinter.filedialog.askopenfile(mode='r', **options) tkinter.filedialog.askopenfiles(mode='r', **options) The above two functions create an "Open" dialog and return the opened file object(s) in read-only mode. tkinter.filedialog.asksaveasfile(mode='w', **options) Create a "SaveAs" dialog and return a file object opened in write- only mode. tkinter.filedialog.askopenfilename(**options) tkinter.filedialog.askopenfilenames(**options) The above two functions create an "Open" dialog and return the selected filename(s) that correspond to existing file(s). tkinter.filedialog.asksaveasfilename(**options) Create a "SaveAs" dialog and return the selected filename. tkinter.filedialog.askdirectory(**options) Prompt user to select a directory. Additional keyword option: *mustexist* - determines if selection must be an existing directory. class tkinter.filedialog.Open(master=None, **options) class tkinter.filedialog.SaveAs(master=None, **options) The above two classes provide native dialog windows for saving and loading files. **Convenience classes** The below classes are used for creating file/directory windows from scratch. These do not emulate the native look-and-feel of the platform. class tkinter.filedialog.Directory(master=None, **options) Create a dialog prompting the user to select a directory. Note: The *FileDialog* class should be subclassed for custom event handling and behaviour. class tkinter.filedialog.FileDialog(master, title=None) Create a basic file selection dialog. cancel_command(event=None) Trigger the termination of the dialog window. dirs_double_event(event) Event handler for double-click event on directory. dirs_select_event(event) Event handler for click event on directory. files_double_event(event) Event handler for double-click event on file. files_select_event(event) Event handler for single-click event on file. filter_command(event=None) Filter the files by directory. get_filter() Retrieve the file filter currently in use. get_selection() Retrieve the currently selected item. go(dir_or_file=os.curdir, pattern='*', default='', key=None) Render dialog and start event loop. ok_event(event) Exit dialog returning current selection. quit(how=None) Exit dialog returning filename, if any. set_filter(dir, pat) Set the file filter. set_selection(file) Update the current file selection to *file*. class tkinter.filedialog.LoadFileDialog(master, title=None) A subclass of FileDialog that creates a dialog window for selecting an existing file. ok_command() Test that a file is provided and that the selection indicates an already existing file. class tkinter.filedialog.SaveFileDialog(master, title=None) A subclass of FileDialog that creates a dialog window for selecting a destination file. ok_command() Test whether or not the selection points to a valid file that is not a directory. Confirmation is required if an already existing file is selected. "tkinter.commondialog" — Dialog window templates ================================================ **Source code:** Lib/tkinter/commondialog.py ====================================================================== The "tkinter.commondialog" module provides the "Dialog" class that is the base class for dialogs defined in other supporting modules. class tkinter.commondialog.Dialog(master=None, **options) show(**options) Render the Dialog window. See also: Modules "tkinter.messagebox", Reading and Writing Files "difflib" — Helpers for computing deltas **************************************** **Source code:** Lib/difflib.py ====================================================================== This module provides classes and functions for comparing sequences. It can be used for example, for comparing files, and can produce information about file differences in various formats, including HTML and context and unified diffs. For comparing directories and files, see also, the "filecmp" module. class difflib.SequenceMatcher This is a flexible class for comparing pairs of sequences of any type, so long as the sequence elements are *hashable*. The basic algorithm predates, and is a little fancier than, an algorithm published in the late 1980’s by Ratcliff and Obershelp under the hyperbolic name “gestalt pattern matching.” The idea is to find the longest contiguous matching subsequence that contains no “junk” elements; these “junk” elements are ones that are uninteresting in some sense, such as blank lines or whitespace. (Handling junk is an extension to the Ratcliff and Obershelp algorithm.) The same idea is then applied recursively to the pieces of the sequences to the left and to the right of the matching subsequence. This does not yield minimal edit sequences, but does tend to yield matches that “look right” to people. **Timing:** The basic Ratcliff-Obershelp algorithm is cubic time in the worst case and quadratic time in the expected case. "SequenceMatcher" is quadratic time for the worst case and has expected-case behavior dependent in a complicated way on how many elements the sequences have in common; best case time is linear. **Automatic junk heuristic:** "SequenceMatcher" supports a heuristic that automatically treats certain sequence items as junk. The heuristic counts how many times each individual item appears in the sequence. If an item’s duplicates (after the first one) account for more than 1% of the sequence and the sequence is at least 200 items long, this item is marked as “popular” and is treated as junk for the purpose of sequence matching. This heuristic can be turned off by setting the "autojunk" argument to "False" when creating the "SequenceMatcher". Changed in version 3.2: Added the *autojunk* parameter. class difflib.Differ This is a class for comparing sequences of lines of text, and producing human-readable differences or deltas. Differ uses "SequenceMatcher" both to compare sequences of lines, and to compare sequences of characters within similar (near-matching) lines. Each line of a "Differ" delta begins with a two-letter code: +------------+---------------------------------------------+ | Code | Meaning | |============|=============================================| | "'- '" | line unique to sequence 1 | +------------+---------------------------------------------+ | "'+ '" | line unique to sequence 2 | +------------+---------------------------------------------+ | "' '" | line common to both sequences | +------------+---------------------------------------------+ | "'? '" | line not present in either input sequence | +------------+---------------------------------------------+ Lines beginning with ‘"?"’ attempt to guide the eye to intraline differences, and were not present in either input sequence. These lines can be confusing if the sequences contain whitespace characters, such as spaces, tabs or line breaks. class difflib.HtmlDiff This class can be used to create an HTML table (or a complete HTML file containing the table) showing a side by side, line by line comparison of text with inter-line and intra-line change highlights. The table can be generated in either full or contextual difference mode. The constructor for this class is: __init__(tabsize=8, wrapcolumn=None, linejunk=None, charjunk=IS_CHARACTER_JUNK) Initializes instance of "HtmlDiff". *tabsize* is an optional keyword argument to specify tab stop spacing and defaults to "8". *wrapcolumn* is an optional keyword to specify column number where lines are broken and wrapped, defaults to "None" where lines are not wrapped. *linejunk* and *charjunk* are optional keyword arguments passed into "ndiff()" (used by "HtmlDiff" to generate the side by side HTML differences). See "ndiff()" documentation for argument default values and descriptions. The following methods are public: make_file(fromlines, tolines, fromdesc='', todesc='', context=False, numlines=5, *, charset='utf-8') Compares *fromlines* and *tolines* (lists of strings) and returns a string which is a complete HTML file containing a table showing line by line differences with inter-line and intra-line changes highlighted. *fromdesc* and *todesc* are optional keyword arguments to specify from/to file column header strings (both default to an empty string). *context* and *numlines* are both optional keyword arguments. Set *context* to "True" when contextual differences are to be shown, else the default is "False" to show the full files. *numlines* defaults to "5". When *context* is "True" *numlines* controls the number of context lines which surround the difference highlights. When *context* is "False" *numlines* controls the number of lines which are shown before a difference highlight when using the “next” hyperlinks (setting to zero would cause the “next” hyperlinks to place the next difference highlight at the top of the browser without any leading context). Note: *fromdesc* and *todesc* are interpreted as unescaped HTML and should be properly escaped while receiving input from untrusted sources. Changed in version 3.5: *charset* keyword-only argument was added. The default charset of HTML document changed from "'ISO-8859-1'" to "'utf-8'". make_table(fromlines, tolines, fromdesc='', todesc='', context=False, numlines=5) Compares *fromlines* and *tolines* (lists of strings) and returns a string which is a complete HTML table showing line by line differences with inter-line and intra-line changes highlighted. The arguments for this method are the same as those for the "make_file()" method. difflib.context_diff(a, b, fromfile='', tofile='', fromfiledate='', tofiledate='', n=3, lineterm='\n') Compare *a* and *b* (lists of strings); return a delta (a *generator* generating the delta lines) in context diff format. Context diffs are a compact way of showing just the lines that have changed plus a few lines of context. The changes are shown in a before/after style. The number of context lines is set by *n* which defaults to three. By default, the diff control lines (those with "***" or "---") are created with a trailing newline. This is helpful so that inputs created from "io.IOBase.readlines()" result in diffs that are suitable for use with "io.IOBase.writelines()" since both the inputs and outputs have trailing newlines. For inputs that do not have trailing newlines, set the *lineterm* argument to """" so that the output will be uniformly newline free. The context diff format normally has a header for filenames and modification times. Any or all of these may be specified using strings for *fromfile*, *tofile*, *fromfiledate*, and *tofiledate*. The modification times are normally expressed in the ISO 8601 format. If not specified, the strings default to blanks. >>> import sys >>> from difflib import * >>> s1 = ['bacon\n', 'eggs\n', 'ham\n', 'guido\n'] >>> s2 = ['python\n', 'eggy\n', 'hamster\n', 'guido\n'] >>> sys.stdout.writelines(context_diff(s1, s2, fromfile='before.py', ... tofile='after.py')) *** before.py --- after.py *************** *** 1,4 **** ! bacon ! eggs ! ham guido --- 1,4 ---- ! python ! eggy ! hamster guido See A command-line interface to difflib for a more detailed example. difflib.get_close_matches(word, possibilities, n=3, cutoff=0.6) Return a list of the best “good enough” matches. *word* is a sequence for which close matches are desired (typically a string), and *possibilities* is a list of sequences against which to match *word* (typically a list of strings). Optional argument *n* (default "3") is the maximum number of close matches to return; *n* must be greater than "0". Optional argument *cutoff* (default "0.6") is a float in the range [0, 1]. Possibilities that don’t score at least that similar to *word* are ignored. The best (no more than *n*) matches among the possibilities are returned in a list, sorted by similarity score, most similar first. >>> get_close_matches('appel', ['ape', 'apple', 'peach', 'puppy']) ['apple', 'ape'] >>> import keyword >>> get_close_matches('wheel', keyword.kwlist) ['while'] >>> get_close_matches('pineapple', keyword.kwlist) [] >>> get_close_matches('accept', keyword.kwlist) ['except'] difflib.ndiff(a, b, linejunk=None, charjunk=IS_CHARACTER_JUNK) Compare *a* and *b* (lists of strings); return a "Differ"-style delta (a *generator* generating the delta lines). Optional keyword parameters *linejunk* and *charjunk* are filtering functions (or "None"): *linejunk*: A function that accepts a single string argument, and returns true if the string is junk, or false if not. The default is "None". There is also a module-level function "IS_LINE_JUNK()", which filters out lines without visible characters, except for at most one pound character ("'#'") – however the underlying "SequenceMatcher" class does a dynamic analysis of which lines are so frequent as to constitute noise, and this usually works better than using this function. *charjunk*: A function that accepts a character (a string of length 1), and returns if the character is junk, or false if not. The default is module-level function "IS_CHARACTER_JUNK()", which filters out whitespace characters (a blank or tab; it’s a bad idea to include newline in this!). >>> diff = ndiff('one\ntwo\nthree\n'.splitlines(keepends=True), ... 'ore\ntree\nemu\n'.splitlines(keepends=True)) >>> print(''.join(diff), end="") - one ? ^ + ore ? ^ - two - three ? - + tree + emu difflib.restore(sequence, which) Return one of the two sequences that generated a delta. Given a *sequence* produced by "Differ.compare()" or "ndiff()", extract lines originating from file 1 or 2 (parameter *which*), stripping off line prefixes. Example: >>> diff = ndiff('one\ntwo\nthree\n'.splitlines(keepends=True), ... 'ore\ntree\nemu\n'.splitlines(keepends=True)) >>> diff = list(diff) # materialize the generated delta into a list >>> print(''.join(restore(diff, 1)), end="") one two three >>> print(''.join(restore(diff, 2)), end="") ore tree emu difflib.unified_diff(a, b, fromfile='', tofile='', fromfiledate='', tofiledate='', n=3, lineterm='\n') Compare *a* and *b* (lists of strings); return a delta (a *generator* generating the delta lines) in unified diff format. Unified diffs are a compact way of showing just the lines that have changed plus a few lines of context. The changes are shown in an inline style (instead of separate before/after blocks). The number of context lines is set by *n* which defaults to three. By default, the diff control lines (those with "---", "+++", or "@@") are created with a trailing newline. This is helpful so that inputs created from "io.IOBase.readlines()" result in diffs that are suitable for use with "io.IOBase.writelines()" since both the inputs and outputs have trailing newlines. For inputs that do not have trailing newlines, set the *lineterm* argument to """" so that the output will be uniformly newline free. The unified diff format normally has a header for filenames and modification times. Any or all of these may be specified using strings for *fromfile*, *tofile*, *fromfiledate*, and *tofiledate*. The modification times are normally expressed in the ISO 8601 format. If not specified, the strings default to blanks. >>> s1 = ['bacon\n', 'eggs\n', 'ham\n', 'guido\n'] >>> s2 = ['python\n', 'eggy\n', 'hamster\n', 'guido\n'] >>> sys.stdout.writelines(unified_diff(s1, s2, fromfile='before.py', tofile='after.py')) --- before.py +++ after.py @@ -1,4 +1,4 @@ -bacon -eggs -ham +python +eggy +hamster guido See A command-line interface to difflib for a more detailed example. difflib.diff_bytes(dfunc, a, b, fromfile=b'', tofile=b'', fromfiledate=b'', tofiledate=b'', n=3, lineterm=b'\n') Compare *a* and *b* (lists of bytes objects) using *dfunc*; yield a sequence of delta lines (also bytes) in the format returned by *dfunc*. *dfunc* must be a callable, typically either "unified_diff()" or "context_diff()". Allows you to compare data with unknown or inconsistent encoding. All inputs except *n* must be bytes objects, not str. Works by losslessly converting all inputs (except *n*) to str, and calling "dfunc(a, b, fromfile, tofile, fromfiledate, tofiledate, n, lineterm)". The output of *dfunc* is then converted back to bytes, so the delta lines that you receive have the same unknown/inconsistent encodings as *a* and *b*. Added in version 3.5. difflib.IS_LINE_JUNK(line) Return "True" for ignorable lines. The line *line* is ignorable if *line* is blank or contains a single "'#'", otherwise it is not ignorable. Used as a default for parameter *linejunk* in "ndiff()" in older versions. difflib.IS_CHARACTER_JUNK(ch) Return "True" for ignorable characters. The character *ch* is ignorable if *ch* is a space or tab, otherwise it is not ignorable. Used as a default for parameter *charjunk* in "ndiff()". See also: Pattern Matching: The Gestalt Approach Discussion of a similar algorithm by John W. Ratcliff and D. E. Metzener. This was published in Dr. Dobb’s Journal in July, 1988. SequenceMatcher Objects ======================= The "SequenceMatcher" class has this constructor: class difflib.SequenceMatcher(isjunk=None, a='', b='', autojunk=True) Optional argument *isjunk* must be "None" (the default) or a one- argument function that takes a sequence element and returns true if and only if the element is “junk” and should be ignored. Passing "None" for *isjunk* is equivalent to passing "lambda x: False"; in other words, no elements are ignored. For example, pass: lambda x: x in " \t" if you’re comparing lines as sequences of characters, and don’t want to synch up on blanks or hard tabs. The optional arguments *a* and *b* are sequences to be compared; both default to empty strings. The elements of both sequences must be *hashable*. The optional argument *autojunk* can be used to disable the automatic junk heuristic. Changed in version 3.2: Added the *autojunk* parameter. SequenceMatcher objects get three data attributes: *bjunk* is the set of elements of *b* for which *isjunk* is "True"; *bpopular* is the set of non-junk elements considered popular by the heuristic (if it is not disabled); *b2j* is a dict mapping the remaining elements of *b* to a list of positions where they occur. All three are reset whenever *b* is reset with "set_seqs()" or "set_seq2()". Added in version 3.2: The *bjunk* and *bpopular* attributes. "SequenceMatcher" objects have the following methods: set_seqs(a, b) Set the two sequences to be compared. "SequenceMatcher" computes and caches detailed information about the second sequence, so if you want to compare one sequence against many sequences, use "set_seq2()" to set the commonly used sequence once and call "set_seq1()" repeatedly, once for each of the other sequences. set_seq1(a) Set the first sequence to be compared. The second sequence to be compared is not changed. set_seq2(b) Set the second sequence to be compared. The first sequence to be compared is not changed. find_longest_match(alo=0, ahi=None, blo=0, bhi=None) Find longest matching block in "a[alo:ahi]" and "b[blo:bhi]". If *isjunk* was omitted or "None", "find_longest_match()" returns "(i, j, k)" such that "a[i:i+k]" is equal to "b[j:j+k]", where "alo <= i <= i+k <= ahi" and "blo <= j <= j+k <= bhi". For all "(i', j', k')" meeting those conditions, the additional conditions "k >= k'", "i <= i'", and if "i == i'", "j <= j'" are also met. In other words, of all maximal matching blocks, return one that starts earliest in *a*, and of all those maximal matching blocks that start earliest in *a*, return the one that starts earliest in *b*. >>> s = SequenceMatcher(None, " abcd", "abcd abcd") >>> s.find_longest_match(0, 5, 0, 9) Match(a=0, b=4, size=5) If *isjunk* was provided, first the longest matching block is determined as above, but with the additional restriction that no junk element appears in the block. Then that block is extended as far as possible by matching (only) junk elements on both sides. So the resulting block never matches on junk except as identical junk happens to be adjacent to an interesting match. Here’s the same example as before, but considering blanks to be junk. That prevents "' abcd'" from matching the "' abcd'" at the tail end of the second sequence directly. Instead only the "'abcd'" can match, and matches the leftmost "'abcd'" in the second sequence: >>> s = SequenceMatcher(lambda x: x==" ", " abcd", "abcd abcd") >>> s.find_longest_match(0, 5, 0, 9) Match(a=1, b=0, size=4) If no blocks match, this returns "(alo, blo, 0)". This method returns a *named tuple* "Match(a, b, size)". Changed in version 3.9: Added default arguments. get_matching_blocks() Return list of triples describing non-overlapping matching subsequences. Each triple is of the form "(i, j, n)", and means that "a[i:i+n] == b[j:j+n]". The triples are monotonically increasing in *i* and *j*. The last triple is a dummy, and has the value "(len(a), len(b), 0)". It is the only triple with "n == 0". If "(i, j, n)" and "(i', j', n')" are adjacent triples in the list, and the second is not the last triple in the list, then "i+n < i'" or "j+n < j'"; in other words, adjacent triples always describe non- adjacent equal blocks. >>> s = SequenceMatcher(None, "abxcd", "abcd") >>> s.get_matching_blocks() [Match(a=0, b=0, size=2), Match(a=3, b=2, size=2), Match(a=5, b=4, size=0)] get_opcodes() Return list of 5-tuples describing how to turn *a* into *b*. Each tuple is of the form "(tag, i1, i2, j1, j2)". The first tuple has "i1 == j1 == 0", and remaining tuples have *i1* equal to the *i2* from the preceding tuple, and, likewise, *j1* equal to the previous *j2*. The *tag* values are strings, with these meanings: +-----------------+-----------------------------------------------+ | Value | Meaning | |=================|===============================================| | "'replace'" | "a[i1:i2]" should be replaced by "b[j1:j2]". | +-----------------+-----------------------------------------------+ | "'delete'" | "a[i1:i2]" should be deleted. Note that "j1 | | | == j2" in this case. | +-----------------+-----------------------------------------------+ | "'insert'" | "b[j1:j2]" should be inserted at "a[i1:i1]". | | | Note that "i1 == i2" in this case. | +-----------------+-----------------------------------------------+ | "'equal'" | "a[i1:i2] == b[j1:j2]" (the sub-sequences are | | | equal). | +-----------------+-----------------------------------------------+ For example: >>> a = "qabxcd" >>> b = "abycdf" >>> s = SequenceMatcher(None, a, b) >>> for tag, i1, i2, j1, j2 in s.get_opcodes(): ... print('{:7} a[{}:{}] --> b[{}:{}] {!r:>8} --> {!r}'.format( ... tag, i1, i2, j1, j2, a[i1:i2], b[j1:j2])) delete a[0:1] --> b[0:0] 'q' --> '' equal a[1:3] --> b[0:2] 'ab' --> 'ab' replace a[3:4] --> b[2:3] 'x' --> 'y' equal a[4:6] --> b[3:5] 'cd' --> 'cd' insert a[6:6] --> b[5:6] '' --> 'f' get_grouped_opcodes(n=3) Return a *generator* of groups with up to *n* lines of context. Starting with the groups returned by "get_opcodes()", this method splits out smaller change clusters and eliminates intervening ranges which have no changes. The groups are returned in the same format as "get_opcodes()". ratio() Return a measure of the sequences’ similarity as a float in the range [0, 1]. Where T is the total number of elements in both sequences, and M is the number of matches, this is 2.0*M / T. Note that this is "1.0" if the sequences are identical, and "0.0" if they have nothing in common. This is expensive to compute if "get_matching_blocks()" or "get_opcodes()" hasn’t already been called, in which case you may want to try "quick_ratio()" or "real_quick_ratio()" first to get an upper bound. Note: Caution: The result of a "ratio()" call may depend on the order of the arguments. For instance: >>> SequenceMatcher(None, 'tide', 'diet').ratio() 0.25 >>> SequenceMatcher(None, 'diet', 'tide').ratio() 0.5 quick_ratio() Return an upper bound on "ratio()" relatively quickly. real_quick_ratio() Return an upper bound on "ratio()" very quickly. The three methods that return the ratio of matching to total characters can give different results due to differing levels of approximation, although "quick_ratio()" and "real_quick_ratio()" are always at least as large as "ratio()": >>> s = SequenceMatcher(None, "abcd", "bcde") >>> s.ratio() 0.75 >>> s.quick_ratio() 0.75 >>> s.real_quick_ratio() 1.0 SequenceMatcher Examples ======================== This example compares two strings, considering blanks to be “junk”: >>> s = SequenceMatcher(lambda x: x == " ", ... "private Thread currentThread;", ... "private volatile Thread currentThread;") "ratio()" returns a float in [0, 1], measuring the similarity of the sequences. As a rule of thumb, a "ratio()" value over 0.6 means the sequences are close matches: >>> print(round(s.ratio(), 3)) 0.866 If you’re only interested in where the sequences match, "get_matching_blocks()" is handy: >>> for block in s.get_matching_blocks(): ... print("a[%d] and b[%d] match for %d elements" % block) a[0] and b[0] match for 8 elements a[8] and b[17] match for 21 elements a[29] and b[38] match for 0 elements Note that the last tuple returned by "get_matching_blocks()" is always a dummy, "(len(a), len(b), 0)", and this is the only case in which the last tuple element (number of elements matched) is "0". If you want to know how to change the first sequence into the second, use "get_opcodes()": >>> for opcode in s.get_opcodes(): ... print("%6s a[%d:%d] b[%d:%d]" % opcode) equal a[0:8] b[0:8] insert a[8:8] b[8:17] equal a[8:29] b[17:38] See also: * The "get_close_matches()" function in this module which shows how simple code building on "SequenceMatcher" can be used to do useful work. * Simple version control recipe for a small application built with "SequenceMatcher". Differ Objects ============== Note that "Differ"-generated deltas make no claim to be **minimal** diffs. To the contrary, minimal diffs are often counter-intuitive, because they synch up anywhere possible, sometimes accidental matches 100 pages apart. Restricting synch points to contiguous matches preserves some notion of locality, at the occasional cost of producing a longer diff. The "Differ" class has this constructor: class difflib.Differ(linejunk=None, charjunk=None) Optional keyword parameters *linejunk* and *charjunk* are for filter functions (or "None"): *linejunk*: A function that accepts a single string argument, and returns true if the string is junk. The default is "None", meaning that no line is considered junk. *charjunk*: A function that accepts a single character argument (a string of length 1), and returns true if the character is junk. The default is "None", meaning that no character is considered junk. These junk-filtering functions speed up matching to find differences and do not cause any differing lines or characters to be ignored. Read the description of the "find_longest_match()" method’s *isjunk* parameter for an explanation. "Differ" objects are used (deltas generated) via a single method: compare(a, b) Compare two sequences of lines, and generate the delta (a sequence of lines). Each sequence must contain individual single-line strings ending with newlines. Such sequences can be obtained from the "readlines()" method of file-like objects. The delta generated also consists of newline-terminated strings, ready to be printed as-is via the "writelines()" method of a file-like object. Differ Example ============== This example compares two texts. First we set up the texts, sequences of individual single-line strings ending with newlines (such sequences can also be obtained from the "readlines()" method of file-like objects): >>> text1 = ''' 1. Beautiful is better than ugly. ... 2. Explicit is better than implicit. ... 3. Simple is better than complex. ... 4. Complex is better than complicated. ... '''.splitlines(keepends=True) >>> len(text1) 4 >>> text1[0][-1] '\n' >>> text2 = ''' 1. Beautiful is better than ugly. ... 3. Simple is better than complex. ... 4. Complicated is better than complex. ... 5. Flat is better than nested. ... '''.splitlines(keepends=True) Next we instantiate a Differ object: >>> d = Differ() Note that when instantiating a "Differ" object we may pass functions to filter out line and character “junk.” See the "Differ()" constructor for details. Finally, we compare the two: >>> result = list(d.compare(text1, text2)) "result" is a list of strings, so let’s pretty-print it: >>> from pprint import pprint >>> pprint(result) [' 1. Beautiful is better than ugly.\n', '- 2. Explicit is better than implicit.\n', '- 3. Simple is better than complex.\n', '+ 3. Simple is better than complex.\n', '? ++\n', '- 4. Complex is better than complicated.\n', '? ^ ---- ^\n', '+ 4. Complicated is better than complex.\n', '? ++++ ^ ^\n', '+ 5. Flat is better than nested.\n'] As a single multi-line string it looks like this: >>> import sys >>> sys.stdout.writelines(result) 1. Beautiful is better than ugly. - 2. Explicit is better than implicit. - 3. Simple is better than complex. + 3. Simple is better than complex. ? ++ - 4. Complex is better than complicated. ? ^ ---- ^ + 4. Complicated is better than complex. ? ++++ ^ ^ + 5. Flat is better than nested. A command-line interface to difflib =================================== This example shows how to use difflib to create a "diff"-like utility. """ Command line interface to difflib.py providing diffs in four formats: * ndiff: lists every line and highlights interline changes. * context: highlights clusters of changes in a before/after format. * unified: highlights clusters of changes in an inline format. * html: generates side by side comparison with change highlights. """ import sys, os, difflib, argparse from datetime import datetime, timezone def file_mtime(path): t = datetime.fromtimestamp(os.stat(path).st_mtime, timezone.utc) return t.astimezone().isoformat() def main(): parser = argparse.ArgumentParser() parser.add_argument('-c', action='store_true', default=False, help='Produce a context format diff (default)') parser.add_argument('-u', action='store_true', default=False, help='Produce a unified format diff') parser.add_argument('-m', action='store_true', default=False, help='Produce HTML side by side diff ' '(can use -c and -l in conjunction)') parser.add_argument('-n', action='store_true', default=False, help='Produce a ndiff format diff') parser.add_argument('-l', '--lines', type=int, default=3, help='Set number of context lines (default 3)') parser.add_argument('fromfile') parser.add_argument('tofile') options = parser.parse_args() n = options.lines fromfile = options.fromfile tofile = options.tofile fromdate = file_mtime(fromfile) todate = file_mtime(tofile) with open(fromfile) as ff: fromlines = ff.readlines() with open(tofile) as tf: tolines = tf.readlines() if options.u: diff = difflib.unified_diff(fromlines, tolines, fromfile, tofile, fromdate, todate, n=n) elif options.n: diff = difflib.ndiff(fromlines, tolines) elif options.m: diff = difflib.HtmlDiff().make_file(fromlines,tolines,fromfile,tofile,context=options.c,numlines=n) else: diff = difflib.context_diff(fromlines, tolines, fromfile, tofile, fromdate, todate, n=n) sys.stdout.writelines(diff) if __name__ == '__main__': main() ndiff example ============= This example shows how to use "difflib.ndiff()". """ndiff [-q] file1 file2 or ndiff (-r1 | -r2) < ndiff_output > file1_or_file2 Print a human-friendly file difference report to stdout. Both inter- and intra-line differences are noted. In the second form, recreate file1 (-r1) or file2 (-r2) on stdout, from an ndiff report on stdin. In the first form, if -q ("quiet") is not specified, the first two lines of output are -: file1 +: file2 Each remaining line begins with a two-letter code: "- " line unique to file1 "+ " line unique to file2 " " line common to both files "? " line not present in either input file Lines beginning with "? " attempt to guide the eye to intraline differences, and were not present in either input file. These lines can be confusing if the source files contain tab characters. The first file can be recovered by retaining only lines that begin with " " or "- ", and deleting those 2-character prefixes; use ndiff with -r1. The second file can be recovered similarly, but by retaining only " " and "+ " lines; use ndiff with -r2; or, on Unix, the second file can be recovered by piping the output through sed -n '/^[+ ] /s/^..//p' """ __version__ = 1, 7, 0 import difflib, sys def fail(msg): out = sys.stderr.write out(msg + "\n\n") out(__doc__) return 0 # open a file & return the file object; gripe and return 0 if it # couldn't be opened def fopen(fname): try: return open(fname) except IOError as detail: return fail("couldn't open " + fname + ": " + str(detail)) # open two files & spray the diff to stdout; return false iff a problem def fcompare(f1name, f2name): f1 = fopen(f1name) f2 = fopen(f2name) if not f1 or not f2: return 0 a = f1.readlines(); f1.close() b = f2.readlines(); f2.close() for line in difflib.ndiff(a, b): print(line, end=' ') return 1 # crack args (sys.argv[1:] is normal) & compare; # return false iff a problem def main(args): import getopt try: opts, args = getopt.getopt(args, "qr:") except getopt.error as detail: return fail(str(detail)) noisy = 1 qseen = rseen = 0 for opt, val in opts: if opt == "-q": qseen = 1 noisy = 0 elif opt == "-r": rseen = 1 whichfile = val if qseen and rseen: return fail("can't specify both -q and -r") if rseen: if args: return fail("no args allowed with -r option") if whichfile in ("1", "2"): restore(whichfile) return 1 return fail("-r value must be 1 or 2") if len(args) != 2: return fail("need 2 filename args") f1name, f2name = args if noisy: print('-:', f1name) print('+:', f2name) return fcompare(f1name, f2name) # read ndiff output from stdin, and print file1 (which=='1') or # file2 (which=='2') to stdout def restore(which): restored = difflib.restore(sys.stdin.readlines(), which) sys.stdout.writelines(restored) if __name__ == '__main__': main(sys.argv[1:]) "dis" — Disassembler for Python bytecode **************************************** **Source code:** Lib/dis.py ====================================================================== The "dis" module supports the analysis of CPython *bytecode* by disassembling it. The CPython bytecode which this module takes as an input is defined in the file "Include/opcode.h" and used by the compiler and the interpreter. **CPython implementation detail:** Bytecode is an implementation detail of the CPython interpreter. No guarantees are made that bytecode will not be added, removed, or changed between versions of Python. Use of this module should not be considered to work across Python VMs or Python releases. Changed in version 3.6: Use 2 bytes for each instruction. Previously the number of bytes varied by instruction. Changed in version 3.10: The argument of jump, exception handling and loop instructions is now the instruction offset rather than the byte offset. Changed in version 3.11: Some instructions are accompanied by one or more inline cache entries, which take the form of "CACHE" instructions. These instructions are hidden by default, but can be shown by passing "show_caches=True" to any "dis" utility. Furthermore, the interpreter now adapts the bytecode to specialize it for different runtime conditions. The adaptive bytecode can be shown by passing "adaptive=True". Changed in version 3.12: The argument of a jump is the offset of the target instruction relative to the instruction that appears immediately after the jump instruction’s "CACHE" entries.As a consequence, the presence of the "CACHE" instructions is transparent for forward jumps but needs to be taken into account when reasoning about backward jumps. Changed in version 3.13: The output shows logical labels rather than instruction offsets for jump targets and exception handlers. The "-O" command line option and the "show_offsets" argument were added. Example: Given the function "myfunc()": def myfunc(alist): return len(alist) the following command can be used to display the disassembly of "myfunc()": >>> dis.dis(myfunc) 2 RESUME 0 3 LOAD_GLOBAL 1 (len + NULL) LOAD_FAST 0 (alist) CALL 1 RETURN_VALUE (The “2” is a line number). Command-line interface ====================== The "dis" module can be invoked as a script from the command line: python -m dis [-h] [-C] [-O] [infile] The following options are accepted: -h, --help Display usage and exit. -C, --show-caches Show inline caches. Added in version 3.13. -O, --show-offsets Show offsets of instructions. Added in version 3.13. If "infile" is specified, its disassembled code will be written to stdout. Otherwise, disassembly is performed on compiled source code received from stdin. Bytecode analysis ================= Added in version 3.4. The bytecode analysis API allows pieces of Python code to be wrapped in a "Bytecode" object that provides easy access to details of the compiled code. class dis.Bytecode(x, *, first_line=None, current_offset=None, show_caches=False, adaptive=False, show_offsets=False) Analyse the bytecode corresponding to a function, generator, asynchronous generator, coroutine, method, string of source code, or a code object (as returned by "compile()"). This is a convenience wrapper around many of the functions listed below, most notably "get_instructions()", as iterating over a "Bytecode" instance yields the bytecode operations as "Instruction" instances. If *first_line* is not "None", it indicates the line number that should be reported for the first source line in the disassembled code. Otherwise, the source line information (if any) is taken directly from the disassembled code object. If *current_offset* is not "None", it refers to an instruction offset in the disassembled code. Setting this means "dis()" will display a “current instruction” marker against the specified opcode. If *show_caches* is "True", "dis()" will display inline cache entries used by the interpreter to specialize the bytecode. If *adaptive* is "True", "dis()" will display specialized bytecode that may be different from the original bytecode. If *show_offsets* is "True", "dis()" will include instruction offsets in the output. classmethod from_traceback(tb, *, show_caches=False) Construct a "Bytecode" instance from the given traceback, setting *current_offset* to the instruction responsible for the exception. codeobj The compiled code object. first_line The first source line of the code object (if available) dis() Return a formatted view of the bytecode operations (the same as printed by "dis.dis()", but returned as a multi-line string). info() Return a formatted multi-line string with detailed information about the code object, like "code_info()". Changed in version 3.7: This can now handle coroutine and asynchronous generator objects. Changed in version 3.11: Added the *show_caches* and *adaptive* parameters. Example: >>> bytecode = dis.Bytecode(myfunc) >>> for instr in bytecode: ... print(instr.opname) ... RESUME LOAD_GLOBAL LOAD_FAST CALL RETURN_VALUE Analysis functions ================== The "dis" module also defines the following analysis functions that convert the input directly to the desired output. They can be useful if only a single operation is being performed, so the intermediate analysis object isn’t useful: dis.code_info(x) Return a formatted multi-line string with detailed code object information for the supplied function, generator, asynchronous generator, coroutine, method, source code string or code object. Note that the exact contents of code info strings are highly implementation dependent and they may change arbitrarily across Python VMs or Python releases. Added in version 3.2. Changed in version 3.7: This can now handle coroutine and asynchronous generator objects. dis.show_code(x, *, file=None) Print detailed code object information for the supplied function, method, source code string or code object to *file* (or "sys.stdout" if *file* is not specified). This is a convenient shorthand for "print(code_info(x), file=file)", intended for interactive exploration at the interpreter prompt. Added in version 3.2. Changed in version 3.4: Added *file* parameter. dis.dis(x=None, *, file=None, depth=None, show_caches=False, adaptive=False) Disassemble the *x* object. *x* can denote either a module, a class, a method, a function, a generator, an asynchronous generator, a coroutine, a code object, a string of source code or a byte sequence of raw bytecode. For a module, it disassembles all functions. For a class, it disassembles all methods (including class and static methods). For a code object or sequence of raw bytecode, it prints one line per bytecode instruction. It also recursively disassembles nested code objects. These can include generator expressions, nested functions, the bodies of nested classes, and the code objects used for annotation scopes. Strings are first compiled to code objects with the "compile()" built-in function before being disassembled. If no object is provided, this function disassembles the last traceback. The disassembly is written as text to the supplied *file* argument if provided and to "sys.stdout" otherwise. The maximal depth of recursion is limited by *depth* unless it is "None". "depth=0" means no recursion. If *show_caches* is "True", this function will display inline cache entries used by the interpreter to specialize the bytecode. If *adaptive* is "True", this function will display specialized bytecode that may be different from the original bytecode. Changed in version 3.4: Added *file* parameter. Changed in version 3.7: Implemented recursive disassembling and added *depth* parameter. Changed in version 3.7: This can now handle coroutine and asynchronous generator objects. Changed in version 3.11: Added the *show_caches* and *adaptive* parameters. distb(tb=None, *, file=None, show_caches=False, adaptive=False, show_offset=False) Disassemble the top-of-stack function of a traceback, using the last traceback if none was passed. The instruction causing the exception is indicated. The disassembly is written as text to the supplied *file* argument if provided and to "sys.stdout" otherwise. Changed in version 3.4: Added *file* parameter. Changed in version 3.11: Added the *show_caches* and *adaptive* parameters. Changed in version 3.13: Added the *show_offsets* parameter. dis.disassemble(code, lasti=-1, *, file=None, show_caches=False, adaptive=False) disco(code, lasti=-1, *, file=None, show_caches=False, adaptive=False, show_offsets=False) Disassemble a code object, indicating the last instruction if *lasti* was provided. The output is divided in the following columns: 1. the line number, for the first instruction of each line 2. the current instruction, indicated as "-->", 3. a labelled instruction, indicated with ">>", 4. the address of the instruction, 5. the operation code name, 6. operation parameters, and 7. interpretation of the parameters in parentheses. The parameter interpretation recognizes local and global variable names, constant values, branch targets, and compare operators. The disassembly is written as text to the supplied *file* argument if provided and to "sys.stdout" otherwise. Changed in version 3.4: Added *file* parameter. Changed in version 3.11: Added the *show_caches* and *adaptive* parameters. Changed in version 3.13: Added the *show_offsets* parameter. dis.get_instructions(x, *, first_line=None, show_caches=False, adaptive=False) Return an iterator over the instructions in the supplied function, method, source code string or code object. The iterator generates a series of "Instruction" named tuples giving the details of each operation in the supplied code. If *first_line* is not "None", it indicates the line number that should be reported for the first source line in the disassembled code. Otherwise, the source line information (if any) is taken directly from the disassembled code object. The *adaptive* parameter works as it does in "dis()". Added in version 3.4. Changed in version 3.11: Added the *show_caches* and *adaptive* parameters. Changed in version 3.13: The *show_caches* parameter is deprecated and has no effect. The iterator generates the "Instruction" instances with the *cache_info* field populated (regardless of the value of *show_caches*) and it no longer generates separate items for the cache entries. dis.findlinestarts(code) This generator function uses the "co_lines()" method of the code object *code* to find the offsets which are starts of lines in the source code. They are generated as "(offset, lineno)" pairs. Changed in version 3.6: Line numbers can be decreasing. Before, they were always increasing. Changed in version 3.10: The **PEP 626** "co_lines()" method is used instead of the "co_firstlineno" and "co_lnotab" attributes of the code object. Changed in version 3.13: Line numbers can be "None" for bytecode that does not map to source lines. dis.findlabels(code) Detect all offsets in the raw compiled bytecode string *code* which are jump targets, and return a list of these offsets. dis.stack_effect(opcode, oparg=None, *, jump=None) Compute the stack effect of *opcode* with argument *oparg*. If the code has a jump target and *jump* is "True", "stack_effect()" will return the stack effect of jumping. If *jump* is "False", it will return the stack effect of not jumping. And if *jump* is "None" (default), it will return the maximal stack effect of both cases. Added in version 3.4. Changed in version 3.8: Added *jump* parameter. Changed in version 3.13: If "oparg" is omitted (or "None"), the stack effect is now returned for "oparg=0". Previously this was an error for opcodes that use their arg. It is also no longer an error to pass an integer "oparg" when the "opcode" does not use it; the "oparg" in this case is ignored. Python Bytecode Instructions ============================ The "get_instructions()" function and "Bytecode" class provide details of bytecode instructions as "Instruction" instances: class dis.Instruction Details for a bytecode operation opcode numeric code for operation, corresponding to the opcode values listed below and the bytecode values in the Opcode collections. opname human readable name for operation baseopcode numeric code for the base operation if operation is specialized; otherwise equal to "opcode" baseopname human readable name for the base operation if operation is specialized; otherwise equal to "opname" arg numeric argument to operation (if any), otherwise "None" oparg alias for "arg" argval resolved arg value (if any), otherwise "None" argrepr human readable description of operation argument (if any), otherwise an empty string. offset start index of operation within bytecode sequence start_offset start index of operation within bytecode sequence, including prefixed "EXTENDED_ARG" operations if present; otherwise equal to "offset" cache_offset start index of the cache entries following the operation end_offset end index of the cache entries following the operation starts_line "True" if this opcode starts a source line, otherwise "False" line_number source line number associated with this opcode (if any), otherwise "None" is_jump_target "True" if other code jumps to here, otherwise "False" jump_target bytecode index of the jump target if this is a jump operation, otherwise "None" positions "dis.Positions" object holding the start and end locations that are covered by this instruction. cache_info Information about the cache entries of this instruction, as triplets of the form "(name, size, data)", where the "name" and "size" describe the cache format and data is the contents of the cache. "cache_info" is "None" if the instruction does not have caches. Added in version 3.4. Changed in version 3.11: Field "positions" is added. Changed in version 3.13: Changed field "starts_line".Added fields "start_offset", "cache_offset", "end_offset", "baseopname", "baseopcode", "jump_target", "oparg", "line_number" and "cache_info". class dis.Positions In case the information is not available, some fields might be "None". lineno end_lineno col_offset end_col_offset Added in version 3.11. The Python compiler currently generates the following bytecode instructions. **General instructions** In the following, We will refer to the interpreter stack as "STACK" and describe operations on it as if it was a Python list. The top of the stack corresponds to "STACK[-1]" in this language. NOP Do nothing code. Used as a placeholder by the bytecode optimizer, and to generate line tracing events. POP_TOP Removes the top-of-stack item: STACK.pop() END_FOR Removes the top-of-stack item. Equivalent to "POP_TOP". Used to clean up at the end of loops, hence the name. Added in version 3.12. END_SEND Implements "del STACK[-2]". Used to clean up when a generator exits. Added in version 3.12. COPY(i) Push the i-th item to the top of the stack without removing it from its original location: assert i > 0 STACK.append(STACK[-i]) Added in version 3.11. SWAP(i) Swap the top of the stack with the i-th element: STACK[-i], STACK[-1] = STACK[-1], STACK[-i] Added in version 3.11. CACHE Rather than being an actual instruction, this opcode is used to mark extra space for the interpreter to cache useful data directly in the bytecode itself. It is automatically hidden by all "dis" utilities, but can be viewed with "show_caches=True". Logically, this space is part of the preceding instruction. Many opcodes expect to be followed by an exact number of caches, and will instruct the interpreter to skip over them at runtime. Populated caches can look like arbitrary instructions, so great care should be taken when reading or modifying raw, adaptive bytecode containing quickened data. Added in version 3.11. **Unary operations** Unary operations take the top of the stack, apply the operation, and push the result back on the stack. UNARY_NEGATIVE Implements "STACK[-1] = -STACK[-1]". UNARY_NOT Implements "STACK[-1] = not STACK[-1]". Changed in version 3.13: This instruction now requires an exact "bool" operand. UNARY_INVERT Implements "STACK[-1] = ~STACK[-1]". GET_ITER Implements "STACK[-1] = iter(STACK[-1])". GET_YIELD_FROM_ITER If "STACK[-1]" is a *generator iterator* or *coroutine* object it is left as is. Otherwise, implements "STACK[-1] = iter(STACK[-1])". Added in version 3.5. TO_BOOL Implements "STACK[-1] = bool(STACK[-1])". Added in version 3.13. **Binary and in-place operations** Binary operations remove the top two items from the stack ("STACK[-1]" and "STACK[-2]"). They perform the operation, then put the result back on the stack. In-place operations are like binary operations, but the operation is done in-place when "STACK[-2]" supports it, and the resulting "STACK[-1]" may be (but does not have to be) the original "STACK[-2]". BINARY_OP(op) Implements the binary and in-place operators (depending on the value of *op*): rhs = STACK.pop() lhs = STACK.pop() STACK.append(lhs op rhs) Added in version 3.11. BINARY_SUBSCR Implements: key = STACK.pop() container = STACK.pop() STACK.append(container[key]) STORE_SUBSCR Implements: key = STACK.pop() container = STACK.pop() value = STACK.pop() container[key] = value DELETE_SUBSCR Implements: key = STACK.pop() container = STACK.pop() del container[key] BINARY_SLICE Implements: end = STACK.pop() start = STACK.pop() container = STACK.pop() STACK.append(container[start:end]) Added in version 3.12. STORE_SLICE Implements: end = STACK.pop() start = STACK.pop() container = STACK.pop() values = STACK.pop() container[start:end] = value Added in version 3.12. **Coroutine opcodes** GET_AWAITABLE(where) Implements "STACK[-1] = get_awaitable(STACK[-1])", where "get_awaitable(o)" returns "o" if "o" is a coroutine object or a generator object with the "CO_ITERABLE_COROUTINE" flag, or resolves "o.__await__". If the "where" operand is nonzero, it indicates where the instruction occurs: * "1": After a call to "__aenter__" * "2": After a call to "__aexit__" Added in version 3.5. Changed in version 3.11: Previously, this instruction did not have an oparg. GET_AITER Implements "STACK[-1] = STACK[-1].__aiter__()". Added in version 3.5. Changed in version 3.7: Returning awaitable objects from "__aiter__" is no longer supported. GET_ANEXT Implement "STACK.append(get_awaitable(STACK[-1].__anext__()))" to the stack. See "GET_AWAITABLE" for details about "get_awaitable". Added in version 3.5. END_ASYNC_FOR Terminates an "async for" loop. Handles an exception raised when awaiting a next item. The stack contains the async iterable in "STACK[-2]" and the raised exception in "STACK[-1]". Both are popped. If the exception is not "StopAsyncIteration", it is re- raised. Added in version 3.8. Changed in version 3.11: Exception representation on the stack now consist of one, not three, items. CLEANUP_THROW Handles an exception raised during a "throw()" or "close()" call through the current frame. If "STACK[-1]" is an instance of "StopIteration", pop three values from the stack and push its "value" member. Otherwise, re-raise "STACK[-1]". Added in version 3.12. BEFORE_ASYNC_WITH Resolves "__aenter__" and "__aexit__" from "STACK[-1]". Pushes "__aexit__" and result of "__aenter__()" to the stack: STACK.extend((__aexit__, __aenter__()) Added in version 3.5. **Miscellaneous opcodes** SET_ADD(i) Implements: item = STACK.pop() set.add(STACK[-i], item) Used to implement set comprehensions. LIST_APPEND(i) Implements: item = STACK.pop() list.append(STACK[-i], item) Used to implement list comprehensions. MAP_ADD(i) Implements: value = STACK.pop() key = STACK.pop() dict.__setitem__(STACK[-i], key, value) Used to implement dict comprehensions. Added in version 3.1. Changed in version 3.8: Map value is "STACK[-1]" and map key is "STACK[-2]". Before, those were reversed. For all of the "SET_ADD", "LIST_APPEND" and "MAP_ADD" instructions, while the added value or key/value pair is popped off, the container object remains on the stack so that it is available for further iterations of the loop. RETURN_VALUE Returns with "STACK[-1]" to the caller of the function. RETURN_CONST(consti) Returns with "co_consts[consti]" to the caller of the function. Added in version 3.12. YIELD_VALUE Yields "STACK.pop()" from a *generator*. Changed in version 3.11: oparg set to be the stack depth. Changed in version 3.12: oparg set to be the exception block depth, for efficient closing of generators. Changed in version 3.13: oparg is "1" if this instruction is part of a yield-from or await, and "0" otherwise. SETUP_ANNOTATIONS Checks whether "__annotations__" is defined in "locals()", if not it is set up to an empty "dict". This opcode is only emitted if a class or module body contains *variable annotations* statically. Added in version 3.6. POP_EXCEPT Pops a value from the stack, which is used to restore the exception state. Changed in version 3.11: Exception representation on the stack now consist of one, not three, items. RERAISE Re-raises the exception currently on top of the stack. If oparg is non-zero, pops an additional value from the stack which is used to set "f_lasti" of the current frame. Added in version 3.9. Changed in version 3.11: Exception representation on the stack now consist of one, not three, items. PUSH_EXC_INFO Pops a value from the stack. Pushes the current exception to the top of the stack. Pushes the value originally popped back to the stack. Used in exception handlers. Added in version 3.11. CHECK_EXC_MATCH Performs exception matching for "except". Tests whether the "STACK[-2]" is an exception matching "STACK[-1]". Pops "STACK[-1]" and pushes the boolean result of the test. Added in version 3.11. CHECK_EG_MATCH Performs exception matching for "except*". Applies "split(STACK[-1])" on the exception group representing "STACK[-2]". In case of a match, pops two items from the stack and pushes the non-matching subgroup ("None" in case of full match) followed by the matching subgroup. When there is no match, pops one item (the match type) and pushes "None". Added in version 3.11. WITH_EXCEPT_START Calls the function in position 4 on the stack with arguments (type, val, tb) representing the exception at the top of the stack. Used to implement the call "context_manager.__exit__(*exc_info())" when an exception has occurred in a "with" statement. Added in version 3.9. Changed in version 3.11: The "__exit__" function is in position 4 of the stack rather than 7. Exception representation on the stack now consist of one, not three, items. LOAD_ASSERTION_ERROR Pushes "AssertionError" onto the stack. Used by the "assert" statement. Added in version 3.9. LOAD_BUILD_CLASS Pushes "builtins.__build_class__()" onto the stack. It is later called to construct a class. BEFORE_WITH This opcode performs several operations before a with block starts. First, it loads "__exit__()" from the context manager and pushes it onto the stack for later use by "WITH_EXCEPT_START". Then, "__enter__()" is called. Finally, the result of calling the "__enter__()" method is pushed onto the stack. Added in version 3.11. GET_LEN Perform "STACK.append(len(STACK[-1]))". Used in "match" statements where comparison with structure of pattern is needed. Added in version 3.10. MATCH_MAPPING If "STACK[-1]" is an instance of "collections.abc.Mapping" (or, more technically: if it has the "Py_TPFLAGS_MAPPING" flag set in its "tp_flags"), push "True" onto the stack. Otherwise, push "False". Added in version 3.10. MATCH_SEQUENCE If "STACK[-1]" is an instance of "collections.abc.Sequence" and is *not* an instance of "str"/"bytes"/"bytearray" (or, more technically: if it has the "Py_TPFLAGS_SEQUENCE" flag set in its "tp_flags"), push "True" onto the stack. Otherwise, push "False". Added in version 3.10. MATCH_KEYS "STACK[-1]" is a tuple of mapping keys, and "STACK[-2]" is the match subject. If "STACK[-2]" contains all of the keys in "STACK[-1]", push a "tuple" containing the corresponding values. Otherwise, push "None". Added in version 3.10. Changed in version 3.11: Previously, this instruction also pushed a boolean value indicating success ("True") or failure ("False"). STORE_NAME(namei) Implements "name = STACK.pop()". *namei* is the index of *name* in the attribute "co_names" of the code object. The compiler tries to use "STORE_FAST" or "STORE_GLOBAL" if possible. DELETE_NAME(namei) Implements "del name", where *namei* is the index into "co_names" attribute of the code object. UNPACK_SEQUENCE(count) Unpacks "STACK[-1]" into *count* individual values, which are put onto the stack right-to-left. Require there to be exactly *count* values.: assert(len(STACK[-1]) == count) STACK.extend(STACK.pop()[:-count-1:-1]) UNPACK_EX(counts) Implements assignment with a starred target: Unpacks an iterable in "STACK[-1]" into individual values, where the total number of values can be smaller than the number of items in the iterable: one of the new values will be a list of all leftover items. The number of values before and after the list value is limited to 255. The number of values before the list value is encoded in the argument of the opcode. The number of values after the list if any is encoded using an "EXTENDED_ARG". As a consequence, the argument can be seen as a two bytes values where the low byte of *counts* is the number of values before the list value, the high byte of *counts* the number of values after it. The extracted values are put onto the stack right-to-left, i.e. "a, *b, c = d" will be stored after execution as "STACK.extend((a, b, c))". STORE_ATTR(namei) Implements: obj = STACK.pop() value = STACK.pop() obj.name = value where *namei* is the index of name in "co_names" of the code object. DELETE_ATTR(namei) Implements: obj = STACK.pop() del obj.name where *namei* is the index of name into "co_names" of the code object. STORE_GLOBAL(namei) Works as "STORE_NAME", but stores the name as a global. DELETE_GLOBAL(namei) Works as "DELETE_NAME", but deletes a global name. LOAD_CONST(consti) Pushes "co_consts[consti]" onto the stack. LOAD_NAME(namei) Pushes the value associated with "co_names[namei]" onto the stack. The name is looked up within the locals, then the globals, then the builtins. LOAD_LOCALS Pushes a reference to the locals dictionary onto the stack. This is used to prepare namespace dictionaries for "LOAD_FROM_DICT_OR_DEREF" and "LOAD_FROM_DICT_OR_GLOBALS". Added in version 3.12. LOAD_FROM_DICT_OR_GLOBALS(i) Pops a mapping off the stack and looks up the value for "co_names[namei]". If the name is not found there, looks it up in the globals and then the builtins, similar to "LOAD_GLOBAL". This is used for loading global variables in annotation scopes within class bodies. Added in version 3.12. BUILD_TUPLE(count) Creates a tuple consuming *count* items from the stack, and pushes the resulting tuple onto the stack: if count == 0: value = () else: value = tuple(STACK[-count:]) STACK = STACK[:-count] STACK.append(value) BUILD_LIST(count) Works as "BUILD_TUPLE", but creates a list. BUILD_SET(count) Works as "BUILD_TUPLE", but creates a set. BUILD_MAP(count) Pushes a new dictionary object onto the stack. Pops "2 * count" items so that the dictionary holds *count* entries: "{..., STACK[-4]: STACK[-3], STACK[-2]: STACK[-1]}". Changed in version 3.5: The dictionary is created from stack items instead of creating an empty dictionary pre-sized to hold *count* items. BUILD_CONST_KEY_MAP(count) The version of "BUILD_MAP" specialized for constant keys. Pops the top element on the stack which contains a tuple of keys, then starting from "STACK[-2]", pops *count* values to form values in the built dictionary. Added in version 3.6. BUILD_STRING(count) Concatenates *count* strings from the stack and pushes the resulting string onto the stack. Added in version 3.6. LIST_EXTEND(i) Implements: seq = STACK.pop() list.extend(STACK[-i], seq) Used to build lists. Added in version 3.9. SET_UPDATE(i) Implements: seq = STACK.pop() set.update(STACK[-i], seq) Used to build sets. Added in version 3.9. DICT_UPDATE(i) Implements: map = STACK.pop() dict.update(STACK[-i], map) Used to build dicts. Added in version 3.9. DICT_MERGE(i) Like "DICT_UPDATE" but raises an exception for duplicate keys. Added in version 3.9. LOAD_ATTR(namei) If the low bit of "namei" is not set, this replaces "STACK[-1]" with "getattr(STACK[-1], co_names[namei>>1])". If the low bit of "namei" is set, this will attempt to load a method named "co_names[namei>>1]" from the "STACK[-1]" object. "STACK[-1]" is popped. This bytecode distinguishes two cases: if "STACK[-1]" has a method with the correct name, the bytecode pushes the unbound method and "STACK[-1]". "STACK[-1]" will be used as the first argument ("self") by "CALL" or "CALL_KW" when calling the unbound method. Otherwise, "NULL" and the object returned by the attribute lookup are pushed. Changed in version 3.12: If the low bit of "namei" is set, then a "NULL" or "self" is pushed to the stack before the attribute or unbound method respectively. LOAD_SUPER_ATTR(namei) This opcode implements "super()", both in its zero-argument and two-argument forms (e.g. "super().method()", "super().attr" and "super(cls, self).method()", "super(cls, self).attr"). It pops three values from the stack (from top of stack down): * "self": the first argument to the current method * "cls": the class within which the current method was defined * the global "super" With respect to its argument, it works similarly to "LOAD_ATTR", except that "namei" is shifted left by 2 bits instead of 1. The low bit of "namei" signals to attempt a method load, as with "LOAD_ATTR", which results in pushing "NULL" and the loaded method. When it is unset a single value is pushed to the stack. The second-low bit of "namei", if set, means that this was a two- argument call to "super()" (unset means zero-argument). Added in version 3.12. COMPARE_OP(opname) Performs a Boolean operation. The operation name can be found in "cmp_op[opname >> 5]". If the fifth-lowest bit of "opname" is set ("opname & 16"), the result should be coerced to "bool". Changed in version 3.13: The fifth-lowest bit of the oparg now indicates a forced conversion to "bool". IS_OP(invert) Performs "is" comparison, or "is not" if "invert" is 1. Added in version 3.9. CONTAINS_OP(invert) Performs "in" comparison, or "not in" if "invert" is 1. Added in version 3.9. IMPORT_NAME(namei) Imports the module "co_names[namei]". "STACK[-1]" and "STACK[-2]" are popped and provide the *fromlist* and *level* arguments of "__import__()". The module object is pushed onto the stack. The current namespace is not affected: for a proper import statement, a subsequent "STORE_FAST" instruction modifies the namespace. IMPORT_FROM(namei) Loads the attribute "co_names[namei]" from the module found in "STACK[-1]". The resulting object is pushed onto the stack, to be subsequently stored by a "STORE_FAST" instruction. JUMP_FORWARD(delta) Increments bytecode counter by *delta*. JUMP_BACKWARD(delta) Decrements bytecode counter by *delta*. Checks for interrupts. Added in version 3.11. JUMP_BACKWARD_NO_INTERRUPT(delta) Decrements bytecode counter by *delta*. Does not check for interrupts. Added in version 3.11. POP_JUMP_IF_TRUE(delta) If "STACK[-1]" is true, increments the bytecode counter by *delta*. "STACK[-1]" is popped. Changed in version 3.11: The oparg is now a relative delta rather than an absolute target. This opcode is a pseudo-instruction, replaced in final bytecode by the directed versions (forward/backward). Changed in version 3.12: This is no longer a pseudo-instruction. Changed in version 3.13: This instruction now requires an exact "bool" operand. POP_JUMP_IF_FALSE(delta) If "STACK[-1]" is false, increments the bytecode counter by *delta*. "STACK[-1]" is popped. Changed in version 3.11: The oparg is now a relative delta rather than an absolute target. This opcode is a pseudo-instruction, replaced in final bytecode by the directed versions (forward/backward). Changed in version 3.12: This is no longer a pseudo-instruction. Changed in version 3.13: This instruction now requires an exact "bool" operand. POP_JUMP_IF_NOT_NONE(delta) If "STACK[-1]" is not "None", increments the bytecode counter by *delta*. "STACK[-1]" is popped. Added in version 3.11. Changed in version 3.12: This is no longer a pseudo-instruction. POP_JUMP_IF_NONE(delta) If "STACK[-1]" is "None", increments the bytecode counter by *delta*. "STACK[-1]" is popped. Added in version 3.11. Changed in version 3.12: This is no longer a pseudo-instruction. FOR_ITER(delta) "STACK[-1]" is an *iterator*. Call its "__next__()" method. If this yields a new value, push it on the stack (leaving the iterator below it). If the iterator indicates it is exhausted then the byte code counter is incremented by *delta*. Changed in version 3.12: Up until 3.11 the iterator was popped when it was exhausted. LOAD_GLOBAL(namei) Loads the global named "co_names[namei>>1]" onto the stack. Changed in version 3.11: If the low bit of "namei" is set, then a "NULL" is pushed to the stack before the global variable. LOAD_FAST(var_num) Pushes a reference to the local "co_varnames[var_num]" onto the stack. Changed in version 3.12: This opcode is now only used in situations where the local variable is guaranteed to be initialized. It cannot raise "UnboundLocalError". LOAD_FAST_LOAD_FAST(var_nums) Pushes references to "co_varnames[var_nums >> 4]" and "co_varnames[var_nums & 15]" onto the stack. Added in version 3.13. LOAD_FAST_CHECK(var_num) Pushes a reference to the local "co_varnames[var_num]" onto the stack, raising an "UnboundLocalError" if the local variable has not been initialized. Added in version 3.12. LOAD_FAST_AND_CLEAR(var_num) Pushes a reference to the local "co_varnames[var_num]" onto the stack (or pushes "NULL" onto the stack if the local variable has not been initialized) and sets "co_varnames[var_num]" to "NULL". Added in version 3.12. STORE_FAST(var_num) Stores "STACK.pop()" into the local "co_varnames[var_num]". STORE_FAST_STORE_FAST(var_nums) Stores "STACK[-1]" into "co_varnames[var_nums >> 4]" and "STACK[-2]" into "co_varnames[var_nums & 15]". Added in version 3.13. STORE_FAST_LOAD_FAST(var_nums) Stores "STACK.pop()" into the local "co_varnames[var_nums >> 4]" and pushes a reference to the local "co_varnames[var_nums & 15]" onto the stack. Added in version 3.13. DELETE_FAST(var_num) Deletes local "co_varnames[var_num]". MAKE_CELL(i) Creates a new cell in slot "i". If that slot is nonempty then that value is stored into the new cell. Added in version 3.11. LOAD_DEREF(i) Loads the cell contained in slot "i" of the “fast locals” storage. Pushes a reference to the object the cell contains on the stack. Changed in version 3.11: "i" is no longer offset by the length of "co_varnames". LOAD_FROM_DICT_OR_DEREF(i) Pops a mapping off the stack and looks up the name associated with slot "i" of the “fast locals” storage in this mapping. If the name is not found there, loads it from the cell contained in slot "i", similar to "LOAD_DEREF". This is used for loading *closure variables* in class bodies (which previously used "LOAD_CLASSDEREF") and in annotation scopes within class bodies. Added in version 3.12. STORE_DEREF(i) Stores "STACK.pop()" into the cell contained in slot "i" of the “fast locals” storage. Changed in version 3.11: "i" is no longer offset by the length of "co_varnames". DELETE_DEREF(i) Empties the cell contained in slot "i" of the “fast locals” storage. Used by the "del" statement. Added in version 3.2. Changed in version 3.11: "i" is no longer offset by the length of "co_varnames". COPY_FREE_VARS(n) Copies the "n" *free (closure) variables* from the closure into the frame. Removes the need for special code on the caller’s side when calling closures. Added in version 3.11. RAISE_VARARGS(argc) Raises an exception using one of the 3 forms of the "raise" statement, depending on the value of *argc*: * 0: "raise" (re-raise previous exception) * 1: "raise STACK[-1]" (raise exception instance or type at "STACK[-1]") * 2: "raise STACK[-2] from STACK[-1]" (raise exception instance or type at "STACK[-2]" with "__cause__" set to "STACK[-1]") CALL(argc) Calls a callable object with the number of arguments specified by "argc". On the stack are (in ascending order): * The callable * "self" or "NULL" * The remaining positional arguments "argc" is the total of the positional arguments, excluding "self". "CALL" pops all arguments and the callable object off the stack, calls the callable object with those arguments, and pushes the return value returned by the callable object. Added in version 3.11. Changed in version 3.13: The callable now always appears at the same position on the stack. Changed in version 3.13: Calls with keyword arguments are now handled by "CALL_KW". CALL_KW(argc) Calls a callable object with the number of arguments specified by "argc", including one or more named arguments. On the stack are (in ascending order): * The callable * "self" or "NULL" * The remaining positional arguments * The named arguments * A "tuple" of keyword argument names "argc" is the total of the positional and named arguments, excluding "self". The length of the tuple of keyword argument names is the number of named arguments. "CALL_KW" pops all arguments, the keyword names, and the callable object off the stack, calls the callable object with those arguments, and pushes the return value returned by the callable object. Added in version 3.13. CALL_FUNCTION_EX(flags) Calls a callable object with variable set of positional and keyword arguments. If the lowest bit of *flags* is set, the top of the stack contains a mapping object containing additional keyword arguments. Before the callable is called, the mapping object and iterable object are each “unpacked” and their contents passed in as keyword and positional arguments respectively. "CALL_FUNCTION_EX" pops all arguments and the callable object off the stack, calls the callable object with those arguments, and pushes the return value returned by the callable object. Added in version 3.6. PUSH_NULL Pushes a "NULL" to the stack. Used in the call sequence to match the "NULL" pushed by "LOAD_METHOD" for non-method calls. Added in version 3.11. MAKE_FUNCTION Pushes a new function object on the stack built from the code object at "STACK[-1]". Changed in version 3.10: Flag value "0x04" is a tuple of strings instead of dictionary Changed in version 3.11: Qualified name at "STACK[-1]" was removed. Changed in version 3.13: Extra function attributes on the stack, signaled by oparg flags, were removed. They now use "SET_FUNCTION_ATTRIBUTE". SET_FUNCTION_ATTRIBUTE(flag) Sets an attribute on a function object. Expects the function at "STACK[-1]" and the attribute value to set at "STACK[-2]"; consumes both and leaves the function at "STACK[-1]". The flag determines which attribute to set: * "0x01" a tuple of default values for positional-only and positional-or-keyword parameters in positional order * "0x02" a dictionary of keyword-only parameters’ default values * "0x04" a tuple of strings containing parameters’ annotations * "0x08" a tuple containing cells for free variables, making a closure Added in version 3.13. BUILD_SLICE(argc) Pushes a slice object on the stack. *argc* must be 2 or 3. If it is 2, implements: end = STACK.pop() start = STACK.pop() STACK.append(slice(start, end)) if it is 3, implements: step = STACK.pop() end = STACK.pop() start = STACK.pop() STACK.append(slice(start, end, step)) See the "slice()" built-in function for more information. EXTENDED_ARG(ext) Prefixes any opcode which has an argument too big to fit into the default one byte. *ext* holds an additional byte which act as higher bits in the argument. For each opcode, at most three prefixal "EXTENDED_ARG" are allowed, forming an argument from two- byte to four-byte. CONVERT_VALUE(oparg) Convert value to a string, depending on "oparg": value = STACK.pop() result = func(value) STACK.append(result) * "oparg == 1": call "str()" on *value* * "oparg == 2": call "repr()" on *value* * "oparg == 3": call "ascii()" on *value* Used for implementing formatted string literals (f-strings). Added in version 3.13. FORMAT_SIMPLE Formats the value on top of stack: value = STACK.pop() result = value.__format__("") STACK.append(result) Used for implementing formatted string literals (f-strings). Added in version 3.13. FORMAT_WITH_SPEC Formats the given value with the given format spec: spec = STACK.pop() value = STACK.pop() result = value.__format__(spec) STACK.append(result) Used for implementing formatted string literals (f-strings). Added in version 3.13. MATCH_CLASS(count) "STACK[-1]" is a tuple of keyword attribute names, "STACK[-2]" is the class being matched against, and "STACK[-3]" is the match subject. *count* is the number of positional sub-patterns. Pop "STACK[-1]", "STACK[-2]", and "STACK[-3]". If "STACK[-3]" is an instance of "STACK[-2]" and has the positional and keyword attributes required by *count* and "STACK[-1]", push a tuple of extracted attributes. Otherwise, push "None". Added in version 3.10. Changed in version 3.11: Previously, this instruction also pushed a boolean value indicating success ("True") or failure ("False"). RESUME(context) A no-op. Performs internal tracing, debugging and optimization checks. The "context" oparand consists of two parts. The lowest two bits indicate where the "RESUME" occurs: * "0" The start of a function, which is neither a generator, coroutine nor an async generator * "1" After a "yield" expression * "2" After a "yield from" expression * "3" After an "await" expression The next bit is "1" if the RESUME is at except-depth "1", and "0" otherwise. Added in version 3.11. Changed in version 3.13: The oparg value changed to include information about except-depth RETURN_GENERATOR Create a generator, coroutine, or async generator from the current frame. Used as first opcode of in code object for the above mentioned callables. Clear the current frame and return the newly created generator. Added in version 3.11. SEND(delta) Equivalent to "STACK[-1] = STACK[-2].send(STACK[-1])". Used in "yield from" and "await" statements. If the call raises "StopIteration", pop the top value from the stack, push the exception’s "value" attribute, and increment the bytecode counter by *delta*. Added in version 3.11. HAVE_ARGUMENT This is not really an opcode. It identifies the dividing line between opcodes in the range [0,255] which don’t use their argument and those that do ("< HAVE_ARGUMENT" and ">= HAVE_ARGUMENT", respectively). If your application uses pseudo instructions or specialized instructions, use the "hasarg" collection instead. Changed in version 3.6: Now every instruction has an argument, but opcodes "< HAVE_ARGUMENT" ignore it. Before, only opcodes ">= HAVE_ARGUMENT" had an argument. Changed in version 3.12: Pseudo instructions were added to the "dis" module, and for them it is not true that comparison with "HAVE_ARGUMENT" indicates whether they use their arg. Deprecated since version 3.13: Use "hasarg" instead. CALL_INTRINSIC_1 Calls an intrinsic function with one argument. Passes "STACK[-1]" as the argument and sets "STACK[-1]" to the result. Used to implement functionality that is not performance critical. The operand determines which intrinsic function is called: +-------------------------------------+-------------------------------------+ | Operand | Description | |=====================================|=====================================| | "INTRINSIC_1_INVALID" | Not valid | +-------------------------------------+-------------------------------------+ | "INTRINSIC_PRINT" | Prints the argument to standard | | | out. Used in the REPL. | +-------------------------------------+-------------------------------------+ | "INTRINSIC_IMPORT_STAR" | Performs "import *" for the named | | | module. | +-------------------------------------+-------------------------------------+ | "INTRINSIC_STOPITERATION_ERROR" | Extracts the return value from a | | | "StopIteration" exception. | +-------------------------------------+-------------------------------------+ | "INTRINSIC_ASYNC_GEN_WRAP" | Wraps an async generator value | +-------------------------------------+-------------------------------------+ | "INTRINSIC_UNARY_POSITIVE" | Performs the unary "+" operation | +-------------------------------------+-------------------------------------+ | "INTRINSIC_LIST_TO_TUPLE" | Converts a list to a tuple | +-------------------------------------+-------------------------------------+ | "INTRINSIC_TYPEVAR" | Creates a "typing.TypeVar" | +-------------------------------------+-------------------------------------+ | "INTRINSIC_PARAMSPEC" | Creates a "typing.ParamSpec" | +-------------------------------------+-------------------------------------+ | "INTRINSIC_TYPEVARTUPLE" | Creates a "typing.TypeVarTuple" | +-------------------------------------+-------------------------------------+ | "INTRINSIC_SUBSCRIPT_GENERIC" | Returns "typing.Generic" | | | subscripted with the argument | +-------------------------------------+-------------------------------------+ | "INTRINSIC_TYPEALIAS" | Creates a "typing.TypeAliasType"; | | | used in the "type" statement. The | | | argument is a tuple of the type | | | alias’s name, type parameters, and | | | value. | +-------------------------------------+-------------------------------------+ Added in version 3.12. CALL_INTRINSIC_2 Calls an intrinsic function with two arguments. Used to implement functionality that is not performance critical: arg2 = STACK.pop() arg1 = STACK.pop() result = intrinsic2(arg1, arg2) STACK.append(result) The operand determines which intrinsic function is called: +------------------------------------------+-------------------------------------+ | Operand | Description | |==========================================|=====================================| | "INTRINSIC_2_INVALID" | Not valid | +------------------------------------------+-------------------------------------+ | "INTRINSIC_PREP_RERAISE_STAR" | Calculates the "ExceptionGroup" to | | | raise from a "try-except*". | +------------------------------------------+-------------------------------------+ | "INTRINSIC_TYPEVAR_WITH_BOUND" | Creates a "typing.TypeVar" with a | | | bound. | +------------------------------------------+-------------------------------------+ | "INTRINSIC_TYPEVAR_WITH_CONSTRAINTS" | Creates a "typing.TypeVar" with | | | constraints. | +------------------------------------------+-------------------------------------+ | "INTRINSIC_SET_FUNCTION_TYPE_PARAMS" | Sets the "__type_params__" | | | attribute of a function. | +------------------------------------------+-------------------------------------+ Added in version 3.12. **Pseudo-instructions** These opcodes do not appear in Python bytecode. They are used by the compiler but are replaced by real opcodes or removed before bytecode is generated. SETUP_FINALLY(target) Set up an exception handler for the following code block. If an exception occurs, the value stack level is restored to its current state and control is transferred to the exception handler at "target". SETUP_CLEANUP(target) Like "SETUP_FINALLY", but in case of an exception also pushes the last instruction ("lasti") to the stack so that "RERAISE" can restore it. If an exception occurs, the value stack level and the last instruction on the frame are restored to their current state, and control is transferred to the exception handler at "target". SETUP_WITH(target) Like "SETUP_CLEANUP", but in case of an exception one more item is popped from the stack before control is transferred to the exception handler at "target". This variant is used in "with" and "async with" constructs, which push the return value of the context manager’s "__enter__()" or "__aenter__()" to the stack. POP_BLOCK Marks the end of the code block associated with the last "SETUP_FINALLY", "SETUP_CLEANUP" or "SETUP_WITH". JUMP JUMP_NO_INTERRUPT Undirected relative jump instructions which are replaced by their directed (forward/backward) counterparts by the assembler. LOAD_CLOSURE(i) Pushes a reference to the cell contained in slot "i" of the “fast locals” storage. Note that "LOAD_CLOSURE" is replaced with "LOAD_FAST" in the assembler. Changed in version 3.13: This opcode is now a pseudo-instruction. LOAD_METHOD Optimized unbound method lookup. Emitted as a "LOAD_ATTR" opcode with a flag set in the arg. Opcode collections ================== These collections are provided for automatic introspection of bytecode instructions: Changed in version 3.12: The collections now contain pseudo instructions and instrumented instructions as well. These are opcodes with values ">= MIN_PSEUDO_OPCODE" and ">= MIN_INSTRUMENTED_OPCODE". dis.opname Sequence of operation names, indexable using the bytecode. dis.opmap Dictionary mapping operation names to bytecodes. dis.cmp_op Sequence of all compare operation names. dis.hasarg Sequence of bytecodes that use their argument. Added in version 3.12. dis.hasconst Sequence of bytecodes that access a constant. dis.hasfree Sequence of bytecodes that access a *free (closure) variable*. ‘free’ in this context refers to names in the current scope that are referenced by inner scopes or names in outer scopes that are referenced from this scope. It does *not* include references to global or builtin scopes. dis.hasname Sequence of bytecodes that access an attribute by name. dis.hasjump Sequence of bytecodes that have a jump target. All jumps are relative. Added in version 3.13. dis.haslocal Sequence of bytecodes that access a local variable. dis.hascompare Sequence of bytecodes of Boolean operations. dis.hasexc Sequence of bytecodes that set an exception handler. Added in version 3.12. dis.hasjrel Sequence of bytecodes that have a relative jump target. Deprecated since version 3.13: All jumps are now relative. Use "hasjump". dis.hasjabs Sequence of bytecodes that have an absolute jump target. Deprecated since version 3.13: All jumps are now relative. This list is empty. Software Packaging and Distribution *********************************** These libraries help you with publishing and installing Python software. While these modules are designed to work in conjunction with the Python Package Index, they can also be used with a local index server, or without any index server at all. * "ensurepip" — Bootstrapping the "pip" installer * Command line interface * Module API * "venv" — Creation of virtual environments * Creating virtual environments * How venvs work * API * An example of extending "EnvBuilder" * "zipapp" — Manage executable Python zip archives * Basic Example * Command-Line Interface * Python API * Examples * Specifying the Interpreter * Creating Standalone Applications with zipapp * Caveats * The Python Zip Application Archive Format "distutils" — Building and installing Python modules **************************************************** Deprecated since version 3.10, removed in version 3.12. This module is no longer part of the Python standard library. It was removed in Python 3.12 after being deprecated in Python 3.10. The removal was decided in **PEP 632**, which has migration advice. The last version of Python that provided the "distutils" module was Python 3.11. "doctest" — Test interactive Python examples ******************************************** **Source code:** Lib/doctest.py ====================================================================== The "doctest" module searches for pieces of text that look like interactive Python sessions, and then executes those sessions to verify that they work exactly as shown. There are several common ways to use doctest: * To check that a module’s docstrings are up-to-date by verifying that all interactive examples still work as documented. * To perform regression testing by verifying that interactive examples from a test file or a test object work as expected. * To write tutorial documentation for a package, liberally illustrated with input-output examples. Depending on whether the examples or the expository text are emphasized, this has the flavor of “literate testing” or “executable documentation”. Here’s a complete but small example module: """ This is the "example" module. The example module supplies one function, factorial(). For example, >>> factorial(5) 120 """ def factorial(n): """Return the factorial of n, an exact integer >= 0. >>> [factorial(n) for n in range(6)] [1, 1, 2, 6, 24, 120] >>> factorial(30) 265252859812191058636308480000000 >>> factorial(-1) Traceback (most recent call last): ... ValueError: n must be >= 0 Factorials of floats are OK, but the float must be an exact integer: >>> factorial(30.1) Traceback (most recent call last): ... ValueError: n must be exact integer >>> factorial(30.0) 265252859812191058636308480000000 It must also not be ridiculously large: >>> factorial(1e100) Traceback (most recent call last): ... OverflowError: n too large """ import math if not n >= 0: raise ValueError("n must be >= 0") if math.floor(n) != n: raise ValueError("n must be exact integer") if n+1 == n: # catch a value like 1e300 raise OverflowError("n too large") result = 1 factor = 2 while factor <= n: result *= factor factor += 1 return result if __name__ == "__main__": import doctest doctest.testmod() If you run "example.py" directly from the command line, "doctest" works its magic: $ python example.py $ There’s no output! That’s normal, and it means all the examples worked. Pass "-v" to the script, and "doctest" prints a detailed log of what it’s trying, and prints a summary at the end: $ python example.py -v Trying: factorial(5) Expecting: 120 ok Trying: [factorial(n) for n in range(6)] Expecting: [1, 1, 2, 6, 24, 120] ok And so on, eventually ending with: Trying: factorial(1e100) Expecting: Traceback (most recent call last): ... OverflowError: n too large ok 2 items passed all tests: 1 test in __main__ 6 tests in __main__.factorial 7 tests in 2 items. 7 passed. Test passed. $ That’s all you need to know to start making productive use of "doctest"! Jump in. The following sections provide full details. Note that there are many examples of doctests in the standard Python test suite and libraries. Especially useful examples can be found in the standard test file "Lib/test/test_doctest/test_doctest.py". Simple Usage: Checking Examples in Docstrings ============================================= The simplest way to start using doctest (but not necessarily the way you’ll continue to do it) is to end each module "M" with: if __name__ == "__main__": import doctest doctest.testmod() "doctest" then examines docstrings in module "M". Running the module as a script causes the examples in the docstrings to get executed and verified: python M.py This won’t display anything unless an example fails, in which case the failing example(s) and the cause(s) of the failure(s) are printed to stdout, and the final line of output is "***Test Failed*** N failures.", where *N* is the number of examples that failed. Run it with the "-v" switch instead: python M.py -v and a detailed report of all examples tried is printed to standard output, along with assorted summaries at the end. You can force verbose mode by passing "verbose=True" to "testmod()", or prohibit it by passing "verbose=False". In either of those cases, "sys.argv" is not examined by "testmod()" (so passing "-v" or not has no effect). There is also a command line shortcut for running "testmod()", see section Command-line Usage. For more information on "testmod()", see section Basic API. Simple Usage: Checking Examples in a Text File ============================================== Another simple application of doctest is testing interactive examples in a text file. This can be done with the "testfile()" function: import doctest doctest.testfile("example.txt") That short script executes and verifies any interactive Python examples contained in the file "example.txt". The file content is treated as if it were a single giant docstring; the file doesn’t need to contain a Python program! For example, perhaps "example.txt" contains this: The ``example`` module ====================== Using ``factorial`` ------------------- This is an example text file in reStructuredText format. First import ``factorial`` from the ``example`` module: >>> from example import factorial Now use it: >>> factorial(6) 120 Running "doctest.testfile("example.txt")" then finds the error in this documentation: File "./example.txt", line 14, in example.txt Failed example: factorial(6) Expected: 120 Got: 720 As with "testmod()", "testfile()" won’t display anything unless an example fails. If an example does fail, then the failing example(s) and the cause(s) of the failure(s) are printed to stdout, using the same format as "testmod()". By default, "testfile()" looks for files in the calling module’s directory. See section Basic API for a description of the optional arguments that can be used to tell it to look for files in other locations. Like "testmod()", "testfile()"’s verbosity can be set with the "-v" command-line switch or with the optional keyword argument *verbose*. There is also a command line shortcut for running "testfile()", see section Command-line Usage. For more information on "testfile()", see section Basic API. Command-line Usage ================== The "doctest" module can be invoked as a script from the command line: python -m doctest [-v] [-o OPTION] [-f] file [file ...] -v, --verbose Detailed report of all examples tried is printed to standard output, along with assorted summaries at the end: python -m doctest -v example.py This will import "example.py" as a standalone module and run "testmod()" on it. Note that this may not work correctly if the file is part of a package and imports other submodules from that package. If the file name does not end with ".py", "doctest" infers that it must be run with "testfile()" instead: python -m doctest -v example.txt -o, --option