|
|
|
|
|
|
|
|
|
|
|
|
|
''' |
|
----------------------------- |
|
dill: serialize all of Python |
|
----------------------------- |
|
|
|
About Dill |
|
========== |
|
|
|
``dill`` extends Python's ``pickle`` module for serializing and de-serializing |
|
Python objects to the majority of the built-in Python types. Serialization |
|
is the process of converting an object to a byte stream, and the inverse |
|
of which is converting a byte stream back to a Python object hierarchy. |
|
|
|
``dill`` provides the user the same interface as the ``pickle`` module, and |
|
also includes some additional features. In addition to pickling Python |
|
objects, ``dill`` provides the ability to save the state of an interpreter |
|
session in a single command. Hence, it would be feasible to save an |
|
interpreter session, close the interpreter, ship the pickled file to |
|
another computer, open a new interpreter, unpickle the session and |
|
thus continue from the 'saved' state of the original interpreter |
|
session. |
|
|
|
``dill`` can be used to store Python objects to a file, but the primary |
|
usage is to send Python objects across the network as a byte stream. |
|
``dill`` is quite flexible, and allows arbitrary user defined classes |
|
and functions to be serialized. Thus ``dill`` is not intended to be |
|
secure against erroneously or maliciously constructed data. It is |
|
left to the user to decide whether the data they unpickle is from |
|
a trustworthy source. |
|
|
|
``dill`` is part of ``pathos``, a Python framework for heterogeneous computing. |
|
``dill`` is in active development, so any user feedback, bug reports, comments, |
|
or suggestions are highly appreciated. A list of issues is located at |
|
https://github.com/uqfoundation/dill/issues, with a legacy list maintained at |
|
https://uqfoundation.github.io/project/pathos/query. |
|
|
|
|
|
Major Features |
|
============== |
|
|
|
``dill`` can pickle the following standard types: |
|
|
|
- none, type, bool, int, float, complex, bytes, str, |
|
- tuple, list, dict, file, buffer, builtin, |
|
- Python classes, namedtuples, dataclasses, metaclasses, |
|
- instances of classes, |
|
- set, frozenset, array, functions, exceptions |
|
|
|
``dill`` can also pickle more 'exotic' standard types: |
|
|
|
- functions with yields, nested functions, lambdas, |
|
- cell, method, unboundmethod, module, code, methodwrapper, |
|
- methoddescriptor, getsetdescriptor, memberdescriptor, wrapperdescriptor, |
|
- dictproxy, slice, notimplemented, ellipsis, quit |
|
|
|
``dill`` cannot yet pickle these standard types: |
|
|
|
- frame, generator, traceback |
|
|
|
``dill`` also provides the capability to: |
|
|
|
- save and load Python interpreter sessions |
|
- save and extract the source code from functions and classes |
|
- interactively diagnose pickling errors |
|
|
|
|
|
Current Release |
|
=============== |
|
|
|
The latest released version of ``dill`` is available from: |
|
|
|
https://pypi.org/project/dill |
|
|
|
``dill`` is distributed under a 3-clause BSD license. |
|
|
|
|
|
Development Version |
|
=================== |
|
|
|
You can get the latest development version with all the shiny new features at: |
|
|
|
https://github.com/uqfoundation |
|
|
|
If you have a new contribution, please submit a pull request. |
|
|
|
|
|
Installation |
|
============ |
|
|
|
``dill`` can be installed with ``pip``:: |
|
|
|
$ pip install dill |
|
|
|
To optionally include the ``objgraph`` diagnostic tool in the install:: |
|
|
|
$ pip install dill[graph] |
|
|
|
To optionally include the ``gprof2dot`` diagnostic tool in the install:: |
|
|
|
$ pip install dill[profile] |
|
|
|
For windows users, to optionally install session history tools:: |
|
|
|
$ pip install dill[readline] |
|
|
|
|
|
Requirements |
|
============ |
|
|
|
``dill`` requires: |
|
|
|
- ``python`` (or ``pypy``), **>=3.8** |
|
- ``setuptools``, **>=42** |
|
|
|
Optional requirements: |
|
|
|
- ``objgraph``, **>=1.7.2** |
|
- ``gprof2dot``, **>=2022.7.29** |
|
- ``pyreadline``, **>=1.7.1** (on windows) |
|
|
|
|
|
Basic Usage |
|
=========== |
|
|
|
``dill`` is a drop-in replacement for ``pickle``. Existing code can be |
|
updated to allow complete pickling using:: |
|
|
|
>>> import dill as pickle |
|
|
|
or:: |
|
|
|
>>> from dill import dumps, loads |
|
|
|
``dumps`` converts the object to a unique byte string, and ``loads`` performs |
|
the inverse operation:: |
|
|
|
>>> squared = lambda x: x**2 |
|
>>> loads(dumps(squared))(3) |
|
9 |
|
|
|
There are a number of options to control serialization which are provided |
|
as keyword arguments to several ``dill`` functions: |
|
|
|
* with *protocol*, the pickle protocol level can be set. This uses the |
|
same value as the ``pickle`` module, *DEFAULT_PROTOCOL*. |
|
* with *byref=True*, ``dill`` to behave a lot more like pickle with |
|
certain objects (like modules) pickled by reference as opposed to |
|
attempting to pickle the object itself. |
|
* with *recurse=True*, objects referred to in the global dictionary are |
|
recursively traced and pickled, instead of the default behavior of |
|
attempting to store the entire global dictionary. |
|
* with *fmode*, the contents of the file can be pickled along with the file |
|
handle, which is useful if the object is being sent over the wire to a |
|
remote system which does not have the original file on disk. Options are |
|
*HANDLE_FMODE* for just the handle, *CONTENTS_FMODE* for the file content |
|
and *FILE_FMODE* for content and handle. |
|
* with *ignore=False*, objects reconstructed with types defined in the |
|
top-level script environment use the existing type in the environment |
|
rather than a possibly different reconstructed type. |
|
|
|
The default serialization can also be set globally in *dill.settings*. |
|
Thus, we can modify how ``dill`` handles references to the global dictionary |
|
locally or globally:: |
|
|
|
>>> import dill.settings |
|
>>> dumps(absolute) == dumps(absolute, recurse=True) |
|
False |
|
>>> dill.settings['recurse'] = True |
|
>>> dumps(absolute) == dumps(absolute, recurse=True) |
|
True |
|
|
|
``dill`` also includes source code inspection, as an alternate to pickling:: |
|
|
|
>>> import dill.source |
|
>>> print(dill.source.getsource(squared)) |
|
squared = lambda x:x**2 |
|
|
|
To aid in debugging pickling issues, use *dill.detect* which provides |
|
tools like pickle tracing:: |
|
|
|
>>> import dill.detect |
|
>>> with dill.detect.trace(): |
|
>>> dumps(squared) |
|
β¬ F1: <function <lambda> at 0x7fe074f8c280> |
|
ββ¬ F2: <function _create_function at 0x7fe074c49c10> |
|
ββ # F2 [34 B] |
|
ββ¬ Co: <code object <lambda> at 0x7fe07501eb30, file "<stdin>", line 1> |
|
βββ¬ F2: <function _create_code at 0x7fe074c49ca0> |
|
βββ # F2 [19 B] |
|
ββ # Co [87 B] |
|
ββ¬ D1: <dict object at 0x7fe0750d4680> |
|
ββ # D1 [22 B] |
|
ββ¬ D2: <dict object at 0x7fe074c5a1c0> |
|
ββ # D2 [2 B] |
|
ββ¬ D2: <dict object at 0x7fe074f903c0> |
|
βββ¬ D2: <dict object at 0x7fe074f8ebc0> |
|
βββ # D2 [2 B] |
|
ββ # D2 [23 B] |
|
β # F1 [180 B] |
|
|
|
With trace, we see how ``dill`` stored the lambda (``F1``) by first storing |
|
``_create_function``, the underlying code object (``Co``) and ``_create_code`` |
|
(which is used to handle code objects), then we handle the reference to |
|
the global dict (``D2``) plus other dictionaries (``D1`` and ``D2``) that |
|
save the lambda object's state. A ``#`` marks when the object is actually stored. |
|
|
|
|
|
More Information |
|
================ |
|
|
|
Probably the best way to get started is to look at the documentation at |
|
http://dill.rtfd.io. Also see ``dill.tests`` for a set of scripts that |
|
demonstrate how ``dill`` can serialize different Python objects. You can |
|
run the test suite with ``python -m dill.tests``. The contents of any |
|
pickle file can be examined with ``undill``. As ``dill`` conforms to |
|
the ``pickle`` interface, the examples and documentation found at |
|
http://docs.python.org/library/pickle.html also apply to ``dill`` |
|
if one will ``import dill as pickle``. The source code is also generally |
|
well documented, so further questions may be resolved by inspecting the |
|
code itself. Please feel free to submit a ticket on github, or ask a |
|
question on stackoverflow (**@Mike McKerns**). |
|
If you would like to share how you use ``dill`` in your work, please send |
|
an email (to **mmckerns at uqfoundation dot org**). |
|
|
|
|
|
Citation |
|
======== |
|
|
|
If you use ``dill`` to do research that leads to publication, we ask that you |
|
acknowledge use of ``dill`` by citing the following in your publication:: |
|
|
|
M.M. McKerns, L. Strand, T. Sullivan, A. Fang, M.A.G. Aivazis, |
|
"Building a framework for predictive science", Proceedings of |
|
the 10th Python in Science Conference, 2011; |
|
http://arxiv.org/pdf/1202.1056 |
|
|
|
Michael McKerns and Michael Aivazis, |
|
"pathos: a framework for heterogeneous computing", 2010- ; |
|
https://uqfoundation.github.io/project/pathos |
|
|
|
Please see https://uqfoundation.github.io/project/pathos or |
|
http://arxiv.org/pdf/1202.1056 for further information. |
|
|
|
''' |
|
|
|
__version__ = '0.3.8' |
|
__author__ = 'Mike McKerns' |
|
|
|
__license__ = ''' |
|
Copyright (c) 2004-2016 California Institute of Technology. |
|
Copyright (c) 2016-2024 The Uncertainty Quantification Foundation. |
|
All rights reserved. |
|
|
|
This software is available subject to the conditions and terms laid |
|
out below. By downloading and using this software you are agreeing |
|
to the following conditions. |
|
|
|
Redistribution and use in source and binary forms, with or without |
|
modification, are permitted provided that the following conditions |
|
are met: |
|
|
|
- Redistributions of source code must retain the above copyright |
|
notice, this list of conditions and the following disclaimer. |
|
|
|
- Redistributions in binary form must reproduce the above copyright |
|
notice, this list of conditions and the following disclaimer in the |
|
documentation and/or other materials provided with the distribution. |
|
|
|
- Neither the names of the copyright holders nor the names of any of |
|
the contributors may be used to endorse or promote products derived |
|
from this software without specific prior written permission. |
|
|
|
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS |
|
"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED |
|
TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR |
|
PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR |
|
CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, |
|
EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, |
|
PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; |
|
OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, |
|
WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR |
|
OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF |
|
ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. |
|
|
|
''' |
|
|