Skip to content
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
45 commits
Select commit Hold shift + click to select a range
19b28fc
CVE-2020-10735: Prevent DoS by very large int()
tiran May 5, 2020
0a96b20
Default to disable, improve tests and docs
tiran Jan 19, 2022
88f6d5d
fix typo
tiran Jan 19, 2022
70c195e
More docs (WIP)
tiran Jan 19, 2022
e17e93b
Basic documentation for sys functions
tiran Jan 19, 2022
fbd14b7
Use ValueError, ignore underscore, scale limit
tiran Jan 20, 2022
dd74d70
Fix CI
tiran Jan 20, 2022
0e01461
Address Greg's review
tiran Aug 1, 2022
0b21e5f
Fix sys.flags len and docs
tiran Aug 1, 2022
3b38abe
Keep the warning, but remove advice about limiting input length in th…
gpshead Aug 2, 2022
37193ed
Renamed the APIs & too many other refactorings.
gpshead Aug 5, 2022
c90b79f
Improve the configuring docs.
gpshead Aug 7, 2022
fea25ea
Stop tying to base10, just use string digits.
gpshead Aug 7, 2022
ac9f22f
Remove the added now-unneeded helper log tbl fn.
gpshead Aug 7, 2022
da72dd1
prevent intdostimeit from emitting errors in test_tools.
gpshead Aug 7, 2022
d7e4d7b
Remove a leftover base 10 reference. clarify.
gpshead Aug 7, 2022
5c7e6d5
versionadded/changed to 3.12
gpshead Aug 7, 2022
61a5bc9
Link to the CVE from the main doc.
gpshead Aug 7, 2022
c15adde
Add a What's New entry.
gpshead Aug 7, 2022
76ae1c2
Add a Misc/NEWS.d entry.
gpshead Aug 7, 2022
1ad88f5
Undo addition to PyConfig to ease backporting.
gpshead Aug 8, 2022
0c83111
Remove the Tools/scripts/ example and timing code.
gpshead Aug 8, 2022
5d39ab6
un-add the <math.h> include (not needed for PR anymore)
gpshead Aug 8, 2022
5b77b3e
Remove added unused imports.
gpshead Aug 8, 2022
de00cdc
Tabs -> Spaces
gpshead Aug 8, 2022
3cc8553
make html and make doctest in Doc pass.
gpshead Aug 8, 2022
da97e65
Raise the default limit and the threshold.
gpshead Aug 10, 2022
ef03a16
Remove xmlrpc.client changes, test-only.
gpshead Aug 12, 2022
e916845
Rearrange the new stdtypes docs, w/limits + caution.
gpshead Aug 13, 2022
101502e
Make a huge int a SyntaxError with lineno when parsing.
gpshead Aug 16, 2022
fa8a58a
Mention the chosen default in the NEWS entry.
gpshead Aug 16, 2022
313ab6d
Properly clear & free the prior exception.
gpshead Aug 16, 2022
614cd02
Add a note to the float.as_integer_ratio() docs.
gpshead Aug 17, 2022
16ad090
Clarify the documentation wording and error msg.
gpshead Aug 17, 2022
4eb72e6
Fix test_idle, it used a long int on a line.
gpshead Aug 17, 2022
da36550
Rename the test.support context manager and document it.
gpshead Aug 19, 2022
f4372cc
Documentation cleanup.
gpshead Aug 19, 2022
c421853
Update attribution in Misc/NEWS.d
gpshead Aug 25, 2022
9f2168a
Regen global strings
tiran Sep 1, 2022
3c8504b
Make the doctest actually run & fix it.
gpshead Sep 1, 2022
1586419
Fix the docs build.
gpshead Sep 2, 2022
94bd3ee
Rename the news file to appease the Bedevere bot.
gpshead Sep 2, 2022
0b91f65
Regen argument clinic after the rebase merge.
gpshead Sep 2, 2022
02776f9
Hexi hexa
tiran Sep 2, 2022
173fa4e
Hexi hexa 2
tiran Sep 2, 2022
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Stop tying to base10, just use string digits.
Renamed to `int_max_str_digits` and simplified the logic per @Y1hgs's
comments on the earlier revision.  Less code, less awkward, and simpler
to explain.

Underscores and the sign are uncounted because that makes for the
easiest implementation.
  • Loading branch information
gpshead committed Sep 2, 2022
commit fea25ea04afbed819c628a6bc2e4f406e7ee49b1
10 changes: 5 additions & 5 deletions Doc/c-api/init_config.rst
Original file line number Diff line number Diff line change
Expand Up @@ -828,13 +828,13 @@ PyConfig

Default: ``0``.

.. c:member:: int int_max_base10_digits
.. c:member:: int int_max_str_digits

If greater than 0, enable int digit limitation. ``-1`` means that
:data:`sys.int_info.default_max_base10_digits` will be used.
If greater than 0, enable int conversion digit limitations. ``-1`` means
that :data:`sys.int_info.default_max_str_digits` will be used.

Configured by the :option:`-X int_max_base10_digits <-X>` command line
flag or the :envvar:`PYTHONINTMAXBASE10DIGITS` environment varable.
Configured by the :option:`-X int_max_str_digits <-X>` command line
flag or the :envvar:`PYTHONINTMAXSTRDIGITS` environment varable.

Default: ``-1``.

Expand Down
10 changes: 6 additions & 4 deletions Doc/library/functions.rst
Original file line number Diff line number Diff line change
Expand Up @@ -911,10 +911,12 @@ are always available. They are listed here in alphabetical order.
The delegation to :meth:`__trunc__` is deprecated.

.. versionchanged:: 3.11
:class:`int` string inputs and string representation can be limited
to help avoid denial of service attacks. A :exc:`ValueError` is raised
when an input or string representation exceeds the limit. See :ref:`int
maximum digits limitation <int_max_base10_digits>` for more information.
:class:`int` string inputs and string representations can be limited to
help avoid denial of service attacks. A :exc:`ValueError` is raised when
the limit is exceeded while converting a string *x* to an :class:`int` or
when converting a :class:`int` into a string would exceed the limit. See
:ref:`int maximum digits limitation <int_max_str_digits>` for more
information.

.. function:: isinstance(object, classinfo)

Expand Down
2 changes: 1 addition & 1 deletion Doc/library/json.rst
Original file line number Diff line number Diff line change
Expand Up @@ -261,7 +261,7 @@ Basic Usage
.. versionchanged:: 3.11
The default implementation of *parse_int* limits the maximum length of
the integer string via the interpreter's :ref:`int maximum digits
limitation <int_max_base10_digits>` mechanism to help avoid denial of
limitation <int_max_str_digits>` mechanism to help avoid denial of
service attacks.

*parse_constant*, if specified, will be called with one of the following
Expand Down
57 changes: 27 additions & 30 deletions Doc/library/stdtypes.rst
Original file line number Diff line number Diff line change
Expand Up @@ -5460,7 +5460,7 @@ types, where they are relevant. Some of these are not reported by the
[<class 'bool'>]


.. _int_max_base10_digits:
.. _int_max_str_digits:

Integer maximum digits limitation
=================================
Expand All @@ -5474,55 +5474,52 @@ power of *2*. Even the best known algorithms for base *10* have sub-quadratic
complexity. Converting a large value such as ``int('1' * 500_000)`` can take
over a second on a fast CPU.

The limit value uses base 10 as a reference point and scales with base. That
means an :class:`int` conversion accepts longer strings for smaller bases and
shorter strings for larger bases. Underscores and the sign in strings don't
count towards the limit.
The limit value is based on the number of digit characters in the input or
output string. That means that higher bases can process larger numbers before
the limit triggers. Underscores and the sign are not counted towards the limit.

When an operation exceeds the limit, a :exc:`ValueError` is raised::

>>> sys.set_int_max_base10_digits(2048)
>>> sys.set_int_max_str_digits(2048)
>>> i = 10 ** 2047
>>> len(str(i))
2048
>>> i = 10 ** 2048
>>> len(str(i))
Traceback (most recent call last):
...
ValueError: exceeds maximum integer base 10 digit limit
ValueError: Exceeds digit limit for string conversions: value has 2049 digits.

Configuring the limit
---------------------

Before Python starts up you can use an environment variable or an interpreter
command line flag to configure the limit:

* :envvar:`PYTHONINTMAXBASE10DIGITS`, e.g.
``PYTHONINTMAXBASE10DIGITS=4321 python3`` to set the limit to ``4321`` or
``PYTHONINTMAXBASE10DIGITS=0 python3`` to disable the limitation.
* :option:`-X int_max_base10_digits <-X>`, e.g.
``python3 -X int_max_base10_digits=4321``
* :data:`sys.flags.int_max_base10_digits` contains the value of
:envvar:`PYTHONINTMAXBASE10DIGITS` or
:option:`-X int_max_base10_digits <-X>`. In case both the env var and the
``-X`` option are set, the ``-X`` option takes precedence. The value of
*-1* indicates that both were unset and the value of
:data:`sys.int_info.default_max_base10_digits` was used during initilization.
* :envvar:`PYTHONINTMAXSTRDIGITS`, e.g.
``PYTHONINTMAXSTRDIGITS=4321 python3`` to set the limit to ``4321`` or
``PYTHONINTMAXSTRDIGITS=0 python3`` to disable the limitation.
* :option:`-X int_max_str_digits <-X>`, e.g.
``python3 -X int_max_str_digits=4321``
* :data:`sys.flags.int_max_str_digits` contains the value of
:envvar:`PYTHONINTMAXSTRDIGITS` or :option:`-X int_max_str_digits <-X>`.
If both the env var and the ``-X`` option are set, the ``-X`` option takes
precedence. A value of *-1* indicates that both were unset, thus a value of
:data:`sys.int_info.default_max_str_digits` was used during initilization.

From code, you can inspect the current limit and set a new one using these
:mod:`sys` APIs:

* :func:`sys.get_int_max_base10_digits` and
:func:`sys.set_int_max_base10_digits` are a getter and setter for
the interpreter-wide limit. Subinterpreters have their own
limit.
* :func:`sys.get_int_max_str_digits` and :func:`sys.set_int_max_str_digits` are
a getter and setter for the interpreter-wide limit. Subinterpreters have
their own limit.

Information about the default and minimum can be found in :attr:`sys.int_info`:

* :data:`sys.int_info.default_max_base10_digits <sys.int_info>` is the
compiled-in default limit.
* :data:`sys.int_info.base10_digits_check_threshold <sys.int_info>` is the
minimum accepted value for the limit.
* :data:`sys.int_info.default_max_str_digits <sys.int_info>` is the compiled-in
default limit.
* :data:`sys.int_info.str_digits_check_threshold <sys.int_info>` is the minimum
accepted value for the limit.

Affected APIs
-------------
Expand All @@ -5549,17 +5546,17 @@ The limitations do not apply to functions with a linear algorithm:
Recommended configuration
-------------------------

The default :data:`sys.int_info.default_max_base10_digits` is expected to be
The default :data:`sys.int_info.default_max_str_digits` is expected to be
reasonable for most applications. If your application requires a different
limit, use Python version and implementation agnostic code to set it.

Example::

>>> import sys
>>> if hasattr(sys, "set_int_max_base10_digits"):
... current_limit = sys.get_int_max_base10_digits()
>>> if hasattr(sys, "set_int_max_str_digits"):
... current_limit = sys.get_int_max_str_digits()
... if not current_limit or current_limit > 4321:
... sys.set_int_max_base10_digits(4321)
... sys.set_int_max_str_digits(4321)


.. rubric:: Footnotes
Expand Down
28 changes: 13 additions & 15 deletions Doc/library/sys.rst
Original file line number Diff line number Diff line change
Expand Up @@ -521,7 +521,7 @@ always available.
:const:`dev_mode` :option:`-X dev <-X>` (:ref:`Python Development Mode <devmode>`)
:const:`utf8_mode` :option:`-X utf8 <-X>`
:const:`safe_path` :option:`-P`
:const:`int_max_base10_digits` :option:`-X int_max_base10_digits <-X>` (:ref:`int maximum digits limitation <int_max_base10_digits>`)
:const:`int_max_str_digits` :option:`-X int_max_str_digits <-X>` (:ref:`int maximum digits limitation <int_max_str_digits>`)
================================== ======================================================================================================

.. versionchanged:: 3.2
Expand All @@ -545,7 +545,7 @@ always available.
Added the ``safe_path`` attribute for :option:`-P` option.

.. versionchanged:: 3.11
Added the ``int_max_base10_digits`` attribute.
Added the ``int_max_str_digits`` attribute.


.. data:: float_info
Expand Down Expand Up @@ -727,10 +727,10 @@ always available.

.. versionadded:: 3.6

.. function:: get_int_max_base10_digits()
.. function:: get_int_max_str_digits()

Return current global value for :ref:`int maximum digits limitation
<int_max_base10_digits>`. See also :func:`set_int_max_base10_digits`
<int_max_str_digits>`. See also :func:`set_int_max_str_digits`

.. versionadded:: 3.11

Expand Down Expand Up @@ -1017,22 +1017,20 @@ always available.
| :const:`sizeof_digit` | size in bytes of the C type used to |
| | represent a digit |
+----------------------------------------+-----------------------------------------------+
| :const:`default_max_base10_digits` | default value for |
| | :func:`sys.get_int_max_base10_digits` when it |
| :const:`default_max_str_digits` | default value for |
| | :func:`sys.get_int_max_str_digits` when it |
| | is not otherwise explicitly configured. |
+----------------------------------------+-----------------------------------------------+
| :const:`base10_digits_check_threshold` | minimum non-zero value for |
| | :func:`sys.set_int_max_base10_digits`, |
| | :envvar:`PYTHONINTMAXBASE10DIGITS`, or |
| | :option:`-X int_max_base10_digits <-X>`. |
| | Supplied positive values less than this will |
| | be silently rounded up to this value. |
| :const:`str_digits_check_threshold` | minimum non-zero value for |
| | :func:`sys.set_int_max_str_digits`, |
| | :envvar:`PYTHONINTMAXSTRDIGITS`, or |
| | :option:`-X int_max_str_digits <-X>`. |
+----------------------------------------+-----------------------------------------------+

.. versionadded:: 3.1

.. versionchanged:: 3.11
Added ``default_max_base10_digits`` and ``base10_digits_check_threshold``.
Added ``default_max_str_digits`` and ``str_digits_check_threshold``.


.. data:: __interactivehook__
Expand Down Expand Up @@ -1333,10 +1331,10 @@ always available.

.. availability:: Unix.

.. function:: set_int_max_base10_digits(n)
.. function:: set_int_max_str_digits(n)

Set global interpreter limit for :ref:`int maximum digits limitation
<int_max_base10_digits>`. See also :func:`get_int_max_base10_digits`
<int_max_str_digits>`. See also :func:`get_int_max_str_digits`

.. versionadded:: 3.11

Expand Down
10 changes: 5 additions & 5 deletions Doc/using/cmdline.rst
Original file line number Diff line number Diff line change
Expand Up @@ -505,8 +505,8 @@ Miscellaneous options
stored in a traceback of a trace. Use ``-X tracemalloc=NFRAME`` to start
tracing with a traceback limit of *NFRAME* frames. See the
:func:`tracemalloc.start` for more information.
* ``-X int_max_base10_digits`` configures :ref:`int maximum digits limitation
<int_max_base10_digits>`. See also :envvar:`PYTHONINTMAXBASE10DIGITS`.
* ``-X int_max_str_digits`` configures :ref:`int maximum digits limitation
<int_max_str_digits>`. See also :envvar:`PYTHONINTMAXSTRDIGITS`.
* ``-X importtime`` to show how long each import takes. It shows module
name, cumulative time (including nested imports) and self time (excluding
nested imports). Note that its output may be broken in multi-threaded
Expand Down Expand Up @@ -585,7 +585,7 @@ Miscellaneous options
The ``-X frozen_modules`` option.

.. versionadded:: 3.11
The ``-X int_max_base10_digits`` option.
The ``-X int_max_str_digits`` option.

.. versionadded:: 3.12
The ``-X perf`` option.
Expand Down Expand Up @@ -768,10 +768,10 @@ conflict.

.. versionadded:: 3.2.3

.. envvar:: PYTHONINTMAXBASE10DIGITS
.. envvar:: PYTHONINTMAXSTRDIGITS

If this variable is set to an integer, it is used to configure the interpreter's
global :ref:`int maximum digits limitation <int_max_base10_digits>`.
global :ref:`int maximum digits limitation <int_max_str_digits>`.

.. versionadded:: 3.11

Expand Down
2 changes: 1 addition & 1 deletion Include/cpython/initconfig.h
Original file line number Diff line number Diff line change
Expand Up @@ -178,7 +178,7 @@ typedef struct PyConfig {
wchar_t *check_hash_pycs_mode;
int use_frozen_modules;
int safe_path;
int int_max_base10_digits; // NOTE(gpshead): do not backport to stable releases due to struct change.
int int_max_str_digits; // NOTE(gpshead): do not backport to stable releases due to struct change.

/* --- Path configuration inputs ------------ */
int pathconfig_warnings;
Expand Down
2 changes: 1 addition & 1 deletion Include/internal/pycore_interp.h
Original file line number Diff line number Diff line change
Expand Up @@ -176,7 +176,7 @@ struct _is {
struct types_state types;
struct callable_cache callable_cache;

int int_max_base10_digits;
int int_max_str_digits;

/* The following fields are here to avoid allocation during init.
The data is exposed through PyInterpreterState pointer fields.
Expand Down
10 changes: 5 additions & 5 deletions Include/internal/pycore_long.h
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@ extern "C" {
*
* 2000 decimal digits fits a ~6643 bit number.
*/
#define _PY_LONG_DEFAULT_MAX_BASE10_DIGITS 2000
#define _PY_LONG_DEFAULT_MAX_STR_DIGITS 2000
/*
* Threshold for max digits check. For performance reasons int() and
* int.__str__ don't checks values that are smaller than this
Expand All @@ -32,11 +32,11 @@ extern "C" {
*
* 333 decimal digits fits a ~1106 bit number.
*/
#define _PY_LONG_MAX_BASE10_DIGITS_THRESHOLD 333
#define _PY_LONG_MAX_STR_DIGITS_THRESHOLD 333

#if ((_PY_LONG_DEFAULT_MAX_BASE10_DIGITS != 0) && \
(_PY_LONG_DEFAULT_MAX_BASE10_DIGITS < _PY_LONG_MAX_BASE10_DIGITS_THRESHOLD))
# error "_PY_LONG_DEFAULT_MAX_BASE10_DIGITS smaller than threshold."
#if ((_PY_LONG_DEFAULT_MAX_STR_DIGITS != 0) && \
(_PY_LONG_DEFAULT_MAX_STR_DIGITS < _PY_LONG_MAX_STR_DIGITS_THRESHOLD))
# error "_PY_LONG_DEFAULT_MAX_STR_DIGITS smaller than threshold."
#endif


Expand Down
10 changes: 5 additions & 5 deletions Lib/test/support/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -2344,11 +2344,11 @@ def sleeping_retry(timeout, err_msg=None, /,


@contextlib.contextmanager
def set_int_max_base10_digits(max_digits):
"""Temporarily change the integer maximum base 10 digits limit."""
current = sys.get_int_max_base10_digits()
def set_int_max_str_digits(max_digits):
"""Temporarily change the int<->str maximum digits limit."""
current = sys.get_int_max_str_digits()
try:
sys.set_int_max_base10_digits(max_digits)
sys.set_int_max_str_digits(max_digits)
yield
finally:
sys.set_int_max_base10_digits(current)
sys.set_int_max_str_digits(current)
30 changes: 15 additions & 15 deletions Lib/test/test_cmd_line.py
Original file line number Diff line number Diff line change
Expand Up @@ -866,36 +866,36 @@ def test_parsing_error(self):
self.assertTrue(proc.stderr.startswith(err_msg), proc.stderr)
self.assertNotEqual(proc.returncode, 0)

def test_int_max_base10_digits(self):
code = "import sys; print(sys.flags.int_max_base10_digits, sys.get_int_max_base10_digits())"
def test_int_max_str_digits(self):
code = "import sys; print(sys.flags.int_max_str_digits, sys.get_int_max_str_digits())"

assert_python_failure('-X', 'int_max_base10_digits', '-c', code)
assert_python_failure('-X', 'int_max_base10_digits=foo', '-c', code)
assert_python_failure('-X', 'int_max_base10_digits=100', '-c', code)
assert_python_failure('-X', 'int_max_str_digits', '-c', code)
assert_python_failure('-X', 'int_max_str_digits=foo', '-c', code)
assert_python_failure('-X', 'int_max_str_digits=100', '-c', code)

assert_python_failure('-c', code, PYTHONINTMAXBASE10DIGITS='foo')
assert_python_failure('-c', code, PYTHONINTMAXBASE10DIGITS='100')
assert_python_failure('-c', code, PYTHONINTMAXSTRDIGITS='foo')
assert_python_failure('-c', code, PYTHONINTMAXSTRDIGITS='100')

def res2int(res):
out = res.out.strip().decode("utf-8")
return tuple(int(i) for i in out.split())

res = assert_python_ok('-c', code)
self.assertEqual(res2int(res), (-1, sys.get_int_max_base10_digits()))
res = assert_python_ok('-X', 'int_max_base10_digits=0', '-c', code)
self.assertEqual(res2int(res), (-1, sys.get_int_max_str_digits()))
res = assert_python_ok('-X', 'int_max_str_digits=0', '-c', code)
self.assertEqual(res2int(res), (0, 0))
res = assert_python_ok('-X', 'int_max_base10_digits=4000', '-c', code)
res = assert_python_ok('-X', 'int_max_str_digits=4000', '-c', code)
self.assertEqual(res2int(res), (4000, 4000))
res = assert_python_ok('-X', 'int_max_base10_digits=100000', '-c', code)
res = assert_python_ok('-X', 'int_max_str_digits=100000', '-c', code)
self.assertEqual(res2int(res), (100000, 100000))

res = assert_python_ok('-c', code, PYTHONINTMAXBASE10DIGITS='0')
res = assert_python_ok('-c', code, PYTHONINTMAXSTRDIGITS='0')
self.assertEqual(res2int(res), (0, 0))
res = assert_python_ok('-c', code, PYTHONINTMAXBASE10DIGITS='4000')
res = assert_python_ok('-c', code, PYTHONINTMAXSTRDIGITS='4000')
self.assertEqual(res2int(res), (4000, 4000))
res = assert_python_ok(
'-X', 'int_max_base10_digits=6000', '-c', code,
PYTHONINTMAXBASE10DIGITS='4000'
'-X', 'int_max_str_digits=6000', '-c', code,
PYTHONINTMAXSTRDIGITS='4000'
)
self.assertEqual(res2int(res), (6000, 6000))

Expand Down
Loading