[Bug] Support for Python 3.12 - Missing distutils

[jessek]

Expected Behavior

The library should function under Python3.12

Actual Behavior

The library attempts to load the distutils module, which was removed in Python3.12. See https://docs.python.org/3.10/library/distutils.html for background on the deprecation and removal of distutils.

Steps to Reproduce the Problem

1. Install mitreattack-python with python3 -m pip install mitreattack-python
2. Run python3 and attempt to load library:

% python3 --version
Python 3.12.4

% python3
Python 3.12.4 (main, Jun  6 2024, 18:26:44) [Clang 15.0.0 (clang-1500.3.9.4)] on darwin
Type "help", "copyright", "credits" or "license" for more information.

>>> from mitreattack.navlayers.core import Layer

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/jessek/tmp/lib/python3.12/site-packages/mitreattack/__init__.py", line 3, in <module>
    from .collections import *
  File "/Users/jessek/tmp/lib/python3.12/site-packages/mitreattack/collections/__init__.py", line 3, in <module>
    from .stix_to_collection import *
  File "/Users/jessek/tmp/lib/python3.12/site-packages/mitreattack/collections/stix_to_collection.py", line 9, in <module>
    from stix2elevator.stix_stepper import step_bundle
  File "/Users/jessek/tmp/lib/python3.12/site-packages/stix2elevator/__init__.py", line 12, in <module>
    from stix.core import STIXPackage
  File "/Users/jessek/tmp/lib/python3.12/site-packages/stix/__init__.py", line 5, in <module>
    from .base import (Entity, EntityList, TypedCollection, TypedList,  # noqa
  File "/Users/jessek/tmp/lib/python3.12/site-packages/stix/base.py", line 20, in <module>
    from . import utils
  File "/Users/jessek/tmp/lib/python3.12/site-packages/stix/utils/__init__.py", line 382, in <module>
    from .parser import *  # noqa
    ^^^^^^^^^^^^^^^^^^^^^
  File "/Users/jessek/tmp/lib/python3.12/site-packages/stix/utils/parser.py", line 7, in <module>
    import mixbox.parser
  File "/Users/jessek/tmp/lib/python3.12/site-packages/mixbox/parser.py", line 5, in <module>
    from distutils.version import StrictVersion
ModuleNotFoundError: No module named 'distutils'

>>>

Possible Solution

This may be related to #163, and removing support for STIX1 will remove the dependency.

I am also experiencing this issue, is there any ETA for a fix?

[billfitzgerald]

Would appreciate feedback on this -

I'm working locally exploring the dataset/tooling in this module and also encountered this issue.

I mitigated it by installing setuptools in the virtual environment where I'm working:

pip install setuptools

The other option was to create a virtual environment using an older version ( lesser than 3.12) of Python.

Is there a preferred way from the perspective of the maintainers to mitigate this?

[jondricek]

Maintainer here. Hello!

Personally I would like nothing more than to update this to work on Python 3.12, but my main focus has not been on that functionality yet. We are semi-stuck on Python 3.10 for a completely unrelated issue, but let me try to connect some dots.

• mitreattack-python (this library) is a dependency of the ATT&CK website (attack.mitre.org) in that we build that website using python, and part of that process uses this library
• Years ago we began using pyScss for doing some CSS magic with our jinja templates. This third party library broke in Python 3.11. We are tracking the issue in our website repository here: Should support python 3.11 onward attack-website#470
  • The fix for the Python 3.11 error was merged in to the master branch in August 2023: fix error: global flags not at the start of the expression at position 1 Kronuz/pyScss#426
  • But the master branch was never pushed to PyPI
  • pyScss is used in an area of the ATT&CK website code that hasn't been touched in a few years, so its scary...
• Considering that mitreattack-python is independent of that website build process, we just need to make sure that this continues to work for both Python 3.10 as well as the newest version of Python (3.12 right now), which should be do-able, but once again we're a bit time constrained in how we focus our efforts.

The issue is not STIX 1 support (#163). The script that is modified in PR #181 (stix_to_collection.py) uses the stix2elevator library (which is capable of STIX1 > STIX 2 conversion as I understand) to upgrade STIX 2.0 to STIX 2.1. Which is something that we do internally with some other tools/processes (related to https://github.com/center-for-threat-informed-defense/attack-workbench-frontend, but that's a whole separate topic). So because we have a different process, we don't use this functionality ourselves. So I guess that's a roundabout way of saying we can get rid of that script entirely.

I will work on this this week, and @stemarks thank you for the PR in the first place. I hope there are no hard feelings if I end up closing the PR and implement the same fix in a slightly different way as I rip out that and potentially some other content as well that we don't use, and using the cover of a breaking change might take the opportunity to do that in a release this week.

[billfitzgerald]

@jondricek - this is all super helpful and informative - thank you!

Based on this, my thinking is that the best short to medium term fix is to create a virtual environment that uses either 3.10 or 3.11 - is that accurate?

While installing setuptools "fixes" the issue, my read is that using v3.10 or 3.11 will provide a more consistent experience until this is addressed in a systematic way - would you agree?

@jondricek no hard feelings at all, I totally understand. I modified this as we needed it to work with 3.12 so thought I would share what I had done in case it helped you guys time wise, and also in case the solution helps anyone else out they can see my changes 👍

[jondricek]

@billfitzgerald yes - i recommend 3.10 for the time being due to the legacy reason above, but i think it should work in 3.11 as well. Honestly, if installing setuptools in the 3.12 environment fixes it for you, that should be fine - however i realize it is, i'll just say obnoxious from our end to force someone to do that. (But i would totally do that in a testing environment on my own machine if I needed to)

@stemarks thanks for lighting a fire under me to do our own version of PEP 594: Removing Dead Batteries (https://peps.python.org/pep-0594/). And thanks for sharing what you did to make it work!

[billfitzgerald]

@jondricek - super helpful, thank you!

RE: "obnoxious from our end"

I mean, it's 2024 - if running a one line command is the worst part of my day, it's an awesome day! :)

Thank you for your input here - very much appreciated.

[tsahil-tencyle]

@jondricek
Hi guys,
we're facing pretty much the same issue with stix2-elevator dependencies.
somewhere down the line they use "weakrefmethod" which breaks setuptools newer versions requirements (they enforces key naming in setup.cfg which weakrefmethod doesn't have)

any update on when (and if) this library is going to be removed?

[jondricek]

Back in my comment in November I mentioned

• pyScss is used in an area of the ATT&CK website code that hasn't been touched in a few years, so its scary...

Long story short, that kicked off a CSS investigation on our team which culminated in a fix that no longer uses that python library anymore. It has been on our development branch for the website for the past month and I think it feels stable by now. So the way I see this issue closing out, along with a timeline is:

• This week (by Friday March 28): identify if we can merge develop into main on the website repository (stretch goal - complete the merge!)
• March 31-April 4: Having merged the website branch, removing the pyScss dependency, ensure that the website repo builds perfectly fine with Python 3.13, updating libraries as needed
• April 7-11: Update mitreattack-python, dropping the stix2elevator dependency, as well as the script that uses it, since it is no longer used.
• By April 11: release updated version of mitreattack-python to PyPI!

Hopefully all this can be done even sooner than this proposed timeline, but I want to be realistic with my team's time and commitments to ongoing efforts as well. Hopefully this sheds some light on how this can unfold!

[tsahil-tencyle]

thanks for the prompt reply
appreciate the effort :)