I don't understand why pytest's collection is so slow.
On our test suite (big django app) it takes about 15s to collect tests. So much that we added a util using ripgrep to find the file and pass it as an argument to pytest when using `pytest -k <testname>`.
boxed 17 minutes ago [-]
I've done some work on making pytest faster, and it's mostly a case of death by a thousand paper cuts. I wrote hammett as an experimental benchmark to compare to.
kinow 2 hours ago [-]
In their case I think they were no specifying any test path. Which would cause pytest to search or tests in multiple directories.
Another thing that can slow down pytest collection and bootstrap is how fixture are loaded. So reducing number or scope of fixtures may help too.
piokoch 2 hours ago [-]
Ehhh, those pesky Python people, complaining and complaining, average Spring Boot application takes 15s to start even looking if the code compiled ;)
thom 2 hours ago [-]
Lest we start to malign the JVM as a whole, my Clojure test suite, which includes functional tests running headless browsers against a full app hitting real Postgres databases, runs end to end in 20s.
throwme_123 1 hours ago [-]
Is Trail of Bits transitioning out of "crypto"?
Imho, they are one of the best auditors out there for smart contracts. Wouldn't be surprising to see some of these talented teams find bigger markets.
ustad 4 hours ago [-]
The article uses pytest - does anyone have similar tips when using pythons builtin unittest?
masklinn 3 hours ago [-]
The sys.monitoring and import optimisation suggestions apply as-is.
If you use standard unittest discovery the third item might apply as well, though probably not to the same degree.
I don’t think unittest has any support for distribution so the xdist stuff is a no.
On the other hand you could use unit test as the API with Pytest as your test runner. Then you can also use xdist. And eventually migrate to the Pytest test api because it’s so much better.
kinow 2 hours ago [-]
I wwsn't familiar with this sys.monitoring option for coverage. Going to give it a try in my test suite. At the moment with docker testcontainers, gh actions test matrix for multiple python versions, and unit + regression + integration tests it is taking about 3-5 minutes.
darkamaul 2 hours ago [-]
Warning, there is a change in coverage 7.7.0 that disables sysmon support for coverage if using branch coverage _and_ a version of Python before 3.14alpha6.
I profiled a huge legacy tests collection using cProfile, and found lots of low hanging fruits. Like some tests were creating 4000x3000 Pillow image in memory just to test how image saving code works (checkign that filename and extension are correct). And hundreds of tests had created this huge image for every test (in the setUp method) because of unittest reliance on inheritance. Reducing size image to 10x5 made the test suite faster for like 5-7% (it was long time ago, so I don't remember exact statistics).
So, I'd run the tests under cProfile first.
bgwalter 42 minutes ago [-]
I get that pytest has features that unittest does not, but how is scanning for test files in a directory considered appropriate for what is called a high security application in the article?
For high security applications the test suite should be boring and straightforward. pytest is full of magic, which makes it so slow.
Python in general has become so complex, informally specified and bug ridden that it only survives because of AI while silencing critics in their bubble.
The complexity includes PSF development processes, which lead to:
I don't disagree that it's "complex, informally specified" (idk about bug ridden or silencing critics), but it's just silly to say it only survives because of AI. It was a top-used language before AI got big for web development, data science and all sorts of scientific analysis, and these haven't gone away: I don't expect Python lost much ground in these fields, if any.
bgwalter 5 minutes ago [-]
Dropbox moved parts from Python to Golang already in 2014. Google fired the Python team last year and I hear that it does not use Python for new code. Instagram is kept afloat by gigantic hacks.
The scientific ecosystem was always there, but relied on heavy marketing to academics, who (sadly) in turn indoctrinate new students to use Python as a first language.
I did forget about sysadmin use cases in Linux distributions, but they could be easily replaced by even Perl, as leaner BSD distributions already do.
On our test suite (big django app) it takes about 15s to collect tests. So much that we added a util using ripgrep to find the file and pass it as an argument to pytest when using `pytest -k <testname>`.
Another thing that can slow down pytest collection and bootstrap is how fixture are loaded. So reducing number or scope of fixtures may help too.
Imho, they are one of the best auditors out there for smart contracts. Wouldn't be surprising to see some of these talented teams find bigger markets.
If you use standard unittest discovery the third item might apply as well, though probably not to the same degree.
I don’t think unittest has any support for distribution so the xdist stuff is a no.
On the other hand you could use unit test as the API with Pytest as your test runner. Then you can also use xdist. And eventually migrate to the Pytest test api because it’s so much better.
[0]: https://coverage.readthedocs.io/en/7.8.0/changes.html#versio...
So, I'd run the tests under cProfile first.
For high security applications the test suite should be boring and straightforward. pytest is full of magic, which makes it so slow.
Python in general has become so complex, informally specified and bug ridden that it only survives because of AI while silencing critics in their bubble.
The complexity includes PSF development processes, which lead to:
https://www.schneier.com/blog/archives/2024/08/leaked-github...
I don't disagree that it's "complex, informally specified" (idk about bug ridden or silencing critics), but it's just silly to say it only survives because of AI. It was a top-used language before AI got big for web development, data science and all sorts of scientific analysis, and these haven't gone away: I don't expect Python lost much ground in these fields, if any.
The scientific ecosystem was always there, but relied on heavy marketing to academics, who (sadly) in turn indoctrinate new students to use Python as a first language.
I did forget about sysadmin use cases in Linux distributions, but they could be easily replaced by even Perl, as leaner BSD distributions already do.