Hi wave I ve been experiencing `pylint` crash whenever I imp Outerbounds #ask-metaflow

Hi :wave: I’ve been experiencing `pylint` crash wh...

late-xylophone-98814

04/18/2023, 5:10 PM

Hi 👋 I’ve been experiencing

pylint

crash whenever I import

janitor

in a metaflow. A minimum example is shown below:

Copy code

import janitor
from metaflow import FlowSpec, step

class LinearFlow(FlowSpec):

    @step
    def start(self):
        self.my_var = 'hello world'
        self.next(self.a)

    @step
    def a(self):
        print('the data artifact is: %s' % self.my_var)
        self.next(self.end)

    @step
    def end(self):
        print('the data artifact is still: %s' % self.my_var)

if __name__ == '__main__':
    LinearFlow()

… and I will get the error:

Copy code

Metaflow 2.7.18 executing LinearFlow for user:jixux
Validating your flow...
    The graph looks good!
Running pylint...
    linear_flow.py:1:0: F0002: /local/home/jixux/test/metaflow/linear_flow.py: Fatal error while checking '/local/home/jixux/test/metaflow/linear_flow.py'. Please open an issue in our bug tracker so we address this. There is a pre-filled template that you can use in '/local/home/jixux/.cache/pylint/pylint-crash-2023-04-18-16-46-18.txt'. (astroid-error)
    Pylint is not happy:
    Fix Pylint warnings listed above or say --no-pylint.

Apparently I can use --no-pylint to override this crash but I really miss the pylint capability. Any suggestions would be helpful~ (I also reported this as a bug in the pylint community)

✅ 1

late-xylophone-98814

04/18/2023, 5:15 PM

Strangely, when I directly run pylint on the flow there’s no crash:

Copy code

# output of `pylint linear_flow.py`:
************* Module linear_flow
linear_flow.py:1:0: C0114: Missing module docstring (missing-module-docstring)
linear_flow.py:4:0: C0115: Missing class docstring (missing-class-docstring)
linear_flow.py:7:4: C0116: Missing function or method docstring (missing-function-docstring)
linear_flow.py:12:4: C0116: Missing function or method docstring (missing-function-docstring)
linear_flow.py:12:4: C0103: Method name "a" doesn't conform to snake_case naming style (invalid-name)
linear_flow.py:13:14: C0209: Formatting a regular string which could be a f-string (consider-using-f-string)
linear_flow.py:17:4: C0116: Missing function or method docstring (missing-function-docstring)
linear_flow.py:18:14: C0209: Formatting a regular string which could be a f-string (consider-using-f-string)
linear_flow.py:8:8: W0201: Attribute 'my_var' defined outside __init__ (attribute-defined-outside-init)
linear_flow.py:1:0: W0611: Unused import janitor (unused-import)

------------------------------------------------------------------
Your code has been rated at 2.31/10 (previous run: 1.54/10, +0.77)

ancient-application-36103

04/18/2023, 5:49 PM

does the error present itself when you comment out the janitor import? trying to make sure that this isn't an issue with your pylint installation

late-xylophone-98814

04/18/2023, 5:49 PM

the error is gone when I comment out

import janitor

late-xylophone-98814

04/18/2023, 5:51 PM

here are my pylint version:

Copy code

pylint 2.17.2
astroid 2.15.3
Python 3.10.0 (default, Mar  3 2022, 09:58:08) [GCC 7.5.0]

late-xylophone-98814

04/18/2023, 5:54 PM

not knowing the details on how metaflow invoke pylint, does the pylint invocation actually dig into the source code of each import?

ancient-application-36103

04/18/2023, 5:59 PM

Nope the pypi imports are not recursively evaluated

late-xylophone-98814

04/18/2023, 6:00 PM

interesting… so the literal difference between the crashing and working version is the single line of

import janitor

4 Views

Open in Slack

Previous Next