https://outerbounds.com/ logo
Join Slack
Powered by
Hello! getting `illegal core dumped` ``` bash: lin...
# ask-metaflow
f
Hello! getting 
illegal core dumped
Copy code
bash: line 1:    37 Illegal instruction     (core dumped) ( python driver.py --with retry:times=3,minutes_between_retries=2 --quiet --metadata=service --environment=local --datastore=s3 --datastore-root=<s3://mtflw-s3-ptlp-prod/metaflow> --event-logger=nullSidecarLogger --monitor=nullSidecarMonitor --no-pylint --with=step_functions_internal step train_model --run-id sfn-$METAFLOW_RUN_ID --task-id ${AWS_BATCH_JOB_ID} --retry-count $((AWS_BATCH_JOB_ATTEMPT-1)) --max-user-code-retries 3 --input-paths sfn-${METAFLOW_RUN_ID}/create_ids/${METAFLOW_PARENT_TASK_ID} ) >> >(python -m metaflow.mflog.tee task $MFLOG_STDOUT) 2>> >(python -m metaflow.mflog.tee task $MFLOG_STDERR >&2)
any idea why?
✅ 1
The workflow was running fine last week. One change that has happened is metaflow version. Can anyone please check if this is recurring issue on the new version?
v
that's a peculiar error. Does it fail immediately or do you see any logs before it crashes?
most likely that would be caused by some other library used in your code
also it could happen if the underlying instances change, e.g. from AMD to Intel, maybe ARM. Has anything changed in your compute environment?
f
Yeah it works when we build it from AMD machine.
I still don’t understand. AMD and Intel both of them are using x86_64 architecture. Why will it matter then? If the CPU architecture from both the company is same, what difference comes in play when we built the images on Intel and run the container on AMD?
v
there are instructions that only Intel supports (e.g. AMX). If your image contains libraries that are compiled to use these instructions (or other Intel-specific stuff), it would produce 
Illegal Instruction
when executed on AMD
I don't know if this is what's happening in your case, but it's one possible explanation
f
So, if I build on AMD machine, then this extension will be missing while building. Then, it would work on Intel machine, right? If that is so, then it will explain why it worked when built on AMD machine and ran on Intel machine. But thanks. I think this is the reason.
👍 1
v
most likely AMD -> Intel would work better
5 Views
Open in Slack
PreviousNext
Powered by