Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[CLI]: wandb.Image breaks plt.imshow #6297

Open
csparker247 opened this issue Sep 15, 2023 · 7 comments · May be fixed by #7279
Open

[CLI]: wandb.Image breaks plt.imshow #6297

csparker247 opened this issue Sep 15, 2023 · 7 comments · May be fixed by #7279
Labels
a:cli Area: Client c:media

Comments

@csparker247
Copy link

csparker247 commented Sep 15, 2023

Describe the bug

I have a script where I log an image using wandb.Image then optionally use matplotlib to plot points on top of the same image. Since point plotting is optional, I import matplotlib inside my point plotting method. In practice, this looks like something like this:

def plot_points(img, pts, step):
  import matplotlib.pyplot as plt
  fig, ax = plt.subplots()
  plt.imshow(img)
  plt.scatter(pts[..., 0], pts[..., 1], marker='.')
  wandb.log({'Plot': plt}, step=step)

wimg = wandb.Image(img, caption='Image')
wandb.log({'Image': wimg}, step=step)
if log_pts:
  plot_points(img, pts, step)

When I run this code on my laptop, this works just fine. However, when I run it on my university's computing cluster, I get AttributeError: module 'PIL' has no attribute 'Image' at the plt.imshow call.

I have no idea why this doesn't fail on all of my machines, but I've tracked the issue down to the import order for matplotlib.pyplot. I have no issues if I import pyplot before using wandb.Image, but I get the AttributeError if I import it afterwards. The following is a minimal code example which reproduces the issue:

import numpy as np
import wandb

# import before calling wandb.Image and all is fine 
# import matplotlib.pyplot as plt

wandb.init()

step=0
img = np.zeros((768, 768))

wimg = wandb.Image(img)
wandb.log({'Plot': wimg}, step=step)

import matplotlib.pyplot as plt
fig, ax = plt.subplots()
plt.imshow(img, cmap='gray')
wandb.log({'Image Plot': plt}, step=step)
# paths redacted
Traceback (most recent call last):
  File "/home/.../minimal.py", line 18, in <module>
    plt.imshow(img, cmap='gray')
  File "/usr/local/.../.venv/lib/python3.10/site-packages/matplotlib/pyplot.py", line 3346, in imshow
    __ret = gca().imshow(
  File "/usr/local/.../.venv/lib/python3.10/site-packages/matplotlib/__init__.py", line 1465, in inner
    return func(ax, *map(sanitize_sequence, args), **kwargs)
  File "/usr/local/.../.venv/lib/python3.10/site-packages/matplotlib/axes/_axes.py", line 5751, in imshow
    im.set_data(X)
  File "/usr/local/.../.venv/lib/python3.10/site-packages/matplotlib/image.py", line 721, in set_data
    if isinstance(A, PIL.Image.Image):
AttributeError: module 'PIL' has no attribute 'Image'

The obvious workaround is to just import pyplot at the start of my script, and I'll probably do that now, but I ultimately want to avoid this since point plotting is optional functionality.

Additional Files

No response

Environment

WandB version: 0.15.10

OS: Ubuntu 22.04 (actually the Docker image nvidia/cuda:11.8.0-cudnn8-devel-ubuntu22.04)

Python version: 3.10.12

Versions of relevant libraries:

  • matplotlib 3.8.0
  • numpy 1.25.2
  • Pillow 10.0.0

I do not have the issue on my laptop with the following configuration:
WandB version: 0.15.10
OS: macOS Ventura 13.5.1 (22G90)
Python version: 3.10.13 (Homebrew)
matplotlib 3.8.0
Pillow 10.0.0

Additional Context

No response

@csparker247 csparker247 added the a:cli Area: Client label Sep 15, 2023
@csparker247
Copy link
Author

csparker247 commented Sep 15, 2023

Looking into this a little bit, I suspect that the issue here might be the lazy loading of PIL.Image:

pil_image = util.get_module(
"PIL.Image",
required='wandb.Image needs the PIL package. To get it, run "pip install pillow".',
)

Copy link

sydholl commented Sep 15, 2023

WandB Internal User commented:
csparker247 commented:
Looking into this a little bit, I suspect that the issue here might be the lazy loading of PIL.Image:

https://github.com/wandb/wandb/blob/00aed12f84d031e59c98691eef8f69d05e1a7e59/wandb/sdk/data_types/image.py#L274C34-L274C34

@csparker247 csparker247 changed the title [CLI]: wand.Image breaks plt.imshow [CLI]: wandb.Image breaks plt.imshow Sep 16, 2023
@rsanandres-wandb
Copy link
Contributor

Hello @csparker247 !

Thank you for providing the detailed bug report! Since this seems to transient in nature, I was also unable to replicate this issue so would you be able to provide the debug.log and debug-internal.log for the SDK bug report I will be creating?

They should be located in the wandb folder in the same directory as where the script was run. The wandb folder has folders formatted as run-DATETIME-ID associated with a single run. Could you retrieve the debug.log and debug-internal.log files from one of these folders specifically from the run that is having issues?

@csparker247
Copy link
Author

Hi @rsanandres-wandb,

Here you go! These are from a new run I just performed using the minimal example above. Note that I went through and redacted some of the sensitive paths, usernames, etc.

debug-internal.log
debug.log

@rsanandres-wandb
Copy link
Contributor

Thank you very much! I will continue with making the bug report with this information.

@csparker247
Copy link
Author

csparker247 commented Sep 22, 2023

For reproducibility's sake, I'm running everything inside a Singularity (now Apptainer) container. The container definition file below is sufficient to reproduce the issue. You run my minimal example like so:

sudo singularity build minimal.sif minimal.def
singularity run minimal.sif python minimal.py

And the contents of minimal.def:

Bootstrap: docker
From: nvidia/cuda:11.8.0-cudnn8-devel-ubuntu22.04

%post -c /bin/bash
  # Basic installs
  apt update
  DEBIAN_FRONTEND=noninteractive apt install -y --no-install-recommends \
      curl \
      git \
      nano \
      tzdata \
      vim \
      python3 \
      python3-venv \
      cuda-compat-11-8
  curl https://bootstrap.pypa.io/get-pip.py | python3

  # Create a venv for this project
  python3 -m venv /usr/local/test/.venv/

  # Activate virtual env
  source /usr/local/test/.venv/bin/activate

  # Update pip
  python3 -m pip install --upgrade pip wheel setuptools

  # Install pyleaf
  python3 -m pip install matplotlib>=3.8 numpy wandb

  # Make writable in overlays
  chmod --recursive a+rw /usr/local/

%runscript
#!/bin/bash
  if [ $# -lt 1 ]; then
      echo "Usage: ./container <command>"
      exit 1
  fi

  source /usr/local/test/.venv/bin/activate
  exec "$@"

@rsanandres-wandb
Copy link
Contributor

Thank you again! I have added this to current bug report.

@soumik12345 soumik12345 linked a pull request Apr 2, 2024 that will close this issue
1 task
@kptkin kptkin added the c:media label Apr 17, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
a:cli Area: Client c:media
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants