Bug: Error on adding recipe from hellofresh #424

Karamellwuerfel · 2024-04-04T09:28:23Z

Is this urgent?

No

What parts are affected

Backend

What is the server version

93

What is the client version

0.4.21

What platform are you using

Linux

What's the problem 🤔

Hi, when I try to import a recipe from hellofresh, a 500 error occurs and it seems to have a problem in the python backend code (see log).

HelloFresh Link to import: https://www.hellofresh.de/recipes/torchietti-in-tomaten-paprika-sosse-5fc66ae618d71c10e64e51d4

Thanks!

Share your logs

[04/Apr/2024:09:21:33 +0000] "POST /api/household/1/recipe/scrape HTTP/2.0" 500 20 "https://kitchen.[secret].de/household/1/recipes" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:124.0) Gecko/20100101 Firefox/124.0"

------

[2024-04-04 09:25:17,040] ERROR in config: argument of type 'NoneType' is not iterable
Traceback (most recent call last):
  File "/opt/venv/lib/python3.11/site-packages/flask/app.py", line 867, in full_dispatch_request
    rv = self.dispatch_request()
         ^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.11/site-packages/flask/app.py", line 852, in dispatch_request
    return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.11/site-packages/flask_jwt_extended/view_decorators.py", line 170, in decorator
    return current_app.ensure_sync(fn)(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/kitchenowl/app/helpers/authorize_household.py", line 26, in decorator
    return func(*args, **kwargs)  # case server admin
           ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/kitchenowl/app/helpers/validate_args.py", line 26, in func_wrapper
    return func(arguments, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/kitchenowl/app/controller/recipe/recipe_controller.py", line 202, in scrapeRecipe
    recipe.name = scraper.title()
                  ^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.11/site-packages/recipe_scrapers/plugins/exception_handling.py", line 64, in decorated_method_wrapper
    return decorated(self, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.11/site-packages/recipe_scrapers/plugins/html_tags_stripper.py", line 75, in decorated_method_wrapper
    decorated_func_result = decorated(self, *args, **kwargs)
                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.11/site-packages/recipe_scrapers/plugins/normalize_string.py", line 34, in decorated_method_wrapper
    return normalize_string(decorated(self, *args, **kwargs))
                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.11/site-packages/recipe_scrapers/plugins/schemaorg_fill.py", line 48, in decorated_method_wrapper
    return decorated(self, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.11/site-packages/recipe_scrapers/hellofresh.py", line 14, in title
    return self.schema.title()
           ^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.11/site-packages/recipe_scrapers/_schemaorg.py", line 103, in title
    return normalize_string(self.data.get("name"))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.11/site-packages/recipe_scrapers/_utils.py", line 144, in normalize_string
    unescaped_string = html.unescape(string)
                       ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/html/__init__.py", line 130, in unescape
    if '&' not in s:
       ^^^^^^^^^^^^
TypeError: argument of type 'NoneType' is not iterable
ERROR:app.config:argument of type 'NoneType' is not iterable
Traceback (most recent call last):
  File "/opt/venv/lib/python3.11/site-packages/flask/app.py", line 867, in full_dispatch_request
    rv = self.dispatch_request()
         ^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.11/site-packages/flask/app.py", line 852, in dispatch_request
    return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.11/site-packages/flask_jwt_extended/view_decorators.py", line 170, in decorator
    return current_app.ensure_sync(fn)(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/kitchenowl/app/helpers/authorize_household.py", line 26, in decorator
    return func(*args, **kwargs)  # case server admin
           ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/kitchenowl/app/helpers/validate_args.py", line 26, in func_wrapper
    return func(arguments, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/kitchenowl/app/controller/recipe/recipe_controller.py", line 202, in scrapeRecipe
    recipe.name = scraper.title()
                  ^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.11/site-packages/recipe_scrapers/plugins/exception_handling.py", line 64, in decorated_method_wrapper
    return decorated(self, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.11/site-packages/recipe_scrapers/plugins/html_tags_stripper.py", line 75, in decorated_method_wrapper
    decorated_func_result = decorated(self, *args, **kwargs)
                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.11/site-packages/recipe_scrapers/plugins/normalize_string.py", line 34, in decorated_method_wrapper
    return normalize_string(decorated(self, *args, **kwargs))
                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.11/site-packages/recipe_scrapers/plugins/schemaorg_fill.py", line 48, in decorated_method_wrapper
    return decorated(self, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.11/site-packages/recipe_scrapers/hellofresh.py", line 14, in title
    return self.schema.title()
           ^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.11/site-packages/recipe_scrapers/_schemaorg.py", line 103, in title
    return normalize_string(self.data.get("name"))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.11/site-packages/recipe_scrapers/_utils.py", line 144, in normalize_string
    unescaped_string = html.unescape(string)
                       ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/html/__init__.py", line 130, in unescape
    if '&' not in s:
       ^^^^^^^^^^^^
TypeError: argument of type 'NoneType' is not iterable

Share your configuration

irrelevant

TomBursch · 2024-04-04T11:18:37Z

Should be fixed with the just published latest version.

Karamellwuerfel · 2024-04-04T12:08:11Z

Hi @TomBursch,
I updated to the last two versions of your images and still get the following error in backend:

kitchenowlback     | [2024-04-04 12:03:48,441] ERROR in config: argument of type 'NoneType' is not iterable
kitchenowlback     | Traceback (most recent call last):
kitchenowlback     |   File "/opt/venv/lib/python3.11/site-packages/flask/app.py", line 870, in full_dispatch_request
kitchenowlback     |     rv = self.dispatch_request()
kitchenowlback     |          ^^^^^^^^^^^^^^^^^^^^^^^
kitchenowlback     |   File "/opt/venv/lib/python3.11/site-packages/flask/app.py", line 855, in dispatch_request
kitchenowlback     |     return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)  # type: ignore[no-any-return]
kitchenowlback     |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
kitchenowlback     |   File "/opt/venv/lib/python3.11/site-packages/flask_jwt_extended/view_decorators.py", line 170, in decorator
kitchenowlback     |     return current_app.ensure_sync(fn)(*args, **kwargs)
kitchenowlback     |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
kitchenowlback     |   File "/usr/src/kitchenowl/app/helpers/authorize_household.py", line 26, in decorator
kitchenowlback     |     return func(*args, **kwargs)  # case server admin
kitchenowlback     |            ^^^^^^^^^^^^^^^^^^^^^
kitchenowlback     |   File "/usr/src/kitchenowl/app/helpers/validate_args.py", line 26, in func_wrapper
kitchenowlback     |     return func(arguments, *args, **kwargs)
kitchenowlback     |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
kitchenowlback     |   File "/usr/src/kitchenowl/app/controller/recipe/recipe_controller.py", line 202, in scrapeRecipe
kitchenowlback     |     recipe.name = scraper.title()
kitchenowlback     |                   ^^^^^^^^^^^^^^^
kitchenowlback     |   File "/opt/venv/lib/python3.11/site-packages/recipe_scrapers/plugins/exception_handling.py", line 64, in decorated_method_wrapper
kitchenowlback     |     return decorated(self, *args, **kwargs)
kitchenowlback     |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
kitchenowlback     |   File "/opt/venv/lib/python3.11/site-packages/recipe_scrapers/plugins/html_tags_stripper.py", line 75, in decorated_method_wrapper
kitchenowlback     |     decorated_func_result = decorated(self, *args, **kwargs)
kitchenowlback     |                             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
kitchenowlback     |   File "/opt/venv/lib/python3.11/site-packages/recipe_scrapers/plugins/normalize_string.py", line 34, in decorated_method_wrapper
kitchenowlback     |     return normalize_string(decorated(self, *args, **kwargs))
kitchenowlback     |                             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
kitchenowlback     |   File "/opt/venv/lib/python3.11/site-packages/recipe_scrapers/plugins/schemaorg_fill.py", line 48, in decorated_method_wrapper
kitchenowlback     |     return decorated(self, *args, **kwargs)
kitchenowlback     |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
kitchenowlback     |   File "/opt/venv/lib/python3.11/site-packages/recipe_scrapers/hellofresh.py", line 14, in title
kitchenowlback     |     return self.schema.title()
kitchenowlback     |            ^^^^^^^^^^^^^^^^^^^
kitchenowlback     |   File "/opt/venv/lib/python3.11/site-packages/recipe_scrapers/_schemaorg.py", line 103, in title
kitchenowlback     |     return normalize_string(self.data.get("name"))
kitchenowlback     |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
kitchenowlback     |   File "/opt/venv/lib/python3.11/site-packages/recipe_scrapers/_utils.py", line 187, in normalize_string
kitchenowlback     |     unescaped_string = html.unescape(string)
kitchenowlback     |                        ^^^^^^^^^^^^^^^^^^^^^
kitchenowlback     |   File "/usr/local/lib/python3.11/html/__init__.py", line 130, in unescape
kitchenowlback     |     if '&' not in s:
kitchenowlback     |        ^^^^^^^^^^^^
kitchenowlback     | TypeError: argument of type 'NoneType' is not iterable
kitchenowlback     | ERROR:app.config:argument of type 'NoneType' is not iterable
kitchenowlback     | Traceback (most recent call last):
kitchenowlback     |   File "/opt/venv/lib/python3.11/site-packages/flask/app.py", line 870, in full_dispatch_request
kitchenowlback     |     rv = self.dispatch_request()
kitchenowlback     |          ^^^^^^^^^^^^^^^^^^^^^^^
kitchenowlback     |   File "/opt/venv/lib/python3.11/site-packages/flask/app.py", line 855, in dispatch_request
kitchenowlback     |     return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)  # type: ignore[no-any-return]
kitchenowlback     |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
kitchenowlback     |   File "/opt/venv/lib/python3.11/site-packages/flask_jwt_extended/view_decorators.py", line 170, in decorator
kitchenowlback     |     return current_app.ensure_sync(fn)(*args, **kwargs)
kitchenowlback     |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
kitchenowlback     |   File "/usr/src/kitchenowl/app/helpers/authorize_household.py", line 26, in decorator
kitchenowlback     |     return func(*args, **kwargs)  # case server admin
kitchenowlback     |            ^^^^^^^^^^^^^^^^^^^^^
kitchenowlback     |   File "/usr/src/kitchenowl/app/helpers/validate_args.py", line 26, in func_wrapper
kitchenowlback     |     return func(arguments, *args, **kwargs)
kitchenowlback     |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
kitchenowlback     |   File "/usr/src/kitchenowl/app/controller/recipe/recipe_controller.py", line 202, in scrapeRecipe
kitchenowlback     |     recipe.name = scraper.title()
kitchenowlback     |                   ^^^^^^^^^^^^^^^
kitchenowlback     |   File "/opt/venv/lib/python3.11/site-packages/recipe_scrapers/plugins/exception_handling.py", line 64, in decorated_method_wrapper
kitchenowlback     |     return decorated(self, *args, **kwargs)
kitchenowlback     |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
kitchenowlback     |   File "/opt/venv/lib/python3.11/site-packages/recipe_scrapers/plugins/html_tags_stripper.py", line 75, in decorated_method_wrapper
kitchenowlback     |     decorated_func_result = decorated(self, *args, **kwargs)
kitchenowlback     |                             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
kitchenowlback     |   File "/opt/venv/lib/python3.11/site-packages/recipe_scrapers/plugins/normalize_string.py", line 34, in decorated_method_wrapper
kitchenowlback     |     return normalize_string(decorated(self, *args, **kwargs))
kitchenowlback     |                             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
kitchenowlback     |   File "/opt/venv/lib/python3.11/site-packages/recipe_scrapers/plugins/schemaorg_fill.py", line 48, in decorated_method_wrapper
kitchenowlback     |     return decorated(self, *args, **kwargs)
kitchenowlback     |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
kitchenowlback     |   File "/opt/venv/lib/python3.11/site-packages/recipe_scrapers/hellofresh.py", line 14, in title
kitchenowlback     |     return self.schema.title()
kitchenowlback     |            ^^^^^^^^^^^^^^^^^^^
kitchenowlback     |   File "/opt/venv/lib/python3.11/site-packages/recipe_scrapers/_schemaorg.py", line 103, in title
kitchenowlback     |     return normalize_string(self.data.get("name"))
kitchenowlback     |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
kitchenowlback     |   File "/opt/venv/lib/python3.11/site-packages/recipe_scrapers/_utils.py", line 187, in normalize_string
kitchenowlback     |     unescaped_string = html.unescape(string)
kitchenowlback     |                        ^^^^^^^^^^^^^^^^^^^^^
kitchenowlback     |   File "/usr/local/lib/python3.11/html/__init__.py", line 130, in unescape
kitchenowlback     |     if '&' not in s:
kitchenowlback     |        ^^^^^^^^^^^^
kitchenowlback     | TypeError: argument of type 'NoneType' is not iterable
kitchenowlback     | [pid: 44|app: 0|req: 11/10] 172.26.0.2 () {64 vars in 1416 bytes} [Thu Apr  4 12:03:43 2024] POST /api/household/1/recipe/scrape => generated 20 bytes in 5333 msecs (HTTP/1.1 500) 2 headers in 98 bytes (3 switches on core 199)

My config uses image: tombursch/kitchenowl-web:latest and image: tombursch/kitchenowl:latest.
In the web-frontend I got the versions

Thx for your help!

TomBursch · 2024-04-04T12:20:41Z

Strange, it works for me both on my personal instance and the public one.

Karamellwuerfel · 2024-04-04T12:27:09Z

My steps to update were

docker-compose pull
docker-compose up --force-recreate --build -d
docker image prune -f

On web-frontend the error looks like this:

Browser network:

Karamellwuerfel · 2024-04-08T06:38:05Z

@thatso can you confirm, imports with hellofresh are working? Try this url: https://www.hellofresh.de/recipes/torchietti-in-tomaten-paprika-sosse-5fc66ae618d71c10e64e51d4 Thanks!

I changed setup to the AIO docker-compose. Versions are the same as after my update, but it's not working for me. Maybe it could be a problem with my nginx proxy.

thatso · 2024-04-08T07:25:06Z

@Karamellwuerfel: did you mix up issues? I don't use hellofresh. 🙄

Karamellwuerfel · 2024-04-08T07:29:59Z

@Karamellwuerfel: did you mix up issues? I don't use hellofresh. 🙄

Oh, I'm sorry!

Karamellwuerfel · 2024-04-20T19:17:36Z

Is it maybe a server configuration issue?

02:~$ ping hellofresh.de
PING hellofresh.de (54.72.116.110) 56(84) bytes of data.

--- hellofresh.de ping statistics ---
56 packets transmitted, 0 received, 100% packet loss, time 56297ms

02:~$ curl hellofresh.de
<html>
<head><title>301 Moved Permanently</title></head>
<body>
<center><h1>301 Moved Permanently</h1></center>
</body>
</html>

02:~$ curl www.hellofresh.de
<!DOCTYPE html>
<!--[if lt IE 7]> <html class="no-js ie6 oldie" lang="en-US"> <![endif]-->
<!--[if IE 7]>    <html class="no-js ie7 oldie" lang="en-US"> <![endif]-->
<!--[if IE 8]>    <html class="no-js ie8 oldie" lang="en-US"> <![endif]-->
<!--[if gt IE 8]><!--> <html class="no-js" lang="en-US"> <!--<![endif]-->
<head>
<title>Attention Required! | Cloudflare</title>
<meta charset="UTF-8" />
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
<meta http-equiv="X-UA-Compatible" content="IE=Edge" />
<meta name="robots" content="noindex, nofollow" />
<meta name="viewport" content="width=device-width,initial-scale=1" />
<link rel="stylesheet" id="cf_styles-css" href="/cdn-cgi/styles/cf.errors.css" />
<!--[if lt IE 9]><link rel="stylesheet" id='cf_styles-ie-css' href="/cdn-cgi/styles/cf.errors.ie.css" /><![endif]-->
<style>body{margin:0;padding:0}</style>


<!--[if gte IE 10]><!-->
<script>
  if (!navigator.cookieEnabled) {
    window.addEventListener('DOMContentLoaded', function () {
      var cookieEl = document.getElementById('cookie-alert');
      cookieEl.style.display = 'block';
    })
  }
</script>
<!--<![endif]-->

</head>

Can someone help me on cloudflare authentication issue?

TomBursch · 2024-04-24T11:34:17Z

Good to know, this has nothing to do with your setup nor with KitchenOwl. Hello fresh is using Cloudflare which, to combat dos attacks, can block bots (which KitchenOwl is when scraping the website). It seems like they flagged your KitchenOwl instance.

Karamellwuerfel added the bug Something isn't working label Apr 4, 2024

TomBursch closed this as completed Apr 4, 2024

TomBursch reopened this Apr 4, 2024

TomBursch mentioned this issue May 7, 2024

Bug: Trouble importing recipes from URL #447

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Bug: Error on adding recipe from hellofresh #424

Bug: Error on adding recipe from hellofresh #424

Karamellwuerfel commented Apr 4, 2024

TomBursch commented Apr 4, 2024

Karamellwuerfel commented Apr 4, 2024

TomBursch commented Apr 4, 2024

Karamellwuerfel commented Apr 4, 2024 •

edited

Karamellwuerfel commented Apr 8, 2024

thatso commented Apr 8, 2024

Karamellwuerfel commented Apr 8, 2024

Karamellwuerfel commented Apr 20, 2024

TomBursch commented Apr 24, 2024 •

edited

Bug: Error on adding recipe from hellofresh #424

Bug: Error on adding recipe from hellofresh #424

Comments

Karamellwuerfel commented Apr 4, 2024

Is this urgent?

What parts are affected

What is the server version

What is the client version

What platform are you using

What's the problem 🤔

Share your logs

Share your configuration

TomBursch commented Apr 4, 2024

Karamellwuerfel commented Apr 4, 2024

TomBursch commented Apr 4, 2024

Karamellwuerfel commented Apr 4, 2024 • edited

Karamellwuerfel commented Apr 8, 2024

thatso commented Apr 8, 2024

Karamellwuerfel commented Apr 8, 2024

Karamellwuerfel commented Apr 20, 2024

TomBursch commented Apr 24, 2024 • edited

Karamellwuerfel commented Apr 4, 2024 •

edited

TomBursch commented Apr 24, 2024 •

edited