Switch between RSS content and Web page content (store both) #1999
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
danielyrovas
commented
Aug 2, 2023
reader/processor/processor.go
Outdated
| @@ -80,14 +80,15 @@ func ProcessFeedEntries(store *storage.Storage, feed *model.Feed, user *model.Us | |||
| logger.Error(`[Processor] Unable to crawl this entry: %q => %v`, entry.URL, scraperErr) | |||
| } else if content != "" { | |||
| // We replace the entry content only if the scraper doesn't return any error. | |||
| entry.Content = content | |||
| // TODO: document change | |||
| entry.WebContent = content | |||
| } | |||
| } | |||
|
|
|||
| rewrite.Rewriter(url, entry, feed.RewriteRules) | |||
There was a problem hiding this comment.
This is problematic, I didn't look at how rewriter gets its data - I'll add a commit which fixes this.
There was a problem hiding this comment.
Rewriter now operates on web_content - as far as I can see rules do not operate on RSS content - only when processing original web page content.
danielyrovas
force-pushed
the
dev-keep-rss-content-and-web-content
branch
from
739e6e2
to
93948d7
Compare
|
I merged the commits and rebased onto main. |
|
Looks like the unit tests are failing after your rebase |
|
Yeah, I'll need to change some of those tests I think. The |
danielyrovas
force-pushed
the
dev-keep-rss-content-and-web-content
branch
2 times, most recently
from
77b36eb
to
d00bcfc
Compare
fguillot
reviewed
Sep 1, 2023
danielyrovas
force-pushed
the
dev-keep-rss-content-and-web-content
branch
3 times, most recently
from
f397706
to
34f4898
Compare
fguillot
reviewed
Sep 2, 2023
danielyrovas
force-pushed
the
dev-keep-rss-content-and-web-content
branch
from
4664e37
to
628dc2a
Compare
1 task
danielyrovas
force-pushed
the
dev-keep-rss-content-and-web-content
branch
from
628dc2a
to
29ee5ec
Compare
|
Hi @fguillot, I've updated my pull request. Do you mind taking a look at it? |
fguillot
reviewed
Apr 4, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Please let me know what you think of this change.
It solves an issue that has irked me when using the interface, which is overwriting the original RSS content with the webpage content which sometimes is not as well formatted as the original RSS content or is empty/junk. It allows people to hit the
downloadbutton just to see what the page looks like when it is downloaded, but still return to the RSS page to continue reading.The feature works like this:
Whenever an entry is loaded, if there is downloaded content available, that will be displayed and the
Downloadbutton becomesShow RSS Content.Otherwise the page loads like normal, showing the
Downloadbutton and the RSS page content.If the
Show RSS Contentbutton is selected, the original RSS content is displayed and the button turns to the regularDownloadbutton.If this
Downloadbutton is selected, the page is processed and saved like normal (re-downloading it even if it has been saved before).The
/fetch-contentAPI method now returns bothcontentandweb_contentas JSON.