Web Development

5787 readers

4 users here now

Welcome to the web development community! This is a place to post, discuss, get help about, etc. anything related to web development

What is web development?

Web development is the process of creating websites or web applications

Rules/Guidelines

Follow the programming.dev site rules
Keep content related to web development
If what you're posting relates to one of the related communities, crosspost it into there to help them grow
If youre posting an article older than two years put the year it was made in brackets after the title

Related Communities

Wormhole

!cool_github_projects@programming.dev

Some webdev blogs

Not sure what to post in here? Want some web development related things to read?

Heres a couple blogs that have web development related content

https://frontendfoc.us/ - [RSS]
https://wesbos.com/blog
https://davidwalsh.name/ - [RSS]
https://www.nngroup.com/articles/
https://sia.codes/posts/ - [RSS]
https://www.smashingmagazine.com/ - [RSS]
https://www.bennadel.com/ - [RSS]
https://web.dev/ - [RSS]

Credits

Icon base by Delapouite under CC BY 3.0 with modifications to add a gradient

founded 3 years ago

MODERATORS

snowe@programming.dev

erlingur@programming.dev

Ategon@programming.dev

Vacant@programming.dev

How to download Lemmy post, as it is shown in the Lemmy UI (feddit.org)

submitted 1 week ago by cows_are_underrated@feddit.org to c/webdev@programming.dev

12 comments fedilink hide all child comments

So, what I want is to download lemmy posts (including comments) as raw html through a script. The problem I have is, that curl by default does not include the css of the page. Is there a way I can download the posts webpages as they are being displayed?

you are viewing a single comment's thread
view the rest of the comments

[–] DmMacniel@feddit.org 1 points 1 week ago (1 children)

Mhm you could run a script through the HTML and look for stylesheet hrefs and download next to them.

[–] cows_are_underrated@feddit.org 2 points 1 week ago (1 children)

I was more looking at doing this with a tool like wget. I got the following command, that downloads the post and its assets, but for some reason it then does not actually work:

'wget --recursive --no-clobber --page-requisites --html-extension --convert-links --restrict-file-names=windows --domains "feddit.org" --no-parent "feddit.org/post" "https://feddit.org/post/33942692"'

The advantage of wget is, that it also accepts multiple links at once, so it is quite easy do download multiple posts at once (which is my goal)

[–] apparia@discuss.tchncs.de 2 points 1 week ago (1 children)

If this is what you are seeing that makes it "not work":

It seems to be implemented in a script that runs when the saved page loads and hides the content, even though that content is actually there. You can tell wget to ignore scripts (which aren't very useful in a download anyway, as they mostly pertain to interactivity) with --reject js. I tried this and was able to view the downloaded post properly.

I've no idea why the script does that, one could probably look at the lemmy-ui source code to figure it out.

[–] cows_are_underrated@feddit.org 1 points 1 week ago

That was my exact problem. Thanks for the solution.