Extremely high number of requests #3

New Issue

seeseemelk · 2025-02-27T06:09:26+01:00

seeseemelk commented

2025-02-27 06:09:26 +01:00

I got this e-mail from Viva+ support:

After reviewing your account activity, it appears that the tool you’re using is making an extremely high number of requests to our platform (every 7 seconds) – far beyond normal user behaviour. This level of traffic has been triggering errors and automated alerts on our system, which ultimately led to the ban.

Every 7 seconds is insane. It should only be doing a scrape every 15 minutes (which frankly is still high. I should probably set this to once every 12 or 24 hours.)

Possible causes for every 7 seconds:

If an error occurs, the program quits and is restarted by the Docker daemon. Docker does not include a restart delay, so vivaplusdl will then immediately try to login again. I should probably add a --slow-startup option or some environment variable to make sure the program holds off for a while before starting again.
Once a video has been scraped, I should double check that we "close" those tabs in Playwright. Perhaps having a million tabs open in the background (if that's even the case) is a problem
The scraping of videos probably is not an issue. It's not rate limited within vivaplusdl, but it should only incur a large number of requests during the initial seeding process, which is a one-time thing. Still, it wouldn't hurt to incorporate some delay.

I got this e-mail from Viva+ support: > After reviewing your account activity, it appears that the tool you’re using is making an extremely high number of requests to our platform (every 7 seconds) – far beyond normal user behaviour. This level of traffic has been triggering errors and automated alerts on our system, which ultimately led to the ban. Every 7 seconds is insane. It should only be doing a scrape every 15 minutes (which frankly is still high. I should probably set this to once every 12 or 24 hours.) Possible causes for every 7 seconds: - If an error occurs, the program quits and is restarted by the Docker daemon. Docker does not include a restart delay, so vivaplusdl will then immediately try to login again. I should probably add a `--slow-startup` option or some environment variable to make sure the program holds off for a while before starting again. - Once a video has been scraped, I should double check that we "close" those tabs in Playwright. Perhaps having a million tabs open in the background (if that's even the case) is a problem - The scraping of videos probably is not an issue. It's not rate limited within vivaplusdl, but it should only incur a large number of requests during the initial seeding process, which is a one-time thing. Still, it wouldn't hurt to incorporate some delay.

seeseemelk commented

2025-02-27 06:09:46 +01:00

The program must not be put into production again until this is resolved

**The program must not be put into production again until this is resolved**

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: seeseemelk/vivaplusdl#3