Skip to content

plugins.bloomberg: add webbrowser API fallback #6107

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

bastimeyer
Copy link
Member

Resolves #6075

Request the input URL using Streamlink's webbrowser API as a fallback mechanic, in case Bloomberg rejects the initial HTTP request being made using the regular HTTPSession. After successfully falling back to the webbrowser API, the user's IP address appears to be whitelisted, so subsequent requests can be made without having to use the webbrowser API.

$ ./script/test-plugin-urls.py bloomberg
:: https://www.bloomberg.com/live
::  272p, 360p, 720p_alt, 720p, 1080p, worst, best
:: https://www.bloomberg.com/live/europe
::  272p, 360p, 540p, 720p, worst, best
:: https://www.bloomberg.com/live/us
::  272p, 360p, 720p_alt, 720p, 1080p, worst, best
:: https://www.bloomberg.com/news/videos/2022-08-10/-bloomberg-surveillance-early-edition-full-08-10-22
::  540p, 720p_alt, 720p, 1080p_alt, 1080p, worst, best

Log output from yesterday when I fully replaced the initial HTTP request. It's now kept though and the webbrowser API is only used as a fallback.

$ streamlink -l debug https://www.bloomberg.com/live best
[cli][debug] OS:         Linux-6.9.9-1-git-x86_64-with-glibc2.40
[cli][debug] Python:     3.12.4
[cli][debug] OpenSSL:    OpenSSL 3.3.1 4 Jun 2024
[cli][debug] Streamlink: 6.8.3+17.gac9bab3c.dirty
[cli][debug] Dependencies:
[cli][debug]  certifi: 2024.7.4
[cli][debug]  isodate: 0.6.1
[cli][debug]  lxml: 5.2.2
[cli][debug]  pycountry: 24.6.1
[cli][debug]  pycryptodome: 3.20.0
[cli][debug]  PySocks: 1.7.1
[cli][debug]  requests: 2.32.3
[cli][debug]  trio: 0.26.0
[cli][debug]  trio-websocket: 0.11.1
[cli][debug]  typing-extensions: 4.12.2
[cli][debug]  urllib3: 2.2.2
[cli][debug]  websocket-client: 1.8.0
[cli][debug] Arguments:
[cli][debug]  url=https://www.bloomberg.com/live
[cli][debug]  stream=['best']
[cli][debug]  --loglevel=debug
[cli][debug]  --player=/usr/bin/mpv
[cli][info] Found matching plugin bloomberg for URL https://www.bloomberg.com/live
[webbrowser.webbrowser][info] Launching web browser: /usr/bin/chromium
[webbrowser.webbrowser][debug] Waiting for web browser process to terminate
[plugins.bloomberg][debug] Found liveId: PHOENIX_US
[utils.l10n][debug] Language code: en_US
[cli][info] Available streams: 272p (worst), 360p, 720p_alt, 720p, 1080p (best)
[cli][info] Opening stream: 1080p (hls)
[cli][info] Starting player: /usr/bin/mpv
[stream.hls][debug] Reloading playlist
[cli][debug] Pre-buffering 8192 bytes
[stream.hls][debug] First Sequence: 72396042; Last Sequence: 72396046
[stream.hls][debug] Start offset: 0; Duration: None; Start Sequence: 72396044; End Sequence: None
[stream.hls][debug] Adding segment 72396044 to queue
[stream.hls][debug] Adding segment 72396045 to queue
[stream.hls][debug] Adding segment 72396046 to queue
[stream.hls][debug] Writing segment 72396044 to output
[stream.hls][debug] Segment 72396044 complete
[cli.output][debug] Opening subprocess: ['/usr/bin/mpv', '--force-media-title=https://www.bloomberg.com/live', '-']
[cli][debug] Writing stream to output
[stream.hls][debug] Writing segment 72396045 to output
[stream.hls][debug] Segment 72396045 complete
[cli][info] Player closed
[stream.segmented][debug] Closing worker thread
[stream.segmented][debug] Closing writer thread
[stream.hls][debug] Writing segment 72396046 to output
[stream.hls][debug] Segment 72396046 complete
[cli][info] Stream ended
[cli][info] Closing currently open stream...

@7k50, @DeltaEpsilon19498 please check... As said, they appear to whitelist your IP address for a certain amount of time when accessing their site regularly from your web browser. What I'm looking for is a debug log output where the fallback does work correctly. So please help me with this.
https://github.com/streamlink/streamlink/blob/master/CONTRIBUTING.md#pull-request-feedback

@bastimeyer bastimeyer added plugin issue A Plugin does not work correctly API: webbrowser labels Aug 3, 2024
@Hakkin
Copy link
Contributor

Hakkin commented Aug 9, 2024

Doesn't work for me on Linux or Windows. I seem to get a bot protection captcha that breaks the script.

$ streamlink --webbrowser-executable "C:\Program Files\Google\Chrome\Application\chrome.exe" -l debug https://www.bloomberg.com/live best
[session][debug] Loading plugin: bloomberg
[cli][debug] OS:         Windows 10
[cli][debug] Python:     3.12.1
[cli][debug] OpenSSL:    OpenSSL 3.0.11 19 Sep 2023
[cli][debug] Streamlink: 6.8.3+31.gd310fc18
[cli][debug] Dependencies:
[cli][debug]  certifi: 2024.7.4
[cli][debug]  isodate: 0.6.1
[cli][debug]  lxml: 5.2.2
[cli][debug]  pycountry: 24.6.1
[cli][debug]  pycryptodome: 3.20.0
[cli][debug]  PySocks: 1.7.1
[cli][debug]  requests: 2.32.3
[cli][debug]  trio: 0.26.2
[cli][debug]  trio-websocket: 0.11.1
[cli][debug]  typing-extensions: 4.12.2
[cli][debug]  urllib3: 2.2.2
[cli][debug]  websocket-client: 1.8.0
[cli][debug] Arguments:
[cli][debug]  url=https://www.bloomberg.com/live
[cli][debug]  stream=['best']
[cli][debug]  --loglevel=debug
[cli][debug]  --webbrowser-executable=C:\Program Files\Google\Chrome\Application\chrome.exe
[cli][info] Found matching plugin bloomberg for URL https://www.bloomberg.com/live
[plugins.bloomberg][info] Could not find JSON data. Falling back to webbrowser API...
[webbrowser.webbrowser][info] Launching web browser: C:\Program Files\Google\Chrome\Application\chrome.exe (headless=False)
[webbrowser.webbrowser][debug] Waiting for web browser process to terminate
[plugins.bloomberg][error] Could not find JSON data. Invalid URL or bot protection...
error: No playable streams found on this URL: https://www.bloomberg.com/live

@DeltaEpsilon19498
Copy link

DeltaEpsilon19498 commented Aug 9, 2024

Yes, I get the same issue. Bloomberg does not like the ip address of the vpn server I am using. See https://gist.github.com/DeltaEpsilon19498/b458c7a9b1d8745f4189391c61658438.

@bastimeyer bastimeyer force-pushed the plugins/bloomberg/webbrowser branch from d310fc1 to fb0d4f6 Compare November 8, 2024 16:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
API: webbrowser plugin issue A Plugin does not work correctly
Projects
None yet
Development

Successfully merging this pull request may close these issues.

plugins.bloomberg: Could not find JSON data. Invalid URL or bot protection
4 participants