Welcome to OGeek Q&A Community for programmer and developer-Open, Learning and Share
Welcome To Ask or Share your Answers For Others

Categories

0 votes
1.1k views
in Technique[技术] by (71.8m points)

beautifulsoup - Python Webscraping, how to navigate on a website?

I am programming a program that should read out certain data from a website and only output certain data (data from a table). However, I ran into a problem. I wrote a program that logs into the website, but from that website I have to go to the next website and then open the document with the data. Unfortunately, I have no idea how I can change the website and then open the document and read out the data. Does anyone have any idea how I could get on there?

from bs4 import BeautifulSoup
import requests

User = ''
Pass = ''
LOGIN_URL = ''
LOGIN_API_URL = ''


def main():
    session_requests = requests.session()
    result = session_requests.get(LOGIN_URL)
    cookies = result.cookies
    soup = BeautifulSoup(result.content, "html.parser")
    auth_token = soup.find("input", {'name': 'logintoken'}).get('value')

    payload = {'username': User, 'password': Pass , 'logintoken':auth_token }

    result = session_requests.post(
    LOGIN_API_URL,
    data=payload,
    cookies=cookies
    )

    #Report successful login
    print("Login succeeded: ", result.ok)
    print("Status code:", result.status_code)
    print(result.text)

    #Get Data



    # Close Session
    requests.session().close()
    print('Session closed')

# Entry point
if __name__ == '__main__':
    main()

question from:https://stackoverflow.com/questions/66059096/python-webscraping-how-to-navigate-on-a-website

与恶龙缠斗过久,自身亦成为恶龙;凝视深渊过久,深渊将回以凝视…
Welcome To Ask or Share your Answers For Others

1 Reply

0 votes
by (71.8m points)

You should read into Selenium with Python. Since there is no specific URL or login details (which you shouldn't post here anyway) it would be quite hard for any of us to create a working example since we don't have anything to work with.

Try the using selenium from the link above and if you have any questions or run into any issues from there come back and ask that specific question.

BS4 and requests can be powerful but selenium emulates a web browser and lets you move through websites like a "human" would. Start there.


与恶龙缠斗过久,自身亦成为恶龙;凝视深渊过久,深渊将回以凝视…
OGeek|极客中国-欢迎来到极客的世界,一个免费开放的程序员编程交流平台!开放,进步,分享!让技术改变生活,让极客改变未来! Welcome to OGeek Q&A Community for programmer and developer-Open, Learning and Share
Click Here to Ask a Question

...