Scrape Data from Multiple Web Pages with Power Query - My Online Training Hub
video description
Date: 2022-04-08
Related videos
Comments and reviews: 10
Mike!
This is a great tutorial! I was able to connect to one table but my second attempt isn't working. The table's data isn't in a specific order so when I go from page 1, 2, 3 etc. it loads them randomly each time and because I'm not on the actual website it's treating each individual page as a completely new query. The order is random so an individual record might be randomly pulled for the page 1 query then again on page 3 leading to duplicates and records that never populate on the page. Any idea how to work around this issue?
Thanks again!
reply
This is a great tutorial! I was able to connect to one table but my second attempt isn't working. The table's data isn't in a specific order so when I go from page 1, 2, 3 etc. it loads them randomly each time and because I'm not on the actual website it's treating each individual page as a completely new query. The order is random so an individual record might be randomly pulled for the page 1 query then again on page 3 leading to duplicates and records that never populate on the page. Any idea how to work around this issue?
Thanks again!
reply
Rodrigo
Dear! Thank you so much for this video.
Could you please share with us how can we do the following:
I have a web based database that is constantly fed. Today it has 300 itens and 15 itens per page, so: 20 pages. But next week, this database may have 600 itens, and due that, 40 pages. How can I automate the function to identify the total number of the pages each time it acesses the web data source?
Thank you!
reply
Dear! Thank you so much for this video.
Could you please share with us how can we do the following:
I have a web based database that is constantly fed. Today it has 300 itens and 15 itens per page, so: 20 pages. But next week, this database may have 600 itens, and due that, 40 pages. How can I automate the function to identify the total number of the pages each time it acesses the web data source?
Thank you!
reply
itech
Incredible overview, thank you so much! Is it possible to do this if you have a site with multiple pages that uses the same URL? I'm trying to scrape data from a public site with multiple pages, but all of them use the same URL - there are no unique identifiers (e.g. page numbers). Any assistance would be greatly appreciated.
reply
Incredible overview, thank you so much! Is it possible to do this if you have a site with multiple pages that uses the same URL? I'm trying to scrape data from a public site with multiple pages, but all of them use the same URL - there are no unique identifiers (e.g. page numbers). Any assistance would be greatly appreciated.
reply
Hamza
Ok i just follow the tutorial and my question is that i only got the first-page list when I close and apply, is this a view only, how to get all scraped data after closing and applying? i only see the first-page column, will I be able to get all the data from all pages? how to view or export the scraped full data?
reply
Ok i just follow the tutorial and my question is that i only got the first-page list when I close and apply, is this a view only, how to get all scraped data after closing and applying? i only see the first-page column, will I be able to get all the data from all pages? how to view or export the scraped full data?
reply
King
Mynda--
- 1:18 - Instead of the 13 HTML Table options listed in your -Navigator- dialog box, when I try to run the same Power BI query on my end, I am getting only 5 Tables (Table 0 through to Table 5; & an additional Document Table.) And these tables there is hardly any data to work with.
Please advise.
reply
Mynda--
- 1:18 - Instead of the 13 HTML Table options listed in your -Navigator- dialog box, when I try to run the same Power BI query on my end, I am getting only 5 Tables (Table 0 through to Table 5; & an additional Document Table.) And these tables there is hardly any data to work with.
Please advise.
reply
Louie
I'm currently new to this stuff. I see that you might be able to customize queries for specific data scraping tasks. I'm interested in learning ethical data scraping techniques. How would it be possible to create custom scraping software? What computing language would you recommend learning in that case?
reply
I'm currently new to this stuff. I see that you might be able to customize queries for specific data scraping tasks. I'm interested in learning ethical data scraping techniques. How would it be possible to create custom scraping software? What computing language would you recommend learning in that case?
reply
Fayyaaz
Good day
Excellent explanation.
I would like you to assist with the following.
There is a webpage that has multiple items but does not have -next page- option. But a continuous list,meaning as one is scrolling down ,more products are added.
How can one code this to see all the products
reply
Good day
Excellent explanation.
I would like you to assist with the following.
There is a webpage that has multiple items but does not have -next page- option. But a continuous list,meaning as one is scrolling down ,more products are added.
How can one code this to see all the products
reply
Xavier
Fantastic content. Thank you. I currently have a URL which provides up to 72 pages of data but it continuously grows. If I make my query retrieve let's say up to 75 pages, the last 3 pages come with error instead of table of course. Is this bad? or can I continue using it that way?
reply
Fantastic content. Thank you. I currently have a URL which provides up to 72 pages of data but it continuously grows. If I make my query retrieve let's say up to 75 pages, the last 3 pages come with error instead of table of course. Is this bad? or can I continue using it that way?
reply
iliyatsekov
Many thanks for the video! What if I have two variable names? My URL includes both a year and a quarter. I created the two variable names but how do I invoke the function to take all quarters from every year?
reply
Many thanks for the video! What if I have two variable names? My URL includes both a year and a quarter. I created the two variable names but how do I invoke the function to take all quarters from every year?
reply
Linda
Thank you very much for bringing this brilliant video. I do have a question, if I am not used to using Power BI and still want to use excel to extract web data like you do in this video, how do I do that?
reply
Thank you very much for bringing this brilliant video. I do have a question, if I am not used to using Power BI and still want to use excel to extract web data like you do in this video, how do I do that?
reply
Add a review, comment
Other channel videos
