Advanced Data Scrapping in Flutter
Recently I was working on a project and I needed to show some data from a website. The data was in table format. I go through many searches over the internet to find a way to scrap data from the website.
Although, I found many solutions, but there was no specific way to get data from the table. I also tried some dart packages but in vain.
But after playing with dart HTML and HTTP Packages I was able to do so.
What is a Website Scraping?
Website scraping, or web data extraction, is the process of extracting or scraping data from websites.
DISCLAIMER: Although this isn’t legal for some websites but here the purpose is solely educational
First things first:
Dart HTML package has a method called parser that is used to parse the website body into the document. The document is used to call the queryselectorall function and it takes CSS path to the specified data target.
But wait, there is something more. The CSS path only works for the JAVA as far as I have worked. Dart doesn’t support the CSS path. I spent more than a week to sort this out.
In dart while scrapping the data, the queryselectorall work by taking up all the tags as it’s children and you have to specify which table’s data you want to show in your application.
Basically it will take all the tags of table in the given website and will start showing all of their results. If there are more than one table. Now what we have to do is specify a child of table whom data we want to get. Here it is:
Now what will happen here? It will get the data from the table number three and show it’s data. But there is still a issue it will get all the rows in the table and you will not be able to get the data of each cell separately.
By luck I was also able to do so.
After specifying the table I created a data variable to get the table rows. The for loop will be iterated to the length of table rows.
Now what is going on? I am taking the row one and it’s data cell or first column by specifying the children[0] to get the name from the table and then the children[1] will get the data of second column. The data[i] is specifying the rows.
Example:
Packages you need:
Import packages as:
Main:
Explanation:
From the Wikipedia I have obtained the table data of universities in Pakistan. In the first for loop I took the number of tables from the website. In the second table I have started the loop from i=1; in order to remove the table headers. Than by specifying the child I have stored the data in lists by using a data controller.
You can find the complete code on GitHub.
Further Reading
- https://data-lessons.github.io/library-webscraping-DEPRECATED/02-csssel/
(Basics of web scraping) - http://www.allitebooks.org/web-scraping-with-python-2nd-edition/
(Excellent book on Python web scraping)
Congrats! you have learnt something new today. Cheers!
Connect with me:
LinkedIn: https://www.linkedin.com/in/asad-khan-86ab92101/
