T77 - Scrape web data and transfer it to your database
|
| Task Description |
Scrape web data and transfer it to your database. For e.g. scrape information from an online book database and build your database
|
| Solution |
 |
Prerequisites
- Let's say you have a list of ISBN numbers for the books for which you want to extract the information like publisher name, paperback, product dimension etc.
- Keep your website from which you want to extract the information, open.
- Keep the database to which you want to transfer your data open. For e.g.. Microsoft Access
|
Screenshot 1
Screenshot 2
Screenshot 3 |
 |
Search for the book by ISBN number
- Keep one notepad window open.
- Click on Record
- Enter one ISBN number in the search box of the website and click on enter.
- It will bring up the book details.
- On the resultant page, type Ctrl + A and Ctrl + C to select the contents of the entire page and copy them
- Paste the copied content to the notepad window by typing Ctrl + V.
|
 |
Extract the required information for one book
- In the notepad window, type Ctrl + HOME to go the beginning of the copied text.
- Type Ctrl + F to bring up the Find window
- Type in 'Publisher:' to look for the Publisher word.
- Once the word is found, hit Escape to remove the Find window.
- Hit the Right arrow to remove the selection.
- Type Shift + End to select the actual Publisher name and copy it using Ctrl + C.
- Now paste this data into your database using Ctrl + V. After you paste it into one field, you can move to the next field by hitting a Right Arrow or TAB key.
- You can repeat the above steps for other headers that you want to extract for e.g. paper back, dimensions etc. and transfer them to the database.
- Stop the recording and save the task.
|
 |
Extract information for all the ISBN numbers
- In order to extract the same information for all the ISBN numbers, you would need to make few changes to the recorded task by editing it in the Task Editor.
- Create a List variable called ISBN-List from Tools->Variable Manager and assign the list of ISBN numbers.
- Alternatively, the list can be read from the text file as well.
- Select your recorded task and click on Edit to open the task in editor.
- Identify the keystroke where you typed the ISBN number.
- Double-click it to open it in editor.
- Replace the actual ISBN number with the $ISBN-List$ variable.
- Save the Keystrokes command.
- Now insert a Loop for List (ISBN-List) around the actions that enter the ISBN number and extract the information. This will repeat your actions for all the ISBN numbers and enter the extracted information to your database.
- Your final database will look something like as shown in the Screenshot 3
|
|
| |
| When you run this task, it will loop through all the ISBN numbers from your list and extracts the required information from the website for that book and transfers it to the database. |
|
 |
| Wow! This program is fantastic. I was given the task to figure out how to get a list of over 1000 contacts off of a website and into Excel. I was trying to write some kind of macro for days through Excel's VB extension and was making no progress. So I went online and found Automation Anywhere, it took me only minutes to write a macro to copy the addresses. I owe Tethys Solutions hours of my life that I was prepared to spend copying and pasting. Thank you so much Tethys" |
- Ben Kinney,
Monterey media
|
| Read Customer Testimonies >> |
|
