Page 1 of 1

Progamming a webcrawler

Posted: Sat Apr 04, 2015 7:54 am
by BlubBlab
I ask myself if anybody has some experience with this here?
I have to programme a task were I login with an AC read 3 values, logoff the HP and go to the next AC.
The results(values) should be saved in an ods. I would love to use a webcrawler instead of an macro.

Maybe I'm to focus using gecko but I don't find really an "how to" for starting.

Re: Progamming a webcrawler

Posted: Sat Apr 04, 2015 11:06 am
by Administrator
I've done it with MicroMacro using a cURL addon, as well as from straight C/C++. Now with MicroMacro 2, you can do it without any additional addons as long as you're willing to learn a bit about how the sockets work and research the HTTP protocol. Looking at the IRC bot code I provided should be of some use. You should also be aware that it doesn't support HTTPS (yet), and will have to figure out how to handle cookies upon request.

Basically all you do is post a command to the server to fetch a specific page, and then read the content it sends back. Parse out what you need using regex.

Re: Progamming a webcrawler

Posted: Wed Apr 08, 2015 12:30 pm
by BlubBlab
Thanks but I ended up using this : http://watin.org/