Progamming a webcrawler

Whatever you want here. Just make sure it's off topic. Which...I guess if the topic is being off topic, then it means you should be on topic. Understand?
Post Reply
Message
Author
User avatar
BlubBlab
Posts: 948
Joined: Fri Nov 30, 2012 11:33 pm
Location: My little Pony cafe

Progamming a webcrawler

#1 Post by BlubBlab » Sat Apr 04, 2015 7:54 am

I ask myself if anybody has some experience with this here?
I have to programme a task were I login with an AC read 3 values, logoff the HP and go to the next AC.
The results(values) should be saved in an ods. I would love to use a webcrawler instead of an macro.

Maybe I'm to focus using gecko but I don't find really an "how to" for starting.
Jack-of-all-trades, but master-of-only of a few :D

My Reps:
https://github.com/BlubBlab/Micromacro-with-OpenCV (My version of MM2 with OpenCV and for MS Visual Studio)
https://github.com/BlubBlab/rom-bot (rombot with no stop WP and advanced human emulation mode and some other extensions)
https://github.com/BlubBlab/Micromacro-2-Bot-Framework ( A work in progress )
My Tools : viewtopic.php?f=10&t=6226

User avatar
Administrator
Site Admin
Posts: 5306
Joined: Sat Jan 05, 2008 4:21 pm

Re: Progamming a webcrawler

#2 Post by Administrator » Sat Apr 04, 2015 11:06 am

I've done it with MicroMacro using a cURL addon, as well as from straight C/C++. Now with MicroMacro 2, you can do it without any additional addons as long as you're willing to learn a bit about how the sockets work and research the HTTP protocol. Looking at the IRC bot code I provided should be of some use. You should also be aware that it doesn't support HTTPS (yet), and will have to figure out how to handle cookies upon request.

Basically all you do is post a command to the server to fetch a specific page, and then read the content it sends back. Parse out what you need using regex.

User avatar
BlubBlab
Posts: 948
Joined: Fri Nov 30, 2012 11:33 pm
Location: My little Pony cafe

Re: Progamming a webcrawler

#3 Post by BlubBlab » Wed Apr 08, 2015 12:30 pm

Thanks but I ended up using this : http://watin.org/
Jack-of-all-trades, but master-of-only of a few :D

My Reps:
https://github.com/BlubBlab/Micromacro-with-OpenCV (My version of MM2 with OpenCV and for MS Visual Studio)
https://github.com/BlubBlab/rom-bot (rombot with no stop WP and advanced human emulation mode and some other extensions)
https://github.com/BlubBlab/Micromacro-2-Bot-Framework ( A work in progress )
My Tools : viewtopic.php?f=10&t=6226

Post Reply

Who is online

Users browsing this forum: No registered users and 5 guests