web crawler - Scraping for research projects: some issues -

-

i scraping websites research project, encountering issues think might useful many users. given well-defined topic (e.g. bird-watching or astro-physics), objectives are:

  • identify important websites spread these ideas
  • crawl representative sample of these websites
  • perform network analysis , thematic analysis of data (e.g. topic models)
  • publish results in academic venue, without publishing of crawled data

to achieve goal, finding following obstacles:

  • sampling method: obviously, it's impossible establish boundaries of sites of interest. impossible know size of dataset, how can establish representativeness of sample? can crawl 10k, 1m, or 10m pages without knowing when should stop.
  • detection/banning issue: crawler, based on scrapy, following robots.txt , tries not hammer servers introducing increasing delays between requests, starting 25ms. however, lot of servers still detect crawler , block it. in sense, sampling process totally biased servers cut me off.
  • legal issues: gray area, feel if don't publish actual pages should on safe side. precautions can take avoid upsetting people, if results of study annoy website owners?

it nice outline methodology researchers, because sure many people encounter these problems start crawling non-trivial number of pages.

thanks suggestions!


Comments

Post a Comment

Popular posts from this blog

c++ - How to add Crypto++ library to Qt project -

-
Image
i downloaded crypto++ source , compiled cryptlib project in visual studio 2013, , added generated .lib file qt project, made .pro file this: qt += core gui qt += sql greaterthan(qt_major_version, 4):qt += widgets target = untitled template = app sources += main.cpp\ mainwindow.cpp headers += mainwindow.h \ databasecontrol.h \ test.h forms += mainwindow.ui win32:config(release, debug|release): libs += -l$$pwd/ -lcryptlib else:win32:config(debug, debug|release): libs += -l$$pwd/ -lcryptlibd else:unix: libs += -l$$pwd/ -lcryptlib includepath += $$pwd/ dependpath += $$pwd/ win32-g++:config(release, debug|release): pre_targetdeps += $$pwd/libcryptlib.a else:win32-g++:config(debug, debug|release): pre_targetdeps += $$pwd/libcryptlibd.a else:win32:!win32-g++:config(release, debug|release): pre_targetdeps += $$pwd/cryptlib.lib else:win32:!win32-g++:config(debug, debug|release): pre_targetdeps += $$pwd/cryptlibd.lib else:unix: pre_targetdeps += $$pw
Read more

jQuery Mobile app not scrolling in Firefox -

-
i have mobile app built in jquery mobile. scrolling fine in chrome & safari, fails scroll in firefox on desktop. have other instances of same app, different features, , works across browsers. life of me, cannot figure this, please help! located here: http://re-brand.us/dsus/index.html seems bug in firefox: bug log . able scroll using keyboard not trackpad on mac. are using mac? i'll keep looking see if can find way around it.
Read more

How to use vim as editor in Matlab GUI -

-
i using matlab r2013b. using gui, command windows , editor in split screen. in matlab preferences > editor/debugger > editor. there option set custom editor instead of matlab editor. set local editor /usr/bin/vim , not able open files @ anymore. there possibility use vim editor in split screen mode? i aware of !vim file.m , not solution looking for, works satisfactory in -nodisplay mode. rather want keep split screen mode. i did not find solution 100% satisfying, using tmux intermediate solution. allows 2 use split screen 2 terminals. in 1 of can open matlab -nodesktop in terminal mode, , in other 1 vim . possible switch between both split screens. you can use functionality of gui matlab typing things commandhistory , who et cetera.
Read more