python - Using TotalOrderPartitioner in Hadoop streaming -

-

i'm using python hadoop streaming project, , need similar functionality provided totalorderpartitioner , inputsampler in hadoop, is, need sample data first , create partition file, use partition file decide k-v pair go reducer in mapper. need in hadoop 1.0.4.

i find hadoop streaming examples keyfieldbasedpartitioner , customized partitioners, use -partitioner option in command tell hadoop use these partitioners. examples found using totalorderpartitioner , inputsampler in java, , need use writepartitionfile() of inputsampler , distributedcache class job. wondering if possible use totalorderpartitioner hadoop streaming? if possible, how can organize code use it? if not, practical implement total partitioner in python first , use it?

did not try, taking example keyfieldbasedpartitioner , replacing:

-partitioner org.apache.hadoop.mapred.lib.keyfieldbasedpartitioner

with

-partitioner org.apache.hadoop.mapreduce.lib.partition.totalorderpartitioner

should work.


Comments

  1. Tejuteju24 July 2018 at 06:28

    Big data in hadoop is the interesting topic and to get some important information. Big data hadoop online training India

    ReplyDelete

Post a Comment

Popular posts from this blog

c++ - How to add Crypto++ library to Qt project -

-
Image
i downloaded crypto++ source , compiled cryptlib project in visual studio 2013, , added generated .lib file qt project, made .pro file this: qt += core gui qt += sql greaterthan(qt_major_version, 4):qt += widgets target = untitled template = app sources += main.cpp\ mainwindow.cpp headers += mainwindow.h \ databasecontrol.h \ test.h forms += mainwindow.ui win32:config(release, debug|release): libs += -l$$pwd/ -lcryptlib else:win32:config(debug, debug|release): libs += -l$$pwd/ -lcryptlibd else:unix: libs += -l$$pwd/ -lcryptlib includepath += $$pwd/ dependpath += $$pwd/ win32-g++:config(release, debug|release): pre_targetdeps += $$pwd/libcryptlib.a else:win32-g++:config(debug, debug|release): pre_targetdeps += $$pwd/libcryptlibd.a else:win32:!win32-g++:config(release, debug|release): pre_targetdeps += $$pwd/cryptlib.lib else:win32:!win32-g++:config(debug, debug|release): pre_targetdeps += $$pwd/cryptlibd.lib else:unix: pre_targetdeps += $$pw...
Read more

c++ - Serialize a class with a Qlist of custom classes as member (using QDataStream) -

-
i'm trying serialize class lesson(my custom class[i removed setters]), includes qlist(question custom class). during test serialization sigsegv(segmentation fault) when deserializing object of qlist inside class lesson. upd1: serialized qlist* , changed qlist, sigsegv still appears. class lesson { public: lesson(); lesson(qstring, qstring, qlist<question>); qstring getname() const; qstring gettext() const; qlist<question>* gettest() const; friend qdatastream &operator<<(qdatastream &out, const lesson &l){ out << l.getname() << l.gettext(); out << l.gettest(); return out; } friend qdatastream &operator>>(qdatastream &in, lesson &l){ qstring name; qstring text; qlist<question> t; in >> name >> text >> t; l = lesson(name, text, t); return in; } friend qdatastream &operator<<(qdatastream &out, const lesson *&l){ out << l...
Read more

Read video using VideoReader function in Matlab? -

-
i want read video folder , extract frames it.i used vidoereader function.but gives error.my code shown below along error. mov=videoreader('11.mp4'); vidframes=read(mov); nframes=mov.numberofframes; i=1:nframes imshow(vidframes(:,:,i),[]); end and error show given below error using videoreader/init file not appear have video data. error in videoreader (line 147) obj.init(filename); error in video (line 7) mov=videoreader('11.mp4'); i think matlab version related problem. faced same problem when using matlab 2013a . however, when changed matlab 2014b problem disappeared.
Read more