hadoop - spring XD rabbit source module fails to process messages, first message stays unacknowledged -

-

i trying simple spring xd application load log events in hdfs. have configured target application spring-ampq/rabbit log4j appender (the org.springframework.amqp.rabbit.log4j.amqpappender class) pump log messages pre-configured exchange. set following stream pull messages , push them hdfs, both soruce , sink modules off-the-shelf xd modules,

stream definition,

xd:>stream create --name demoq1 --definition "rabbit | hdfs --rollover=15 --directory=/user/root" --deploy

created , deployed new stream 'demoq1'

xd:>stream list   stream name  stream definition                                   status   -----------  --------------------------------------------------  --------   demoq1       rabbit | hdfs --rollover=15 --directory=/user/root  deployed

amqp appender publishing messages exchange , routing demoq1 queue, rabbit source picking first message , gets stuck, not acknowledge message. reason?

in container log, see this: "failed write message payload hdfs" ?

if so, need use type conversion between modules. rabbit source hdfs sink messages byte arrays.

your stream definition be,

stream create --name demoq1 --definition "rabbit --outputtype=text/plain | hdfs --rollover=15 --directory=/user/root" --deploy

or,

stream create --name demoq1 --definition "rabbit | hdfs --inputtype=text/plain --rollover=15 --directory=/user/root" --deploy

note outputtype or inputtype option in source/sink respectively. in case, hdfs sink's hdfsstoremessagehandler expects payload of type string.

for more details on type conversion, please check out: https://github.com/spring-projects/spring-xd/wiki/type-conversion


Comments

Post a Comment

Popular posts from this blog

jQuery Mobile app not scrolling in Firefox -

-
i have mobile app built in jquery mobile. scrolling fine in chrome & safari, fails scroll in firefox on desktop. have other instances of same app, different features, , works across browsers. life of me, cannot figure this, please help! located here: http://re-brand.us/dsus/index.html seems bug in firefox: bug log . able scroll using keyboard not trackpad on mac. are using mac? i'll keep looking see if can find way around it.
Read more

c++ - How to add Crypto++ library to Qt project -

-
Image
i downloaded crypto++ source , compiled cryptlib project in visual studio 2013, , added generated .lib file qt project, made .pro file this: qt += core gui qt += sql greaterthan(qt_major_version, 4):qt += widgets target = untitled template = app sources += main.cpp\ mainwindow.cpp headers += mainwindow.h \ databasecontrol.h \ test.h forms += mainwindow.ui win32:config(release, debug|release): libs += -l$$pwd/ -lcryptlib else:win32:config(debug, debug|release): libs += -l$$pwd/ -lcryptlibd else:unix: libs += -l$$pwd/ -lcryptlib includepath += $$pwd/ dependpath += $$pwd/ win32-g++:config(release, debug|release): pre_targetdeps += $$pwd/libcryptlib.a else:win32-g++:config(debug, debug|release): pre_targetdeps += $$pwd/libcryptlibd.a else:win32:!win32-g++:config(release, debug|release): pre_targetdeps += $$pwd/cryptlib.lib else:win32:!win32-g++:config(debug, debug|release): pre_targetdeps += $$pwd/cryptlibd.lib else:unix: pre_targetdeps += $$pw
Read more

php array slice every 2th rule -

-
i have array on php outputs in txt file: printing grid -- 1 values -- undef = -9.99e+08 20.2 \nprinting grid -- 1 values -- undef = -9.99e+08 102.0 \nprinting grid -- 1 values -- undef = -9.99e+08 55.1 \nprinting grid -- 1 values -- undef = -9.99e+08 -18.3 \n what point, need nummeric values @ beginning of each rule. 20.2, 102.0, 55.1, -18,3 etc... example, there can more or less in txt file. now do? have tested array slice function can't exclude rule: printing grid -- 1 values -- undef = -9.99e+08 this array slice code... $arraygood = array_slice($arraybad, 1, 4); and foreach loop create text file: $file = fopen("array.txt","w"); foreach ($arraygood $meting => $waarde) { echo fwrite($file,$waarde . '\n'); } fclose($file); thanks! as posted in comments, understanding have array of strings similar following: $lines = array( "printing grid -- 1 values -- undef = -9.99e+08", "20.2 \nprinting grid --
Read more