Spring XD stream is a big data solution for how to ingestion. Each Stream is defined by 3 kinds of module: Source, Sink & Processor.
The tutorial will introduce Spring XD Stream.
Relate Post: Spring XD Batch Job – How to Start
Contents
I. Technology for Spring XD Stream
– Java 8
– Spring XD 1.3.1.Release
II. Concepts
A Spring XD is define by 3 kinds of modules: SOURCE | PROCESSOR 1 | PROCESSOR 2 | … PROCESSOR n | SINK. 2 modules Source & Sink are mandatory, and Processor is option.
Stream Modules are built with Spring Integration.
1. Source
A source is a message flow, contains a direct channel named output which is fed by an inbound adapter (configured with a poller, or triggered event)
List of sources supported by Spring XD, you can see at spring-xd-1.3.1.RELEASE\xd\modules\source:
File FTP GemFire Continuous Query (gemfire-cq) GemFire Source (gemfire) HTTP JDBC Source (jdbc) JMS Kafka Mail MongoDB Source (mongodb) MQTT RabbitMQ Reactor IP (reactor-ip) SFTP Stdout Capture Syslog Tail TCP TCP Client (tcp-client) Time Trigger Source (trigger) Twitter Search (twittersearch) Twitter Stream (twitterstream) |
We can create a customized source module using Spring Integration.
2. Processor
A processor is a message flow that contains a direct channel: input and a subscribable channel: output. It is use for transformation on the message.
List of Processors supported by Spring XD, you can see at \spring-xd-1.3.1.RELEASE\xd\modules\processor:
Aggregator Filter Header Enricher (header-enricher) HTTP Client (http-client) JSON to Tuple (json-to-tuple) Object to JSON (object-to-json) Script Shell Splitter Transform |
3. Sink
A sink is a message flow that contains a direct channel: input and an outbound adapter, or service activator used to provide the message to an external resource.
List of sinks supported by Spring XD, you can see at \spring-xd-1.3.1.RELEASE\xd\modules\sink:
Dynamic Router (router) File Sink (file) FTP Sink (ftp) GemFire Server GPFDIST Cassandra Hadoop (HDFS) (hdfs) HDFS Dataset (Avro/Parquet) (hdfs-dataset) JDBC Kafka Sink (kafka) Log Mail Mongo MQTT Sink (mqtt) Null Sink (null) RabbitMQ Redis Shell Sink (shell) Splunk Server (splunk) TCP Sink (tcp) |
III. Practices
Create a simple stream: http | transform –expression=payload.toUpperCase() | log
Relate Post: Spring XD Batch Job – How to Start
1. Start Spring XD with single-node mode
Open a CMD
Go to: ./spring-xd-1.3.1.RELEASE\xd\bin
Run xd-singlenode
2. Create a Stream
Go to: .\spring-xd-1.3.1.RELEASE\shell\bin
Run xd-shell
Then run: stream create --definition "http | transform --expression=payload.toUpperCase() | log" --name myprocstrem --deploy
Now open Browser, go to http://localhost:9393/admin-ui/#/streams/definitions
Stream is deployed:
You also can view on Containers:
3. Check Result
Create a Post from xd-shell
http post --target http://localhost:9000 --data "hello"
On xd-singlenode console, view log:
For undeploy or destroy the stream, you can use commandlines or web admins:
With command line:
xd:> stream undeploy --name myprocstrem xd:> stream destroy --name myprocstrem |
Last updated on June 9, 2017.