Welcome to OGeek Q&A Community for programmer and developer-Open, Learning and Share
Welcome To Ask or Share your Answers For Others

Categories

0 votes
432 views
in Technique[技术] by (71.8m points)

publish subscribe - solution for large scale data streaming and time series storage

I have to implement the following system on-premise and looking for the right software solution:

  1. Thousands of nodes at different location collecting telemetry, mainly traffic stats by protocol and performance metrics like node cpu, mem utilization etc.
  2. These stats needs to be sent every several seconds over network to a central repository. Every node sends several records of stats looking like: timestap, metric1,metric2... metricN. I can live with a record being lost every now and then.
  3. These records need to be enriched before storage with added meta data like the cities where node resides, the customer they belong to so the final record would look like: timestamp, city, customer,... metric1,metric2... metricN
  4. A webapp queries the data store to present graphs of metrics vs time based on aggregations by customer, city, node etc.

I know there are a lot of tools set to accomplish this task but a looking for a combination that would work well together. I'm looking for a solution that would include:

  1. A tool for transporting the data from the nodes to the central repo in an encrypted manner
  2. A tool to efficiently enrich the data (most of enrichment is based on static 1:1 matches or range matches)
  3. A timeseries store for holding the data and allowing efficient queries by time & metadata

与恶龙缠斗过久,自身亦成为恶龙;凝视深渊过久,深渊将回以凝视…
Welcome To Ask or Share your Answers For Others

1 Reply

0 votes
by (71.8m points)
等待大神答复

与恶龙缠斗过久,自身亦成为恶龙;凝视深渊过久,深渊将回以凝视…
OGeek|极客中国-欢迎来到极客的世界,一个免费开放的程序员编程交流平台!开放,进步,分享!让技术改变生活,让极客改变未来! Welcome to OGeek Q&A Community for programmer and developer-Open, Learning and Share
Click Here to Ask a Question

...