Today is the tour stop of the Elasticon Tour that swung into Seattle. Myself and some of the Home Depot Quote Center team headed up via the Geek Train for the event. We arrived the night before so we could get up and actually be awake and ready for the event.
Just to note, a good clean place to stay, that isn’t overpriced like most of Seattle is the Pioneer Square Hotel - usually about $110-120 bucks. If you’re in town for a conference, sometimes it’s even worth skipping the “preferred hotels” and staying here. But I digress…
When the team and I walked in we waited a little bit for registration to get started. We stood around and chatted with some of our other cohort. Once the registration did open, we strolled into the main public space and started checking out some demos.
From what I could figure out from just watching the demo is that StreamSets is a ingest engine. That’s simple enough to determine just taking a look at their site. But being able to watch the demo also enlightened me to the way the interface IDE (the thing in the dark pictures above) worked.
The IDE provided ways to connect to ingestion data with minimal schema and actually start to flow the ingestion of this data through the engine. One of the key things that caught my attention at this point was the tie in with Kafka and Hadoop with the respective ingest and egress of data to sources ranging from AWS S3 to things like Elastic’s engine or other various sources that I’ll be working with in the coming months.
For more information about StreamSets here are a few other solid articles:
- Elastic Search Plus StreamSets for Reliable Data Ingestion
- Introducing the Streamsets Data Collector
…and connect to keep up with what StreamSets is doing:
…and install instructions:
…and most importantly, the code:
Beats (Not the Lousy Headphones)
Recently I installed Nagios as I will be doing a lot of systems monitoring, management, and general devops style work in the coming weeks to build out solid site reliability. Nagios will theoretically do a lot of the things I need it to do, but then I stumbled into the recently released Beats by Elastic Search (not by Dre, see above links in the title).
I won’t even try to explain Beats, because it is super straight forward. I do suggest checking out the site if you’re even slightly interested, but if you just want the quick lowdown, here’s a quote that basically summarizes the tool.
“Beats is the platform for building lightweight, open source data shippers for many types of operational data you want to enrich with Logstash, search and analyze in Elasticsearch, and visualize in Kibana. Whether you’re interested in log files, infrastructure metrics, network packets, or any other type of data, Beats serves as the foundation for keeping a beat on your data.”
So there ya go, something that collects a ton - if not almost all of - the data that I need to manage and monitor the infrastructure, platforms, network, and more that I’m responsible for. I’m currently diving in, but here’s a few key good bits about Beats that I’m excited to check out.
1 - PacketBeat
This is the real-time network packet analyzer that integrates with Elasticsearch and provides the respective analytics you’d expect. It gives a level of visibility with Beats between all the network servers and such that will prospectively give me insight to were our series of tubes or getting clogged up. I’m looking forward to seeing our requests mapped up with our responses! ;)
2 - FileBeat
This is a log data shipper based on the Logstash-Forwarder. At least it was at one point, it appears to look like it is less and less based on it. This beat monitors log directories for log files, tails the fails, and forwards them to Logstash. This completes another important part of what I need to systemically monitor within our systems.
Random fascinating observations:
- Did I mention Beats is written in Go? Furtherering Derek’s tweet from 2012! ;)
Prediction: Go will become the dominant language for systems work in IaaS, Orchestration, and PaaS in 24 months. #golang— Derek Collison (@derekcollison) September 11, 2012
- Beats has a cool logo, and the design of the tooling is actually solid, as if someone cared about how one would interact with the tools. I’ll see how this holds up as I implement a sample implementation of things with Beats & the various data collectors.
More References & Reading Material for Beats:
That’s it for the highlights so far. If anything else catches my eye this evening at the Elasticon Tour, I’ll get started rambling about it too!