francisco jesús gómez & carlos juan diaz - sinfonier: storm builder for security...
TRANSCRIPT
/sin’fɒnjə/
Security Intelligence
Army Knowledge Online (www.us.army.mil) FM 2-0 INTELLIGENCE
/sin’fɒnjə/
/sin’fɒnjə/The Intelligence Cycle
Direction
http://www.cni.es/es/queescni/ciclo/
Collection
Analysis
Dissemination
/sin’fɒnjə/ This is NOT OSINTThis is Copy & Paste
http://tinyurl.com/pavtula
http://tinyurl.com/npegzok
http://tinyurl.com/q2ag2b9
February 26, 2014
What is Intelligence?
Quite simply, intelligence is the information our nation’s leaders need to keep our country safe.
Our leaders, like the President, make policy decisions based on this intelligence.
/sin’fɒnjə/Intelligence (Kids’ Zone)
https://www.cia.gov/kids-page/6-12th-grade/who-we-are-what-we-do/what-is-intelligence.html
• The generation of knowledge in support of decision makers
TroubleshootingAnticipation
• Intelligence is people (but not all people are intelligent):– Methodologies– Tools– Techniques
/sin’fɒnjə/Intelligence
sheer volumen of information
volatile
time saving
gather structure
enrichclassify
store
real time
analyze
/sin’fɒnjə/Tools are Essential
integrate
/sin’fɒnjə/
Storm Builder for Security Intelligence
/sin’fɒnjə/Storm
“Apache Storm is a free and open source distributed realtime computation system. Storm makes it easy to reliably process unbounded streams of data, doing for realtime processing what Hadoop did for batch processing. Storm is simple, can be used with any programming language, and is a lot of fun to use! “
http://storm.incubator.apache.org/
/sin’fɒnjə/Visual Programming
http://blog.interfacevision.com/design/design-visual-progarmming-languages-snapshots/
/sin’fɒnjə/Module: Types
SPOUT BOLT DRAIN
/sin’fɒnjə/Module: Types
SPOUT
“A spout is a source of streams in a computation. Typically a spout reads from a queueing broker such as Kestrel, RabbitMQ, or Kafka, but a spout can also generate its own stream or read from somewhere like the Twitter streaming API. Spout implementations already exist for most queueing systems.”
/sin’fɒnjə/Module: Types
BOLT
“A bolt processes any number of input streams and produces any number of new output streams. Most of the logic of a computation goes into bolts, such as functions, filters, streaming joins, streaming aggregations, talking to databases, and so on.”
/sin’fɒnjə/Module: Types
DRAIN?
/sin’fɒnjə/
Define a Module
Load to Storm
Use in a Topology
Upload your Code
Share on Sinfonier
Module: Life Cycle
/sin’fɒnjə/
Make a Topology
Run on Storm
Check Dashboard
Show results
Topology
cat /var/log/named/query.log | grep "IN A" | awk '{ print $6 }' | awk -F"#" '{print $1}' |sort -n | uniq -c | sort -rn | head | awk '{ printf $1",";system("curl -s http://freegeoip.net/csv/"$2 | cut –d”,” –f3 )}’
curl --retry 3 --insecure -s https://www.rootedcon.es/ | grep -E 'href="http://.*rootedcon\.es'| awk -F"href=\"" '{print $2}' | sed 's|\".*||g' | xargs curl -s -o /dev/null --write-out "%{http_code}:%{size_download}\n"| awk -F":" '{ if ( $1 == "200") { print "RSS size: " $2} }'
crontab -l# m h dom mon dow command@reboot /usr/bin/python /home/charlie/.ave_phoenix.py30 7,15,23 * * * /home/charlie/vigila/gauchap.sh –tweet fotos 2>&1 >/dev/null
/sin’fɒnjə/ Shell Scripting
/sin’fɒnjə/
Demo & Use cases
/sin’fɒnjə/TweetMon
/sin’fɒnjə/TorrentPeer
/sin’fɒnjə/Crawler
/sin’fɒnjə/Roadmap
Fun & Profit
Community
/sin’fɒnjə/We Want You
/sin’fɒnjə/
Become a Beta Tester
http://sinfonier-project.net/
http://tinyurl.com/sinfonier