• About Me Card

Max Hemingway

~ Musings as I work through life, career and everything.

Max Hemingway

Category Archives: Big Data

An A-Z Guide to being an Architect

07 Thursday Jan 2016

Posted by Max Hemingway in Architecture, Big Data, Cloud, Development, DevOps/OpsDev, Enterprise Architecture, Governance, Innovation, IoT, Open Source, Productivity, Programming, Security, Social Media, Tools

≈ Leave a comment

Tags

Architecture, Cloud, CPD, Data, Development, DevOps, Innovation, IoT, Knowledge, learning, Open Source, OpsDev, Productivity, Programming, Social Media

Back in 2008 Microsoft published An A-Z Guide to ABCBeing an Architect in their Architecture Journals.

Here is my take on an updated A to Z Guide to being an Architect. A couple of these may be similar.

A – Architect

Having the right level of skills as an Architect or engaging an Architect with the right level of skills will depend on the work needing to be undertaken. There are several types of Architect with some specialising in certain areas and others being multi domain skilled. The list below covers some of the different types of Architect- this is not an exhaustive list:

  • Enterprise Architect
  • Information Architect
  • Solutions Architect
  • Software Architect
  • Systems Architect

B – Blueprints

Following Blueprints and Patterns either published by vendors (such as the Microsoft Blueprints) or developed internally around your products and services will ensure repeat-ability and cost control around the design process.

Some examples showing different pattern types can be found at Architecture Patterns

C – Contextual Web Era

The up and coming 4th Platform area is the Contextual Web Era

  • 1st Platform – Mainframe Era
  • 2nd Platform – Client Server Era
  • 3rd Platform – Cloud Era
  • 4th Platform – Contextual Web Era

This is an up and coming era with lots of new innovation and developments. Keeping up with developments is key going forward for any architect to understand designs/solutions, art of the possible now and future, innovation and for developing roadmaps for solutions.

D – DevOps

To quote Wikipedia – “DevOps (a clipped compound of “development” and “operations”) is a culture, movement or practice that emphasizes the collaboration and communication of both software developers and other information-technology (IT) professionals while automating the process of software delivery and infrastructure changes”. Having knowledge of DevOps, OpsDev and Agile assist with Architecting a solution for a business understanding their practices and modes of interacting with technology to meet business requirements. A Good book on the subject of DevOps is “The Phoenix Project” by Gene Kim.

E – Enterprise Architecture

EA (Enterprise Architecture) is a blueprint that defines how a business can meet its objectives and strategy. This is achieved by conducting analysis, design, planning, recommendations and implementations through an Enterprise Architecture Framework

Enterprise Architecture Wikibook

F – Four Two Zero One Zero

42010 is the ISO Standard that most frameworks adhere to. Working to a Framework brings structure to your designs and life cycles.

There are a number of frame works available such as:

  • DoDAF
  • MoDAF
  • TOGAF
  • Zachman
  • Other Frameworks are available

Enterprise Architecture Wikipedia Book

G – Governance

Governance is an important part of architecture as it

  • Ensures Conformance
  • Controls Variance
  • Maintains Vitality
  • Enables Communication
  • Sets Direction
  • Issue Resolution
  • Provides Guidance and Prioritisation
  • Promotes Best Practise
  • Minimises Risk
  • Protects IT environments from tactical IT changes, project solutions, and strategic proposals that are not in an organisations global best interest
  • Controlling Technical Diversity, Over-Engineering and Unnecessary Complexity
  • Ensures projects can proceed quickly & efficiently
  • Control over IT spend
  • Quality Standards
  • Efficient and optimal use of resources and increase the effectiveness of IT processes

H – Hands On

It is important to be current and understand the technologies you are architecting. There are lots of options available to get your hands dirty using technology from using Cloud Servers to virtual machines on your compute device. There are other computing devices such as the Raspberry PI that provide a cheap alternative to standing up small farms to learn on.

I – IoT

IoT (Internet of Things) is where physical things are connected by the internet using embedded sensors, software, networks and electronics. This allows the items to be managed, controlled and reported on. My blog posts on IoT Device Security Considerations and Security Layers goes into more detail on this subject.

J – Juxtaposition

Juxtaposition is something an architect should be doing to compare things/items/artefacts etc.
noun;
1. an act or instance of placing close together or side by side, especially for comparison or contrast.
2.the state of being close together or side by side.

Source:http://dictionary.reference.com/browse/juxtaposition

K – Knowledge

I would class Skills with Knowledge. It is important as an Architect to ensure that your skills/knowledge are up to date and where you are unsure of a technology, you have a plan to address and skill up. Build a good CPD (Continuing Professional Development) plan and work towards completing it.

L – Language

With the move to cloud it is important to ensure your scripting skills are up to date as most cloud platforms use scripting to assist with the deployment of environments. This is also true of other DevOps/OpsDev applications. If you are unsure on what to learn this guide may help you – Learn a Programming Language – But which one?

M -Micro Segmentation

Micro Segmentation allows a business to use Networks, Compute and Storage to automate and deliver complex solutions by carving up and using the infrastructure. This segments part of the infrastructures to specific functions/tasks. It can also be used in a security context to segment networks, firewalls, compute and storage to increase security and reduce cyber attacks.  VMware have produced a book “Micro Segmentation for Dummies” that can be downloaded from here.

N – Next Generation

Next Generation refers to the next stage or development to something such as a new release of hardware or software. Next Generation is becoming a common term now to define products and artefacts, an example being Next Generation Firewalls.

O – Open Source

Open Source has been available for a long time with software such a Linux, however there is a bigger shift towards using Open Source and acceptance by businesses. Some examples of Open Source that is now mainstream within business include;

  • Ansible
  • Chef
  • Docker
  • Puppet

P – Performance

Performance can cover people as well as solutions / systems. Performance metrics should be set out at the inception of an engagement then monitored and reported on. This will be a factor in driving Continuous Improvement going forward as well as forecasting / planning for future upgrades and expansion.

Q – Quality

Quality is a huge subject and has a lot if standards governing it and how it affects all aspects of business and architecture. Knowing which standards and how they affect a solution will assist in the whole architecture lifecycle. There are also a number of tools available to help you;

  • Architecture Frameworks
  • ITIL
  • Six Sigma

There is also a level of pride and satisfaction in producing a quality solution and system achieving the objectives and requirements set out by the business.

R- Roadmap

Any architecture/solution should have a roadmap to set out its future. Roadmaps should include items such as:

  • Current state
  • Future state
  • Innovation
  • Upgrades / Releases
  • New Features / Functions
  • End of Life / Replacement

S – SMAC

SMAC stands for Social, Mobile, Analytics, Cloud. SMAC is an acronym that covers the areas and concepts when these four technologies are brought together to drive innovation in business. A good description of SMAC written by a colleague can be found here Acronyms SMAC.

T – Transformation

The majority, if not all systems will undergo a form of transformation. This may be in the form of a simple upgrade or to a complex redesign and migration to something else.

U – UX

UX (User eXperience) affects how people interact with your architecture / design and how they feel about it (emotions and attitudes). With the boom in apps and the nearing Contextual Web Era, UX is one of the most important factors to getting an architecture used. If your users don’t like the system they may find something else to use that they like.

V – Vision

Understanding the vision of your customer and their business is the driving factor for any architecture.

On working with your customer you should look to become a Trusted Advisor and also with your colleagues. A great book on the subject is The Trusted Advisor by David Maister. The book covers 3 main areas which discusses perspectives on trust, the structure of trust building and putting trust to work.

W – WWW

The internet is a key delivery mechanism for systems. Knowing how this works and key components to the internet should be understood such as:

  • IPV4 – IPV6
  • DNS
  • Routing
  • Connectivity
  • Security

X – X86

X86 – is a standard that every knows as its one of the most common platform types available.

Y – Year

Year is for the longevity of the solution you are designing. How many years are your expecting it to last What are the Business Requirements, statutory obligations, depreciation etc that need to be planned in. Consider things like End of Life, Maintenance and Upgrades on hardware and software from a solution point of view.

Z – Zero Defects

The best solution is the one with zero defects, but reaching this goal can be a challenge and can also consume a lot of expense. The best way to ensure Zero Defects is to use:

  • Best Practice
  • Reference Architectures
  • Blueprints/Patterns
  • Checklists
  • Reuse
  • Lessons Learnt

This is my current A to Z and some of the entries may be different in your version so “What is in your A to Z of being an Architect?”

I will look to write some further blog posts on the areas listed in this A to Z

Share this:

  • Twitter
  • Facebook
  • LinkedIn
  • Email
  • Pinterest

Like this:

Like Loading...

The Internet of Security and Things

08 Tuesday Sep 2015

Posted by Max Hemingway in Big Data, Cloud, IoT, Security

≈ Leave a comment

Tags

Cloud, Data, IoT, Security

How secure is the Internet of Things?

Traditionally we have been used to MThingsalware protection and Anti-Virus on our PC’s, then moving to laptops and other devices. Now on phones and slowly moving towards
the Internet of Things.  One article in the news today caught my eye where it is reported that Malware is being found pre-installed on devices, in this case Mobile Phones. G Data Report

It would seem that the hackers are trying to get the jump on the industry well before the devices are falling into the hands of the consumer. This is not the first time such incidents have been reported.

The race for Internet of Things sensors, devices and “Things” is growing fast, however with these incidents of Malware being found, how long will it be before code is appearing on chips on sensors that shouldn’t be there.

There are lots of Operating Systems available for the IOT. These can be classed as the mainstream ones that appear in the news and everyone knows such as Microsoft, Raspberry Pi, Linux etc, to the less know ones that are used on chipsets such as Contiki, TinyOS, Nano-RK.   (See https://maxhemingway.com/2015/04/14/iot-operating-systems/).

There are a number of challenges for the IoT industry, businesses and consumers (this list is not exhaustive);

  • Authentication
  • Data Capture
  • Encryption
  • Intrusion – Application, Network and Physical
  • Location tracking
  • Malware/Anti-Virus
  • Service disruption
  • Taking control of devices

These threats will drive the Internet of Security to protect the Internet of Things.

Cisco is looking to tackle some of these by running a Security Grand Challenge to offer prizes to the best security solutions.

More competitions and challenges will probably emerge as the industries try to understand and protect against the risks and use a crowd source model to help protect the IoT.

Share this:

  • Twitter
  • Facebook
  • LinkedIn
  • Email
  • Pinterest

Like this:

Like Loading...

Learning Data Science – Useful References

14 Tuesday Jul 2015

Posted by Max Hemingway in Big Data, Data Science, Machine Learning, Open Source

≈ 1 Comment

Tags

Big Data, Data, Data Science, Knowledge, Machine Learning

Firstly thanks to Tim Osterbuhr who prompteLearningd me to create this list of resources that I have found useful in learning about Data Science after he read my blog post on Learning Data Science. Tim has also provided some of the likes below as well.

Here is the list of Useful References for Learning Data Science. (This list is be no means exhaustive)

From my Blog

  • Learning Data Science
  • Data Science in the Cloud ebook
  • Data Science and Information Theory
  • Data Mining Courses
  • Open Source, Open Human, Open Data, Open Sesame!
  • Data Scientist Skill Set
  • R {swirls} – Learning R by doing
  • Correlation does not imply causation
  • Statistical Inference Resources

From Around the Web

  • 6 checkpoints to ensure regression model validity for analytics
  • Algorithms: Design and Analysis
  • Analyzing Big Data with Twitter
  • Big Data Analytics: Descriptive Vs. Predictive Vs. Prescriptive
  • Data Analysis
  • Data Mining for the Masses
  • Data Science Course
  • Google Visualization API Reference
  • k-means clustering
  • Occam’s Razor
  • PCA Step by Step
  • Regression Equation: What it is and How to use it
  • Using JavaScript visualization libraries with R

Public Data Sets

  • http://www.cs.cmu.edu/~./enron/
  • http://www.secviz.org/content/the-davix-live-cd
  • http://www.caida.org/data/overview/
  • http://www.secviz.org/content/visual-analytics-workshop-with-worlds-leading-security-visualization-expert-0
  • http://snap.stanford.edu/data/
  • http://analytics.ncsu.edu/
  • https://code.google.com/p/google-refine/

Data Science Books

  • 9 Free Books for Learning Data Mining & Data Analysis
  • 16 Free Data Science Books
  • 27 free data mining books

Happy to add other links from readers to this list.

Share this:

  • Twitter
  • Facebook
  • LinkedIn
  • Email
  • Pinterest

Like this:

Like Loading...

Techdays Online Azure Special

02 Tuesday Jun 2015

Posted by Max Hemingway in Architecture, Big Data, Cloud, DevOps/OpsDev, IoT, Machine Learning

≈ Leave a comment

Tags

Architecture, Big Data, Cloud, DevOps, IoT, Machine Learning, Open Source, OpsDev

Microsoft are running a Techdays Online Azure Special over the next 3 days

Registration is at https://info.microsoft.com/UK-Azure-WBNR-FY15-06Jun-Azure-Techdays-2015-Registration.html

  • June 02, 2015 09:00 AM – TechDays Online Azure Special Day One: Keynotes, IOT, Hybrid and Open Source
  • June 03, 2015 09:00 AM – TechDays Online Azure Special Day Two: Apps, Architecture, Big Data and Machine Learning
  • June 04, 2015 09:00 AM – TechDays Online Azure Special Day Three: Cloud Infrastructure and Dev Ops

Hopefully the sessions will be available offline after the event for reference and catch up.

Books

Share this:

  • Twitter
  • Facebook
  • LinkedIn
  • Email
  • Pinterest

Like this:

Like Loading...

Open Source Web Crawlers and Data Sets

15 Friday May 2015

Posted by Max Hemingway in Big Data, Data Science

≈ 1 Comment

Tags

Big Data, Data, Data Science

webA great list of 50 Open Source Web Crawlers has been produced by Baiju NT on a Big Data Blog

Web Crawlers are useful in gathering data from other sites when performing research, although caution should be used as with today’s levels of protection some sites defenses may consider your data gathering as an attack.

Its probably best to check first if any data sets exist with the data you are looking for.

https://www.quandl.com/ is a search engine for data sets that has listed 12 million data sets.

There are lots of data sets available from governments such as http://data.gov.uk/ in the UK.

If its a smaller list of good data sources is needed have a look at http://www.kdnuggets.com/datasets/index.html

Sources:

  • https://www.quandl.com/
  • http://www.kdnuggets.com/datasets/index.html
  • http://bigdata-madesimple.com/top-50-open-source-web-crawlers-for-data-mining/

Share this:

  • Twitter
  • Facebook
  • LinkedIn
  • Email
  • Pinterest

Like this:

Like Loading...

Data Mining Courses

28 Tuesday Apr 2015

Posted by Max Hemingway in Big Data, Data Science

≈ 1 Comment

Tags

Big Data, Data, Data Science, learning

mineVia Coursera the University of Illinois at Urbana-Champaign is running a specialisation on Data Mining.  As with all Coursera courses, you don’t have to take the specialisation, but can take the courses individually or one after each other. Taking the courses outside of the specialisation means that you wont get to complete the capstone project and earn your certificate at the end.

This track is made up 5 courses covering:

Pattern Discovery in Data Mining

  • Introduction to data mining
  • Concepts and challenges in pattern discovery and analysis
  • Scalable pattern discovery algorithms
  • Pattern evaluation
  • Mining flexible patterns in multi-dimensional space
  • Mining sequential patterns
  • Mining graph patterns
  • Pattern-based classification
  • Application examples of pattern discovery

Text Retrieval and Search Engines

  • Introduction to text data mining
  • Basic concepts in text retrieval
  • Information retrieval models
  • Implementation of a search engine
  • Evaluation of search engines
  • Advanced search engine technologies

Cluster Analysis in Data Mining

  • Basic concept and introduction
  • Partitioning methods
  • Hierarchical methods
  • Density-based methods
  • Probabilistic models and EM algorithm
  • Spectral clustering
  • Clustering high dimensional data
  • Clustering streaming data
  • Clustering graph data and network data
  • Constraint-based clustering and semi-supervised clustering
  • Application examples of cluster analysis

Text Mining and Analytics

  • Overview of text analytics and applications
  • Extending a search engine to support text analytics (text categorization, text clustering, text summarization)
  • Topic mining and analysis with statistical topic models
  • Opinion mining and summarization
  • Integrative analysis of text and structured data

Data Visualization

  • Visualization Infrastructure (graphics programming and human perception)
  • Basic Visualization (charts, graphs, animation, interactivity)
  • Visualizing Relationships (hierarchies, networks)
  • Visualizing Information (text, databases)

These courses would complement the courses from John Hopkins on Data Science

Source: https://www.coursera.org/specialization/datamining/20

Share this:

  • Twitter
  • Facebook
  • LinkedIn
  • Email
  • Pinterest

Like this:

Like Loading...

Big Data – 4V’s + Verification

27 Monday Apr 2015

Posted by Max Hemingway in Big Data, Data Science, IoT

≈ Leave a comment

Tags

Big Data, Data, Data Science, Infographic, IoT

IBM have released an Infographic on the “Four V’s of Big Data” which covers:

  • Volume – Scale of Data
  • Variety – Different forms of Data
  • Velocity – Analysis of Streaming Data
  • Veracity – Uncertainty of Data

4-Vs-of-big-data

There should be another V for “Verification” which covers the questions you ask of the data in order to obtain the results. A check should also be made on the data to look at the inference of the results as different views or questions asked in a slightly different way could produce completely different outcomes in the data.

Having the right data is important and ensuring the data gathered and collected is relevant to the business questions you are asking. Two stats in the infographic stick out for me on this:

  • $3.1 Trillion a year on poor data quality
  • 40 Zetabytes of data created by 2020

Perhaps with the right Verification there may not be so much uncertainty (Veracity) and a huge saving to businesses reducing a high loss in money, time and incorrect data.

Share this:

  • Twitter
  • Facebook
  • LinkedIn
  • Email
  • Pinterest

Like this:

Like Loading...

Do you know Big Data?

07 Tuesday Apr 2015

Posted by Max Hemingway in Big Data, Data Science, Tools

≈ Leave a comment

Tags

Big Data, Data, Data Science, Knowledge

Whilst looking into some suitable questions to ask about Big Data, I can across an excellent poster titled “Do you know Big Data?” produced by Altamira.

The poster covers a set of questions that help you question Big Data and a Big Data project.

  • What is Big Data?
  • What types of Big Data are there?
  • How do we extract knowledge from Big Data?
  • What do we do with knowledge we extract?
  • What types of Visual Techniques are there?
  • What types of Statistical Algorithms are there?
  • How big is Big Data?
  • What is a Data Scientist?
  • How do we implement Big Data solutions?
  • How do we address privacy and ethics in Big Data?
  • How do we secure Big Data?
  • What are leading Big Data tools?
  • What questions should we ask about Databases?
  • What questions about Predictive Tools?

bigdata

A useful tool as a starting place to research further elements of Big Data.

Share this:

  • Twitter
  • Facebook
  • LinkedIn
  • Email
  • Pinterest

Like this:

Like Loading...

Technology Couch Podcast

Technology Couch Podcast

Topical discussions with different guests on Technology

Chat and views on latest Technology trends, news and what is currently hot in the industry

Max Hemingway

  • Listen on Apple Podcasts
  • Podcast RSS Feed

RSS Feed

RSS Feed RSS - Posts

Currently Reading

@HemingwayReads

Other Publications I contribute to

https://sparrowhawkbushcraft.com/

Recent Posts

  • My Virtual Selfie – Avatars and Identity Security
  • Air Launching Satellites into Space – UK First
  • Our Acceptance of Modern Technologies
  • Pen based Productivity Tools – The Chronodex 2023
  • Top 10 Tech Podcasts for 2023

Categories

  • 21st Century Human
  • 3D Printing
  • Applications
  • Architecture
  • Arduino
  • Automation
  • BCS
  • Big Data
  • Certification
  • Cloud
  • Cobotics
  • Connected Home
  • Data
  • Data Fellowship
  • Data Science
  • Development
  • DevOps/OpsDev
  • Digital
  • DigitalFit
  • Drone
  • Enterprise Architecture
  • F-TAG
  • Governance
  • Health
  • Innovation
  • IoT
  • Machine Learning
  • Micro:Bit
  • Mindset
  • Mobiles
  • Networks
  • Open Source
  • Podcasts
  • Productivity
  • Programming
  • Quantum
  • Raspberry Pi
  • Robotics
  • Scouting
  • Scouts
  • Security
  • Smart Home
  • Social Media
  • Space
  • STEM
  • Tools
  • Uncategorized
  • Wearable Tech
  • Windows
  • xR

Archives

Reading Shelf

Archives

Recent Posts

  • My Virtual Selfie – Avatars and Identity Security
  • Air Launching Satellites into Space – UK First
  • Our Acceptance of Modern Technologies
  • Pen based Productivity Tools – The Chronodex 2023
  • Top 10 Tech Podcasts for 2023

Top Posts & Pages

  • Data Fellowship - BCS Level 4 Certificate in Data Analysis Tools
  • A formula for Innovation
  • Have you tried R yet?
  • Personal Knowledge Management System
  • The Nature and Cycle of CPD
  • Data Fellowship - BCS Level 4 Diploma in Data Analysis Concepts
  • My Virtual Selfie - Avatars and Identity Security

Category Cloud

21st Century Human Architecture Automation Big Data Cloud Cobotics Data Data Science Development DevOps/OpsDev Digital DigitalFit Enterprise Architecture Governance Innovation IoT Machine Learning Mindset Open Source Podcasts Productivity Programming Raspberry Pi Robotics Security Social Media STEM Tools Uncategorized Wearable Tech

Tags

# 3D Printing 21st Century Human Applications Architecture Automation BCS Big Data Blockchain Certification Cloud Cobot Cobotics Coding Communication Connected Home Continuous Delivery CPD Data Data Fellowship Data Science Delivery Development DevOps Digital DigitalFit Digital Human Docker Drone Email Encryption Enterprise Architecture Framework GTD Hashtag Infographic Information Theory Innovation IoT Journal Knowledge learning Machine Learning Micro:Bit MicroLearning Mindset Mixed Reality Networks Open Source OpsDev PKMS Podcasts Productivity Programming Proving It R RaspberryPI Robot Robotics Scouts Security Smart Home Social Media Standards Statistical Inference STEM Technology Couch Podcast Thinking Tools Training Visualisation Voice Wearable Tech Windows xR

License

Creative Commons Licence
This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.

Meta

  • Register
  • Log in
  • Entries feed
  • Comments feed
  • WordPress.com

Blog at WordPress.com.

  • Follow Following
    • Max Hemingway
    • Join 71 other followers
    • Already have a WordPress.com account? Log in now.
    • Max Hemingway
    • Customize
    • Follow Following
    • Sign up
    • Log in
    • Report this content
    • View site in Reader
    • Manage subscriptions
    • Collapse this bar
 

Loading Comments...
 

    %d bloggers like this: