All Posts (375)
-
Read more…
Imagine there are two girls standing in front of you – The first girl is cute, beautiful, interesting and has the smile that any guy would die for. And the other girl is average-looking, quiet, not-so-impressive… no different from the ones that you usually see in the restaurant cash counter. Which girl will you call out for a date? If you’re like me, you will choose the attractive girl. You see, life is full of options and making the right choice is what matters the most.
If you’re a Java developer, then you probably have more choices to make – like the switch from Java to Hadoop.
Big data and Hadoop are the two most popular buzzwords in the industry. Chances are that you have come across these two terms on the Java payscale forums or seen your senior colleagues making the switch to get bigger paychecks. I’ll tell you what, the upgrade from Java to Hadoop is not just about staying updated with the latest technology or getting appraisals – it’s about being competent…
-
Read more…
In this post I will share some tips I learned after using the Apache Hadoop environment for some years, and doing many many workshops and courses. The information here considers Apache Hadoop around version 2.9, but it could definably be extended to other similar versions.
These are considerations for when building or using a Hadoop cluster. Some are considerations over the Cloudera distribution. Anyway, hope it helps!
- Don't use Hadoop for millions of small files. It overloads the namenode and makes it slower. It is not difficult to overload the namenode. Always check capability vs number of files. Files on Hadoop usually should be more than 100 MB.
- You have to have a 1 GB of memory for around 1 million files in the namenode.…
-
Read more…
The SAP engineers as well as the software professionals at Sapphire Now have offlate been discussing the possible restrictions and likelihood for SAP S/4HANA for acting as a robust platform primarily for the purpose of managing the enterprise based strategies for Internet of Things as well as Big Data.
The general notion in the mindset of the people is that Internet of Things (IoT) is capable of sending out a tidal wave of enormous quantities of data in any enterprise. It is has been projected that billions of controllers and sensors, who are all capable of connecting to some other machine for the instruction and analysis, are eagerly waiting for the creation of trillions of data events along with transactions. The SAP managers and engineers has gone ahead and shared their views through a series of conversations when they were probed on how an organization could simply reconcile huge data quantities with in-memory built-for-speed…
-
Read more…
Here's a selection of Hadoop-related articles worth checking out. Enjoy the reading!
-
Read more…
This tutorial is provided by Guru99. Originally posted here.
Apache HADOOP is a framework used to develop data processing applications which are executed in a distributed computing environment.
In this tutorial we will learn,
- Components of Hadoop
- Features Of 'Hadoop'
- Network Topology In Hadoop
Similar to data residing in a local file system of personal computer system, in Hadoop, data resides in a distributed file system which is called as a Hadoop Distributed File system.
Processing model is based on 'Data Locality' concept wherein computational logic is sent to cluster nodes(server) containing data. This computational logic is nothing but a compiled version of a program written in a high level language such as Java. Such a program, processes data stored in Hadoop…
-
Read more…
Guest blog post by Mohammad Tariq Iqbal
Quite often, while working with Hbase, I used to feel how cool it would be to have a database that can replicate my data to datacenters across the world consistently. So that I can take the pleasure of global availability and geographic locality. And also which will save my data even in case of some catastrophe or natural disaster. Which supports general-purpose transactions, and provides a SQL-based query language. And which has features of an SQL database as well. But it was only untill recently I found out that it is not an imagination anymore.
I was sitting with a senior+friend of mine at a Cafe… -
Read more…
Originally posted here by Bernard Marr.
When you learn about Big Data you will sooner or later come across this odd sounding word: Hadoop - but what exactly is it?
Put simply, Hadoop can be thought of as a set of open source programs and procedures (meaning essentially they are free for anyone to use or modify, with a few exceptions) which anyone can use as the "backbone" of their big data operations.
I'll try to keep things simple as I know a lot of people reading this aren't software engineers, so I hope I don't over-simplify anything - think of this as a brief guide for someone who wants to know a bit more about the nuts and bolts…
-
Read more…
Guest blog post by Bernard Marr
Hadoop – the software framework which provides the necessary tools to carry out Big Data analysis – is widely used in industry and commerce for many Big Data related tasks.
It is open source, essentially meaning that it is free for anyone to use for any purpose, and can be modified for any use. While designed to be user-friendly, in its “raw” state it still needs considerable specialist knowledge to set up and run.
Because of this a large number of commercial versions have come onto the market in recent years, as vendors have created their own versions designed to be more easily used, or supplied alongside consultancy services to get you crunching through your data in no time.…
-
Read more…
Guest blog post by Michael Walker
Batch data processing is an efficient way of processing high volumes of data is where a group of transactions is collected over a period of time. Data is collected, entered, processed and then the batch results are produced (Hadoop is focused on batch data processing). Batch processing requires separate programs for input, process and…
-
Read more…
Guest blog post by Michael Walker
Big data analytical ecosystem architecture is in early stages of development. Unlike traditional data warehouse / business intelligence (DW/BI) architecture which is designed for structured, internal data, big data systems work with raw unstructured and semi-structured data as well as internal and external data sources. Additionally, organizations may need both batch and (near) real-time data processing capabilities from big data systems.…
-
Read more…
Data is an advantage for industries as it benefits them make up-to-date choices. Strange! Data is being produced at an extraordinary rate and establishments are hording it like there’s not any tomorrow, generating enormous data groups we call big data. But is big data serving these businesses or is it just obscuring the decision-making procedure? We will find out.
Big data has numerous applications and, collective with analytics, is cast-off to find responses to glitches in a variation of businesses. For organisations, it can benefit them comprehend customer behavior and get most out of business procedures, all of which, in concept, should help administrators make sound choices to drive business development. But like so numerous things that complete good in concept, it’s not precisely working out for numerous organisations. In a worldwide review of over 300 C-level initiative administrators by Chartered Global Management Accountant (CGMA), which was complemented…
-
Read more…
Price discrimination and downward demand spiral are widely used analytical concepts/practices in the Airlines and Hospitality industries respectively, long before the term Big Data Analytics was even coined. Incidentally, these concepts have been taught in global elite b-schools for decades. So, how come Analytics, which has been there in practice for decades experience a meteoric rise suddenly? To answer this question, we need to get the Big Picture. Given below are key factors that led to huge buzz around analytics today.
- Proliferation of Data Sources – Every day we create…
-
Read more…
The advent of sharing economy has brought a sea change in the way urban populace commute locally. The Ubers, Lyfts and many other local players have made taxi riding convenient, affordable and safe. These rides have emerged as a strong alternative to the public transport clocking millions of rides per month in some cities. The emergence of hyper-local delivery models to optimize the supply chain has also led to a large number of daily trips by these vehicles.
These developments have mandated the installations of either standalone or smartphone app-based GPS devices to keep track of and better regulate these rides and a fleet of taxis. These GPS systems spew a ton of data generating up to GBs of…
-
Read more…
The sudden increase in the volume of data from the order of gigabytes to zettabytes has created the need for a more organized file system for storage and processing of data. The demand stemming from the data market has brought Hadoop in the limelight making it one of biggest players in the industry. Hadoop Distributed File System (HDFS), the commonly known file system of Hadoop and Hbase (Hadoop’s database) are the most topical and advanced data storage and management systems available in the market.
What are HDFS and HBase?
HDFS is fault-tolerant by design and supports rapid data transfer between nodes even during system failures. HBase is a non-relational and open source Not-Only-SQL database that runs on top of Hadoop. HBase comes under CP type of CAP (Consistency, Availability, and Partition Tolerance) theorem.
HDFS is most suitable for performing…
-
Read more…
Introduction
By now, you have probably heard of the Hadoop Distributed File System (HDFS), especially if you are data analyst or someone who is responsible for moving data from one system to another. However, what are the benefits that HDFS has over relational databases?
HDFS is a scalable, open source solution for storing and processing large volumes of data. HDFS has been proven to be reliable and efficient across many modern data centers.
HDFS utilizes commodity hardware along with open source…
-
Read more…
Data is a key asset of any company, particularly transactional data which holds business secrets such as financial or health records. Data is most vulnerable in transit between the server that stores it and that client that requests it.
The standard approach to ensuring security is to encrypt data on the server and use the SSL-enabled HTTPS protocol to secure data in transport. However, what if we could increase the level of security even further, by using HTTPS and sending data in an encrypted format over the communication line, only to decrypt data on clients who have valid certificates? That approach would make a traditional man-in-the-middle (MITM) attack much more difficult.…
-
Read more…
Guest blog post by Bill Vorhies
Summary: The shortage of data scientists is driving a growing number of developers to fully Automated Predictive Analytic platforms. Some of these offer true One-Click Data-In-Model-Out capability, playing to Citizen Data Scientists with limited or no data science expertise. Who are these players and what does it mean for the profession of data science?
In a recent poll the question was raised “Will Data Scientists be replaced by software, and if so, when?” The consensus answer:
Data Scientists automated and unemployed by 2025.
Are we really just grist…
-
Read more…
Guest blog post by Bernard Marr
With an ever-growing number of businesses turning to Big Data and analytics to generate insights, there is a greater need than ever for people with the technical skills to apply analytics to real-world problems.
Computer programming is still at the core of the skillset needed to create algorithms that can crunch through whatever structured or unstructured data is thrown at them. Certain languages have proven themselves better at this task than others. Here’s a brief overview of 10 of the most popular and widely used.…
-
Read more…
Find below the list of Hadoop interview questions and answers jotted down to help job seekers
Question: What is Hadoop and its workings?
Answer: When “Big Data” appeared as problematic, Apache Hadoop changed as an answer to it. Apache Hadoop is a context which offers us numerous facilities or tools to store and development of Big Data. It benefits in analysing Big Data and creation business decisions out of it, which can’t be done professionally and successfully using old-style systems.
Question: What is the usage of Hadoop?
Answer: With Hadoop, the employer can run requests on the systems that have thousands of bulges scattering through countless terabytes. Rapid data dispensation and assignment among nodes helps continuous operation even when a node fails averting system let-down.
Question: On what idea, the Hadoop framework runs?
Answer: Hadoop Framework acts upon the subsequent two core…
-
Read more…
Suppose you need to create a high-load project based on a PHP MVC framework. You would probably use caching wherever possible. Maybe you would build the project in a single file, or maybe even write your own MVC framework with minimal functionality, or rewrite some parts of another framework. While, yes, this works, it’s a little bit tricky, isn’t it? Fortunately, there is one more solution that makes most of these manipulations unnecessary (save for the cache, perhaps), and this solution is called the PhalconPHP framework.
What Is PhalconPHP?
PhalconPHP is an MVC framework for PHP written in C and supplied as a compiled PHP extension. This is what makes it one of the fastest frameworks available (to be completely honest the fastest one is Yaf, but it is a micro framework and…
Resources
Research
