Downloaded 245 times
Uploaded byApache Apex
Intro to Big Data Hadoop
The document provides an overview of big data, its definitions, and characteristics, highlighting the challenges and the need for alternative processing methods due to its volume, velocity, and variety. It also introduces distributed systems, describing them as collections of independent computers that function as a single system, emphasizing the importance of transparency and collaboration among components. Additionally, it discusses the implications of big data on decision-making and operational efficiency.
More Related Content
Similar to Intro to Big Data Hadoop
![Vibe Coding vs. Spec-Driven Development [Free Meetup]](https://cdn.slidesharecdn.com/ss_thumbnails/vibecodingvsspecdrivendevelopment-251209105622-43f455e7-thumbnail.jpg?width=640&height=640&fit=bounds)
Intro to Big Data Hadoop
- 1. Dr. Sandeep G.Deshmukh Introduction to
- 2. Contents ❑ Big Data ❑Distributed Systems ❑ Hadoop ➢ Hadoop Distributed File System (HDFS) ➢ MapReduce 2
- 3.
- 4.
- 5. Big data isdata that exceeds the processing capacity of conventional database systems. The data is too big, moves too fast, or doesn’t fit the strictures of your database architectures. To gain value from this data, you must choose an alternative way to process it. https://www.oreilly.com/ideas/what-is-big-data Definition
- 6. Quantity of data Datasets too large to store and analyze using traditional databases Volume
- 7. Velocity Speed at whichdata is generated Speed at which data is moving around and analyzed Analyze data while it is being generated without even putting it into databases
- 8. Variety Different types ofdata that we can use
- 9. Veracity Messiness or trustworthinessof the data Volume makes up for quality Eg. Tweets with spelling mistakes, short words ( u -> you, thr-> there)
- 10. Value Getting value outof Big Data!!!
- 11. Definition “Big data” is high-volume,-velocity and -variety information assets that demand cost-effective, innovative forms of information processing for enhanced insight and decision making By Gartner
- 12. Definition Big data isa term for data sets that are so large or complex that traditional data processing applications are inadequate Challenges include analysis, capture, data curation, search,sharing, storage, transfer, visualization, querying, updating and information privacy. The term often refers simply to the use of predictive analytics or certain other advanced methods to extract value from data, and seldom to a particular size of data set. Accuracy in big data may lead to more confident decision making, and better decisions can result in greater operational efficiency, cost reduction and reduced risk. Wikipedia
- 13. Use Case: BigData in Oil & Gas Drilling http://analytics-magazine.org/images/stories/novdec12/big-data.jpg
- 14. Use Case: Uber- Pay Surge Pricing if Battery is Low
- 15. ● A BriefHistory of Big Data Everyone Should Read ● Beyond Volume, Variety and Velocity is the Issue of Big Data Veracity ● What is big data? - OpenSource.com ● What is big data? - O’Reilly ● 5 Big Data Use Cases To Watch ● Best Big Data Analytics Use Cases ● The 5 game changing big data use cases ● Big Data - The 5 Vs Everyone Must Know ● Top SlideShare Presentations on Big Data Further Reading
- 16.
- 17. A distributed systemis a collection of independent computers that appears to its users as a single coherent system. Distributed Systems: Principles and Paradigms, 2nd Edition, Andrew S. Tanenbaum, Maarten Van Steen, 2006 http://www.mypearsonstore.com/bookstore/distributed-systems-principles-and-paradigms-9780132392273?xid=PSED Definition
- 18. Distributed Systems: Principlesand Paradigms, 2nd Edition, Andrew S. Tanenbaum, Maarten Van Steen, 2006
- 19. Transparency Description Access Hidedifferences in data representation and how a resource is accessed Location Hide where a resource is located Migration Hide that a resource may move to another location Relocation Hide that a resource may be moved to another location while in use Replication Hide that a resource is replicated Concurrency Hide that a resource may be shared by several competitive users Failure Hide the failure and recovery of a resource Forms of Transparency in Distributed Systems
- 20. ● A distributedsystem consists of components (i.e., computers) that are autonomous ● Users (be they people or programs) think they are dealing with a single system. This means that one way or the other the autonomous components need to collaborate. How to establish this collaboration lies at the heart of developing distributed systems.
- 21. A distributed systemis a model in which components located on networked computers communicate and coordinate their actions by passing messages. The components interact with each other in order to achieve a common goal. Three significant characteristics of distributed systems are: concurrency of components, lack of a global clock, and independent failure of components. Wikipedia https://www.oreilly.com/ideas/what-is-big-data Definition
- 22. ● Distributed Computing- Wikipedia ● Distributed computing ● Characteristics of distributed system Further Reading
- 23.
- 24. Big Data Primers:Size does matter
- 25. Big Data Primers:Vertical Vs Horizontal Scaling Vertical Scaling Horizontal Scaling
- 26. Big Data Primers:The scale of infrastructure