Downloaded 11 times
Uploaded byBarbara Fusinska
Analysing GitHub commits with R
This document discusses analyzing GitHub commit data using R. It covers capturing GitHub event data from sources like the GitHub Archive and API, processing the data in R, and analyzing metrics like the distribution of programming languages across active repositories. It also discusses using tools like Google BigQuery and Azure Machine Learning with R for querying larger datasets and building visualizations and models in the cloud. The overall goals are to demonstrate obtaining insights into open source project activity and technologies using R for data analysis.
More Related Content
Similar to Analysing GitHub commits with R
![[Databeers] 27-11-2014 “Githubland”. Luis Osa](https://cdn.slidesharecdn.com/ss_thumbnails/slidesgithubland-150108030230-conversion-gate02-thumbnail.jpg?width=640&height=640&fit=bounds)
More from Barbara Fusinska
Analysing GitHub commits with R
- 1. Analysing GitHub commitswith R @BasiaFusinska barbarafusinska.com [email protected]
- 2. About me Programmer Math enthusiast Sweettooth @BasiaFusinska • @BasiaFusinska • barbarafusinska.com • [email protected] https://github.com/BasiaFusinska/RTalk
- 3.
- 4. Agenda • Data analysis •Capturing GitHub data • Using R in Data Analysis • Azure ML
- 5. Data analysis process RawData Processed Data Data Analysis & Visualization Exploratory Data Analysis Data Capture
- 6. What do wewant to find out?
- 7.
- 8.
- 9.
- 10.
- 12. Event Data Data Source (Events/Logs/Files) Store& reorganize Query
- 13.
- 14.
- 15. Why R? • RossIhaka & Robert Gentleman • Name: – First letter of names – Play on the name of S – S-PLUS – commercial alternative • Open source • Nr 1 for statistical computing
- 16.
- 17. R Environment • Rproject – console environment – http://www.r-project.org/ • IDE – Any editor – RStudio http://www.rstudio.com/products/rstudio/download/
- 18.
- 19.
- 21.
- 23.
- 25. Distribution of activerepositories per language
- 26. What is anactive repository?
- 27.
- 28.
- 29.
- 30.
- 32. Task: Reading PullRequests 1. Read the file line by line and extract only pull request events 2. Extract id and language information 3. Count and visualise language distribution Data: 1h GitHub Archive Events from 01-01-2015, 3 PM
- 33.
- 34.
- 35.
- 36.
- 37.
- 39.
- 40.
- 41.
- 42. Now everything issorted…
- 43. Goal: Analyze ACTUALactive repositories
- 44.
- 45. Language information • Activerepositories – Create, Push and PullRequest events • Missing language information: – Google BigQuery – GitHub API • Process various data sources
- 46.
- 47. Different sources ofdata • GitHub Archive: – id, – url in a form: https://api.github.com/repos/:name – (rare cases) language • Google BigQuery: – no id, – url in a form: https://github.com/:name – language
- 48. Task: Reading ActiveRepositories 1. Read the file line by line and extract only create, push and pull request events 2. Extract id and url information 3. Read Google BigQuery data from saved file 4. Combine repositories data and Google data base on the same url and fill in missing language information 5. Count and visualise language distribution
- 49.
- 50.
- 51.
- 52.
- 53.
- 54.
- 55. Active repositories perlanguages
- 56.
- 57. Task: Retrieve languageinfo for repository • GET /repos/:owner/:repo/languages • Owner: BasiaFusinska • Repo: RWorkshop
- 58.
- 59. Task: Calling GitHubSearch • GET /search/repositories • Querying: q parameter • Paging: page parameter
- 60.
- 61.
- 62.
- 63. Big Data inR • What’s _Big_Data_ anyway? • R processes data in memory • Bring down only the data you need • Streaming the data from database
- 64. Azure Machine LearningStudio
- 65. Azure ML RScript
- 66.
- 67. Reading pull requestsexperiments
- 68.
- 69.
- 70. Pull requests -visualization
- 71. To summarize… • Datascience – not a rocket science, shaving the yak • Different sources – different truths • Capturing & storing data • Data science UI – visualization is the key • Desktop - hypothesis, development • Cloud – production
- 72. What’s next? • DataExploration in R, workshop • basiafusinska.com, blog • katacoda.com, interactive learning platform
- 73.