Implemented API revisions by antaresc · Pull Request #57 · datacommonsorg/api-python

antaresc · 2019-07-29T12:07:45Z

No description provided.

datacommons/utils.py

datacommons/examples/places_basic.py

shifucun · 2019-07-29T17:34:18Z

datacommons/places.py

+  payload = utils.format_response(res)
+
+  # Create the results and format it appropriately
+  result = utils.format_expand_payload(payload, 'place', must_exist=dcids)


what is must_exists used for?

Ah it ensures that certain dcids are returned in the payload. I'll remove this once the right output is returned by the mixer.

shifucun · 2019-07-29T17:34:46Z

datacommons/populations.py

-  if new_col_name in self._dataframe:
-    raise ValueError('{} is already a column name.'.format(new_col_name))
+def get_populations(dcids, population_type, pv_map, reload=False):
+  """ Returns a list of populations associated with the given dcids.


this could be optional map for populations with no pvs

Good point, changed

Ok for now, but left a comment in notebook about "pv_map" not being an informative var name. Can change later after discussion

Spaceenter · 2019-07-29T21:17:15Z

datacommons/core.py

+    return payload['inArcs']
+  return []
+
+def get_property_values(dcids, prop, outgoing=True, value_type=None, reload=False, limit=utils._MAX_LIMIT):


line size <=80

ditto for the rest of the PR.

Spaceenter · 2019-07-29T21:18:53Z

datacommons/core.py

+
+  Args:
+    dcid: The node to get triples for.
+    reload: Whether or not to send the query to cache.


This is confusing. Reading this feels like "reload=true" sends query to cache, which is not true.

Agreed, fixed to say "A flag that sends the query without hitting cache when set."

Removed reload from the API

Spaceenter · 2019-07-29T21:19:58Z

datacommons/examples/core_basic.py

+
+  # Print the first 10 triples associated with Santa Clara County
+  print_header('Triples for "containedInPlace" of Santa Clara County')
+  triples = dc.get_triples(dcid, reload=True)


No need to set "reload=True"?

Removed reload from the API

Spaceenter · 2019-07-29T21:21:50Z

datacommons/examples/populations_pandas.py

+pd.options.display.max_rows = 10
+pd.options.display.max_colwidth = 30
+
+def print_header(label):


Seems like we have print_header in every example py file. Could it be extracted out to a util lib?

Yeah this sounds good.

Spaceenter · 2019-07-29T21:22:48Z

datacommons/places.py

+
+  Args:
+    dcids: List of dcids to get contained in places of.
+    place_type: The type of place to query for.


s/place/places

tjann · 2019-07-30T20:36:16Z

datacommons/core.py

+
+# ----------------------------- WRAPPER FUNCTIONS -----------------------------
+
+def get_property_labels(dcid, outgoing=True, reload=False):


I guess the assumption is that a node will never have too many properties. For now that seems more than reasonable, but stands out because almost all other methods have a limit on return length.

tjann · 2019-07-31T00:08:12Z

datacommons/populations.py

-                     new_col_name,
-                     observation_date,
-                     measured_property,
-                     stats_type=None,


Nit: can you swap measured_property and obsevation_date, then move stats_type down? That way we have logical flow of observation_date followed by observation_period

* Begin API rewrite * Implemented core functions. * Implemented pandas helpers * Finished implementation of core functions and added examples * Dropped old examples * Implemented get_places_in * Implemented get_population and get_observation * Temporary fix until colab supports 0.25 * Addressed comments from PR.

* Added ability to specify childhoodLocation * Never change python * Adding dockerfiles * Updated workspace * Updated workspace * Fixed test * Added verbose output * Python3 * Incremented version number * feature/api-version-2 (#47) * Corrected namespace for Intangible, hyperlinked a definition. * Added schema.datacommons.org/County. * Added documentation for dcid and provenance. * Added documentation for area, timezone, freebaseId, geonamesId, and wikidataId. * Fixed name of geonamesId. * Removed 'current' from the definition of timezone. * Fixed the empty column bug (#24) * Add GNIS property. * Add fipsId property. * Use updated caching api * Allow space in column name * Remove print * Fixed <br> to <br/> for xmllint (our rdf parsing is xhtml-based). (we may need to handle html in comments differently too) * Preparation for use with sdoapp * tweaks * Test of changes visibility * Test changes * Added config file to repo * Mods to roduce correct (vocabUri based) RDFa * Update datacommons.py Add a 'typeOf' literal when querying for the type of the container place for get_places_in(). * Add missing "()" to a query in get_places_in(). * Update datacommons client to be compatible with new schema * Added ability to specify childhoodLocation (#34) * Added ability to specify childhoodLocation * Never change python * Add CensusTract to get_places_in API * Handle error case * Add comment * added empty examples file for consistancy * Adjust config for release * Added temporary test config * Add deploymeny yaml files * Added draft deployment instructions file * Change deployment file from html to md * For observations, use observation_date instead of start_time/end_time * Use prod client * Use orient split in read and save dataframe so index is not saved in json * Add measurementMethod in get_observation * Update client library to fit for new mixer string_value * Fix header prefix and date format * docstring typos and consistency * Reimplemented base API * Implemented places extension * Fixed bug * Added bio stub * Finished re-implementing bio extension... n o w t o t e s t. * Added examples and fixed DCFrame bugs * added populations stub for pop extension * implemented get_pop and get_obs in populations API n o w t o t e s t * Fixed dangling line in places * BioExtension demo works * Some tweaks * Fixed header comment * missing comma * fixed typo self._col_type -> self._col_types * seed and new col val already have ? append to beg * fixing populations library and updating infra * similar fix for get_obs for extra ? for colvar should have fixed this when I fixed get_populations... * get useful prints in test/examples * fat fingers on copy and paste * places.py self._col_type -> self._col_types * Implemented draft of weather API extension * Added weather example * Weather API works * Added bio mixer specs and tweaked bio API * Fixed places bug * Make SchoolDistrict containedIn County * Feature/api version 2 (#50) * Corrected namespace for Intangible, hyperlinked a definition. * Added schema.datacommons.org/County. * Added documentation for dcid and provenance. * Added documentation for area, timezone, freebaseId, geonamesId, and wikidataId. * Fixed name of geonamesId. * Removed 'current' from the definition of timezone. * Fixed the empty column bug (#24) * Add GNIS property. * Add fipsId property. * Use updated caching api * Allow space in column name * Remove print * Fixed <br> to <br/> for xmllint (our rdf parsing is xhtml-based). (we may need to handle html in comments differently too) * Preparation for use with sdoapp * tweaks * Test of changes visibility * Test changes * Added config file to repo * Mods to roduce correct (vocabUri based) RDFa * Update datacommons.py Add a 'typeOf' literal when querying for the type of the container place for get_places_in(). * Add missing "()" to a query in get_places_in(). * Update datacommons client to be compatible with new schema * Added ability to specify childhoodLocation (#34) * Added ability to specify childhoodLocation * Never change python * Add CensusTract to get_places_in API * Handle error case * Add comment * added empty examples file for consistancy * Adjust config for release * Added temporary test config * Add deploymeny yaml files * Added draft deployment instructions file * Change deployment file from html to md * For observations, use observation_date instead of start_time/end_time * Use prod client * Use orient split in read and save dataframe so index is not saved in json * Add measurementMethod in get_observation * Update client library to fit for new mixer string_value * Fix header prefix and date format * docstring typos and consistency * Reimplemented base API * Implemented places extension * Fixed bug * Added bio stub * Finished re-implementing bio extension... n o w t o t e s t. * Added examples and fixed DCFrame bugs * added populations stub for pop extension * implemented get_pop and get_obs in populations API n o w t o t e s t * Fixed dangling line in places * BioExtension demo works * Some tweaks * Fixed header comment * missing comma * fixed typo self._col_type -> self._col_types * seed and new col val already have ? append to beg * fixing populations library and updating infra * similar fix for get_obs for extra ? for colvar should have fixed this when I fixed get_populations... * get useful prints in test/examples * fat fingers on copy and paste * places.py self._col_type -> self._col_types * Implemented draft of weather API extension * Added weather example * Weather API works * Added bio mixer specs and tweaked bio API * Fixed places bug * Added data cleaning helpers * Implemented DCNode constructor and get_property_values (#51) * Added cloud build files * Implemented DCNode constructor and get_property_values * Added additional error to constructor * Remove limit adjustment * Fixed a bug in eq * Implemented DCQuery, DCNode, DCFrame (#53) * Added additional error to constructor * Remove limit adjustment * Fixed a bug in eq * Implemented basic implementation of get_triples * DCNode implemented * Implemented DCFrame * Implemented DCQuery * Implemented get_places_in mixin * Fixed bug in DCFrame * fix a few errors on the first look at the code * Rename DCFrame, DCNode, DCQuery to Frame, Node, Query, and fix relevant calls. * Remove duplicate definitions of constants. * Remove methods that are not likely to be used often. * Implemented API revisions (#57) * Begin API rewrite * Implemented core functions. * Implemented pandas helpers * Finished implementation of core functions and added examples * Dropped old examples * Implemented get_places_in * Implemented get_population and get_observation * Temporary fix until colab supports 0.25 * Addressed comments from PR. * add absolute path import for py2 compatibility, other imports for google readibility * Updated docstrings to be more clear (#58) * Fixed a bug in populations example (#59) * added flexibility to use single col dataframe, convert to series for user * Added unit tests (#61) * Added unit tests to the Python Client API * Remove commented code * Added requests to requirements * Added line to Dockerfile * Updated fields to match new mixer version. * Added comments and added in PY2 support imports. * Clarified comment in get_populations * Clarified triple. * update API endpoint * Added sphinx documentation (#62) * Fixed comment in core example * Added Sphinx documentation draft * Updated conf.py * Fixed unit tests * Updated main splash page * Separated files out * Subtitles * Fixed get_places_in comments * Fixed comments on all functions * Formatted dcid * Point resources to dev2 * Updated client API to reflect new return format of GetPropertyValues (#63) * Added more documentation (#64) * Added more documentation * Fixed docs bug * Fixed grammar * Check in R API Client code (#69) * update gitignore with R stuff * check in R code * beta note * Removed out as key from GetPropertyValues (#70)

antaresc added 7 commits July 29, 2019 00:51

Begin API rewrite

cbd5177

Implemented core functions.

107ec71

Implemented pandas helpers

a7a1f3d

Finished implementation of core functions and added examples

bb73e17

Dropped old examples

b3665ac

Implemented get_places_in

8223612

Implemented get_population and get_observation

80a7753

antaresc requested review from Spaceenter, shifucun and tjann July 29, 2019 12:08

Temporary fix until colab supports 0.25

1d0ee3f

shifucun reviewed Jul 29, 2019

View reviewed changes

datacommons/utils.py Show resolved Hide resolved

shifucun reviewed Jul 29, 2019

View reviewed changes

Spaceenter approved these changes Jul 29, 2019

View reviewed changes

tjann approved these changes Jul 30, 2019

View reviewed changes

tjann reviewed Jul 31, 2019

View reviewed changes

Addressed comments from PR.

25edc7e

antaresc merged commit 51919c4 into datacommonsorg:dev2 Aug 1, 2019


		# ----------------------------- WRAPPER FUNCTIONS -----------------------------

		def get_property_labels(dcid, outgoing=True, reload=False):

Conversation

antaresc commented Jul 29, 2019

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Comments