Adjust to ixmp Backend/Model refactor by khaeru · Pull Request #249 · iiasa/message_ix

khaeru · 2019-10-25T06:53:27Z

iiasa/ixmp#182 modularizes ixmp.core into abstract Backend and Model classes, and concrete JDBCBackend and GAMSModel classes. This PR adjusts message_ix accordingly.

How to review

Read the docs preview: https://message-ix-khaeru.readthedocs.io/en/feature-ixmp-backend/
Run the tests.
Look at the code/diff.
- The CI-related files (.appveyor.yml, .travis.yml, ci/*, setup.cfg and setup.py) can be skimmed/ignored; they mainly copy what is done in ixmp.

Details

message_ix jumps stack layers in a few places by communicating directly with ixmp_source.

Click to show source…

- MsgScenario.getTypeList in `cat_list`: https://github.com/iiasa/message_ix/blob/570f79833eddf046f7779db3f68fb86c0b13b0cc/message_ix/core.py#L94 - MsgScenario.addCatEle in `add_cat`: https://github.com/iiasa/message_ix/blob/570f79833eddf046f7779db3f68fb86c0b13b0cc/message_ix/core.py#L111 - MsgScenario.getCatEle in `cat`: https://github.com/iiasa/message_ix/blob/570f79833eddf046f7779db3f68fb86c0b13b0cc/message_ix/core.py#L123 - MsgScenario.getFirstModelYear in `firstmodelyear`: https://github.com/iiasa/message_ix/blob/570f79833eddf046f7779db3f68fb86c0b13b0cc/message_ix/core.py#L272 - MsgScenario.getTecActYrs. Note this was incorrectly in **ixmp**.Scenario; iiasa/ixmp#182 deletes it in ixmp.

The code here:

Modifies message_ix.Scenario:
- Adds a method ._backend(name, ...) that calls backend API methods named ixmp.Backend.ms_{name}(...).
- Moves .years_active() from ixmp.Scenario.
  - Instead of calling the ixmp_source method getTecActYrs (closed source; perpetuates a stack level violation in the Backend), I've just reimplemented this (open-source) in message_ix.
- Deletes .remove_solution()—it only duplicated ixmp.Scenario.
- Significantly simplifies .clone(). Now:
  - message_ix.Scenario.clone only exists as a placeholder for its docstring.
  - ixmp.Scenario.clone handles valid combinations of arguments; so these are the same for ixmp.Scenario and message_ix.Scenario
  - JDBCBackend.s_clone raises exceptions for specific limitations of ixmp_source: https://github.com/iiasa/ixmp/blob/962f739bf71a1718b1c3fdf8f6e73e983986fc6e/ixmp/backend/jdbc.py#L321-L333
Deletes message_ix.model_settings and adds message_ix.models containing two subclasses of ixmp.GAMSModel named MESSAGE and MESSAGE_MACRO.
Also:
- message_ix.default_model_paths is simplified using pathlib. This should eventually be delegated to ixmp.config.Config.
- Handling of CPLEX optfiles is moved into the new MESSAGE model class.
- ixmp documentation is not repeated in message_ix; instead, linked with intersphinx.
- CI configuration is revised to mirror ixmp; this includes adding Codecov, thus closes Set up Codecov CI configuration and badge #238.

PR checklist

Tests added.
Documentation added.
Release notes updated.

codecov · 2019-10-26T16:24:14Z

Codecov Report

❗ No coverage uploaded for pull request base (master@742fb89). Click here to learn what that means.
The diff coverage is 93.93%.

@@            Coverage Diff            @@
##             master     #249   +/-   ##
=========================================
  Coverage          ?   66.72%           
=========================================
  Files             ?       15           
  Lines             ?     1139           
  Branches          ?        0           
=========================================
  Hits              ?      760           
  Misses            ?      379           
  Partials          ?        0

Impacted Files	Coverage Δ
message_ix/cli.py	`65.21% <0%> (ø)`
message_ix/testing.py	`100% <100%> (ø)`
message_ix/__init__.py	`100% <100%> (ø)`
message_ix/default_path_constants.py	`100% <100%> (ø)`
message_ix/tools/add_year/__init__.py	`62.43% <50%> (ø)`
message_ix/core.py	`89.17% <94.11%> (ø)`
message_ix/models.py	`97.05% <97.05%> (ø)`

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 742fb89...d18f6f6. Read the comment docs.

gidden

Hey @khaeru, this looks absolutely superb. I couldn't find any suggestions in the code specifically, but I did notice one doc page didn't render as expected (or I think so).

One question here - how will this play with the now languishing MACRO PR #223? Is it worth pulling that in as is (given that it is no worse and arguably better than present day) and then adjusting this PR as needed? I'm not sure I have the bandwith to do it the other way around, and am afraid that it simply die on the vine.

gidden · 2019-11-01T08:24:18Z

+   The MESSAGE Python class encapsulates the GAMS code for the core MESSAGE mathematical formulation.
+   The *model_options* arguments are received from :meth:`.Scenario.solve`, and—except for *solve_options*—are passed on to the parent class :class:`~ixmp.model.gams.GAMSModel`; see there for a full list of options.
+
+.. autoclass:: MESSAGE_MACRO


This looks to be broken on RTD? https://message-ix-khaeru.readthedocs.io/en/feature-ixmp-backend/api.html#message_ix.models.MESSAGE.defaults

Good eye. The code is working as intended, but that's indeed not what we want the user to be seeing in the documentation. I'll try for a fix.

khaeru · 2019-11-01T09:29:52Z

how will this play with the MACRO PR #223?

A rebase/merge from master might be needed after this goes in. I'll add some comments over there.

khaeru · 2019-11-01T11:01:30Z

See the above comment on IAMconsortium/pyam#281; IAMconsortium/pyam#280 broke the build, so I will now try to add a workaround.

khaeru · 2019-11-01T11:35:21Z

Okay, that worked. Thanks @gidden for the quick review.

khaeru added 6 commits October 25, 2019 01:18

Add message_ix.models module

d21a60b

Adjust tools.add_year for ixmp API changes

24cea0a

Add Scenario._backend, .years_active; delete .remove_solution

5f183fd

Delete Scenario.clone

f24e21e

Use Backend.ms_cat_set_elements

130280b

Use Backend.ms_* methods; remove Scenario._jobj shim

c5d5f28

khaeru self-assigned this Oct 25, 2019

khaeru mentioned this pull request Oct 25, 2019

Add Backend, JDBCBackend, Model, GAMSModel classes iiasa/ixmp#182

Merged

15 tasks

khaeru added 22 commits October 25, 2019 17:24

Re-add thin Scenario.clone() wrapper

cc1994c

Tidy CI files per ixmp

6541ee7

Fix pip VCS URLs

3930a85

Update Travis caching and env.sh

74918c8

Update ci/requirements.txt

01666a9

Update ci/requirements.txt 2

6c39d70

Update ci/requirements.txt 3

b047994

Add ci/appveyor-install.ps1 to echo ixmp

14ec6b7

On AppVeyor, update pip using conda

e413a0a

On Travis, update pip using conda

3db579a

Add numpy to explicit requirements

732f10d

Add conda info to Travis for debugging

c2c8c0f

Travis: install dependencies using conda

935a2b1

Split requirements.txt

0e232a7

Give conda --yes

6e717ae

Add pip to testing conda env

0195dab

Install conda-requirements into testing conda env

190d19d

Install pip-requirements into testing conda environment

c2e4596

Update doc/requirements.txt

6ff7c40

Add to conda-requirements

6aa1356

Adjust docs cross-reference to ixmp

2ac6df8

Convert Path to str in message_ix.cli

109ad59

khaeru added this to the 2.0 milestone Oct 26, 2019

Append conda-forge to default channels for JPype

8ced964

khaeru added 13 commits October 27, 2019 00:16

Provide firstmodelyear using Scenario.cat

5724faf

Adjust coveragerc

7b3ad94

Update RELEASE_NOTES

3e588c5

Coerce to int in Scenario.firstmodelyear

fcc13f3

Remove Scenario._backend (now handled by ixmp)

eeee361

Adjust add_par signature in testing.make_westeros

3ea2c0a

Compute years_active in Scenario, rather than using backend

896e28b

Update documentation; remove parallel copy of ixmp API docs

a311227

Move CPLEX optfile code to MESSAGE class; adjust docs

b58c108

Fix small docs typos

cc6f7a3

Further docs improvements; remove _init_scenario (no-op)

d7503ff

Return CI to ixmp@master

4ddb2eb

Proofread docs and comments

81f1506

gidden approved these changes Nov 1, 2019

View reviewed changes

khaeru added 2 commits November 1, 2019 11:06

Update API docs per @gidden reivew

2cb403b

Further improve API docs

4631353

khaeru mentioned this pull request Nov 1, 2019

scatter plotting tests fail for recent matplotlib version IAMconsortium/pyam#281

Closed

Work around IAMconsortium/pyam#280 / IAMconsortium/pyam#281

d18f6f6

khaeru merged commit 3bd21e5 into iiasa:master Nov 1, 2019

khaeru deleted the feature/ixmp-backend branch November 1, 2019 11:39

This was referenced Nov 1, 2019

Travis cron CI is broken #252

Closed

Disentangle message_ix and ixmp_source #254

Open

Connect MACRO explicitly to MESSAGEix Scenarios #223

Merged

OFR-IIASA mentioned this pull request Dec 11, 2019

years_active() returns wrong years #278

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Adjust to ixmp Backend/Model refactor#249

Adjust to ixmp Backend/Model refactor#249
khaeru merged 47 commits into
iiasa:masterfrom
khaeru:feature/ixmp-backend

khaeru commented Oct 25, 2019 •

edited

Loading

Uh oh!

codecov Bot commented Oct 26, 2019 •

edited

Loading

Uh oh!

gidden left a comment

Uh oh!

gidden Nov 1, 2019

Uh oh!

khaeru Nov 1, 2019

Uh oh!

khaeru commented Nov 1, 2019

Uh oh!

khaeru commented Nov 1, 2019

Uh oh!

khaeru commented Nov 1, 2019

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

khaeru commented Oct 25, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

How to review

Details

PR checklist

Uh oh!

codecov Bot commented Oct 26, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

gidden left a comment

Choose a reason for hiding this comment

Uh oh!

gidden Nov 1, 2019

Choose a reason for hiding this comment

Uh oh!

khaeru Nov 1, 2019

Choose a reason for hiding this comment

Uh oh!

khaeru commented Nov 1, 2019

Uh oh!

khaeru commented Nov 1, 2019

Uh oh!

khaeru commented Nov 1, 2019

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

khaeru commented Oct 25, 2019 •

edited

Loading

codecov Bot commented Oct 26, 2019 •

edited

Loading