Skip to content

Docs: Add details related to testing in docs #2596

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
kmr-srbh wants to merge 4 commits into
base: main
Choose a base branch
from
Draft

Docs: Add details related to testing in docs #2596

Changes from 1 commit
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
053f888
Decrease indentation of sections to improve visual clarity
kmr-srbh Mar 11, 2024
f4b8c88
Add detailed information on testing
kmr-srbh Mar 11, 2024
407bb77
Update installation.md
kmr-srbh Mar 11, 2024
ba4d09f
Merge branch 'main' into docs-enhancement
kmr-srbh Mar 21, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Add detailed information on testing
  • Loading branch information
@kmr-srbh
kmr-srbh authored Mar 11, 2024
commit f4b8c88b1a7ad4aa2d64ea0e1e3487b1d16163b4
284 changes: 151 additions & 133 deletions doc/src/installation.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,172 +3,190 @@
Follow the steps below to install and run LPython on Linux, Windows or macOS.

## Prerequisites
- ### Install Conda
Follow the instructions provided [here](https://github.com/conda-forge/miniforge/#download) to install Conda on your platform (Linux, macOS and Windows) using a conda-forge distribution called Miniforge.

For Windows, these are additional requirements:
- Miniforge Prompt
- Visual Studio (with "Desktop Development with C++" workload)

- ### Set up your system
- Linux
- Run the following command to install some global build dependencies:

```bash
sudo apt-get install build-essential binutils-dev clang zlib1g-dev
```
- Windows
- Download and install [Microsoft Visual Studio Community](https://visualstudio.microsoft.com/downloads/) for free.

- Run the Visual Studio Installer. Download and install the "Desktop Development with C++" workload which will install the Visual C++ Compiler (MSVC).

- Launch the Miniforge prompt from the Desktop. It is recommended to use MiniForge instead of Powershell as the main terminal to build and write code for LPython. In the MiniForge Prompt, initialize the MSVC compiler using the below command:

```bash
call "C:\Program Files\Microsoft Visual Studio\2022\Community\Common7\Tools\VsDevCmd" -arch=x64
```

You can optionally test MSVC via:

```bash
cl /?
link /?
```

Both commands must print several pages of help text.

- Windows with WSL
- Install Miniforge Prompt and add it to path:
```bash
wget https://github.com/conda-forge/miniforge/releases/latest/download/Miniforge3-Linux-x86_64.sh -O miniconda.sh
bash miniconda.sh -b -p $HOME/conda_root
export PATH="$HOME/conda_root/bin:$PATH"
conda init bash # (shell name)
```
- Open a new terminal window and run the following commands to install dependencies:
```bash
conda create -n lp -c conda-forge llvmdev=11.0.1 bison=3.4 re2c python cmake make toml clangdev git
```

- Optionally, you can change the directory to a Windows location using `cd /mnt/[drive letter]/[windows location]`. For e.g. - `cd mnt/c/Users/name/source/repos/`.
### Install Conda
Follow the instructions provided [here](https://github.com/conda-forge/miniforge/#download) to install Conda on your platform (Linux, macOS and Windows) using a conda-forge distribution called Miniforge.


- ### Clone the LPython repository
Make sure you have `git` installed. Type the following command to clone the repository:
For Windows, these are the additional requirements:
- Miniforge Prompt
- Visual Studio (with "Desktop Development with C++" workload)

### Set up your system
#### Linux
Run the following command to install some global build dependencies:
```bash
sudo apt-get install build-essential binutils-dev clang zlib1g-dev
```
#### Windows
- Download and install [Microsoft Visual Studio Community](https://visualstudio.microsoft.com/downloads/) for free.

- Run the Visual Studio Installer. Download and install the "Desktop Development with C++" workload which will install the Visual C++ Compiler (MSVC).

- Launch the Miniforge prompt from the Desktop. It is recommended to use MiniForge instead of Powershell as the main terminal to build and write code for LPython. In the MiniForge Prompt, initialize the MSVC compiler using the below command:

```bash
git clone https://github.com/lcompilers/lpython.git
cd lpython
call "C:\Program Files\Microsoft Visual Studio\2022\Community\Common7\Tools\VsDevCmd" -arch=x64
```

You may also use GitHub Desktop to do the same.

## Building LPython
- ### Linux and macOS
- Create a Conda environment:
You can optionally test MSVC via:

```bash
conda env create -f environment_unix.yml
conda activate lp
```
```bash
cl /?
link /?
```

- Generate the prerequisite files and build in Debug Mode:
Both commands must print several pages of help text.

```bash
# if you are developing on top of a forked repository; please run following command first
# ./generate_default_tag.sh
#### Windows with WSL
- Install Miniforge Prompt and add it to path:
```bash
wget https://github.com/conda-forge/miniforge/releases/latest/download/Miniforge3-Linux-x86_64.sh -O miniconda.sh
bash miniconda.sh -b -p $HOME/conda_root
export PATH="$HOME/conda_root/bin:$PATH"
conda init bash # (shell name)
```
- Open a new terminal window and run the following commands to install dependencies:
```bash
conda create -n lp -c conda-forge llvmdev=11.0.1 bison=3.4 re2c python cmake make toml clangdev git
```

- Optionally, you can change the directory to a Windows location using `cd /mnt/[drive letter]/[windows location]`. For e.g. - `cd mnt/c/Users/name/source/repos/`.


### Clone the LPython repository
Make sure you have `git` installed. Type the following command to clone the repository:

```bash
git clone https://github.com/lcompilers/lpython.git
cd lpython
```

You may also use GitHub Desktop to do the same.

## Building LPython
### Linux and macOS
- Create a Conda environment:

```bash
conda env create -f environment_unix.yml
conda activate lp
```

./build0.sh
./build1.sh
```
- Generate the prerequisite files and build in Debug Mode:

- ### Windows
- Create a Conda environment using the pre-existing file:
```bash
# If you are developing on top of a forked repository, please run following command first - ./generate_default_tag.sh

```bash
conda env create -f environment_win.yml
conda activate lp
```

- Generate the prerequisite files and build in Release Mode:
./build0.sh
./build1.sh
```

```bash
call build0.bat
call build1.bat
```
- ### Windows with WSL
### Windows
- Create a Conda environment using the pre-existing file:

- Activate the Conda environment:
```bash
conda activate lp
```
```bash
conda env create -f environment_win.yml
conda activate lp
```

- Run the following commands to build the project:
```bash
./build0.sh
cmake -DCMAKE_BUILD_TYPE=Debug -DWITH_LLVM=yes -DCMAKE_INSTALL_PREFIX=`pwd`/inst .\
make -j8
```
- Generate the prerequisite files and build in Release Mode:

## Tests
```bash
call build0.bat
call build1.bat
```
### Windows with WSL

- ### Linux and macOS
- Activate the Conda environment:
```bash
conda activate lp
```

- Run tests:
- Run the following commands to build the project:
```bash
./build0.sh
cmake -DCMAKE_BUILD_TYPE=Debug -DWITH_LLVM=yes -DCMAKE_INSTALL_PREFIX=`pwd`/inst .\
make -j8
```

```bash
ctest
./run_tests.py
```
## Testing
The LPython codebase broadly contains 3 types of tests:
1. **Integration tests:** Check whether the code compiles correctly and gives the required output at runtime.
2. **Compile time tests:** Check whether the code compiles correctly and generates the required compile time values for compile time evaluations.
3. **Error tests:** Check whether the code compiles correctly and throws the required error for invalid operations.

- Update test references:
```
./run_tests.py -u
```
Follow the instructions provided below to understand the method of writing and running tests.

- Run integration tests:
### Writing tests
Of the 3 types stated above, not all are always required to be written together. Follow the steps provided below to understand how each test is written:

```bash
cd integration_tests
./run_tests.py
```
- In case you have recently updated macOS, you may get a warning like below in some test cases:
```bash
ld: warning: object file (test_list_index2.out.tmp.o) was built for newer macOS version (14.0) than being linked (13.3)
```
This leads to mismatch of hashes with expected output in some test cases, this can be resolved by updating command line tools:
#### Integration tests
The integration tests are located inside `./integration_tests`. To write an integration test, follow the steps provided below:
1. **Write tests:** This is almost always the case when you want to include tests for new additions to a module, built-in functions or data-type attributes. If it does, just add a new function which tests for the changes you made. Add a function call for your function at the required position.

```bash
git clean -dfx
sudo rm -rf /Library/Developer/CommandLineTools # make sure you know what you're doing here
sudo xcode-select --install
./build.sh
./run_tests.py
```
If the file does not exist, create one and follow the steps stated above.

- Speed up Integration Tests on macOS
2. **Update test references:** After you are done writing your tests, update the test references through the following script.
```bash
./run_tests.py -u
```

Integration tests run slowly because Apple checks the hash of each
executable online before running.
#### Compile time tests
TODO: Learn more

You can turn off that feature in the Privacy tab of the Security and Privacy item of System Preferences > Developer Tools > Terminal.app > "allow the apps below to run software locally that does not meet the system's security
policy."
#### Error tests
The error tests are located inside `./tests/errors/`. You are required to write an error test for every such action for which your code intentionally raises an error. For example, if your code raises a ZeroDivisionError, you need to write an error test for that. An error test is simply a piece of code which causes the required error to be raised.

The procedure of writing an error test is the same as integration tests. One also needs to update the test references for the same.

### Running tests
#### Linux and macOS
- Run ctest:

- ### Windows
```bash
ctest
./run_tests.py
```
- Run integration tests:

- Run integration tests
```bash
cd integration_tests
./run_tests.py
```
> [!NOTE]
> In case you have recently updated macOS, you may get a warning like below in some test cases:
>
> ```bash
> ld: warning: object file (test_list_index2.out.tmp.o) was built for newer macOS version (14.0) than being linked (13.3)
> ```
> This leads to mismatch of hashes with expected output in some test cases, this can be resolved by updating command line tools:
>
> ```bash
> git clean -dfx
> sudo rm -rf /Library/Developer/CommandLineTools # make sure you know what you're doing here
> sudo xcode-select --install
> ./build.sh
> ./run_tests.py
> ```

> [!TIP]
> Integration tests run slowly on macOS because Apple checks the hash of each executable online before running.
>
> You can turn off that feature in the Privacy tab of the Security and Privacy item of `System Preferences > Developer Tools > Terminal.app > "Allow the apps below to run software locally that does not meet the system's security policy."`


#### Windows
- Run integration tests

```bash
python run_tests.py --skip-run-with-dbg
```
```bash
python run_tests.py --skip-run-with-dbg
```

- Update reference tests
- Update reference tests

```bash
python run_tests.py -u --skip-run-with-dbg
```
```bash
python run_tests.py -u --skip-run-with-dbg
```

## Examples (Linux and macOS)

Expand Down