1. Setup and Installation

The purpose of this tutorial is to demonstrate the proper setup of your Python environment so that hs_process properly runs. In this tutorial, we will:

  1. Install hs-process

  2. Confirm you are using the correct Python installation

  3. Verify that the installation was successful by executing a test script


1.1. Installing with Anaconda/conda

With Anaconda installed, please proceed to install hs_process via conda-forge:

conda install -c conda-forge hs-process

This is the easiest, most complete, and most reliable method for installation because it installs all of the dependencies from the same conda-forge channel. Before installing, it is recommended to create a new environment (see the Python Environment setup below).

1.1.1. [Option 2] Installing via pip

hs_process can be installed via pip, but first the dependencies should be installed via the conda install command from the conda forge channel. Although pip install hs_process will try to install package dependencies available on PyPI, it is not generally recommended to install a package via pip if that package exists in the Anaconda Package Repository. In the Anaconda Console, execute each of the following commands:

  1. conda install -c conda-forge geopandas -y (*link*)

  2. conda install -c conda-forge seaborn -y (*link*)

  3. conda install -c conda-forge spectral -y (*link*)

Now that all of the dependencies are installed via conda, hs_process can finally be installed via pip:

  1. pip install hs_process (*link*) will install the remaining dependencies via pip. Note that pip should only be used after as many dependencies as possible are installed via conda install - see here for more information.


1.2. Confirming your environment

It can be easy to accidentally open a Python IDE (e.g., Spyder or Jupyter Notebook) for a Python environment different that where hs_process and its dependencies were installed. The following Python command can be used to confirm you are using the Python executable in your new environment:

[1]:
import sys

sys.executable
[1]:
'C:\\Users\\nigo0024\\AppData\\Local\\Continuum\\anaconda3\\envs\\spec\\python.exe'

1.3. Testing hs_process installation

To verify hs_process and its dependencies are properly installed, the test.py can be run. First get local access to test.py:

  1. Fork, clone, or download the hs_process repository.

  2. Download sample files and place into the newly forked/cloned/downloaded hs_process\test\testdata directory (do not change the file names).

*Note*: There is a hyperspectral datacube that is ~750 MB, and this is why this data is not placed in the testdata folder by default (Github doesn’t allow large files). The small files should be included from Github, but the .spec and .bip files must be copied over. After copying the sample files to your testdata directory, run test.py from your command prompt:

  1. python <path to hs_process\test\test.py> to run through some basic tests to be sure all dependencies are installed properly.


[2]:
%run F:\nigo0024\Documents\GitHub\hs_process\test\test.py  # optionally, a script can be run from Jupyter with this syntax
test_bip_extensions (test_hsio.Test_hsio_get_fname_hdr) ... ok
test_spec_extensions (test_hsio.Test_hsio_get_fname_hdr) ... ok
test_failure (test_hsio.Test_hsio_read_cube) ... expected failure
test_names (test_hsio.Test_hsio_read_cube) ... ok
test_readability (test_hsio.Test_hsio_read_cube) ... ok
test_split (test_hsio.Test_hsio_read_cube) ... skipped 'demonstrating skipping'
test_names (test_hsio.Test_hsio_read_spec) ... ok
test_readability (test_hsio.Test_hsio_read_spec) ... ok
test_byteorder (test_hsio.Test_hsio_set_io_defaults) ... ok
test_dtype (test_hsio.Test_hsio_set_io_defaults) ... ok
test_ext (test_hsio.Test_hsio_set_io_defaults) ... ok
test_force (test_hsio.Test_hsio_set_io_defaults) ... ok
test_instance_independence (test_hsio.Test_hsio_set_io_defaults) ... ok
test_interleave (test_hsio.Test_hsio_set_io_defaults) ... ok
test_return_values (test_hsio.Test_hsio_set_io_defaults) ... ok
test_write_cube (test_hsio.Test_hsio_write_cube) ... ok
test_write_spec (test_hsio.Test_hsio_write_spec) ...
C:\Users\nigo0024\AppData\Local\Temp\tmpl3m_8qps\Wells_rep2_20180628_16h56m_pika_gige_7.bip.hdr
Saving C:\Users\nigo0024\AppData\Local\Temp\tmpl3m_8qps\Wells_rep2_20180628_16h56m_pika_gige_7.bip
ok
test_write_tif_multi (test_hsio.Test_hsio_write_tif) ... ok
test_write_tif_single (test_hsio.Test_hsio_write_tif) ...
Saving C:\Users\nigo0024\AppData\Local\Temp\tmpvpeb_d9y\Wells_rep2_20180628_16h56m_pika_gige_7-mean.spec
Either `projection_out` is `None` or `geotransform_out` is `None` (or both are). Retrieving projection and geotransform information by loading `hsio.fname_in` via GDAL. Be sure this is appropriate for the data you are trying to write.

Either `projection_out` is `None` or `geotransform_out` is `None` (or both are). Retrieving projection and geotransform information by loading `hsio.fname_in` via GDAL. Be sure this is appropriate for the data you are trying to write.

ok

----------------------------------------------------------------------
Ran 19 tests in 1.248s

OK (skipped=1, expected failures=1)

1.4. Other information

1.4.1. Install Python

There are many ways to download and install Python. The Anaconda distribution is recommended becuase it can simplify the installation of package dependencies and streamlines the process of setting up the virtual environment. Anaconda also comes with Spyder and Jupyter notebooks, each of which make working with Python quite easy.

Download and install Python 3.7 via Anaconda. When installing, choose *install for only me* instead of install for all users, as this simplifies insallation of package dependencies.

Note: Due to dependency conflicts, hs_process seems to only work with Python 3.6 and Python 3.7 (as of February 2020).


1.4.2. Create a new environment (optional)

Creating a new environment is not strictly necessary, but given that installing other geospatial packages from different channels may cause dependency conflicts, it can be good practice to install the geospatial stack in a clean environment starting fresh.

A Python environment refers to the ecosystem consisting of a particular installed version of Python, as well as any third-party packages it has access to. In Python, every time a package is installed (e.g., pip install some_third_party_package, conda install -c conda-forge some_third_party_package, etc.), your Python environment is gaining functionality that builds upon the Python Standard Library.

This is great because the possibilities of what can be accomplished using Python is virtually limitless. Ideally, the list of packages in a given environment should only include those required in the project we are working on, not all packages that have ever been installed and are needed for any project you’ve ever worked on (this can be dozens or even hundreds of packages). A downside with too many packages is that it is inefficient to share your environment with someone else that would like to use the program. As time goes on, there may also be compatibility issues among package dependencies and your code that can be avoided by creating a tidy little Python environment for your project.

With Anaconda installed, it is first recommended (but not required) to create a new Python 3.7 environment before installing package dependencies. Open an Anaconda console from the start menu as an administrator by clicking Start->Anaconda, right-click Anaconda Console, and choose to Run as Administrator. In the Anaconda Console, execute the following commands:

  1. conda create -n spec python=3.7 to create an Anaconda Python 3.7 environment named spec (you can name this whatever you’d like).

  2. conda activate spec to activate the newly configured environment. IMPORTANT: Anytime you want to use hs_process, be sure to run conda activate spec prior to opening your Python IDE via the command line (e.g. Spyder, Jupyter Notebook, etc.). Otherwise, open your IDE via Anaconda Navigator, being sure the correct environment is active.


1.4.3. Install Spyder and/or Jupyter Notebook

Open Anaconda Navigator from the start menu by clicking Start -> Anaconda -> Anaconda Navigaor. If Spyder or Jupyter Notebook are not installed on your new “spec” environment (if you created a new environment), select the “Install” button, then “Launch” to open the application to run Python in your selected environment.


1.4.4. System Requirements

hs_process was developed using Python 3 (on Windows 10 OS), so it is recommended to install and use Python 3. This tutorial has only been tested on Windows. It will likely work on other major platforms (i.e., Windows, Linux, Mac), but you will have to be sure to get the required dependencies installed and working on your own.


1.4.5. Software and Libraries

The following software and libraries are required to leverage the full API of hs_process:

Software

Libraries

  1. GeoPandas

  2. Seaborn (if only for visualizing data in the API examples)

  3. Spectral Python

The following libraries are also dependencies of hs_process, but they are also dependencies of GeoPandas, Seaborn, and Spectral. Therefore they will be installed during the install procedures of the above libraries.

  • GDAL *GeoPandas has a GDAL dependency

  • Numpy *GDAL has a numpy dependency

  • Pandas *GeoPandas has a Pandas dependency

  • Matplotlib *Seaborn has a Matplotlib dependency

  • Shapely *GeoPandas has a Shapely dependency


1.4.6. Other methods for installing hs_process

IMPORTANT: Before installing hs_process via any of the following methods, be sure the GeoPandas and Fiona dependencies are installed via conda (see GeoPandas and Fiona dependencies below).

  • pip install git+https://github.com/tnigon/hs_process *(link)* will install the latest development version directly from Github.

OR another option is to clone the Github repository and install from your local copy. After navigating to the directory of your cloned local copy:

  • pip install .

The recommended folder directory for the hs_process package is in the site-packages folder in your Python Path (alongside all other Python packages).


1.4.7. GeoPandas and Fiona dependencies

IMPORTANT: The GeoPandas and Fiona dependencies MUST installed via conda. GeoPandas has a dependency on Fiona, and because of some nuances with the pip version of Fiona*, GeoPandas MUST be installed via Anaconda (at least on Windows OS).

  • conda install -c conda-forge geopandas *(link)* will install both GeoPandas and fiona because GeoPandas has a fiona dependency.

*During installation, the pip version of Fiona requires that it is able to discover the locations of header files and libraries needed to compile its C extnesions, and these must be provided by the user when installing on Windows. For more information on this behavior, please refer to the Fiona documentation.