hs_process package

Submodules

hs_process.batch module

class hs_process.batch.batch(base_dir=None, search_ext='.bip', dir_level=0, lock=None, progress_bar=False)

Bases: object

Class for batch processing hyperspectral image data. Makes use of segment, spatial_mod, and spec_mod to batch process many datacubes in a given directory. Supports options to save full datacubes, geotiff renders, as well as summary statistics and/or reports for the various tools.

Note

It may be a good idea to review and understand the defaults, hsio, hstools, segment, spatial_mod, and spec_mod classes prior to using the batch module.

cube_to_spectra(fname_list=None, base_dir=None, search_ext='bip', dir_level=0, base_dir_out=None, folder_name='cube_to_spec', name_append='cube-to-spec', write_geotiff=True, out_dtype=False, out_force=None, out_ext=False, out_interleave=False, out_byteorder=False)

Calculates the mean and standard deviation for each cube in fname_list and writes the result to a “.spec” file.

Parameters

• fname_list (list, optional) – list of filenames to process; if left to None, will look at base_dir, search_ext, and dir_level parameters for files to process (default: None).

• base_dir (str, optional) – directory path to search for files to spectrally clip; if fname_list is not None, base_dir will be ignored (default: None).

• search_ext (str) – file format/extension to search for in all directories and subdirectories to determine which files to process; if fname_list is not None, search_ext will be ignored (default: ‘bip’).

• dir_level (int) – The number of directory levels to search; if None, searches all directory levels (default: 0).

• base_dir_out (str) – directory path to save all processed spectra; if set to None, a folder named according to the folder_name parameter is added to base_dir

• folder_name (str) – folder to add to base_dir_out to save all the processed datacubes (default: ‘cube_to_spec’).

• name_append (str) – name to append to the filename (default: ‘cube-to-spec’).

• write_geotiff (bool) – whether to save the masked RGB image as a geotiff alongside the masked datacube.

• out_XXX – Settings for saving the output files can be adjusted here if desired. They are stored in batch.io.defaults, and are therefore accessible at a high level. See hsio.set_io_defaults() for more information on each of the settings.

Note

The following batch example builds on the API example results of the spatial_mod.crop_many_gdf function. Please complete the spatial_mod.crop_many_gdf example to be sure your directory (i.e., base_dir) is populated with multiple hyperspectral datacubes. The following example will be using datacubes located in the following directory: F:\nigo0024\Documents\hs_process_demo\spatial_mod\crop_many_gdf

Example

Load and initialize the batch module, checking to be sure the directory exists.

>>> import os
>>> from hs_process import batch
>>> data_dir = r'F:\nigo0024\Documents\hs_process_demo'
>>> base_dir = os.path.join(data_dir, 'spatial_mod', 'crop_many_gdf')
>>> print(os.path.isdir(base_dir))
>>> hsbatch = batch(base_dir, search_ext='.bip', progress_bar=True)  # searches for all files in ``base_dir`` with a ".bip" file extension
True

Use batch.cube_to_spectra to calculate the mean and standard deviation across all pixels for each of the datacubes in base_dir.

>>> hsbatch.cube_to_spectra(base_dir=base_dir, write_geotiff=False, out_force=True)
Processing file 39/40: 100%|██████████| 40/40 [00:03<00:00, 13.28it/s]------------------------------------------------| 0.0%

Use seaborn to visualize the spectra of plots 1011, 1012, and 1013. Notice how hsbatch.io.name_plot is utilized to retrieve the plot ID, and how the “history” tag is referenced from the metadata to determine the number of pixels whose reflectance was averaged to create the mean spectra. Also remember that pixels across the original input image likely represent a combination of soil, vegetation, and shadow.

>>> import seaborn as sns
>>> import re
>>> fname_list = [os.path.join(base_dir, 'cube_to_spec', 'Wells_rep2_20180628_16h56m_pika_gige_7_plot_1011-cube-to-spec-mean.spec'),
                  os.path.join(base_dir, 'cube_to_spec', 'Wells_rep2_20180628_16h56m_pika_gige_7_plot_1012-cube-to-spec-mean.spec'),
                  os.path.join(base_dir, 'cube_to_spec', 'Wells_rep2_20180628_16h56m_pika_gige_7_plot_1013-cube-to-spec-mean.spec')]
>>> ax = None
>>> for fname in fname_list:
>>>     hsbatch.io.read_spec(fname)
>>>     meta_bands = list(hsbatch.io.tools.meta_bands.values())
>>>     data = hsbatch.io.spyfile_spec.load().flatten() * 100
>>>     hist = hsbatch.io.spyfile_spec.metadata['history']
>>>     pix_n = re.search('<pixel number: (.*)>', hist).group(1)
>>>     if ax is None:
>>>         ax = sns.lineplot(x=meta_bands, y=data, label='Plot '+hsbatch.io.name_plot+' (n='+pix_n+')')
>>>     else:
>>>         ax = sns.lineplot(x=meta_bands, y=data, label='Plot '+hsbatch.io.name_plot+' (n='+pix_n+')', ax=ax)
>>> ax.set_xlabel('Wavelength (nm)', weight='bold')
>>> ax.set_ylabel('Reflectance (%)', weight='bold')
>>> ax.set_title(r'API Example: `batch.cube_to_spectra`', weight='bold')

segment_band_math(fname_list=None, base_dir=None, search_ext='bip', dir_level=0, base_dir_out=None, folder_name='band_math', name_append='band-math', write_geotiff=True, method='ndi', wl1=None, wl2=None, wl3=None, b1=None, b2=None, b3=None, list_range=True, plot_out=True, out_dtype=False, out_force=None, out_ext=False, out_interleave=False, out_byteorder=False)

Batch processing tool to perform band math on multiple datacubes in the same way. batch.segment_band_math is typically used prior to batch.segment_create_mask to generate the images/directory required for the masking process.

Parameters

• method (str) – Must be one of “ndi” (normalized difference index), “ratio” (simple ratio index), “derivative” (deriviative-type index), or “mcari2” (modified chlorophyll absorption index2). Indicates what kind of band math should be performed on the input datacube. The “ndi” method leverages segment.band_math_ndi(), the “ratio” method leverages segment.band_math_ratio(), and the “derivative” method leverages segment.band_math_derivative(). Please see the segment documentation for more information (default: “ndi”).

• wl1 (int, float, or list) – the wavelength (or set of wavelengths) to be used as the first parameter of the band math index; if list, then consolidates all bands between two wavelength values by calculating the mean pixel value across all bands in that range (default: None).

• wl2 (int, float, or list) – the wavelength (or set of wavelengths) to be used as the second parameter of the band math index; if list, then consolidates all bands between two wavelength values by calculating the mean pixel value across all bands in that range (default: None).

• b1 (int, float, or list) – the band (or set of bands) to be used as the first parameter of the band math index; if list, then consolidates all bands between two band values by calculating the mean pixel value across all bands in that range (default: None).

• b2 (int, float, or list) – the band (or set of bands) to be used as the second parameter of the band math index; if list, then consolidates all bands between two band values by calculating the mean pixel value across all bands in that range (default: None).

• list_range (bool) – Whether bands/wavelengths passed as a list is interpreted as a range of bands (True) or for each individual band in the list (False). If list_range is True, b1/wl1 and b2/wl2 should be lists with two items, and all bands/wavelegths between the two values will be used (default: True).

• plot_out (bool) – whether to save a histogram of the band math result (default: True).

• write_geotiff (bool) – whether to save the masked RGB image as a geotiff alongside the masked datacube.

Note

The following batch example builds on the API example results of the spatial_mod.crop_many_gdf function. Please complete the spatial_mod.crop_many_gdf example to be sure your directory (i.e., base_dir) is populated with multiple hyperspectral datacubes. The following example will be using datacubes located in the following directory: F:\nigo0024\Documents\hs_process_demo\spatial_mod\crop_many_gdf

Example

Load and initialize the batch module, checking to be sure the directory exists.

>>> import os
>>> from hs_process import batch
>>> base_dir = r'F:\nigo0024\Documents\hs_process_demo\spatial_mod\crop_many_gdf'
>>> print(os.path.isdir(base_dir))
True
>>> hsbatch = batch(base_dir, search_ext='.bip')  # searches for all files in ``base_dir`` with a ".bip" file extension

Use batch.segment_band_math to compute the MCARI2 (Modified Chlorophyll Absorption Ratio Index Improved; Haboudane et al., 2004) spectral index for each of the datacubes in base_dir. See Harris Geospatial for more information about the MCARI2 spectral index and references to other spectral indices.

>>> folder_name = 'band_math_mcari2-800-670-550'  # folder name can be modified to be more descriptive in what type of band math is being performed
>>> method = 'mcari2'  # must be one of "ndi", "ratio", "derivative", or "mcari2"
>>> wl1 = 800
>>> wl2 = 670
>>> wl3 = 550
>>> hsbatch.segment_band_math(base_dir=base_dir, folder_name=folder_name,
                              name_append='band-math', write_geotiff=True,
                              method=method, wl1=wl1, wl2=wl2, wl3=wl3,
                              plot_out=True, out_force=True)
Bands used (``b1``): [198]
Bands used (``b2``): [135]
Bands used (``b3``): [77]
Wavelengths used (``b1``): [799.0016]
Wavelengths used (``b2``): [669.6752]
Wavelengths used (``b3``): [550.6128]
Saving F:\nigo0024\Documents\hs_process_demo\spatial_mod\crop_many_gdfand_math_mcari2-800-670-550\Wells_rep2_20180628_16h56m_pika_gige_7_plot_1011-band-math-mcari2-800-670-550.bip
...

batch.segment_band_math creates a new folder in base_dir (in this case the new directory is F:\nigo0024\Documents\hs_process_demo\spatial_mod\crop_many_gdfand_math_mcari2-800-670-550) which contains several data products. The first is band-math-stats.csv: a spreadsheet containing summary statistics for each of the image cubes that were processed via batch.segment_band_math; stats include pixel count, mean, standard deviation, median, and percentiles across all image pixels.

Second is a geotiff file for each of the image cubes after the band math processing. This can be opened in QGIS to visualize in a spatial reference system, or can be opened using any software that supports floating point .tif files.

Third is the band math raster saved in the .hdr file format. Note that the data conained here should be the same as in the .tif file, so it’s a matter of preference as to what may be more useful. This single band .hdr can also be opend in QGIS.

Fourth is a histogram of the band math data contained in the image. The histogram illustrates the 90th percentile value, which may be useful in the segmentation step (e.g., see batch.segment_create_mask).

segment_composite_band(fname_list=None, base_dir=None, search_ext='bip', dir_level=0, base_dir_out=None, folder_name='composite_band', name_append='composite-band', write_geotiff=True, wl1=None, b1=None, list_range=True, plot_out=True, out_dtype=False, out_force=None, out_ext=False, out_interleave=False, out_byteorder=False)

Batch processing tool to create a composite band on multiple datacubes in the same way. batch.segment_composite_band is typically used prior to batch.segment_create_mask to generate the images/directory required for the masking process.

Parameters

• wl1 (int, float, or list) – the wavelength (or set of wavelengths) to be used as the first parameter of the band math index; if list, then consolidates all bands between two wavelength values by calculating the mean pixel value across all bands in that range (default: None).

• b1 (int, float, or list) – the band (or set of bands) to be used as the first parameter of the band math index; if list, then consolidates all bands between two band values by calculating the mean pixel value across all bands in that range (default: None).

• list_range (bool) – Whether bands/wavelengths passed as a list is interpreted as a range of bands (True) or for each individual band in the list (False). If list_range is True, b1/wl1 and b2/wl2 should be lists with two items, and all bands/wavelegths between the two values will be used (default: True).

• plot_out (bool) – whether to save a histogram of the band math result (default: True).

• write_geotiff (bool) – whether to save the masked RGB image as a geotiff alongside the masked datacube.

segment_create_mask(fname_list=None, base_dir=None, search_ext='bip', dir_level=0, mask_dir=None, base_dir_out=None, folder_name='mask', name_append='mask', write_datacube=True, write_spec=True, write_geotiff=True, mask_thresh=None, mask_percentile=None, mask_side='lower', out_dtype=False, out_force=None, out_ext=False, out_interleave=False, out_byteorder=False)

Batch processing tool to create a masked array on many datacubes. batch.segment_create_mask is typically used after batch.segment_band_math to mask all the datacubes in a directory based on the result of the band math process.

Parameters

• mask_thresh (float or int) – The value for which to mask the array; should be used with side parameter (default: None).

• mask_percentile (float or int) – The percentile of pixels to mask; if percentile``=95 and ``side``='lower', the lowest 95% of pixels will be masked following the band math operation (default: ``None; range: 0-100).

• mask_side (str) – The side of the threshold for which to apply the mask. Must be either ‘lower’, ‘upper’, ‘outside’, or None; if ‘lower’, everything below the threshold will be masked; if ‘outside’, the thresh / percentile parameter must be list-like with two values indicating the lower and upper bounds - anything outside of these values will be masked out; if None, only the values that exactly match the threshold will be masked (default: ‘lower’).

• geotiff (bool) – whether to save the masked RGB image as a geotiff alongside the masked datacube.

Note

The following batch example builds on the API example results of spatial_mod.crop_many_gdf and batch.segment_band_math. Please complete each of those API examples to be sure your directories (i.e., base_dir, and mask_dir) are populated with image files. The following example will be masking datacubes located in: F:\nigo0024\Documents\hs_process_demo\spatial_mod\crop_many_gdf based on MCARI2 images located in: F:\nigo0024\Documents\hs_process_demo\spatial_mod\crop_many_gdf\band_math_mcari2-800-670-550

Example

Load and initialize the batch module, ensuring base_dir is a valid directory

>>> import os
>>> from hs_process import batch
>>> base_dir = r'F:\nigo0024\Documents\hs_process_demo\spatial_mod\crop_many_gdf'
>>> print(os.path.isdir(base_dir))
True
>>> hsbatch = batch(base_dir, search_ext='.bip')  # searches for all files in ``base_dir`` with a ".bip" file extension

There must be a single-band image that will be used to determine which datacube pixels are to be masked (determined via the mask_dir parameter). Point to the directory that contains the MCARI2 images.

>>> mask_dir = os.path.join(base_dir, 'band_math_mcari2-800-670-550')
>>> print(os.path.isdir(mask_dir))
True

Indicate how the MCARI2 images should be used to determine which hyperspectal pixels are to be masked. The available parameters for controlling this are mask_thresh, mask_percentile, and mask_side. We will mask out all pixels that fall below the MCARI2 90th percentile.

>>> mask_percentile = 90
>>> mask_side = 'lower'

Finally, indicate the folder to save the masked datacubes and perform the batch masking via batch.segment_create_mask

>>> folder_name = 'mask_mcari2_90th'
>>> hsbatch.segment_create_mask(base_dir=base_dir, mask_dir=mask_dir,
                                folder_name=folder_name,
                                name_append='mask-mcari2-90th', write_geotiff=True,
                                mask_percentile=mask_percentile,
                                mask_side=mask_side)
Saving F:\nigo0024\Documents\hs_process_demo\spatial_mod\crop_many_gdf\mask_mcari2_90th\Wells_rep2_20180628_16h56m_pika_gige_7_plot_1011-mask-mcari2-90th.bip
Saving F:\nigo0024\Documents\hs_process_demo\spatial_mod\crop_many_gdf\mask_mcari2_90th\Wells_rep2_20180628_16h56m_pika_gige_7_plot_1011-mask-mcari2-90th-spec-mean.spec
...

batch.segment_create_mask creates a new folder in base_dir named according to the folder_name parameter (in this case the new directory is F:\nigo0024\Documents\hs_process_demo\spatial_mod\crop_many_gdf\mask_mcari2_90th) which contains several data products. The first is mask-stats.csv: a spreadsheet containing the band math threshold value for each image file. In this example, the MCARI2 value corresponding to the 90th percentile is listed.

fname	plot_id	lower-pctl-90
…	1011	0.83222
…	1012	0.81112
…	1013	0.74394

…etc.

Second is a geotiff file for each of the image cubes after the masking procedure. This can be opened in QGIS to visualize in a spatial reference system, or can be opened using any software that supports floating point .tif files. The masked pixels are saved as null values and should render transparently.

Third is the full hyperspectral datacube, also with the masked pixels saved as null values. Note that the only pixels remaining are the 10% with the highest MCARI2 values.

Fourth is the mean spectra across the unmasked datacube pixels. This is illustrated above by the green line plot (the light green shadow represents the standard deviation for each band).

spatial_crop(fname_sheet=None, base_dir=None, search_ext='bip', dir_level=0, base_dir_out=None, folder_name='spatial_crop', name_append='spatial-crop', write_geotiff=True, method='single', gdf=None, out_dtype=False, out_force=None, out_ext=False, out_interleave=False, out_byteorder=False)

Iterates through a spreadsheet that provides necessary information about how each image should be cropped and how it should be saved.

If gdf is passed (a geopandas.GoeDataFrame polygon file), the cropped images will be shifted to the center of appropriate ‘plot_id’ polygon.

Parameters

• fname_sheet (fname, pandas.DataFrame, or None, optional) – The filename of the spreadsheed that provides the necessary information for fine-tuning the batch process cropping. See below for more information about the required and optional contents of fname_sheet and how to properly format it. Optionally, fname_sheet can be a Pandas.DataFrame. If left to None, base_dir and gdf must be passed.

• base_dir (str, optional) – directory path to search for files to spatially crop; if fname_sheet is not None, base_dir will be ignored (default: None).

• base_dir_out (str, optional) – output directory of the cropped image (default: None).

• folder_name (str, optional) – folder to add to base_dir_out to save all the processed datacubes (default: ‘spatial_crop’).

• name_append (str, optional) – name to append to the filename (default: ‘spatial-crop’).

• write_geotiff (bool, optional) – whether to save an RGB image as a geotiff alongside the cropped datacube.

• method (str, optional) – Must be one of “single” or “many_gdf”. Indicates whether a single plot should be cropped from the input datacube or if many/multiple plots should be cropped from the input datacube. The “single” method leverages spatial_mod.crop_single() and the “many_gdf” method leverages spatial_mod.crop_many_gdf(). Please see the spatial_mod documentation for more information (default: “single”).

• gdf (geopandas.GeoDataFrame, optional) – the plot names and polygon geometery of each of the plots; ‘plot_id’ must be used as a column name to identify each of the plots, and should be an integer.

• out_XXX – Settings for saving the output files can be adjusted here if desired. They are stored in batch.io.defaults, and are therefore accessible at a high level. See hsio.set_io_defaults() for more information on each of the settings.

Tips and Tricks for fname_sheet when gdf is not passed

If gdf is not passed, fname_sheet may have the following required column headings that correspond to the relevant parameters in spatial_mod.crop_single() and spatial_mod.crop_many_gdf():

1. “directory”

2. “name_short”

3. “name_long”

4. “ext”

5. “pix_e_ul”

6. “pix_n_ul”.

With this minimum input, batch.spatial_crop will read in each image, crop from the upper left pixel (determined as pix_e_ul/pix_n_ul) to the lower right pixel calculated based on crop_e_pix/crop_n_pix (which is the width of the cropped area in units of pixels).

Note

crop_e_pix and crop_n_pix have default values (see defaults.crop_defaults()), but they can also be passed specifically for each datacube by including appropriate columns in fname_sheet (which takes precedence over defaults.crop_defaults).

fname_sheet may also have the following optional column headings:

1. “crop_e_pix”

2. “crop_n_pix”

3. “crop_e_m”

4. “crop_n_m”

5. “buf_e_pix”

6. “buf_n_pix”

7. “buf_e_m”

8. “buf_n_m”

9. “gdf_shft_e_m”

10. “gdf_shft_n_m”

11. “plot_id_ref”

12. “study”

13. “date”

More fname_sheet Tips and Tricks

1. These optional inputs passed via fname_sheet allow more control over exactly how the images are to be cropped. For a more detailed explanation of the information that many of these columns are intended to contain, see the documentation for spatial_mod.crop_single() and spatial_mod.crop_many_gdf(). Those parameters not referenced should be apparent in the API examples and tutorials.

2. If the column names are different in fname_sheet than described here, defaults.spat_crop_cols() can be modified to indicate which columns correspond to the relevant information.

3. The date and study columns do not impact how the datacubes are to be cropped, but if this information exists, batch.spatial_crop adds it to the filename of the cropped datacube. This can be used to avoid overwriting datacubes with similar names, and is especially useful when processing imagery from many dates and/or studies/locations and saving them in the same directory. If “study”, “date”, and “plot_id” are all passed, this information is used to formulate the output file name; e.g., study_wells_date_20180628_plot_527-spatial-crop.bip. If either “study” or “date” is missing, the populated variables wil be appended to the end of the hsio.name_short string; e.g., plot_9_3_pika_gige_1_plot_527-spatial-crop.bip.

4. Any other columns can be added to fname_sheet, but batch.spatial_crop() does not use them in any way.

Note

The following batch example only actually processes a single hyperspectral image. If more datacubes were present in base_dir, however, batch.spatial_crop would process all datacubes that were available.

Note

This example uses spatial_mod.crop_many_gdf to crop many plots from a datacube using a polygon geometry file describing the spatial extent of each plot.

Example

Load and initialize the batch module, checking to be sure the directory exists.

>>> import os
>>> import geopandas as gpd
>>> import pandas as pd
>>> from hs_process import batch
>>> base_dir = r'F:\nigo0024\Documents\hs_process_demo'
>>> print(os.path.isdir(base_dir))
>>> hsbatch = batch(base_dir, search_ext='.bip', dir_level=0,
                    progress_bar=True)  # searches for all files in ``base_dir`` with a ".bip" file extension
True

Load the plot geometry as a geopandas.GeoDataFrame

>>> fname_gdf = r'F:\nigo0024\Documents\hs_process_demo\plot_bounds.geojson'
>>> gdf = gpd.read_file(fname_gdf)

Perform the spatial cropping using the “many_gdf” method. Note that nothing is being passed to fname_sheet here, so batch.spatial_crop is simply going to attempt to crop all plots contained within gdf that overlap with any datacubes in base_dir.

Passing fname_sheet directly is definitely more flexible for customization. However, some customization is possible while not passing fname_sheet. In the example below, we set an easting and northing buffer, as well as limit the number of plots to crop to 40. These defaults trickle through to spatial_mod.crop_many_gdf(), so by setting them on the batch object, they will be recognized when calculating crop boundaries from gdf.

>>> hsbatch.io.defaults.crop_defaults.buf_e_m = 2  # Sets buffer in the easting direction (units of meters)
>>> hsbatch.io.defaults.crop_defaults.buf_n_m = 0.5
>>> hsbatch.io.defaults.crop_defaults.n_plots = 40  # We can limit the number of plots to process from gdf
>>> hsbatch.spatial_crop(base_dir=base_dir, method='many_gdf',
                         gdf=gdf, out_force=True)

Because fname_list was passed instead of fname_sheet, there is not a way to infer the study name and date. Therefore, “study” and “date” will be omitted from the output file name. If you would like output file names to include “study” and “date”, please pass fname_sheet with “study” and “date” columns.

Processing file 39/40: 100%|██████████| 40/40 [00:02<00:00, 17.47it/s]

A new folder was created in base_dir - F:\nigo0024\Documents\hs_process_demo\spatial_crop - that contains the cropped datacubes and the cropped geotiff images. The Plot ID from the gdf is used to name each datacube according to its plot ID. The geotiff images can be opened in QGIS to visualize the images after cropping them.

The cropped images were brightened in QGIS to emphasize the cropped boundaries. The plot boundaries are overlaid for reference (notice the 2.0 m buffer on the East/West ends and the 0.5 m buffer on the North/South sides).

spectra_combine(fname_list=None, base_dir=None, search_ext='bip', dir_level=0, base_dir_out=None, out_dtype=False, out_force=None, out_ext=False, out_interleave=False, out_byteorder=False)

Batch processing tool to gather all pixels from every image in a directory, compute the mean and standard deviation, and save as a single spectra (i.e., a spectra file is equivalent to a single spectral pixel with no spatial information).

Parameters

• fname_list (list, optional) – list of filenames to process; if left to None, will look at base_dir, search_ext, and dir_level parameters for files to process (default: None).

• base_dir (str, optional) – directory path to search for files to spectrally clip; if fname_list is not None, base_dir will be ignored (default: None).

• search_ext (str) – file format/extension to search for in all directories and subdirectories to determine which files to process; if fname_list is not None, search_ext will be ignored (default: ‘bip’).

• dir_level (int) – The number of directory levels to search; if None, searches all directory levels (default: 0).

• base_dir_out (str) – directory path to save all processed datacubes; if set to None, a folder named according to the folder_name parameter is added to base_dir (default: None).

• out_XXX – Settings for saving the output files can be adjusted here if desired. They are stored in batch.io.defaults, and are therefore accessible at a high level. See ``hsio.set_io_defaults() for more information on each of the settings.

Note

The following example will load in several small hyperspectral radiance datacubes (not reflectance) that were previously cropped manually (via Spectronon software). These datacubes represent the radiance values of grey reference panels that were placed in the field to provide data necessary for converting radiance imagery to reflectance. These particular datacubes were extracted from several different images captured within ~10 minutes of each other.

Example

Load and initialize the batch module, checking to be sure the directory exists.

>>> import os
>>> from hs_process import batch
>>> base_dir = r'F:\nigo0024\Documents\hs_process_demo\cube_ref_panels'
>>> print(os.path.isdir(base_dir))
True
>>> hsbatch = batch(base_dir)

Combine all the radiance datacubes in the directory via batch.spectra_combine.

>>> hsbatch.spectra_combine(base_dir=base_dir, search_ext='bip',
                            dir_level=0)
Combining datacubes/spectra into a single mean spectra.
Number of input datacubes/spectra: 7
Total number of pixels: 1516
Saving F:\nigo0024\Documents\hs_process_demo\cube_ref_panels\spec_mean_spy.spec

Visualize the combined spectra by opening in Spectronon. The solid line represents the mean radiance spectra across all pixels and images in base_dir, and the lighter, slightly transparent line represents the standard deviation of the radiance across all pixels and images in base_dir.

Notice the lower signal at the oxygen absorption region (near 770 nm). After converting datacubes to reflectance, it may be desireable to spectrally clip this region (see spec_mod.spectral_clip())

spectra_derivative(fname_list=None, base_dir=None, search_ext='spec', dir_level=0, base_dir_out=None, folder_name='spec_derivative', name_append='spec-derivative', order=1, out_dtype=False, out_force=None, out_ext=False, out_interleave=False, out_byteorder=False)

Batch processing tool to calculate the numeric spectral derivative for multiple spectra.

Parameters

• fname_list (list, optional) – list of filenames to process; if left to None, will look at base_dir, search_ext, and dir_level parameters for files to process (default: None).

• base_dir (str, optional) – directory path to search for files to process; if fname_list is not None, base_dir will be ignored (default: None).

• search_ext (str) – file format/extension to search for in all directories and subdirectories to determine which files to process; if fname_list is not None, search_ext will be ignored (default: ‘bip’).

• dir_level (int) – The number of directory levels to search; if None, searches all directory levels (default: 0).

• base_dir_out (str) – directory path to save all processed spectra; if set to None, a folder named according to the folder_name parameter is added to base_dir

• folder_name (str) – folder to add to base_dir_out to save all the processed datacubes (default: ‘spec_derivative’).

• name_append (str) – name to append to the filename (default: ‘spec-derivative’).

• order (int) – The order of the derivative (default: 1).

• out_XXX – Settings for saving the output files can be adjusted here if desired. They are stored in batch.io.defaults, and are therefore accessible at a high level. See ``hsio.set_io_defaults() for more information on each of the settings.

Note

The following batch example builds on the API example results of the batch.cube_to_spectra function. Please complete both the spatial_mod.crop_many_gdf and batch.cube_to_spectra examples to be sure your directory (i.e., base_dir) is populated with multiple hyperspectral spectra. The following example will be using spectra located in the following directory: F:\nigo0024\Documents\hs_process_demo\spatial_mod\crop_many_gdf\cube_to_spec

Example

Load and initialize the batch module, checking to be sure the directory exists.

>>> import os
>>> from hs_process import batch
>>> data_dir = r'F:\nigo0024\Documents\hs_process_demo'
>>> base_dir = os.path.join(data_dir, 'spatial_mod', 'crop_many_gdf', 'cube_to_spec')
>>> print(os.path.isdir(base_dir))
>>> hsbatch = batch(base_dir, search_ext='.spec', progress_bar=True)

Use batch.spectra_derivative to calculate the central finite difference (i.e., the numeric spectral derivative) for each of the .spec files in base_dir.

>>> order = 1
>>> hsbatch.spectra_derivative(base_dir=base_dir, order=order, out_force=True)

Use seaborn to visualize the derivative spectra of plots 1011, 1012, and 1013.

>>> import seaborn as sns
>>> import re
>>> fname_list = [os.path.join(base_dir, 'spec_derivative', 'Wells_rep2_20180628_16h56m_pika_gige_7_plot_1011-spec-derivative-order-{0}.spec'.format(order)),
                  os.path.join(base_dir, 'spec_derivative', 'Wells_rep2_20180628_16h56m_pika_gige_7_plot_1012-spec-derivative-order-{0}.spec'.format(order)),
                  os.path.join(base_dir, 'spec_derivative', 'Wells_rep2_20180628_16h56m_pika_gige_7_plot_1013-spec-derivative-order-{0}.spec'.format(order))]
>>> ax = None
>>> for fname in fname_list:
>>>     hsbatch.io.read_spec(fname)
>>>     meta_bands = list(hsbatch.io.tools.meta_bands.values())
>>>     data = hsbatch.io.spyfile_spec.open_memmap().flatten() * 100
>>>     hist = hsbatch.io.spyfile_spec.metadata['history']
>>>     pix_n = re.search('<pixel number: (?s)(.*)>] ->', hist).group(1)
>>>     if ax is None:
>>>         ax = sns.lineplot(meta_bands, 0, color='gray')
>>>         ax = sns.lineplot(x=meta_bands, y=data, label='Plot '+hsbatch.io.name_plot+' (n='+pix_n+')')
>>>     else:
>>>         ax = sns.lineplot(x=meta_bands, y=data, label='Plot '+hsbatch.io.name_plot+' (n='+pix_n+')', ax=ax)
>>> ax.set(ylim=(-1, 1))
>>> ax.set_xlabel('Wavelength (nm)', weight='bold')
>>> ax.set_ylabel('Derivative reflectance (%)', weight='bold')
>>> ax.set_title(r'API Example: `batch.spectra_derivative`', weight='bold')

spectra_to_csv(fname_list=None, base_dir=None, search_ext='spec', dir_level=0, base_dir_out=None, name='stats-spectra', multithread=False)

Reads all the .spec files in a direcory and saves their reflectance information to a .csv. batch.spectra_to_csv is identical to batch.spectra_to_df except a .csv file is saved rather than returning a pandas.DataFrame.

Parameters

• fname_list (list, optional) – list of filenames to process; if left to None, will look at base_dir, search_ext, and dir_level parameters for files to process (default: None).

• base_dir (str, optional) – directory path to search for files to spectrally clip; if fname_list is not None, base_dir will be ignored (default: None).

• search_ext (str) – file format/extension to search for in all directories and subdirectories to determine which files to process; if fname_list is not None, search_ext will be ignored (default: ‘bip’).

• dir_level (int) – The number of directory levels to search; if None, searches all directory levels (default: 0).

• base_dir_out (str) – directory path to save all processed datacubes; if set to None, file is saved to base_dir

• name (str) – The output filename (default: “stats-spectra”).

• multithread (bool) – Whether to leverage multi-thread processing when reading the .spec files. Setting to True should speed up the time it takes to read all .spec files.

Note

The following example builds on the API example results of batch.segment_band_math() and batch.segment_create_mask()_. Please complete each of those API examples to be sure your directory (i.e., ``F:nigo0024Documentshs_process_demospatial_modcrop_many_gdfmask_mcari2_90th`) is populated with image files.

Example

Load and initialize the batch module, checking to be sure the directory exists.

>>> import os
>>> from hs_process import batch
>>> base_dir = r'F:\nigo0024\Documents\hs_process_demo\spatial_mod\crop_many_gdf\mask_mcari2_90th'
>>> print(os.path.isdir(base_dir))
True
>>> hsbatch = batch(base_dir)

Read all the .spec files in base_dir and save them to a .csv file.

>>> hsbatch.spectra_to_csv(base_dir=base_dir, search_ext='spec',
                           dir_level=0)
Writing mean spectra to a .csv file.
Number of input datacubes/spectra: 40
Output file location: F:\nigo0024\Documents\hs_process_demo\spatial_mod\crop_many_gdf\mask_mcari2_90th\stats-spectra.csv

When stats-spectra.csv is opened in Microsoft Excel, we can see that each row is a .spec file from a different plot, and each column is a particular spectral band/wavelength.

spectra_to_df(fname_list=None, base_dir=None, search_ext='spec', dir_level=0, multithread=False)

Reads all the .spec files in a direcory and returns their data as a pandas.DataFrame object. batch.spectra_to_df is identical to batch.spectra_to_csv except a pandas.DataFrame is returned rather than saving a .csv file.

Parameters

• fname_list (list, optional) – list of filenames to process; if left to None, will look at base_dir, search_ext, and dir_level parameters for files to process (default: None).

• base_dir (str, optional) – directory path to search for files to spectrally clip; if fname_list is not None, base_dir will be ignored (default: None).

• search_ext (str) – file format/extension to search for in all directories and subdirectories to determine which files to process; if fname_list is not None, search_ext will be ignored (default: ‘bip’).

• dir_level (int) – The number of directory levels to search; if None, searches all directory levels (default: 0).

• multithread (bool) – Whether to leverage multi-thread processing when reading the .spec files. Setting to True should speed up the time it takes to read all .spec files.

Note

The following example builds on the API example results of batch.segment_band_math() and batch.segment_create_mask()_. Please complete each of those API examples to be sure your directory (i.e., ``F:nigo0024Documentshs_process_demospatial_modcrop_many_gdfmask_mcari2_90th`) is populated with image files.

Example

Load and initialize the batch module, checking to be sure the directory exists.

>>> import os
>>> from hs_process import batch
>>> base_dir = r'F:\nigo0024\Documents\hs_process_demo\spatial_mod\crop_many_gdf\mask_mcari2_90th'
>>> print(os.path.isdir(base_dir))
True
>>> hsbatch = batch(base_dir)

Read all the .spec files in base_dir and load them to df_spec, a pandas.DataFrame.

>>> df_spec = hsbatch.spectra_to_df(base_dir=base_dir, search_ext='spec',
                                    dir_level=0)
Writing mean spectra to a ``pandas.DataFrame``.
Number of input datacubes/spectra: 40

When visualizing df_spe in Spyder, we can see that each row is a .spec file from a different plot, and each column is a particular spectral band.

It is somewhat confusing to conceptualize spectral data by band number (as opposed to the wavelenth it represents). hs_process.hs_tools.get_band can be used to retrieve spectral data for all plots via indexing by wavelength. Say we need to access reflectance at 710 nm for each plot.

>>> df_710nm = df_spec[['fname', 'plot_id', hsbatch.io.tools.get_band(710)]]

spectral_clip(fname_list=None, base_dir=None, search_ext='bip', dir_level=0, base_dir_out=None, folder_name='spec_clip', name_append='spec-clip', wl_bands=[[0, 420], [760, 776], [813, 827], [880, 1000]], out_dtype=False, out_force=None, out_ext=False, out_interleave=False, out_byteorder=False)

Batch processing tool to spectrally clip multiple datacubes in the same way.

Parameters

• fname_list (list, optional) – list of filenames to process; if left to None, will look at base_dir, search_ext, and dir_level parameters for files to process (default: None).

• base_dir (str, optional) – directory path to search for files to spectrally clip; if fname_list is not None, base_dir will be ignored (default: None).

• search_ext (str) – file format/extension to search for in all directories and subdirectories to determine which files to process; if fname_list is not None, search_ext will be ignored (default: ‘bip’).

• dir_level (int) – The number of directory levels to search; if None, searches all directory levels (default: 0).

• base_dir_out (str) – directory path to save all processed datacubes; if set to None, a folder named according to the folder_name parameter is added to base_dir

• folder_name (str) – folder to add to base_dir_out to save all the processed datacubes (default: ‘spec-clip’).

• name_append (str) – name to append to the filename (default: ‘spec-clip’).

• wl_bands (list or list of lists) – minimum and maximum wavelenths to clip from image; if multiple groups of wavelengths should be cut, this should be a list of lists. For example, wl_bands=[760, 776] will clip all bands greater than 760.0 nm and less than 776.0 nm; wl_bands = [[0, 420], [760, 776], [813, 827], [880, 1000]] will clip all band less than 420.0 nm, bands greater than 760.0 nm and less than 776.0 nm, bands greater than 813.0 nm and less than 827.0 nm, and bands greater than 880 nm (default).

• out_XXX – Settings for saving the output files can be adjusted here if desired. They are stored in batch.io.defaults, and are therefore accessible at a high level. See ``hsio.set_io_defaults() for more information on each of the settings.

Note

The following batch example builds on the API example results of the batch.spatial_crop function. Please complete the batch.spatial_crop example to be sure your directory (i.e., base_dir) is populated with multiple hyperspectral datacubes. The following example will be using datacubes located in the following directory: F:\nigo0024\Documents\hs_process_demo\spatial_crop

Example

Load and initialize the batch module, checking to be sure the directory exists.

>>> import os
>>> from hs_process import batch
>>> base_dir = r'F:\nigo0024\Documents\hs_process_demo\spatial_crop'
>>> print(os.path.isdir(base_dir))
True
>>> hsbatch = batch(base_dir, search_ext='.bip', progress_bar=True)  # searches for all files in ``base_dir`` with a ".bip" file extension

Use batch.spectral_clip to clip all spectral bands below 420 nm and above 880 nm, as well as the bands near the oxygen absorption (i.e., 760-776 nm) and water absorption (i.e., 813-827 nm) regions.

>>> hsbatch.spectral_clip(base_dir=base_dir, folder_name='spec_clip',
                          wl_bands=[[0, 420], [760, 776], [813, 827], [880, 1000]],
                          out_force=True)
Processing 40 files. If this is not what is expected, please check if files have already undergone processing. If existing files should be overwritten, be sure to set the ``out_force`` parameter.
Processing file 39/40: 100%|██████████| 40/40 [00:01<00:00, 26.68it/s]

Use seaborn to visualize the spectra of a single pixel in one of the processed images.

>>> import seaborn as sns
>>> fname = os.path.join(base_dir, 'Wells_rep2_20180628_16h56m_pika_gige_7_1011-spatial-crop.bip')
>>> hsbatch.io.read_cube(fname)
>>> spy_mem = hsbatch.io.spyfile.open_memmap()  # datacube before clipping
>>> meta_bands = list(hsbatch.io.tools.meta_bands.values())
>>> fname = os.path.join(base_dir, 'spec_clip', 'Wells_rep2_20180628_16h56m_pika_gige_7_1011-spec-clip.bip')
>>> hsbatch.io.read_cube(fname)
>>> spy_mem_clip = hsbatch.io.spyfile.open_memmap()  # datacube after clipping
>>> meta_bands_clip = list(hsbatch.io.tools.meta_bands.values())
>>> ax = sns.lineplot(x=meta_bands, y=spy_mem[26][29], label='Before spectral clipping', linewidth=3)
>>> ax = sns.lineplot(x=meta_bands_clip, y=spy_mem_clip[26][29], label='After spectral clipping', ax=ax)
>>> ax.set_xlabel('Wavelength (nm)', weight='bold')
>>> ax.set_ylabel('Reflectance (%)', weight='bold')
>>> ax.set_title(r'API Example: `batch.spectral_clip`', weight='bold')

Notice the spectral areas that were clipped, namely the oxygen and water absorption regions (~770 and ~820 nm, respectively). There is perhaps a lower signal:noise ratio in these regions, which was the merit for clipping those bands out.

spectral_mimic(fname_list=None, base_dir=None, search_ext='bip', dir_level=0, base_dir_out=None, folder_name='spec_mimic', name_append='spec-mimic', sensor='sentinel-2a', df_band_response=None, col_wl='wl_nm', center_wl='peak', out_dtype=False, out_force=None, out_ext=False, out_interleave=False, out_byteorder=False)

Batch processing tool to spectrally mimic a multispectral sensor for multiple datacubes in the same way.

Parameters

• fname_list (list, optional) – list of filenames to process; if left to None, will look at base_dir, search_ext, and dir_level parameters for files to process (default: None).

• base_dir (str, optional) – directory path to search for files to spectrally resample; if fname_list is not None, base_dir will be ignored (default: None).

• search_ext (str) – file format/extension to search for in all directories and subdirectories to determine which files to process; if fname_list is not None, search_ext will be ignored (default: ‘bip’).

• dir_level (int) – The number of directory levels to search; if None, searches all directory levels (default: 0).

• base_dir_out (str) – directory path to save all processed datacubes; if set to None, a folder named according to the folder_name parameter is added to base_dir

• folder_name (str) – folder to add to base_dir_out to save all the processed datacubes (default: ‘spec_bin’).

• name_append (str) – name to append to the filename (default: ‘spec-bin’).

• sensor (str) – Should be one of [“sentera_6x”, “micasense_rededge_3”, “sentinel-2a”, “sentinel-2b”, “custom”]; if “custom”, df_band_response and col_wl must be passed.

• df_band_response (pd.DataFrame) – A DataFrame that contains the transmissivity (%) for each sensor band (as columns) mapped to the continuous wavelength values (as rows). Required if sensor is “custom”, ignored otherwise.

• col_wl (str) – The column of df_band_response denoting the wavlengths (default: ‘wl_nm’).

• center_wl (str) – Indicates how the center wavelength of each band is determined. If center_wl is “peak”, the point at which transmissivity is at its maximum is used as the center wavelength. If center_wl is “weighted”, the weighted average is used to compute the center wavelength. Must be one of [“peak”, “weighted”] (default: "peak").

• out_XXX – Settings for saving the output files can be adjusted here if desired. They are stored in batch.io.defaults, and are therefore accessible at a high level. See ``hsio.set_io_defaults() for more information on each of the settings.

Note

The following batch example builds on the API example results of the batch.spatial_crop function. Please complete the batch.spatial_crop example to be sure your directory (i.e., base_dir) is populated with multiple hyperspectral datacubes. The following example will be using datacubes located in the following directory: F:\nigo0024\Documents\hs_process_demo\spatial_crop

Example

Load and initialize the batch module, checking to be sure the directory exists.

>>> import os
>>> from hs_process import batch
>>> base_dir = r'F:\nigo0024\Documents\hs_process_demo\spatial_crop'
>>> print(os.path.isdir(base_dir))
True
>>> hsbatch = batch(base_dir, search_ext='.bip', progress_bar=True)  # searches for all files in ``base_dir`` with a ".bip" file extension

Use batch.spectral_mimic to spectrally mimic the Sentinel-2A multispectral satellite sensor.

>>> hsbatch.spectral_mimic(
    base_dir=base_dir, folder_name='spec_mimic',
    name_append='sentinel-2a',
    sensor='sentinel-2a', center_wl='weighted')
Processing 40 files. If existing files should be overwritten, be sure to set the ``out_force`` parameter.
Processing file 39/40: 100%|██████████| 40/40 [00:04<00:00,  8.85it/s]

Use seaborn to visualize the spectra of a single pixel in one of the processed images.

>>> import seaborn as sns
>>> fname = os.path.join(base_dir, 'Wells_rep2_20180628_16h56m_pika_gige_7_1011-spatial-crop.bip')
>>> hsbatch.io.read_cube(fname)
>>> spy_mem = hsbatch.io.spyfile.open_memmap()  # datacube before mimicking
>>> meta_bands = list(hsbatch.io.tools.meta_bands.values())
>>> fname = os.path.join(base_dir, 'spec_mimic', 'Wells_rep2_20180628_16h56m_pika_gige_7_1011-sentinel-2a.bip')
>>> hsbatch.io.read_cube(fname)
>>> spy_mem_sen2a = hsbatch.io.spyfile.open_memmap()  # datacube after mimicking
>>> meta_bands_sen2a = list(hsbatch.io.tools.meta_bands.values())
>>> ax = sns.lineplot(x=meta_bands, y=spy_mem[26][29], label='Hyperspectral (Pika II)', linewidth=3)
>>> ax = sns.lineplot(x=meta_bands_sen2a, y=spy_mem_sen2a[26][29], label='Sentinel-2A "mimic"', marker='o', ms=6, ax=ax)
>>> ax.set_xlabel('Wavelength (nm)', weight='bold')
>>> ax.set_ylabel('Reflectance (%)', weight='bold')
>>> ax.set_title(r'API Example: `batch.spectral_mimic`', weight='bold')

spectral_resample(fname_list=None, base_dir=None, search_ext='bip', dir_level=0, base_dir_out=None, folder_name='spec_bin', name_append='spec-bin', bandwidth=None, bins_n=None, out_dtype=False, out_force=None, out_ext=False, out_interleave=False, out_byteorder=False)

Batch processing tool to spectrally resample (a.k.a. “bin”) multiple datacubes in the same way.

Parameters

• fname_list (list, optional) – list of filenames to process; if left to None, will look at base_dir, search_ext, and dir_level parameters for files to process (default: None).

• base_dir (str, optional) – directory path to search for files to spectrally resample; if fname_list is not None, base_dir will be ignored (default: None).

• search_ext (str) – file format/extension to search for in all directories and subdirectories to determine which files to process; if fname_list is not None, search_ext will be ignored (default: ‘bip’).

• dir_level (int) – The number of directory levels to search; if None, searches all directory levels (default: 0).

• base_dir_out (str) – directory path to save all processed datacubes; if set to None, a folder named according to the folder_name parameter is added to base_dir

• folder_name (str) – folder to add to base_dir_out to save all the processed datacubes (default: ‘spec_bin’).

• name_append (str) – name to append to the filename (default: ‘spec-bin’).

• bandwidth (float or int) – The bandwidth of the bands after spectral resampling is complete (units should be consistent with that of the .hdr file). Setting bandwidth to 10 will consolidate bands that fall within every 10 nm interval.

• bins_n (int) – The number of bins (i.e., “bands”) to achieve after spectral resampling is complete. Ignored if bandwidth is not None.

• out_XXX – Settings for saving the output files can be adjusted here if desired. They are stored in batch.io.defaults, and are therefore accessible at a high level. See ``hsio.set_io_defaults() for more information on each of the settings.

Note

The following batch example builds on the API example results of the batch.spatial_crop function. Please complete the batch.spatial_crop example to be sure your directory (i.e., base_dir) is populated with multiple hyperspectral datacubes. The following example will be using datacubes located in the following directory: F:\nigo0024\Documents\hs_process_demo\spatial_crop

Example

Load and initialize the batch module, checking to be sure the directory exists.

>>> import os
>>> from hs_process import batch
>>> base_dir = r'F:\nigo0024\Documents\hs_process_demo\spatial_crop'
>>> print(os.path.isdir(base_dir))
True
>>> hsbatch = batch(base_dir, search_ext='.bip', progress_bar=True)  # searches for all files in ``base_dir`` with a ".bip" file extension

Use batch.spectral_resample to bin (“group”) all spectral bands into 20 nm bandwidth bands (from ~2.3 nm bandwidth originally) on a per-pixel basis.

>>> hsbatch.spectral_resample(
    base_dir=base_dir, folder_name='spec_bin',
    name_append='spec-bin-20', bandwidth=20)
Processing 40 files. If existing files should be overwritten, be sure to set the ``out_force`` parameter.
Processing file 39/40: 100%|██████████| 40/40 [00:00<00:00, 48.31it/s]
...

Use seaborn to visualize the spectra of a single pixel in one of the processed images.

>>> import seaborn as sns
>>> fname = os.path.join(base_dir, 'Wells_rep2_20180628_16h56m_pika_gige_7_1011-spatial-crop.bip')
>>> hsbatch.io.read_cube(fname)
>>> spy_mem = hsbatch.io.spyfile.open_memmap()  # datacube before resampling
>>> meta_bands = list(hsbatch.io.tools.meta_bands.values())
>>> fname = os.path.join(base_dir, 'spec_bin', 'Wells_rep2_20180628_16h56m_pika_gige_7_1011-spec-bin-20.bip')
>>> hsbatch.io.read_cube(fname)
>>> spy_mem_bin = hsbatch.io.spyfile.open_memmap()  # datacube after resampling
>>> meta_bands_bin = list(hsbatch.io.tools.meta_bands.values())
>>> ax = sns.lineplot(x=meta_bands, y=spy_mem[26][29], label='Hyperspectral (Pika II)', linewidth=3)
>>> ax = sns.lineplot(x=meta_bands_bin, y=spy_mem_bin[26][29], label='Spectral resample (20 nm)', marker='o', ms=6, ax=ax)
>>> ax.set_xlabel('Wavelength (nm)', weight='bold')
>>> ax.set_ylabel('Reflectance (%)', weight='bold')
>>> ax.set_title(r'API Example: `batch.spectral_resample`', weight='bold')

spectral_smooth(fname_list=None, base_dir=None, search_ext='bip', dir_level=0, base_dir_out=None, folder_name='spec_smooth', name_append='spec-smooth', window_size=11, order=2, stats=False, out_dtype=False, out_force=None, out_ext=False, out_interleave=False, out_byteorder=False)

Batch processing tool to spectrally smooth multiple datacubes in the same way.

Parameters

• fname_list (list, optional) – list of filenames to process; if left to None, will look at base_dir, search_ext, and dir_level parameters for files to process (default: None).

• base_dir (str, optional) – directory path to search for files to spectrally clip; if fname_list is not None, base_dir will be ignored (default: None).

• search_ext (str) – file format/extension to search for in all directories and subdirectories to determine which files to process; if fname_list is not None, search_ext will be ignored (default: ‘bip’).

• dir_level (int) – The number of directory levels to search; if None, searches all directory levels (default: 0).

• base_dir_out (str) – directory path to save all processed datacubes; if set to None, a folder named according to the folder_name parameter is added to base_dir

• folder_name (str) – folder to add to base_dir_out to save all the processed datacubes (default: ‘spec-smooth’).

• name_append (str) – name to append to the filename (default: ‘spec-smooth’).

• window_size (int) – the length of the window; must be an odd integer number (default: 11).

• order (int) – the order of the polynomial used in the filtering; must be less than window_size - 1 (default: 2).

• stats (bool) – whether to compute some basic descriptive statistics (mean, st. dev., and coefficient of variation) of the smoothed data array (default: False)

• out_XXX – Settings for saving the output files can be adjusted here if desired. They are stored in batch.io.defaults, and are therefore accessible at a high level. See ``hsio.set_io_defaults() for more information on each of the settings.

Note

The following batch example builds on the API example results of the batch.spatial_crop function. Please complete the batch.spatial_crop example to be sure your directory (i.e., base_dir) is populated with multiple hyperspectral datacubes. The following example will be using datacubes located in the following directory: F:\nigo0024\Documents\hs_process_demo\spatial_crop

Example

Load and initialize the batch module, checking to be sure the directory exists.

>>> import os
>>> from hs_process import batch
>>> base_dir = r'F:\nigo0024\Documents\hs_process_demo\spatial_crop'
>>> print(os.path.isdir(base_dir))
True
>>> hsbatch = batch(base_dir, search_ext='.bip')  # searches for all files in ``base_dir`` with a ".bip" file extension

Use batch.spectral_smooth to perform a Savitzky-Golay smoothing operation on each image/pixel in base_dir. The window_size and order can be adjusted to achieve desired smoothing results.

>>> hsbatch.spectral_smooth(base_dir=base_dir, folder_name='spec_smooth',
                            window_size=11, order=2)
Processing 40 files. If this is not what is expected, please check if files have already undergone processing. If existing files should be overwritten, be sure to set the ``out_force`` parameter.
Spectrally smoothing: F:\nigo0024\Documents\hs_process_demo\spatial_crop\Wells_rep2_20180628_16h56m_pika_gige_7_1011-spatial-crop.bip
Saving F:\nigo0024\Documents\hs_process_demo\spatial_crop\spec_smooth\Wells_rep2_20180628_16h56m_pika_gige_7_1011-spec-smooth.bip
Spectrally smoothing: F:\nigo0024\Documents\hs_process_demo\spatial_crop\Wells_rep2_20180628_16h56m_pika_gige_7_1012-spatial-crop.bip
Saving F:\nigo0024\Documents\hs_process_demo\spatial_crop\spec_smooth\Wells_rep2_20180628_16h56m_pika_gige_7_1012-spec-smooth.bip
...

Use seaborn to visualize the spectra of a single pixel in one of the processed images.

>>> import seaborn as sns
>>> fname = os.path.join(base_dir, 'Wells_rep2_20180628_16h56m_pika_gige_7_1011-spatial-crop.bip')
>>> hsbatch.io.read_cube(fname)
>>> spy_mem = hsbatch.io.spyfile.open_memmap()  # datacube before smoothing
>>> meta_bands = list(hsbatch.io.tools.meta_bands.values())
>>> fname = os.path.join(base_dir, 'spec_smooth', 'Wells_rep2_20180628_16h56m_pika_gige_7_1011-spec-smooth.bip')
>>> hsbatch.io.read_cube(fname)
>>> spy_mem_clip = hsbatch.io.spyfile.open_memmap()  # datacube after smoothing
>>> meta_bands_clip = list(hsbatch.io.tools.meta_bands.values())
>>> ax = sns.lineplot(x=meta_bands, y=spy_mem[26][29], label='Before spectral smoothing', linewidth=3)
>>> ax = sns.lineplot(x=meta_bands_clip, y=spy_mem_clip[26][29], label='After spectral smoothing', ax=ax)
>>> ax.set_xlabel('Wavelength (nm)', weight='bold')
>>> ax.set_ylabel('Reflectance (%)', weight='bold')
>>> ax.set_title(r'API Example: `batch.spectral_smooth`', weight='bold')

Notice how the “choppiness” of the spectral curve is lessened after the smoothing operation. There are spectral regions that perhaps had a lower signal:noise ratio and did not do particularlly well at smoothing (i.e., < 410 nm, ~770 nm, and ~820 nm). It may be wise to perform batch.spectral_smooth after batch.spectral_clip.

hs_process.segment module

class hs_process.segment.segment(spyfile)

Bases: object

Class for aiding in the segmentation and/or masking of image data to filter out pixels that are of least interest.

band_math_derivative(wl1=None, wl2=None, wl3=None, b1=None, b2=None, b3=None, spyfile=None, list_range=True, print_out=True)

Calculates a derivative-type spectral index from two input bands and/or wavelengths. Bands/wavelengths can be input as two individual bands, two sets of bands (i.e., list of bands), or range of bands (i.e., list of two bands indicating the lower and upper range).

Definition:

array_der = (wl1 - wl2) / (wl2 - wl3)

Parameters

• wl1 (int, float, or list) – the wavelength (or set of wavelengths) to be used as the first parameter of the derivative index; if list, then consolidates all bands between two wavelength values by calculating the mean pixel value across all bands in that range (default: None).

• wl2 (int, float, or list) – the wavelength (or set of wavelengths) to be used as the second parameter of the derivative index; if list, then consolidates all bands between two wavelength values by calculating the mean pixel value across all bands in that range (default: None).

• wl3 (int, float, or list) – the wavelength (or set of wavelengths) to be used as the third parameter of the derivative index; if list, then consolidates all bands between two wavelength values by calculating the mean pixel value across all bands in that range (default: None).

• b1 (int, float, or list) – the band (or set of bands) to be used as the first parameter of the derivative index; if list, then consolidates all bands between two band values by calculating the mean pixel value across all bands in that range (default: None).

• b2 (int, float, or list) – the band (or set of bands) to be used as the second parameter of the derivative index; if list, then consolidates all bands between two band values by calculating the mean pixel value across all bands in that range (default: None).

• b3 (int, float, or list) – the band (or set of bands) to be used as the third parameter of the derivative index; if list, then consolidates all bands between two band values by calculating the mean pixel value across all bands in that range (default: None).

• spyfile (SpyFile object or numpy.ndarray) – The datacube to process; if numpy.ndarray or None, loads band information from self.spyfile (default: None).

• list_range (bool) – Whether bands/wavelengths passed as a list is interpreted as a range of bands (True) or for each individual band in the list (False). If list_range is True, b1/wl1 and b2/wl2 should be lists with two items, and all bands/wavelegths between the two values will be used (default: True).

• print_out (bool) – Whether to print out the actual bands and wavelengths being used in the NDI calculation (default: True).

Returns

2-element tuple containing

• array_der (numpy.ndarray): Derivative band math array.

• metadata (dict): Modified metadata describing the derivative array (array_der).

Example

Load hsio and segment modules

>>> import numpy as np
>>> import os
>>> from hs_process import hsio
>>> from hs_process import segment
>>> data_dir = r'F:\nigo0024\Documents\hs_process_demo'
>>> fname_in = os.path.join(data_dir, 'Wells_rep2_20180628_16h56m_pika_gige_7-Radiance Conversion-Georectify Airborne Datacube-Convert Radiance Cube to Reflectance from Measured Reference Spectrum.bip.hdr')
>>> io = hsio(fname_in)
>>> my_segment = segment(io.spyfile)

Calculate the MERIS Terrestrial Chlorophyll Index (MTCI; Dash and Curran, 2004) via segment.band_math_derivative

>>> array_mtci, metadata = my_segment.band_math_derivative(wl1=754, wl2=709, wl3=681, spyfile=io.spyfile)
Band 1: [176]  Band 2: [154]  Band 3: [141]
Wavelength 1: [753.84]  Wavelength 2: [708.6784]  Wavelength 3: [681.992]

>>> array_mtci.shape
(617, 1300)
>>> np.nanmean(array_mtci)
9.401104

Show MTCI image via hsio.show_img

>>> io.show_img(array_mtci, vmin=-2, vmax=15)

band_math_mcari2(wl1=None, wl2=None, wl3=None, b1=None, b2=None, b3=None, spyfile=None, list_range=True, print_out=True)

Calculates the MCARI2 (Modified Chlorophyll Absorption Ratio Index Improved; Haboudane et al., 2004) spectral index from three input bands and/or wavelengths. Bands/wavelengths can be input as two individual bands, two sets of bands (i.e., list of bands), or range of bands (i.e., list of two bands indicating the lower and upper range).

Definition:

array_mcari2 = ((1.5 * (2.5 * (wl1 - wl2) - 1.3 * (wl1 - wl3))) / np.sqrt((2 * wl1 + 1)**2 - (6 * wl1 - 5 * np.sqrt(wl2)) - 0.5))

Parameters

• wl1 (int, float, or list) – the wavelength (or set of wavelengths) to be used as the first parameter of the MCARI2 index; if list, then consolidates all bands between two wavelength values by calculating the mean pixel value across all bands in that range (default: None).

• wl2 (int, float, or list) – the wavelength (or set of wavelengths) to be used as the second parameter of the MCARI2 index; if list, then consolidates all bands between two wavelength values by calculating the mean pixel value across all bands in that range (default: None).

• wl3 (int, float, or list) – the wavelength (or set of wavelengths) to be used as the third parameter of the MCARI2 index; if list, then consolidates all bands between two wavelength values by calculating the mean pixel value across all bands in that range (default: None).

• b1 (int, float, or list) – the band (or set of bands) to be used as the first parameter of the MCARI2 index; if list, then consolidates all bands between two band values by calculating the mean pixel value across all bands in that range (default: None).

• b2 (int, float, or list) – the band (or set of bands) to be used as the second parameter of the MCARI2 index; if list, then consolidates all bands between two band values by calculating the mean pixel value across all bands in that range (default: None).

• b3 (int, float, or list) – the band (or set of bands) to be used as the third parameter of the MCARI2 index; if list, then consolidates all bands between two band values by calculating the mean pixel value across all bands in that range (default: None).

• spyfile (SpyFile object or numpy.ndarray) – The datacube to process; if numpy.ndarray or None, loads band information from self.spyfile (default: None).

• list_range (bool) – Whether bands/wavelengths passed as a list is interpreted as a range of bands (True) or for each individual band in the list (False). If list_range is True, b1/wl1 and b2/wl2 should be lists with two items, and all bands/wavelegths between the two values will be used (default: True).

• print_out (bool) – Whether to print out the actual bands and wavelengths being used in the NDI calculation (default: True).

Returns

2-element tuple containing

• array_mcari2 (numpy.ndarray): MCARI2 spectral index band math array.

• metadata (dict): Modified metadata describing the MCARI2 index array (array_mcari2).

Example

Load hsio and segment modules

>>> import numpy as np
>>> import os
>>> from hs_process import hsio
>>> from hs_process import segment
>>> data_dir = r'F:\nigo0024\Documents\hs_process_demo'
>>> fname_in = os.path.join(data_dir, 'Wells_rep2_20180628_16h56m_pika_gige_7-Radiance Conversion-Georectify Airborne Datacube-Convert Radiance Cube to Reflectance from Measured Reference Spectrum.bip.hdr')
>>> io = hsio(fname_in)
>>> my_segment = segment(io.spyfile)

Calculate the MCARI2 spectral index (Haboudane et al., 2004) via segment.band_math_mcari2

>>> array_mcari2, metadata = my_segment.band_math_mcari2(wl1=800, wl2=670, wl3=550, spyfile=io.spyfile)
Band 1: [198]  Band 2: [135]  Band 3: [77]
Wavelength 1: [799.0016]  Wavelength 2: [669.6752]  Wavelength 3: [550.6128]

>>> np.nanmean(array_mcari2)
0.57376945

Show MCARI2 image via hsio.show_img

>>> io.show_img(array_mcari2)

band_math_ndi(wl1=None, wl2=None, b1=None, b2=None, spyfile=None, list_range=True, print_out=True)

Calculates a normalized difference spectral index from two input bands and/or wavelengths. Bands/wavelengths can be input as two individual bands, two sets of bands (i.e., list of bands), or range of bands (i.e., list of two bands indicating the lower and upper range).

Definition:

array_ndi = (wl1 - wl2) / (wl1 + wl2)

Parameters

• wl1 (int, float, or list) – the wavelength (or set of wavelengths) to be used as the first parameter of the normalized difference index; if list, then consolidates all bands between two wavelength values by calculating the mean pixel value across all bands in that range (default: None).

• wl2 (int, float, or list) – the wavelength (or set of wavelengths) to be used as the second parameter of the normalized difference index; if list, then consolidates all bands between two wavelength values by calculating the mean pixel value across all bands in that range (default: None).

• b1 (int, float, or list) – the band (or set of bands) to be used as the first parameter of the normalized difference index; if list, then consolidates all bands between two band values by calculating the mean pixel value across all bands in that range (default: None).

• b2 (int, float, or list) – the band (or set of bands) to be used as the second parameter of the normalized difference index; if list, then consolidates all bands between two band values by calculating the mean pixel value across all bands in that range (default: None).

• spyfile (SpyFile object or numpy.ndarray) – The datacube to process; if numpy.ndarray or None, loads band information from self.spyfile (default: None).

• list_range (bool) – Whether bands/wavelengths passed as a list is interpreted as a range of bands (True) or for each individual band in the list (False). If list_range is True, b1/wl1 and b2/wl2 should be lists with two items, and all bands/wavelegths between the two values will be used (default: True).

• print_out (bool) – Whether to print out the actual bands and wavelengths being used in the NDI calculation (default: True).

Returns

2-element tuple containing

• array_ndi (numpy.ndarray): Normalized difference band math array.

• metadata (dict): Modified metadata describing the normalized difference array (array_ndi).

Example

Load hsio and segment modules

>>> import numpy as np
>>> import os
>>> from hs_process import hsio
>>> from hs_process import segment
>>> data_dir = r'F:\nigo0024\Documents\hs_process_demo'
>>> fname_in = os.path.join(data_dir, 'Wells_rep2_20180628_16h56m_pika_gige_7-Radiance Conversion-Georectify Airborne Datacube-Convert Radiance Cube to Reflectance from Measured Reference Spectrum.bip.hdr')
>>> io = hsio(fname_in)
>>> my_segment = segment(io.spyfile)

Calculate the Normalized difference vegetation index using 10 nm bands centered at 800 nm and 680 nm via segment.band_math_ndi

>>> array_ndvi, metadata = my_segment.band_math_ndi(wl1=[795, 805], wl2=[675, 685], spyfile=io.spyfile)
Effective NDI: (800 - 680) / 800 + 680)

>>> np.nanmean(array_ndvi)
0.8184888

Show NDVI image via hsio.show_img

>>> io.show_img(array_ndvi)

band_math_ratio(wl1=None, wl2=None, b1=None, b2=None, spyfile=None, list_range=True, print_out=True)

Calculates a simple ratio spectral index from two input band and/or wavelengths. Bands/wavelengths can be input as two individual bands, two sets of bands (i.e., list of bands), or a range of bands (i.e., list of two bands indicating the lower and upper range).

Definition:

array_ratio = (wl1/wl2)

Parameters

• wl1 (int, float, or list) – the wavelength (or set of wavelengths) to be used as the first parameter of the ratio index; if list, then consolidates all bands between two wavelength values by calculating the mean pixel value across all bands in that range (default: None).

• wl2 (int, float, or list) – the wavelength (or set of wavelengths) to be used as the second parameter of the ratio index; if list, then consolidates all bands between two wavelength values by calculating the mean pixel value across all bands in that range (default: None).

• b1 (int, float, or list) – the band (or set of bands) to be used as the first parameter (numerator) of the ratio index; if list, then consolidates all bands between two band values by calculating the mean pixel value across all bands in that range (default: None).

• b2 (int, float, or list) – the bands (or set of bands) to be used as the second parameter (denominator) of the ratio index; if list, then consolidates all bands between two bands values by calculating the mean pixel value across all bands in that range (default: None).

• spyfile (SpyFile object or numpy.ndarray) – The datacube to process; if numpy.ndarray or None, loads band information from self.spyfile (default: None).

• list_range (bool) – Whether a band passed as a list is interpreted as a range of bands (True) or for each individual band in the list (False). If list_range is True, b1/wl1 and b2/wl2 should be lists with two items, and all bands/wavelegths between the two values will be used (default: True).

• print_out (bool) – Whether to print out the actual bands and wavelengths being used in the NDI calculation (default: True).

Returns

2-element tuple containing

• array_ratio (numpy.ndarray): Ratio band math array.

• metadata (dict): Modified metadata describing the ratio array (array_ratio).

Example

Load hsio and segment modules

>>> import numpy as np
>>> import os
>>> from hs_process import hsio
>>> from hs_process import segment
>>> data_dir = r'F:\nigo0024\Documents\hs_process_demo'
>>> fname_in = os.path.join(data_dir, 'Wells_rep2_20180628_16h56m_pika_gige_7-Radiance Conversion-Georectify Airborne Datacube-Convert Radiance Cube to Reflectance from Measured Reference Spectrum.bip.hdr')
>>> io = hsio(fname_in)
>>> my_segment = segment(io.spyfile)

Calculate a red/near-infrared band ratio using a range of bands (i.e., mimicking a broadband sensor) via segment.band_math_ratio

>>> array_ratio, metadata = my_segment.band_math_ratio(wl1=[630, 690], wl2=[800, 860], list_range=True)
Effective band ratio: (659/830)

>>> np.nanmean(array_ratio)
0.10981177

Notice that 29 spectral bands were consolidated (i.e., averaged) to mimic a single broad band. We can take the mean of two bands by changing list_range to False, and this slightly changes the result.

>>> array_ratio, metadata = my_segment.band_math_ratio(wl1=[630, 690], wl2=[800, 860], list_range=False)
Effective band ratio: (660/830)

>>> np.nanmean(array_ratio)
0.113607444

Show the red/near-infrared ratio image via hsio.show_img

>>> io.show_img(array_ratio, vmax=0.3)

composite_band(wl1=None, b1=None, spyfile=None, list_range=True, print_out=True)

Calculates a composite band from a range of bands or wavelengths. Bands/wavelengths can be input as individual bands, a set of bands (i.e., list of bands), or a range of bands (i.e., list of two bands indicating the lower and upper range).

Parameters

• wl1 (int, float, or list) – the wavelength (or set of wavelengths) to consolidate; if list, then all wavelengths between two wavelength values are consolidated by calculating the mean pixel value across all wavelengths in that range (default: None).

• b1 (int, float, or list) – the band (or set of bands) to consolidate; if list, then all bands between two band values are consolidated by calculating the mean pixel value across all bands in that range (default: None).

• spyfile (SpyFile object or numpy.ndarray) – The datacube to process; if numpy.ndarray or None, loads band information from self.spyfile (default: None).

• list_range (bool) – Whether a band passed as a list is interpreted as a range of bands (True) or for each individual band in the list (False). If list_range is True, b1/wl1 and b2/wl2 should be lists with two items, and all bands/wavelegths between the two values will be used (default: True).

• print_out (bool) – Whether to print out the actual bands and wavelengths being used in the NDI calculation (default: True).

Returns

2-element tuple containing

• array_b1 (numpy.ndarray): Consolidated array.

• metadata (dict): Modified metadata describing the consolidated array (array_b1).

load_spyfile(spyfile)

Loads a SpyFile (Spectral Python object) for data access and/or manipulation by the hstools class.

Parameters

spyfile (SpyFile object) – The datacube being accessed and/or manipulated.

Example

Load hsio and segment modules

>>> from hs_process import hsio
>>> from hs_process import segment
>>> fname_in = r'F:\nigo0024\Documents\hs_process_demo\Wells_rep2_20180628_16h56m_pika_gige_7-Convert Radiance Cube to Reflectance from Measured Reference Spectrum.bip.hdr'
>>> io = hsio(fname_in)
>>> my_segment = segment(io.spyfile)

Load datacube via segment.load_spyfile

>>> my_segment.load_spyfile(io.spyfile)
>>> my_segment.spyfile
Data Source:   'F:\nigo0024\Documents\hs_process_demo\Wells_rep2_20180628_16h56m_pika_gige_7-Convert Radiance Cube to Reflectance from Measured Reference Spectrum.bip'
    # Rows:            617
    # Samples:        1300
    # Bands:           240
    Interleave:        BIP
    Quantization:  32 bits
    Data format:   float32

hs_process.spatial_mod module

class hs_process.spatial_mod.spatial_mod(spyfile, gdf=None, **kwargs)

Bases: object

Class for manipulating data within the spatial domain (e.g., cropping a datacube by a geographical boundary).

crop_many_gdf(spyfile=None, gdf=None, **kwargs)

Crops many plots from a single image by comparing the image to a polygon file (geopandas.GoeDataFrame) that contains plot information and geometry of plot boundaries.

Parameters

• spyfile (SpyFile object, optional) – The datacube to crop; if None, loads datacube and band information from spatial_mod.spyfile (default: None).

• gdf (geopandas.GeoDataFrame, optional) – the plot IDs and polygon geometery of each of the plots; ‘plot_id’ must be used as a column name to identify each of the plots, and should be an integer; if None, loads geodataframe from spatial_mod.gdf (default: None).

• kwargs –

  Can be any of the keys in self.defaults.crop_defaults: plot_id_ref (int, optional): the plot ID of the reference plot.

  plot_id_ref is required if passing pix_e_ul, pix_n_ul, or n_plots because it is used as the reference point for any of the adjustments/modifications dictated by said parameters. plot_id_ref must be present in the gdf, and the extent of plot_id_ref must intersect the extent of the datacube (default: None).

  pix_e_ul (int, optional): upper left pixel column (easting) of

  plot_id_ref; this is used to calculate the offset between the GeoDataFrame geometry and the approximate image georeference error (default: None).

  pix_n_ul (int, optional): upper left pixel row (northing) of

  plot_id_ref; this is used to calculate the offset between the GeoDataFrame geometry and the approximate image georeference error (default: None).

  crop_e_m (float, optional): length of each row (easting

  direction) of the cropped image in map units (e.g., meters; default: None).

  crop_n_m (float, optional): length of each column (northing

  direction) of the cropped image in map units (e.g., meters; default: None)

  crop_e_pix (int, optional): number of pixels in each row in the

  cropped image (default: None).

  crop_n_pix (int, optional): number of pixels in each column in

  the cropped image (default: None).

  buf_e_m (float, optional): The buffer distance in the easting

  direction (in map units; e.g., meters) to be applied after calculating the original crop area; the buffer is considered after crop_X_m / crop_X_pix. A positive value will reduce the size of crop_X_m / crop_X_pix, and a negative value will increase it (default: None).

  buf_n_m (float, optional): The buffer distance in the northing

  direction (in map units; e.g., meters) to be applied after calculating the original crop area; the buffer is considered after crop_X_m / crop_X_pix. A positive value will reduce the size of crop_X_m / crop_X_pix, and a negative value will increase it (default: None).

  buf_e_pix (int, optional): The buffer distance in the easting

  direction (in pixel units) to be applied after calculating the original crop area (default: None).

  buf_n_pix (int, optional): The buffer distance in the northing

  direction (in pixel units) to be applied after calculating the original crop area (default: None).

  gdf_shft_e_m (float): The distance to shift the cropped

  datacube from the upper left/NW plot corner in the east direction (negative value will shift to the west). Only relevent when gdf is passed. This shift is applied after the offset is applied from buf_X (default: 0.0).

  gdf_shft_n_m (float): The distance to shift the cropped

  datacube from the upper left/NW plot corner in the north direction (negative value will shift to the south). Only relevent when gdf is passed. This shift is applied after the offset is applied from buf_X (default: 0.0).

  gdf_shft_e_pix (int): The pixel units to shift the cropped

  datacube from the upper left/NW plot corner in the east direction (negative value will shift to the west). Only relevent when gdf is passed. This shift is applied after the offset is applied from buf_X (default: 0.0).

  gdf_shft_n_pix (int): The pixel units to shift the cropped

  datacube from the upper left/NW plot corner in the north direction (negative value will shift to the south). Only relevent when gdf is passed. This shift is applied after the offset is applied from buf_X (default: 0.0).

  n_plots (int, optional): number of plots to crop, starting with

  plot_id_ref and moving from West to East and North to South. This can be used to limit the number of cropped plots (default; None).

Returns

• df_plots (pandas.DataFrame) – data for which to crop each plot; includes ‘plot_id_ref’, ‘pix_e_ul’, and ‘pix_n_ul’ columns. This data can be passed to spatial_mod.crop_single to perform the actual cropping.

Return type

pandas.DataFrame

Note

If pix_e_ul or pix_n_ul are passed, the pixel offset from the northwest corner of plot_id_ref will be calculated. This offset is then applied to all plots within the extent of the image to systematically shift the actual upper left pixel locations for each plot, effectively shifting the easting and/or northing of the upper left pixel of the hyperspectral datacube to match that of the gdf. If the shift should only apply to a select number of plots, n_plots can be passed to restrict the number of plots that are processed.

Note

Either the pixel coordinate or the map unit coordinate should be passed for crop_X_Y and buf_X_Y in each direction (i.e., easting and northing). Do not pass both.

Example

Load the hsio and spatial_mod modules

>>> import geopandas as gpd
>>> import os
>>> from hs_process import hsio
>>> from hs_process import spatial_mod

Read datacube and spatial plot boundaries

>>> fname_in = r'F:\nigo0024\Documents\hs_process_demo\Wells_rep2_20180628_16h56m_pika_gige_7-Radiance Conversion-Georectify Airborne Datacube-Convert Radiance Cube to Reflectance from Measured Reference Spectrum.bip.hdr'
>>> fname_in = r'F:\nigo0024\Documents\hs_process_demo\Wells_rep2_20180628_16h56m_pika_gige_7_nohist-Radiance Conversion-Georectify Airborne Datacube-Convert Radiance Cube to Reflectance from Measured Reference Spectrum.bip.hdr'

>>> fname_gdf = r'F:\nigo0024\Documents\hs_process_demo\plot_bounds.geojson'
>>> gdf = gpd.read_file(fname_gdf)
>>> io = hsio(fname_in)
>>> my_spatial_mod = spatial_mod(
        io.spyfile, base_dir=io.base_dir, name_short=io.name_short,
        name_long=io.name_long)
>>> dir_out = os.path.join(io.base_dir, 'spatial_mod', 'crop_many_gdf')
>>> name_append = '-crop-many-gdf'

Get instructions on how plots should be cropped via spatial_mod.crop_many_gdf(); note that a pandas.DataFrame is returned with information describing how each plot should be cropped.

>>> df_plots = my_spatial_mod.crop_many_gdf(spyfile=io.spyfile, gdf=gdf)
>>> df_plots.head(5)
    plot_id_ref  pix_e_ul  pix_n_ul  crop_e_pix  crop_n_pix
0          1018       478         0         229          76
1           918       707         0         229          76
2           818       936         0         229          76
3           718      1165         0         229          76
4           618      1394         0         229          76
...

Use the data from the first row of df_plots to crop a single plot from the original image (uses spatial_mod.crop_single)

>>> pix_e_ul=113
>>> pix_n_ul=0
>>> crop_e_pix=229
>>> crop_n_pix=75
>>> plot_id_ref=1018
>>> array_crop, metadata = my_spatial_mod.crop_single(
        pix_e_ul=pix_e_ul, pix_n_ul=pix_n_ul, crop_e_pix=crop_e_pix, crop_n_pix=crop_n_pix,
        spyfile=io.spyfile, plot_id_ref=plot_id_ref)

Save the cropped datacube and geotiff to a new directory

>>> fname_out = os.path.join(dir_out, io.name_short + '_plot_' + str(1018) + name_append + '.' + io.defaults.envi_write.interleave)
>>> fname_out_tif = os.path.join(dir_out, io.name_short + '_plot_' + str(1018) + '.tif')
>>> io.write_cube(fname_out, array_crop, metadata=metadata, force=True)
>>> io.write_tif(fname_out_tif, spyfile=array_crop, metadata=metadata)
Saving F:\nigo0024\Documents\hs_process_demo\spatial_mod\crop_many_gdf\Wells_rep2_20180628_16h56m_pika_gige_7_plot_1018-crop-many-gdf.bip
Either `projection_out` is `None` or `geotransform_out` is `None` (or both are). Retrieving projection and geotransform information by loading `hsio.fname_in` via GDAL. Be sure this is appropriate for the data you are trying to write.

Using a for loop, use spatial_mod.crop_single and hsio.write_cube to crop by plot and save cropped datacubes to file

>>> for idx, row in df_plots.iterrows():
>>>     io.read_cube(fname_in, name_long=io.name_long,
                     name_plot=row['plot_id_ref'],
                     name_short=io.name_short)
>>>     my_spatial_mod.load_spyfile(io.spyfile)
>>>     array_crop, metadata = my_spatial_mod.crop_single(
                pix_e_ul=row['pix_e_ul'], pix_n_ul=row['pix_n_ul'],
                crop_e_pix=row['crop_e_pix'], crop_n_pix=row['crop_n_pix'],
                buf_e_m=2.0, buf_n_m=0.75,
                plot_id_ref=row['plot_id_ref'])
>>>     fname_out = os.path.join(dir_out, io.name_short + '_plot_' + str(row['plot_id_ref']) + name_append + '.bip.hdr')
>>>     fname_out_tif = os.path.join(dir_out, io.name_short + '_plot_' + str(row['plot_id_ref']) + '.tif')
>>>     io.write_cube(fname_out, array_crop, metadata=metadata, force=True)  # force=True to overwrite the plot_1018 image
>>>     io.write_tif(fname_out_tif, spyfile=array_crop, metadata=metadata)
Saving F:\nigo0024\Documents\hs_process_demo\crop_many_gdf\Wells_rep2_20180628_16h56m_pika_gige_7_plot_1018.bip
Saving F:\nigo0024\Documents\hs_process_demo\crop_many_gdf\Wells_rep2_20180628_16h56m_pika_gige_7_plot_918.bip
Saving F:\nigo0024\Documents\hs_process_demo\crop_many_gdf\Wells_rep2_20180628_16h56m_pika_gige_7_plot_818.bip
Saving F:\nigo0024\Documents\hs_process_demo\crop_many_gdf\Wells_rep2_20180628_16h56m_pika_gige_7_plot_718.bip
Saving F:\nigo0024\Documents\hs_process_demo\crop_many_gdf\Wells_rep2_20180628_16h56m_pika_gige_7_plot_618.bip
Saving F:\nigo0024\Documents\hs_process_demo\crop_many_gdf\Wells_rep2_20180628_16h56m_pika_gige_7_plot_1017.bip
Saving F:\nigo0024\Documents\hs_process_demo\crop_many_gdf\Wells_rep2_20180628_16h56m_pika_gige_7_plot_917.bip
Saving F:\nigo0024\Documents\hs_process_demo\crop_many_gdf\Wells_rep2_20180628_16h56m_pika_gige_7_plot_817.bip
Saving F:\nigo0024\Documents\hs_process_demo\crop_many_gdf\Wells_rep2_20180628_16h56m_pika_gige_7_plot_717.bip
Saving F:\nigo0024\Documents\hs_process_demo\crop_many_gdf\Wells_rep2_20180628_16h56m_pika_gige_7_plot_617.bip
...

Open cropped geotiff images in QGIS to visualize the extent of the cropped images compared to the original datacube and the plot boundaries (the full extent image is darkened and displayed in the background:

crop_single(pix_e_ul=0, pix_n_ul=0, crop_e_pix=None, crop_n_pix=None, crop_e_m=None, crop_n_m=None, buf_e_pix=None, buf_n_pix=None, buf_e_m=None, buf_n_m=None, spyfile=None, plot_id_ref=None, gdf=None, gdf_shft_e_pix=None, gdf_shft_n_pix=None, gdf_shft_e_m=None, gdf_shft_n_m=None, name_append='spatial-crop-single')

Crops a single plot from an image. If plot_id_ref and gdf are explicitly passed (i.e., they will not be loaded from spatial_mod class), the “map info” tag in the metadata will be adjusted to center the cropped area within the appropriate plot geometry.

Parameters

• pix_e_ul (int, optional) – upper left pixel column (easting) to begin cropping (default: 0).

• pix_n_ul (int, optional) – upper left pixel row (northing) to begin cropping (default: 0).

• crop_e_m (float, optional) – length of each row (easting direction) of the cropped image in map units (e.g., meters; default: None).

• crop_n_m (float, optional) – length of each column (northing direction) of the cropped image in map units (e.g., meters; default: None)

• crop_e_pix (int, optional) – number of pixels in each row in the cropped image (default: None).

• crop_n_pix (int, optional) – number of pixels in each column in the cropped image (default: None).

• buf_e_m (float, optional) – The buffer distance in the easting direction (in map units; e.g., meters) to be applied after calculating the original crop area; the buffer is considered after crop_X_m / crop_X_pix. A positive value will reduce the size of crop_X_m / crop_X_pix, and a negative value will increase it (default: None).

• buf_n_m (float, optional) – The buffer distance in the northing direction (in map units; e.g., meters) to be applied after calculating the original crop area; the buffer is considered after crop_X_m / crop_X_pix. A positive value will reduce the size of crop_X_m / crop_X_pix, and a negative value will increase it (default: None).

• buf_e_pix (int, optional) – The buffer distance in the easting direction (in pixel units) to be applied after calculating the original crop area (default: None).

• buf_n_pix (int, optional) – The buffer distance in the northing direction (in pixel units) to be applied after calculating the original crop area (default: None).

• spyfile (SpyFile object or numpy.ndarray) – The datacube to crop; if numpy.ndarray or None, loads band information from self.spyfile (default: None).

• plot_id_ref (int) – the plot ID of the area to be cropped (default: None).

• gdf (geopandas.GeoDataFrame) – the plot names and polygon geometery of each of the plots; ‘plot_id’ must be used as a column name to identify each of the plots, and should be an integer.

• gdf_shft_e_m (float) – The distance to shift the cropped datacube from the upper left/NW plot corner in the east direction (negative value will shift to the west). Only relevent when gdf is passed. This shift is applied after the offset is applied from buf_X (default: 0.0).

• gdf_shft_n_m (float) – The distance to shift the cropped datacube from the upper left/NW plot corner in the north direction (negative value will shift to the south). Only relevent when gdf is passed. This shift is applied after the offset is applied from buf_X (default: 0.0).

• gdf_shft_e_pix (int) – The pixel units to shift the cropped datacube from the upper left/NW plot corner in the east direction (negative value will shift to the west). Only relevent when gdf is passed. This shift is applied after the offset is applied from buf_X (default: 0.0).

• gdf_shft_n_pix (int) – The pixel units to shift the cropped datacube from the upper left/NW plot corner in the north direction (negative value will shift to the south). Only relevent when gdf is passed. This shift is applied after the offset is applied from buf_X (default: 0.0).

• name_append (str) – NOT YET SUPPORTED; name to append to the filename (default: ‘spatial-crop-single’).

Returns

2-element tuple containing

• array_crop (numpy.ndarray): Cropped datacube.

• metadata (dict): Modified metadata describing the cropped hyperspectral datacube (array_crop).

Example

Load and initialize the hsio and spatial_mod modules

>>> from hs_process import hsio
>>> from hs_process import spatial_mod
>>> fname_in = r'F:\nigo0024\Documents\hs_process_demo\Wells_rep2_20180628_16h56m_pika_gige_7-Convert Radiance Cube to Reflectance from Measured Reference Spectrum.bip.hdr'
>>> io = hsio(fname_in)
>>> my_spatial_mod = spatial_mod(
        io.spyfile, base_dir=io.base_dir, name_short=io.name_short,
        name_long=io.name_long)

Crop an area with a width (easting) 200 pixels and a height (northing) of 50 pixels, with a northwest/upper left origin at the 342nd column (easting) and 75th row (northing).

>>> pix_e_ul = 342
>>> pix_n_ul = 75
>>> array_crop, metadata = my_spatial_mod.crop_single(pix_e_ul, pix_n_ul, crop_e_pix=200, crop_n_pix=50)

Save as a geotiff using io.write_tif, then load into QGIS to visualize.

>>> fname_tif = r'F:\nigo0024\Documents\hs_process_demo\spatial_mod\crop_single\crop_single.tif'
>>> io.write_tif(fname_tif, array_crop, metadata=metadata)
Either `projection_out` is `None` or `geotransform_out` is `None` (or both are). Retrieving projection and geotransform information by loading `hsio.fname_in` via GDAL. Be sure this is appropriate for the data you are trying to write.

Open cropped geotiff image in QGIS to visualize the extent of the cropped image compared to the original datacube and the plot boundaries (the full extent image is darkened and displayed in the background):

load_spyfile(spyfile, **kwargs)

Loads a SpyFile (Spectral Python object) for data access and/or manipulation by the hstools class.

Parameters

• spyfile (SpyFile object) – The datacube being accessed and/or manipulated.

• base_dir (str) – to be used by the plot gdf attribute data.

• name_short (str) – to be used by the plot gdf attribute data.

• name_long (str) – to be used by the plot gdf attribute data.

Example

Load and initialize the hsio and spatial_mod modules

>>> from hs_process import hsio
>>> from hs_process import spatial_mod
>>> fname_in = r'F:\nigo0024\Documents\hs_process_demo\Wells_rep2_20180628_16h56m_pika_gige_7-Convert Radiance Cube to Reflectance from Measured Reference Spectrum.bip.hdr'
>>> io = hsio(fname_in)
>>> my_spatial_mod = spatial_mod(
        io.spyfile, base_dir=io.base_dir, name_short=io.name_short,
        name_long=io.name_long)

Load datacube using spatial_mod.load_spyfile

>>> my_spatial_mod.load_spyfile(
        io.spyfile, base_dir=io.base_dir, name_short=io.name_short,
        name_long=io.name_long)
>>> my_spatial_mod.spyfile
Data Source:   'F:\nigo0024\Documents\hs_process_demo\Wells_rep2_20180628_16h56m_pika_gige_7-Convert Radiance Cube to Reflectance from Measured Reference Spectrum.bip'
    # Rows:            617
    # Samples:        1300
    # Bands:           240
    Interleave:        BIP
    Quantization:  32 bits
    Data format:   float32

hs_process.spec_mod module

class hs_process.spec_mod.spec_mod(spyfile)

Bases: object

Class for manipulating data within the spectral domain, which is usually pixel-based.

load_spyfile(spyfile)

Loads a SpyFile (Spectral Python object) for data access and/or manipulation by the hstools class.

Parameters

spyfile (SpyFile object) – The datacube being accessed and/or manipulated.

Example

Load and initialize the hsio and spec_mod modules

>>> from hs_process import hsio
>>> from hs_process import spec_mod
>>> fname_in = r'F:\nigo0024\Documents\hs_process_demo\Wells_rep2_20180628_16h56m_pika_gige_7-Convert Radiance Cube to Reflectance from Measured Reference Spectrum.bip.hdr'
>>> io = hsio(fname_in)
>>> my_spec_mod = spec_mod(io.spyfile)

Load datacube

>>> my_spec_mod.load_spyfile(io.spyfile)
>>> my_spec_mod.spyfile
Data Source:   'F:\nigo0024\Documents\hs_process_demo\Wells_rep2_20180628_16h56m_pika_gige_7-Convert Radiance Cube to Reflectance from Measured Reference Spectrum.bip'
    # Rows:            617
    # Samples:        1300
    # Bands:           240
    Interleave:        BIP
    Quantization:  32 bits
    Data format:   float32

spec_derivative(spyfile_spec=None, order=1)

Calculates the numeric derivative spectra from spyfile_spec.

The derivavative spectra is calculated as the slope (rise over run) of the input spectra, and is normalized by the wavelength unit.

Parameters

• spyfile_spec – The spectral spyfile object to calculate the derivative for.

• order (int) – The order of the derivative (default: 1).

Example

Load and initialize hsio

>>> import os
>>> from hs_process import hsio
>>> from hs_process import spec_mod
>>> data_dir = r'F:\nigo0024\Documents\hs_process_demo'
>>> fname_hdr_spec = os.path.join(data_dir, 'Wells_rep2_20180628_16h56m_pika_gige_7_plot_611-cube-to-spec-mean.spec.hdr')
>>> io = hsio()
>>> io.read_spec(fname_hdr_spec)
>>> my_spec_mod = spec_mod(io.spyfile_spec)

Calculate the numeric derivative.

>>> spec_dydx, metadata_dydx = my_spec_mod.spec_derivative(order=1)

>>> io.write_spec('spec_derivative_order-1.spec.hdr', spec_dydx, df_std=None, metadata=metadata_dydx)

Plot the numeric derivative spectra and compare against the original spectra.

>>> import numpy as np
>>> import seaborn as sns
>>> sns.set_style("ticks")
>>> wl_x = np.array([float(i) for i in metadata_dydx['wavelength']])
>>> y_ref = io.spyfile_spec.open_memmap()[0,0,:]*100
>>> ax1 = sns.lineplot(wl_x, y_ref)
>>> ax2 = ax1.twinx()
>>> ax2 = sns.lineplot(wl_x, 0, ax=ax2, color='gray')
>>> ax2 = sns.lineplot(wl_x, spec_dydx[0,0,:]*100, ax=ax2, color=sns.color_palette()[1])
>>> ax2.set(ylim=(-0.8, 1.5))
>>> ax1.set_xlabel('Wavelength (nm)', weight='bold')
>>> ax1.set_ylabel('Reflectance (%)', weight='bold')
>>> ax2.set_ylabel('Reflectance derivative (%)', weight='bold')
>>> ax1.set_title(r'API Example: `hstools.spec_derivative`', weight='bold')

spectral_clip(wl_bands=[[0, 420], [760, 776], [813, 827], [880, 1000]], spyfile=None)

Removes/clips designated wavelength bands from the hyperspectral datacube.

Parameters

• wl_bands (list or list of lists) – minimum and maximum wavelenths to clip from image; if multiple groups of wavelengths should be cut, this should be a list of lists. For example, wl_bands=[760, 776] will clip all bands greater than 760.0 nm and less than 776.0 nm; wl_bands = [[0, 420], [760, 776], [813, 827], [880, 1000]] will clip all band less than 420.0 nm, bands greater than 760.0 nm and less than 776.0 nm, bands greater than 813.0 nm and less than 827.0 nm, and bands greater than 880 nm (default).

• spyfile (SpyFile object or numpy.ndarray) – The data cube to clip; if numpy.ndarray or None, loads band information from spec_mod.spyfile (default: None).

Returns

2-element tuple containing

• array_clip (numpy.ndarray): Clipped datacube.

• metadata (dict): Modified metadata describing the clipped hyperspectral datacube (array_clip).

Example

Load and initialize hsio and spec_mod

>>> import os
>>> from hs_process import hsio
>>> from hs_process import spec_mod
>>> data_dir = r'F:\nigo0024\Documents\hs_process_demo'
>>> fname_hdr = os.path.join(data_dir, 'Wells_rep2_20180628_16h56m_pika_gige_7-Radiance Conversion-Georectify Airborne Datacube-Convert Radiance Cube to Reflectance from Measured Reference Spectrum.bip.hdr')
>>> io = hsio()
>>> io.read_cube(fname_hdr)
>>> my_spec_mod = spec_mod(io.spyfile)

Using spec_mod.spectral_clip, clip all spectral bands below 420 nm and above 880 nm, as well as the bands near the oxygen absorption (i.e., 760-776 nm) and water absorption (i.e., 813-827 nm) regions.

>>> array_clip, metadata_clip = my_spec_mod.spectral_clip(
        wl_bands=[[0, 420], [760, 776], [813, 827], [880, 1000]])

Plot the spectra of the unclippe hyperspectral image and compare to that of the clipped image for a single pixel.

>>> import seaborn as sns
>>> from ast import literal_eval
>>> spy_hs = my_spec_mod.spyfile.open_memmap()  # datacube before smoothing
>>> meta_bands = list(io.tools.meta_bands.values())
>>> meta_bands_clip = sorted([float(i) for i in literal_eval(metadata_clip['wavelength'])])
>>> ax = sns.lineplot(x=meta_bands, y=spy_hs[200][800]*100, label='Before spectral clipping', linewidth=3)
>>> ax = sns.lineplot(x=meta_bands_clip, y=array_clip[200][800]*100, label='After spectral clipping', ax=ax)
>>> ax.set_xlabel('Wavelength (nm)', weight='bold')
>>> ax.set_ylabel('Reflectance (%)', weight='bold')
>>> ax.set_title(r'API Example: `spec_mod.spectral_clip`', weight='bold')

spectral_mimic(sensor='sentinel-2a', df_band_response=None, col_wl='wl_nm', center_wl='peak', spyfile=None)

Mimics the response of a multispectral sensor based on transmissivity of sensor bands across a range of wavelength values by calculating its weighted average response and interpolating the hyperspectral response.

Parameters:

sensor (str): Should be one of

[“sentera_6x”, “micasense_rededge_3”, “sentinel-2a”, “sentinel-2b”, “custom”]; if “custom”, df_band_response and col_wl must be passed.

df_band_response (pd.DataFrame): A DataFrame that contains the

transmissivity (%) for each sensor band (as columns) mapped to the continuous wavelength values (as rows). Required if sensor is “custom”, ignored otherwise.

col_wl (str): The column of df_band_response denoting the

wavlengths (default: ‘wl_nm’).

center_wl (str): Indicates how the center wavelength of each

band is determined. If center_wl is “peak”, the point at which transmissivity is at its maximum is used as the center wavelength. If center_wl is “weighted”, the weighted average is used to compute the center wavelength. Must be one of [“peak”, “weighted”] (default: "peak").

spyfile (SpyFile object): The datacube being accessed and/or

manipulated.

Returns:

2-element tuple containing

• array_multi (numpy.ndarray): Mimicked datacube.

• metadata (dict): Modified metadata describing the mimicked spectral array (array_multi).

Example:

Load and initialize hsio and spec_mod

>>> import os
>>> from hs_process import hsio
>>> from hs_process import spec_mod
>>> data_dir = r'F:\nigo0024\Documents\hs_process_demo'
>>> fname_hdr = os.path.join(data_dir, 'Wells_rep2_20180628_16h56m_pika_gige_7-Radiance Conversion-Georectify Airborne Datacube-Convert Radiance Cube to Reflectance from Measured Reference Spectrum.bip.hdr')

>>> data_dir2 = r'G:\BBE\AGROBOT\Shared Work\hs_process_results\data

ef_closest_panelcrop_plot’

>>> fname_hdr = os.path.join(data_dir2, 'study_aerffield_date_20190708_plot_5110-crop-plot.bip.hdr')
>>> array = io.spyfile.open_memmap()

>>> io = hsio()
>>> io.read_cube(fname_hdr)
>>> my_spec_mod = spec_mod(io.spyfile)

Use spec_mod.spectral_mimic to mimic the Sentinel-2A spectral response function.

>>> array_s2a, metadata_s2a = my_spec_mod.spectral_mimic(sensor='sentinel-2a', center_wl='weighted')

Plot the mean spectral response of the hyperspectral image to that of the mimicked Sentinel-2A image bands (mean calculated across the entire image).

>>> import seaborn as sns
>>> from ast import literal_eval
>>> spy_hs = my_spec_mod.spyfile.open_memmap()  # datacube before smoothing
>>> meta_bands = list(io.tools.meta_bands.values())
>>> meta_bands_s2a = sorted([float(i) for i in literal_eval(metadata_s2a['wavelength'])])

>>> ax = sns.lineplot(x=meta_bands, y=spy_hs[200][800]*100, label='Hyperspectral (Pika II)', linewidth=3)
>>> ax = sns.lineplot(x=meta_bands_s2a, y=array_s2a[200][800]*100, label='Sentinel-2A "mimic"', marker='o', ms=6, ax=ax)
>>> ax.set_xlabel('Wavelength (nm)', weight='bold')
>>> ax.set_ylabel('Reflectance (%)', weight='bold')
>>> ax.set_title(r'API Example: `spec_mod.spectral_mimic`', weight='bold')

Use spec_mod.spectral_mimic to mimic the Sentera 6x spectral configuration and compare to both hyperspectral and Sentinel-2A.

>>> array_6x, metadata_6x = my_spec_mod.spectral_mimic(sensor='sentera_6x', center_wl='peak')
>>> meta_bands_6x = sorted([float(i) for i in literal_eval(metadata_6x['wavelength'])])
>>> ax = sns.lineplot(x=meta_bands, y=spy_hs[200][800]*100, label='Hyperspectral (Pika II)', linewidth=3)
>>> ax = sns.lineplot(x=meta_bands_s2a, y=array_s2a[200][800]*100, label='Sentinel-2A "mimic"', marker='o', ms=6, ax=ax)
>>> ax = sns.lineplot(x=meta_bands_6x, y=array_6x[200][800]*100, label='Sentera 6X "mimic"', marker='o', ms=8, ax=ax)
>>> ax.set_xlabel('Wavelength (nm)', weight='bold')
>>> ax.set_ylabel('Reflectance (%)', weight='bold')
>>> ax.set_title(r'API Example: `spec_mod.spectral_mimic`', weight='bold')

And finally, mimic the Micasense RedEdge-MX and compare to hyperspectral, Sentinel-2A, and Sentera 6X.

>>> array_re3, metadata_re3 = my_spec_mod.spectral_mimic(sensor='micasense_rededge_3', center_wl='peak')
>>> meta_bands_re3 = sorted([float(i) for i in literal_eval(metadata_re3['wavelength'])])
>>> ax = sns.lineplot(x=meta_bands, y=spy_hs[200][800]*100, label='Hyperspectral (Pika II)', linewidth=3)
>>> ax = sns.lineplot(x=meta_bands_s2a, y=array_s2a[200][800]*100, label='Sentinel-2A "mimic"', marker='o', ms=6, ax=ax)
>>> ax = sns.lineplot(x=meta_bands_6x, y=array_6x[200][800]*100, label='Sentera 6X "mimic"', marker='o', ms=8, ax=ax)
>>> ax = sns.lineplot(x=meta_bands_re3, y=array_re3[200][800]*100, label='Micasense RedEdge 3 "mimic"', marker='o', ms=8, ax=ax)
>>> ax.set_xlabel('Wavelength (nm)', weight='bold')
>>> ax.set_ylabel('Reflectance (%)', weight='bold')
>>> ax.set_title(r'API Example: `spec_mod.spectral_mimic`', weight='bold')

spectral_resample(bandwidth=None, bins_n=None, spyfile=None)

Performs pixel-wise resampling of spectral bands via binning

(calculates the mean across all bands within each bandwidth

region for each image pixel).

Parameters

• bandwidth (float or int) – The bandwidth of the bands after spectral resampling is complete (units should be consistent with that of the .hdr file). Setting bandwidth to 10 will consolidate bands that fall within every 10 nm interval.

• bins_n (int) – The number of bins (i.e., “bands”) to achieve after spectral resampling is complete. Ignored if bandwidth is not None.

• spyfile (SpyFile object) – The datacube being accessed and/or manipulated.

Returns

2-element tuple containing

• array_bin (numpy.ndarray): Binned datacube.

• metadata (dict): Modified metadata describing the binned spectral array (array_bin).

Example

Load and initialize hsio and spec_mod

>>> import os
>>> from hs_process import hsio
>>> from hs_process import spec_mod
>>> data_dir = r'F:\nigo0024\Documents\hs_process_demo'
>>> fname_hdr = os.path.join(data_dir, 'Wells_rep2_20180628_16h56m_pika_gige_7-Radiance Conversion-Georectify Airborne Datacube-Convert Radiance Cube to Reflectance from Measured Reference Spectrum.bip.hdr')
>>> io = hsio()
>>> io.read_cube(fname_hdr)
>>> my_spec_mod = spec_mod(io.spyfile)

Use spec_mod.spectral_resample to “bin” the datacube to bands with 20 nm bandwidths.

>>> array_bin, metadata_bin = my_spec_mod.spectral_resample(bandwidth=20)

Plot the mean spectral response of the hyperspectral image to that of the binned image bands (mean calculated across the entire image).

>>> import seaborn as sns
>>> from ast import literal_eval
>>> spy_hs = my_spec_mod.spyfile.open_memmap()  # datacube before smoothing
>>> meta_bands = list(io.tools.meta_bands.values())
>>> meta_bands_bin = sorted([float(i) for i in literal_eval(metadata_bin['wavelength'])])

>>> ax = sns.lineplot(x=meta_bands, y=spy_hs[200][800]*100, label='Hyperspectral (Pika II)', linewidth=3)
>>> ax = sns.lineplot(x=meta_bands_bin, y=array_bin[200][800]*100, label='Spectral resample (20 nm)', marker='o', ms=6, ax=ax)
>>> ax.set_xlabel('Wavelength (nm)', weight='bold')
>>> ax.set_ylabel('Reflectance (%)', weight='bold')
>>> ax.set_title(r'API Example: `spec_mod.spectral_resample`', weight='bold')

spectral_smooth(window_size=11, order=2, spyfile=None)

Performs Savitzky-Golay smoothing on the spectral domain.

Parameters

• window_size (int) – the length of the window; must be an odd integer number (default: 11).

• order (int) – the order of the polynomial used in the filtering; must be less than window_size - 1 (default: 2).

• spyfile (SpyFile object or numpy.ndarray) – The data cube to clip; if numpy.ndarray or None, loads band information from spec_mod.spyfile (default: None).

Returns

2-element tuple containing

• array_smooth (numpy.ndarray): Clipped datacube.

• metadata (dict): Modified metadata describing the smoothed hyperspectral datacube (array_smooth).

Note

Because the smoothing operation is performed for every pixel individually, this function may take several minutes for large images.

Example

Load and initialize hsio and spec_mod

>>> import os
>>> from hs_process import hsio
>>> from hs_process import spec_mod
>>> data_dir = r'F:\nigo0024\Documents\hs_process_demo'
>>> fname_hdr = os.path.join(data_dir, 'Wells_rep2_20180628_16h56m_pika_gige_7-Radiance Conversion-Georectify Airborne Datacube-Convert Radiance Cube to Reflectance from Measured Reference Spectrum.bip.hdr')
>>> io = hsio()
>>> io.read_cube(fname_hdr)
>>> my_spec_mod = spec_mod(io.spyfile)

Use spec_mod.spectral_smooth to perform a Savitzky-Golay smoothing operation across the hyperspectral spectral signature.

>>> array_smooth, metadata_smooth = my_spec_mod.spectral_smooth(
        window_size=11, order=2)

Plot the spectra of an individual pixel to visualize the result of the smoothing procedure.

>>> import seaborn as sns
>>> spy_hs = my_spec_mod.spyfile.open_memmap()  # datacube before smoothing
>>> meta_bands = list(io.tools.meta_bands.values())
>>> meta_bands_smooth = sorted([float(i) for i in metadata_smooth['wavelength']])
>>> ax = sns.lineplot(x=meta_bands, y=spy_hs[200][800]*100, label='Before spectral smoothing', linewidth=3)
>>> ax = sns.lineplot(x=meta_bands_smooth, y=array_smooth[200][800]*100, label='After spectral smoothing', ax=ax)
>>> ax.set_xlabel('Wavelength (nm)', weight='bold')
>>> ax.set_ylabel('Reflectance (%)', weight='bold')
>>> ax.set_title(r'API Example: `spec_mod.spectral_smooth`', weight='bold')

hs_process.utilities module

class hs_process.utilities.defaults

Bases: object

Class containing default values and/or settings for various hs_process tools/functions.

crop_defaults

Default values for performing spatial cropping on images. crop_defaults is referenced by the spatial_mod.crop_single() function to get default values if various user-parameters are not passed or are left to None. In this way, defaults.crop_defaults can be modified once by the user to avoid having to pass the same parameter(s) repeatedly if executing spatial_mod.crop_single() many times, such as in a for loop.

crop_defaults.directory

File directory of the input image to be cropped (default: None).

Type

str

crop_defaults.name_short

Part of the datacube name that is generally not repeated across many datacubes captured at the same time. In the name_long example above, name_short = “plot_101_pika_gige_2”. The part of the filename that is name_short should end with a dash (but should not include that dash as it belongs to name_long; default: None).

Type

str

crop_defaults.name_long

Part of the datacube name that tends to be long and is repeated across many datacubes captured at the same time. This is an artifact of Resonon/Spectronon software, and may be desireable to shorten and/or make more informative. For example, a datacube may have the following name: “plot_101_pika_gige_2-Radiance From Raw Data-Georectify Airborne Datacube-Reflectance from Radiance Data and Measured Reference Spectrum.bip” and another datacube captured in the same campaign may be named: “plot_102_pika_gige_1-Radiance From Raw Data-Georectify Airborne Datacube-Reflectance from Radiance Data and Measured Reference Spectrum.bip” name_long should refer to everything after the first dash (including the first dash) up to the file extension (“.bip”): name_long = “-Radiance From Raw Data-Georectify Airborne Datacube-Reflectance from Radiance Data and Measured Reference Spectrum” (default: None).

Type

str

crop_defaults.ext

File extension to save the cropped image (default: ‘bip’).

Type

str

crop_defaults.pix_e_ul

upper left pixel column (easting) to begin cropping (default: 0).

Type

int

crop_defaults.pix_n_ul

upper left pixel row (northing) to begin cropping (default: 0).

Type

int

crop_defaults.buf_e_pix

The buffer distance in the easting direction (in pixel units) to be applied after calculating the original crop area (default: 0).

Type

int

crop_defaults.buf_n_pix

The buffer distance in the northing direction (in pixel units) to be applied after calculating the original crop area (default: 0).

Type

int

crop_defaults.buf_e_m

The buffer distance in the easting direction (in map units; e.g., meters) to be applied after calculating the original crop area; the buffer is considered after crop_X_m/crop_X_pix. A positive value will reduce the size of crop_X_m/crop_X_pix, and a negative value will increase it (default: None).

Type

float

crop_defaults.buf_n_m

The buffer distance in the northing direction (in map units; e.g., meters) to be applied after calculating the original crop area; the buffer is considered after crop_X_m/crop_X_pix. A positive value will reduce the size of crop_X_m/crop_X_pix, and a negative value will increase it (default: None).

Type

float

crop_defaults.crop_e_pix

number of pixels in each row in the cropped image (default: 90).

Type

int

crop_defaults.crop_n_pix

number of pixels in each column in the cropped image (default: 120).

Type

int

crop_defaults.crop_e_m

length of each row (easting direction) of the cropped image in map units (e.g., meters; default: None).

Type

float

crop_defaults.crop_n_m

length of each column (northing direction) of the cropped image in map units (e.g., meters; default: None).

Type

float

crop_defaults.plot_id_ref

the plot ID of the area to be cropped (default: None).

Type

int

envi_write

Attributes for writing ENVI datacubes to file, following the convention of the Spectral Python envi.save_image() parameter options for writing an ENVI datacube to file.

envi_write.dtype

The data type with which to store the image. For example, to store the image in 16-bit unsigned integer format, the argument could be any of numpy.uint16, 'u2', 'uint16', or 'H' (default: np.float32).

Type

numpy.dtype or str

envi_write.force

If hdr_file or its associated image file exist, force=True will overwrite the files; otherwise, an exception will be raised if either file exists (default: False).

Type

bool

envi_write.ext

The extension to use for saving the image file; if not specified, a default extension is determined based on the interleave. For example, if interleave``='bip', then ``ext is set to ‘bip’ as well. If ext is an empty string, the image file will have the same name as the .hdr, but without the ‘.hdr’ extension (default: None).

Type

str

envi_write.interleave

The band interleave format to use for writing the file; interleave should be one of ‘bil’, ‘bip’, or ‘bsq’ (default: ‘bip’).

Type

str

envi_write.byteorder

Specifies the byte order (endian-ness) of the data as written to disk. For little endian, this value should be either 0 or ‘little’. For big endian, it should be either 1 or ‘big’. If not specified, native byte order will be used (default: None).

Type

int or str

spat_crop_cols

Default column names for performing batch spatial cropping on images. Useful when batch processing images via batch.spatial_crop(). batch.spatial_crop() takes a parameter fname_sheet, which can be a filename to a spreadsheet or a pandas.DataFrame. defaults.spat_crop_cols should be modified if the column names in fname_sheet are different than what is expected (see documentation for batch.spatial_crop() to know the expected column names).

spat_crop_cols.directory

column name for input directory (default: ‘directory’).

Type

str

spat_crop_cols.fname

column name for input fname (default: ‘fname’).

Type

str

spat_crop_cols.name_short

column name for input image’s name_short (default: ‘name_short’).

Type

str

spat_crop_cols.name_long

column name for input image’s name_long (default: ‘name_long’).

Type

str

spat_crop_cols.ext

column name for file extension of input image (default: ‘ext’).

Type

str

spat_crop_cols.pix_e_ul

column name for pix_e_ul (default: ‘pix_e_ul’).

Type

str

spat_crop_cols.pix_n_ul

column name for pix_n_ul (default: ‘pix_n_ul’).

Type

str

spat_crop_cols.alley_size_e_pix

column name for alley_size_e_pix (default: ‘alley_size_e_pix’).

Type

str

spat_crop_cols.alley_size_n_pix

column name for alley_size_n_pix (default: ‘alley_size_n_pix’).

Type

str

spat_crop_cols.alley_size_e_m

column name for alley_size_e_m (default: ‘alley_size_e_m’).

Type

str

spat_crop_cols.alley_size_n_m

column name for alley_size_n_m (default: ‘alley_size_n_m’).

Type

str

spat_crop_cols.buf_e_pix

column name for buf_e_pix (default: ‘buf_e_pix’).

Type

str

spat_crop_cols.buf_n_pix

column name for buf_n_pix (default: ‘buf_n_pix’).

Type

str

spat_crop_cols.buf_e_m

column name for buf_e_m (default: ‘buf_e_m’).

Type

str

spat_crop_cols.buf_n_m

column name for buf_n_m (default: ‘buf_n_m’).

Type

str

spat_crop_cols.crop_e_pix

column name for crop_e_pix (default: ‘crop_e_pix’).

Type

str

spat_crop_cols.crop_n_pix

column name for crop_n_pix (default: ‘crop_n_pix’).

Type

str

spat_crop_cols.crop_e_m

column name for crop_e_m (default: ‘crop_e_m’).

Type

str

spat_crop_cols.crop_n_m

column name for crop_n_m (default: ‘crop_n_m’).

Type

str

spat_crop_cols.plot_id_ref

column name for plot_id (default: ‘crop_n_pix’).

Type

str

spat_crop_cols.n_plots_x

column name for n_plots_x (default: ‘n_plots_x’).

Type

str

spat_crop_cols.n_plots_y

column name for n_plots_y (default: ‘n_plots_y’).

Type

str

spat_crop_cols.n_plots

column name for n_plots (default: ‘n_plots’).

Type

str

class hs_process.utilities.hsio(fname_in=None, name_long=None, name_plot=None, name_short=None, str_plot='plot_', individual_plot=False, fname_hdr_spec=None)

Bases: object

Class for reading and writing hyperspectral data files, as well as accessing, interpreting, and modifying its associated metadata. With a hyperspectral data file loaded via hsio, there is simple functionality to display the datacube image as a multi-band render, as well as for saving a datacube as a 3-band geotiff. hsio relies heavily on the Spectral Python package.

read_cube(fname_hdr=None, overwrite=True, name_long=None, name_short=None, name_plot=None, individual_plot=False)

Reads in a hyperspectral datacube using the Spectral Python package.

Parameters

• fname_hdr (str) – filename of datacube to be read (default: None).

• overwrite (bool) – Whether to overwrite any of the previous user-passed variables, including name_long, name_plot, and name_short. If variables are already set and overwrite is False, they will remain the same. If variables are set and overwrite is True, they will be overwritten based on typcial file naming conventions of Resonon/Spectronon software. Any of the user-passed variables (e.g., name_long, etc.) will overwrite those that were set previously whether overwrite is True or False (default: False).

• name_long (str) – Spectronon processing appends processing names to the filenames; this indicates those processing names that are repetitive and can be deleted from the filename following processing (default: None).

• name_short (str) – The base name of the image file (see note above about name_long; default: None).

• name_plot (str) – numeric text that describes the plot number (default: None).

• individual_plot (bool) – Indicates whether image (and its filename) is for an individual plot (True), or for many plots (False; default: False).

Note

hs_process will search for name_long, name_plot, and name_short based on typical file naming behavior of Resonon/ Spectronon software. If any of these parameters are passed by the user, however, that will take precedence over “searching the typical file nameing behavior”.

Example

Load and initialize hsio

>>> from hs_process import hsio
>>> fname_hdr = r'F:\nigo0024\Documents\hs_process_demo\Wells_rep2_20180628_16h56m_pika_gige_7-Radiance Conversion-Georectify Airborne Datacube-Convert Radiance Cube to Reflectance from Measured Reference Spectrum.bip.hdr'
>>> io = hsio()  # initialize an instance of the hsio class (note there are no required parameters)

Load datacube using hsio.read_cube

>>> io.read_cube(fname_hdr)
>>> io.spyfile
Data Source:   'F:\nigo0024\Documents\hs_process_demo\Wells_rep2_20180628_16h56m_pika_gige_7-Radiance Conversion-Georectify Airborne Datacube-Convert Radiance Cube to Reflectance from Measured Reference Spectrum.bip'
    # Rows:            617
    # Samples:        1300
    # Bands:           240
    Interleave:        BIP
    Quantization:  32 bits
    Data format:   float32

Check name_long, name_short, and name_plot values derived from the filename

>>> io.name_long
'-Convert Radiance Cube to Reflectance from Measured Reference Spectrum'

>>> io.name_plot
'7'

>>> io.name_short
'Wells_rep2_20180628_16h56m_pika_gige_7'

read_spec(fname_hdr_spec, overwrite=True, name_long=None, name_short=None, name_plot=None)

Reads in a hyperspectral spectrum file using the using the Spectral Python package.

Parameters

fname_hdr_spec (str) – filename of spectra to be read.

Example

Load and initialize hsio

>>> from hs_process import hsio
>>> fname_hdr = r'F:\nigo0024\Documents\hs_process_demo\Wells_rep2_20180628_16h56m_pika_gige_7_plot_611-cube-to-spec-mean.spec.hdr'
>>> io = hsio()  # initialize an instance of the hsio class (note there are no required parameters)

Load datacube using hsio.read_spec

>>> io.read_spec(fname_hdr)
>>> io.spyfile_spec
Data Source:   'F:\nigo0024\Documents\hs_process_demo\Wells_rep2_20180628_16h56m_pika_gige_7_plot_611-cube-to-spec-mean.spec'
    # Rows:              1
    # Samples:           1
    # Bands:           240
    Interleave:        BIP
    Quantization:  32 bits
    Data format:   float32

Check name_long, name_short, and name_plot values derived from the filename

>>> io.name_long
'-cube-to-spec-mean'

>>> io.name_short
'Wells_rep2_20180628_16h56m_pika_gige_7_plot_611'

>>> io.name_plot
'611'

set_io_defaults(dtype=False, force=None, ext=False, interleave=False, byteorder=False)

Sets any of the ENVI file writing parameters to hsio; if any parameter is left unchanged from its default, it will remain as-is (i.e., it will not be set).

Parameters

• dtype (numpy.dtype or str) – The data type with which to store the image. For example, to store the image in 16-bit unsigned integer format, the argument could be any of numpy.uint16, ‘u2’, ‘uint16’, or ‘H’ (default=``False``).

• force (bool) – If hdr_file or its associated image file exist, force=True will overwrite the files; otherwise, an exception will be raised if either file exists (default=``None``).

• ext (str) – The extension to use for saving the image file; if not specified, a default extension is determined based on the interleave. For example, if interleave``='bip', then ``ext is set to ‘bip’ as well. If ext is an empty string, the image file will have the same name as the .hdr, but without the ‘.hdr’ extension (default: False).

• interleave (str) – The band interleave format to use for writing the file; interleave should be one of ‘bil’, ‘bip’, or ‘bsq’ (default=``False``).

• byteorder (int or str) – Specifies the byte order (endian-ness) of the data as written to disk. For little endian, this value should be either 0 or ‘little’. For big endian, it should be either 1 or ‘big’. If not specified, native byte order will be used (default=``False``).

Example

Load and initialize hsio

>>> from hs_process import hsio
>>> io = hsio()  # initialize an instance of the hsio class

Check defaults.envi_write

>>> io.defaults.envi_write
{'dtype': numpy.float32,
 'force': False,
 'ext': '',
 'interleave': 'bip',
 'byteorder': 0}

Modify force parameter and recheck defaults.envi_write

>>> io.set_io_defaults(force=True)
>>> io.defaults.envi_write
{'dtype': numpy.float32,
 'force': True,
 'ext': '',
 'interleave': 'bip',
 'byteorder': 0}

show_img(spyfile=None, band_r=120, band_g=76, band_b=32, vmin=None, vmax=None, cmap='viridis', cbar=True, inline=True)

Displays a datacube as a 3-band RGB image using Matplotlib.

Parameters

• spyfile (SpyFile object or numpy.ndarray) – The data cube to display; if None, loads from self.spyfile (default: None).

• band_r (int) – Band to display on the red channel (default: 120)

• band_g (int) – Band to display on the green channel (default: 76)

• band_b (int) – Band to display on the blue channel (default: 32)

• vmin/vmax (scalar, optional) – The data range that the colormap covers. By default, the colormap covers the complete value range of the supplied data (default: None).

• cmap (str) – The Colormap instance or registered colormap name used to map scalar data to colors. This parameter is ignored for RGB(A) data (default: “viridis”).

• cbar (bool) – Whether to include a colorbar in the image (default: True).

• inline (bool) – If True, displays in the IPython console; else displays in a pop-out window (default: True).

Note

The inline parameter points to the hsio.show_img function, and is only expected to work in an IPython console (not intended to be used in a normal Python console).

Example

Load hsio and spatial_mod modules

>>> from hs_process import hsio # load hsio
>>> from hs_process import spatial_mod # load spatial mod

Load the datacube using hsio.read_cube

>>> fname_hdr = r'F:\nigo0024\Documents\hs_process_demo\Wells_rep2_20180628_16h56m_pika_gige_7-Radiance Conversion-Georectify Airborne Datacube-Convert Radiance Cube to Reflectance from Measured Reference Spectrum.bip.hdr'
>>> io = hsio()  # initialize an instance of the hsio class
>>> io.read_cube(fname_hdr)

Perform simple spatial cropping via spatial_mod.crop_single

>>> my_spatial_mod = spatial_mod(io.spyfile)  # initialize spatial_mod instance to crop the image
>>> array_crop, metadata = my_spatial_mod.crop_single(pix_e_ul=250, pix_n_ul=100, crop_e_m=8, crop_n_m=3)

Show an RGB render of the cropped image using hsio.show_img

>>> io.show_img(array_crop)

write_cube(fname_hdr, spyfile, metadata=None, dtype=None, force=None, ext=None, interleave=None, byteorder=None)

Wrapper function that accesses the Spectral Python package to save a datacube to file.

Parameters

• fname_hdr (str) – Output header file path (with the ‘.hdr’ extension).

• spyfile (SpyFile object or numpy.ndarray) – The hyperspectral datacube to save. If numpy.ndarray, then metadata (dict) should also be passed.

• metadata (dict) – Metadata to write to the ENVI .hdr file describing the hyperspectral data cube being saved. If SpyFile object is passed to spyfile, metadata will overwrite any existing metadata stored by the SpyFile object (default=None).

• dtype (numpy.dtype or str) – The data type with which to store the image. For example, to store the image in 16-bit unsigned integer format, the argument could be any of numpy.uint16, ‘u2’, ‘uint16’, or ‘H’ (default=np.float32).

• force (bool) – If hdr_file or its associated image file exist, force=True will overwrite the files; otherwise, an exception will be raised if either file exists (default=False).

• ext (None or str) – The extension to use for saving the image file. If not specified or if set to an empty string (e.g., ext=''), a default extension is determined using the same name as fname_hdr, except without the “.hdr” extension. If fname_hdr is provided without the “non-.hdr” extension (e.g., “bip”), then the extension is determined from the interleave parameter. For example, if interleave``='bip', then ``ext is set to ‘bip’ as well. Use of ext is not recommended; instead, just set fname_hdr with the correct extension or use interleave to set the extension (default: None; determined from fname_hdr or interleave).

• interleave (str) – The band interleave format to use for writing the file; interleave should be one of ‘bil’, ‘bip’, or ‘bsq’ (default=’bip’).

• byteorder (int or str) – Specifies the byte order (endian-ness) of the data as written to disk. For little endian, this value should be either 0 or ‘little’. For big endian, it should be either 1 or ‘big’. If not specified, native byte order will be used (default=None).

Note

If dtype, force, ext, interleave, and byteorder are not passed, default values will be pulled from hsio.defaults. Thus, hsio.defaults can be modified prior to calling hsio.write_cube() to avoid having to pass each of thes parameters in the hsio.write_cube() function (see the hsio.set_io_defaults() function for support on setting these defaults and for more information on the parameters). Each of these parameters are passed directly to the Spectral Python envi.save_image() function. For more information, please refer to the Spectral Python documentation.

Example

Load hsio and spatial_mod modules

>>> import os
>>> import pathlib
>>> from hs_process import hsio  # load hsio
>>> from hs_process import spatial_mod  # load spatial mod
>>> data_dir = r'F:\nigo0024\Documents\hs_process_demo'
>>> fname_hdr_in = os.path.join(data_dir, 'Wells_rep2_20180628_16h56m_pika_gige_7-Radiance Conversion-Georectify Airborne Datacube-Convert Radiance Cube to Reflectance from Measured Reference Spectrum.bip.hdr')
>>> io = hsio()  # initialize the hsio class
>>> io.read_cube(fname_hdr_in)

Perform simple spatial cropping via spatial_mod.crop_single to generate a new datacube.

>>> my_spatial_mod = spatial_mod(io.spyfile)  # initialize spatial_mod instance to crop the image
>>> array_crop, metadata = my_spatial_mod.crop_single(pix_e_ul=250, pix_n_ul=100, crop_e_m=8, crop_n_m=3)

Save the datacube using hsio.write_cube

>>> fname_hdr = os.path.join(data_dir, 'hsio', 'Wells_rep2_20180628_16h56m_pika_gige_7-hsio-write-cube-cropped.bip.hdr')
>>> pathlib.Path(os.path.dirname(fname_hdr)).mkdir(parents=True, exist_ok=True)
>>> io.write_cube(fname_hdr, array_crop, metadata=metadata, force=True)

Load the datacube into Spectronon for visualization

write_spec(fname_hdr_spec, df_mean, df_std=None, metadata=None, dtype=None, force=None, ext=None, interleave=None, byteorder=None)

Wrapper function that accesses the Spectral Python package to save a single spectra to file.

Parameters

• fname_hdr_spec (str) – Output header file path (with the ‘.hdr’ extension). If the extension is explicitely specified in fname_hdr_spec and the ext parameter is also specified, fname_hdr_spec will be modified to conform to the extension set using the ext parameter.

• df_mean (pandas.Series or numpy.ndarray) – Mean spectra, stored as a df row, where columns are the bands.

• df_std (pandas.Series or numpy.ndarray, optional) – Standard deviation of each spectra, stored as a df row, where columns are the bands. This will be saved to the .hdr file.

• dtype (numpy.dtype or str) – The data type with which to store the image. For example, to store the image in 16-bit unsigned integer format, the argument could be any of numpy.uint16, ‘u2’, ‘uint16’, or ‘H’ (default=np.float32).

• force (bool) – If hdr_file or its associated image file exist, force=True will overwrite the files; otherwise, an exception will be raised if either file exists (default=False).

• ext (None or str) – The extension to use for saving the image file. If not specified or if set to an empty string (e.g., ext=''), a default extension is determined using the same name as fname_hdr_spec, except without the “.hdr” extension. If fname_hdr_spec is provided without the “non-.hdr” extension (e.g., “bip”), then the extension is determined from the interleave parameter. For example, if interleave``='bip', then ``ext is set to ‘bip’ as well. Use of ext is not recommended; instead, just set fname_hdr_spec with the correct extension or use interleave to set the extension (default: None; determined from fname_hdr_spec or interleave).

• interleave (str) – The band interleave format to use for writing the file; interleave should be one of ‘bil’, ‘bip’, or ‘bsq’ (default=’bip’).

• byteorder (int or str) – Specifies the byte order (endian-ness) of the data as written to disk. For little endian, this value should be either 0 or ‘little’. For big endian, it should be either 1 or ‘big’. If not specified, native byte order will be used (default=None).

• metadata (dict) – Metadata to write to the ENVI .hdr file describing the spectra being saved; if None, will try to pull metadata template from hsio.spyfile_spec.metadata or hsio.spyfile.metadata (default=None).

Example

Load and initialize hsio

>>> from hs_process import hsio # load hsio
>>> fname_hdr_in = r'F:\nigo0024\Documents\hs_process_demo\Wells_rep2_20180628_16h56m_pika_gige_7-Radiance Conversion-Georectify Airborne Datacube-Convert Radiance Cube to Reflectance from Measured Reference Spectrum.bip.hdr'
>>> io = hsio()  # initialize the hsio class (note there are no required parameters)
>>> io.read_cube(fname_hdr_in)

Calculate spectral mean via hstools.mean_datacube

>>> spec_mean, spec_std, _ = io.tools.mean_datacube(io.spyfile)
>>> fname_hdr_spec = r'F:\nigo0024\Documents\hs_process_demo\hsio\Wells_rep2_20180628_16h56m_pika_gige_7-mean.spec.hdr'

Save the new spectra to file via hsio.write_spec

>>> io.write_spec(fname_hdr_spec, spec_mean, spec_std)
Saving F:\nigo0024\Documents\hs_process_demo\hsio\Wells_rep2_20180628_16h56m_pika_gige_7-mean.spec

Open Wells_rep2_20180628_16h56m_pika_gige_7-mean.spec in Spectronon for visualization

write_tif(fname_tif, spyfile=None, metadata=None, fname_in=None, projection_out=None, geotransform_out=None, show_img=False)

Wrapper function that accesses the GDAL Python package to save a small datacube subset (i.e., three bands or less) to file.

Parameters

• fname_tif (str) – Output image file path (with the ‘.tif’ extension).

• spyfile (SpyFile object or numpy.ndarray, optional) – The data cube to save. If numpy.ndarray, then metadata (dict) should also be passed. If None, uses hsio.spyfile (default: None).

• metadata (dict) – Metadata information; if geotransform_out is not passed, “map info” is accessed from metadata and geotransform_out is created from that “map info”.

• fname_in (str, optional) – The filename of the image datacube to be read in initially. This is potentially useful if projection_out and/or geotransform_out are not passed and a numpy.ndarray is passed as the spyfile - in this case, write_tif() uses fname_in to load the fname_in datacube via GDAL, which can in turn be used to load the projection or geotransform information for the output geotiff (default: None).

• projection_out (str) – The GDAL projection to use while writing the geotiff. Applied using gdal.driver.dataset.SetProjection() (default: None; hsio.projection_out)

• geotransform_out (str) – The GDAL geotransform to use while writing the geotiff. Applied using gdal.driver.dataset.SetGeoTransform() (default: None; hsio.geotransform_out)

• show_img (bool or str) – Whether to display a render of the image being saved as a geotiff. Must be False (does not display the image), “inline” (displays the image inline using the IPython console), or “popout” (displays the image in a pop-out window; default: “inline”).

Example

Load and initialize hsio

>>> from hs_process import hsio  # load hsio
>>> fname_hdr_in = r'F:\nigo0024\Documents\hs_process_demo\Wells_rep2_20180628_16h56m_pika_gige_7-Radiance Conversion-Georectify Airborne Datacube-Convert Radiance Cube to Reflectance from Measured Reference Spectrum.bip.hdr'
>>> io = hsio()  # initialize the hsio class
>>> io.read_cube(fname_hdr_in)

Save an RGB render of the datacube to file via hsio.write_tif

>>> fname_tif = r'F:\nigo0024\Documents\hs_process_demo\hsio\Wells_rep2_20180628_16h56m_pika_gige_7.tif'
>>> io.write_tif(fname_tif, spyfile=io.spyfile, fname_in=fname_hdr_in)
Either `projection_out` is `None` or `geotransform_out` is `None` (or both are). Retrieving projection and geotransform information by loading `hsio.fname_in` via GDAL. Be sure this is appropriate for the data you are trying to write.
Clipping input data to the valid range for imshow with RGB data ([0..1] for floats or [0..255] for integers).

Open Wells_rep2_20180628_16h56m_pika_gige_7.tif in QGIS with the plot boundaries overlaid

class hs_process.utilities.hstools(spyfile)

Bases: object

Basic tools for manipulating Spyfiles and accessing their metadata.

Parameters

spyfile (SpyFile object) – The datacube being accessed and/or manipulated.

clean_md_sets(metadata=None)

Converts metadata items that are expressed as a list to be expressed as a dictionary.

Parameters

metadata (dict, optional) – Metadata dictionary to clean

Returns

metadata_out (dict) – Cleaned metadata dictionary.

Return type

dict

Example

Load and initialize hsio

>>> from hs_process import hsio
>>> fname_in = r'F:\nigo0024\Documents\hs_process_demo\Wells_rep2_20180628_16h56m_pika_gige_7-Radiance Conversion-Georectify Airborne Datacube-Convert Radiance Cube to Reflectance from Measured Reference Spectrum.bip.hdr'
>>> io = hsio(fname_in)

Create sample metadata with “wavelength” expressed as a list of strings

>>> metadata = {'samples': 1300,
                'lines': 617,
                'bands': 4,
                'file type': 'ENVI Standard',
                'wavelength': ['394.6', '396.6528', '398.7056',
                '400.7584']}

Clean metadata using hstools.clean_md_sets. Notice how wavelength is now expressed as a str representation of a dict, which is required for properly writing the metadata to the .hdr file via save_image() in Spectral Python.

>>> io.tools.clean_md_sets(metadata=metadata)
{'samples': 1300,
 'lines': 617,
 'bands': 4,
 'file type': 'ENVI Standard',
 'wavelength': '{394.6, 396.6528, 398.7056, 400.7584}'}

del_meta_item(metadata, key)

Deletes metadata item from SpyFile object.

Parameters

• metadata (dict) – dictionary of the metadata

• key (str) – dictionary key to delete

Returns

metadata (dict) – Dictionary containing the modified metadata.

Return type

dict

Example

Load and initialize hsio

>>> from hs_process import hsio
>>> fname_in = r'F:\nigo0024\Documents\hs_process_demo\Wells_rep2_20180628_16h56m_pika_gige_7-Radiance Conversion-Georectify Airborne Datacube-Convert Radiance Cube to Reflectance from Measured Reference Spectrum.bip.hdr'
>>> io = hsio(fname_in)

Create sample metadata

>>> metadata = {'samples': 1300,
                'lines': 617,
                'bands': 4,
                'file type': 'ENVI Standard',
                'map info': '{UTM, 1.0, 1.0, 421356.76707299997, 4844936.7317699995, 0.04, 0.04, 15, T, WGS-84, units  meters, rotation  0.000}',
                'wavelength': ['394.6', '396.6528', '398.7056',
                '400.7584']}

Delete “map info” from metadata using hstools.del_met_item

>>> io.tools.del_meta_item(metadata, 'map info')
{'samples': 1827,
 'lines': 617,
 'bands': 4,
 'file type': 'ENVI Standard',
 'wavelength': ['394.6', '396.6528', '398.7056', '400.7584']}

dir_data()

Retrieves the data directory from “site packages”.

get_UTM(pix_e_ul, pix_n_ul, utm_x, utm_y, size_x, size_y)

Calculates the new UTM coordinate of cropped plot to modify the “map info” tag of the .hdr file.

Parameters

• pix_e_ul (int) – upper left column (easting) where image cropping begins.

• pix_n_ul (int) – upper left row (northing) where image cropping begins.

• utm_x (float) – UTM easting coordinates (meters) of the original image (from the upper left).

• utm_y (float) – UTM northing coordinates (meters) of the original image (from the upper left).

• size_x (float) – Ground resolved distance of the image pixels in the x (easting) direction (meters).

• size_y (float) – Ground resolved distance of the image pixels in the y (northing) direction (meters).

Returns

2-element tuple containing

• utm_x_new (float): The modified UTM x coordinate (easting) of cropped plot.

• utm_y_new (float): The modified UTM y coordinate (northing) of cropped plot.

Example

Load and initialize hsio

>>> from hs_process import hsio
>>> fname_in = r'F:\nigo0024\Documents\hs_process_demo\Wells_rep2_20180628_16h56m_pika_gige_7-Radiance Conversion-Georectify Airborne Datacube-Convert Radiance Cube to Reflectance from Measured Reference Spectrum.bip.hdr'
>>> io = hsio(fname_in)

Retrieve UTM coordinates and pixel sizes from the metadata

>>> map_info_set = io.spyfile.metadata['map info']
>>> utm_x = io.tools.get_meta_set(map_info_set, 3)
>>> utm_y = io.tools.get_meta_set(map_info_set, 4)
>>> spy_ps_e = float(map_info_set[5])
>>> spy_ps_n = float(map_info_set[6])

Calculate the UTM coordinates at the 100th easting pixel and 50th northing pixel using hstools.get_UTM

>>> ul_x_utm, ul_y_utm = io.tools.get_UTM(100, 50,
                                          utm_x, utm_y,
                                          spy_ps_e,
                                          spy_ps_n)
>>> ul_x_utm
441360.80707299995
>>> ul_y_utm
4855934.691769999

get_band(target_wl, spyfile=None)

Finds the band number of the closest target wavelength.

Parameters

• target_wl (int or float) – the target wavelength to retrieve the band number for (required).

• spyfile (SpyFile object, optional) – The datacube being accessed and/or manipulated; if None, uses hstools.spyfile (default: None).

Returns

key_band (int) – band number of the closest target wavelength (target_wl).

Return type

int

Example

Load and initialize hsio

>>> from hs_process import hsio
>>> fname_in = r'F:\nigo0024\Documents\hs_process_demo\Wells_rep2_20180628_16h56m_pika_gige_7-Radiance Conversion-Georectify Airborne Datacube-Convert Radiance Cube to Reflectance from Measured Reference Spectrum.bip.hdr'
>>> io = hsio(fname_in)

Use hstools.get_band to find the band number corresponding to 703 nm

>>> io.tools.get_band(703, io.spyfile)
151

get_band_index(band_name)

. Returns the index of band from “band names”.

Parameters

band_name (int or list) – the target band to retrieve the band index for (required).

Returns

band_idx (int) – band index of the passed band name (band_name).

Return type

int or list

Example

Load and initialize hsio

>>> from hs_process import hsio
>>> fname_in = r'F:\nigo0024\Documents\hs_process_demo\Wells_rep2_20180628_16h56m_pika_gige_7-Radiance Conversion-Georectify Airborne Datacube-Convert Radiance Cube to Reflectance from Measured Reference Spectrum.bip.hdr'
>>> io = hsio(fname_in)

Using hstools.get_band_index, find the band index of bands 4, 43, and 111.

>>> io.tools.get_band_index([4, 43, 111])
[3, 42, 110]

get_band_num(band_idx)

Adds 1 to band_idx and returns the band number(s).

Parameters

band_idx (int or list) – the target band index(es) to retrive the band number for (required).

Returns

band_num (int or list): band number of the passed band index (band_idx).

Return type

int or list

Example

Load and initialize hsio

>>> from hs_process import hsio
>>> fname_in = r'F:\nigo0024\Documents\hs_process_demo\Wells_rep2_20180628_16h56m_pika_gige_7-Radiance Conversion-Georectify Airborne Datacube-Convert Radiance Cube to Reflectance from Measured Reference Spectrum.bip.hdr'
>>> io = hsio(fname_in)

Using hstools.get_band_num, find the band number located at the 4th, 43rd, and 111th index values.

>>> io.tools.get_band_num([4, 43, 111])
[5, 44, 112]

get_band_range(range_wl, index=True, spyfile=None)

Retrieves the band index or band name for all bands within a wavelength range.

Parameters

• range_wl (list) – the minimum and maximum wavelength to consider; values should be int or float.

• index (bool) – Indicates whether to return the band number (False; min=1) or to return index number (True; min=0) (default: True).

Returns

band_list (list): A list of all bands (either index or number, depending on how index is set) between a range in wavelength values.

Return type

list

Example

Load and initialize hsio

>>> from hs_process import hsio
>>> data_dir = r'F:\nigo0024\Documents\hs_process_demo'
>>> fname_in = os.path.join(data_dir, 'Wells_rep2_20180628_16h56m_pika_gige_7-Radiance Conversion-Georectify Airborne Datacube-Convert Radiance Cube to Reflectance from Measured Reference Spectrum.bip.hdr')
>>> io = hsio(fname_in)

Find the band name of all bands between 700 and 710 nm

>>> io.tools.get_band_range([700, 710], index=False, spyfile=io.spyfile)
[150, 151, 152, 153, 154]

Find the band index values of all bands between 700 and 710 nm via hstools.get_band_range

>>> io.tools.get_band_range([700, 710], index=True, spyfile=io.spyfile)
[149, 150, 151, 152, 153]

Sometimes “band names” are not integers in sequential order. To demonstrate the utility of the index parameter, let’s take a look at Sentinel 2A mimicked imagery.

>>> data_dir = r'F:\nigo0024\Documents\hs_process_demo\spec_mod'
>>> fname_in = os.path.join(data_dir, 'Wells_rep2_20180628_16h56m_pika_gige_7-mimic-s2a.bip.hdr')
>>> io = hsio(fname_in)

Find the band name of all bands between 760 and 840 nm

>>> io.tools.get_band_range([760, 840], index=False, spyfile=io.spyfile)
['S2A_SR_AV_B7', 'S2A_SR_AV_B8']

Find the band index values of all bands between 760 and 840 nm via hstools.get_band_range

>>> io.tools.get_band_range([760, 840], index=True, spyfile=io.spyfile)
[6, 7]

get_center_wl(wl_list, spyfile=None, wls=True)

Gets band numbers and mean wavelength from all wavelengths (or bands) in wl_list.

Parameters

• wl_list (list) – the list of bands to get information for (required).

• spyfile (SpyFile object) – The datacube being accessed and/or manipulated; if None, uses hstools.spyfile (default: None).

• wls (bool) – whether wavelengths are passed in wl_list or if bands are passed in wl_list (default: True - wavelenghts passed).

Returns

2-element tuple containing

• bands (list): the list of bands (band number) corresponding to wl_list.

• wls_mean (float): the mean wavelength from wl_list.

Example

Load and initialize hsio

>>> from hs_process import hsio
>>> fname_in = r'F:\nigo0024\Documents\hs_process_demo\Wells_rep2_20180628_16h56m_pika_gige_7-Radiance Conversion-Georectify Airborne Datacube-Convert Radiance Cube to Reflectance from Measured Reference Spectrum.bip.hdr'
>>> io = hsio(fname_in)

Using hstools.get_center_wl, find the bands and actual mean wavelength of the bands closest to 700 and 710 nm.

>>> bands, wls_mean = io.tools.get_center_wl([700, 710], wls=True)
>>> bands
[150, 155]
>>> wls_mean
705.5992

get_meta_set(meta_set, idx=None)

Reads metadata “set” (i.e., string representation of a Python set; common in .hdr files), taking care to remove leading and trailing spaces.

Parameters

• meta_set (str) – the string representation of the metadata set

• idx (int) – index to be read; if None, the whole list is returned (default: None).

Returns

metadata_list (list or str): List of metadata set items (as str), or if idx is not None, the item in the position described by idx.

Return type

list or str

Example

Load and initialize hsio

>>> from hs_process import hsio
>>> fname_in = r'F:\nigo0024\Documents\hs_process_demo\Wells_rep2_20180628_16h56m_pika_gige_7-Radiance Conversion-Georectify Airborne Datacube-Convert Radiance Cube to Reflectance from Measured Reference Spectrum.bip.hdr'
>>> io = hsio(fname_in)

Retrieve the “map info” set from the metadata via hstools.get_meta_set

>>> map_info_set = io.spyfile.metadata['map info']
['UTM',
 '1.0',
 '1.0',
 '441357.287073',
 '4855944.7717699995',
 '0.04',
 '0.04',
 '15',
 'T',
 'WGS-84',
 'units  meters',
 'rotation  0.000']

get_spectral_mean(band_list, spyfile=None)

Gets the spectral mean of a datacube from a list of bands.

Parameters

• band_list (list) – the list of bands to calculate the spectral mean for on the datacube (required).

• spyfile (SpyFile object or numpy.ndarray) – The datacube being accessed and/or manipulated; if None, uses hstools.spyfile (default: None).

Returns

array_mean (numpy.array or pandas.DataFrame): The mean reflectance from spyfile for the bands in band_list.

Return type

numpy.array or pandas.DataFrame

Example

Load and initialize hsio

>>> from hs_process import hsio
>>> fname_in = r'F:\nigo0024\Documents\hs_process_demo\Wells_rep2_20180628_16h56m_pika_gige_7-Radiance Conversion-Georectify Airborne Datacube-Convert Radiance Cube to Reflectance from Measured Reference Spectrum.bip.hdr'
>>> io = hsio(fname_in)

Calculate the spectral mean of the datacube via hstools.get_spectral_mean using all bands between 800 and 840 nm

>>> band_list = io.tools.get_band_range([800, 840], index=False)
>>> array_mean = io.tools.get_spectral_mean(band_list, spyfile=io.spyfile)
>>> io.show_img(array_mean)

get_wavelength(target_band, spyfile=None)

Returns actual wavelength of the closest target band.

Parameters

• target_band (int or float) – the target band to retrieve wavelength number for (required).

• spyfile (SpyFile object, optional) – The datacube being accessed and/or manipulated; if None, uses hstools.spyfile (default: None).

Returns

key_wavelength (float) – wavelength of the closest target band (target_band).

Return type

float

Example

Load and initialize hsio

>>> from hs_process import hsio
>>> fname_in = r'F:\nigo0024\Documents\hs_process_demo\Wells_rep2_20180628_16h56m_pika_gige_7-Radiance Conversion-Georectify Airborne Datacube-Convert Radiance Cube to Reflectance from Measured Reference Spectrum.bip.hdr'
>>> io = hsio(fname_in)

Use hstools.get_wavelength to find the wavelength value corresponding to the 151st band

>>> io.tools.get_wavelength(151, io.spyfile)
702.52

get_wavelength_range(range_bands, index=True, spyfile=None)

Retrieves the wavelengths for all bands within a band range.

Parameters

• range_bands (list) – the minimum and maximum band number to consider; values should be int.

• index (bool) – Indicates whether the bands in range_bands denote the band number (False; min=1) or the index number (True; min=0) (default: True).

Returns

wavelength_list (list): A list of all wavelengths between a range in band numbers or index values (depending how index is set).

Return type

list

Example

Load and initialize hsio

>>> from hs_process import hsio
>>> fname_hdr = r'F:\nigo0024\Documents\GitHub\hs_process\hs_process\data\Wells_rep2_20180628_16h56m_test_pika_gige_7-Convert Radiance Cube to Reflectance from Measured Reference Spectrum.bip.hdr'
>>> io = hsio(fname_hdr)

Find the wavelengths from the 16th to 21st bands

>>> io.tools.get_wavelength_range([16, 21], index=False, spyfile=io.spyfile)
[425.392, 427.4448, 429.4976, 431.5504, 433.6032, 435.656]

Find the wavelengths from the 16th to the 21st index

>>> io.tools.get_wavelength_range([16, 21], index=True, spyfile=io.spyfile)
[427.4448, 429.4976, 431.5504, 433.6032, 435.656, 437.7088]

load_spyfile(spyfile)

Loads a SpyFile (Spectral Python object) for data access and/or manipulation by the hstools class.

Parameters

spyfile (SpyFile object) – The datacube being accessed and/or manipulated.

Example

Load and initialize hsio

>>> from hs_process import hsio
>>> fname_in = r'F:\nigo0024\Documents\hs_process_demo\Wells_rep2_20180628_16h56m_pika_gige_7-Radiance Conversion-Georectify Airborne Datacube-Convert Radiance Cube to Reflectance from Measured Reference Spectrum.bip.hdr'
>>> io = hsio(fname_in)

Load a new datacube using hstools.load_spyfile

>>> io.tools.load_spyfile(io.spyfile)
>>> io.tools.spyfile
Data Source:   'F:\nigo0024\Documents\hs_process_demo\Wells_rep2_20180628_16h56m_pika_gige_7-Radiance Conversion-Georectify Airborne Datacube-Convert Radiance Cube to Reflectance from Measured Reference Spectrum.bip'
    # Rows:            617
    # Samples:        1300
    # Bands:           240
    Interleave:        BIP
    Quantization:  32 bits
    Data format:   float32

mask_array(array, metadata, thresh=None, percentile=None, side='lower')

Creates a masked numpy array based on a threshold value. If array is already a masked array, that mask is maintained and the new mask(s) is/ are added to the original mask.

Parameters

• array (numpy.ndarray) – The data array to mask.

• thresh (float or list) – The value for which to base the threshold; if thresh is list and side is None, then all values in thresh will be masked; if thresh is list and side is not None, then only the first value in the list will be considered for thresholding (default: None).

• percentile (float) – The percentile of pixels to mask; if percentile = 95 and side = ‘lower’, the lowest 95% of pixels will be masked prior to calculating the mean spectra across pixels (default: None; range: 0-100).

• side (str) – The side of the threshold for which to apply the mask. Must be either ‘lower’, ‘upper’, ‘outside’, or None; if ‘lower’, everything below the threshold will be masked; if ‘outside’, the thresh / percentile parameter must be list-like with two values indicating the lower and upper bounds - anything outside of these values will be masked out; if None, only the values that exactly match the threshold will be masked (default: ‘lower’).

Returns

2-element tuple containing

• array_mask (numpy.ndarray): The masked numpy.ndarray based on the passed threshold and/or percentile value.

• metadata (dict): The modified metadata.

Example

Load and initialize hsio

>>> from hs_process import hsio
>>> fname_in = r'F:\nigo0024\Documents\hs_process_demo\Wells_rep2_20180628_16h56m_pika_gige_7-Radiance Conversion-Georectify Airborne Datacube-Convert Radiance Cube to Reflectance from Measured Reference Spectrum.bip.hdr'
>>> io = hsio(fname_in)

Retrieve the image band at 800 nm using hstools.get_band and hsio.spyfile.open_memmap

>>> band = io.tools.get_band(800)
>>> array = io.spyfile.open_memmap()[:, :, band]

Create a masked array of all values below the 75th percentile via hstools.mask_array

>>> array_mask, metadata = io.tools.mask_array(array, io.spyfile.metadata, percentile=75, side='lower')

See that the “history” tage in the metadata has been modified

>>> metadata['history'][-158:]
"hs_process.mask_array[<label: 'thresh?' value:None; label: 'percentile?' value:75; label: 'side?' value:lower; label: 'unmasked_pct?' value:24.9935170178282>]"

Visualize the unmasked array using hsio.show_img. Set vmin and vmax to ensure the same color scale is used in comparing the masked vs. unmasked arrays.

>>> vmin = array.min()
>>> vmax = array.max()
>>> io.show_img(array, vmin=vmin, vmax=vmax)

Visualize the unmasked array using hsio.show_img

>>> io.show_img(array_mask, vmin=vmin, vmax=vmax)

mean_datacube(spyfile, mask=None, nodata=0)

Calculates the mean spectra for a datcube; if mask is passed (as a numpy.ndarray), then the mask is applied to spyfile prior to computing the mean spectra.

Parameters

• spyfile (SpyFile object or numpy.ndarray) – The hyperspectral datacube to mask.

• mask (numpy.ndarray) – the mask to apply to spyfile; if mask does not have similar dimensions to spyfile, the first band (i.e., first two dimensions) of mask will be repeated n times to match the number of bands of spyfile (default: None).

• nodata (float or None) – If None, treats all pixels cells as they are repressented in the numpy.ndarray. Otherwise, replaces nodata with np.nan and these cells will not be considered when calculating the mean spectra.

Returns

3-element tuple containing

• spec_mean (SpyFile.SpyFile object): The mean spectra from the input datacube.

• spec_std (SpyFile.SpyFile object): The standard deviation of the spectra from the input datacube.

• datacube_masked (numpy.ndarray): The masked numpy.ndarray; if mask is None, datacube_masked is identical to the SpyFile data array.

Example

Load and initialize hsio

>>> from hs_process import hsio
>>> fname_in = r'F:\nigo0024\Documents\hs_process_demo\Wells_rep2_20180628_16h56m_pika_gige_7-Radiance Conversion-Georectify Airborne Datacube-Convert Radiance Cube to Reflectance from Measured Reference Spectrum.bip.hdr'
>>> io = hsio(fname_in)

Retrieve the image band at 800 nm using hstools.get_band and hsio.spyfile.open_memmap, then mask out all pixels whose value falls below the 75th percentile.

>>> band = io.tools.get_band(800)
>>> array = io.spyfile.open_memmap()[:, :, band]
>>> array_mask, metadata = io.tools.mask_array(array, io.spyfile.metadata, percentile=75, side='lower')

Calculate the spectral mean from the remaining (i.e., unmasked) pixels using hstools.mean_datacube.

>>> spec_mean, spec_std, datacube_masked = io.tools.mean_datacube(io.spyfile, mask=array_mask)

Save using hsio.write_spec and hsio.write_cube, then load into Spectronon software for visualization.

>>> fname_hdr_spec = r'F:\nigo0024\Documents\hs_process_demo\hstools\Wells_rep2_20180628_16h56m_pika_gige_7-mean_800nm_75th.spec.hdr'
>>> fname_hdr_cube = r'F:\nigo0024\Documents\hs_process_demo\hstools\Wells_rep2_20180628_16h56m_pika_gige_7-mean_800nm_75th.bip.hdr'
>>> io.write_spec(fname_hdr_spec, spec_mean, spec_std, metadata=metadata, force=True)
Saving F:\nigo0024\Documents\hs_process_demo\hstools\Wells_rep2_20180628_16h56m_pika_gige_7-mean_800nm_75th.spec
>>> io.write_cube(fname_hdr_cube, datacube_masked, metadata=metadata, force=True)
Saving F:\nigo0024\Documents\hs_process_demo\hstools\Wells_rep2_20180628_16h56m_pika_gige_7-mean_800nm_75th.bip

modify_meta_set(meta_set, idx, value)

Modifies metadata “set” (i.e., string representation of a Python set; common in .hdr files) by converting string to list, then adjusts the value of an item by its index.

Parameters

• meta_set (str) – the string representation of the metadata set

• idx (int) – index to be modified; if None, the whole meta_set is returned (default: None).

• value (float, int, or str) – value to replace at idx

Returns

set_str (str): Modified metadata set string.

Return type

str

Example

Load and initialize hsio

>>> from hs_process import hsio
>>> fname_in = r'F:\nigo0024\Documents\hs_process_demo\Wells_rep2_20180628_16h56m_pika_gige_7-Radiance Conversion-Georectify Airborne Datacube-Convert Radiance Cube to Reflectance from Measured Reference Spectrum.bip.hdr'
>>> io = hsio(fname_in)

Retrieve the “map info” set from the metadata via hstools.get_meta_set

>>> map_info_set = io.spyfile.metadata['map info']
>>> map_info_set
['UTM',
 '1.0',
 '1.0',
 '441357.287073',
 '4855944.7717699995',
 '0.04',
 '0.04',
 '15',
 'T',
 'WGS-84',
 'units  meters',
 'rotation  0.000']

Modify the value at index position 4 from 4855944.7717699995 to 441300.2 using hstools.modify_meta_set.

>>> io.tools.modify_meta_set(map_info_set, idx=4, value=441300.2)
'{UTM, 1.0, 1.0, 441357.287073, 441300.2, 0.04, 0.04, 15, T, WGS-84, units  meters, rotation  0.000}'

plot_histogram(array, fname_fig=None, title=None, xlabel=None, percentile=90, bins=50, fontsize=16, color='#444444')

Plots a histogram with the percentile value labeled.

Parameters

• array (numpy.ndarray) – The data array used to create the histogram for; if array is masked, the masked pixels are excluded from the histogram.

• fname_fig (str, optional) – The filename to save the figure to; if None, the figure will not be saved (default: None).

• title (str, optional) – The plot title (default: None).

• xlabel (str, optional) – The x-axis label of the histogram (default: None).

• percentile (scalar, optional) – The percentile to label and illustrate on the histogram; if percentile = 90, the band/spectral index value at the 90th percentile will be labeled on the plot (default: 90; range: 0-100).

• bins (int, optional) – Number of histogram bins (default: 50).

• fontsize (scalar) – Font size of the axes labels. The title and text annotations will be scaled relatively (default: 16).

• color (str, optional) – Color of the histogram columns (default: “#444444”)

Returns

Figure object showing the histogram.

Return type

fig (matplotlib.figure)

Example

Load and initialize hsio

>>> from hs_process import hsio
>>> fname_in = r'F:\nigo0024\Documents\hs_process_demo\Wells_rep2_20180628_16h56m_pika_gige_7-Radiance Conversion-Georectify Airborne Datacube-Convert Radiance Cube to Reflectance from Measured Reference Spectrum.bip.hdr'
>>> io = hsio(fname_in)

Retrieve the image band at 800 nm using hstools.get_band and hsio.spyfile.open_memmap

>>> band = io.tools.get_band(800)
>>> array = io.spyfile.open_memmap()[:, :, band]

Create a masked array of all values below the 5th percentile via hstools.mask_array

>>> array_mask, metadata = io.tools.mask_array(array, io.spyfile.metadata, percentile=5, side='lower')

Visualize the histogram of the unmasked pixels (i.e., those greater than the 5th percentile) using hstools.plot_histogram

>>> title = 'Reflectance at 800 nm'
>>> xlabel = 'Reflectance (%)'
>>> fig = io.tools.plot_histogram(array_mask, title=title, xlabel=xlabel)

Module contents

hs_process is a Python package for processing and manipulating aerial hyperspectral imagery.

hs_process emphasizes the ability to batch process datacubes, with the overall goal of keeping the processing pipeline as “hands-off” as possible. There is also a focus of maintaining the ability to record some of the subjective aspects of image processing.

class hs_process.batch(base_dir=None, search_ext='.bip', dir_level=0, lock=None, progress_bar=False)

Bases: object

Class for batch processing hyperspectral image data. Makes use of segment, spatial_mod, and spec_mod to batch process many datacubes in a given directory. Supports options to save full datacubes, geotiff renders, as well as summary statistics and/or reports for the various tools.

Note

It may be a good idea to review and understand the defaults, hsio, hstools, segment, spatial_mod, and spec_mod classes prior to using the batch module.

cube_to_spectra(fname_list=None, base_dir=None, search_ext='bip', dir_level=0, base_dir_out=None, folder_name='cube_to_spec', name_append='cube-to-spec', write_geotiff=True, out_dtype=False, out_force=None, out_ext=False, out_interleave=False, out_byteorder=False)

Calculates the mean and standard deviation for each cube in fname_list and writes the result to a “.spec” file.

Parameters

• fname_list (list, optional) – list of filenames to process; if left to None, will look at base_dir, search_ext, and dir_level parameters for files to process (default: None).

• base_dir (str, optional) – directory path to search for files to spectrally clip; if fname_list is not None, base_dir will be ignored (default: None).

• search_ext (str) – file format/extension to search for in all directories and subdirectories to determine which files to process; if fname_list is not None, search_ext will be ignored (default: ‘bip’).

• dir_level (int) – The number of directory levels to search; if None, searches all directory levels (default: 0).

• base_dir_out (str) – directory path to save all processed spectra; if set to None, a folder named according to the folder_name parameter is added to base_dir

• folder_name (str) – folder to add to base_dir_out to save all the processed datacubes (default: ‘cube_to_spec’).

• name_append (str) – name to append to the filename (default: ‘cube-to-spec’).

• write_geotiff (bool) – whether to save the masked RGB image as a geotiff alongside the masked datacube.

• out_XXX – Settings for saving the output files can be adjusted here if desired. They are stored in batch.io.defaults, and are therefore accessible at a high level. See hsio.set_io_defaults() for more information on each of the settings.

Note

The following batch example builds on the API example results of the spatial_mod.crop_many_gdf function. Please complete the spatial_mod.crop_many_gdf example to be sure your directory (i.e., base_dir) is populated with multiple hyperspectral datacubes. The following example will be using datacubes located in the following directory: F:\nigo0024\Documents\hs_process_demo\spatial_mod\crop_many_gdf

Example

Load and initialize the batch module, checking to be sure the directory exists.

>>> import os
>>> from hs_process import batch
>>> data_dir = r'F:\nigo0024\Documents\hs_process_demo'
>>> base_dir = os.path.join(data_dir, 'spatial_mod', 'crop_many_gdf')
>>> print(os.path.isdir(base_dir))
>>> hsbatch = batch(base_dir, search_ext='.bip', progress_bar=True)  # searches for all files in ``base_dir`` with a ".bip" file extension
True

Use batch.cube_to_spectra to calculate the mean and standard deviation across all pixels for each of the datacubes in base_dir.

>>> hsbatch.cube_to_spectra(base_dir=base_dir, write_geotiff=False, out_force=True)
Processing file 39/40: 100%|██████████| 40/40 [00:03<00:00, 13.28it/s]------------------------------------------------| 0.0%

Use seaborn to visualize the spectra of plots 1011, 1012, and 1013. Notice how hsbatch.io.name_plot is utilized to retrieve the plot ID, and how the “history” tag is referenced from the metadata to determine the number of pixels whose reflectance was averaged to create the mean spectra. Also remember that pixels across the original input image likely represent a combination of soil, vegetation, and shadow.

>>> import seaborn as sns
>>> import re
>>> fname_list = [os.path.join(base_dir, 'cube_to_spec', 'Wells_rep2_20180628_16h56m_pika_gige_7_plot_1011-cube-to-spec-mean.spec'),
                  os.path.join(base_dir, 'cube_to_spec', 'Wells_rep2_20180628_16h56m_pika_gige_7_plot_1012-cube-to-spec-mean.spec'),
                  os.path.join(base_dir, 'cube_to_spec', 'Wells_rep2_20180628_16h56m_pika_gige_7_plot_1013-cube-to-spec-mean.spec')]
>>> ax = None
>>> for fname in fname_list:
>>>     hsbatch.io.read_spec(fname)
>>>     meta_bands = list(hsbatch.io.tools.meta_bands.values())
>>>     data = hsbatch.io.spyfile_spec.load().flatten() * 100
>>>     hist = hsbatch.io.spyfile_spec.metadata['history']
>>>     pix_n = re.search('<pixel number: (.*)>', hist).group(1)
>>>     if ax is None:
>>>         ax = sns.lineplot(x=meta_bands, y=data, label='Plot '+hsbatch.io.name_plot+' (n='+pix_n+')')
>>>     else:
>>>         ax = sns.lineplot(x=meta_bands, y=data, label='Plot '+hsbatch.io.name_plot+' (n='+pix_n+')', ax=ax)
>>> ax.set_xlabel('Wavelength (nm)', weight='bold')
>>> ax.set_ylabel('Reflectance (%)', weight='bold')
>>> ax.set_title(r'API Example: `batch.cube_to_spectra`', weight='bold')

segment_band_math(fname_list=None, base_dir=None, search_ext='bip', dir_level=0, base_dir_out=None, folder_name='band_math', name_append='band-math', write_geotiff=True, method='ndi', wl1=None, wl2=None, wl3=None, b1=None, b2=None, b3=None, list_range=True, plot_out=True, out_dtype=False, out_force=None, out_ext=False, out_interleave=False, out_byteorder=False)

Batch processing tool to perform band math on multiple datacubes in the same way. batch.segment_band_math is typically used prior to batch.segment_create_mask to generate the images/directory required for the masking process.

Parameters

• method (str) – Must be one of “ndi” (normalized difference index), “ratio” (simple ratio index), “derivative” (deriviative-type index), or “mcari2” (modified chlorophyll absorption index2). Indicates what kind of band math should be performed on the input datacube. The “ndi” method leverages segment.band_math_ndi(), the “ratio” method leverages segment.band_math_ratio(), and the “derivative” method leverages segment.band_math_derivative(). Please see the segment documentation for more information (default: “ndi”).

• wl1 (int, float, or list) – the wavelength (or set of wavelengths) to be used as the first parameter of the band math index; if list, then consolidates all bands between two wavelength values by calculating the mean pixel value across all bands in that range (default: None).

• wl2 (int, float, or list) – the wavelength (or set of wavelengths) to be used as the second parameter of the band math index; if list, then consolidates all bands between two wavelength values by calculating the mean pixel value across all bands in that range (default: None).

• b1 (int, float, or list) – the band (or set of bands) to be used as the first parameter of the band math index; if list, then consolidates all bands between two band values by calculating the mean pixel value across all bands in that range (default: None).

• b2 (int, float, or list) – the band (or set of bands) to be used as the second parameter of the band math index; if list, then consolidates all bands between two band values by calculating the mean pixel value across all bands in that range (default: None).

• list_range (bool) – Whether bands/wavelengths passed as a list is interpreted as a range of bands (True) or for each individual band in the list (False). If list_range is True, b1/wl1 and b2/wl2 should be lists with two items, and all bands/wavelegths between the two values will be used (default: True).

• plot_out (bool) – whether to save a histogram of the band math result (default: True).

• write_geotiff (bool) – whether to save the masked RGB image as a geotiff alongside the masked datacube.

Note

The following batch example builds on the API example results of the spatial_mod.crop_many_gdf function. Please complete the spatial_mod.crop_many_gdf example to be sure your directory (i.e., base_dir) is populated with multiple hyperspectral datacubes. The following example will be using datacubes located in the following directory: F:\nigo0024\Documents\hs_process_demo\spatial_mod\crop_many_gdf

Example

Load and initialize the batch module, checking to be sure the directory exists.

>>> import os
>>> from hs_process import batch
>>> base_dir = r'F:\nigo0024\Documents\hs_process_demo\spatial_mod\crop_many_gdf'
>>> print(os.path.isdir(base_dir))
True
>>> hsbatch = batch(base_dir, search_ext='.bip')  # searches for all files in ``base_dir`` with a ".bip" file extension

Use batch.segment_band_math to compute the MCARI2 (Modified Chlorophyll Absorption Ratio Index Improved; Haboudane et al., 2004) spectral index for each of the datacubes in base_dir. See Harris Geospatial for more information about the MCARI2 spectral index and references to other spectral indices.

>>> folder_name = 'band_math_mcari2-800-670-550'  # folder name can be modified to be more descriptive in what type of band math is being performed
>>> method = 'mcari2'  # must be one of "ndi", "ratio", "derivative", or "mcari2"
>>> wl1 = 800
>>> wl2 = 670
>>> wl3 = 550
>>> hsbatch.segment_band_math(base_dir=base_dir, folder_name=folder_name,
                              name_append='band-math', write_geotiff=True,
                              method=method, wl1=wl1, wl2=wl2, wl3=wl3,
                              plot_out=True, out_force=True)
Bands used (``b1``): [198]
Bands used (``b2``): [135]
Bands used (``b3``): [77]
Wavelengths used (``b1``): [799.0016]
Wavelengths used (``b2``): [669.6752]
Wavelengths used (``b3``): [550.6128]
Saving F:\nigo0024\Documents\hs_process_demo\spatial_mod\crop_many_gdfand_math_mcari2-800-670-550\Wells_rep2_20180628_16h56m_pika_gige_7_plot_1011-band-math-mcari2-800-670-550.bip
...

batch.segment_band_math creates a new folder in base_dir (in this case the new directory is F:\nigo0024\Documents\hs_process_demo\spatial_mod\crop_many_gdfand_math_mcari2-800-670-550) which contains several data products. The first is band-math-stats.csv: a spreadsheet containing summary statistics for each of the image cubes that were processed via batch.segment_band_math; stats include pixel count, mean, standard deviation, median, and percentiles across all image pixels.

Second is a geotiff file for each of the image cubes after the band math processing. This can be opened in QGIS to visualize in a spatial reference system, or can be opened using any software that supports floating point .tif files.

Third is the band math raster saved in the .hdr file format. Note that the data conained here should be the same as in the .tif file, so it’s a matter of preference as to what may be more useful. This single band .hdr can also be opend in QGIS.

Fourth is a histogram of the band math data contained in the image. The histogram illustrates the 90th percentile value, which may be useful in the segmentation step (e.g., see batch.segment_create_mask).

segment_composite_band(fname_list=None, base_dir=None, search_ext='bip', dir_level=0, base_dir_out=None, folder_name='composite_band', name_append='composite-band', write_geotiff=True, wl1=None, b1=None, list_range=True, plot_out=True, out_dtype=False, out_force=None, out_ext=False, out_interleave=False, out_byteorder=False)

Batch processing tool to create a composite band on multiple datacubes in the same way. batch.segment_composite_band is typically used prior to batch.segment_create_mask to generate the images/directory required for the masking process.

Parameters

• wl1 (int, float, or list) – the wavelength (or set of wavelengths) to be used as the first parameter of the band math index; if list, then consolidates all bands between two wavelength values by calculating the mean pixel value across all bands in that range (default: None).

• b1 (int, float, or list) – the band (or set of bands) to be used as the first parameter of the band math index; if list, then consolidates all bands between two band values by calculating the mean pixel value across all bands in that range (default: None).

• list_range (bool) – Whether bands/wavelengths passed as a list is interpreted as a range of bands (True) or for each individual band in the list (False). If list_range is True, b1/wl1 and b2/wl2 should be lists with two items, and all bands/wavelegths between the two values will be used (default: True).

• plot_out (bool) – whether to save a histogram of the band math result (default: True).

• write_geotiff (bool) – whether to save the masked RGB image as a geotiff alongside the masked datacube.

segment_create_mask(fname_list=None, base_dir=None, search_ext='bip', dir_level=0, mask_dir=None, base_dir_out=None, folder_name='mask', name_append='mask', write_datacube=True, write_spec=True, write_geotiff=True, mask_thresh=None, mask_percentile=None, mask_side='lower', out_dtype=False, out_force=None, out_ext=False, out_interleave=False, out_byteorder=False)

Batch processing tool to create a masked array on many datacubes. batch.segment_create_mask is typically used after batch.segment_band_math to mask all the datacubes in a directory based on the result of the band math process.

Parameters

• mask_thresh (float or int) – The value for which to mask the array; should be used with side parameter (default: None).

• mask_percentile (float or int) – The percentile of pixels to mask; if percentile``=95 and ``side``='lower', the lowest 95% of pixels will be masked following the band math operation (default: ``None; range: 0-100).

• mask_side (str) – The side of the threshold for which to apply the mask. Must be either ‘lower’, ‘upper’, ‘outside’, or None; if ‘lower’, everything below the threshold will be masked; if ‘outside’, the thresh / percentile parameter must be list-like with two values indicating the lower and upper bounds - anything outside of these values will be masked out; if None, only the values that exactly match the threshold will be masked (default: ‘lower’).

• geotiff (bool) – whether to save the masked RGB image as a geotiff alongside the masked datacube.

Note

The following batch example builds on the API example results of spatial_mod.crop_many_gdf and batch.segment_band_math. Please complete each of those API examples to be sure your directories (i.e., base_dir, and mask_dir) are populated with image files. The following example will be masking datacubes located in: F:\nigo0024\Documents\hs_process_demo\spatial_mod\crop_many_gdf based on MCARI2 images located in: F:\nigo0024\Documents\hs_process_demo\spatial_mod\crop_many_gdf\band_math_mcari2-800-670-550

Example

Load and initialize the batch module, ensuring base_dir is a valid directory

>>> import os
>>> from hs_process import batch
>>> base_dir = r'F:\nigo0024\Documents\hs_process_demo\spatial_mod\crop_many_gdf'
>>> print(os.path.isdir(base_dir))
True
>>> hsbatch = batch(base_dir, search_ext='.bip')  # searches for all files in ``base_dir`` with a ".bip" file extension

There must be a single-band image that will be used to determine which datacube pixels are to be masked (determined via the mask_dir parameter). Point to the directory that contains the MCARI2 images.

>>> mask_dir = os.path.join(base_dir, 'band_math_mcari2-800-670-550')
>>> print(os.path.isdir(mask_dir))
True

Indicate how the MCARI2 images should be used to determine which hyperspectal pixels are to be masked. The available parameters for controlling this are mask_thresh, mask_percentile, and mask_side. We will mask out all pixels that fall below the MCARI2 90th percentile.

>>> mask_percentile = 90
>>> mask_side = 'lower'

Finally, indicate the folder to save the masked datacubes and perform the batch masking via batch.segment_create_mask

>>> folder_name = 'mask_mcari2_90th'
>>> hsbatch.segment_create_mask(base_dir=base_dir, mask_dir=mask_dir,
                                folder_name=folder_name,
                                name_append='mask-mcari2-90th', write_geotiff=True,
                                mask_percentile=mask_percentile,
                                mask_side=mask_side)
Saving F:\nigo0024\Documents\hs_process_demo\spatial_mod\crop_many_gdf\mask_mcari2_90th\Wells_rep2_20180628_16h56m_pika_gige_7_plot_1011-mask-mcari2-90th.bip
Saving F:\nigo0024\Documents\hs_process_demo\spatial_mod\crop_many_gdf\mask_mcari2_90th\Wells_rep2_20180628_16h56m_pika_gige_7_plot_1011-mask-mcari2-90th-spec-mean.spec
...

batch.segment_create_mask creates a new folder in base_dir named according to the folder_name parameter (in this case the new directory is F:\nigo0024\Documents\hs_process_demo\spatial_mod\crop_many_gdf\mask_mcari2_90th) which contains several data products. The first is mask-stats.csv: a spreadsheet containing the band math threshold value for each image file. In this example, the MCARI2 value corresponding to the 90th percentile is listed.

fname	plot_id	lower-pctl-90
…	1011	0.83222
…	1012	0.81112
…	1013	0.74394

…etc.

Second is a geotiff file for each of the image cubes after the masking procedure. This can be opened in QGIS to visualize in a spatial reference system, or can be opened using any software that supports floating point .tif files. The masked pixels are saved as null values and should render transparently.

Third is the full hyperspectral datacube, also with the masked pixels saved as null values. Note that the only pixels remaining are the 10% with the highest MCARI2 values.

Fourth is the mean spectra across the unmasked datacube pixels. This is illustrated above by the green line plot (the light green shadow represents the standard deviation for each band).

spatial_crop(fname_sheet=None, base_dir=None, search_ext='bip', dir_level=0, base_dir_out=None, folder_name='spatial_crop', name_append='spatial-crop', write_geotiff=True, method='single', gdf=None, out_dtype=False, out_force=None, out_ext=False, out_interleave=False, out_byteorder=False)

Iterates through a spreadsheet that provides necessary information about how each image should be cropped and how it should be saved.

If gdf is passed (a geopandas.GoeDataFrame polygon file), the cropped images will be shifted to the center of appropriate ‘plot_id’ polygon.

Parameters

• fname_sheet (fname, pandas.DataFrame, or None, optional) – The filename of the spreadsheed that provides the necessary information for fine-tuning the batch process cropping. See below for more information about the required and optional contents of fname_sheet and how to properly format it. Optionally, fname_sheet can be a Pandas.DataFrame. If left to None, base_dir and gdf must be passed.

• base_dir (str, optional) – directory path to search for files to spatially crop; if fname_sheet is not None, base_dir will be ignored (default: None).

• base_dir_out (str, optional) – output directory of the cropped image (default: None).

• folder_name (str, optional) – folder to add to base_dir_out to save all the processed datacubes (default: ‘spatial_crop’).

• name_append (str, optional) – name to append to the filename (default: ‘spatial-crop’).

• write_geotiff (bool, optional) – whether to save an RGB image as a geotiff alongside the cropped datacube.

• method (str, optional) – Must be one of “single” or “many_gdf”. Indicates whether a single plot should be cropped from the input datacube or if many/multiple plots should be cropped from the input datacube. The “single” method leverages spatial_mod.crop_single() and the “many_gdf” method leverages spatial_mod.crop_many_gdf(). Please see the spatial_mod documentation for more information (default: “single”).

• gdf (geopandas.GeoDataFrame, optional) – the plot names and polygon geometery of each of the plots; ‘plot_id’ must be used as a column name to identify each of the plots, and should be an integer.

• out_XXX – Settings for saving the output files can be adjusted here if desired. They are stored in batch.io.defaults, and are therefore accessible at a high level. See hsio.set_io_defaults() for more information on each of the settings.

Tips and Tricks for fname_sheet when gdf is not passed

If gdf is not passed, fname_sheet may have the following required column headings that correspond to the relevant parameters in spatial_mod.crop_single() and spatial_mod.crop_many_gdf():

1. “directory”

2. “name_short”

3. “name_long”

4. “ext”

5. “pix_e_ul”

6. “pix_n_ul”.

With this minimum input, batch.spatial_crop will read in each image, crop from the upper left pixel (determined as pix_e_ul/pix_n_ul) to the lower right pixel calculated based on crop_e_pix/crop_n_pix (which is the width of the cropped area in units of pixels).

Note

crop_e_pix and crop_n_pix have default values (see defaults.crop_defaults()), but they can also be passed specifically for each datacube by including appropriate columns in fname_sheet (which takes precedence over defaults.crop_defaults).

fname_sheet may also have the following optional column headings:

1. “crop_e_pix”

2. “crop_n_pix”

3. “crop_e_m”

4. “crop_n_m”

5. “buf_e_pix”

6. “buf_n_pix”

7. “buf_e_m”

8. “buf_n_m”

9. “gdf_shft_e_m”

10. “gdf_shft_n_m”

11. “plot_id_ref”

12. “study”

13. “date”

More fname_sheet Tips and Tricks

1. These optional inputs passed via fname_sheet allow more control over exactly how the images are to be cropped. For a more detailed explanation of the information that many of these columns are intended to contain, see the documentation for spatial_mod.crop_single() and spatial_mod.crop_many_gdf(). Those parameters not referenced should be apparent in the API examples and tutorials.

2. If the column names are different in fname_sheet than described here, defaults.spat_crop_cols() can be modified to indicate which columns correspond to the relevant information.

3. The date and study columns do not impact how the datacubes are to be cropped, but if this information exists, batch.spatial_crop adds it to the filename of the cropped datacube. This can be used to avoid overwriting datacubes with similar names, and is especially useful when processing imagery from many dates and/or studies/locations and saving them in the same directory. If “study”, “date”, and “plot_id” are all passed, this information is used to formulate the output file name; e.g., study_wells_date_20180628_plot_527-spatial-crop.bip. If either “study” or “date” is missing, the populated variables wil be appended to the end of the hsio.name_short string; e.g., plot_9_3_pika_gige_1_plot_527-spatial-crop.bip.

4. Any other columns can be added to fname_sheet, but batch.spatial_crop() does not use them in any way.

Note

The following batch example only actually processes a single hyperspectral image. If more datacubes were present in base_dir, however, batch.spatial_crop would process all datacubes that were available.

Note

This example uses spatial_mod.crop_many_gdf to crop many plots from a datacube using a polygon geometry file describing the spatial extent of each plot.

Example

Load and initialize the batch module, checking to be sure the directory exists.

>>> import os
>>> import geopandas as gpd
>>> import pandas as pd
>>> from hs_process import batch
>>> base_dir = r'F:\nigo0024\Documents\hs_process_demo'
>>> print(os.path.isdir(base_dir))
>>> hsbatch = batch(base_dir, search_ext='.bip', dir_level=0,
                    progress_bar=True)  # searches for all files in ``base_dir`` with a ".bip" file extension
True

Load the plot geometry as a geopandas.GeoDataFrame

>>> fname_gdf = r'F:\nigo0024\Documents\hs_process_demo\plot_bounds.geojson'
>>> gdf = gpd.read_file(fname_gdf)

Perform the spatial cropping using the “many_gdf” method. Note that nothing is being passed to fname_sheet here, so batch.spatial_crop is simply going to attempt to crop all plots contained within gdf that overlap with any datacubes in base_dir.

Passing fname_sheet directly is definitely more flexible for customization. However, some customization is possible while not passing fname_sheet. In the example below, we set an easting and northing buffer, as well as limit the number of plots to crop to 40. These defaults trickle through to spatial_mod.crop_many_gdf(), so by setting them on the batch object, they will be recognized when calculating crop boundaries from gdf.

>>> hsbatch.io.defaults.crop_defaults.buf_e_m = 2  # Sets buffer in the easting direction (units of meters)
>>> hsbatch.io.defaults.crop_defaults.buf_n_m = 0.5
>>> hsbatch.io.defaults.crop_defaults.n_plots = 40  # We can limit the number of plots to process from gdf
>>> hsbatch.spatial_crop(base_dir=base_dir, method='many_gdf',
                         gdf=gdf, out_force=True)

Because fname_list was passed instead of fname_sheet, there is not a way to infer the study name and date. Therefore, “study” and “date” will be omitted from the output file name. If you would like output file names to include “study” and “date”, please pass fname_sheet with “study” and “date” columns.

Processing file 39/40: 100%|██████████| 40/40 [00:02<00:00, 17.47it/s]

A new folder was created in base_dir - F:\nigo0024\Documents\hs_process_demo\spatial_crop - that contains the cropped datacubes and the cropped geotiff images. The Plot ID from the gdf is used to name each datacube according to its plot ID. The geotiff images can be opened in QGIS to visualize the images after cropping them.

The cropped images were brightened in QGIS to emphasize the cropped boundaries. The plot boundaries are overlaid for reference (notice the 2.0 m buffer on the East/West ends and the 0.5 m buffer on the North/South sides).

spectra_combine(fname_list=None, base_dir=None, search_ext='bip', dir_level=0, base_dir_out=None, out_dtype=False, out_force=None, out_ext=False, out_interleave=False, out_byteorder=False)

Batch processing tool to gather all pixels from every image in a directory, compute the mean and standard deviation, and save as a single spectra (i.e., a spectra file is equivalent to a single spectral pixel with no spatial information).

Parameters

• fname_list (list, optional) – list of filenames to process; if left to None, will look at base_dir, search_ext, and dir_level parameters for files to process (default: None).

• base_dir (str, optional) – directory path to search for files to spectrally clip; if fname_list is not None, base_dir will be ignored (default: None).

• search_ext (str) – file format/extension to search for in all directories and subdirectories to determine which files to process; if fname_list is not None, search_ext will be ignored (default: ‘bip’).

• dir_level (int) – The number of directory levels to search; if None, searches all directory levels (default: 0).

• base_dir_out (str) – directory path to save all processed datacubes; if set to None, a folder named according to the folder_name parameter is added to base_dir (default: None).

• out_XXX – Settings for saving the output files can be adjusted here if desired. They are stored in batch.io.defaults, and are therefore accessible at a high level. See ``hsio.set_io_defaults() for more information on each of the settings.

Note

The following example will load in several small hyperspectral radiance datacubes (not reflectance) that were previously cropped manually (via Spectronon software). These datacubes represent the radiance values of grey reference panels that were placed in the field to provide data necessary for converting radiance imagery to reflectance. These particular datacubes were extracted from several different images captured within ~10 minutes of each other.

Example

Load and initialize the batch module, checking to be sure the directory exists.

>>> import os
>>> from hs_process import batch
>>> base_dir = r'F:\nigo0024\Documents\hs_process_demo\cube_ref_panels'
>>> print(os.path.isdir(base_dir))
True
>>> hsbatch = batch(base_dir)

Combine all the radiance datacubes in the directory via batch.spectra_combine.

>>> hsbatch.spectra_combine(base_dir=base_dir, search_ext='bip',
                            dir_level=0)
Combining datacubes/spectra into a single mean spectra.
Number of input datacubes/spectra: 7
Total number of pixels: 1516
Saving F:\nigo0024\Documents\hs_process_demo\cube_ref_panels\spec_mean_spy.spec

Visualize the combined spectra by opening in Spectronon. The solid line represents the mean radiance spectra across all pixels and images in base_dir, and the lighter, slightly transparent line represents the standard deviation of the radiance across all pixels and images in base_dir.

Notice the lower signal at the oxygen absorption region (near 770 nm). After converting datacubes to reflectance, it may be desireable to spectrally clip this region (see spec_mod.spectral_clip())

spectra_derivative(fname_list=None, base_dir=None, search_ext='spec', dir_level=0, base_dir_out=None, folder_name='spec_derivative', name_append='spec-derivative', order=1, out_dtype=False, out_force=None, out_ext=False, out_interleave=False, out_byteorder=False)

Batch processing tool to calculate the numeric spectral derivative for multiple spectra.

Parameters

• fname_list (list, optional) – list of filenames to process; if left to None, will look at base_dir, search_ext, and dir_level parameters for files to process (default: None).

• base_dir (str, optional) – directory path to search for files to process; if fname_list is not None, base_dir will be ignored (default: None).

• search_ext (str) – file format/extension to search for in all directories and subdirectories to determine which files to process; if fname_list is not None, search_ext will be ignored (default: ‘bip’).

• dir_level (int) – The number of directory levels to search; if None, searches all directory levels (default: 0).

• base_dir_out (str) – directory path to save all processed spectra; if set to None, a folder named according to the folder_name parameter is added to base_dir

• folder_name (str) – folder to add to base_dir_out to save all the processed datacubes (default: ‘spec_derivative’).

• name_append (str) – name to append to the filename (default: ‘spec-derivative’).

• order (int) – The order of the derivative (default: 1).

• out_XXX – Settings for saving the output files can be adjusted here if desired. They are stored in batch.io.defaults, and are therefore accessible at a high level. See ``hsio.set_io_defaults() for more information on each of the settings.

Note

The following batch example builds on the API example results of the batch.cube_to_spectra function. Please complete both the spatial_mod.crop_many_gdf and batch.cube_to_spectra examples to be sure your directory (i.e., base_dir) is populated with multiple hyperspectral spectra. The following example will be using spectra located in the following directory: F:\nigo0024\Documents\hs_process_demo\spatial_mod\crop_many_gdf\cube_to_spec

Example

Load and initialize the batch module, checking to be sure the directory exists.

>>> import os
>>> from hs_process import batch
>>> data_dir = r'F:\nigo0024\Documents\hs_process_demo'
>>> base_dir = os.path.join(data_dir, 'spatial_mod', 'crop_many_gdf', 'cube_to_spec')
>>> print(os.path.isdir(base_dir))
>>> hsbatch = batch(base_dir, search_ext='.spec', progress_bar=True)

Use batch.spectra_derivative to calculate the central finite difference (i.e., the numeric spectral derivative) for each of the .spec files in base_dir.

>>> order = 1
>>> hsbatch.spectra_derivative(base_dir=base_dir, order=order, out_force=True)

Use seaborn to visualize the derivative spectra of plots 1011, 1012, and 1013.

>>> import seaborn as sns
>>> import re
>>> fname_list = [os.path.join(base_dir, 'spec_derivative', 'Wells_rep2_20180628_16h56m_pika_gige_7_plot_1011-spec-derivative-order-{0}.spec'.format(order)),
                  os.path.join(base_dir, 'spec_derivative', 'Wells_rep2_20180628_16h56m_pika_gige_7_plot_1012-spec-derivative-order-{0}.spec'.format(order)),
                  os.path.join(base_dir, 'spec_derivative', 'Wells_rep2_20180628_16h56m_pika_gige_7_plot_1013-spec-derivative-order-{0}.spec'.format(order))]
>>> ax = None
>>> for fname in fname_list:
>>>     hsbatch.io.read_spec(fname)
>>>     meta_bands = list(hsbatch.io.tools.meta_bands.values())
>>>     data = hsbatch.io.spyfile_spec.open_memmap().flatten() * 100
>>>     hist = hsbatch.io.spyfile_spec.metadata['history']
>>>     pix_n = re.search('<pixel number: (?s)(.*)>] ->', hist).group(1)
>>>     if ax is None:
>>>         ax = sns.lineplot(meta_bands, 0, color='gray')
>>>         ax = sns.lineplot(x=meta_bands, y=data, label='Plot '+hsbatch.io.name_plot+' (n='+pix_n+')')
>>>     else:
>>>         ax = sns.lineplot(x=meta_bands, y=data, label='Plot '+hsbatch.io.name_plot+' (n='+pix_n+')', ax=ax)
>>> ax.set(ylim=(-1, 1))
>>> ax.set_xlabel('Wavelength (nm)', weight='bold')
>>> ax.set_ylabel('Derivative reflectance (%)', weight='bold')
>>> ax.set_title(r'API Example: `batch.spectra_derivative`', weight='bold')

spectra_to_csv(fname_list=None, base_dir=None, search_ext='spec', dir_level=0, base_dir_out=None, name='stats-spectra', multithread=False)

Reads all the .spec files in a direcory and saves their reflectance information to a .csv. batch.spectra_to_csv is identical to batch.spectra_to_df except a .csv file is saved rather than returning a pandas.DataFrame.

Parameters

• fname_list (list, optional) – list of filenames to process; if left to None, will look at base_dir, search_ext, and dir_level parameters for files to process (default: None).

• base_dir (str, optional) – directory path to search for files to spectrally clip; if fname_list is not None, base_dir will be ignored (default: None).

• search_ext (str) – file format/extension to search for in all directories and subdirectories to determine which files to process; if fname_list is not None, search_ext will be ignored (default: ‘bip’).

• dir_level (int) – The number of directory levels to search; if None, searches all directory levels (default: 0).

• base_dir_out (str) – directory path to save all processed datacubes; if set to None, file is saved to base_dir

• name (str) – The output filename (default: “stats-spectra”).

• multithread (bool) – Whether to leverage multi-thread processing when reading the .spec files. Setting to True should speed up the time it takes to read all .spec files.

Note

The following example builds on the API example results of batch.segment_band_math() and batch.segment_create_mask()_. Please complete each of those API examples to be sure your directory (i.e., ``F:nigo0024Documentshs_process_demospatial_modcrop_many_gdfmask_mcari2_90th`) is populated with image files.

Example

Load and initialize the batch module, checking to be sure the directory exists.

>>> import os
>>> from hs_process import batch
>>> base_dir = r'F:\nigo0024\Documents\hs_process_demo\spatial_mod\crop_many_gdf\mask_mcari2_90th'
>>> print(os.path.isdir(base_dir))
True
>>> hsbatch = batch(base_dir)

Read all the .spec files in base_dir and save them to a .csv file.

>>> hsbatch.spectra_to_csv(base_dir=base_dir, search_ext='spec',
                           dir_level=0)
Writing mean spectra to a .csv file.
Number of input datacubes/spectra: 40
Output file location: F:\nigo0024\Documents\hs_process_demo\spatial_mod\crop_many_gdf\mask_mcari2_90th\stats-spectra.csv

When stats-spectra.csv is opened in Microsoft Excel, we can see that each row is a .spec file from a different plot, and each column is a particular spectral band/wavelength.

spectra_to_df(fname_list=None, base_dir=None, search_ext='spec', dir_level=0, multithread=False)

Reads all the .spec files in a direcory and returns their data as a pandas.DataFrame object. batch.spectra_to_df is identical to batch.spectra_to_csv except a pandas.DataFrame is returned rather than saving a .csv file.

Parameters

• fname_list (list, optional) – list of filenames to process; if left to None, will look at base_dir, search_ext, and dir_level parameters for files to process (default: None).

• base_dir (str, optional) – directory path to search for files to spectrally clip; if fname_list is not None, base_dir will be ignored (default: None).

• search_ext (str) – file format/extension to search for in all directories and subdirectories to determine which files to process; if fname_list is not None, search_ext will be ignored (default: ‘bip’).

• dir_level (int) – The number of directory levels to search; if None, searches all directory levels (default: 0).

• multithread (bool) – Whether to leverage multi-thread processing when reading the .spec files. Setting to True should speed up the time it takes to read all .spec files.

Note

The following example builds on the API example results of batch.segment_band_math() and batch.segment_create_mask()_. Please complete each of those API examples to be sure your directory (i.e., ``F:nigo0024Documentshs_process_demospatial_modcrop_many_gdfmask_mcari2_90th`) is populated with image files.

Example

Load and initialize the batch module, checking to be sure the directory exists.

>>> import os
>>> from hs_process import batch
>>> base_dir = r'F:\nigo0024\Documents\hs_process_demo\spatial_mod\crop_many_gdf\mask_mcari2_90th'
>>> print(os.path.isdir(base_dir))
True
>>> hsbatch = batch(base_dir)

Read all the .spec files in base_dir and load them to df_spec, a pandas.DataFrame.

>>> df_spec = hsbatch.spectra_to_df(base_dir=base_dir, search_ext='spec',
                                    dir_level=0)
Writing mean spectra to a ``pandas.DataFrame``.
Number of input datacubes/spectra: 40

When visualizing df_spe in Spyder, we can see that each row is a .spec file from a different plot, and each column is a particular spectral band.

It is somewhat confusing to conceptualize spectral data by band number (as opposed to the wavelenth it represents). hs_process.hs_tools.get_band can be used to retrieve spectral data for all plots via indexing by wavelength. Say we need to access reflectance at 710 nm for each plot.

>>> df_710nm = df_spec[['fname', 'plot_id', hsbatch.io.tools.get_band(710)]]

spectral_clip(fname_list=None, base_dir=None, search_ext='bip', dir_level=0, base_dir_out=None, folder_name='spec_clip', name_append='spec-clip', wl_bands=[[0, 420], [760, 776], [813, 827], [880, 1000]], out_dtype=False, out_force=None, out_ext=False, out_interleave=False, out_byteorder=False)

Batch processing tool to spectrally clip multiple datacubes in the same way.

Parameters

• fname_list (list, optional) – list of filenames to process; if left to None, will look at base_dir, search_ext, and dir_level parameters for files to process (default: None).

• base_dir (str, optional) – directory path to search for files to spectrally clip; if fname_list is not None, base_dir will be ignored (default: None).

• search_ext (str) – file format/extension to search for in all directories and subdirectories to determine which files to process; if fname_list is not None, search_ext will be ignored (default: ‘bip’).

• dir_level (int) – The number of directory levels to search; if None, searches all directory levels (default: 0).

• base_dir_out (str) – directory path to save all processed datacubes; if set to None, a folder named according to the folder_name parameter is added to base_dir

• folder_name (str) – folder to add to base_dir_out to save all the processed datacubes (default: ‘spec-clip’).

• name_append (str) – name to append to the filename (default: ‘spec-clip’).

• wl_bands (list or list of lists) – minimum and maximum wavelenths to clip from image; if multiple groups of wavelengths should be cut, this should be a list of lists. For example, wl_bands=[760, 776] will clip all bands greater than 760.0 nm and less than 776.0 nm; wl_bands = [[0, 420], [760, 776], [813, 827], [880, 1000]] will clip all band less than 420.0 nm, bands greater than 760.0 nm and less than 776.0 nm, bands greater than 813.0 nm and less than 827.0 nm, and bands greater than 880 nm (default).

• out_XXX – Settings for saving the output files can be adjusted here if desired. They are stored in batch.io.defaults, and are therefore accessible at a high level. See ``hsio.set_io_defaults() for more information on each of the settings.

Note

The following batch example builds on the API example results of the batch.spatial_crop function. Please complete the batch.spatial_crop example to be sure your directory (i.e., base_dir) is populated with multiple hyperspectral datacubes. The following example will be using datacubes located in the following directory: F:\nigo0024\Documents\hs_process_demo\spatial_crop

Example

Load and initialize the batch module, checking to be sure the directory exists.

>>> import os
>>> from hs_process import batch
>>> base_dir = r'F:\nigo0024\Documents\hs_process_demo\spatial_crop'
>>> print(os.path.isdir(base_dir))
True
>>> hsbatch = batch(base_dir, search_ext='.bip', progress_bar=True)  # searches for all files in ``base_dir`` with a ".bip" file extension

Use batch.spectral_clip to clip all spectral bands below 420 nm and above 880 nm, as well as the bands near the oxygen absorption (i.e., 760-776 nm) and water absorption (i.e., 813-827 nm) regions.

>>> hsbatch.spectral_clip(base_dir=base_dir, folder_name='spec_clip',
                          wl_bands=[[0, 420], [760, 776], [813, 827], [880, 1000]],
                          out_force=True)
Processing 40 files. If this is not what is expected, please check if files have already undergone processing. If existing files should be overwritten, be sure to set the ``out_force`` parameter.
Processing file 39/40: 100%|██████████| 40/40 [00:01<00:00, 26.68it/s]

Use seaborn to visualize the spectra of a single pixel in one of the processed images.

>>> import seaborn as sns
>>> fname = os.path.join(base_dir, 'Wells_rep2_20180628_16h56m_pika_gige_7_1011-spatial-crop.bip')
>>> hsbatch.io.read_cube(fname)
>>> spy_mem = hsbatch.io.spyfile.open_memmap()  # datacube before clipping
>>> meta_bands = list(hsbatch.io.tools.meta_bands.values())
>>> fname = os.path.join(base_dir, 'spec_clip', 'Wells_rep2_20180628_16h56m_pika_gige_7_1011-spec-clip.bip')
>>> hsbatch.io.read_cube(fname)
>>> spy_mem_clip = hsbatch.io.spyfile.open_memmap()  # datacube after clipping
>>> meta_bands_clip = list(hsbatch.io.tools.meta_bands.values())
>>> ax = sns.lineplot(x=meta_bands, y=spy_mem[26][29], label='Before spectral clipping', linewidth=3)
>>> ax = sns.lineplot(x=meta_bands_clip, y=spy_mem_clip[26][29], label='After spectral clipping', ax=ax)
>>> ax.set_xlabel('Wavelength (nm)', weight='bold')
>>> ax.set_ylabel('Reflectance (%)', weight='bold')
>>> ax.set_title(r'API Example: `batch.spectral_clip`', weight='bold')

Notice the spectral areas that were clipped, namely the oxygen and water absorption regions (~770 and ~820 nm, respectively). There is perhaps a lower signal:noise ratio in these regions, which was the merit for clipping those bands out.

spectral_mimic(fname_list=None, base_dir=None, search_ext='bip', dir_level=0, base_dir_out=None, folder_name='spec_mimic', name_append='spec-mimic', sensor='sentinel-2a', df_band_response=None, col_wl='wl_nm', center_wl='peak', out_dtype=False, out_force=None, out_ext=False, out_interleave=False, out_byteorder=False)

Batch processing tool to spectrally mimic a multispectral sensor for multiple datacubes in the same way.

Parameters

• fname_list (list, optional) – list of filenames to process; if left to None, will look at base_dir, search_ext, and dir_level parameters for files to process (default: None).

• base_dir (str, optional) – directory path to search for files to spectrally resample; if fname_list is not None, base_dir will be ignored (default: None).

• search_ext (str) – file format/extension to search for in all directories and subdirectories to determine which files to process; if fname_list is not None, search_ext will be ignored (default: ‘bip’).

• dir_level (int) – The number of directory levels to search; if None, searches all directory levels (default: 0).

• base_dir_out (str) – directory path to save all processed datacubes; if set to None, a folder named according to the folder_name parameter is added to base_dir

• folder_name (str) – folder to add to base_dir_out to save all the processed datacubes (default: ‘spec_bin’).

• name_append (str) – name to append to the filename (default: ‘spec-bin’).

• sensor (str) – Should be one of [“sentera_6x”, “micasense_rededge_3”, “sentinel-2a”, “sentinel-2b”, “custom”]; if “custom”, df_band_response and col_wl must be passed.

• df_band_response (pd.DataFrame) – A DataFrame that contains the transmissivity (%) for each sensor band (as columns) mapped to the continuous wavelength values (as rows). Required if sensor is “custom”, ignored otherwise.

• col_wl (str) – The column of df_band_response denoting the wavlengths (default: ‘wl_nm’).

• center_wl (str) – Indicates how the center wavelength of each band is determined. If center_wl is “peak”, the point at which transmissivity is at its maximum is used as the center wavelength. If center_wl is “weighted”, the weighted average is used to compute the center wavelength. Must be one of [“peak”, “weighted”] (default: "peak").

• out_XXX – Settings for saving the output files can be adjusted here if desired. They are stored in batch.io.defaults, and are therefore accessible at a high level. See ``hsio.set_io_defaults() for more information on each of the settings.

Note

The following batch example builds on the API example results of the batch.spatial_crop function. Please complete the batch.spatial_crop example to be sure your directory (i.e., base_dir) is populated with multiple hyperspectral datacubes. The following example will be using datacubes located in the following directory: F:\nigo0024\Documents\hs_process_demo\spatial_crop

Example

Load and initialize the batch module, checking to be sure the directory exists.

>>> import os
>>> from hs_process import batch
>>> base_dir = r'F:\nigo0024\Documents\hs_process_demo\spatial_crop'
>>> print(os.path.isdir(base_dir))
True
>>> hsbatch = batch(base_dir, search_ext='.bip', progress_bar=True)  # searches for all files in ``base_dir`` with a ".bip" file extension

Use batch.spectral_mimic to spectrally mimic the Sentinel-2A multispectral satellite sensor.

>>> hsbatch.spectral_mimic(
    base_dir=base_dir, folder_name='spec_mimic',
    name_append='sentinel-2a',
    sensor='sentinel-2a', center_wl='weighted')
Processing 40 files. If existing files should be overwritten, be sure to set the ``out_force`` parameter.
Processing file 39/40: 100%|██████████| 40/40 [00:04<00:00,  8.85it/s]

Use seaborn to visualize the spectra of a single pixel in one of the processed images.

>>> import seaborn as sns
>>> fname = os.path.join(base_dir, 'Wells_rep2_20180628_16h56m_pika_gige_7_1011-spatial-crop.bip')
>>> hsbatch.io.read_cube(fname)
>>> spy_mem = hsbatch.io.spyfile.open_memmap()  # datacube before mimicking
>>> meta_bands = list(hsbatch.io.tools.meta_bands.values())
>>> fname = os.path.join(base_dir, 'spec_mimic', 'Wells_rep2_20180628_16h56m_pika_gige_7_1011-sentinel-2a.bip')
>>> hsbatch.io.read_cube(fname)
>>> spy_mem_sen2a = hsbatch.io.spyfile.open_memmap()  # datacube after mimicking
>>> meta_bands_sen2a = list(hsbatch.io.tools.meta_bands.values())
>>> ax = sns.lineplot(x=meta_bands, y=spy_mem[26][29], label='Hyperspectral (Pika II)', linewidth=3)
>>> ax = sns.lineplot(x=meta_bands_sen2a, y=spy_mem_sen2a[26][29], label='Sentinel-2A "mimic"', marker='o', ms=6, ax=ax)
>>> ax.set_xlabel('Wavelength (nm)', weight='bold')
>>> ax.set_ylabel('Reflectance (%)', weight='bold')
>>> ax.set_title(r'API Example: `batch.spectral_mimic`', weight='bold')

spectral_resample(fname_list=None, base_dir=None, search_ext='bip', dir_level=0, base_dir_out=None, folder_name='spec_bin', name_append='spec-bin', bandwidth=None, bins_n=None, out_dtype=False, out_force=None, out_ext=False, out_interleave=False, out_byteorder=False)

Batch processing tool to spectrally resample (a.k.a. “bin”) multiple datacubes in the same way.

Parameters

• fname_list (list, optional) – list of filenames to process; if left to None, will look at base_dir, search_ext, and dir_level parameters for files to process (default: None).

• base_dir (str, optional) – directory path to search for files to spectrally resample; if fname_list is not None, base_dir will be ignored (default: None).

• search_ext (str) – file format/extension to search for in all directories and subdirectories to determine which files to process; if fname_list is not None, search_ext will be ignored (default: ‘bip’).

• dir_level (int) – The number of directory levels to search; if None, searches all directory levels (default: 0).

• base_dir_out (str) – directory path to save all processed datacubes; if set to None, a folder named according to the folder_name parameter is added to base_dir

• folder_name (str) – folder to add to base_dir_out to save all the processed datacubes (default: ‘spec_bin’).

• name_append (str) – name to append to the filename (default: ‘spec-bin’).

• bandwidth (float or int) – The bandwidth of the bands after spectral resampling is complete (units should be consistent with that of the .hdr file). Setting bandwidth to 10 will consolidate bands that fall within every 10 nm interval.

• bins_n (int) – The number of bins (i.e., “bands”) to achieve after spectral resampling is complete. Ignored if bandwidth is not None.

• out_XXX – Settings for saving the output files can be adjusted here if desired. They are stored in batch.io.defaults, and are therefore accessible at a high level. See ``hsio.set_io_defaults() for more information on each of the settings.

Note

The following batch example builds on the API example results of the batch.spatial_crop function. Please complete the batch.spatial_crop example to be sure your directory (i.e., base_dir) is populated with multiple hyperspectral datacubes. The following example will be using datacubes located in the following directory: F:\nigo0024\Documents\hs_process_demo\spatial_crop

Example

Load and initialize the batch module, checking to be sure the directory exists.

>>> import os
>>> from hs_process import batch
>>> base_dir = r'F:\nigo0024\Documents\hs_process_demo\spatial_crop'
>>> print(os.path.isdir(base_dir))
True
>>> hsbatch = batch(base_dir, search_ext='.bip', progress_bar=True)  # searches for all files in ``base_dir`` with a ".bip" file extension

Use batch.spectral_resample to bin (“group”) all spectral bands into 20 nm bandwidth bands (from ~2.3 nm bandwidth originally) on a per-pixel basis.

>>> hsbatch.spectral_resample(
    base_dir=base_dir, folder_name='spec_bin',
    name_append='spec-bin-20', bandwidth=20)
Processing 40 files. If existing files should be overwritten, be sure to set the ``out_force`` parameter.
Processing file 39/40: 100%|██████████| 40/40 [00:00<00:00, 48.31it/s]
...

Use seaborn to visualize the spectra of a single pixel in one of the processed images.

>>> import seaborn as sns
>>> fname = os.path.join(base_dir, 'Wells_rep2_20180628_16h56m_pika_gige_7_1011-spatial-crop.bip')
>>> hsbatch.io.read_cube(fname)
>>> spy_mem = hsbatch.io.spyfile.open_memmap()  # datacube before resampling
>>> meta_bands = list(hsbatch.io.tools.meta_bands.values())
>>> fname = os.path.join(base_dir, 'spec_bin', 'Wells_rep2_20180628_16h56m_pika_gige_7_1011-spec-bin-20.bip')
>>> hsbatch.io.read_cube(fname)
>>> spy_mem_bin = hsbatch.io.spyfile.open_memmap()  # datacube after resampling
>>> meta_bands_bin = list(hsbatch.io.tools.meta_bands.values())
>>> ax = sns.lineplot(x=meta_bands, y=spy_mem[26][29], label='Hyperspectral (Pika II)', linewidth=3)
>>> ax = sns.lineplot(x=meta_bands_bin, y=spy_mem_bin[26][29], label='Spectral resample (20 nm)', marker='o', ms=6, ax=ax)
>>> ax.set_xlabel('Wavelength (nm)', weight='bold')
>>> ax.set_ylabel('Reflectance (%)', weight='bold')
>>> ax.set_title(r'API Example: `batch.spectral_resample`', weight='bold')

spectral_smooth(fname_list=None, base_dir=None, search_ext='bip', dir_level=0, base_dir_out=None, folder_name='spec_smooth', name_append='spec-smooth', window_size=11, order=2, stats=False, out_dtype=False, out_force=None, out_ext=False, out_interleave=False, out_byteorder=False)

Batch processing tool to spectrally smooth multiple datacubes in the same way.

Parameters

• fname_list (list, optional) – list of filenames to process; if left to None, will look at base_dir, search_ext, and dir_level parameters for files to process (default: None).

• base_dir (str, optional) – directory path to search for files to spectrally clip; if fname_list is not None, base_dir will be ignored (default: None).

• search_ext (str) – file format/extension to search for in all directories and subdirectories to determine which files to process; if fname_list is not None, search_ext will be ignored (default: ‘bip’).

• dir_level (int) – The number of directory levels to search; if None, searches all directory levels (default: 0).

• base_dir_out (str) – directory path to save all processed datacubes; if set to None, a folder named according to the folder_name parameter is added to base_dir

• folder_name (str) – folder to add to base_dir_out to save all the processed datacubes (default: ‘spec-smooth’).

• name_append (str) – name to append to the filename (default: ‘spec-smooth’).

• window_size (int) – the length of the window; must be an odd integer number (default: 11).

• order (int) – the order of the polynomial used in the filtering; must be less than window_size - 1 (default: 2).

• stats (bool) – whether to compute some basic descriptive statistics (mean, st. dev., and coefficient of variation) of the smoothed data array (default: False)

• out_XXX – Settings for saving the output files can be adjusted here if desired. They are stored in batch.io.defaults, and are therefore accessible at a high level. See ``hsio.set_io_defaults() for more information on each of the settings.

Note

The following batch example builds on the API example results of the batch.spatial_crop function. Please complete the batch.spatial_crop example to be sure your directory (i.e., base_dir) is populated with multiple hyperspectral datacubes. The following example will be using datacubes located in the following directory: F:\nigo0024\Documents\hs_process_demo\spatial_crop

Example

Load and initialize the batch module, checking to be sure the directory exists.

>>> import os
>>> from hs_process import batch
>>> base_dir = r'F:\nigo0024\Documents\hs_process_demo\spatial_crop'
>>> print(os.path.isdir(base_dir))
True
>>> hsbatch = batch(base_dir, search_ext='.bip')  # searches for all files in ``base_dir`` with a ".bip" file extension

Use batch.spectral_smooth to perform a Savitzky-Golay smoothing operation on each image/pixel in base_dir. The window_size and order can be adjusted to achieve desired smoothing results.

>>> hsbatch.spectral_smooth(base_dir=base_dir, folder_name='spec_smooth',
                            window_size=11, order=2)
Processing 40 files. If this is not what is expected, please check if files have already undergone processing. If existing files should be overwritten, be sure to set the ``out_force`` parameter.
Spectrally smoothing: F:\nigo0024\Documents\hs_process_demo\spatial_crop\Wells_rep2_20180628_16h56m_pika_gige_7_1011-spatial-crop.bip
Saving F:\nigo0024\Documents\hs_process_demo\spatial_crop\spec_smooth\Wells_rep2_20180628_16h56m_pika_gige_7_1011-spec-smooth.bip
Spectrally smoothing: F:\nigo0024\Documents\hs_process_demo\spatial_crop\Wells_rep2_20180628_16h56m_pika_gige_7_1012-spatial-crop.bip
Saving F:\nigo0024\Documents\hs_process_demo\spatial_crop\spec_smooth\Wells_rep2_20180628_16h56m_pika_gige_7_1012-spec-smooth.bip
...

Use seaborn to visualize the spectra of a single pixel in one of the processed images.

>>> import seaborn as sns
>>> fname = os.path.join(base_dir, 'Wells_rep2_20180628_16h56m_pika_gige_7_1011-spatial-crop.bip')
>>> hsbatch.io.read_cube(fname)
>>> spy_mem = hsbatch.io.spyfile.open_memmap()  # datacube before smoothing
>>> meta_bands = list(hsbatch.io.tools.meta_bands.values())
>>> fname = os.path.join(base_dir, 'spec_smooth', 'Wells_rep2_20180628_16h56m_pika_gige_7_1011-spec-smooth.bip')
>>> hsbatch.io.read_cube(fname)
>>> spy_mem_clip = hsbatch.io.spyfile.open_memmap()  # datacube after smoothing
>>> meta_bands_clip = list(hsbatch.io.tools.meta_bands.values())
>>> ax = sns.lineplot(x=meta_bands, y=spy_mem[26][29], label='Before spectral smoothing', linewidth=3)
>>> ax = sns.lineplot(x=meta_bands_clip, y=spy_mem_clip[26][29], label='After spectral smoothing', ax=ax)
>>> ax.set_xlabel('Wavelength (nm)', weight='bold')
>>> ax.set_ylabel('Reflectance (%)', weight='bold')
>>> ax.set_title(r'API Example: `batch.spectral_smooth`', weight='bold')

Notice how the “choppiness” of the spectral curve is lessened after the smoothing operation. There are spectral regions that perhaps had a lower signal:noise ratio and did not do particularlly well at smoothing (i.e., < 410 nm, ~770 nm, and ~820 nm). It may be wise to perform batch.spectral_smooth after batch.spectral_clip.

class hs_process.defaults

Bases: object

Class containing default values and/or settings for various hs_process tools/functions.

crop_defaults

Default values for performing spatial cropping on images. crop_defaults is referenced by the spatial_mod.crop_single() function to get default values if various user-parameters are not passed or are left to None. In this way, defaults.crop_defaults can be modified once by the user to avoid having to pass the same parameter(s) repeatedly if executing spatial_mod.crop_single() many times, such as in a for loop.

crop_defaults.directory

File directory of the input image to be cropped (default: None).

Type

str

crop_defaults.name_short

Part of the datacube name that is generally not repeated across many datacubes captured at the same time. In the name_long example above, name_short = “plot_101_pika_gige_2”. The part of the filename that is name_short should end with a dash (but should not include that dash as it belongs to name_long; default: None).

Type

str

crop_defaults.name_long

Part of the datacube name that tends to be long and is repeated across many datacubes captured at the same time. This is an artifact of Resonon/Spectronon software, and may be desireable to shorten and/or make more informative. For example, a datacube may have the following name: “plot_101_pika_gige_2-Radiance From Raw Data-Georectify Airborne Datacube-Reflectance from Radiance Data and Measured Reference Spectrum.bip” and another datacube captured in the same campaign may be named: “plot_102_pika_gige_1-Radiance From Raw Data-Georectify Airborne Datacube-Reflectance from Radiance Data and Measured Reference Spectrum.bip” name_long should refer to everything after the first dash (including the first dash) up to the file extension (“.bip”): name_long = “-Radiance From Raw Data-Georectify Airborne Datacube-Reflectance from Radiance Data and Measured Reference Spectrum” (default: None).

Type

str

crop_defaults.ext

File extension to save the cropped image (default: ‘bip’).

Type

str

crop_defaults.pix_e_ul

upper left pixel column (easting) to begin cropping (default: 0).

Type

int

crop_defaults.pix_n_ul

upper left pixel row (northing) to begin cropping (default: 0).

Type

int

crop_defaults.buf_e_pix

The buffer distance in the easting direction (in pixel units) to be applied after calculating the original crop area (default: 0).

Type

int

crop_defaults.buf_n_pix

The buffer distance in the northing direction (in pixel units) to be applied after calculating the original crop area (default: 0).

Type

int

crop_defaults.buf_e_m

The buffer distance in the easting direction (in map units; e.g., meters) to be applied after calculating the original crop area; the buffer is considered after crop_X_m/crop_X_pix. A positive value will reduce the size of crop_X_m/crop_X_pix, and a negative value will increase it (default: None).

Type

float

crop_defaults.buf_n_m

The buffer distance in the northing direction (in map units; e.g., meters) to be applied after calculating the original crop area; the buffer is considered after crop_X_m/crop_X_pix. A positive value will reduce the size of crop_X_m/crop_X_pix, and a negative value will increase it (default: None).

Type

float

crop_defaults.crop_e_pix

number of pixels in each row in the cropped image (default: 90).

Type

int

crop_defaults.crop_n_pix

number of pixels in each column in the cropped image (default: 120).

Type

int

crop_defaults.crop_e_m

length of each row (easting direction) of the cropped image in map units (e.g., meters; default: None).

Type

float

crop_defaults.crop_n_m

length of each column (northing direction) of the cropped image in map units (e.g., meters; default: None).

Type

float

crop_defaults.plot_id_ref

the plot ID of the area to be cropped (default: None).

Type

int

envi_write

Attributes for writing ENVI datacubes to file, following the convention of the Spectral Python envi.save_image() parameter options for writing an ENVI datacube to file.

envi_write.dtype

The data type with which to store the image. For example, to store the image in 16-bit unsigned integer format, the argument could be any of numpy.uint16, 'u2', 'uint16', or 'H' (default: np.float32).

Type

numpy.dtype or str

envi_write.force

If hdr_file or its associated image file exist, force=True will overwrite the files; otherwise, an exception will be raised if either file exists (default: False).

Type

bool

envi_write.ext

The extension to use for saving the image file; if not specified, a default extension is determined based on the interleave. For example, if interleave``='bip', then ``ext is set to ‘bip’ as well. If ext is an empty string, the image file will have the same name as the .hdr, but without the ‘.hdr’ extension (default: None).

Type

str

envi_write.interleave

The band interleave format to use for writing the file; interleave should be one of ‘bil’, ‘bip’, or ‘bsq’ (default: ‘bip’).

Type

str

envi_write.byteorder

Specifies the byte order (endian-ness) of the data as written to disk. For little endian, this value should be either 0 or ‘little’. For big endian, it should be either 1 or ‘big’. If not specified, native byte order will be used (default: None).

Type

int or str

spat_crop_cols

Default column names for performing batch spatial cropping on images. Useful when batch processing images via batch.spatial_crop(). batch.spatial_crop() takes a parameter fname_sheet, which can be a filename to a spreadsheet or a pandas.DataFrame. defaults.spat_crop_cols should be modified if the column names in fname_sheet are different than what is expected (see documentation for batch.spatial_crop() to know the expected column names).

spat_crop_cols.directory

column name for input directory (default: ‘directory’).

Type

str

spat_crop_cols.fname

column name for input fname (default: ‘fname’).

Type

str

spat_crop_cols.name_short

column name for input image’s name_short (default: ‘name_short’).

Type

str

spat_crop_cols.name_long

column name for input image’s name_long (default: ‘name_long’).

Type

str

spat_crop_cols.ext

column name for file extension of input image (default: ‘ext’).

Type

str

spat_crop_cols.pix_e_ul

column name for pix_e_ul (default: ‘pix_e_ul’).

Type

str

spat_crop_cols.pix_n_ul

column name for pix_n_ul (default: ‘pix_n_ul’).

Type

str

spat_crop_cols.alley_size_e_pix

column name for alley_size_e_pix (default: ‘alley_size_e_pix’).

Type

str

spat_crop_cols.alley_size_n_pix

column name for alley_size_n_pix (default: ‘alley_size_n_pix’).

Type

str

spat_crop_cols.alley_size_e_m

column name for alley_size_e_m (default: ‘alley_size_e_m’).

Type

str

spat_crop_cols.alley_size_n_m

column name for alley_size_n_m (default: ‘alley_size_n_m’).

Type

str

spat_crop_cols.buf_e_pix

column name for buf_e_pix (default: ‘buf_e_pix’).

Type

str

spat_crop_cols.buf_n_pix

column name for buf_n_pix (default: ‘buf_n_pix’).

Type

str

spat_crop_cols.buf_e_m

column name for buf_e_m (default: ‘buf_e_m’).

Type

str

spat_crop_cols.buf_n_m

column name for buf_n_m (default: ‘buf_n_m’).

Type

str

spat_crop_cols.crop_e_pix

column name for crop_e_pix (default: ‘crop_e_pix’).

Type

str

spat_crop_cols.crop_n_pix

column name for crop_n_pix (default: ‘crop_n_pix’).

Type

str

spat_crop_cols.crop_e_m

column name for crop_e_m (default: ‘crop_e_m’).

Type

str

spat_crop_cols.crop_n_m

column name for crop_n_m (default: ‘crop_n_m’).

Type

str

spat_crop_cols.plot_id_ref

column name for plot_id (default: ‘crop_n_pix’).

Type

str

spat_crop_cols.n_plots_x

column name for n_plots_x (default: ‘n_plots_x’).

Type

str

spat_crop_cols.n_plots_y

column name for n_plots_y (default: ‘n_plots_y’).

Type

str

spat_crop_cols.n_plots

column name for n_plots (default: ‘n_plots’).

Type

str

class hs_process.hsio(fname_in=None, name_long=None, name_plot=None, name_short=None, str_plot='plot_', individual_plot=False, fname_hdr_spec=None)

Bases: object

Class for reading and writing hyperspectral data files, as well as accessing, interpreting, and modifying its associated metadata. With a hyperspectral data file loaded via hsio, there is simple functionality to display the datacube image as a multi-band render, as well as for saving a datacube as a 3-band geotiff. hsio relies heavily on the Spectral Python package.

read_cube(fname_hdr=None, overwrite=True, name_long=None, name_short=None, name_plot=None, individual_plot=False)

Reads in a hyperspectral datacube using the Spectral Python package.

Parameters

• fname_hdr (str) – filename of datacube to be read (default: None).

• overwrite (bool) – Whether to overwrite any of the previous user-passed variables, including name_long, name_plot, and name_short. If variables are already set and overwrite is False, they will remain the same. If variables are set and overwrite is True, they will be overwritten based on typcial file naming conventions of Resonon/Spectronon software. Any of the user-passed variables (e.g., name_long, etc.) will overwrite those that were set previously whether overwrite is True or False (default: False).

• name_long (str) – Spectronon processing appends processing names to the filenames; this indicates those processing names that are repetitive and can be deleted from the filename following processing (default: None).

• name_short (str) – The base name of the image file (see note above about name_long; default: None).

• name_plot (str) – numeric text that describes the plot number (default: None).

• individual_plot (bool) – Indicates whether image (and its filename) is for an individual plot (True), or for many plots (False; default: False).

Note

hs_process will search for name_long, name_plot, and name_short based on typical file naming behavior of Resonon/ Spectronon software. If any of these parameters are passed by the user, however, that will take precedence over “searching the typical file nameing behavior”.

Example

Load and initialize hsio

>>> from hs_process import hsio
>>> fname_hdr = r'F:\nigo0024\Documents\hs_process_demo\Wells_rep2_20180628_16h56m_pika_gige_7-Radiance Conversion-Georectify Airborne Datacube-Convert Radiance Cube to Reflectance from Measured Reference Spectrum.bip.hdr'
>>> io = hsio()  # initialize an instance of the hsio class (note there are no required parameters)

Load datacube using hsio.read_cube

>>> io.read_cube(fname_hdr)
>>> io.spyfile
Data Source:   'F:\nigo0024\Documents\hs_process_demo\Wells_rep2_20180628_16h56m_pika_gige_7-Radiance Conversion-Georectify Airborne Datacube-Convert Radiance Cube to Reflectance from Measured Reference Spectrum.bip'
    # Rows:            617
    # Samples:        1300
    # Bands:           240
    Interleave:        BIP
    Quantization:  32 bits
    Data format:   float32

Check name_long, name_short, and name_plot values derived from the filename

>>> io.name_long
'-Convert Radiance Cube to Reflectance from Measured Reference Spectrum'

>>> io.name_plot
'7'

>>> io.name_short
'Wells_rep2_20180628_16h56m_pika_gige_7'

read_spec(fname_hdr_spec, overwrite=True, name_long=None, name_short=None, name_plot=None)

Reads in a hyperspectral spectrum file using the using the Spectral Python package.

Parameters

fname_hdr_spec (str) – filename of spectra to be read.

Example

Load and initialize hsio

>>> from hs_process import hsio
>>> fname_hdr = r'F:\nigo0024\Documents\hs_process_demo\Wells_rep2_20180628_16h56m_pika_gige_7_plot_611-cube-to-spec-mean.spec.hdr'
>>> io = hsio()  # initialize an instance of the hsio class (note there are no required parameters)

Load datacube using hsio.read_spec

>>> io.read_spec(fname_hdr)
>>> io.spyfile_spec
Data Source:   'F:\nigo0024\Documents\hs_process_demo\Wells_rep2_20180628_16h56m_pika_gige_7_plot_611-cube-to-spec-mean.spec'
    # Rows:              1
    # Samples:           1
    # Bands:           240
    Interleave:        BIP
    Quantization:  32 bits
    Data format:   float32

Check name_long, name_short, and name_plot values derived from the filename

>>> io.name_long
'-cube-to-spec-mean'

>>> io.name_short
'Wells_rep2_20180628_16h56m_pika_gige_7_plot_611'

>>> io.name_plot
'611'

set_io_defaults(dtype=False, force=None, ext=False, interleave=False, byteorder=False)

Sets any of the ENVI file writing parameters to hsio; if any parameter is left unchanged from its default, it will remain as-is (i.e., it will not be set).

Parameters

• dtype (numpy.dtype or str) – The data type with which to store the image. For example, to store the image in 16-bit unsigned integer format, the argument could be any of numpy.uint16, ‘u2’, ‘uint16’, or ‘H’ (default=``False``).

• force (bool) – If hdr_file or its associated image file exist, force=True will overwrite the files; otherwise, an exception will be raised if either file exists (default=``None``).

• ext (str) – The extension to use for saving the image file; if not specified, a default extension is determined based on the interleave. For example, if interleave``='bip', then ``ext is set to ‘bip’ as well. If ext is an empty string, the image file will have the same name as the .hdr, but without the ‘.hdr’ extension (default: False).

• interleave (str) – The band interleave format to use for writing the file; interleave should be one of ‘bil’, ‘bip’, or ‘bsq’ (default=``False``).

• byteorder (int or str) – Specifies the byte order (endian-ness) of the data as written to disk. For little endian, this value should be either 0 or ‘little’. For big endian, it should be either 1 or ‘big’. If not specified, native byte order will be used (default=``False``).

Example

Load and initialize hsio

>>> from hs_process import hsio
>>> io = hsio()  # initialize an instance of the hsio class

Check defaults.envi_write

>>> io.defaults.envi_write
{'dtype': numpy.float32,
 'force': False,
 'ext': '',
 'interleave': 'bip',
 'byteorder': 0}

Modify force parameter and recheck defaults.envi_write

>>> io.set_io_defaults(force=True)
>>> io.defaults.envi_write
{'dtype': numpy.float32,
 'force': True,
 'ext': '',
 'interleave': 'bip',
 'byteorder': 0}

show_img(spyfile=None, band_r=120, band_g=76, band_b=32, vmin=None, vmax=None, cmap='viridis', cbar=True, inline=True)

Displays a datacube as a 3-band RGB image using Matplotlib.

Parameters

• spyfile (SpyFile object or numpy.ndarray) – The data cube to display; if None, loads from self.spyfile (default: None).

• band_r (int) – Band to display on the red channel (default: 120)

• band_g (int) – Band to display on the green channel (default: 76)

• band_b (int) – Band to display on the blue channel (default: 32)

• vmin/vmax (scalar, optional) – The data range that the colormap covers. By default, the colormap covers the complete value range of the supplied data (default: None).

• cmap (str) – The Colormap instance or registered colormap name used to map scalar data to colors. This parameter is ignored for RGB(A) data (default: “viridis”).

• cbar (bool) – Whether to include a colorbar in the image (default: True).

• inline (bool) – If True, displays in the IPython console; else displays in a pop-out window (default: True).

Note

The inline parameter points to the hsio.show_img function, and is only expected to work in an IPython console (not intended to be used in a normal Python console).

Example

Load hsio and spatial_mod modules

>>> from hs_process import hsio # load hsio
>>> from hs_process import spatial_mod # load spatial mod

Load the datacube using hsio.read_cube

>>> fname_hdr = r'F:\nigo0024\Documents\hs_process_demo\Wells_rep2_20180628_16h56m_pika_gige_7-Radiance Conversion-Georectify Airborne Datacube-Convert Radiance Cube to Reflectance from Measured Reference Spectrum.bip.hdr'
>>> io = hsio()  # initialize an instance of the hsio class
>>> io.read_cube(fname_hdr)

Perform simple spatial cropping via spatial_mod.crop_single

>>> my_spatial_mod = spatial_mod(io.spyfile)  # initialize spatial_mod instance to crop the image
>>> array_crop, metadata = my_spatial_mod.crop_single(pix_e_ul=250, pix_n_ul=100, crop_e_m=8, crop_n_m=3)

Show an RGB render of the cropped image using hsio.show_img

>>> io.show_img(array_crop)

write_cube(fname_hdr, spyfile, metadata=None, dtype=None, force=None, ext=None, interleave=None, byteorder=None)

Wrapper function that accesses the Spectral Python package to save a datacube to file.

Parameters

• fname_hdr (str) – Output header file path (with the ‘.hdr’ extension).

• spyfile (SpyFile object or numpy.ndarray) – The hyperspectral datacube to save. If numpy.ndarray, then metadata (dict) should also be passed.

• metadata (dict) – Metadata to write to the ENVI .hdr file describing the hyperspectral data cube being saved. If SpyFile object is passed to spyfile, metadata will overwrite any existing metadata stored by the SpyFile object (default=None).

• dtype (numpy.dtype or str) – The data type with which to store the image. For example, to store the image in 16-bit unsigned integer format, the argument could be any of numpy.uint16, ‘u2’, ‘uint16’, or ‘H’ (default=np.float32).

• force (bool) – If hdr_file or its associated image file exist, force=True will overwrite the files; otherwise, an exception will be raised if either file exists (default=False).

• ext (None or str) – The extension to use for saving the image file. If not specified or if set to an empty string (e.g., ext=''), a default extension is determined using the same name as fname_hdr, except without the “.hdr” extension. If fname_hdr is provided without the “non-.hdr” extension (e.g., “bip”), then the extension is determined from the interleave parameter. For example, if interleave``='bip', then ``ext is set to ‘bip’ as well. Use of ext is not recommended; instead, just set fname_hdr with the correct extension or use interleave to set the extension (default: None; determined from fname_hdr or interleave).

• interleave (str) – The band interleave format to use for writing the file; interleave should be one of ‘bil’, ‘bip’, or ‘bsq’ (default=’bip’).

• byteorder (int or str) – Specifies the byte order (endian-ness) of the data as written to disk. For little endian, this value should be either 0 or ‘little’. For big endian, it should be either 1 or ‘big’. If not specified, native byte order will be used (default=None).

Note

If dtype, force, ext, interleave, and byteorder are not passed, default values will be pulled from hsio.defaults. Thus, hsio.defaults can be modified prior to calling hsio.write_cube() to avoid having to pass each of thes parameters in the hsio.write_cube() function (see the hsio.set_io_defaults() function for support on setting these defaults and for more information on the parameters). Each of these parameters are passed directly to the Spectral Python envi.save_image() function. For more information, please refer to the Spectral Python documentation.

Example

Load hsio and spatial_mod modules

>>> import os
>>> import pathlib
>>> from hs_process import hsio  # load hsio
>>> from hs_process import spatial_mod  # load spatial mod
>>> data_dir = r'F:\nigo0024\Documents\hs_process_demo'
>>> fname_hdr_in = os.path.join(data_dir, 'Wells_rep2_20180628_16h56m_pika_gige_7-Radiance Conversion-Georectify Airborne Datacube-Convert Radiance Cube to Reflectance from Measured Reference Spectrum.bip.hdr')
>>> io = hsio()  # initialize the hsio class
>>> io.read_cube(fname_hdr_in)

Perform simple spatial cropping via spatial_mod.crop_single to generate a new datacube.

>>> my_spatial_mod = spatial_mod(io.spyfile)  # initialize spatial_mod instance to crop the image
>>> array_crop, metadata = my_spatial_mod.crop_single(pix_e_ul=250, pix_n_ul=100, crop_e_m=8, crop_n_m=3)

Save the datacube using hsio.write_cube

>>> fname_hdr = os.path.join(data_dir, 'hsio', 'Wells_rep2_20180628_16h56m_pika_gige_7-hsio-write-cube-cropped.bip.hdr')
>>> pathlib.Path(os.path.dirname(fname_hdr)).mkdir(parents=True, exist_ok=True)
>>> io.write_cube(fname_hdr, array_crop, metadata=metadata, force=True)

Load the datacube into Spectronon for visualization

write_spec(fname_hdr_spec, df_mean, df_std=None, metadata=None, dtype=None, force=None, ext=None, interleave=None, byteorder=None)

Wrapper function that accesses the Spectral Python package to save a single spectra to file.

Parameters

• fname_hdr_spec (str) – Output header file path (with the ‘.hdr’ extension). If the extension is explicitely specified in fname_hdr_spec and the ext parameter is also specified, fname_hdr_spec will be modified to conform to the extension set using the ext parameter.

• df_mean (pandas.Series or numpy.ndarray) – Mean spectra, stored as a df row, where columns are the bands.

• df_std (pandas.Series or numpy.ndarray, optional) – Standard deviation of each spectra, stored as a df row, where columns are the bands. This will be saved to the .hdr file.

• dtype (numpy.dtype or str) – The data type with which to store the image. For example, to store the image in 16-bit unsigned integer format, the argument could be any of numpy.uint16, ‘u2’, ‘uint16’, or ‘H’ (default=np.float32).

• force (bool) – If hdr_file or its associated image file exist, force=True will overwrite the files; otherwise, an exception will be raised if either file exists (default=False).

• ext (None or str) – The extension to use for saving the image file. If not specified or if set to an empty string (e.g., ext=''), a default extension is determined using the same name as fname_hdr_spec, except without the “.hdr” extension. If fname_hdr_spec is provided without the “non-.hdr” extension (e.g., “bip”), then the extension is determined from the interleave parameter. For example, if interleave``='bip', then ``ext is set to ‘bip’ as well. Use of ext is not recommended; instead, just set fname_hdr_spec with the correct extension or use interleave to set the extension (default: None; determined from fname_hdr_spec or interleave).

• interleave (str) – The band interleave format to use for writing the file; interleave should be one of ‘bil’, ‘bip’, or ‘bsq’ (default=’bip’).

• byteorder (int or str) – Specifies the byte order (endian-ness) of the data as written to disk. For little endian, this value should be either 0 or ‘little’. For big endian, it should be either 1 or ‘big’. If not specified, native byte order will be used (default=None).

• metadata (dict) – Metadata to write to the ENVI .hdr file describing the spectra being saved; if None, will try to pull metadata template from hsio.spyfile_spec.metadata or hsio.spyfile.metadata (default=None).

Example

Load and initialize hsio

>>> from hs_process import hsio # load hsio
>>> fname_hdr_in = r'F:\nigo0024\Documents\hs_process_demo\Wells_rep2_20180628_16h56m_pika_gige_7-Radiance Conversion-Georectify Airborne Datacube-Convert Radiance Cube to Reflectance from Measured Reference Spectrum.bip.hdr'
>>> io = hsio()  # initialize the hsio class (note there are no required parameters)
>>> io.read_cube(fname_hdr_in)

Calculate spectral mean via hstools.mean_datacube

>>> spec_mean, spec_std, _ = io.tools.mean_datacube(io.spyfile)
>>> fname_hdr_spec = r'F:\nigo0024\Documents\hs_process_demo\hsio\Wells_rep2_20180628_16h56m_pika_gige_7-mean.spec.hdr'

Save the new spectra to file via hsio.write_spec

>>> io.write_spec(fname_hdr_spec, spec_mean, spec_std)
Saving F:\nigo0024\Documents\hs_process_demo\hsio\Wells_rep2_20180628_16h56m_pika_gige_7-mean.spec

Open Wells_rep2_20180628_16h56m_pika_gige_7-mean.spec in Spectronon for visualization

write_tif(fname_tif, spyfile=None, metadata=None, fname_in=None, projection_out=None, geotransform_out=None, show_img=False)

Wrapper function that accesses the GDAL Python package to save a small datacube subset (i.e., three bands or less) to file.

Parameters

• fname_tif (str) – Output image file path (with the ‘.tif’ extension).

• spyfile (SpyFile object or numpy.ndarray, optional) – The data cube to save. If numpy.ndarray, then metadata (dict) should also be passed. If None, uses hsio.spyfile (default: None).

• metadata (dict) – Metadata information; if geotransform_out is not passed, “map info” is accessed from metadata and geotransform_out is created from that “map info”.

• fname_in (str, optional) – The filename of the image datacube to be read in initially. This is potentially useful if projection_out and/or geotransform_out are not passed and a numpy.ndarray is passed as the spyfile - in this case, write_tif() uses fname_in to load the fname_in datacube via GDAL, which can in turn be used to load the projection or geotransform information for the output geotiff (default: None).

• projection_out (str) – The GDAL projection to use while writing the geotiff. Applied using gdal.driver.dataset.SetProjection() (default: None; hsio.projection_out)

• geotransform_out (str) – The GDAL geotransform to use while writing the geotiff. Applied using gdal.driver.dataset.SetGeoTransform() (default: None; hsio.geotransform_out)

• show_img (bool or str) – Whether to display a render of the image being saved as a geotiff. Must be False (does not display the image), “inline” (displays the image inline using the IPython console), or “popout” (displays the image in a pop-out window; default: “inline”).

Example

Load and initialize hsio

>>> from hs_process import hsio  # load hsio
>>> fname_hdr_in = r'F:\nigo0024\Documents\hs_process_demo\Wells_rep2_20180628_16h56m_pika_gige_7-Radiance Conversion-Georectify Airborne Datacube-Convert Radiance Cube to Reflectance from Measured Reference Spectrum.bip.hdr'
>>> io = hsio()  # initialize the hsio class
>>> io.read_cube(fname_hdr_in)

Save an RGB render of the datacube to file via hsio.write_tif

>>> fname_tif = r'F:\nigo0024\Documents\hs_process_demo\hsio\Wells_rep2_20180628_16h56m_pika_gige_7.tif'
>>> io.write_tif(fname_tif, spyfile=io.spyfile, fname_in=fname_hdr_in)
Either `projection_out` is `None` or `geotransform_out` is `None` (or both are). Retrieving projection and geotransform information by loading `hsio.fname_in` via GDAL. Be sure this is appropriate for the data you are trying to write.
Clipping input data to the valid range for imshow with RGB data ([0..1] for floats or [0..255] for integers).

Open Wells_rep2_20180628_16h56m_pika_gige_7.tif in QGIS with the plot boundaries overlaid

class hs_process.hstools(spyfile)

Bases: object

Basic tools for manipulating Spyfiles and accessing their metadata.

Parameters

spyfile (SpyFile object) – The datacube being accessed and/or manipulated.

clean_md_sets(metadata=None)

Converts metadata items that are expressed as a list to be expressed as a dictionary.

Parameters

metadata (dict, optional) – Metadata dictionary to clean

Returns

metadata_out (dict) – Cleaned metadata dictionary.

Return type

dict

Example

Load and initialize hsio

>>> from hs_process import hsio
>>> fname_in = r'F:\nigo0024\Documents\hs_process_demo\Wells_rep2_20180628_16h56m_pika_gige_7-Radiance Conversion-Georectify Airborne Datacube-Convert Radiance Cube to Reflectance from Measured Reference Spectrum.bip.hdr'
>>> io = hsio(fname_in)

Create sample metadata with “wavelength” expressed as a list of strings

>>> metadata = {'samples': 1300,
                'lines': 617,
                'bands': 4,
                'file type': 'ENVI Standard',
                'wavelength': ['394.6', '396.6528', '398.7056',
                '400.7584']}

Clean metadata using hstools.clean_md_sets. Notice how wavelength is now expressed as a str representation of a dict, which is required for properly writing the metadata to the .hdr file via save_image() in Spectral Python.

>>> io.tools.clean_md_sets(metadata=metadata)
{'samples': 1300,
 'lines': 617,
 'bands': 4,
 'file type': 'ENVI Standard',
 'wavelength': '{394.6, 396.6528, 398.7056, 400.7584}'}

del_meta_item(metadata, key)

Deletes metadata item from SpyFile object.

Parameters

• metadata (dict) – dictionary of the metadata

• key (str) – dictionary key to delete

Returns

metadata (dict) – Dictionary containing the modified metadata.

Return type

dict

Example

Load and initialize hsio

>>> from hs_process import hsio
>>> fname_in = r'F:\nigo0024\Documents\hs_process_demo\Wells_rep2_20180628_16h56m_pika_gige_7-Radiance Conversion-Georectify Airborne Datacube-Convert Radiance Cube to Reflectance from Measured Reference Spectrum.bip.hdr'
>>> io = hsio(fname_in)

Create sample metadata

>>> metadata = {'samples': 1300,
                'lines': 617,
                'bands': 4,
                'file type': 'ENVI Standard',
                'map info': '{UTM, 1.0, 1.0, 421356.76707299997, 4844936.7317699995, 0.04, 0.04, 15, T, WGS-84, units  meters, rotation  0.000}',
                'wavelength': ['394.6', '396.6528', '398.7056',
                '400.7584']}

Delete “map info” from metadata using hstools.del_met_item

>>> io.tools.del_meta_item(metadata, 'map info')
{'samples': 1827,
 'lines': 617,
 'bands': 4,
 'file type': 'ENVI Standard',
 'wavelength': ['394.6', '396.6528', '398.7056', '400.7584']}

dir_data()

Retrieves the data directory from “site packages”.

get_UTM(pix_e_ul, pix_n_ul, utm_x, utm_y, size_x, size_y)

Calculates the new UTM coordinate of cropped plot to modify the “map info” tag of the .hdr file.

Parameters

• pix_e_ul (int) – upper left column (easting) where image cropping begins.

• pix_n_ul (int) – upper left row (northing) where image cropping begins.

• utm_x (float) – UTM easting coordinates (meters) of the original image (from the upper left).

• utm_y (float) – UTM northing coordinates (meters) of the original image (from the upper left).

• size_x (float) – Ground resolved distance of the image pixels in the x (easting) direction (meters).

• size_y (float) – Ground resolved distance of the image pixels in the y (northing) direction (meters).

Returns

2-element tuple containing

• utm_x_new (float): The modified UTM x coordinate (easting) of cropped plot.

• utm_y_new (float): The modified UTM y coordinate (northing) of cropped plot.

Example

Load and initialize hsio

>>> from hs_process import hsio
>>> fname_in = r'F:\nigo0024\Documents\hs_process_demo\Wells_rep2_20180628_16h56m_pika_gige_7-Radiance Conversion-Georectify Airborne Datacube-Convert Radiance Cube to Reflectance from Measured Reference Spectrum.bip.hdr'
>>> io = hsio(fname_in)

Retrieve UTM coordinates and pixel sizes from the metadata

>>> map_info_set = io.spyfile.metadata['map info']
>>> utm_x = io.tools.get_meta_set(map_info_set, 3)
>>> utm_y = io.tools.get_meta_set(map_info_set, 4)
>>> spy_ps_e = float(map_info_set[5])
>>> spy_ps_n = float(map_info_set[6])

Calculate the UTM coordinates at the 100th easting pixel and 50th northing pixel using hstools.get_UTM

>>> ul_x_utm, ul_y_utm = io.tools.get_UTM(100, 50,
                                          utm_x, utm_y,
                                          spy_ps_e,
                                          spy_ps_n)
>>> ul_x_utm
441360.80707299995
>>> ul_y_utm
4855934.691769999

get_band(target_wl, spyfile=None)

Finds the band number of the closest target wavelength.

Parameters

• target_wl (int or float) – the target wavelength to retrieve the band number for (required).

• spyfile (SpyFile object, optional) – The datacube being accessed and/or manipulated; if None, uses hstools.spyfile (default: None).

Returns

key_band (int) – band number of the closest target wavelength (target_wl).

Return type

int

Example

Load and initialize hsio

>>> from hs_process import hsio
>>> fname_in = r'F:\nigo0024\Documents\hs_process_demo\Wells_rep2_20180628_16h56m_pika_gige_7-Radiance Conversion-Georectify Airborne Datacube-Convert Radiance Cube to Reflectance from Measured Reference Spectrum.bip.hdr'
>>> io = hsio(fname_in)

Use hstools.get_band to find the band number corresponding to 703 nm

>>> io.tools.get_band(703, io.spyfile)
151

get_band_index(band_name)

. Returns the index of band from “band names”.

Parameters

band_name (int or list) – the target band to retrieve the band index for (required).

Returns

band_idx (int) – band index of the passed band name (band_name).

Return type

int or list

Example

Load and initialize hsio

>>> from hs_process import hsio
>>> fname_in = r'F:\nigo0024\Documents\hs_process_demo\Wells_rep2_20180628_16h56m_pika_gige_7-Radiance Conversion-Georectify Airborne Datacube-Convert Radiance Cube to Reflectance from Measured Reference Spectrum.bip.hdr'
>>> io = hsio(fname_in)

Using hstools.get_band_index, find the band index of bands 4, 43, and 111.

>>> io.tools.get_band_index([4, 43, 111])
[3, 42, 110]

get_band_num(band_idx)

Adds 1 to band_idx and returns the band number(s).

Parameters

band_idx (int or list) – the target band index(es) to retrive the band number for (required).

Returns

band_num (int or list): band number of the passed band index (band_idx).

Return type

int or list

Example

Load and initialize hsio

>>> from hs_process import hsio
>>> fname_in = r'F:\nigo0024\Documents\hs_process_demo\Wells_rep2_20180628_16h56m_pika_gige_7-Radiance Conversion-Georectify Airborne Datacube-Convert Radiance Cube to Reflectance from Measured Reference Spectrum.bip.hdr'
>>> io = hsio(fname_in)

Using hstools.get_band_num, find the band number located at the 4th, 43rd, and 111th index values.

>>> io.tools.get_band_num([4, 43, 111])
[5, 44, 112]

get_band_range(range_wl, index=True, spyfile=None)

Retrieves the band index or band name for all bands within a wavelength range.

Parameters

• range_wl (list) – the minimum and maximum wavelength to consider; values should be int or float.

• index (bool) – Indicates whether to return the band number (False; min=1) or to return index number (True; min=0) (default: True).

Returns

band_list (list): A list of all bands (either index or number, depending on how index is set) between a range in wavelength values.

Return type

list

Example

Load and initialize hsio

>>> from hs_process import hsio
>>> data_dir = r'F:\nigo0024\Documents\hs_process_demo'
>>> fname_in = os.path.join(data_dir, 'Wells_rep2_20180628_16h56m_pika_gige_7-Radiance Conversion-Georectify Airborne Datacube-Convert Radiance Cube to Reflectance from Measured Reference Spectrum.bip.hdr')
>>> io = hsio(fname_in)

Find the band name of all bands between 700 and 710 nm

>>> io.tools.get_band_range([700, 710], index=False, spyfile=io.spyfile)
[150, 151, 152, 153, 154]

Find the band index values of all bands between 700 and 710 nm via hstools.get_band_range

>>> io.tools.get_band_range([700, 710], index=True, spyfile=io.spyfile)
[149, 150, 151, 152, 153]

Sometimes “band names” are not integers in sequential order. To demonstrate the utility of the index parameter, let’s take a look at Sentinel 2A mimicked imagery.

>>> data_dir = r'F:\nigo0024\Documents\hs_process_demo\spec_mod'
>>> fname_in = os.path.join(data_dir, 'Wells_rep2_20180628_16h56m_pika_gige_7-mimic-s2a.bip.hdr')
>>> io = hsio(fname_in)

Find the band name of all bands between 760 and 840 nm

>>> io.tools.get_band_range([760, 840], index=False, spyfile=io.spyfile)
['S2A_SR_AV_B7', 'S2A_SR_AV_B8']

Find the band index values of all bands between 760 and 840 nm via hstools.get_band_range

>>> io.tools.get_band_range([760, 840], index=True, spyfile=io.spyfile)
[6, 7]

get_center_wl(wl_list, spyfile=None, wls=True)

Gets band numbers and mean wavelength from all wavelengths (or bands) in wl_list.

Parameters

• wl_list (list) – the list of bands to get information for (required).

• spyfile (SpyFile object) – The datacube being accessed and/or manipulated; if None, uses hstools.spyfile (default: None).

• wls (bool) – whether wavelengths are passed in wl_list or if bands are passed in wl_list (default: True - wavelenghts passed).

Returns

2-element tuple containing

• bands (list): the list of bands (band number) corresponding to wl_list.

• wls_mean (float): the mean wavelength from wl_list.

Example

Load and initialize hsio

>>> from hs_process import hsio
>>> fname_in = r'F:\nigo0024\Documents\hs_process_demo\Wells_rep2_20180628_16h56m_pika_gige_7-Radiance Conversion-Georectify Airborne Datacube-Convert Radiance Cube to Reflectance from Measured Reference Spectrum.bip.hdr'
>>> io = hsio(fname_in)

Using hstools.get_center_wl, find the bands and actual mean wavelength of the bands closest to 700 and 710 nm.

>>> bands, wls_mean = io.tools.get_center_wl([700, 710], wls=True)
>>> bands
[150, 155]
>>> wls_mean
705.5992

get_meta_set(meta_set, idx=None)

Reads metadata “set” (i.e., string representation of a Python set; common in .hdr files), taking care to remove leading and trailing spaces.

Parameters

• meta_set (str) – the string representation of the metadata set

• idx (int) – index to be read; if None, the whole list is returned (default: None).

Returns

metadata_list (list or str): List of metadata set items (as str), or if idx is not None, the item in the position described by idx.

Return type

list or str

Example

Load and initialize hsio

>>> from hs_process import hsio
>>> fname_in = r'F:\nigo0024\Documents\hs_process_demo\Wells_rep2_20180628_16h56m_pika_gige_7-Radiance Conversion-Georectify Airborne Datacube-Convert Radiance Cube to Reflectance from Measured Reference Spectrum.bip.hdr'
>>> io = hsio(fname_in)

Retrieve the “map info” set from the metadata via hstools.get_meta_set

>>> map_info_set = io.spyfile.metadata['map info']
['UTM',
 '1.0',
 '1.0',
 '441357.287073',
 '4855944.7717699995',
 '0.04',
 '0.04',
 '15',
 'T',
 'WGS-84',
 'units  meters',
 'rotation  0.000']

get_spectral_mean(band_list, spyfile=None)

Gets the spectral mean of a datacube from a list of bands.

Parameters

• band_list (list) – the list of bands to calculate the spectral mean for on the datacube (required).

• spyfile (SpyFile object or numpy.ndarray) – The datacube being accessed and/or manipulated; if None, uses hstools.spyfile (default: None).

Returns

array_mean (numpy.array or pandas.DataFrame): The mean reflectance from spyfile for the bands in band_list.

Return type

numpy.array or pandas.DataFrame

Example

Load and initialize hsio

>>> from hs_process import hsio
>>> fname_in = r'F:\nigo0024\Documents\hs_process_demo\Wells_rep2_20180628_16h56m_pika_gige_7-Radiance Conversion-Georectify Airborne Datacube-Convert Radiance Cube to Reflectance from Measured Reference Spectrum.bip.hdr'
>>> io = hsio(fname_in)

Calculate the spectral mean of the datacube via hstools.get_spectral_mean using all bands between 800 and 840 nm

>>> band_list = io.tools.get_band_range([800, 840], index=False)
>>> array_mean = io.tools.get_spectral_mean(band_list, spyfile=io.spyfile)
>>> io.show_img(array_mean)

get_wavelength(target_band, spyfile=None)

Returns actual wavelength of the closest target band.

Parameters

• target_band (int or float) – the target band to retrieve wavelength number for (required).

• spyfile (SpyFile object, optional) – The datacube being accessed and/or manipulated; if None, uses hstools.spyfile (default: None).

Returns

key_wavelength (float) – wavelength of the closest target band (target_band).

Return type

float

Example

Load and initialize hsio

>>> from hs_process import hsio
>>> fname_in = r'F:\nigo0024\Documents\hs_process_demo\Wells_rep2_20180628_16h56m_pika_gige_7-Radiance Conversion-Georectify Airborne Datacube-Convert Radiance Cube to Reflectance from Measured Reference Spectrum.bip.hdr'
>>> io = hsio(fname_in)

Use hstools.get_wavelength to find the wavelength value corresponding to the 151st band

>>> io.tools.get_wavelength(151, io.spyfile)
702.52

get_wavelength_range(range_bands, index=True, spyfile=None)

Retrieves the wavelengths for all bands within a band range.

Parameters

• range_bands (list) – the minimum and maximum band number to consider; values should be int.

• index (bool) – Indicates whether the bands in range_bands denote the band number (False; min=1) or the index number (True; min=0) (default: True).

Returns

wavelength_list (list): A list of all wavelengths between a range in band numbers or index values (depending how index is set).

Return type

list

Example

Load and initialize hsio

>>> from hs_process import hsio
>>> fname_hdr = r'F:\nigo0024\Documents\GitHub\hs_process\hs_process\data\Wells_rep2_20180628_16h56m_test_pika_gige_7-Convert Radiance Cube to Reflectance from Measured Reference Spectrum.bip.hdr'
>>> io = hsio(fname_hdr)

Find the wavelengths from the 16th to 21st bands

>>> io.tools.get_wavelength_range([16, 21], index=False, spyfile=io.spyfile)
[425.392, 427.4448, 429.4976, 431.5504, 433.6032, 435.656]

Find the wavelengths from the 16th to the 21st index

>>> io.tools.get_wavelength_range([16, 21], index=True, spyfile=io.spyfile)
[427.4448, 429.4976, 431.5504, 433.6032, 435.656, 437.7088]

load_spyfile(spyfile)

Loads a SpyFile (Spectral Python object) for data access and/or manipulation by the hstools class.

Parameters

spyfile (SpyFile object) – The datacube being accessed and/or manipulated.

Example

Load and initialize hsio

>>> from hs_process import hsio
>>> fname_in = r'F:\nigo0024\Documents\hs_process_demo\Wells_rep2_20180628_16h56m_pika_gige_7-Radiance Conversion-Georectify Airborne Datacube-Convert Radiance Cube to Reflectance from Measured Reference Spectrum.bip.hdr'
>>> io = hsio(fname_in)

Load a new datacube using hstools.load_spyfile

>>> io.tools.load_spyfile(io.spyfile)
>>> io.tools.spyfile
Data Source:   'F:\nigo0024\Documents\hs_process_demo\Wells_rep2_20180628_16h56m_pika_gige_7-Radiance Conversion-Georectify Airborne Datacube-Convert Radiance Cube to Reflectance from Measured Reference Spectrum.bip'
    # Rows:            617
    # Samples:        1300
    # Bands:           240
    Interleave:        BIP
    Quantization:  32 bits
    Data format:   float32

mask_array(array, metadata, thresh=None, percentile=None, side='lower')

Creates a masked numpy array based on a threshold value. If array is already a masked array, that mask is maintained and the new mask(s) is/ are added to the original mask.

Parameters

• array (numpy.ndarray) – The data array to mask.

• thresh (float or list) – The value for which to base the threshold; if thresh is list and side is None, then all values in thresh will be masked; if thresh is list and side is not None, then only the first value in the list will be considered for thresholding (default: None).

• percentile (float) – The percentile of pixels to mask; if percentile = 95 and side = ‘lower’, the lowest 95% of pixels will be masked prior to calculating the mean spectra across pixels (default: None; range: 0-100).

• side (str) – The side of the threshold for which to apply the mask. Must be either ‘lower’, ‘upper’, ‘outside’, or None; if ‘lower’, everything below the threshold will be masked; if ‘outside’, the thresh / percentile parameter must be list-like with two values indicating the lower and upper bounds - anything outside of these values will be masked out; if None, only the values that exactly match the threshold will be masked (default: ‘lower’).

Returns

2-element tuple containing

• array_mask (numpy.ndarray): The masked numpy.ndarray based on the passed threshold and/or percentile value.

• metadata (dict): The modified metadata.

Example

Load and initialize hsio

>>> from hs_process import hsio
>>> fname_in = r'F:\nigo0024\Documents\hs_process_demo\Wells_rep2_20180628_16h56m_pika_gige_7-Radiance Conversion-Georectify Airborne Datacube-Convert Radiance Cube to Reflectance from Measured Reference Spectrum.bip.hdr'
>>> io = hsio(fname_in)

Retrieve the image band at 800 nm using hstools.get_band and hsio.spyfile.open_memmap

>>> band = io.tools.get_band(800)
>>> array = io.spyfile.open_memmap()[:, :, band]

Create a masked array of all values below the 75th percentile via hstools.mask_array

>>> array_mask, metadata = io.tools.mask_array(array, io.spyfile.metadata, percentile=75, side='lower')

See that the “history” tage in the metadata has been modified

>>> metadata['history'][-158:]
"hs_process.mask_array[<label: 'thresh?' value:None; label: 'percentile?' value:75; label: 'side?' value:lower; label: 'unmasked_pct?' value:24.9935170178282>]"

Visualize the unmasked array using hsio.show_img. Set vmin and vmax to ensure the same color scale is used in comparing the masked vs. unmasked arrays.

>>> vmin = array.min()
>>> vmax = array.max()
>>> io.show_img(array, vmin=vmin, vmax=vmax)

Visualize the unmasked array using hsio.show_img

>>> io.show_img(array_mask, vmin=vmin, vmax=vmax)

mean_datacube(spyfile, mask=None, nodata=0)

Calculates the mean spectra for a datcube; if mask is passed (as a numpy.ndarray), then the mask is applied to spyfile prior to computing the mean spectra.

Parameters

• spyfile (SpyFile object or numpy.ndarray) – The hyperspectral datacube to mask.

• mask (numpy.ndarray) – the mask to apply to spyfile; if mask does not have similar dimensions to spyfile, the first band (i.e., first two dimensions) of mask will be repeated n times to match the number of bands of spyfile (default: None).

• nodata (float or None) – If None, treats all pixels cells as they are repressented in the numpy.ndarray. Otherwise, replaces nodata with np.nan and these cells will not be considered when calculating the mean spectra.

Returns

3-element tuple containing

• spec_mean (SpyFile.SpyFile object): The mean spectra from the input datacube.

• spec_std (SpyFile.SpyFile object): The standard deviation of the spectra from the input datacube.

• datacube_masked (numpy.ndarray): The masked numpy.ndarray; if mask is None, datacube_masked is identical to the SpyFile data array.

Example

Load and initialize hsio

>>> from hs_process import hsio
>>> fname_in = r'F:\nigo0024\Documents\hs_process_demo\Wells_rep2_20180628_16h56m_pika_gige_7-Radiance Conversion-Georectify Airborne Datacube-Convert Radiance Cube to Reflectance from Measured Reference Spectrum.bip.hdr'
>>> io = hsio(fname_in)

Retrieve the image band at 800 nm using hstools.get_band and hsio.spyfile.open_memmap, then mask out all pixels whose value falls below the 75th percentile.

>>> band = io.tools.get_band(800)
>>> array = io.spyfile.open_memmap()[:, :, band]
>>> array_mask, metadata = io.tools.mask_array(array, io.spyfile.metadata, percentile=75, side='lower')

Calculate the spectral mean from the remaining (i.e., unmasked) pixels using hstools.mean_datacube.

>>> spec_mean, spec_std, datacube_masked = io.tools.mean_datacube(io.spyfile, mask=array_mask)

Save using hsio.write_spec and hsio.write_cube, then load into Spectronon software for visualization.

>>> fname_hdr_spec = r'F:\nigo0024\Documents\hs_process_demo\hstools\Wells_rep2_20180628_16h56m_pika_gige_7-mean_800nm_75th.spec.hdr'
>>> fname_hdr_cube = r'F:\nigo0024\Documents\hs_process_demo\hstools\Wells_rep2_20180628_16h56m_pika_gige_7-mean_800nm_75th.bip.hdr'
>>> io.write_spec(fname_hdr_spec, spec_mean, spec_std, metadata=metadata, force=True)
Saving F:\nigo0024\Documents\hs_process_demo\hstools\Wells_rep2_20180628_16h56m_pika_gige_7-mean_800nm_75th.spec
>>> io.write_cube(fname_hdr_cube, datacube_masked, metadata=metadata, force=True)
Saving F:\nigo0024\Documents\hs_process_demo\hstools\Wells_rep2_20180628_16h56m_pika_gige_7-mean_800nm_75th.bip

modify_meta_set(meta_set, idx, value)

Modifies metadata “set” (i.e., string representation of a Python set; common in .hdr files) by converting string to list, then adjusts the value of an item by its index.

Parameters

• meta_set (str) – the string representation of the metadata set

• idx (int) – index to be modified; if None, the whole meta_set is returned (default: None).

• value (float, int, or str) – value to replace at idx

Returns

set_str (str): Modified metadata set string.

Return type

str

Example

Load and initialize hsio

>>> from hs_process import hsio
>>> fname_in = r'F:\nigo0024\Documents\hs_process_demo\Wells_rep2_20180628_16h56m_pika_gige_7-Radiance Conversion-Georectify Airborne Datacube-Convert Radiance Cube to Reflectance from Measured Reference Spectrum.bip.hdr'
>>> io = hsio(fname_in)

Retrieve the “map info” set from the metadata via hstools.get_meta_set

>>> map_info_set = io.spyfile.metadata['map info']
>>> map_info_set
['UTM',
 '1.0',
 '1.0',
 '441357.287073',
 '4855944.7717699995',
 '0.04',
 '0.04',
 '15',
 'T',
 'WGS-84',
 'units  meters',
 'rotation  0.000']

Modify the value at index position 4 from 4855944.7717699995 to 441300.2 using hstools.modify_meta_set.

>>> io.tools.modify_meta_set(map_info_set, idx=4, value=441300.2)
'{UTM, 1.0, 1.0, 441357.287073, 441300.2, 0.04, 0.04, 15, T, WGS-84, units  meters, rotation  0.000}'

plot_histogram(array, fname_fig=None, title=None, xlabel=None, percentile=90, bins=50, fontsize=16, color='#444444')

Plots a histogram with the percentile value labeled.

Parameters

• array (numpy.ndarray) – The data array used to create the histogram for; if array is masked, the masked pixels are excluded from the histogram.

• fname_fig (str, optional) – The filename to save the figure to; if None, the figure will not be saved (default: None).

• title (str, optional) – The plot title (default: None).

• xlabel (str, optional) – The x-axis label of the histogram (default: None).

• percentile (scalar, optional) – The percentile to label and illustrate on the histogram; if percentile = 90, the band/spectral index value at the 90th percentile will be labeled on the plot (default: 90; range: 0-100).

• bins (int, optional) – Number of histogram bins (default: 50).

• fontsize (scalar) – Font size of the axes labels. The title and text annotations will be scaled relatively (default: 16).

• color (str, optional) – Color of the histogram columns (default: “#444444”)

Returns

Figure object showing the histogram.

Return type

fig (matplotlib.figure)

Example

Load and initialize hsio

>>> from hs_process import hsio
>>> fname_in = r'F:\nigo0024\Documents\hs_process_demo\Wells_rep2_20180628_16h56m_pika_gige_7-Radiance Conversion-Georectify Airborne Datacube-Convert Radiance Cube to Reflectance from Measured Reference Spectrum.bip.hdr'
>>> io = hsio(fname_in)

Retrieve the image band at 800 nm using hstools.get_band and hsio.spyfile.open_memmap

>>> band = io.tools.get_band(800)
>>> array = io.spyfile.open_memmap()[:, :, band]

Create a masked array of all values below the 5th percentile via hstools.mask_array

>>> array_mask, metadata = io.tools.mask_array(array, io.spyfile.metadata, percentile=5, side='lower')

Visualize the histogram of the unmasked pixels (i.e., those greater than the 5th percentile) using hstools.plot_histogram

>>> title = 'Reflectance at 800 nm'
>>> xlabel = 'Reflectance (%)'
>>> fig = io.tools.plot_histogram(array_mask, title=title, xlabel=xlabel)

class hs_process.segment(spyfile)

Bases: object

Class for aiding in the segmentation and/or masking of image data to filter out pixels that are of least interest.

band_math_derivative(wl1=None, wl2=None, wl3=None, b1=None, b2=None, b3=None, spyfile=None, list_range=True, print_out=True)

Calculates a derivative-type spectral index from two input bands and/or wavelengths. Bands/wavelengths can be input as two individual bands, two sets of bands (i.e., list of bands), or range of bands (i.e., list of two bands indicating the lower and upper range).

Definition:

array_der = (wl1 - wl2) / (wl2 - wl3)

Parameters

• wl1 (int, float, or list) – the wavelength (or set of wavelengths) to be used as the first parameter of the derivative index; if list, then consolidates all bands between two wavelength values by calculating the mean pixel value across all bands in that range (default: None).

• wl2 (int, float, or list) – the wavelength (or set of wavelengths) to be used as the second parameter of the derivative index; if list, then consolidates all bands between two wavelength values by calculating the mean pixel value across all bands in that range (default: None).

• wl3 (int, float, or list) – the wavelength (or set of wavelengths) to be used as the third parameter of the derivative index; if list, then consolidates all bands between two wavelength values by calculating the mean pixel value across all bands in that range (default: None).

• b1 (int, float, or list) – the band (or set of bands) to be used as the first parameter of the derivative index; if list, then consolidates all bands between two band values by calculating the mean pixel value across all bands in that range (default: None).

• b2 (int, float, or list) – the band (or set of bands) to be used as the second parameter of the derivative index; if list, then consolidates all bands between two band values by calculating the mean pixel value across all bands in that range (default: None).

• b3 (int, float, or list) – the band (or set of bands) to be used as the third parameter of the derivative index; if list, then consolidates all bands between two band values by calculating the mean pixel value across all bands in that range (default: None).

• spyfile (SpyFile object or numpy.ndarray) – The datacube to process; if numpy.ndarray or None, loads band information from self.spyfile (default: None).

• list_range (bool) – Whether bands/wavelengths passed as a list is interpreted as a range of bands (True) or for each individual band in the list (False). If list_range is True, b1/wl1 and b2/wl2 should be lists with two items, and all bands/wavelegths between the two values will be used (default: True).

• print_out (bool) – Whether to print out the actual bands and wavelengths being used in the NDI calculation (default: True).

Returns

2-element tuple containing

• array_der (numpy.ndarray): Derivative band math array.

• metadata (dict): Modified metadata describing the derivative array (array_der).

Example

Load hsio and segment modules

>>> import numpy as np
>>> import os
>>> from hs_process import hsio
>>> from hs_process import segment
>>> data_dir = r'F:\nigo0024\Documents\hs_process_demo'
>>> fname_in = os.path.join(data_dir, 'Wells_rep2_20180628_16h56m_pika_gige_7-Radiance Conversion-Georectify Airborne Datacube-Convert Radiance Cube to Reflectance from Measured Reference Spectrum.bip.hdr')
>>> io = hsio(fname_in)
>>> my_segment = segment(io.spyfile)

Calculate the MERIS Terrestrial Chlorophyll Index (MTCI; Dash and Curran, 2004) via segment.band_math_derivative

>>> array_mtci, metadata = my_segment.band_math_derivative(wl1=754, wl2=709, wl3=681, spyfile=io.spyfile)
Band 1: [176]  Band 2: [154]  Band 3: [141]
Wavelength 1: [753.84]  Wavelength 2: [708.6784]  Wavelength 3: [681.992]

>>> array_mtci.shape
(617, 1300)
>>> np.nanmean(array_mtci)
9.401104

Show MTCI image via hsio.show_img

>>> io.show_img(array_mtci, vmin=-2, vmax=15)

band_math_mcari2(wl1=None, wl2=None, wl3=None, b1=None, b2=None, b3=None, spyfile=None, list_range=True, print_out=True)

Calculates the MCARI2 (Modified Chlorophyll Absorption Ratio Index Improved; Haboudane et al., 2004) spectral index from three input bands and/or wavelengths. Bands/wavelengths can be input as two individual bands, two sets of bands (i.e., list of bands), or range of bands (i.e., list of two bands indicating the lower and upper range).

Definition:

array_mcari2 = ((1.5 * (2.5 * (wl1 - wl2) - 1.3 * (wl1 - wl3))) / np.sqrt((2 * wl1 + 1)**2 - (6 * wl1 - 5 * np.sqrt(wl2)) - 0.5))

Parameters

• wl1 (int, float, or list) – the wavelength (or set of wavelengths) to be used as the first parameter of the MCARI2 index; if list, then consolidates all bands between two wavelength values by calculating the mean pixel value across all bands in that range (default: None).

• wl2 (int, float, or list) – the wavelength (or set of wavelengths) to be used as the second parameter of the MCARI2 index; if list, then consolidates all bands between two wavelength values by calculating the mean pixel value across all bands in that range (default: None).

• wl3 (int, float, or list) – the wavelength (or set of wavelengths) to be used as the third parameter of the MCARI2 index; if list, then consolidates all bands between two wavelength values by calculating the mean pixel value across all bands in that range (default: None).

• b1 (int, float, or list) – the band (or set of bands) to be used as the first parameter of the MCARI2 index; if list, then consolidates all bands between two band values by calculating the mean pixel value across all bands in that range (default: None).

• b2 (int, float, or list) – the band (or set of bands) to be used as the second parameter of the MCARI2 index; if list, then consolidates all bands between two band values by calculating the mean pixel value across all bands in that range (default: None).

• b3 (int, float, or list) – the band (or set of bands) to be used as the third parameter of the MCARI2 index; if list, then consolidates all bands between two band values by calculating the mean pixel value across all bands in that range (default: None).

• spyfile (SpyFile object or numpy.ndarray) – The datacube to process; if numpy.ndarray or None, loads band information from self.spyfile (default: None).

• list_range (bool) – Whether bands/wavelengths passed as a list is interpreted as a range of bands (True) or for each individual band in the list (False). If list_range is True, b1/wl1 and b2/wl2 should be lists with two items, and all bands/wavelegths between the two values will be used (default: True).

• print_out (bool) – Whether to print out the actual bands and wavelengths being used in the NDI calculation (default: True).

Returns

2-element tuple containing

• array_mcari2 (numpy.ndarray): MCARI2 spectral index band math array.

• metadata (dict): Modified metadata describing the MCARI2 index array (array_mcari2).

Example

Load hsio and segment modules

>>> import numpy as np
>>> import os
>>> from hs_process import hsio
>>> from hs_process import segment
>>> data_dir = r'F:\nigo0024\Documents\hs_process_demo'
>>> fname_in = os.path.join(data_dir, 'Wells_rep2_20180628_16h56m_pika_gige_7-Radiance Conversion-Georectify Airborne Datacube-Convert Radiance Cube to Reflectance from Measured Reference Spectrum.bip.hdr')
>>> io = hsio(fname_in)
>>> my_segment = segment(io.spyfile)

Calculate the MCARI2 spectral index (Haboudane et al., 2004) via segment.band_math_mcari2

>>> array_mcari2, metadata = my_segment.band_math_mcari2(wl1=800, wl2=670, wl3=550, spyfile=io.spyfile)
Band 1: [198]  Band 2: [135]  Band 3: [77]
Wavelength 1: [799.0016]  Wavelength 2: [669.6752]  Wavelength 3: [550.6128]

>>> np.nanmean(array_mcari2)
0.57376945

Show MCARI2 image via hsio.show_img

>>> io.show_img(array_mcari2)

band_math_ndi(wl1=None, wl2=None, b1=None, b2=None, spyfile=None, list_range=True, print_out=True)

Calculates a normalized difference spectral index from two input bands and/or wavelengths. Bands/wavelengths can be input as two individual bands, two sets of bands (i.e., list of bands), or range of bands (i.e., list of two bands indicating the lower and upper range).

Definition:

array_ndi = (wl1 - wl2) / (wl1 + wl2)

Parameters

• wl1 (int, float, or list) – the wavelength (or set of wavelengths) to be used as the first parameter of the normalized difference index; if list, then consolidates all bands between two wavelength values by calculating the mean pixel value across all bands in that range (default: None).

• wl2 (int, float, or list) – the wavelength (or set of wavelengths) to be used as the second parameter of the normalized difference index; if list, then consolidates all bands between two wavelength values by calculating the mean pixel value across all bands in that range (default: None).

• b1 (int, float, or list) – the band (or set of bands) to be used as the first parameter of the normalized difference index; if list, then consolidates all bands between two band values by calculating the mean pixel value across all bands in that range (default: None).

• b2 (int, float, or list) – the band (or set of bands) to be used as the second parameter of the normalized difference index; if list, then consolidates all bands between two band values by calculating the mean pixel value across all bands in that range (default: None).

• spyfile (SpyFile object or numpy.ndarray) – The datacube to process; if numpy.ndarray or None, loads band information from self.spyfile (default: None).

• list_range (bool) – Whether bands/wavelengths passed as a list is interpreted as a range of bands (True) or for each individual band in the list (False). If list_range is True, b1/wl1 and b2/wl2 should be lists with two items, and all bands/wavelegths between the two values will be used (default: True).

• print_out (bool) – Whether to print out the actual bands and wavelengths being used in the NDI calculation (default: True).

Returns

2-element tuple containing

• array_ndi (numpy.ndarray): Normalized difference band math array.

• metadata (dict): Modified metadata describing the normalized difference array (array_ndi).

Example

Load hsio and segment modules

>>> import numpy as np
>>> import os
>>> from hs_process import hsio
>>> from hs_process import segment
>>> data_dir = r'F:\nigo0024\Documents\hs_process_demo'
>>> fname_in = os.path.join(data_dir, 'Wells_rep2_20180628_16h56m_pika_gige_7-Radiance Conversion-Georectify Airborne Datacube-Convert Radiance Cube to Reflectance from Measured Reference Spectrum.bip.hdr')
>>> io = hsio(fname_in)
>>> my_segment = segment(io.spyfile)

Calculate the Normalized difference vegetation index using 10 nm bands centered at 800 nm and 680 nm via segment.band_math_ndi

>>> array_ndvi, metadata = my_segment.band_math_ndi(wl1=[795, 805], wl2=[675, 685], spyfile=io.spyfile)
Effective NDI: (800 - 680) / 800 + 680)

>>> np.nanmean(array_ndvi)
0.8184888

Show NDVI image via hsio.show_img

>>> io.show_img(array_ndvi)

band_math_ratio(wl1=None, wl2=None, b1=None, b2=None, spyfile=None, list_range=True, print_out=True)

Calculates a simple ratio spectral index from two input band and/or wavelengths. Bands/wavelengths can be input as two individual bands, two sets of bands (i.e., list of bands), or a range of bands (i.e., list of two bands indicating the lower and upper range).

Definition:

array_ratio = (wl1/wl2)

Parameters

• wl1 (int, float, or list) – the wavelength (or set of wavelengths) to be used as the first parameter of the ratio index; if list, then consolidates all bands between two wavelength values by calculating the mean pixel value across all bands in that range (default: None).

• wl2 (int, float, or list) – the wavelength (or set of wavelengths) to be used as the second parameter of the ratio index; if list, then consolidates all bands between two wavelength values by calculating the mean pixel value across all bands in that range (default: None).

• b1 (int, float, or list) – the band (or set of bands) to be used as the first parameter (numerator) of the ratio index; if list, then consolidates all bands between two band values by calculating the mean pixel value across all bands in that range (default: None).

• b2 (int, float, or list) – the bands (or set of bands) to be used as the second parameter (denominator) of the ratio index; if list, then consolidates all bands between two bands values by calculating the mean pixel value across all bands in that range (default: None).

• spyfile (SpyFile object or numpy.ndarray) – The datacube to process; if numpy.ndarray or None, loads band information from self.spyfile (default: None).

• list_range (bool) – Whether a band passed as a list is interpreted as a range of bands (True) or for each individual band in the list (False). If list_range is True, b1/wl1 and b2/wl2 should be lists with two items, and all bands/wavelegths between the two values will be used (default: True).

• print_out (bool) – Whether to print out the actual bands and wavelengths being used in the NDI calculation (default: True).

Returns

2-element tuple containing

• array_ratio (numpy.ndarray): Ratio band math array.

• metadata (dict): Modified metadata describing the ratio array (array_ratio).

Example

Load hsio and segment modules

>>> import numpy as np
>>> import os
>>> from hs_process import hsio
>>> from hs_process import segment
>>> data_dir = r'F:\nigo0024\Documents\hs_process_demo'
>>> fname_in = os.path.join(data_dir, 'Wells_rep2_20180628_16h56m_pika_gige_7-Radiance Conversion-Georectify Airborne Datacube-Convert Radiance Cube to Reflectance from Measured Reference Spectrum.bip.hdr')
>>> io = hsio(fname_in)
>>> my_segment = segment(io.spyfile)

Calculate a red/near-infrared band ratio using a range of bands (i.e., mimicking a broadband sensor) via segment.band_math_ratio

>>> array_ratio, metadata = my_segment.band_math_ratio(wl1=[630, 690], wl2=[800, 860], list_range=True)
Effective band ratio: (659/830)

>>> np.nanmean(array_ratio)
0.10981177

Notice that 29 spectral bands were consolidated (i.e., averaged) to mimic a single broad band. We can take the mean of two bands by changing list_range to False, and this slightly changes the result.

>>> array_ratio, metadata = my_segment.band_math_ratio(wl1=[630, 690], wl2=[800, 860], list_range=False)
Effective band ratio: (660/830)

>>> np.nanmean(array_ratio)
0.113607444

Show the red/near-infrared ratio image via hsio.show_img

>>> io.show_img(array_ratio, vmax=0.3)

composite_band(wl1=None, b1=None, spyfile=None, list_range=True, print_out=True)

Calculates a composite band from a range of bands or wavelengths. Bands/wavelengths can be input as individual bands, a set of bands (i.e., list of bands), or a range of bands (i.e., list of two bands indicating the lower and upper range).

Parameters

• wl1 (int, float, or list) – the wavelength (or set of wavelengths) to consolidate; if list, then all wavelengths between two wavelength values are consolidated by calculating the mean pixel value across all wavelengths in that range (default: None).

• b1 (int, float, or list) – the band (or set of bands) to consolidate; if list, then all bands between two band values are consolidated by calculating the mean pixel value across all bands in that range (default: None).

• spyfile (SpyFile object or numpy.ndarray) – The datacube to process; if numpy.ndarray or None, loads band information from self.spyfile (default: None).

• list_range (bool) – Whether a band passed as a list is interpreted as a range of bands (True) or for each individual band in the list (False). If list_range is True, b1/wl1 and b2/wl2 should be lists with two items, and all bands/wavelegths between the two values will be used (default: True).

• print_out (bool) – Whether to print out the actual bands and wavelengths being used in the NDI calculation (default: True).

Returns

2-element tuple containing

• array_b1 (numpy.ndarray): Consolidated array.

• metadata (dict): Modified metadata describing the consolidated array (array_b1).

load_spyfile(spyfile)

Loads a SpyFile (Spectral Python object) for data access and/or manipulation by the hstools class.

Parameters

spyfile (SpyFile object) – The datacube being accessed and/or manipulated.

Example

Load hsio and segment modules

>>> from hs_process import hsio
>>> from hs_process import segment
>>> fname_in = r'F:\nigo0024\Documents\hs_process_demo\Wells_rep2_20180628_16h56m_pika_gige_7-Convert Radiance Cube to Reflectance from Measured Reference Spectrum.bip.hdr'
>>> io = hsio(fname_in)
>>> my_segment = segment(io.spyfile)

Load datacube via segment.load_spyfile

>>> my_segment.load_spyfile(io.spyfile)
>>> my_segment.spyfile
Data Source:   'F:\nigo0024\Documents\hs_process_demo\Wells_rep2_20180628_16h56m_pika_gige_7-Convert Radiance Cube to Reflectance from Measured Reference Spectrum.bip'
    # Rows:            617
    # Samples:        1300
    # Bands:           240
    Interleave:        BIP
    Quantization:  32 bits
    Data format:   float32

class hs_process.spatial_mod(spyfile, gdf=None, **kwargs)

Bases: object

Class for manipulating data within the spatial domain (e.g., cropping a datacube by a geographical boundary).

crop_many_gdf(spyfile=None, gdf=None, **kwargs)

Crops many plots from a single image by comparing the image to a polygon file (geopandas.GoeDataFrame) that contains plot information and geometry of plot boundaries.

Parameters

• spyfile (SpyFile object, optional) – The datacube to crop; if None, loads datacube and band information from spatial_mod.spyfile (default: None).

• gdf (geopandas.GeoDataFrame, optional) – the plot IDs and polygon geometery of each of the plots; ‘plot_id’ must be used as a column name to identify each of the plots, and should be an integer; if None, loads geodataframe from spatial_mod.gdf (default: None).

• kwargs –

  Can be any of the keys in self.defaults.crop_defaults: plot_id_ref (int, optional): the plot ID of the reference plot.

  plot_id_ref is required if passing pix_e_ul, pix_n_ul, or n_plots because it is used as the reference point for any of the adjustments/modifications dictated by said parameters. plot_id_ref must be present in the gdf, and the extent of plot_id_ref must intersect the extent of the datacube (default: None).

  pix_e_ul (int, optional): upper left pixel column (easting) of

  plot_id_ref; this is used to calculate the offset between the GeoDataFrame geometry and the approximate image georeference error (default: None).

  pix_n_ul (int, optional): upper left pixel row (northing) of

  plot_id_ref; this is used to calculate the offset between the GeoDataFrame geometry and the approximate image georeference error (default: None).

  crop_e_m (float, optional): length of each row (easting

  direction) of the cropped image in map units (e.g., meters; default: None).

  crop_n_m (float, optional): length of each column (northing

  direction) of the cropped image in map units (e.g., meters; default: None)

  crop_e_pix (int, optional): number of pixels in each row in the

  cropped image (default: None).

  crop_n_pix (int, optional): number of pixels in each column in

  the cropped image (default: None).

  buf_e_m (float, optional): The buffer distance in the easting

  direction (in map units; e.g., meters) to be applied after calculating the original crop area; the buffer is considered after crop_X_m / crop_X_pix. A positive value will reduce the size of crop_X_m / crop_X_pix, and a negative value will increase it (default: None).

  buf_n_m (float, optional): The buffer distance in the northing

  direction (in map units; e.g., meters) to be applied after calculating the original crop area; the buffer is considered after crop_X_m / crop_X_pix. A positive value will reduce the size of crop_X_m / crop_X_pix, and a negative value will increase it (default: None).

  buf_e_pix (int, optional): The buffer distance in the easting

  direction (in pixel units) to be applied after calculating the original crop area (default: None).

  buf_n_pix (int, optional): The buffer distance in the northing

  direction (in pixel units) to be applied after calculating the original crop area (default: None).

  gdf_shft_e_m (float): The distance to shift the cropped

  datacube from the upper left/NW plot corner in the east direction (negative value will shift to the west). Only relevent when gdf is passed. This shift is applied after the offset is applied from buf_X (default: 0.0).

  gdf_shft_n_m (float): The distance to shift the cropped

  datacube from the upper left/NW plot corner in the north direction (negative value will shift to the south). Only relevent when gdf is passed. This shift is applied after the offset is applied from buf_X (default: 0.0).

  gdf_shft_e_pix (int): The pixel units to shift the cropped

  datacube from the upper left/NW plot corner in the east direction (negative value will shift to the west). Only relevent when gdf is passed. This shift is applied after the offset is applied from buf_X (default: 0.0).

  gdf_shft_n_pix (int): The pixel units to shift the cropped

  datacube from the upper left/NW plot corner in the north direction (negative value will shift to the south). Only relevent when gdf is passed. This shift is applied after the offset is applied from buf_X (default: 0.0).

  n_plots (int, optional): number of plots to crop, starting with

  plot_id_ref and moving from West to East and North to South. This can be used to limit the number of cropped plots (default; None).

Returns

• df_plots (pandas.DataFrame) – data for which to crop each plot; includes ‘plot_id_ref’, ‘pix_e_ul’, and ‘pix_n_ul’ columns. This data can be passed to spatial_mod.crop_single to perform the actual cropping.

Return type

pandas.DataFrame

Note

If pix_e_ul or pix_n_ul are passed, the pixel offset from the northwest corner of plot_id_ref will be calculated. This offset is then applied to all plots within the extent of the image to systematically shift the actual upper left pixel locations for each plot, effectively shifting the easting and/or northing of the upper left pixel of the hyperspectral datacube to match that of the gdf. If the shift should only apply to a select number of plots, n_plots can be passed to restrict the number of plots that are processed.

Note

Either the pixel coordinate or the map unit coordinate should be passed for crop_X_Y and buf_X_Y in each direction (i.e., easting and northing). Do not pass both.

Example

Load the hsio and spatial_mod modules

>>> import geopandas as gpd
>>> import os
>>> from hs_process import hsio
>>> from hs_process import spatial_mod

Read datacube and spatial plot boundaries

>>> fname_in = r'F:\nigo0024\Documents\hs_process_demo\Wells_rep2_20180628_16h56m_pika_gige_7-Radiance Conversion-Georectify Airborne Datacube-Convert Radiance Cube to Reflectance from Measured Reference Spectrum.bip.hdr'
>>> fname_in = r'F:\nigo0024\Documents\hs_process_demo\Wells_rep2_20180628_16h56m_pika_gige_7_nohist-Radiance Conversion-Georectify Airborne Datacube-Convert Radiance Cube to Reflectance from Measured Reference Spectrum.bip.hdr'

>>> fname_gdf = r'F:\nigo0024\Documents\hs_process_demo\plot_bounds.geojson'
>>> gdf = gpd.read_file(fname_gdf)
>>> io = hsio(fname_in)
>>> my_spatial_mod = spatial_mod(
        io.spyfile, base_dir=io.base_dir, name_short=io.name_short,
        name_long=io.name_long)
>>> dir_out = os.path.join(io.base_dir, 'spatial_mod', 'crop_many_gdf')
>>> name_append = '-crop-many-gdf'

Get instructions on how plots should be cropped via spatial_mod.crop_many_gdf(); note that a pandas.DataFrame is returned with information describing how each plot should be cropped.

>>> df_plots = my_spatial_mod.crop_many_gdf(spyfile=io.spyfile, gdf=gdf)
>>> df_plots.head(5)
    plot_id_ref  pix_e_ul  pix_n_ul  crop_e_pix  crop_n_pix
0          1018       478         0         229          76
1           918       707         0         229          76
2           818       936         0         229          76
3           718      1165         0         229          76
4           618      1394         0         229          76
...

Use the data from the first row of df_plots to crop a single plot from the original image (uses spatial_mod.crop_single)

>>> pix_e_ul=113
>>> pix_n_ul=0
>>> crop_e_pix=229
>>> crop_n_pix=75
>>> plot_id_ref=1018
>>> array_crop, metadata = my_spatial_mod.crop_single(
        pix_e_ul=pix_e_ul, pix_n_ul=pix_n_ul, crop_e_pix=crop_e_pix, crop_n_pix=crop_n_pix,
        spyfile=io.spyfile, plot_id_ref=plot_id_ref)

Save the cropped datacube and geotiff to a new directory

>>> fname_out = os.path.join(dir_out, io.name_short + '_plot_' + str(1018) + name_append + '.' + io.defaults.envi_write.interleave)
>>> fname_out_tif = os.path.join(dir_out, io.name_short + '_plot_' + str(1018) + '.tif')
>>> io.write_cube(fname_out, array_crop, metadata=metadata, force=True)
>>> io.write_tif(fname_out_tif, spyfile=array_crop, metadata=metadata)
Saving F:\nigo0024\Documents\hs_process_demo\spatial_mod\crop_many_gdf\Wells_rep2_20180628_16h56m_pika_gige_7_plot_1018-crop-many-gdf.bip
Either `projection_out` is `None` or `geotransform_out` is `None` (or both are). Retrieving projection and geotransform information by loading `hsio.fname_in` via GDAL. Be sure this is appropriate for the data you are trying to write.

Using a for loop, use spatial_mod.crop_single and hsio.write_cube to crop by plot and save cropped datacubes to file

>>> for idx, row in df_plots.iterrows():
>>>     io.read_cube(fname_in, name_long=io.name_long,
                     name_plot=row['plot_id_ref'],
                     name_short=io.name_short)
>>>     my_spatial_mod.load_spyfile(io.spyfile)
>>>     array_crop, metadata = my_spatial_mod.crop_single(
                pix_e_ul=row['pix_e_ul'], pix_n_ul=row['pix_n_ul'],
                crop_e_pix=row['crop_e_pix'], crop_n_pix=row['crop_n_pix'],
                buf_e_m=2.0, buf_n_m=0.75,
                plot_id_ref=row['plot_id_ref'])
>>>     fname_out = os.path.join(dir_out, io.name_short + '_plot_' + str(row['plot_id_ref']) + name_append + '.bip.hdr')
>>>     fname_out_tif = os.path.join(dir_out, io.name_short + '_plot_' + str(row['plot_id_ref']) + '.tif')
>>>     io.write_cube(fname_out, array_crop, metadata=metadata, force=True)  # force=True to overwrite the plot_1018 image
>>>     io.write_tif(fname_out_tif, spyfile=array_crop, metadata=metadata)
Saving F:\nigo0024\Documents\hs_process_demo\crop_many_gdf\Wells_rep2_20180628_16h56m_pika_gige_7_plot_1018.bip
Saving F:\nigo0024\Documents\hs_process_demo\crop_many_gdf\Wells_rep2_20180628_16h56m_pika_gige_7_plot_918.bip
Saving F:\nigo0024\Documents\hs_process_demo\crop_many_gdf\Wells_rep2_20180628_16h56m_pika_gige_7_plot_818.bip
Saving F:\nigo0024\Documents\hs_process_demo\crop_many_gdf\Wells_rep2_20180628_16h56m_pika_gige_7_plot_718.bip
Saving F:\nigo0024\Documents\hs_process_demo\crop_many_gdf\Wells_rep2_20180628_16h56m_pika_gige_7_plot_618.bip
Saving F:\nigo0024\Documents\hs_process_demo\crop_many_gdf\Wells_rep2_20180628_16h56m_pika_gige_7_plot_1017.bip
Saving F:\nigo0024\Documents\hs_process_demo\crop_many_gdf\Wells_rep2_20180628_16h56m_pika_gige_7_plot_917.bip
Saving F:\nigo0024\Documents\hs_process_demo\crop_many_gdf\Wells_rep2_20180628_16h56m_pika_gige_7_plot_817.bip
Saving F:\nigo0024\Documents\hs_process_demo\crop_many_gdf\Wells_rep2_20180628_16h56m_pika_gige_7_plot_717.bip
Saving F:\nigo0024\Documents\hs_process_demo\crop_many_gdf\Wells_rep2_20180628_16h56m_pika_gige_7_plot_617.bip
...

Open cropped geotiff images in QGIS to visualize the extent of the cropped images compared to the original datacube and the plot boundaries (the full extent image is darkened and displayed in the background:

crop_single(pix_e_ul=0, pix_n_ul=0, crop_e_pix=None, crop_n_pix=None, crop_e_m=None, crop_n_m=None, buf_e_pix=None, buf_n_pix=None, buf_e_m=None, buf_n_m=None, spyfile=None, plot_id_ref=None, gdf=None, gdf_shft_e_pix=None, gdf_shft_n_pix=None, gdf_shft_e_m=None, gdf_shft_n_m=None, name_append='spatial-crop-single')

Crops a single plot from an image. If plot_id_ref and gdf are explicitly passed (i.e., they will not be loaded from spatial_mod class), the “map info” tag in the metadata will be adjusted to center the cropped area within the appropriate plot geometry.

Parameters

• pix_e_ul (int, optional) – upper left pixel column (easting) to begin cropping (default: 0).

• pix_n_ul (int, optional) – upper left pixel row (northing) to begin cropping (default: 0).

• crop_e_m (float, optional) – length of each row (easting direction) of the cropped image in map units (e.g., meters; default: None).

• crop_n_m (float, optional) – length of each column (northing direction) of the cropped image in map units (e.g., meters; default: None)

• crop_e_pix (int, optional) – number of pixels in each row in the cropped image (default: None).

• crop_n_pix (int, optional) – number of pixels in each column in the cropped image (default: None).

• buf_e_m (float, optional) – The buffer distance in the easting direction (in map units; e.g., meters) to be applied after calculating the original crop area; the buffer is considered after crop_X_m / crop_X_pix. A positive value will reduce the size of crop_X_m / crop_X_pix, and a negative value will increase it (default: None).

• buf_n_m (float, optional) – The buffer distance in the northing direction (in map units; e.g., meters) to be applied after calculating the original crop area; the buffer is considered after crop_X_m / crop_X_pix. A positive value will reduce the size of crop_X_m / crop_X_pix, and a negative value will increase it (default: None).

• buf_e_pix (int, optional) – The buffer distance in the easting direction (in pixel units) to be applied after calculating the original crop area (default: None).

• buf_n_pix (int, optional) – The buffer distance in the northing direction (in pixel units) to be applied after calculating the original crop area (default: None).

• spyfile (SpyFile object or numpy.ndarray) – The datacube to crop; if numpy.ndarray or None, loads band information from self.spyfile (default: None).

• plot_id_ref (int) – the plot ID of the area to be cropped (default: None).

• gdf (geopandas.GeoDataFrame) – the plot names and polygon geometery of each of the plots; ‘plot_id’ must be used as a column name to identify each of the plots, and should be an integer.

• gdf_shft_e_m (float) – The distance to shift the cropped datacube from the upper left/NW plot corner in the east direction (negative value will shift to the west). Only relevent when gdf is passed. This shift is applied after the offset is applied from buf_X (default: 0.0).

• gdf_shft_n_m (float) – The distance to shift the cropped datacube from the upper left/NW plot corner in the north direction (negative value will shift to the south). Only relevent when gdf is passed. This shift is applied after the offset is applied from buf_X (default: 0.0).

• gdf_shft_e_pix (int) – The pixel units to shift the cropped datacube from the upper left/NW plot corner in the east direction (negative value will shift to the west). Only relevent when gdf is passed. This shift is applied after the offset is applied from buf_X (default: 0.0).

• gdf_shft_n_pix (int) – The pixel units to shift the cropped datacube from the upper left/NW plot corner in the north direction (negative value will shift to the south). Only relevent when gdf is passed. This shift is applied after the offset is applied from buf_X (default: 0.0).

• name_append (str) – NOT YET SUPPORTED; name to append to the filename (default: ‘spatial-crop-single’).

Returns

2-element tuple containing

• array_crop (numpy.ndarray): Cropped datacube.

• metadata (dict): Modified metadata describing the cropped hyperspectral datacube (array_crop).

Example

Load and initialize the hsio and spatial_mod modules

>>> from hs_process import hsio
>>> from hs_process import spatial_mod
>>> fname_in = r'F:\nigo0024\Documents\hs_process_demo\Wells_rep2_20180628_16h56m_pika_gige_7-Convert Radiance Cube to Reflectance from Measured Reference Spectrum.bip.hdr'
>>> io = hsio(fname_in)
>>> my_spatial_mod = spatial_mod(
        io.spyfile, base_dir=io.base_dir, name_short=io.name_short,
        name_long=io.name_long)

Crop an area with a width (easting) 200 pixels and a height (northing) of 50 pixels, with a northwest/upper left origin at the 342nd column (easting) and 75th row (northing).

>>> pix_e_ul = 342
>>> pix_n_ul = 75
>>> array_crop, metadata = my_spatial_mod.crop_single(pix_e_ul, pix_n_ul, crop_e_pix=200, crop_n_pix=50)

Save as a geotiff using io.write_tif, then load into QGIS to visualize.

>>> fname_tif = r'F:\nigo0024\Documents\hs_process_demo\spatial_mod\crop_single\crop_single.tif'
>>> io.write_tif(fname_tif, array_crop, metadata=metadata)
Either `projection_out` is `None` or `geotransform_out` is `None` (or both are). Retrieving projection and geotransform information by loading `hsio.fname_in` via GDAL. Be sure this is appropriate for the data you are trying to write.

Open cropped geotiff image in QGIS to visualize the extent of the cropped image compared to the original datacube and the plot boundaries (the full extent image is darkened and displayed in the background):

load_spyfile(spyfile, **kwargs)

Loads a SpyFile (Spectral Python object) for data access and/or manipulation by the hstools class.

Parameters

• spyfile (SpyFile object) – The datacube being accessed and/or manipulated.

• base_dir (str) – to be used by the plot gdf attribute data.

• name_short (str) – to be used by the plot gdf attribute data.

• name_long (str) – to be used by the plot gdf attribute data.

Example

Load and initialize the hsio and spatial_mod modules

>>> from hs_process import hsio
>>> from hs_process import spatial_mod
>>> fname_in = r'F:\nigo0024\Documents\hs_process_demo\Wells_rep2_20180628_16h56m_pika_gige_7-Convert Radiance Cube to Reflectance from Measured Reference Spectrum.bip.hdr'
>>> io = hsio(fname_in)
>>> my_spatial_mod = spatial_mod(
        io.spyfile, base_dir=io.base_dir, name_short=io.name_short,
        name_long=io.name_long)

Load datacube using spatial_mod.load_spyfile

>>> my_spatial_mod.load_spyfile(
        io.spyfile, base_dir=io.base_dir, name_short=io.name_short,
        name_long=io.name_long)
>>> my_spatial_mod.spyfile
Data Source:   'F:\nigo0024\Documents\hs_process_demo\Wells_rep2_20180628_16h56m_pika_gige_7-Convert Radiance Cube to Reflectance from Measured Reference Spectrum.bip'
    # Rows:            617
    # Samples:        1300
    # Bands:           240
    Interleave:        BIP
    Quantization:  32 bits
    Data format:   float32

class hs_process.spec_mod(spyfile)

Bases: object

Class for manipulating data within the spectral domain, which is usually pixel-based.

load_spyfile(spyfile)

Loads a SpyFile (Spectral Python object) for data access and/or manipulation by the hstools class.

Parameters

spyfile (SpyFile object) – The datacube being accessed and/or manipulated.

Example

Load and initialize the hsio and spec_mod modules

>>> from hs_process import hsio
>>> from hs_process import spec_mod
>>> fname_in = r'F:\nigo0024\Documents\hs_process_demo\Wells_rep2_20180628_16h56m_pika_gige_7-Convert Radiance Cube to Reflectance from Measured Reference Spectrum.bip.hdr'
>>> io = hsio(fname_in)
>>> my_spec_mod = spec_mod(io.spyfile)

Load datacube

>>> my_spec_mod.load_spyfile(io.spyfile)
>>> my_spec_mod.spyfile
Data Source:   'F:\nigo0024\Documents\hs_process_demo\Wells_rep2_20180628_16h56m_pika_gige_7-Convert Radiance Cube to Reflectance from Measured Reference Spectrum.bip'
    # Rows:            617
    # Samples:        1300
    # Bands:           240
    Interleave:        BIP
    Quantization:  32 bits
    Data format:   float32

spec_derivative(spyfile_spec=None, order=1)

Calculates the numeric derivative spectra from spyfile_spec.

The derivavative spectra is calculated as the slope (rise over run) of the input spectra, and is normalized by the wavelength unit.

Parameters

• spyfile_spec – The spectral spyfile object to calculate the derivative for.

• order (int) – The order of the derivative (default: 1).

Example

Load and initialize hsio

>>> import os
>>> from hs_process import hsio
>>> from hs_process import spec_mod
>>> data_dir = r'F:\nigo0024\Documents\hs_process_demo'
>>> fname_hdr_spec = os.path.join(data_dir, 'Wells_rep2_20180628_16h56m_pika_gige_7_plot_611-cube-to-spec-mean.spec.hdr')
>>> io = hsio()
>>> io.read_spec(fname_hdr_spec)
>>> my_spec_mod = spec_mod(io.spyfile_spec)

Calculate the numeric derivative.

>>> spec_dydx, metadata_dydx = my_spec_mod.spec_derivative(order=1)

>>> io.write_spec('spec_derivative_order-1.spec.hdr', spec_dydx, df_std=None, metadata=metadata_dydx)

Plot the numeric derivative spectra and compare against the original spectra.

>>> import numpy as np
>>> import seaborn as sns
>>> sns.set_style("ticks")
>>> wl_x = np.array([float(i) for i in metadata_dydx['wavelength']])
>>> y_ref = io.spyfile_spec.open_memmap()[0,0,:]*100
>>> ax1 = sns.lineplot(wl_x, y_ref)
>>> ax2 = ax1.twinx()
>>> ax2 = sns.lineplot(wl_x, 0, ax=ax2, color='gray')
>>> ax2 = sns.lineplot(wl_x, spec_dydx[0,0,:]*100, ax=ax2, color=sns.color_palette()[1])
>>> ax2.set(ylim=(-0.8, 1.5))
>>> ax1.set_xlabel('Wavelength (nm)', weight='bold')
>>> ax1.set_ylabel('Reflectance (%)', weight='bold')
>>> ax2.set_ylabel('Reflectance derivative (%)', weight='bold')
>>> ax1.set_title(r'API Example: `hstools.spec_derivative`', weight='bold')

spectral_clip(wl_bands=[[0, 420], [760, 776], [813, 827], [880, 1000]], spyfile=None)

Removes/clips designated wavelength bands from the hyperspectral datacube.

Parameters

• wl_bands (list or list of lists) – minimum and maximum wavelenths to clip from image; if multiple groups of wavelengths should be cut, this should be a list of lists. For example, wl_bands=[760, 776] will clip all bands greater than 760.0 nm and less than 776.0 nm; wl_bands = [[0, 420], [760, 776], [813, 827], [880, 1000]] will clip all band less than 420.0 nm, bands greater than 760.0 nm and less than 776.0 nm, bands greater than 813.0 nm and less than 827.0 nm, and bands greater than 880 nm (default).

• spyfile (SpyFile object or numpy.ndarray) – The data cube to clip; if numpy.ndarray or None, loads band information from spec_mod.spyfile (default: None).

Returns

2-element tuple containing

• array_clip (numpy.ndarray): Clipped datacube.

• metadata (dict): Modified metadata describing the clipped hyperspectral datacube (array_clip).

Example

Load and initialize hsio and spec_mod

>>> import os
>>> from hs_process import hsio
>>> from hs_process import spec_mod
>>> data_dir = r'F:\nigo0024\Documents\hs_process_demo'
>>> fname_hdr = os.path.join(data_dir, 'Wells_rep2_20180628_16h56m_pika_gige_7-Radiance Conversion-Georectify Airborne Datacube-Convert Radiance Cube to Reflectance from Measured Reference Spectrum.bip.hdr')
>>> io = hsio()
>>> io.read_cube(fname_hdr)
>>> my_spec_mod = spec_mod(io.spyfile)

Using spec_mod.spectral_clip, clip all spectral bands below 420 nm and above 880 nm, as well as the bands near the oxygen absorption (i.e., 760-776 nm) and water absorption (i.e., 813-827 nm) regions.

>>> array_clip, metadata_clip = my_spec_mod.spectral_clip(
        wl_bands=[[0, 420], [760, 776], [813, 827], [880, 1000]])

Plot the spectra of the unclippe hyperspectral image and compare to that of the clipped image for a single pixel.

>>> import seaborn as sns
>>> from ast import literal_eval
>>> spy_hs = my_spec_mod.spyfile.open_memmap()  # datacube before smoothing
>>> meta_bands = list(io.tools.meta_bands.values())
>>> meta_bands_clip = sorted([float(i) for i in literal_eval(metadata_clip['wavelength'])])
>>> ax = sns.lineplot(x=meta_bands, y=spy_hs[200][800]*100, label='Before spectral clipping', linewidth=3)
>>> ax = sns.lineplot(x=meta_bands_clip, y=array_clip[200][800]*100, label='After spectral clipping', ax=ax)
>>> ax.set_xlabel('Wavelength (nm)', weight='bold')
>>> ax.set_ylabel('Reflectance (%)', weight='bold')
>>> ax.set_title(r'API Example: `spec_mod.spectral_clip`', weight='bold')

spectral_mimic(sensor='sentinel-2a', df_band_response=None, col_wl='wl_nm', center_wl='peak', spyfile=None)

Mimics the response of a multispectral sensor based on transmissivity of sensor bands across a range of wavelength values by calculating its weighted average response and interpolating the hyperspectral response.

Parameters:

sensor (str): Should be one of

[“sentera_6x”, “micasense_rededge_3”, “sentinel-2a”, “sentinel-2b”, “custom”]; if “custom”, df_band_response and col_wl must be passed.

df_band_response (pd.DataFrame): A DataFrame that contains the

transmissivity (%) for each sensor band (as columns) mapped to the continuous wavelength values (as rows). Required if sensor is “custom”, ignored otherwise.

col_wl (str): The column of df_band_response denoting the

wavlengths (default: ‘wl_nm’).

center_wl (str): Indicates how the center wavelength of each

band is determined. If center_wl is “peak”, the point at which transmissivity is at its maximum is used as the center wavelength. If center_wl is “weighted”, the weighted average is used to compute the center wavelength. Must be one of [“peak”, “weighted”] (default: "peak").

spyfile (SpyFile object): The datacube being accessed and/or

manipulated.

Returns:

2-element tuple containing

• array_multi (numpy.ndarray): Mimicked datacube.

• metadata (dict): Modified metadata describing the mimicked spectral array (array_multi).

Example:

Load and initialize hsio and spec_mod

>>> import os
>>> from hs_process import hsio
>>> from hs_process import spec_mod
>>> data_dir = r'F:\nigo0024\Documents\hs_process_demo'
>>> fname_hdr = os.path.join(data_dir, 'Wells_rep2_20180628_16h56m_pika_gige_7-Radiance Conversion-Georectify Airborne Datacube-Convert Radiance Cube to Reflectance from Measured Reference Spectrum.bip.hdr')

>>> data_dir2 = r'G:\BBE\AGROBOT\Shared Work\hs_process_results\data

ef_closest_panelcrop_plot’

>>> fname_hdr = os.path.join(data_dir2, 'study_aerffield_date_20190708_plot_5110-crop-plot.bip.hdr')
>>> array = io.spyfile.open_memmap()

>>> io = hsio()
>>> io.read_cube(fname_hdr)
>>> my_spec_mod = spec_mod(io.spyfile)

Use spec_mod.spectral_mimic to mimic the Sentinel-2A spectral response function.

>>> array_s2a, metadata_s2a = my_spec_mod.spectral_mimic(sensor='sentinel-2a', center_wl='weighted')

Plot the mean spectral response of the hyperspectral image to that of the mimicked Sentinel-2A image bands (mean calculated across the entire image).

>>> import seaborn as sns
>>> from ast import literal_eval
>>> spy_hs = my_spec_mod.spyfile.open_memmap()  # datacube before smoothing
>>> meta_bands = list(io.tools.meta_bands.values())
>>> meta_bands_s2a = sorted([float(i) for i in literal_eval(metadata_s2a['wavelength'])])

>>> ax = sns.lineplot(x=meta_bands, y=spy_hs[200][800]*100, label='Hyperspectral (Pika II)', linewidth=3)
>>> ax = sns.lineplot(x=meta_bands_s2a, y=array_s2a[200][800]*100, label='Sentinel-2A "mimic"', marker='o', ms=6, ax=ax)
>>> ax.set_xlabel('Wavelength (nm)', weight='bold')
>>> ax.set_ylabel('Reflectance (%)', weight='bold')
>>> ax.set_title(r'API Example: `spec_mod.spectral_mimic`', weight='bold')

Use spec_mod.spectral_mimic to mimic the Sentera 6x spectral configuration and compare to both hyperspectral and Sentinel-2A.

>>> array_6x, metadata_6x = my_spec_mod.spectral_mimic(sensor='sentera_6x', center_wl='peak')
>>> meta_bands_6x = sorted([float(i) for i in literal_eval(metadata_6x['wavelength'])])
>>> ax = sns.lineplot(x=meta_bands, y=spy_hs[200][800]*100, label='Hyperspectral (Pika II)', linewidth=3)
>>> ax = sns.lineplot(x=meta_bands_s2a, y=array_s2a[200][800]*100, label='Sentinel-2A "mimic"', marker='o', ms=6, ax=ax)
>>> ax = sns.lineplot(x=meta_bands_6x, y=array_6x[200][800]*100, label='Sentera 6X "mimic"', marker='o', ms=8, ax=ax)
>>> ax.set_xlabel('Wavelength (nm)', weight='bold')
>>> ax.set_ylabel('Reflectance (%)', weight='bold')
>>> ax.set_title(r'API Example: `spec_mod.spectral_mimic`', weight='bold')

And finally, mimic the Micasense RedEdge-MX and compare to hyperspectral, Sentinel-2A, and Sentera 6X.

>>> array_re3, metadata_re3 = my_spec_mod.spectral_mimic(sensor='micasense_rededge_3', center_wl='peak')
>>> meta_bands_re3 = sorted([float(i) for i in literal_eval(metadata_re3['wavelength'])])
>>> ax = sns.lineplot(x=meta_bands, y=spy_hs[200][800]*100, label='Hyperspectral (Pika II)', linewidth=3)
>>> ax = sns.lineplot(x=meta_bands_s2a, y=array_s2a[200][800]*100, label='Sentinel-2A "mimic"', marker='o', ms=6, ax=ax)
>>> ax = sns.lineplot(x=meta_bands_6x, y=array_6x[200][800]*100, label='Sentera 6X "mimic"', marker='o', ms=8, ax=ax)
>>> ax = sns.lineplot(x=meta_bands_re3, y=array_re3[200][800]*100, label='Micasense RedEdge 3 "mimic"', marker='o', ms=8, ax=ax)
>>> ax.set_xlabel('Wavelength (nm)', weight='bold')
>>> ax.set_ylabel('Reflectance (%)', weight='bold')
>>> ax.set_title(r'API Example: `spec_mod.spectral_mimic`', weight='bold')

spectral_resample(bandwidth=None, bins_n=None, spyfile=None)

Performs pixel-wise resampling of spectral bands via binning

(calculates the mean across all bands within each bandwidth

region for each image pixel).

Parameters

• bandwidth (float or int) – The bandwidth of the bands after spectral resampling is complete (units should be consistent with that of the .hdr file). Setting bandwidth to 10 will consolidate bands that fall within every 10 nm interval.

• bins_n (int) – The number of bins (i.e., “bands”) to achieve after spectral resampling is complete. Ignored if bandwidth is not None.

• spyfile (SpyFile object) – The datacube being accessed and/or manipulated.

Returns

2-element tuple containing

• array_bin (numpy.ndarray): Binned datacube.

• metadata (dict): Modified metadata describing the binned spectral array (array_bin).

Example

Load and initialize hsio and spec_mod

>>> import os
>>> from hs_process import hsio
>>> from hs_process import spec_mod
>>> data_dir = r'F:\nigo0024\Documents\hs_process_demo'
>>> fname_hdr = os.path.join(data_dir, 'Wells_rep2_20180628_16h56m_pika_gige_7-Radiance Conversion-Georectify Airborne Datacube-Convert Radiance Cube to Reflectance from Measured Reference Spectrum.bip.hdr')
>>> io = hsio()
>>> io.read_cube(fname_hdr)
>>> my_spec_mod = spec_mod(io.spyfile)

Use spec_mod.spectral_resample to “bin” the datacube to bands with 20 nm bandwidths.

>>> array_bin, metadata_bin = my_spec_mod.spectral_resample(bandwidth=20)

Plot the mean spectral response of the hyperspectral image to that of the binned image bands (mean calculated across the entire image).

>>> import seaborn as sns
>>> from ast import literal_eval
>>> spy_hs = my_spec_mod.spyfile.open_memmap()  # datacube before smoothing
>>> meta_bands = list(io.tools.meta_bands.values())
>>> meta_bands_bin = sorted([float(i) for i in literal_eval(metadata_bin['wavelength'])])

>>> ax = sns.lineplot(x=meta_bands, y=spy_hs[200][800]*100, label='Hyperspectral (Pika II)', linewidth=3)
>>> ax = sns.lineplot(x=meta_bands_bin, y=array_bin[200][800]*100, label='Spectral resample (20 nm)', marker='o', ms=6, ax=ax)
>>> ax.set_xlabel('Wavelength (nm)', weight='bold')
>>> ax.set_ylabel('Reflectance (%)', weight='bold')
>>> ax.set_title(r'API Example: `spec_mod.spectral_resample`', weight='bold')

spectral_smooth(window_size=11, order=2, spyfile=None)

Performs Savitzky-Golay smoothing on the spectral domain.

Parameters

• window_size (int) – the length of the window; must be an odd integer number (default: 11).

• order (int) – the order of the polynomial used in the filtering; must be less than window_size - 1 (default: 2).

• spyfile (SpyFile object or numpy.ndarray) – The data cube to clip; if numpy.ndarray or None, loads band information from spec_mod.spyfile (default: None).

Returns

2-element tuple containing

• array_smooth (numpy.ndarray): Clipped datacube.

• metadata (dict): Modified metadata describing the smoothed hyperspectral datacube (array_smooth).

Note

Because the smoothing operation is performed for every pixel individually, this function may take several minutes for large images.

Example

Load and initialize hsio and spec_mod

>>> import os
>>> from hs_process import hsio
>>> from hs_process import spec_mod
>>> data_dir = r'F:\nigo0024\Documents\hs_process_demo'
>>> fname_hdr = os.path.join(data_dir, 'Wells_rep2_20180628_16h56m_pika_gige_7-Radiance Conversion-Georectify Airborne Datacube-Convert Radiance Cube to Reflectance from Measured Reference Spectrum.bip.hdr')
>>> io = hsio()
>>> io.read_cube(fname_hdr)
>>> my_spec_mod = spec_mod(io.spyfile)

Use spec_mod.spectral_smooth to perform a Savitzky-Golay smoothing operation across the hyperspectral spectral signature.

>>> array_smooth, metadata_smooth = my_spec_mod.spectral_smooth(
        window_size=11, order=2)

Plot the spectra of an individual pixel to visualize the result of the smoothing procedure.

>>> import seaborn as sns
>>> spy_hs = my_spec_mod.spyfile.open_memmap()  # datacube before smoothing
>>> meta_bands = list(io.tools.meta_bands.values())
>>> meta_bands_smooth = sorted([float(i) for i in metadata_smooth['wavelength']])
>>> ax = sns.lineplot(x=meta_bands, y=spy_hs[200][800]*100, label='Before spectral smoothing', linewidth=3)
>>> ax = sns.lineplot(x=meta_bands_smooth, y=array_smooth[200][800]*100, label='After spectral smoothing', ax=ax)
>>> ax.set_xlabel('Wavelength (nm)', weight='bold')
>>> ax.set_ylabel('Reflectance (%)', weight='bold')
>>> ax.set_title(r'API Example: `spec_mod.spectral_smooth`', weight='bold')