diff --git a/.gitignore b/.gitignore new file mode 100644 index 0000000..f07e50c --- /dev/null +++ b/.gitignore @@ -0,0 +1,43 @@ +# Python +__pycache__/ +*.py[cod] +*$py.class +*.so +.Python +build/ +develop-eggs/ +dist/ +downloads/ +eggs/ +.eggs/ +lib/ +lib64/ +parts/ +sdist/ +var/ +wheels/ +*.egg-info/ +.installed.cfg +*.egg +MANIFEST + +# Virtual environments +venv/ +env/ +ENV/ +.venv + +# IDEs +.vscode/ +.idea/ +*.swp +*.swo +*~ + +# OS +.DS_Store +Thumbs.db + +# Test results +results/ +*.log \ No newline at end of file diff --git a/README.md b/README.md index c12bb06..dbaf300 100644 --- a/README.md +++ b/README.md @@ -1,26 +1,31 @@ # Python Packaging with Deephaven -This example demonstrates how to create and deploy Python packages that use Deephaven. It shows you how to package both command-line tools and reusable libraries using modern Python packaging standards. +This repository demonstrates how to create and deploy Python packages that use Deephaven. It shows three complete packaging scenarios following the official [Python Packaging User Guide](https://packaging.python.org/en/latest/guides/writing-pyproject-toml/) recommendations. This example accompanies the [Packaging custom code and dependencies](https://deephaven.io/core/docs/how-to-guides/sysadmin/setuptools-deployment/) guide in the Deephaven documentation. ## What you'll learn -This example shows you how to: - - Create installable Python packages with Deephaven dependencies -- Build command-line tools that process data with Deephaven - Package reusable library code for other projects +- Build command-line tools with entry point scripts - Manage dependencies with `pyproject.toml` +- Use the src-layout structure - Distribute packages as wheel archives -## Project structure +## Prerequisites + +- Python 3.8 or later +- pip (Python package installer) +- Basic familiarity with Python packaging + +## Repository structure -The example includes three complete packaging scenarios: +This repository contains three complete packaging scenarios: ### 1. Library-only package (`my_dh_library/`) -A reusable library with Deephaven query functions that other projects can import. +Reusable library code without CLI tools. Other projects import your modules. ``` my_dh_library/ @@ -33,68 +38,74 @@ my_dh_library/ └── README.md ``` +**Usage:** +```python +from my_dh_library.queries import filter_by_threshold +``` + ### 2. CLI-only package (`my_dh_cli/`) -Command-line tools for processing data with Deephaven. +Command-line tool without exposing library code. ``` my_dh_cli/ ├── src/ -│ └── my_dh_package/ +│ └── my_dh_cli/ │ ├── __init__.py │ ├── __main__.py -│ ├── cli.py -│ └── processor.py +│ └── cli.py ├── pyproject.toml -├── data/ -│ └── sample.csv └── README.md ``` +**Usage:** +```python +# Use within a Python session with server running +from my_dh_cli.cli import my_dh_query +result = my_dh_query("input_data.csv", verbose=True) +``` + ### 3. Combined package (`my_dh_toolkit/`) -Both reusable library code and command-line tools in one package. +Both reusable library code and command-line tools. ``` my_dh_toolkit/ ├── src/ -│ └── my_dh_package/ +│ └── my_dh_toolkit/ │ ├── __init__.py │ ├── __main__.py │ ├── cli.py +│ ├── processor.py │ ├── queries.py │ └── utils.py ├── pyproject.toml └── README.md ``` -## Prerequisites +**Usage:** +```python +# As a library +from my_dh_toolkit.queries import filter_by_threshold +``` -- Python 3.8 or later -- pip (Python package installer) -- Basic familiarity with Python packaging +```python +# As CLI functions (within Python session) +from my_dh_toolkit import my_dh_query, batch_process +result = my_dh_query("input_data.csv", verbose=True) +batch_process("data/", "results/", verbose=True) +``` ## Quick start Clone the repository: ```shell -git clone https://github.com/deephaven-examples/python-packaging.git -cd python-packaging +git clone https://github.com/deephaven-examples/deephaven-python-packaging.git +cd deephaven-python-packaging ``` -Choose an example to try: - -### Try the CLI package - -```shell -cd my_dh_cli -pip install -e . -my-dh-query data/sample.csv --verbose -my-dh-process data/ --output results/ -``` - -### Try the library package +### Try the library-only package ```shell cd my_dh_library @@ -105,173 +116,249 @@ python Then in Python: ```python +# Start the Deephaven server +from deephaven_server import Server +server = Server(port=10000, jvm_args=["-Xmx4g"]) +server.start() + +# Now use the library functions from my_dh_library.queries import filter_by_threshold from deephaven import read_csv data = read_csv("../data/sample.csv") filtered = filter_by_threshold(data, "Score", 75.0) -print(f"Filtered to {filtered.size} rows") +``` + +### Try the CLI-only package + +> [!NOTE] +> CLI tools require a Deephaven server running in the same Python process. The examples below show how to use the CLI functions within a Python session where the server is already started. True standalone CLI commands (run from a separate terminal) are not practical with Deephaven due to JVM initialization requirements. + +```shell +cd my_dh_cli +pip install -e . +python +``` + +Then in Python: + +```python +# Start the Deephaven server +from deephaven_server import Server +server = Server(port=10000, jvm_args=["-Xmx4g"]) +server.start() + +# Now use the CLI function +from my_dh_cli.cli import my_dh_query +result = my_dh_query("../data/sample.csv", verbose=True) +print(f"Processed {result.size} rows") ``` ### Try the combined package +> [!NOTE] +> Like the CLI-only package, the CLI commands require a Deephaven server in the same Python process. Use the library functions within a Python session. + ```shell cd my_dh_toolkit pip install -e . +python +``` + +Then in Python: + +```python +# Start the Deephaven server +from deephaven_server import Server +server = Server(port=10000, jvm_args=["-Xmx4g"]) +server.start() # Use as a library -python -c "from my_dh_toolkit.queries import filter_by_threshold; print('Library imported successfully')" +from my_dh_toolkit.queries import filter_by_threshold +from deephaven import read_csv -# Use as CLI tools -my-dh-query ../data/sample.csv -my-dh-process ../data/ --output results/ +data = read_csv("../data/sample.csv") +filtered = filter_by_threshold(data, "Score", 75.0) + +# Or use the CLI functions +from my_dh_toolkit import my_dh_query, batch_process +result = my_dh_query("../data/sample.csv", verbose=True) +batch_process("../data/batch/", "./output", verbose=True) ``` -## What's included +## Sample data -### Command-line tools +The `data/` directory contains sample CSV files for testing: -The CLI examples demonstrate: +- `sample.csv` - Single file with Name, Score, Value, and Category columns +- `batch/` - Multiple CSV files for batch processing examples -- **Entry point scripts** - Commands installed to your PATH -- **Module execution** - Running with `python -m package_name` -- **Argument parsing** - Using Click for robust CLI interfaces -- **Multiple commands** - Single package with multiple tools -- **Verbose output** - Optional detailed logging +## Key concepts + +### Package structure -### Library modules +All examples use the **src-layout**, which is the recommended structure for Python packages: -The library examples show: +``` +my_project/ +├── src/ +│ └── my_package/ +│ ├── __init__.py +│ └── module.py +├── pyproject.toml +└── README.md +``` -- **Reusable query functions** - Common Deephaven operations -- **Type hints** - Proper function signatures -- **Public API exports** - Clean import patterns -- **Documentation** - Docstrings for all functions +The src-layout keeps source code separate from tests and configuration files. -### Configuration +### Entry point scripts -All examples include: +Entry point scripts are defined in `[project.scripts]` and become available after installation: -- **`pyproject.toml`** - Modern Python packaging configuration -- **Dependency management** - Automatic installation of Deephaven and other requirements -- **Version constraints** - Ensuring compatible package versions -- **Entry points** - Mapping command names to Python functions +```toml +[project.scripts] +my-command = "my_package.module:function" +``` -## Building and distributing +After `pip install`, you can run `my-command` from anywhere. -Each example can be built into a distributable wheel: +### Module execution -```shell -cd my_dh_cli # or any example directory -pip install build -python -m build -``` +Add a `__main__.py` file to support running packages with `python -m`: -This creates a `.whl` file in the `dist/` directory that can be: +```python +from my_package.cli import app -- Installed locally: `pip install dist/my_dh_cli-0.1.0-py3-none-any.whl` -- Distributed to others -- Published to PyPI: `python -m twine upload dist/*` +if __name__ == "__main__": + app() +``` -## Running the examples +This allows running without installation: `python -m my_package` -### Development mode +### Dependencies -Install in editable mode to make changes without reinstalling: +Dependencies are specified in `pyproject.toml`: -```shell -pip install -e . +```toml +[project] +dependencies = [ + "deephaven-server>=0.35.0", + "click>=8.0.0", +] ``` -### Regular installation +These are automatically installed when users install your package. -Install from the built wheel: +## Building and distributing + +Build a distributable wheel: ```shell -pip install dist/package_name-0.1.0-py3-none-any.whl +cd my_dh_cli # or any package directory +pip install build +python -m build ``` -### Without installation +This creates a `.whl` file in `dist/` that can be: -Run directly from source using module execution: +- Installed locally: `pip install dist/my_dh_cli-0.1.0-py3-none-any.whl` +- Distributed to others +- Published to PyPI: `python -m twine upload dist/*` -```shell -python -m my_dh_package input_data.csv -``` +## Packaging scenarios -## Sample data +### When to use library-only -The `data/` directory contains sample CSV files for testing: +- Creating reusable code for other projects +- No command-line interface needed +- Code will be imported, not executed -- `sample.csv` - Small dataset with Name, Age, and Score columns -- `batch/` - Multiple CSV files for batch processing examples +**Example:** Data processing utilities, query functions, helper classes -You can use your own CSV files with these examples. +### When to use CLI-only -## Key concepts +- Building command-line tools for end users +- No library code to expose +- Want clean command names -### Entry point scripts vs module execution +**Example:** Data conversion tools, file processors, automation scripts -The examples demonstrate two ways to run Python packages: +### When to use combined -1. **Entry point scripts** - Commands defined in `[project.scripts]` that become available after installation - ```shell - my-dh-query data.csv - ``` +- Need both library and CLI functionality +- Want to provide multiple interfaces +- Library functions useful on their own -2. **Module execution** - Running packages with `python -m` without installation - ```shell - python -m my_dh_package data.csv - ``` +**Example:** Data analysis toolkit with both API and CLI -See the [Execution patterns](https://deephaven.io/core/docs/how-to-guides/sysadmin/setuptools-deployment/#execution-patterns) section of the guide for when to use each method. +## Execution patterns -### Package structure +### Entry point scripts (recommended for CLI tools) -All examples use the **src-layout**, which is the recommended structure for Python packages. This keeps source code separate from tests and configuration files. +**Configure in `pyproject.toml`:** +```toml +[project.scripts] +my-command = "my_package.module:function" +``` -### Dependencies +**Run after installation:** +```shell +my-command +``` -The examples show how to: +**Benefits:** +- Clean command names +- Available system-wide +- Standard Python packaging approach -- Specify required packages (like `deephaven-server`) -- Set version constraints -- Define optional dependencies for features like visualization or testing +### Module execution (useful for development) -## Related documentation +**Add `__main__.py`:** +```python +from my_package.cli import app -- [Packaging custom code and dependencies](https://deephaven.io/core/docs/how-to-guides/sysadmin/setuptools-deployment/) - Complete guide -- [Install and use Python packages](https://deephaven.io/core/docs/how-to-guides/install-and-use-python-packages/) -- [Use the Deephaven Python package](https://deephaven.io/core/docs/how-to-guides/deephaven-python-package/) -- [Python Packaging User Guide](https://packaging.python.org/en/latest/guides/writing-pyproject-toml/) -- [Click documentation](https://click.palletsprojects.com/) +if __name__ == "__main__": + app() +``` + +**Run without installation:** +```shell +python -m my_package +``` + +**Benefits:** +- No installation required +- Useful for development and testing +- Works from source directory ## Troubleshooting ### Command not found after installation -If your command isn't found after installation: - -- Ensure the installation completed without errors +- Ensure installation completed without errors - Check that the installation directory is in your PATH - Try reinstalling: `pip install --force-reinstall .` ### Import errors -If you encounter import errors: - - Verify all dependencies are installed: `pip list` - Check that you're using Python 3.8 or later - Ensure Deephaven is installed: `pip install deephaven-server` ### Module not found errors -If Python can't find your modules: - - Verify `__init__.py` files exist in all package directories - Check that package names in `[project.scripts]` match your directory structure - Try reinstalling in editable mode: `pip install -e .` +## Related documentation + +- [Packaging custom code and dependencies](https://deephaven.io/core/docs/how-to-guides/sysadmin/setuptools-deployment/) - Complete guide +- [Install and use Python packages](https://deephaven.io/core/docs/how-to-guides/install-and-use-python-packages/) +- [Use the Deephaven Python package](https://deephaven.io/core/docs/how-to-guides/deephaven-python-package/) +- [Python Packaging User Guide](https://packaging.python.org/en/latest/guides/writing-pyproject-toml/) +- [Click documentation](https://click.palletsprojects.com/) + ## Note The code in this repository is built for Deephaven Community Core v0.35.0 or later. For the latest Deephaven version, see [deephaven.io](https://deephaven.io/). diff --git a/data/batch/file1.csv b/data/batch/file1.csv new file mode 100644 index 0000000..fe9fae6 --- /dev/null +++ b/data/batch/file1.csv @@ -0,0 +1,4 @@ +Name,Score,Value,Category +Alice,85,120,A +Bob,92,150,B +Charlie,78,95,A diff --git a/data/batch/file2.csv b/data/batch/file2.csv new file mode 100644 index 0000000..e270a59 --- /dev/null +++ b/data/batch/file2.csv @@ -0,0 +1,4 @@ +Name,Score,Value,Category +Diana,88,110,C +Eve,95,180,B +Frank,72,85,A diff --git a/data/batch/file3.csv b/data/batch/file3.csv new file mode 100644 index 0000000..692c4a2 --- /dev/null +++ b/data/batch/file3.csv @@ -0,0 +1,5 @@ +Name,Score,Value,Category +Grace,91,160,C +Henry,83,105,B +Iris,89,140,A +Jack,76,90,C diff --git a/data/sample.csv b/data/sample.csv new file mode 100644 index 0000000..1d1b856 --- /dev/null +++ b/data/sample.csv @@ -0,0 +1,11 @@ +Name,Score,Value,Category +Alice,85,120,A +Bob,92,150,B +Charlie,78,95,A +Diana,88,110,C +Eve,95,180,B +Frank,72,85,A +Grace,91,160,C +Henry,83,105,B +Iris,89,140,A +Jack,76,90,C diff --git a/my_dh_cli/README.md b/my_dh_cli/README.md new file mode 100644 index 0000000..66e3a4a --- /dev/null +++ b/my_dh_cli/README.md @@ -0,0 +1,57 @@ +# My Deephaven CLI + +A CLI-only package providing command-line tools for data processing with Deephaven. This package is designed to be installed and run as a command-line tool. + +## Installation + +```shell +pip install . +``` + +Or in editable mode for development: + +```shell +pip install -e . +``` + +## Usage + +> [!NOTE] +> CLI functions require a Deephaven server running in the same Python process. Use the functions within a Python session where the server is already started. + +```shell +pip install -e . +python +``` + +Then in Python: + +```python +# Start the Deephaven server +from deephaven_server import Server +server = Server(port=10000, jvm_args=["-Xmx4g"]) +server.start() + +# Now use the CLI function +from my_dh_cli.cli import my_dh_query +result = my_dh_query("data/sample.csv", verbose=True) +print(f"Processed {result.size} rows") +``` + +## Commands + +### my-dh-query + +Process a CSV file with Deephaven. + +**Arguments:** +- `input_file` - Path to the CSV file to process + +**Options:** +- `--verbose, -v` - Enable verbose output + +## Requirements + +- Python 3.8 or later +- Deephaven Server 0.35.0 or later +- Click 8.0.0 or later diff --git a/my_dh_cli/data/batch/file1.csv b/my_dh_cli/data/batch/file1.csv new file mode 100644 index 0000000..fe9fae6 --- /dev/null +++ b/my_dh_cli/data/batch/file1.csv @@ -0,0 +1,4 @@ +Name,Score,Value,Category +Alice,85,120,A +Bob,92,150,B +Charlie,78,95,A diff --git a/my_dh_cli/data/batch/file2.csv b/my_dh_cli/data/batch/file2.csv new file mode 100644 index 0000000..e270a59 --- /dev/null +++ b/my_dh_cli/data/batch/file2.csv @@ -0,0 +1,4 @@ +Name,Score,Value,Category +Diana,88,110,C +Eve,95,180,B +Frank,72,85,A diff --git a/my_dh_cli/data/batch/file3.csv b/my_dh_cli/data/batch/file3.csv new file mode 100644 index 0000000..692c4a2 --- /dev/null +++ b/my_dh_cli/data/batch/file3.csv @@ -0,0 +1,5 @@ +Name,Score,Value,Category +Grace,91,160,C +Henry,83,105,B +Iris,89,140,A +Jack,76,90,C diff --git a/my_dh_cli/data/sample.csv b/my_dh_cli/data/sample.csv new file mode 100644 index 0000000..1d1b856 --- /dev/null +++ b/my_dh_cli/data/sample.csv @@ -0,0 +1,11 @@ +Name,Score,Value,Category +Alice,85,120,A +Bob,92,150,B +Charlie,78,95,A +Diana,88,110,C +Eve,95,180,B +Frank,72,85,A +Grace,91,160,C +Henry,83,105,B +Iris,89,140,A +Jack,76,90,C diff --git a/my_dh_cli/pyproject.toml b/my_dh_cli/pyproject.toml new file mode 100644 index 0000000..32c7f84 --- /dev/null +++ b/my_dh_cli/pyproject.toml @@ -0,0 +1,20 @@ +[build-system] +requires = ["setuptools>=61.0", "wheel"] +build-backend = "setuptools.build_meta" + +[project] +name = "my_dh_cli" +version = "0.1.0" +description = "Command-line tool for data processing" +readme = "README.md" +requires-python = ">=3.8" +dependencies = [ + "deephaven-server>=0.35.0", + "click>=8.0.0", +] + +[project.scripts] +my-dh-query = "my_dh_cli.cli:app" + +[tool.setuptools.packages.find] +where = ["src"] diff --git a/my_dh_cli/src/my_dh_cli/__init__.py b/my_dh_cli/src/my_dh_cli/__init__.py new file mode 100644 index 0000000..cd9785f --- /dev/null +++ b/my_dh_cli/src/my_dh_cli/__init__.py @@ -0,0 +1,3 @@ +"""My Deephaven package for data processing.""" + +__version__ = "0.1.0" diff --git a/my_dh_cli/src/my_dh_cli/__main__.py b/my_dh_cli/src/my_dh_cli/__main__.py new file mode 100644 index 0000000..ff7364e --- /dev/null +++ b/my_dh_cli/src/my_dh_cli/__main__.py @@ -0,0 +1,4 @@ +from my_dh_cli.cli import app + +if __name__ == "__main__": + app() diff --git a/my_dh_cli/src/my_dh_cli/cli.py b/my_dh_cli/src/my_dh_cli/cli.py new file mode 100644 index 0000000..3e05e1c --- /dev/null +++ b/my_dh_cli/src/my_dh_cli/cli.py @@ -0,0 +1,49 @@ +import click + + +def my_dh_query(input_file: str, verbose: bool = False): + """Read a CSV file and perform a simple query operation on the data.""" + from deephaven import read_csv + from pathlib import Path + + input_path = Path(input_file) + + if not input_path.exists(): + raise click.ClickException(f"Input file does not exist: '{input_path}'") + if not input_path.is_file(): + raise click.ClickException(f"Input path is not a file: '{input_path}'") + + if verbose: + click.echo(f"Processing {input_file}...") + + try: + source = read_csv(input_file) + except Exception as e: + raise click.ClickException(f"Failed to read CSV file '{input_file}': {e}") + + column_names = [col.name for col in source.columns] + if "Score" not in column_names: + raise click.ClickException( + f"File '{input_path.name}' is missing required column 'Score'. " + f"Available columns: {', '.join(column_names)}" + ) + + result = source.update(formulas=["DoubleScore = Score * 2"]) + + if verbose: + click.echo(f"Processed {result.size} rows") + + return result + + +@click.command() +@click.argument("input_file", type=click.Path(exists=True)) +@click.option("--verbose", "-v", is_flag=True, help="Enable verbose output") +def app(input_file: str, verbose: bool) -> None: + """Process data with Deephaven.""" + result = my_dh_query(input_file, verbose) + click.echo("Processing complete!") + + +if __name__ == "__main__": + app() diff --git a/my_dh_library/README.md b/my_dh_library/README.md new file mode 100644 index 0000000..4362a1c --- /dev/null +++ b/my_dh_library/README.md @@ -0,0 +1,55 @@ +# My Deephaven Library + +A library-only package providing reusable Deephaven query functions. This package contains no CLI tools - it's designed to be imported and used as a library in other Python projects. + +## Installation + +```shell +pip install . +``` + +Or in editable mode for development: + +```shell +pip install -e . +``` + +## Usage + +> [!NOTE] +> All Deephaven functionality requires a running server. Start the server before importing Deephaven modules. + +Import and use the library functions in your Python code: + +```python +# Start the Deephaven server +from deephaven_server import Server +server = Server(port=10000, jvm_args=["-Xmx4g"]) +server.start() + +# Now use the library functions +from my_dh_library.queries import filter_by_threshold, add_computed_columns +from deephaven import read_csv + +data = read_csv("data/sample.csv") +filtered = filter_by_threshold(data, "Score", 75.0) +enhanced = add_computed_columns(filtered) +``` + +## Available Functions + +### Query Functions (`my_dh_library.queries`) + +- `filter_by_threshold(table, column, threshold)` - Filter table rows where column value exceeds threshold +- `add_computed_columns(table)` - Add commonly used computed columns to a table +- `summarize_by_group(table, group_col, value_col)` - Create summary statistics grouped by a column + +### Utility Functions (`my_dh_library.utils`) + +- `validate_columns(table, required_columns)` - Check if table has all required columns +- `get_table_info(table)` - Get basic information about a table + +## Requirements + +- Python 3.8 or later +- Deephaven Server 0.35.0 or later diff --git a/my_dh_library/data/batch/file1.csv b/my_dh_library/data/batch/file1.csv new file mode 100644 index 0000000..fe9fae6 --- /dev/null +++ b/my_dh_library/data/batch/file1.csv @@ -0,0 +1,4 @@ +Name,Score,Value,Category +Alice,85,120,A +Bob,92,150,B +Charlie,78,95,A diff --git a/my_dh_library/data/batch/file2.csv b/my_dh_library/data/batch/file2.csv new file mode 100644 index 0000000..e270a59 --- /dev/null +++ b/my_dh_library/data/batch/file2.csv @@ -0,0 +1,4 @@ +Name,Score,Value,Category +Diana,88,110,C +Eve,95,180,B +Frank,72,85,A diff --git a/my_dh_library/data/batch/file3.csv b/my_dh_library/data/batch/file3.csv new file mode 100644 index 0000000..692c4a2 --- /dev/null +++ b/my_dh_library/data/batch/file3.csv @@ -0,0 +1,5 @@ +Name,Score,Value,Category +Grace,91,160,C +Henry,83,105,B +Iris,89,140,A +Jack,76,90,C diff --git a/my_dh_library/data/sample.csv b/my_dh_library/data/sample.csv new file mode 100644 index 0000000..1d1b856 --- /dev/null +++ b/my_dh_library/data/sample.csv @@ -0,0 +1,11 @@ +Name,Score,Value,Category +Alice,85,120,A +Bob,92,150,B +Charlie,78,95,A +Diana,88,110,C +Eve,95,180,B +Frank,72,85,A +Grace,91,160,C +Henry,83,105,B +Iris,89,140,A +Jack,76,90,C diff --git a/my_dh_library/pyproject.toml b/my_dh_library/pyproject.toml new file mode 100644 index 0000000..3fa6b90 --- /dev/null +++ b/my_dh_library/pyproject.toml @@ -0,0 +1,16 @@ +[build-system] +requires = ["setuptools>=61.0", "wheel"] +build-backend = "setuptools.build_meta" + +[project] +name = "my_dh_library" +version = "0.1.0" +description = "Reusable Deephaven query functions" +readme = "README.md" +requires-python = ">=3.8" +dependencies = [ + "deephaven-server>=0.35.0", +] + +[tool.setuptools.packages.find] +where = ["src"] diff --git a/my_dh_library/src/my_dh_library/__init__.py b/my_dh_library/src/my_dh_library/__init__.py new file mode 100644 index 0000000..6e191c0 --- /dev/null +++ b/my_dh_library/src/my_dh_library/__init__.py @@ -0,0 +1,7 @@ +"""My Deephaven package for data processing.""" + +__version__ = "0.1.0" + +from my_dh_library.queries import filter_by_threshold, add_computed_columns, summarize_by_group + +__all__ = ["filter_by_threshold", "add_computed_columns", "summarize_by_group"] diff --git a/my_dh_library/src/my_dh_library/queries.py b/my_dh_library/src/my_dh_library/queries.py new file mode 100644 index 0000000..eb0adfc --- /dev/null +++ b/my_dh_library/src/my_dh_library/queries.py @@ -0,0 +1,35 @@ +"""Reusable Deephaven query functions.""" + +from deephaven.table import Table +from deephaven import agg +from .utils import validate_columns + + +def filter_by_threshold(table: Table, column: str, threshold: float) -> Table: + """Filter table rows where column value exceeds threshold.""" + validate_columns(table, [column], raise_error=True) + return table.where(f"{column} > {threshold}") + + +def add_computed_columns(table: Table) -> Table: + """Add commonly used computed columns to a table.""" + validate_columns(table, ["Value"], raise_error=True) + return table.update( + [ + "DoubleValue = Value * 2", + "IsHigh = Value > 100", + ] + ) + + +def summarize_by_group(table: Table, group_col: str, value_col: str) -> Table: + """Create summary statistics grouped by a column.""" + validate_columns(table, [group_col, value_col], raise_error=True) + return table.agg_by( + [ + agg.sum_(f"Sum = {value_col}"), + agg.avg(f"Avg = {value_col}"), + agg.count_("Count"), + ], + by=[group_col], + ) diff --git a/my_dh_library/src/my_dh_library/utils.py b/my_dh_library/src/my_dh_library/utils.py new file mode 100644 index 0000000..f38c859 --- /dev/null +++ b/my_dh_library/src/my_dh_library/utils.py @@ -0,0 +1,39 @@ +"""Utility functions for working with Deephaven tables.""" + +from deephaven.table import Table + + +def validate_columns(table: Table, required_columns: list[str], raise_error: bool = False) -> bool: + """Check if table has all required columns. + + Args: + table: The table to validate + required_columns: List of column names that must be present + raise_error: If True, raises ValueError when columns are missing + + Returns: + True if all columns are present, False otherwise + + Raises: + ValueError: If raise_error is True and columns are missing + """ + table_columns = [col.name for col in table.columns] + missing = [col for col in required_columns if col not in table_columns] + + if missing: + if raise_error: + raise ValueError( + f"Column(s) {missing} not found in table. " + f"Available columns: {', '.join(table_columns)}" + ) + return False + return True + + +def get_table_info(table: Table) -> dict: + """Get basic information about a table.""" + return { + "num_rows": table.size, + "num_columns": len(table.columns), + "columns": [col.name for col in table.columns], + } diff --git a/my_dh_toolkit/README.md b/my_dh_toolkit/README.md new file mode 100644 index 0000000..bb91728 --- /dev/null +++ b/my_dh_toolkit/README.md @@ -0,0 +1,104 @@ +# My Deephaven Toolkit + +A combined package providing both reusable library code and command-line functions for Deephaven. This package can be used as both a library (imported in Python code) and as CLI functions (called within a Python session). + +## Installation + +```shell +pip install . +``` + +Or in editable mode for development: + +```shell +pip install -e . +``` + +## Usage as a Library + +> [!NOTE] +> All Deephaven functionality requires a running server. Start the server before importing Deephaven modules. + +Import and use the library functions in your Python code: + +```python +# Start the Deephaven server +from deephaven_server import Server +server = Server(port=10000, jvm_args=["-Xmx4g"]) +server.start() + +# Now use the library functions +from my_dh_toolkit.queries import filter_by_threshold, add_computed_columns +from my_dh_toolkit import my_dh_query, batch_process +from deephaven import read_csv + +data = read_csv("data/sample.csv") +filtered = filter_by_threshold(data, "Score", 75.0) + +# Or use the exported functions +result = my_dh_query("data/sample.csv", verbose=True) +``` + +## Usage as CLI Functions + +> [!NOTE] +> CLI functions require a Deephaven server running in the same Python process. Use them within a Python session where the server is already started. + +```python +# Start the Deephaven server +from deephaven_server import Server +server = Server(port=10000, jvm_args=["-Xmx4g"]) +server.start() + +# Use the CLI functions +from my_dh_toolkit import my_dh_query, batch_process +result = my_dh_query("data/sample.csv", verbose=True) +batch_process("data/batch/", "./output", verbose=True) +``` + +## Commands + +### my-dh-query + +Process a single CSV file with Deephaven. + +**Arguments:** +- `input_file` - Path to the CSV file to process + +**Options:** +- `--verbose, -v` - Enable verbose output + +### my-dh-process + +Batch process multiple CSV files from a directory. + +**Arguments:** +- `directory` - Directory containing CSV files to process + +**Options:** +- `--output, -o` - Output directory (default: ./output) +- `--verbose, -v` - Enable verbose output + +## Available Functions + +### Query Functions (`my_dh_toolkit.queries`) + +- `filter_by_threshold(table, column, threshold)` - Filter table rows where column value exceeds threshold +- `add_computed_columns(table)` - Add commonly used computed columns to a table +- `summarize_by_group(table, group_col, value_col)` - Create summary statistics grouped by a column + +### Utility Functions (`my_dh_toolkit.utils`) + +- `validate_columns(table, required_columns)` - Check if table has all required columns +- `get_table_info(table)` - Get basic information about a table + +### Exported Functions (`my_dh_toolkit`) + +- `my_dh_query(input_file, verbose)` - Read and process a CSV file +- `batch_process(directory, output_dir, verbose)` - Process multiple CSV files + +## Requirements + +- Python 3.8 or later +- Deephaven Server 0.35.0 or later +- Click 8.0.0 or later diff --git a/my_dh_toolkit/data/batch/file1.csv b/my_dh_toolkit/data/batch/file1.csv new file mode 100644 index 0000000..fe9fae6 --- /dev/null +++ b/my_dh_toolkit/data/batch/file1.csv @@ -0,0 +1,4 @@ +Name,Score,Value,Category +Alice,85,120,A +Bob,92,150,B +Charlie,78,95,A diff --git a/my_dh_toolkit/data/batch/file2.csv b/my_dh_toolkit/data/batch/file2.csv new file mode 100644 index 0000000..e270a59 --- /dev/null +++ b/my_dh_toolkit/data/batch/file2.csv @@ -0,0 +1,4 @@ +Name,Score,Value,Category +Diana,88,110,C +Eve,95,180,B +Frank,72,85,A diff --git a/my_dh_toolkit/data/batch/file3.csv b/my_dh_toolkit/data/batch/file3.csv new file mode 100644 index 0000000..692c4a2 --- /dev/null +++ b/my_dh_toolkit/data/batch/file3.csv @@ -0,0 +1,5 @@ +Name,Score,Value,Category +Grace,91,160,C +Henry,83,105,B +Iris,89,140,A +Jack,76,90,C diff --git a/my_dh_toolkit/data/sample.csv b/my_dh_toolkit/data/sample.csv new file mode 100644 index 0000000..1d1b856 --- /dev/null +++ b/my_dh_toolkit/data/sample.csv @@ -0,0 +1,11 @@ +Name,Score,Value,Category +Alice,85,120,A +Bob,92,150,B +Charlie,78,95,A +Diana,88,110,C +Eve,95,180,B +Frank,72,85,A +Grace,91,160,C +Henry,83,105,B +Iris,89,140,A +Jack,76,90,C diff --git a/my_dh_toolkit/pyproject.toml b/my_dh_toolkit/pyproject.toml new file mode 100644 index 0000000..81bc624 --- /dev/null +++ b/my_dh_toolkit/pyproject.toml @@ -0,0 +1,21 @@ +[build-system] +requires = ["setuptools>=61.0", "wheel"] +build-backend = "setuptools.build_meta" + +[project] +name = "my_dh_toolkit" +version = "0.1.0" +description = "Deephaven library and CLI tools" +readme = "README.md" +requires-python = ">=3.8" +dependencies = [ + "deephaven-server>=0.35.0", + "click>=8.0.0", +] + +[project.scripts] +my-dh-query = "my_dh_toolkit.cli:app" +my-dh-process = "my_dh_toolkit.processor:process" + +[tool.setuptools.packages.find] +where = ["src"] diff --git a/my_dh_toolkit/src/my_dh_toolkit/__init__.py b/my_dh_toolkit/src/my_dh_toolkit/__init__.py new file mode 100644 index 0000000..31bf609 --- /dev/null +++ b/my_dh_toolkit/src/my_dh_toolkit/__init__.py @@ -0,0 +1,8 @@ +"""My Deephaven package for data processing.""" + +__version__ = "0.1.0" + +from my_dh_toolkit.cli import my_dh_query +from my_dh_toolkit.processor import batch_process + +__all__ = ["my_dh_query", "batch_process"] diff --git a/my_dh_toolkit/src/my_dh_toolkit/__main__.py b/my_dh_toolkit/src/my_dh_toolkit/__main__.py new file mode 100644 index 0000000..c7407a9 --- /dev/null +++ b/my_dh_toolkit/src/my_dh_toolkit/__main__.py @@ -0,0 +1,4 @@ +from my_dh_toolkit.cli import app + +if __name__ == "__main__": + app() diff --git a/my_dh_toolkit/src/my_dh_toolkit/cli.py b/my_dh_toolkit/src/my_dh_toolkit/cli.py new file mode 100644 index 0000000..3e05e1c --- /dev/null +++ b/my_dh_toolkit/src/my_dh_toolkit/cli.py @@ -0,0 +1,49 @@ +import click + + +def my_dh_query(input_file: str, verbose: bool = False): + """Read a CSV file and perform a simple query operation on the data.""" + from deephaven import read_csv + from pathlib import Path + + input_path = Path(input_file) + + if not input_path.exists(): + raise click.ClickException(f"Input file does not exist: '{input_path}'") + if not input_path.is_file(): + raise click.ClickException(f"Input path is not a file: '{input_path}'") + + if verbose: + click.echo(f"Processing {input_file}...") + + try: + source = read_csv(input_file) + except Exception as e: + raise click.ClickException(f"Failed to read CSV file '{input_file}': {e}") + + column_names = [col.name for col in source.columns] + if "Score" not in column_names: + raise click.ClickException( + f"File '{input_path.name}' is missing required column 'Score'. " + f"Available columns: {', '.join(column_names)}" + ) + + result = source.update(formulas=["DoubleScore = Score * 2"]) + + if verbose: + click.echo(f"Processed {result.size} rows") + + return result + + +@click.command() +@click.argument("input_file", type=click.Path(exists=True)) +@click.option("--verbose", "-v", is_flag=True, help="Enable verbose output") +def app(input_file: str, verbose: bool) -> None: + """Process data with Deephaven.""" + result = my_dh_query(input_file, verbose) + click.echo("Processing complete!") + + +if __name__ == "__main__": + app() diff --git a/my_dh_toolkit/src/my_dh_toolkit/processor.py b/my_dh_toolkit/src/my_dh_toolkit/processor.py new file mode 100644 index 0000000..526c766 --- /dev/null +++ b/my_dh_toolkit/src/my_dh_toolkit/processor.py @@ -0,0 +1,65 @@ +import click +from pathlib import Path + + +def batch_process(directory: str, output_dir: str, verbose: bool = False) -> None: + """Process multiple CSV files from a directory.""" + from deephaven import read_csv, write_csv + + input_path = Path(directory) + output_path = Path(output_dir) + + if not input_path.exists(): + raise click.ClickException(f"Input directory does not exist: '{input_path}'") + if not input_path.is_dir(): + raise click.ClickException(f"Input path is not a directory: '{input_path}'") + + try: + output_path.mkdir(parents=True, exist_ok=True) + except PermissionError: + raise click.ClickException(f"Permission denied: Cannot create output directory '{output_path}'") + except OSError as e: + raise click.ClickException(f"Failed to create output directory '{output_path}': {e}") + + csv_files = list(input_path.glob("*.csv")) + + if verbose: + click.echo(f"Found {len(csv_files)} CSV files to process") + + for csv_file in csv_files: + if verbose: + click.echo(f"Processing {csv_file.name}...") + + table = read_csv(str(csv_file)) + + column_names = [col.name for col in table.columns] + if "Score" not in column_names: + raise click.ClickException( + f"File '{csv_file.name}' is missing required column 'Score'. " + f"Available columns: {', '.join(column_names)}" + ) + + processed = table.update(formulas=["ProcessedScore = Score * 2"]) + + output_file = output_path / f"processed_{csv_file.name}" + try: + write_csv(processed, str(output_file)) + except Exception as e: + raise click.ClickException(f"Failed to write output file '{output_file}': {e}") + + if verbose: + click.echo(f" Processed {processed.size} rows -> {output_file.name}") + + +@click.command() +@click.argument("directory", type=click.Path(exists=True, file_okay=False)) +@click.option("--output", "-o", default="./output", help="Output directory") +@click.option("--verbose", "-v", is_flag=True, help="Enable verbose output") +def process(directory: str, output: str, verbose: bool) -> None: + """Batch process CSV files with Deephaven.""" + batch_process(directory, output, verbose) + click.echo("Batch processing complete!") + + +if __name__ == "__main__": + process() diff --git a/my_dh_toolkit/src/my_dh_toolkit/queries.py b/my_dh_toolkit/src/my_dh_toolkit/queries.py new file mode 100644 index 0000000..eb0adfc --- /dev/null +++ b/my_dh_toolkit/src/my_dh_toolkit/queries.py @@ -0,0 +1,35 @@ +"""Reusable Deephaven query functions.""" + +from deephaven.table import Table +from deephaven import agg +from .utils import validate_columns + + +def filter_by_threshold(table: Table, column: str, threshold: float) -> Table: + """Filter table rows where column value exceeds threshold.""" + validate_columns(table, [column], raise_error=True) + return table.where(f"{column} > {threshold}") + + +def add_computed_columns(table: Table) -> Table: + """Add commonly used computed columns to a table.""" + validate_columns(table, ["Value"], raise_error=True) + return table.update( + [ + "DoubleValue = Value * 2", + "IsHigh = Value > 100", + ] + ) + + +def summarize_by_group(table: Table, group_col: str, value_col: str) -> Table: + """Create summary statistics grouped by a column.""" + validate_columns(table, [group_col, value_col], raise_error=True) + return table.agg_by( + [ + agg.sum_(f"Sum = {value_col}"), + agg.avg(f"Avg = {value_col}"), + agg.count_("Count"), + ], + by=[group_col], + ) diff --git a/my_dh_toolkit/src/my_dh_toolkit/utils.py b/my_dh_toolkit/src/my_dh_toolkit/utils.py new file mode 100644 index 0000000..f38c859 --- /dev/null +++ b/my_dh_toolkit/src/my_dh_toolkit/utils.py @@ -0,0 +1,39 @@ +"""Utility functions for working with Deephaven tables.""" + +from deephaven.table import Table + + +def validate_columns(table: Table, required_columns: list[str], raise_error: bool = False) -> bool: + """Check if table has all required columns. + + Args: + table: The table to validate + required_columns: List of column names that must be present + raise_error: If True, raises ValueError when columns are missing + + Returns: + True if all columns are present, False otherwise + + Raises: + ValueError: If raise_error is True and columns are missing + """ + table_columns = [col.name for col in table.columns] + missing = [col for col in required_columns if col not in table_columns] + + if missing: + if raise_error: + raise ValueError( + f"Column(s) {missing} not found in table. " + f"Available columns: {', '.join(table_columns)}" + ) + return False + return True + + +def get_table_info(table: Table) -> dict: + """Get basic information about a table.""" + return { + "num_rows": table.size, + "num_columns": len(table.columns), + "columns": [col.name for col in table.columns], + } diff --git a/setuptools-deployment.md b/setuptools-deployment.md new file mode 100644 index 0000000..8af03a6 --- /dev/null +++ b/setuptools-deployment.md @@ -0,0 +1,774 @@ +--- +title: Packaging custom code and dependencies +sidebar_label: Python packaging +--- + +[Python packaging](https://packaging.python.org/en/latest/guides/writing-pyproject-toml/) enables you to create distributable packages containing custom code, command-line tools, and managed dependencies. Deephaven's pip-installable packages integrate seamlessly with modern Python packaging tools, allowing you to build reusable libraries and executable scripts that leverage Deephaven's query engine. This guide walks through the concepts and patterns for packaging Deephaven-based Python projects. + +Python packaging with [`pyproject.toml`](https://packaging.python.org/en/latest/guides/writing-pyproject-toml/) provides: + +- **Reusable libraries** - Package query functions and utilities for import by other projects. +- **Command-line tools** - Build executable scripts with entry point definitions. +- **Dependency management** - Automatically install Deephaven and required packages. +- **Distribution** - Share code as wheel archives via PyPI or direct distribution. +- **Version control** - Specify compatible dependency versions for reproducible installations. + +## Example repository + +The examples in this guide use the [deephaven-python-packaging](https://github.com/deephaven-examples/deephaven-python-packaging) repository. It demonstrates three complete packaging scenarios with working code, sample data, and comprehensive documentation. + +To explore the examples, clone the repository: + +```bash +git clone https://github.com/deephaven-examples/deephaven-python-packaging.git +cd deephaven-python-packaging +``` + +The repository contains three example packages: + +- `my_dh_library/` - Library-only package with reusable query functions. +- `my_dh_cli/` - CLI-only package with command-line tools. +- `my_dh_toolkit/` - Combined package with both library and CLI functionality. + +## Package structure + +Modern Python packages use the **src-layout**, which is the recommended structure by the [Python Packaging Authority](https://packaging.python.org/en/latest/guides/writing-pyproject-toml/). This layout keeps source code separate from tests and configuration files: + +``` +my_dh_project/ +├── src/ +│ └── my_dh_package/ +│ ├── __init__.py +│ ├── queries.py +│ └── utils.py +├── pyproject.toml +└── README.md +``` + +### Key components + +- **`src/`** - Source directory containing the package code. +- **`my_dh_package/`** - The Python package (directory name used in imports). +- **`__init__.py`** - Makes the directory importable and can export public API. +- **`pyproject.toml`** - Defines package metadata, dependencies, and entry points. +- **Module files** - Python files containing your functions and classes. + +The package name under `src/` determines how users import your code. For example, with `src/my_dh_library/`, users import via `from my_dh_library import ...`. + +## Packaging scenarios + +Different projects have different needs. The example repository demonstrates three common scenarios: + +### Library-only package + +Package reusable code without CLI tools. Other projects import your modules. + +**Structure:** + +``` +my_dh_library/ +├── src/ +│ └── my_dh_library/ +│ ├── __init__.py +│ ├── queries.py +│ └── utils.py +├── pyproject.toml +└── README.md +``` + +**Usage:** + +```python +from my_dh_library.queries import filter_by_threshold, add_computed_columns +from deephaven import read_csv + +data = read_csv("data.csv") +filtered = filter_by_threshold(data, "Score", 75.0) +``` + +**When to use:** + +- Creating reusable utilities for other projects. +- No command-line interface needed. +- Code will be imported, not executed directly. + +### CLI-only package + +Package executable command-line tools without exposing library code. + +**Structure:** + +``` +my_dh_cli/ +├── src/ +│ └── my_dh_cli/ +│ ├── __init__.py +│ ├── __main__.py +│ └── cli.py +├── pyproject.toml +└── README.md +``` + +**Usage:** + +```python +# CLI functions are used within a Python session +from my_dh_cli.cli import my_dh_query +result = my_dh_query("data.csv", verbose=True) +``` + +**When to use:** + +- Building command-line tools for data processing +- Want clean function interfaces +- No library code to expose to other projects + +### Combined package + +Package both reusable library code and command-line tools. + +**Structure:** + +``` +my_dh_toolkit/ +├── src/ +│ └── my_dh_toolkit/ +│ ├── __init__.py +│ ├── __main__.py +│ ├── cli.py +│ ├── processor.py +│ ├── queries.py +│ └── utils.py +├── pyproject.toml +└── README.md +``` + +**Usage:** + +```python +# As a library +from my_dh_toolkit.queries import filter_by_threshold +from my_dh_toolkit import my_dh_query + +# Use library functions +data = read_csv("data.csv") +filtered = filter_by_threshold(data, "Score", 75.0) + +# Or use CLI functions +result = my_dh_query("data.csv", verbose=True) +``` + +**When to use:** + +- Need both library and CLI functionality +- Want to provide multiple interfaces to the same code +- Library functions are useful independently + +## Create a new package + +
+Step-by-step instructions for creating packages from scratch + +This section walks through creating each type of package from scratch. + +### Create a library-only package + +Create the directory structure: + +```bash +mkdir -p my_dh_library/src/my_dh_library +cd my_dh_library +``` + +Create `pyproject.toml`: + +```toml +[build-system] +requires = ["setuptools>=61.0", "wheel"] +build-backend = "setuptools.build_meta" + +[project] +name = "my_dh_library" +version = "0.1.0" +description = "Reusable Deephaven query functions" +readme = "README.md" +requires-python = ">=3.8" +dependencies = [ + "deephaven-server>=0.35.0", +] + +[tool.setuptools.packages.find] +where = ["src"] +``` + +Create `src/my_dh_library/__init__.py`: + +```python +"""My Deephaven package for data processing.""" + +__version__ = "0.1.0" + +from my_dh_library.queries import filter_by_threshold, add_computed_columns, summarize_by_group + +__all__ = ["filter_by_threshold", "add_computed_columns", "summarize_by_group"] +``` + +Create `src/my_dh_library/utils.py`: + +```python +"""Utility functions for working with Deephaven tables.""" + +from deephaven.table import Table + + +def validate_columns(table: Table, required_columns: list[str], raise_error: bool = False) -> bool: + """Check if table has all required columns. + + Args: + table: The table to validate + required_columns: List of column names that must be present + raise_error: If True, raises ValueError when columns are missing + + Returns: + True if all columns are present, False otherwise + + Raises: + ValueError: If raise_error is True and columns are missing + """ + table_columns = [col.name for col in table.columns] + missing = [col for col in required_columns if col not in table_columns] + + if missing: + if raise_error: + raise ValueError( + f"Column(s) {missing} not found in table. " + f"Available columns: {', '.join(table_columns)}" + ) + return False + return True + + +def get_table_info(table: Table) -> dict: + """Get basic information about a table.""" + return { + "num_rows": table.size, + "num_columns": len(table.columns), + "columns": [col.name for col in table.columns], + } +``` + +Create `src/my_dh_library/queries.py`: + +```python +"""Reusable Deephaven query functions.""" + +from deephaven.table import Table +from deephaven import agg +from .utils import validate_columns + + +def filter_by_threshold(table: Table, column: str, threshold: float) -> Table: + """Filter table rows where column value exceeds threshold.""" + validate_columns(table, [column], raise_error=True) + return table.where(f"{column} > {threshold}") + + +def add_computed_columns(table: Table) -> Table: + """Add commonly used computed columns to a table.""" + validate_columns(table, ["Value"], raise_error=True) + return table.update( + [ + "DoubleValue = Value * 2", + "IsHigh = Value > 100", + ] + ) + + +def summarize_by_group(table: Table, group_col: str, value_col: str) -> Table: + """Create summary statistics grouped by a column.""" + validate_columns(table, [group_col, value_col], raise_error=True) + return table.agg_by( + [ + agg.sum_(f"Sum = {value_col}"), + agg.avg(f"Avg = {value_col}"), + agg.count_("Count"), + ], + by=[group_col], + ) +``` + +Create `README.md` with installation and usage instructions. + +### Create a CLI-only package + +Create the directory structure: + +```bash +mkdir -p my_dh_cli/src/my_dh_cli +cd my_dh_cli +``` + +Create `pyproject.toml`: + +```toml +[build-system] +requires = ["setuptools>=61.0", "wheel"] +build-backend = "setuptools.build_meta" + +[project] +name = "my_dh_cli" +version = "0.1.0" +description = "Command-line tool for data processing" +readme = "README.md" +requires-python = ">=3.8" +dependencies = [ + "deephaven-server>=0.35.0", + "click>=8.0.0", +] + +[project.scripts] +my-dh-query = "my_dh_cli.cli:app" + +[tool.setuptools.packages.find] +where = ["src"] +``` + +Create `src/my_dh_cli/__init__.py`: + +```python +"""My Deephaven package for data processing.""" + +__version__ = "0.1.0" +``` + +Create `src/my_dh_cli/__main__.py`: + +```python +from my_dh_cli.cli import app + +if __name__ == "__main__": + app() +``` + +Create `src/my_dh_cli/cli.py`: + +```python +import click + + +def my_dh_query(input_file: str, verbose: bool = False): + """Read a CSV file and perform a simple query operation on the data.""" + from deephaven import read_csv + from pathlib import Path + + input_path = Path(input_file) + + if not input_path.exists(): + raise click.ClickException(f"Input file does not exist: '{input_path}'") + if not input_path.is_file(): + raise click.ClickException(f"Input path is not a file: '{input_path}'") + + if verbose: + click.echo(f"Processing {input_file}...") + + try: + source = read_csv(input_file) + except Exception as e: + raise click.ClickException(f"Failed to read CSV file '{input_file}': {e}") + + column_names = [col.name for col in source.columns] + if "Score" not in column_names: + raise click.ClickException( + f"File '{input_path.name}' is missing required column 'Score'. " + f"Available columns: {', '.join(column_names)}" + ) + + result = source.update(formulas=["DoubleScore = Score * 2"]) + + if verbose: + click.echo(f"Processed {result.size} rows") + + return result + + +@click.command() +@click.argument("input_file", type=click.Path(exists=True)) +@click.option("--verbose", "-v", is_flag=True, help="Enable verbose output") +def app(input_file: str, verbose: bool) -> None: + """Process data with Deephaven.""" + result = my_dh_query(input_file, verbose) + click.echo("Processing complete!") + + +if __name__ == "__main__": + app() +``` + +Create `README.md` with installation and usage instructions. + +### Create a combined package + +Create the directory structure: + +```bash +mkdir -p my_dh_toolkit/src/my_dh_toolkit +cd my_dh_toolkit +``` + +Create `pyproject.toml`: + +```toml +[build-system] +requires = ["setuptools>=61.0", "wheel"] +build-backend = "setuptools.build_meta" + +[project] +name = "my_dh_toolkit" +version = "0.1.0" +description = "Deephaven library and CLI tools" +readme = "README.md" +requires-python = ">=3.8" +dependencies = [ + "deephaven-server>=0.35.0", + "click>=8.0.0", +] + +[project.scripts] +my-dh-query = "my_dh_toolkit.cli:app" +my-dh-process = "my_dh_toolkit.processor:process" + +[tool.setuptools.packages.find] +where = ["src"] +``` + +Create `src/my_dh_toolkit/__init__.py`: + +```python +"""My Deephaven package for data processing.""" + +__version__ = "0.1.0" + +from my_dh_toolkit.cli import my_dh_query +from my_dh_toolkit.processor import batch_process + +__all__ = ["my_dh_query", "batch_process"] +``` + +Create `src/my_dh_toolkit/__main__.py`: + +```python +from my_dh_toolkit.cli import app + +if __name__ == "__main__": + app() +``` + +Create the library modules (`queries.py`, `utils.py`) using the same code as the library-only package. + +Create `src/my_dh_toolkit/cli.py` using the same code as the CLI-only package. + +Create `src/my_dh_toolkit/processor.py`: + +```python +import click +from pathlib import Path + + +def batch_process(directory: str, output_dir: str, verbose: bool = False) -> None: + """Process multiple CSV files from a directory.""" + from deephaven import read_csv, write_csv + + input_path = Path(directory) + output_path = Path(output_dir) + + if not input_path.exists(): + raise click.ClickException(f"Input directory does not exist: '{input_path}'") + if not input_path.is_dir(): + raise click.ClickException(f"Input path is not a directory: '{input_path}'") + + try: + output_path.mkdir(parents=True, exist_ok=True) + except PermissionError: + raise click.ClickException(f"Permission denied: Cannot create output directory '{output_path}'") + except OSError as e: + raise click.ClickException(f"Failed to create output directory '{output_path}': {e}") + + csv_files = list(input_path.glob("*.csv")) + + if verbose: + click.echo(f"Found {len(csv_files)} CSV files to process") + + for csv_file in csv_files: + if verbose: + click.echo(f"Processing {csv_file.name}...") + + table = read_csv(str(csv_file)) + + column_names = [col.name for col in table.columns] + if "Score" not in column_names: + raise click.ClickException( + f"File '{csv_file.name}' is missing required column 'Score'. " + f"Available columns: {', '.join(column_names)}" + ) + + processed = table.update(formulas=["ProcessedScore = Score * 2"]) + + output_file = output_path / f"processed_{csv_file.name}" + try: + write_csv(processed, str(output_file)) + except Exception as e: + raise click.ClickException(f"Failed to write output file '{output_file}': {e}") + + if verbose: + click.echo(f" Processed {processed.size} rows -> {output_file.name}") + + +@click.command() +@click.argument("directory", type=click.Path(exists=True, file_okay=False)) +@click.option("--output", "-o", default="./output", help="Output directory") +@click.option("--verbose", "-v", is_flag=True, help="Enable verbose output") +def process(directory: str, output: str, verbose: bool) -> None: + """Batch process CSV files with Deephaven.""" + batch_process(directory, output, verbose) + click.echo("Batch processing complete!") + + +if __name__ == "__main__": + process() +``` + +Create `README.md` with installation and usage instructions. + +
+ +## Configure `pyproject.toml` + +The [`pyproject.toml`](https://packaging.python.org/en/latest/guides/writing-pyproject-toml/) file defines your package configuration. + +### Configuration options + +Here's a detailed breakdown of `pyproject.toml` for a library-only package: + +```toml +[build-system] +requires = ["setuptools>=61.0", "wheel"] +build-backend = "setuptools.build_meta" + +[project] +name = "my_dh_library" +version = "0.1.0" +description = "Reusable Deephaven query functions" +readme = "README.md" +requires-python = ">=3.8" +dependencies = [ + "deephaven-server>=0.35.0", +] + +[tool.setuptools.packages.find] +where = ["src"] +``` + +### Key sections + +- **`[build-system]`** - Specifies setuptools as the build backend +- **`[project]`** - Package metadata and dependencies +- **`name`** - Project name (used for `pip install`) +- **`dependencies`** - Required packages installed automatically +- **`[tool.setuptools.packages.find]`** - Tells setuptools to find packages in `src/` + +For CLI packages, add a `[project.scripts]` section: + +```toml +[project.scripts] +my-dh-query = "my_dh_cli.cli:app" +``` + +This creates a command-line entry point that calls the `app` function from `my_dh_cli.cli`. + +## Managing dependencies + +Dependencies are specified in the `dependencies` field: + +```toml +[project] +dependencies = [ + "deephaven-server>=0.35.0", + "click>=8.0.0", + "pandas>=2.0.0", +] +``` + +### Version constraints + +Use version specifiers to control which versions are acceptable: + +- `>=0.35.0` - Minimum version (0.35.0 or higher) +- `>=2.0.0,<3.0.0` - Version range (2.x only) +- `~=1.24.0` - Compatible release (>=1.24.0, <1.25.0) +- `==1.0.0` - Exact version (not recommended for libraries) + +### Optional dependencies + +Define optional feature sets that users can install separately: + +```toml +[project.optional-dependencies] +visualization = [ + "matplotlib>=3.7.0", + "seaborn>=0.12.0", +] +dev = [ + "pytest>=7.0.0", + "black>=23.0.0", +] +``` + +Users can install optional dependencies: + +```bash +pip install my_dh_library[visualization] +pip install my_dh_library[visualization,dev] +``` + +## Installation and usage + +### Install a package + +Install from source in editable mode for development: + +```bash +cd my_dh_library +pip install -e . +``` + +Or install normally: + +```bash +pip install . +``` + +### Use a library package + +After installation, import and use the library functions: + +```python +# Start the Deephaven server +from deephaven_server import Server +server = Server(port=10000, jvm_args=["-Xmx4g"]) +server.start() + +# Import and use library functions +from my_dh_library.queries import filter_by_threshold +from deephaven import read_csv + +data = read_csv("data.csv") +filtered = filter_by_threshold(data, "Score", 75.0) +``` + +> [!NOTE] +> All Deephaven functionality requires a running server. Start the server before importing Deephaven modules. + +### Use CLI functions + +CLI functions must be used within the same Python session as the server: + +```python +# Start the Deephaven server +from deephaven_server import Server +server = Server(port=10000, jvm_args=["-Xmx4g"]) +server.start() + +# Use CLI functions +from my_dh_cli.cli import my_dh_query +result = my_dh_query("data.csv", verbose=True) +``` + +## Building and distributing + +Build a distributable wheel: + +```bash +cd my_dh_library +pip install build +python -m build +``` + +This creates a `.whl` file in `dist/` that can be: + +- Installed locally: `pip install dist/my_dh_library-0.1.0-py3-none-any.whl` +- Distributed to others +- Published to PyPI: `python -m twine upload dist/*` + +## Best practices + +### Package structure + +- Use the src-layout for all packages +- Keep package names lowercase with underscores +- Match the package directory name to the import name +- Include `__init__.py` in all package directories + +### Dependencies + +- Specify minimum versions for Deephaven and critical dependencies +- Use version ranges for flexibility +- Group related optional dependencies +- Document any system-level dependencies + +### Documentation + +- Include a comprehensive README.md +- Document all public functions and classes +- Provide usage examples +- Explain server initialization requirements + +### Testing + +- Write tests for all public functions +- Test with different Deephaven versions +- Include sample data for testing +- Document how to run tests + +## Server initialization + +Deephaven requires a running server before using any Deephaven functionality. The server must be initialized in the same Python process that uses Deephaven: + +```python +from deephaven_server import Server + +# Initialize and start the server +server = Server(port=10000, jvm_args=["-Xmx4g"]) +server.start() + +# Now you can import and use Deephaven +from deephaven import read_csv +data = read_csv("data.csv") +``` + +### Key points + +- Each Python process has its own JVM +- Starting a server in one terminal doesn't help another terminal +- CLI tools must run in the same session as the server +- The server uses approximately 4GB of memory by default (configurable via `jvm_args`) + +## Next steps + +The [deephaven-python-packaging](https://github.com/deephaven-examples/deephaven-python-packaging) repository provides complete, working examples of all three packaging scenarios. Clone the repository and explore the examples to see how to structure your own Deephaven packages. + +Each example includes: + +- Complete source code +- Configured `pyproject.toml` +- Sample data files +- Comprehensive README +- Usage examples + +## Related documentation + +- [Install and use Python packages](https://deephaven.io/core/docs/how-to-guides/install-and-use-python-packages/) +- [Use the Deephaven Python package](https://deephaven.io/core/docs/how-to-guides/deephaven-python-package/) +- [Python Packaging User Guide](https://packaging.python.org/en/latest/guides/writing-pyproject-toml/) +- [Creating command-line tools](https://packaging.python.org/en/latest/guides/creating-command-line-tools/) +- [Setuptools documentation](https://setuptools.pypa.io/) +- [Click documentation](https://click.palletsprojects.com/)