devdocs.md
1 # Development Documentation 2 3 This comprehensive guide covers Planar's architecture, development setup, contribution guidelines, and best practices for extending the framework. 4 5 ## Architecture Overview 6 7 ### System Architecture 8 9 Planar follows a modular architecture with clear separation of concerns: 10 11 ``` 12 ┌─────────────────────────────────────────────────────────────┐ 13 │ User Interface Layer │ 14 ├─────────────────────────────────────────────────────────────┤ 15 │ Planar.jl (Main) │ PlanarInteractive.jl │ CLI Tools │ 16 ├─────────────────────────────────────────────────────────────┤ 17 │ Strategy Layer │ 18 ├─────────────────────────────────────────────────────────────┤ 19 │ Strategies.jl │ User Strategies │ StrategyStats│ 20 ├─────────────────────────────────────────────────────────────┤ 21 │ Execution Layer │ 22 ├─────────────────────────────────────────────────────────────┤ 23 │ SimMode.jl │ PaperMode.jl │ LiveMode.jl │ Executors.jl│ 24 ├─────────────────────────────────────────────────────────────┤ 25 │ Data & Exchange Layer │ 26 ├─────────────────────────────────────────────────────────────┤ 27 │ Exchanges.jl │ Data.jl │ Fetch.jl │ Processing.jl │ Metrics.jl│ 28 ├─────────────────────────────────────────────────────────────┤ 29 │ Foundation Layer │ 30 ├─────────────────────────────────────────────────────────────┤ 31 │Engine.jl│Instruments.jl│OrderTypes.jl│Collections.jl│Lang.jl│ 32 └─────────────────────────────────────────────────────────────┘ 33 ``` 34 35 ### Core Components 36 37 #### Engine Module 38 The foundation of Planar, providing: 39 - **Asset Management**: `AbstractAsset`, `Asset`, `Derivative` types 40 - **Instance System**: `AssetInstance` linking assets to [exchanges](exchanges.md) 41 - **Strategy Framework**: `Strategy{Mode}` parametric types 42 - **Core Abstractions**: Base types and interfaces 43 44 #### Execution Modes 45 Three distinct execution environments: 46 - **SimMode**: Fast synchronous [backtesting](guides/execution-modes.md#simulation)-mode) with [OHLCV data](guides/../guides/data-management.md#ohlcv-data) 47 - **PaperMode**: Real-time [simulation](guides/execution-modes.md#simulation-mode) with live data feeds 48 - **LiveMode**: Actual trading with [exchanges](exchanges.md) APIs 49 50 #### Data Pipeline 51 Comprehensive [data management](guides/../guides/data-management.md): 52 - **Fetch**: Data acquisition from [exchanges](exchanges.md) and external sources 53 - **Processing**: Data cleaning, resampling, and transformation 54 - **Storage**: Zarr-based large dataset storage with LMDB indexing 55 - **Watchers**: Real-time data monitoring and alerting 56 57 #### Exchange Integration 58 Unified [exchange](exchanges.md) interface: 59 - **[CCXT](exchanges.md#ccxt-integration) Integration**: 100+ [exchange](exchanges.md) support via Ccxt.jl 60 - **Custom Exchanges**: Framework for implementing proprietary APIs 61 - **Order Management**: Unified order types across exchanges 62 63 ### Module Relationships 64 65 ```mermaid 66 graph TD 67 A[Planar.jl] --> B[Engine.jl] 68 A --> C[Strategies.jl] 69 A --> D[SimMode.jl] 70 A --> E[PaperMode.jl] 71 A --> F[LiveMode.jl] 72 73 C --> B 74 D --> B 75 E --> B 76 F --> B 77 78 B --> G[Instruments.jl] 79 B --> H[OrderTypes.jl] 80 B --> I[Executors.jl] 81 82 D --> J[Data.jl] 83 E --> J 84 F --> J 85 86 J --> K[Fetch.jl] 87 J --> L[Processing.jl] 88 89 F --> M[Exchanges.jl] 90 E --> M 91 92 M --> N[ExchangeTypes.jl] 93 94 O[PlanarInteractive.jl] --> A 95 O --> P[Plotting.jl] 96 O --> Q[Opt.jl] 97 ``` 98 99 ### Data Flow Architecture 100 101 ```mermaid 102 sequenceDiagram 103 participant U as User Strategy 104 participant S as Strategy Engine 105 participant E as Execution Mode 106 participant D as Data Layer 107 participant X as Exchange 108 109 U->>S: Initialize Strategy 110 S->>D: Load Historical Data 111 D->>S: Return [OHLCV](guides/../guides/data-management.md#ohlcv-data) Data 112 113 loop Trading Loop 114 S->>U: Call Strategy Logic 115 U->>S: Generate Signals 116 S->>E: Execute Orders 117 E->>X: Place Orders (Live/Paper) 118 X->>E: Order Confirmation 119 E->>S: Update Positions 120 S->>D: Log Trade Data 121 end 122 ``` 123 124 ## Development Setup 125 126 ### Prerequisites 127 128 - **[Julia](https://julialang.org/) 1.11+**: Latest stable [Julia](https://julialang.org/) version 129 - **Git**: With submodule support 130 - **Python 3.8+**: For [CCXT](exchanges.md#ccxt-integration) integration (managed via CondaPkg) 131 - **Docker** (optional): For containerized development 132 133 ### Initial Setup 134 135 1. **Clone Repository**: 136 ```bash 137 git clone --recurse-submodules https://github.com/defnlnotme/Planar.jl 138 cd Planar.jl 139 ``` 140 141 2. **Environment Setup**: 142 ```bash 143 # Enable direnv (recommended) 144 direnv allow 145 146 # Or manually set environment 147 export JULIA_PROJECT=Planar 148 export JULIA_NUM_THREADS=$(nproc) 149 ``` 150 151 3. **Install Dependencies**: 152 153 ### Development Environment 154 155 #### Recommended Julia Setup 156 157 158 #### IDE Configuration 159 160 **VS Code with Julia Extension**: 161 ```json 162 { 163 "julia.environmentPath": "Planar", 164 "julia.enableTelemetry": false, 165 "julia.execution.resultDisplay": "both", 166 "julia.lint.run": true, 167 "julia.format.indent": 4, 168 "julia.format.margin": 92 169 } 170 ``` 171 172 **Vim/Neovim with LanguageServer.jl**: 173 ```lua 174 require'lspconfig'.julials.setup{ 175 settings = { 176 julia = { 177 environmentPath = "Planar", 178 format = { 179 indent = 4, 180 margin = 92 181 } 182 } 183 } 184 } 185 ``` 186 187 ## PlanarDev Utilities 188 189 The `PlanarDev` package assists developers by providing helper functions for working with Planar and for conducting tests. 190 191 ### Precompilation Control 192 193 To skip precompilation for selected modules, set the `JULIA_NOPRECOMP` environment variable: 194 195 196 Alternatively, you can manage environment variables with `direnv` (refer to the `.envrc` in the repository). To disable precompilation entirely for certain packages, use `JULIA_NOPRECOMP=all`. This is recommended only when altering low-level components of the module stack. Remember to clear the compilation cache when changing precompilation settings: 197 198 199 The `Exchanges` and `Fetch` packages contain a `compile.jl` file to generate precompile statements using [CompileBot.jl](https://github.com/aminya/CompileBot.jl). This is particularly useful for precompilation tasks that involve numerous web requests. However, this method is not currently used as it does not compile as many methods as `PrecompileTools`. 200 201 !!! warning "Custom Precompilation" 202 For custom method precompilation, enclose your code with `py_start_loop` and `py_stop_loop` from the Python package to prevent Pkg from stalling due to lingering threads. 203 204 ### Method Invalidation Strategy 205 206 The order of `using ...` statements when loading modules can influence method invalidation. To minimize invalidation, arrange the module imports starting with the ones most likely to cause invalidations to the ones least likely. For instance, placing `using Python` at the beginning can expedite loading times: 207 208 209 Modules known for heavy invalidations: 210 211 - Python 212 - Ccxt (initiates the Python async loop) 213 - Data (relies on Zarr and DataFrames) 214 - Plots (depends on Makie) 215 216 To reduce invalidations, include external modules in only one local package and then use that package as a dependency in other local packages. For instance, if `DataFrames` is a dependency of the local package `Data`, and you want to use `DataFrames` in the `Stats` package, do not add `DataFrames` to `Stats` dependencies. Instead, use `Data` and import `DataFrames` from there: 217 218 219 ### Handling Segfaults 220 221 In rare cases involving complex multi-threaded scenarios, disable and re-enable the garbage collector (GC) around the loading of Planar to avoid segmentation faults: 222 223 224 Refer to [PythonCall.jl issue #201](https://github.com/cjdoris/PythonCall.jl/issues/201) for more details. 225 226 ### Dependency Management 227 228 When adding dependencies, ensure that a dependency is only included in one subpackage. If you need the same dependency in another subpackage, add the first subpackage as the dependency, not the external module. 229 230 The order of `using` or `import` statements within packages is crucial. Always import external dependencies before internal ones to minimize method invalidations. 231 232 233 ## See Also 234 235 - **[Exchanges](exchanges.md)** - Exchange integration and configuration 236 - **[Config](config.md)** - Exchange integration and configuration 237 - **[Overview](troubleshooting/index.md)** - Troubleshooting: Troubleshooting and problem resolution 238 - **[Optimization](optimization.md)** - Performance optimization techniques 239 - **[Performance Issues](troubleshooting/performance-issues.md)** - Troubleshooting: Performance optimization techniques 240 - **[Data Management](guides/../guides/data-management.md)** - Guide: Data handling and management 241 242 ## Contribution Guidelines 243 244 ### Code Style and Standards 245 246 #### Julia Formatting 247 Planar uses JuliaFormatter with Blue style: 248 249 250 Configuration in `.JuliaFormatter.toml`: 251 ```toml 252 style = "blue" 253 margin = 92 254 indent = 4 255 always_for_in = true 256 whitespace_typedefs = true 257 whitespace_ops_in_indices = true 258 remove_extra_newlines = true 259 ``` 260 261 #### Naming Conventions 262 263 - **Types**: PascalCase (`MyCustomType`) 264 - **Functions**: snake_case (`my_function`) 265 - **Constants**: UPPER_CASE (`MY_CONSTANT`) 266 - **Modules**: PascalCase (`MyModule`) 267 - **Variables**: snake_case (`my_variable`) 268 269 #### Documentation Standards 270 271 All public functions must have docstrings: 272 273 274 ### Testing Patterns 275 276 #### Unit Testing Structure 277 278 279 #### Integration Testing 280 281 282 #### Property-Based Testing 283 284 285 ### Extension Best Practices 286 287 #### Creating New Modules 288 289 1. **Module Structure**: 290 ``` 291 MyNewModule/ 292 ├── Project.toml 293 ├── Manifest.toml 294 ├── src/ 295 │ ├── MyNewModule.jl 296 │ ├── types.jl 297 │ ├── functions.jl 298 │ └── utils.jl 299 ├── test/ 300 │ ├── runtests.jl 301 │ └── test_functions.jl 302 └── README.md 303 ``` 304 305 2. **Project.toml Template**: 306 ```toml 307 name = "MyNewModule" 308 uuid = "generate-new-uuid" 309 authors = ["Your Name <your.email@example.com>"] 310 version = "0.1.0" 311 312 [deps] 313 # Only essential dependencies 314 Engine = {path = "../Engine"} 315 316 [compat] 317 julia = "1.11" 318 ``` 319 320 3. **Module Definition**: 321 322 #### Implementing Custom Strategies 323 324 325 #### Performance Optimization Guidelines 326 327 1. **Type Stability**: 328 329 2. **Memory Allocation**: 330 331 3. **Benchmarking**: 332 333 ### Debugging and Profiling 334 335 #### Debugging Strategies 336 337 1. **Logging**: 338 339 2. **Interactive Debugging**: 340 341 3. **Testing Utilities**: 342 343 #### Performance Profiling 344 345 346 ### Continuous Integration 347 348 #### GitHub Actions Workflow 349 350 ```yaml 351 # .github/workflows/ci.yml 352 name: CI 353 on: 354 push: 355 branches: [main, develop] 356 pull_request: 357 branches: [main] 358 359 jobs: 360 test: 361 runs-on: ubuntu-latest 362 strategy: 363 matrix: 364 julia-version: ['1.11', '1.12'] 365 366 steps: 367 - uses: actions/checkout@v3 368 with: 369 submodules: recursive 370 371 - uses: julia-actions/setup-julia@v1 372 with: 373 version: ${{ matrix.julia-version }} 374 375 - uses: julia-actions/cache@v1 376 377 - uses: julia-actions/julia-buildpkg@v1 378 with: 379 project: Planar 380 381 - uses: julia-actions/julia-runtest@v1 382 with: 383 project: Planar 384 385 - uses: julia-actions/julia-processcoverage@v1 386 387 - uses: codecov/codecov-action@v3 388 ``` 389 390 ### Release Process 391 392 #### Version Management 393 394 1. **Semantic Versioning**: Follow SemVer (MAJOR.MINOR.PATCH) 395 - MAJOR: Breaking changes 396 - MINOR: New features, backward compatible 397 - PATCH: Bug fixes, backward compatible 398 399 2. **Release Checklist**: 400 - [ ] Update version in Project.toml 401 - [ ] Update CHANGELOG.md 402 - [ ] Run full test suite 403 - [ ] Update documentation 404 - [ ] Create release tag 405 - [ ] Build and test Docker images 406 407 #### Documentation Updates 408 409 410 ### Community Guidelines 411 412 #### Issue Reporting 413 414 When reporting issues, include: 415 - Julia version and Planar version 416 - Minimal reproducible example 417 - Expected vs actual behavior 418 - Error messages and stack traces 419 - System information (OS, architecture) 420 421 #### Pull Request Process 422 423 1. Fork the repository 424 2. Create feature branch from `develop` 425 3. Make changes with tests 426 4. Update documentation 427 5. Submit pull request with clear description 428 6. Address review feedback 429 7. Merge after approval 430 431 #### Code Review Standards 432 433 - **Functionality**: Does the code work as intended? 434 - **Performance**: Are there performance implications? 435 - **Style**: Does it follow project conventions? 436 - **Tests**: Are there adequate tests? 437 - **Documentation**: Is it properly documented? 438 - **Breaking Changes**: Are breaking changes justified and documented? 439 440 This comprehensive development guide ensures consistent, high-quality contributions to the Planar ecosystem.