1  #+TITLE: Asteroid Radio - Development Guide
  2  #+AUTHOR: Asteroid Radio Development Team
  3  #+DATE: 2025-10-26
  4  
  5  * Development Setup
  6  
  7  #+BEGIN_QUOTE
  8  *Note on Package Managers*: Examples use =apt= (Debian/Ubuntu). Replace with your distribution's package manager (=dnf=, =pacman=, =zypper=, =apk=, etc.).
  9  #+END_QUOTE
 10  
 11  ** Prerequisites
 12  
 13  *** System Dependencies
 14  - SBCL (Steel Bank Common Lisp)
 15  - Quicklisp package manager
 16  - Git version control
 17  - Docker and Docker Compose
 18  - PostgreSQL (for production database)
 19  - taglib for metadata extraction
 20  
 21  *** Lisp Dependencies (via Quicklisp)
 22  
 23  All Lisp dependencies are automatically installed via Quicklisp when you run =(ql:quickload :asteroid)=:
 24  
 25  **** Core Framework
 26  - =radiance= - Web framework and module system
 27  - =slynk= - SLIME/SLY development server
 28  - =i-log4cl= - Logging interface
 29  - =r-clip= - CLIP template processor (Radiance)
 30  - =r-simple-rate= - Rate limiting (Radiance)
 31  - =r-simple-profile= - User profiles (Radiance)
 32  - =r-data-model= - Data modeling (Radiance)
 33  
 34  **** Utilities & Libraries
 35  - =lass= - CSS preprocessing in Lisp
 36  - =cl-json= - JSON encoding/decoding
 37  - =alexandria= - Common Lisp utilities
 38  - =local-time= - Time and date handling
 39  - =taglib= - Audio metadata extraction
 40  - =ironclad= - Cryptographic functions
 41  - =babel= - Character encoding conversion
 42  - =cl-fad= - File and directory operations
 43  - =bordeaux-threads= - Portable threading
 44  - =drakma= - HTTP client
 45  - =usocket= - Universal socket library
 46  - =cl-ppcre= - Perl-compatible regular expressions
 47  
 48  **** Radiance Interfaces
 49  - =:auth= - Authentication interface
 50  - =:database= - Database interface
 51  - =:user= - User management interface
 52  
 53  *** Ubuntu/Debian Installation
 54  #+BEGIN_SRC bash
 55  # Install system packages
 56  sudo apt update
 57  sudo apt install sbcl git docker.io docker-compose postgresql libtagc0-dev
 58  
 59  # Add user to docker group
 60  sudo usermod -a -G docker $USER
 61  # Log out and back in for group changes to take effect
 62  
 63  # Install Quicklisp (if not already installed)
 64  curl -O https://beta.quicklisp.org/quicklisp.lisp
 65  sbcl --load quicklisp.lisp --eval "(quicklisp-quickstart:install)" --quit
 66  
 67  # Note: PostgreSQL runs in Docker for development
 68  # See docs/POSTGRESQL-SETUP.org for database configuration
 69  #+END_SRC
 70  
 71  ** Project Setup
 72  
 73  *** Clone Repository
 74  #+BEGIN_SRC bash
 75  git clone https://github.com/fade/asteroid.git
 76  cd asteroid
 77  #+END_SRC
 78  
 79  *** Install Lisp Dependencies
 80  #+BEGIN_SRC bash
 81  # Start SBCL and load the system
 82  sbcl
 83  (ql:quickload :asteroid)
 84  #+END_SRC
 85  
 86  *** ASDF Configuration (Optional but Recommended)
 87  For easier development, configure ASDF to find the asteroid system:
 88  #+BEGIN_SRC bash
 89  # Create ASDF source registry configuration
 90  mkdir -p ~/.config/common-lisp
 91  cat > ~/.config/common-lisp/source-registry.conf
 92  ;; -*-lisp-*-
 93  (:source-registry
 94   (:tree "/path/to/your/projects/")
 95   :inherit-configuration)
 96  #+END_SRC
 97  
 98  This allows you to load the asteroid system from any directory without changing paths.
 99  
100  * Development Workflow
101  
102  ** Local Development Server
103  
104  *** Starting Development Environment
105  #+BEGIN_SRC bash
106  # Start Docker streaming services
107  cd docker/
108  docker compose up -d
109  
110  # Verify containers are running
111  docker compose ps
112  
113  # View logs
114  docker compose logs -f
115  
116  # Start RADIANCE web server (local development)
117  sbcl --eval "(ql:quickload :asteroid)" --eval "(asteroid:start-server)"
118  #+END_SRC
119  
120  *** Development URLs
121  - *Web Interface*: http://localhost:8080/asteroid/
122  - *Admin Panel*: http://localhost:8080/asteroid/admin
123  - *User Management*: http://localhost:8080/asteroid/admin/users
124  - *Web Player*: http://localhost:8080/asteroid/player
125  - *API Base*: http://localhost:8080/api/asteroid/
126  - *Live Stream*: http://localhost:8000/asteroid.mp3
127  - *Icecast Admin*: http://localhost:8000/admin/ (admin/asteroid_admin_2024)
128  
129  ** Music Library Management
130  
131  *** Directory Structure
132  The music directory is located directly under the asteroid root directory:
133  #+BEGIN_SRC
134  asteroid/music/                 # Music directory (can be symlink)
135  ├── artist1/
136  │   ├── album1/
137  │   │   ├── track1.mp3
138  │   │   └── track2.flac
139  │   └── album2/
140  │       └── track3.ogg
141  └── artist2/
142      └── single.wav
143  #+END_SRC
144  
145  The =music/= directory can be:
146  - A regular directory with music files
147  - A symlink to your actual music collection
148  - Multiple subdirectories or symlinks within it
149  
150  *** Recursive Scanning Capabilities
151  The Asteroid application includes built-in recursive directory scanning:
152  - *Function*: =scan-music-library= in =stream-media.lisp=
153  - *Supports*: MP3, FLAC, OGG, WAV formats
154  - *Recursive*: Automatically scans all subdirectories
155  - *Metadata*: Extracts title, artist, album, duration using taglib
156  - *Database*: Stores track information in RADIANCE database
157  
158  *** Adding Music to Development Environment
159  #+BEGIN_SRC bash
160  # Option 1: Copy music files directly
161  cp -r /path/to/your/music/* music/
162  
163  # Option 2: Symlink entire music directory
164  ln -s /path/to/existing/music music
165  
166  # Option 3: Symlink subdirectories within music/
167  mkdir -p music
168  ln -s /path/to/collection1 music/collection1
169  ln -s /path/to/collection2 music/collection2
170  
171  # Option 4: Mount remote directory (for large collections)
172  # Edit docker-compose.yml to change volume mount:
173  # volumes:
174  #   - /mnt/remote-music:/app/music:ro
175  
176  # Trigger library scan via API
177  curl -X POST http://localhost:8080/api/asteroid/admin/scan-library
178  #+END_SRC
179  
180  ** Code Organization
181  
182  *** Main Components
183  - =asteroid.lisp= - Main server with RADIANCE routes and API endpoints
184  - =asteroid.asd= - System definition with dependencies
185  - =template/= - CLIP HTML templates for web interface
186  - =static/= - CSS stylesheets and static assets
187  - =asteroid-radio.liq= - Liquidsoap streaming configuration
188  
189  *** Key Modules
190  - *Web Routes*: RADIANCE framework with =#@= URL patterns
191  - *Database*: RADIANCE DB abstraction for track metadata
192  - *Streaming*: Docker containers with Icecast2 and Liquidsoap
193  - *File Processing*: Metadata extraction and library management
194  - *Docker Integration*: Containerized streaming infrastructure
195  
196  ** Development Practices
197  
198  *** Code Style
199  - Use 2-space indentation for Lisp code
200  - Follow Common Lisp naming conventions
201  - Document functions with docstrings
202  - Use meaningful variable and function names
203  
204  *** Database Development
205  #+BEGIN_SRC lisp
206  ;; Always use quoted symbols for field names
207  (db:select 'tracks (db:query (:= 'artist "Artist Name")))
208  
209  ;; Primary key is "_id" internally, "id" in JSON responses
210  (gethash "_id" track-record)
211  #+END_SRC
212  
213  *** Template Development with CLIP
214  
215  Asteroid Radio uses CLIP (Common Lisp HTML Processor) for templating. Templates are in the =template/= directory.
216  
217  **** Custom =data-text= Attribute Processor
218  
219  We define a custom CLIP attribute processor in =template-utils.lisp= for dynamic text replacement:
220  
221  #+BEGIN_SRC lisp
222  ;; Defined in template-utils.lisp
223  (clip:define-attribute-processor data-text (node value)
224    "Process data-text attribute - replaces node text content with clipboard value"
225    (plump:clear node)
226    (plump:make-text-node node (clip:clipboard value)))
227  #+END_SRC
228  
229  **** Using =data-text= in Templates
230  
231  In your HTML templates (=.chtml= files):
232  
233  #+BEGIN_SRC html
234  <!-- The data-text attribute gets replaced with the value from the plist -->
235  <h1 data-text="page-title">Default Title</h1>
236  <span data-text="username">Guest</span>
237  <p data-text="status-message">Loading...</p>
238  #+END_SRC
239  
240  **** Rendering Templates from Lisp
241  
242  In your route handlers:
243  
244  #+BEGIN_SRC lisp
245  (define-page my-page #@"/my-page" ()
246    (render-template-with-plist "my-template"
247                                :page-title "My Page"
248                                :username (user:username (auth:current))
249                                :status-message "Ready"))
250  #+END_SRC
251  
252  **** How It Works
253  
254  1. =render-template-with-plist= passes keyword arguments to CLIP
255  2. CLIP processes the template and finds =data-text= attributes
256  3. The custom processor replaces the node's text with the value from the "clipboard" (keyword args)
257  4. Default text in the HTML is replaced with dynamic content
258  
259  **** CLIP Documentation
260  
261  - **CLIP GitHub**: https://github.com/Shinmera/clip
262  - **Attribute Processors**: Custom processors extend CLIP's functionality
263  - **Standard CLIP**: Uses =lquery= for more complex DOM manipulation
264  - **Our Approach**: Simple =data-text= processor for most use cases
265  
266  **** Template Development Tips
267  
268  - Keep templates in =template/= directory
269  - Use =data-text= for simple text replacement
270  - Test template changes with browser refresh (templates are cached)
271  - Clear cache during development: =(clear-template-cache)=
272  - Maintain responsive design principles
273  
274  *** CSS Development with LASS
275  - CSS is generated dynamically from =static/asteroid.lass= using LASS (Lisp Augmented Style Sheets)
276  - Edit the =.lass= file, not the generated =.css= file
277  - CSS is automatically compiled when the server starts via =compile-styles= function
278  - Use Lisp syntax for CSS: =(body :background "#0a0a0a" :color "#00ffff")=
279  - Supports nested selectors, variables, and programmatic CSS generation
280  
281  ** Testing
282  
283  *** Manual Testing Checklist
284  - [ ] Web interface loads correctly
285  - [ ] Admin panel functions work
286  - [ ] File upload and processing works
287  - [ ] Live stream plays audio
288  - [ ] Database queries return expected results
289  - [ ] API endpoints respond correctly
290  
291  *** Docker Container Testing
292  #+BEGIN_SRC bash
293  # Check container status
294  docker compose ps
295  
296  # Test stream connectivity
297  curl -I http://localhost:8000/asteroid.mp3
298  
299  # Test with media player
300  vlc http://localhost:8000/asteroid.mp3
301  
302  # Check container logs
303  docker compose logs icecast
304  docker compose logs liquidsoap
305  #+END_SRC
306  
307  *** API Testing
308  
309  Asteroid Radio includes a comprehensive automated test suite:
310  
311  #+BEGIN_SRC bash
312  # Run full test suite
313  ./test-server.sh
314  
315  # Run with verbose output
316  ./test-server.sh -v
317  
318  # Test specific endpoints manually
319  curl http://localhost:8080/api/asteroid/status
320  curl http://localhost:8080/api/asteroid/tracks
321  curl -X POST http://localhost:8080/api/asteroid/player/play -d "track-id=123"
322  #+END_SRC
323  
324  See [[file:TESTING.org][Testing Guide]] for complete documentation.
325  
326  *** API Endpoint Structure
327  
328  All API endpoints use Radiance's =define-api= macro and follow this pattern:
329  
330  - Base URL: =/api/asteroid/=
331  - Response format: JSON
332  - Authentication: Session-based for protected endpoints
333  
334  See [[file:API-ENDPOINTS.org][API Endpoints Reference]] for complete API documentation.
335  
336  ** Debugging
337  
338  *** Common Development Issues
339  
340  **** Stream Not Playing
341  - Check Docker container status: =docker compose ps=
342  - Check Liquidsoap container logs: =docker compose logs liquidsoap=
343  - Check Icecast2 container logs: =docker compose logs icecast=
344  - Verify music files exist in =docker/music/library/=
345  - Restart containers: =docker compose restart=
346  
347  **** Database Errors
348  - Ensure proper field name quoting in queries
349  - Check RADIANCE database configuration
350  - Verify database file permissions
351  
352  **** Template Rendering Issues
353  - Check CLIP template syntax
354  - Verify template file paths
355  - Test with simplified templates first
356  
357  *** Debug Configuration
358  #+BEGIN_SRC bash
359  # Enable verbose logging in Docker containers
360  # Edit docker/liquidsoap/asteroid-radio.liq
361  settings.log.level := 4
362  settings.log.stdout := true
363  settings.log.file := true
364  settings.log.file.path := "/var/log/liquidsoap/asteroid.log"
365  
366  # View real-time container logs
367  docker compose logs -f liquidsoap
368  docker compose logs -f icecast
369  #+END_SRC
370  
371  ** Contributing Guidelines
372  
373  *** Branch Strategy
374  - =main= - Stable production code
375  - =develop= - Integration branch for new features
376  - =feature/*= - Individual feature development
377  - =bugfix/*= - Bug fixes and patches
378  
379  *** Commit Messages
380  - Use clear, descriptive commit messages
381  - Reference issue numbers when applicable
382  - Keep commits focused on single changes
383  
384  *** Pull Request Process
385  1. Create feature branch from =develop=
386  2. Implement changes with tests
387  3. Update documentation if needed
388  4. Submit pull request with description
389  5. Address code review feedback
390  6. Merge after approval
391  
392  *** Code Review Checklist
393  - [ ] Code follows project style guidelines
394  - [ ] Functions are properly documented
395  - [ ] No hardcoded values or credentials
396  - [ ] Error handling is appropriate
397  - [ ] Performance considerations addressed
398  
399  ** Development Tools
400  
401  *** Recommended Editor Setup
402  - *Emacs*: SLIME for interactive Lisp development
403  
404  *** Useful Development Commands
405  #+BEGIN_SRC lisp
406  ;; Reload system during development
407  (ql:quickload :asteroid :force t)
408  
409  ;; Restart RADIANCE server
410  (radiance:shutdown)
411  (asteroid:start-server)
412  
413  ;; Clear database for testing
414  (db:drop 'tracks)
415  (asteroid:setup-database)
416  #+END_SRC
417  
418  ** Performance Considerations
419  
420  *** Development vs Production
421  - Use smaller music libraries in =docker/music/= for faster testing
422  - Enable debug logging in Docker containers only when needed
423  - Consider memory usage with large track collections in containers
424  - Test with realistic concurrent user loads using Docker scaling
425  - Use =docker compose.dev.yml= for development-specific settings
426  
427  *** Optimization Tips
428  - Cache database queries where appropriate
429  - Optimize playlist generation for large libraries
430  - Monitor memory usage during development
431  - Profile streaming performance under load
432  
433  * Configuration Files
434  - =radiance-core.conf.lisp= - RADIANCE framework configuration
435  - =docker/liquidsoap/asteroid-radio.liq= - Liquidsoap streaming setup
436  - =docker/icecast.xml= - Icecast2 server configuration
437  - =docker/docker-compose.yml= - Container orchestration
438  
439  ** Docker Development
440  #+BEGIN_SRC bash
441  # Start development containers
442  cd docker/
443  docker compose up -d
444  
445  # Build development container with changes
446  docker compose up --build
447  
448  # Access container shell for debugging
449  docker compose exec liquidsoap bash
450  docker compose exec icecast bash
451  
452  # Stop all containers
453  docker compose down
454  #+END_SRC
455  
456  * Troubleshooting
457  
458  ** Development Environment Issues
459  
460  *** SBCL/Quicklisp Problems
461  - Ensure Quicklisp is properly installed
462  - Check for conflicting Lisp installations
463  - Verify system dependencies are installed
464  
465  *** Docker Container Issues
466  - Check container status: =docker compose ps=
467  - Verify Docker daemon is running: =docker info=
468  - Check container logs: =docker compose logs [service]=
469  - Restart containers: =docker compose restart=
470  
471  *** Network Access Issues
472  - Check firewall settings for ports 8000, 8080
473  - Verify WSL networking configuration if applicable
474  - Test container networking: =docker compose exec liquidsoap ping icecast=
475  - Check port binding: =docker compose port icecast 8000=
476  
477  *** File Permission Issues
478  - Ensure =docker/music/= directory is accessible
479  - Check ownership: =ls -la docker/music/=
480  - Fix permissions: =sudo chown -R $USER:$USER docker/music/=
481  - Verify container volume mounts in =docker-compose.yml=
482  - For remote mounts: ensure network storage is accessible
483  
484  *** Music Library Issues
485  - Check if music files exist: =find docker/music/ -name "*.mp3" -o -name "*.flac"=
486  - Verify supported formats: MP3, FLAC, OGG, WAV
487  - Test recursive scanning: =curl -X POST http://localhost:8080/asteroid/api/scan-library=
488  - Check database for tracks: =curl http://localhost:8080/asteroid/api/tracks=
489  - For large collections: avoid network mounts, use local storage (see memory about 175+ files causing timeouts)
490  
491  ** Getting Help
492  - Check existing issues in project repository
493  - Review RADIANCE framework documentation
494  - Consult Liquidsoap manual for streaming issues
495  - Join our IRC chat room: **#asteroid.music** on **irc.libera.chat**
496  - Ask questions in project discussions
497  
498  This development guide provides the foundation for contributing to Asteroid Radio. For deployment and production considerations, see the Installation Guide and Performance Testing documentation.
499  
500  * Development Stack Links
501  
502  ** Core Technologies
503  - **SBCL** (Steel Bank Common Lisp): https://www.sbcl.org/
504  - **Quicklisp** (Common Lisp package manager): https://www.quicklisp.org/
505  - **ASDF** (Another System Definition Facility): https://common-lisp.net/project/asdf/
506  
507  ** Web Framework & Libraries
508  - **RADIANCE** (Web framework): https://shirakumo.github.io/radiance/
509  - **CLIP** (HTML templating): https://shinmera.github.io/clip/
510  - **LASS** (CSS in Lisp): https://shinmera.github.io/LASS/
511  - **Alexandria** (Utility library): https://alexandria.common-lisp.dev/
512  - **Local-Time** (Time handling): https://common-lisp.net/project/local-time/
513  
514  ** Audio & Streaming
515  - **Docker** (Containerization): https://www.docker.com/
516  - **Icecast2** (Streaming server): https://icecast.org/
517  - **Liquidsoap** (Audio streaming): https://www.liquidsoap.info/
518  - **TagLib** (Audio metadata): https://taglib.org/
519  
520  ** Database & Data
521  - **cl-json** (JSON handling): https://common-lisp.net/project/cl-json/
522  - **cl-fad** (File/directory utilities): https://edicl.github.io/cl-fad/
523  - **Ironclad** (Cryptography): https://github.com/sharplispers/ironclad
524  - **Babel** (Character encoding): https://common-lisp.net/project/babel/
525  
526  ** Development Tools
527  - **Emacs** (Editor): https://www.gnu.org/software/emacs/
528  - **SLIME** (Emacs Lisp IDE): https://common-lisp.net/project/slime/
529  - **Slynk** (SLIME backend): https://github.com/joaotavora/sly
530  - **Git** (Version control): https://git-scm.com/
531  
532  ** System Libraries
533  - **Bordeaux-Threads** (Threading): https://common-lisp.net/project/bordeaux-threads/
534  - **Drakma** (HTTP client): https://edicl.github.io/drakma/
535  - **CIFS-Utils** (Network file systems): https://wiki.samba.org/index.php/LinuxCIFS_utils
536  
537  ** Documentation & Standards
538  - **Common Lisp HyperSpec**: http://www.lispworks.com/documentation/HyperSpec/Front/
539  - **Docker Compose**: https://docs.docker.com/compose/
540  - **Org Mode** (Documentation format): https://orgmode.org/