Storage Add

README.md

@@ -4,7 +4,7 @@ emoji: 🎼
 colorFrom: gray
 colorTo: red
 sdk: gradio
-sdk_version: 5.34.1
+sdk_version: 5.34.2
 python_version: 3.12.8
 app_file: app.py
 pinned: true
@@ -40,6 +40,16 @@ This allows us to follow the same arraingement of the original melody.
 
 **Thank you Huggingface for the community grant to run this project**!!
 
+## Key Features
+
+- **Unlimited Audio Generation**: Generate music of any length by seamlessly stitching together segments
+- **User History**: Save and manage your generated music and access it later
+- **File Storage**: Generated files are automatically stored in a Hugging Face repository with shareable URLs
+- **Rich Metadata**: Each generated file includes detailed metadata about the generation parameters
+- **API Access**: Generate music programmatically using the REST API
+- **Background Customization**: Use custom images and settings for your music videos
+- **Melody Conditioning**: Use existing music to guide the generation process
+
 # Audiocraft
 ![docs badge](https://github.com/facebookresearch/audiocraft/workflows/audiocraft_docs/badge.svg)
 ![linter badge](https://github.com/facebookresearch/audiocraft/workflows/audiocraft_linter/badge.svg)
@@ -67,8 +77,6 @@ We use 20K hours of licensed music to train MusicGen. Specifically, we rely on a
 
 ## Installation
 Audiocraft requires Python 3.9, PyTorch 2.1.0, and a GPU with at least 16 GB of memory (for the medium-sized model). To install Audiocraft, you can run the following:
-
-```shell
 # Best to make sure you have torch installed first, in particular before installing xformers.
 # Don't run this if you already have PyTorch installed.
 pip install 'torch>=2.1'
@@ -76,8 +84,6 @@ pip install 'torch>=2.1'
 pip install -U audiocraft  # stable release
 pip install -U git+https://git@github.com/facebookresearch/audiocraft#egg=audiocraft  # bleeding edge
 pip install -e .  # or if you cloned the repo locally
-```
-
 ## Usage
 We offer a number of way to interact with MusicGen:
 1. A demo is also available on the [`facebook/MusicGen`  HuggingFace Space](https://huggingface.co/spaces/Surn/UnlimitedMusicGen) (huge thanks to all the HF team for their support).
@@ -88,9 +94,46 @@ We offer a number of way to interact with MusicGen:
   updated with contributions from @camenduru and the community.
 6. Finally, MusicGen is available in 🤗 Transformers from v4.31.0 onwards, see section [🤗 Transformers Usage](#-transformers-usage) below.
 
-### More info about Top-k, Top-p, Temperature and Classifier Free Guidance from ChatGPT
-6. Finally, MusicGen is available in 🤗 Transformers from v4.31.0 onwards, see section [🤗 Transformers Usage](#-transformers-usage) below.
+### Advanced Usage
+
+#### Programmatic Generation via API
+
+The `predict_simple` API endpoint allows generating music without using the UI:
+import requests
+
+# Example API call
+response = requests.post(
+    "https://huggingface.co/spaces/Surn/UnlimitedMusicGen/api/predict_simple",
+    json={
+        "model": "stereo-medium",  # Choose your model
+        "text": "Epic orchestral soundtrack with dramatic strings and percussion",
+        "duration": 60,  # Duration in seconds
+        "topk": 250,
+        "topp": 0,  # 0 means use topk instead
+        "temperature": 0.8,
+        "cfg_coef": 4.0,
+        "seed": 42,  # Use -1 for random seed
+        "overlap": 2,  # Seconds of overlap between segments
+        "video_orientation": "Landscape"  # or "Portrait"
+    }
+)
 
+# URLs to the generated content
+video_url, audio_url, seed = response.json()
+#### Custom Background Images
+
+You can use your own background images for the music video:
+
+1. Upload an image through the UI
+2. Or specify an image URL in the API call:response = requests.post(
+    "https://huggingface.co/spaces/Surn/UnlimitedMusicGen/api/predict_simple",
+    json={
+        # ... other parameters
+        "background": "https://example.com/your-image.jpg",
+        "video_orientation": "Landscape"
+    }
+)
+### More info about Top-k, Top-p, Temperature and Classifier Free Guidance from ChatGPT
 
 Top-k: Top-k is a parameter used in text generation models, including music generation models. It determines the number of most likely next tokens to consider at each step of the generation process. The model ranks all possible tokens based on their predicted probabilities, and then selects the top-k tokens from the ranked list. The model then samples from this reduced set of tokens to determine the next token in the generated sequence. A smaller value of k results in a more focused and deterministic output, while a larger value of k allows for more diversity in the generated music.
 
@@ -102,7 +145,53 @@ Classifier-Free Guidance: Classifier-Free Guidance refers to a technique used in
 
 These parameters, such as top-k, top-p, temperature, and classifier-free guidance, provide different ways to influence the output of a music generation model and strike a balance between creativity, diversity, coherence, and control. The specific values for these parameters can be tuned based on the desired outcome and user preferences.
 
-## API
+## API and Storage Integration
+
+UnlimitedMusicGen now offers enhanced API capabilities and file storage integration with Hugging Face repositories:
+
+### REST API Access
+
+The application exposes a simple REST API endpoint through Gradio that allows you to generate music programmatically:
+import requests
+
+# Basic API call example
+response = requests.post(
+    "https://your-app-url/api/predict_simple",
+    json={
+        "model": "medium",
+        "text": "4/4 120bpm electronic music with driving bass",
+        "duration": 30,
+        "temperature": 0.7,
+        "cfg_coef": 3.75,
+        "title": "My API Generated Track"
+    }
+)
+
+# The response contains URLs to the generated audio/video
+video_url, audio_url, seed = response.json()
+print(f"Generated music video: {video_url}")
+print(f"Generated audio file: {audio_url}")
+print(f"Seed used: {seed}")
+### File Storage
+
+Generated files are automatically uploaded to a Hugging Face dataset repository, providing:
+
+- Persistent storage of your generated audio and video files
+- Shareable URLs for easy distribution
+- Organization by user, timestamp, and metadata
+- Automatic handling of file paths and naming
+
+The storage system supports various file types including audio (.wav, .mp3), video (.mp4), and images (.png, .jpg).
+
+### Background Image Support
+
+You can now provide custom background images for your music videos:
+- Upload from your device
+- Use URL links to images (automatically downloaded and processed)
+- Choose between landscape and portrait orientations
+- Add title and generation settings overlay with customizable fonts and colors
+
+## Python API
 
 We provide a simple API and 10 pre-trained models. The pre trained models are:
 - `small`: 300M model, text to music only - [🤗 Hub](https://huggingface.co/facebook/musicgen-small)
@@ -121,14 +210,8 @@ In order to use MusicGen locally **you must have a GPU**. We recommend 16GB of m
 GPUs will be able to generate short sequences, or longer sequences with the `small` model.
 
 **Note**: Please make sure to have [ffmpeg](https://ffmpeg.org/download.html) installed when using newer version of `torchaudio`.
-You can install it with:
-```
-apt-get install ffmpeg
-```
-
+You can install it with:apt-get install ffmpeg
 See after a quick example for using the API.
-
-```python
 import torchaudio
 from audiocraft.models import MusicGen
 from audiocraft.data.audio import audio_write
@@ -145,22 +228,14 @@ wav = model.generate_with_chroma(descriptions, melody[None].expand(3, -1, -1), s
 
 for idx, one_wav in enumerate(wav):
     # Will save under {idx}.wav, with loudness normalization at -14 db LUFS.
-    audio_write(f'{idx}', one_wav.cpu(), model.sample_rate, strategy="loudness", loudness_compressor=True)
-```
-## 🤗 Transformers Usage
+    audio_write(f'{idx}', one_wav.cpu(), model.sample_rate, strategy="loudness", loudness_compressor=True)## 🤗 Transformers Usage
 
 MusicGen is available in the 🤗 Transformers library from version 4.31.0 onwards, requiring minimal dependencies 
 and additional packages. Steps to get started:
 
 1. First install the 🤗 [Transformers library](https://github.com/huggingface/transformers) from main:
-
-```
 pip install git+https://github.com/huggingface/transformers.git
-```
-
 2. Run the following Python code to generate text-conditional audio samples:
-
-```py
 from transformers import AutoProcessor, MusicgenForConditionalGeneration
 
 
@@ -174,26 +249,16 @@ inputs = processor(
 )
 
 audio_values = model.generate(**inputs, max_new_tokens=256)
-```
-
 3. Listen to the audio samples either in an ipynb notebook:
-
-```py
 from IPython.display import Audio
 
 sampling_rate = model.config.audio_encoder.sampling_rate
 Audio(audio_values[0].numpy(), rate=sampling_rate)
-```
-
 Or save them as a `.wav` file using a third-party library, e.g. `scipy`:
-
-```py
 import scipy
 
 sampling_rate = model.config.audio_encoder.sampling_rate
 scipy.io.wavfile.write("musicgen_out.wav", rate=sampling_rate, data=audio_values[0, 0].numpy())
-```
-
 For more details on using the MusicGen model for inference using the 🤗 Transformers library, refer to the 
 [MusicGen docs](https://huggingface.co/docs/transformers/main/en/model_doc/musicgen) or the hands-on 
 [Google Colab](https://colab.research.google.com/github/sanchit-gandhi/notebooks/blob/main/MusicGen.ipynb).
@@ -237,16 +302,12 @@ Yes. We will soon release the training code for MusicGen and EnCodec.
 
 Check [@camenduru tutorial on Youtube](https://www.youtube.com/watch?v=EGfxuTy9Eeo).
 
-## Citation
-```
-@article{copet2023simple,
+## Citation@article{copet2023simple,
       title={Simple and Controllable Music Generation},
       author={Jade Copet and Felix Kreuk and Itai Gat and Tal Remez and David Kant and Gabriel Synnaeve and Yossi Adi and Alexandre Défossez},
       year={2023},
       journal={arXiv preprint arXiv:2306.05284},
 }
-```
-
 ## License
 * The code in this repository is released under the MIT license as found in the [LICENSE file](LICENSE).
 * The weights in this repository are released under the CC-BY-NC 4.0 license as found in the [LICENSE_weights file](LICENSE_weights).

app.py

@@ -34,10 +34,12 @@ import modules.user_history
 from modules.version_info import versions_html, commit_hash, get_xformers_version
 from modules.gradio import *
 from modules.file_utils import get_file_parts, get_filename_from_filepath, convert_title_to_filename, get_unique_file_path, delete_file, download_and_save_image
+from modules.constants import IS_SHARED_SPACE, HF_REPO_ID
+from modules.storage import upload_files_to_repo
 
 MODEL = None
 MODELS = None
-IS_SHARED_SPACE = "Surn/UnlimitedMusicGen" in os.environ.get('SPACE_ID', '')
+#IS_SHARED_SPACE = "Surn/UnlimitedMusicGen" in os.environ.get('SPACE_ID', '')
 INTERRUPTED = False
 UNLOAD_MODEL = False
 MOVE_TO_CPU = False
@@ -640,12 +642,12 @@ def predict_simple(model: str, text: str, duration: int = 10, dimension: int = 2
             settings_font_size (int, optional): Font size for settings text.
             settings_animate_waveform (bool, optional): Animate waveform in video.
             video_orientation (str, optional): Video orientation
-            return_history_json (bool, optional): Whether to return history JSON instead of typical output. Default to False.
+            return_history_json (bool, optional): Return history JSON instead of typical output. Default to False.
 
         Returns:
              tp.List[tp.Tuple[str, str, str]]: [waveform_video_path, wave_file_path, seed_used]
     """
-    profile_username_to_send = "Satoshi Nakamoto" 
+    profile_username_to_send = "default_user" 
 
     if not profile:
         profile = modules.user_history.get_profile
@@ -663,12 +665,35 @@ def predict_simple(model: str, text: str, duration: int = 10, dimension: int = 2
             profile_username_to_send = actual_profile_data
 
     UMG_result = predict(model, text, melody_filepath=None, duration=duration, dimension=dimension, topk=topk, topp=topp, temperature=temperature, cfg_coef=cfg_coef, background=background, title=title, settings_font=settings_font, settings_font_color=settings_font_color, seed=seed, overlap=overlap, prompt_index=prompt_index, include_title=include_title, include_settings=include_settings, harmony_only=False, profile=profile, segment_length=segment_length, settings_font_size=settings_font_size, settings_animate_waveform=settings_animate_waveform, video_orientation=video_orientation, excerpt_duration=3.5,  return_history_json=return_history_json)
+
+    # upload to storage and return urls
+    folder_name = f"user_uploads/{profile_username_to_send}"
     if return_history_json:
-        #content = [UMG_result["image_path"], UMG_result["video_path"], UMG_result["audio_path"], UMG_result["metadata"]["Seed"]]
-        #content = {"content":[UMG_result], "isError":False, "message":"Music generated successfully."}
-        #content = {fix_path(UMG_result["image_path"]), fix_path(UMG_result["video_path"]), fix_path(UMG_result["audio_path"]), UMG_result["metadata"]["Seed"]}        
+        # use modules.storage.upload_files_to_repo to get urls for image_path, video_path, audio_path
+        upload_result = upload_files_to_repo(
+            files=[UMG_result["video_path"],UMG_result["audio_path"], UMG_result["image_path"]],
+            repo_id=HF_REPO_ID, # constants.py value of dataset repo
+            folder_name=f"{folder_name}/{UMG_result['metadata']['title']}/{UMG_result['metadata']['Seed']}/{time.strftime('%Y%m%d%H%M%S')}",
+            create_permalink=False,
+            repo_type="dataset"
+        )
+        if upload_result:
+            UMG_result["video_path"] = upload_result[0][1]  # Assuming [(response, link) for link in individual_links]
+            UMG_result["audio_path"] = upload_result[1][1]
+            UMG_result["image_path"] = upload_result[2][1]
         content = UMG_result["video_path"], UMG_result["audio_path"], UMG_result["metadata"]["Seed"]
         UMG_result = content
+    else:
+        # use modules.storage.upload_files_to_repo to get urls for video_path, audio_path
+        upload_result = upload_files_to_repo(
+            files=[UMG_result[0],UMG_result[1]],
+            repo_id=HF_REPO_ID, # constants.py value of dataset repo
+            folder_name=f"{folder_name}/{UMG_result[2]}/{time.strftime('%Y%m%d%H%M%S')}",
+            create_permalink=False,
+            repo_type="dataset"
+        )
+        if upload_result:
+            UMG_result = upload_result[0][1], upload_result[1][1], UMG_result[2]
     
     return UMG_result
 

modules/constants.py

@@ -0,0 +1,40 @@
+# modules/constants.py
+# constants.py contains all the constants used in the project
+import os
+from pathlib import Path
+from dotenv import load_dotenv
+
+# Load environment variables from .env file
+dotenv_path = Path(__file__).parent.parent / '.env'
+load_dotenv(dotenv_path)
+
+IS_SHARED_SPACE = "Surn/UnlimitedMusicGen" in os.environ.get('SPACE_ID', '')
+
+HF_API_TOKEN = os.getenv("HF_API_TOKEN")
+if not HF_API_TOKEN:
+    raise ValueError("HF_TOKEN is not set. Please check your .env file.")
+try:
+    if os.environ['TMPDIR']:
+        TMPDIR = os.environ['TMPDIR']
+    else:
+        TMPDIR = os.path.join(os.path.dirname(os.path.abspath(__file__)), 'tmp')
+except:
+    TMPDIR = os.path.join(os.path.dirname(os.path.abspath(__file__)), 'tmp')
+
+os.makedirs(TMPDIR, exist_ok=True)
+
+model_extensions = {".glb", ".gltf", ".obj", ".ply"}
+model_extensions_list = list(model_extensions)
+image_extensions = {".png", ".jpg", ".jpeg", ".webp"}
+image_extensions_list = list(image_extensions)
+audio_extensions = {".mp3", ".wav", ".ogg", ".flac", ".aac"}
+audio_extensions_list = list(audio_extensions)
+video_extensions = {".mp4"}
+video_extensions_list = list(video_extensions)
+upload_file_types = model_extensions_list + image_extensions_list + audio_extensions_list + video_extensions_list
+
+# Constants for URL shortener
+HF_REPO_ID = os.getenv("HF_REPO_ID")
+if not HF_REPO_ID:
+    HF_REPO_ID = "Surn/Storage"  # Replace with your Hugging Face repository ID
+SHORTENER_JSON_FILE = "shortener.json"  # The name of your JSON file in the repo

modules/storage.md

@@ -0,0 +1,156 @@
+# Storage Module (`modules/storage.py`) Usage Guide
+
+The `storage.py` module provides helper functions for:
+- Generating permalinks for 3D viewer projects.
+- Uploading files in batches to a Hugging Face repository.
+- Managing URL shortening by storing (short URL, full URL) pairs in a JSON file on the repository.
+- Retrieving full URLs from short URL IDs and vice versa.
+- Handle specific file types for 3D models, images, video and audio.
+
+## Key Functions
+
+### 1. `generate_permalink(valid_files, base_url_external, permalink_viewer_url="surn-3d-viewer.hf.space")`
+- **Purpose:**  
+  Given a list of file paths, it looks for exactly one model file (with an extension defined in `model_extensions`) and exactly two image files (extensions defined in `image_extensions`). If the criteria are met, it returns a permalink URL built from the base URL and query parameters.
+- **Usage Example:**from modules.storage import generate_permalink
+
+valid_files = [
+    "models/3d_model.glb",
+    "images/model_texture.png",
+    "images/model_depth.png"
+]
+base_url_external = "https://huggingface.co/datasets/Surn/Storage/resolve/main/saved_models/my_model"
+permalink = generate_permalink(valid_files, base_url_external)
+if permalink:
+    print("Permalink:", permalink)
+### 2. `generate_permalink_from_urls(model_url, hm_url, img_url, permalink_viewer_url="surn-3d-viewer.hf.space")`
+- **Purpose:**  
+  Constructs a permalink URL by combining individual URLs for a 3D model (`model_url`), height map (`hm_url`), and image (`img_url`) into a single URL with corresponding query parameters.
+- **Usage Example:**from modules.storage import generate_permalink_from_urls
+
+model_url = "https://example.com/model.glb"
+hm_url = "https://example.com/heightmap.png"
+img_url = "https://example.com/source.png"
+
+permalink = generate_permalink_from_urls(model_url, hm_url, img_url)
+print("Generated Permalink:", permalink)
+### 3. `upload_files_to_repo(files, repo_id, folder_name, create_permalink=False, repo_type="dataset", permalink_viewer_url="surn-3d-viewer.hf.space")`
+- **Purpose:**  
+  Uploads a batch of files (each file represented as a path string) to a specified Hugging Face repository (e.g. `"Surn/Storage"`) under a given folder.
+  The function's return type is `Union[Dict[str, Any], List[Tuple[Any, str]]]`.
+  - When `create_permalink` is `True` and exactly three valid files (one model and two images) are provided, the function returns a dictionary:```
+{
+    "response": <upload_folder_response>,
+    "permalink": "<full_permalink_url>",
+    "short_permalink": "<shortened_permalink_url_with_sid>"
+}
+```  - Otherwise (or if `create_permalink` is `False` or conditions for permalink creation are not met), it returns a list of tuples, where each tuple is `(upload_folder_response, individual_file_link)`.
+  - If no valid files are provided, it returns an empty list `[]` (this case should ideally also return the dictionary with empty/None values for consistency, but currently returns `[]` as per the code).
+- **Usage Example:**
+
+  **a. Uploading with permalink creation:**from modules.storage import upload_files_to_repo
+
+files_for_permalink = [
+    "local/path/to/model.glb",
+    "local/path/to/heightmap.png",
+    "local/path/to/image.png"
+]
+repo_id = "Surn/Storage" # Make sure this is defined, e.g., from constants or environment variables
+folder_name = "my_new_model_with_permalink"
+
+upload_result = upload_files_to_repo(
+    files_for_permalink, 
+    repo_id, 
+    folder_name, 
+    create_permalink=True
+)
+
+if isinstance(upload_result, dict):
+    print("Upload Response:", upload_result.get("response"))
+    print("Full Permalink:", upload_result.get("permalink"))
+    print("Short Permalink:", upload_result.get("short_permalink"))
+elif upload_result: # Check if list is not empty
+    print("Upload Response for individual files:")
+    for res, link in upload_result:
+        print(f"  Response: {res}, Link: {link}")
+else:
+    print("No files uploaded or error occurred.")
+  **b. Uploading without permalink creation (or if conditions for permalink are not met):**from modules.storage import upload_files_to_repo
+
+files_individual = [
+    "local/path/to/another_model.obj",
+    "local/path/to/texture.jpg"
+]
+repo_id = "Surn/Storage"
+folder_name = "my_other_uploads"
+
+upload_results_list = upload_files_to_repo(
+    files_individual, 
+    repo_id, 
+    folder_name, 
+    create_permalink=False # Or if create_permalink=True but not 1 model & 2 images
+)
+
+if upload_results_list: # Will be a list of tuples
+    print("Upload results for individual files:")
+    for res, link in upload_results_list:
+        print(f"  Upload Response: {res}, File Link: {link}")
+else:
+    print("No files uploaded or error occurred.")
+### 4. URL Shortening Functions: `gen_full_url(...)` and Helpers
+The module also enables URL shortening by managing a JSON file (e.g. `shortener.json`) in a Hugging Face repository. It supports CRUD-like operations:
+- **Read:** Look up the full URL using a provided short URL ID.
+- **Create:** Generate a new short URL ID for a full URL if no existing mapping exists.
+- **Update/Conflict Handling:**  
+If both short URL ID and full URL are provided, it checks consistency and either confirms or reports a conflict.
+
+#### `gen_full_url(short_url=None, full_url=None, repo_id=None, repo_type="dataset", permalink_viewer_url="surn-3d-viewer.hf.space", json_file="shortener.json")`
+- **Purpose:**  
+  Based on which parameter is provided, it retrieves or creates a mapping between a short URL ID and a full URL.  
+  - If only `short_url` (the ID) is given, it returns the corresponding `full_url`.  
+  - If only `full_url` is given, it looks up an existing `short_url` ID or generates and stores a new one.  
+  - If both are given, it validates and returns the mapping or an error status.
+- **Returns:** A tuple `(status_message, result_url)`, where `status_message` indicates the outcome (e.g., `"success_retrieved_full"`, `"created_short"`) and `result_url` is the relevant URL (full or short ID).
+- **Usage Examples:**
+
+  **a. Convert a full URL into a short URL ID:**from modules.storage import gen_full_url
+from modules.constants import HF_REPO_ID, SHORTENER_JSON_FILE # Assuming these are defined
+
+full_permalink = "https://surn-3d-viewer.hf.space/?3d=https%3A%2F%2Fexample.com%2Fmodel.glb&hm=https%3A%2F%2Fexample.com%2Fheightmap.png&image=https%3A%2F%2Fexample.com%2Fsource.png"
+
+status, short_id = gen_full_url(
+    full_url=full_permalink, 
+    repo_id=HF_REPO_ID, 
+    json_file=SHORTENER_JSON_FILE
+)
+print("Status:", status)
+if status == "created_short" or status == "success_retrieved_short":
+    print("Shortened URL ID:", short_id)
+    # Construct the full short URL for sharing:
+    # permalink_viewer_url = "surn-3d-viewer.hf.space" # Or from constants
+    # shareable_short_url = f"https://{permalink_viewer_url}/?sid={short_id}"
+    # print("Shareable Short URL:", shareable_short_url)
+  **b. Retrieve the full URL from a short URL ID:**from modules.storage import gen_full_url
+from modules.constants import HF_REPO_ID, SHORTENER_JSON_FILE # Assuming these are defined
+
+short_id_to_lookup = "aBcDeFg1"  # Example short URL ID
+
+status, retrieved_full_url = gen_full_url(
+    short_url=short_id_to_lookup, 
+    repo_id=HF_REPO_ID, 
+    json_file=SHORTENER_JSON_FILE
+)
+print("Status:", status)
+if status == "success_retrieved_full":
+    print("Retrieved Full URL:", retrieved_full_url)
+## Notes
+- **Authentication:** All functions that interact with Hugging Face Hub use the HF API token defined as `HF_API_TOKEN` in `modules/constants.py`. Ensure this environment variable is correctly set.
+- **Constants:** Functions like `gen_full_url` and `upload_files_to_repo` (when creating short links) rely on `HF_REPO_ID` and `SHORTENER_JSON_FILE` from `modules/constants.py` for the URL shortening feature.
+- **File Types:** Only files with extensions included in `upload_file_types` (a combination of `model_extensions` and `image_extensions` from `modules/constants.py`) are processed by `upload_files_to_repo`.
+- **Repository Configuration:** When using URL shortening and file uploads, ensure that the specified Hugging Face repository (e.g., defined by `HF_REPO_ID`) exists and that you have write permissions.
+- **Temporary Directory:** `upload_files_to_repo` temporarily copies files to a local directory (configured by `TMPDIR` in `modules/constants.py`) before uploading.
+- **Error Handling:** Functions include basic error handling (e.g., catching `RepositoryNotFoundError`, `EntryNotFoundError`, JSON decoding errors, or upload issues) and print messages to the console for debugging. Review function return values to handle these cases appropriately in your application.
+
+---
+
+This guide provides the essential usage examples for interacting with the storage and URL-shortening functionality. You can integrate these examples into your application or use them as a reference when extending functionality.

modules/storage.py

@@ -0,0 +1,327 @@
+# modules/storage.py
+__version__ = "0.1.1" # Added version
+import os
+import urllib.parse
+import tempfile
+import shutil
+import json
+import base64
+from huggingface_hub import login, upload_folder, hf_hub_download, HfApi
+from huggingface_hub.utils import RepositoryNotFoundError, EntryNotFoundError
+from modules.constants import HF_API_TOKEN, upload_file_types, model_extensions, image_extensions, audio_extensions, video_extensions, HF_REPO_ID, SHORTENER_JSON_FILE
+from typing import Any, Dict, List, Tuple, Union
+
+# see storage.md for detailed information about the storage module and its functions.
+
+def generate_permalink(valid_files, base_url_external, permalink_viewer_url="surn-3d-viewer.hf.space"):
+    """
+    Given a list of valid files, checks if they contain exactly 1 model file and 2 image files.
+    Constructs and returns a permalink URL with query parameters if the criteria is met.
+    Otherwise, returns None.
+    """
+    model_link = None
+    images_links = []
+    audio_links = []
+    video_links = []
+    for f in valid_files:
+        filename = os.path.basename(f)
+        ext = os.path.splitext(filename)[1].lower()
+        if ext in model_extensions:
+            if model_link is None:
+                model_link = f"{base_url_external}/{filename}"
+        elif ext in image_extensions:
+            images_links.append(f"{base_url_external}/{filename}")
+        elif ext in audio_extensions:
+            audio_links.append(f"{base_url_external}/{filename}")
+        elif ext in video_extensions:
+            video_links.append(f"{base_url_external}/{filename}")
+    if model_link and len(images_links) == 2:
+        # Construct a permalink to the viewer project with query parameters.
+        permalink_viewer_url = f"https://{permalink_viewer_url}/"
+        params = {"3d": model_link, "hm": images_links[0], "image": images_links[1]}
+        query_str = urllib.parse.urlencode(params)
+        return f"{permalink_viewer_url}?{query_str}"
+    return None
+
+def generate_permalink_from_urls(model_url, hm_url, img_url, permalink_viewer_url="surn-3d-viewer.hf.space"):
+    """
+    Constructs and returns a permalink URL with query string parameters for the viewer.
+    Each parameter is passed separately so that the image positions remain consistent.
+    
+    Parameters:
+        model_url (str): Processed URL for the 3D model.
+        hm_url (str): Processed URL for the height map image.
+        img_url (str): Processed URL for the main image.
+        permalink_viewer_url (str): The base viewer URL.
+    
+    Returns:
+        str: The generated permalink URL.
+    """
+    import urllib.parse
+    params = {"3d": model_url, "hm": hm_url, "image": img_url}
+    query_str = urllib.parse.urlencode(params)
+    return f"https://{permalink_viewer_url}/?{query_str}"
+
+def upload_files_to_repo(
+    files: List[Any],
+    repo_id: str,
+    folder_name: str,
+    create_permalink: bool = False,
+    repo_type: str = "dataset",
+    permalink_viewer_url: str = "surn-3d-viewer.hf.space"
+) -> Union[Dict[str, Any], List[Tuple[Any, str]]]:
+    """
+    Uploads multiple files to a Hugging Face repository using a batch upload approach via upload_folder.
+
+    Parameters:
+        files (list): A list of file paths (str) to upload.
+        repo_id (str): The repository ID on Hugging Face for storage, e.g. "Surn/Storage".
+        folder_name (str): The subfolder within the repository where files will be saved.
+        create_permalink (bool): If True and if exactly three files are uploaded (1 model and 2 images),
+                                 returns a single permalink to the project with query parameters.
+                                 Otherwise, returns individual permalinks for each file.
+        repo_type (str): Repository type ("space", "dataset", etc.). Default is "dataset".
+        permalink_viewer_url (str): The base viewer URL.
+
+    Returns:
+        Union[Dict[str, Any], List[Tuple[Any, str]]]:
+            If create_permalink is True and files match the criteria:
+                dict: {
+                    "response": <upload response>,
+                    "permalink": <full_permalink URL>,
+                    "short_permalink": <shortened permalink URL>
+                }
+            Otherwise:
+                list: A list of tuples (response, permalink) for each file.
+    """
+    # Log in using the HF API token.
+    login(token=HF_API_TOKEN) # Corrected from HF_TOKEN to HF_API_TOKEN
+    
+    valid_files = []
+    permalink_short = None
+    
+    # Ensure folder_name does not have a trailing slash.
+    folder_name = folder_name.rstrip("/")
+    
+    # Filter for valid files based on allowed extensions.
+    for f in files:
+        file_name = f if isinstance(f, str) else f.name if hasattr(f, "name") else None
+        if file_name is None:
+            continue
+        ext = os.path.splitext(file_name)[1].lower()
+        if ext in upload_file_types:
+            valid_files.append(f)
+    
+    if not valid_files:
+        # Return a dictionary with None values for permalinks if create_permalink was True
+        if create_permalink:
+            return {
+                "response": "No valid files to upload.",
+                "permalink": None,
+                "short_permalink": None
+            }
+        return [] 
+    
+    # Create a temporary directory; copy valid files directly into it.
+    with tempfile.TemporaryDirectory(dir=os.getenv("TMPDIR", "/tmp")) as temp_dir:
+        for file_path in valid_files:
+            filename = os.path.basename(file_path)
+            dest_path = os.path.join(temp_dir, filename)
+            shutil.copy(file_path, dest_path)
+        
+        # Batch upload all files in the temporary folder.
+        # Files will be uploaded under the folder (path_in_repo) given by folder_name.
+        response = upload_folder(
+            folder_path=temp_dir,
+            repo_id=repo_id,
+            repo_type=repo_type,
+            path_in_repo=folder_name,
+            commit_message="Batch upload files"
+        )
+    
+    # Construct external URLs for each uploaded file.
+    base_url_external = f"https://huggingface.co/datasets/{repo_id}/resolve/main/{folder_name}"
+    individual_links = []
+    for file_path in valid_files:
+        filename = os.path.basename(file_path)
+        link = f"{base_url_external}/{filename}"
+        individual_links.append(link)
+    
+    # If permalink creation is requested and exactly 3 valid files are provided,
+    # try to generate a permalink using generate_permalink().
+    if create_permalink: # No need to check len(valid_files) == 3 here, generate_permalink will handle it
+        permalink = generate_permalink(valid_files, base_url_external, permalink_viewer_url)
+        if permalink:
+            status, short_id = gen_full_url(
+                full_url=permalink,
+                repo_id=HF_REPO_ID, # This comes from constants
+                json_file=SHORTENER_JSON_FILE # This comes from constants
+            )
+            if status in ["created_short", "success_retrieved_short", "exists_match"]:
+                permalink_short = f"https://{permalink_viewer_url}/?sid={short_id}"
+            else: # Shortening failed or conflict not resolved to a usable short_id
+                permalink_short = None 
+                print(f"URL shortening status: {status} for {permalink}")
+
+            return {
+                "response": response,
+                "permalink": permalink,
+                "short_permalink": permalink_short
+            }
+        else: # generate_permalink returned None (criteria not met)
+            return {
+                "response": response, # Still return upload response
+                "permalink": None,
+                "short_permalink": None
+            }
+
+    # Otherwise, return individual tuples for each file.
+    return [(response, link) for link in individual_links]
+
+def _generate_short_id(length=8):
+    """Generates a random base64 URL-safe string."""
+    return base64.urlsafe_b64encode(os.urandom(length * 2))[:length].decode('utf-8')
+
+def _get_json_from_repo(repo_id, json_file_name, repo_type="dataset"):
+    """Downloads and loads the JSON file from the repo. Returns empty list if not found or error."""
+    try:
+        login(token=HF_API_TOKEN)
+        json_path = hf_hub_download(
+            repo_id=repo_id,
+            filename=json_file_name,
+            repo_type=repo_type,
+            token=HF_API_TOKEN  # Added token for consistency, though login might suffice
+        )
+        with open(json_path, 'r') as f:
+            data = json.load(f)
+        os.remove(json_path) # Clean up downloaded file
+        return data
+    except RepositoryNotFoundError:
+        print(f"Repository {repo_id} not found.")
+        return []
+    except EntryNotFoundError:
+        print(f"JSON file {json_file_name} not found in {repo_id}. Initializing with empty list.")
+        return []
+    except json.JSONDecodeError:
+        print(f"Error decoding JSON from {json_file_name}. Returning empty list.")
+        return []
+    except Exception as e:
+        print(f"An unexpected error occurred while fetching {json_file_name}: {e}")
+        return []
+
+def _upload_json_to_repo(data, repo_id, json_file_name, repo_type="dataset"):
+    """Uploads the JSON data to the specified file in the repo."""
+    try:
+        login(token=HF_API_TOKEN)
+        api = HfApi()
+        # Use a temporary directory specified by TMPDIR or default to system temp
+        temp_dir_for_json = os.getenv("TMPDIR", tempfile.gettempdir())
+        os.makedirs(temp_dir_for_json, exist_ok=True)
+
+        with tempfile.NamedTemporaryFile(mode="w+", delete=False, suffix=".json", dir=temp_dir_for_json) as tmp_file:
+            json.dump(data, tmp_file, indent=2)
+            tmp_file_path = tmp_file.name
+        
+        api.upload_file(
+            path_or_fileobj=tmp_file_path,
+            path_in_repo=json_file_name,
+            repo_id=repo_id,
+            repo_type=repo_type,
+            commit_message=f"Update {json_file_name}"
+        )
+        os.remove(tmp_file_path) # Clean up temporary file
+        return True
+    except Exception as e:
+        print(f"Failed to upload {json_file_name} to {repo_id}: {e}")
+        if 'tmp_file_path' in locals() and os.path.exists(tmp_file_path):
+            os.remove(tmp_file_path) # Ensure cleanup on error too
+        return False
+
+def _find_url_in_json(data, short_url=None, full_url=None):
+    """
+    Searches the JSON data.
+    If short_url is provided, returns the corresponding full_url or None.
+    If full_url is provided, returns the corresponding short_url or None.
+    """
+    if not data: # Handles cases where data might be None or empty
+        return None
+    if short_url:
+        for item in data:
+            if item.get("short_url") == short_url:
+                return item.get("full_url")
+    if full_url:
+        for item in data:
+            if item.get("full_url") == full_url:
+                return item.get("short_url")
+    return None
+
+def _add_url_to_json(data, short_url, full_url):
+    """Adds a new short_url/full_url pair to the data. Returns updated data."""
+    if data is None: 
+        data = []
+    data.append({"short_url": short_url, "full_url": full_url})
+    return data
+
+def gen_full_url(short_url=None, full_url=None, repo_id=None, repo_type="dataset", permalink_viewer_url="surn-3d-viewer.hf.space", json_file="shortener.json"):
+    """
+    Manages short URLs and their corresponding full URLs in a JSON file stored in a Hugging Face repository.
+
+    - If short_url is provided, attempts to retrieve and return the full_url.
+    - If full_url is provided, attempts to retrieve an existing short_url or creates a new one, stores it, and returns the short_url.
+    - If both are provided, checks for consistency or creates a new entry.
+    - If neither is provided, or repo_id is missing, returns an error status.
+
+    Returns:
+        tuple: (status_message, result_url)
+               status_message can be "success", "created", "exists", "error", "not_found".
+               result_url is the relevant URL (short or full) or None if an error occurs or not found.
+    """
+    if not repo_id:
+        return "error_repo_id_missing", None
+    if not short_url and not full_url:
+        return "error_no_input", None
+
+    login(token=HF_API_TOKEN) # Ensure login at the beginning
+    url_data = _get_json_from_repo(repo_id, json_file, repo_type)
+
+    # Case 1: Only short_url provided (lookup full_url)
+    if short_url and not full_url:
+        found_full_url = _find_url_in_json(url_data, short_url=short_url)
+        return ("success_retrieved_full", found_full_url) if found_full_url else ("not_found_short", None)
+
+    # Case 2: Only full_url provided (lookup or create short_url)
+    if full_url and not short_url:
+        existing_short_url = _find_url_in_json(url_data, full_url=full_url)
+        if existing_short_url:
+            return "success_retrieved_short", existing_short_url
+        else:
+            # Create new short_url
+            new_short_id = _generate_short_id()
+            url_data = _add_url_to_json(url_data, new_short_id, full_url)
+            if _upload_json_to_repo(url_data, repo_id, json_file, repo_type):
+                return "created_short", new_short_id 
+            else:
+                return "error_upload", None
+
+    # Case 3: Both short_url and full_url provided
+    if short_url and full_url:
+        found_full_for_short = _find_url_in_json(url_data, short_url=short_url)
+        found_short_for_full = _find_url_in_json(url_data, full_url=full_url)
+
+        if found_full_for_short == full_url: 
+            return "exists_match", short_url 
+        if found_full_for_short is not None and found_full_for_short != full_url: 
+            return "error_conflict_short_exists_different_full", short_url
+        if found_short_for_full is not None and found_short_for_full != short_url:
+            return "error_conflict_full_exists_different_short", found_short_for_full
+        
+        # If short_url is provided and not found, or full_url is provided and not found,
+        # or neither is found, then create a new entry with the provided short_url and full_url.
+        # This effectively allows specifying a custom short_url if it's not already taken.
+        url_data = _add_url_to_json(url_data, short_url, full_url)
+        if _upload_json_to_repo(url_data, repo_id, json_file, repo_type):
+            return "created_specific_pair", short_url
+        else:
+            return "error_upload", None
+                
+    return "error_unhandled_case", None # Should not be reached