Spaces:
Running
Running
| title: Link Lazarus Method | |
| emoji: 🔗 | |
| colorFrom: blue | |
| colorTo: red | |
| sdk: streamlit | |
| sdk_version: "1.32.2" | |
| app_file: wikipedia_dead_links_streamlit.py | |
| pinned: false | |
| # The Link Lazarus Method: Wikipedia Dead Link Finder by metehan.ai - Streamlit Version | |
| A Streamlit web application for finding and logging dead (broken) external links in Wikipedia articles, identifying potentially available domains for registration, and saving them to a dedicated database. | |
| ## Features | |
| - **Multiple Search Methods**: | |
| - Search by text to find Wikipedia articles | |
| - Search by category to find related articles | |
| - **Dead Link Detection**: Checks external links for HTTP errors or connection issues | |
| - **Domain Availability**: Identifies which domains from dead links might be available for registration | |
| - **Restricted TLD Filtering**: Automatically identifies and excludes restricted domains (.edu, .gov, etc.) | |
| - **Available Domains Database**: Maintains a separate database of potentially available domains | |
| - **Real-time Logging**: Saves dead links and available domains to JSON files as they're found | |
| - **Result Visualization**: Displays results in an interactive table with filtering options | |
| - **Export to CSV**: Download results as a CSV file | |
| - **Web Archive Filter**: Automatically ignores links from web.archive.org | |
| - **Configurable**: Adjust settings via the sidebar | |
| ## Requirements | |
| - Python 3.6+ | |
| - Required packages listed in `requirements_streamlit.txt` | |
| ## Installation | |
| ``` | |
| pip install -r requirements_streamlit.txt | |
| ``` | |
| ## Usage | |
| Run the Streamlit app: | |
| ``` | |
| streamlit run wikipedia_dead_links_streamlit.py | |
| ``` | |
| The application will open in your default web browser with three main tabs: | |
| ### 1. Search by Text | |
| - Enter search terms to find Wikipedia articles containing that text | |
| - View search results with snippets | |
| - Process all found pages to check for dead links and available domains | |
| ### 2. Search by Category | |
| - Enter a category name to find Wikipedia categories | |
| - Select a category to crawl its pages | |
| - Find dead links and available domains within those pages | |
| ### 3. Available Domains | |
| - View all potentially available domains found during searches | |
| - Filter domains by status (potentially available, expired, etc.) | |
| - See details about each domain including where it was found | |
| - Download the list as a CSV file | |
| ## How Domain Availability Works | |
| The app uses these methods to determine if a domain might be available: | |
| 1. **WHOIS Lookup**: Checks if the domain has registration information | |
| 2. **Expiration Check**: Identifies domains with expired registration dates | |
| 3. **DNS Lookup**: Verifies if the domain has active DNS records | |
| 4. **TLD Restriction Check**: Identifies restricted TLDs that cannot be freely registered | |
| Domains are flagged as potentially available if: | |
| - No WHOIS registration data is found | |
| - The domain's expiration date has passed | |
| - No DNS records exist for the domain | |
| - The domain does NOT have a restricted TLD (.edu, .gov, .mil, etc.) | |
| ### Restricted TLDs (Optional) | |
| The following TLDs are recognized as restricted and will never be reported as available, if you choose to filter them: | |
| - .edu - Educational institutions | |
| - .gov - Government entities | |
| - .mil - Military organizations | |
| - .int - International organizations | |
| - Country-specific restrictions like .ac.uk, .gov.uk, etc. | |
| **Note**: For definitive availability, you should verify with a domain registrar. The tool provides a starting point for identifying potential opportunities. | |
| ## Configuration Options | |
| - **Log file path**: Where to save the dead links JSON results | |
| - **Available domains file**: Where to save the available domains database | |
| - **Max concurrent requests**: Number of links to check simultaneously | |
| - **Max pages to process**: Limit the number of articles to process | |
| ## Output Files | |
| The app generates two main JSON files: | |
| 1. **wikipedia_dead_links.json**: Contains details about all dead links found | |
| 2. **available_domains.json**: Contains only the potentially available domains and where they were found | |
| You can also download results as CSV files directly from the app. Make sure follow on X @metehan777 and LinkedIn www.linkedin.com/in/metehanyesilyurt for the upcoming updates and more tips&tools. |