Scrapers: Difference between revisions

From Official Kodi Wiki
Jump to navigation Jump to search
(Convert to Portal)
No edit summary
 
(One intermediate revision by the same user not shown)
Line 3: Line 3:


= Using Scrapers =
= Using Scrapers =
Scrapers are used to contact online information providers to download metadata for video and music items to be added to the library.
Scrapers are used to contact online information providers to download metadata for video and music items that are to be added to the library.


A number of scrapers are installed by default when installing {{kodi}}. Other scrapers are available for installation from the official repository. The below table lists all official scrapers and their compatability.
A number of scrapers are installed by default when installing {{kodi}}. Other scrapers are available for installation from the official repository. The tables at the bottom of this page list all official scrapers and their compatability.


{| style="width: 100%;"
|-
| style="width: 5%; vertical-align:top;" | [[File:language_globe_icon.png|50px|left]]
| style="width: 44%; vertical-align:top;" |'''{{resize|1.3em| [[HOW-TO:Create_Video_Library|Creating your Video Library]]}}'''
Setting up your video library for the first time? Follow this How To guide
| style="width: 2%;" |
| style="width: 5%; vertical-align:top;" | [[File:language_globe_icon.png|50px|left]]
| style="width: 44%; vertical-align:top;" | '''{{resize|1.3em| [[HOW-TO:Create_Music_Library|Creating your Music Library]]}}'''
Setting up your music library for the first time? Follow this How To guide
|-
| style="width: 5%; vertical-align:top;" | [[File:language_globe_icon.png|50px|left]]
| style="width: 44%; vertical-align:top;" | '''{{resize|1.3em| [[Incorrect_and_missing_videos|Video scraping problems]]}}'''
Something not working as expected? Check this page for common solutions.
| style="width: 2%;" |
| style="width: 5%; vertical-align:top;" |
| style="width: 44%; vertical-align:top;" |
|-
| style="width: 5%; vertical-align:top;" | [[File:language_globe_icon.png|50px|left]]
| style="width: 44%; vertical-align:top;" | '''{{resize|1.3em| [[Changing Scrapers]]}}'''
| style="width: 2%;" |
| style="width: 5%; vertical-align:top;" | [[File:language_globe_icon.png|50px|left]]
| style="width: 44%; vertical-align:top;" | '''{{resize|1.3em| [[:Category:Scraper|List of all Scraper pages]]}}'''
|}
= Scraper Development =
Historically, {{kodi}} scrapers were XML based. While XML scrapers have been used successfully for well over a decade, their use has been deprecated, since the release of {{kodi}} v19 Matrix, in favour of Python based scrapers.
XML scrapers have become difficult to develop and maintain as they are no longer suited to the advanced REST API's used by the information providers, which results in frequent maintenance and updates to cope with API changes.
The following pages are available to help with the development process of scrapers. Although XML scrapers are deprecated, and no future XML scrapers will be accepted into the official repository, the two pages dedicated to XML scrapers contain useful information.
{| style="width: 100%;"
|-
| style="width: 5%; vertical-align:top;" | [[File:Crystal Project Network Connection Manager.png|50px|left]]
| style="width: 44%; vertical-align:top;" |'''{{resize|1.3em| [[Python_movie_scraper_development|Developing a Movie scraper in Python]]}}'''
| style="width: 2%;" |
| style="width: 5%; vertical-align:top;" | [[File:Crystal Project Network Connection Manager.png|50px|left]]
| style="width: 44%; vertical-align:top;" | '''{{resize|1.3em| [[Scraper development XML|XML Scraper Development]]}}'''
XML scrapers are deprecated. This page may still contain useful development information.
|-
| style="width: 5%; vertical-align:top;" | [[File:Crystal Project Network Connection Manager.png|50px|left]]
| style="width: 44%; vertical-align:top;" |'''{{resize|1.3em| [[Python_tv_scraper_development|Developing a TV Show scraper in Python]]}}'''
| style="width: 2%;" |
| style="width: 5%; vertical-align:top;" | [[File:Crystal Project Network Connection Manager.png|50px|left]]
| style="width: 44%; vertical-align:top;" | '''{{resize|1.3em| [[HOW-TO:Write_media_scrapers|Writing an XML scraper]]}}'''
XML scrapers are deprecated. This page may still contain useful development information.
|}
= Official Scrapers =
{|
{|
! style="background-color:salmon;" | Scrapers compatible with v18 Leia
! style="background-color:salmon;" | Scrapers compatible with v19 Matrix
! style="background-color:#a0e75a;" | Scrapers compatible with v19 Matrix
! style="background-color:#a0e75a;" | Scrapers compatible with v20 Nexus
! style="background-color:skyBlue;" | Scrapers compatible with v20 Nexus
! style="background-color:skyBlue;" | Scrapers compatible with v21 Omega
|-
|-
|
|
{| class="mw-collapsible mw-collapsed wikitable" style="margin: 0.0em 0 0.0em 0em; {{border-radius|5px}}; {{box-shadow|0px|1px|3px|#878787}};"
{| class="mw-collapsible mw-collapsed wikitable" style="margin: 0.0em 0 0.0em 0em; {{border-radius|5px}}; {{box-shadow|0px|1px|3px|#878787}};"
! scope="row" style="width:500px; background-color:#dfedf0; text-align:left;"| List of {{red|Album}} scrapers compatible with v18 Leia
! scope="row" style="width:500px; background-color:#dfedf0; text-align:left;"| List of {{red|Album}} scrapers
|-
|-
|
|
<dpl>
<dpl>
  category = Leia_add-on_repository
  category = Matrix_add-on_repository
  category = Album_scraper_add-ons
  category = Album_scraper_add-ons
notcategory = Broken add-ons
</dpl>
</dpl>
|}
|}
|
|
{| class="mw-collapsible mw-collapsed wikitable" style="margin: 0.0em 0 0.0em 0em; {{border-radius|5px}}; {{box-shadow|0px|1px|3px|#878787}};"
{| class="mw-collapsible mw-collapsed wikitable" style="margin: 0.0em 0 0.0em 0em; {{border-radius|5px}}; {{box-shadow|0px|1px|3px|#878787}};"
! scope="row" style="width:500px; background-color:#dfedf0; text-align:left;"| List of {{red|Album}} scrapers compatible with v19 Matrix
! scope="row" style="width:500px; background-color:#dfedf0; text-align:left;"| List of {{red|Album}} scrapers
|-
|-
|
|
<dpl>
<dpl>
  category = Matrix_add-on_repository
  category = Nexus_add-on_repository
  category = Album_scraper_add-ons
  category = Album_scraper_add-ons
notcategory = Broken add-ons
</dpl>
</dpl>
|}
|}
|
|
{| class="mw-collapsible mw-collapsed wikitable" style="margin: 0.0em 0 0.0em 0em; {{border-radius|5px}}; {{box-shadow|0px|1px|3px|#878787}};"
{| class="mw-collapsible mw-collapsed wikitable" style="margin: 0.0em 0 0.0em 0em; {{border-radius|5px}}; {{box-shadow|0px|1px|3px|#878787}};"
! scope="row" style="width:500px; background-color:#dfedf0; text-align:left;"| List of {{red|Album}} scrapers compatible with v20 Nexus
! scope="row" style="width:500px; background-color:#dfedf0; text-align:left;"| List of {{red|Album}} scrapers
|-
|-
|
|
<dpl>
<dpl>
  category = Nexus_add-on_repository
  category = Omega_add-on_repository
  category = Album_scraper_add-ons
  category = Album_scraper_add-ons
notcategory = Broken add-ons
</dpl>
</dpl>
|}
|}
Line 45: Line 104:
|
|
{| class="mw-collapsible mw-collapsed wikitable" style="margin: 0.0em 0 0.0em 0em; {{border-radius|5px}}; {{box-shadow|0px|1px|3px|#878787}};"
{| class="mw-collapsible mw-collapsed wikitable" style="margin: 0.0em 0 0.0em 0em; {{border-radius|5px}}; {{box-shadow|0px|1px|3px|#878787}};"
! scope="row" style="width:500px; background-color:#dfedf0; text-align:left;"| List of {{red|Artist}} scrapers compatible with v18 Leia
! scope="row" style="width:500px; background-color:#dfedf0; text-align:left;"| List of {{red|Artist}} scrapers
|-
|-
|
|
<dpl>
<dpl>
  category = Leia_add-on_repository
  category = Matrix_add-on_repository
  category = Artist_scraper_add-ons
  category = Artist_scraper_add-ons
notcategory = Broken add-ons
</dpl>
</dpl>
|}
|}
|
|
{| class="mw-collapsible mw-collapsed wikitable" style="margin: 0.0em 0 0.0em 0em; {{border-radius|5px}}; {{box-shadow|0px|1px|3px|#878787}};"
{| class="mw-collapsible mw-collapsed wikitable" style="margin: 0.0em 0 0.0em 0em; {{border-radius|5px}}; {{box-shadow|0px|1px|3px|#878787}};"
! scope="row" style="width:500px; background-color:#dfedf0; text-align:left;"| List of {{red|Artist}} scrapers compatible with v19 Matrix
! scope="row" style="width:500px; background-color:#dfedf0; text-align:left;"| List of {{red|Artist}} scrapers
|-
|-
|
|
<dpl>
<dpl>
  category = Matrix_add-on_repository
  category = Nexus_add-on_repository
  category = Artist_scraper_add-ons
  category = Artist_scraper_add-ons
notcategory = Broken add-ons
</dpl>
</dpl>
|}
|}
|
|
{| class="mw-collapsible mw-collapsed wikitable" style="margin: 0.0em 0 0.0em 0em; {{border-radius|5px}}; {{box-shadow|0px|1px|3px|#878787}};"
{| class="mw-collapsible mw-collapsed wikitable" style="margin: 0.0em 0 0.0em 0em; {{border-radius|5px}}; {{box-shadow|0px|1px|3px|#878787}};"
! scope="row" style="width:500px; background-color:#dfedf0; text-align:left;"| List of {{red|Artist}} scrapers compatible with v20 Nexus
! scope="row" style="width:500px; background-color:#dfedf0; text-align:left;"| List of {{red|Artist}} scrapers
|-
|-
|
|
<dpl>
<dpl>
  category = Nexus_add-on_repository
  category = Omega_add-on_repository
  category = Artist_scraper_add-ons
  category = Artist_scraper_add-ons
notcategory = Broken add-ons
</dpl>
</dpl>
|}
|}
Line 76: Line 138:
|
|
{| class="mw-collapsible mw-collapsed wikitable" style="margin: 0.0em 0 0.0em 0em; {{border-radius|5px}}; {{box-shadow|0px|1px|3px|#878787}};"
{| class="mw-collapsible mw-collapsed wikitable" style="margin: 0.0em 0 0.0em 0em; {{border-radius|5px}}; {{box-shadow|0px|1px|3px|#878787}};"
! scope="row" style="width:500px; background-color:#dfedf0; text-align:left;"| List of {{red|Music Video}} scrapers compatible with v18 Leia
! scope="row" style="width:500px; background-color:#dfedf0; text-align:left;"| List of {{red|Music Video}} scrapers
|-
|-
|
|
<dpl>
<dpl>
  category = Leia_add-on_repository
  category = Matrix_add-on_repository
  category = Music_video_scraper_add-ons
  category = Music_video_scraper_add-ons
notcategory = Broken add-ons
</dpl>
</dpl>
|}
|}
|
|
{| class="mw-collapsible mw-collapsed wikitable" style="margin: 0.0em 0 0.0em 0em; {{border-radius|5px}}; {{box-shadow|0px|1px|3px|#878787}};"
{| class="mw-collapsible mw-collapsed wikitable" style="margin: 0.0em 0 0.0em 0em; {{border-radius|5px}}; {{box-shadow|0px|1px|3px|#878787}};"
! scope="row" style="width:500px; background-color:#dfedf0; text-align:left;"| List of {{red|Music Video}} scrapers compatible with v19 Matrix
! scope="row" style="width:500px; background-color:#dfedf0; text-align:left;"| List of {{red|Music Video}} scrapers
|-
|-
|
|
<dpl>
<dpl>
  category = Matrix_add-on_repository
  category = Nexus_add-on_repository
  category = Music_video_scraper_add-ons
  category = Music_video_scraper_add-ons
notcategory = Broken add-ons
</dpl>
</dpl>
|}
|}
|
|
{| class="mw-collapsible mw-collapsed wikitable" style="margin: 0.0em 0 0.0em 0em; {{border-radius|5px}}; {{box-shadow|0px|1px|3px|#878787}};"
{| class="mw-collapsible mw-collapsed wikitable" style="margin: 0.0em 0 0.0em 0em; {{border-radius|5px}}; {{box-shadow|0px|1px|3px|#878787}};"
! scope="row" style="width:500px; background-color:#dfedf0; text-align:left;"| List of {{red|Music Video}} scrapers compatible with v20 Nexus
! scope="row" style="width:500px; background-color:#dfedf0; text-align:left;"| List of {{red|Music Video}} scrapers
|-
|-
|
|
<dpl>
<dpl>
  category = Nexus_add-on_repository
  category = Omega_add-on_repository
  category = Music_video_scraper_add-ons
  category = Music_video_scraper_add-ons
notcategory = Broken add-ons
</dpl>
</dpl>
|}
|}
Line 107: Line 172:
|
|
{| class="mw-collapsible mw-collapsed wikitable" style="margin: 0.0em 0 0.0em 0em; {{border-radius|5px}}; {{box-shadow|0px|1px|3px|#878787}};"
{| class="mw-collapsible mw-collapsed wikitable" style="margin: 0.0em 0 0.0em 0em; {{border-radius|5px}}; {{box-shadow|0px|1px|3px|#878787}};"
! scope="row" style="width:500px; background-color:#dfedf0; text-align:left;"| List of {{red|Movie}} scrapers compatible with v18 Leia
! scope="row" style="width:500px; background-color:#dfedf0; text-align:left;"| List of {{red|Movie}} scrapers
|-
|-
|
|
<dpl>
<dpl>
  category = Leia_add-on_repository
  category = Matrix_add-on_repository
  category = Movie_scraper_add-ons
  category = Movie_scraper_add-ons
notcategory = Broken add-ons
</dpl>
</dpl>
|}
|}
|
|
{| class="mw-collapsible mw-collapsed wikitable" style="margin: 0.0em 0 0.0em 0em; {{border-radius|5px}}; {{box-shadow|0px|1px|3px|#878787}};"
{| class="mw-collapsible mw-collapsed wikitable" style="margin: 0.0em 0 0.0em 0em; {{border-radius|5px}}; {{box-shadow|0px|1px|3px|#878787}};"
! scope="row" style="width:500px; background-color:#dfedf0; text-align:left;"| List of {{red|Movie}} scrapers compatible with v19 Matrix
! scope="row" style="width:500px; background-color:#dfedf0; text-align:left;"| List of {{red|Movie}} scrapers
|-
|-
|
|
<dpl>
<dpl>
  category = Matrix_add-on_repository
  category = Nexus_add-on_repository
  category = Movie_scraper_add-ons
  category = Movie_scraper_add-ons
notcategory = Broken add-ons
</dpl>
</dpl>
|}
|}
|
|
{| class="mw-collapsible mw-collapsed wikitable" style="margin: 0.0em 0 0.0em 0em; {{border-radius|5px}}; {{box-shadow|0px|1px|3px|#878787}};"
{| class="mw-collapsible mw-collapsed wikitable" style="margin: 0.0em 0 0.0em 0em; {{border-radius|5px}}; {{box-shadow|0px|1px|3px|#878787}};"
! scope="row" style="width:500px; background-color:#dfedf0; text-align:left;"| List of {{red|Movie}} scrapers compatible with v20 Nexus
! scope="row" style="width:500px; background-color:#dfedf0; text-align:left;"| List of {{red|Movie}} scrapers
|-
|-
|
|
<dpl>
<dpl>
  category = Nexus_add-on_repository
  category = Omega_add-on_repository
  category = Movie_scraper_add-ons
  category = Movie_scraper_add-ons
notcategory = Broken add-ons
</dpl>
</dpl>
|}
|}
Line 138: Line 206:
|
|
{| class="mw-collapsible mw-collapsed wikitable" style="margin: 0.0em 0 0.0em 0em; {{border-radius|5px}}; {{box-shadow|0px|1px|3px|#878787}};"
{| class="mw-collapsible mw-collapsed wikitable" style="margin: 0.0em 0 0.0em 0em; {{border-radius|5px}}; {{box-shadow|0px|1px|3px|#878787}};"
! scope="row" style="width:500px; background-color:#dfedf0; text-align:left;"| List of {{red|TV Show}} scrapers compatible with v18 Leia
! scope="row" style="width:500px; background-color:#dfedf0; text-align:left;"| List of {{red|TV Show}} scrapers
|-
|-
|
|
<dpl>
<dpl>
  category = Leia_add-on_repository
  category = Matrix_add-on_repository
  category = TV_scraper_add-ons
  category = TV_scraper_add-ons
notcategory = Broken add-ons
</dpl>
</dpl>
|}
|}
|
|
{| class="mw-collapsible mw-collapsed wikitable" style="margin: 0.0em 0 0.0em 0em; {{border-radius|5px}}; {{box-shadow|0px|1px|3px|#878787}};"
{| class="mw-collapsible mw-collapsed wikitable" style="margin: 0.0em 0 0.0em 0em; {{border-radius|5px}}; {{box-shadow|0px|1px|3px|#878787}};"
! scope="row" style="width:500px; background-color:#dfedf0; text-align:left;"| List of {{red|TV Show}} scrapers compatible with v19 Matrix
! scope="row" style="width:500px; background-color:#dfedf0; text-align:left;"| List of {{red|TV Show}} scrapers
|-
|-
|
|
<dpl>
<dpl>
  category = Matrix_add-on_repository
  category = Nexus_add-on_repository
  category = TV_scraper_add-ons
  category = TV_scraper_add-ons
notcategory = Broken add-ons
</dpl>
</dpl>
|}
|}
|
|
{| class="mw-collapsible mw-collapsed wikitable" style="margin: 0.0em 0 0.0em 0em; {{border-radius|5px}}; {{box-shadow|0px|1px|3px|#878787}};"
{| class="mw-collapsible mw-collapsed wikitable" style="margin: 0.0em 0 0.0em 0em; {{border-radius|5px}}; {{box-shadow|0px|1px|3px|#878787}};"
! scope="row" style="width:500px; background-color:#dfedf0; text-align:left;"| List of {{red|TV Show}} scrapers compatible with v20 Nexus
! scope="row" style="width:500px; background-color:#dfedf0; text-align:left;"| List of {{red|TV Show}} scrapers
|-
|-
|
|
<dpl>
<dpl>
  category = Nexus_add-on_repository
  category = Omega_add-on_repository
  category = TV_scraper_add-ons
  category = TV_scraper_add-ons
notcategory = Broken add-ons
</dpl>
</dpl>
|}
|}
|}
|}
{| style="width: 100%;"
|-
| style="width: 5%; vertical-align:top;" | [[File:language_globe_icon.png|50px|left]]
| style="width: 44%; vertical-align:top;" |'''{{resize|1.3em| [[HOW-TO:Create_Video_Library|Creating your Video Library]]}}'''
Setting up your video library for the first time? Follow this How To guide
| style="width: 2%;" |
| style="width: 5%; vertical-align:top;" | [[File:language_globe_icon.png|50px|left]]
| style="width: 44%; vertical-align:top;" | '''{{resize|1.3em| [[HOW-TO:Create_Music_Library|Creating your Music Library]]}}'''
Setting up your music library for the first time? Follow this How To guide
|-
| style="width: 5%; vertical-align:top;" | [[File:language_globe_icon.png|50px|left]]
| style="width: 44%; vertical-align:top;" | '''{{resize|1.3em| [[Incorrect_and_missing_videos|Video scraping problems]]}}'''
Something not working as expected? Check this page for common solutions.
| style="width: 2%;" |
| style="width: 5%; vertical-align:top;" |
| style="width: 44%; vertical-align:top;" |
|-
| style="width: 5%; vertical-align:top;" | [[File:language_globe_icon.png|50px|left]]
| style="width: 44%; vertical-align:top;" | '''{{resize|1.3em| [[Changing Scrapers]]}}'''
| style="width: 2%;" |
| style="width: 5%; vertical-align:top;" | [[File:language_globe_icon.png|50px|left]]
| style="width: 44%; vertical-align:top;" | '''{{resize|1.3em| [[:Category:Scraper|List of all Scraper pages]]}}'''
|}
= Scraper Development =
Historically, {{kodi}} scrapers were XML based. While XML scrapers have been used successfully for well over a decade, their use has been deprecated, since the release of {{kodi}} v19 Matrix, in favour of Python based scrapers.
XML scrapers have become difficult to develop and maintain as they are no longer suited to the advanced REST API's used by the information providers, which results in frequent maintenance and updates to cope with API changes.
The following pages are available to help with the development process of scrapers. Although XML scrapers are deprecated, and no future XML scrapers will be accepted into the official repository, the two pages dedicated to XML scrapers contain useful information.
{| style="width: 100%;"
|-
| style="width: 5%; vertical-align:top;" | [[File:Crystal Project Network Connection Manager.png|50px|left]]
| style="width: 44%; vertical-align:top;" |'''{{resize|1.3em| [[Python_movie_scraper_development|Developing a Movie scraper in Python]]}}'''
| style="width: 2%;" |
| style="width: 5%; vertical-align:top;" | [[File:Crystal Project Network Connection Manager.png|50px|left]]
| style="width: 44%; vertical-align:top;" | '''{{resize|1.3em| [[Scraper development XML|XML Scraper Development]]}}'''
XML scrapers are deprecated. This page may still contain useful development information.
|-
| style="width: 5%; vertical-align:top;" | [[File:Crystal Project Network Connection Manager.png|50px|left]]
| style="width: 44%; vertical-align:top;" |'''{{resize|1.3em| [[Python_tv_scraper_development|Developing a TV Show scraper in Python]]}}'''
| style="width: 2%;" |
| style="width: 5%; vertical-align:top;" | [[File:Crystal Project Network Connection Manager.png|50px|left]]
| style="width: 44%; vertical-align:top;" | '''{{resize|1.3em| [[HOW-TO:Write_media_scrapers|Writing an XML scraper]]}}'''
XML scrapers are deprecated. This page may still contain useful development information.
|}






{{updated|19}}
{{updated|20}}


[[Category:Karellen]]
[[Category:Karellen]]

Latest revision as of 01:33, 24 January 2023

Home icon grey.png   ▶ Development
▶ Music library
▶ Video library		 ▶ Scrapers


Using Scrapers

Scrapers are used to contact online information providers to download metadata for video and music items that are to be added to the library.

A number of scrapers are installed by default when installing Kodi. Other scrapers are available for installation from the official repository. The tables at the bottom of this page list all official scrapers and their compatability.


Language globe icon.png
 Creating your Video Library

Setting up your video library for the first time? Follow this How To guide

Language globe icon.png
 Creating your Music Library

Setting up your music library for the first time? Follow this How To guide

Language globe icon.png
 Video scraping problems

Something not working as expected? Check this page for common solutions.

Language globe icon.png
 Changing Scrapers
Language globe icon.png
 List of all Scraper pages


Scraper Development

Historically, Kodi scrapers were XML based. While XML scrapers have been used successfully for well over a decade, their use has been deprecated, since the release of Kodi v19 Matrix, in favour of Python based scrapers.

XML scrapers have become difficult to develop and maintain as they are no longer suited to the advanced REST API's used by the information providers, which results in frequent maintenance and updates to cope with API changes.

The following pages are available to help with the development process of scrapers. Although XML scrapers are deprecated, and no future XML scrapers will be accepted into the official repository, the two pages dedicated to XML scrapers contain useful information.


Crystal Project Network Connection Manager.png
 Developing a Movie scraper in Python
Crystal Project Network Connection Manager.png
 XML Scraper Development

XML scrapers are deprecated. This page may still contain useful development information.

Crystal Project Network Connection Manager.png
 Developing a TV Show scraper in Python
Crystal Project Network Connection Manager.png
 Writing an XML scraper

XML scrapers are deprecated. This page may still contain useful development information.


Official Scrapers

Scrapers compatible with v19 Matrix Scrapers compatible with v20 Nexus Scrapers compatible with v21 Omega
List of Album scrapers
List of Album scrapers
List of Album scrapers
List of Artist scrapers
List of Artist scrapers
List of Artist scrapers
List of Music Video scrapers
List of Music Video scrapers
List of Music Video scrapers
List of Movie scrapers
List of Movie scrapers
List of Movie scrapers
List of TV Show scrapers
List of TV Show scrapers
List of TV Show scrapers


Retrieved from "https://kodi.wiki/index.php?title=Scrapers&oldid=243957"