Scrapers: Difference between revisions
m (Remove links) |
(Convert to Portal) |
||
Line 1: | Line 1: | ||
{{mininav|[[Development]]|[[ | {{mininav|[[Development]] {{l2| [[Music library]] }} {{l2|[[Video library]]}} }} | ||
{{ | |||
= Using Scrapers = | |||
Scrapers are used to contact online information providers to download metadata for video and music items to be added to the library. | |||
A number of scrapers are installed by default when installing {{kodi}}. Other scrapers are available for installation from the official repository. The below table lists all official scrapers and their compatability. | |||
== | {| | ||
! style="background-color:salmon;" | Scrapers compatible with v18 Leia | |||
! style="background-color:#a0e75a;" | Scrapers compatible with v19 Matrix | |||
! style="background-color:skyBlue;" | Scrapers compatible with v20 Nexus | |||
|- | |||
| | |||
{| class="mw-collapsible mw-collapsed wikitable" style="margin: 0.0em 0 0.0em 0em; {{border-radius|5px}}; {{box-shadow|0px|1px|3px|#878787}};" | |||
! scope="row" style="width:500px; background-color:#dfedf0; text-align:left;"| List of {{red|Album}} scrapers compatible with v18 Leia | |||
|- | |||
| | |||
<dpl> | |||
category = Leia_add-on_repository | |||
category = Album_scraper_add-ons | |||
</dpl> | |||
|} | |||
| | |||
{| class="mw-collapsible mw-collapsed wikitable" style="margin: 0.0em 0 0.0em 0em; {{border-radius|5px}}; {{box-shadow|0px|1px|3px|#878787}};" | |||
! scope="row" style="width:500px; background-color:#dfedf0; text-align:left;"| List of {{red|Album}} scrapers compatible with v19 Matrix | |||
|- | |||
| | |||
<dpl> | |||
category = Matrix_add-on_repository | |||
category = Album_scraper_add-ons | |||
</dpl> | |||
|} | |||
| | |||
{| class="mw-collapsible mw-collapsed wikitable" style="margin: 0.0em 0 0.0em 0em; {{border-radius|5px}}; {{box-shadow|0px|1px|3px|#878787}};" | |||
! scope="row" style="width:500px; background-color:#dfedf0; text-align:left;"| List of {{red|Album}} scrapers compatible with v20 Nexus | |||
|- | |||
| | |||
<dpl> | |||
category = Nexus_add-on_repository | |||
category = Album_scraper_add-ons | |||
</dpl> | |||
|} | |||
|- | |||
| | |||
{| class="mw-collapsible mw-collapsed wikitable" style="margin: 0.0em 0 0.0em 0em; {{border-radius|5px}}; {{box-shadow|0px|1px|3px|#878787}};" | |||
! scope="row" style="width:500px; background-color:#dfedf0; text-align:left;"| List of {{red|Artist}} scrapers compatible with v18 Leia | |||
|- | |||
| | |||
<dpl> | |||
category = Leia_add-on_repository | |||
category = Artist_scraper_add-ons | |||
</dpl> | |||
|} | |||
| | |||
{| class="mw-collapsible mw-collapsed wikitable" style="margin: 0.0em 0 0.0em 0em; {{border-radius|5px}}; {{box-shadow|0px|1px|3px|#878787}};" | |||
! scope="row" style="width:500px; background-color:#dfedf0; text-align:left;"| List of {{red|Artist}} scrapers compatible with v19 Matrix | |||
|- | |||
| | |||
<dpl> | |||
category = Matrix_add-on_repository | |||
category = Artist_scraper_add-ons | |||
</dpl> | |||
|} | |||
| | |||
{| class="mw-collapsible mw-collapsed wikitable" style="margin: 0.0em 0 0.0em 0em; {{border-radius|5px}}; {{box-shadow|0px|1px|3px|#878787}};" | |||
! scope="row" style="width:500px; background-color:#dfedf0; text-align:left;"| List of {{red|Artist}} scrapers compatible with v20 Nexus | |||
|- | |||
| | |||
<dpl> | |||
category = Nexus_add-on_repository | |||
category = Artist_scraper_add-ons | |||
</dpl> | |||
|} | |||
|- | |||
| | |||
{| class="mw-collapsible mw-collapsed wikitable" style="margin: 0.0em 0 0.0em 0em; {{border-radius|5px}}; {{box-shadow|0px|1px|3px|#878787}};" | |||
! scope="row" style="width:500px; background-color:#dfedf0; text-align:left;"| List of {{red|Music Video}} scrapers compatible with v18 Leia | |||
|- | |||
| | |||
<dpl> | |||
category = Leia_add-on_repository | |||
category = Music_video_scraper_add-ons | |||
</dpl> | |||
|} | |||
| | |||
{| class="mw-collapsible mw-collapsed wikitable" style="margin: 0.0em 0 0.0em 0em; {{border-radius|5px}}; {{box-shadow|0px|1px|3px|#878787}};" | |||
! scope="row" style="width:500px; background-color:#dfedf0; text-align:left;"| List of {{red|Music Video}} scrapers compatible with v19 Matrix | |||
|- | |||
| | |||
<dpl> | |||
category = Matrix_add-on_repository | |||
category = Music_video_scraper_add-ons | |||
</dpl> | |||
|} | |||
| | |||
{| class="mw-collapsible mw-collapsed wikitable" style="margin: 0.0em 0 0.0em 0em; {{border-radius|5px}}; {{box-shadow|0px|1px|3px|#878787}};" | |||
! scope="row" style="width:500px; background-color:#dfedf0; text-align:left;"| List of {{red|Music Video}} scrapers compatible with v20 Nexus | |||
|- | |||
| | |||
<dpl> | |||
category = Nexus_add-on_repository | |||
category = Music_video_scraper_add-ons | |||
</dpl> | |||
|} | |||
|- | |||
| | |||
{| class="mw-collapsible mw-collapsed wikitable" style="margin: 0.0em 0 0.0em 0em; {{border-radius|5px}}; {{box-shadow|0px|1px|3px|#878787}};" | |||
! scope="row" style="width:500px; background-color:#dfedf0; text-align:left;"| List of {{red|Movie}} scrapers compatible with v18 Leia | |||
|- | |||
| | |||
<dpl> | |||
category = Leia_add-on_repository | |||
category = Movie_scraper_add-ons | |||
</dpl> | |||
|} | |||
| | |||
{| class="mw-collapsible mw-collapsed wikitable" style="margin: 0.0em 0 0.0em 0em; {{border-radius|5px}}; {{box-shadow|0px|1px|3px|#878787}};" | |||
! scope="row" style="width:500px; background-color:#dfedf0; text-align:left;"| List of {{red|Movie}} scrapers compatible with v19 Matrix | |||
|- | |||
| | |||
<dpl> | |||
category = Matrix_add-on_repository | |||
category = Movie_scraper_add-ons | |||
</dpl> | |||
|} | |||
| | |||
{| class="mw-collapsible mw-collapsed wikitable" style="margin: 0.0em 0 0.0em 0em; {{border-radius|5px}}; {{box-shadow|0px|1px|3px|#878787}};" | |||
! scope="row" style="width:500px; background-color:#dfedf0; text-align:left;"| List of {{red|Movie}} scrapers compatible with v20 Nexus | |||
|- | |||
| | |||
<dpl> | |||
category = Nexus_add-on_repository | |||
category = Movie_scraper_add-ons | |||
</dpl> | |||
|} | |||
|- | |||
| | |||
{| class="mw-collapsible mw-collapsed wikitable" style="margin: 0.0em 0 0.0em 0em; {{border-radius|5px}}; {{box-shadow|0px|1px|3px|#878787}};" | |||
! scope="row" style="width:500px; background-color:#dfedf0; text-align:left;"| List of {{red|TV Show}} scrapers compatible with v18 Leia | |||
|- | |||
| | |||
<dpl> | |||
category = Leia_add-on_repository | |||
category = TV_scraper_add-ons | |||
</dpl> | |||
|} | |||
| | |||
{| class="mw-collapsible mw-collapsed wikitable" style="margin: 0.0em 0 0.0em 0em; {{border-radius|5px}}; {{box-shadow|0px|1px|3px|#878787}};" | |||
! scope="row" style="width:500px; background-color:#dfedf0; text-align:left;"| List of {{red|TV Show}} scrapers compatible with v19 Matrix | |||
|- | |||
| | |||
<dpl> | |||
category = Matrix_add-on_repository | |||
category = TV_scraper_add-ons | |||
</dpl> | |||
|} | |||
| | |||
{| class="mw-collapsible mw-collapsed wikitable" style="margin: 0.0em 0 0.0em 0em; {{border-radius|5px}}; {{box-shadow|0px|1px|3px|#878787}};" | |||
! scope="row" style="width:500px; background-color:#dfedf0; text-align:left;"| List of {{red|TV Show}} scrapers compatible with v20 Nexus | |||
|- | |||
| | |||
<dpl> | |||
category = Nexus_add-on_repository | |||
category = TV_scraper_add-ons | |||
</dpl> | |||
|} | |||
|} | |||
{| style="width: 100%;" | |||
|- | |||
| style="width: 5%; vertical-align:top;" | [[File:language_globe_icon.png|50px|left]] | |||
| style="width: 44%; vertical-align:top;" |'''{{resize|1.3em| [[HOW-TO:Create_Video_Library|Creating your Video Library]]}}''' | |||
Setting up your video library for the first time? Follow this How To guide | |||
== | | style="width: 2%;" | | ||
[[File: | | style="width: 5%; vertical-align:top;" | [[File:language_globe_icon.png|50px|left]] | ||
| style="width: 44%; vertical-align:top;" | '''{{resize|1.3em| [[HOW-TO:Create_Music_Library|Creating your Music Library]]}}''' | |||
Setting up your music library for the first time? Follow this How To guide | |||
|- | |||
| style="width: 5%; vertical-align:top;" | [[File:language_globe_icon.png|50px|left]] | |||
| style="width: 44%; vertical-align:top;" | '''{{resize|1.3em| [[Incorrect_and_missing_videos|Video scraping problems]]}}''' | |||
Something not working as expected? Check this page for common solutions. | |||
| style="width: 2%;" | | |||
| style="width: 5%; vertical-align:top;" | | |||
| style="width: 44%; vertical-align:top;" | | |||
|- | |||
| style="width: 5%; vertical-align:top;" | [[File:language_globe_icon.png|50px|left]] | |||
| style="width: 44%; vertical-align:top;" | '''{{resize|1.3em| [[Changing Scrapers]]}}''' | |||
== | | style="width: 2%;" | | ||
| style="width: 5%; vertical-align:top;" | [[File:language_globe_icon.png|50px|left]] | |||
| style="width: 44%; vertical-align:top;" | '''{{resize|1.3em| [[:Category:Scraper|List of all Scraper pages]]}}''' | |||
|} | |||
= Scraper Development = | |||
Historically, {{kodi}} scrapers were XML based. While XML scrapers have been used successfully for well over a decade, their use has been deprecated, since the release of {{kodi}} v19 Matrix, in favour of Python based scrapers. | |||
XML scrapers have become difficult to develop and maintain as they are no longer suited to the advanced REST API's used by the information providers, which results in frequent maintenance and updates to cope with API changes. | |||
The following pages are available to help with the development process of scrapers. Although XML scrapers are deprecated, and no future XML scrapers will be accepted into the official repository, the two pages dedicated to XML scrapers contain useful information. | |||
{| style="width: 100%;" | |||
|- | |||
| style="width: 5%; vertical-align:top;" | [[File:Crystal Project Network Connection Manager.png|50px|left]] | |||
| style="width: 44%; vertical-align:top;" |'''{{resize|1.3em| [[Python_movie_scraper_development|Developing a Movie scraper in Python]]}}''' | |||
| style="width: 2%;" | | |||
| style="width: 5%; vertical-align:top;" | [[File:Crystal Project Network Connection Manager.png|50px|left]] | |||
| style="width: 44%; vertical-align:top;" | '''{{resize|1.3em| [[Scraper development XML|XML Scraper Development]]}}''' | |||
XML scrapers are deprecated. This page may still contain useful development information. | |||
|- | |||
| style="width: 5%; vertical-align:top;" | [[File:Crystal Project Network Connection Manager.png|50px|left]] | |||
| style="width: 44%; vertical-align:top;" |'''{{resize|1.3em| [[Python_tv_scraper_development|Developing a TV Show scraper in Python]]}}''' | |||
| style="width: 2%;" | | |||
| style="width: 5%; vertical-align:top;" | [[File:Crystal Project Network Connection Manager.png|50px|left]] | |||
| style="width: 44%; vertical-align:top;" | '''{{resize|1.3em| [[HOW-TO:Write_media_scrapers|Writing an XML scraper]]}}''' | |||
XML scrapers are deprecated. This page may still contain useful development information. | |||
|} | |||
{{updated|19}} | |||
[[Category:Karellen]] | |||
[[Category: | |||
[[Category:Scraper]] | [[Category:Scraper]] | ||
[[Category:Music library]] | |||
[[Category:Video library]] | [[Category:Video library]] | ||
[[Category: | [[Category:Disambiguation_pages]] |
Revision as of 04:44, 29 May 2022
Development Music library Video library |
Scrapers |
Using Scrapers
Scrapers are used to contact online information providers to download metadata for video and music items to be added to the library.
A number of scrapers are installed by default when installing Kodi. Other scrapers are available for installation from the official repository. The below table lists all official scrapers and their compatability.
Scrapers compatible with v18 Leia | Scrapers compatible with v19 Matrix | Scrapers compatible with v20 Nexus | ||||||
---|---|---|---|---|---|---|---|---|
|
|
| ||||||
|
|
| ||||||
|
|
| ||||||
|
|
| ||||||
|
|
|
Creating your Video Library
Setting up your video library for the first time? Follow this How To guide |
Creating your Music Library
Setting up your music library for the first time? Follow this How To guide | |||
Video scraping problems
Something not working as expected? Check this page for common solutions. |
||||
Changing Scrapers | List of all Scraper pages |
Scraper Development
Historically, Kodi scrapers were XML based. While XML scrapers have been used successfully for well over a decade, their use has been deprecated, since the release of Kodi v19 Matrix, in favour of Python based scrapers.
XML scrapers have become difficult to develop and maintain as they are no longer suited to the advanced REST API's used by the information providers, which results in frequent maintenance and updates to cope with API changes.
The following pages are available to help with the development process of scrapers. Although XML scrapers are deprecated, and no future XML scrapers will be accepted into the official repository, the two pages dedicated to XML scrapers contain useful information.
Developing a Movie scraper in Python | XML Scraper Development
XML scrapers are deprecated. This page may still contain useful development information. | |||
Developing a TV Show scraper in Python | Writing an XML scraper
XML scrapers are deprecated. This page may still contain useful development information. |